Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback
Authors: Yifu Yuan
Jianye HAO
Yi Ma
Zibin Dong
Hebin Liang
Jinyi Liu
Zhixin Feng
Kai Zhao
YAN ZHENG
Venue: ICLR 2024 poster (2024)
Keywords: RLHF
Diverse Human Feedback
Reinforcement Learning
Primary area: reinforcement learning
TL;DR: we introduce Uni-RLHF, a universal and comprehensive benchmark suite tailored for RLHF.

Abstract: Reinforcement Learning with Human Feedback (RLHF) has received significant attention for performing tasks without the need for costly manual reward design by aligning human preferences. It is crucial to consider diverse human feedback types and various learning methods in different environments. However, quantifying progress in RLHF with diverse feedback is challenging due to the lack of standardized annotation platforms and widely used unified benchmarks. To bridge this gap, we introduce **Uni-RLHF**, a comprehensive system implementation tailored for RLHF. It aims to provide a complete workflow from *real human feedback*, fostering progress in the development of practical problems. Uni-RLHF contains three packages: 1) a universal multi-feedback annotation platform, 2) large-scale crowdsourced feedback datasets, and 3) modular offline RLHF baseline implementations. Uni-RLHF develops a user-friendly annotation interface tailored to various feedback types, compatible with a wide range of mainstream RL environments. We then establish a systematic pipeline of crowdsourced annotations, resulting in large-scale annotated datasets comprising more than 15 million steps across 30 popular tasks. Through extensive experiments, the results in the collected datasets demonstrate competitive performance compared to those from well-designed manual rewards. We evaluate various design choices and offer insights into their strengths and potential areas of improvement. We wish to build valuable open-source platforms, datasets, and baselines to facilitate the development of more robust and reliable RLHF solutions based on realistic human feedback. The website is available at https://uni-rlhf.github.io/.

Published as a conference paper at ICLR 2024
UNI-RLHF: UNIVERSAL PLATFORM AND BENCH- MARK SUITE FOR REINFORCEMENT LEARNING WITH DIVERSE HUMAN FEEDBACK
Yifu Yuan', Jianye Hao Yi Ma', Zibin Dong', Hebin Liang', Jinyi Liu', Zhixin Feng' Kai Zhao', Yan Zheng'
{yuanyf, jianye. hao, mayi, zibindong, lianghebin, jyliu, jingzhel353 kaizhao janzheng)etiju eduu.co cn CCollege of Intelligence and Computing, Tiajjin University
ABSTRACT
Reinforcement Learning with Human Feedback (RLHF) has received significant attention for performing tasks without the need for costly manual reward design by aligning human preferences. is crucial consider diverse human feedback types and various learning methods in different environments. However, quanti- fying progress in RLHF with diverse feedback is challenging due the lack of standardized annotation platforms and widely used unified benchmarks. To bridge this gap, we introduce Uni-RLHF, comprehensive system implementation tai- lored for RLHF aims to provide complete workflow from real human feed- back, fostering progress in the development of practical problems. Uni-RLHF contains three packages: a universal multi-feedback annotation platform. 2) large-scale crowdsourced feedback datasets, and 3) modular offline RLHF base- line implementations. Uni-RLHF develops user-friendly annotation interface tailored to various feedback types, compatible with a wide range of mainstream RL environments. We then establish systematic pipeline of crowdsourced anno- tations, resulting in large-scale annotated datasets comprising more than 15 mil- lion steps across 30 popular tasks. Through extensive experiments, the results in the collected datasets demonstrate competitive performance compared to those from well-designed manual rewards. We evaluate various design choices and of- fer insights into their strengths and potential areas of improvement. We wish build valuable open- spuncsource platforms, datasets, and baselines to facilitate the de- velopment of more robust and reliable RLHF solutions based realistic human feedback. The website is available at https / /uni- rlhf, github. io/.
INTRODUCTION
Reinforcement learning (RL) in decision making has demonstrated promising capabilities in many practical scenarios (Feng et al., 2023; Kaufmann et al., 2023; Ni et al., 2023). However, designing dense and comprehensive reward signals for RL requires significant human efforts and extensive domain knowledge (Zhu et al., 2020; Hadfield-Menell et al., 2017; Hu et al., 2020). Moreover, agents can easily exploit designed rewards to achieve high returns in unexpected ways, leading to reward hacking phenomena (Skalse et al., 2022). RLHF (Ibarz et al., 2018; Christiano et al., 2017; Knox & Stone, 2009) serves as a powerful paradigm that transforms human feedback into guidance signals. It eliminates the need for manual reward function design, playing an important role in scenarios such as robotics control (Hiranaka al., 2023) and language model training (Ouyang al., 2022). Beyond the reward modeling, the integration of diverse human feedback into reinforcement learning (Ghosal et al., 2023; Guan et al., 2021a) has become the spotlight, as it can align the agent's behavior with the complex and variable intentions humans (Lindner & El-Assady, 2022; Casper et al., 2023).
Despite the increasing significance of RLHF, the scarcity of universal tools and unified benchmarks specifically designed for RLHF hinders a comprehensive study into learning from various types
"Corresponding authors: Jianye Hao (jiany.haoQtruu.ddu.

---

Published as a conference paper at ICLR 2024
of feedback. Several RLHF benchmarks (Lee al., 2021b; Shin et al., 2023; Soniels-Koch & Freedman, 2022) for comparative feedback have been proposed, but they all utilize predefined hand- engineered reward functions from the environment under conditions that simulate absolute expert and infinite synthetic feedback. However, in reality, scenarios akin to unbiased expert feedback are scarce. There is a growing use of crowdsourced annotations (Ouyang et al., 2022; Touvron et al., 2023) a substitute cope with the increas- ing scale of feedback. A significant issue is general unreliability and confusion among large-scale crowdsourced annotators, leading cognitive inconsistency and even adversarial attacks (Lee et al., 10; BryIk et al., 2019; Ghosal et al., 2023). Therefore, studying the RLHF algorithm design with human teachers instead of script teachers is more compatible with the practical problem. Moreover, the absence of reliable annotation tools that can support diverse types of feedback across environ- ments, as well as efficient pipelines for crowdsourcing, compounds the challenges of conducting research on human feedback.
scripted teachers, conducting experiments
To bridge the gap, we introduce Uni-RLHF, a comprehensive system implementation for RLHF, comprising of three packages: 1) universal annotation platform supporting diverse human feed- back for decision making tasks, 2) large-scale and reusable crowdsourced feedback datasets, and 3) modular offline RL baselines from human feedback. Uni-RLHF offers streamlined interface designed to facilitate the study of learning across variety of feedback types. Uni-RLHF offers a consistent API for different types of feedback. It uses a query sampler to load and visualize a set of available segments from the datasets. The annotators then provide appropriate feedback labels, which are transformed into a standardized format. Uni-RLHF is compatible with mainstream RI. environments and demonstrates strong scalability.
Crowdsourced label collection often challenging and costly. When gathering human feedback la- bels instead of script labels, various factors such as different experimental setups, and the number of annotators can introduce significant variability in final performance. Therefore, standardized data collection process and reliable feedback datasets can greatly alleviate the burden on researchers. We have established a systematic pipeline including crowdsourced annotation and data filters, providing 30 feedback datasets (=15M steps) of varying difficulty levels that can be reused.
Furthermore, we provide Offline RLHF baselines that integrate mainstream reinforcement learning algorithms and different feedback modeling methods. We then evaluate design choices and compare the performance of baseline algorithms on the collected datasets. We empirically found that Uni- RLHF achieves competitive performance with well-designed manual task reward functions. This offers a solid foundation and benchmark for the RLHF community.
Our contributions are as follows:
We introduce a universal RLHF annotation platform that supports diverse feedback types and corresponding standard encoding formats.
We develop crowdsourced feedback collection pipeline with data filters, which facilitates the cre- ation of reusable, large-scale feedback datasets, serving as a valuable starting point for researchers in the field.
We conduct offline RL baselines using collected feedback datasets and various design choices. Ex- perimental results show that training with human feedback achieve competitive performance compared to well-designed reward functions, and effectively align with human preferences.
RELATED WORK
Reinforcement Learning from Human Feedback (RLHF) leverages human feedback to guide the learning process of reinforcement learning agents (Akrour et al., 2011; Pilarski et al., 2011; Wilson al., 2012; Liu et al., 2024: Zhou et al., 2024), aiming to address the challenges in situations where environment information is limited and human input is required to enhance the efficiency and perfor- mance (Wirth et al., 2017; Zhang al., 2019). RLHF covers a variety of areas such preference- based RL (Christiano et al., 2017), inverse RL (Abbeel & Ng. 2004; Ng et al., 2000) and active learning (Chen et al., 2020) with various scopes of feedback types. Comparative feedback (Chris- tiano et al., 2017; Lee et al., 202 Park et al., 2022: Liang et al., 2022) facilitate a pairwise com- parison amongst objectives, culminating in a preference for trajectories, and further transform these rankings into the targets of the reward model. The TAMER (Knox & Stone, 2009; 2012; Warnell et al., 2018; Cederborg et al., 2015) and COACH (MacGlashan et al., 2017; Arumugam et al., 2019) series frameworks furnishes quantifiable evaluative feedback (binary right/wrong signals) indi-
2

---

Published as a conference paper at ICLR 2024
vidual steps. Despite the excellent generality of comparison and evaluation, the power of expression lacking (Guan et al., 2023). Recent works (Guan et al., 202 Teso & Kersting, 2019) have em- ployed visual explanation feedback encoding task-related features to further broaden the channels of duman--achinn interaction. Atrey al. (2019); Liang et al. (2023) enhances learning efficiency through the incorporation of a saliency map provided by humans. Besides, language (Sharma et al., 2022) and relative attribute (Guan et al., 2023) corrective guidance furnishes richer semantic mode of communication, thereby aligning with user intentions with greater precision.
Benchmarks and Platforms for RLHE. Though diverse types of feedback are widely proposed and applied, prominent limitation that there are lack of generalized targeted benchmarks for deploying and evaluating different types of human feedback. Freire et al. (2020) proposed DE- RAIL benchmark for preference-based learning, but it's limited simple diagnostic tasks. Lee al. (2021a) considered more complex robotic tasks and designed simulated teachers with irrationality. Shin al. (2023) evaluated offline preference-based RL on parts of D4RL (Fu al., 2020) tasks, but these studies primarily focus on synthesizing simulated feedback using scripted teachers. Kim et al. (2023) experimented with small batches of expert feedback and found high disagreement rates between synthesis and human labels, suggesting synthetic labels may be misleading. Besides, Shah et al. (2019) and Ghosal et al. (2023) suggested using real human feedback can improve the learning of human-aliin -maanee reward models. In novel approach, we use large-scale crowdsourced labeling for systematic analysis and experimentation across multiple environments, Another key limitation is the lack of universal platforms for multiple feedback types annotation. While some researchers have open-sourced their tools (Guan et al., 2021a; Kim et al., 2023), they are suitable only for individual, small-scale annotations and specific feedback types. presenting significant challenges for scaling up to a large number of environments. Biyik et al. (2022) provided labeling tools for active learning which are limited in low dimension tasks. A concurrent work, RIHF-Blender (Metz et al., 2023), implemented an interactive application for investigating human feedback, but lacks scalability for more environments and has not yet been evaluated in a user study.
3 UNIVERSAL PLATFORM FOR REINFORCEMENT LEARNING WITH DIVERSE FEEDBACK TYPES
To align RLHF methodologies with practical problems and cater to researchers' needs for systematic studies of various feedback types within a unified context, we introduce the Uni-RLHF system. We start with the universal annotation platform, which supports various types of human feedback along with a standardized encoding format for diverse human feedback. Using this platform, we have established a pipeline for crowdsourced feedback collection and filtering, amassing large-scale crowdsourced labeled datasets and setting a unified research benchmark. The complete system framework is shown in Fig. 1.
3.1 IMPLEMENTATION FOR MULTI-FEEDBACK ANNOTATION PLATFORM
Uni employs a client-server architecture, built upon the Flask back-end service and the Vue² front-end framework. It provides an interactive interface for employers and annotators, supporting multi-user annotation. Its intuitive design enables researchers to focus on the task, not the tool. As illustrated in Fig. 1 (upper left), Uni-RLHF packs up abstractions for RLHF annotation workflow, where the essentials include: interfaces supporting a wide range of online environments and offline datasets, a query sampler that determines which data to display, interface, enabling annotators view available trajectory segments and provide feedback responses and feedback translator that transforms diverse feedback labels into a standardized format.
an interactive user
Environment and Datasets Interface. The Uni-RIHF supports both online and offline training modes. During online training, Uni-RLHF samples data from the buffer periodically and utilizes its built-in render function to present the segments through the user interface. This facilitates asyn- chronous training of the RL agent and the feedback model. The complete training details are shown in Appendix D. In offline mode, we gather access to several widely recognized reinforcement learn- ing datasets. Some representative tasks from these environments thermore, Uni-RLHF allows easy customization and integration of new offline datasets by simply adding three functions Load.datssess. sample and vi sualizz
visualized in Table 1. Fur-
phttps://g thub. coub.ccom/vejj//core
3

---

Published as a conference paper at ICLR 2024
Table I: Overview of the representative offline datasets supported by Uni-RLHF
Benchmark
Domain
Aposts
Toral Tasks
Mijovo
Walkerăd, Hopper. HallChretah
9
DARI,
Adrost
Door, Pcn, Hammer
9
SMARTS (hoou al., 2020)
Automatic
Cruise, Cut-in, Lefe
V-DRRL (La al., 2022)
DMCeninel
Walker, Cheetah, Homanoid
4
Beisvert et 2023)
Online
Feedback
Platform
Datasets
Chillin Interaaction
Offline Dutasets
Visualization
Empleyer
Insraction Mode
Crowdsources
1
Query Type
Replay
Bulfer
Provide
Task Description
Feedback Type
Kelease
Query Sampler
Feedback Translator
Random
Disegreement
Ex-ont
tiA &
Schedule
Castomization
User Inserface
SQQ
Large scale Datasets
Filher
Ei-past
RM Structure Choice
Offline Baselines
(g).aa)
ag
CNN structure
Human Reward Model (RM)
Haman Ferdhack
Transformer
Linear
Causal Transfommer
(s, (s,a,'', .P)
Linear
Algorithm Choice
Preference Attention Layer
Lineer
Linear
CQL
RI Policy
Offlnne Daanssets
MLP strcture
CNN structure
Transfummer suruciure
Figure 1: Overview of the Uni-RLHF system. Uni-RLHF consists of three components including the platform, the datasets, and the offline RLHF baselines.
Query Sampler. The query sampler component determines how sample the data presented to the user interface. Choosing suitable query sampler to annotate can help improve model performance (Shin et al., 2023). We propose four configurable types of sampling methods:
Random: Random sampling is the random selection of specified number. We regard random sampling as the most common baseline, and this simple approach can achieve competitive perfor- mance active learning sampling in numerous environments (Lee et al., 2021b).
Disagreement: Disagreement sampling is primarily applicable for comparative feedback. We cal- culate the variance of the queries using the probability p that trajectory is superior to trajectory 2 according the feedback model which encourages the selection of informative queries exhibiting maximum divergence (Christiano et al., 2017).
Schedule: Schedule sampling prefers sampling near on-policy query from the replay buffer during online training, which adapts to the gradually improving policy performance and mitigates the issue of query-policy misalignment (Hu et al., 2023).
Customization: Customized sampling allows for task and customizable metries sampling that facilitates targeted queries.
Interactive User Interface. Uni-RLHF provides users with clean visual interface, offering an array of interactive modalities for various feedback types. The user interface consists of a seg- ment display, task description, feedback interaction, and additional information module. The feed- back interaction supports different levels of granularity, including selection, swiping, dragging, and keyframe capturing. Notably, we also support focused framing of images, which can be applied to the visual feedback, and the annotations can be saved the PASCAL VOC (Everingham et al., 2010) format. See our homepage for more visualization.
The feedback translation component receives various feedback from the user interface, which is encoded into consistent feedback labels based on the standardized feedback en- coding framework proposed in Section 3.2. Post-processed labels are then matched with queries and linked shared storage space, such an SQL. database. This configuration allows the application
Feedback Translator.
4

---

Published as a conference paper at ICLR 2024
to scale, enabling multiple annotators to carry out tasks without interference. This setup facilitates
large-scale distributed human experiments.
3.2 STANDARDIZED FEEDBACK ENCODING FORMAT FOR REINFORCEMENT LEARNING
To capture and utilize diverse and heterogeneous feedback labels from annotators, we analyze a range of research and propose a standardized feedback encoding format along with possible training methodologies. We examine the standard interaction for an RL agent with an environment discrete time. At each time step f, the agent receives state S from the environment and chooses an based on the policy a ~ (- st). Then the environment returns a reward r, (optional) and the next state S-4-1 We discuss five common feedback types, explaining how annotators interact with these types and how they can be encoded. Additionally, we briefly outline the potential forms and applications of reinforcement learning that integrate various forms human feedback. We introduce the encoding format in this section and refer to Appendix F for the corresponding training methods.
@ Comparative Feedback. Comparative feedback is the most com- monly used feedback type for reward modeling because of its simplicity and universality. Crowdsourcing tends to prefer relative comparisons over absolute evaluations. Users are presented with either two trajectory segments for comparison or multiple objectives for ranking. We define segment as sequence of time-indexed observations (S., S -- with length H. Given a pair of segments (o",o") annotators give feed- back indicating their preference. This preference, denoted as Veomp, can be expressed as 0 o' a² > 00 or o² = o.. We encode the labels Boomp E {{1, 0), (0, 1), (0.5, 0.5) and deposit them as triples (a o Neomp) in the feedback datasets D. Following the Bradley-Tere model (Bradley & Terry, 1952), comparative feedback can be utilized to train reward function Fy that aligns with human preferences. See training details in Appendix F.I.
Comsurative Pezibork
@ Attribbute Feedback. In many cases, evaluating the quality of out- comes using a single scalar value is challenging due to the multifaceted nature of numerous problems. For instance, stable robotic movement quires consideration of multiple factors, including speed, stride length, and postural pattern (Zhu et al., 2023). Similarly, autonomous driving necessitates the simultaneous consideration of speed, safety, and rule- adherence (Huang et al., 2021; 2023). Thus assessment mechanism that evaluates attributes from multiple perspectives is needed guide the agent towards multi-objective optimization. Inspired by Guan et al. (2023): Dong et al. (2023), have extended comparative feedback into attribute feedback from multi-perspective evaluation. First, we predefine a {a) ",,; .r.} represents a set of A: relative attribute about agent's s behaviors. Given pair of segments (o o"). The annotator will conduct relative evaluation of two segments for each given attribute, which is encoded as yau = (y/,... .... y/x) We then establish a learnable attribute strength mapping model that translates trajectory behaviors into latent space vectors. This approach simplifies the problem to a binary comparison and we can distill the reward model similar to comparative feedback. See training details in Appendix F22
Aftribute
Feedback
Velecte
Stop size
Human
and yi E {11, 0). (0. 1), (0.5, 0.5)
Evaluative Feedback. In this feedback mode, annotators observe segment of segment and assign a numerical score other quantifi- able judgment. Evaluative feedback can offer informative absolute eval- uations of samples and be less demanding than comparative feedback. Some studies provide feedback to the agent for a specific state, rather than the entire trajectory. potentially leading to a reward- seward-greed dilemma. Therefore, define n discrete ratings for trajectory levels (White al., 2023). As shown in the right figure, three ratings are available: good, bad, and average. Thus we receive corresponding human label (a,, Deval), where Geval {0, ..., 1} is level of assessment. See training details in Appendix F.3.
Graloative
Feeiberk
PaR Visual Feedback. Visual feedback offers a strong way for annotators to interact with agents naturally and efficiently. Annotators identify key task-relevant objects or areas within the image, guiding the agents toward critical elements for decision-making via vi-
5

---

Published as a conference paper at ICLR 2024
sual explanation. We define the feedback as (o, Dvisuat), where Vvisual Box; is a the bounding box position [Xiinn Ymin, Tmax: Mmax ). To reduce the burden on the annotators, similar to Gmelin et al. (2023) and Guan et al. (2021b), we provide an object detector that tre-highlights and tracks all objects in the environment, and the annotator only needs to deselect irrelevant We use the detector for atari provided by Delfosse et al. (2023) objects. ample. As shown in the right figure, all objects including people, a key, monsters, and ladders are pre- -ra-ffr move interference targets like ladders. Given an RGB image observation or E H x3, the saliency encoder maps the original input to the saliency map my E [0. x| which highlights important parts of the image based on feedback labels. Then the policy trained with inputs that consist of the RGB images and the saliency predictions as an extra channel. See training details in Appendix F.5.
[Box1; .... Boxm }-
ex-
the annotations only need to re-
Keypoint Feedback. Frequently, only few decision points are im- portant long-horizon tasks (L.iu al., 2023). As illustrated the right figure, the agent (red arrow) must pick up the key and unlock the door to reach the target location (green circle). Guidance towards key ob- servations such as picking up the key and opening the door can reduce redundant exploration and improve learning efficiency. We annotate key time steps they = {11, t2, ...,.... fn) } from segment a, where n is the number of keypoints. Subsequently, we use this feedback to train keypoint pre- dictor and use historical trajectories to infer key states, which serve as guidance for the agent. See training details in Appendix F.4.
Keyfnist Feedback
Pick up kee
Open the dooar
3.3 LARGE-SCALE CROWDSOURCER ANNOTATION PIPELINE
To validate the case of use of various aspects of the Uni-RLHF, we implemented large-scale crowd- sourced annotation tasks for feedback label collection, using widely recognized Offline RL datasets. After completing the data collection, amount of noisy crowdsourced data. Our goal is to construct crowdsourced data annotation pipelines around the sourced data annotation and filtering. We aim to foster collaboration and reduce the high cost of feedback from RLHF research, facilitating wider adoption of RLHF solutions through reusable crowdsourced feedback datasets. See Appendix for details of datasets.
conducted two rounds of data filtering to minimize the
facilitating the creation of large-scale annotated datasets via parallel crowd-
Datasets Collection Workflow. To generate label and conduct primary evaluations, we assembled a team of about 100 crowdsourced workers, each of whom was paid for their work. These annotators come from a broad array of professions, ranging in age from 18 to 45 and boasting diverse academic backgrounds. None had prior experience in reinforcement learning or robotics research. We aim to simulate a scenario like practical applications of low-cost and non-expert crowdsourced annotation. Using comparative feedback annotation as an example, each annotator is assigned task along with instructions about the task description and judgment standards. To enhance the understanding of task objectives, experts (authors) provide five example annotations with detailed analysis. We refer to Appendix H for full instructions.
Collecting large-scale feedback often results in suboptimal and noisy data (Luce, 2012). To improve learning performance under human irrationality and bias, we adopt an ex-ante approach. During the feedback collection phase, we incorporate a small amount of expert validation set and discard labels that fall below the accuracy threshold. Furthermore, we check labels during the annotation process and summarize the overall problems to optimize the task description. This process aligns the intent of crowd- sourcing with experts. Another approach ex-post methods is label noise learning (Wang et al., 2022a; Jeon et al., 2020), which improves per- formance by understanding and modeling the generation mechanism of noise in the samples. This method is orthogonal to our filtering mecha- nism.
Ex-ante Filters.
Figure 2: Annotation ac- curacy in left-c task
To verify the effectiveness of each component in the annotation pipeline, we initially sampled 300 trajectory segments of the left-c task in SMARTS for expert annotation, referred to oracle. We

---

Published as a conference paper at ICLR 2024
had five crowdsourcing instances, each annotating 100 trajectories in three distinct settings. naive implies only seeing the task description, +example allows for viewing five expert-provided annota- tion samples and detailed analysis, and +filter adds filters to the previous conditions. The experi- mental results, displayed in Fig. 2, revealed that each component significantly improved annotation accuracy, ultimately achieving a 98% agreement rate with the expert annotations.
4 EVALUATING BENCHMARKS FOR OFFLINE RLHF
Finally, we conducted numerous experiments on downstream decision-makinn tasks, utilizing the collected crowdsourced feedback datasets to verify the reliability of the Uni-RLHF system. Notably, due to the wide applicability and low crowdsoureing training cost of comparative feedback. we conducted a more detailed study of comparative feedback in Section 4.1, systematically establishing comprehensive research benchmarks. We then performed additional experimental verification of attribute feedhack in Section 4.2 and keypoint feedback in Appendix C.5 as primer. We encourage the community to contribute and explore more extensive baselines and usage for additional feedback types within the Uni-RLHF system.
4.1
EVALUATING OFFLINE RL WITH COMPARATIVE FEEDBACK
Setups. As shown in Fig. (bottom), we used crowdsourcing comparative feedback for reward learning (see Appendix F.1 for more training details). Then we used the trained reward model to label reward-free offline datasets and train offline RL policy. Because the reward learning process and offline RL were fully decoupled. our methods can simply be combined with any offline RL algo- rithms. We used three of the most commonly used offline RL algorithms the backbone, including Implicit Q-Learning (IQL. Kostrikov et al. (2021)), Conservative Q-Learning (CQL,, Kumar et al. (2020)) and TD3BC (Fujimoto & Gu, 2021). We adopt implementations of these algorithms from the CORL (Tarasov et al., 2022) library and reuse the architecture and hyperparameters for differ- ent environments. To ensure comprehensive evaluation and benchmarking that accurately reflects the capabilities of the offline RLHF algorithm across a wide range of tasks and complexities, we selected wide range of tasks of varied types, encompassing locomotion, manipulation, and navi- gation tasks from D4RL (Fu et al.. 2020), as well visual control tasks from Atar-Atari (Seno, 2023) and realistic autonomous driving scenarios from SMARTS (Zhou et al., 2020). More envi- ronment and dataset details can be found in Appendix A. As baselines, modeling via three reward model architectures, denoted as MLP, CNN and TFM (Transformer) structure (see Fig. bottom right). MLP and CNN architectures are similar to Christiano et al. (2017); Lee et al. (2021a). MIP is for vector inputs, while CNN for image inputs. Following the Kim et al. (2023), we also implemented similar transformer architecture to estimate the weighted sum of Mar-Morko rewards. refer to Appendix for more details about RM architectures.
evaluated preference
4.1.1 D4RL EXPERIMENTS
expert scnre-random score underlying original reward from the benchmark. We used Oracle to represent models trained using hand-designed task rewards. In addition, we assessed two different methods of acquiring labels: one crowd-sourced labels obtained by crowd-sourcing through the Uni-RLHF system, denoted as CS, and the other is synthetic labels generated by script teachers based on ground truth task rewards, which can be considered as expert labels, denoted as ST. The performance is reported in Table 2, we focus on the performance of algorithms that crowdsourced labels because they best match the realistic problem setting.
We measured expert-normalized scores = 100 X scure random scure
Based on these results, we can draw several valuable observations. Firstly, among the three algo- rithmic foundations, IQL demonstrated the most consistent and highest average performance across environments when utilizing the same crowd-sourced datasets. This attributable not only to the high performance of the IQL baseline but also to the consistency of IQL's performance. When comparing the results of IQL-CS and IQL-Oracle, we found that IQL demonstrates competitive performance against Oracle in most of tasks and even surpasses Oracle in some cases. On the con- trary, CQL occasionally experiences policy collapse, leading to drastic performance drop, as exemplified in the application of the CQL-CS-MLP method to the hopper -m-rr. Secondly, After comparing the results of CS-TFM and CS-MLP (marked as blue), we empirically found that CS- TFM exhibits superior stability and average performance compared to CS-MLP, particularly in the context of sparse reward task settings. We hypothesize that this is due to TFM's modeling approach and its ability to capture key events, which align more closely with human judgment. Thirdly, our experiments show that crowd-sourced labels (CS) can achieve competitive performance compared to synthetic labels (ST) in most environments. Especially with CQL and TD3BC backbone, we can
7

---

Published as a conference paper at ICLR 2024
Table 2: Average normalized scores of various RL. baselines with human feedback D4RL datasets.
The scores are averaged across 3 seeds. In each dataset, the method which of the CS-MLP and CS-
TFM methods is better is highlighted in blue. The best performance of all methods are marked with If standard deviation is intersected with the best method, it is also marked with * For a complete set of results including mean and standard deviation, please refer to Appendix E.
Dataset
COL
TD3BC
Orade
SE-LPP
ST-TFM
CS-MLP
CS-TFM
Doale
ST-MLP
SI-TFM
CS-MLP
CS-TFM
Omal:
ST-MEP
S1-TFM
CS-MLP
CS-TFM
walker20 m
$0.91
75.39
75.4
79.36
80.75*
75.62
76.0
77.22
90.91
56.0
25.3
54.11"
walker2d m-r
$2.15*
68.6
60.33
67.3
56.52
73.09*
-0.3
33.18
20.6
1.82
82.15"
82.8*
24.3
47.2
61.94
walker2d
111.72
109,8
109.16
109.4
109.12
109.56
108.9
108.83
92.8
98.96
111.72
130.4
110.13
74.5
110.75
boppes-o
67.53
51.8
37.47
50.8
67.81
59.08
57.1
44.11
54.7
63.47'
60.37
58.6
62,89
48.0
99,42*
hepper-mer
97.43
70.1
64-42
22.65
95.11
2.1
2.08
1.8
$2.97
64.42
44.4
25.8
hopper-mee
107.42
107.7
109.16
94.3
111 439
99.26
57.5
57.27
57.4
57,05
101.15
103.7*
104,14
97.4
91.18
halfchtetahh m
48.31"
47.0
45.10
43.3
43.24
47.04
43.9
43.26
43.4
43.5
43.10
50.3"
43.05
34.8
46.62
44.46"
40.63
38.0
39,
45.04
42.8
40.73
41.9
40,97
44.34"
44.2
ballcherabbm-se
94,74*
92.2
92.91
91.00
92. 20
95.634
69.0
63.84
62.7
64.86
90.78
94.14
78.99
73.8
50.83
mujoco average
73.29
52,09
55.65
74.6
51.86
anemune
77.00
71.39
76.67
74.22
08.64
92.75"
93.71
91.71"
63.95
91.34"
70.75
93.51"
92.99*
90.25
92.30*
artmase-ud
54.25
51.66
59.67
54.60
63.82
37.25
34.05
25.11
6.77
22.75
44.75
73.19
51.88
59.58
65.75
74.24*
71.67
72.31
65.25
65.754
7.98
62.39'
60.26*
64.67*
0.25*
0.21
0.00-
0.25*
0,59
andmaro-m-d
73,75*
65.74
66.00
62.69
64.91
67.25
17.50
63.27
46.95
69. 749
0.25
3,33*
0.99
0.10
0.30
antmaze
42.00
40.79
43.33*
19.36"
64.63*
20.75
1.70
18.45
16.45*
19.33
0.00
0.07
0.00
0.00
0.00
astmaze- 1-d
30.25
49.24"
29.67
21.97
29.67
20.50
20.88
12.39
0.00
33.00
0.00
0.00
0.00
0.00
0.00
avenge
57.17
57.67
55.04
56.12
45.55
37.06
50.14
28.38
23.75
pen human
78.49*
50.15
63.66
57.26
06.07
13.71
9.80
20.31"
6.53
23.73
3,58
3,94"
3.94
3.71*
2.81
pas-clined
83.42*
59.92
64.55
62.94
62.26
3.721
2.88
3.18
5.13
10844
14.52*
6.71
19.13
pen-expent
128.05
132.85
127.20
120.55
122.42
-1.41
34*
119.60
121,14
122.41
122.53
14.41
34.62
11.45
30.78
door-human
3.36
6.'
3.22
4.68
4.92
10.31
5.81
-0.13*
-0.32
.0.33
doce-cloned
3,07*
-0.08
-0.06
-0,10
-0.02
-0.33*
-0.34
-0.34*
-0.34*
-0.34
0.29*
-0.34
-0.34
-0.34
-0.34
door-expert
106.65
105.35
105.05
105.2) 72
105 (0
-0.32
103.90
103.32
102.63
103.15
-0.33*
-0.34*
-0.33*
-0.34
-0.34
bummer human
1,79*
1.43*
1,85*
1:03
0.54
0.35
1.41"
0.70*
1.13"
1.02*
1.00*
0.96*
0.46
1,02*
hamite-cioned
1.50*
0.70*
1.87
0.67
0.73
0.30
0.28
0.29*
0.284
0.29
0.25
0.25
0.27
3.45*
0.35*
kammeet-cspett
127.36
120.66
117.60*
2.22
3.03*
adroit average
59.43
55.46
55.39
49.33
$4.08
2.1
42.39
41,57
40.2
42.20
14.23
2.64
5.4
1.83
5.82
observe a significant improvement in some cases. This suggests that while synthetic labels are not subject issues such as noise and irrationality, they are prone shortsighted behaviors and cannot provide flexible and variable evaluations based on a global view like humans.
4.1.2 ATARI EXPERIMENTS
We conducted experiments in five image- based Atari environments, with detailed results presented in Table 3. We col- lected 2,000 pairs of annotated compara- tive feedback for each task and trained re- ward model for the CNN structure. Next, we chose the CQL of discrete version by d3ripy' codebase and verified the perfor- mance of Oracle, ST and CS with the same hyperparameters. Across all tasks, we found that models trained using crowd- sourced labels consistently outperformed those trained with synthetic labels, achieving performance levels comparable to the Oracle. We hypothesize that this superior performance of crowd-sourced labels could be due to their ability to capture nuanced details within the game environment, details that may prove challenging to encap- sulate through simple integral reward functions. This potentially results in more accurate evaluations as compared to those derived from synthetic labels.
Table 3: Averaged scores of DiscreteCQL 5 atari tasks with different reward functions. The result shows the average and standard deviation averaged over 3 seeds. We use highlighting to mark the bet- ter one of the ST and CS labels.
Dataset
DiscreteCOL-Oracl DiscreteCOL-ST
DiscreteCQL-CS
Boxing-m
78.2+7.8
62,447.8
64.5+10.6
Breakout-m
79.9242.0
3.0:1.0
33.5+10.8
Enduro-m
196.9#4.9
189.5+8.3
191.5+8.3
Pong-m
14.522.6
18.542.2
18.612.1
Qbert-m
11013.2:1655.9
8170.8+2387..9
11301.7#313.3
Table 4: Success rate of IQL with different
4.1.3 SMARTS EXPERIMENTS
reward model in SMARTS with 3 seeds.
Task IQL-Oracle
IQL-ST IQL-CS
As shown in Table 4, we studied three typical scenarios in autonomous driving scenarios. The reward function design is particularly complex because it requires more extensive expertise and balancing multiple factors. We
cruise
0.71+0.03
0.55+0.01
0.62+0.03
left-c
0,5340.03
0.52+0.03
0.70+0.03
cutin
0.85+0.04
0.81=0.03
0.80+0.03
average
0.70
0.63
0.71
8

---

Published as a conference paper at ICLR 2024
Iargee:
Farger:
1200
1004
1000
190000 90.7
750
190
250-
234
250
Figure 3: Learning curves of three sub-tasks with different objectives. The target attribute strengths are set to [1.0, 0.5, 0.5, 0.5, 1.0), [0.5, 1.0, 0.5, 0.5, 1.0] and 00.5, 0.5, 0.5, 1.0, 1.0]. respectively. The policy can continuously optimize multiple objectives during the training process.
thee
Figure 4: We visualized the behavior switching by adjusting the target attributes every 200 steps. The attribute values for speed were to [0.1, 1.0, 0.5, 0.1, L.0]. and for height, they were set [1.0, 0.6, 1.0, 0.1, 1.0). The corresponding changes in attributes can be clearly observed in the curves. We refer to homepage for the visualisations of walker.
carefully designed the task rewards incorporating eight factors, which includes some events and mul- tiple penalty (see Table 5 for details). We empirically demonstrate that can achieve competitive performance simply by crowdsourced annotations, compared to carefully designed reward functions or scripted teachers. We provide a multi-indicator metries for evaluation (speed and comfort) and visualisation in Appendix C.I. The model using human feedback showed more conservative and rule-following policy which was consistent with human intentions.
4.2 EVALUATING OFFLINE RL WITH ATTRIBUTE FEEDBACK
Setups. We trained an attribute-con conditioned reward model attribute feedback datasets for multi- objective optimization (see Appendix F.2 for more training details). To utilize this reward model for training, we adopted version of TD3BC, we concatenated the target relative attribute strengths to the inputs of both the policy and value networks. During training, we first evaluated the attribute strengths of the trajec- tories in the datasets, then obtained the rewards corresponding each time step the trajectories using the reward model, and finally used evaluated attribute strengths and rewards to train TD3BC.
TD3BC approach. Specifically, based on the original
Results. In our experiments, trained TD3BC on the walker environment from DMControl, of which pre-defined attributes are (speed, stride, leg preference, torso he height, humanness) (See Appendix H.2 for more details of attribute definition and feedback collection). We selected three sets of target attribute strengths, representing objectives of making the walker move faster, have a larger stride, or have a higher torso. During training, we validated the algorithm's 's performance on these three sub-tasks and showed the learning curve Fig. 3. The results showed that using the further tested the trained model's ability switch behaviors. We ran the model for 1000 steps and adjusted the target attributes every 200 steps, recording the walker's 's speed and torso height in Fig. 4. reward model could adjust
reward model could indeed achieve multi-objective training. We
The results showed that the policy trained using the attribute-condition its behavior based on the given target objectives, achieving the goal of multi-objecti multi-objecctve optimization.
CONCLUSION, CHALLENGE, AND FUTURE DIRECTIONS
To alleviate the difficulties of widely adopting RLHF solutions in applications, we propose compre- hensive system Uni-RLHF. We first implement the platform that supports multiple feedback types along with a feedback standard encoding format. Subsequently, we carry out large-scale crowd- sourced annotation creating reusable offline feedback datasets. Finally, we integrate mainstream of- fline reinforcement learning algorithms with human feedback, evaluate the performance of baseline algorithms on the collected datasets, and establish reliable benchmark. Although We attempted to cover a wide range of factors in RLHF, there are some limitations. For example, we conduct large- scale experiments with only the most common Comparative Feedback and we lack deep analyses of human irrationality. We summarize some future research directions in Appendix G.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (Grant Nos. 92370132), the National Key R&D Program of China (Grant No. 0022D00164022 and the Xi- aomi Young Talents Program of Xiaomi Foundation. The authors thank Peilong Han for his help in building and deploying the platform.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, ICML, 2004.
Riad Akrour, Mare Schoenauer, and Michele Sebag. Preference- baaeee policy learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD, 2011.
Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L Littman. Deep reinforcement learning from policy-dependent human feedback. arXiv preprint axXiv:1902.04257, 2019.
Akanksha Atrey. Kaleigh Clary, and David Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. arXiv preprint urXiv:1912.05743, 2019.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
Erdem Bıysk, Malayandi Palan, Nicholas C Landolfi, Dylan P Losey, and Dorsa Sadigh. Asking easy questions: A user-friendly approach to active reward learning. arXiv preprint arXiv:1910.044. 1910.04363, 2019.
Erdem Biyik, Aditi Talati, and Dorsa Sadigh. Aprel: A library for active preference- reward learning algorithms. In ACM/IEEE International Conference on Human- Robot Interaction, HRI, 2022.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3324-34 1952.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.1522 15217 2023.
Thomas Cederborg, Ishaan Grover, Charles L.. Isbell Jr., and Andrea Lockerd Thomaz. Policy shap- ing with human teachers. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI, 2015.
Si-An Chen, Voot Tangkaratt, Hsuan- Tien Lin, and Masashi Sugiyama. Active deep q-learning with demonstration. Machine Learning, 109:1699-111 2020.
Maxime Chevalicr-Boisvert,, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CORR, abs/2306. 13831, 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforeement learning from human preferences. Advances in neural information processing sys- tems, NIPS, 2017.
Oliver Daniels-Koch and Rachel Freedman. The expertise problem: Learning from specialized feedback. arXiv preprint arXiv:2211.066 0,66699 2022.
Quentin Delfosse, Jannis Blüml, Bjarne Gregori, Sebastian Sztwiertnia, and Kristian Kerst- ing. Ocatari: Object-centric atari 2600 reinforcement learning environments. arXiv preprint arXiv:2306.ne 200.4969660 2023.
10

---

Published as a conference paper at ICLR 2024
Zibin Dong. Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng. Yujing Hu, Tangjie Lv, Changjie Fan, and Zhipeng Hu. Aligndiff: Aligning diverse human preferences via behavior-customisabe diffusion model. arXiv preprint arXiv:23
2023.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-338, 2010.
Shuo Feng, Haowei Sun, Xintao Yan, Haojie Zhu, Zhengxia Zou, Shengyin Shen, and Henry X Liu. Dense reinforcement learning for safety validation of autonomous vehicles. Nature, 615(7953): 620-627, 2023.
Pedro Freire, Adam Gleave, Sam Toyer, and Stuart Russell. Derail: Diagnostic environments for reward and imitation learning. arXiv preprint arXiv:2012.01365, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning.
Advances in neural information processing systems, 22-21122- 2021.
Gaurav R Ghosal, Matthew Zurek, Daniel $ Brown, and Anca D Dragan. The effect of modeling human rationality level on learning rewards from multiple feedback types. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Kevin Gmelin, Shikhar Bahl, Russell Mendonca, and Deepak Pathak. Efficient rl via disentangled environment and agent representations. International conference on machine learning. 2023.
Lin Guan, Mudit Verma, Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems, NeurlPS, 2021a.
Lin Guan, Mudit Verma, Suna Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation. Advances Neural Information Processing Systems, 2021b.
Lin Guan, Karthik Valmeekam, and Subbarao Kambhampati. Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine Soft actor-eritic: Off-policy
maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, 2018.
Dylan Madfield-Menell, Smitha Milli, Picter Abbecl, Stuart J. Russell, and Anca D. Dragan. Inverse reward design. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, 2017.
Ayano Hiranaka, Minjune Hwang, Sharon Lee, Chen Wang, Li Fei-Fei, Jiajun Wu. and Ruohan Zhang. Primitive skill-based robot learning from human evaluative feedback. arXiv preprint arXiv:2307. 15801, 2023.
Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing- Shan Jia, and Ya-Qin Zhang. Query-police misalign- ment in preference-based reinforcement learning. arXiv preprint arXiv:2305.17400 2023.
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 33:15931-15941 2020.
Junning Huang. Sirui Xie, Jiankai Sun, Qiurui Ma, Chunxiao Liu, Dahua Lin, and Bolei Zhou. Learning a decision module by imitating driver's control behaviors. In Conference on Robot Learning, pp. 1-10. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2024
Zhiyu Huang, Haochen Liu, Jingda Wu, and Chen Lv. Differentiable integrated motion prediction and planning with learnable cost function for autonomous driving. IEEE transactions on neural networks and learning systems, 2023.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 33:4415- 4426, 2020.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):982- 992-987,,. 2023.
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Pref- erence transformer: Modeling human preferences using transformers for RL. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, 2009.
W Bradley Knox and Peter Stone. Reinforcement learning from simultaneous human and mdp reward. In AAMAS, 2012.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint 10Xiv:2110.06 2021.
Srinivas SS Kruthiventi, Kumar Ayush, and R Venkatesh Babu. Deepfix: A fully convolutional neural network for predicting human eye fixations. IEEE Transactions on Image Processing, 26
(9)4446-4456, 2017.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 13:1179-1191, 2020.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: teefback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. Proceedings of the 38th Inter- national Conference on Machine Learning, ICML, 2021a.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. arXiv preprint urXiv:2111.03026 2021b.
Jianxiong Li. Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang. Mind the gap: Offline policy optimization for imperfect rewards. arXiv preprint arXiv:2302.01667 2023.
Anthony Liang, Jesse Thomason, and Erdem Biyik. Visarl: by human saliency. In ICRA2023 Workshop Pretraining for Robotics (PT4R), 2023.
Visual reinforcement learning guided
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbecl. Reward uncertainty for exploration in preference- based reinforcement learning. In The Tenth International Conference on Learning Representations, ICLR, 2022.
David Lindner and Mennatallah El-Assady. Humans are not boltzmann distributions: Challenges and opportunities for modelling human feedback and interaction in reinforcement learning. arXiv preprint arXiv:2206. 13316, 2022.
Haozhe Liu, Mingchen Zhuge, Bing Li, Yuhui Wang, Francesco Faccio, Bernard Ghanem, and Jürgen Schmidhuber. Learning to identify critical states for reinforcement learning from videos. arXiv preprint arXiv:2308 208779230..07795, 2023.
Jinyi Liu, Yifu Yuan, Jianye Hao, Fei Ni. Lingzhi Fu, Yibin Chen, and Yan Zheng. Enhancing robotic manipulation with ai feedback from multimodal large language models. arXiv preprint arXiv:2402. 14245, 2024.
12

---

Published as a conference paper at ICLR 2024
Nian Liu, Junwei Han, and Ming Hiuan Yang. Picanet: Learning pixel-wise contextual attention for saliency detection. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 3089-3098, 2018.
Cong Lu. Philip Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. arXiv preprint arXiv:2206.04779, 2022.
Duncan Luce. Individual choice behavior: A theoretical analysis. Courier Corporation, 2012.
James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew Taylor, and Michael L Littman. Interactive learning from policy-dependent human feedback. In International conference on machine learning, 2017.
Yannick Metz, David Lindner, Raphaël Baur, Daniel Keim, and Mennatallah El-Assady. Rihf- blender: A configurable interactive interface for learning from diverse human feedback. arXiv preprint arXiv:2308. 04332, 2023.
Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In ICML, 2000.
Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. Metadif- fuser: Diffusion model as conditional planner for offline meta-rl. In International Conference on Machine Learning, pp. 26087-26105. PMLR, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. arXiv preprint arXiv:2203. 10050, 2022.
Patrick M Pilarski, Michael R Dawson Thomas Degris, Farbod Fahimi, Jason P Carey, and Richard Sutton. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. In IEEE international conference on rehabilitation robotics, 2011.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018
Amir Rasouli, Soheil Alizadeh, Iuliia Kotseruba, Yi Ma, Hebin Liang, Yuan Tian, Zhiyu Huang, Haochen Liu, Jingda Wu, Randy Goebel, Tianpei Yang, Matthew E. Taylor, Liam Paull, and Xi Chen. Driving smarts competition at neurips 2022: Insights and outcome. In Proceedings of the NeurIPS 2022 Competitions Track, 2022.
Takuma Seno. D4rl atari. https: /ttps:///githii wwt.//oooo//etae..eeeeee tttp//aaae//rrrraaaaaa
2023.
Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca Dragan. On the feasibility of learning, rather than assuming, human biases for reward inference. In International Conference Machine Learning, 2019.
Pratyusha Sharma, Balakumar Sundaralingam,, Valts Blukis, Chris Paxton, Tucker Hermans, An- tonio Torralba, Jacob Andreas, and Dieter Fox. Correcting robot plans with natural language feedback. arXiv preprint arXiv:2204.05/ 206.00886.., 2022.
Daniel Shin, Anca D Dragan, and Daniel S Brown. Benchmarks and algorithms for offline preference-based reward learning. arXiv preprint arXiv:2301.01392, 2023.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances in Neural Information Processing Systems, 35:9460- 9470009947,,-01 2022.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae- Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEFE Transactions on Neural Networks and Learning Systems, 2022.
13

---

Published as a conference paper at ICLR 2024
Jonathan Sorg, Richard L Lewis, and Satinder Singh. Reward design via online gradient ascent.
Advances in Neural Information Processing Systems, 23, 2010.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. arXiv:2210.07105, 2022.
arXiv preprint
Stefano Teso and Kristian Kersting. Explanatory interactive machine learning. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.002288 2023.
Haobo Wang, Ruixuan Xiao, Yiwen Dong, Lei Feng, and Junbo Zhao. Promix: combating label noise via maximizing clean sample utility, arXiv preprint arXiv:2207.10276. 2022a.
Zhen Wang, Marko Jusup, Rui-Wu Wang, Lei Shi, Yoh Iwasa, Yamir Moreno, and Jürgen Kurths. Onymity promotes cooperation in social dilemma experiments. Science advances, 3(3):e1601444, 2017.
Zhen Wang, Marko Jusup. Lei Shi, Joung-Hun Lee, Yoh Iwasa, and Stefano Boccaletti. Exploiting cognitive bias promotes cooperation in social dilemma experiments. Nature communications, 9 (1):2954, 2018.
Zhen Wang, Marko Jusup, Hao Guo, Lei Shi, Sunčana Gežek, Madhur Anand, Matjaž Perc, Chris T Bauch, Jürgen Kurths, Stefano Boccaletti, et al. Communicating sentiment and outlook reverses inaction against collective risks. Proceedings of the National Academy of Sciences, 117(30): 17650-17655, 2020.
Zhen Wang. Chunjiang Mu, Shuyue Hu, Chen Chu, and Xuelong Li. Modelling the dynamics of regret minimization in large agent populations: master equation approach. In IJCAI, pp. 534- 540, 2022b.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interac- tive agent shaping in nigh-dimensional state spaces. In Proceedings of the AAAI conference artificial intelligence, 2018
Devin White, Mingkang Wu, Ellen Novoseller, Vernon Lawhern, Nick Waytowich, and Yongcan Cao. Rating-based reinforcement learning. arXiv preprint arXiv:2307. 16348, 2023.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. Advances in neural information processing systems, NIPS, 2012.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference- based reinforcement learning methods. The Journal of Machine Learning Research, 18: 13::1 136:46, 2017.
Wanqi Xue, Bo An, Shuicheng Yan, and Zhongwen Xu. Reinforcement learning from diverse human preferences. arXiv preprini arXiv:2301.11774, 2023.
Ruohan Zhang, Faraz Torabi, Lin Guan, Dana H. Ballard, and Peter Stone. Leveraging human guid- ance for deep reinforeement learning tasks. In Proceedings of the Eventy-Eightt International Joint Conference on Artificial Intelligence, IJCAI, 2019.
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Mont- gomery Alban, Iman Fadakar, Zheng Chen, al. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving. arXiv preprint arXiv:2010.09776, 2020.
Xinglin Zhou, Yifu Yuan, Shaofu Yang, and Jianye Hao. Mentor: Guiding hierarchical re-
inforcement learning with human feedback and dynamic distance constraint. arXiv preprint arXiv:2402.1 14244, 2024.
Baiting Zhu, Meihua Dang, and Aditya Grover. Scaling pareto-efficient decision making via offline multi-objective rl. arXiv preprint arXiv:2305.00561 2023.
14

---

Published as a conference paper at ICLR 2024
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Ku- mar, and Sergey Levine. The ingredients of real world robotic reinforcement learning. In 8th International Conference Learning Representations, ICLR, 2020.
15

---

Published as a conference paper at ICLR 2024
A
ENVIRONMENT AND DATASETS DETAILS
In this section, we describe the details of all environments and corresponding tasks.
A.1 THE DETAILS OF THE DATASETS FOR D4RL
Gym Mujoco tasks consist of the HalfCheetah, Hopper, and Walker2d datasets from the D4RL (Fu et al., 2020) offiin RL benchmark. Walker2d is bipedal robot control task, where the goal is to maintain the balance of the robot body and move as fast possible. Hopper is a single-legged robot control task where the goal is to make the robot jump as fast as possible. Halfcheetah is to make a simulated robot perform a running motion that resembles the movement of cheetah while trying to maximize the distance traveled within a fixed time period. We use the medium, medium-expert, and medium-replay datasets in our evaluations, which consist of (a mixture of different) policies with varying levels of optimality. We use the v2 version of the environment in our experiments.
AntM involves controlling an 8-DoF ant quadruped robot to navigate through mazes and reach a desired goal. The agent receives sparse rewards of +1/0 based on whether it successfully reaches the goal or not. We study each method using the following datasets: large-diverse, large-play, medium- diverse, medium-play, umaze-diverse, and umaze. The difference between diverse and play datasets the optimality of the trajectories they contain. The diverse datasets consist of trajectories directed towards random goals from random starting points, whereas the play datasets comprise trajectories directed towards specific locations that may not necessarily correspond the goal. We used the v2 version of the environment in our experiments.
Adroit contains series of continuous and ppare--reaate robotic environments to control 24-DoF simulated Shadow Hand robot to twirl a pen, hammer a nail, open a door, or move a ball. This domain was selected to measure the effect of narrow expert data distributions and human demon- strations on a sparse reward, high-dimensional robotic manipulation task. There are three types of datasets for each environment: expert, human, and cloned. We use the v1 version of the environment in our experiments.
A.2 THE DETAILS OF THE DATASETS FOR ATARI
We evaluate offline RLHF algorithms on five image-based Atari games (Pong, Breakout, Qbert, Boxing and Enduro). using the dataset released by D4RL_Atari The dataset for each task was categorized into three levels of difficulty: mixed denotes datasets collected at the first 1M steps, medium denotes datasets collected between 9M steps and 10M steps, and expert denotes datasets collected at the last IM steps. We evaluate the algorithms using the medium difficulty dataset be- cause the expert dataset only requires good imitation performance without reward function learning.
A.3 THE DETAILS OF SMARTS ENVIRONMENTS AND DATASETS COLLECTION
Scalable Multi-Agent Reinforcement Learning Training School (SMARTS) (Zhou et al., 2020) is simulation platform designed for deep learning applications the domain autonomous driving. striving to provide users with realistic and diverse driving interaction scenarios.
In this study, we selected three representative and challenging driving scenarios from the Driving SMARTS 2022 benchmark (Rasouli et al., 2022). Fach scenario corresponds to a distinct driving situation and demands specific driving skills for appropriate handling. Within these scenarios, the infrastructure is hicrarchically organized into road, lane, and waypoint structures. At the highest level, the transportation network is structured as graph where each node represents individ- ual road. Each road can further encompass one or multiple lanes, and subsequently, each lane subdivided into a series of waypoints.
Upon environment initialization, given a start point and goal, the simulation environment generates a road-level global driving route for the agent. The timestep for these scenarios is uniformly set at 0. At each timestep, the agent, using the global route, road features, and the dynamic states of the ego vehicle and neighboring vehicles, decides on actions that lead the ego vehicle to accelerate, decelerate, or change lanes. The environment then updates the states of both the ego vehicle and
16

---

Published as a conference paper at ICLR 2024
Table 5: Task Reward Function Design of Smarts
Item
Value
Definition
Distance-
Distance-Traveled Traveled Reward
Single-step travelling distance
Encouraging vehicles to travel towards goal ego vehicle reach the goal
Reach Goal Reward
+20
Near Goal Reward
min(dis.ke cessstoggooo. tot.stt.ggo////aoo /oal/
ego vehicle close the goal
Collision Penalty
-40
Collision
Mistake Penalty
Ego vehicle triggers off road, off lane wrong way The ego vehicle in right lane
Lane Selection Reward
+0.3
Lane Change Penalty
The ego vehicle changes lane
Time Penalty
-0.3
Each time step gives fixed penalty term
other vehicles in the environment based on the agent's decision and other influential factors, such as the behavior of other vehicles, ensuring real-time dynamic information provision for the agent.
The specifics of the three selected scenarios within SMARTS are described as follows:
The ego vehicle is tasked to travel on three-lane straight road. During its journey, it must adjust its speed and change lanes when necessary to avoid potential collisions, ensuring successful arrival at the end of a specified lane.
cruise:
cut-in: The ego vehicle is tasked to travel three-lane straight road where it encounters distinct challenges compared to the standard cruise scenario. Environmental vehicles in this setting may display more aggressive behaviors like lann-changing or sudden deceleration, aiming to obstruct the ego vehicle's 's progression. As a result, the ego vehicle must adeptly navigate these challenges to ensure safe arrival at its designated goal.
left-c: The ego vehicle is tasked to commence its journey from a single-lane straight road, progress towards an intersection, and execute left turn. Following the turn, it is expected to merge into designated lane on a two-lane straight road.
Datasets collection. In each of the three driving scenarios, we employed online reinforcement learning algorithms to train two agents, each designed specifically for the respective scenario. The first agent demonstrates medium driving proficiency, achieving success rate ranging from 40% to 80% in its designated scenario. In contrast, the second agent exhibits expert-level performance, attaining a success rate of 95% or higher in the same scenario. For dataset construction, 800 driving trajectories were collected using the intermediate agent, while an additional 200 were gathered via the expert agent. By integrating the two datasets, we compiled a mixed dataset encompassing 1 1,000 driving trajectories.
Task Reward Function Design. To direct the ego vehicle efficiently to its goal while prioritizing safety, We went through a number of tweaks and iterations, and eventually established a multifaceted reward function, marked as Oracle in Table 4.
This function consistently applied across the three distinct driving scenarios. The function encom- passes several components: distance-travele reward, reach/near goal reward, mistake event penalty, lenesselcction reward, and time penalty. The specifics of each component are detailed below and dis- played in the Table 5:
Distance-Traveled Reward: This reward promotes the forward progression of the ego vehicle towards its goal. It is quantified by the distance the vehicle travels along the lane that directs towards the goal.
This is a sparse reward, allocated exclusively at the terminal step. Should the ego vehicle successfully reach the goal, a reward of +20 is granted. Furthermore, if the ego vehicle concludes its task within 20m proximity of the goal, a "near-goal" reward is dis- pensed, determined as the lesser value between dis to to_goal/5 5 and 10. This structure accentuates the importance of closely approaching the goal, fostering an active drive towards the destination.
Reach/Near Goal Reward:
Mistake Event Penalty: This penalty addresses undesirable actions undertaken by the ego ve- hicle. Specifically, significant penalty of -40 is levied if the ego vehicle collides with another vehicle. For individual transgressions, such as coming into contact with road edges or road shoul- ders, instances of driving against the flow of traffic, penalty of -6 is imposed.
Lane-Selection Reward: This encapsulates both a reward and a counteracting penalty. The pos- itive reward, valued at +0.3, is conferred whenever the ego vehicle operates within the lane that
17

---

Published as a conference paper at ICLR 2024
leads it to the goal, thus encouraging correct lane-changing conducive to the goal's 's direction. Con- versely, penalty -1 aims to inhibit unwarranted or frequent lane-changing actions, imposed on each instance the ego vehicle transitions between lanes.
Time Penalty: To emphasize efficiency and motivate the vehicle swiftly reach its destination, a consistent minor penalty of -0.3 is applied at every timestep.
B REWARD MODEL STRUCTURE AND LEARNING DETAILS
We implemented the structure of the three reward models with the following implementation details:
MLP: We use a three-layer neural network with 256 hidden units each using ReLU as an activation ensemble of three reward models and
function. To improve the stability in reward learning, constrain the output using the tanh function. Each model is trained by optimizing the cross- cross-entropy loss defined in Eq. (4) using Adam optimizer with the initial learning rate of 0.0003.
CNN: We make the following changes to the MLP structure: 1) We process the image input using a 3-layer CNN with kernel sizes (8, 4, 3). stride 24, 1)). and (32, 64, 64) filters per layer. 2) We empirically demonstrate that the tanh activation function exhibits instability in visual environments and is prone to gradient vanishing. so we do not use the activation function for the output. 3) We use a lower learning rate of 0.0001.
For our experiments, we modified the publicly available codebase of Preference Transformer (PTy² and re-implemented our cansformer-baseee reward model. Unlike other struc- tures, PT is better able to capture key events in the long-term behaviors of agents through a prefer- ence predictor P |o > a"| based on the weighted sum of non-Markovian rewards:
Transformer:
exp (Ee
(((s) a.) ) +
a.) ()) :)))
P P > :
(1)
(5% (((5 ( :16
Σje(o.1) exp
E, w
i ) i
As shown in Fig. (bottom right), PT consists of Causal Transformer and Preference attention layer components. In all experiments, we use causal transformers (GPT architecture) (Radford et al., 2018) with one layer and four self-attention heads followed by bidirectional self-attention layer with single self-attention head. Given trajectory state-action segment a of length H, the embedding is generated by a linear layer. Then the embedding is fed into the causal transformer network and produces the embedding of the output {xr} where the th output corresponds to t of the inputs. Then, the preference attention layer receives the hidden embedding from the output of causal transformer {x+ ) and generates rewards é, and importance weights wi. We define the output 2; as the attention weight values of the i -th query and the keys based on the self-attention mechanism.
H
2 softmax ((():)))) ft
Σ
1 Σ softmax (I/g, kp) ke)) (e)
Then we can be defined as w: with MI. structure, PT requires a trajectory as input to generate rewards. Following the codebase of PT, when predicting Tt, take the trajectory of length H up to time as input and choose the corresponding last output as Ty.
Unlike the reward model
H
Hyperparameters for training transformer structure reward models are shown in Table 6.
C ADDITIONAL ANALYSIS AND EVALUATION EXPERIMENTS
C.I ADDITIONAL ANALYSIS AND VISUALISATION OF SMARTS EXPERIMENTS
In order to evaluate the performance of the model from multiple perspectives, we have introduced two additional metrics: (1) the average driving speed of the vehicle, to assess the operational effi ciency, which is the larger the better, and (2) the comfort level of the vehicle, characterized by the
Transfomerr
18

---

Published as a conference paper at ICLR 2024
Table 6: Hyperparameters for training transformer structure reward models.
Hyperparameter
Value
Embedding dimension
256
Number of attention heads
4
Number of layers Dropout Number of ensembles
0.1
Batch size Optimizer
64
Adam
Learning rate
10
Weight decay
10-4
Table 7:
Multi-indicator Evaluation in the SMARTS. Larger values for success and speed better, while smaller values for comfort are better.
IQL-Oracle
IQL-CrowdSource
success rate?
speedf
comfort,
success rate?
speedf
comfort,
left-c
0.53 # 0.03
9.75 = 0.32
7.10 = 0,06
0.70 + 0.03
10.04 + 0.16
6.98 + 0.26
cruise
0.71 0.03
13.65 + 0.03
1.85 = 0.08
0.62 + 0.03
12.61 + 0.44
1.84 = 0.49
cutin
0.85 0.04
13.84 + 0.05
0.95 0.23
0.80 + 0.03
13.90 = 0.01
0.86 = 0.05
0.70
12.42
3.30
0.71
12.19
3.23
sum of the vehicle's linear jerk (the rate change of linear acceleration) and angular jerk (the rate of change of angular acceleration), which is the smaller the hetter.
in the SMARTS environment. Fig. 5 presents an instance of a left-tur-crooss scenario in which the CS model succeeds, while the Oracle model fails due to collision. In this scenario, the ego vehicle is required to start from a single-lane straight road, make a left turn at the cross intersection, proceed onto another two- lane straight road, and ultimately reach a goal located at the end of one of these two lanes. When navigating the intersection, if there stopped or slow-moving neighboring vehicle in front of the ego vehicle, awaiting its turn to pass through, the ego vehicle must decelerate to prevent a collision with the vehicle ahead. The first controlled by the Oracle model. The second row displays the trajectory of the ego vehicle under the control of the CS model. The figure reveals that the Oracle model fails to timely decelerate the ego vehicle, resulting collision with the vehicle ahead, whereas the CS model opts to stop and wait for the vehicle in front, eventually passing through the intersection smoothly.
Next, provide visualization of an instance of the left turn
of the Fig. 5 shows the driving trajectory of the ego vehicle
C.2 ANALYSIS AND VISUALIZATION OF THE LEARNED REWARD MODEL
As shown in Fig. 6, evaluate the quality of reward model in the training process. Experiments show that the reward model can align with human intent and achieve a high and stable prediction accuracy. The results in Fig. demonstrate that the trained reward model is able to accurately predict step reward for each state and has strong correlation with the carefully hand-designed reward function.
C.3 ABLATION OF DIFFERENT QUERY SAMPLER IN OFFLINE RLHF
To better validate the role of different samplers, collected crowdsoureing labels and con- ducted the experiments to demonstrate the performance of different sampling methods. In each experiment, we collect Nquery = 100 annotation labels (comparative feedback) using different sam- pler separately and use IOL-MLL-CrowwSourre CrowdSoureed method for simplicity. Specifically, we consider the following query sampler:
Random Sampler: Randomly select Nquery pairs of trajectory segments from the dataset.
19

---

Published as conference paper at ICLR 2024
Frame
Frame
Frame 35
Frame 53
Frame
Frame 89
Figure 5: Visualization an instance of the left scene in the SMARTS environment. The first
row shows the driving trajectory of the ego vehicle controlled by the Oracle model. The second row displays the trajectory of the ego vehicle under the control of the CS model. CS model opts to stop and wait while Oracle model results in collision.
waller2d-medium-mpppl
hespeiimeduu--epayy.2
0.9
0.8
30
1.0
20
40
25
2pors
Epech
Eboth
Epoch
Figure Learning curves reward model on D4RI. MuloCo tasks. Fach experiments was trained with crowdsourced labels and run for 3 seeds.
Disagreement Sampler: Firstly, we generate Nmn = 20 pairs of trajectory segments randomly, then train the initial reward models and measure the variance across ensemble of reward models prediction. Then we select the most N = 40 high-variance segments. After two iterations, we obtain Nquery 100 annotation labels and the final reward model is trained.
We modify the customization interface to achieve the entropy sampler, which measure the entropy of preference prediction. The other experiments process consistent with the disagreement sampler.
Entropy Sampler:
Table 8 shows the normalized scores of MMLL-CCCoddSureed with various query sampler. We observe that suitable label sampler could improve the task performance, this also consistent with similar conclusion by Shin et al. (2023). The disagreement sampler outperforms random sampling, however, the training phase requires multiple rounds of iterations, which introduces little additional sampling cost.
C.4 HUMAN EVALUATION FOR ATTRIBUTE FEEDBACK EXPERIMENTS
RLHF is most usefuu domains where only the humans understand the good behaviour like hu- manness attribute. Therefore, we conducted human evaluation experiments for the agents trained by attribute feedback. We first collected 3 trajectories with different humanness, which 1Ó.1, 0.6, 1] and invited five human evaluators. The human evaluators performed a blind selection to judge which video is most human-like and which video is least human-like. If the result is consistent with the preset labels, it means that the agents can follow the humanness indicator well.
to
Table 8: Normalized scores of IQL-MLLPCCrowSSoue CrowdSourced with various query sampler across three seeds.
Dataset
Random Sampler
Disagreement Sampler
Entropy Sampler
antmaze- siattee-eediimdiiiie diverse-v2 hopper- seceee-mediim--ere replay-v22
46.95 + 11.91 42.75 5.27
64.43:16.57 63.28-18.58
62.69-7.51 44.53 6.83
20

---

Published as a conference paper at ICLR 2024
weller/--medium-eeperea
hopper-medium-eeppe
Predktte heroee
0.4
0.2
00
0.2
0.0
-0.1
Ground Couth Roword
Ground F. Roward
Ground Truth Reward
Groundaruth Roward
Visualization of the correspondence between predicted reward and ground truth step re- wards on 4 D4RL MuJoCo tasks.
Figure 7:
Table 9: Normalized scores between TD3BC and TD3BC-keypoint in the maze2d environments. Each run with seeds. We observe that adding keypoint guidance achieves significant results in long horizon navigation tasks with sparse reward.
Dataset
TD3BC
TD3BC-keypoint
maze2d-medium-vl maze2d-large-vl
59.5433.3
131.7=33.3
97.1+25.4
150.6+60.2
Finally, all people correctly chose the highest humanness trajectory, and only person chose the lowest humanness trajectory incorrectly. The experimental results confirm that agents through RLHF training are able to follow the abstraction metrics well. We provide video clip in Fig. 10 and full videos on the homepage.
C.5 EVALUATING OFFLiNE RL WITH KEYPOINT FEEDBACK
We trained an keypoint predictor keypoint feedback datasets in the Maze2d environments (see Appendix F.4 for more training details). Then we adopted an kcypoint-conditionee TD3BC ap- proach, where the state of each step treats the keypoint as subgoal. Compared to the original TD3BC, we concatenated the predicted goal to the inputs of both the policy and value networks. The results are shown in Table 9. TD3BC achieves a significant improvement over baseline because the agents are able utilize the information provided by the keypoint as a subgoal.
D ONLINE RL TRAINING WITH REAL HUMAN FEEDBACK
Reploy Buffer
Feelback Buffer
query
Online Interaction
relabol
vispalize
Online Asynchronous Labeling Process
provide feedback
Figure 8: Online Asynchronous Labeling Process of Uni-RLHF. We show Uni-RLHF can learn from asynchronous human feedback and annotators interact with agents through remote web inter- face.
As shown in Fig. Uni-RLHF accommodates the online asynchronous reinforcement learning train- ing process. Much of the previous work (Christiano et al., 2017; Lee et al., 2021a; Park et al., 2022) assumes that online RL training and annotation (even with synthetic labels) are performed alternately, which can lead to low training efficiency for practical applications. Uni-RLHF builds asynchronous pipelines that allow training and annotation to occur simultaneously. It employs
21

---

Published as a conference paper at ICLR 2024
Time
Figure 9: Multiple front flip demonstration of walker trained via online RLHF. We show two random case of walker, which is trained using 200 queries of real human feedback. We refer to our homepage for more video visualizations.
sampler to extract data from the replay buffer, which is then stored in the feedback buffer as pending trajectory samples for annotation. Then various annotators can label these samples through the in- terface without interrupting the training process of the agents. When the number new annotations reaches predefined threshold, the reward function undergoes an update, thus relabelling the data stored the replay buffer.
We validate the effectiveness of the online mode of Uni-RLHF, which allow agents to learn novel behaviors where suitable reward function is difficult design. We demonstrate the behaviour of multiple continuous front flip based on the walker environment as shown in Fig. 9. For enabling multiple front flip skill, we train the SAC (Haarnoja et al., 2018) agent with online setting following Lee al. (2021a). First, we pre-train the exploration policy with an intrinsic reward for the first 2e4 steps to collect diverse behaviors. Collecting a greater diversity of trajectories helps to enhance the value of initial human feedback. Then, every le4 steps, we randomly sample pairs of trajectories from the replay buffer into the feedback buffer to wait for the annotation. The annotators free label without interfering with the walker training, and every 50 new labels will retrain the reward model and relabel the reward in the replay buffer. Finally, we give totally 200 queries of human feedback for walker front flips experiments and observe that walker master the continuous multiple front flip fluently. The other details and the hyperparameters are the same as Lee et al. (2021a).
E FULL RESULTS ON THE D4RL
We report complete average normalized scores on the D4RI, datasets. The scores are from the final 10 evaluations (100 evaluations for the antmaze domain) with 3 seeds, representing mean # std. "-r", "-r","-m"," "-m-""""" and "-m-e" is short for random, medium, medium-replay, and medium-expert, respectively. IQL results are presented in Table 10, CQL results are presented in Table 11, and TD3BC results are presented in Table 12.
F DETAILS OF STANDARDIZED FEEDBACK ENCODING FORMAT
F.1 COMPARATIVE FEEDBACK
We define segment sequence of time-indexed observations (Sk, ...y 8k+ 11 with length H. Given a pair of segments (o,o"), a human teacher gives feedback indicating which segment is preferred, so the preference Ycomp could be indicated の a!. o² > σ0 or o² σ", which represents equally preferred. We encode the labels Yecmp as Yeomp E (11,0), (0, (O.)))))))) and deposit them as triples (a (ewan)) in the feedback Subsequently, we treat the preference for trajectories as the training objective to optimize the reward function Pus- Following the the Bradley- Terry model (Bradley & Terry, 1952), we model human preference based on Fr as follows:
exp a)))
Polo x =0 Eie(o.1) exp(E, Poi (s), a)))
(3)
22

---

Published as a conference paper at ICLR 2024
Table 10: Average normalized scores D4RL datasets using IQL.
IQL
Dataset
Oracle
ST-MLP
ST-TFM
CS-MLP
CS-TFM
walker2d-m
80.91 = 3.17
73.7 = 11.9
75.39 a: 6.96
78.4 a 0.5
79.36
walker2d-m-r
82.15 = 3.03
68.6
60.33 10.31
67.3 4.8
56.52
walker2d-m-c
111.72 0.86
0.
109.16
109.12
hopper-m
67.53
51.8
37.47 10.57
4.2
6..81
hopper-m-r
97.43
70.1 14.3
21.52
8.11
22.65
hopper-m-e
107.42
1.8.7
109.16
94.3
111.43
halfcheetah-m
48.31 0.22
4.10
0.11
4.2
43.24 0.34
halfcheetah-m-rr
44.46
4.6
38.0
39.49 0.41
halfcheetah-m--e
94.74
9..2
+ 92.91
91.0
92.20
mujoco Average
81.63
73.7
69.9
73.29
69.09
antmaze-u-v2
77.00 # 5.52
71.59 # 7.21
74.67 # 5.86
74.22 + 4.22
68.44 4,96
antmaze-u-d-v2
54.25 = 5.54
51.66 14.04
59.67 + 5.03
54.60 4.78
63.82 = 5.60
antmaze-m-p-v2
65.75
71.67 # 5.77
72.31 4.10
65.25 4.10
antmaze-m-d-vv
73.75 #: 5,45
65.74 7.01
à 66.00
62.69
64.91 # 4.41
antmaze-l-p-v2
42.00 4.53
40.79 4.89
43.33
49,
8.98
44.63
antmaze-l-d-v2
30.25 3.63
49.24 6.03
29.67 = 12.66
21.97 11.87
29.67 9.39
antmaze average
57.17
58.91
57.67
55.94
56.12
pen-human
78.49 = 8.21
50.15#15.81
63,66 20.96
57.26=28 86
60.07+10.45
pen-cloned
83.42 # 8.19
59.92 1.12
64.65 # 25.4
62.94=23 .4444
62.26+20.86
pen-expert
128.05 9.22
132.8524.60
127.29 11.38
120.15214.78
122.42+11.9
dour-human
3.26 1.83
3.46
6.8 1.7
5.0524.34
3.22+2.45
door-cloned
1.75
-0.08
-0.06
0.10:0.0118
-0.01640.082
door-expert
106.65
105.35=0.37
105.05
105.7240.20
105.00 0.18
hammer-human
1.79
1.43 1.04
2.04
1,03=0.13
0.54 0.41
hammer-cloned
1.50 0.69
0.70 0.33
1.87
0.67=0.31
0.73+0.21
hammer-experr
128.68 + 0.33
127.420.17
127.36 0.54
91.22-62.33
126.5 = 2.07
adroit average
59.43
53.46
55.39
49.33
54.08
Table 11: Average normalized scores on D4RL datasets using CQL
CQL
Dataset
Oracle
ST-MLP
ST-TFM
CS-MLP
CS-TFM
walker2d-m
80.75 3.28
76.9+2.3
75.62 2.25
76.0 0.9
2.42
walker2d-ma m-r
73.09 t: 13.22
-0.3+.00
33.18
20.6 36.3
1.82
walker2d-m-e
109.56 0.39
108.920.4
108.83 0.32
92.8
98.
hopper-m
59.08 + 3.77
57.1+6.5
44.04
54.7
63.47
hopper-m-r
95.11 + 5.27
2.120.4
2.08 + 0.46
1.8
$2.97 = 13.04
hopper-m-e
99.26 10.91
57.5+2.9
57.27 + 2.85
57.4
$7.05 + 4.83
halfcheetab-m
47.04 0.22
43.9#0.2
43.26 + 0.25
4.1
43.5 0.24
halfchectah-m--r
45.04 0.27
42.82..3
+.173
4.1
40.97 1.48
halfcheetah-m-ee
95.63 0.42
69.0+7.8
63.84+7.89
62.7
64.86
mujoco Average
78.28
50.9
52.09
50.14
55.65
antmaze-u-v2
92.75 # 1.92
93.71 10:+1... 1.99
91.71 = 2.89
63.95 10.11
91.34
uatmaze-u-d-v2
37.25 3.70
34.05 28.05
25.11 10.44
6.77
22.75 8.67
antmaze-m-p-v2
65.75 11.61
7.98 + 13.07
62.39 + 5.64
6.526
64.67 7.32
antmaze-m-d-v2
67.25 3.56
17.50
63.27 5.61
46.95 11.91
69.74 2.12
antmaze-l-p-v2
20.75 -0.754..22
1.70
18.45
44.45 +
+
antmaze-l-d-v2
20.50 13.24
20.88 8.55
12.39 + 3.76
0.00 0.00
33.00 + 5.57
antmaze average
50.71
29.3
45.55
37.06
50.14
pen-human
13.71 16.98
9,80&14.08
20.31+17.11
6.53
23.77+14.23
pen-cloned
1.06
3.82 8.51
7.71
2.88+8.42
3.18+7.55
pen-expert
.1.41
138.3445.23
119.6016.45
121.14412.63
122.41+10.92
door-human
5.53
5.89
4.92 6.22
10.31+6.46
8.81 +
door-cloned
-0.33
0.00
-0.34 + 0.00
-0,3410.00
-0.44 0.00
door-expert
-0.32
103.90+0.80
103,3240.22
102.63+4.92
103.1545.88
hammer-human
0.14
0.85 + 0.27
1.11 + 1.14
0.70+0.18
1.13 + 0.34
hammer--loned
0.00
0.28 + 0.01
0.29 + 0.01
0.28+0.01
0.29+0.01
hammer-expert
0.00
120.12
120.66+5.67
117.65+4.71
117.60+2.56
adroit average
42.39
41.57
40.2
42.22
23

---

Published as a conference paper at ICLR 2024
Table 12: Average normalized scores D4RL datasets using TD3BC
TD3BC
Dataset
Oracle
ST-MPP
ST-TFM
CS-MLP
CS-TFM
walker2d-m
80.91 = 3.17
86.0+1.5
0
26.3
84.11
walker2d-m-r
82.15 3.03
82.8+15.4
2.22
47.2
61.
walker2d-m-e
111.72 0.86
110.420.9
110.13
59.1
110.75
hopper-m
60.37 # 3.49
58.647.8
62.89 20.53
40.0
99.42
hopper-m-r
64.42 #: 21.52
44.4
24.35 7.44
25.8 12.3
4
hopper-m-e
101.17 9:07
103.728.3
104.14 1.82
97.4 15.3
91.18 27.56
halfcheetah-m
48 0.18
50.3+0.4
48.06 + 6.27
3.38
46.62 * 6.45
halfcheetah-ns
44.84 0.59
44,2+0.4
36,87 0,48
3..
29.58 5.64
half
90.78 6.04
94.1+1.1
78.99 6.82
7.1
80.83 3.46
hujoco Average
76.45
74.8
63.33
51.86
71.76
anlmaze-u-v2
70.75 + 39.18
93.51 2.32
92.90 3.08
90.25
92.30 + 1.90
antmaze-u-d-v2
44.75 + 11.61
73.19 4.03
36.45 + 5.24
51.88 = 6
59.58 = 22.59
antmaze-m-p-v2
0.25 0,43
0.21 + 0.33
0,00 # 0.00
0.25 + 0.34
0.39 = 0.57
antmazc-m-dd-2
0.25 0.43
3.33 + 1.27
0.39
0.10 0.14
0.32 0.41
antmaze-l-p-v2
0.00 0.00
0.07 + 0.12
0.00
0.00 0.00
0.00 0.00
antmaze-l-d-v2
0.00 0.00
0.00) 0.00
0.00 0.00
0.00 0.00
0.00 0.00
antmaze average
19.33
28.38
21.62
23.75
25.43
pen-human
-3.8842,10
-3.94+0.27
-3.94+0,24
-3.71=0.20
-2.81+2.00
pen-cloned
5.1345.28
10.84 + 20.05
14.52 = 16.07
6.7147.66
19.13 = 12.21
pen-expert
122.53 + 21.27
14.91
34.6246.79
11.45+23.09
30,28
door-human
-0.13+00.07
+..32
-0.32+0.03
-0.33+0.01
-0.32+0.02
door-cloned
0.29+0.59
0.00
-0.34+0.00
-0.3420.00
-0.34+0.00
door-expert
+ 0.01
-0.3440.00
0.03
-0.3420.00
-0.34.0.00
hammer-human
1.02 0.24
1.00 0.12
0,96
0.46+0.07
1.02=0.41
hammer-cloned
0,25 + 0.01
0.25 + 0.01
0.20:00.5
0,4520.23
0.35+0.17
hammer-ex
3.11 + 0.03
2.22+1.557
3.13 + 0.02
2.13+1.69
3.03 # 0.27
adroit average
14.22
2.64
5.4
1.83
5.62
where o² > oa denotes the event that segment is preferable to segment j and (s at) o'. order align the reward model with human preferences, the update of T transformed into minimizing the following cross-entropy loss:
L(bb D) = - Eoo 1100 a ) ((0) log P P > o'1 + P > -o
(4)
where The term y0) and y()) represents the corresponding value in 3eomp. During the experimental process, the agent can be trained in parallel with the reward model, undergoing iterative updates. It worth noting that Yeomp can simply extended degree to which A is better than B (Bai et al., 2022). For example, Dcomp (0.75, 0.25) represents that A is better than B with a smaller degree than (1, 0). Also, the expansion method can simply be trained along the Eq. (4) without any changes.
finer-grained division of labels including the
F.2 ATTRIBUTE FEEDBACK
o First, we predefine a {ar,................. "e......,00 .ak] represents a set of & relative attributes of the agent's behaviors. Given a pair of segments (o o-), The annotator will conduct relative evalua- tion of two segments for each given attribute, which encoded as Yaur (y1. yk) and E (), (0, 1), (0.5.0.5) 0.5)]. Then we establish a learnale attribute strength mapping model (G) = a = fe
yax] E [0,1] 13k that maps trajectory behaviors to latent space vectors, where ah indicates the corresponding attribute. Specifically, the probability that annotators believe that segment のº exhibits stronger attribute performance on a; defined as follows:
exp Co (ao);
pau Jo o1] ESE(1 exp 6 (α,);
where Ço (7); represents i-th element of Ep. To facilitate the multi perspectveee human intention of the attribute mapping model, train čg using modified objective:
L(Ca D)
Σ
Σ
1i(1) log po: (0o X a1) + (2) log pa (01 > aa] (6)
1E(1, (DDDOL..EED
24

---

Published as a conference paper at ICLR 2024
Once such an attribute strength mapping model is established, normalize the latent space vectors Ço (C3 ?g. (0.mii) (S8,ma m0 We then derive evaluations of the trajectories based on the target attributes Vojs required for the task (relative strength of desired attributes, constrained between 0 and 1), simplifying it into a binary comparison:
y( o1 upp =
(1,0), if [[( - uppel la 1 (())()))
(01) (0,1), if (ol) op 11:0 (01)
opt
Subsequently, we establish a conditional reward model Fg (St, ar, v) using the pseudo-labels ob- tained from the transformation. We modified equations Eq. (3) and Eq. (4) to distill the reward model follows:
exp(E, Fes (s), a,, va))
Polot + 0' Eie(o,1) exp a())
(8)
L(v, D) = -B of Meccop) ) ((0))og P [o"h > v + log P [ot > 0 b
(9)
F.3 EVALUATIVE FEEDBACK
6 Following the RbRL (Rating-based Reinforcement Learning) (White et al., 2023), define n discrete ratings for trajectory levels, such as good, bad and average. Therefore we receive cor- responding human label (a, (eeva). where Ileval E {O, 1} is level of assessment. Let return R(or Σ, (st, ).). determine rating categories =: Ro R₁ .... S R, 1 divided by evaluation distribution. Based on the boundary conditions, we can estimate the probability Po (1) of each segment being assigned to the i-th rating, defined as:
Riti)
(i = Σjzo (R))) RRy1))
(10)
Then multi-class cross-entropy loss train the reward model:
L(v, D) - -E (o.
Σ v() (o)
(11)
F,4 KEYPOINT FEEDBACK
We annotate key state time steps Ykey = (t,, tt.....tt.. f from segment a, where n is the number of keypoints. Subsequently, employ this feedback train a keypoint predictor 9o and use histor- ical trajectories to inference key states, serving as guidance for the agent. Our predictor (-(-) aims to predict the most adjacent keypoints and is trained in supervised manner with regression loss:
C(o.D) = (B )~D Σ (
(s)) - 11
(12)
where (key E Jkey and Stay is the keypoint closest to the current s.. Then the policy are trained with inputs that consist of the states and keypoint predictions Ts( 9 (st)). As a result, the agents are able to obtain additional explicit goal guidance during training process.
additional concatenation, marked as
F.5 VISUAL FEEDBACK
We define the feedback as (St, Dvisual, where St E and Jvissull [Boxi..... Boxm Box; the bounding box position (æmin: Yminy Tmax: Mmax). Following the Liang et al. (2023), we are able to convert the annotated visual highlights into saliency map using saliency predictor 9a- There have been many saliency prediction model in the literature, e.g. PiCANet (Liu et al., 2018), Deepfix (Kruthiventi et al., 2017). We can then use the human feedback labels to bootstrap the saliency predictor ga go)). Given an RGB image observation o, H saliency encoder map the original input the saliency map my [0, H which highlight important parts of the
25

---

Published as a conference paper at ICLR 2024
image based feedback labels. There several methods of aggregating additional saliency visual representations. The CNN encoder and the policy are jointly trained with inputs (or, m4) that consist of the RGB images and saliency predictions as an additional channel. Another possible approach is use the RGB image x predicted saliency map as an imput. Utilising saliency information enables more effective focus on relevant and important representations in the image tasks.
G SOME CHALLENGES AND POTENTIAL FUTURE DIRECTIONS
We outline some challenges and potential future directions as follows:
Performance Metrics: Although we performed evaluation experiments of the RLHF method, is still evaluated using predefined reward signals, which may not be available in the application. How quantify human intentions and explain real-world factors is an important research direction.
Estimations of Human Irrationality and Bias: Irrationality in human feedback can cause great difficulties in modeling feedback models (Lee al., 2021b; Wang al., 2022b; 2017; 2018; 2020), and future research may focus on reducing the negative effects of noise, e.g. noise label learn- ing (Song et al., 2022), and strong model constraints (Xue et al., 2023).
Imperfect Reward Correction: Modelling environmental reward through human feedback is easily caused by partially correct rewards (Li et al., 2023). How to second-bood reward modeling through reward shaping (Sorg et al., 2010) and domain knowledge is area ripe for research.
Reinforcement Learning with Multi-feedback Integration: When multiple feedback types are labeled, it is extremely interesting to jointly apply multiple feedback modalities in the same task or to compare different feedback performances.
H INSTRUCTIONS FOR CROWDSOURCING ANNOTATORS
In this section, we list the full instructions for each feedback mode for crowdsourcing. In order to improve the consistency of all participant's understanding of the task objectives, for each task, the expert performed 5 annotations as examples and gave detailed process of analysis. Note that in Atari environments, also ask the annotators play for at least ten minutes to better familiarize themselves with the task goals.
H.1 COMPARATIVE FEEDBACK
The crowdsourced annotators will watch each pair of video clips and choose which one of the videos is more helpful for achieving the goal of the agent according to the following task descrip- tion. If the annotators cannot make a deterministic decision, they are allowed choose equally.
Options:
(1) Choose the left side of the video on the left is more helpful for achieving the goal of the agent.
(2) Choose equal if both videos perform similarly.
(3) Choose the right side of the video on the right is more helpful for achieving the goal of the agent.
Walker2d--ecdium-22 v2, Walker22--mcdiuum-rplaaa-
Description:
What you see is two videos of a robot walking. The goal is to move as much to the right possible while minimizing energy costs.
If robot is about to fall or walking abnormally (e.g., walking on one leg, slipping), while the other robot is walking normally, even the distance traveled is greater than the other robot, should be considered a worse choice.
you think both robots are walking steadily and normally, then choose the better video based on the distance the robot moves.
26

---

Published as a conference paper at ICLR 2024
If both videos show the robots almost falling walking abnormally, then the robot that maintains a normal standing posture for a longer time or travels a greater distance should be considered the better choice.
If the above rules still cannot make you decide the preference of the segments, you are allowed select equal options for both video segments.
2o Hopper-necdiun--eplayy-22 Hopee
Description:
What you see is two-part video of jumping robot. The goal is move as far to the right as possible while minimizing energy costs.
two videos are jumping steadily and normally, choose the video based on the distance the robot moves.
both robots
one robot is jumping normally while the other robot has an unstable center gravity, unstable landing, is about to fall, consider the robot with normal jumping as the better choice.
If both robots in the videos have experienced situations such as unstable center of gravity, unstable landing, or about to fall, consider the robot that maintains normal jumping posture for longer period moves greater distance as the better choice.
If the above rules still cannot make you decide the preference of the segments, you are allowed select equal options for both video segments.
Halfchhetah--mediun-222. Haffehheeahh-nedddun-rrraaa-- Hatchheettannneeeeeeeeeeerrrr
Description:
What you see is videos of the running of the cheetah robot. The goal is move as much as possible to the right while minimizing energy costs.
If both robots are rumning steadily and normally the two videos, choose the better video based on the distance the robot moves.
one robot is running normally, while the other robot has an unstable center gravity, flips over, or is about to fall, consider the robot running normally as the better choice.
If both robots in the two videos have experienced abnormalities, consider the robot that maintains normal running posture for longer time moves greater distance as the better choice.
If the above rules still cannot make you decide the preference of the segments, you allowed to select equal options for both video segments.
Antzaze-umaze-v2,
diverse--ddvrse-e22 mediaz play-v2, Antmaze- latmaze-large-llay-eea Antmare-large-diieree--
medium-diverse-v2.
Description:
videos of ants navigating maze. The objective is reach the red target position as quickly as possible without falling down.
What you seeing
If both robots are walking normally and making progress toward the target, choose the better video based on the distance to the endpoint.
one ant is walking normally while the other ant falls down, stops, or walks in the wrong direction of the target, consider the ant robot walking normally as the better choice.
If both ants are in abnormal states and there is a significant difference in the distance to the target, consider the ant closer to the target as the better choice.
If the above rules still cannot make you decide the preference of the segments, you are allowed select equal options for both video segments.
27

---

Published as a conference paper at ICLR 2024
Pen-human-vi, Pen-cloned-vl
Description:
What you see is dexterous hand whose goal is to manipulate the pen to align its direction and angle with the target pen appearing nearby.
If in segment, the angle of the pen in hand eventually becomes consistent with the angle of the target pen nearby, while in another segment, the pen hand has not yet adjusted to the angle of the target pen, then the former is considered a better choice.
both segments have the pen in hand in the process of normal adjustment to the target angle, both are in abnormal states (stuck, rotating in the wrong direction, etc.), the segment with a direction and angle closer to the target pen is a better choice.
If the pen in hand is already almost aligned with the direction and angle of the target pen in one segment, while in another segment, the pen in hand is still in the process of adjustment, then the segment that has already achieved the goal is considered better choice.
If the above rules still cannot make you decide the preference of the segments, you are allowed select equal options for both video segments.
Door-human-vl, Door-cloned-yl
Description:
What you see is dexterous hand, the goal is to release the door lock and open the door.
When the door contacts the door stop at the other end, the task is considered complete.
the two scenarios, the angle the door is larger (the maximum angle being the contact with the door stop at the other end), it should be considered better.
In the two scenarios, if the angle at which the door is opened is the same, then the scenario where the door is opened faster should be considered better.
If the above rules still cannot make you decide the preference of the segments, you are allowed select equal options for both video segments.
Hammer hhmmnaer-umma-v/// v1, Hammer-cloned-e
Description:
What you see is dexterity. The goal is to pick up the hammer and drive the nail into the plank. When the entire length of the nail is inside the plank, the task is successful.
In the two segments, the deeper the nail is driven into the plank, the better it should be considered.
In the two segments, if the nail is driven into the plank to the same degree, the scenario that completes the action of nailing the nail smoothly and seamlessly should be considered better.
the two segments, if one segment has nails that are consistently fully hammered in, while the other segment has nails gradually being hammered into the wooden board, then the first one is considered the better choice.
In the two segments, if a segment experiences anomalies such as the hammer slipping out hand, is considered the worse choice.
If the above rules still cannot make you decide the preference of the segments, you allowed to select equal options for both video segments.
Boxing-medium-v0
Description:
28

---

Published as a conference paper at ICLR 2024
You fight opponent in boxing ring. You score points for hitting the opponent. If you score 100 points, your opponent knocked out.
You play as a white boxer, the more times you hit the opponent and the fewer times you are hit by the opponent, the better choice it is considered.
To better understand the game, we recommend that you watch this video" and try playing the game for at least 10 minutes using the application".
Breakout-medium-vo
Description:
The dynamics are similar to pong: You move a paddle and hit the ball in a brick wall at the top of the sereen. Your goal is destroy the brick wall. You try break through the wall and let the ball wreak havoc on the other side. In two scenarios, scoring more should be considered better choice.
To better understand the game, we recommend that you watch this video and try playing the game for least 10 minutes using the application.
Enduro-medhum-vv0
Description:
You are racer in the National Enduro, a long-distance endurance race. You must overtake a certain amount of cars each day to stay in the race. In two scenarios, scoring more should be considered better choice.
To better understand the game, recommend that you watch this video and try playing the game for least 10 minutes using the application.
Pong-medium-v0
Description:
You control the right paddle, you compete against the left paddle controlled by the com- puter. You each try to keep deflecting the ball away from your goal and into your oppo- nent's goal. The segment of winning more and losing less considered better choice.
To better understand the game, recommend that you watch this video and try playing the game for least 10 minutes using the application.
Qhert-medium-v0
Description:
You are bert.. Your goal to change the color of all the cubes on the pyramid to the pyramid's 'destination' color. To do this, you must hop on each cube on the pyramid one time while avoiding nasty creatures that lurk there.
You score points for changing the color of the cubes to their destination colors by defeating enemies. two scenarios, scoring more should be considered better choice.
To better understand the game, recommend that you watch this video and try playing the game for least 10 minutes using the application.
29

---

Published as a conference paper at ICLR 2024
Cruise (Straight road cruising), Cut-in (Dangerous lane change avoidance)
Description:
What you see is two segments of videos showing autonomous vehicles driving on straight road with three lanes. The goal is reach the endpoint located on specified lane quickly as possible while ensuring safety. The following rules are prioritized in descending order:
If one vehicle reaches the endpoint while the other does not, the former is considered better. one vehicle goes out of bounds or collides while the other does not, the latter is considered better.
If there a significant difference in average speed between the two vehicles, i.e., differ- ence greater than 3m/s, the faster vehicle is considered better.
one vehicle consistently stays on the lane where the target point located while the other does not, the former is considered better.
there is no significant difference in both the driving time on the target lane and the average speed of the vehicles (less than 0.2m/s), it is allowed to choose equal options.
left-turn-cross (Intersection turning)
Description:
What you see is two segments of videos showing autonomous vehicles driving in simulated environment. The goal is to complete the turns as quickly as possible while ensuring safety and reaching the endpoint located on the designated lane. The following rules are prioritized in decreasing order:
If one vehicle reaches the finish line while the other does not, the former is considered better. one vehicle goes out of bounds or collides while the other does not, the latter is considered better.
the previous rules cannot determine a comparison, the one with faster average speed is considered better.
If there is no significant difference in the average speeds of the two vehicles (less than 0.2m/s), is allowed choose equal options.
H.2 ATTRIBUTE FEEDBACK
The crowdsourced annotators will watch each pair of video clips and determine the relative strength of the performance of the agents in each attribute based on the behavioral attribute defi- nitions. Select the equal option if the annotators believes that the agents behave consistently on an attribute.
Dakeerrol-Walker
Options:
Choose the left side if the left video has stronger attribute performance than the right video.
Choose equal if both videos perform consistently.
Choose the right side if the left video has weaker attribute performance than the right video.
Definitions walker attribute:
The speed which the agent moves the right. The greater the distance moved to the right, the faster the speed.
Speed:
Stride:
The maximum distance between the agent's feet. When the agent exhibits abnor- mal behaviors such as falling, shaking, or unstable standing, weaker performance should be selected.
Leg Preference: If the agent prefers the left leg and exhibits a walking pattern where the left leg drags the right leg, stronger performance should be selected. If the agent prefers
30

---

Published as a conference paper at ICLR 2024
the right leg and exhibits walking pattern where the right leg drags the left leg, weaker performance should be selected. the agent walks with both legs in a normal manner, equal should be selected.
The torso height of the agent. If the average height of the agent's 's torso during movement higher, stronger performance should be selected. Conversely, weaker performance should be selected if the average height is lower. which is higher, select equal.
Torso height:
difficult to discern
Humanness:
The similarity between agent behavior and humans. When the agent is closer human movement, stronger performance should be selected. When the agent exhibits abnormal behaviors such as falling, shaking, or unstable standing, weaker performance should be selected.
H.3 KEYPOINT FEEDBACK
Mazeed-medium-yy v1, Maze2d-large-vl
Instruction:
Crowdsourced annotators will watch a video clip and accurately label the keypoints in the video based on the task description. keypoints are frames in the video that contain important information or significant scene transitions, such as the appearance a person, movement of object, key turning points, camera switches. Please watch the video carefully and accu- rately label all the keypoints using the provided tool the time point where each keypoints appears.
Description:
The maze2d domain requires 2D agent learn to navigate in the maze to reach target goal and stay there. You should mark up several keypoints, mainly involving key movements, turns, and other dynamics. If you think the entire video is not helpful for completing the task, you can choose skip
I DETAILS OF ANNOTATED FEEDBACK DATASETS
We establish systematic pipeline of crowdsourced annotations, resulting in large-scale annotated datasets comprising more than 15 million steps across 30 popular tasks. Documentation of the datasets, including detailed information on environments, tasks, number of queries, and length of queries can be found in the following Table 13.
31

---

Published as a conference paper at ICLR 2024
Table 13: Annotated Feedback Datasets details,
Domain
Env Name
Queries Num
Queries Length
FPS
walker2d-m-v2
2000
200
50
walker2d-m-r-vv2
2000
200
50
walker2d-m-e-v2
2000
200
50
hopper-m-v2
2000
200
50
Mujoco
hopper-m-r-v2
2000
200
50
hopper-m-e-v2
2000
200
50
halfchectah-mm-v2
2000
200
50
halfcheettah-m---v2
2000
200
50
halfcheetah-mmee-22
2000
200
50
antmaze-umaze-vv2
2000
200
50
anthaaz--umazz-diivrrsee2
2000
200
50
antmaze-nnedium-plaa--22
2000
200
50
Antmaze
antnaze-mediun-ddivers--22
2000
200
50
antmaze-large-plaa--2
2000
200
50
antmaze-large-diverss--2
2000
200
50
pen-human-vl
2000
50
15
pen-cloned-vl
2000
door-human-vl
2000
50
15
Adroit
door-cloned-vl
2000
50
15
hammer-human-vi
2000
50
15
hammer-cloned-vl
2000
50
15
cutin
2000
50
15
Smarts
left_c
2000
50
15
cruise
2000
50
15
breakout-m
2000
40
20
boxing-m
2000
40
20
Atari
enduro-m
2000
40
20
pong-m
2000
20
qbert-m
2000
40
20
Deepmind Control
walker (attribute feedback)
4000
200
50
32

---

Published as a conference paper at ICLR 2024
1.0
0.6
0.1
Time
1.0
0.6
0.1
1.0
0.6
0.1
1.0
0.6
0.1
1.0
0.6
0.1
1.0
0.6
0.1
1.0
0.6
0.1
1.0
0.6
0.1
1.0
0.1
1.0
0.6
0.1
Figure 10: Video clips of different humanness attribute levels. From left to right the target attributes are set to [1.0, 0.6, 0.1].
33


Reviews:
[{"review_id": "AGzVOBphhB", "reviewer": "Reviewer_3bfZ", "summary": "The paper introduces a new framework to collect human feedback for RLHF. The framework allows for five different types of feedback to be collected with a convenient user interface, depending on the task at hand. A dataset of feedback of three of these types is collected using crowdsourced human workers. This dataset is then used to evaluate multiple existing offline RLHF approaches, where a trained reward model is used to label trajectories from a dataset, and then an offline RL algorithm is used to produce a policy. The evaluation shows that the collected data for comparative and attribute feedback is of good quality, and allows for the learned policy to perform at a quality comparable to a policy trained from hand-crafted reward models. In some cases the policies trained from human feedback outperform those trained from hand-crafted reward models.", "questions": "- In table 2, the results are normalized against an \"expert\", as the formula in the beginning of Section 4.1.1 shows. How is the expert trained? It is interesting that some of the methods in the table outperform the expert.\n- I found the attribute training model in Appendix E.2 confusing. The learned log-preferences $\\hat{\\zeta}^\\alpha$ in eq. (7) probably also need the subscript $\\theta$.  The relative strengths of attributes $v^\\alpha_{opt}$ are provided as hyperparameters. These strengths are in $[0,1]$. Why, then, in eq. (7) the attributes are checked for being close to $\\hat{\\zeta}^\\alpha$, which is supposed to become the probability of preference according to the respective attribute only after the softmax operation (5)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In my view, the main strength of the paper is the open-sourced dataset and the tool for feedback collection. These will bring great value to the community and allow for easy benchmarking of offline RLHF methods. The extensive evaluation on D4RL is another strong point of the paper. It will provide a reasonable baseline for future offline RLHF approaches. Finally, the dataset with collected attribute feedback can be used for multi-objective learning, an area where fewer datasets in general are available.", "weaknesses": "- The claim for a \"standardized feedback encoding format along with corresponding training methodologies\" in Section 3.2 might be overly ambitious. Most importantly, the training methodologies are only provided in the appendix for two out of the five feedback types. Comparative feedback interface also does not allow the user to indicate the strength of preference, as done e.g. in the Anthropic's dataset for language model alignment [1]. The extension of the methodology in Appendix E.1 to this type seems straightforward (introduce y=(0.75,0.25), for example), and it could be beneficial to mention this.\n- Related to the previous point, the paper provides too few details on the way Atari experiments were performed. Appendix G and Section 4.1.1 imply that comparative feedback was collected, but Figure 5 (d) in the appendix -- that visual feedback was used instead. The highlighting in Table 8 should be explained. My guess is that the best of the ST and CS labels is highlighted.\n- There is no dataset or benchmarking for evaluative feedback, hence it is hard to assess the usefulness of this part of the system. It is unclear whether data for keypoint and visual feedback on the Atari environments was collected. I believe that the datasets for comparative and attributive feedback are already a good enough contribution, and an interface for other feedback types is a nice-to-have extra, so this point does not make me give this paper a negative rating.\n- Star annotations in Table 2 are incomplete. As I understand it, higher is better in the entire table. If that is the case, stars that should mark that the method performs better than the oracle are missing in several places: (hopper-m, CQL, CS-TFM), or (antmaze-u-d, IQL, CS-MLP). There are more missing stars in the table.\n- Blue annotations in Table 2 are also significantly wrong, if I understand them correctly. Blue should mark the methods where crowdsourced labels (CS) show better results than the synthetic labels (ST) for the corresponding method. Then, for example, (walker2d-m-r, IQL, CS-MLP) should not be colored in blue, since (walker2d-m-l, IQL, ST-MLP) shows a better score (109.8 > 109.4). I counted 10 instances of such mislabeling in the IQL group alone. This is especially important since the paper claims that \"the performance of models trained using crowd-sourced labels (CS) tends to slightly outperform those trained with synthetic labels (ST) in most environments, with significant improvements observed in some cases.\" Once the blue labels are revised, it is questionable whether this claim still holds. Other offline RL approaches (CQL and TD3BC) show that CS labels are better more consistently, but these approaches perform worse than IQL, so the results for them are correspondingly less important.\n- Some CS-TFM results are missing from Table 2. I did not find the explanation for this, maybe I missed it? It would be helpful to see these results in the final version, to better assess the claim of the paper that \"CS-TFM exhibits superior stability and average performance compared to CS-MLP\".\n- For the SMARTS experiment in Section 4.1.2, the paper claims that \"the best experimental performance can be obtained simply by crowdsourcing annotations, compared to carefully designed reward functions or scripted teachers.\" I would say that from the three tasks used one cannot conclude that carefully designed rewards perform worse than crowdsourced annotations. In table 3, the former outperforms the latter on two out of three methods. The claim seems to be made based on average success rate, which only differs by 0.01, less than one standard deviation.\n- In Figure 4, speed and torso height are plotted against time. Every 200 steps, the speed attribute value changes (1 to 0.1 and back), so we see the respective changes in speed on the first plot. The relative strength of the \"torso height\" attribute, however, does not change (stays at 0.1), and the torso height parameters do not change much between the changes. It would be more interesting to see the results where the strength of torso height also changes, so that we can see that it influences the plot.\n- Generally speaking, RLHF is most useful in domains where only the humans understand the true reward. This is the case, for example, with \"helpfulness\" of an LLM, or with \"humanness\" of the gait of a walking robot. An important evaluation, then, is to see whether the human evaluators prefer the policies trained with an RLHF approach in terms of these hard-to-define measures. This paper, however, does not present such an evaluation. \"Humanness\" comparisons are collected as shown in Section 4.2, but it is never compared to a policy that is trained without taking humanness into account.\n\nOverall, these weaknesses do not seem to me to be strong enough to motivate a rejection. The dataset and tool contributions are strong, and the problems with baseline evaluations can be clarified in the final version of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g3ZuKK2HOO", "reviewer": "Reviewer_FdzJ", "summary": "Uni-RLHF is a comprehensive system for reinforcement learning with diverse human feedback. It includes an Enhanced-RLHF platform that supports multiple feedback types, environments, and parallel user annotations, along with a feedback standard encoding format. The system also provides large-scale datasets, offline RLHF baselines, and human feedback datasets for relabeling and human-aligned reward models.", "questions": "Please refer to weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Uni-RLHF provides a universal platform for RLHF that supports multiple feedback types, environments, and parallel user annotations. The system includes a feedback standard encoding format that facilitates the integration of diverse feedback types.\n2. Uni-RLHF provides large-scale datasets and offline RLHF baselines for evaluating the performance of RL algorithms with human feedback. The system also includes human feedback datasets for relabeling and human-aligned reward models, which can improve the efficiency and effectiveness of RLHF.\n3. Uni-RLHF can foster progress in the development of practical problems in RLHF by providing a complete workflow from real human feedback.", "weaknesses": "1. Uni-RLHF's large-scale crowdsourced feedback datasets may contain noise and bias, which can affect the performance of RL algorithms trained on these datasets.\n2.  The system's offline RLHF baselines may not be optimized for specific applications, which can limit their usefulness in practical settings.\n3. The system's reliance on human feedback may introduce additional costs and delays in the RL development process, compared to purely synthetic feedback.\nUni-RLHF is an engineering-focused system that emphasizes incremental improvements rather than groundbreaking innovations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "erePGJBnR5", "reviewer": "Reviewer_BKHb", "summary": "This work introduces Uni-RLHF, an eco-system for Reinforcement Learning with Human Feedback to facilitate the data collection with human annotators, the sharing of datasets and the RL alignment.\nIn particular, the annotation platform named Enhanced-RLHF supports various feedback types.\nThen, the paper investigates different offline RL strategies, and show that the reward models trained on their crowdsourcing labels lead to better performances than when using synthetic labels, and can approximate the Oracle rewards. This is done on motion control/manipulation tasks such as D4RL or Atari or Smarts. They aim for fair evaluation of RLHF strategies.", "questions": "- could you please clarify the differences with RLHF-Blender ? in which case we should use one or the other ?\n- How did you select the hyperparameters?\n- Have you applied any strategy to refine the dataset quality?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The work effectively succesfully presents a comprehensive system to deal with the data collection process with diverse human feedback types.\n- The motivation is clear and interesting: indeed, RLHF appears nowadays as a go-to strategy to ensure the reliability of AI systems. Therefore the proposed eco-system can be of interest to some researchers.\n- Crowdsourced labels are sufficient to approximate Oracle-based reward, showing the flexibility of RLHF even in those D4RL datasets.", "weaknesses": "- The main weakness is the limitation to control/locomotion tasks. More real world tasks/other modalities (such as text), and also larger architectures are required to make this eco-system more attractive.\n- The benchmark only compares offline RL, thus for example the de facto online strategy for RLHF (PPO) is not ablated.\n- The different query samplers are not ablated.\n- Only the comparative feedback is ablated. It would have been interested to compare the quality of the reward models.\n- While the authors do acknowledge this, providing a data cleaning procedure (or data filters) in the eco-system would be very positive to foster its applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)