Reward Learning from Multiple Feedback Types
Authors: Yannick Metz
Andras Geiszl
Raphaël Baur
Mennatallah El-Assady
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
RLHF
Machine Learning
Multi-Type Feedback
Primary area: reinforcement learning
TL;DR: We present a simulation framework and reward model implementations for diverse feedback of multiple types, and show their effectiveness

Abstract: Learning rewards from preference feedback has become an important tool in the alignment of agentic models. Preference-based feedback, often implemented as a binary comparison between multiple completions, is an established method to acquire large-scale human feedback. However, human feedback in other contexts is often much more diverse. Such diverse feedback can better support the goals of a human annotator, and the simultaneous use of multiple sources might be mutually informative for the learning process or carry type-dependent biases for the reward learning process.
Despite these potential benefits, learning from different feedback types has yet to be explored extensively.
In this paper, we bridge this gap by enabling experimentation and evaluating multi-type feedback in a wide set of environments. We present a process to generate high-quality simulated feedback of six different types. Then, we implement reward models and downstream RL training for all six feedback types.
Based on the simulated feedback, we investigate the use of types of feedback across ten RL environments and compare them to pure preference-based baselines. We show empirically that diverse types of feedback can be utilized and lead to strong reward modeling performance. This work is the first strong indicator of the potential of multi-type feedback for RLHF.

Published as a conference paper at ICLR 2025
REWARD LEARNING FROM MULTIPLE FEEDBACK TYPES
Yannick Metz! András Geiszl,, Raphaël Baur?, Mennatallah El-Assady²
'University of Konstanz, Germany
yannick met met 2 (uunni-- -oontiiknntnnn.... de 2ETH Zurich, Switzerland ageiszl@inf sthz.cc, maophael.bauz.. menna. mema.elassady/ea.. wtw.aiiieth...... ch
ABSTRACT
Learning rewards from preference feedback has become important tool in the alignment of agentic models. Preference-based feedback, often implemented as binary comparison between multiple completions, is established method acquire large-scale human feedback. However, human feedback in other contexts is often much more diverse. Such diverse feedback can better support the goals of a human annotator, and the simultaneous use of multiple sources might be mutually informative for the learning process or carry type-dependent biases for the reward learning process. Despite these potential beneñits, learning from different feedback types has yet to be explored extensively. In this paper, we bridge this gap by enabling experimentation and evaluating multi-type feedback in a broad set of environments. We present process to generate high-quality simulated feedback of six different types. Then, we implement reward models and downstream RL training for all six feedback types. Based on the simulated feedback, we investigate the use of types of feedback across RI. environments and compare them to pure baselines. We show empirically that diverse types of feedback can be utilized and lead strong reward modeling performance. This work is the first strong indicator of the potential of multi-type feedback for RLHF.
preference-bas
INTRODUCTION
Reinforcement learning from human feedback (RLHF) is a powerful tool to train agents when it difficult to specify a reward function or when human knowledge can improve training efficiency. Recently, using multiple forms of human feedback for reward modeling has come into focus (Jeon)
ct al., 2020 Ghosal et al. 2023 Ibarz et al. 2018 Brysk et al., 2022a Mehta & Losey 2022). Using diverse sources of information opens up several possibilities: (1) feedback from different sources allows for correcting potential biases in the data; (2) the feedback type can be adapted to a partic- ular task user based on preferences, knowledge state, or available input modalities; (3) agents can actively select an appropriate type of feedback during training to optimize learning (Jeon et al. 2020). However, there are only few empirical investigations on the characteristies of different feed- back types (Mehta & Losey| [2022] Ghosal et al. 2023 Bryık et al. 2022a). Even though there have been some efforts to unify different types of reward (Jeon al.] [220),00)) and attempts learn from a combination of two or three feedback types (Ibarz et al. 2018 Mehta & Losey) (022). the use of a larger number of different feedback types has not been fully explored. This paper addresses three key questions: (1) How can we define, model, and simulate different explicit types of human feedback consistently? (2) What characteristies do reward functions learned from these feedback types exhibit? (3) Can combine reward models from different feedback types for robust learning?
Contributions - To answer these, we contribute by (I) Implementing synthetic generation and reward models for six distinct human feedback types, along with joint-reward modeling approach based on a reward function ensemble (section 3). (2) Empirically investigating the effectiveness and complementarity of these feedback types (section 4), and (3) Analyzing the performance of joint training with multiple feedback types, highlighting the potential of learning from diverse human feedback in future applications (section 5).

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Reinforcement Learning from Human Feedback Using human feedback the sole or an additional source of reward information has gained traction in research (Ng et al. 20000] Knox & Stone 2009 Griffihh al. 2013 Christiano al. 2017]. especially for the alignment of large language models (Ouyang et al. 2022). This paper focuses on learning rewards solely from human feedback,
excluding approaches like interactive reward shaping (Knox & Stone 2009 Warnell et al., 2017).
So far, the generation of synthetic feedback has been primarily explored in the area of preferences (Bryık et al. 2022b) Christiano al. (017). In their foundational work for modern RLHF, (Christiano et al. (017)) utilize simple oracle model based on underlying ground truth reward, which uses the sum of rewards over segments to compute preferences. To account for human error, they assume fixed miss-labeling probability of 10%, implemented via preference label switching. (Bryık et al., 2022b) implement a comprehensive toolbox to simulate preferences with adaptable levels of noise/irrationalii and methods to collect data from humans. We extend these efforts into a more versatile multi-type feedback setting.
Synthetic Generation of Human Feedback -
Combining Multiple Reward Functions and Multi-Type Feedback Different types of feedback, ranging from ratings, demonstrations (Ng et al. 2000;; Abbeel & Ng, 2004). comparisons (Wirth al. 2017) interruptions (HadficldMMenell al.) (017)) or even language and narrations (Fish et al.
2018 Sumers et al. 2022b has been proposed. A wide range of different types of feedback can be interpreted via a common framework of eeward-rational choice (Jeon al.. 2020). There has been some work on combining multiple feedback types, most notably demonstrations and preferences Ibarz al. [2018); Bryık et al. (2022a). However, the authors used separate phases of demonstrations and preference-baset learning. Most similar in spirit is work by Mehta & Losey] who also train single reward model based on the reward rational choice framework. However, they restrict themselves to two (three) feedback types, including demonstrations and preferences (as well as stops).
Sumers et al. investigate evaluative, instructive, and descriptive language feedback in a simple bandit environment. We and more diverse and dynamic feedback.
work as extension of these efforts towards more complex environments
Recently, there has been some work on extending the space of possible feedback by presenting implementations for collection from human annotators (Metz et al. 2023] Yuan et al.) 2024). However, these works do not investigate the full implementation of diverse reward models and have not analyzed reward learning and RL from diverse feedback.
3 DEFINING AND SYNTHESIZING MULTI-TYPE FEEDBACK
We start our investigation by defining relevant feedback types, describing our approach to simulate feedback of these different types, and discussing the implementation of reward models.
Preliminaries - We assume a RLHF scenario with an agent that acts within an environment, observing states 8; S C R-V and performing actions 0, E A C RR The agents select actions according to policy m, which throughout training is optimized with respect to a learned reward function f S A + R. In RLHF, the reward function estimator is updated based on human feedback. In RLHF, humans often provide feedback over trajectories ç e E, with E being the set of all possible trajectories. Trajectories acting within the environment. Providing feedback for trajectories requires fewer human interactions, as human labelers do not need to label every state-action pair. Finally, in this paper, we refer to an expert policy Re. we is the policy that maximizes the true reward function, which is not observable in humans. We use TT H) = II+ ++ x (8t) as a shorthand for the probability of a policy acting over a segment.
sequences of states and actions, e.g., generated by the agent
3.1 FEEDBACK TYPES
In the scope of this study, decided six exemplary types of feedback motivated by existing work (Metz et al.) 2023). We briefly present how these six types of feedback can be defined individually.
2

---

Published as a conference paper at ICLR 2025
3.1.1 EVALUATIVE FEEDBACK
F1: Rating Feedback We define this type of feedback for cases in which a human gives numerical or otherwise quantifiable judgment of target, e.g., binary feedback
numeric score (Arzate Cruz Igarashi 2020). We define evaluative feedback as mapping from a target 7 to a scalar value in a value set fb V C R. Common choices for the value set can be binary score V == -1,1 or rating V = 1,,,, 10, which we choose for our experiments:
Feval = y E = (1,2,3,... 10)
F2: Comparative Feedback - Here, a user makes relative judgment, i.c., a pairwise comparison between two targets ranking of multiple targets. This type of preference-based feedback is widely used because is assumed that it is easier for humans to give comparative judgment compared to absolute scores (Wirth al. 2017). Comparative feedback is set of targets with associated ordering relation. We define an instance comparative feedback as preference over two segments:
FeomplE1 E2) = Ugb E [,)
3.1.2 INSTRUCTIONAL FREDBACK
F3: Demonstrative Feedback - The human is asked to provide a reference of optimal behavior that the agent should imitate (Ng al.) (0000). We generally assume that these demonstrations optimal w.r.t. the "internal" human reward function. A demonstration is not conditioned on any specific trajectory but rather independent and given for the entire trajectory space.
Fdemo [) = with { E~ Te
with T being final state (e.g., terminal environment state).
F4: Corrective Feedback - Here, the user has a trajectory showcasing imperfect behavior as reference and needs provide an improved trajectory. This can be done either implicitly, e.g., by pushing robot into a correct position (Mehta & Losey 2022), or explicitly via specifying better action. We define correction as:
Feorr Feorr(Eu.++)) = AEGCH with ELL Atat E64+
3.1.3 DESCRIPTIVE FEEDBACK
F5: Descriptive Feedback The definition of descriptive feedback is open, and could interpret any declarative statement about an MDP as descriptive (Rodriguez-Sanchez et al. 2023). We chose a formulation in which descriptive feedback is a qualitative judgment about features, compared to (Sumers et al., 2022a), i.c., information about "state-action pairs like these are..." As such, descriptive feedback should not inform about single trajectory but instead, serve as way to assign reward information about set of features:
Fdese S, A) uyb R
F6: Descriptive Preferences - As an extension to descriptions that mirror comparative and corrective feedback, which can be interpreted relative versions of evaluative and demonstrative feedback, respectively, we propose descriptive preferences. Descriptive preferences are similar to feature preferences proposed before Yuan et al. (2024).
Fres.preff ( C S. C A), ( C S c A)) = upb (,,
These six feedback types are a non-exhaustive selection human feedback, and many more can be imagined (Jeon al.) 2020 Kaufmann et al.) 2024 Metz et al. 2023). However, argue that our selection represents a set of fundamental, general, and explicit human feedback types. Many other types of human feedback are implicit (Kovač et al. 202 Kaufmann al. 2024). i.e., need to be reduced to an explicit reward signal eventually, y, or collect meta- ineta-information that support the interpretation of human feedback (Kaufmann al. 2024).

---

Published as a conference paper at ICLR 2025
Popon Models
Evaleativ Featbark
Grovnit-cpimaii with pulccy
State Action Choerding
Clockyyinnt
& FR
R X.Mennsl
Fw Sero
vy & Bgirgg)
Description Fauid)) (ptts)).aaa) plac),
Rdlettifffor
Siekig via 1ize
Demonstration
Descriptive Preferetce
swpp Co)
Correction
Kuting
Favele)
Bins HirlE)
(r Ta (()), agn/r(n) rtes))
Companison
Fa Gol (g)) G
VeodbarkDataree
Dp
Faages
Generation of Simulated Feedback of different types: Based on an existing expert model and rollout buffer, we generate six types of feedback, including ratings and comparisons, demos and corrections, as well as descriptions and deseriptive preferences.
Figure 1:
3.2 SIMULATING MULTI-TYPE HUMAN FEEDBACK
We base our experimental evaluation on simulated feedback of different types to enable larger- scale environments and ensure reproducibility. While there exists some work simulate individual feedback types (e.g., preference feedback for locomotion environments, Atari (Brown et al. (209bb) or language modeling (Dubois et al., 2024 pipelines
provide feedback for multiple feedback types has been limited so far (uuan et al. 2024 Metz et al., 2023). Yuan et al. describe five feedback types but only collected annotated datasets for two feedback types.
To tackle these challenges, we implement lightweight software library to generate feedback of six types mentioned above. The library fully interoperable with well-established RL frameworks such as Gymnavium (Towers et al. 2024), Stable Baselines (Raffin et al.) (202)), and Imitation (Gleave et al. 2022). provides utilities for synthetic feedback generation, reward models, and agent training.
3.2. GENERATION PROCESS
We describe a method to generate fixed labeled feedback datasets for reward model training. Our approach can also adapted for dynamic feedback generation with minimal modifications, /Figure I gives an overview of the synthetic generation process. We aim to create synthetic feedback for trajectory data based on the six abovementioned types.
As the foundation for feedback datasets, we sample segments across 20 model checkpoints from four expert RL models for each environment to ensure diversity, with random start indices and truncation at the end of episodes (Details in /susscction B.6).
Evaluative Feedback - We simulate ratings (I-10) based on discounted segment returns, creating a more realistic scenario than raw rewards. Using calibration set Ecal εu. &1. ..., implement equal-width binning of discounted returns, assigning scores decrementally from highest (10) to lowest (1) bins (distribution shown in Figure 1Tp. Online feedback generation compares incoming returns against this binning, with optional dynamic updates to maintain distribution consistency. For comparative feedback, derive pairwise comparisons from reward differences, excluding pairs with similar rewards (difference < 10% of return standard deviation).
For both corrective and demonstrative feedback, we utilize best-performing expert checkpoints. We generate expert trajectories from saved initial states using policy m.s selecting the
Instructive Feedback
samples from multiple expert models. For corrective feedback. we retain only corrections achieving higher discounted returns than original segments (comparison in (FFiuure 15). When state resets are unavailable, single-step corrections via expert action queries provide an alternative. Demonstrative feedback utilizes these high-quality corrections independently.
Descriptive Feedback We implement clustering using mini-batch k-means on concatenated state-action pairs. The number of clusters matches other feedback types' sample sizes, providing comparable trajectory space resolution. Each cluster's mean representative serves as the reward model target, with averaged rewards as description scores. Descriptive preferences are derived from reward differences between randomly sampled cluster pairs.
"The code available
feedbacckee
4

---

Published as a conference paper at ICLR 2025
3.3 INTRODUCING FEEDBACK NOISE
We implement a consistent noise scheme across feedback types based on human feedback rational- ity modeling (Jeon et al., 2020 Ghosal al. 2023). To ensure consistency, rather than directly in- troducing noise for each individual feedback type. we introduce noise into the underlying reward distribution, which then influences generated feedback.
For evaluative feedback, we add noise via truncated Gaussian distribution (u, lower, upper):
fbbe = fy + NT 0, B 10, 10)
Preference labels are flipped according to perturbed rewards, effectively reducing rationality by increasing incorrect preference probability. More nuanced than random flips, this approach primarily affects segments with similar rewards.
For descriptive feedback, the noise scale is determined by cluster reward ranges:
f r(c) (r a |max(re)) min((r)),, mán(re), max(re))
For demonstrative feedback, we perturb states and actions based on the observed standard deviations in the dataset:
{ = (s, + 1.Nr(p=0,,0=0 B NT = 0, 3 3
In Appenddxx B we detail the feedback dataset generation process and its characteristics to ensure reproducibility. We see the documentation of the generated datasets key part of our approach, as influences down-stream analysis. alternative regret-based scheme is discussed subsecttion B.5) Real human feedback may contain various biases, and encourage human-subject studies to determine appropriate values for different feedback types.
3.4 REWARD FUNCTIONS FROM FEEDBACK TYPES
For our baseline implementation of single reward functions, we closely follow the established methodology from RLHF: For feedback mapping to sealar values (Ratings and Descriptions), our reward model is optimized with an MSF-loss between the predicted reward & and annotated scalar feedback values ypb. For relative feedback (such as Comparisons, Corrections and Descriptive Preferences), we optimize the reward function with cross-entropy loss between predicted and annotated pairwise preferences.
(MSE(F) = Σ (p)o(E)
Σ f()))2 SEE
EED
(MLE(F)
Σ
log log +
+ PIg e'}.
with Plgt > {2] following the established Bradley- Terry model Bradley & Terry 1952
exp.3 3 Σ, f(ot, at)
P > = exp exp f(o,, a.) + exp 3 3, a3)
(3)
To train the reward model, we also model Demonstrative feedback preferences between the optimal demonstrations and trajectories based on random policies, i.e., random behavior, In our experiments, sampling against random behavior has been stronger than sampling against other sub-optimal rollouts. We follow the reward (rational) choice formalism (Jeon et al. 2020) for relative feedback. We discuss the formalization of our proposed feedback types under this framework Appendix A For our assume optimal reward, i.c., rationality coefficient of 3
baseline experiments, we will investigate learning from sub-optimal, more irrational feedback later. line with existing work (Christiano et al. 2017). optimize our reward model over trajectory segments &, with maximal length of 50 steps, except for descriptive feedback, which uses single states.
However,
In our analysis, pre-train reward models based trajectories collected from different checkpoints of an RL model, similar to (Brown et al.) 2019a).. This approach should lead to the model learning the reward distribution for representative set of state-action pairs. For details, refer to Appendix D)
5

---

Published as a conference paper at ICLR 2025
( MSE
(c,, Ey. ~), MLLE Comparisons
(Erand: Eis ~), LMLE
Ratings
Demonstrations (Se,11 Oe,1 (8e,2, (c.2)), ,,, <), LM
(Ci, Et 3), LALL
(8c, Gcs Tc), MMSE Descriptions
Corrections
Descriptive Comparison
Table I: A summary the reward model implementations for different feedback types, i.e., the reward inputs and loss function used to optimize the reward model.
4 INVESTIGATING THE EFFECTIVENESS OF DIFFERENT FEEDBACK TYPES
We investigate the effectiveness and characteristics reward learning across different feedback types, focusing on three key research questions: (1) How do various feedback types compare to sampled pairwise comparisons under both optimal and perturbed conditions? (2) What distinct characteristics emerge in reward functions and agent behavior across different feedback types?
(3) Does combining multiple feedback types offer advantages over single-type approaches? We explore this through an ensemble of feedback-type reward functions in RL agent training.
4.1 TRAINING SETUP
We evaluate approach on established benchmark environments: Mujoco locomotion environments (HalfCheetah-v5, Walker2d-v5, Swimmer-v5, Ant-v5, Hopper-v5, Humanoid-v5), a MetaWorld Yu al. (2019) environments (sweep-into-vy) and three discrete action space environment from High- wayEmv Leurent (2018): merge-vO, highway-fast-v0, roundabout-vO.
For expert model training, we use PPO (Schulman al.) 2017) and SAC (Haarnoja al. 2018). implementing tuned hyperparameters from available implementations (Rattin) 2020; Cobbe et al.
2019 Machado al. 2018). The reward model consists of a six six-layer MLP with ReLU activations.
Our feedback datasets comprise 10000 segments of 50 steps or 10000 descriptive clusters. We generate five separate feedback datasets control for dataset composition variations due random sampling and environment stochasticity. Reward models undergo supervised pre-training on labeled datasets (see Appendix D).
We evaluate learned reward functions using a simplified RLHF setup with single pre-trained reward model. Agent training follows a standard PPO configuration (details AAppeddix Fj. with results aggregated across five random seeds unless specified otherwise. As an additional baseline, we train policies via behavioral cloning on the collected demonstration datasets (details in subsection E.5)
4.2 RL WITH LEARNED REWARD FUNCTIONS FROM OPTIMAL. FEEDBACK
Initial reward model training across all feedback types shows effective learning with decreasing validation loss curves (detailed in AAppeddix D). All reward types demonstrate steady loss reduction even when noise is introduced, with specific effects discussed in the following section.
Figure 2 demonstrates that different feedback types achieve competitive performance across envi- ronments. Some feedback types can match the expert model performance for three environments with ground- truth feedback. However, performance is significantly worse in other environments like Swimmer-v5 or Ant-v5 than training with a ground- -trth reward function. Descriptive feedback and descriptive preferences generally yield the best results, though specific environments show excep- tions for instance, demonstrative feedback outperforms others in Swimmer-v5.
4.3 SENSITIVITY TO FEEDBACK VALUE NOISE
While initial analysis assumed optimal feedback relative to ground-truth rewards, real human feedback inevitably contains errors. We investigate this by introducing controlled noise into the feedback data, using comparable perturbation scales across feedback types.
Figure 3 jillustrates varying sensitivity to noise across feedback types. While the validation loss of reward function decreases across feedback types, downstream RL performance varies by feedback

---

Published as a conference paper at ICLR 2025
Sw Pere
AB-55
Re:povs
compartiin
CE reroaad
2000
4333
5000
2000
4000
250
306
3030
4200
3030
2435
3000
2000
2005
6053
(a) Episode Returns for all Mujoco-Euve Environments
roundatiout
highway fest.vo
metanoo sweep-intowv
Evalustive
Comparation
Conrectvve Descriotive Behav doningg
Max
50
Episode
(b) Episode Returns/Success rate for all Highway-Enn and Metaworld
Figure 2: RL from individual feedback type reward models. The area boundaries indicate minimum and maximum values out of the sampled runs. Results are averaged over five random seeds/ feedback datasets. For full training curves see subsectionn E.2
Rpward Mocel Volidadion Logs Curves HolCChoctah vs Surdiie
Compenatee
Coreecine
(a) Reward Model Validation Loss: -5
(b) RL Episode Returns: Half-Cheetall Cheetah-v5
Figure 3: Showcasing the influence of noise on different feedback types in the HalfCheetah-v5 environment.
type. For Half-Cheetahee feedback seems more susceptible, whereas descriptive feedback is more robust. Notably, moderate noise can sometimes improve downstream RI. performance despite negatively affecting reward learning, particularly for corrective feedback. We hypothesize this acts as form of label smoothing, encouraging the model to focus on more distinctly different pairs rather than highly similar ones affected by noise. We provide additional results in subsectiin E.3
Summary: Our experiments demonstrate that various feedback types are effective across different environments and training stages. Comparative feedback is matched or even surpassed by other types across environments. While descriptive feedback shows consistently strong performance, its real-world collection presents challenges, though approaches using state clustering have been proposed Zhhng al. (2022). These findings suggest potential benefits in leveraging multiple feedback types to exploit their respective strengths in specific scenarios.
4.4 ANALYZING REWARD FUNCTIONS FROM DIFFERENT FEEDBACK TYPES
Beyond downstream RL agent performance, we analyze the Iearned reward functions to better understand the characteristies of different feedback types.
Correlation with Ground-Truth Reward Function - Function-Fiiurre 4 jillustrates pairwise correlations between tround-truth and learned reward functions, computed separate validation dataset aggregated across five feedback datasets.
Most feedback types demonstrate high correlation with the ground-truth reward function in Mujoco,
7

---

Published as a conference paper at ICLR 2025
Esslustive
45.27
Compuratio
1.09
Desoripive
40.08
68.69
DeSi mrce
Nateelad Phat honeee
0.75
00.35
46,6
18.20
Deviemment
00.32
+6.35
$0.83
0.23
0.33
60.38
+8.23
+8.18
60.22
00.13
68:25
0053
0.2
0.2 0.8 0.6 0.3
(a) Aggregated reward function correlations (Mujoco, five seeds, optimal). Details see sub
(b) Relationship between the correlation of ground-trun re- ward function, learning reward function, and downstream ward (normalized by expert rewards).
section D.5
Figure 4: Correlation of Learned Reward Functions for Different Feedback Types: (a) Correlation differs between types (b) We do not observe strong relationship between GT correlation and rewards.
except for corrections and demonstrations, which show greater divergence due to their dependence on expert policy rather than direct derivation from ground-truth. As shown Figure 4(b), the weak correlation between ground-truth reward similarity and final reward suggests that alternative feedback types can uncover different yet effective reward functions. Furthermore, high correlation is itself not indicative of downstream performance.
Effectiveness for Different Noise Levels Figure demoonstattes varying noise sensitivity across feedback types in two environments. Increased noise generally reduces correlation with ground-truth rewards. Descriptive feedback shows more stability to noise, consistent with its robust downstream RL performance. In contrast, evaluative and comparative rewards exhibit larger performance drops. Demonstrative feedback has lower correlation and is less affected by noise. These effects are also visible across other environments subsection D.6
Comparanan
(a) Correlation with ground truth reward function over different noise levels for HalfCheetah-v5
(b) Correlation with ground truth reward function over different noise levels for Walker-v5
Figure 5: Robustness of different reward functions for different noise levels: With increasing noise levels, see drop in performance for types, but to different degree.
Comparing Predictions over Sequences Analyzing reward predictions across 100-step trajec- tories reveals interesting patterns. Figure demonstrates that a high correlation with ground-truth rewards does not necessarily translate to strong RI performance. For instance, models achieve near- expert performance in the Humanoid environment despite low reward correlation, particularly with demonstrative feedback. This suggests that effective RL training can occur with various reward func- tions independent of their ground-truth correlation.
Summary: - Our analysis reveals that most feedback types achieve high ground -trth accuracy, excluding demonstrative and corrective. However, uncorrelated reward functions, like demon- strative feedback, can be highly informative and enable learning in downstream RL. The varying performance characteristics across feedhack types, especially with introduced noise, suggest comple-
8

---

Published as a conference paper at ICLR 2025
(a) Soquential reward predictions for Ant-v5
(b) Sequential reward predictions for Humanoid-v5
Figure Reward predictions from learned reward functions: (a) Ant-v5 shows high correlation
with ground-truth rewards yet suboptimal agent performance. (b) Humanoid-v5 maintains strong downstream performance despite lower reward correlations than other Mujoco environments.
mentary strengths that could be combined.
JOINT-MODELING TO LEARN REWARDS FROM MULTI-TYPE HUMAN FEEDBACK
While different feedback types effectively learn reward functions, their performance varies across scenarios and noise conditions. We explore proof-of-concept approach combining multiple feedback types through ensemble methods. Averaging Model Using our pre-trained reward models, we implement basic ensemble that averages predictions from all six feedback types. Uncertainty- Weighted Ensemble - extend the basic ensemble by incorporating epistemic uncertainty to mitigate underfitting. Using standard deviations from individual reward model ensembles Durasov et al. (202I), we weigh each feedback type's predictions based on model inverse uncertainty to leverage complementary strengths.
5.1 RESULTS
Figure demonstrates euvironment--peecfii success. The ensemble matches the best individual feedback performance in HalfCheetah-v5, significantly outperforming the single-feedback average. However, Walker2d-v5 shows limited improvement, performing at or below average, indicating room for improvement in complementary learning. These results indicate that the combination of feedback types might harness the information within the set of feedback types. However, it might fail in other scenarios.
Our preliminary experiments (see subsection C.2) suggest that selective feedback type combinations outperform full ensembles, opening promising directions for optimizing feedback type selection and combination strategies. As shown in Figure 7 this approach has limitations, including potentially improper uncertainty calibration, though we expect provide full utility and present additional challenges in an online learning setup where specific feedback types might be queried, and we strongly encourage further exploration in this research area.
6 DISCUSSION AND OUTLOOK
Our work yields two key insights: (1) Multiple feedback types effectively enable reward learning and RL, with noise-robust results across environments, challenging the presumed superiority of comparative feedback. (2) While simultaneous learning from multiple reward models shows promise, further research is needed to realize its potential fully.
Our implementation advances multi-type feedback beyond theoretical frameworks (Jeon et al. 2020). We identify several key directions for future work:

---

Published as a conference paper at ICLR 2025
5000
1503
Shat Tewas
2000
1000
2000
2000
Sosel (companative) lonal on (deariictivn)
besdiig (comparateed Eaneline (seuriptive)
0.0
022
0.0
0.8
1.2
60
02
0/4
lec
(a) HalfChretah-v5
(b) Walker2d-v5
3003
2003
4003
1032
Uncortality Ensemble
Aecrogo
toxel ne (porppanative) Socce ne descriittrel
2003
Encemble
Uncertennty Craemble
(comparathee)
0.8
Glesel
Jeb
0.2
0.6
1.0
12
Ant-v5
Clobal Steps
(d) Humanoid-v5
Figure 7: Comparison between the ensemble (with and without uncertainty- -baee scaling), average returns across all single feedback RL runs, descriptive and comparative baselines (averaged over 12 runs). The joint modeling approach can match the best single type but also has failure cases.
Dynamic Reward Models - Transitioning from fixed pre-traine nee-trained to continuously updating models could enable more dynamic learning processes. This approach could incorporate complex querying mechanics (Zhan et al. ;; Jeon et al. (2020); Metz et al. (2023) and leverage different feedback types at various training stages, building
weighted ensemble implementation.
Scaling to Complex Domains - While our evaluation spans multiple RL environments, extending these findings more complex domains remains crucial, particularly LLM alignment and fine- tuning. Additionally, expanding to more feedback types presents opportunities and challenges- integration is straightforward, efficient strategies for managing multiple feedback sources need investigation.
Human Feedback Integration A natural extension transitioning real human feedback. where adapting synthetic reward generation to observed human feedback patterns could improve experimental validity. In particular, we base our simulation on simple assumptions, such as modeling noise via an additive Gaussian distribution. Fitting characteristics to measurements from human experiments promises more realistic experiments the future.
Exploring RL from AI Feedback - Strongly related to the concept of synthetic feedback discussed in this paper is the use of RL from AI Feedback (RLAIF) Lee et al. (2024); Bai ct al. (002).. As human feedback annotations may be hard to acquire at scale, RLAIF based limited human guidance can vastly improve efficiency [iu et al. (2024); Lee et al. (2024). Multi-type feedback can enable ways of integrating various sources of human and Al feedback. Our framework allows the investigation and comparison of new sources and comparison of their effectiveness.
CONCLUSION
This paper demonstrates the effectiveness of multi-type human feedback for reward learning, moving beyond traditional binary preferences. We defined six feedback types and described protocol to generate synthetic feedback. We systematically evaluate learned reward functions and reveal some of their characteristics and complementary strengths in reward learning and downstream RL performance. Our analysis of joint reward models suggests that joint modeling can perform well but needs further exploration. We encourage researchers to explore more diverse human-Al interaction paradigms and contribute to the development of more capable and reliable Al systems.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILIT STATEMENT
Source code for all experiments is available the supplementary materials, and will be open sourced. We used the Stable- Baselines:] R et al. (202T) library (Version 2.3.2) to train all RL expert models with the provided default architectures for MLP- and CNN-based policies. Furthermore, we used the RL Baselines3 Zoo Rafiin (2020) framework (Version 2.3.2) to train the agents, and used the included hyperparamters to train both the expert models and downstream RL models with learned reward models. Hyperparameters and scores of expert models reported in subsectto B.6 Reward models were implemented and trained as Pytorch Lightning-moder modellss, with the training and architecture details reported in (Appendix D
All experiments were performed on machines with single Nvidia V/00 GPUs. Most tasks were CPU- bound and required relatively little GPU memory. We estimate that reproducing all experiments results reported in the paper, takes approximately 500 hours on single node with 4 GPUs and 64 cores. The runtime for feedback generation of 10,000 steps is approximately around 60 minutes (depending on network inference time and environment complexity), fitting of reward models takes 10-30 minutes. The training times of (expert) RL models are highly dependent on the scenario (timesteps and availability of parallel simulation environments). With learned reward function, training of Mujoco environment with SAC and million environment steps takes around one hour. Training with reward model ensemble does only impose modest runtime increase, but potentially a large memory increase (if all reward models are kept in GPU memory).
ACKNOWLEDGEMENTS
The authors acknowledge support by the state of Baden- Württemberg through bwHPC. Part of this research was conducted with financial support from a DAAD Research grants for doctoral students.
REFERENCES
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Eventy-First International Conference on Machine Learning. ICML '04, pp. 1, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1381138385. doi: 10.1 10.1145/1015330.1011
//doi wareiiorg//ae 11..111.11151 1145/1015330 1015430
Christian Arzate Cruz and Takeo Igarashi. A survey on interactive reinforcement learning: Design principles and open challenges. DIS 2020 Proceedings of the 2020 ACM Designing Interactive Systems Conference, pp. 1 1299, July 2020. doi: 10.1 145/3357226.3335552 Uttptttss:// arxiv.org//aie abs 21000000000 4vvl arXiv: 2105. 12949 Publisher: Association for Computing
Machinery, Inc ISBN: 9781450369749.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton. Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan, Constitutional ai: Harmlessness from ai feedback, 2022. URL https: /wwpvv/avvvv///////////a.//aaaav org/ abs/ 22w222222222 08073
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, (3(4)))324 345, 1952.
Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations, July 2019a. URL http::pp/aaaxii.oog//aaaaao////a
06387 19X.v:1904.06387 [cs, stat].
Daniel S. Brown, Wonjoon Goo, and Scott Niekum. Better-than-demmonttaate imitation learning via

---

Published as a conference paper at ICLR 2025
automatically-rankee demonstrations, In Proceedings of the 3rd Conference on Robot Learning, 2019b.
Eugene Bykovets, Yannick Metz, Mennatallah El-Assady, Daniel Keim, and Joachim M Buh- mann. How to enable uncertainty estimation in proximal policy optimization. arXiv preprint arXiv:2210.03649, 2022.
Erdem BIyrk, Dylan P. Losey, Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1))45561
January 2022a. ISSN 0278-3649, 1741-3176, doi: 10. 10.1177702783344921110 1041652. URL http:
//journals.
www /ttp://o.......1117/72/333333330
Erdem BIyik, Aditi Talati, and Dorsa Sadigh. Aprel: A library for active preference-based reward learning algorithms, 2022b.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. arXiv preprint arXiv: 1912.01588, 2019.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Nikita Durasov, Timur Bagautdinov, Pierre Baque and Pascal Fua. Masksembles for uncertainty estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 13539-1333 13548, 2021.
Hsiao- Yu Fish, Tung Adam, W Harley, Liang-Kang Huang, and Katerina Fragkiadaki. Reward Learning from Narrated Demonstrations. 2018. arXiv: 1804.106920 10692v1.
Gaurav R. Ghosal, Matthew Zurek, Daniel S. Brown, and Anca D. Dragan. The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types, August 2023. URL
http://arxiv/ org/ abs /2w/82228... 10687 arXiv:2208. 10687 [es].
Adam Gleave, Mohammad Taufeeque, Juan Rocamonde, Erik Jenner, Steven H. Wang. Sam Toyer, Maximilian Ernestus, Nora Belrose, Scott Emmons, and Stuart Russell. imitation: Clean imitation learning implementations. 12Xi::2211.119222 [cs.LG], 2022. URL httpp:://axiv..rt org/
abs 0ass/2211.1191
Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles I. Ishell, and Andrea Thomaz. Policy Shaping: Integrating Human Feedback with Reinforcement Learning. Advances in Neural Information Processing Systems, 2013.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pp. 1861-1870. PMLR, 2018.
Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The off-switch game, In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI' 17, pp. 220-227. AAAI Press, 2017. ISBN 9780999241 103.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018
Hong Jun Jeon, Smitha Milli, and Anea D. Dragan. Reward-rational (implicit) choice: A unifying
12

---

Published as a conference paper at ICLR 2025
formalism for reward learning, December 2020. URL http://axxxv.crr//abbb1 04833 arXiv:2002.04833 [cs].
Timo Kaufmann, Paul Weng. Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback, 2024.
wttp:///.gggbbb.....////// 14925
W. Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In The Fifth International Conference on Knowledge Capture, September 2009, URL http://wwx. www. cs utexas wwtp:edu///uuaaeee LatpaaaaaaaaKKKAAPAP knox
W Bradley Knox, Stephane Hatgis- Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessan- dro Allievi. Models of human preference for learning reward functions. arXiv preprint
2xXiv:2206.02231,, 2022.
Grgur Kovač, Rémy Portelas, Katja Hofmann, and Pierre- Yves Oudeyer. SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents. July 2021. doi: 10.48550/ arXiv:
Mxxiv.2107.00956 URL htt.::ttts:/arrtt wwt.:/orrrgabb//o/go /2007 2107
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with feedback, 2024. URL tattps://arxe
00rg/abs23300 00267
Edouard Leurent. An environment for autonomous driving decision decision-making https://github
com/ e wwweeeeeenttteee /hi ghway-enw 2018.
Jinyi Liu, Yifu Yuan, Jianye Hao, Fei Ni, Lingzhi Fu, Yibin Chen, and Yan Zheng. Enhancing robotic manipulation with ai feedback from multimodal large language models, 2024. URL https
//arxiv.org/aaa abs/ a24/ 14245
James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew E Taylor, and Michael Littman. Interactive learning from policy-dependen human feedback. In International conference on machine learning, pp. 2085-22552202 2294. PMLR, 2017.
Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: evaluation protocols and open problems for general agents. J. Artif. Int. Res., 61(1):523-562 jan 2018. ISSN 1076-9757.
Shaunak A. Mehta and Dylan P. Losey. Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human- Robot Interaction, July 2022. URL httt:://axxv..rzz abs /bzs/200..m 03395 arXiv:2207.0333 [cs].
Yannick Metz, David Lindner, Raphaël Baur, Daniel Keim, and Mennatallah El-Assady. Rlhf-blendes A configurable interactive interface for learning from diverse human feedback. arXiv preprint arXiv:2308.0445 04333, 2023.
Andrew Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, pp. 2, 2000.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, March 2022. org/ abs faabb//2203.002 22155.20.05555 arXiv:2203.02155 [cs].
URL
Antonin Raffin. RI baselines3 zoo. 2020.
DDR RM/ wM/rl-baselines zoo
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-basclines:: Reliable reinforcement learning implementations, Journal of Machine Learning Research, 22(268):1-8, 2021. URL httt::://mll..og/ppppte http:///.........pp/////ooooooo/
20- 1364.31444500 html
Rafael Rodriguez-Sanchez, Benjamin Adin Spiegel, Jennifer Wang, Roma Patel, Stefanie Tellex,
13

---

Published as a conference paper at ICLR 2025
and George Konidaris. Rlang: declarative language for describing partial world knowledge to reinforcement learning agents. In International Conference on Machine Learning, pp. 29161- 29178. PMLR, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint orXiv:/707.06344 2017.
Theodore R. Sumers, Robert D. Hawkins, Mark K. Ho, Thomas L. Griffiths, and Dylan Hadfield- Menell. How talk so Al will learn: Instructions, descriptions, and autonomy, October 2022a.
URL http: org a2we 07870 arXiv:2206.07870 [cs].
Theodore R. Sumers, Robert D. Hawkins, Mark K. Ho, Thomas L.. Griffiths, and Dylan Hadfield- Menell. Linguistic communication as (inverse) reward design. April 2022b. doi: 10. 10.44850//rxiiv 2204.05091. URL httpp:://axxiv..r///aaaa// 2204.05091v1.
05091v1
arXiv: 2204.05091 ISBN:
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. CoRR, abs/1703.01365, 2017. URL http:: org/ abs//b/3.. 01365
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407. 2024.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces. 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp. 1545-1553, September 2017. doi: 10.48550/arxiv. 1ww.xxi..779.10000 10163. URL https
//arxiv. httpa//..rrgab///////////// 10163v2 9781577358008.
arXiv: 1709.10166 Publisher: AAAI press ISBN:
B. P. Welford. Note on method for calculating corrected sums of squares and products. Techno- metrics, 4(3)419-420, 1962 doi: 10. 0001100000440170661166...0000 tandfon lnn www.co//////ooo abss
URL htttp:://wwwt
1080 1.tp:/...10/00001111111111111111
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. survey of preference- based reinforcement learning methods. J. Mach. Learn. Res., 18(1):4945-499) jan 2017. ISSN 1532-4435.
Wanqi Xue, Bo An, Shuicheng Yan, and Zhongwen Xu. Reinforcement learning from diverse human preferences. arXiv preprini arXiv:2301.11774, 2023.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019. URL https: arwiv. http://awxwww...aaaaaaaa ab901910 1910
10897
Yifu Yuan, Jianye Hao, Yi Ma, Zibin Dong, Hebin Liang, Jinyi Liu, Zhixin Feng, Kai Zhao, and Yan Zheng. Uni-RLHF: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. In The Twelfth International Conference on Learning Representations, ICLR, 2024. URL / net/ attpeeeewweeeeeeeeeeerrrrrneeeed
Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason Lee. How to query human feedback efficiently in rl? 2023.
David Zhang, Micah Carroll, Andreea Bobu, and Anca Dragan. Tife-Efficien Reward Learning via Visually Assisted Cluster Ranking, November 2022. RRLlitttp//aaxxy
2212
00169 arXiv:2212.00166 [cs].
14

---

Published as a conference paper at ICLR 2025
A DETAILS ON THE FORMALIZATION OF FEEDBACK TYPES
This study investigates six exemplary feedback types motivated by existing work MMetz et al. (023). Here, we will briefly model these feedback types according to the reward rational choice formalism presented in previous work (Jeon al. 2020). Table and Table show the choice set, grounding functions and constraint -aaseeddtt---ed formulations respectively. The formalism proposes that feedback types can be described by explicit implicit of (feedback) choices, that C humans optimize over, and a grounding function vb -> fz. which maps these choices c E C to a distribution over trajectories.
Table 2: Feedback types, choice sets, and grounding functions
Feedback type
(Choice set)
3 ::C - (Grounding func.)
Rating
(g.fre) (<) x a. b]. a. bl C C
(E, f1. =
Comparative
& 6 E2
id
Corrective
(E) E (E) x (5c E E EE}
(5 AC) = E A
Demonstrative
Ed E
id
Descriptive
(f. S. A
[3, E
fay fa
Descr Pref.
Jaiis E (fa,1 ES E 1,, E A)))
18,0
E
Ja fn
Table 3: Feedback types and corresponding constraints
Feedback type
Constraint
Evaluative
r(g)= fp
(r(())2((3) if { {1
Comparative
(r(E2) r(1) if & = E
Corrective
22)
Demonstrative
r(Ea) r()) EEE
Description
e(((s.0)EEss=11 EEss=faa fal) U/b
(((((sa){E3) = Se,1, fa,1) r(((s, E = Sa a=, if i=1
Descriptive Preference
(r(((sa)) E S0,2, fa,2)) r(((s,a) E = Ja,1))
if i= 2
B ANALYSIS OF SYNTHETIC GENERATED FEEDBACK
this first appendix, we want to highlight the characteristics of different types synthetic feedback. We want to discuss details because the simulated feedback is integral to our investigation.
B. DETAILS OF EVALUATIVE FEEDBACK
Evaluative feedback is based on discounted segments returns (i.e., the discounted sum of step rewards within segment). The discount factors 7 match the discount factors of the expert models (as reported in inssubeection B.6). Figure 8 and Figure show the distribution of discounted segments returns for the feedback datasets. The distribution of rewards segments from a set of checkpoints throughout training. To generate ratings, we use an equal-width binning approach of returns, and assign ratings of to 10 for each bin.
result of the data collection strategy, i.c. collecting
Figure 8: Histogram of total discounted reward distribution (for 10000 randomly sampled segments with max. length 50/ truncated at episode end). Each color represents different generated feedback dataset.
15

---

Published as a conference paper at ICLR 2025
Algorithm I Generation of Evaluative and Comparative Feedback
Require: Environment E, Expert models & fei...., en Novice model checkpoints θ
(0,,...,,n. Steps per checkpoint T, Segment length L, Discount factor ?
1: S + Ø
Set of segments
2: for 0 do
3:
f- NoviceModel(0)
4:
Sg E.reset()
5:
F < Ø C t Ø
D Feedback and state copy collection
6:
for = to T
7:
C E.gel_state()
8:
0t - Ty (1-1) St (astep(ar)
9:
10:
F+ FU ((5--1, 4e, T2,
11:
end for
12: S+S S CreateSegments FF, L)
end for
14:
for o E
S do
Vo
[ Eeer (oo)
16:
VE
T Ceeg (o
Rn Σ1- y'r
Discounted reward sum b Evaluative feedback
18:
Fenal f- 10 BinIndex Ro, 10)
end for
20: Feomp & GGePreffeenccPPiiss RRo)
> Comparative feedback
return Feval, Fcomp
Figure 9: Cont. Histogram of total discounted reward distribution (for 10000 randomly sampled segments with max. length 50/ truncated at episode end). Each color represents different generated feedback dataset.
Figure shows the extracted ratings (plotted 0-9) in relation to the underlying rewards.
Figure 10: Comparison between ratings and total discounted reward (for 10000 randomly sampled segments with max. length 50/ truncated episode end). Colors indicated the origin model of the rating (out of the expert model ensemble). Segments are sorted by discounted return.
Comparative feedback is directly extracted from the and therefore shares the characteristics with vating-based feedback. contains the pseudo-code describing the feedback generation process for evaluative (rating) feedback, and comparisons.
Figure 11 and 12 the return difference between preferred and non non-preferred segments in the dataset. As the segments with higher return preferred (in the optimal non-noise case), we only values in the upper-left triangle. We see, that the segments pairs have wide distribution of return
16

---

Published as a conference paper at ICLR 2025
differences, i.e. the dataset contains pairs with very similar dissimilar returns, i.e. very similar dissimilar levels of performance with respect to the ground truth reward function. Furthermore, pairs collected across wide range of returns, indicating the effectiveness of our expert model-based segment sampling approach.
Fur Feveeb
Feleree
Figure 11: Scatter plot of discounted rewards for preference pairs: On the y-axis, we plot the rewards of the preferred segments, the x-axis are the rewards of the non-preferred segments. Because the return of the preferred result in the optimal case (Noise 0.0) is always preferred, we only observe points above the diagonal.
Peefemnce Aradids
Podfre da Pails Rowand
curdbeere Podive Reuank
ouarg Fradernce Rewards
Continuation: Scatter plot of discounted rewards for preference pairs: On the y-axis, we plot the rewards the preferred segments, on the x-axis are the rewards of the nre-preferred segments. Because the return of the preferred result in the optimal case (Noise 0.0) is always preferred, we only observe points above the diagonal.
Figure 12:
Figure 13 and Figure 14 show histograms of the segment pair return differences for different environ- ments.
17

---

Published as a conference paper at ICLR 2025
Exvasd Diteeca Palals
Bonad Dfferroos Nots
1300
106
Fevent Pala
2843 feeedERerrri PMlr
Sowurd Difenrndo Not
300
323
Frand
Figure 13: A histogram showing the distribution of return differences between preference pairs (for 10000 randomly sampled segments with max. length 50/ truncated at episode end).
Reward SPheneoehh
Bosurd
Ansard Pair
300
D00
Figure 14: Continuation: A histogram showing the distribution of return differences between preference pairs (for 10000 randomly sampled segments with max. length 50/ truncated episode end).
18

---

Published as a conference paper at ICLR 2025
Algorithm 2 Generation of Demonstrative and Corrective Feedback
Require:
Environment E, Expert models & fei..... Cn State copies C Cm Seg- Segment length L, Discount factor y
J. Sm Ensure: Demonstrative feedback F derao, Corrective feedback Frorr
ments = {$1....,
Fdemo t Ø, Feors +- Ø for i {l,..., m do
2:
3:
D+
Current demonstrations Current expert returns
4:
R Ø
5:
for E & do
6:
E.set
7:
d
Current demonstration
8:
for j E {1.... [l...., L, do a + c(s) E.step(a)
9:
10:
8' - du a()
11:
12:
8 s' end for
13:
14:
D D U {d]
15:
R+ RU 1
7 (1)) d}
16:
end for
17:
k + arg max
18:
d' + Dr. & Rx
Torig Σl- 7're: a;, E st
20:
if r" Torig then
Fdemo t- Fdemo {dr Fcorr t- F Foor U ((s,, d")}
22:
23:
end
24:
end for
25:
return Fdemo: Feorr
B.2 DISTRIBUTION AND EXAMPLES OF DEMONSTRATIVE AND CORRECTIVE FEEDBACK
A key novelty over existing work is the ability to also generate demonsstrations//ation advice and corrections dynamically for given segments based on direct access to the expert policy, 2 contains pseudo-code for the generation of demonstrative and corrective feedback. However, this approach raises several questions, which we should investigate:
(1) How does the distribution of generated camoostrations/correttinns look like? (2) How do they compare the existing episodes within the dataset? (3) How does the choice of an expert policy influence the generated feedback?
Comparison of Returns between Learner and Experts First, we want make sure that the expert model policies that we use to correct and demonstrate advanced behavior, actually meaningfully improve over the baseline learner performance. As we can see |Figure 15 and Figure 16 expert policies can improve performance over the leamnr. However, depending on the situation, we see wide range of achieved rewards for both policy types. This caused by the initial start state of trajectory, which might be highly non-optimal, and the expert policy might only be able to recover from a failure state but not achieve fully optimal reward.
Examples for Learner and Corrected Trajectories - To further illustrate the difference between weak learner trajectories and more optimal expert trajectories, we give some examples: Figure 17- Figure 18 highlight the differences in behavior by the weak and strong policy. As expected, we can identify that expert policies better exploit good initial states and are often able to recover from failure cases. More video examples provided in the supplementary material.
B.3 DETAILS OF DESCRIPTIVE FEEDBACK
In subsection 3.1 we have defined descriptive feedback as a mapping from abstract state-/action- feature values feedback value. We have implemented this feedback type via clustering approach:
19

---

Published as a conference paper at ICLR 2025
Sepment
Figure 15: Comparison of achieved segment return between learner and expert demonstrations that we use as corrective and demonstrative feedback.
Figure 16:
Comparison of achieved segment return between learner and expert demonstrations that we use as corrective and demonstrative feedback.
All collected state-action pairs (s, a) D are clustered into N clusters (with being the number of feedback instances/queries, In our experiment, match the number of clusters with the number of segments, e.g., 10.000. We utilize mini-batch k-means clustering with a batch size of 1000 to cluster the (s, -)-iiis.. We then average the observation, action, and optimality gaps for cluster to get a single feedback instance (s, a, 0y)) that describes part of the state space without referring to specific state.
20

---

Published as a conference paper at ICLR 2025
Figure 17: HalfCheetah-v5 Comparison of key-frames between learner and expert demonstrations, that we as corrective and demonstrative feedback: Top Row states from the learner segment, bottom row is the expert segment.
Figure 18: Ant-v5- Comparison of key-frames between learner and expert demonstrations that we corrective and demonstrative feedback: Top Row is stated from the learner segment, bottom row is the expert segment.
In Figure 19
we show sub-sample of state-action pairs (5000 samples) alongside sample of the computed averaged cluster representatives (500) and the associated step reward for each pair marked in red. In Figure 20 show t-sne projections of the full set clusters alongside the associated descriptive rewards
B.4 LIMITATIONS AND ALTERNATIVE FORMULATIONS
Lastly, the generation scheme has some limitations. The biggest limitation in our mind, is the potential mismatch between optimality of the evaluatiie//descritive feedback, which is based on the underlying reward function, and the demonstrativeecorretiie feedback that is based on expert policies. While these experts policies were optimized with respect to the equivalent ground-truth discounted reward functions, they might still be sub-optimal compared to an optimal policy. However, in none of our explored environments, have access to optimal expert policy.
The feedback datasets were collected over trajectories sampled by rolling out the models from various checkpoints of the expert policies. This can be seen both positive negative thing: On the positive thing, this ensures a better comparability between feedback types (including demonsttraivee/corectiie feedback(, because this means feedback is available for roughly the same space of trajectories across types, and should therefore transport comparable underlying biases and non-optimality. On the flip- side, this means that the learned reward functions are only well-defined for the trajectories in the
21

---

Published as a conference paper at ICLR 2025
Algorithm 3 Generation of Descriptive Feedback
Require: Segments S = {81,....
. Sn) :....... Descriptive feedback Fdese: Descriptive preferences F Fdeso.pref.
Number of feedback instances k
Ensure:
1: X' + 2, 4- 2
2: for S do
3:
X' - X' U CCCnaatenateee a;) (Zi, T;) E
4: R R (ri (Ii, ai rs) Es}
5: end for
6: C KMe (X, k)
B Cluster assignments
7: X + [ा Ejcc Xj :: .l,,....,..))
Cluster representatives
8: R + ( Lje
(L....
D. Cluster mean rewards
9: Fdese + -[ iiE fl....,kk) k})
D Cluster descriptions
10: Fdese.pref. & GetPreferencePairs(t* RR) return F deses F Jrej.
FCheetah.uS:
anc Representatives: Ground-Truth Reward
Originel Poinas
Claster Representatiom
leature
-00
60
Figure 19: t-SNE projection state-action pairs, colored with the associated ground-truth reward. Overlayed, marked by X computed cluster representatives. These cluster representatives, along- side the average reward of underlying pairs, are used as descriptive feedback.
datasets, i.e. also better-than-expert trajectories that deviate significantly from the observed behavior, must be considered out-of-distribution We therefore do not expect that trained RL models can easily surpass expert performance. Indeed, this is mostly supported by the experimental results.
Our method is well suited for generating fixed, offline training datasets. However, approach requires some modification to generate online oracle feedback. This includes initializing the histogram initial dataset, but updating it based online data. Similarly, the clustering
of ratings based used to generate descriptive feedback must also be re-computed on dynamic buffer to adapt to online trajectories. In the provided code repository, oracle which can be used for online RLHF with simulated multi-type feedback.
provide an implementation of dynamic feedback
Alternative Formulations: During the design process, discussed several alternative ways
22

---

Published as a conference paper at ICLR 2025
(a) HaliCheetab-v5
(b) Swimmer-v5
States bolere somenddl
TSAE Stntecation reaart
effewents
(c) Walker2d-v5
(d) Hopper-v5
States Iedend
DM Frmuch
TIME inelean hyceaeii
Ant-v5
Humanoid-v5
7/58
T:ME Festrs
Sefriitn Bemond
2000
g merge-v0
(h) highway-fast-vO
TIME raandi
(i) roundabout-v0
(i) Metaworld-Sween Seaaworld-SSeep-Into--22
Figure 20: Descriptive Feedback Analysis for the benchmarked environments. Each subfigure shows the T-SNE visualization of states (left) and the distribution of rewards (right) for the respective environment.
23

---

Published as a conference paper at ICLR 2025
generate synthetic feedback:
(Evaluative feedback) Disagreement with expert policy: A possible alternative using the expert's value function is to base the feedback directly on the policy. Suppose the selected action observed in the segment deviates from the agent's prediction. In that case, assume it is against the intended series actions that the expert agent prefers. However, this formulation does not have room for alternative solution strategies that might be recognized by a human annotator, including segments that are even better than the expert. In both cases, the optimality-gau saaiiy-gga-baasss sap-based approach will lead low and potentially even negative gaps, which will be converted into high-score feedback.
(Descriptive feedback) The formulation of descriptive feedback is more undetermined than the other types. We explored an alternative approach to assign scores or preferences features instead of segments using feature attribution methods of the value estimator. This allows us to identify features that are relevant for high/low value. We can then interpret the difference in prediction for the presence/absence of an important feature as a reward assignment. However, we have found the attributions generated by methods such as Integrated Gradients (Sundararajan et al. 2017) overly noisy to be effective in our context. However, in our experiments, the attributions maps proved rather unreliable and only provided a weak signal due to being similar across states. Furthermore, they were only suited to create description of state, and not acttonatiion pairs.
An additional approach based on regret, that explored more extensively, is presented in the following.
B.5 ALTERNATIVE GENERATION OF EVALUATIVE AND COMPARATIVE FEEDBACK WITH OPTIMALITY GAPS
We extensively explored to use the value function of an expert model as a source of feedback. Existing work (Brown et al.) 2019a Xue al., 2023 Christiano al., 2017) utilizing simulated feedback (often called an oracle), use the ground-truth reward function of the environment to simulate human feedback.
In actor-critic RL algorithms like SAC HHarnojj et al. 2018) or PPO (Schulman et al. (017). we can utilize the value (s) or v-value ( a) E Ie estimate directly to generate feedback values, and the learned policy for demonstrative feedback. Similarly, we can directly transfer this approach to purely value-based RL like Q-Learning
We explored an alternative way to generate evaluative feedback on notion similar to regret, i.c., we rate a segment not based on the actual collected environment reward but instead on whether it showcases optimal behavior from the perspective of an expert model. This notion aligns more closely with reward behavior observed in humans, which have been shown to rate relative to their expectations (MacGlashan al., 2017; Knox al., 2022; Jeon et al., 2020). Furthermore, feedback based on ground-truth rewards is not entirely suited for sparse-reward tasks, as we might only observe few rewards during typical segment, resulting in weak learning signal. The observed regret is computed by comparing the expected future rewards with the observed ones:
H-1 Y' HVc(H)
Dopt Ve(su)
We want to briefly clarify the relation between the formal regret as commonly defined in RL research and our definition of
Regret and Optimality Gap
Lemma B.1. Equality of Regret and Optimality Gap Under the assumption that the expert policy is optimal, the optimality gap opt is equivalent to the expected regret for a segment in hireihorizzon tasks.
Proof. We can infer this from the common definition of regret RØ over time horizon H (in our case, the length of segment). With ground-truth reward function r(st, az) and (st, at ) being the
24

---

Published as a conference paper at ICLR 2025
segment:
H
Rh == E
(st a.) r(s;, az)
1=0
E
(5t: 07) Σ (stt. ai]
t=0
1=
T Σ R(s
[
r(5 07)
Σ
r(se, aï)
Σ r(s at)
M+1
t-
11
E[
Σ r(s), (7))
Σ R(s),a)))
1=#+1
0=0
H
= (su) (*HH))
r(s,, arl
1=U
H
Vo=Ve EVVc(so) - Ve(sH)
rst at)
t0
However, in infinite horizon tasks, i.e., Mujoco is treated as infinite horizon task in the common implementations of PPO, we often observe that (*())) and V<(+)) are very similar or even equiva- lent, except for states that show catastrophic failure, because states cannot be easily differentiated from each other and therefore will have very similar value estimates. Theorem B.I also does not hold for the infinite horizon case due infinite sums.
In practice, this means that the optimality gap Dopt.e is similar to the negative sum of (discounted) ground truth rewards, i.e., reduces to the standard implementations of feedback.
Another complicating factor is environmental stochasticity. Due to these factors, we chose the term optimality gap instead of regret to indicate the distinction.
Comparison with Environment Ground-Truth Feedback - As described above, we use regret- based optimality gap w.r.t,, an expert value function, as our foundation for evaluative feedback. As a first sanity check for the validity of this generated feedback, we investigate the relationship between these optimality gaps and sums of ground-trui -ruth reward (akin to existing work); Visible a linear correlation between achieved rewards and the negative optimality gap. In particular for non-episodic tasks with low influence of stochasticity like Mujoco-envinee enviromments,,, negative optimality gaps, and discounted total rewards almost co-inside, with the imperfect nature of value estimates leading the introduction of some noise. For some of the other environments, we observe both the difficulty and promise of our approach over ground-truth reward: Return (i.e., the sum of (discounted) rewards over segment) is an uninformative measure in many environments with sparse rewards (like Atari). In contrast, the simulated reward can serve as a stronger signal.
The effect of value- ffnucion ensembles on simulated feedback - In the main paper, we use the ground truth reward function generate feedback. We have experimented with using multiple trained agents to get more reliable value estimates, including broader coverage and the ability to detect out- of-distribution samples that may lead faulty results. We observed that some models tend to slightly over- under-estimate the return, leading to slightly overlapping bins. We argue that this expert model noise compatible with the expected behavior of human labels. Such label noise has been reported in crowd-sourcing tasks by human annotators. We average the optimality gap predictions across four models to achieve representative results.
We see that the value estimates for austere environments often close together, and the averaged estimates have less noise, which means the estimated optimality gaps more closely mirror the observed ground-truth returns. For more complex environments, estimates vary more. In the future, might want to investigate further increasing the number of expert value functions for even more reliable optimality gap estimates.
25

---

Published as a conference paper at ICLR 2025
B.6 EXPERT POLICY: RL HYPERPARAMETER SETTINGS
To train expert models, we chose the default hyperparameter settings of RL training given in Stable- Baselines3 Zoo (Rafinn 2020).
Table 4: RL-Training hyperparameters for Mujoco expert agents trained with PPO.
Hyperparameters for Mujoco Experiments: PPO
Environment
Ant-v5
Swimmer-v5
Walker2d-v5
Algorithm
SAC
Network Arch.
3-layer MLP
Hidden size
256
256
64
Training Timesteps
le6
1e6
1c6
Num. Parallel Env.
4
Batch Size
64
256
32
Learning Rate
2.063e-5
6e-4
5e-05
Num. Epochs
20
10
20
Sampled Steps per Env.
512
1024
512
GAE lambda
0.92
0.98
0.95
Gamma
0.98
0.9999
0.99
Ent. Coeff
0.0004
0.0554757
0.0005
VF Coeff.
0.581
0.38782
0.872
Max. Grad Norm
0.8
0.6
1.0
Clip Range
0.1
0.3
0.1
Obs. Normalization
false
Reward Normalization
false
Table 5: RI-TTrinning hyperparameters for Mujoco expert agents trained with SAC.
Hyperparameters for Mujoco Experiments: SAC
Environment
Ant-v5
Hopper-v5
Humanoid-v5
Algorithm
SAC
Network Arch.
3-layer MLP
Hidden size
256
256
256
Training Timesteps
le6
le6
2e6
Num. Parallel Env.
1
1
1
Batch Size
256
256
256
Learning Rate
3e-4
3e-4
3e-4
Gamma
0.99
0.99
0.99
Ent. Coeff
"auto"
"auto"
"auto"
Obs. Normalization
false
Reward Normalization
false
we report the final evaluation returns by the trained expert policies (no- deterministic sampling, ten episodes). We ran five separate seeds for each environment. We selected four best-performing seeds as part of model ensemble for simulated feedback.
Finally, in Table
26

---

Published as a conference paper at ICLR 2025
Table 6: RL-Training hyperparameters for Highway- Env expert agents trained with PPO.
Hyperparameters for Highway-Env: PPO
Environment
merge-v0
hishway-fast-v5
roundabout-v5
Algorithm
PPO
Network Arch.
3-layer MLP
Hidden size
256
256
256
Training Timesteps
le5
le5
2c5
Num. Parallel Env.
64
64
64
Batch Size
32
32
32
Learning Rate
5e-4
5e-4
5e-4
Num. Epochs
Gamma
0.8
0.8
0.8
Ent. Coeff
0.0
0.0
0.0
Obs. Normalization
false
Reward Normalization
false
Table 7: RL-Training hyperparameters for Metaworld expert agents trained with SAC.
Hyperparameters for Metaworld Experiments: SAC
Environment
Ant-v5
Algorithm
SAC
Network Arch.
3-layer MIP
Hidden size
256
Training Timesteps
le6
Num. Parallel Env.
Batch Size
256
Learning Rate
3e-4
Gamma
0.99
Ent. Coeff
"auto"
Obs. Normalization
false
Reward Normalization
false
Table 8: Sorted expert evaluation scores for selected environments, with averages. We only retain the top four expert models generate synthetic feedback.
Walker2d-v5 H
Swimmer-v5
Ant-v5
Hopper-v5
Humanoid-v5
5549.8062
5860.0417
359.1049
4306.0373
3556.5895
6254.5279
5254.2713
5071.9133
355.3227
4132.0158
3268.0585
5893.5920
5124.0117
4018.3773
320.6619
4122.3491
3250.1369
5853.7701
5095.0633
3717.2990
317.2805
3989.7485
3205.6200
5682.9198
4907.9425
3098.4731
295.3936
3601.7154
3191.8214
5403.7185
5186.2190
4353.2209
329.5527
4030.3732
3294.4453
5817.7057
27

---

Published as a conference paper at ICLR 2025
B.7 VISUALIZING THE ARTIFICIAL NOISE IN GENERATED FEEDBACK
Finally, we want visualize the effect of introduced noise on the generated feedback. Figure 21 and Figure shows the effect of truncated noise added onto the reward distribution. Figure shows the impact of noise on the descriptive feedback. We showcase the adaptations for one of the datasets in the HalfCheetah-v5 environment.
HARE SANEE -S Original HARE Newarrs
Saber 8:34
Metwe
Newwo
Figure 21: Scatterplot displaying original and perturbed rewards: With increasing noise, the underly- ing reward distribution gets perturbed.
Figure 22: Histogram displaying original and perturbed rewards: With increasing noise, the underly- ing reward distribution gets perturbed.
van Mate E21)
Iree on
Figure 23:
Scatterplot displaying the rewards of preference pairs: For optimal feedback, pairs always have perfect relationship between good and bad reward. For increased noise, more and more pairs get flipped, i.c., indicate a wrong preference w.r.t. the ground-truth reward function.
Our library includes utilities to investigate and visualize the details of the datasets, including the perturbed data. We argue that this kind of transparency is crucial for reproducibility.
As a future extension of our work, we may develop common reporting standards for the content of feedback datasets, including sample and rating distributions, well as underlying assumptions.
28

---

Published as a conference paper at ICLR 2025
HolO vs Flect of Neice Bate Renoee Pellirriii
Neagr
Pea
S
Pepgrt Morn
Figure 24: Displaying the effect of artificial noise on descriptive feedback: The assigned cluster reward values are perturbed by the introduced noise.
C EXPERIMENTS WITH SIMPLIFIED REWARDS
In preparation for the main experiments reported in this paper, we ran trials using method that used the reward predictions for each step separately. In this setup, we adopted reward models trained from five types (evaluative, comparative, descriptive, corrective, and demonstrative) of synthetically generated feedback to provide dense rewards to an agent trained using the SAC algorithm in the Half Cheetah v3 environment. To train the reward models, training procedure described carlier, where we used feedback given on single steps as inputs to the neural network that was trained to predict reward for that step. In the case of the runs using ensemble networks, the individual model outputs were averaged to get the final reward for the current step.
employed slightly modified version of the
Sings
Segus
(a) Simple reward networks
(b) Ensemble reward networks
Figure 25: RL Training Curves for training from different individual feedback reward models (displayed are the ground-truth rewards). The line labeled Expert represents the performance of the expert policies used for generating feedback.
C.1 USING REWARD FUNCTION ENSEMBLES
The summary of mean reward curves from 4 training runs is shown in figure 25 First, we compare the training the simple reward model networks (left) to those trained using ensembles (right). In these figures, can see that, in some cases, using ensembles dramatically improves performance. For example, after introducing ensembles to the comparative and descriptive feedback models, their performance reached that of the agent trained using environment rewards (labeled Expert), and the standard deviation of the rewards achieved during the four runs also decreased. In other cases, such as for evaluative, demonstrative, and corrective feedback models, the performance of the trained RI.
29

---

Published as a conference paper at ICLR 2025
agent decreased.
C.2 COMBINATION OF FEEDBACK TYPES BY AVERAGING
In addition to using the feedback models by themselves, we can combine them by averaging their reward predictions. Additionally, since the magnitudes of these outputs not the same scale and can vary, we also need to normalize them before averaging. We keep rolling mean and standard deviation for this and update them using Welford's algorithm (Welford 1962.
Cosd
DesMiutionn Lmmal
(a) Combination of 2 feedback types
(b) Combination of feedback types
(c) Combination of 4 feedback types
Figure 26: RL Training Curves for training from different combined feedback reward models (displayed are the ground-truth rewards). The line labeled as sachhhhhhhe represents the performance of the expert policies used for generating feedback.
The results of runs using different numbers of feedback types are in figure 26 From the training sessions using the combination of 2 feedback types, we can see that some significantly improved the performance over agents trained using their feedback components, reaching mean rewards similar to what the expert model obtained while also improving the standard deviation of rewards throughout the different runs. Specifically, all combinations that did not include the corrective feedback type achieved better scores than both of their single feedback components.
We get similar picture by looking at the figures of combinations of feedback types, where if do not include the worst-performing corrective feedback, then the rewards obtained by the trained RL agent significantly increase. This might also be the reason for the relatively poor performance of the agent trained with four feedback types, including the corrective feedback.
Overall, combining multiple modalities of feedback, the agents' scores were higher than those obtained using the worst single feedback component but in many cases, better than all which indicates that the strengths of these individual modalities are combined through mixed-feedback training.
30

---

Published as a conference paper at ICLR 2025
D REWARD MODEL TRAINING DETAILS
In this appendix, we want to discuss details of reward model training, including ablations and hyperparameters settings.
D.1 TRAINING DETAILS
Rewards were trained on the collected feedback datasets in a standard, supervised way. For the Mujoco environments, we used 6-layer MLP with 256 hidden units and ReLu activation function. We use the Adam optimizer with learning rate of le and weight decay le 2. Models are trained for 100 epochs, with early stopping (5 patience epochs) enabled. We use a batch size of 128.
Based on best practices from previous work and our preliminary experiments (see subssectio C.I). we use reward function ensembles instead of single monolithic models and average the predictions of the submodels. We implemented these ensembles by using Masksembles layers (Durasov et al. 2021] after each fully-connected layer of the network. Similar to techniques like Monte-Carlo Dropout, model with Masksembles computes multiple predictions of partially overlapping models during inference. We chose a mask number of with a scale factor of 1.8, corresponding to four ensemble members. Using Masksembles reduced the required training compute and parameter count compared to "true" reward model ensemble while achieving comparable performance in many cases (Durasov
al. 2021 Bykovets al. 2022).
D.2
LEARNING FROM NOISY FEEDBACK
For our baseline experiments, we have analyzed reward modeling and RL. performance for optimal feedback (w.r.t. the expert models) (Griffith et al., 2013). However, expecting real human feedback to be optimal is a very strong assumption that can generally not be met. We investigated the performance of different feedback types under increasing noise the generated labels. In order to enable meaningful comparison between different feedback types, we designed a consistent perturbation approach across feedback types:
Basic approach: - We modify the underlying reward distribution by adding Gaussian noise of varying degree. The values for evaluative feedback and cluster description reward are updated accordingly, and preferences are flipped if the rewards change. The level of noise is controlled by single parameter 3. First, the range of the reward distribution computed, e.g, [Tmin Tmaa] ]. For evaluative feedback and corrections, this is the range of discounted segment returns; for descriptive feedback this coincides with the range of single cluster rewards. Each single feedback value is then perturbed by additive noise sampled from a truncated gaussian distribution with f Vyb and Tmmmn]
3 |r mmaz-
Truncated Gaussian Distribution: We use truncated distribution, to ensure comparability with the binning based approach to generate rating feedback, as well as to respect bounds of observation- s/actions, which is relevant for demonstrative feedback. Compared fixed clipping values, sampling from truncated Gaussian distribution has lower variance: Based on subset of our data, we expect that we need
Gaussian distribution with
brief analysis of both approaches deviation sigma of the truncated normal distribution Ny to around four times the standard deviation of an un- un-truncated normal distribution A to have a comparable level of perturbation (as measured by the 7.2-norm between original and perturbed value distribution). The 3-values reported in this paper are for the truncated normal distribution, thus for replicating the results with non-truncated gaussian distribution for noise would therefore correspond roughly to a value 4.
the standard
Rating Feedback - We perturb rating feedback, by adding from truncated Gaussian distribution with mean at the original rating, and the standard deviation controlled by the parameter 3 IR+, and truncate the values at the edges of the rating scale. This simulates the effect of human user "incorrectly" switching between two neighboring ratings.
Comparative, Corrective, Descriptive Preferences - For these preference-based feedback types, we assign labels according to the newly perturbed segment return distribution. In effect, this means that preference pairs with similar performance difference in performance, which matches our intuition. However, this approach might actually underestimate human error, that be observed in real labeling tasks due miss-clicks miss-
more often flipped than examples with clear
31

---

Published as a conference paper at ICLR 2025
understanding.
Demonstrative Feedback - Because we model demonstrative feedback effectively as preference- based feedback, could also employ label flipping here. However, this does not represent the type of eιror human labeler would make. Instead, for demonstrative feedback, noise should occur in sub- optimal demonstrations, i.e., sub-optimal action selection. For the sake of simplicity, being able to add distortion to already collected data, we do not perform full rollouts with a distorted policy. Instead we apply additive Gaussian noise on the observations and actions, simulating imperfect demonstrations. We acknowledge that this is a simplistic assumption and will consider further investigations and encourage further investigations to enable more coman-aligned perturbations.
Descriptive Feedback - As descriptive feedback maps to scalar rewards, similar to rating feedback, we use uniform noise schema for rating feedback. Because there are now fixed bounds for ratings (i.e., 1-10), we infer the feedback range based on the collected segments within the datasets 'D and single-step rewards. Feedback scores are then modified by uniform sampling from truncated normal distribution with width (mazp(r) mi) conforming to the existing range of / E IR+.
To summarize, we can introduce different noise levels to the different feedback types, controlled by parameter 3, corresponding to a comparable degree of perturbation. For our experiments, we investigate the behavior of reward functions within a range of 33 from 0.1 3.0, f-value of 3.0 already corresponds to very severe change in distribution.
32

---

Published as a conference paper at ICLR 2025
D.3 REWARD MODEL TRAINING RESULTS
Shown in Figure 27 and Figure 28 almost all reward models train effectively with optimal feedback, achieving low validation loss. We conclude that the reward models can fit the reward data high degree, and in turn that our chosen architecture and training protocol are generally appropriate for the task.
(a) Rating Feedback
(b) Comparative Feedback
302
2002
1000
(c) Demonstrative Feedback
(d) Corrective Feedback
Ievanc valies
5000
(e) Descriptive Feedback
(f) Descriptive Preference Feedback
Figure 27: Validation Loss Curves for Different Feedback Types (Mujoco): Steps is the number of optimizer steps. Averaged over five feedback datasets.
When analyzing results for Meta World and Highway, observe more variance compared to the Mujoco-Enve
33

---

Published as a conference paper at ICLR 2025
Mode Saizee Crom bobaster Profec
Dowel Mold Win Uno Ouer
(a) Rating Feedback
(b) Comparative Feedback
bewart Mode: velicarce Lees Caives for Dansoettative Feedback
Farverd
Nodel Medel
dation Las Cunee Comeetia Feadhact
(c) Demonstrative Feedback
(d) Corrective Feedback
Eavard Pedeit valder Las Ourves for Descripive Feedbect
Savard Podet ss Matiot Lees Carves for Drsc tiphvievve preferrzz DE Foetbech
(c) Descriptive Feedback
(f) Descriptive Preference Feedback
Figure 28: Continuation of Validation Loss Curves for Different Feedback Types (Highway-Env,
Meta Worldy: Steps is the number of optimizer steps. Averaged over five feedback datasets.
34

---

Published as a conference paper at ICLR 2025
D.4 REWARD MODEL TRAINING RESULTS WITH NOISE
As shown above, with optimal feedback, all reward models train effectively. achieving low validation loss. In the following plots, we show the reward model validation loss for different levels of noise, with 3 = {0., 0.25, 0.5, 0. J. averaged across 3 seeds. We can observe, that the introduced noise leads to clear decrease in validation loss.
Reward Model: Validation Loss Curves for HallCheetah-v5 Comparative
Eveluative
Demonstrative
6-
250
5.0
T5E Shaps
1000
1250
1908
253
500
150
1000
1251
1500
3.00
255
500
704
1250
1901
Stags
30823
Corrective
Descripcive
Descriptive preference
Valas
0.7
8:3
225
0.75
Valeo
Hila 5.7
Staps
Stepe
Figure 29: Reward Learning Loss Curve for HalrCheetah-v5 at different levels of noise
Reward Model: Vallidation
Walker2d-v5
Evaluative
Comcarative
Demonstrative
3.30
8.33
Hobec 0:15
1250
1000
75
Staps
Steps
50621
Corrective
Descriptive
Descriptiv_sreeeere preference
E25
03
03
150
MHC
3004
1290
3508
252
130
1200
124
252
130
758
3008
1250
1563
Figure 30: Reward Learning Loss Curve for Walker2d-v5 at different levels of noise
35

---

Published as a conference paper at ICLR 2025
Reward Model: Vallidaiin Loss Curves for Swimmer-v5
Evelustive
Demonstrative
20:5
Menoc
bab:00 vale qus V46
Mooe
Nolle 2:
153
Maooc 0.75
175
12.0
C3
3569
Corrective
Descriptive
Descriptive preference
0-7
Holoe 8-3
L5
02-
01-
03
01-
250
365
150
1800
1250
1963
.00
150
1000
125
252
140
1306
1399
1200
Figure 31: Reward Learning Loss Curve for Swimmer-v5 at different levels of noise
Reward Model: Validation Loss Curves for Ant-v5
Foslustive
Companative
Demonstrative
8-2s
5.2
0.25
0.10
59
6.00
3.00
252
Stops
Stepe
90015
Descripcive
preference
Hobe 8.3
Mos
05
04
253
1000
Figure 32: Reward Learning Loss Curve for Ant-v5 at different levels of noise
Reward Model: Validation Loss Curves for Hopper-v5
Notee Eoclustion
Companation
Demonstratioo Eye
2.20
Melee 77
Sober 011
Hobe 1.75
12.5
152-
2.5
250
530
750
1304
1350
200
400
000
1000
250
506
150
1306
1250
1300
Corrective
Descriptive
Descriptiveeprreeeeeee preference
2.1
225
0.2
Steee
Stops
340gs
Figure 33: Reward Learning Loss Curve for Hopper-v5 different levels of noise
36

---

Published as a conference paper at ICLR 2025
Reward Model: Validation Loss Curves for Humanoid-vs
Foslustive
Comparative
Demonstrative
180
3.2
2.50
5%%
2.22
3.25
275
160
0.09
766 1290
Steds
Stops
Steds
preference
Hoae Heree
Ho E.1
Mos
32
430
Figure 34: Reward Learning Loss Curve for Humanoid-v5 at different levels of noise
37

---

Published as a conference paper at ICLR 2025
D.5 CORRELATION BETWEEN
AND LEARNED REWARD FUNCTIONS
0:16
6:H4
0.70
8:77
0.90
0:80
0.35
0.50
0.73
0:80
0:13
0.41
0.57
0.73
0.52
0.71
0.66
0.55
0.73
0-91
0:40
0:73
0.03
0.11
0.75
0-90
20.23
(a) Noise: 0.25
(b) Noise: 0.5
46.35
0.50
0.
5:72
D:11
0.33
0:94
0:40
3:38
0.48
0.00
5.75
0:17
0.55
0.63
0:00
0.49
3.08
0.26
0.75
Fdt
0.89
6.32
6:16
0.57
0.67
0.07
0.00
0:44
0:48
0.41
0.75
0:42
28.38
20.18
(c) Noise: 0.75
(d) Noise: 1.5
2:14
2:34
6.25
2:36
0.22
0.40
2:21
8.06
014
C13
3.70
6:58
021
0.42
0.20
20.17
28.29
20.18
20.19
2:33
8.08
0.23
643
.0.18
48.37
40.19
.0.32
40.25
.0.27
(e) Noise: 3.0
Figure 35: Summary of pairwise reward function correlations between feedback types at different noise levels Correlations averaged across all six Mujoco environments, seeds per and noise level)
38

---

Published as a conference paper at ICLR 2025
D.5.1 CORRELATION ACROSS FEEDBACK TYPES FOR INDIVIDUAL ENVIRONMENTS
Anoute
200
017
640
(a) HalfChectah-v5
(b) Walker-v5
(c) Swimmer-v5
(d) Ant-v5
6:1
500
611
(e) Hopper-v5
Humanoid-v5
Figure 36: Pairwise correlations between the ground truth and learned reward functions from different feedback types. Optimal feedback without noise.
39

---

Published as a conference paper at ICLR 2025
Conetie
(a) HalfCheetah-v5
(b) Walker-v5
Parilee Coniee pondion
504
(c) Swimmer-v5
(d) Ant-v5
(c) Hopper-v5
(f) Humanoid-v5
Figure 37: Pairwise correlations between the ground truth and learned reward functions from different feedback types. Noise level 0.5.
40

---

Published as a conference paper at ICLR 2025
D.6 INFLUENCE OF NOISE ON CORRELATION WITH GROUND- REWARD FUNCTION
Copparive
(a) HalfCheetah--5
(b) Walker-v5
(c) Swimmer-v5
(d) Ant-v5
Conechive
Pedtrancc
(c) Hopper-v5
(f) Humanoid-v5
Figure 38: Influence of noise correlation with ground-truth reward function
41

---

Published as a conference paper at ICLR 2025
D.7 SEQUENTIAL REWARD PREDICTIONS
www
(a) HallCheetah-vS
(b) Walker-v5
(c) Swimmer-v5
Ant-v5
(e) Hopper-v5
Humanoid-v5
Figure 39: Ground Truth reward function and reward model predictions for randomly sampled sequences.
42

---

Published as a conference paper at ICLR 2025
E AGENT TRAINING DETAILS
E. TRAINING CONFIGURATION
For the training of downstream RL-agents, we chose the same hyperparamter configuration as for the initial expert models (see Table and Table
We implemented the reward function as a wrapper around the Synunssium-environnoee which replaces the ground-truth environment rewards with the prediction of the reward model. reward is predicted for each observed state-action pair. As mentioned above, for our analysis, we use single pre-trained reward model. instead of a consciously updated reward model.
For the environments trained with PPO, we chose to standardize the rewards by subtracting running mean, and dividing by the standard deviation. This follows the existing hyperparameters RL for these environments, available in KkableBaselincee Zoo (Raffin et al. 2021). The SAC agents are trained with unnormalized rewards for all runs.
E.2 DETAILED REWARD CURVES FOR RL TRAINING WITH OPTIMAL REWARD MODELS
6030
Tvarr ariccee
300
HalfCheetah-v5
(b) Walker-v5
Sutardk
Dakatcc
Lompotie
2003
-250
(c) Swimmer-v5
(d) Ant-v5
4070
3000
106
(e) H5
(I) Humanoid-vS
Figure 40: RL reward learning curves for optimal rewards. The shaded area indicates the maximum and minimum reward curves. Rewards are aggregated over 5 feedback sets.
43

---

Published as a conference paper at ICLR 2025
Druur on
(a) merge-v0
roundabout-v5
140
384
13
14
(c) highway-fast-v0
(d) metawword-sseee--ntt--
Figure 41: RL reward learning curves for optimal rewards. The shaded area indicates the maximum and minimum reward curves, Rewards are aggregated over 5 feedback sets.
44

---

Published as a conference paper at ICLR 2025
E.3 DETAILED REWARD CURVES FOR RL TRAINING WITH NOISY REWARD MODELS
The following plots show the downstream RL performance for agents trained with reward function of multiple noise levels. Rewards are aggregated over 3 seeds per noise level. We can identify two trends: (1) Often, RL performance is surprisingly robust, even as reward model are way less accurate, and multiple feedback types are competitive in this regard. Descriptive feedback seems to be the most robust, only being heavily influenced by noise in one environment (Swimmer-v5), but actually improving its's performance over optimal feedback, which we attribute to an implicit regularization effect. (2) We do see that different environments can be more or less susceptible to introduced noise in general. Environments like Hopper-v5 and HalfCheetah-v5 seem more robust than an environment like Swimmer-v5.
Compatetie
Nobe 0.0
Nebs 0.3
Nebas 0.15
5000
-003
3000
4000
0.75
3033
3003
Noise E-
3000
Nales 2-0
2033
2013
Naise ELL.
2000
1033
1003
Noiee 6-5
1000
Noiie 0:25
Neloe 5.75
3.0
02
0.4
0.6
0.6
0.0
0.
0.4
0.6 Steps
0.
15
0.0
0.2
0.4
0.6
0.8
Corrective
Deccripiive
Descriptive, preference
0033
0003
6000
3023
0.35
5002
5000
1000
1000
4000
3003
Hose D.1.
Mase
Hosec 0:23
Moke 1:23
Nolee 8.3
Steps
Steps
Steae
Figure 42: RL Reward Curves for HalfCheetah-v5 at different levels of noise
Evaluative
Comparative
Demenstrative
4003
Noree
4000
0.25
Horie
0.25
3032
holkaa O.S.
3000
Wolce 2i5
5000
Tacise
Roward
Moiso 0.73
Mase 9.79
10003
1007
1000
Correctivn
preference
Nolde 0.5
Rebe O.S
Nelde 0.5
Nelic 0.3
hobe 0.3
Relic 0.2
Nelic 0.15
Rebe
heac 0.15
3032
3003
Danaaaa
Rawaard
hobe
0.15
2003
2003
2000
1003
1000
1000
3.0
0.2
0.6
0.5
2.0
02
0.4
0.6
0.8
1.2
0.0
0.2
0.5
Stopo
Stops
Figure 43: RL Reward Curves for Walker2d-v5 at different levels of noise.
45

---

Published as a conference paper at ICLR 2025
Evsluative
Companitvve
Semonstrratvve
Hoten
Helen 0.25
Hauu 0.3
150
150
130
Hobe
Holes 0.75
Conrective
Descripivve preference
Moke 0.0
Nebre 0.
Nove 0.0
Volen 0.3
Notra 6.5
holee 0.5
Nobe
No
1:41
100
100
0.2
3.0
0.0
0.2
08
1.0
0.2
0-4
0.5
2.5
3.0
Figure 44: RI. Reward Curves for Swimmer-v5 at different levels of noise
Evakustive
Comparatio aire
Derronatrative
12:0
Mebe 0.1
4303-
2000
Nobic
3000
Mence 0.25
3007
3000
Notue 0.25
2000
Molie
Holla
-1000
-1303
-1000
Steps
Stepss
Corractive
Descriotive
Dencrigh ve preference
4000
Nokia
4202
3000
Nobis D:25
3002
3000
2200
Prlie 2.0
Noke 0.
-1160
-1292-
holiee 2.5
-1000
Holee 0.3 Notue
Steps
Steps
Stopc
Figure 45: RL Reward Curves for Ant-v5 at different levels noise
3000
3000
2500
2000
Meloe
Modee Mosue
1000
L000
Macue 0.29
1000
0.75
0.75
Makia
1.5
Stepps
Stegss
Descriative
Descriptive_eeddnee perfrittieeeeeeeeeeecee
Notoc 0.3
2000
2000
Noken 0.75
1500
1000
Nokis 0.25
1000
Malua 0.25
00
0.2
0.4
1.5
0.2
0.4
0.0
0.4 2.6 0.0
Steps
Steps
Stops
Figure 46: RL Reward Curves for Hopper-v5 at different levels of noise
46

---

Published as a conference paper at ICLR 2025
Evaltaative
Campor stive
Demorsttatiio
2000
20000
1000
1000
1000
Steps
Steps
Steps
Cornectioe 1M
Describlive
Descriptive prefesence
3000
5000
5000
Y
2000
2000
11000
1000
D600
S7200
Stops
Stops
Figure 47: RL Reward Curves for Humanoid at different levels of noise
47

---

Published as a conference paper at ICLR 2025
E.4 DETAILED REWARD CURVES FOR RL TRAINING WITH VARYING AMOUNTS OF FEEDBACK
The following plots show the downstream RL performance for agents trained with varying amounts of feedback. We created datasets with fewer feedback instances by sampling random segments/state clusters from the full dataset. Rewards are aggregated over 3 seeds per amount of feedback.
5032
4033
4000
2003
Nanond
1037
#Toxc 5033
#Tbxc 2507
3200
artt 1260
#rtedk 2500
1000
#foett 2500
fbock 5000
#Taac 1253
afta 1750
#Tbuck 1230
-1007
Stops
Steps
Stops
*Teec 10000
aebe 30000
5000
4032
Tbace 1250
4200
ah-lisk J0C00
efback 3250
3003
Tbacc
3000
1500
3000
wrbb 750
2000
k7.kkck 1250
oftb 750
-1003
-1200
-1003
5.50
0.75
3.00
0.00
1.50
0.33
0.50
Figure 48: RL Rewards for HalfCheetah-v5 with different amounts of feedback.
beck 30000
beck 30000
4000
#bock 10000
Beck 5000
5000
5000
2200
busk 2500
25:00
buct
750
Rewaadd
2033
2003
2000
3.0
02
0.6
0.5
1.5
3.0
02
0.4
0.6
0.8
1.2
0.0
0.2
0.4
0.5
0.8
Steps
Staps
bebbck 5000
5000
#fbact 5000
3033
ell bect 2200
3250
3003
2500
3250
3000
#ibbct 2000
#lback 1250
1000
1000
0.0
022
0.4
0.6
0.8
Staps
Steps
Stace
Figure 49: RL Rewards for Walker2d-v5 with different amounts of feedback.
We find that all feedback types are sensitive to decreasing number of feedback instances. However we still can highlight some particular observations: Comparative feedback is relatively efficient, showing better performance with few feedback instances. Other types, like evaluative and descriptive feedback show noticeable drop-off for fewer instances. However, sub-sampling might impact evaluative and descriptive feedback types an particular manner: Because the reward histograms, and the clustering are computed for the entire dataset, sampling from the full dataset is not representative of recomputed statistics/a recomputed coarse clustering. We therefore report this experiment as an important indictor for the stability of different feedback types, and encourage future exploration.
48

---

Published as a conference paper at ICLR 2025
KRENK 1000
200
2500
250
2000
293
120
1250
256
1250
Rewardd
-30
100-
302
#lacc 32803
#facc 3590
#laacc
#Foucc 750
10
0.6
oftedh 10000
#Tteck 10000
#faucc 32903
250
eltbeck 1250
#iback 1250
#Raacc 3253
110
192
759
100
I00
300
5.0
0.0
0.1
1.0
80
0.2
0.8
3.6
0.
0.2
0.4
0.6
2.5
1.0
Figure 50: RI. Rewards for Swimmer-v5 with different amounts of feedback.
3000
#rbckk 5000
arbeck 2500
2002
3000
#fbect $000
#ffocc 2390
2000
2002
2000
750
1000
3300
#Pilyck 1eice
1000
entaect
ertbach 1250
Steps
Steps
Steps
2000
#lback 5000
brbck 2500
3002
#tback 2500
My "M
3000
2000
2393-
2000
1000
3300
1000
#Taecc 32803
1000
1303
1000
#Rlsac 3250
753
Steps
348
Steps
Stopc
Figure 51: RL Rewards for Ant-v5 with different amounts of feedback.
DH 10303
2500
2250
2000
2000
#ffeect 5002
1000
#Rlicce 2307
#lloc 2530
2252
2210
0.
05
Stepes
Stegrs
Stegss
3500
2500
3500
#Hbock 50.00
#Tascc 30000
2000
#becc 352
1500
1000
#dbuuc 2503
1000
2233
00
02
26
1.5
0.2
0.4
0.0
0.4 0.6
00
Steps
Steps
Stops
Figure 52: RL Rewards for Hopper-v5 with different amounts of feedback.
49

---

Published as a conference paper at ICLR 2025
#Tbect 13000
3000
#Tbecc L250
0000
2000
10903
Coasct 5023
Easscc 5200
#fousk 2333
2200
#Rloe 752
PRAAACC 7700
0.01
0:25
0.75
1.2
0.50
0.25
0.50
0.39
1.00
335
0.0
0.5
DEbacc 13000
3300
4000
4000
4000
3000
1000
#Plocc 9839
#io 9200
2328
PRiacc 2000
1000
#ftoock 752
#fbscc 750
0.22
825
0.59 0.75
3.00
1.35
0.23 3.55 0.58 0.75 1.00 1.2 Steps
0.0
2.5
Stopo
Stops
Figure 53: RL Rewards for Humanoid with different amounts of feedback.
E.5 BEHAVIORAL CLONING BASELINES
For each environment, we also train baseline via supervised behavioral cloning (BC), As the training dataset, we utilize the demonstrations, generated for the feedback data. Segments are flattened, i.e. 10,100 segments of 50 steps each lead to 500,000 state-action pairs for training of the policy. For the policy network, we use a two-layer neural network with 32 units (as the default in the imitation library Gleave et al. (2022)) Additionally, entropy-regulaarizatoo is used to avoid over-fitting of the supervised policies. We have trained networks with a batch size of 32, and 20 maximum epochs on the dataset. However, we often did not see significant improvement after around 5 episodes.
F JOINT-REWARD MODEL DETAILS
F.1 TRAINING CONFIGURATION
Similarly the other training runs, chose default training parameters. However, more extensive hyper-parameter search might yield better performance the future.
For training. the RL algorithm is unmodified from a normal training setup. We only adapted the reward function. As before, we implement learning with the multi-type ensemble via a wrapper around the environment. This time, instead of single reward function, load a set of reward functions.
For a state-action pair, each individual reward model is queried and predicts a reward. As mentioned above, we standardize the individual reward functions, they be in dissimilar value ranges. We do this by standardizing each individual reward function, by subtracting the rolling mean, and dividing by the rolling standard deviation. We compute the rolling standard deviation with Welford's algorithm Welford (1962).
Given the ensemble of single feedback
reward functions
Ryb
{fevet, F fommp ddemos foorr, dessor ddeser freeerr.re...... we implement two variants:
Averaging: We just average the normalized reward into a joint reward prediction
r(s, a) = RRbbl Piype ERy
Σ
Ttype (s, a)
Uncertainty-Weiglle Ensemble: Each single feedback reward model is an ensemble in itself relying the Masksembles architecture). We can therefore receive approximate posterior probability for each feedback type, i.e. a mean and standard deviation between the sub-model predictions. We experimented with using these standard deviations as uncertainties for each feedback type. We then weighted the reward predictions with the inverted standard deviation, i.e., feedback type with lower uncertainty for state-action
50

---

Published as a conference paper at ICLR 2025
pair contributes to the prediction.
Ei=1
V
1=
51


Reviews:
[{"review_id": "vo34p4RxLh", "reviewer": "Reviewer_PQUp", "summary": "This paper thoroughly investigates learning from different types of human feedback, including defining various types of human feedback, detailing how to collect synthetic feedback for research purposes, and training reward models based on synthetic feedback. It also analyzes the performance of joint training with multiple feedback types and examines the effectiveness of different feedback types.", "questions": "1. Figure 3 obscures the text.   \n2. Some figures are not analyzed, and there are more important results in the appendix, requiring a restructuring of the paper.        \n3. Mean and standard deviation are not provided.             \n4. Why is the correlation of some reward functions so low?         \n5.Is the method for training reward models online? (i.e., continuously updated online with new samples)", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.Studying different types of human feedback is extremely important and can promote research development in the RLHF community.  \n2.Provides methods for generating various types of synthetic feedback for future RLHF research.         \n3.For the first time, it proposes training with multiple feedback sources while considering human noise.        \n4.The various analyses of reward models in the main paper and the appendix are comprehensive.", "weaknesses": "1.The workload of this paper is substantial, covering many key points, which results in relatively preliminary research on each type of feedback. The characteristics of different feedback types are not well demonstrated. Can you describe several key feedback types or explain which feedback types are more suitable for specific scenarios?      \n2.The first half of the paper is well-written, but the experimental organization in the latter half is chaotic, making it difficult to draw clear conclusions.\n\nOverall, I believe this paper is highly valuable, but the current version appears somewhat hasty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iBkfzGM8IG", "reviewer": "Reviewer_rXcY", "summary": "The paper develops a lightweight software library for generating six feedback types—rating, comparative, demonstration, corrective, descriptive, and descriptive preference—in the field of Reinforcement Learning from Human Feedback (RLHF). The library is compatible with established RL frameworks, including Gymnasium, Stable Baselines, and Imitation.\n\nThe paper introduces noise into the synthetic feedback. Experimental results are presented on basic Gym Mujoco locomotion tasks. In these experiments, the authors compare the learned reward functions and the agent’s performance across different feedback types, both with and without noise. Additionally, the authors present a joint reward feedback method, i.e., an ensemble of rewards, and compare it to single feedback type baselines. This approach performs well in the HalfCheetah-v5 environment but is less effective in Walker2d-v5.", "questions": "1) How diverse is the distribution of trajectories in line 197? Can you provide quantification of the exploration?\n\n2) Is there any analysis for Figure 2, specifically why demonstrative and corrective feedback appear significantly different from other methods?\n\n3) How can you ensure it is fair to compare different feedback types given that noise is generated separately for each type?\n\n4) What are the hardware requirements and other support needed, along with the expected training time and latency based on these specifications?\n\n5) Are there any results available for environments beyond Mujoco?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "With the development of RLHF, there has been an exponential growth of research, especially in interdisciplinary applications, such as large language models (LLMs). I appreciate that it is important to have a standard library of feedback types commonly found in established RL frameworks. This is undoubtedly helpful for both new and experienced researchers in this rapidly evolving field. The choice of RL methods (e.g., PPO) and environments used in the paper are standard and widely accepted in the literature.\n\nFurthermore, by encompassing multiple feedback types, this library has the potential to standardize RLHF research, making studies more comparable and reproducible across the field. Unlike prior work that often focuses on single feedback types or limited noise modeling, this toolkit provides a broader, more robust framework, which could foster progress in handling realistic feedback conditions in reinforcement learning.", "weaknesses": "1) Limited Analysis of Feedback Types and Noise Effects: The paper provides only a shallow analysis of the results across different feedback types and the impact of adding noise. The authors introduce Gaussian noise as a way to simulate realistic inconsistencies in human feedback, which is a valid approach. However, they assume that the added noise will uniformly challenge the agent’s learning process, yet they provide limited empirical support to demonstrate the nuanced effects of this noise. In reinforcement learning, robustness to noise is complex and context-dependent; simply adding noise doesn’t necessarily simulate real-world variability comprehensively. This is particularly relevant in cases where the agent encounters unfamiliar scenarios, as it may lack the generalization needed to adapt successfully, which is not addressed here. It would be beneficial if the authors quantified the noise across feedback types and analyzed how this type-specific noise impacts learning stability and performance. Without such quantification, comparing the robustness of different feedback types remains somewhat speculative and may not provide fair insights. As highlighted by Casper et al. (2023), inconsistencies in human feedback are a fundamental limitation in reinforcement learning from human feedback, underscoring the need for a systematic approach to evaluating robustness in such noisy environments.\n\nReference: Casper, Stephen, et al. \"Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback.\" Transactions on Machine Learning Research, 2023.\n\n2) Reward Ensemble Approach is Underdeveloped: The paper attempts to improve reward learning through a reward ensemble (combining different feedback types), which is an intriguing direction. However, the ensemble approach lacks depth and does not yield substantial improvements. The authors themselves acknowledge that rewards from different feedback types cannot be averaged simply, and yet, the methods presented for combining them are fairly straightforward. Given the challenging nature of ensemble reward learning, the results are not strong enough to warrant this as a core contribution. As such, this component appears more exploratory than foundational, and it could benefit from either more sophisticated ensemble techniques or a more extensive evaluation to validate its potential utility.\n\n3) Insufficient Details as a Software Library: As a contribution to reinforcement learning tools, the paper lacks critical details expected from a software library description. While the library supports diverse RL environments beyond the basic Gym Mojoco tasks, only these tasks are presented in the main text. Further, there is little mention of essential details such as hardware requirements, training time, or memory usage, which are crucial for reproducibility and practical use by researchers. Although some hyper-parameter details are provided in the appendix, this is insufficient for a full understanding of the library’s operational requirements and expected performance.\n\n4) Typos and Formatting Issues\n\n- Figure 4b title should read “RL Episode Returns.”\n\n- Incomplete sentences in the paragraph near line 380.\n\n- Hidden text in line 356.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FpXWEO3xsn", "reviewer": "Reviewer_oTK8", "summary": "The paper presents a benchmark or toolkit to use to simulate multiple types of feedback in the context of learning preferences from human feedback as well as analyzes one approach to combining multiple sources of feedback (ensemble of rewards). The different sources of feedback include ratings (scalar scores per trajectory), binary preference labels, demonstrations, corrections, descriptions of high value state-action pairs, and descriptions of high value features. The paper describes in detail how each sources of feedback is simulated using an environment's ground truth reward as the human proxy. The bulk of experiments focus on comparing the reward models learned from the different sources of feedback by looking at correlation between learned reward functions and downstream policy performance according to the ground truth reward. Policy performance and reward model accuracy are evaluated in the face of feedback noise, with how noise is applied varying between the different sources of feedback. For evaluating how to combine the multiple sources of feedback, the authors present two approaches to combining an ensemble of reward models, where different mini reward model ensembles are trained from different sources of feedback. The primary take away is that there is no one best source of feedback across tasks.", "questions": "* In Figure 5, the results are described as averaged over \"3 feedback datasets\", what are the sources of the different feedback datasets? Are these different random seeds? Earlier it was stated that 5 random seeds were used.\n* How well do the feedback ensemble rewards correlate with the ground truth reward function?\n* In Section 4.2 you talk about sampling against random behavior being stronger than agains sub-optimal rollouts for demonstrative feedback. This is an interested conclusion and I would have expected the opposite. Why do you think this is the case?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses an important problem, which is providing a toolkit/benchmark for people to use to research learning from different types of feedback and how to combine them. The approach is similar to what has already been used to learn from binary preference labels alone, which makes it an easy toolkit for people to pick up and understand how it works. \n- The experiments and results demonstrate that accounting for multiple source of feedback is neither straightforward nor trivial, and work is needed to understand the strengths and benefits of each.\n-The proposed toolkit does not rely on actual humans in the loop making it something many people can use easily for initial proofs of concepts. \n- The toolkit provides mechanisms to have different sources of noise, which is crucial as we know human feedback is noisy. The code is note yet provided, but from the description in the paper, it seems like running experiments with different amounts of noise should be fairly straight forward.", "weaknesses": "**High-level overview:**\n\nThere are two main weaknesses of this paper discussed at a high level here, but more details below. The work is valuable and necessary, but the needed level of rigor isn't there yet.\n     (1) the toolkit it not validated against any studies involving humans, therefore it is impossible to know if the conclusions drawn in the paper reflect characteristics of the feedback or of the implementation\n     (2) the text in the paper, especially in the second half describing experiments and results, contradicts itself and presented results making it difficult to draw conclusions and have take aways or learnings.\n\n**The details:**\n* The way the paper is written suggests that both the multi-feedback toolkit and the method for combing the multiple sources of feedback are both equal contributions. I would push back that the main contribution is the toolkit and the proposed method is rather trivial with no true baselines to show its benefit or contribution. It would have been better to limit the feedback combination methods as a tool for proof of concept showing how the toolkit can be used. \n* The authors draw conclusions about the usefulness of different types of feedback (e.g. Figure 2 reward model correlation and demonstrative and corrective having the worst correlation with the ground truth reward). However, in the absence of a study to validate the toolkit against human feedback makes it impossible to validate if this is a function of the feedback, or the choices they have made in how the feedback is implemented.\n* Only one environment is evaluated therefore it is not possible to assess how general the proposed toolkit and feedback simulation method are. At least one other should be included. MetaWorld is popular for the preference learning and would be a natural fit as it is included in [BPref] (https://github.com/rll-research/BPref).\n* There are multiple places in the paper where the authors seem to contradict either themselves or the presented results:\n     * at the start of Section 4.4, the first sentence sounds like there was not online reward model adaptation, but then the last two sentences to that first paragraph make it sound like there was: \"...we utilize simple pre-trained reward models instead of continuously adapting the models...\" versus \"The reward model is continuously updated...\"\n     * the text in the main body does not mentioned that the reward model correlations for other tasks beyond half cheetah have are weaker with different patterns among the feedback types. The full nuance of the results are not represented, and instead strong language and conclusions are drawn about a single task.\n     * the statement on lines 325 - 326 \"...all feedback types individually can learn reward functions that closely match the ground-truth reward...\" is not true for all feedback types according to the Figure 2 and those in the appendix. In Figure 2, demonstrative and corrective have a correlation of 0.61 and 0.5, which is a medium match. In Figure 25, these feedback types have correlations as low as a 0.18.\n     * on lines 410 - 411, it is stated that \"...no feedback type is obviously more or less robust to noise.\" however, then looking at Figure 4(b) and Figure 5(b), the performance gap between learning curves across different amounts of noise varies across feedback types, which suggests that some are more sensitive that. others. To back up the claim, \"obviously\" needs to be quantified to make it clear the threshold for what qualifies as \"more or less robust\".\n     * section 5 opens by stating that different feedback types struggle in different scenarios, but all of the discussion in Section 4 talks about how there is no clear difference between the different sources of feedback.\n* The first half of the paper motivating and describing the multiple feedback toolkit is very well written, methodical, and easy to follow. However, there is a switch part way through the paper when it transitions to talking about experiments with the toolkit. Here the presentation and writing changes drastically with things like: \n     * Figure 3 overlapping the text of the main body\n     * an incomplete sentence (page 8 line 380)\n     * what seems to be misconnected analysis and results figures (e.g. figures 4 and 5 - the text seems to talk about figure 4, but references figure 5 and figure 4 is not mentioned in the paper as far as I can tell)\n     * mislabelled and titled figures where the captions disagree with the figure axis/title (e.g. Figure 2 - HalfCheetah-v5 versus Swimmer-v5; Figure 5 - swimmer v5 and walker2d-v5 versus half cheetah-v5; and figure 4 - \"reward model validation curves\", but it looks like policy returns)\n     * Figure 4 left plot, the y-axis scale are on different scales, making it tricky to compare across feedback sources.\n     * using both a : and - on page 8 line 429\n     * it is stated that results are over 5 random seeds, but there are no standard deviation results\n     * there are numerous places with typos (e.g. \"as the be considered\" line 256) that need to be addressed.\n     * not all lines in Figure 3 are labelled nor described.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WSWQhLQO7C", "reviewer": "Reviewer_vPP3", "summary": "This paper explores reward learning utilising multiple different types of feedback, covering evaluative, instructional, and descriptive forms.\n\nThey propose a method on how to generate these forms of feedback synthetically, how to learn from these feedback methods in isolation, and also how to combine these reward signals to learn from many types of feedback simultaneously.\n\nAdditionally, they investigate the different properties of these feedback methods such as their correlation, and ability to provide a reward signal for learning an RL policy.", "questions": "Does the method of reward modelling proposed work to learn both an RL policy and a reward model *without pre-training the reward model*?\n\nWhen utilising demonstrative and preference feedback, a common method is training first on the demonstrations, e.g. using MaxEnt or SFT / Behavioural cloning, and then fine-tuning on the preferences with RLHF.\nIt would be interesting to see a comparison to this baseline method.\n\nHow do reward function correlations vary across environments and across trajectories?\n\nIt would be good to run multiple seeds and plot mean and standard error/deviation in the figures.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Research is well motivated.\n\nExisting work is nicely leveraged and inter-twinned to form basis of paper without re-inventing the wheel.\n\nPaper is reasonably well written, with ideas are clearly communicated.\n\nThe proposed method to generate synthetic feedback seems good.\n\nThe proposed method to learn from multiple feedback types, namely training separate reward models and combining their scores, is clean, simple, and easy to understand.\n\nThe noise model used seems reasonable.\n\nPaper considers alternative formulations of human feedback, namely regret-based.\n\nExploring the reward model correlations is good analysis.\n\nFor the most part, the paper tests on many environments with many noise levels, giving strong evidence for some of their claims.", "weaknesses": "Learning a reward model for each feedback type presents some limitations not considered by the authors.\n* If these reward models are large or computationally expensive (e.g. in an foundation model finetuning setting where each one may be foundation model sized), then having one for every feedback type might not be very scalable or practical.\n* The resulting reward signal cannot incorporate information only deducible from considering multiple feedback types at once.\nFor example, there may be some patterns or generalisations only apparent when considering all feedback.\n* If there's only small amounts of one type of feedback, its corresponding reward model might be very inaccurate, and if the uncertainty for the uncertainty-weighting is not well calibrated, it could lead to that reward model making the overall reward signal worse.\n\nThe paper only considers synthetically generated feedback and does not evaluate / verify with real human feedback.\n\nThe paper focuses on continuous control tasks, and does not test on any discrete or contextual bandit environments.\nThis is important to test as this more closely corresponds to the LLM fine-tuning setting, a key application of RLHF and related methods.\n\nInterpreting the demonstrations as being preferred only to random trajectories seriously limits the information extracted from them compared to standard demonstration learning approaches like MaxEnt.\nDespite the authors claims suggesting this was \"stronger than sampling against other sub-optimal rollouts\", the amount of interesting state space explored by random policies decreases exponentially as the environment gets larger and more complicated.\nFor example, in the language modelling setting, interpreting your demonstrated responses as better than random tokens would probably not improve your pre-trained model compared to supervised fine-tuning.\n\nThe feedback methods, except one, all use the same fixed segment length.\nThis may be sub-optimal if different feedback methods work better with different sized segments.\n\nThe reward model correlation analysis is only done on one environment and are averaged over the whole trajectory.\nIt's not clear how consistent these correlations are across different environments, and it may be interesting to see how these correlations change over the course of a typical trajectory (E.g. maybe some are well correlated to begin with, and then become de-correlated).\n\nGraphs have no error bars, and it's not clear how many random seeds have been averaged over in the RL experiments.\nIf there are not multiple random seeds, then very little of the results presented can be assumed to be statistically significant.\n\nThe reward models are pre-trained before being used to train a policy.\nThis is very atypical for RLHF and limits the conclusions that can be drawn from the results.\n\nIn figure 3, not showing the score for learning from the ground truth reward as a function of environment time steps limits comparisons between learning from the feedback modes vs the ground truth itself.\n\nIt's not clear how amounts of each type of feedback are controlled to be comparable.\nIt seems the amount of each type available is very dependent and sensitive to parameters of the synthetic generation process.\n\nFor uncertainty-weighted ensembles, the uncertainty of an ensemble that has been trained on feedback that only constrains reward differences (e.g. comparative) is not well-calibrated.\n\nI do not believe some of the claims in the discussion and conclusion are well-supported.\n* On point (2), line 495/496, combining rewards is only tested on two environments and only performs well in one of them.\n* Line 523/524, the comparison of these characteristics has been very limited.\n* Line 526 to 528, learning from multiple feedback types being \"very effective\" is not supported by the evidence presented in figure 6.\n\n## Errata:\n* Figure 2 caption and subcaptions disagree on what environment the results are from (Swimmer-v5 vs HalfCheetah(-v5)).\n* Figure 3 partially covers some text\n* The paragraph at the start of section 4.4 conflicts with itself: \"instead of continuously adapting ... is continuously updated\"\n* Line 380, sentence abruptly ends and is unfinished\n* The axes and legends for figures 2,3,4, and 5 (especially 3), are too small to clearly read, especially on a printed copy of the paper.\n* The lighter shaded lines in figures 4, 5, and 6 are hard to see.\n* There is no reference to figure 4 in the main text of the paper, nor analysis of what it shows.\n* Line 454, \"As stated before, each individual reward model ... is an ensemble in itself\". This does not appear to have been stated before.\n* Line 467, the text refers to the \"Hopper-v5\" environment, but the figure under discussion, 6, only contains the HalfCheetahv5 and Walker2d-v5 environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)