Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning
Authors: Calarina Muslimani
Matthew E. Taylor
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Human-in-the-loop
Preference learning
Learning from scalar feedback
Primary area: reinforcement learning

Abstract: To create useful reinforcement learning (RL) agents, step zero is to design a suitable reward function that captures the nuances of the task. However, reward engineering can be a difficult and time-consuming process.  
Instead, human-in-the-loop RL methods hold the promise of learning reward functions from human feedback. Despite recent successes, many of the human-in-the-loop RL methods still require numerous human interactions to learn successful reward functions.
To improve the feedback efficiency of human-in-the-loop RL methods (i.e., require less human interaction), this paper introduces Sub-optimal Data Pre-training, SDP, an approach that leverages reward-free, sub-optimal data to improve scalar- and preference-based RL algorithms. In SDP, we start by pseudo-labeling all low-quality data with the minimum environment reward. Through this process, we obtain reward labels 
to pre-train our reward model without requiring human labeling or preferences. 
This pre-training phase provides the reward model a head start in learning, enabling it to recognize that low-quality transitions should be assigned low rewards. Through extensive experiments with both simulated and human teachers, we find that SDP can at least meet, but often significantly improve, state of the art human-in-the-loop RL performance across a variety of simulated robotic tasks.

Published as a conference paper at ICLR 2025
LEVERAGING SUB-OPTIMAL DATA FOR HUMAN-IN- THE-LOOP REINFORCEMENT LEARNING
Calarina Muslimani University of Alberta musl iman ua berba.a... ca
Matthew E. Taylor
University of Alberta
Alberta Machine Intelligence Institute mat mat wetthhew.rr e www..eety.oorruaabrata.ccooo
ABSTRACT
To create useful reinforcement learning (RL) agents, step zero is to design suit- able reward function that captures the nuances of the task. However, reward engineering can be difficult and time-consuming process. Instead, human-in- too-loop RL methods hold the promise of learning reward functions from human feedback. Despite recent successes, many of the human momman-in-theee in-the--loop RL methods still require numerous human interactions to learn successful reward functions. To improve the feedback efficiency of human-in-the-loww RI. methods (i.e., require less human interaction), this paper introduces Sub-optimal Data Pre- Fre-training, SDP, approach that leverages reward-fiee free, sub-optimal data to improve scalar- and preference- baaedd RL. algorithms. In SDP, we start by pseudo-labeling all low- quality data with the minimum environment reward. Through this process, we obtain reward labels to pre-train our reward model without requiring human la- beling preferences. This pre-training phase provides the reward model a head start in learning, enabling to recognize that low-quality transitions should be as- signed low rewards. Through extensive experiments with both simulated and hu- man teachers, we find that SDP can at least meet, but often significantly improve. state of the art cumann-n--te--opee RL performance across a variety of simulated robotic tasks.
INTRODUCTION
In reinforcement learning (RL), an agent's objective is to interact with an environment and maximize its total (discounted) expected reward. The reward hypothesis further maintains that well-specified reward function is sufficient for an agent to learn to solve a task (Sutton & Barto 2018). However, defining reward function that precisely captures all task complexities is often tedious and non- trivial (Booth et al. (023). There have been notable examples of reward misspecification, in which RL agents discovered and exploited unintended shortcuts in the reward function (Skalse al., 2022). One notorious example is the CoastRunners game, in which the goal should be to finish a boat race as fast as possible an RL agent instead gained the most reward by spinning its boat circle despite concurrently catching on fire and crashing into other boats (Clark & Amodei 2016).
promising alternative is learn reward functions directly from human feedback. In this paradigm, humans can provide feedback in the form of preferences or scalar signals, which can then be used learn reward function that consistent with human desires (Daniel et al.) 2014 Christiano et al. 2017). Despite recent progress, existing preference- and scalar-based RI. methods still suffer from high human labeling costs that can require thousands of human queries to learn an adequate reward function (Christiano et al.) 2017). Prior work attempts to mitigate this issue through several mechanisms, including active learning (Lee et al.) 2021a),, data augmentation (Park et al. 2022). semi-supervised learning (Park al. 2022), and meta- leeannle (Hejna III & Sadigh 2023).
Alternatively, our work draws on recent advances in offline RL that have demonstrated the value of low-quality data (Yu al. (021). However, its potential in human-in-the tn-the-loop RL remains unex- plored. As low-quality data is often readily accessible or easy to obtain, this work addresses this gap by asking the question:

---

Published as a conference paper at ICLR 2025
Can leverage sub-optimal, unlabeled data to improve learning in human-in--tee -ntthe-loop RL methods?
To that end, we present Sub-optimal Data Pre-training, SDP, a tool for human-in-the-logg RL al- gorithmss to increase human feedback efficiency. SDP leverages sub-optimal trajectories by pseudo- labeling all transitions with the minimum environment reward. The now pseudo-labeled sub-optimal data serves two purposes. First, we pre-train& a regression- basedd reward model by applying standard supervised learning to minimize the mean squared loss. Intuitively, this pre-training provides the reward model a head start, biasing it towards assigning lower reward values to these low-quality transitions. Second, we initialize the RL agent's replay buffer with the sub-optimal data and make learning updates to the RL agent. This process changes the RL agent's 's policy and provides differ- ent behaviors for the human to provide feedback on (relative learning with no initial sub-optimal data). This ensures that when the human teacher provides feedback, their time is used efficiently. avoiding redundant feedback on the existing sub-optimal data. Afterward, we follow the standard preference- or scalar- based RL protocol.
This paper's core contribution showing that we can harness the availability low-quality, reward- free data for human-in-the-loe in-the-loop RL approaches by pseudo-labeling it with minimum rewards and treating it as a prior for learning reward models. We first validate the utility of SDP in extensive simulated teacher experiments, combining it with four scalar- and preference-based RL algorithms.
These experiments show that SDP significantly improves the feedback efficiency in complex tasks from both the DeepMind Control (DMControl) (Tassa et al.] 2018) and Meta- World (Yu et al. 2020) suites. Crucially, we further highlight the real-world applicability of SDP by demonstrating its success with human teachers in a 16- 16-person user study. Overall, this work takes important step toward considering how human-in-the-loop RI. approaches can take advantage of readily available sub-optimal data.
2 RELATED WORK
Human-in-the-Logg RL Several approaches in human-in-the-loop RL allow agents to leverage human feedback to adapt or learn new behavior. Learning from demonstration one such methodol- ogy that allows human to provide examples of desired agent behavior (Argall et al. (2009). Human demonstration data has been used to shape the environment's reward function (Brys et al.) (015))
develop reward function from scratch (Abbeel & Ng 2004), or bias the agent's 's policy towards certain actions [Taylor et al., 201T). Although demonstrations can be a rich source of feedback, they often expensive obtain and may require domain experts (Dragan & Srinivasa 2012).
feedback where a teacher provides preferences between two more sets of agent behavior (Christiano et al. 2017). Preference learning has been popularized in recent years as it can require less effort and expertise compared to providing demon- strations. To further reduce the amount of human interaction required, several strategies have been introduced. This has included combining preferences with demonstrations (Ibarz et al.) 2018 Bryik al. 2022). unsupervised pre-training (Lee et al., aa), bi-level optimization (Liu et al. 2022). semi-supervised learning (Park et (002)) data augmentation (Park et al. 2002). uncertainty- based exploration (Liang al., 2022). meta-learning (Hejna III & Sadigh 2023). and active learning ap- proaches (Hu et al., 2024). Despite its popularity, some argue that comparison feedback might not capture the full intricacies of human preferences, as oftentimes the human is limited to choosing between two options (Daniel et al. 2014 White et al.) 2024).
Another approach is learning from preference-
As a result, another body of work focuses on learning from scalar feedback where human teachers can provide scalar signals to evaluate an agent's behavior (Knox & Stone| [2009] Griffith et a 2013 Loftin al. [0166] White et al. 2024). Several works use scalar feedback to either learn a reward model (Daniel et al. 2014] Cabi et al.) 2020)
action-value function (Knox & Stone, 2009
2013 Warnell et al. 2018) via regression.
Learning from Sub-Optimal Data SDP aims to leverage sub-optimal data for scalar- and preference-bact RL algorithms. However, learning from low-quality data or negative examples has been applied in other areas of RL and imitation learning (Chen et al. 2021 Tangkaratt et al. [021 In standard RL, several works use sub-optimal demonstrations to initialiee a policy (Taylor et al.
2011 Hesster et al. 2018
Gao al. 2019). In goal-conditioned RI., Finnsight-Experiiencee
2

---

Published as a conference paper at ICLR 2025
uses failed episodes by treating them success with respect to a different goal (Andrychowicz iet al. 2017). In inverse reinforcement learning (IRL), Shiarlis et al. (2016) proposed a constrained optimization formulation that can accommodate both successful and failed demonstrations, Brown (et al. (2019) makes use of ranked demonstrations to learn a reward function in IRI.. Later work in IRI. automatically generates ranked trajectories by adding increasing amounts of noise learned policy (Brown et al., 2020). Lastly, in offline RL, |Singh et al. (2020) leverages sub-optimal tran- sitions from multiple prior tasks and assigns reward labels according to the current task reward function. Our work is most closely related to Yu et al. (2022), which leverages reward-free free, sub- optimal data in the offline RL setting by pseudo-labeling all transitions with zero and adding them to the RL agent's replay buffer. However, we found that directly applying this approach to the numan-in-the-lowe RL setting was ineffective (see Appendix D
BACKGROUND
In the RL paradigm, agents interact with an enviromment to maximize the total (discounted) expected reward it can achieve. This interaction is modeled as a Markov Decision Process (MDP) which con- sists of (S, A, r,), At every time-step f, the agent receives state 8 from the environment and chooses action d, E A. The transition function, T, determines the probability of transitioning to state 81+1 and receiving reward T++1, given the agent was in state s; and executed action aN The environment then provides the agent Tt+1- The agent attempts to learn policy, π S - A, that maximizes the expected return E|G] = Ei= " r which is defined as the expected sum of discounted future rewards with discount factor y E [0, 1).
3.1 REWARD LEARNING FROM HUMAN FEEDBACK
This paper assumes that we are in a reward-fres free paradigm, an MDP/R setting, where our goal is to (1) learn a reward function, f, from human feedback and (2) learn a policy that maximizes the total (discounted) expected f. We follow the standard reward learning framework that uses supervised learning to learn a parameterized reward function, fg. with parameters 0 (Christiano et al. 2017). In both scalar- and preference-basee settings, we consider trajectory segments a, where consists of sequence of states and actions: {st, ae, S141, at+1, St+k: dt with k as the segment size.
Preference-based Reward Learning In preference- learning, two segments, σº and a', are compared by a teacher, yielding y E {0, 1., Specifically, if the teacher preferred segment of over segment σ then y is set 1, and if the converse is true y is set to O. If both segments equally preferred, then y is set to 0.5. As feedback is collected, is stored as tuples (o", a- y) in the reward model data set DRM- In general, a' > σ², then the segment o² is preferred by the teacher over segment od, We follow the Bradlcy-Terry model (Bradley & Terry 1952) to define a preference predictor using the reward function Ag:
exp E, Fo Po > 0d) Eie(o,1) exp(E, (o af))
Intuitively, this model assumes that the probability of the teacher preferring a segment depends exponentially the total sum of predicted rewards along the segment. To train the reward function, we can use supervised learning where the teacher provides the labels y. More specifically, we update tg by minimizing the standard binary cross-entropy objective:
L C (0, D)
Eia wDDD [[1 y) logPo(o" ot) + y log on]]
(2)
Scalar-based Reward Learning The primary difference between scalar and preference-based re- ward learning is that in scalar-based learning, the human teacher assigns numerical ratings to trajec- tory segments. In this setting, the comparisons between segments are implicit. More concretely, a teacher assigns scalar value y segment of, and as feedback is collected, it is stored tuples (a y) in the reward model data set DRM- We then apply standard regression and update ña by minimizing the mean squared error:
L MSE (0, D) = BOoD (1 o()))
(3)
3

---

Published as a conference paper at ICLR 2025
Human Studies in Human in the Loop RL To evaluate human-in-the-loe RL algorithms, common protocol is the use of simulated teachers, where feedback is provided according to a ground truth reward function. This can be useful, as it offers efficient and controlled evaluation setting. However, we argue that is essential to evaluate human-in-the-loog RI. algorithms with real human feedback. This is especially important in light of recent work finding discrepancies between prefer- ence learning algorithms when using simulated teacher feedback versus human feedback (Metcalf al. 2024).
4 SUB-OPTIMAL DATA PRE-TRAINING
In this section, we present SDP, tool that leverages sub-optimal trajectories to improve the feedback efficiency for human-in-the-loop RI.. We refer to sub-optimal trajectories as sequences of (8, a) pairs such that:
(8t,, az) - Tmin <e AE E [1,++M] for some small
>> 0,
(4)
where l is the segment size and Tmin is the minimum possible environment reward. Equation essentially expresses that the rewards achieved along a sub-optimal trajectory should be close to T'in- However, it is important to note that in practice we do not have access the ground truth reward. This prevents us from directly identifying sub-optimal trajectories using Equation| Instead, we rely on selecting trajectories that we estimate will align with this criterion, such as gathering trajectories via a random policy.
Once sub-optimal trajectories collected, take the approach of pseudo-labeling all transitions with T'in- The goal of SDP is then to use this pseudo-labele data to create a prior for reward models in human-in-the-lae conjunction with any off-the-shelf human-in-the-loop RI algorithm that learns a reward function from human feedback.
RL methods (see Figure []. Its simplicity enables SDP to be used in
SDP comprises two phases: (I) the reward model pre-training phase and (2) the agent update phase. In the reward model pre-training phase, we first gather data set, Deab. of N sub-optimal state, action transitions. We then pseudo-label all transitions in Dsub with rewards of Tmin- resulting in Daub = {&s, ass Tmin Ji-1 Daub then used to optimize the reward model Ae with the mean squared loss in Equation 3] As result, the reward model řo learns to associate all sub-optimal transitions with a low reward. Without such a prior, the reward model would initially have random estimates for these transitions; while the only way to improve such estimates is to obtain feedback from a teacher. Therefore, the reward model pre-training phase provides a valuable reward initialization without requiring any human feedback.
Next, the agent update phase, we initialize the RL agent's replay buffer Dagent with Suah. The RL agent then briefly interacts with its environment and performs gradient updates according its loss functions. The agent update process changes the RL agent's policy and generates new transitions, which are then stored in both the agent's 's replay buffer Dagent and the reward model's 's data set DRM- important to note that in standard scalar- and preference- based reward learning, we query the teacher for feedback on trajectory segments sampled from DRM- Therefore, adding new transitions into DRM during the agent update phase is necessary to ensure that the teacher does not provide redundant feedback the original sub-optimal transitions (as DRM was empty prior to the agent update phase). When it is time for the teacher to provide their first set of feedback, the feedback can cover a different region of the state and action space, relative the original sub-optimal data. In Ap- pendix D.I] Figure & we empirically show that the agent update phase changes the RI. agent's 's policy by performing policy rollouts and analyzing the differences in state distributions. See Algorithm for the complete pscudocode.
At first glance, labeling sub-optimal transitions with an incorrect reward may seem problematic, as incorrect labels do introduce statistical bias into the reward model and the RL agent's 's value network. However, as the transitions are sub-optimal, we observe that the bias for using an incorrect reward is low (see Figure in Appendix D.2). Moreover, by using the sub-optimal transitions, we increase the overall amount of data used by both models, which can decrease the models' variance, as shown in the offline RL setting Yuu al. 2022
4

---

Published as a conference paper at ICLR 2025
(s.
(8.
(s. a.
Reward model pre-training phase
Agent replay buffer
Reward model
get d.llddd
Agent update phase
take action
Figure I: Overview of SDP: After obtaining data set of sub-optimal trajectories, we pseudo-label the transitions with rewards of Tmin (c.g.. Tmin 0). We then pre-train the reward model fe using this data set. During the agent update phase, initialize the RL. agent's 's replay buffer with the same pseudo-labeled data set. The agent then interacts in the environment and makes learning updates to obtain new behaviors for a teacher to give feedback.
Algorithm SDP
Require: Reward model fg &- 0 randomly initialized, Reward model data set DRM + 0, RL agent with replay buffer Dagent f 0, Sub-optimal data set Dauh with reward labels Tmin
1: // REWARD MODEL PRE-TRAIN PHASE
2: for each gradient step do Optimize ig on Daub with MSE (Equation
3:
4: end for
5: // AGENT UPDATE PHASE
6: Dugent + Dsub
7: for each time-step t do Collect 8141 by taking action a. x (8t)
8:
9: Store (st, at, Po, 8t+1) ) in Dagent
Store (st, DRM
11:
Update RL agent with Dagent
12: end for
13: Begin scalar- preference-baseeeeeeeee RL using pre-trained fg. RL agent, and D Dagent DRM
5 EXPERIMENTS
This section considers the following four research questions (RQ's):
RQ Can SDP improve upon existing scalar- and preference-based RL methods?
RQ 2:
Can SDP effectively leverage sub-optimal trajectories from different tasks improve per- formance on target task?
RQ 3:
Can SDP be used with real human feedback?
RQ 4:
How sensitive is SDP various hyperparameters?
5

---

Published as a conference paper at ICLR 2025
5.1 EXPERIMENTAL DESIGN
To demonstrate the versatility of SDP, we apply SDP to both preference and scalar-based RL ap- proaches. However, as preference feedback can be less consuming than scalar feedback, we primarily concentrate on preference-basee RI. in our experiments, exploring scalar feedback in smaller capacity. For the preference- -aaee experiments, we combine SDP with four contemporary preference-based algorithms: PEBBLE (Lee al. [2221a), RUNE (Liang et al., 2022), SURF (Park) et al. 2022), and MRN (Liu et al., 2022). We benchmark the performance of the four algorithms augmented with SDP against their original versions without SDP, well against SAC. We treat SAC (Haarnoja et al. 2018) as an oracle (i.e., upper bound) because it learns while accessing the ground truth reward function, which is unavailable to the other algorithms. For the scalar-based experiments, we combine SDP with R-PEBBLE (a regression variant of PEBBLE). We compare SDP + R-PEBBLE against R-PEBBLE, Deep TAMER (Warnell et al. 2018) scalar feedback RI. algorithm), and SAC. We note that SAC is the core RL algorithm used across all basclines.
Implementation Details For SDP, we collected sub-optimal trajectories via a random policy. In particular, we used 50000 state, action transitions for all experiments in Section 5.2 Note that we do not require explicit access to a sub-optimal policy; we only require state, action transitions from said policy. Moreover, to ensure a fair comparison across algorithms, we maintained equal feedback budgets for all algorithms within each environment, while adjusting the budget across environments reflect their degree of difficulty. See Appendix [or a complete overview of the implementation process and specific hyperparameters for all algorithms.
Evaluation We show average offline performance (i.e., freeze the policy and evaluate it with no exploration) over ten episodes using either the ground truth reward function (DMControl experi- ments) or the success rate (Meta-World experiments). It is important to note that only SAC has access to the ground truth reward function. We perform this evaluation every 10000 training steps. To systemically evaluate performance, ing of a single trajectory segment or preferences between two trajectory segments according to the ground truth reward function. To thoroughly test the effectiveness of SDP, we perform evaluations on four robotic locomotion tasks from the DMControl Suite: Walker-walk, Cheetah-run, Quadruped- walk, and Cartpole-swingup and five robotic manipulation tasks from Meta-Worid: Hammer, Door- unlock, Door-lock, Drawer-open, and Window-open. In our experiments, the results are averaged over five seeds with shaded regions or error bars indicating 95% confidence intervals. To test for significant differences in final performance (i.e., the undiscounted return) and learning efficiency (i.e., the total area under the return curve, AUC), we perform Welch f-tests (equal variances not assumed) with a p-value of 0.05. See Appendices D.8 and D.9 Tables performance and AUC across all experiments.
simulated teacher that provides either scalar rat-
14 for summary of final
5.2 LOCOMOTION AND MANIPULATION RESULTS
Preference Feedback Experiments We first address RQ 1 by evaluating the utility of SDP in the preference-based RL setting. Considering all four preference-based algorithms in the nine environ- ments, SDP significantly (p < 0.05) improved learning (i.e., either final performance or AUC) in 23 out of the 36 experiments (see Figure 2p. In the remaining experiments, there were no significant differences in the performance between SDP and the baseline algorithms. The addition of SDP (i.e., SDP + base algorithm) never statistically hurt performance.
Scalar Feedback Experiments Continuing our investigation into RQ I, we now evaluate the performance of SDP in the scalar-based RL setting. We performed evaluations in Walker-walk, Cheetah-run, and Quadruped-walk, In Figure 3 we found that SDP (purple curve) significantly im- proves either the final performance or AUC compared to R-PEBBLE (navy curve) and Deep TAMER (yellow curve). More impressively, we found that SDP achieves comparable final performance to SAC (red curve), which uses the ground truth reward function, using as little as 60 feedback queries.
Leveraging Different Task Data in SDP Our previous experiments showed that SDP can lever- age sub-optimal data from the target task to improve numan-in-the-loon RL methods. Now, ad- dressing RQ 2, we investigate whether SDP can similarly use sub-optimal data from related tasks to

---

Published as a conference paper at ICLR 2025
Cartpole-Swingupp Feedback=48
Walker-walk, Feedback 100
Cheetah-run, Feedback 200
AUC x 101
4.0
x 16
4.0
5.0
30-
x3.--
x
3.0
1.0-
1.0-
Quadruped-wall 5e0
Hammer, Feedback=7500
Door-unlock, Feedback=500
AUC
6.0
A
0.6
0.4
2.0
0.2
-0.2
Door-lock, Feedback=500
Drawer-open, Feedback- 1000
Window-Open, Feedback=200
03
1.0
0.5
AUC x
0.8
e 0.4
0.6
0.3
033
04-
u.2
024
2.22
<0.1
0.0
0.0
0.0
-3.2
-0.1
PEBBLE
SDP + PEBBLE
RUNE
SDP RUNE
SURF
SDP SURF
MRN
5DP MRN
SAC
Figure 2: Results from the preference feedback experiments in the DMControl and World suites show mean AUC + 95% confidence intervals. indicates that SDP + the base preference learning algorithm achieves a statistically greater score than the base preference learning algorithm.
Cheetah-run, Feedback
Walker-walk, Feedback
Quadruped Feedback
1000
Episodic return
acn
1200
000
8c0
400
403
100
200
200
Environment steps (x 10*)
R-PEB3LE
Deep
Figure 3: In the scalar feedback experiments on the DMControl environments, SDP significantly outperforms R-PEBBLE and Deep TAMER and achieves comparable performance SAC.
improve performance on the target task. We perform three preference learning experiments, com- paring PEBBLE with SDP + PEBBLE, using sub-optimal data from a different prior task that has the same virtual robot: (1) Walker-stand for Walker-walk, (2) Quadruped-wall for Quadruped-rum and (3) Drawer-open for Door-open. To obtain the sub-optimal data for the prior tasks, we gath- ered transitions from partially trained policies opposed using random policies. Each partially trained policy achieved a final score of approximately 15-20% of that achieved by fully trained policy. This ensured that the distribution of sub-optimal data differed between the prior and target tasks. See Appendix A.2 for further details on the experiment setup. Figure demonstrates that in all three tested environments, SDP successfully leverage sub-optimal data from related tasks (green curve) as it achieved similar performance to SDP when leveraging target task data (purple curve).
5.3 PREFERENCE LEARNING WITH HUMAN FEEDBACK
To evaluate human-in-the-lorr in-the-loop RL algorithms, argue that is critical to understand their efficacy with human teachers. However, human user studies do not appear to be widely adopted in the current literature. To empirically investigate this, we conducted survey of 45 preference learning studies

---

Published as a conference paper at ICLR 2025
Walker-stand to Walker-walk
Quadruped-walk to Quadruped-run
Drawer-open to Door-open
Episodic return
700
600
Episodic return
500
600
Rate (%)
120
300
400
80
300
60
suo
200
20
10
30
40
50
80
100
60
80
100
Environment steps (x 10")
PEBBLE 5DP. S.b opt data - Drewer open 5DP. S.b ob--ppttooo cata Door open
SJP. Prior sub oct dsta - Welker stand
S0P: Sub opt. data Quadruped walk SDP. Sub-opt data Quadruped run
SJP, Sub-opt data Walker waik
Figure 4: This highlights that SDP can leverage sub-optimal data from different prior tasks as performed comparable to SDP when using target task sub-optimal data.
from 2012 to 2024 to understand the prevalence of human user studies in the preference learning literature. We found that fewer than 50% tested their proposed algorithms with human participants who were not also the authors. Moreover, of those studies that did involve human participants, only 41% included non-expert individuals, while none provided sufficient demographic information about their participants (c.g.. gender, race/ethnicity, level of education). These limitations raise significant concerns about whether and how current preference learning algorithms generalize to different populations.
To that end, we now address RQ 3 and evaluate the efficacy of SDP with real human feedback. We conduct an were collected: 18-24 (6), 25-30 (7), and 50-70 (3). Participants self-identified their membership in racial groups: South Asian (6), East Asian (3), White (4), Middle Eastern or North African (2), and multi-racial (1). The participants' highest educational attainments were: high school diploma (3), Bachelor's ' degrees (8), and Master's degrees (5), and their expertise varied across AI/ML (12), other computer science topics (1), and non-computer science fields (3). This highlights the diversity of our participants in terms of demographies and expertise.
human- -ubject study of 16 participants (9 male, 7 female). Age ranges
This user study compares the performance of SDP and PEBBLE in two DMControl enviromments: Pendulum-swingup (with 7 participants) and Cartpole-wwingup (with 12 participants). We focused our comparison to PEBBLE as performed comparably to the other preference learning baselines in Section 5.2 We between-subjects experimental design each participant provides prefer- ences for single seed of both SDP and PEBBLE algorithms. The preference budgets for Pendulum- swingup and Cartpole-swingup were 40 and 48, respectively. We selected these environments specif- ically because they could be solved with fewer preferences. aiming to reduce the overall time com- mitment required from participants. Each trial for single environment lasted approximately 1-1.5 hours. See Appendix
B for more details including user instructions and interface.
We visualize both algorithms' final performance and AUC in Figure 5 We found that in both en- vironments SDP (purple plots) maintains significant (p < 0.05) performance gains over PEBBLE (blue plots) in terms of either final performance or AUC. In Cartpole-swingup, we also observed consistent performance from SDP regardless of whether the teacher was human simulated. This suggests that our prior results with simulated teachers can generalize to settings where human feed- back is provided. Moreover, we observed no significant differences in SDP's effectiveness across demographic factors, including gender, age, educational, and computer science background (see Appendix C Tables and ( This finding is particularly encouraging for the potential real-world deployment of SDP, highlighting its usability for non-expert users across diverse demographics.
5.4 ABLATION AND SENSITIVITY STUDIES
To further understand the effectiveness of SDP, we perform further analysis of SDP across three dimensions: (1) the phases of SDP, (2) the number of feedback queries, and (3) the amount of sub-optimal data. This analysis aims to address RQ 4 and provide a deeper understanding of the factors influencing SDP's performance. For these experiments, we focus on SDP + R-PEBBLE in the Walker-walk environment. Additional results for Cheetah-run can be found in Appendix D.5
See Appendix B.2 for more details.
8

---

Published as a conference paper at ICLR 2025
AUC
Final Performance
1000
1.1
Final Performance
600
AlC
0.6
200
.0.2
Cartaole swingup
Pondulum
200
Cartpole
Pendulum
PEBBLE
SDP+PEBBLE
Figure 5: This demonstrates that SDP can significantly outperform PEBBLE terms of both AUC (left) and final performance (right) even when human teachers are providing preferences. denotes a statistically significant difference between SDP and PEBBLE.
Walker-walk, Phase Ablation
Walker-walk, Feedback Size Ablation
Walk Walker-walk, Data Size Ablation
Episodic return
1000
800
600
400
200
Environment steps 10*)
ATEUULL Fousback Feedbace
Figure 6: These figures show ablation and sensitivity studies of SDP in Walker-walk.
SDP Component Analysis First, evaluate the effect of each phase of SDP individually, the reward model pre-train phase, and the agent update phase. Figure 6 (leftmost) demonstrates the importance of using both phases in SDP for scalar-based RL approaches. We found that the SDP variants that only use one of the phases (green and gray curves) result in worse performance than the full SDP (purple curve).
Effect of Feedback Amount We evaluate SDP and R-PEBBLE with feedback budgets [60, 100, 200] to analyze the impact of feedback quantity on performance. As shown Fig- ure 6 (middle), SDP (purple curves) consistently outperforms R-PEBBLE (navy curves), further demonstrating its effectiveness across varying feedback levels.
Effect of Sub-Optimal Data Amount We evaluate the performance of SDP using varying amounts of sub-optimal transitions E [5000, 15000, 50000). Figure 6 (rightmost) indicates that while 5000 transitions (gray curve) led the poorest performance, increasing this amount to 15000 or 50000 (green and purple curves) yielded comparable improved results, suggesting that more sub-optimal data can benefit SDP.
CONCLUSION
In this work, we present SDP, an approach that improves the feedback efficiency for human-in- the-loop RL algorithms. SDP is specifically designed to leverage reward-free, sub-optimal data for scalar- and preference-basee RL approaches. By pseudo-labeling low-quality data with the mini- mum environment rewards, can pre-train the reward model without the need for human labeling. This provides the reward model with a head start in learning. This head start allows the reward model to learn to associate low-quality transitions with low reward values, even before receiving any actual

---

Published as a conference paper at ICLR 2025
human feedback. Our simulated teacher experiments in DMControl and Meta-World suites demon- strate that SDP can significantly improve a variety of preference- and ccalarbbase reward learning algorithms. Importantly,
further validate the real-world applicability of SDP by demonstrating its success in a 16-person user study. This work takes an important step towards considering how sub-optimal data can be leveraged for human-in-the-lne RI.
ACKNOWLEDGMENTS
Part of this work has taken place the Intelligent Robot Learning Lab the University of Alberta, which is supported in part by research grants from Alberta Innovates; Alberta Machine Intelligence Institute (Amii); a Canada CIFAR Al Chair, Amii; Digital Research Alliance of Canada; Mitacs; and the National Science and Engineering Research Council.
ETHICS STATEMENT
The goal of preference and scalar-based RL is to learn reward functions that encode human prefer- ences. However, this necessitates interaction with human users, raising concerns about potential bi- ases in the collected preferences. these preferences are primarily drawn from a non-representative group, the resulting RL system may unfairly prioritize the desires or needs of that group over others. To that end, we performed a human subject study that was reviewed and approved by an external ethics committee. This ensured the responsible and ethical collection of human preferences. In ad- dition, we took care to gather participants from a wide range both demographic and educational backgrounds, which we detail in Section 5.3
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide link to our code repository: https://
github. ccm/ cmus Limammm/aai SDP ICLR We also provide pseudocode in Section 4 Moreover,
outline extensive training and evaluation details in Section 5.1 and Appendix A.2 We also outline Lastly, for the human subject study, we provide details on the instructions provided to the participants in Appendix B as well description of the preference interface used. In our released codebase, also provide the specific code files used to run the experiments in the user study. In addition, an anonymized version of the human subject data will be released, as stated in our ethics approval.
all hyperparameters used for all algorithms and environments in Appendix A.2 Tables
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship Learning via Inverse Reinforcement Learning. In International Conference on Machine Learning, 2004.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight Experience Re- play. Neural Information Processing Systems, 2017.
Brenna Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A Survey of Robot Learn- ing from Demonstration. Robotics and Autonomous Systems, 2009.
Peter Barnett, Rachel Freedman, Justin Svegliato, and Stuart Russell. Active Reward Learning from Multiple Teachers. In Workshop Artificial Intelligence Safety al AAAI Conference Artificial Intelligenc, 2023.
Erdem Biyik and Dorsa Sadigh. Batch Active Preference-Barte Learning of Reward Functions. In Conference on Robot Learning. 2018.
Erdem Bıylk, Dylan P Losey, Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning Reward Functions from Diverse Sources Human Feedback: Optimally Inte- grating Demonstrations and Preferences. The International Journal Robotics Research, 2022.
10

---

Published as a conference paper at ICLR 2025
Yaniv Blumenfeld, Dar Gilboa, and Daniel Soudry. Beyond Signal Propagation: Is Feature Diver- sity Necessary in Deep Neural Network Initialization? In International Conference on Machine Learning. 2020.
Serena Booth, W. Bradley Knox, Julie Shah, Scott Nickum, Peter Stone, and Alessandro Allievi. The Perils of Trial-and-Error Reward Design: Misdesign through Overfitting and Invalid Task Specifications. In AAAI Conference on Artificial Intelligence, 2023.
Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs. Biometrika, 1952.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating Beyond Subop- timal Demonstrations via Inverse Reinforcement Learning from Observations In International Conference on Machine Learning. 2019.
Daniel S. Brown, Wonjoon Goo, and Scott Nickum. Better- Demonstrator Imitation Learning via Automatically-Rankes Demonstrations. In Conference on Robot Learning. 2020.
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor, and Ann Nowé. Reinforcement Learning from Demonstration through Shaping. International Joint Conference on Artificial Intelligence, 2015.
Erdem Bryik, Nicolas Huynh, Mykel J. Kochenderfer, and Dorsa Sadigh. Active Preference-Bane Gaussian Process Regression for Reward Learning. In Robotics: Science and Systems, 2020.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, et al. Scaling Data-Driven Robotics with Reward Sketching and Batch Reinforcement Learning. Robotics: Science and Systems, 2020.
Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from Suboptimal Demonstration via Self SuperSuper Reward Regression. In Conference Robot Learning, 2021.
Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, and Fei-Yue Wang. RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences. In International Con- ference on Machine Learning, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Neural Information Processing Systems, 2017.
Jack Clark and Dario Amodei. Faulty Reward Functions in the Wild, 2016.
Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, and Jan Peters. Active Reward Learning. In Roboties: Science and Systems, 2014.
Oliver Daniels-Kocd Koch and Rachel Freedman. The Expertise Problem: Learning from Specialized Feedback. In ML Safety Workshop at Neural Information Processing Systems, 2022.
Anca Dragan and Siddhartha Srinivasa. Formalizing Assistive Teleoperation. Robotics: Science and Systems, 2012.
Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement Learn- ing from Imperfect Demonstrations. In International Conference on Machine Learning, 2019.
Joseph Giovanelli, Alexander Tornede, Tanja Tornede, and Marius Lindauer. Interactive Hyperpa- rameter Optimization in Multi-objective Problems via Preference Learning. In AAAI Conference on Artificial Intelligence, 2024.
Shane Griffith. Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. Policy Shaping: Integrating Human Feedback with Reinforcement Learning, In Neural Informa- fion Processing Systems, 2013.
11

---

Published as a conference paper at ICLR 2025
Tuomas Haarmoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor Critic::: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In International Con- ference Machine Learning, 2018.
Donald Joseph Hejna III and Dorsa Sadigh. Few-Shot Preference Learning for Human- -iraan----th--ooy RL. In Conference on Robot Learning, 2023.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep Q-learning from Demonstrations: In AAAI Conference Artificial Intelligence, 2018.
Simon Holk, Daniel Marta, and Iolanda Leite. PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learning. In International Conference Human- Robot Interaction, 2024.
Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang, Query-Policy Misalign- ment Preference- Based Reinforcement Learning. International Conference on Learning Representations, 2024.
Minyoung Hwang, Gunmin Lee, Hogun Kee, Chan Woo Kim, Kyungjae Lee, and Songhwai Oh. Sequential Preference Ranking for Efficient Reinforcement Learning from Human Feedback. In Neural Information Processing Systems, 2023a.
Minyoung Hwang, Luca Weihs, Chanwoo Park, Kimin Lee, Aniruddha Kembhavi, and Kiana Ehsani. Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferences. In Conference on Computer Vision and Puttern Recognition, 2023b.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from Human Preferences and Demonstrations in Atari. In Neural Information Processing Systems, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
W Bradley Knox and Peter Stone. Interactively Shaping Agents via Human Reinforcement. In International Conference on Knowledge Capture, 2009.
W Bradley Knox and Peter Stone. Learning Non-Myopically from Human-Generated Reward. In International Conference Intelligent User Interfaces, 2013.
W. Bradley Knox, Stephane Hatgis-Kessell. Serena Booth, Scott Niekum, Peter Stone, and Alessan- dro Allievi. Models of Human Preference for Learning Reward Functions. Transactions Machine Learning Research, 2023.
Kimin Lee, Laura Smith, and Pieter Abbeel. PEBBLE: Bredback-Efficinn Interactive Reinforce- ment Learning via Relabeling Experience and Unsupervised Pre-training In International Con- ference Machine Learning, 2021a
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-Pref: Benehmarking Preference-Batee Reinforcement Learning. In Neural Information Processing Systems Datasets and Benchmarks Track, 2021b.
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward Uncertainty for Exploration in Preference- based Reinforcement Learning. In International Conference on Learning Representa- tions, 2022.
Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-RewarddNert Implicitly Differen- tiable Reward Learning for Preference-basee Reinforcement Learning. In Neural Information Processing Systems, 2022.
Runze Liu, Yali Du, Fengshuo Bai, Jiafei Lyu, and Xiu Li. PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation. International Conference Machine Learning, 2024.
12

---

Published as a conference paper at ICLR 2025
Yi Liu, Gaurav Datta, Ellen Novoseller, and Daniel S. Brown. Efficient Preference-Basee Rein- forcement Learning Using Learned Dynamics Models. In International Conference on Robotics and Automation, 2023.
Robert Loftin, Bei Peng, James MacGlashan, Michael L Littman, Matthew E Taylor, Jeff Huang, and David L Roberts. Learning Behaviors via Human-Delivered Discrete Feedback: Modeling Implicit Feedback Strategies Speed up Learning. Autonomous Agents and Multiagent Systems, 2016.
Daniel Marta, Simon Holk, Christian Pek. Jana Tumova, and Iolanda Leite. VARIQuery: VAE Begment-Based Active Learning for Query Selection in Preference-Basee Reinforcement Learn- ing. In International Conference on Intelligent Robots and Systems, 2023a.
Daniel Marta, Simon Holk, Christian Pek, Jana Tumova, and Iolanda Leite. Aligning Human Pref-
erences with Baseline Objectives in Reinforcement Learning. In International Conference Robotics and Automation, 2023b.
Daniel Marta, Simon Holk, Christian Pek, and Iolanda Leite. SEQUEL: Seni--Sperviied
Preference-based RL with Query Synthesis via Latent Interpolation. In International Conference on Robotics and Automation, 2024.
Thommen George Karimpanal Maxence Hussonnois and Santu Rana. Controlled Diversity with Preference: Towards Learning a Diverse Set of Desired Skills. In International Conference Autonomous Agents and Mulliagent Systems, 2023.
Shaunak A Mehta and Dylan P Losey. Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human- Interaction. ACM Transactions on Human-Robo In- teraction, 2024.
Katherine Metcalf, Miguel Sarabia, Natalie Mackraz, and Barry-John Theobald. Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards. In Conference on Robot Learning. 2023.
Katherine Metcalf, Miguel Sarabia, Masha Fedzechkina, and Barry-John Theobald. Can You Rely on Synthetic Labellers in Preference-Batte Reinforcement Learning? It's Complicated. In AAAI Conference on Artificial Intelligence, 2024.
Vivek Myers, Erdem Biyik, Nima Anari, and Dorsa Sadigh. Learning Multimodal Rewards from Rankings. In Conference on Robot Learning, 2022.
Vivek Myers, Erdem Biyik, and Dorsa Sadigh. Active Reward Learning from Online Preferences. In International Conference on Robotics and Automation, 2023.
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-eefficinnt Preference- based Reinforcement Learning. In International Conference Learning Representations, 2022.
Zhizhou Ren, Anji Liu, Yitao Liang, Jian Peng, and Jianzhu Ma. Efficient Meta Reinforcement Learning for Preference-bert based Fast Adaptation. In Neural Information Processing Systems, 2022.
Kyriacos Shiarlis, Joao Messias, and Shimon Whiteson. Inverse Reinforcement Learning from Fail- ure. In International Conference Autonomous Agents and Multiagent Systems, 2016.
Avi Singh. Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Con- necting New Skills to Past Experience with Offline Reinforcement Learning. In Conference on Robot Learning, 2020.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and Character- izing Reward Gaming. In Neural Information Processing Systems, 2022.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A Mini- maximalist Approach to Reinforcement Learning from Human Feedback. In International Con- ference Machine Learning, 2024.
13

---

Published as a conference paper at ICLR 2025
Voot Tangkaratt, Nontawat Charoenphakdee, and Masashi Sugiyama. Robust Imitation Learning from Noisy Demonstrations. In International Conference on Artificial Intelligence and Statistics, 2021.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind Control Suite. CoRR, 2018.
Matthew E. Taylor, Halit Bener Suay, and Sonia Chernova. Integrating Reinforcement Learning with Human Demonstrations of Varying Ability. In International Conference on Autonomous Agents and Multiagent Systems, 2011.
Mudit Verma and Katherine Metcalf. Hindsight PRIORs for Reward Learning from Human Prefer- ences. In International Conference on Learning Representations, 2024.
Mudit Verma, Siddhant Bhambri, and Subbarao Kambhampati. Exploiting Unlabeled Data for Feed- back Efficient Human Preference based Reinforcement Learning. In Workshop Representation Learning for Responsible Human- Human-Centric Centric AI at AAAI Conference on Artificial Intelligenc, 2023,
Ngo Anh Vien, Wolfgang Ertel, and Tae Choong Chung. Learning via Human Feedback in Contin- uous State and Action Spaces. Applied Intelligence, 2013.
Ren-Jian Wang. Ke Xue, Yutong Wang, Peng Yang, Haobo Fu, Qiang Fu, and Chao Qian. Di- versity from Human Feedback. In Workshop on Agent Learning in Open-Endedness at Neural Information Processing Systems, 2023.
Ruiqi Wang, Weizheng Wang, and Byung-Cheol Min. Feedback-efficient Active Preference Learn- ing for Socially Aware Robot Navigation. In International Conference on Intelligent Robots and Systems, 2022.
Xiaofei Wang, Kimin Lee, Kourosh Hakhamaneshi, Pieter Abbeel, and Michael Laskin. Skill Pref- erences: Learning to Extract and Execute Robotic Skills from Human Feedback. In Conference on Robot Learning. 2021.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep TAMER: Interactive Agent Shaping in Sighnniimmsinnal State Spaces. In AAAI Conference on Artificial Intelligence, 2018.
Devin White, Mingkang Wu, Ellen Novoseller, Vernon Lawhern, Nicholas Waytowich, and Yong- can Cao. Rating-based Reinforcement Learning. In AAAI Conference Artificial Intelligence, 2024.
Nils Wilde, Erdem Biyık, Dorsa Sadigh, and Stephen L Smith. Learning Reward Functions from Scale Feedback. In Conference on Robot Learning, 2021.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A Bayesian Approach for Policy Learning from Trajectory Preference Queries. Neural Information Processing Systems, 2012.
Wanqi Xue, Bo An, Shuicheng Yan, and Zhongwen Xu. Reinforcement Learning from Diverse Human Preferences. In International Joint Conference on Artificial Intelligence, 2024.
Tianhe Yu, Deirdre Quillen, Zhampeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A Benchmark and Evaluation for Multi-task and Meta Reinforcement Learn- ing. In Conference Robot Learning, 2020.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative Data Sharing for Multi-task Offline Reinforcement Learning. Neural Information Processing Systems, 2021.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to Leverage Unlabeled Data in Offline Reinforcement Learning. In International Conference on Machine Learning, 2022.
14

---

Published as a conference paper at ICLR 2025
Guoxi Zhang and Hisashi Kashima. Learning State Importance for Preference-basee based Reinforcement Learning. Machine Learning, 2024.
Jiawei Zhao, Florian Tobias Schaefer, and Anima Anandkumar. ZerO Initialization: Initializing
Neural Networks with only Zeros and Ones. Transactions Machine Learning Research, 2022.
Tianchen Zhu, Yue Qiu, Haoyi Zhou, and Jianxin Li. Decoding Global Preferences: Temporal and Cooperative Dependency Modeling in Multi-Agent Preference-Basee Reinforcement Learning. In AAAI Conference on Artificial Intelligene, 2024.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A SIMULATED EXPERIMENT DETAILS
A.1 BENCHMARKS
PEBBLE PEBBLE has two primary components: unsupervised exploration and off-policy learn- ing with relabeling. The purpose of the unsupervised exploration phase is to collect diverse expe- riences for human teacher provide feedback More specifically, PEBBLE optimizes state entropy to explore the environment. Furthermore, PEBBLE uses off-policy reinforcement learning to learn policy. PEBBLE specifically uses off-policy RL algorithm they more sample efficient compared to their on-policy counterparts, Then as the reward model changes, PEBBLE relabels all transitions in the RL agent's replay buffer with the latest reward model. This is integral as the reward model is non-stationary, and relabeling the transitions stabilizes the learning process.
To adapt PEBBLE the scalar feedback setting, we make one minor change to the reward model. In the scalar feedback setting, we use scripted teacher that provides a scalar rating of a single trajectory segment. Therefore, the only update PEBBLE is with respect to the loss function. we use the mean-squared error loss in Equation Instead of using the cross-entropy loss in Equation 2
RUNE RUNE is a preference learning algorithm (built on top of PEBBLE) that uses an uncertainty-base based exploration strategy to improve feedback efficiency. To encourage exploration for the SAC agent, RUNE adds an intrinsic reward component based on the standard deviation in the reward model.
SURF SURF another preference learning algorithm (built on top of PEBBLE) that improves feedback efficiency by using semi-supervised and data augmentation approaches. To incorporate semi-supervised learning, SURF generates pseudo-labels for unlabeled trajectories by querying the learned reward model. If the reward model confidently (e.g., low output standard deviation) pre- dicts the pseudo-label, then the trajectory. label pair added to the reward model training data set. Further, SURF proposes new data augmentation technique that crops sub-seque sequences of trajectories.
MRN MRN a preference learning algorithm also integrated with PEBBLE. Unlike other PbRL algorithms, MRN is bi-level optimization algorithm in which the actor and critic are updated in the inner loop, and the reward model updated the outer loop. Importantly, the reward model takes into account the performance of the critic on the preference data.
Deep TAMER In scalar feedback experiments, also consider the Deep TAMER bench- mark. In Deep TAMER, scalar feedback is used to learn a human reward function via regression. Then the agent acts greedily with respect to this reward function. Furthermore, the original im- plementation of Deep TAMER was built on top of DQN. Therefore, there was no separate actor- network. In addition, Deep TAMER only used discrete feedback 1,]
To make Deep TAMER a fair benchmark, we made a few adjustments. To start, we allow Deep TAMER learn from real-valued feedback as done the other scalar-based experiments. However, instead of using the ground truth reward function as feedback, we use the state-action values from a fully-trained SAC agent. We do this because, in TAMER (and Deep TAMER), the teacher intended to provide feedback representative of the return. Secondly, in Deep TAMER, feedback is provided per (state, action) pair. Therefore, to make sure Deep TAMER received the same amount of feedback as the other baselines, we used trajectory segment size x feedback amount for Deep TAMER only. The other benchmarks receive a scalar feedback value that is the sum of rewards along trajectory segment. Third, learn the reward model
standard regression as described in Section 3.1 Lastly, as our testing environments are continuous state and action, we learn a separate actor policy, similarly done in (Vien et al. 2013).
16

---

Published as a conference paper at ICLR 2025
A.2 TRAINING DETAILS
In all of our experiments, we use the hyperparameters in [ the reward models used in all benchmarks. For the agent update phase of SDP, additional hyperparameter is associated with the number of environment interactions made before the standard preference/scalar feedback learning loop begins. However, for simplicity, we kept the same value as the existing feedback frequency hyperparameter. as feedback frequency also dictates the number of environment interactions made between feedback sessions
Furthermore, we use most of the existing reward model hyperparameters used in PEBBLE, however, we adjusted the following four hyperparameters: feedback frequency, amount of feedback per ses- sion, trajectory segment size (only for Meta-world), and activation function for the final NN layer. We adjusted the first two hyperparameters because PEBBLE originally used a significantly larger feedback budget, therefore budget. We used a different trajectory segment size for Meta-world because we wanted to keep the segment sizes the same across both the DMControl and Meta-world environments. Moreover, found that the output activation function could significantly affect learning, therefore we tested all benchmarks using both Tanh (original activation used) and Leaky-ReLLJ and chose the reward model that achieved better final performance. For the RUNE and SURF baselines, we use any hyperpa- rameters associated with their specific algorithm according to the original paper (see Table 2 For fair comparison with SDP, we provide all human-in -eee-iiitttttte tlo-loop baselines (e.g.. PEBBLE, R-PEBBLE, Deep TAMER, RUNE, and SURF) with the sub-optimal data set be used in both the reward model and by the RI. agent.
wanted the feedback schedule better reflect smaller feedback
Furthermore, select trajectory segments for the teacher to provide feedback on, we use uniform sampling in the DMControl tasks and disagreement sampling in the Meta-world tasks. Disagreement sampling is a popular active learning approach in which trajectories with higher uncertainty (based on an ensemble of neural networks) are more likely to be sampled (Christiano et al. 2017). As for the SAC hyperparameters, we use the values found in Tables
For the experiments in which we leveraged sub-optimal data from different task (i.e., Walker-stand, Quadruped-walle Drawer-open), we gathered 50,000 transitions from partially trained policies. We note that for these experiments, In these experiments, the prior and target tasks were environments in which the simulated robot was identical. The only difference is the environmental reward. Therefore, the random policy for both environments would be the same. To truly demonstrate transfer, we wanted to ensure we obtained low-quality transitions of the prior task that were different from the target task.
purposely did not use transitions gathered from random policy.
Each partially trained policy achieved a final score of approximately 15-20% of that achieved by a fully trained policy. More specifically, we used the following procedure train the SAC policies. First, for Walker-stand, we trained SAC policy for 5,000 time steps, and the average final perfor- mance was approximately 194. Second, for Quadruped-wand walk, we trained a SAC policy for 100,000 timesteps, and the average final performance was approximately 184. Lastly, for Drawer-open, we trained a SAC policy for 50,000 time steps, and the average final success rate was approximately 14%.
17

---

Published as a conference paper at ICLR 2025
HYPERFARAMETER
VALUE
SEGMENT SIZE
50
RANDOM STEPS
(LE., SUB OPTIMAL DATA TRANSITIONS)
50000
UNSUPERVISED EXPLORATION STEPS
9000
CONSIDER UNsUPERVISED EXPLORATIONS
SUB-OPTIMAL.
FALSE (WALKER-WAY CARTROLF.SWINGIN QUADRUPED WALK PARI.)
TRUE (OTHERS)
FREQUENCY OF FEEDBACK
20000 (DMCONTROL)
10000 (WINDOW-OPEN, DOOR-UNLOCK, DOOR-LOCK)
5000 (HAMMER, DRAWER-OPENT
FEEDBACK BUDGETS IN ABLATIONS
2000 (DOOR-OPEN)
RUN, WALK)
PEEDRACK QUERIES
(HAMMER, DRAWER-OPEN)
8 |CARTPOLEESWIKGUP) (OTHERS)
SAMPLING SCHEME
DISAGREEMENT SAMPLING (METWWORL))
UNIFORM SAMPLING (DMCONTROL)
TRAINING EPOCHS
200 (WINDOW-OPEN, SURF AND SDP SURF)
(OTHERS)
LEARNING RATE
3 x10-1
INTERMEDIATE NEURAL NETWORK ACTIVATION
LEAKY RELU
BATCH SizG
128
NEURONS PER HIDDEN LAYER
128
Loss FUNCTION
MEAN SQUARED ERROR (SCALAR FEEDBACK) ENTROPY Loss FEEDBACK)
Reward Model Pre-training LAST LAYER NEURAL NETWORK ACTIVATION
TANII
For Human--n-the--le Algorithm
LAST LAYER NEURAL NETWORK ACTIVATION
LEAKY RELU (PEBBLE, RUNE, SURF)
TANH (SDP HAMMER, DRAWER WALKER (ALKER-WALK) LEAKY RELU (SDP OTHERS)
Table 1: Hyperparameters for the reward model used in all experiments (both preference and scalar feedback variants).
18

---

Published as a conference paper at ICLR 2025
HYPERPARAMETER
VALUE
Specific RUNE Hyperparameters
BETA SCHEDULE
LINEAR DECAY
BETA INIT
0.05
BETA DECAY
10-5
Specific SURF Hyperparameters
THRESHOLD X
0.99
UNLABELED BATCH RATIO
4
LOSs WEIGHT
MIN/MAX LENGTH OF CROPPED SEGMENT
[45, 55]
SEGMENT LENGTH BEFORE CROPPING
60
Specific MRN Hyperparameters
META STEPS
1000 (WALKER-WALA))
3000 (QUADRUPED-WALK)
10000 (DOOR-OPEN)
5000 (OTHERS)
Table 2: Baseline Specific Hyperparameters.
HYPERPARAMETER
VALUE
OPTIMIZER
ADAM
KKngmaa & Ba 2015
DISCOUNT
0.99
ALPHA LEARNING RATE
10-4
ACTOR BETAS
0.9, 0.999
CRITIC BETAS
0.9, 0.999
ALPHA BETAS
0.9, 0.999
TARGET SMOOTHING COEFFICIENT
0.005
ACTOR UPDATE FREQUENCY
CRITIC TARGET UPDATE FREQUENCY
INIT TEMPERATURE
0.1
NETWORK TYPE
MLP
NONLINEARITY
RELU
GRADIENT UPDATES PER STEP
Table 3: Hyperparameters for SAC that shared by all algorithms. SAC is the standard RL algorithm that learns from the environment's true reward signal, not via preference learning.
19

---

Published as a conference paper at ICLR 2025
HYPERPARAMETER
VALUE
DMControl
BATCH SIZE
512
1024 (OTHERS)
HIDDEN LAYERS
2
NEURONS PER HIDDEN LAYER
256 (CARTPOLE-SWINGUP)
1024 (OTHERS)
ACTOR/CRITIC LEARNING RATE
10-5 (CHEETAH-RUN PREFERENCE FEEDBACK)
10-4 (CHEETAH- RUN SCALAR FEEDBACK)
5 x 10-4 (WALKER-WALK)
10 (OTHERS)
TRAINING STEPS
0.5 10th (WALKER-WALK, CARTPOLE-SWINGUP) 10 (OTHERS)
Metaworld BATCH SIZE
512
HIDDEN LAYERS
3
NEURONS PER HIDDEN LAYER
256
ACTOR/CRITIC LEARNING RATE
10-4
TRAINING STEPS
0.5 x 106 (DOOR-LOCK AND WINDOW-OPEN)
10c (OTHERS)
Table 4: Specific SAC hyperparameters that were tuned for the DMControl and Metaworld exper-
iments. The majority of these hyperparameters were selected from the PEBBLE repo (Lee et al.
202
HYPERPARAMETER
VALUE
SAC
TRAINING STEPS
0.3 x 106
BATCH SIZE
1024
HIDDEN LAYERS
2
NEURONS PER HIDDEN LAYER
1024
ACTOR/CRITIC LEARNING RATE
10
Reward Model
CONSIDER UNSUPERVISED EXPLORATIONS SUB-OPTIMAL
FALSE 20000
FREQUENCY OF FEEDBACK
FEEDBACK QUERIES PER SESSION
10 (PENDULUM-SWINGU))
8 (CARTPOLE-SWINGUP)
SAMPLING SCHEME
UNIFORM SAMPLING
BATCH SIZE LAST LAYER NEURAL NETWORK ACTIVATION
128
LEAKY RELU
Table 5; Tuned hyperparameters for human feedback experiments.
20

---

Published as a conference paper at ICLR 2025
B HUMAN SUBJECT STUDY DETAILS
Our study was approved by an external ethics committee. In total, 16 users participated in our study, 3 of which participated in both Pendulum-swingup and Cartpole-swingup tasks. As for the amount of participation per task, 7 users provided preferences in Pendulum-swingup and 12 users provided preferences in Cartpole-swingup. During the timeline of user study, initially had participants interacting in Pendulum Vendulum-swingup. however we found that it was very simple, therefore we decided as more difficult task. Each participant provided preferences for single run of the environment, as each session took approximately 1 to 1.5 hours. Table shows the specific hyperparameters used.
include
B. PROTOCOL STEPS
The study took place in person, where the participant first provided the consent from review and sign. Then, participants and the researcher, together, reviewed the instructions outlining the objective of the study. The instructions had two components. First, they outlined participant's goal in the study. Second, as done in Christiano et al. (2017), we provided guide on what constitutes good and bad behavior in each domain.
Pendulum-swingup The first set of instructions for the study are as follows:
1. Objective: for the pole to swing up and balance.
2. Agent controls how much torque (or force or twist) to apply
3. It does matter which way the pole swings, left right, long the pole balances.
4. Your task: You will two clips of behaviors, and your job is to select the clip you think is better given the above objective. If you think both clips are identical in behavior, you can select equally preferable.
The second set of instructions (i.e., advice) for the study are as follows:
1. The first priority is for the pole to begin swinging back and forth (i.e., picking up momen- tum). Therefore, the video clip where the pole has swung higher is better. Even if the agent is not behaving well in either clips, if you can tell that the pole is higher in one clip than the other, it is better prefer that clip.
2. In general, if the pole is swinging rapidly in circle (e.g.. complete 360), then this is usually worse behavior than if the pole is barely moving.
Cartpole-swingup The first set of instructions for the study are follows:
1. Pole is attached to a cart.
2. Agent can move the cart left and right.
3. Goal is for the agent to move the cart such that the pole swings up and balances
4. Your task: You will see two clips of behaviors, and your job is select the clip you think is better given the above objective. If you think both clips are identical in behavior, you can select equally preferable.
The second set of instructions (i.e., advice) for the study follows:
1. The first priority is for the pole to begin swinging back and forth (i.e., picking up momen- tum). Therefore, the video clip where the pole has swung higher is better. Even if the agent is not behaving well in either clips, if you can tell that the pole is higher in one clip than the other, it better to prefer that clip.
2. In general, the pole is swinging rapidly in circle (e.g.. complete 360), then this usually worse behavior than if the pole is barely moving.
3. If the pole balancing then falls over, then is is better behavior than if the pole is barely moving.
21

---

Published as a conference paper at ICLR 2025
During the instruction period, participants also watched video clips of what successful and unsuc- cessful behavior looks like. In addition, participants were able to ask questions about the environ- ment's objective.
Once the instruction period was complete, participants completed a practice round in providing preferences. This was included users could gain familiarity with the interface. Figure shows a screenshot of the interface used. Participants would use keyboard input to move from one video clip the next. Then, participants had the opportunity rewatch either clips many times they liked. Afterwards, participants chose which clip they preferred (or equally preferred) via keyboard input. Each video clip consisted of 50 step segment, which was approximately 2 seconds long. The total study time was (1 - 1.5) hours. After the study was complete, participants filled out demographie survey which included questions pertaining their age, race/ethnicity, education level completed, and area of expertise.
Preference Comparsion # 1, Trajectory A
muyy caamtag.
westes about beginl
Secttrg resty tretectriie
lice" are rady preferesoes tragensery
erecs encar are raaly glve profereices to trajectirs
was prpecctory sanva
pocr wat scowt:
eepniciing toopetario
anuch tragectory zue proters
9
orater the trodectrtts eqailyy, prece
Time step 50
+
Q
rrss boer nesty plve preferesce to trddecery
Mganr1
Preference Comparsion # 1, Trajectory B
reothct snstoe
letting trafecteries
tress vee snr is see pelle apttr
vraleisss A.
scallty,
nau referee tratectery we epaltr.
156
Time step 50
catting rass venfrmeeees at batcy eeterree
Fater
+ Q
tros Exter
ancferron Tragetery
Figure 7: This shows the user interface used for the human subject study. Participants would view each video clip, sequentially, then decide which clip they preferred.
22

---

Published as a conference paper at ICLR 2025
B.2 PREFERENCE LEARNING SURVEY DETAILS
Table 6 provides a list of all papers included in our preference learning survey. We now describe the criteria for inclusion and exclusion in our survey. To find papers, we used Google Search with key words: preference learning. reinforcement learning from human feedback, and preference based reinforcement learning. We also found papers by reviewing those that cited popular preference learning works. This included Christiano et al. (017) Lee ct al. (202la). as these were two of the first papers that popularized preference learning. We included conference, journal and workshop papers in our survey. We did not include papers that were strictly on Arxiv. To keep the survey within scope, we did not include any preference learning works related to large language or foundation models. We also did not include any works that were entirely theoretical (e.g., empirical results were provided).
23

---

Published as a conference paper at ICLR 2025
Table 6: Articles used in Preference Learning Survey from Section 5.3 was provided.
NA indicates no information
PAPER REFERENCE
HUMAN USERS (NON-AUTHOR)
NUMBER USERS
Myers al.
2023)
Yes
22
Metcalf et 20244
Yes
50
Myers ct al. 2022)
Yes
50
Christaaoo ct al. 2017
Yes
NA
Hejna III & Sadigh
2023
Yes
Hwang al. 2023bb
Yes
Biyik & Sadigh
2018
Yes
10
Hwang al.
2023a
Yes
Bryık et al.
20200
Yes
10
Wilde
2021
Yes
18
B1yık ct al.
2022
Yes
15
Ibarz et al.
2018
Yes
NA
Knox et al.
2023
Yes
143
Holk cl al.
2024)
Yes
32
Metcalf
2023)
Yes
40
Marta et al.
2024)
Yes
NA
Marta et al.
2023a
Yes
70
Marta ct at.
2023b
Yes
20
Ren et al.
2022)
Yes
NA
Mehta & Losey
2024)
Yes
15
White al. (0244
Yes
20
Wang et al.
2022
Yes
10
Wang et al.
2023)
No
Zhang & Kashima
2024
No
Lee
2021a)
No
Liang et al.
|002
No
Park al. 2022)
No
Hu ct al.
2024)
No
Liu ct al.
2022
No
Liu al.
2023
No
Xue et al.
2024
No
Daniels-Koch & Freedman
2022
No
Verma et al.
2023)
(2023)
No
Barnett et (2233
No
Verma & Metcali
(0224)
2024)
No
Metcall al. 2023
No
Swamy et all
2024
No
Lee et
202
No
Wang et 2021
No
Maxence Hussonnois & Rana
2023
No
Wilson al. (2012)
2012)
No
[0224)
No
Giovanelli al.
2024
No
Zhu et al. 2024)
No
Cheng et al.
2024
No
24

---

Published as a conference paper at ICLR 2025
C ADDITIONAL HUMAN TEACHER RESULTS
TASK
FEEDRACK
BACKGROUND
AUC
P VALUE
FINAL PERFORMANCE
P VALUE
Non-CS
8258, 44 1898.47
648.88 35.17
CARTPOLE-SWINGUP
48
CS
10208.1 2311.16
0.183
614.14 114.81
0.692
Non-CS
3540.70
327.65 407.80
PENDULUM SWINGUP
CS
10188.70 70.88 3858.94
0.258
641.68 237.76
0.23
Table This table shows the AUC and final performance of SDP (mean 95% confidence intervals) for CS and non-CS participants in the human subject study.
Task
Feedback
Demographic
AUC
Value
CARTPOLE-SWINGE
48
FEMALE
$720.03
0.085
MALE
11770.22 1669.96
KENDULUM-SWINGLE
40
FEMALE
$
0.331
MALE
10134.67 4822.22
CARTPOLE-SWING
48
AGE 18-30
10109.54 2370.61
0.313
AGE 50-70
10923.05 1222.88 1292.83
SRIDULUM-SWINGHHH
40
AGE 18-30
8619.07 33811448
0.318
AGE 50-70
10607.93 5249.72
CARTPOLE-SWINGUP
48
EDUCATION: BACHELORS OR HIGHER
11044.55 # 1783.25
0.794
EDUCATION: HIGH SCHOOL DIPLOMA
6247.99 99 # 5186.47
PENDULJU--SINNGUP 40
EDUCATION: BACHELORS OR HIGHER
10188.80 88 + 3858.95
0.742
EDUCATION: HIGH SCHOOL DIPLOMA
7677.85 = 3540.7
Table 8: This table shows the AUC of SDP (mean + 95% confidence intervals) for different de- mographic conditions. This table highlights that SDP can be effective irrespective of demographic background (gender, age, and educational background).
Tables 7 highlight that there no significant difference in the performance of SDP when human teachers have differing backgrounds. This includes background in CS, educational level, age, and gender. We note that we did not perform statistical analysis comparing racial groups as some experiments had only one participant of particular racial background, making comparisons less meaningful.
25

---

Published as a conference paper at ICLR 2025
D ADDITIONAL SIMULATED TEACHER RESULTS
For simplicity, in all additional experiments in this section, we only compare SDP + PEBBLE with PEBBLE (or R-PEBBLE).
D.1 STUDY OF AGENT UPDATE PHASE
Figure emphasizes how the agent update phase does result in new transitions, therefore the teacher provides feedback to transitions that are different from the original sub-optimal transitions used for pre-training.
8
Figure 8: Door open, preference learning exp. These plots show how the agent's policy has changed from the reward model pre-training phase (purple histograms) to the agent update phase (blue his- tograms), thereby resulting in a different distribution for the state features.
D.2 TRUE REWARD VALUES OF RANDOM POLICY
Figure shows the true reward values for sub-optimal data gathered through random policy in the DMControl suite. This emphasizes that the true reward value is close to the value we pseudo-label the sub-optimal transitions with (i.e., zero), therefore SDP should not yield large incorrect reward bias.
26

---

Published as a conference paper at ICLR 2025
10000-
8000
Frequency
6000
4000
2000
0
0.0015 <
0.0596
0.1176 R 0.1177
0.1757 R 0.2337
0.2337 <R R < 0.2918
0.0596
R < 0.1117
Reward value
(a) Walker-walk.
10000-
8000
Frequency
6000
4000
2000
0-
0.0014
0.0028 R 0.0042
0.0042 R R 0.0056
0.0056 R 0.007
R
0.0014
R < 0.0028
Reward value
(b) Cheetah-run.
5000
Frequency
4000
3000
2000
1000
0-
0.0
0.1999 R
0. 3999 R
0.5998 <
0.7998 <
R 0.1999
0.3999
0.7998
Reward value
(c) Quadruped-wall,
Figure 9: Distribution of true reward values for transitions obtained with a random policy
27

---

Published as a conference paper at ICLR 2025
D.3 ZERO WEIGHT STUDY
To understand the effect of each component of SDP, we performed two additional ablations. First, in the reward model pre-training phase, the goal is for the reward model to learn output zero. However, a trivial means to achieve an output of zero is to set all weights and biases in the neural network zero. Therefore, we compare the full SDP to SDP using zero-weight initialization as a replacement for the reward model pre-training. We found that using a zero-weight initialization for the reward model in place of the pre-training phase results in significantly degraded performance (see the green curve in Figure 10 r01-right.. This is not surprising, as previous works have found that a zero-weight initialization can negatively affect the training neural networks (Blumenfeld et al.) 2020 Zhao et al. 2022). In addition, Figure [ demonstrates that the reward model pre-training phase does not produce reward model weights of zero, emphasizing why we do not experience the same performance degradation that occurs if use vero-weight initialization.
D.4 FAKE INPUT STUDY
Second, SDP makes use of state, action transitions that are gathered through a sub-optimal policy. Therefore, these are transitions that an agent experiences while interacting with its environment. However, as previously noted, the goal of the reward model pre-training phase is for the reward model learn to output zero. Therefore, it necessary for the reward model to pre-train on inputs that are real environment transitions? Instead, can we pre-train the reward model on transitions that did not result from an agent-enviionmenn interaction? To test this, we created "fake" inputs of size dim(state) + dim(action), and for each input dimension, We obtained 50,000 "fake" transitions and used this data for the reward model pre-training phase. goal is understand the effect of the type of inputs the reward model In this experiment, pre- pre-training phase, therefore we kept the agent update phase as is (i.e., provided the true sub-optimal data for this phase). We then compared SDP using true transitions SDP using "fake" transitions in Figures left and T2 We found that the full SDP (purple curve) in which we pre-train the reward model on true sub-optimal transitions yields significantly greater final performance than SDP using "fake" " transitions (green curve). This highlights the importance of using true sub-optimal transitions in SDP.
randomly sampled a value from N(0, 1).
Walker-walk, Fake Input Ablation
Walker-walk, O Weights Ablation
Episodic retur
1000
1000
800
800
400
400
200
200
10
20
40
20
30
50
Environment steps (x 104)
SDP, 'Fake' transitions
SDP, Zero weight Init
Figure 10: Zero weights and fake input studies in Walker-walk
28

---

Published as a conference paper at ICLR 2025
0.2
0.1
Parameter value
0.0
-0.1
-0.2
-0.3
20000
40000
60000
80000
100000
120000 140000
(a) Scalar feedback experiment in Walker-walk.
Figure 11: This figure shows the reward model weights of SDP after the reward model pre-training phase. This demonstrates that the reward model pre-train phase of SDP does result in zero neural network weights.
Cheetah-run, Fake Input Ablation
Episodic return
PEBBLE
800
SDP,
600
400
200
0
20
40
60
80
100
Environment steps (x 104)
Figure 12: Fake Input Study in Cheetah Run Preference Learning
D.5 SDP COMPONENT STUDIES
Figure I shows additional phase ablations in the preference learning experiments done on the Cheetan-run environment. This highlights the importance of using both phases in SDP. In addition, we ran an experiment in Walker-walk where we directly apply the procedure Yu et al. (2022) uses for leveraging sub-optimal data in the offline RL setting. They simply store the sub-optimal transi- tions in the RL agent's s replay buffer with the pseudo reward label and follow standard offline RL. We repeated this with the other difference of following standard preference learning (PEBBLE) afterwards. We found that the average final performance was 104.32 with a 95% confidence interval of 46.11 12, which close to 4 times less than the final performance of SDP.
29

---

Published as a conference paper at ICLR 2025
Cheetah-run, Phase Ablation
Episodic return
PEBBLE
800
SDP
SDP Reward Pre-train Only
SDP Agent Update Only
600
400
200
0
20
40
60
80
100
Environment steps (x 104)
Figure 13: Phase Ablation Study in Cheetah Run Preference Learning
Figure 14 shows an additional ablation over the amount of prior data used in SDP. We observed similar performance gains as described in section 5.4
Cheetah-run, Data Size Ablation
Episodic return
SDP, Prior data amount 50k
800
SDP, Prior data amount 15k
SDP, Prior data amount 5k
600
400
200
0
0
20
40
60
80
100
Environment steps (x 104)
Figure 14: Prior Data Amount Study in Cheetah Run Preference Learning
30

---

Published as a conference paper at ICLR 2025
D.6 EFFECT OF RELABELLING REPLAY BUFFER
SDP is combined with four preference learning algorithms, PEBBLE, RUNE, SURF, and MRN. core feature of these algorithms is that every time the reward model is updated, all transitions inside the RL agent's s replay buffer are updated using the latest reward model. Figure 13 shows the effects of relabeling the sub-optimal data (i.e., the reward labels) with the latest learned reward model. This means the reward labels remain frozen at zero throughout the entire training process. This ablation was demonstrate that if the sub-optimal data in the agent's replay buffer is not updated with the latest reward model, then performance will suffer. This is likely the case because incorrect reward bias from the pseudo-labeling process persists, whereas when we relabel the transitions with the updated reward model, the incorrect reward bias may reduce over time.
Walker-walk, Relabeling Sub-opt Data Abla
Episodic return
SDP
000
SDP, relabeling sub-opt data
R-PEBBLE
600
400
200
0
0
10
20
30
40
50
Environment steps (x 104)
Figure 15: Relabeling sub-opt data study: Walker-walk, scalar feedback
D.7 EFFECT OF TRANSITION QUALITY IN SDP
Moreover, show that the effectiveness of SDP relies the of sub-optimal data transitions. If we use high-quality data transitions (i.e., transitions that came from a fully trained RL agent policy), SDP will fail (see Figure (6).. This unsurprising result confirms that pseudo-labeling high-reward transitions with zero can significantly hurt the reward model and the agent's 's performance
Cheetah-run, High Quality Ablation
Episodic return
PEBBLE
800
SDP
SDP, Prior data High Quality
600
400
200
0
20
40
60
80
100
Environment steps (x 104)
Figure 16: Using high-quality data in SDP study: Cheetah-run, preference feedback
31

---

Published as a conference paper at ICLR 2025
D.8 SCALAR-BASED EXPERIMENT STATISTICS
Tables 9 and 10 provide a summary of the mean final performance and the mean area under the curve for all environments and benchmarks in the scalar feedback setting
LEARNING FROM SCALAR FEEDBACK
Task
SDP
R-PEBBLE
DEEP TAMER
WALKER-WALK
931.31 = 36.59
362.99 181.23#
33.10 = 11.10*
CHEETAH-RUN
862.72 + 49.13
522.10 238.87*
30.71 + 16.26*
QUADRUPED-WALK
777.38 + 156.74
543.92 + 193.21*
+4.7448..33
Table 9: This table shows the mean final performance plus and minus the 95% confidence interval. indicates SDP achieves significantly greater mean final performance.
LEARNING FROM SCALAR FREDBACK
Task
SDP
R-PEBBLE
DEEP TAMER
WALKKR-WALK
34981.66 + 1559.24
13147.1 + 6102.31
1823.42 + 578.48*
CHEETAH-RUN
55144.89 + 4266.266699 94
33557.58 + 15185.47*
2459.23 + 1226.34
QUADRUPED-WALK
58370.85 10759.00
37076.52 # 11968.48*
7365.68 3556.27
Table 10: This table shows the mean area under the learning curve (AUC) plus and minus the 95%
confidence interval. indicates SDP achieves significantly greater AUC.
32

---

Published as a conference paper at ICLR 2025
D.9 PREFERENCE LEARNING EXPERIMENT STATISTICS
Tables I1 14 provide a summary of the mean final performance and the mean area under the curve for all environments and benchmarks in the preference feedback setting.
TASK
FEEDBACK
METHOD
FINAL RETURN
VALUE
PEBBLE
226.4 # 136.35
0.007
SDP + PEBBLE
533.94 # 57.71
RUNE
263.48 # 176.2
SDP + RUNE
561.77 108.17
0.021
CARTPOLE-SWINGUP 48
SURF
83.12
SDP + SURF
3
0.039
MRN
151.05
SDP + MRN
644.03 5
0.0*
PEBBLE
173.9 100.26
0.001
SDP + PEBBLE
490.07 # 50.07
RUNE
160.6 # 61.85
0.0*
SDP + RUNE
575.71 104.52
WALKER-WALK
100
SURF
202.1
0.005
SDP + SURF
4
MRN
58..14
0.0*
SDP + MRN
85.53
PEBBLE
582.51 + 67.62
0.062
SDP + PEBBLE
704.63 # 102.27
RUNE
681.59 = 68.05
0.645
SDP + RUNE
616.78 279.34
CHEETAH-RUM
200
SURF
0.266
SDP + SURF
6
MRN
728.36 # 38.69
0.305
SDP + MRN
7
PEBBLE
350.24 # 228.91
0.017*
SDP + PEBBLE
753.18 # 104.79
RUNE
363.65 # 163.93
0.009
SDP + RUNE
704.1
QUADRUPED-WALK 500
SURF
550.51 2211.19
0.089
SDP + SURF
767.78 # 140.67
MRN
200.12 63.04
0.0*
SDP + MRN
733.83 # 62.87
Table 11: This table shows the final performance (mean + 95% confidence intervals) for all DM- Control preference learning experiments. indicates significant differences.
33

---

Published as a conference paper at ICLR 2025
TASK
FEEDBACK
METHOD
AUC
VALUE
PEBBLE
0.0*
SDP + PEBBLE
21519.9
+
RUNE
8817.21
0.004
SDP RUNE
2
CARTPOLE-SWINGUP 48
SURF
$593.44:1999.
0.074
SDP SURF
10288.22 16
16
MRN
6774.95 1783.95
0.0*
SDP MRN
22564.32 # 3846.6
PEBBLE
7571.79 2903.09
0.001 *
SDP PEBBLE
18458.81 # 1224.69
RUNE
6 2566.88
0.0
SDP + RUNE
21059.76 3056.28
WALKER-WALK
100
SURF
0.001 8
SDP SURF
16700.51 1449.62
MRN
7711.05 # 1295.46
0.0*
SDP MRN
25184.26 3071.65
PEBBLE
39245.94 5189.16
0.006
SDP + PEBBLE
51121.68 # 3488.87
RUNE
45824.63 4459.04
0.792
SDP RUNE
37202.31 16330.83
CHEETAH-RUN
200
SURF
40522. 00522.1849989.22 2989.22
0.218
SDP SURF
42777.1#3775.07
MRN
0.106
SDP MRN
19968.24411877
PEBBLE
20584.95 # 9814.49
0.018
SDP PEBBLE
42044.34 11183.84
RUNE
22861.89 9861.31
0.082
SDP RUNE
32664.79 3929.32
QUADRUPED-WALE WALK
500
SURF
0.074
S SURF
44176.44 8404.76
MRN
14805.07 # 1860.39
0.002
SDP MRN
35209.32 = 6225.09
Table 12: This table shows the AUC (mean 95% confidence intervals) for all DMControl prefer- ence learning experiments. * indicates significant differences.
34

---

Published as a conference paper at ICLR 2025
TASK
FEEDBACK
METHOD
FINAL RETURN
VALUE
PEBBLE
10.0 + 13.58
0.003*
SDP + PEBBLE
66.0 21.18
RUNE
4.0
0.001 *
SDP + RUNE
68.0 17.0
HAMMER
7500
SURF
0.0
0.033
SDP + SURF
25.0
MRN
4.0
0.027
SDP + MRN
46.0 27.5
PEBBLE
42.0 + 34.35 34.35
0.066
SDP + PEBBLE
80.0 15.68
RUNE
26.0 22.59
0.02*
SDP + RUNE
3
DOOR-UNLOCK
500
SURF
0.354
SDP + SURF
22.17
MRN
=
0.399
SDP + MRN
56.0 37.02 37.02
PEBBLE
38.0 + 30.06
0.035#
SDP + PEBBLE
80.0 7.84
RUNE
12.0+32.00
0.793
SDP + RUNE
40.0 + 30.87
DOOR-LOCK
500
SURF
0.684
SDP + SURF
60.0 + 13.58
MRN
66.0 + 26.93
0.268
SDP + MRN
78.0 + 17.88
PEBBLE
4.0 7.01
0.045
SDP + PEBBLE
36.0 + 25.16
RUNE
0.0
0.001 *
SDP + RUNE
14.24
DPAWER-OPEN 1000
SURF
20.0
0.018
SDP + SURF
80.07
MRN
20.0
0.136
SDP + MRN
PEBBLE
34.0 37.44
0.234
SDP + PEBBLE
54.0 26.35
RUNE
12.0 + 21.04
0.014
SDP + RUNE
38.0 + 24.42
WINDOW-OPEN 200
SURF
+.0020000 20.44
0.098
SDP + SURF
66.0 + 26.35
MRN
46.0 + 32.61
0.208
SDP + MRN
68.0 + 31.06
Table 13: This table shows the final performance (mean = 95% confidence intervals) for all Meta- world preference learning experiments. * indicates significant differences.
35

---

Published as a conference paper at ICLR 2025
TASK
FEEDBACK
METHOD
AUC
P VALUE
PEBBLE
424.0 + 175.18 18
0.017
SDP PEBBLE
2222.0 + 1009.47
RUNE
490.0 300.85
0.001
SDP + RUNE
2520.0 627.47 627.477
HAMMER
7500
SURF
482.0 #15.59
0.006
SDP SURF
3042.0 1073.54
MRN
494.0
0.016
SDP + MRN
2494.0 1108.52
PEBBLE
2142.0 - 1738.06
0.031 x
SDP PEBBLE
4838.0 # 1251.13
RUNE
1272.0 1275.71
0.047
SDP + RUNE
1836.0 2063.68
DOOR-UNLOCK
500
SURF
1580.0 = 2462.83
0.348
SDP SURF
4802.0 # 1537.81
MRN
3082.0 # 2232.43
0.342
SDP + MRN
3842.0 + 2230.38
PEBBLE
876.0 + 647.87
0.008
SDP PEBBLE
2300.0 135.0
RUNE
1342.0 + 590.05
0.809
SDP RUNE
884.0 # 633.87
DOOR-LOCK
500
SURF
1892.0 # 842.85
0.429
SDP + SURF
1986.0 184.83
MRN
0.306
SDP MRN
2236.0 479.9
PEBBLE
314.0 533.02
0.117
SDP + PEBBLE
1702.0 + 1706.
RUNE
38.0 + 49.71
0.004
SDP RUNE
3794.0 + 1306.6
DRAWER-OPEN 1000
SURF
696.0 1167.88
0.015
SDP SURF
4078.0 1839.07
MRN
1550.0 2690.98
0.211
SDP MRN
3206.0#2114.16
PEBBLE
794.0 1015.04
0.131
SDP PEBBLE
1698.0 + 822.8
RUNE
230.0 = 334.14
0.01*
SDP RUNE
948.0 + 631.92
WINDOW-OPEN 200
SURF
242.0 + 248.44
0.064
SDP +URF
2048.0 0...0-20 + 867.93
MRN
1252.0 + 796.8
0.223
SDP + MRN
1728.0 + 666.48
Table 14: This table shows the area under the curve (mean + 95% confidence intervals) for all Metaworld preference learning experiments. indicates significant differences.
36


Reviews:
[{"review_id": "nxmTOSwxUE", "reviewer": "Reviewer_RKAm", "summary": "The submission proposed a simple framework, namely Sub-optimal Data Pre-training (SDP), that leverages reward-free, suboptimal data to warm-start the reward model as well as the RL agent replay buffer. Through simulation experiments with both simulated teacher policy and real human subjects, the authors claim that the proposed SDP can significantly improves the efficiency of human feedbacks under Human-in-the-loop (HitL) reinforcement learning setting.", "questions": "- Can authors provide more thorough explanation on why introduced bias into the reward model will not affect the performance?\n- Can authors explain why Deep Tamer is not learning at all in Figure 3?\n- In Figure 6 Phase Ablation, what is the difference between SDP Agent Update Only and R-PEBBLE? Why SDP Reward Pre-train Only has better performance over SDP Agent Update Only? A more detailed explanation on the difference between these baselines will be helpful.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 1, "strengths": "- The main idea of this submission is straightforward and intuitive. Utilizing sub-optimal data is a popular way to improve sample efficiency.\n- The submission is well-written and easy to follow. Visualizations are clear and helpful.\n- The demonstrated experiments are reasonably comprehensive, invluding robotic locomotion and manipulation tasks.\n- The submission includes a 16-people human subject study.\n- The result analysis is well-formulated with multiple seeds and significant tests.", "weaknesses": "- My main concern about the submission is that the contribution is incremental without significant advance. Leveraging a set of sub-optimal data generated with randomly initialized policy as a warm-start for the reward model is straight forward, and seems to me the only major contribution. The authors follow the standard RLHF framework such as Bradley-Terry model, etc. This work failed to address any of the existing challenges, such as the assumption of linear reward feature combinations, the assumption of Boltzmann rationality of the human demonstrator, etc.\n\n- I remain somewhat unconvinced in the claim made in line 205 - line 210. Pseudo-labeling all low-quality transitions with the same minimum environmental rewards indeed introduces bias into the reward model, and will further lead to reward ambiguity. The empirical results in the DMControl suite is not sufficient to claim that the bias for using an incorrect reward is low. In fact, I doubt the effectiveness in pseudo-labeling low-quality samples in more complex real-world robotic tasks with higher dimensions. It would be helpful if the authors can include additional experiments on higher-dimensional simulation or real-world experiments to showcase the effectiveness.\n\n- The authors took a great effort to include a well-formulated human subject study. I don't think it deserves any extra credits as the human subjects are only asked to compare two clips and provide preference labels, instead of provide direct interventions (learning from human intervention). I think the user study with diverse human subjects will be mostly useful if they can provide direct intervention and correction trajectories. A single preference label over two trajectory pairs might not fully reflect human's intention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pLoW6I7qmY", "reviewer": "Reviewer_xKys", "summary": "Use reward-free sub-optimal data for pretraining human in the loop RL by adding a reward. The reward is trained by assigning the minimum environment reward to all states in the reward free suboptimal data, and iteratively adding interactions. The online interactions can be labeled either with preference-based RL or with scalar rewards. This suboptimal-data aware method is then evaluated over several experiments against other human in the loop algorithms.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The algorithm is straighforward.\n\nThe method is well explained and clear.\n\nThe empirical analysis is good.\n\nRunning human in the loop with human participants is admirable.", "weaknesses": "The assumption made in Equation 4 is quite restritive, since this states that the per-state reward must be low for every state in a suboptimal trajectory. There are many tasks for which a sub-optimal trajectory may receive significant non-trivial reward, suhc as achieving an intermediate goal, while still being significantly suboptimal. In addition, knowledge of the minimum environment reward can also be a limiting assumption, though less so.\n\nThe method may be a little bit too straightforward, as the concept of applying a small, fixed, low reward given to unlabeled interaction data has been explored in the offline/off-policy RL literature, which has a lot of overlap with human in the loop RL. Since the method proposes very little to convert this interaction data into the context of human in the loop, this makes the algorithm itself of low novelty. It seems like knowledge about the scale or type of human in the loop data (preference or scalar) would provide some insight into a better method for assigning the suboptimal data.\n\nHow does relabeling the suboptimal data with the preference learned rewards through training compare? This seems like a logical experiment that could provide insight into how much the suboptimality assumption can be relaxed.\n\nWhile the baseline evaluation is extensive, it seems like a missing area are methods that utilize suboptimal data. As a comparison, pretraining with an offline RL method that can leverage suboptimal data smoothly would be preferred, at least as an upper bound on performance.\n\nWhile user load is a serious concern, a participant study that investigated more complex domains, and experiments in more complex domains such as Franka Kitchen or 2D minecraft would illustrate the efficacy of this method. In these control tasks it is not particularly realistic to have a large amount of entirely reward-free data, whereas in a robotic kitchen this is more plasible. Since this method is fundamental, an experiment along this line would make sense.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FcWLEeVgm9", "reviewer": "Reviewer_85xh", "summary": "The paper introduces Sub-optimal Data Pre-training (SDP), a method for improving human-in-the-loop reinforcement learning by pre-training reward models with sub-optimal, reward-free data to enhance feedback efficiency and performance.", "questions": "1. Is learning with the SDP method more effective than learning with sparse rewards?\n2. Does SDP require more data for more complex environment?\n3. How well does the SDP approach perform when applied to high-dimensional action or state spaces, such as humanoid?\n4. Does the human teacher need to undergo training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This SDP approch is novel\n2. This paper has some real world validation (human study)", "weaknesses": "1. The performance of the SDP method is highly dependent on the quality and representativeness of the sub-optimal data used for pre-training. Limited data availability or random policies generated from the same initial seeds can negatively impact SDP performance.\n2. The human study in this paper did not account for human variance. For instance, would differences in education levels affect the quality of the labels?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SjcDoZsgqK", "reviewer": "Reviewer_TrDy", "summary": "This paper presents a method to initialize a reward network for RL with human feedback. Their method centers around using random exploration to then label sub-optimal trajectories with low rewards and therefore filter those state action pairs from exploration once the human preference phase begins. Their results show that this method is able to improve results in baselines and is agnostic to which human preference algorithm used.", "questions": "There are no baseline comparison here. Is this the only work that you could augment the reward function with? I know there are other methods which do data augmentation possibly for the reward function, is there no such reasonable comparisons? If no that is fine but I would imagine there is (maybe not limit yourself to only pre-trained options if needed?). \n\nHow do we determine r_min?\n\nWhat future work is there?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Research Problem. The research problem is at a high level “how to use unlabeled data to warm start RL with human preferences”. Since human preferences (or even AI preferences) are a reasonable and increasingly common method for learning reward models as opposed to hand coding them improving such algorithms with cheap data is a good research problem.\n\nNovelty. The novelty seems reasonable from the related work section.\n\nSignificance/Contribution. The significance of this paper is solid. It has a high practicality in many empirical applications and I am of the opinion that works that improve data efficiency are critical in RL research.\n\nAlgorithm Details. The algorithm itself isn’t extremely complex but the authors do a good job explaining it.\n\nExperimental Selection. The experiments ran emphasize the point of the paper well. \n\nPresentation of Results. All figures are very well done and very clear.\n\nAblation Studies. Sensitivity study is done with hyperparams and shows relative robustness. \n\nEmpirical Results. Good, results indicate this is a successful method and general strategy for improving preferences.\n\nFailure Analysis. There is no failure analysis in this paper. I wish for the Rune + SDP baseline you would talk about why it doesn’t help in this case. Is there a particular part of rune that makes it work worse? In what future algorithms would this strategy not work.\n\nClarity is generally good. See weakness for improvements.", "weaknesses": "I think the related work section could be improved. I understand that this work is human in the loop RL but I feel the human in the loop RL section is lacking. None of this section is actually compared to this work and doesn’t emphasize what makes this work novel. I understand that it isn’t directly related to most of these, but then I believe you should choose papers that this work relates more directly to (improving reward models for RL). I would recommend you shorten the Human in the loop section and add a section based on improving reward models.\n\nI have a few issues with the experimental section. You say over 5 seeds. Does that mean each bar is only the average/CI of 5 runs? If I just missed the number let me know, if I didn’t miss it and that is correct, I’d recommend running these many more times (I realize it might take a while but 5 isn’t enough). You say the that the addition of SDP never huts performance. Your plots in Figure 2 suggests otherwise. It generally improves performance but for rune SDP hurts performance in a few of the tasks. If you mean never hurts performance by a statistically significant margin this is fine but I would prefer you say this explicitly as this is slightly misleading in my opinion. \n\nIs it possible to hypothesize why not only does this improve sample efficiency but from Figure 4 seems to also improve finals results? \n\nClarity: I wish the authors would more explicitly/intuitively define some of the terms. I was able to figure it out eventually but helping the reader would be great. Specifically:\n-\t“Low quality transitions”\n-\t“Minimum environment reward”\n-\t“Feedback efficiency” \nI wish the authors didn’t use the term “RQ”. Instead, if it possible, redefine them in the paragraph where you discuss them. No reason to make the reader jump around.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)