Query-Policy Misalignment in Preference-Based Reinforcement Learning
Authors: Xiao Hu
Jianxiong Li
Xianyuan Zhan
Qing-Shan Jia
Ya-Qin Zhang
Venue: ICLR 2024 spotlight (2024)
Keywords: preference-based reinforcement learning
human feedback efficiency
query-policy misalignment
Primary area: reinforcement learning
TL;DR: We identify a long-neglected issue of existing PbRL studies: Query-Policy Misalignment, and propose a minimalist approach to address it.

Abstract: Preference-based reinforcement learning (PbRL) provides a natural way to align RL agents’ behavior with human desired outcomes, but is often restrained by costly human feedback. To improve feedback efficiency, most existing PbRL methods focus on selecting queries to maximally improve the overall quality of the reward model, but counter-intuitively, we find that this may not necessarily lead to improved performance. To unravel this mystery, we identify a long-neglected issue in the query selection schemes of existing PbRL studies: Query-Policy Misalignment. We show that the seemingly informative queries selected to improve the overall quality of reward model actually may not align with RL agents’ interests, thus offering little help on policy learning and eventually resulting in poor feedback efficiency. We show that this issue can be effectively addressed via policy-aligned query and a specially designed hybrid experience replay, which together enforce the bidirectional query-policy alignment. Simple yet elegant, our method can be easily incorporated into existing approaches by changing only a few lines of code. We showcase in comprehensive experiments that our method achieves substantial gains in both human feedback and RL sample efficiency, demonstrating the importance of addressing query-policy misalignment in PbRL tasks.

Published as a conference paper at ICLR 2024
QUERY-POLICY MISALIGNMENT IN PREFERENCE- BASED REINFORCEMENT LEARNING
Xiao Hu' .Janxiong Li Xianyuan Zhan'
Qing-Shan Jia't & Ya-Qin Zhang'
1 Tsinghua University, Beijing, China 2 Shanghai Artificial Intelligence Laboratory, Shanghai, China (hu nau-x21,20 1i 1i-jx21) @mails. ts tsinghua edu, en, zhanxianyuanea tss.gginnai....ssstia...a. edu, on
ABSTRACT
Preference-basee reinforcement learning (PbRL) provides natural way to align RL agents' behavior with human desired outcomes, but is often restrained by costly human feedback. To improve feedback efficiency, most existing PhRi. methods focus on selecting queries to maximally improve the overall quality of the reward model, but counter-intuitivelyy we find that this may not necessarily lead to improved performance. To unravel this mystery, issue in the query selection schemes of existing PbRL studies: Query-Police Misalignment. We show that the seemingly informative queries selected to improve the overall quality of reward model actually may not align with RL agents' interests, thus offering little help on policy learning and eventually resulting in poor feedback efficiency. We show that this issue can be effectively addressed via policy-aligned query and specially designed hybrid experience replay, which together enforce the bidirectional query-policy alignment. Simple yet elegant, our method can be easily incorporated into existing approaches by changing only few lines of code. We showcase in comprehensive experiments that our method achieves substantial gains in both human feedback and RL sample efficiency, demonstrating the importance of addressing query-policy misalignment in PbRL tasks.Code is available at ht https: : github. com/ hwwphhhtx/iaeaeee/ QPA.
identify a long-neglected
INTRODUCTION
Reward plays an imperative role in every reinforcement learning (RL) problem. It specifies the learning objective and incentivizes agents to acquire correct behaviors. With well-designed rewards, RL has achieved remarkable success solving many complex tasks (Mnih al., 2015; Silver et al., 2017b; Degrave et al., 2022). However, designing suitable reward function remains longstanding challenge (Abel et al., 2021; Li et al., 2023; Sorg, 2011). Due to human cognitive bias and system complexity (Hadfield-Menel) et al., 2017). it is diffficult to accurately convey complex behaviors through numerical rewards, resulting in unsatisfactory or even hazardous agent behaviors.
Preference-based RL (PbRL), also known as RL from human feedback (RLHF), promises learning re- ward functions autonomously without the need for tedious band-engieeered reward design (Christiano al., 2017; Lee et al., 202 la;b; Park et al., 2022; Liang et al., 2022; Shin et al., 2023; Tien al., 2023). Instead of using provided rewards, PbRL queries a (human) overseer to provide preferences between pair of agent's behaviors, and the RL agent seeks maximize reward function that is trained to be consistent with human preferences. This approach provides more natural way for humans to communicate their desired outcomes RL agents, enabling more desirable behaviors (Christiano et al., 2017). However, labeling large number of preference queries requires tremendous human effort, inhibiting its wide application in real-world scenarios (Lee et al., 2021b; Park et al., 2022; Liang et al., 2022). Thus, in PbRL, the feedback efficiency is of utmost importance.
To enable feedback-efficies PbRL, overseer's preference and which ones not to, in order to extract as much information as possible from each preference labeling process (Christiano et al., 2017: Lee al., 2021b; Biyik & Sadigh, 2018; Biyik et al., 2020). Motivated by this, existing works focus on querying the most "informative"
crucial to carefully select which behaviors query the
"The two authors contributed equally.
CCoreeponnencce Xianyuan Zhan, Qing-Shan Jia and Ya-Qin Zhang

---

Published as a conference paper at ICLR 2024
Query-Policy Misalignment
Historical behavior
Do hurt human!
Now, how can grasp these blocks?
Querry
Joe Select "Informative" behaviors to query oversaer
Overseer
Bob
Bob still don't ton't know how to pick up these blocks!
Figure 1: Illustration of query-policy misalignment. Bob's current focus is on grasping the blocks. However, the overseer advises him
harm to humans instead of providing guidance grasping techniques.
behaviors for preferences that are likely to maximally rectify the overall reward model, such as sampling according to ensemble disagreements, mutual information, or behavior entropy (Leee et al., 2021b; Shin et al., 2023). However, it is also observed that these carefully designed query schemes often only marginally outperform the simplest scheme that randomly selects behaviors to query human preferences (Ibarz et al., 2018; Lee et al., 2021b). This counter-intuitive phenomenon brings about a puzzling question: Why are these seemingly informative queries actually not effective in PbRL training?
In this paper, we identify a negleeeee issue in the query schemes of existing approaches that is responsible for their ineffectiveness: Query-Policy Misalignment. Take Figure Bob is a robot attempting to pick up some blocks, while Joe is a querier selecting the behaviors that he considers the most informative for the reward model to query the overseer. However, the chosen behaviors may not align with Bob's s current interests. So, even after the query, Bob is still clueless about how pick up blocks, which means that a valuable query opportunity might be wasted. In Section 4, we provide concrete experiments and find that such misalignment is prevalent in previous query schemes. Specifically, we observe that the queried behaviors often fall outside the scope the current policy's 's visitation distribution, indicating that they are less likely to be encountered by the current RI. agent, and thus are not what the current agent's interests. Therefore, the queried behaviors have little impact on the current policy learning and result in poor feedback efficiency.
illustration,
Interestingly, we find that the query-policy misalignment issue can be easily addressed by policy- aligned query selection, which can be implemented by making simple modification to existing query schemes. This technique ensures that the query scheme only selects the recent behaviors of RL agents, which in turn enables the overseer to provide timely feedback relevant behaviors for policy learning rather than some irrelevant experiences. By making this minimalist modification to the query schemes of existing methods, we showcase substantial improvements in terms of both feedback and sample efficiency as compared to the base schemes. Further leveraging the insight from query-policy misalignment, we introduce simple technique, called hybrid experience replay, that simply updates RL agents using experiences uniformly sampled from the entire replay buffer and some recent experiences. Intuitively, this technique ensures that the RL agent updates more frequently on the region where human preferences have been recently labeled, thereby further aligning the policy learning with the recent human preferences.
In summary, the combination of policy-aligned query selection and hybrid experience replay estab- lishes bidirectional query-policy alignment, making every query accountable for policy learning. Notably, these techniques can be easily incorporated into existing PbRI. approaches (Lee et al., 2021 Park et al., 2022) with minimal modifications. We evaluate our proposed method on benchmark environments in DeepMind Control Suite (DMControl) (Tassa et al., 2018) and Meta World (Yu et al., 2020). Simple yet elegant, experimental results demonstrate significant feedback and sample effi- ciency gains, highlighting the effectiveness of our proposed method and the importance of addressing query-policy misalignment in PbRL.
RELATED WORK
PbRL provides natural approach for humans (oracle overseer) to communicate desired behaviors with RL agents by making relative judgments between pair of behaviors (Akrour et al., 2011; Christiano et al., 2017; Stiennon et al., 2020; Wu et al,, 2021). However,
Pilarski et al., 2011;
2

---

Published as a conference paper at ICLR 2024
acquiring preferences is typically costly, imposing high demands on feedback efficiency (Lee et al., 2021b; Park et al., 2022; Liang et al., 2022).
Query selection schemes in PbRL. It is widely acknowledged that the query selection scheme plays a crucial role in PbRL for improving feedback efficiency (Christiano et al., 2017: Biyik & Sadigh, 2018; Biyik al., 2020; Ibarz et al., 2018; Lee et al., 2021b). Motivated by this idea, prior works commonly assess the information quality of queries using metrics such as entropy (Biyik & Sadigh, 2018; Ibarz et al., 2018: Lee et al., 2021b), the L2 distance in feature space (Biyik et al., 2020) or ensemble disagreement of the reward model (Christiano et al., 2017; Ibarz et al., 2018; Lee al., 202 Park al., 2022: Liang al., 2022). Based on these metrics, researchers often employ complex sampling approaches such as greedy sampling (Biyik & Sadigh, 2018), K-medoids algorithm (Biyik & Sadigh, 2018; Rdusseeun & Kaufman, 1987), or Poisson disk sampling (Bridson, 2007; Biyik et al., 2020), etc., to sample the most "informative" queries. Despite adding extra computational costs, is observed that these complex schemes often benefit little to policy learning. and sometimes perform similarly to the simplest scheme that directly queries humans with randomly selected queries (Lee et al., 2021b; Ibarz et al., 2018). In this paper, we identify a common issue with the existing schemes, query-policy misalignment, which shows that the selected seemingly "informative" queries may not align well with the current interests of RL agents, providing an explanation of why existing schemes often lead to less improved feedback efficiency.
Other techniques for improving feedback efficiency. In addition to designing effective query selection schemes, there are other efforts improve the feedback efficiency of PbRL from various perspectives. For instance, some works focus on ensuring that the initial queries are feasible for humans to provide high-quality preferences by initializing RL agents with imitation learning (lbarz et al., 2018) or mnsupervised-pretraiin exploration can improve both sample and feedback efficiency. Recently. Park et al. (2022) applies pseudo-labeling (Lee et al., 2013) in semi-supervised learning along with temporal cropping data augmentation to remedy the limited human preferences and achieves SOTA performances. Note that our proposed method is orthogonal to these techniques and can be used in conjunction with any of them with minimal code modifications.
(Lee et al., 2021b). Liang et al. (2022) shows that adequate
Local decision-aware model learning. The key idea of our proposed method is to learn the reward mode precisely within the distribution of the current policy, rather than inadequately within the entire global state-action space. Some model-based decision decision-making methods share similar idea with ours, emphasizing the importance of learning a critical local dynamics model instead of a global one enhance sample complexity. Specifically, a line of model- policy search work aims to learn the local dynamics model on current states or trajectories to expedite convergence (Levine & Abbeel, 2014; Levine et al., 2015; Fu et al., 2016, 6; Lioutikov et al., 2014; Bagnell & Schneider, 2001; Atkeson et al., 1997). Some work on value-equivalent models focuses on learning the dynamics model relevant for value estimation (Grimm al., 2020; Srinivas al., 2018; Oh et al., 2017: Silver et al., 2017a: 7a; Tamar et al., 2016). Several based RL. methods propose to learn the model centering on the current policy or task (Wang et al., 2023; Lambert al., 2020; Ma al., 2023).
3 PRELIMINARY
The RL problem is typically specified as Markov Decision Process (MDP) (Puterman, 2014), which is defined by a tuple M == (S,Ar 7). S, A represent the state and action space, r: S x A - IR the reward function, T: S 1 A - S is the dynamics and 7 E (0, 1) is the discount factor. The goal of RL is to learn a policy π : S + A that maximizes the expected cumulative discounted reward.
Off-policy actor-critic RL. To tackle the nigh-dimensional state-action space, off- actor-critic RI. algorithms typically maintain parametric Q-function Qo(s, a) and a parametric policy n (a)s), which are optimized via alternating between policy evaluation (Eq. (1)) and policy improvement (Eq. (2)) steps. The policy evaluation step seeks to enforce Qo(s, a) to be consistent with the empirical Bellman operator that backs up samples (s, a, s') stored in replay buffer D, while the policy improvement step improves Th via maximizing the learned Q-value:
g++ + arg argmin min F. Feaa.'
(F(0,)) + +yE [Q" (d. - 2(0.0)))]
(1)
Q
1+1 arg max Eow EewDaartial)
[ [s,
(2)
Prefereece-based RL. Different from the standard RL setting, the reward signal is not available in PbRL.. Instead, (human) overseer provides preferences between pairs of trajectory segments,
3

---

Published as a conference paper at ICLR 2024
3rd
4th
5tht
10
IB No CI suriiiyy Dhery
distibutiio
3.6
10
Leamedd neward
8.0
6.0
50
4.0
24
2.0
3.0
1.8
-2.
1.2
84
4.
(o) description,
5
30
0
0.0
Grand Inth meden
--14
0300 C00 coneent Quattty
Vistation distibbution. d1
2,4
Vistat didr Buttoon
10
Leamned reward
Desired Behavior
1.40
2.1
7.5
-1,
2.5
2.0
2.2
1.0
12
00
(b) Desired behavior.
10
distritutioo
Vistation
Miskation distributionn df
90
3.8
3.5
2:3
23
54
72
1.2
14
36
Now colice ouen Unbeer
00
Celiestmmple (2.65)
10
00
0.0
4
10
4
-1.
Learning curves.
[4) Visualization of training processes of different query selection schemes and the final learned rewards.
Figure 2: Impacts of query-policy misalignment in PbRL training. (a). 2D navigation task. RL agent should navigate the goal. (b). The desired behavior of this task is to move to the goal in straight line. (c). The learning curves of different query selection methads. (d). Existing query selection methods often select queries that lie outside the visitation distribution the current policy.
and the agent leverages these feedbacks to learn a reward function P.. S x A + RR to be con- sistent with the provided preferences. A trajectory segment is sequence of states and actions (8k, ak, ... I.-I,-1; Ob (.++1-1) } E (S x A) which is typically shorter than the whole trajectories. Given pair of segments (a a ) the overseer provides feedback signal y indicating which segment the overseer prefer, i.e., E {0, 1}. where 0 indicates the overseer prefers segment o" over a², 1 otherwise. Following the Bradley-Terry model (Bradley & Terry, 1952). we can model preference predictor Ps using the reward function as)
exp E, Po (s).
Pa [ot > =0] Eie(o.1) exp Ee Pip (s),
where 0² > oº denotes the overseer prefers a³ than 0 Typically, Fs is optimized by minimizing the cross-entropy loss of preference predictor P. and the true preference label y.
Creward
E
[(1 - y) log Ps. [o" - ylog Ps [o-> o
where D' denotes the preference buffer which stores the history overseer's preferences {(o", o², .v)}. Most recent PbRL methods (Lee al., 202 Park et al., 2022; Liang et al., 2022) are built upon off-policy actor-critic RL algorithms to enhance sample and feedback efficiency. In these methods, pairs of segments (a o²) are selected from trajectories in the off- off-policy RL replay buffer D. These selected pairs are then sent to the overseer for preference query, yielding feedback (a", o .v). These feedback instances are subsequently stored the separate preference buffer D for reward learning. The query selection scheme refers to the strategy that decides which pair of segments (a", o!) should be selected from D for preference query prior to the reward learning. An effective query selection scheme is of paramount importance to achieve high feedback efficiency in PbRL.
4 QUERY-POLICY MISALIGNMENT
motivating example. In this section, we conduct an intuitive experiment to demonstrate a prevalent but long-neglected issue: query-policy misalignment, which accounts for the poor feed- back efficiency of existing query selection schemes. Specifically, as illustrated in Figure 2(a), we consider 2D continuous space with (x, v) coordinates defined on [0, 10 For each step, the RL agent can move Ar and Ay ranging from [-1,1]. We want the agent to navigate from the start to the goal quickly as possible. We run PEBBLE (Lee et al., 2021b), a pop- ular PbRL method, with two widely used query selection schemes in previous studies: uni- form query selection that randomly selects segments to query preferences and disagreement
4

---

Published as a conference paper at ICLR 2024
query selection that selects the segments with the largest ensemble disagreement of preference
predictors (Lee et al., 2021b; Christiano et al., 2017; Ibarz et al., 2018; Park et al., 2022).
We also run PEBBLE with policy-aligned query selection that selects the recently collected segments from the policy- environment interactions and will be further investigated in later content. We track the distribution d' of current agent's 's policy π throughout the training process and plot the selected segments of different query selection schemes in Figure 2(d). Please refer to the Appendix C.1 for detailed experimental setups.
Visitation distribution d
10
2.1
b.
a
1.5
0.9
0.6
We observe that the selected segments of existing query selec- tion schemes typically fall outside the scope of the visitation distribution d' (marked with green circles). We refer to this phenomenon as query-policy misalignment, as illustrated in Fig- ure 3. Such misalignment wastes valuable feedback because the overseer provides preferences on experiences that are less likely encountered by, or in other words, irrelevant to the current RL agent's ' learning. Therefore, although these selected segments may improve the overall quality of the reward model in the full state-action space, they contribute little to current policy training and potentially cause feedback inefficiency. By contrast, policy-aligned selection selects fresh segments that are recently visited by the current RL policy, enabling timely feedback on the current status of the policy and leading to significant performance gain, as shown in Figure 2(c). Furthermore, while the learned reward of policy-aligned query selection may differ from the ground truth reward, it provides the most useful information to guide the agent to navigate toward the right direction and more strictly discourages detours as compared the vague per-step ground truth reward. This suggests that the learned reward captures more targeted information in solving the task while also blocking out less useful information in the ground truth reward, thus enabling more effective policy learning. For additional evidence substantiating the existence of the query-policy misalignment issue and its contribution to feedback inefficiency, please refer to the Appendix D.1.
0.3
0.0
Quory-Policy Misalignment
Figure 3:
Query-policy misalignment. Existing query selection methods often select queries that lie outside the visita- tion distribution of the current policy.
As mentioned earlier, existing PbRL methods struggle with feedback inefficiency caused by query- policy misalignment. To address this issue, we propose an elegant method: QPA (Query-Policy Alignment). The key idea of QPA is that rather than learning the reward model across the entire global atate-action space inadequately as existing methods do, it is more effective to focus on learning the reward model precisely within the distribution of the current policy d", utilizing the same amount of human feedback. On the other hand, the Q-function should also be accurately approximated around d= We present an error bound in Appendix to intuitively explain why this idea is reasonable.
5 QUERY-POLICY ALIGNMENT FOR PREFERENCE--BASEE RL (QPA)
In this section, introduce an elegant solution: QPA, which can effectively address query-policy misalignment and is also compatible with existing off-policy PbRL methods with only 20 lines of code modifications. Please see Algorithm I for the outline of our method.
5.1 POLICY-ALIGNED QUERY SELECTION
In contrast to existing query selection schemes, we highlight that the segment query selection should be aligned with the on-policy distribution. In particular, it crucial to ensure that the pairs of segments (o", o²) selected for preference queries stay close to the current policy's 's visitation distribution d", By assigning more overseer's feedback segments obtained from on-policy trajectories of the current policy n, we aim to enhance the accuracy of the preference (reward) predictor within the on on-policy distribution d= We refer this query scheme as policy-aligned query selection.
In practice, a natural approach to implement policy-aligned query selection is to utilize the current policy л to interact with the environment and generate a set of trajectories. Then, pairs of segments (o",a1) ot) can be selected from these trajectories to obtain overseer's 's preference query. While such "absolute" policy-aligned query selection ensures that all selected segments conform to the on-policy distribution d", it may have a negative impact the sample efficiency of off-policy RL due to the additional on-policy rollout. Instead of performing the "absolute" policy-alignen query selection, an alternative is to select segments that within or "near" on-policy distribution. As we mentioned in Section 3, in typical off- off-po PbRL, (o", o') are selected in trajectories sampled from the RL replay buffer D. A simple yet effective way perform policy-aligned query selection is to
5

---

Published as a conference paper at ICLR 2024
QPA (20% query explore)
(50% query explore)
QPA
Waleresaak (eeddbace 1002
Wellbe-gn (teedback
130 Quddhupedd welk (Beedbacc 1000
1000 Suaddupedd mun (teebbeck: 1080
Epcodd Return
Epponde Return
Episode
Epscod
500
250
20
20
Environment (x10f)
023
356
(x10f)
Eriiil 0x10f)
Envtronment 1X20%)
Figure 4: Learning curves of QPA and QPA (20% or 50% query explore) on locomotion tasks. In QPA, 100% of the queries are sampled from the policy-aligned buffer 2 In QPA (20% or 50% query explore), we sample 20% or 50% queries from the entire replay buffer D, while the remaining 80% or 50% of queries are sampled from D The declining performance of QPA (20% 50% query explore) suggests that allocating some queries to the entire replay buffer in an attempt to explore high-reward state regions can compromise feedback efficiency.
choose (o9,o)) from the most recent trajectories stored in D. For the sake of clarity, we refer to the buffer that stores the most recent trajectories as the policy-aligned buffer DD It's worth noting that this simple approach strikes a balance between policy-aligne query and sample efficiency in RL. Furthermore, it is particularly easy to implement and enables a minimalist modification to existing PbRL methods. Take the well-known and publicly-available B- B-pref (Lee et al., 2021a) PbRL implementation framework as an example, all that's required is to reduce the size of the firstnnffrst-out query selection buffer, which can be thought of as a knockoff of DP
The use of policy-aligned query confines the query selection to the policy-alignes buffer D which can avoid the uninformative exploration of query selection within the entire replay buffer D. We show in Figure 4 that the excessive exploration of query selection could hurt feedback efficiency. Querying segments from entire replay buffer D in an attempt to explore high-reward state regions can only get the high-reward segments by chance. Allocating more queries D cannot guarantee finding high-reward regions, but potentially resulting in wasted human feedback. In contrast, the queried segments only sampled in DF can provide valuable information for learning the rewards within on- on-policy distribution, resulting in continual improvement in current policy. Given the limited feedback typically encountered in real- world scenarios, it is preferable to help current policy to get more local guidance using policy-alignes query selection. Please refer to Appendix D.4 for more discussions and experiments about the exploration in PbRL.
5.2 HYBRID EXPERIENCE REPLAY
Following the policy-aligned query selection and reward learning procedure, it's also important to ensure that value learning is aligned with the on-policy distribution. Specifically, more attention should be paid improving the veracity of Q-function within on-policy distribution d", where the preference (reward) predictor performs well in the preceding step.
To update the Q-function, existing off-policy PbRL algorithms perform the empirical Bellman iteration (1) by simply sampling transitions (s, a, s') uniformly from the replay buffer D. Although the aforementioned policy-aligned query selection can effectively facilitate the learning of the reward function within d",, the Q-function may not be accurately approximated on d' due to inadequate empirical Bellman iterations using (s, a, a') transitions drawn from d= Taking inspiration from combined Q-learning (Zhang & Sutton, 2017) and some prioritized experience replay methods that consider on-policyness (Liu et al., 2021; Sinha et al., 2022), we devise a hybrid experience replay mechanism. To elaborate, we still sample transitions (s, s') uniformly, but from two different sources. Specifically, half of the uniformly sampled transitions are drawn from D, while the other half are drawn from the policy-ali policy-aligned buffer DM In other words, the hybrid sample ratio between Dea and D in experience replay is set to 0.5. The proposed mechanism can provide assurance that the Q-function is updated adequately near dd
5.3 DATA AUGMENTATION FOR REWARD LEARNING
Besides the above two key designs, we also adopt the temporal data augmentation technique for reward learning (Park et al., 2022). To further clarify, we randomly subsample several shorter pairs of snippets (6", x²) from the queried segments (o o² .y), and put these (6", _-, w) into the preference buffer D for optimizing the cross-entropy loss in (4). Data augmentation has been widely used in many deep RL (Kostrikov et al., 2020; Laskin et al., 2020a;b) algorithms, and also has been seamlessly integrated into previous PbRL algorithms for consistency regularization (Park et al., 2022). Diverging from the practical implementation in Park et al. (2022), we generate multiple (0",0'.v)

---

Published as a conference paper at ICLR 2024
instances from single (o oo, y)) as opposed to generating only one (ō ,', .)) instance from each (o a yy. which effectively expands the preference dataset. An in-depth ablation analysis of the data augmentation technique is provided in Section 6.3 and Appendix D.5.
6 EXPERIMENT
In this section, we present extensive evaluations on 6 locomotion tasks in DMControl (Tassa et al., 2018) and 3 robotic manipulation tasks in MetaWorld (Yu et al., 2020).
6.1 IMPLEMENTATION AND EXPERIMENT SETUPS
QPA can be incorporated into any off- off-policy PbRL algorithms. We implement QPA on top of the widely-adopted P6RL backbone framework B-Pref (Lee et al., 202 We configure the policy- aligned query selection with a policy-aligned buffer size of N = 10. The hybrid experience replay is implemented with a sample ratio of w 0.5, and we set the data augmentation ratio at 7 = 20.
In our experiments, demonstrate the efficacy of QPA in comparison to PEBBLE (Lee et al., 2021b), the SOTA method, SURF (Park et al., 2022), and on-policy PbRL method PrePPO (Christiano et al., 2017). As QPA, PEBBLE, and SURF all employ SAC (Haarnoja et al., 2018) for policy learning, we utilize SAC with ground truth reward as reference performance upper bound for these approaches. For PEBBLE and SURF, we employ the disagreement query selection scheme in their papers that selects segments with the largest ensemble disagreement of reward models (Lee et al., 2021b; Park al., 2022; Ibarz et al., 2018). To be specific, we train an ensemble of three reward networks Ty with varying random initializations and select (o": o.)) based on the variance of the preference predictor Pe. While leveraging an ensemble of reward models for query selection may offer improved robustness and efficacy in complex tasks as observed in (Lee et al., 202 lb; Ibarz et al., 2018) 8), the additional computational cost incurred by multiple reward models can be unacceptable in scenarios where the reward model is particularly large, e.g., large language models (LLM). Hence in QPA, we opt to use a single reward model and employ policy-aligned query selection (simply randomly selects segments from the on-policy buffer DP After all feedback is provided and the reward learning phase is complete, we switch from the hybrid experience replay to the commonly used uniform experience replay for policy evaluation in QPA.
In each task, QPA, SURF, PEBBLE and PrePPO utilize the same amount of total preference queries and feedback frequency for a fair comparison. We use an oracle scripted overseer provide pref- erences based on the cumulative ground truth rewards of each segment defined in the benchmarks. Using the scripted ground truth overseer allows us evaluate the performance of PbRL algorithms quantitatively, unbiasedly and quickly, which is common practice in previous PbRL literature (Chris- tiano et al., 2017; Lee et al., 202 Park al., 2022; Liang et al., 2022). We perform 10 evaluations on locomotion tasks and 100 evaluations on robotic manipulation tasks across 5 runs every 1 environment steps and report the mean (solid line) and 95% confidence interval (shaded regions) of the results, unless otherwise specified. Please see Appendix C.2 for more experimental details.
6.2 BENCHMARK TASKS PERFORMANCE
Locomotion tasks in DMControl suite. DMControl (Tassa et al., 2018) provides diverse high- dimensional locomotion tasks based on MuJoCo physics (Todorov et al., 2012). We choose 6 complex tasks in DMControl: Walker_walk, Walker_run, Cheetah_run, Quadruped_wadr Humanoid _ttidddd_tddddiiiii Figure 5 shows the learning curves of SAC (green), QPA (red), SURF (brown), and PEBBLE (blue) on these tasks. As illustrated in Figure 5, QPA enjoys significantly better feedback efficiency and outperforms SURF and PEBBLE by substantial margin on all the tasks. To be men- tioned, the SOTA method SURF adopts seeudo-labeling based semi-supervised learning technique to enhance feedback efficiency. By contrast, QPA removes these complex designs and achieves consistently better performance with a minimalist algorithm. To further evaluate feedback efficiency of QPA, we have included additional experiment results in Appendix D.3, showcasing the perfor- mance of these methods under varying total amounts of feedback and different feedback frequencies. Surprisingly, we observe that in some complex tasks (c.g., Quadraped _walk, Quadruped_ruue can even surpass SAC with ground truth reward during the early training stages, despite experiencing stagnation of performance improvement as feedback provision is halted in the later stages. We provide additional experiment results in Section 6.4 to further elaborate on this phenomenon.
Quadruped_ _unn
QPA
Robotic manipulation tasks in Meta-world. We conduct experiments on complex manipulation tasks in Meta-world (Yu et al., 2020): Door-unlock, Drawer-open, Door-open. The learning curves
7

---

Published as a conference paper at ICLR 2024
presented in Figure 6. Similar prior works (Christiano et al., 2017; Lee et al., 202 b;;,b;a; Park et al., 2022; Liang et al., 2022), we employ the ground truth success rate as a metric to quantify the performance of these methods. Once again, these results provide further evidence that QPA effectively enhances feedback efficiency across a diverse range of complex tasks.
The dashed black line depicted in Figure 5, 6 and subsequent figures represents the last feedback collection step. The stopping step for feedback collection is set to align with that of previous work (Park et al., 2022), facilitating straightforward cross-referencing for readers. We conduct additional experiments with different feedback stop steps in Appendix D.3. While increased feedback enhances the performance of PbRL methods, the limited amount of total feedback enables a more effective evaluation of the feedback efficiency of these PbRL methods. In Meta World environment, the variance of these PbRL algorithms increases, which has also been observed other PhRL. literature (Lee et al., 2021b; Park et al., 2022; Liang et al., 2022). For clear comparison, we provide Table 5 in Appendix D.3 to summarize the mean and standard deviation of these PbRL methods.
with ground reward
PEBELE
1030
Walkern-all (fendback-1000
3000
Cheetai--un ('eedback-=10)
1000
Walker-run (feedhack- 2501
Eptsade Return
790
Epicode neturn
350-
Episode Return
250-
500
300
200-
230-
200
800
853
Enviranment Steps 6x10%)
Environment (x10%)
Environment Steps (x107)
1000 Quadruped walk (feedback=1000)
I0e0 Quadruped-run (feedback=1000)
se Humenood---o (feedbacker 10000)
Egisode Return
750
Ealcode Ratur
150-
-00
500
Episode 200
250
250
81c
033
Environment Steps (x10*)
Environment Steps (x10%)
Environment Steos (x10*)
Figure 5: Learning curves on locomotion tasks as measured on the ground truth reward. The dashed black line
represents the last feedback collection step.
with
SURF
PEBBLE
Daorsnlacck (foddbakk- 2000)
Drawer-apen (feedbark- 3000)
Doar-open (fondback- 30001
300
100
Success Rate
Success Rate
Success Rate
23
8.00
8:38
Environmenti Steps (x109)
Environment (x10%)
Environment Steps (x10%)
Figure 6: Learning curves on robotic manipulation tasks as measured on the ground truth success rate. The
dashed black line represents the last feedback collection step.
6.3 ABLATION STUDY
To evaluate the impact of each component in QPA, we incrementally apply policy-aligned query (PA), hybrid experience replay (HR), and data augmentation (DA) to the backbone algorithm PEBBLE. Figure 7 confirms that policy-aligned query has a positive impact on final results. Moreover, the combination of policy-aligned query and experience replay proves to indispensable for the success of our method. While data augmentation does not always guarantee performance improvement, it tends to enhance performance in most cases. In certain tasks, using only the hybrid experience replay technique can result in slightly improved performance. This could be attributed to the advantage of on- on-policyness (Liu et al., 2021; Sinha et al., 2022) that is facilitated by hybrid experience replay.
QPA also exhibits good hyperparameter robustness and achieves consistent performance improvement over SURF and PEBBLE across various tasks with different parameter values of the query buffer size N, data augmentation ratio and hybrid experience replay sample ratio w. We provide detailed
8

---

Published as a conference paper at ICLR 2024
QRA (PK: HR: DR w]
QRA HR: DR: w/0)
(PK. w. W.H.: DA: m/o)
(PAL I/o, W. W/O)
Wialetrooakc (6esshaccc1100)
Wallee--un ('oodbackk-100)
Choetahhrun fleedback 100)
Shadtrpossoak (loodback 1000)
126
Eptooto Rotur
Eplodde Return
Edisods
30
100
230
230
150
Environment ox10f)
Environment bx10f)
Environment Steps 0x10fs
Enstronment bx30f)
Figure 7: Contribution of each technique in QPA, i.c., policy- query (PA), hybrid experience replay
(HR), and data augmentation (DA).
ablation results on hyperparameters in Appendix D.5. In all the tasks presented in Figure 5 and Figure 6, we utilize data augmentation ratio of r = 20 and a hybrid experience replay sample ratio of 0.5. For the majority of tasks, we set the size of the policy-aligned buffer as N = 10.
6.4 ADDITIONAL. BENEFIT OF THE REWARD LEARNED BY QPA
As observed in Section 6.2, although SAC with ground truth reward is expected to attain higher performance compared to PbRL algorithms built upon SAC, it is found that QPA surpass during the early training stages in certain tasks. To further investigate this phenomenon, we conduct experiments that increase the total amount of feedback and allow the overseer to provide feedback throughout the training process. As illustrated in Figure 8, QPA often exhibits faster learning compared to SAC with ground truth reward in this setting. and in Quadruped rum task even achieves higher scores. This phenomenon may be attributed to the ability to encode the long-ferm horizon information of the reward function learned by QPA, which can be more beneficial for the current policy to learn and successfully solve the task. Such a property is also uncovered in the motivating example in Section 4. This intriguing and noteworthy phenomenon also highlights the possibility of PbRL methods with learned rewards outperforming RL methods with per-step ground truth rewards. We hope that this observation will inspire further investigations into the essence of learned rewards in PbRL and foster the development of more feedback- PbRL methods in the future.
SAC with ground trush reverd
S4C with proand buth reward (bodaack-1000
54G with poound muth revard Heedfadd-11800
1000
10000
Episode Return
750
Epicode Weturr
750
Epioadie Return
4:30
200-
b00
+00
853
87
60
Environment Steps (x10f)
Environment Steps (x20%)
Environment Steps (x10*)
(a) Quadruped_ walk
(b) Quadruped run
(c) Humanoid stand
Figure 8: Learning curves of QPA compared to SAC with ground truth reward given more overseer feedback
throughout the entire training process.
7 CONCLUSION AND DISCUSSION
This paper addresses a long-neglecte issue in existing PbRL studies, namely query-policy mis- alignment, which hinders the existing query selection schemes from effectively improving feedback efficiency. To tackle this issue, we propose bidirectional query-policy alignment (QPA) approach that incorporates policy-aligned query selection and hybrid experience replay. QPA can be imple- mented with minimal code modifications on existing PbRL algorithms. Simple yet effective, extensive evaluations on DMControl and MetaWorld benchmarks demonstrate substantial gains of QPA in terms of feedback and sample efficiency, highlighting the importance of addressing the query-policy misalignment issue in PbRL research. However, note that the query-policy misalignment issue is inherently not present in on-policy PbRL methods, as they naturally select on-policy segments to query preferences. These methods, however, suffer from severe sample inefficiency compared to ooflpol PbRL methods. In contrast, QPA approach not only enables sample-efficien off- off-policy learning, but also achieves high feedback efficiency. presenting a superior solution for the practical implementation of PbRL. in real-world scenarios.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is supported by the National Key Research and Development Program of China under Grant (2022YFB2002904, 2022YFA1004600), funding from Asialnfo Technologies and the National Natural Science Foundation of China under Grant (No. 62125304).
REFERENCES
David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of markov reward. Advances in Neural Information Processing Systems, 34:7799-7812. 2021.
Riad Akrour, Mare Schoenauer, and Michele Sebag. Preference-baset policy learning. In Proceedings of the 201 Ith European Conference on Machine Learning and Knowledge Discovery in Databases- Volume Part I, pp. 12 12-27, 2011.
Christopher G Atkeson, Andrew W Moore, and Stefan Schaal. Locally weighted learning for control. Lazy learning, pp. 75-113, 1997.
J Andrew Bagnell and Jeff G Schneider. Autonomous helicopter control using reinforcement learning policy search methods. In Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No. 0/CH37164), volume 2, pp. 1615-1620. IEEE, 2001.
Erdem Biyik and Dorsa Sadigh. Batch active preference- based learning of reward functions. In Conference on robot learning, pp. 519-528. PMLR, 2018.
Erdem Biyik, Nicolas Huynh, Mykel Kochenderfer, and Dorsa Sadigh. Active preference-base gaussian process regression for reward learning. In Roboties: Science and Systems, 2020.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4) 1952.
Robert Bridson. Fast poisson disk sampling in arbitrary dimensions. SIGGRAPH sketches, 10(1):1, 2007.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414-41) 2022.
Justin Fu, Sergey Levine, and Pieter Abbeel. One-shot learning of manipulation skills with online dynamics adaptation and ncural network priors. In 2016 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4016. IEEE, 2016.
Christopher Grimm, André Barreto, Satinder Singh, and David Silver. The value equivalence principle for model-based reinforcement learning. Advances in Neural Information Processing Systems, 33: 5541-5552. 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning. pp. 1861-1878 PMLR, 2018.
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004 / 13649, 2020.
10

---

Published as a conference paper at ICLR 2024
Nathan Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective mismatch in model-based reinforcement learning. In Learning for Dynamics and Control, pp. 761-770. PMLR, 2020.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp. 5639-5650. PMLR, 2020a.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in neural information processing systems, 33:
2020b.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Kimin Lee, Laura Smith, Anea Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021a.
Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference Machine Learning, pp. 6152-6163. PMLR, 2021b.
Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamies. Advances in neural information processing systems, 27, 2014.
Sergey Levine, Nolan Wagener, and Pieter Abbeel. Learning contact-rich manipulation skills with guided policy search. arXiv preprint 1rXivv1501.0500 I, 2015.
Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang Mind the gap: Offline policy optimization for imperfect rewards. In The Eleventh International Conference on Learning Representations, 2023.
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. International Conference on Learning Representations, 2022.
Rudolf Lioutikov, Alexandros Paraschos, Jan Peters, and Gerhard Neumann. Sample-based informationl-thcoree stochastic optimal control. In 2014 IEEE international conference on robotics and automation (ICRA), pp. 3896-3902. IEEE, 2014.
Runze Liu, Fengshuo Bai. Yali Du, and Yaodong Yang. neta-reward--et: Implicitly differentiable reward learning for preference-based reinforcement learning. Advances in Neural Information Processing Systems, 35:22270-22284, 2022.
Xu-Hui Liu, Zhenghai Xue, Jingcheng Pang, Shengyi Jiang, Feng Xu, and Yang Yu. Regret mini- mization experience replay in off-policy reinforcement learning. Advances in Neural Information Processing Systems, 14117604-17655 2021.
Yecheng Jason Ma, Kausik Sivakumar, Jason Yan, Osbert Bastani, and Dinesh Jayaraman. Learning policy-aware models for model-based reinforcement learning via transition occupancy matching. In Learning for Dynamics and Control Conference, pp. 259-271. PMLR, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 5 518(7540):59-533,, 2015.
Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. Advances neural information processing systems, 30, 2017.
Jongjin Park, Younggyo Sco, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efficien preference- reinforcement learning. In International Conference Learning Representations, 2022.

---

Published as a conference paper at ICLR 2024
Patrick M Pilarski, Michael R Dawson, Thomas Degris, Farbod Fahimi, Jason P Carey, and Richard S Sutton. Online human training of a myyoelectric prosthesis controller via actor-critic reinforcement learning. In 2011 IEEE international conference rehabilitation robotics, pp. 1-7. IEEE, 2011.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
LKPJ Rdusseeun and Kaufman. Clustering by means of medoids. In Proceedings of the statistical data analysis based on the 1.I norm conference, neuchatel, switzerland, volume 31, 1987.
Daniel Shin, Anca Dragan, and Daniel S. Brown. Benchmarks and algorithms for offline preference- based reward learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8855
David Silver, Hado Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac- Amold, David Reichert, Neil Rabinowitz, Andre Barreto, et al. The predictron: End-to-end learning and planning. International Conference on Machine Learning, pp. 3191-3199, PMLR, 2017a.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676)3354 359, 2017b.
Samarth Sinha, Jiaming Song, Animesh Garg, and Stefano Ermon. Experience replay with likelihood- free importance weights. Learning for Dynamics and Control Conference, pp. 110-123. PMLR, 2022.
Jonathan Daniel Sorg. The optimal reward problem: Designing effective reward for bounded agents. PhD thesis, University of Michigan, 2011.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal plan- ning networks: Learning generalizable representations for visuomotor control. In International Conference on Machine Learning, pp. 4732 4741. PMLR, 2018.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 1000803-011220000 2020.
Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks. Advances in neural information processing systems, 29, 2016.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 1rXiv:1801.00690 2018
Jeremy Tien, Jerry Zhi-Yang Yang He, Zackory Erickson, Anca Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-based reward learning. In The Eleventh International Conference on Learning Representations, 2023.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSI international conference on intelligent robots and systems, pp. 5026-500 5033. IEEE, 2012.
Xiyao Wang, Wichayapom Wongkamjan, Ruonan Jia, and Furong Huang. Live in the moment: Learning dynamics model adapted to evolving policy. In International Conference on Machine Learning, pp. 36470-36493. PMIR, 2023.
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109. 10862, 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100, PMLR, 2020.
Shangtong Zhang and Richard S Sutton. A deeper look at experience replay. arXiv preprint 2017.
12

---

Published as a conference paper at ICLR 2024
A PROOFS
In this section, we present emror bound to intuitively explain why the key idea behind the two techniques in QPA (e.g. policy-aligned query selection and hybrid experience replay ) is reasonable. Note that we do not aim to provide tighter bound but rather to offer an insightful theoretical interpretation to query-policy alignment.
Given the learned reward function Fwg, the current stochastic policy and its state-action visitation distribution d=, we denote OF, as the Q-function of associated with Fw and QF, as the estimated O-function obtained from the policy evaluation step in Eq. (1), which serves as an approximation of OF, O denotes the Q-function of π with true reward r. Define the distribution-dependees norms (1)( == Exzp [x))] have the following error bound:
Given the two conditions lif - rlla- Sc c cand 1
Q lla I a, the value approximation error
||Q7 Q7, laa upper bounded as:
107 - Q lla S 1- ta
(5)
Proof. By repeatedly applying triangle inequality, we have
||Q Q 11. = 1|Q - Q7, + Q7, - Q7 lla
S 11 Q7, aa. + +1Q7, QF, 1la
((aaaaad" (5. -Q,, (s,a) + +o = (ssaaaaa (1) + I. (s', (s - BB OF (8', a')) +
(6)
< 711²" - 07 d +a
sct 76 + 7 + + 7 t
1-7 +a
B ALGORITHM PROCEDURE
We provide the procedure QPA in Algorithm 1. QPA is compatible with existing off-policy PbRL methods, with only 20 lines of code modifications on top of the widely-adopted PbRL backbone framework B-Pref (Lee et al., 202
C EXPERIMENTAL DETAILS
C.I 2D NAVIGATION EXPERIMENT IN SECTION 4
In this section, we present the detailed task descriptions and implementation setups of the motivating example in Section 4.
Task description. As illustrated in Figure 2 (a), we consider a 2D continuous space with (æ, y) coordinates defined on 1010, 10 For each step, the RL agent move Ar and Ay ranging from [-1, The objective for the agent is to navigate from the starting point (1,1) to the goal location (10, 10) as quickly as possible. The hand-engineered reward function (ground truth reward
in Figure 2 (b)) to provide preferences is defined as the negative distance to the goal, i.e., r(8, a)
(ar 10)² + (y 10)²
13

---

Published as a conference paper at ICLR 2024
Algorithm 1: QPA
Input
FFrequency of overseer feedback K, number of queries per feedback session M Policy-aligned buffer N. data augmentation ratio r, hybrid sample ratio w
Initialize: Initialize replay buffer D, query buffer DD, policy-aligned buffer DP with size (Option) Unsupervised pretraining (Lee et al., 2021b)
for each iteration do
Collect store new experience iteration %KK 0 then
DU ((s,,, D + D ((s,,,n)
/+ ppwllcyylllicoooo inged query selection (see Section 5.1) [ Don
Query for preferences [w) store preference + D U w)) for each gradient step do
Sample minibatch preferences B + {(oo, a v) ==1 D
/ Data augmentation for zeward learning (see Section 5.3) AXT Generate augmented preferences 8 {(00, a' w))
based on B
Optimize Creward in Eq. (4) w.E.t F6 using &
11
Relabel the rewards (Lee et al., 2021b) in D
12
for each gradient step
* Hybrid experience replay (see Section 5.2)
Sample minibatch Dmini {(s, r, ) 8)) D, Diini ((,,a,,,8)) D
Optimize SAC agent using Diini mini
Implementation details. We train PEBBLE (Lee et al., 2021b) using different query selection schemes: uniform query selection, disagreement query selection, and policy-aligned query selection obtain one pair of segments to query overseer
(see Section 5.1). In each feedback session, preferences. The total amount of feedback set to 8. Each segment contains 5 transition steps. For all schemes, select the 1st queries using uniform query selection according to PEBBLE implementation. After the 1st query selection, we start selecting queries using different selection schemes. The 2nd selected pairs of segments of different schemes are tracked in Figure 3, and the 3rd to 5th selected pairs are tracked in Figure 2 (d).
C.2 DMCONTROL AND META- WORLD EXPERIMENTS
(a) Walker walk
(b) Walker run
(c) Cheetah_run
(d) Quadruped walk (e) Quadruped_
Humanoid stand
Figure 9: Rendered images of locomotion tasks from DMControl.
C.2.1
TASK DESCRIPTIONS
Locomotion tasks in DMControl suite. DMControl (Tassa et al., 2018) provides diverse high- dimensional locomotion tasks. For our study, we choose complex tasks Walker_walk, Walker_run Cheetah Quadruped _aak,, Quadruped_ree run, Humanoid _tand as depicted in Figure 9. In Walker wwak and Walk kun,_rrm,kk the ground truth reward a combination terms encouraging an upright torso and forward velocity. The observation space is 24 dimensional, and the action space is 6 dimensional. In Cheetah run, the ground truth reward is linearly proportional to the forward velucity up to maximum of 10m/s. The observation space is 17 dimensional, and the action space is
14

---

Published as a conference paper at ICLR 2024
dimensional. In Quadruped walk and Quadruped Jun, the ground truth reward includes the terms encouraging an upright torso and forward velocity. The observation space is 58 dimensional, and the action space is 12 dimensional. In Humanoid stand, the ground truth reward is composed of terms that encourage an upright torso, a high head height, and minimal control. The observation space 67 dimensional, and the action space is 21 dimensional.
Robotic manipulation tasks in Meta-world. Meta-world (Yu et al., 2020) provides diverse high-dimennional robotic manipulation tasks. For all Meta-world tasks, the observation space is 39 dimensional and the action space is 4 dimensional. For our study, we choose 3 complex tasks Door unlock, Drawer _penn Door open as depicted in Figure For Door hbor_umlock, the goal is to unlock the door by rotating the lock counter-clocky For Drawer prawer_open, the goal is to open drawer and the initial drawer position randomized. For Door_oper oopen, the goal is to open door with a revolving joint and the initial door position is randomized. Please refer to (Yu et al., 2020) for detailed descriptions of the ground truth rewards.
and the initial door position is randomized.
(a) Door unlock
(b) Drawer_ open
(c) Door_open
Figure 10: Rendered images of robotic manipulation tasks from Meta-world.
C.2.2 IMPLEMENTATION DETAILS
Implementation framework. We implement QPA top of the widely-adopted PbRL backbone framework B-Pref' (Lee et al., 10)..). B-Pref provides standardized implementation of PEBBLE (Lee al., 2021b), which can be regarded as the fundamental backbone algorithm for off-policy actor-critic PbRL algorithms. Therefore, similar to prior works (SURF (Park et al., 2022), RUNE (Liang et al., 2022), etc), we opt for PEBBLE as our backbone algorithm as well. B-Pref implements 3 distinct buffers: the RL replay buffer D. the query selection buffer D' and the preference buffer D D stores historical trajectories for off-policy RL agent's training. D' a copy of excluding the predicted reward. It is specifically utilized for segment query selection. D stores the historical feedback (a", a² y) for reward training. Every time the reward model F.. is updated, all of the past experience stored in D is relabeled accordingly. We implement SURF using their officially released code?, which is also built upon B-Pref.
Query selection scheme. For PEBBLE and SURF, employ the ensemble disagreement query selection scheme in their papers. To be specific, we train an ensemble of three reward networks Pg with varying random initializations. When selecting segments from D', firstly uniformly sample large {(o", o')} batch from D'. Subsequently, we choose (a o²) from this batch based on the highest variance of the preference predictor Pv. For QPA, we simply reduce the size D'. This minimal modification ensures that the first-in-firt-oout buffer D' exclusively stores the most recent trajectories, effectively emulating the characteristics the policy-aligned buffer DRA In QPA, instead of utilizing ensemble disagreement query based on multiple reward models, reward model and employ policy-aligned query selection (simply randomly selects segments from the policy-aligned buffer DR This approach significantly reduces the computational cost compared to ensemble disagreement query selection, particularly in scenarios where the reward model is notably large, such as large language models (LLMs). Although leveraging an ensemble of reward models for query selection may offer improved robustness and efficacy in complex tasks as observed in (Lee et al., 2021b; Ibarz et al., 2018), we showcase that QPA, employing the simple policy-aligned query selection, can substantially outperform SURF and PEBBLE with ensemble disagreement query selection.
single
hhttps://gpenreviiet net htwpe//reeeeeeeeeetooooooeeaaae. 1
15

---

Published as a conference paper at ICLR 2024
Data augmentation. In the officially released code of SURF, they implement the temporal data augmentation, which generates one (ō O¹, y) instance from each (a", o y) pair. In contrast to their implementation, in QPA, generate multiple (0".0) instances from single (o",a', a vy
pair, which effectively expands the preference dataset. Specifically, we sample multiple pairs of snippets (o 6*) from the queried segments , y). These pairs of snippets (Ø", o²)) consist of sequences of observations and actions, but they are shorter than the corresponding segments (a", o²). In each pair of snippets (6", o²), o has the same length as 6², but they have different initial states.
Hyperparameter setting, QPA, PEBBLE, and SURF all employ SAC (Soft Actor-Critic) (Haarnoja et al., 2018) for policy learning and share the same hyperparameters of SAC. We provide the full list of hyperparameters of SAC in Table 1. Both QPA and SURF utilize PEBBLE as the off-policy PbRI. backhone algorithm and share the same hyperparameters
PEBBLE as listed in Table 2. The additional hyperparameters of SURF based on PEBBLE are set according to their paper and are listed Table 3. The additional hyperparameters of QPA are presented in Table
Table Hyperparameters of SAC
Hyperparameter
Value
Hyperparameter
Value
Discount
0.99
Critic target update freq Critic EMA
Init temperature
0.1
0.005
Alpha learning rate
le-4
Actor learning rate
5e-4 (Walker_wall
Critic learning rate
5e-4
(Walker walk, Cheetah run,
Cheetah run,
Walker run)
Walker_run)
1e-4 (Other tasks)
le-4
(Other tasks)
Actor hidden dim
1024
Critic hidden dim
1024
Actor hidden layers
Critic hidden layers
2
Actor activation function Optimizer
ReLU
Critic activation function
ReLU
Adam
Bacth size
1024
Table 2: Hyperparameters of PEBBLE
Hyperparameter
Value
Length of segment
50
Unsupervised pre-training steps
9000
Total feedback
100 (Walker walk, Cheetah run, Walker run)
1000 (Quadruped_warl walk, Quadruped_ walk)
2000 (Door_unloe) ___locllkkk
3000 (Drawer
crewer_open, open, Door_open)
10000 (Humanoid_saee _tandddddsttddd)
Frequency of feedback
5000 (Humanoid_starin Drawer_open, Door_open)
20000 (Walker_walk, Cheetah. run, Walker run, Door_unlode
30000 (Quadruped walk, Quadruped walk)
# of queries per session
10 Walkerr_alk Cheetah_run run, Walker_run)
30 (Drawer_open Door_open)
50 (Humanoid stand)
100 (Quadruped_ walk, Quadruped_ walk, Door coor_unlock)
Size of query selection buffer
100
16

---

Published as a conference paper at ICLR 2024
Table Additional hyperparameters of SURF
Hyperparameter
Value
Unlabeled batch ratio
Threshold
0.999 (Cheetah_ run), 0.99 (Other tasks)
Loss weight
Min/Max length of cropped segment
[45, 55]
Segment length before cropping
60
Table Additional hyperparameters of QPA
Hyperparameter
Value
Size of policy-ale aplicy-aligned buffer
30 (Drawer _oeereppppeeee_ee Door soor__nlock), 60 (Door_open),
10 (Other tasks)
Data augmentation ratio r
20
Hybrid experience replay sample ratio
0.5
Min/Max length subsampled snippets
[35, 45]
D MORE EXPERIMENTAL RESULTS
In this section, we present more experimental results. For each task, we perform evaluations across 5 rums every 10* environment steps and report the mean (solid line) and 95% confidence interval (shaded regions) of the results, unless otherwise specified.
D.1 QUERY-POLICY MISALIGNMENT AND ITS ROLE IN CAUSING FEEDBACK INEFFICIENCY.
In section 4, we present a motivating example to clarify the query-policy misalignment issue existing PbRL methods. In this section, provide more experimental results in complex environment to show that:
the query-policy misalignment issue does exist in typical PbRL methods and does cause the feedback inefficiency;
using only policy-aligend query selection address the query query-policy misalignment can result in a significant performance improvement.
we compare the two method: PEBBLE and PEBBLE + policy-aligned query selection. The policy-aligned query selection is presented in Section 5.1. To assess the extent to which the queried segments align with the distribution using the queried segments at each query time. Figure (a) shows that the segments queried by PEBBLE exhibit low log likelihood of n indicating that these segments fall outside the distribution of the current policy æ. Figure 11 (a) and (b) demonstrate that by only using the misalignment leads to a considerable improvement in feedback efficiency. As the technique policy- be directly added PEBBLE without other modifications address
As all the recent off-policy PbRL methods are built upon PEBBLE.
of the current policy क, we compute the log likelihood
aligend query selection to address the query-polig
aligned query selection query-par query-policy misalignment and improve feedback efficiency, the query-policy misalignment issue indeed the key factor causing feedback inefficiency.
D.2 ADDITIONAL COMPARISON
this section, we demonstrate the efficacy of QPA in comparison another SOTA method MRN (Liu et al., 2022), All the basic settings remain consistent with the experimental setup outlined in Section 6.1. The additional hyperparameters (such as the bi-level update frequency) PEBBLE
MRN based on according their paper. Figure 12 shows that in most tasks, QPA consistently outperforms MRN. is worth noting that MRN adopts the performance of the Q-function as the learning target to formulate bi-level optimization problem, which is orthogonal our methods.
17

---

Published as a conference paper at ICLR 2024
PEBBLE + policy--ligned query selection
PEBBLE
Walkenwalk (feedback=100)
Cheetah:run (feedback=100)
Walker run (feedback=220)
100
-15
-2.0
-2.
-3.0
Environment Steps (x109)
Environment Steps (x10f)
Environment Steps (x10%)
(a) Log likelihood of current policy ят on the queried segments in 3 locomotion tasks.
1995
Wa Walker-walk (feedback-100)
Cheetah-run (feedback=100)
Walker-run (feedbeck=250)
Episade Return
Episade Return
250
Episode Return
750
500
500
500
80
0.25
Environment Steps (x10P)
Environment Steps (x10*)
Environment Steps (x10%)
(b) Learning curves of episode return in above locomotion tasks.
The blue line represents the typical PhRL method: PEBBLE, upon which most recent PhRL. methods are built. The red line represents PEBBLE + policy-aligned query selection. The experiments conducted Kalker__wlke Cheetah_ run, Walker the log likelihood current policy m. The segments queried by PEBBLE exhibit a low log likelihood of indicating that these segments fall outside the distribution of the current policy æ, which supports the existence of query-policy misalignment. When PEBBLE incorporates our proposed technique policy-aligned query selection in Section 5.1, there is substantial inerease in the log likelihood of и. (b) The performance of PEBBLE + policy-aligned query selection significantly surpasses that of PEBBLE in the corresponding task.
Figure 11:
(a) We the queried segments at every query time compute
query selection can be easily incorporate into MRN to further improve the feedback
efficiency.
PEBBLE
policy- query selection
3000
Walker-walk [feedback=1000
run ffeedback-100)
ffeedback=2500
Episode Return
750
Episode Return
Episode Retur
290
200
150
0.3
tubs
025
0001
0.75
Environment Steps 6x10%
Environment Steps 61109
Environment Steps (x109)
Quadruped-walt (feedback-1000)
Quadruped-run foedback
nec Humanoid-ooe stand (feedhack- 10000)
Episode Return
Episade Return
Eplsode Return
82c
0.5
3.00
8.0
250
0.75
1.0
Environment Steps (x10%)
Environment Steps 6410%)
Environment Stegs (x10%)
Figure 12: Learning curves on locomotion tasks of QPA, PEBBLE, MRN. and MRN policy-aligned query selection. The dashed black line represents the last feedback collection step.
18

---

Published as a conference paper at ICLR 2024
D.3 RESULTS UNDER VARYING TOTAL FEEDBACK AND FEEDBACK FREQUENCIES
To further demonstrate the feedback efficiency of QPA, compare its performance with that of SURF and PEBBLE under varying total feedback and different feedback frequencies on certain locomotion tasks. As illustrated in Figure 13-17, QPA (red) consistently outperforms SURF (brown) and PEBBLE (blue) across a wide range of total feedback quantities and feedback frequencies. The dashed black line depicted in these figures represents the last feedback collection step.
SAC ground truth reward
QPA
SURF
PEBBLE
Walker walk (feddacck-80)
Walker walk (feedbock-100)
Walker.wa l (feebbckk 150
3053
1000
3860
755
Egisode Return
Eplcode Return
350
Epizode
$42
100
540
200
250
250
Environment Steps (110))
Environment Steps (x10")
Environment Steps (x10*)
Figure 13: Learning curves Walker walk under varying total amounts of feedback {60, 100, 150).
(twedback-00)
Epicode Return
Eplaode Return
Eplszde
Epiodde
ods
15
Envvcomment Steps bi2ff)
Envionment Steps bx10f)
Environment Steps 0x10f)
Environment Stops 1x20%)
Figure 14: Learning curves Cheetah under varying total amounts of feedback {60, 100, 140, 250).
Quadruped-run (teddback=600)
Quadruped-run [eedbact--1000
Quadruped-run (fedtback-1700)
Episodle Beturm
Episode Return
Episode
252
200
0.25
050
6T7
102
825
0:30
3.00
0.23
2.00
0.75
Environment Steps (x10))
Environment Steps (x10f)
Environment Steps (110))
Figure 15: Learning curves on Quadruped_ rwn under varying total amounts of feedback (600, 1000, 1700].
Walker run (feedback-100)
scre Quadruped walk (feedback- -7000
Humenaid stand (feedback=5000
1000
Epapode Return
Episoda Return
150
Episode Retur
600
300
8.0
626
0.3
1.00
850
0.2
250
0.55
Environment Stops bx10
Environmant Stops (x10%)
Environment Steps 0110))
Figure 16: Learning curves on other tasks under different total amounts of feedback.
feedback frequency- 15000
feedback frequency=3000
1000
1000
Episode Return
750
100
Episede
250
250
Environment Steps (x10%)
Environment Stegs (X10%)
Figure 17: Learning curves Walker walk under varying feedback frequencies.
Additionally, we provide a summary in Table 5 showcasing the average episode return or success rate of all PbRL methods across all domains, including standard deviation, based the final 10 100 evaluations and 5 seeds.
19

---

Published as a conference paper at ICLR 2024
Table Average episode return / success rate of all PbRL methods with standard deviation over final 10/100 evaluations and 5 seeds. QPA consistently outperforms other methods across a wide range of tasks and given feedback amounts.
Task Total feethhck
SAC with ground iruth reward
PEBBLE
SURF
QPA
Walker walk--00
949.21=21.94
388, 163.00
610.91+170.83
802.655
Walker_wallk 60
949.21=21.94
205 105.23+103.00
544.064105.24 24
672.19:128.99
Walker_ Walker_walk-1e walk-150
9499211=21.9 1.9
570.682168.21
829 $29.95438.0 05
862.39:127.88 .6
Walker
832.44:21.91
234.48:147.477
467.63+152.58
Dhectah_run-100
812.35=60.47
121.44::62.35
462.44+167.7
652.56+109..0
Cheetah run-60
812.35=60.47
301.13 144.21
378, 778.764255.74 74
528.91 16
Cheetah
812.35=660.47
499.01=203.62
590,98a61
647.55+664.62
waadruped_walkkIiooo walk-1000
925 25.20.09
491.04=301.0
457. 157.5661688 16.89688
6699755::01.61
Quadruped_raae run-1000
660.44x128.7
409 100 133.88
431 03+94 44
495.71 24
cendruped_run-diie
660.44:122.7
166.932148.51
342.07:103.44
403.95:99.88
Humanoid stand-10000 10000
503.1 13+85,85
266 23.9
332.67+166. 66. 18
467. 367.09932.411
Humamoid sland 5000
503.13
28.1 18.11
44.75+77.99
79.39288.63
Doox-umlack-2000
100.0:0.0
74.0=37.65
80,2+37.63
Drawer-open-Be 3000
91,6+16.8
58.4=40.96
47.0=43.41
65,0=43.25
Door peee-ooee--300
100.0=0.0
79.6#39.8
50.0=41.91
84.4
D.4 THE EXPLORATION IN QPA
As detailed in Figure 4 and Section 5. the exploration of query selection in PbRL can cause feedback inefficiency. The experiments in Section 6.2 and Appendix D.3 also demonstrate that, in general, policy-aligned query selection is more feedback-efficient compared to methods such as uniform query or disagreement query, which, some extent, "encourage" the exploration of query selection.
Using the policy-aligned query selection, which confines the queries to the local policy-aligned buffer DPs, provides more local guidance for current policy, enhancing feedback efficiency. However, adopting policy-aligned query selection doesn't tt imply a lack of explorative capability in QPA. In fact, SAC's intrinsic encouragement of exploration can enhance the policy performance of QPA.
We would like to highlight that the exploration of state-action space in traditional RL algorithms and the exploration of query selection PbRL algorithms are two separate and distinct aspects. While the exploration of query selection potentially causes feedback inefficiency, the exploration ability in RL algorithms can help improve the sample efficiency. QPA adopts SAC as the backbone RL algorithm. The MaxEntropy policy of SAC helps QPA explore the unknown regions, while the policy-aligned query helps QPA exploit the limited human feedback. We conduct supplementary experiments in Figure 18 that remove the MaxEntropy of SAC. The results show that the intrinsic encouragement of exploration in SAC plays an important role in policy performance.
QPA MaxEntropy
QPA
Wallerwak (foddacco1100
Cheetah-run 1ooddock--10)
De0 Quadruped walk (feedbaci 1000)
1000 Quadruped #eedbbck-11000
Edto00 Robum
Eptoode Setum
Epcods
800
Edicode
202
Enviroomment Strps bx10f)
Enviromennt
Ex10")
Environment
Strres Oc30")
Enstronment bx10")
Figure 18: Learning curves of QPA and the modified QPA that removes the MaxEntropy of SAC (QPA w/o MaxEntropy).
D.5 ADDITIONAL ABLATION STUDY
We emphasize that the outstanding performance of QPA, as demonstrated in Section 6, was not achieved by meticulously selecting QPA's hyperparameters, On the contrary, as indicated in Table 4, we consistently use a data augmentation ratio = 20, a hybrid experience replay of w = 0.5 across all tasks and use a policy-aligned buffer size of N = 10 in the majority of tasks. To delve deeper into the impact of hyperparameters on QPA's performance, we conduct an extensive ablation study across range of tasks.
20

---

Published as a conference paper at ICLR 2024
Effect of data augmentation ratio The data augmentation ratio, denoted as т, represents the number of instances (ō", _.. .8) generated from a single pair (o .y.y). To explore the impact of the data augmentation ratio on QPA's performance, we evaluate the performance of QPA under different data augmentation ratios о E {0, 10, 20, 100). Figure 19 illustrates that QPA consistently demonstrates superior performance across diverse range of data augmentation ratios. While larger data augmentation ratio does not necessarily guarantee improved performance, it is worth noting that т 20 generally a favorable choice in most cases.
GRA 25-
OPA 10
OPA (T-00) 0P% 11 1001
GRA IT.
OPA 10
SUMF
QP4 [T- 100+
GMA 1201
PEBOLE
PEABLE
1300
1000
Episode Return
Episode Return
755-
Epicade Return
300
-00
200
250
230
250
602
0.25
6:30
0.75
100
8:0
0.25
0.33
2.13
3.00
Environment Steps (x109)
Environment Steps (x109)
Environment Steps (x10%)
(a) maaker_walkk
(b) Cheetah_naas run
(c) Walker_run run
(e-20)000 20) OPA (c- 201
QPA1T-0)
GP (T= 10
OPA IT-
SURF
QPA 12)
OPA 1285
SURE
0MA (C 2001
TEBSLE
1000
1:00
Episade Boturn
Egioade Return
Episode Return
402
853
8.
1.5
Environment Steps (x110))
Environment Stops (x10")
Environment Steps (x10%)
(d) Quadruped walk
(e) Quadruped run
(I) Humanoid stand
Figure 19: Learning curves locomotion tasks under different data augmentation ratios 7 {0, 10, 20, 100}
of QPA.
Effect of policy-aligned buffer size N. The size of the on-policy buffer, denoted as N, signifies the number of the recent trajectories stored in the fost-in-first-uu buffer DR A larger N implies that the policy-aligned buffer DP contain additional historical trajectories generated by a past policy x' that may be significantly different from the current policy æ, potentially deviating from the main idea of policy-aligned query selection. Therefore, it is reasonable for QPA to opt for smaller value of N. To further investigate how the on-policy buffer size affects QPA's performance, we evaluate the performance of QPA under different on-policy buffer sizes N (5, 10, 50). Figure 20 demonstrates QPA consistently showcases superior performance, particularly when using a smaller value for N.
In Quadruped_ run, it is observed that QPA with N 50 does not exhibit improved performance. This outcome can be attributed to the fact that the larger value of N compromises the essence of near "on-policy" within D This once again highlights the significance of the policy-aligned query selection principle. A smaller value of N may not always result in a significant improvement in performance. This could be because the pair of segments selected from a very small aelined buffer is more likely to be similar to each other, resulting in less informative queries. Overall, it is generally considered favorable to select N = 10 for achieving better performance.
Effect of hybrid experience replay sample ratio replay, denoted as w, signifies the proportion of transitions sourced from the policy-aligned buffer DeS during the policy evaluation (empirical Bellman iteration) step. When = O, all transitions are sourced from the entire replay buffer D. Conversely, when w = 1, all transitions are sourced from the policy-aligne aligned buffer DM We provide an additional ablation study in Figure 21 to investigate the effect of w. Only sampling the transitions from the entire replay buffer D (w = 0.8) for Bellman iterations diminishes QPA's performance, which validates that the hybrid experience replay technique
The sample ratio of hybrid experience
21

---

Published as a conference paper at ICLR 2024
DR% 1N-55
GRA IN 100
SURT P2D0LC
074
PERSLE
I8 10
SJIP
1000
IN
1300
GR%, (9-133
1000
OPA IW 100
Episode Return
Episode Retuin
752
Epicode Retuin
500-
100)
252
250-
250
0.5
0.75
85o
0:25
Environment Steps 01109)
Environment Steps (x109)
Environment Steps (x10%)
(a) Walker walk
(b) Cheetah run
(c) Walker run
UPA IN 100
SUAP FEBOLE
CPA
SUPP PEBBLE
ORA Ii
SU5r PEBSLE
GPA 51
OPA 0W- 5)
(RS -
1000
1003
0000
Epizade Return
150
Esisode Return
730
Episode Return
000
500
300
900
230
8.0
9.25
0:50
0.75
1000
800
0.23
0.50
100
65
05
20
Environment Steps (x10*)
Environment Steps (x10%)
Environment Steps (x10%)
(d) Quadruped_ walk
(e) Quadruped run
(I) Humanoid stand
Figure 20: Learning curves on locomotion tasks under different sizes of policy-alignee buffer N E 45, 50} of QPA.
can enhance performance. When larger proportion of transitions sampled from the policy-aligned buffer (w = 0.8), QPA's performance remains close to QPA with w 0.5 The hybrid experience replay not only benefits the value learning within on on-polic distribution, but also improves sample efficiency compared to on-policy PbRL algorithms. Overall, w 0.5 is generally favorable choice.
OPA (w 0.8)
OPA (w 0.5)
OPA (w
1800
Wallerwalk (feedbacc=110)
Cheetah--un Meothock-1000
Quadruped walk (feodhacc- 1000
Quadrupod-run Heeathack-1000
Epicodo Ratur
Eploode Retum
Epioad
035
152
85
Environment Steps (x10")
Enviromeent Steps 0x10f)
Environment Steps 0410%)
Envrrommnt Steps 1x20%)
Figure 21: Learning curves locomotion tasks under different hybrid experience replay sample ratio {0. 0.,, 0.8).
E HUMAN EXPERIMENTS
In this section, we compare the agent trained with real human preferences (provided by the authors) to the agent trained with hand-engineered preferences (provided by the hand-engineered reward function Appendix 1.2.1) on the Cheetah_rui run task. We report the training results in Figure 22. Please refer to the supplementary material for the videos of agent training processes.
Avoiding reward exploitation via human preferences. As shown in Figure 22, the agent trained with real human preferences exhibits more natural behavior, while the agent trained with hand- engineered preferences often behaves more aggressively and may even roll over. This is because the wand-engineered reward function is based solely on the linear proportion of forward velocity, without fully considering the agent's 's posture. Consequently, it can be easily exploited by the RL agent. Take Figure 23 as example: when comparing the behaviors of "Stand still" and "Recline".
22

---

Published as a conference paper at ICLR 2024
(a) Agent trained with human preferences
(b) Agent trained with hand-engineered preferences
Figure 22: Human experiments on Cheetah_ run task.
(a) Stand still
(b) Recline
Figure 23: A pair of segments.
QPA
PEBBLE
Episode Return
450
300
150
8.00
0.25
0.50
0.75
1.00
Environment Steps (x106)
Figure 24: Learning curves of QPA and PEBBLE on Cheetah run under real human preferences.
humans would typically not prefer the latter, as it clearly contradicts the desired behavior of "running forward". However, in our experiments, we observe that the hand-engineered overseer would favor "Recline" over "Stand still", as the hand-engineered preferences only consider the forward velocity and overlook the fact that the agent lying down while still maintaining some forward velocity. This highlights the advantages of RL with real human feedback over standard RL that training with hand-engineered rewards.
The effectiveness of QPA under real human preference. We also provide the learning curve of QPA compared to PEBBLE under 100 real human preferences in Figure 24. The preferences labeled by two different humans, which might introduce inconsistencies and contradictions in preferences. Figure 24 shows that using feedback from real humans, our method also notably improves feedback efficiency compared to PEBBLE.
23

---

Published as a conference paper at ICLR 2024
Experimental details and user interface. The total feedback, frequency of feedback and number of queries per session in real human experiments remain consistent with the experimental setup outlined in Section 6.1 and Table 2. A subtle distinction is that human users have the choice to skip certain queries when they find it challenging to determine their preferred segment, unlike the scripted annotator. These skipped queries not accounted for in the total feedback.
Our implementation of real human PbRL builds upon the widely-adopted PbRL backbone frame- work B-Pref (Lee al., 2021a). In the original B-Pref code, scripted preferences based the cumulative ground truth rewards of each segment defined in the benchmarks are used after each query selection. However, to enable real humans provide preferences and furmish labeling instruc- tions and a user interface, we' ve mainly modified the get llaeel function. The modified function now accepts additional inputs: physics_ _eg and video retoorrrcorerr. The physics_ seg is obtained from emypphysics_ physics s_stttst__________ (ate__iems() in DMControl environments (Tassa et al., 2018). a class for physies rendering. We provide the Python code of modified ger ger_label function as follow.
import numpy np import torch
get human label.
sa_t: the segment; physice_seg: the physics information of the segment; env: the environment; video_recorders the recorder for physics rendering.
def yet_label (self, sa ta, sa_t_2, physics_sedd segl, soysics_seg2, env, video_recorder))
get human label
haben_ np.zeros (sa_t_1, (sa_t_1.shape) shape [0])
for seg veg_index in range (physics_sene shape [0]):
# render the pairs of segments and save the video
video_recorder[0]- for in range (physicc__eell [eeg)____________nnee.ee
enabled-True)
with env. eav.physies. reset reseieseetett_ccntettttlill context env www.physics.set_tteee state styysssssessttteeeeesstsssss.sss (pwysics_sesseepppeeesess /seg index [i])
video recorderi0).recorddeee video_recorder
(1c (cnv, enabled-True)
for in range
[ang___inx]]]]_ shape
with env. enw.phyysics..essttccnttet env. env.physics. (physicsssessepeeeee seg__inded [i])
(ideo_reorder11..xccooeeeo vicec_recorder save
save ('vg2 mp42)
labeling True
provide labeling instruction and query human for preferences while (labeling):
print("\n")
print (" ("------
print ("Feedback number:", seg_inden
# preference:
# 0: segment 0 is better
# 1: segment is better
other number: hard judge, skip this pair of segment
while True:
A check if it is 0/1/number type preference
try:
rational label input (aput("Preee ("Preference: 0 other number")
rational_labet int (rational_label
break
except:
print ("Wrong label type. Please enter 0/1/other number. \n")
24

---

Published as a conference paper at ICLR 2024
print
human_labell [sog_indcx] rational_labe labeling False
the hard-to-judge pairs of segments
cancel np, where (human_labelss Lamannlabelss = np.deleter (human_ Labels, cancel, axis=0)
and human_iabelss-1)) [0]
1a = delete (sa_ts t_1, cancel, axis=0) sa___ 2 np, delete (sa 2, cancel, axis=0)
print ivalld query number:", len (ww.aahlaaelllng return sa_t_l, sa_t_2, human_labels
The user interface is shown in Figure 25. Users have the option to input either 0 or 1 to indicate their preference, or they can input any other number to skip this labeling.
segSumpp
segi
videits chrstaluuuu.)
videow destaltuuu
Feedbakk number:
Ireforence:
Figure 25: User interface for providing user preferences.
25


Reviews:
[{"review_id": "PM9pnNQMm7", "reviewer": "Reviewer_LzS2", "summary": "In this work, the Authors set to improve feedback efficiency in RLHL (RL from human feedback). To this end, they propose to (1) use only recent behaviors of RL agents for queries and to (2) update RL agents using recent experiences alongside experiences uniformly sampled from the replay buffer. The Authors test their proposed sampling approaches in several existing RLHF models on a number of control benchmarks where they show improved performance in comparison to baseline approaches.", "questions": "See above\n\n_____________________\nPost-rebuttal. The Authors have diligently addressed my concerns and, to my knowledge, did a good job of addressing the other Reviewers' concerns. Also, I see that this paper is of a much higher quality than a typical paper at this venue this year that has been assigned the same score as I initially put here. To reflect on both of these facts, I increase my score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper addresses an important problem of improving the feedback efficiency in RLHF. As RLHF is widely used today, in particular, with LLMs (large language models), the results offered in this work are important, timely, and definitely relevant to the community.\n\nThe simplicity of the approach is also a plus. As the Authors state in the paper, their sampling strategies require minor additions to the existing models’ code, on the order of a couple dozen lines of code, so their approach is easy to implement.\n\nThe proposed approach has been tested on a variety of control tasks where it shows improvement in performance compared to the baseline models.\n\nOverall, I think this submission is a solid work with a clear message and thoroughly conducted experiments.", "weaknesses": "I wasn’t able to fully follow some of the logic regarding the justification of the sampling scheme design. Throughout Pages 5 and 6, I understood the equations describing the error bounds for the Q-functions and the rewards but I couldn’t see the formal connection between these bounds and the proposed sampling schemes. If there *is* a formal connection, I suggest elaborating on it, e.g. in the statement in 5.1. that goes: “By assigning more observers feedback <…> we aim to enhance the accuracy of the preference (reward) predictor <…> This aligns with the intuition from the condition <…> in Eq. (5).” Similarly, in 5.2, the claim: “The proposed mechanism can provide assurance that the Q-function is updated adequately near $d^\\pi$” may need further elaboration formally supporting it. If there *isn’t* a formal connection between these claims and Eq. (5), I suggest removing Eq. (5) and the related references as, in the current standing, they may be confusing to a reader like myself.\n\nAt the same time, similar ideas have been explored in literature. The field of decision-aware model learning (or value equivariance model learning) used the idea that, in model-based RL, learning a model that’s accurate everywhere might be unnecessary and, instead, learning a model that’s only accurate in task-critical regions of the state space may offer a way of improving models’ sample complexity. Although these approaches, to my knowledge, have not been applied to RLHF, they pursued similar goals (i.e. improving the sample complexity) by using similar methods (i.e. focusing on the prediction accuracy for task-relevant states and transitions). I suggest discussing these (and similar) lines of work and their relation to the proposed sampling approach in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U2Y5Io2z5q", "reviewer": "Reviewer_EEB5", "summary": "The paper addresses the problem of query-policy misalignment in preference-based reinforcement learning (PbRL) and introduces a novel method, QPA, consisting of two main techniques, policy-aligned query selection, and hybrid experience replay, to improve the efficiency of human feedback in PbRL.", "questions": "* The paper could be strengthened by testing the proposed method under real-world conditions, using actual human annotators instead of an oracle, to test the generalizability of their results.\n\n* Providing more implementation details or pseudocode for the proposed method would add value to the paper and make it easier for others to understand, replicate, and build upon the proposed method.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper offers a fresh viewpoint on an underexplored issue in PbRL and proposes an interesting solution.\n\n* The conducted experiments are extensive, covering a range of tasks and comparing against multiple benchmark methods.\n\n* The paper is written well, the problem is clearly defined, the proposed approach is thoroughly explained, and the empirical evaluation is meticulously conducted.\n\n* The results show the effectiveness of the proposed method in achieving significant gains in performance.", "weaknesses": "* Query-policy misalignment not conclusively proven: While the paper proposes that query-policy misalignment is an interesting hypothesis to cause a problem in PbRL, there is no comprehensive evaluation or solid evidence to confirm this proposal.\n\n* Lack of real human experiments: The testing and validation conducted in the paper relied on a scripted annotator which is not representative of real-world users, which could limit the generalizability and applicability of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oxJw1Xj82J", "reviewer": "Reviewer_EPnJ", "summary": "Summary: The paper addresses the challenge of Query-Policy Misalignment in Preference-based Reinforcement Learning (PbRL). PbRL is a method where reinforcement learning (RL) agents align their behavior based on human preferences. However, the efficiency of these models is often restricted due to costly human feedback. The paper identifies that most existing PbRL methods aim to improve the reward model's quality by selecting queries but may not necessarily enhance the RL agent's performance. The authors introduce the concept of policy-aligned query and hybrid experience replay as a solution. These methods focus on improving the alignment between the queries chosen and the current interests of the RL agent, thereby enhancing feedback efficiency. This is done by simply sampling from recent trajectories.", "questions": "Could the proposed methods be adapted or combined with on-policy PbRL techniques to achieve even better results?\nAre there specific scenarios or domains where the proposed method may not be as effective?\nHow does the system handle scenarios where human feedback might be inconsistent or contradictory?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel perspective, highlighting the overlooked issue of Query-Policy Misalignment in PbRL.\n2. The proposed solution of policy-aligned query selection and hybrid experience replay is simple to implement and requires minimal changes to existing PbRL systems.\n3. Comprehensive experiments on well-established benchmarks like DMControl and MetaWorld prove the substantial benefits of the proposed method in terms of feedback and sample efficiency.", "weaknesses": "1. The focus is predominantly on off-policy PbRL methods, with limited exploration of on-policy PbRL methods, which naturally select on-policy segments to query preferences.\n2. The approach is quite simple and I’m not sure if it’s novel compared to methods that are more similar to on-policy methods. \n3. The paper should compare to Liu et al. Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning. NeurIPS ‘22, which is the current SoTA for PbRL.\n4. I thought the Figure 1, 2, 3 could be improved and explained better, both in the text, and in the figure, and in the caption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)