Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
Authors: Katherine Metcalf
Miguel Sarabia
Natalie Mackraz
Barry-John Theobald
Venue: CoRL 2023 Poster (2023)
Keywords: human-in-the-loop learning
preference-based RL
RLHF

Abstract: Preference-based reinforcement learning (PbRL) aligns a robot behavior with human preferences via a reward function learned from binary feedback over agent behaviors. We show that encoding environment dynamics in the reward function improves the sample efficiency of PbRL by an order of magnitude. In our experiments we iterate between: (1) encoding environment dynamics in a state-action representation $z^{sa}$ via a self-supervised temporal consistency task, and (2) bootstrapping the preference-based reward function from $z^{sa}$, which results in faster policy learning and better final policy performance. For example, on quadruped-walk, walker-walk, and cheetah-run, with 50 preference labels we achieve the same performance as existing approaches with 500 preference labels, and we recover 83% and 66% of ground truth reward policy performance versus only 38% and 21% without environment dynamics. The performance gains demonstrate that _explicitly encoding environment dynamics improves preference-learned reward functions_.

Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
Katherine Metcalf
Miguel Sarabia Apple, California, USA
Natalie Mackraz
Barry-John Theobald
(kmetcalf, miguelsdc, natalie barryjomm.tthebalddjappp...
Abstract: Preference-basee reinforcement learning (PbRL) aligns a robot behav- ior with human preferences via a reward function learned from binary feedback over agent behaviors. We show that dynamics-aware reward functions improve the sample efficiency of PbRL by an order of magnitude. In our experiments we iterate between: (1) learning a dynamics-aware state-action representation 2 via self-supervised temporal consistency task, and (2) bootstrapping the preference- based reward function from 2nh which results in faster policy learning and better final policy performance. For example, on quadruped-walks walker-walk, and cheetah-run, with 50 preference labels we achieve the same performance as ex- isting approaches with 500 preference labels, and we recover 83% and 66% ground truth reward policy performance versus only 38% and 21%. The per- formance gains demonstrate the benefits of explicitly learning a dynamics-aware reward model. Repo: https://githut: wwt.coooaaaaaoo hamm/apple/nl--eed
Keywords: human-in-the-loop learning, preference-based RI., RLHF
1 Introduction
The quality of reinforcement learned (RI.) policy depends on the quality of the reward func- tion used to train it. However, specifying reliable numerical reward function is challenging. For example, a robot may learn to maximize a defined reward function without actually com- pleting a desired task, known as reward hacking [1, 2]. Instead, preference-based reinforcement learning (PhRL) infers the reward values by way of preference feedback used to train policy [3, 5, 7, 8, 10, 11, 12]. Using preference feedback avoids the need to manually define absolute numerical reward values (e.g. in TAMER [13]) and is easier provide than corrective feedback (e.g. in DAGGER [14]). However, many existing PbRL. methods require either demonstra- tions [5], which are not always feasible to provide, or an impractical number of feedback samples [3, 4, 11, 12, 15].
We target sample-efficient reward function learning by exploring the benefits of dynamics-aware reeereece--eaamee reward functions Rewards Encoding Environment Dynamics (REED) (Sec- tion 4.1). Fast alignment between robot behaviors and human needs is essential for robots operating on real world domains. Given the difficulty people face when providing feedhack for single state- action pair [13], and the importance of defining preferences over transitions instead of single state- action pairs [4], is likely that people's 's internal reward functions are defined over outcomes rather than state-action pairs. We hypothesize that: (1) modelling the relationship between state, action, and next-state triplets is essential to learn preferences over transitions, (2) encoding awareness of dynamics with a temporal consistency objective will allow the reward function to better generalize states and actions with similar outcomes, and (3) exposing the reward model to all transitions experienced by the policy during training will result in more stable reward estimations during re- ward and policy learning. Therefore, we incorporate environment dynamics via a self-supervised temporal consistency task using the state-of-the- self-predictive representations (SPR) [16] as one such method for capturing environment dynamics.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

We evaluate the benefits of dynamics-awareness awareness using the current state-of-the-an in preference learning [3, 4]. In our experiments, which follow Lee et al. [10], REED reward functions out- perform non- non-REED reward functions across different preference dataset sizes, quality of preference labels, observation modalities, and tasks (Section 5). REED reward functions lead to faster policy training and reduce the number of preference samples needed (Section 6) supporting our hypotheses about the importance of environments dynamics for preference-leamee reward functions.
2 Related Work
Learning from Human Feedback. Learning reward functions from preference- basedd feedback [7. 17, 18, 19, 20, 21, 22, 23] has been used to address the limitations of learning policies directly from human feedback [24, 25, 26] by inferring reward functions from either task success [27, 28, 29] or real-valued reward labels [30, 31]. Learning policies directly from human feedback is inefficient as near constant supervision is commonly assumed. Inferring reward functions from task success feedback requires examples of success, which can be difficult to acquire in complex and multi-step task domains. Finally, people have difficulty providing reliable, real-valued reward labels. PbRL was extended to deep RL. domains by Christiano et al. [3], then improved upon and made more effi cient by PEBBLE [4] followed by SURF [11], Reta--Rearrd-Ne (MRN) [12], and RUNE [15]. To reduce the feedback complexity of PhRL, PEBBLE [4] sped up policy learning via (1) intrinsically- motivated exploration, and (2) relabelling the experience replay buffer. Both techniques improved the sample complexity of the policy and the trajectories generated by the policy, which were then used to seek feedback. SURF [1 reduced feedback complexity by incorporating augmentations and sheudo-labellingg into the reward model learning. RUNE [15] improved feedback sample complexity by guiding policy exploration with reward uncertainty. MRN [12] incorporated policy performance in reward model updates, but further investigation required to ensure that the method does not allow the policy influence and bias how the reward function is learned, two concerns called out in [32]. SIRL [33] adds an auxiliary contrastive objective encourage the reward function to learn similar representations for behaviors human labellers consider to be similar. However, this approach requires extra feedback from human teachers to provide information about which behaviors are sim- ilar to one another. Additionally, preference-learning has also been incorporated into data-driven skill extraction and execution in the absence of a known reward function [34]. Of the extensions and improvements to PbRL, only MRN [12] and SIRL [33], like REED, explore the benefits of auxiliary information.
Encoding Environment Dynamics. Prior work has demonstrated the benefits of encoding environ- ment dynamics in the state-action representation of a policy [16, 35, 36), and reward functions for imitation learning [37, 38] and inverse reinforcement learning [39, 40]. Additionally, it is common for dynamics models to predict both the next state and the environment's reward [35], which sug- gests is important imbue the reward function with awareness of the dynamics. The primary self- supervised approach to learning dynamics model is predict the latent next state [16, 35, 41, 42), and the current state-of-the- offtthe-rrt in data efficient RL [16, 36] uses SPR [16] to do exactly this. Unlike prior work in imitation and inverse reinforcement learning, we explicitly evaluate the benefits of dynamics-aware auxiliary objectives versus auxiliary objective induced regularization effect.
3 Preference-based Reinforcement Learning
RL trains an agent to achieve tasks via environment interactions and reward signals [43]. For each time step the environment provides a state s; used by the agent to select an action according to its policy 4, ~ (a)ss). Then a, is applied to the environment, which returns a next state according to its transition function $1+1 ~ T(st, a.) and a reward r(st, at). The agent's 's goal is to learn a policy ne maximizing the expected discounted return, Ex= " **(+st at+x). In PbRL [3, 4, 5, 7, 8, 9, 10] ns trained with a reward function f distilled from preferences Ps iteratively queried from a teacher, where re is assumed to be latent factor explaining the preference Pv. A buffer B of transitions is accumulated as gg learns and explores.

---

A labelled preference dataset Preet is acquired by querying teacher for preference labels every K steps of policy training and is stored as triplets (o², Up). where o² and o² are trajectory segments (sequences of state-action pairs) of length I, and Up is a preference label indicating which, if any, of the trajectories is preferred [10]. To query the teacher, the M maximally informative pairs of trajectory segments (e.g. pairs that most reduce model uncertainty) are sampled from B, sent the teacher for preference labelling, and stored in peet [10, 23, 44, 45]. Typically 'D peel is used to update i on schedule conditioned on the training steps for T (e.g. every time the teacher is queried).
The preference triplets (a-,o², Up) create a supervised preference prediction task to approximate re with P [3, 4, 19). The prediction task follows the Bradley-Tery model [46] for a stochastic teacher and assumes that the preferred trajectory has a higher cumulative reward according to the teacher's ri- The probability of the teacher preferring o² over a² (o σ²) is formalized as:
exp E, (s) (sss,aj) at)
Eie(1.2) exp E, Pos (s), Po 0²1
(1)
where of is the state at time step of trajectory 21, and aj is the corresponding action taken. The parameters va of Fwv are optimized such that the binary cross-entropy over 'D Pret is minimized:
- (ot,o".8p))
E
1y log Po o'] + (1) log Paloo > =
(2)
While Polot > o²] and L defined over trajectory segments, fw operates over individual (st, ar) pairs. Each reward estimation in Equation is made independently of the other (St, ax) pairs in the trajectory and Polo o²] simply sums the independently estimated rewards. Therefore, environ- ment dynamics, or the outcome of different actions in different states, are not explicitly encoded in the reward function, limiting its ability to model the relationship between state-action pairs and the values associated with their outcomes. By supplementing the supervised preference prediction task with a self-supervieed temporal consistency task (Section 4.1), we take advantage of all transitions experienced by Ta to learn a state-action representation in way that explicitly encodes environment dynamics, and can be used to learn solve the preference prediction task.
4 Dynamics-Aware Reward Function
In this section, we present our approach to encoding dynamics-awarence via a temporal consistency task into the state-action representation of preference-leareed reward function. There are many methods for encoding dynamics and we show results using SPR, the current state of the art. The main idea is to learn a state-action representation that is predictive of the latent representation of the next state using self-supervised temporal consistency task and all transitions experienced by z Preferences are then learned with a linear layer over the state-action representation.
4.1 Rewards Encoding Environment Dynamics (REED)
We use the SPR [16] self-supervised temporal consistency task to learn state-action representations that are predictive of likely future states and thus environment dynamics. The state-action represen- tations are then bootstrapped to solve the preference prediction task in Equation (see Figure 1 for an overview of the architecture). The SPR network is parameterized by and 0, where v is shared with A and is unique to the SPR network. At train time, batches of (st, at, St41) triplets are sampled from buffer B and encoded: [s(S,, 1/sa) zf, fa(a,, v))) - =f [=f =ē, thsa) + = and fa (st+e, v/a) 2i+1- The embedding 2f41 is used to form our target for Equations 3 and 4. A dynamics function ga(2;" A2) + 2/41 then predicts the latent representation of the next state 2f41- The functions (-).. (.).,, and ga(-) are multi-layer perceptrons (MLPs), and f-a(-) concatenates =ī and 2 along the feature dimension before encoding them with a MLP. To encourage knowledge of environment dynamics in =;" ga(-) is kept simple, e.g. a linear layer.
Following [16], projection head (- pppo) is used to project both the predicted and target next state representations to smaller latent spaces via bottleneck layer and a prediction head
3

---

predicted
PREFERENCE-LEANEE
MLP
REWARD FUNCTION
PEEF-PREICTIVEE REPRESENTATION (SPR)
action
Action Enccoer GA (9,)
Replay Buffeer
sample
Encocit Ful
Dynanics ..0)
Projection Haral Opra)
Apral + Opra)
state
State Encoder vd
Projection
Viy
XA .W.
ligeol -Ppr))
Figure Architecture for self- self-predictive representation (SPR) objective [16] (in yellow), and Snterence-learnee reward function (in blue). Modules in green shared between SPR and the preference-laane leareedd reward function.
( (pe) is used to predict the target projections: f441 = Ame (h (ppoo (2++1, Ppro): 0pre) and yf++1 (pr42211: p. Both Apro and Apr are modelled using linear layers.
The benefits of REED should be independent of the self seff-supervieed objective function. Therefore, we present results for two different self-supervised objectives: Distillation (i.e. SimSiam with loss ( [36, 47] and Contrastive (i.e. SimCLR with loss L [48, 49, 50], referred to as Distillation REED and Contrastive REED respectively. LS and _ are defined as:
S - (o
(3)
(xp sg( 1))//
_ log
2N E [on exp(cos (1)- 0()/T)
In LSS, stop gradient operation, sg((..), applied to 3 and then 0'+ pushed be consistent with v441 via negative cosine similarity loss. In L stop gradient operation is applied to y and then V641 is pushed to be predictive of which candidate next state is the true next state via the NT-Xent loss.
Rather than applying augmentations to the input, temporally adjacent states are used create the different views [16, 36, 49, 50). Appendix D details the architectures for the SPR component net- works.
State-Action Fusion Reward Network. REED requires a modification to the reward network ar- chitecture used by Christiano al. [3] and PEBBLE [4] as latent state representations are compared. Instead of concatenating raw state-action features, we separately encode the state, fa(-), and action, fa()). before concatenating the embeddings and passing them the body of our reward network. For the purposes of comparison, refer to the modified reward network the state-action fusion (SAF) reward network. For architecture details, see Appendix D.
4.2 Incorporating REED into PbRL
The self- supe-supe temporal consistency task is used update the parameters and 0 each time the reward network is updated (every K steps of policy training, Section 3). All transitions in the buffer B are used to update the state-action representation = which effectively increases the amount of data used to train the reward function from M K preference triplets to all state-action pairs experienced by the policy'. REED precedes selecting and presenting the M queries to the teacher for feedback. Updating 1 and 0 prior to querying the teacher exposes se larger space of environment dynamics (all transitions collected since the last model update), which enables the model to learn more about the world prior selecting informative trajectory pairs for the teacher to label. The state-action representation nd and lincar prediction layer used to solve the
Note the reward function is still trained with M K triplets, but the sattonction encoder has the opportunity better capture the dynamics of the environment.

---

preference prediction task (Equation 1). After each update to fg.. T is trained on the updated Pp-- See Appendix for REED incorporated into PrefPPO [3] and PEBBLE [4].
Experimental Setup
Our experimental results in Section 6 demonstrate that learning dynamics-aware reward function explicitly improves PbRL policy performance for both state-space and image-space observations. To verify that the performance improvements are due to dynamics-awarenes, rather than just the inclusion of a self-supervised auxiliary task, we compared against an image-augmentation bbased auxiliary task (Image Aug.). The experiments and results are provided in Appendix G) and show that indeed the performance improvements are due specifically to encoding environment dynamics in the reward function. Additionally, we compare against the SURF [11], RUNE [15], and MRN [12] extensions to PEBBLE.
We follow the experiments outlined by the B-Pref benchmark [10]. Models are evaluated on the DeepMind Control Suite (DMC) [51] and MetaWorld [52] environment simulators. DMC provides locomotion tasks with varying degrees of difficulty and MetaWorld provides object manipulation tasks. For each DMC and MetaWorld task, we evaluate performance on varying amounts of feed- back, i.c. different preference dataset sizes, and different labelling strategies for the synthetic teacher. The number of queries (M) presented to the teacher every K steps is set such that for a given task, teacher feedback always stops at the same episode. Feedback is provided by simulated teachers following [3, 4, 10, 11, 34), where six labelling strategies are used to evaluate model performance under different types and amounts of label noise. The teaching strategics were first proposed by B-Pref [10]. An overview of the labelling strategies provided in Appendix B.
Following Christiano et al. [3] and PEBBLE [4), Pg is modelled with an ensemble of three networks with a corresponding ensemble for the SPR auxiliary task. The ensemble is key for disagreement- based query sampling (Appendix A) and has been shown to improve final policy performance [10]. All queried segments are of a fixed length (2 = 50)2. The Adam optimizer [53] with 81 = 0.9, 82 = 0.999, and no Iæ-regulariiztion [54] is used to train the reward functions. For all PEBBLE- related methods, intrinsic policy training is reported in the learning curves and occurs over the first 9000 steps. The batch size for training on the preference dataset M, matching the number of queries presented to the teacher, and varies based on the amount of feedback. For details about model architectures, hyper-parametery, and the image augmentations used in the mmage-augmennation self- supervised auxiliary task, refer to Appendices D, E, and G. None of the hyper-parametedd nor archi- tectures are altered from the original SAC [55], PPO [56], PEBBLE [4], PrefPPO [4], Meta-Reward Net [12], RUNE [15], nor SURF [11] papers. The policy and preference learning implementations provided in the B-Pref repository [57] are used for all experiments.
6 Results
The synthetic preference labellers allow policy performance to be evaluated against the ground truth reward function and reported as mean and standard deviation over 10 runs. Both learning curves and mean normalized returns are reported, where mean normalized returns [10] are given by:
Σ To (sts π (a:)) ro (8,, T (a:))
normalized returns T
(5)
where T' is the number of policy training training steps or episodes, ry is the ground truth reward function, T is the policy trained on the learned reward function, and π rv the policy trained on the ground truth reward function.
For
2 Fixed segments lengths are not strictly necessary, and, when evaluating with simulated humans, are harmful when the reward is constant step penalty.
5

---

Table 1: Mean and + s.d. normalized return (Equation 5) over 10 random seeds with the oracle labeller and disagreement sampling. The best result for each condition is in bold. BASE refers to the PEBBLE or PrefPPO baseline, +DISTILL distillation REED, and +CONTRAST contrastive REED. SURF, RUNE, and MRN are baselines. Results are normalized relative to SAC. See Appendices H.2 and H.4 for all tasks, feedback amounts, and PREFPPO results.
PEBBLE
TASK
FEED.
BASE
+DITTILL
+CONTRAST
SURF [11]
RUNE [15]
[12]
WALKER
500
0.74 1 0.18
0.86 0.20
0.90 1 0.17
0.78 1 0.12
0.76 0.20
0.77 1 0.20
50
0.21
0.10
0.66 0.24
0.62 0.22
0.47 14770.13
0.23
0.38
0.12
QUADRUPED
500
0.56 + 0.21
1.10 0.21
0.80 0.188
1.10
1.10 0.21
WALK
50
0.38 +: 0.26
0.65 = 0.16
0.31 # 0.18
0.48 + 0.19
0.41 0.21
0.83 : 0.12
CHEETAH
500
0,86 + 0.14
0.88 0.22
0.94 + 0.21
0.56 0,16
0.61 + 0.17
0.80 = 0.16
RUN
50
0.35 0.11
0.63 0.23
0.70 + 0.28
0.35 + 0.18
0.32 0.12
0.38 0.16
BUTTON
10k
0.66 1: 0.26
Collapses Collapses
0.65 1 0.27
0.68 1 0.29
0.45 0.21
0.59 1 0.27
PRESS
2.5K
0.37 4:
0.25
0.18
0.22 +
0.35
SWEEP
0.28
Collapses Collapses
0.47 0.23
0.48
0.26
0.15
0.28
INTO
2.5K
0.15 + 0.09
0.21 + 0.
0.25 + 0.13
0.16 0.11
0.22 = 0.12
MEAN
0.46
0.47
0.64
0.55
0.46
0.57
Walker Walk
Quadruped Walk
Cheatah Run
Button Press
Sweep Into
1000
100
2500
500
50
1000
108-
10000
50
10
0.5
Steps (x1e66)
Steps txle6)
Steps (x1e6)
Steps (xle6)
Steps (xle6)
sac
mm
rune
pebale
pibble-distt
contr
pdo
prefspo
prefope dit..
prefopo
Figure/ 2: Learning curves for three DMC and two MetaWorld tasks with 50 and 500 (DMC) and 2.5k and 10k (MetaWorld) pieces of feedback, for state-space observations, disagreement sampling, and oracle labels. Refer to Appendices H.I and H.3 for more tasks and feedback amounts.
Learning curves for state-space observations for SAC and PPO trained on the ground truth reward, PEBBLE, PrefPPO, PEBBLE + REED, PrefPPO + REED, Meta-Reward-Neet SURF, and RUNE are shown Figure 2. and mean normalized returns [10] are shown in Table (Appendix H.2 and H.4 for PrefPPO). The learning curves show that reward functions learned using REED consistently outper- form the baseline methods for locomotive tasks, and for manipulation tasks REED methods are con- sistently a top performer, especially for smaller amounts of feedback. On average across tasks and feedback amounts, REED methods outperform baselines (MEAN in Table 1).
prebpc--d
prepposcceets
Figure 3: Mean normalized return across or- acle, noisy, mistake, and equal labellers Lee et al. [10] on quadruped-walk with state- space observations for 50, 500, and 1000 pieces of feedback.
Learning curves for image-space observations presented for the PEBBLE and PEBBLE+REED methods in Figure 4, and mean normalized returns [10] in Appendix H.4 The impact of preference label quality on policy performance for PEBBLE and PEBBLE+REED shown in Figure 3 and Table On average, across labeller strategies, REED-baasd methods outperform baselines.
Figures 2, 3 and show that REED improves the speed of policy learning and the final performance of the learned policy relative to REn-REED methods. The increase in policy performance is observed across environments, labelling strategies, amount of feedback, and observation type. We ablated the

---

Walker Walk
Quadruped Walk
Chestai Run
Button Press
Sweep Into
1000
100
2500
500
1000
108
10000
500
50
Steps (X1e6)
Steps (X1e6)
Steps (X1e6)
Steps (x1e6)
Steps (x1e6)
38C
pebble
pebblerdist
pabbiencontr
prolppo
prefppo-distil.
prafpporcont:
Figure 4: Learning curves for three DMC and two MetaWorld tasks with 50 and 500 (DMC) and 2.5k and 10k (MetaWorld) pieces of feedback, for imagc-space observations, disagreement sam- pling, and oracle labels. Only PEBBLE is evaluated for the image-space due to the poor state-space performance of PrefPPO. Results for more tasks and feedback amounts available in Appendices H.1 and H.3.
impact of the modified reward architecture and found that the REED performance improvements are not due to the modified reward network architecture (Appendix F). Refer to Appendix for results across all combinations of task, feedback amount, and teacher labelling strategies for state-space observations 1 and H.2), and image-space MetaWorld results more tasks (drawer open, drawer close, window open, and door open) and feedback amounts (H.3 and H.4). The trends observed in the subset of tasks included in Table I, and Figures 2 and 4 are also observed in the additional tasks and experimental conditions in the Appendices.
6.1 Source of Improvements
Table 2:
Mean normalized returns across feedback amounts, tasks, and labeller types (oracle, mistake, noisy, and equal).
There is no clear advantage between the Distillation and Contrastive REED objectives on the DMC loco- motion tasks, suggesting the improved policy perfor- mance stems from encoding awareness of dynamics rather than any particular self-supervised objective. However in the
PEBBLE +DISTILL +CONTRAST SURE RUNE MRN
0.53
0.47
0.68
0.54
0.46
0.50
object manipulation tasks, Distillation REED tends collapse with Con- trastive REED being the more robust method. From comparing SAC, PEBBLE, PEBBLE+REED, and PEBBLE+Image Aug. (Appendix G.3), we see that PEBBLE+Image Aug. improves perfor- mance over PEBBLE with large amounts of feedback (e.g. 4.2 times higher mean normalized re- turns for walker-walk at 1000 pieces of feedback), but does not have a large effect on performance for lower-feedback regimes (e.g. 5.6% mean normalized returns with PEBBLE+Image Aug. versus 5.5% with PEBBLE for walker-walk at 50 pieces of feedback). In contrast, incorporating REED always yields higher performance than both the baseline and PEBBLE+ DessLEE+Image Aug. regardless of the amount of feedback. For results analyzing the generalizability and stability of reward function learning when using a dynamics awwae aware auxiliary objective, see Appendix I.
7 Discussion and Limitations
The benefits dynamics awareness are especially pronounced for labelling types that introduce in- correct labels (i.e. mistake and noisy) (Figure 3 and Appendix H) and smaller amounts of preference feedback. For example, state-space observation DMC tasks with 50 pieces of feedback, REED methods more closely recover the performance of the policy trained on the ground truth reward re- covering 62 66% versus 21% % on walker-walk, and 65 85% versus 38% on quadruped-wall for PEBBLE- based methods (Table 1). Additionally, PEBBLE+REED methods retain policy perfor- mance with factor of 10 fewer pieces of feedback compared to PEBBLE. Likewise, when con- sidering image-space observations, PEBBLE+REED methods trained with times less feedback exceed the performance of base PEBBLE on all DMC tasks. For instance, PEBBLE+Contrastvve REED achieves a mean normalized return of 53% with 50 pieces of feedback whereas baseline PEBBLE reaches 36% on the same task with 500 pieces of feedback.
7

---

The policy improvements are smaller for REED reward functions on Meta World tasks than they are for DMC tasks and are generally smaller for PrefPPO than PEBBLE due to a lack of data diversity in the buffer B used to train on the temporal consistency task. For PrefPPO lack of data diversity is due to slow learning and for MerlW a high similarity between observations. In particular, Distillation REED methods on state-space observations frequently suffer representation collapse and are not reported here. The objective, in this case SimSiam, learns degenerate solution, where states encoded by constant function and actions are ignored due to the source and target views having near perfect cosine similarity. However, representation collapse is not observed for the image-space observations, and baseline performance is retained with one quarter the amount of feedback when training with PEBBLE+REED methods. the amount of feedback is kept constant, we notice a 25% to 70% performance improvement over the baseline for all PEBBLE+REED methods in the Button Press task.
The benefits of dynamics awareness can be compared against the benefits of other approaches to improving feedback sample complexity, specifically pseudo-labelling (in SURF) [11], guiding pol- icy exploration with reward uncertainty (in RUNE) [15], and incorporating policy performance into reward updates (in MRN) [12]. REED methods consistently outperform SURF, RUNE, and MRN on the DMC tasks demonstrating the importance of dynamics awareness locomotion tasks. On the Meta MetaWorld object manipulation tasks, REED frequently outperforms SURF, RUNE, and MRN, especially for smaller amounts of feedback, but the performance gains smaller than for DMC. Smaller performance gains on MetaWorld relative to other sample efficiency methods is in line with general REED findings for MetaWorld (above) that relate to the slower environment dynamics. However, it is important to call out that all four methods are complementary and can be combined.
Across tasks and feedback amounts, policy performance higher for rewards that are learned on the state-space observations compared those learned image-space observations. There several tasks, such cheetah-run and sweep into, for which PEBBLE, and therefore all REED experiments that build on PEBBLE, are not able to learn reward functions that lead to reasonable policy performance when using the image-space observations.
The results demonstrate the benefits and importance of environment dynamics to preference-leaneed reward functions.
Limitations The limitations of REED are: (1) more complex tasks still require a relatively large number of preference labels, (2) extra compute and time are required, (3) Distillation REED can collapse when observations have high similarity, and (4) redundant transitions in the buffer B from slow policy learning or state spaces with low variability result in over-fitting on the temporal consis- tency task.
Conclusion
We have demonstrated the benefits of dynamics awareness in a preference-leareed reward for PbRL, especially when feedback limited noisy. Across experimental conditions, found REED methods retain the performance of PEBBLE with 10-fold decrease in feedback. The benefits are observed across tasks, observation modalities, and labeller types. Additionally, we found that, com- pared to the other PbRL extensions targeting sample efficiency, REED most consistently produced the largest performance gains, especially for smaller amounts feedback. The resulting sample ef- ficiency is necessary for learning reward functions aligned with user preferences in practical robotic settings.
8

---

References
[1] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete prob- lems in Al safety. arXiv preprint arXiv: 1606.06600 2016.
[2] D. Hadeield-Menlll, S. Milli, P. Abbeel, S. Russell, and A. Dragan. Inverse reward design. Advances in Neural Information Processing Systems, 30, 2017.
[3] P. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. volume 30, 2017.
[4] K. Lee, L. Smith, and P. Abbeel. PEBBLE: Freddbac--efficienn interactive reinforcement learn- ing via relabeling experience and unsupervised pre-training. In Proceedings of the Interna- tional Conference on Machine Learning, pages 6152-6163. PMLR, 2021.
[5] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in Atari. Advances in Neural Information Processing Systems, 31, 2018.
[6] D. Hadfield-Menell, S. Russell, P. Abbeel, and A. Dragan. Cooperative inverse reinforcement learning. Advances in Neural Information Processing Systems, 29, 2016.
[7] J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg- Scalable agent alignment via reward modeling: A research direction. arXiv preprint
2018.
[8] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and
P. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[9] J. Wu, L. Ouyang. D. Ziegler, N. Stiennon, R. Lowe, J. Leike, and P. Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862, 2021.
[10] K. Lee, L. Smith, A. Dragan, and P. Abbeel. B-Pref: Benchmarking preference-based rein- forcement learning. Neural Information Processing Systems, 2021.
[11] Park, Y. Seo, J. Shin, H. Lee, Abbeel, and K. Lee. SURF: semi-supervised reward learning with data augmentation for feedhack-efficient preference-based reinforcement learning. arXiv preprint arXiv:2203. 10050, 2022.
[12] R. Liu, Bai, Y. Du, and Y. Yang. Meta-Reeard--Net Implicitly differentiable reward learn- ing for preference-bare based reinforcement learning. Advances in Neural Information Processing Systems, 35, 2022.
[13] W. Knox and P. Stone. Tamer: Training an agent manually via evaluative reinforcement. In Proceedings of the International Conference on Development and Learning, pages 292-297. IEEE, 2008.
[14] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion no-regret online learning, In Proceedings of the International Conference Artificial Intelligence and Statistics, pages 627-635, 2011.
[15] X. Liang, K. Shu, K. Lee, and P. Abbeel. Reward uncertainty for exploration in preference- based reinforcement learning. 2022.
[16] M. Schwarzer, A. Anand, R. Goel, R. Hjelm, A. Courville, and Bachman. Data-efficient reinforcement learning with self- predictre representations, In Proceedings of the International Conference Learning Representations, 2020.
[17] R. Akrour, M. Schoenauer, and M. Sebag. Preference-based policy learning. In Proceedings of the Joint European Conference Machine Learning and Knowledge Discovery in Databases, pages 12-27. Springer, 2011.

---

[18] R. Akrour, M. Schoenauer, and M. Sebag. April: Active preference learning- based reinforce- ment learning. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 1 116-131. Springer, 2012.
[19] A. Wilson, A. Fern, and P. Tadepalli. A Bayesian approach for policy learning from trajectory preference queries. Advances in Neural Information Processing Systems, 25, 2012.
[20] H. Sugiyama, T. Meguro, and Y. Minami. Preference-learning based inverse reinforcement learning for dialog control. In Proceedings of Interspeech, 2012.
[21] C. Wirth and J. Fürkrraz. Preference-basee based reinforcement learning: A preliminary survey. In Proceedings of the Workshop on Reinforcement Learning from Generalized Feedback: Beyond Numeric Rewards, 2013.
[22] C. Wirth, J. Fürnkranz, and G. Neumann. Model-free preference-based reinforcement learning. In Proceedings of the Conference on Artificial Intelligence (AAAI), 2016.
[23] D. Sadigh, A. Dragan, S. Sastry, and S. Seshia. Active preference-based learning of reward functions. 2017.
[24] Pilarski, M. Dawson, T. Degris, F. Fahimi, Carey, and R. Sutton, Online human training of myoelectric prosthesis controller via actor-critic reinforcement learning. In Proceedings of the International Conference Rehabilitation Robotics, pages 1-7. IEEE, 2011.
[25] J. MacGlashan, M. Ho, R. Loftin, B. Peng, G. Wang, D. Roberts, M. Taylor, and M. Littman. Interactive learning from policy-dependent human feedback. In Proceedings of the Interna- tional Conference on Machine Learning, pages 2285-2294 PMLR, 2017.
[26] D. Arumugam, J. Lee, S. Saskin, and M. Littman. Deep reinforcement learning from policy- dependent human feedback. arXiv preprint arXiv: 2019.
[27] M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. Johnson, and S. Levine. Solar: Deep structured representations for model- reinforcement learning. In Proceedings of the International Conference on Machine Learning, pages 7444-7453. PMLR, 2019.
[28] A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine. End-to-end robotic reinforcement learning without reward engineering, arXiv preprint 1041v:190..07844 2019.
[29] L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine. AVID: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint
2019.
[30] W. Knox and P. Stone. Interactively shaping agents via human reinforcement: The TAMER framework. In Proceedings of the International Conference on Knowledge Capture, pages 9-16, 2009.
[31] G. Warnell, N. Waytowich, V. Lawhern, and P. Stone. Deep Tamer: Interactive agent shaping in high-dimensional state spaces. In Proceedings of the Conference Artificial Intelligence (AAAI), volume 32, 2018.
[32] Armstrong, Leike, L. Orseau, and S. Legg. Pitfalls of learning a reward function online. arXiv preprint 2xXxi:2004.1364. 2020.
[33] A. Bobu, Y. Liu, R. Shah, D. S. Brown, and A. D. Dragan. Sirl: Similarity-basee implicit representation learning. In Proceedings of the 2023 ACM/IEEE International Conference on Himann-Robor Interaction, pages 0644-574220 2023.
[34] X. Wang, K. Lee, K. Hakhamaneshi, Abbeel, and M. Laskin. Skill preferences: Learning to extract and execute robotic skills from human feedback. In Proceedings of the Conference Robot Learning, pages 1259-1268. PMLR, 2021.
10

---

[35] A. Zhang, R. McAllister, R. Calandra, Y. Gal, and S. Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020.
[36] W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y. Gao. Mastering Atari games with limited data. Advances in Neural Information Processing Systems, 34, 2021.
[37] D. Brown, R. Coleman, R. Srinivasan, and S. Niekum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference on Machine Learning, pages 1165-1177. PMLR, 2020.
[38] H. Sikchi, A. Saran, W. Goo, and S. Niekum. A ranking game for imitation learning, arXiv preprint arXiv:2202.03488/ 2022.
[39] D. Brown, W. Goo, P. Nagarajan, and S. Nickum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. In International conference on machine learning, pages 783-792. PMLR, 2019.
[40] L. Chen, R. Paleja, and M. Gombolay. Learning from suboptimal demonstration via self- supervised reward regression. In Conference on robot learning. pages 1262-1277. PMLR, 2021.
[41] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the International Conference on Machine Learning, pages 2565.2 PMLR, 2019.
[42] A. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep rein- forcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33:741-752, 2020.
[43] R. Sutton and A. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[44] E. Biyik and D. Sadigh. Batch active preference-based learning of reward functions. In Pro- ceedings of the Conference on Robot Learning, pages 519-528. PMLR, 2018.
[45] E. Biyik, N. Huynh, M. Kochenderfer, and D. Sadigh. Active meference-basee gaussian pro- cess regression for reward learning. In Proceedings of the Robotics: Science and Systems, 2020.
[46] R. Bradley and M. Temy. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
[47] X. Chen and K. He. Exploring simple Siamese representation learning. In Proceedings of the Conference Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[48] T. Chen, Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learn- ing of visual representations. In Proceedings of the International Conference Machine Learning, pages 1597-1607. PMLR, 2020.
[49] A. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 18Xiv:8877003744 2018.
[50] B. Mazoure, R. Tachet des Combes, T. Doan, Bachman, and R. Hjelm. Deep reinforcement and InfoMax learning. Advances in Neural Information Processing Systems, 33:3686-3698, 2020.
[51] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Li, D. Casas, D. Budden, A. Abdolmaleki, J. Merel, 2018.
A. Lefrancq, et al. Deepmind control suite. arXiv preprint
[52] T. Yu, D. Quillen, Z. He, R. Julian, Hausman, C. Finn, and S. Levine. Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning. In Proceedings of the Conference Robot Learning, pages 1094-1100 100. PMLR, 2020.
11

---

[53] D. Kingma and J. Ba. ADAM: A method for stochastic optimization. volume 3, 2015.
[54] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-pent Segh-performance deep learning library. Advances neural information processing systems, 32, 2019.
[55] T. Haarnoja, A. Zhou, Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum tropy deep reinforcement learning with stochastic actor. In Proceedings of the International Conference Machine Learning, pages 1861-1870. PMLR, 2018.
[56] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.
[57] K. Lee, L.. Smith, A. Dragan, and Abbeel. B-Pref, 2021. URL httpt:htt//gghhiiee com/ rll- All-rrseearchBPrres
[58] 7. Mandi, F. Liu, K. Lee, and Abbeel. Towards more generalizable one-shot visual imitation learning. Proceedings of the International Conference on Robotics and Automation, pages 2434-2444. IEEE, 2022.
[59] Goyal, S. Niekum, and R. Mooney. Pix12r: Guiding reinforcement learning using natural language by mapping pixels rewards. In Proceedings of the Conference on Robot Learning, pages 485-497, PMLR, 2021.
A Disagreement Sampling
For all experiments in this paper, disagreement sampling is used select which trajectory pairs will be presented to the teacher for preference labels. Disagreement-baase sampling selects trajectory pairs as follows: (1) N segments sampled uniformly from the replay buffer; (2) the M pairs of segments with the largest variance in preference prediction across the reward network ensemble are sub-sampled. Disagreement-basee sampling is used as it reliably resulted in highest performing policies compared to the other sampling methods discussed in Lee et al. [10].
B Labelling Strategies
An overview of the six labelling strategies is provided below, ordered from least noisy (see [10] for details and configuration specifics):
1. oracle prefers the trajectory segment with the larger return and equally prefers both seg- ments when their returns are identical
2. skip - follows oracle, except randomly selects 10% of the M query pairs to discard from the preference dataset Dpref
3. myopic follows oracle, except compares discounted returns (? 0.9) placing more weight transitions at the end of the trajectory
4. equal follows oracle, except marks trajectory segments equally preferable when the difference in returns is less than 0.5% of the average ground truth returns observed during the last K policy training steps
5. mistake- - follows oracle, except randomly selects 10% of the M query pairs and assigns in- correct labels in a structured way (e.g., a preference for segment two becomes a preference for segment one)
6. noisy randomly assigns labels with probability proportional the relative returns asso- ciated with the pair, but labels the segments equally preferred when they have identical returns
12

---

C REED Algorithm
The REED task is specified in Algorithm the context the PEBBLE preference-learning al- gorithm. The main components of the PEBBLE algorithm are included, with our modifications identified the comments. For the original and complete specification of PEBBLE, please see [4] Algorithm 2.
Algorithm PEBBLE + REED Training Procedure
1: Given:
2:
K D teacher feedback frequency
3:
D t queries per feedback
4:
Initializes:
5:
Qo D parameters for Q-function
6:
Fi leaarned reward function
7:
(P (R).)) D self-future consistency (10 parameters shared with P.)
8:
Drref <- 0 preference dataset
9:
Dspr +- 00 SPR dataset
10: unsupervised policy training and exploration
11: B, n + EXPLORE [4] Algorithm 1
12: joint policy and reward training
13: for policy train step do
14:
if step % K = 0 then
15:
Daic <- Dat UB & update SPR dataset
16:
for each SPR gradient step do
17:
(St, 0t. 8t+1)} ~ Date sample minibatch + SFC JOR_FORW ((tt, ats $(41)3) Section D.2
optimize Creed with respect to SPR(w..) > Equations (3) and (4)
20:
F - SPR bo copy shared SPR parameters to reward model
update Prect fw. and Bb following [4] Algorithm 2 [lines 9 18]
22:
update B, no. and Qo following [4] Algorithm 2 [lines 20 27]
13

---

D Architectures
The network architectures are specified PyTorch 1.13. For architecture hyper-paramelers, c.g. hidden size and number of hidden layers, see Appendix E.2
D.1 Self-Predictive Representations Network
The SPR network is implemented in PyTorch. The architectures for the mext_state.projecae and consistency predictor when image observations are used come from [58]. The image encoder architecture comes from [59]. The SPR network initialized as follows:
def build_spr_netvore
self, state_size: int,
sizze: int, action_size: int, action.eabed_size: int,
hidden_size: int,
conisssencyypFoojeciooo...e int,
caga3tenggy.cmppriioo..dddd.ee int,
sith_conaisttenryppeeicctto.oo.n bool, num_layers: int .mage_observations: bool,
ingg..hidee.nuu.ccannl.ss int
The network architecture and build logic the complete the REED self self superviidd temporal consistency task based SPR.
Args:
state_size: number of features defining the agent's state space state_ catt__embed_sie: number of dimensions in the state embedding action_size: number features defining the agent's actions action_embed_size number dimensions in the action embedding hidden_size: number of dimensions in the hidden layers of state-action embedding network
consistency_Projjctiooo.ii1 number of units used compare the predicted and target latent next
state
segasttecc_comprisso.hhdeeeee__ number of units in the hidden
layers of the proje and the consistency when using the contrastive objective the consistency prediction is not used, but is when using the distillation objective
with.coniistenyy.rrdittt.....e
num_layers: number of hidden layers used to embed the state-action representation
image_ whether image observations are used. If image used, state-space
observations observations are used image__hident num cuw.nennells: the number of channels to use in the image encoder's hidden layers
build the network that will encode the state features image_ onservotservttoon:::
state_conv_encodee = mn. Sequential
mn. Conv2d
state_size [0]
image_hidden, num_ chaneenelss
3, stride=1
14

---

Re.UeLU() MaxPoo2l (2, 2),
nn. Conv2d (
image hht__hhidd____eee nu__ caadeen_lum_channllls.
jmage.hidden.nun.cheeee
stride=1
)
nn ReLU MaxPool2d (2, 2),
nn. (onv2d
num num_ cum_neeneess,
image_hidden, num _ channels,
3,
stride=1
)
nn ReLU), M..Maaooll2ddc (2, 2)
conv out_ torch flatten
_atate_ c
torch rand (size=[1] list (ize))
)) ()
self. statste_teeeeodee nn. Sequentiaa
nn Linea
(convoooccct out_size,
LeakyRel nw..eekgeell(eeaatteeee.... sloeeiiee
else:
self.a atate_encoder = torch. mn. Sequeniial (
torch mn. Linear (state_ Shate__ebed__iie
torch. nn. LeakyReLU LeakyReLU (togatiee
build the network that will encode the action features self action, encoder action_encoder
torch Sequential torch mn Linear (action_size. action, @etiio__ebed__ee babed_size)) torch. nn. morch..n.LeakkBeee. LeakyBeUU (negative
# build the network that models the relationship between the state and action embeddings
state_action_encode hidden_in_sie
@etiio__mbed_size state_ for i in range (num_layers): state_action_encoderr apper (
torch mn Li..aa (hiddeniniii hidden_size),
state innte_aactionnnncoerr.co ( torch an aarrdaan...eaaaalllaaeeeeeee.eeo
hidden_in_sizz inze hidden_size
torch. mn Sequential (*state_ action_ encoder))))) # this is a single dense layer because want to focus as much of the useful semantic information possible in the state-action representation
self.star wttttttttttioneenneee
self wtw.tstpteepcetttt.. torch. nn. Linear hidden_size, state_ embed_sizee
Lmage_obseervttinn
aelf next wtw.sstttteeeeeeeee mn Sequential (
mn (atalloorriddttttt_eeeeebee
15

---

mn R..U (implace=Trur)) Linear
saate_embe__iie, conssstangy.compproooo....d size
),
ReLU ReUU (inplace=True),
nn. Linear
conaistancy.projeeciio...se
)
if _ith htt..ooniiieeeyyyprrciioooo.eeee
self consisteency__redccco. mn. Sequential (
Mm.eeU Liner
conssttency.projjcciooo..ee
consaseacy.ccopttioooohhttatllld
),
Linear
conssscacyyprrojccico.....z
)
else:
predictor None
else:
self next_ stxt_staateprojectr torch. m.cce.....iner. saate_smbe__siee
consiistency.proectiinn...e
self consisstoncyy_predccooo nn. Linear
consisteecyyppojjetitn....t
conaisteecyy_rroecttto......
else:
self www..coossttrccypeeeiitt.. None
forward pass through the SFC network follows:
def apr_ (self,,
transitions: Earironsent/ransiiiinnnattt Atb..cosssseacyyyrrcccooo......
bool):
The logic for a forvard pass through the SPR network. Arga:
batch of environment transitions composed of states, actions, and next states with.coassstttcyypeccciooo.....t when using the contrastive objective the consistency prediction is not used, but is when using the distillation objective
transitions:
Returns:
predicted embedding of the next state - in SimSiam paper next state embedding (detached from graph) in SimSian paper dimensionality: (batch, time step)
encode the state, the action, and the state pair St -
embed self state_ (transitions. states)
16

---

int actions_embed self (raaaadatttt actions (1) at
state_action_eme embeds torch. concat
[states embed, actions_ dim
state actaaccctnenncce embed self atate actiine_cdion
predict and project the representation of the next state zi - 4+1
_ext_state__tt self next_ see.t actite
next sext_state_pred = self. www.neeetttteeettteeeeeeet projjector (next_ if with cotsistensst pereecccpppeiitiioncc head:
next_state_pred self consistency (next_atate_pred)
we don't want gradients to back propagaee into the learned # parameters from anything we do with the next state with torch. norch.no-gra();;
# -
# embed the next state
next pext_state_enet embed = self.state, encoder (transitions nwx._extttee _sattss # project the next state embedding into space where it can be # compared with the predicted next state projected_nxt_stee
self wolf.next_state_eeeee nettsstatee.pojeeccor
from the SimSiam paper, this is p and return next_state_pred,
D.2 SAF Reward Network
The architecture of the SAF Reward Network is a subset of the SFC network with the addition of linear to map the state-action representation to predicted rewards. The SFC network is implemented in Py Torch and is initialized following the below build method:
def muild_eaf_netvork network
self, state_size: int, state_embed_ int,
action_size: int, action_embed_size: int,
hidden_size: int,
num_layers: int,
fhallaattivattioo.typ::: str,
image_observaticn bool,
:
Args:
state_size: number of features defining the agent's state space state_embed_size: number of dimensions in the state embedding action_size: number of features defining the agent's actions betion.embed.size: number hidden_size: number of dimensions in the hidden layers of state-action embedding network
dimensions
the action embedding
nua_layers: number of hidden layers used to embed the state-action representation
final ahnatlactivttionttyp the activation used on the final layer image_ whether image observations are used. If image observations are not used, state-space observations are used
17

---

Ltage_liider..mmcchaneee
the number of channels to use in the image encoder's hidden layers
# build the network that will encode the state features if ciage_ooserrations
state_conv_encoe
Sequential
nn Conv2d
state_size [0],
image_hidden num channels, 3,
stride-1
nn ReLU
nn. MaxPool2d (2, 2),
Conv2d image_ Srage_hidden_mmmchame
image_hidden Saciieen_uummmee cum_ceemels
3,
stride-1
),
nn ReLU nn MaxPool2d 2),
Conv2d Srage_hiiden_mmmchhaniee 3, stride=1
imag__hideem.nnnccmaeeee
nn ReLU ()
nn MaxPool2d 2)
conv_out_size torch. flatten(
sttat__cov_eencoeee
torch rand (size= (istastte_sizeg
)) s)ze
self state_ encoder
Sequentiai
state_ conv convdeecoderrrrrroooooooecccceee
nn Linear (conv_out_siaee state__mbed_siio) nn LeakyReRlLJ (ngateeeoo
else:
self sstate_encode torch mn. Sequential (
torch. mn. Linear (state_size, state_ embed_size).
torch. mn LeakyReUU sogative_slope-le-2)
build the network that will encode the action features self action_ torch. nn Sequential torch un Linear (action_size action_embed_size), norch.nn.Leakye.. Leech....Laaakkeuueeeeeeeea (ngaaeaeoeeettineooooooo=21122
# build the network that models the relationship between the # state and action embeddings state_action_eecodd =
hidden_in_size
action_embed_size state_embed_
for i in range :nun_layers):
wot.aaactooo.ecccer.epppoa torch un (widen hidden_size),
state__ction_ append torch. nn heatyyeeelinggtiveee...goeeee
18

---

hidden_in_size hidden_size
self.state_actine acce torch Sequential (#statee_ction_
build the prediction head and select a final activation
(hidden_size, 1) if fiaalactivation.tpee =- "tanh":
self. prediction_head -
self self.fimal
torch.nn Tanh
elif fmal_activation..pp = torch. mn Sigmoid
"sig":
self
else:
self wwt.alaaciiaatiin.noono.
torch.nn.RelU
A forward pass through the SAF network is as follows:
def eaf forf transitions: EaviromeenTraastiiionteaa)))
Args:
transitions:
batch environment transitions composed of states, actions, and next states
Returns:
predicted embedding of the next state - in SimSiam paper next state embedding (detached from graph) in SimSiam paper dimensionality: (batch, time step)
encode the state, the action, and the state- pair # -
states_embed self wtw.sstttteeccctceee (transitions statess states)
-
actions_ embed self acctionnencooeeeo (transitions actiionnnsnatiiss actions)
embes = torch. concat ( # (1,4))22
[states_embed, actions_embed], dim=-1
state actte_action_omtt self. atate_actioonencom
acate_action_embed
return self final iinallactivation
self. predictiiio
E Hyper-parameters
E.1 Train Hyper-parameters
This section specifies the hyper-parameters (e.g. learning rate, batch size, etc) used for the experi- ments and results (Section 6). The SAC, PPO, PEBBLE, and PrefPPO experiments all match those used in [55], [56], and [4] respectively. The SAC and PPO hyper-parameters are specified in Table 3, the PEBBLE and PrefPPO hyper-parameters are given in Table 4, and the seper-parameter, used to train on the REED task are Table
The image-space models were trained on images of size 50x50. For PEBBLE and REED on DMC tasks, color images were used, and for Meta World tasks, grayscale images used. All pixel values were scaled the range (0.0,1.0.
For the basge-based REED methods, we found that larger value was important for the Meta- World experiments compared to the DMC experiments due to slower environment dynamics. In Meta World the differences between subsequent observations are more similar than DMC. In the state-space, the mean cosine similarity between all observations accumulated in the replay buffer was 0.9. For the image-space observations, the mean cosine similarity 0.7. Additionally, for sweep into, due to the similarity in World observations, the slower environment dynamics, and
19

---

difficulty of tasks like sweep into, we found it beneficial to update on the REED objective every 5th update to the reward model order to avoid over fitting on the REED objective and reducing the accuracy of the f preference predictions.
Table 3: Training hyper- ppear--rraetters for SAC [55] and PPO [56].
HYPERRPPARAEEEE
VALUE
SAC
le-3 (cheetah), 5c-4 (walker),
Learning rate
le-4 (quadruped), 3e-4 (MetaWorld)
Batch size
512 (DMC), 1024 (MetaWorld)
Total timesteps
500k, 1M (quadruped, sweep into)
Optimizer
Adam [53]
Critic EMA
5c-3
Critic target update freq.
(B,, Ba)
(0.9, 0.999)
Initial Temperature
0.1
Discount
0.99
PPO
Learning rate
(DMC), 3e-4 (MetaWorld)
Batch size
128 (all but chectah), 512 (cheetah)
500k (cheetah, walker, button press), 1M (quadruped, sweep into)
Total timesteps
(sweep into), 16 (chectah, quadruped), (walker, sweep into)
Envs per worker
Optimizer
Adam [53]
Discount
0.99
Clip range
0.2
Entropy bonus
0.0
GAE parameter
0.92
Timesteps per rollout
250 (MetaWorld) 500 (DMC)
20

---

Table 4: Training nyper-parameters for PEBBLE [4] and PrefPPO [3, 4]. The only hyper- parameere that differs between PEBBLE and PrefPPO is the DMC learning rate. The batch size for the reward network changes based per total feedback amount to match the number of queries M sent to the teacher for labelling each feedback session.
HYPER-PARAMETER
VALUE
Learinng PEBBLE
3e-4
Leamingg rate PrefPPO
5e-4 (DMC), (MetaWorld)
Optimizer
Adam [53]
Segment length /
50 (DMC), 25 (MetaWorld)
1k/00, 500/50, 200/20, 100/10, 50/5 (DMC) 20k/100, 10k/50, 5k/25, 2.5k/12 (MetaWorl))
Feedback amount number queries (M)
20k (walker, cheetah), 30k (quadruped). (MetaWorkd)
Steps between queries (K)
Table 5: Training syper-parameters for REED with the SPR objective [16] (Section 4.1). The REED hyper-parameter were used with both the PEBBLE [4] and PreiPPO [3, 4] preference-learinng algo- rithms. Hlyper-parameters are by environment/task and shared by the two SSL objectives: Distilla- tion versus Contrastive (Section 4.1). Training on the REED task occurred every K steps (specified Table 4) prior to updating on the preference task. The SPR objective predicts future latent states k steps in the future. While our that k vs. & > had no real impact on learning quality for these sate-action feature spaces. For Contrastive REED experiments (state-spar space and image-space observations), т 0.005. In general, image-based experiments REED were less sensitive the syper-parameters than the state-space experiments experiments.
sweep evaluated multiple values for k. we found
ENVIRONMENT LEARNING RATE EPOCHS PER UPDATE
BATCH SIZE
OPTIMIZER
STATE-SPACE OBSERVATIONS
Walker
le-3
20
12
SGD
Cheetah
le-3
20
12
SGD
Quadruped
le-4
20
128
Adam [53]
Button Press
lc-4
10
128
Adam [53]
Sweep Into
5e-5
5
256
Adam [53]
IMAGE-SPACE OBSERVATIONS
Walker
le-4
5
256
Adam [53]
Cheetah
le-4
5
256
Adam [53]
Quadruped
le-4
5
256
Adam [53]
Button Press
le-4
5
256
Adam [53]
5
Sweep Into
1e-4
5
512
Adam [53]
5
Drawer Open
1e-4
5
256
Adam [53]
5
Drawer Close
le-4
5
256
Adam [53]
5
Window Open
le44
5
256
Adam [53]
5
Door Open
lc-4
5
256
Adam [53]
5
21

---

E.2 Architecture Hyper-parameters
The network hyper-parameters (e.g. hidden dimension, number of hidden layers, etc) used for the experiments and results (Section 6) are specified in Table 6.
Table 6: Architecture prper-parameters for SAC [55], PPO [56], the base reward model (used for PEBBLE [4] and PrePPO [3, 4J), the SAF reward model (Section 4.1). and the SPR model (Section 4.1). The hyper-parameters reported here are intended to inform the values to used to initialize the architectures in Appendix D.
model are indicated with "N/A". The SPR model is what REED uses construct the self-supervised temporal consistency task. The base reward model is used with PEBBLE and PrefPPO in Lee et al. [4] and [10]. The SAF reward network is used for all REED conditions in Section 6. The "Final Activation" refers to the activation function used just prior to predicting the reward for a given state action pair. The action embedding sizes are the same for the state-space and image-space observations.
not relevant
HYPER PARAMETER
SAC
PPO
BASE REWARD
SAF REWARD SPR NET
STATE SPACE OBSERVATIONS
20 (walker),
20 (walker),
(cheetah),
17 (cheetah),
State emhed size
N/A
N/A
N/A
78 (quadruped),
78 (quadruped),
30 (MetaWorld)
30 (MetaWorld)
10 (walker),
10 (walker),
6 (cheetah),
6 (cheetah),
Action embed size
N/A
N/A
N/A
12 (quadruped),
12 (quadruped),
(MetaWorld)
MeraWWold)
5 (walker),
(cheetah),
Comparison units
N/A
N/A
N/A
N/A
10 (quadruped),
5 (MetaWorld)
(DMC).
Num. hidden
3
3
3 (MetaWorld)
1024 (DMC),
Units per layer
256
256
256
256
256 (MetaWorld)
Final activation
N/A
N/A
tanh
tanh
N/A
IMAGE- SPACE OBSERVATIONS
(walker),
20 (walker),
17 (cheetah),
17 (chectah),
State embed size
N/A
N/A
N/A
78 (quadruped),
78 (quadruped),
30 (MetaWorld)
30 (MetaWorld)
Comparison units
N/A
N/A
N/A
N/A
128
(DMC),
Num. hidden
(MetaWorld)
1024 (DMC),
Units per layer
256
256
256 (MetaWorld)
Final activation
N/A
N/A
tanh
tanh
N/A
22

---

SAF Reward Net Ablation
Walker Walk
Quadruped Walk
Button Press
Sweep Into
1000
Success Rate
100
Returns
500
50
0
0.0
0.50
0.0
sac
pebble
pebble+saf
pebble+simsiam
cenble+contr.
Figure 5: Ablation of the SAF reward net for malker-walk, quadruped-wall -wak, sweep into, and button press with 500 (walker and quadruped) and 5k (sweep into and button press) teacher-labelled queries with disagreement-basse sampling and the oracle labelling strategy.
We present results ablating the impact of our modified SAF reward network architecture in Table 7, Section 4.1, State-Action Fusion Reward Network, for details. In our ablation, replace the original PEBBLE reward network architecture from [4] with our SAF network and then evaluate on the joint experimental condition with no other changes to reward function learning. We evaluate the impact of the SAF reward network on the walker-walk, quadruped -wak,,,, sweep into, and button press tasks. Policy and reward function learning is evaluated across feedback amounts and labelling styles. All hyper-paramette
match those used in all other experiments the paper (see Appendix E). We compare PEBBLE with the SAF reward network architecture (PEBBLE SAF) against SAC trained the ground truth reward, PEBBLE with the original architecture (PEBBLE), PEBBLE with Distillation REED (PEBBLE+Dist)) and PEBBLE with Contrastive REED (PEBBLE+Contr).
The inclusion of the SAF reward network architecture does not meaningfully impact policy perfor- mance. In general, across domains and experimental conditions, PEBBLE + SAF performs on par with slightly worse than PEBBLE. The lack of performance improvements suggest that the per- formance improvements observed when the auxiliary temporal consistency objective due the auxiliary objective and not the change in network architecture.
23

---

Table The impact of the SAF reward network ablated. Ratio of policy performance learned versus ground truth rewards for walker-walk, quadruped-walk, sweep into, and button press across preference learning methods, labelling methods and feedback amounts (with disagreement sampling).
EQUAL
NOISY
WALKER-WALK
PEBBLE
0.85 (0.
0.76 (0.21)
0.88 (0.16)
0.85 (0.17)
0.79 (0. 18)
0.81 (0. (0.18)
0.83
+SAF
(0.19)
0.62
0.88
0.81 (0.19)
0.74
0.78
1K
-DIST.
0.9 (0.16)))
0.37 (0.2)
0.91 (0.12)
0.89 (0.16)
0.8 (0.17)
0.8% (0.17)
0.86
+CONTR.
(.9.6)))
(0.2)
(0.12)
(0.16)
(0.17)
(0.17)
PEBBLE
(0.188
0.61 (0.17)
0.84 (0.19)
0.75 (0.19)
0.67 (0.19)
0.69 (0.19)
0.72
+SAF
0.68
0,51 (0.13)
0.76 (0.17)
0.68 (0.17)
0.56 (0.15)
0.68 (0.17)
0.65
-DIST
(0.2)
0.71 (0.2)
0.87 (0.2)
0.87 (0.2)
0.82 (0.22)
0.84 (0.2)
0
-CONTR.
0.9
(0.14)
(..17))
0.88 (0.16)
0.88
(0.17)
(0.12)
0.67 (0.2)
(0.17)
0.43 (3.13)
(0. (3)
+SAF
0.53 (0.16)
0.41 (0.15)
0.59 (0.18)
0.53 (0.16)
0.36 (0.1)
0.48 (0.14)
0.48
+DIST.
(0.23)
0.6 (9.16)
0.85 (0.21)
0.8 (0.24)
0.75 (0.26)
(0.24)
0.77
+CONTR.
0.85 (0.19)
0.73 (0.23)
0.85 (0.19)
(0.2)
0.79
0.85 (0.22)
0.82
QUADRUPED-
PEBBLE
0.94 (0.15)
(0.19)
(0.26)
(.0(0.1))
0.93 (0.13)
(0.19)
+SAF
0.97 (0.15)
0.45 (0.17)
(0.22)
0.87 (0.19)
0.76 (0.13)
0.59 (0.14)
0.81
+DIST.
(0.31
0.47 (0.19)
(0.37)
1.3 (0.26)
1.2 (0.18)
0.96 (0.15)
1.09
+CONTR.
(.3
(0.16)
(0.29)
1.3
1.13
PEBBLE
(0.15)
(0.19)
(0.15)
(0.14)
0.73 (0.18)
(0.25)
+SAP
(0.16)
0.44 (0.19)
0.99 (0.23)
0.9 (0.19)
0.63 (0.15)
0.6 (0.2)
0.72
1K
-DIST.
(.190.
(0.14)
(0,22)
(.33)
1.1 (0.21)
1.0 (0.15)
1.04
-CONTR.
1.1
0.63 (0.16)
1,2 (0.29)
(0.19)
1.1 (..19))))
0.83 (0.14)
0.99
PEBBLE
(0.21)
0.48 (0.21)
0.66 (0.2)
0.64 (0.15)
0.47 (0.22)
0.48 (0.23)
0.55
+SAF
0.63
0.4 (0.22)
0.85 (0.14)
0.75 (0.19)
0.56 (0.18)
0.5 (0.19)
0.61
500
+DIST.
(0.2)
(0.22)
-CONTR.
1.1 (0.21)
0.64 (0.11)
1.1 (0.22)
1.1 (0.17)
1.0 (0.17)
0.85 (0.14)
0.97
PEBBLE
0.53 (0.18)
0.36 (0.23)
0.64 (0.15)
0.62 (0.16)
0.46 (0.22)
0.47 (0.21)
0.51
250
+SAF
0.51 0.51 (0.2)
0.36 (0.22)
0.73 (0.18)
0.53 (0.17)
0.53 (0.19)
0.45 (0.
0.52
-DIST.
0.98
0.58 (0.18)
(0.19)
0.79 (0.12)
0.9
0.77
0.84
-CONTR.
0.98 (0.15)
0.58 (0.18)
1.0 (0.19)
0.79 (0.12)
0.9 (0.18)
0.77 (0.16)
0.84
BUTTON PRESS
PEBBLE
0.72 (0.26)
0.57 (0.26)
0.77 (0.25)
0.75 (0.26)
0.68 (0.21)
0.72 (0.24)
0.70
20K
+SAF
0.77 (0.23)
0.72 (0.28)
0.84 (0.23)
0.75 (0.24)
0.78 (0.21)
0.77 (0.22)
0.77
+CONTR.
(0.25)
(0.28)
(0.27)
(0.27)
(0.24)
(0.26)
(0.26)
(0.21)
(0.27)
(0.26)
0.67 (0.24)
(0.26)
10K
+SAF
(0.25)
0.66 (0.26)
0.74 (0.23)
0.71 (0.25)
0.67 (0.19)
0.71 (0.25)
0.70
+CONTR.
0.65 (0.27)
0.61 (0.3)
0.66 (0.27)
0.62 (0.26)
0.6 (0.25)
0.68 (0.28)
0.64
PEBBLE
0.48 (0.21)
0.31 (0. 12)
0.56 (0.25)
0.54 (0.24)
0.59 (0.23)
0.52 (0.23)
0.50
0.55
(0.26)
(0.24)
(0.21)
(0.24)
+CONTR.
0.55 (0.24)
0.54 (0.26)
0.65 (0.27)
0.63 (0.26)
0.57 (0.24)
0.63 (0.28)
0.60
PEBBLE
0.37 (0.18)
0.21 (0.088)
0.44 (0.21)
0.34 (0.15)
0.4 (0.1))
0.34 (0.18)
0.35
2.5%
+SAF
0.58 (0.26)
0.38 (0.17)
0.61 (0.26)
0.54 (0.23)
0.52 (0.21)
0.54 (0.2)
0.53
+CONTR.
0.49 (0.25)
0.42 (0.22)
0.52 (0.24)
0.5 (0.23)
0.44 (0.
0.45 (0.21)
0.47
PEBBLE
0.53 (0.25)
0.26 (0.15)
0.51 (0.23)
0.52 (0.27)
0.47 (0.28)
0.47 (0.26)
0.46
20K
+SAF
0.5 (0.24)
0.36 (0.15)
0.47 (0.22)
0.39 (0.19)
0.49 (0.21)
0.6 (0.21)
0.47
+CONTR.
0.5 (0.22)
0.36 (0.13)
0.41 (0.2)
0.6 (0.22)
0.54 (0.21)
0.61 (0.25)
0.50
PEBBLE
0.28 (0.12)
0.22 (0.13)
0.45 (0.21)
0.33 (0.17)
0.47 (0.25)
0,51 (0.24)
0.38
(D.2)
(0.19)
(0.2)
(0.17)
(0.2)
(0.24)
*CONTR.
0.47 (0.23)
0.3 (0.14)
0.45 (0.24)
0.32 (0.21)
0.42 (0.22)
0.44 (0.21)
0.40
PEBBLE
0.17 (0.09))
0.17 (0.089)
0.28 (0.19)
0.24 (0.15)
0.23 (0.13)
0.22 (0.12)
0.22
5K
+SAP
(0.15)
0.2 (3.1))
0.4 (0.23)
0.38 (0.17)
0.19 (I)
0.41 (0.2)
0.32
-CONTR.
0.34 (0.14)
0.23 (0.19)
(0.24)
0.37 (0.2)
0.4
0.44
0.38
PEBBLE
0.15 (D.086)
0.13 (0.076)
(0.1)
(0.075)
2.5K
+SAF
0.33 (0.19)
0.12 (0.082)
0.32 (0.17)
0.18 (0.09)
0.27 (0.11)
0.22 (0.14)
0.25
-CONTR.
0.21 (0.13)
0.19 (0.22)
0.29 (0.17)
0.17 (0.09)
0.25 (0.15)
0.28 (0.16)
0.23

---

G Image Aug. Task Details
We present results ablating the impact environment dynamics on top of the PEBBLE model to show how much of the REED gains come from encoding environment dynamics versus incor- porating an auxiliary task. In our ablation, we replace the REED auxiliary task with an image- augmentation-based self-supervised learning auxiliary task that compares a batch of image obser- vation states with augmented versions of the same observations using either LSS (Equation 3), or I.C (Fquation 4). We compare the impact of the SSI. data augmentation auxiliary task (PEBBLE Img. Aug.) with the impact of PEBBLE + REED on the image-based PEBBLE preference learning algorithm using the walker-walk, quadruped-walk, and cheetah-run DMC tasks. Policy and reward function learning is evaluated across feedback amounts and with the oracle labeler. All hyper- parameters match those specified in Appendix E, with the exception of those listed below (Table 8).
The Img. Aug. task learns representations of the state, not state-action pairs, as is done in REED, and so the Img. Aug. representations do not encode environment dynamics. To separate out states and actions, the Img. Aug. task uses the SAF reward model architecture. The data augmentations match those used [58].
The PEBBLE Img. Aug. algorithm is the same PEBBLE + REED (Algorithm C), except, in- stead of updating the SPR network using the temporal dynamics task, the SSL Image Augmentation network (Appendix G.2) is updated using the mmage-augmentation task. The state encoder is shared between the reward and the SSL Image Augmentation networks.
The inclusion of an auxiliary task to improve the state encodings does improve the performance of PEBBLE, but does not improve as much with the encoded environment dynamies. This can be seen across feedback types and DMC environments (see Figure 6 and Table 12). The performance improvement against the PEBBLE baseline suggests that having the auxiliary task does have some benefit. However, the performance improvements of REED against the image augmentation aux- iliary task suggest that the performance improvements observed when the auxiliary task encodes environment dynamics gives meaningful improvements.
G.1 Data Augmentation Parameters
This section presents the PEBBLE + Img. Aug. method for creating the augmented image observa- tions. We evaluate the impact of both the "weak" and "strong" image augmentations used in [58]. The augmentation parameters for both the weak and strong styles are given in Table 9,
25

---

Table 8: Image Augmentation hyper-parameters for the PEBBLE reward + SSL model that differ from the parameters outlined in Appendix E.
HYPER- PARAMERR
VALUE
REWARD MODEL
Learning Rate
le-4
Grayscale Images
False
Normalize Images
True
SSL DATA AUGMENTATION MODEL
Learning Rate
5e-5
Grayscale Images
False
Normalize Images
True
Use Strong Augmentations (Table 9)
False
Batch Size
256
Loss
Distillation
Table 9: Data Augmentation copper-aaramttes for PEBBLE+SSL as specified in MOSAIC [58].
HYPER-PARAMETER
VALUE
WEAK AUGMENTATIONS
Random Jitter (p)
0.01
Normalization (jl, .7)
[0.485, 0.456, 0.406), [0.229, 0.224, 0.225]
Random Resize Crop (scale min/max, ratio)
[0.7, 1.0], [1.8, 1.8)
STRONG AUGMENTATIONS
Random Jitter (p)
0.01
Random Grayscale (p)
0.01
Random Horizontal Flip (p)
0.01
Normalization (p,, r))
[0.485, 0.456, 0.406), [0.229. 0.224, 0.225]
Random Gaussian Blur (a min/max, p)
10.1, 2.0), 0.01
Random Resize Crop (scale min/max, ratio)
[0.6. 1.0]. [1.8, 1.8]
26

---

G.2 SSL Data Augmentation Architecture
The SSL Image Augmentation network is implemented in PyTorch. The architecture for the con- Eedency-prediceee comes from [58]. The image encoder architecture comes from [59]. The SSL network is initialized as follows:
def build_ sealld_ss_.eetork(ees (self,
state_size: L. [int], state_ int,
consisteeccy.rrojetioo...tt int, caniiatency..comrrssss.ddddd size: int,
utth.consistencyyprreiitttt head: bool, maage_hiddmmnmmmchaee
int):
The network architecture and build logic the complete the REED self -sueerviiee temporal consistency task based on SPR. Args:
state_size, dimensionality of the states
state_embed_size: number of dimensions in the state embedding number of units used compare the
conisstecyy.rrojcciio...e predicted and target latent next
state
cossistacy..copprriooo...ooo. hideen number of units in the hidden
layers of the pext_stato.projector and the consistancy_predctor when using the contrastive of objective the consistency prediction is not used, but is when using the distillation objective
with with.ccosistteeyyprediiioo....e
hmage_hidden_nun_c
the number of channels to use in the image encoder's hidden layers
Build the network that will encode the state features. ciate__cov__encode un un. Sequential nn un.ReLU() un Ma.Maxool22d (2, 2),
(onved [0], image_ hidden nun channellss, stride=1),
nn nn. Conv2d
num_ channels, image_ fece__hiddn__nue channels, 3, stride=1), un ReLU (), nn MaxPool2d (2, 2),
Conv2d
image_ bagge_hidden_nun.oo channels, mmage_hidden.em num_ channels. 3, stride=1),
ReLU nn MaxPoool22 2)
conv_out_size torch flatten welf.sttae_encode mn un. Sequential
(torch. rand (size=[1] list (state_siz
Linear (conv_ state_ smbee__emed___ee)) nn
slope--e-2)
self munsiistency_projectt
Sequential( и W)),
# Rearrange ('B emaette_embd__siee) nn. ReLU (inplace=True), nn
# Rearrange ('BT d H W -> BT (d H w)*
nn nn. Linear (state,
cetiatteeccyccoppriooooo wtt..y.ooooooooiin..iiiiii....i. hidden_size), nn. ReLU (inplace=
27

---

Linear (onnisseayy.coaprroooo...dddaaae www.Layrrorrmcooaaae. [aapppllcccccaaaaaaaaaarrreoooaa i
from: otapa///eeaaaaaaaaaaaaaaaaaaaaaa if oth..conaastteyy..rddciooo..odd.
self oeI..ccassstteeyypprecctttorrrr" in.Sequential
mn.eell(aaplaaacTTTee)) mn oai.herrcconsstteepppppeeeooiiio nn m_AAeul((npplacc=rrrraa mn else: ww..ahoccccconnttseeeec...oooorp
condistency.poojeriio.....
self consisteencypprddictor None
A forward pass through the SSL network is follows: def ssl_forvard (self, observations: Rurhamanneebbeerrttiisttttt aath.conniitttccyppeelcctooooo_t bool):
The logic for forward pass through the SSL network.
Args: observations: a batch of environment raw observations and augmented observations
with wtth.cosiisteeyyppeeicioooo,
when using the contrastive objective the consistency prediction not used, but when using the distillation objective
Returns:
predicted embedding the augmented state the augmented state embedding (detac) dimensionality: (batch, time step)
Encode the observations.
observations_embed self. state_ (observations states)
Predict the augmented observations.
if aith.consistterrypprcctttt... head:
augaeneeel_bberrvtttooo.prpr self colf.coassstttncypppeeicc... wwelf. state_ (obse
else:
aggaented_.bberrrttiooo..rr pttp:/ppraasssasssaaaaapopppppp (orseraatiiooeeeesse.eeo _mmbedd
self state,
we don't gradients back-propagat into the learned parameters from anything with torch. torch.no_ee no_grad
# embed the augmented observation
= colf.state state_encoder (observations augmented_states) # project the augmented observation embedding into space where it can be compar.
projectel_aguantcclbbberaaat outp
self peef. (oggeeteoorcmgeeeetcdccoaaaaaaa
return orgpapa.....aaiaoooo.oooooaanrer
28

---

G.3 Image Aug. Auxiliary Task Results
We present results comparing the impact of REED's temporal auxiliary task and the Image Aug. auxiliary task. Learning curves (Figure 6) and normalized returns (Table 10) are provided for image- space observation walker-wall walk, cheetah-run, and quadruped tasks across different amounts of feedback. We compare the contributions of the Image Aug. auxiliary task PEBBLE against SAC trained on the ground truth reward, PEBBLE, PEBBLE Distillation REED (+Dist.), and PEBBLE Contrastive REED (+Contr.). Results are reported for the Image Aug. task using the distillation objective (Equation 3) as +Dist.+lmg. Aug.
The inclusion of the Image Aug. auxiliary task improves performance relative to PEBBLE, but does reach the level of performance achieved by REED. The gap policy performance between REED and the Image Aug. auxiliary task suggests that encoding environment dynamies in the reward function and not including auxiliary task that trains all policy experiences is the cause of the performance gains observed from REED.
Walker Walk
Quadruped Walk
Cheetah Run
1000
50
500
0
1000
100
500
0
1000
250
500
0
1000
500
500
1000
1000
500
0.0
0.50
D.0
0.5
Steps (x1e6)
Steps (x1e6)
Steps (x1e6)
pebble
petéle-dist:
petblei-cont
petble petble_img.aug jingaug.
Figure 6: Episode returns learning curves for walker walk, quadruped walk, and cheetah run across preference-based RL methods and feedback amounts for image-based observations. The oracle labeller is used generate preference feedback. Mean policy returns are plotted along the y-axis with number of steps (in units of 1000) along the x-axis. There is one plot per environment (grid columns) and feedback amount (grid rows) with corresponding results per learning methods in each plot. The learning methods evaluated are SAC trained on the ground truth reward, PEBBLE, PEBBLE with the Image Aug. auxiliary task (pehble+img. aug.), PEBBLE Distillation REED (pebble+dist.), and PEBBLE + Contrastive REED (pebble+contr.). From top to bottom, the rows correspond to 2.5k, 5k, 10k, and 20k pieces of teacher feedback. From left right, the columns correspond to walker walk, quadruped walk, and cheetah run.
29

---

Table 10: Ratio of policy performance learned versus ground truth rewards for walker-walk, cheetah-run, and quadruped-walk across feedback amounts (with disagreement sampling). The results reported means (standard deviations) over 10 random seeds.
DMC
TASK
METHOD
50
100
250
500
1000
MEAN
PEBBLE
0.06 (0.02)
0.07 (0.02)
0.09 (0.02)
0.11 (0.03)
0.11 (0.03)
0.09
WALKER WALK
+DIST.
0.18 (0.03)
0.33 (0.10)
0.40 (0.09)
0.57 (0.16)
0.68 (0.23)
0.43
+CONTR.
0,28 (0.12)
0.35 (0.13)
0.46 (0.14)
0.58 (0.14)
0.61 (6)16)
0.46
+DIST.+MMG.AG.
0.06 (0.02)
0.10 (0.02)
0.16 (0.02)
(0.03)
0.46 (0.11)
0.20
PERBLE
0.28 (0.23)
0.23 (0.19)
0.23 (0.15)
0.23 (0.19)
0.47 (0.09)
0.29
QUADRUPED WALK
+DIST.
0,56 (0.15)
0.59 (0.16)
0.61 (0.12)
0.71 (0.14)
0.72 (0.19)
0.64
*CONTR.
0.53 (0.16)
0.64 (0.10)
0.69 (0.17)
0,73 (0.22)
0.71 (0.16)
0.66
+IGST.+ING.AUG.
0.43 (0.19)
0.35 (0.
0.42 (0.17)
0.68 (0.
0.62 (0.16)
0.50
PERBLE
0.01 (0.01)
0.02 (0.01)
0.02 (0.02)
0.02 (0.03)
0.03 (0.02)
0.02
CHEBTAII-UUN
+DIST.
(0.03)
(0.04)
(0.02)
0.12 (0.04)
(0.07)
+CONTR.
0.05 (0.02)
0.14 (0.04)
0.14 (0.04)
0.23 (0.09)
0.16 (0.06)
0.14
DIST.+IMG.AUG
0.02 (0.01)
0.05 (0.02)
0.14 (0.05)
0.11 (0.04)
0.12 (0.05)
0.09
30

---

H
Complete Joint Results
Results are presented for all tasks, feedback amounts, and teacher labelling styles. The benefits of SPR rewards are greatest: 1) for increasingly more challenging tasks, 2) when there limited feedback available, and 3) when the labels are increasingly noisy.
H.1 State-space Observations Learning Curves
Learning curves provided for walker-walk (Figure 7), cheetah-run (Figure 8), quadruped-waid (Figure 9), button press (Figure 10), and sweep into (Figure 11) across feedback amounts and all teacher labelling strategies for state-space observations.
aQual
misake
mnyopic
noay
skip
1820
520
520
300
500
320
1020
1000
3.0
0,500
0.500
030
0.300
0.300
3.5
580 2po pebblee
pebble-rconte
prefpon
prefopo timolam
pretspo-contr
Figure 7: Returns learning curves for walker-walk across preference-basee RL methods, labelling style, and feedback amount for state-space observations. Mean policy returns are plotted along the y-axis with number of steps (in units of 1000) along the x-axis. There is one plot per labelling style (grid columns) and feedback amount (grid rows) with corresponding results per learning methods in each plot. From top to bottom, the rows correspond to 50, 100, 500, and 1000 pieces of teacher feedback. From left right, the columns correspond equal, noisy, mistake, myopic, oracle, and skip labelling styles (see Appendix B for details).
31

---

mistake
320
100
520
1520
200
1000
1000
520
0.0
Staps (xle6)
Stapa (clue)
Steps (cle6)
Staps (x2e6)
Staps (x1e6)
Staps (xle5)
5po
pobblc
pebotc-Sinsiam
posbic-contr.
protppo
profsgo-simmtamm
prefspom contr.
Figure 8: Returns learning curves for cheetah-run across preference-based RI. methods, labelling style, and feedback amount for state-space observations. Mean policy returns are plotted along the y-axis with number of steps (in units of 1000) along the x-axis. There is one plot per labelling style (grid columns) and feedback amount (grid rows) with corresponding results per learning methods each plot. From top to bottom, the rows correspond to 50, 100, 500, and 1000 pieces of teacher feedback. From left to right, the columns correspond to equal, noisy, mistake, myopic, oracle, and skip labelling styles (see Appendix for details).
32

---

mistak
1020
320
1020
200
520
500
1020
5:0
Steps (x1e6)
Staps (xlu6)
Staps (x1e5)
Staps (xle6)
Staps (x2e5)
Stups (cle5) pretspo-contr:
500
pobblc
pebatc-Sinsiam
contic-contr.
protppo
profsgo-slmatmm
Figure 9: Returns learning curves for quadruped-walk across preference-based RI. methods, la- belling style, and feedback amount for state-space observations. Mean policy returns are plotted along the y-axis with number of steps (in units of le6) along the x-axis. There is one plot per la- belling style (grid columns) and feedback amount (grid rows) with corresponding results per learning methods each plot. From top to bottom, the rows correspond to 50, 100, 500, 1000, and 2000 pieces of teacher feedback. From left to right, the columns correspond to equal, noisy, mistake, myopic, oracle, and skip labelling styles (see Appendix for details).
33

---

equal
myopic
noisy
2500
50
5000
100
10000
50
20000
50
0.2
0.9.0
0.5.0
0.5.0
0.5.0
D.5.0
0.5
Stops (x1e6)
Stops (xle6)
Stops (c1e6)
Steps (c1e6)
Steps (c1e6)
Steps (c16))
pgo
prefppo
prefppo+contr:
Figure 10: Success rate learning curves for button press across preference-based RL methods, belling style, and feedback amount for state-space observations. Mean success rates are plotted along the y-axis with number of steps (in units 1000) along the x-axis. There one plot per la- belling style (grid columns) and feedback amount (grid rows) with corresponding results per learning methods each plot. From top bottom, the rows correspond to 2.5k. 5k, 10k, and 20k pieces of teacher feedback. From left right, the columns correspond to equal, noisy, mistake, myopic, cle, and skip labelling styles (see Appendix B for details).
34

---

equal
nois
2500
50
5000
10000
50
20000
50
Steps (x166)
Steps (xle6)
Steps (x
Steps (xle6)
Steps (1166)
Steps (Xle6)
pgo
prefppo
prefppo+contr
Figure 11: Success rate learning curves for sweep into across preference-based RL methods, belling style, and feedback amount for state-space observations. Mean success rates are plotted along the y-axis with number of steps (in units 1000) along the x-axis. There one plot per la- belling style (grid columns) and feedback amount (grid rows) with corresponding results per learning methods each plot. From top bottom, the rows correspond to 2.5k. 5k, 10k, and 20k pieces of teacher feedback. From left right, the columns correspond to equal, noisy, mistake, myopic, ora- cle, and skip labelling styles (see Appendix B for details).
35

---

H.2 State-space Normalized Returns
The normalized returns (see Section 6 and Equation 5) for walker-walk, quadruped-walk, cheetah- run, button press, and sweep into, across all teacher labelling styles and a larger range feedback amounts for state-space observations, are given in Table 11.
36

---

Table 11: Ratio of policy performance learned versus ground truth rewards for walker-walk, cheetah-run, quadruped-wall, sweep into, and button press across preference learning methods, labelling methods and feedback amounts (with disagreement sampling). The results are reported as means (standard deviations) over 10 random seeds.
FERDRACK METROD
ORACLE
MISTAKE
EQUAL
SKIP
MYOPIC
NOISY
MEAN
WALKER WALK
PEBBLE
0.85 (0.17)
0.76 (0.21)
0.88 (0.16)
0.85 (0.17)
0.79 (0.18)
0.81 (0.18)
0.83
+DIST.
0.9 (0.16)
0.77 (0.2)
0.91 (0.12)
0.89 (0.16)
0.8 (0.17)
0.88 (0.17)
0.86
+CONTR.
0.77 (0.2)
0.91 (0.12)
0.89 (0.16)
(0.17))7)
0.88 (0.17)
0.86
IK
PREFPPO
(0.029)
+DIST.
1.1 (0.025)
0.92 (0.05)
1.0 (0.031)
1.0 (0.02)
0.99 (0.035)
0.96 (0.046)
1.00
+CONTR.
(0.029)
(0.043)
(0.021)
(0.028)
(0.033)
(0.045)
1.00
PEBBLE
0.74 (0.18)
0.61 (0.17)
0.84 (0.19)
0.75 (0.19)
0.67 (0.19)
0.69 (0.19)
0.72
DIST.
0.86 (0.2)
0.71 (0.2)
0.87 (0.2)
0.87
0.82 (0.22)
0.84 (0.2)
0.83
+ONTR..
0.9 (0.17)
0.81 (0.19)
0.9 (0.14)
0,9 (0.17)
0.88 (0.16)
0.88 (0.18)
0.88
500
PREFPPO
0.95 (0.052)
0.83 (0.087)
0.95 (0.044)
0.96 (0.058)
0.89 (0.076)
0.88 (0.069)
0.91
+DIST.
0.88 (0.074)
0.81 (0.084)
0.98 (0.025)
0.95 (0.027)
(0.073)
0.9 (0.069)
0.90
+CONTR.
0.93 (0.061)
0.85 (0,077)
0.95 (0.046)
0.92 (0.059)
0.82 (0.085)
0.77 (0.098)
0.88
PEBBLE
0.52 (0.17)
0.46 (0.15)
0.67 (0.2)
0.54 (0.15)
0.46 (0.13)
0.45 (0.14)
0.52
DIST.
0.74 (0.2)
0.65 (0.21)
0.79 (0.22)
0.74 (7.400.1
0.64 (0.21)
0.8 (0.25)
0.73
+CONTR.
(0.2)
(0.2)
(0.18)
(0.2)
(0.19)
(0.21)
200
PREXPPO
0.93 (0.058)
(0.076)
0.93 (0.027)
0.88 (0. (0.045)
000079)
(6)
+DIST.
0.78 (0.087)
0.68 (0.11)
0.81 (0.062)
0.77 (0.081)
0.77 (0.077)
0.78 (0.053)
0.77
+CONTR.
0.81 (0.088)
0.81 (0.079)
0.86 (0.044)
0.85 (0.055)
0.76 (0.13)
0.74 (0.1)
0.80
PEBBLE
0.34 (0.11)
0.31 (0.11)
0.37 (0.1)
0.37 (0.12)
0.29 (0.085)
0.41 (0.13)
0.35
+DIST.
(0.23)
(0.2)
(0.25)
(0.23)
(0.19)
(0.24)
+CONTR.
0.78 (0.21)
0.69 (0.22)
0.74 (0.22)
0.74 (0.
0.68 (0.19)
0.74 (0.22)
0.73
100
PREPPPO
0.68 (0.08)
0.59 (0.093)
0.73 (0.065)
0.73 (0.065)
0.58 (0.11)
0.68 (0.072)
0.67
+DIST.
0.67 (0.08)
0.63 (0.11)
0.71 (0.075)
0.71 (0.084)
0.63 (0.099)
0.63 (0.094)
0.66
+CONTR.
0.72 (0.084)
0.49 (0.14)
0.71 (0.063)
0.65 (0.091)
0.64 (0.1)
0.58 (0.13)
0.63
PEBBLE
(0.1)
(0.11)
(0.11)
DIST.
0.66 (0.24)
0.44 (0.13)
0.6 (0.21)
0.64 (0.24)
0.44 (0.12)
0.48 (0.15)
0,54
+CONTR.
(0.22)
0.44 (0.11)
0.72 (0.22)
(0.22)
0.54 (0.16)
(0.17)
0.58
50
PREFPPO
0,51 (0.13)
0.41 (0.18)
0.57 (0.12)
0.59 (0.11)
U.51 (0.14)
0.51 (0.14)
0.52
+DIST.
(0.13)
0.41
0.6 (0.13)
0.56 (0.13)
0.58
0.48 (0.13)
0.54
+CONTR.
0.58 (0.12)
0.54 (0.15)
0.62 (0.12)
0.63 (0.11)
0.5 (0.13)
0.57 (0.12)
0.57
Continnes on next puge...

---

from previous page
FEEDBACK
METHOD
ORACLE
MISTAKE
EQUAL
SKIP
MYOPIC
NoISY
MEAN
CHEFTAH-RUN
PEBBLE
(0.18)
(0.2)
(0.17)
(0.16)
(0.18)
+DSST.
0.93 (0.2)
0.83 18)
0.88 (0.14)
0.92 (0.16)
1.0 (0.18)
0.86 (0.21)
0.90
+CONRR.
(0.
0.84 (0.18)
0.89 (0.17)
(0.17)
(0.20)
(0.13)
IK
PREFPPO
0.71 (0.064)
0.72 (0.086)
0.76 (0.069)
0.71 (0.076)
0.75 (0.066)
0.69 (0.093)
0.72
+DIST.
0.67 (0.054)
(0.076)
0.75 (0.08)
0.66 (0.064)
0.79 (0.084)
0.65 (0.081)
0.70
+CONTR.
0.7 (0.069)
0.8 (0.11)
0.72 (0.07)
0.71 (0.065)
0.78 (0.089)
0.65 (0.083)
0.73
PEBBLE
0.86 (0.14)
(0.18)
0.86 (0.19)
0.71 (0.16)
0.79 (0.16)
0.71 (0.15)
0.79
(0.22)
(0.18)
0.93 (0.15)
(0.15)
0.85 (0.14)
(0.19)
+CONTR.
0.94 (0.21)
0.72 (0.14)
0.9 (0.21)
0.89 (0.18)
0.93 (0.18)
0.82 (0.16)
0.87
500
PREFPPO
0.62 (0.043)
0.63 (0.047)
0,77 (0.089)
0.66
0.66 (0.04)
(0.09)
0.67
+DIST.
0.67 (0.062)
0.74 (0.14)
0.7 (0.072)
0.63 (0.069)
0.67 (0.076)
0.68 (0.081)
0.68
(0.062)
(0.072)
(0.082)
(0.065)
(0.047)
(0.073)
PEBBLE
0.71 (0.23)
0.62 (0.22)
0.71 (0.24)
0.57 (0.2)
0.75 (0.18)
(0.22)
0.66
+DIST.
0.77 (0.28)
0.61 (0.19)
0.77 (0.22)
0.79 (0.25)
0.76 (0.25)
0.65 (0.27)
0.72
+CONTR.
0.73 (0.22)
0.67 (0.21)
0.83 (0.25)
0.8 (0.24)
0.83 (0.19)
0.76 (0.23)
0.77
200
33
PREXPPO
0.57 (0.042)
0.13 (0.13)
0.66 (0.059)
0.54 (0.052)
0.64 (0.073)
0.56 (0.099)
0.62
(0.066)
0.53 (0.049)
(0.054)
(0.085)
+CONTR.
0.62 (0.071)
0.49 (0.046)
0.61 (0.06)
0.63 (0.1)
0.56 (0.062)
0.58 (0.048)
0.58
PEBBLE
0.4 (0.14)
0.4 (0.13)
0.61 (0.22)
0.47 (0.2)
0.55 (0.21)
0.42 (0.14)
0.48
+DIST.
0.69 (0.26)
0.59 (0.22)
0.79 (0.28)
0.65 (0.29)
0.67 (0.26)
0.53 (0.21)
0.65
+CONTR.
0.64 (0.28)
0.65 (0.21)
0.78 (0.21)
(0.29)
0.81 (0.26)
0.72 (0.25)
0.72
100
PREFPPO
(0.036)
(0.061)
(0.051)
(0.054)
(0.054)
DIST.
0.49 (0.036)
0.47 (0.088)
0.49 (0.046)
0.49 (0.038)
0.52 (0. (0.04)
0,52 (0.081)
0.50
+CONTR.
0.54 (0.037)
0.51 (0.06)
0,59 (0.085)
0.5 (0.059)
0.59 (0.061)
0.45 (0.039)
0.53
PEBBLE
0.35 (0.11)
0.26 (0.098)
0.39 (0.14)
0.39 (0.12)
0.4 (D.15)
0.24 (0.089)
0.34
DDSS.
0.63 (0.23)
0.59 (0.27)
0.69 (0.25)
0.62 (0.23)
0.68 (0.31)
0.46 (0.22)
0.61
+CONTR.
0,7 (0.28)
0.51 (0.21)
0.72 (0.28)
0.53 (0.2)
0.66 (0.28)
0.66 (0.28)
0.63
PREFPPO
0.5 (0.066)
0.49 (0.07)
0.44 (0.076)
0.4 (0.038)
0.34 (0.062)
0.3 (0.066)
0.41
+DIST.
0.44 (0.041)
(0.039)
0.44 (0.082)
0.3 (0.063)
(0.085)
(0.11)
0.41
+CONTR.
0.47 (0.051)
0.42 (0.039)
0.47 (0.093)
0.43 (0.038)
0.46 (0.039)
0.31 (0.061)
0.43
QUADRUPED-WALX
(0.15)
0.55 (0.19)
1.1 (0.26)
(0. 16)
0.93 (0.13)
(0.19)
2K
+DIST.
1.3 (0.31)
0.47 (0.19)
1.4 (0.37)
1.3 (0.26)
1.2 (0.18)
0.96 (0.15)
1.09
Continacy page...

---

from previous page
FEEDBACK
METHOD
ORACLE
MISTAKE
EQUAL
SKIP
MYOPIC
NOISY
MEAN
+CONTR.
1.3 (0.25)
0.7 (0.16)
1.2 (0.24)
1.3 (0.29)
1.3 (0.28)
1.0
1.13
PREFPPO
(0.18)
(0.18)
(0.25
1.07
+DIST.
(0.22)
1.0 (0.2)
1.2 (0.23)
1.1 (0.24)
1.1 (0.26)
0.91 (0.1)
1.06
+CONTR.
1.0 (0.24)
(0.2)
1.4 (0.3)
1.2 (0.29)
1.1 (0.38)
(0.32)
1.28
PEBBLE
0.86 (0)15)
0.53 (0.19)
0.88 (0. 15)
0.91 (0.14)
0.73 (0.18)
0.48 (0.25)
0.73
+DIST.
(0.19)
0.59 (0.
1.2 (0.22)
(0.3)
1.1 (0.21)
(0.15))))
1.04
+CONTR.
1.1 (0.19)
0.63 (0.16)
1.2 (0.29)
1.1 (0.19)
1.1 (0.19)
0.83 (0.14)
0.99
PREPPPO
(0,17)
0.88 (0.17)
1.1 (0.15)
0.98 (0.21)
0.89 (0.18)
0.83 (0.17)
0.92
+DIST.
(0.21)
0.88 (0.23)
(0,27.
1.2 (0.23)
1.1 (0.26)
1.1 (0.16)
1.11
+CONTR.
1.1 (0.19)
0.68 (0.28)
1.2 (0.25)
1.1 (0.2)
0.82 (0.31)
0.56 (0.25)
0.82
PERBLE
0.56 (0.21)
0.48 (0.21)
0.66 (0.2)
0.64 (0.15)
0.47 (0.22)
0.48 (0.23)
0.55
+DSST.
1.1 (0.21)
0.58 (0.16)
1.2 (0.24)
1.0 (0.22)
1.0 (0.19)
0.68 (0.16)
0.93
+CONTR.
(0.21)
0.64 (0.11)
(0.22)
(0.17)
(..17)
(0.14)
500
PREPPPO
0,8 (0.18)
0.81 (0.22)
0.96 (0.12)
0.72 (0.18)
0.74 (0.24)
(0.17)
0.82
+DIST.
1.1 (0.2)
0.76 (0.19)
1.0 (0.2)
1.1 (0.25)
0.89 (0.21)
0.81 (0.25)
0.95
+CONTR.
1.1 (0.21)
0.63 (0.25)
0.9 (0.28)
0.89 (0.22)
0.88 (0.16)
1.5 (0.48)
0.95
39
PEBBLE
0.54 (0.19)
0.49 (0.22)
0.64 (0.15)
0.46 (0.2)
0.43 (0.22)
0.48 (0.23)
0,51
17)
0.57 (0.17)
(0.16)
(0.14)
0.76 (0.11)
(0.15)
+ONTR.
0.95 (0.15)
0.53 (0.16)
0.86 (0.
0.88 (0.14)
0.77 (0.12)
0.74 (0.16)
0.79
200
PREFPPO
(0.23)
0.59 (0.28)
0.82 (0.17)
0.65 (0.27)
0.8 (0.25)
0.82 (0.27)
0.73
+DIST.
0.89 (0.15)
0.79 (0.23)
0.95 (0.18)
0.87 (0.16)
0.76 (0.26)
0.79 (0.17)
0.84
(0.33)
(0.15)
(0.21)
(0.18)
(0.24)
PEBBLE
(0.17)
(0.14)
(0.22)
DIST.
0.78 (0.
0.54 (0.2)
0.98 (0.19)
0.72 (0.15)
0.75 (0.16)
0.67 (0.
0.74
+CONTR.
0.67 (0.18)
0.47 (0.2)
0.89 (0.14)
0.76 (0.15)
0.79 (0.17)
(0.19)
100
PREPPPO
0.56 (0.31)
0.81 (0.31)
0.66 (0.22)
0.62 (0.28)
0.51 (0.31)
0.6 (0.29)
0.63
0.76
(0.26)
+CONTR.
0.91 (0.19)
0.52 (0.42)
0.61 (0.32)
0.6 (0.27)
0.99 (0.21)
0.53 (0.37)
0.69
PEBBIEE
0.38 (0.26)
0.4 (0.23)
0.49 (0.2)
0.42 (0.25)
0.42 (0.26)
0.36 (0.26)
0.41
+DIST.
0.65 (0.16)
0.47 (0.24)
0.77 (0.14)
0.68 (0.18)
0.67 (0.2)
0.56 (0.19)
0.63
+CONTR.
0.83 (0.12)
0.49 (0.23)
0.8 (0.14)
0.65 (0.18)
0.69 (0.16)
0.62 (0.19)
0.68
50
PREPPPO
0.68
0.64 (0.28)
0.58
(0.26)
(0.3)
0.56
+DSST.
0.9 (0.19)
0.71 (0.29)
0.9 (0.25)
0.83 (0.18)
0.68 (0.26)
0.77 (0.29)
0.80
(0.34)
(0.29)
(0.27)
(0.16)
(0.44)
(0.35)
Continues page...

---

from previous page
FEEDBACK
METHOD
ORACLE
MISTAKE
EQUAL
SKIP
MYOPIC
NoISY
MEAN
BUTTON PRESS
PEBBLE
(0.26)
(0.25)
(0.26)
(0.24)
+CONTR.
0.65 (0.25)
0.61 (0.28)
0.67 (0.27)
0.67 (0.27)
0.67 (0.24)
0.69 (0.26)
0.66
20X
PREPPPO
0.18 (0.03)
0,18 (0.04)
0.21 (0.03)
0.18 (0.03)
0.17 (0.04)
0.17 (0.04)
0.18
+CONTR.
0.22 (0.03)
0.17 (0.03)
0.22 (0.02)
0.17 (0.03)
0.19 (0.03)
0.17 (0.04)
0.19
PEBBLE
0.66 (0.26)
0.47 (0.21)
0.67 (0.27)
0.63 (0.26)
0.67 (0.24)
0.6 (0.26)
0.62
(0.27)
(0.3)
(0.26)
(0.25)
(0.28)
10K
PREFPPO
(0.03)
0.14 (0.04)
(0.03)
(0.04)
(0.03)
(0.04)
0.17
+CONTR.
0.15 (0.04)
0.12 (0.05)
0.18 (0.03)
0.17 (0.03)
0.17 (0.03)
0.16 (0.03)
0.16
PERBLE
0.48 (0.21)
0.31 (0.12)
0.56 (0.25)
0.54 (0.24)
0.59 (0.23)
0.52 (0.23)
0.50
+CONTR.
0.55 (0.24)
0.54 (0.26)
0.65 (0.27)
0.63 (0.26)
0.57 (0.24)
0.63 (0.28)
0.60
PREFPPO
0.15 (0.04)
0.13 (0.05)
0.19 (0.03)
0.16 (0.04)
0.16 (0.04)
0.14 (0.04)
0.15
+CONTR.
0.14 (0.04)
0.13 (0.05)
0.18 (0.03)
0.14 (0.04)
0.14 (0.04)
0.14 (0.03)
0.14
PEBBLE
0.37 (0.18)
0.21 (0.088)
0.44 (0.21)
0.34 (0.15)
0.4 (0.17)
0.34 (0.18)
0.35
+CONTR.
0.49 (0.25)
0.42 (0.22)
0.52 (0.24)
0.5 (0.23)
0.44 (0.17)
0.45 (0.21)
0.47
40
2.5K
PREPPPO
(0.05)
(0.05)
(0.05)
(0.05)
CONTR.
0.14 (0.04)
0.11 (0.05)
0.15 (0.04)
0.1 (0.04)
0.14 (0.04)
0.13 (0.04)
0.13
SWEEP-INTO
PEBBLE
(0.25)
0.26 (0.15)
(0.23)
(0.27)
(0.26)
+CONTR.
0.5 (0.22)
0.36 (0.13)
0.41 (0.2)
0.6 (0.22)
0.54 (0.21)
0.61 (0.25)
0.50
PREPPPO
0,16 (0.046)
0.14 (0.047)
0,16 (0.069)
0.18 (0.065)
0.19 (0.063)
0.08 (0.026)
0.15
+CONTR.
0.23 (0.064)
0.11 (0.042)
0.2 (0.058)
0.2 (0.051)
0.19 (0.054)
0.1 (0.034)
0.17
PEBBLE
0.28 (0.12)
0,22 (0.13)
0.45 (0.21)
0,33 (0.17)
0.47 (0.25)
0.51 (0.24)
0.38
(0.23)
0.3 (0.14)
(0.24)
(0.21)
(0.22)
(0.21)
10K
PREFPPO
(0.048)
(0.064)
(0.05)
(0.049)
(0.055)
(0.024)
+CONTR.
0.11 (0.034)
0.12 (0.046)
0.15 (0.046)
0.16 (0.056)
0.11 (0.052)
0.054 (0.029)
0.12
PERBLE
0.17 (0.099)
0.17 (0.089)
0.28 (0.19)
0.24 (0.15)
0.23 (0.13)
0.22 (0.12)
0.22
+CONTR.
0.34 (0)
0.23 (0.19)
0.52 (0.24)
0,37 (0.2)
0.4 (0.24)
0.44 (0.18)
0.38
PREFPPO
0.1 (0.039)
0.078 (0.027)
0.1 (0.032)
0.092 (0.025)
0.1 (0.038)
0.051 (0.019)
0.09
+CONTR.
(0.052)
(0.034)
(0.032)
(0.076)
(0.026)
(0.026)
2.5K
PEBBLE
0.15 (0.086)
0.13 (0.076)
0.16 (0.1)
0.16 (0.088)
0.18 (0.075)
0.25 (0.11)
0.17
Continaes on next page...

---

from previous page
FEEDBACK
METHOD
ORACLE
MISTAKE
EQUAL
SKIP
MYOPIC
NoIsY
MEAN
+CONTR.
0.21 (0.13)
0.19 (0.22)
0.29 (0.17)
0.17 (0.092)
0.25 (0.15)
0.28 (0.16)
0.23
2.5K
PREFPPO
0.092 (0.032)
0.15 (0.051)
(0.049)
(0.035)
0.032 (0.022)
+CONTR.
0.058 (0.019)
0.048 (0.018)
0.11 (0.032)
0.072 (0.035)
0.07 (0.016)
0,036 (0.017)
0.07

---

H.3 Image-space Observations Learning Curves
Learning curves are provided for the walker-walk, cheetah-run, and quadruped-walk DMC tasks (Figure 12), and for the button press, sweep into, drawer open, drawer close, window open, and door close MetaWorke tasks (Figure 13) across feedback amounts for image-space observations. For all image-based results, the orable labeller is used provide the preference feedback.
Walker Walk
Quadruped Walk
Cheetah Run
1000
50
500
0
1000
100
500
1000
250
500
0
1000
500
500
0
1000
1000
500
0.0
0.50
D.0
0.5
Steps (x1e6)
Steps (x1e6)
Steps (x1e6)
pobble
pebble+ dist.
pcotic++contr
Figure 12: Episode returns learning curves for walker walk, quadruped walk, and cheetah run across preference-based RL methods and feedback amounts for image-based observations. The oracle labeller is used to generate preference feedback. Mean policy returns are plotted along the y-axis with number of steps (in units of 1000) along the x-axis. There is one plot per environment (grid columns) and feedback amount (grid rows) with corresponding results per learning methods in each plot. From top bottom, the rows correspond to 2.5k, 5k, 10k, and 20k pieces of teacher feedback. From left to right, the columns correspond to walker walk, quadruped walk, and cheetah run.
42

---

Button Press
Sieeep Into
Drawer Open
Drawer Close
Window Open
Door Open
100
2500
5000
50
1000
30
20000
50
0.0
0.50
10.0
0'.00
0.500
0.00
0.5
Stops (x1e6)
Steps (1166)
Stops (X166)
Steps (x1e6)
Steps Oc1e6)
(xle6)
SBc
pebble
pebble+ dist
pebble+contt.
Figure 13: Success rate learning curves for button press, sweep into, drawer open, drawer close, window open, and door close across preference-based RL methods and feedback amounts for image-based observations. The oracle labeller used to generate preference feedback. Mean suc- cess rates plotted along the y-axis with number of steps (in units of 1000) along the x-axis. There one plot per environment (grid columns) and feedback amount (grid rows) with corresponding re- sults per learning methods in each plot. From top to bottom, the rows correspond 2.5k, 5k, 10k, and 20k pieces of teacher feedback. From left to right, the columns correspond to button press, sweep into, drawer open, drawer close, window open, and door close.
43

---

H.4 Image-space Observations Normalized Returns
The normalized retrrn (see Section ?? and Equation 5) for walker-walk, quadruped-walk, cheetah- run, button press, sweep into, drawer open, drawer close, window open, and door close across larger range of feedback amounts for image-space observations are given Table 12.
Table 12: Ratio of policy performance on learned versus ground truth rewards for the image- observations space walker-walk, cheetah-run, quadruped-wale sweep into,button press, drawer open, drawer close, window open, and door open tasks across preference learning meth- ods, labelling methods and feedback amounts (with disagreement sampling). The results are re- ported as means (standard deviations) over 10 random seeds.
DMC
TASK
METHOD
50
100
250
500
1000
MEAN
PEBBLE
0.06 (0.02)
0.07 (0.02)
0.09 (0.02)
0.11 (0.03)
0.11 (0.03)
0.09
WALKER WALK
+DIST.
0.18 (0.03)
0.33 (0.10)
0.40 (0.09)
0.57 (0.16)
0.68 (0.23)
0.43
CONTR.
0.28
0.35
0.46 (0.14)
0.58
0.61 (0.16)
0.46
PEBBLE
(0.23)
(0.19)
(0.15)
(0.19)
(0.09)
WLADRUPED-WAKK
DDST.
0.56
0.59 (6.16)
0.61 (0.12)
0.71 (0.14)
0.72 (0.19)
0.64
+CONTR.
(0.16)
(0.097)
0.69 (0.17)
0.73 (0.22)
(0. (0.16)
0.66
PEBBLE
0.01 (0.01)
0.02 (0.01)
0.02 (0.02)
0.02 (0.03)
0.03 (0.02)
0.02
+DIST.
0.07 (0.03)
0.07 (0.03)
0.07 (0.02)
0.12
0.18 (0.07)
0.10
+CONTR.
0.05 (0.02)
0.14 (0.04)
0.14 (0.04)
0.23 (0.09)
0.16 (0.06)
0.14
MeTAWORLD
TASK
METHOD
2.5K
10K
20K
MRAN
PEBBLE
(0.07)
0.20 (0.07)
(0.09)
(0.11)
BUTTON PRESS
+DIST.
0.25 (0.04)
0.35 (0.09)
0.46 (0.16)
0.58 (0.23)
0.41
+CONTR.
0.27 (0.04)
0.35 (0.10)
0.34 (0.07)
0.42 (0.11)
0.35
PEBBLE
0.03 (0.02)
0.05 (0.03)
0.04 (0.04)
0.10 (0.08)
0.06
(0.10)
+CONTR.
(0.11)
0.12 (0.06)
0.11 (0.07)
0.12 (0.05)
0.11
PEBBLE
0.39 (0.10)
0.50 (0.12)
0.55 (0.11)
0.60 (0.14)
0.51
DRAWER OPEN
+DIST.
0.55 (0.07)
0.60 (0.11)
0.65 (0.08)
0.70 (0.08)
0.63
CONTR.
0.60 (0.11)
0.60 (0.12)
0.67 (0.08)
0.71 (0.08)
0.64
PEBBLE
(0.22)
(0.16)
(0.17)
DRAWER CLOSE
+DIST.
0.95 (0.11)
0.97 (0.12)
0.93 (0.09)
0.97 (0.06)
0.96
+CONTR.
0.97 (0.13)
0.93 (0.11)
0.93 (0.13)
0.95 (0.08)
0.95
PEBBLE
0.23 (0.16)
0.18 (0.078)
0.22 (0.1)
0.35 (0.15)
0.25
OPEN
(0.13)
(0.17)
(0.19)
*CONTR.
0.26 (0.11)
0.46 (0.20)
0.48 (0.19)
0.58 (0.19)
0.44
PEBBLE
0.17 (0.06)
0.26 (0.06)
0.23
0.26 (0.06)
0.23
DOOR OPEN
+DIST.
0.34 (0.07)
0.33 (0.08)
0.34 (0.09)
0.49 (0.13)
0.37
+CONTR.
0.19
0.34 (0.15)
0.33 (0.07)
0,40 (0.05)
0.32
44

---

Stability and Generalization Benefits
L.1 Reward Model Stability
50
100
500
Quadruped Walker
Walk
0.05
0.00
Walk
0.05
0.00
equal
mistake
myopic
noisy
orade
Skip
equal
mistake
myopic
noisy
oracle
skip
equal
mistake
myopic
noisy
oracle
Skip
pebble
pebble+simsiae
Figure 14: Variance in predicted f.. reward as fy is learned and updated in conjunction with Ta- There is one plot per feedback amount (columns) and environment (rows) with corresponding re- sults per learning method and labelling strategy in each plot. The learning methods assessed are PEBBLE and PEBBLE with the SimSiam temporal consistency objective. The labelling strategies (see Appendix B for details) marked along the x-axis.
We evaluate reward model stability across updates by computing the variance in predicted rewards for each transition in B across reward model updates. We expect lower variance to translate into more stability and less reward non-stationarity, resulting in better policy performance. Mean and standard deviation in predicted rewards are provided for each model update over 10 random seeds in Figure 14. A representative subset of the conditions (walker-walk and quadruped-walk from Section 6 are used to evaluate reward model stability.
Figure 14 shows that for fewer feedback samples, the predicted rewards more stable across reward updates for REED methods. For larger amounts of feedback (2 500). where REED vs. non-REED reward policy performance is closer, the amount of predicted reward variability does not differ greatly between REED and non-REED reward functions. Therefore, the benefits REED methods are most pronounced when preference feedback limited.
These reward stability results partially explain why REED leads to better policy performance. Next, we investigate whether performance differences are solely due the interplay between reward and policy learning, or if the difference is also due to differences in overall reward quality.
I.2 Reward Reuse
We assess reward re-usability on a representative subset the conditions from Section 6 by: (1) learning preference-based reward function following [3] and [4]; (2) freezing the reward function; (3) training SAC policy from seratch using the frozen reward function. Reward function reuse is evaluated by comparing policy performance to: (a) SAC trained on the ground truth reward, and (b) SAC learned jointly with the reward function.
Figure shows that, when reusing a reward function, REED improves policy performance relative non-REED methods. When environment dynamics are encoded in the reward function, perfor- mance closely matches exceeds that of both policies trained on the ground truth and policies trained jointly with the reward function.
We see different trends when comparing the reused case to the joint case across environments. For walker-walk, the policy trained on the reused reward function typically slightly under performs the policy trained conjunction with the reward function (with the exception of feedback = 200 with the REED reward function) suggesting the REED and non-REED reward functions are over-fitting to the transitions they trained on. For quadruped -aakk, policies trained the reused REED reward function outperform the policy trained jointly with the REED reward function for feedback amounts
45

---

50
200
500
1000
1000
walker walk
0.500
0.5.0
0.5.0
quadruped walk
1000
0
10
10
10
Steps (x1e6)
Steps (x1e6)
Steps (x1e6)
Steps (x1e6)
sac
joint pebble
reuse pebble
joint rebble+reed
reuse pebble+reed
Figure 15: Preference-learred reward reuse policy learning curves for walker-walk and quadruped- walk comparing SAC on the ground truth reward function vs. on the PEBBLE learned reward func- tion on the PEBBLE with SimSiam reward function with the oracle labeller across feedback amount. Mean policy returns are plotted along the y-axis with number of steps along the x-axis.
200 and matches for 200. To our surprise, SAC is able to learn faster on the REED reward function than the ground truth reward function whenever the amount of feedback is 200. Whereas for the non non-REED reward function, the policy trained on the reused reward function under- performs, matches, or out-performs the policy learned jointly with the reward function depending on the amount of feedback. The results suggest that the REED method are less prone to over-fitting.
1.2.1 Complete Reward Reuse Results
The complete reward reuse results for walker-walk (Figure 16). quadruped-we -alk (Figure 17), and cheetah-run (Figure 18) across feedback amounts and teacher labelling strategies.
46

---

equal
mistake
myopic
noisy
oracle
skip
1000
8
10000
100
500
10004
200
10000
500
10004
1000
Steps (xleb)
Steps (x1e6)
Steps (xlee)
Steps (x1eb)
Steps (x1e6)
Steps (xteb)
Figure 16: Learning curves for walker-walk comparing joint reward and policy learning with policy learning using previously learned reward funetion across preference-based RL method, labelling style, and feedback amount. Mean policy returns are plotted along the y-axis with number of steps (in units of 1000) along the x-axis. There is one plot per labelling style (grid columns) and feedback amount (grid rows) with corresponding results per learning methods in each plot. From top to bottom, the rows correspond to 50, 100, 200, 500, and 1000 pieces of teacher feedback. From left to right, the columns correspond to equal, noisy, mistake, myopic, oracle, and skip labelling styles (see Appendix for details).
47

---

equal
mistake
myopic
noisy
oracle
skip
1000
8
100l
130
1008
200
100
500
500
1008
1000
500
1008
2000
Steps (xleb)
Steps (K1e6)
Steps (x1e6)
Steps (x1e6)
Steps [K1 (6)
Steps (x1e6)
Figure 17: Learning curves for quadruped-wall comparing joint reward and policy learning with policy learning using previously learned reward function across preference-based RL method, labelling style, and feedback amount. Mean policy returns are plotted along the y-axis with number of steps (in units of 1000) along the x-axis. There is one plot per labelling style (grid columns) and feedback amount (grid rows) with corresponding results per learning methods in each plot. From top to bottom, the rows correspond to 50, 100, 200, 500, 1000, and 2000 pieces of teacher feedback. From left right, the columns correspond to equal, noisy, mistake, myopic, oracle, and skip labelling styles (see Appendix b for details).
48

---

equal
mistake
myopic
noisy
oracle
skip
1000
10000
100
500
10004
200
10000
500
10004
1000
Steps (xleb)
Steps (x1e6)
Steps (xlee)
Steps (x1eb)
Steps (xte6)
Steps (x1eb)
pobbicereed
Figure 18: Learning curves for cheetah-run comparing joint reward and policy learning with policy learning using previously learned reward funetion across preference-based RL method, labelling style, and feedback amount. Mean policy returns are plotted along the y-axis with number of steps (in units of 1000) along the x-axis. There is one plot per labelling style (grid columns) and feedback amount (grid rows) with corresponding results per learning methods in each plot. From top to bottom, the rows correspond to 50, 100, 200, 500, and 1000 pieces of teacher feedback. From left to right, the columns correspond to equal, noisy, mistake, myopic, oracle, and skip labelling styles (see Appendix for details).
49


Reviews:
[{"review_id": "zl50lBdgyT", "reviewer": "Reviewer_XueR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors using self-supervised objectives to learn a representation of the environment dynamics, which improves the result of preference-based learning. The learning objective can be plugged into existing reward learning frameworks such as PEBBLE. The results are competitive with other reward-learning techniques (especially when labeler is noisy) and highlight the importance of considering environment dynamics.\n", "summary_of_recommendation": "The author provides and straight-forward extension (learning joint state-action representations) to preference-based RL and show its advantage through extensive end-to-end experiments. While the proposed framework is solid, the paper could really benefit from further investigation and analysis of HOW the learned representations benefit reward learning. The authors can do this with concrete examples, or leverage interpretability tools.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Questions:\n1. Can you provide evidence to support the claims that the reward functions learned with REED are more stable and generalizable?\n2. Can you provide some concrete example to illustrate why the jointly-learned state-action representation can improve reward learning?\n\n\n\n\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1. The paper provides extensive evaluation over preference learning benchmarks, as well as different types of labeling issues such as noisy, mistake and myopic.\n2. The authors studies two self-supervised learning objectives to disentangle the overall improve from a specific objective choice, which better highlights the advantage of encoding environment dynamics in general.\n\nCons:\n1. Missing stability and generalizability results. While the intro claims that leveraging environment dynamics improves the generalizability (contribution 2) and stability (contribution 3) of the learned reward functions, I was not able to find the evidences in the experiments. While the authors are able to show the results on better final performance of the learning method, these attributes needs to be separately validated.\n2. More insight into better performance. While the authors are able to show that state-action representations learned with REED can yield better final performance, it remains unclear to me how this is portrayed in the benchmark environments. Does the learned reward function bypass certain limitations of previous methods? Can the authors provide some examples, or further interpretability study of the learned reward functions? While it is straightforward to understand that using environment dynamics SHOULD improve reward learning, more investigations can help reveal HOW.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "QgasrPgAv-", "reviewer": "Reviewer_zDek", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper seeks to improve preference-based reinforcement learning by learning the reward function based on a self-supervised representation that has been trained to predict dynamics. Results show that the proposed approach is often much more sample efficient than prior PbRL techniques.", "summary_of_recommendation": "Overall, I think the strengths outweigh the weaknesses so I would recommend a weak reject. However, there are no actual robotics experiments and some of the simulation experiments make me question the value of the proposed approach.\n\n---\nPost rebuttal: I thank the authors for their responses and maintain my score of weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Missing references:\nPrior work has looked at learning latent representations for PbRL using a forward dynamics loss:\nBrown et al. \"Safe imitation learning via fast bayesian reward inference from preferences.\" ICML, 2020.\n\nPrior work has looked at using contrastive learning to learn representations for PbRL:\nBobu et al. \"SIRL: Similarity-based Implicit Representation Learning.\" HRI. 2023.\n\nWhy not use an exponential moving average like prior work BYOL and SPR?\n\nWhy use a linear projection followed by a linear prediction? This is redundant since two linear transformations can be represented by a single linear transformation.\n\nOn line 142 the term \"different views\" is not defined or explained.\n\nWhy first encode the states and actions separately?\n\nTypo in Table 1: REED.SURF -- missing space\n\nI'm not convinced that Table 1 shows a clear win for REED. SURF should be bolded for the BUTTON Press and SWEEP into tasks but is not. This means that REED methods are only significantly better on 5/10 tasks for PEBBLE and only on 4/10 tasks for PrefPPO. Especially in PrefPPO it seems that Base is the best method.\n\nWhy do we want to encode dynamics? Prior work has often only learned state-based reward functions. Indeed. AIRL argues that using actions will lead to learning the wrong reward function---one that is entangled with the dynamics function. How does your proposed approach perform when the dynamics change? It seems that REED would be especially sensitive to this as well as changes in embodiment. These issues should be discussed more in the limitations.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n+The proposed method can take advantage of all experience by refining its latent state-action representation.\n+Good range of experiments and ablations. The inclusion of learning curves and final normalized performance is nice.\n+Good discussion and limitations sections.\n+ Overall I like the idea of the paper and I think it covers an important problem.\n\nWeaknesses:\n- The authors say that giving feedback is difficult for single state-action pairs and cite TAMER; however, the TAMER papers shows that this is not hard and is a practical method for training RL systems.\n-Missing some relevant prior work.\n-Some of the design choices seem to introduce redundancy or are not adequately motivated.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "9VXqyAUc_Wm", "reviewer": "Reviewer_D6aA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduced the notion of the encoding environment dynamics through a temporal consistency task during reward function learn. The authors claim that this inew method REED mproves the sample-efficiency of a preference-based reinforcement learning algorithm.\nThey compare their method with 3 other  reward-learning extensions to PEBBLE and 2 other PbRL algorithms (PEBBLE and PrefPPO) and find that in many cases their method outperforms the baselines. They additionally perform a set of studies to examine the effect of labeling errors,  feedback amounts etc. and compare the PbRL algorithms PEBBLE and PrefPPO against REED. They find that including the environment dynamics information improves the results.", "summary_of_recommendation": "I am familiar with works in the space of Preference based reinforcement learning and reinforcement learning in general", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. Can the authors provide comparison of other PEBBLE extensions (SURF, RUNE, MRN) to REED for the labeling noise experiment?\n2. Can the authors plot the final reward for each of the compared methods (PEBBLE, SURF, RUNE, MRN, REED) against the number of samples (feedbacks)? (It seems from the appendix that the DMC experiments were performed with six different amounts of feedback - but SURF, RUNE, MRN were not included in the evaluation)\n3. Please address the following:\n3-a. In line 138, \"and then $\\tilde y^d_{t+1}$ is pushed to be consistent with $z^s_{t+1}$ via a negative cosine similarity loss.\" $z^s_{t+1}$ should probably be $y^d_{t+1}$.\n3-b. In Table 1., the SURF column seems to have larger mean values for the BUTTON PRESS and SWEEP examples, yet - +CONTRAST has been bolded.\n3-c. In Figure 3, x- ticks on the left hand pictures look incorrect. - for example those for the cheetah run start with 10.0 and end at 0.5\n3-d. incomplete plot for pebble in quadruped walk  (Figure 3 ) column 2. were all the steps not finished?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper was clear for the most part and easy to read. While other works have also used auxillary and consistency losses for improvement of RL algorithms, I haven’t seen it used in the context of PbRL (except maybe a newish paper that proposes Model based PbRL). Additionally, this method performs an extensive comparison with other methods. \n\n**Strengths**:\n- comparison with several baselines\n- study examining the advantage of dynamics encoding for noisy labels is especially interesting\n\n**Weaknesses**:\n- Comparison of other PEBBLE extensions (SURF, RUNE, MRN) for the labeling noise experiment was not performed.\n- Possible typos and errors in the evaluation tables.\n- The relevance of the dynamics  encoding especially in the context of sampling efficiency can be made clearer if the number of samples were plotted against the final reward for each of the compared methods (PEBBLE, SURF, RUNE, MRN, REED)\n\n**Comments**:\n- Since this method is learning a model anyway, instead of using SAC, has a model-based RL method investigated?\n- instead of using an auxillary loss for dynamics learning, can $s_{t+1}$ be directly used as an input to the reward function?  ", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "2mO_dUedqS", "reviewer": "Reviewer_8gBh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes to learn reward functions using a learned representation of states that can predict the dynamics of the environment. Two kinds of objectives are explored for learning state-representation - Distillation and Contrastive. The reward is then constructed as a linear or a simple function of the learned representations. The authors compare their method on the B-pref benchmark suite and show improved performance.", "summary_of_recommendation": "I am having a hard time understanding the novelty of the paper. As I pointed out, similar ideas have already been executed in prior work but I appreciated the experimental evaluation of this paper.\n\n\nUpdate: Thanks for the response, I have updated my score. The author's reply about updating the paper to reflect novelty clearly (that objective they proposed has been used before and contribution is the empirical evaluation between using dynamics aware representation loss vs non-dynamics aware representation loss for PbRL). I am still conflicted about the name of the method which is misleading. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. The reason for incremental improvement in PrefPPO is given to be the lack of diversity in replay buffer. Why not maintain another buffer that stores all previous transitions for the purpose of training state representations?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. The authors examine a new architecture for predicting reward functions where the state representations are learned using SPR (self-predictive representations).\n2. The authors perform extensive experiments demonstrating the superiority of their method in the 5 environments considering Varying preferences, and varying noise strategy in labelling preferences and adding their proposed modifications to an off-policy PbRL variant (Pebble) and on-policy (PrefPPO). \n\nWeaknesses:\n1. Lack of novelty: I am not sure as to why the proposed modification is novel. The idea of using representations predictive of environment dynamics for learning reward functions have been explored before. eg. (Sec 5.2 in https://arxiv.org/pdf/2002.09089.pdf). Basically, a literature review of a whole line of work in learning from preferences seems to be ignored:\na. https://arxiv.org/abs/1904.06387\nb. https://arxiv.org/pdf/2002.09089.pdf\nc. https://arxiv.org/abs/2202.03481\nd. https://arxiv.org/abs/2010.11723\n\n2. Experimental results: For a majority of tasks in Table 1, it seems that the improvement by REED falls within the standard deviation of results from other methods. Are the results statistically significant?\n\n3. Rewards encoding environment dynamics: I am not sure how this claim can be made. Traditionally, in prior literature, the rewards that are shaped by environment dynamics are referred to as Potential based rewards (see https://people.eecs.berkeley.edu/~russell/papers/icml99-shaping.pdf). But, in this paper, only the representation of states encode environment dynamics, not rewards that are shaped by dynamics. Reward shaping has theoretical guarantee on learning performance but learning a state dependent on reward has no clear impact. Second, even the raw state directly encodes environment dynamics as it is a Markov decision process. I am not sure if the claims of the paper hinges on reward encoding environment dynamics. An alternate hypothesis for the performance gains might just be the regularization induced by the bottleneck of learning state features.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper initially got mixed reviews with identified weaknesses in terms of novelty and significance of the results, and lack of ablation studies. Yet, these concerns have been addressed in the rebuttal and all reviewer see the merits of the presented approach. Yet, as suggestd by reviewer 8gBh the focus should be put more on the analysis of the results and less on the algorithmic novelty as similar predictive state respresentations have been already proposed. I agree with the reviewers that the paper contans interesting insights of how to construct state representations for preference-based RL and recommend acceptance.
Decision: Accept (Poster)