What Matters to You? Towards Visual Representation Alignment for Robot Learning
Authors: Thomas Tian
Chenfeng Xu
Masayoshi Tomizuka
Jitendra Malik
Andrea Bajcsy
Venue: ICLR 2024 poster (2024)
Keywords: Robot learning
Preference learning
Visual reward learning
Representation alignment
Primary area: applications to robotics, autonomy, planning

Abstract: When operating in service of people, robots need to optimize rewards aligned with end-user preferences. Since robots will rely on raw perceptual inputs, their rewards will inevitably use visual representations. Recently there has been excitement in using representations from pre-trained visual models, but key to making these work in robotics is fine-tuning, which is typically done via proxy tasks like dynamics prediction or enforcing temporal cycle-consistency. However, all these proxy tasks bypass the human’s input on what matters to them, exacerbating spurious correlations and ultimately leading to behaviors that are misaligned with user preferences. In this work, we propose that robots should leverage human feedback to align their visual representations with the end-user and disentangle what matters for the task. We propose Representation-Aligned Preference-based Learning (RAPL), a method for solving the visual representation alignment problem and visual reward learning problem through the lens of preference-based learning and optimal transport. Across experiments in X MAGICAL and in robotic manipulation, we find that RAPL’s reward consistently generates preferred robot behaviors with high sample efficiency, and shows strong zero-shot generalization when the visual representation is learned from a different embodiment than the robot’s.

Published as a conference paper at ICLR 2024
WHAT MATTERS TO You? TOWARDS VISUAL REPRE- SENTATION ALIGNMENT FOR ROBOT LEARNING
Ran Tian', Chenfeng Xu', Masayoshi Tomizuka' Jitendra Malik', Andrea Bajesy²
UC Berkeley CCaneeiie Mellon University
ABSTRACT
When operating service of people, robots need to optimize rewards aligned with end-user preferences. Since robots will rely raw perceptual inputs, their rewards will inevitably use visual representations. Recently there has been excite- ment in using representations from pre-trained visual models, but key to making these work in robotics is fine-tuning, which is typically done via proxy tasks like dynamics prediction enforcing temporal cycle-consistency. However, all these proxy tasks bypass the human's 's input on what matters to them, exacerbating spuri- ous correlations and ultimately leading behaviors that are misaligned with user preferences. In this work, we propose that robots should leverage human feedback align their visual representations with the end-user and disentangle what mat- Learning
ters for the task. We propose Representation-Aligee (RAPL), method for solving the visual representation aligmment problem and visual reward learning problem through the lens of meference-basee learning and optimal transport. Across experiments X-MAGICAL and robotic manipula- tion, we find that RAPL's reward consistently generates preferred robot behaviors with high sample efficiency, and shows strong zero-shot generalization when the visual representation is learned from different embodiment than the robot's.
Preference-
INTRODUCTION
Imagine that a robot manipulator is tasked with cleaning your kitchen countertop. To be successful, its reward function should model your preferences: what you think an orderly countertop looks like (e.g.. plates should be stacked by color), how objects should be handled during cleaning (e.g., cups should be moved one-at-a-time but food scraps should be pushed in large groups), and what parts of the countertop should be avoided (e.g., always stay away from the expensive espresso machine).
long-standing approach to this problem has been through inverse reinforcement learning (IRL), where the robot infers reward from demonstrations. Fundamental work in IRL (Abbeel & Ng, 2004: Ziebart et al., 2008) argues for matching features between the expert and the learner during reward inference, more formally known optimizing Integral Probability Metries (IPMs) (Sun et al., 2019; Swamy et al., 2021). While there are many ways to do this matching, optimal transport has recently been used optimize IPMs defined on high-dimensional feature spaces in a principled manner (Xiao et al., 2019; Dadashi et al., 2021; Papagiannis & Li, 2022; Luo et al., 2023).
The question remains, what are good features for the learner to match? While traditional work used hand-engineered features defined on low-dimensional states of the world that the robot could do state estimation on (e.g. object pose) (Levine al., 2011: Finn et al., 2016), large vision models trained via sttesspprriiscduuuuprvissee learning promise representations for robotics that directly operate on raw image inputs and capture more nuanced features automatically (e-g., object color and shape) (Xiao et al., 2022; Ma et al., 2023; Karamcheti et al., 2023). However, automatically extracting relevant visual features that incentivize preferred robot behavior and minimize spurious correlations remains an open challenge (Zhang et al., 2020). Previous works use proxy tasks during representation learning to imbue some prior human knowledge about attributes that are relevant (Brown et al., 2020), such assuming access action labels and doing behavior cloning (Haldar et al., 2023a;b) or performing temporal cycle- consisecn learning (Dadashi et al., 2021). The former requires additional signals that may be hard obtain (i.e., actions), while the latter assumes that

---

Published as a conference paper at ICLR 2024
Visual Representation A Alignment
P(
ΦH)
Policy Learning via O'T-based
Homan Visnal Representatin
9H
max N
Figure 1: Representation-Alinnn Preference-basse Learning (RAPL), an action-free visual representation learning method using casy-to-provide human preference feedback on video demos. Using the human preference triplets, the robot goes from paying attention to the end-effector (oft) to paying attention to the objects and the goal region (ØTH) at the end of alignment. The aligned repre- sentation is used to construct an optimal transport- -ased visual reward for robot behavior learning.
temporal consistency signal is sufficient to extract what matters to the task, potentially ignoring other features that matter to the demonstrator (e.g., avoiding undesired regions).
Since robots ultimately operate in service of people, recent works advocate that the robot should explicitly engage in a representation alignment process with the end-user. These works leverage user feedback such human-driven feature selection (Bullard et al., 2018; Luu-Duc & Miura, 2019), interactive feature construction (Bobu et al., 2021; Katz et al., 2021) or similarity-uueries (Bobu al., 2023a) learn human-centric representations. However, these approaches either operate on pre-defined feature sets or in low-dimensional state spaces (e.g., positions). In the visual domain, (Zhang al., 2020) uses manually- -efined reward signals to learn representations which disentangle high versus low reward images; unfortunately, when the main objective is reward learning in the first place, this approach is not feasible.
Instead of hoping to extract human-centric visual representations via proxy tasks that bypass human input or require action labels per video frame, we propose that robots use easy-to-provis proviiiv human preference feedback- the human is asked to compare two more video demonstrations to align their visual representations with what matters to the end-user (Figure 1).
We first formalize the visual representation alignment problem for robotics as metric learning problem in the human's 's representation space. We then propose Representation-Aligue Preference- based Learning (RAPL), tractable video-only method for solving the alignment problem and learn- ing visual robot rewards via optimal transport. Across experiments in X-Magical (Zakka et al., 2022) and in robotic manipulation, we find that RAPL's reward consistently generates preferred robot be- haviors with high sample efficiency, and shows strong zero-shot generalization when the visual representation learned different embodiment than the robot's.
PROBLEM SETUP
Human Policy, We consider scenarios where the robot R wants to learn how to perform a task for human H. The human knows the desired reward r* which encodes their preferences for the task. The human acts via approximately optimal policy ग O + U4 based on their underlying reward function. Instead of directly consuming the raw perceptual input, research suggests that humans naturally build visual representations of the world (Bonnen et al., 2021) that focus task-relevant attributes (Callaway et al., 2021). We model the human's representation model as Φn O - 4 mapping from the perceptual input o O the human's latent space 2H which captures their task and preferencee-elleant features.
Robot Policy. We seek learn robot policy के O UR which maps from image observations to actions that maximizes a reward:
arg max E FR
Bowp(o mR)
Σ t=0
r' r(oR('')))
where E [0, is the discount factor and 10" ..... is the image observation trajectory induced by the robot's policy. The robot's 's reward r relies on a visual representation, ΦR O - dR. which maps from the image observation this could be hand cradterafted, such as distances to objects from the agent's end-effector (Ziebart et al., 2008; Levine al., 201 I; Finn et al., 2016), the output of encoder pre-trained on large-scale
O to dower-dimensional latent space, PR. In general,
2

---

Published as a conference paper at ICLR 2024
datasets (Chen et al., 2021; Ma et al., 2023). Before the robot optimize for TR. it is faced with two questions: what visual representation ØR should it use to encode observations, and which reward should it optimize to align its behavior with zH?
3 RAPL: REPRESENTATION-ALIIC
PREFERENCE-BASEE LEARNING
In this work, we leverage pre-trained visual encoders but advocate that robots fine-tune them with preference-bee based human feedback to extract features that are relevant for how the end-user likes the task to be performed. Preference-basee feedback, where a user is asked to compare two more trajectories, has been shown to be easier for end-users to provide compared to direct labelling or giving near-optimal demonstrations (Wirth et al., 2017). However, we are the first to use preference- based human feedback to align pre-trained visual models with user preference for robot learning.
Once the robot has aligned representation, what reward should it optimize to generate preferred behaviors? To reduce the sample complexity. we use optimal transport methods to design a visual reward (Villani et al., 2009), and focus the preference feedback exclusively fine-tuning the rep- resentation, Since optimal transport methods are way to optimize Integral Probability Metries in principled manner, the transport plan exactly yields reward which maximizes feature matching between the learner and the expert in our aligned representation space.
In this section, we first formally state the visual representation alignment problem by drawing upon recent work in cognitive science (Sucholutsky & Griffiths, 2023) and then detail our approximate so- lution, pppresentation-Aligne Preference-basee Learning (RAPL), through the lens of preference- based learning and optimal transport. In addition to this section's content, we also provide more details about comparisons between our work and previous work in Appendix A.2 and A.I.
3.1 THE VISUAL REPRESENTATION ALIGNMENT PROBLEM FOR ROBOTICS
We follow the formulation in Sucholutsky & Griffiths (2023) and bring this to the robot learning domain. Intuitively, visual representation alignment is defined as the degree to which the output of the robot's encoder, dR, matches the human's 's internal representation, őH, for the same image observation, o E O, during task execution. We utilize a triplet-based definition of representation alignment in (Jamieson & Nowak, 2011) and (Sucholutsky & Griffiths, 2023).
Definition (Triplet-based Representation Space) Let = {o' be a sequence of image observations over timesteps.
> be given representation model, and á(o) == "t be the corresponding embedding trajectory. For some distance metric d(.,-) and two observation trajectories o' and o', let d()(o'), o(o²)) be the distance between their em- bedding trajectories. The triplet-based representation space of & is:
S = d(o(o'), o(o0)) < (((o(o'),<(o'"))) i. Ez).
where E is the set of all possible image trajectories for the task of interest.
Intuitively, this states that the visual representation ó helps the agent determine how similar two videos are in lower-dimensional space. For all possible triplets of videos that the agent could see, it can determine which videos are more similar and which videos are less similar using its embedding space. The set Sy contains all such similarity triplets.
Definition (Visual Representation Aligument Problem) Recall that 6n and ΦR are the human and robot's 's visual representations respectively. The representation alignment problem defined as learning a Φn which minimizes the difference between the two agents' representation spaces, as measured by a function È which penalizes divergence between the two representation spaces:
min (S)R::Sp))) $n
3.2 REPRESENTATION INFERENCE VIA PREFERENCE-BASED LEARNING
Although this formulation sheds light on the underlying problem. solving Equation 3 exactly is impossible since the functional form of the human's representation on is unavailable and the set
3

---

Published as a conference paper at ICLR 2024
Són is infinite. Thus, approximate the problem by constructing subset Sog Son of triplet queries. Since we seek a representation that is relevant to the human's 's preferences, we ask the human to rank these triplets based on their peefeence-basse notion of similarity (e.g., r" (on((')) > r" (""(((")) > r" (őn(o*) = > o² > o²). With these rankings, we implicitly learn on via neural network trained on these triplets.
We interpret human's preference over the triplet (o' o² o²) E Son via the Bradley- model (Bradley & Terry, 1952), where o' is treated as anchor and o of are compared to the anchor in terms of similarity as in Equation 2:
c e=d(du(o"): (oa()"))
P(o' - o > o* @H) e-d(ou(o"), @n(())) + e-d(ou(o"), o=(o")))
(4)
A natural idea would be leverage the human's preference feedback do direct reward prediction (i.e.. model both d and Φн via single neural network to approximate r*), as is done in tradi- tional preference-basee reward learning (Christiano et al., 2017). However, (Section 5.2) that directly learning a high-quality visual reward from preference queries requires prohibitive amount of human feedback that unrealistic expect from end-users (Brown et al., 2019; Bobu et al., 2023b).
find empirically
Instead, we focus all the preference feedback on just representation aligmment. But, this raises the question what distance measure d should we use? In this work, we use optimal transport principled way to measure the feature matching between any two videos. For any video o and for a given representation Φ, let the induced empirical embedding distribution be p = T Σt= &o(ot)> where 6o(ot) is a Dirac distribution centered on d(o'). Optimal transport finds the optimal coupling 4a E IR T that transports one embedding distribution, Ps. to another video embedding distribution, Py. with minimal cost. This approach has well develoeee suite of numerical solution techniques (Peyré et al., 2019) which we leverage in practice (for details on this see App. A.3).
Our final optimization is maximum likelihood estimation problem:
zn max dn
Σ
PPo' oo o* dh).
(5)
(ot EE8
Since the robot seeks a visual representation that is aligned with the human's, we set: OR := ÓH.
3.3
PREFERENCE-ALICE
ROBOT BEНAAIIR VIA OPTIMAL TRANSPORT
Given our aligned visual representation, we seek a robot policy TR whose behavior respects the end-user's preferences. Traditional IRL methods (Abbeel & Ng. 2004; Ziebart et al., 2008) are built upon matching features between the expert and the learner. Leveraging this insight, we use optimal transport methods since the optimal transport plan is equivalent to defining reward function that encourages this matching (Kantorovich & Rubinshtein, 1958).
Specifically, we seek to match the embedded observation occupancy measure induced by the robot's policy TR. and the embedded observation occupancy measure of a human's preferred video demon- stration, 04, Thus, the optimal transport plan yields the reward which is optimized in Equation I:
T
r r(okioR) ΦR)
= Σ e=11
c((o((o)) (R (o(
(6)
where μ² is the optimal coupling between the empirical embedding distributions induced by the robot and the expert. This reward has been successful in prior vision-based robot learning (Haldar et al., 2023b;a; Guzey et al., 2023) with the key difference in our setting being that we use RAPL's aligned visual representation ΦR for feature matching (for details on the difference between our approach and prior OT based visual reward see Appendix A.
4 EXPERIMENTAL DESIGN
We design a series of experiments to investigate RAPL's ability to learn visual rewards and generate preferred robot behaviors.
4

---

Published as a conference paper at ICLR 2024
Preference Dataset: Son. While the ultimate test is learning from real end-user feedback, in this work we use a simulated human model as a first step. This allows us to easily ablate the size of the preference dataset, and gives us privileged access r* for direct comparison. In all environments, the simulated human constructs the preference dataset SΦn by sampling triplets of videos uniformly random from the set' of video observations E and then ranking them with their reward y* as in Equation 4.
Independent & Dependent Measures. Throughout our experiments we vary the visual reward signal used for robot policy optimization and the preference dataset size used for representation learning. We measure robot task success as a binary indicator of if the robot completed the task with high reward r"
Controlling for Confounds. Our ultimate goal is have visual robot policy, тR, that takes as input observations and outputs actions. However, to rigorously compare policies obtained from different visual rewards, we need to disentangle the effect of the reward signal from any other policy design choices, such the input encoders and architecture. To have a fair comparison, we follow the approach from (Zakka et al., 2022; Kumar et al., 2023) and input the privileged ground-truit state into all policy networks, but vary the visual reward signal used during policy optimization. Across all methods, we use an identical reinforcement learning setup and Soft-Actor Critic for training (Haarnoja al., 2018) with code base from (Zakka al., 2022).
In Section 5, we first control the agent's embodiment to be consistent between both representation learning and robot optimization (e.g., assume that the robot shows video triplets of itself to the hu- man and the human ranks them). In Section 6, we relax this assumption and consider the more realistic cross-embodiment scenario where the representation learning performed different embodiment than the robot's. s. For all policy learning experiments, we use 10 expert demon- strations the demonstration D+ + for generating the reward (more details in Appendix A.3).
videos of
5 RESULTS: FROM REPRESENTATION TO BEHAVIOR ALIGNMENT
We first experiment in the toy X-Magical environment (Zakka et al., 2022), and then move to the realistic IsaacGym simulator.
5.1 X-MAGICAL
Tasks. We design two tasks inspired by kitchen coun- tertop cleaning. The robot always has to push objects to a goal region (e.g., trash can), shown in pink at the top of the scene in Figure' 2. In the avoiding task, the end-user prefers that the robot and objects never enter an off-limits zone during pushing (blue box in left Fig- ure 2). In the grouping task, the end-user prefers that objects are always pushed efficiently together (instead one-at-a-time) towards the goal region (center, Figure 2).
Avoiding
Grouping
Grouping
Privileged State Reward. For avoiding, the true state is 7D: planar robot position (PR IR²) and ori-
Figure 2: X-Magical & IsaacGym tasks.
entation (OR), planar position of the object (Pots E R2), distance between goal region and ob- ject, (dohj2goal), and distance between the off-limits reward is: r ravoid (s)
and the object (dots2obj). The human's - dgoat2obj - T(dabs2obj < deafety), where dsafety is a safety dis- tance and I an indicator function giving when the condition true. For grouping, the state is 9D: = (PR, CR. Pubjl: Pobja, dgualZobji, dgoal2obj2). The human's 's reward is: ngroup (s) max mmaxxdoaal2obji.
dgoalzobjz) dasiiabbbjiddddlobbppp
|/Poaj1 - Pobje|12.
Baselines. We compare visual reward, RAPL, against (1) GT, an oracle policy obtained under r", (2) RLHE, which is vanilla preference-bait based reward learning (Christiano et al., 2017; Brown
'To minimize the bias of this set representation learning, we construct E such that the reward distribution of this set under r", is approximately uniform. Future work should investigate the impact of this set further, e.g., (Sadigh et al., 2017).
5

---

Published as a conference paper at ICLR 2024
Avoiding
Grouping
(A)
IBI
pageres
progrrsr
limis
Distke
anly prakss lage!
adhave Ladk cenonng "afiimts ere
Prefernd
poodes eégert tegetker
Figure 3: X-Magical. (left & right) examples of preferred and disliked videos for each task. (center) reward associated with each video under each method. RAPL's predicted reward follows the GT pattern: low reward when the behavior disliked and high reward when the behavior are preferred. RLHF and TCC assign high reward to disliked behavior (e.g., (D)).
Avoiding
Grouping
0.8
Stocrss nite
0.8
0.6
0.4
0.2
100
50
100
150
200
250
Epoch
Figure 4: X-Magical. Policy evaluation success rate during policy learning. Colored lines are the mean and variance of the evaluation success RAPL can match GT in the avoiding task and outperforms baseline visual rewards in grouping task.
al. 2019) that directly maps an image observation scalar reward, and (3) TCC (Zakka al., 2022: Kumar et al., 2023) which finetunes a pre-trained encoder via temporal cycle consistency constraints using 500 task demonstrations and then uses L2 distance between the current image embedding and the goal image embedding reward. We use the same preference dataset with 150 triplets for training RLHF and RAPL.
Visual backbone. We use the same setup in (Zakka al., 2022) with the ResNet- backbone pre-trained on ImageNet. The original classification head is replaced with linear layer that outputs 32-dimensional vector our embedding space, 4n := $2 The TCC representation model is trained with 500 demonstrations using the code from (Zakka et al., 2022). Both RAPL and RLHF only fine-tune the last linear layer. All representation models frozen during policy learning.
Hypothesis. RAPL better at capturing preferences beyond task progress compared to direct reward prediction RLHF TCC visual reward, yielding higher success rate.
Results. Figure 3 shows the rewards over time for three example video observations in the avoid (left) and group task (right). Fach video is marked preferred by the end-user's ground-truth reward or disliked. Across all examples, RAPL's rewards are highly correlated with the GT rewards: when the behavior in the video is disliked, then reward is low; when the behavior preferred, then the reward is increasing. TCC's reward is correlated with the robot making spatial progress (i.e., plot (E) and (F) where observations get closer to looking like the goal image), but incorrectly predicts high reward when the robot makes spatial progress but violates the human's 's preference ((C) and (D) Figure 3). RLHF performs comparably to RAPL, with slight suboptimality in scenarios (C) and (D). Figure 4 shows the policy evaluation success rate during RI training with each reward function (solid line is the mean, shaded area the standard deviation, over 5 trials with different random seeds.). Across all environments, RAPL performs comparably to GT (avoid success: 80%, baselines with better sample efficiency,
group success: so 60%) and significantly outperforms RAPL takes 10 epochs to reach 70% success rate in the avoid task (GT requires 100) and takes 100 epochs to reach 40% success rate in the avoid task (GT requires 150), supporting our hypothesis.

---

Published as a conference paper at ICLR 2024
Reward of Profornd Demo
Optimal Transport Plan
dRAM.
Prcc
Expert Ttmno
Reward of Distiien Demo
Profernl Demo
Diailel Damo
CTT
RATI
MNP-OT TCC-TT
Figure 5: Manipulation. (center) Expert, prefemed, and disliked video demo. (left) reward associ- ated with each video under each method. RAPL's predicted reward follows the GT pattern. RLHF assigns high reward to disliked behavior. (right) OT coupling for each representation. Columns embedded frames of expert demo. Rows of top matrices are embedded frames of preferred demo; of bottom matrices are embedded frames of disliked demo. Peaks exactly along the diagonal indicate that the frames of the two videos are aligned in the latent space; uniform values the ma- trix indicate that the videos cannot be aligned (i.e., all frames equally "similar" to the next). RAPL matches this structure: diagonal peaks for disliked, while baselines show diffused values no matter the videos being compared.
and uniform for expert-and-
5.2 ROBOT MANIPULATION
In the X-Magical toy environment, RAPL outperformed progress-based visual rewards, but direct preference-based reward prediction was competitive baseline. Moving to the more realistic robot manipulation environment, we want to 1) disentangle the benefit of our fine-tuned representation from the optimal transport reward structure, and 2) understand if our method still outperforms direct reward prediction in a more complex environment?
Task, Privileged State & Reward. We design a robot manipulation task in the IsaaeGym physics simulator (Makoviychuk et al., 2021). We replicate the tabletop grouping scenario, where a Franka robot arm needs to learn that the end-user prefers objects be pushed efficiently together (instead of one-at-a-time) to the goal region (light blue region in right of Figure 2). The state is I8D; robot proprioception (fjoints E RR 10) 3D object positions (Pobjia)), and object distances to goal (d (.goal2abj..2.. The grouping reward is identical in Section 5.1.
Baselines. In addition to comparing RAPL against (1) GT and (2) RLHF, we ablate the representa- tion model but control the visual reward structure. We consider five additional baselines that all use optimal transport- based reward but operate on different representations: (3) MVP-OT which learss image representation via masked visual pre-training: (4) Mene-Tuned--VP-OO.. which fine-tunes MVP representation using images from the task environment; (5) R3M-OT, which is a ResNet-18 encoder (Nair al., 2022) pre-trained objective that combines time contrastive learning, rideo-languee aligmment, and sparsity penalty; (6): ImageNet-OT, which is a ResNet-15 encoder pre-traiie pre-trained on ImageNet: (7) TCC-OT (Dadashi et al., 2021) which embeds images via the TCC representation trained with 500 task demonstrations. We use the same preference dataset with 150 triplets for training RLHF and RAPI.
the Ego4D data (Grauman al., 2022) via learning
Visual model backbone. All methods except MVP-OT and ResNet-18 visual backbone and have the same training setting as the one the X-Magical experi- ment. MVP-OT and Fine- Tww---unndd------
share the same
use a off-the-shelf visual transformer (Xiao et al., 2022) pre-trained on the Ego4D data set (Grauman et al., 2022). All representation models are frozen during robot policy learning.
Hypotheses. HI: RAPL's higher success rate is driven by its aligned visual representation. H2: RAPL outperforms RLHF with lower amounts of human preference queries.
Results: Reward prediction & policy learning, the center Figure 5 we show three video demos: an expert video demonstration, preferred video, and a disliked video. On the right of Figure 5, we visualize the optimal transport plan comparing the expert video to the disliked and preferred videos under visual representations @RAPL, @ToC-OT, ΦMYr-OT. Intuitively, peaks exactly

---

Published as a conference paper at ICLR 2024
along the diagonal indicate that the frames of the two videos aligned in the latent space; uni- form values in the matrix indicate that the two videos cannot be aligned (i.e., all frames are equally "similar" to the next). RAPL's 'sss representation induces precisely this structure: diagonal peaks when comparing two preferred videos and uniform when comparing a preferred and disliked video. In- terestingly,
diffused peak regions in all transport plans under both TCC-OT and MVP-OT representations, indicating their representations struggle to align preferred behaviors and disentan- gle disliked behaviors. This is substantiated by the left of Figure 5, which shows the learned reward over time of preferred video and disliked video. Across all examples, RAPL rewards are highly correlated to GT rewards while baselines struggle to disambiguate, supporting H1. A more de- tailed quantitative analysis on the relationship between the learned visual reward and the end-user's ground-truth reward is shown Appendix A.8.
Figure 6 shows the policy evaluation history dur- ing RL training with each reward function. We see RAPL performs comparably to GT (succ. rate: 70%) while all baselines struggle to achieve a success rate of more than 10%. It's surprising that RLHF fails in a more realistic environment since its objective is similar ours, but without explic- itly considering representation alignment. To further investigate this, we apply a linear probe on the final embedding and visualize the image heatmap of what each method's final embedding pays attention to Figure 1 in the Appendix. RLHF is biased towards paying attention to irrelevant areas that can induce spurious correlations; in contrast RAPL focuses on the task-relevant objects and the goal region.
0.8
Success
0.4
0.
150
250
Epoch
GI RAPL
RLHF
MVP-OT
TCC-OT
Fine- Tunedd--d-- R3M-OT
ImageNet-OT
Figure 6: Manipulation. Success during robot policy learning for each visual reward.
Results: Sample complexity. We further study the sensitivity of RAPL. and RI.HF to the preference query dataset size. For RLHF, we double the preference query dataset during reward model training to 300 queries (detailed in App. A.5). Pol- icy performance is improved, indicating that with more feed- back data, preference-batee
Sp000ss rale
reward prediction could yield an aligned policy. Nevertheless, RAPL outperforms RLHF by 75% with 50% less training data, supporting H2. While all the RAPL results above used 150 preference queries to train the representation, we also train a visual representation with 100, 50, and 25 preference queries. Figure 7 shows the success rate of the robot manipulation policy for each version of RAPL. We note that RAPL achieves a 45% success rate even when trained on only 25 preference queries.
150
100
50
25
Figure Manipulation. RAPL sample efficiency analysis. RAPI. can achieve Rs 45% success rate with only 25 preference queries.
Additional Complex Manipulation Task with Visual Distractors. Finally, we implemented more complex robot manipulation task to further validate RAPL'S ability to disentangle both seman- tic and lew-level visual features that underlie an end-user's preferences under visual distractors. The detailed task description and results are reported in Appendix A.7. We find that RAPL significantly outperforms all baselines in this more complex manipulation and cefferenne-learning task.
RESULTS: ZERO-SHOT GENERALIZATION ACROSS EMBODIMENTS
So far, the preference feedback used for aligning the visual representation was given on videos o E E generated the same embodiment that of the robot. However, in reality, the human could give preference feedback on videos of a different embodiment than the specific robot's. We investigate our approach can generalize to changes in the embodiment between the preference dataset Sμ and the robot policy optimization.
Tasks & Baselines. We use the same setup for each environment as in Section 5.
Cross-Domain Agents. In X- X-Magical, reward functions always trained the short stick agent, but the learning agent is a gripper in avoid and a medium stick in grouping task. In manipulation train RAPL and RL RLHF on videos of the Franka, but deploy the rewards on the Kuka robot.
8

---

Published as a conference paper at ICLR 2024
Avoiding
Grouping
Suooas mite
0.8
0.8
04
Sucecs
04
0.4
02
150
100
150
Epoch
Epoch
RLHF
Figure 8: Pross-Embodimente X- X-Magical. Policy evaluation success rate during policy learning. Colored lines are the mean and variance of the evaluation success rate. RAPL achieves a comparable success rate compared to GT with high learning efficiency, and outperforms RLHF and TCC.
Hypothesis. RAPL enables zero-shot cross-ebbodiment generalization of the visual reward com- pared to other visual rewards.
Results. Figure 8 and Figure 9 show the policy evaluation histories during RL training with each re- ward function in the cross- eeboddmebod X-Magical environment and the manipulation environment. We see that in all cross-embodiment scenarios, RAPL achieves a comparable success rate compared to GT and significantly outperforms baselines which struggle to achieve more than zero success rate, supporting H1. See more results in App. A.6.
We note interesting finding in the X-Magical grouping task when the representation trained videos of the short stick agent, but the learn- ing agent is the medium stick agent (see Fig- 14 in Appendix). Because the short stick agent is so small, it has a harder time keep- ing the objects grouped together; in-domain re- sults from Section 5 show success rate of z 60% (see Figure 4). In theory, with well- specified reward, the task success rate should increase when the medium stick agent does the task, since it is better suited to push objects to- gether. Interestingly, when the short stick visual representation is transferred zero-shot to the medium stick, we see precisely this: RAPL's task success rate improves by 33% under cross- embodiment transfer (suce. rate R 80%). This finding indicates that RAPL can learn task-relevant features that guide correct task execution even new embodiment.
0.6
Sucvess
0.4
0.2
150
250
350
Epoch
GT
RAPL
RLHF
MVP-OT
TCC-OT
Fine-Tuned-MVP--C R3M-OT
ImageNel-OT
Cross-Enin Embodiment: Manipulation. Colored lines are the mean and variance of the success rate during policy learning.
Figure 9:
CONCLUSION
In this work, presented video-only, preference-based learning method for solving the visual representation alignment problem. We demonstrated that with an aligned visual representation, reward learning via optimal transport feature matching can generate successful robot behaviors with high sample efficiency, and shows strong zero-shot generalization when the visual reward is learned on a different embodiment than the robot's.
Although in this work we focused on controlled simulation experiments, future work should validate RAPL with real human feedback and robotic hardware experiments. Though our method shows bet- ter sample efficiency than RLHH, asking humans for preference queries should be done strategically (e.g., via active learning), should be robust to noisy feedback, and could be improved by leverag- ing multi-modality (e.g., preferences & language feedback). While our current approach was an offline fine-tuning method, future work onto online visual reward fine-tuning from human feedback is an exciting direction. Finally, incorporating feedback from multiple humans (e.g.. crowddsourced multimodal preferences) are also an exciting future direction.

---

Published as a conference paper at ICLR 2024
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 2004.
Erdem Biyik and Dorsa Sadigh. Batch active preference-based learning of reward functions, In
Conference on robot learning, pp. 519-528. PMLR, 2018.
Andreea Bobu, Marius Wiggert, Claire Tomlin, and Anca D Dragan. Feature expansive reward learn- ing: Rethinking human input. In Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction, pp. 216-224, 2021.
Andreea Bobu, Yi Liu, Rohin Shah, Daniel S Brown, and Anca D Dragan, Sirl: Similarity-basee implicit representation learning. International Conference on Human Robot Interaction, 2023a.
Andreea Bobu, Andi Peng, Pulkit Agrawal, Julie Shah, and Anca D Dragan. Aligning robot and human representations. arXiv preprint arXiv:2302.01922 2023b.
Tyler Bonnen, Daniel LK Yamins, and Anthony D Wagner. When the ventral visual stream is not enough: A deep learning account of medial temporal lobe involvement in perception. Neuron, 109(17):2755-277) 2021.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Nickum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pp. 783-792. PMLR, 2019.
Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Nickum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference on Machine Learning, pp. 1165-1177. PMIR, 2020.
Kalesha Bullard, Sonia Cheroov,, and Andrea L Thomaz. Hlmann-divve feature selection for robotic agent learning classification tasks from demonstration. In 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 6923-6930. IEEE, 2018
Frederick Callaway, Antonio Rangel, and Thomas L Griffiths. Fixation patterns in simple choice reflect optimal information sampling. PLoS computational biology, 17(3)e 1008863, 2021.
Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" in-the-wild" human videos. Robotics: Science and Systems, 2021.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Robert Dadashi, Léonard Hussenot, Matthiuu Geist, and Olivier Pietquin. Primal wasserstein imita- tion learning. International Conference on Robot Learning, 2021.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. International conference on machine learning, pp. 49-58, PMLR, 2016.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Irmak Guzey, Yinlong Dai, Ben Evans, Soumith Chintala, and Lerrel Pinto. See to touch: Learning tactile dexterity through visual incentives. arXiv preprint arXiv:2309. /2300, 2023.
Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
10

---

Published as a conference paper at ICLR 2024
Siddhant Haldar, Vaibhav Mathur, Denis Yarats, and Letrel Pinto. Watch and match: Supercharg- ing imitation with regularized optimal transport. In Conference on Robot Learning, pp. 32 43. PMLR. 2023a.
Siddhant Haldar, Jyothish Pari, Anant Rai, and Lerrel Pinto. from one minute of demonstrations, Robotics: Science and Systems, 2023b.
Teach robot to fish: Versatile imitation
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.096 00w.2200600968,,, 2021.
Kevin G Jamicson and Robert Nowak. Low-dimensional embedding using adaptively selected or- dinal data. In 2011 49th Annual Allerton Conference Communication, Control, and Computing (Allerton), pp. 1077-1084. IEEE, 2011.
Leonid Vasilevich Kantorovich and SG Rubinshtein. On space of totally additive functions. Vestnik of the St. Petersburg University: Mathmattics, 13(7):52-59, 1958.
Siddharth Karamcheti, Suraj Nair, Annie S Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. arXiv preprint 2:Xi::2302212766. 2023.
Sydney M Katz, Amir Malcki, Erdem Bıyık, and Mykel J Kochenderfer. Preference- based learning of reward function features. arXiv preprint arXiv:2103.02727, 2021.
Sateesh Kumar, Jonathan Zamora, Nicklas Hansen, Rishabh Jangir, and Xiaolong Wang. Graph inverse reinforcement learning from diverse videos. In Conference Robot Learning, pp. 55- 66. PMLR, 2023.
Sergey Levine, Zoran Popovic, and Vladlen Koltun. Nonlinear inverse reinforcement learn- ing with gaussian processes. J. Shawe-Taylon Taylor, R. Zemel, P. Bartlett, Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL https: http::/proooeedd....n..eeeeeeeee httpeerpss..cppppeeee........ fflles/ hrreee/oo222.oooooeeeeeerrrrrea
Yicheng Luo, zhengyao jiang, Samuel Cohen, Edward Grefenstette, and Marc Peter Deisen- roth. Optimal transport for offline imitation learning. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https:/ http::://pperrvvie.... net/ forem/ormmmmm/ did= MhuFzFsrfvH.
Hoai Luu-Due and Jun Miura. An incremental feature set refinement in programming by demon- stration scenario. In 2019 IEEE 4th International Conference on Advanced Robotics and Mecha- tronics (ICARM), pp. 372-377. IEEE, 2019.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. International Conference Learning Representations, 2023.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A univer- sal visual representation for robot manipulation. Conference on Robot Learning, 2022.
Georgios Papagiannis and Yumpeng Li. Imitation learning with sinkhorn distances. In Joint Eu- ropean Conference on Machine Learning and Knowledge Discovery in Databases, pp. 116-131. Springer, 2022.
11

---

Published as a conference paper at ICLR 2024
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications data science. Foundations and Trends@ in Machine Learning, 11(5-6):3556007,222 2019.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference- based learn- ing of reward functions. Robotics: Science and Systems, 2017.
Pierre Sermanet, Kelvin Xu, and Sergey Levine. Unsupervised perceptual rewards for imitation learning. Robotics: Science and Systems, 2016.
Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg. Concept2Rohot: Learning manipulation concepts from instructions and human demonstrations. In Proceedings of Robotics: Science and Systems (RS5), 2020.
Daniel Shin, Anca D Dragan, and Daniel S Brown. Benchmarks and algorithms for offline preference-based reward learning. Transactions on Machine Learning, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3000 2020.
Ilia Sucholutsky and Thomas L Griffiths. Aligmment with human representations supports robust few-shot learning. arXiv preprint 19X1 2023.
Wen Sun, Anirudh Vemula, Byron Boots, and Drew Bagnell. Provably efficient imitation learning from observation alone. In International conference on machine learning, pp. 6036-6045. PMLR, 2019.
Gokul Swamy, Sanjiban Choudhury, Andrew Bagnell, and Steven Wu. Of moments and match- ing: A game-theoretic framework for closing the imitation gap. In International Conference Machine Learming, pp. 10022-10032. PMLR, 2021.
Ajay Kumar Tanwani, Pierre Sermanet, Andy Yan, Raghav Anand, Mariano Phielipp, and Ken Goldberg. Motion2vec: Semi-supervised representation learning from surgical videos. 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 2174-2181. IEEE, 2020.
Jeremy Tien, Jerry Zhi- Yang He, Zackory Erickson, Anca D Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference- reward learning. International Con- ference Learning Representations, 2022.
Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2009.
Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, al. survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136);1-46, 2017.
Huang Xiao, Michael Herman, Joerg Wagner, Sebastian Ziesche, Jalal Etesami, and Thai Hong Linh. Wasserstein adversarial imitation learning, arXiv preprint arXiv:1906.088 081 13, 2019.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre- pre-training for motor control. arXiv preprint arXiv:2203.cm 06173, 2022.
Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. In Computer Vision-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII 16, pp. 1-19. Springer, 2020.
Chenfeng Xu, Shijia Yang. Tomer Galanti, Bichen Wu, Xiangyu Yue, Bohan Zhai Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Image2point: 3d point-cloud understanding with 2d image pretrained models. In European Conference on Computer Vision, pp. 638-656, Springer, 2022.
Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. Xirl: emboddmebdd inverse reinforcement learning. Conference on Robot Learning, pp. 537-546. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2024
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invari- ant representations for reinforcement learning without reconstruction. International Conference Learning Representations, 2020.
David Zhang, Micah Carroll, Andreea Bobu, and Anca Dragan. Time-efficient reward learning via visually assisted cluster ranking, arXiv preprint arXiv:2212.00165 2022.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse Chicago, IL, USA, 2008
reinforcement learning. In Aaai, volume 8, pp.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.I MOTIVATING QUESTIONS
Inspired by the appendix of (Karamcheti et al., 2023), in this section, we list some motivating ques- tions that may arise from reading the main paper.
QI. The experiments all consider preferences beyond task progress. If the end-user's preference is only progress, can RAPL achieve comparable performance compared to the SOTA TCC-based visual reward (Zakka et al., 2022)?
To investigate this, we return to the X- X-Magical group- ing task (middle plots in Figure 2) where short stick robot needs to push two objects to goal. We removed the grouping preference so the ground truth task reward is consistent with the original benchmark in (Zakka et al., 2022). We trained RAPL with 150 preference queries and compare it with the TCC reward model trained us- ing 500 demonstrations. In Figure 10, we show the pol- icy evaluation success rate during policy learning. We see that RAPL has comparable final success rate compared to TCC and has more stable policy training, showing that it can learn superset of preferences when compared to TCC.
TCC
0.6
Success
0.4
0.
Epoch
Figure 10: X-Magical. Progress-only
reward success rate.
Q2. What makes RAPL different from prior robot learning works that use optimal transport (OT) based visual rewards?
Indeed, OT-based visual rewards have become increasingly popular for learning robot manipulation (Haldar et al., 2023b;a; Guzey et al., 2023). However, key to making the OT-based visual reward successful in Haldar et al. (2023b) fine-tuning the representation model via behavior cloning tasks. This helps the model to capture some task-relevant information at the cost of requiring action labels. Furthermore, by relying on action labels, is unclear if the learned reward can generalize different embodiment. Instead, our approach learns the representation only using preference queries (no action labels) and can generalize to embodiments.
Q3. Why do MVP-OT and TCC-OT achieve near 0 success rate in the robot manipulation experi- ments in Figure 6 and Figure 9?
Recall that both MVP-OT and TCC-OT use optimal transport to match the embedding distribu- tion of the robot and the expert, but they vary which visual representation they use to obtain the embedding.
The MVP encoder is trained via masked autoencoding (He et al., 2022) to reconstruct heavily masked video frames. As such, it captures representations amenable to per-pixel reconstruction. Prior work (Karamcheti et al., 2023) has demonstrated that this representation struggles with higher- level problems (e.g., language-based imitation). We hypothesize this is why MVP-OT struggles to capture preference-relevan features and does not lead to aligned robot behaviors. Our results are also consistent with the experiments in (Haldar al., 2023b) where an OT-based visual reward with a pre-trained MVP representation model gives near 0 success rate for manipulation tasks.
The TCC encoder trained via temporal cycle-consistency constraints, and as such captures repre- sentations that encode solely task progress (e.g.. distance to the goal image). Such representation works well when goal reaching the only preference of the end user. our tabletop grouping task, the end user cares about goal reaching, but they also prefer moving the two objects together to goal region over moving the objects one-by-one. Thus if the robot happens to push one object towards the goal during policy learning, TCC-OT will reward this behavior (since this image is getting "closer" to the goal image) even though this is not preferred by the user.
A.2 EXTENDED RELATED WORK
Visual robot rewards promise to capture task preferences directly from videos. Self-supervised approaches leverage task progress inherent in video demonstrations to learn how "far" the robot
14

---

Published as a conference paper at ICLR 2024
is from completing the task (Zakka et al., 2022; Kumar et al., 2023; Ma et al., 2023) while other approaches identify task segments and measure distance to these subgoals (Sermanet et al., 2016; Tanwani et al., 2020; Shao et al., 2020; Chen et al., 2021). However, these approaches fail to model preferences during task execution that go beyond progress (e.g., spatial regions to avoid during movement). Fundamental work in IRL uses feature matching between the expert and the learer in terms of the expected state visitation distribution to infer rewards (Abbeel & Ng. 2004; Ziebart al., 2008), and recent work in optimal transport has shown how scale this matching to high dimensional state spaces (Xiao et al., 2019; Dadashi et al., 2021: Papagiannis & Li, 2022; Luo et al., 2023). However, key to making this matching work from high- nigh-dimenssonal visual input spaces is a good visual embedding. Previous works used proxy tasks, such as behavior cloning (Haldar et al., 2023a;b) or temporal cycle-consistency learning (Dadashi et al., 2021), to train the robot's visual representation. In contrast to prior works that rely on hard-to-obtain action labels using only self- supervised signal, we propose an OT-based visual reward that is trained purely on videos (no action labels needed) that ranked by the end-user's 's preferences.
Preference- learning. While demonstrations have been the data of choice for reward learning in the past, an increasingly popular approach is to use preference-baseee learning (Christiano et al., 2017: Sadigh et al., 2017; Biyik & Sadigh, 2018; Wirth et al., 2017: Brown et al., 2019; Stiemnon al., 2020; Zhang et al., 2022; Shin al., 2023). Here the human is asked to compare two (or more) trajectories (or states), and then the robot infers a map from ranked trajectories to a scalar reward. This feedback is often easier for people give than kinesthetic teaching or fine-grained feedback (Shin et al., 2023). At the same time, prior works and our experiments show that directly predicting the reward from preference queries and high-dimmnsional input suffers from high sample inefficiency and causal confusion (Bobu et al., 2023b; Tien et al., 2022). To mitigate this issue, (Brown et al., 2020) augments multiple self- supervised objectives like inverse dynamics prediction or enforcing temporal cycle-consistency with the preference learning loss, but this requires additional signals like actions and the additional self-supervised objective may bias the learned rewards towards capturing spurious correlations.
Representation alignment in robot learning. Representation aligmment studies the agreement between the representations of two learning agents. As robots will ultimately operate in service of people, representation alignment is becoming increasingly important for robots to interpret the world in the same way feature selection (Bullard et al., 2018; Luu-Duc & Miura, 2019), interactive feature construction (Bobu et al., 2021; Katz et al., 2021). similarity-implieit representation learning (Bobu et al., 2023a), to learn aligned representations for robot behavior learning. But they either operate on manually defined feature set or learning features in low-dimensional state space settings (e.g., positions). In the visual domain, (Zhang et al., 2020) uses a per-image reward signal to align the image representation with the preferences encoded in the reward signal; however, when the main objective is learning the human's 's reward then assuming a priori access to such reward signal is not feasible. Instead, our work utilizes human preference feedback to align the robot's visual representations with the end user and optimal transport as our embedding-bee bbasdd reward function.
do. Previous work has leveraged user feedback, such nuran-driven
A.3 OPTIMAL TRANSPORT BASED REWARD
Setup. Let o = T= be a trajectory of observations, where T is the trajectory length. Let D+ C Sby be dataset of preferred videos from the preference video dataset and DD. be the set of videos induced by given robot policy RR. We denote ò Rh.,,, + R an observation encoder that maps a x w RGB image to a n, dimensional embedding. For any video o, let the induced empirical embedding distribution be = + Et= Sok(a"), where (ot(ot) is Dirac distribution centered on (n)
Background. Optimal transport finds the optimal coupling μ² E T that transports the robot embedding distribution, PR. of a robot video OR E DD to the expert video embedding distribution, e+ for 04 D4, with minimal cost (as measured by a distance function, e.g. cosine distance). This comes down to an optimization problem that minimizes the Wasserstein distance between the two distributions:
T
T
11" = arg min PEMiPR
Σ
(((0))
(7)
15

---

Published as a conference paper at ICLR 2024
where M(PR, p+) {1r E RTXT, p1 pR. p+} is the set of coupling matrices and C R x R - is a cost function defined in the embedding space (e.g,, cosine distance). The optimal transport plan gives rise to the following reward signal that incentivizes the robot to stay within the expert demonstration distribution by explicitly minimizing the distance between the observation distribution and expert distribution:
r(oRi ΦR)
Σ ((dR(
(8)
Regularized optimal transport. Solving the above optimization in Equation 7 exactly is generally intractable for high dimensional distributions. In practice, we solve a entropy regularized version of the problem following the Sinkhorn algorithm (Peyré et al., 2019) which is amenable to fast optimization:
T E ((0())) cH()).
1 arg min p
E
where denotes the entropy term that regularizes the optimization and is the associated weight,
Choosing an 0+ to match with TR. The reward (8) requires matching the robot to an expert observation video. To choose this expert observation, follow the approach from (Haldar al., 2023a). During policy optimization, given robot's trajectory's observation induced by the robot policy TR. we select the the "closest" expert demonstration of D4 match the robot behavior with. This demonstration selection happens via:
T
0 arg min min o+ED pEM(PR.P+)
Σ
((()k)
(10)
P=
Φ PRLHF-150
Φ ØRLHF-3UU
PRAPL
Franka
Kuka
Figure 11: Manipulation: Attention Map. Visualization attention map for RLHF- 150 demos, RLHF-300 demos, and RAPL with 150 demos for both Franka and Kuka (cross-embodiment) im- ages. Each entry of the figure shows two image snapshots from the relevant demonstration set with the attention map overlaid. Bright yellow areas indicate image patches that contribute most the final embedding; darker purple patches indicate less contribution. á #RLHF-160 is biased towards paying attention irrelevant areas that can induce spurious correlations; contrast RAPL learns to focus on the task-relevant objects and the goal region. ØRLHF-SK 's attention is slightly shifted to objects while still pays high attention the robot embodiment.
A.4 ATTENTION MAP FOR RAPL AND RLHF
In Figure 11 visualize the attention map with novel use of linear permutation and kernel 2D feature map (i.e., activation
inflation (Xu et al., 2022). Specifically, we use channel- map) attention map (Xu et al., 2020). Different from previous works that operate 2D feature maps, our approach utilizes a linear mapping W e RCi, xC.ut on a ID feature, which is average- Mathematically, the procedure can be formulated
pooled from 2D feature Fan E xCx
16

---

Published as a conference paper at ICLR 2024
FID W E, HxW F2D
(11)
H W
where FID E Roui is the aligned features from our proposed RAPL by W E R XCut, Inspired by Xu al. (2022), we can inflate the ID linear mapping W E RCCCu keep the kernel size I, i.e. W E RCix XCout >> Winfiate E 'R RCin equation can be equally represented as
XCout into 2D and
Then above
W Winflate x F2n, D F
82n
(12)
We average the F2D Roui XXx in channel dimension, and visualize the output as our attention map. A visualization of the full process is shown Figure 12.
Features from pre- trained
Weights from RAPL W E B
TD feature
G
Czs
avgPool.
Can
Architecture
Cjn
Cout
Gin
Cout
H
F20
W
Apply W RCuusGin per image patch
Heatmap
Cn
Gi
Caut
x
avgPool
H
Attention Map Visualization
H
f 20
H
W
W
W
W
Figure 12: Attention Map Visualization Method. (top) Visualization of our RAPL architecture. (bottom) Visualization of our process for creating 2D attention map.
A.5 ADDITIONAL RLHF RESULTS: ABLATION ON FEEDBACK DATASET SIZE
In Section 5.2, it's surprising that RLHF fails to learn robot policy in a more realistic environment since its ob- jective is similar to ours, but without explicitly consider- ing representation alignment. To further investigate this, apply linear probe the final embedding and visu- alize the image heatmap of what RAPL's (our represen- tation model trained with 150 training samples), RLHF- 150's (RLHF trained with 150 samples). and RLHF- 300's (RLHF trained with 300 samples samples) final embedding pays attention to in Figure 11.
RAPL RLHF-150 RLHF-300
Succes
0.4
0.2
We see that ΦRAPL learns focus on the objects, the contact region, and the goal region while paying less at- biased towards
150
250
350
Epoch
tention to the robot arm; 0RLHF-150 paying attention to irrelevant areas that can induce spu- rious correlations (such as the robot arm and background area); &RLH 's attention is slightly shifted to objects while still pays high attention to the robot embodiment.
Figure 13: Manipulation. RAPL out- performs RLHF by 75% with 50% less training data.
When deploying FRLHF-300 in Franka manipulation policy learning, observe that policy per- formance is slightly improved (indicating that with more feedback data, preference- based reward prediction could yield to aligned policy), but RAPL still outperforms RLHF 75% with 50% less training data, supporting the hypothesis: RAPL outperforms RLHF with lower amounts of human preference queries.
17

---

Published as a conference paper at ICLR 2024
A.6 ADDITIONAL BROSS-EMBODINIE RESULTS: X-MAGICAL & KUKA MANIPULATION
Figure 14 shows the rewards over time for the three cross- cooss-embodiment video observations (marked as preferred by the end-user's ground-truth reward disliked) in the avoid (left) and group task (right), Across all examples, RAPL rewards are highly correlated with the GT rewards even when deployed on cross-embodiment robot.
Cross-Domaine Avoiding
XAIL
BLAF
TOL
Cross-Domain: Grouping
oxter crikes prggos
(A)
IBI
Divions
progrese
Deskel
exiers ner'ss "wfiness"
Decleed
only pasko large
Padional
exteriey
70
Figure 14: Cross-Embodiment: X-Magical. RAPL distinguish preferred and disliked videos in the cross-embodiment setting.
Figure 15 shows the rewards over time for the two cross-embodiment video observations (marked as preferred by the end- user's ground-truth reward or disliked). Across all examples, RAPL 's rewards are highly correlated with the GT rewards even when deploved on cross enood-mmbodmment robot.
Reward of Kuka Preferral Demo
Cross- Furhudiment: Optimal Transport Plan
ORAn
free
Franka
20
Reward Kuka Disliked Demo C
Kuka Proforral Damo
Kikka (tellawd Demoo
Figure 15: Fross-Emboddimnt:: Manipulation. (center) Expert video on Franka robot, preferred video on Kuka, and disliked Kuka video demo. (left) Reward associated with each video under each method. RAPL's predicted reward generalizes to the Kuka robot and follows the GT pattern. (right) OT plan for each representation. Columns are embedded frames of expert demo on Franka robot. Rows of top matrices embedded frames of preferred demo Kuka; rows of bottom matrices are embedded frames of disliked demo on Kuka. Peaks exactly along the diagonal indicate that the frames of the two videos aligned in the latent space; uniform values in the matrix indicate that the two videos cannot be aligned (i.e., all frames are equally "similar" to the next). RAPL matches this structure: diagonal peaks for appert-and-reeferte baselines show diffused values no matter the videos being compared.
and uniform for expert-and-dislikee. while
A.7 ADDITIONAL COMPLEX ROBOT MANIPULATION TASK WITH VISUAL DISTRACTORS
In this section, we consider more complex robot manipulation task to further validate RAPL's abil- ity to disentangle visual features that underlic an end-user's preferences. We increase the difficulty of the the robot manipulation environment described in subsection 5.2 by adding visual distrac- tors that are irrelevant the human's preferences (left figure in Figure 16). The environment has
18

---

Published as a conference paper at ICLR 2024
Sucesss rale
0.6
0.4
0.2
150
250
Epoch
GT
RAPL
RLIFF
MVP-OT
R3M-OT
ImageNet-Or
Figure 16: Clutter task: Complex robot manipulation with visual distractors. (left) Visualiza- tion of preferences. Green (preferred behavior): the Franka arm grasps the green rectangular prism and uses that to push the red rectangular prism to the goal region. Red (negative behavior): the Franka arm fails to grasp the rectangular prism. (right) Policy training results for our method and additional baselines.
multiple objects on the table various colors-red, green, and goal-region-blue saseiioon-lee--e bluu-buluuueneee and some of objects cubes while others rectangular prisms. The Franka robot needs to learn that the end-user prefers to push the rectangular objects (instead the cubes) efficiently together (instead of one-at-a-time) the goal region.
Task Complexity & Feature Entanglement. Compared to the manipulation task described in subsection 5.2, the Franka arm needs to learn representations that disentangle both semantic preference (grouping) and low-level preference (shape) from RGB images under visual distractors (object color). In addition, the Franka arm needs learn grasp the rectangular prism first and use that to push the second rectangular prism, as it is difficult to directly push the rectangular prism stably using the robot finger gripper, thus increasing the task difficulty.
Privileged State & Reward. The state is 34D: robot proprioception (0jolnts E 12, 3D object po- sitions (P Pobji. . Pobj.....4), and object distances goal (d ggalloobj/2 d dgoalzobsti...
The simulated
human's 's reward is:
group (5) max max(dyoaaa dgonl2abjrcet.. (goallnbjjoc..a)
|Pobsttrs Pobj l|2
0.1
Pobj
Pobjiiui Lint 1|2-
Baselines. In addition to comparing RAPL against (1) GT and (2) RLHIF, we ablate the representa- tion model but control the visual reward structure. We consider four additional baselines that all use optimal transport-base based reward but operate different representations: (3): MVP-OT which is a off-the-shelf visual transformer (Xiao et al., 2022) pre-trained on the Ego4D data set (Grauman al., 2022) via masked visual pre-training: (4): Pine-Tuned-MVER
which fine-tunes MVP-OT using images from the task environment via LoRA (Hu et al., 2021): : (5): R3M-OT which is an off- the-shelf ResNet-18 encoder (Nair al., 2022) pre-trained on the Ego4D data set (Grauman al., 2022) via learning objective that combines time contrastive learning, video-language alignment, and a sparsity penalty; (6): ImageNet-OT which is an off-the-shelf ResNet- 18 encoder pre-trined on ImageNet. We use the same preference dataset with 150 triplets for training RLHF and RAPL.
Results. The right plot Figure 16 shows the policy evaluation history during RL training with each reward function. We see RAPL performs comparably to GT (succ. rate: & 65%) while all baselines struggle achieve a success rate of more than as 10% with the same number epochs.
A.8 SPEARMAN'S CORRELATION BETWEEN GT AND LEARNED VISUAL REWARDS
In this section, conduct quantitative analysis to investigate the relationship between the learned visual reward and the end-user's 's ground-truth reward. Specifically, for each robot manipulation task, compute the average Spearman's correlation coefficient between the learned visual reward
19

---

Published as a conference paper at ICLR 2024
and the end-user's ground-truth reward across 100 video trajectories. We can observe from Table that our learned visual reward shows a stronger correlation to the end-user's ground-truth reward compared to baselines.
Spearman's 's Correlation
Franka Group (Sec. 5.2)
Kuka Group (Sec. A.6)
Franka Clutter (Sec. A.7)
RAPL
0.59
0.47
0.61
RLHF
0.38
0.31
0.26
MVP-OT
-0.1
0.02
0.08
Fine-
0.19
0.02
0.11
Tuned-
MVP-OT
ImageNet-
-0.09
0.12
-0.02
OT
R3M-OT
0.03
-0.14
-0.17
Table Spearman's 's rank correlation coefficient between the GT reward and each learned reward.
A.9 ROBOT MANIPULATION:: RLHF PERCEIVED Vs. TRUE SUCCESS
We investigated if the poor RLHF performance in Fig- ure 6 can be attributed to poor RL optimization or to poor visual reward structure. We compared the true success rate of the RLHF policy (under the true human's mea- sure of success) the "perceived" performance under the RLHF reward. These results are visualized in Figure 17: purple the true success rate and black is the "perceived" reward under the RI.HF learned reward. We see that after 350 epochs, the RLHF learned reward perceives the pol- icy achieving high reward. However, shown in the manuscript's s Figure 6 and in Figure 17, the true success rate is still near zero. This indicates that the RL optimiza- tion is capable of improving over time, but is optimizing poor reward signal that does not correlate with the true
RLHF
0.8
-02
Stersoo rate
-0.4
0.6
0.2
0.8
150 Epoch
250
350
Figure 17: Manipulation. True success rate of the RLHF policy (purple) versus the "perceived" performance under the RLHF reward (black).
measure of success.
20


Reviews:
[{"review_id": "Y4Jwrtlk1x", "reviewer": "Reviewer_NUZE", "summary": "The authors proposes Representation Aligned Preference-based Learning (RAPL).\nIt is a video-only tractable method to use human feedback preferences between three trajectories to align visual representations with what matter to the tasks.\n\nRAPL first asks humans to rate preferences over a set of video triplets.\n\nIt then uses the Bradley-Terry model to interpret and leverage the preferences, but not directly to reward prediction.\nInstead, it focuses all the data on representation alignment on the video pre-trained visual features (ResNet18).\nIt uses the optimal transport method, which transports one video embedding distribution to another, with minimal cost.\nGiven the aligned representation RAPL then uses optimal transport to design a visual reward for the robot policy.\n\nRAPL is then tested against ground truth policy, TCC, and vanilla RLHF. \nWe first tested the same embodiment for both training and testing.\nThis first test is done on a toy X-MAGICAL environment, and then a realistic IsaacGym simulator.\nFor the former, the task is kitchen top cleaning, with avoiding off-limits zone. \nWith the triplet training size of 150, RAPL is able not only to make spatial progress, but to be close to human preference. \nThis results in a higher overall binary success rate on the task.\n\nThe IsaacGym simulator also shows similar results, where the RAPL features are aligned with GT on preferred videos, and no alignment on dislikes.\nFurther analysis shows that RAPL focuses on task-relevant objects and regions is the reason for the significantly higher final task success rate.\n\nRAPL is then tested on zero-shot generalization setup, where the testing robot embodiment is different from that of training.\nThis second test is also done in X_MAGICAL, and then the robot simulator. For both RAPL is the best, closest to the ground truth performance.\nIn fact, on some combination (X-MAGICAL train on short stick, test on medium stick), RAPL improves its performance of the previous non-cross modal setup.", "questions": "How does the visual attention map (figure 11, supplementary material) computed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper presented a novel preference-based representation alignment and robot policy learning techniques.\n\nThe testing section is quite thorough and goes in-depth on the various nuances of what is being learned and accomplished in the actual robot task.", "weaknesses": "No discussion on what is still hard to do or not reliable.\nAlso, evaluation based on difficulty level of the configuration, shape, number of objects needed to be cleared would help toward answering the previous question.\nAnother is to compare with a second, very different, task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VpFASnzk5D", "reviewer": "Reviewer_sCYJ", "summary": "The paper proposes a new method called Representation-Aligned Preference-based Learning (RAPL) to align robot’s visual representation using human preferences. The paper formally defines visual representation alignment problem for robotics as minimizing divergence between human’s latent preference and robot’s representation space. RAPL uses human preference rankings over video triplets to learn representations aligned with preferences of humans and it then uses optimal transport to map the aligned representation to the reward for training. Authors present results in simulation on table top manipulation and 2 block rearrangement tasks. They demonstrate RAPL outperforms and is sample efficient compared to RLHF and temporal consistency based methods in all environments. In addition, they also show RAPL enables zero-shot generalization of rewards to new embodiments for both tasks.", "questions": "1. In Figure 6. the RLHF results are quite poor. Have authors tried scaling training for RLHF for the same experiment to figure out how long does it take to reach RAPL/GT performance?\n\nMy major concern is evaluation on simple object movement tasks and no evaluation on tasks with multi-modal preferences if authors add more experiments that'd address my concerns I’d be happy to update the rating", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Idea of aligning visual representations to human preferences and using that to extract preference based rewards is interesting and novel\n2. Experiments are well designed and demonstrate RAPL outperforms vanilla RLHF in the studied simpler manipulation and block movement tasks is promising.\n3. Results demonstrating zero-shot generalization to new embodiements and sample efficiency gains further strengthen benefits of using RAPL.\n4. Paper is well written and easy to follow", "weaknesses": "1. Experimental setup is promising but the evaluation is done on simpler tasks. With just 150 triplet dataset used for both the tasks to learn preferences the tasks being studied brings up a question about how well this approach works on more complex tasks? For example, the block movement task in x-magical environment only requires agent to avoid the blue box in the environment. This can essentially be treated as a simple obstacle to avoid in the environment which is simple to learn in general. Some example tasks authors could use for evaluation are: block stacking task with constraints where a user prefers stacking blocks based on size i.e. largest block is at lowest followed by smaller blocks irrespective of the color of boxes. This setup tests generalization ability of the policy to different colors and ability to reason about object size based on preferences. My point being preference-based learning needs to be evaluated with tasks that have distractors that are difficult to pick up. It’d be nice if authors can add more experiments on different manipulation tasks with such properties\n2. One baseline I’d like to see in addition to RAPL is how well does a simple contrastive pretrained visual representation as reward with same optimal transport logic performs in comparison to RAPL. It is unclear how beneficial RAPL’s triplet sampling is for the current set of tasks being considered. \n3. For the cross-embodiment experiments the simple tasks of block movement doesn’t necessarily highlight how well the proposed approach is performing. I’d be interested in seeing similar results in simple object pick and place task similar to one asked in W1.\n4. Human preferences are multi-modal in nature i.e. different people have different preferences. The current setup doesn’t consider multi-modal preferences and show experiments comparing different methods in such a setup which seems like a big flaw in current experimental setup. I’d like authors to consider adding experiments that’d demonstrate effectiveness of RAPL under such setting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1STPeDvoKD", "reviewer": "Reviewer_vWH8", "summary": "This paper proposes Representation-Aligned Preference-based Learning (RAPL), a method which leverages human feedback to align visual representations with the end-user. Preference-based learning and optimal transport are used for this alignment and for the policy learning. Experimental results in two different environments reveal that RAPL outperforms prior work by learning a policy which both leads to higher rewards and has better sample efficiency.", "questions": "- How is the proposed approach better than GT for \"Avoiding\"?\n- There seem to be error bars on many of the plots -- how many trials is this over?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Well-written. The paper is very well-written and easy to follow. The figures very much aid in understanding the paper.\n- Experiments. The experiments reveal that the proposed methodology outperforms baseline methods by non-trivial margins. The experiments are done across a number of environments, and evaluate the proposed methodology in a number of different ways.\n- Ablations and Visualizations. Ablations are conducted and visualizations are included to better understand how and why the proposed methodology performs better than the baselines.", "weaknesses": "- Missing / weak baselines.\n  - Zhang et al (2020) is described to be the most similar to the proposed methodology. Why is this not compared to?\n  - The main baseline appears to be TCC (Zakka et al., 2022; Kumar et al., 2023). Which of these (Zakka et al., 2022 vs Kumar et al., 2023) is used as the baseline? Furthermore, both Zakka et al., 2022 and Kumar et al., 2023 report results on {long-stick, medium-stick, short-stick, gripper} tasks -- why are these tasks not used to report results, for a completely fair comparison? This would be the most revealing comparison to the baselines.\n  - A comparison to other pre-training work is made by comparing to MVP-OT, which is revealing. MVP is trained on Ego4D, so there is a massive distribution shift, perhaps explaining the poor performance. For a fairer comparison, could Masked Visual Pre-training be performed on data from the environments directly (e.g. using the trajectories)? \n  - While the visual backbone models are initialized with ImageNet pre-trained weights, may be revealing to include this (without any further training) as another baseline.\n- Data is simulated. A simulated human model is used, rather than learning from real end-user feedback. \n- Figure 3 qualitatively shows the reward correlation -- quantification of such correlation using e.g. Spearman's correlation coefficient, compared to baselines, would be revealing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zOZXKAnGNF", "reviewer": "Reviewer_2gN6", "summary": "The paper attempts to utilize human feedback to synchronize visual representation with the end-user's priorities for the task at hand. The authors present the Representation-Aligned Preference-based Learning (RAPL) method to tackle challenges in visual representation alignment and visual reward learning. Trials conducted in XMAGICAL and robotic manipulation demonstrate that RAPL consistently produces desired robot behaviors with notable sample efficiency and robust zero-shot generalization.", "questions": "All listed in the weakness section. I would consider changing my rating, if the author could address my questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper pioneers the formalization of the visual representation alignment challenge in robotics, framing it as metric learning within the human representation space. Their proposed RAPL targets resolving the alignment issue and facilitates learning visual robot rewards through optimal transport. Experimental results further demonstrate that RAPL consistently yields desired robot behaviors, maintaining high sample efficiency. Lastly, they also show zero-shot generalization across embodiments.", "weaknesses": "1. The task complexity may not adequately highlight the significance of human feedback in robot learning.\n2. While the approach captures human-like preferences, there's no concrete evidence to validate its representation of genuine human preferences in real-world tasks. Enhancing the complexity of preference-based feedback might better align pre-trained visual models with user preferences.\n3. The author should elaborate in the main paper on their method of utilizing privileged access to state information for dataset generation.\n4. While the efficacy of this approach is demonstrated with the X-Magical and ISACCGym robots, it would be beneficial to see its applicability in other robotic embodiment or real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)