Trajectory Improvement and Reward Learning from Comparative Language Feedback
Authors: Zhaojing Yang
Miru Jun
Jeremy Tien
Stuart Russell
Anca Dragan
Erdem Biyik
Venue: CoRL 2024 (2024)
Keywords: Learning from Human Language Feedback
Reward Learning
Human-Robot Interaction
TL;DR: We propose an approach to learn reward function from comparative language feedback, which outperforms traditional preference-based learning method

Abstract: Learning from human feedback has gained traction in fields like robotics and natural language processing in recent years. While prior works mostly rely on human feedback in the form of comparisons, language is a preferable modality that provides more informative insights into user preferences. In this work, we aim to incorporate comparative language feedback to iteratively improve robot trajectories and to learn reward functions that encode human preferences. To achieve this goal, we learn a shared latent space that integrates trajectory data and language feedback, and subsequently leverage the learned latent space to improve trajectories and learn human preferences. To the best of our knowledge, we are the first to incorporate comparative language feedback into reward learning. Our simulation experiments demonstrate the effectiveness of the learned latent space and the success of our learning algorithms. We also conduct human subject studies that show our reward learning algorithm achieves a 23.9% higher subjective score on average and is 11.3% more time-efficient compared to preference-based reward learning, underscoring the superior performance of our method. Our website is at https://liralab.usc.edu/comparative-language-feedback/.

Trajectory Improvement and Reward Learning from Comparative Language Feedback
Zhaojing Yang' Miru Jun'
Jeremy Tien?
Stuart J. Russell
Anca Dragan' Erdem Bıyik'
'University of Southern California UUniversity of California, Berkeley
Learning from human feedback has gained traction in fields like robotics and natural language processing in recent years. While prior works mostly rely on human feedback in the form of comparisons, language is a prefer- able modality that provides more informative insights into user preferences. In this work, we aim to incorporate comparative language feedback to iteratively improve robot trajectories and to learn reward functions that encode human preferences. To achieve this goal, we learn shared latent space that integrates trajectory data and language feedback, and subsequently leverage the learned latent space to improve trajectories and learn human preferences. To the best of our knowledge, we are the first to incorporate comparative language feedback into reward learning. Our sim- ulation experiments demonstrate the effectiveness of the learned latent space and the success of our learning algorithms. We also conduct human subject studies that show our reward learning algorithm achieves 23.9% higher subjective score on average and is 11.33 11.3% more tine-ffficiett compared preference based reward learning, underscoring the superior performance of our method. Our website is at https://lirala: usc. edu/ chtp//ampaaaiive -ww..aa.gg---f-eeaeaa... feedback/.
Abstract:
Keywords: Learning from human feedback, reward learning, HRI
Introduction
Learning from human feedback has gained significant popularity in robotics, leading to the study of different forms of human feedback: demonstrations [1, 2, 3, 4), preference comparisons [5, 6, 7, 8), rankings [9], physical corrections [10], visual saliency maps [11], human language [12, 13], etc. Among these, preference comparisons grew popular for its simplicity and ease of use, especially compared demonstrations [7]. Preference comparisons often involve users choosing between a pair of choices. Using these selections to learn a reward function and train a policy is known as reinforeement learning from human feedback (RLHF) [6] or more generally preference-bar learning [5, 14]. It has proven applicable to broad range of fields ranging from robotics [7, 15] to natural language processing [16], from traffic routing [17] to human-computer interaction [18].
Despite their successes, preference comparisons suffer from problems [19] such as the unreliability of human data and the limited information bandwidth, i.e., each pairwise comparison contains at most bit of information [20]. There has been research to provide a better interface [21], allowing the users specify their preferences for every feature, but they require features to be designed by hand. As an alternative form of human feedback, comparative language is considerably more informative than preference comparisons, allowing users prioritize specific aspects. For example, it allows users to naturally indicate their preference about speed by simply saying, "the robot should move faster," making more intuitive and interpretable.
In this work, we aim to leverage comparative language feedback to learn the human preferences, i.c., their reward functions. In pursuit of this objective, first leam shared latent space that aligns tra- jectories and language feedback. This alignment enables the robot to comprehend human language feedback, leverage it for adapting its behavior to learn and better align with the human's 'ss prefer- ences. To test the effectiveness of our approach, we conduct experiments in two simulation environ- ments and human-subject study with real robot. The results suggest that reward learning from
8th Conference on Robot Learning (CoRI. 2024), Munich, Germany.

---

want the robot to pick up spoon
Which do I prefer (A or B)?
How do I want change it?
B
"Move farther from stove."
Learning from Comparison
Learning from Language
Figure I: An image from our human subject studies where the human user wants the robot to pick up the spoon. Compared to traditional comparison preference learning, our language preference learning enables users to give more informative feedback, which helps the rohot to capture human preferences more efficiently.
comparative language feedback outperforms traditional preference comparisons in performance and time-efficienny, and is favored by most of the users.
2 Related Work
Before formally defining the problem, we will first review existing works in learning from human feedback, preference-basee learning, and robot learning from human language feedback.
Learning from Human Feedback. Several promising approaches leverage human feedback to train robots [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 22]. However, the most popular methods of learning from demonstrations and rankings involve trade-off between informativeness and ease of use [6, 19, 23, 24]. Furthermore, none has taken advantage of the expressive nature of language to develop a method of reward learning from comparative language feedback. Addressing this gap will provide more scalable methods for training robots, particularly in complex environments where traditional feedback forms may be insufficient or impractical.
Preference-based Learning. Based preference comparison feedback, preference-bare learning widely used for its logical intuitiveness and ease of use. Learning from rankings beneficial to deep reinforcement learning [6]. but suffers from several shortcomings [19] such as the reliability of human data and difficulties of choosing from equivalent choices. Works have sought to improve its shortcomings in time- and sample-effciency by actively generating pairs based on information gain [20] or volume removal [5], and doing these in batches of pairs [25]. Sikchi et al. [26] and Brown et al. [27] integrate preference feedback into imitation learning, demonstrating superior performance in their respective approaches. However, the fundamental limitation remains that each pairwise comparison provides at most 1 bit of information [20]. In addition, users struggle to choose between two similar trajectories [7]. In contrast, our method of comparative language feedback offers more informative input after observing just one trajectory, while maintaining intuitiveness and ease of use.
Robot Learning with Human Language Feedback. Benefiting from advancements natural lan- guage processing, works have leveraged language for adjusting robot trajectories [13, 28, 29, 30, 31, 32), fine-tuning language models [12], and reward shaping [33, 34]. For example, Shi et [31] use language-conditioned behavior cloning (LCBC) for corrective language commands and improving policies. Lynch et al. [35] describe an approach for real-time guidance from humans using natural language in order to achieve goal. Cui et al. [36] introduce an approach to use human language feedback to correct robot manipulation in real-time via shared autonomy. Goyal et al. [33] utilize human language combined with past action sequences to generate rewards, and in a separate work, Goyal et al. [34] leverage natural language to map image observations to rewards. However, these works focus on using human language the instruction correction to the robot, none of them has explored learning the reward function of humans entirely from comparative language feedback, which offers benefits over policy learning with generalizability and explainability. Additionally, our method iteratively updates the learned reward function for better preference alignment, providing an advantage over one-shot corrections [13, 29].
3 Problem Definition
Having reviewed the related literature, we are now ready to formally define the problem. We model the robot as a decision making agent in standard finite-horizon Markov decision pro-

---

Escode:
Wlob
Uneeaker
11
Sum
97-1 1
10-0
d/B
Ercodes
Φ 4 b >
Reward Leamiing
(squak)
Eneedkr
Align
Gendrad
Eacodat
"Move cosee
75
4/1
"Pick up the cute batua
V/
lke cabe"
(a) Model Architecture
(b) Language-based Reward Learning
Figure 2: Overview of our approach. (a) Architecture of the model that learns a shared latent space between trajectories and comparative language feedback. (b) Comparative language-based reward learning.
cess (MDP). Each robot trajectory consists of state-action pairs for T time steps': 7 {s5, 40), (=1,a)) ...,... (CT-1,0T-1))
A reward function R(st, ae), which is only known by the human user, encodes human preferences regarding the task. The reward
trajectory is defined as: R(r) = - R(st, Qz). ) For each trajectory, the human may provide language feedback that attempts improve the trajectory (based on the reward function) in one aspect, e.g., speed, distance to objects, height of the robot's arm, etc. As an example, the human might say "move farther from the stove" (see Figure 1) if the robot's trajectory would be higher-reward in that way. Our goal is to develop a framework where we learn the reward function based on such feedback.
Unfortunately, this task is doomed without any information about how language feedback relates to the different aspects of the task. In this work, we learn this relation with an offline pretraining phase where we collect dataset that consists of pairs of robot trajectories and a language label describing how the two trajectories differ. We use this dataset to learn encoders that map trajectories and language feedback onto shared latent space. We leverage these encoders learn the preferences (reward functions) of different users based on their language feedback.
In the next section, we detail our approach to learning the latent space and explain how this learned latent space utilized for iterative trajectory improvements and reward learning.
4 Our Approach
As shown in Figure 2, our approach is composed of two stages: First, we learn a shared latent space where robot trajectories and human language feedback are aligned. Second, leverage this learned latent space to (1) improve robot trajectory or (2) learn human preferences.
4.1 Learning the Shared Latent Space
To learn a shared latent space for trajectories and language feedback, we collect a dataset of pair of trajectories and is a language utterance that
r,,,7", tuples, where T" and 7 describes the difference between the two trajectories. Note that this language utterance does not necessarily align with the human's preferences about the robot: it just describes difference be- tween the trajectories - it is possible to have I "move faster" if the robot moves faster in B than 7 even though users want the robot move slowly.
After collecting such a dataset, we propose the model visualized in Figure to learn the shared latent space between trajectories and language utterances. Similar to [37]. use neural network to encode each state-action pair (st, a;) from the pair of trajectories, 7 and -" to embeddings of and o respectively. We want these embeddings to contain information about aspects of the trajectories that the human may care and give feedback about.
To achieve this, we align the difference between the embeddings Φf and B with embeddings th of the language feedback by using the following loss function:
Latign (T B 1) = log (sigmoid (w/7 (eB +^1))
(1)
'Our work trivially extends to the cases where trajectory length not tixed.
3

---

where the sigmoid can be considered as the probability that the language utterance I is inputted for the pair of trajectories -a and n Intuitively, when o - and 1 align, i... have the same direction for fixed magnitudes, the loss is minimized. This alignment will enable us to acquire the embedding of an improved trajectory as o + when given an initial trajectory 7 and comparative language feedback I.
Instead of training a language encoder from scratch, which would require prohibitively large and diverse dataset of -,,,,,,) tuples, we use a pretrained TS model [38]. To align embeddings of trajectories with those of language feedback, we first freeze T5 model and train the trajectory encoder. Subsequently, we perform co finneuuning of both components.
Additionally, incorporate normalization term into the loss function to help balance the mag- nitude of the embeddings. This term constrains the norms of the trajectory embeddings to remain below 1 and the norm of the language embeddings be close to 1:
L (T 7B ,1) = a (max{1l0<) 1,0} + max [ 1,0)) b- 1112 111
(2)
where a and b are two hyperparameters.
Overall, the objective we in the model training consists of two terms:
(3) which we use to train the trajectory encoder and finetune T5. Training this architecture gives us the ability to encode any trajectory and language utterance in the same latent space. In the next subsection, we demonstrate how this latent space is useful for iteratively improving robot trajectories and for learning human preferences based on their language feedback.
U(TA,7B () = ( ,,,,,, + Lnomm (7 -B,),,
4.2 Utilizing the Learned Latent Space
The shared latent space aligns robot trajectories with human language feedback. This alignment enables an intuitive understanding of user preferences. We will now explore two primary ways to leverage this learned latent space: first, iteratively improve the robot's trajectory, and second, to accurately learn user preferences.
4.2.1 Iterative Trajectory Improvement
Firstly, we leverage the latent space to iteratively improve an initial suboptimal robot trajectory. We start with showing the initial trajectory 70 to the user and asking for language feedback 1 Upon receiving human's language feedback, we use our trained encoders to compute the trajectory embedding 66 and language embedding thro We then find the improved trajectory 7 such that its difference with 7 best aligns with the human's 's language feedback based on cosine similarity:
w/TT (d' )d)
y1 argmax 7' ||0/ | 1
In this work, we search over a discrete, predefined set of trajectories to solve the optimization in Eq. (4) for compu- tational efficiency. It is, however, possible to use reinforcement learning or model--reeicction control algorithms to solve this optimization the expense of increased computational cost.
We iteratively continue this process for N iterations to obtain 70, _0,7'
4.2.2 Reward Learning from Comparative Language Feedback
In addition to improving trajectories, we also utilize the learned latent space to learn the user's preference, i.e., their reward function. Previous approaches ask users to label their preference from a pair of trajectories, which requires the users to watch two trajectories in order to give at most 1 bit of information. In our language- based reward learning approach, for each query i, we only show users one trajectory T; to collect language feedback I, based on their preferences. Given one trajectory, we construct an improved trajectory f; de, Φr, + th,, Note that 7 is just an imaginary trajectory that maps to oi,, + vh, in the learned latent space. Then following the Bradley- Terry model [39], which is commonly used in preference-based learning, model a preference predictor with the reward function rgg as:
expr)(0+.) Pe(f, > T3)= exprę(0*.) + expre((r,)
(5)

---

where is neural network parameterized with &. The reward function rg is trained by minimizing the following negative loglikelihood loss:
LEsplicit
log Pe(fi > Ti)
(6)
We now make an important observation about the comparative language feedback. When the user tells the robot to move farther from the stove, it indeed indicates a preference between the original trajectory and the improved trajectory that moves farther from the stove. However, this feedback contains much more information than this comparison. The user could use any comparative language utterance teach the robot, but they specifically selected one about the distance to the stove. This means, with high probability, the improvement the robot may get from this feedback is higher than that of any other comparative language feedback.
Mathematically, this indicates preference for 4 over : Or. = or. + vi where is any language utterance other than 4, Again apply the Bradley- -enry model utilize this implicit preference: exprr((++.)
PG 7.) rx + exprr((
Based on this, we sample language feedback from a set of pre-collected language utterances other than 4, and minimize the following loss about the chosen language feedback:
Limplicit = k j=0
log Pe(f: > Fi.s)
(8)
Overall, the loss for reward learning from comparative language feedback is follows:
k-1
Lnevard LExplicit + Implicit
Σ
(log Pç(f, x Ti)
log Pş(fs f.)))
(9)
x=0
5=0
Training the reward function with this loss enables us to efficiently capture human preferences through comparative language feedback.
Experiments
We conducted simulation experiments and human subject studies to validate methods with di- verse tasks and features that humans may give feedback about.
5.1 Simulation Experiments
Synthetic humans. For simulation experiments, we synthesize the comparative language feedback from simulated human with simplified reward function R:
R(se, ot) w 0(s az)
(10)
where 0 is a function that maps a state-action pair to vector of high-level features (e.g.. speed, distance to the stove). We similarly define ((r) denote the sum of features over trajectory т. w
is a vector of weights that maps the features to a scalar reward value. Both 0 and w are unknown to our algorithm.
Given a true reward function w, let r" be the optimal tra- jectory under reward Then, the synthetic human gives the noisy language feedback ( when shown trajectory 7
( (softmax c -(0-)
(11)
where 0* and 0 denote the true cumulative features of I' and 7 respectively, O denotes element-wise multipli- cation. The function takes sample from the output of the softmax, which is computed across different features we designed for the simulated humans (see Appendix A), and outputs a language feedback that corresponds to the sampled feature. The language feedback output is chosen randomly from all language utterances of that feature in the GPT-augmented dataset (see Appendix E).
the Figure 3: Each dataset sample is pair of
trajectories and a language feedback.
Environments. We experimented in two simulation environments: Robosuite [40] and Meta-World [41]. Robosuite has a Jaco robot arm at a table with a cube and a bottle, and the task is to pick up the cube. The states are given 640 x 480 RGB images (resized to 224 224) and 25-dimensional
5

---

proprioception. The action space is 4-dimensional (the end-effectoo always points down). Meta- World has a Sawyer robot arm at table, and the task is to push button on the side of the table. The states are given as 480 x 320 RGB images (resized to 224 x 224) and 20-dimensional proprioception. The action space is 4-dimensional (the end-effector always points down).
Learning the Latent Space. Lists of language feedback were created for hand-crafted features (see Appendix A) to indicate a change (e.g.,
Robosuite
Meta-World
0.6
Reward
Reward
('Move higher', 'Move lower'} for height). They were augmented with GPT 3.5 [42] to 629 sentences for Robosuite and 660 for Meta- World. The splits are 480, 74, 75 and 492, 84, 84 for the training. validation, and test sets. Note these features are only for syn- thetic dataset creation - training our architec- ture does not require hand-designed features.
04-
04
Improwed Trajectory
Optimal Trajectory
n0
on
10
16
15
Iterations
Iterations
Results of experiments where we use simu- lated human language feedback to iteratively improve a robot trajectory (mean I std over 100 runs). The dashed line represents average reward of optimal trajectories.
Figure 4:
We trained RL policies with randomized weights w using stratified sampling [43] over the features, then generated rollout trajectories with timesteps = 500 for each environment. For Robosuite, we generated 448 unique rollouts and 324 for Meta- World. The splits were 359, 44, 45 and 260, 32, 32 for training. validation, and test sets. Trajectories were paired within each set and matched with a language feedback that deseribes the change in each feature from 7 to - (see Figure 3).
We train the encoders with the loss in Eq. (3), and use accuracy as the metric evaluate:
Acc [[[=A,7B,1 ui 6o_ > 0]
(12)
Total number of samples
Training the encoders with the training set of trajectories and language utterances, we observed a test accuracy of 84.9% in Robosuite and 82.9% in Meta-World, This indicates the trajectory and lan- guage embeddings are well-aligned. We found co-finetuinng both trajectory encoder and language model, versus utilizing frozen language model and only training the trajectory encoder, helps map trajectories and language utterances to the same latent space (see Table 2 in the Appendix).
Iterative Trajectory Improvement. Next, we conducted iterative trajectory improvement exper- iments based on the method we presented in Section 4.2.1. We set the number of iteration steps N 15 and repeat the full experiment over 100 random seeds. The true rewards of these trajec- tories are shown in Fig. 4. In both environments, we consistently improve the trajectories, which showcases the effectiveness of the learned latent space and the improvement algorithm. However, it can be noted that our algorithm cannot reach the performance of the optimal trajectory. This is because every improvement iteration completely independent from the previous iterations and the robot may get stuck in a loop between good, but non-optimal, trajectories.
Our reward learning algorithm presented in Section 4.2.2, on the other hand, utilizes the entire history of human feedback, so it should not suffer from this problem. We will now demonstrate this.
Reward Learning. We again randomly initialize the reward weights to simulate human feedback. For each environment, we simulate three synthetic humans, and run the experiments with three random seeds for each simulated human. We use the loss in Eq. (8) to learn the reward function. We adopt cross-entropy CE(Pun: tw: P&)) as the evaluation metric, where
exp w ((rA)
((
(13) Another metric is the true reward value of the optimal trajectory selected from the test set based on the learned reward, reflecting how close the learned reward is to the true reward. All reward values are normalized between 0 and 1.
(T x expw ((74)((44)) exp 0(7B) P5(TT )イ
(xpr + expr
We compare our method against reward learning from comparisons, where for each query, the simu- lated user chooses a preferred trajectory from pair (3) and with probability shown in Eq. (13).
As indicated in Figure 5a, the cross-entropy of our method decreases faster than that of comparison preference learning in both environments, demonstrating that our approach converges more quickly than the baseline. Meanwhile, Figure 5b shows that the true reward of optimal trajectories reaches

---

Robosuite
Meta-World
Robosuite
Meta-World
Cross-Entropy Entropy
Crose Entropy
Reward
Reward
05
Language
Language
24
Compatison
Comparison
Queries
Number Queries
Number Queries
Number of Querles
(a) Cross-Entropy
(b) True Reward Values of Optimal Trajectory
Figure 5: Results of reward learning, averaged over 3 seeds. (a) Cross Entropy: our method converges faster.
(b) True Reward of Optimal Trajectory: Our approach.
a value of in shorter time with our language-base based method, especially in Meta-World, further supporting that it learns the reward function more efficiently.
Ablation Study. We conducted an ablation study evaluate the different components of the loss function (Eq. (8)). As indicated in Figure 6a, the combination of both components outperforms using each component individually, demonstrating the validity of our loss function design. Also, using both components fits the true reward function better, as shown in Figure 6b.
Robosuite
Meta-World
Robosuite
Meta-World
07:
Crose Entropy
Cross Entrepy
Reward 7o
Reward
05
Logitr Leniee
0..
Laptec + Lempiih
Lomic
Liipich
Limptat
0.4
Lieatee
100
Number Queries
Number of Queries
Number of Queries
Number of Queries
(a) Cross-Entropy
(b) True Reward Values of Optimal Trajectory
Figure 6: Results of Ablation Study. (a) Baseline Comparison: our method consistently outperforms the base-
line. (b) Ablation Study: Using two components is better than applying each of them alone.
5.2 User Studies
To verify the effectiveness of our approach, we conducted human subject studies by recruiting 10 subjects (4 female, male) from varying backgrounds and observing them to interact with our real robot setup. Our study has been approved by the IRB office of the University of Southern California.
Real Robot Setup. We used WidowX 250 6DOF robot with the setup of [44] in front of kitchen set (Figure 7). The task is to pick up a spoon while avoiding going over a pan on the stove. The states given 480 x 320 images resized 224 x 224, and 22-dimensional proprioception. The actions are 7-dimensional (6DOF+gripper). To train the latent space, we collected dataset of 321 trajectories with varying levels of success at picking up the spoon, avoiding the pan, and speed. These trajectories were divided into 192 for training, 64 for validation, and 65 for testing. 496 GPT- augmented sentences were split into 297 for training, 99 for validation, and 100 for testing. These sentences were paired with the trajectories in the same manner as in the simulation experiments. Another set of 32 trajectories was collected for the user studies following the pretraining phase.
Study 1: Trajectory Improvement. Users start with suboptimal trajectory from the dataset and give language feedback for a maxi- mum of 10 iterations until they are satisfied.
Improve Trajectory:
Improve Trajectory:
Average Score
Average Attribute Scores
Iterations to Sutisfaction
Study 2: Preference Learning. For 20 itera- tions, users give feedback (preference compar-
Experience Speed Satisfaction Attributes
Number of llerations
ison or comparative language) on random tra- jectories and rate the optimal trajectory from the dataset with respect to the learned reward after every 5 iterations. To avoid bias, 5 users began with the comparison based method and 5 with the comparative language based method.
Figure 8: (Left) Average attribute scores. (Right) Itera- tions to satisfaction.
After each study, users were asked to complete post-study surveys (see Appendix B).
7

---

Figure 7: (Left) Closeup view of the kitchen set with the spoon hanging on the wall, above the pan on the stove. (Right) WidowX 250 6DOF robot arm, user, and text prompt for experiments.
Preference L Learning:
Preference Learning:
Table 1: User satisfaction aspects, not
Average Score for Attributes
Average Time per Query
mutually exclusive between methods
Average Score
5-
Compasen
Language
Time per Query
75-
Satisfactory
3-
Method
Language Comparison
25
Time Efficient
10
1/10
Experience Speed Adaptation Trust
Attributes
LanguageComparison Method
Convenient
Adaptable
6/10
8/10
3/
8/10 10
Figure 9: (Leff) User ratings. (Right) Time per query.
Results. Our post-study surveys reveal that the first study has consistently positive responses for user experience and speed of adaptation, but has two peaks in iterations to satisfaction (Fig. 8). We conjecture that the dataset of 32 trajectories may not have contained trajectories that the users de- sired. As for the second study, our Singuage-based method scored better than the comparison-based method for all attributes. The average score of the language-basee method over all attributes is 23.9% higher than the comparison-based method, illustrating the superior performance of our ap- proach. In terms of time-efficiency, our method takes 11.3% less time for users to answer each query as shown in Fig. 9, even though it includes the time it took for users to type their feedback. Indeed, users found the language method to be more time efficient and adaptable but found compar- ison method to be more convenient (Table 1). Fig. 10 shows the given by the learned reward function. To quantitatively assess the learning efficiency, we follow prior work in active learning [9, 45, 46] by checking the area under curve (AUC) for both lines. We found that the AUC for the comparative language line is statistically significantly higher than that of preference comparison line (p- < 0.05). This indicates that our munguage-basedd approach more efficiently captures human preferences compared to the comparison method.
rating of optimal trajectories
An unexpected observation from Fig. 10 is that humans' rat- ings decreased from 5 queries to 10 queries while using com- parative language feedback. This may be because of the rela- tively small subject size: it is possible that the average rating after 5 iterations, where the standard deviation is indeed large,
Preference Leariing
Very Bad 5 Very Good]
Optimal Rating Trajectories
Average Rating
Language
Comparison
is higher than should be. Another potential explanation is that humans may be more lenient to the robot after 5 iter- ations than 10 iterations, especially in comparative language feedback where they only watch the robot 5 times give their feedback as opposed to 10 times in preference comparisons.
Iteration
Figure 10: Average ratings of the opti- mal trajectory shown every 5 iterations for each method.
Conclusion
In this paper, we presented a robot learning framework to learn human preferences using compara- tive language feedback. For this, we aligned trajectories with language feedback in a shared latent space, then used it to improve trajectories and learn preferences. Experiments in simulation environ- ments and real-world user studies suggest that our approach consistently outperforms comparison preference learning and is favored by most users for aspects such time efficiency and adaptability.
Limitations and Future Work. Even though our user studies show some generalizability in the comparative language feedback, our model is limited by the feedback about the objects seen in the pretraining data. Additionally, we sample queries uniformly at random during reward learning- Based on these limitations, the future works include: (1) use image annotations to generate language feedback containing all objects, and (2) implement active learning to reduce the number of iterations required to reach the optimal trajectory.
8

---

Acknowledgments
This work was partially supported by NSF HCC grant #23 and a gift in support of the Center for Juman-Compatibee Al at UC Berkeley from the Open Philanthropy Foundation.
References
[1] D. Sadigh, S. S. Sastry, S. A. Seshia, and A. D. Dragan. Planning for autonomous cars that leverage effects on human actions. In Proceedings of Robotics: Science and Systems (RSS), June 2016.
[2] Ross, G. Gordon, and D. Bagnell. reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[3] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. 2019 International Conference on Robotics and Automation (ICRA), pages 8077-8083. IEFF, 2019.
[4] R. Hoque, A. Balakrishna, E. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. In 5th Annual Conference Robot Learning, 2021.
[5] D. Sadigh, A. D. Dragan, S. S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. Proceedings of Robotics: Science and Systems (RSS), July 2017. doi: 1..5077/RSS.2117.IIII.
[6] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[7] E. Biyik, D. P. Losey, M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1):45-67 67, 2022.
[8] N. Wilde, E. Biyik, D. Sadigh, and S. L. Smith. Learning reward functions from scale feedback. In Proceedings of the Sth Conference on Robot Learning (CoRL), 2021.
[9] V. Myers, E. Biyik, N. Anari, and D. Sadigh. Learning multimodal rewards from rankings. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
[10] A. Bajesy, D. Losey, M. K. O'malley, and A. D. Dragan. Learning robot objectives from physical human interaction. In Conference on Robot Learning, pages 217-226. PMLR, 2017.
[11] A. Liang, J. Thomason, and E. Biyik. Visarl: Visual reinforcement learning guided by human saliency. In ICRA2023 Workshop on Pretraining for Robotics (PT4R), 2023.
[12] A. Campos and J. Shern. Training language models with language feedback. In ACL Work- shop on Learning with Natural Language Supervision. 2022., 2022.
[13] P. Sharma, B. Sundaralingam, V. Blukis, C. Paxton, T. Hermans, A. Torralba, J. Andreas, and D. Fox. Correcting robot plans with natural language feedback. In Proceedings of Roboties: Science and Systems (RSS), 2022.
[14] C. Wirth, R. Akrour, G. Neumann, J. Fürnkranz, et al. A survey of preference-base based reinforce- ment learning methods. Journal of Machine Learning Research, 18(136):1-46 2017.
[15] Y. Wang, Z. Sun, J. Zhang. Z. Xian, E. Biyik, D. Held, and Z. Erickson. R1-vim-f: Reinforce- ment learning from vision language foundation model feedback. In International Conference on Machine Learning (ICML), 2024.
[16] I. Ouyang, J. WI, X. Jiang. D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.

---

[17] E. Biyık, D. A. Lazar, R. Pedarsani. and D. Sadigh. Incentivizing efficient equilibria in traffic networks with mixed autonomy, IEEE Transactions on Control of Network Systems, 8(4): 1717-1729, 2021.
[18] N. Dennler, D. Delgado, D. Zeng, S. Nikolaidis, and M. Matarić. The rosid tool: Empower- ing users to design multimodal signals for human-robot collaboration. In 18th International Symposium on Experimental Robotics (ISER), 2023.
[19] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire. T. Wang, S. Marks, C.-R. Segerie, M. Carroll, A. Peng, P. Christof- fersen, M. Damani,
Slocum, U. Anwar, A. Siththaranjan, M. Nadeau, E. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, E. Biyik, A. Dragan, D. Krueger, D. Sadigh, and D. Hadfield- Open problems and fundamental limitations of reinforce- ment learning from human feedback. Transactions Machine Learning Research (TMLR), 2024.
[20] E. Biyik, M. Palan, N. C. Landolfi, D. P. Losey, and D. Sadigh. Asking easy questions: A user- friendly approach to active reward learning. In Proceedings of the 3rd Conference on Robot Learning (CoRL.), 2019.
[21] C. Basu, M. Singhal, and A. D. Dragan. Learning from richer human guidance: Augment- ing comparison-baxe learning with feature queries. Proceedings of the 2018 ACM/IEEE International Conference on Human-Rone Robot Interaction, pages 132-140, 2018.
[22] J. Spencer. S. Choudhury, M. Barnes, M. Schmittle, M. Chiang. P. Ramadge, and S. Srinivasa. Expert intervention learning: An online framework for robot learning from explicit and implicit human feedback. Autonomous Robots, pages 1-15, 2022.
[23] Abbeel and A. Ng. Apprenticeship learning via inverse reinforcement learning. Proceedings, Twenty-First International Conference on Machine Learning, 1CML 2004, 09 2004. doi: 10. 00119777-0038--016--- 8_417
[24] B. Akgün, M. Cakmak, K. Jiang, and A. L. Thomaz. Keyframe-based learning from demon- stration. International Journal of Social Robotics, 4:343 355, 2012. URL https: //api. semanticscholar. www//orgppoueeoooooo
10004846.
[25] E. Biyik, N. Anari, and D. Sadigh. Batch active learning of reward functions from human preferences. ACM Transactions Human-Robot Interaction.
[26] H. Sikchi, A. Saran, W. Goo, and S. Niekum. A ranking game for imitation learning. Transac- tions on Machine Learning Research, 2023.
[27] D. S. Brown, W. Goo, and S. Nickum. Better-hhan-demonstaaoo imitation learning via automatically-ranko. demonstrations. In Conference on robot learning, pages 330-359. PMLR, 2020.
[28] A. Bucker, L. Figueredo, S. Haddadinl, A. Kapoor, S. Ma, and R. Bonatti. Reshaping robot trajectories using natural language commands: study of multi-modal data alignment using transformers. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 978 984. IEEE, 2022.
[29] A. Bucker, L. Figueredo, S. Haddadin, A. Kapoor, S. Ma, S. Vemprala, and R. Bonatti. Latte: Language trajectory transformer. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7287-7294. IEEE, 2023.
[30] J. Yow, N. P. Garg, M. Ramanathan, W. T. Ang, et al. Extract-explainabll trajectory corrections from language inputs using textual description of features. arXiv preprint arX1v.2401.03701, 2024.
[31] I. Shi, Z. Hu, T. Z. Zhao, A. Sharma, K. Pertsch, Luo, S. Levine, and C. Finn. Yell at your robot: Improving on-the-fly from language corrections, 2024.
10

---

[32] M. Han, Y. Zhu, S.-C. Zhu, Y. N. Wu, and Y. Zhu. Interpret: Interactive predicate learning from language feedback for generalizable task planning. arXiv preprint arXiv:2405. 19758, 2024.
[33] P. Goyal, S. Nickum, and R. J. Mooney. Using natural language for reward shaping in rein- forcement learning, 2019.
[34] Goyal, S. Niekum, and R. Mooney. PixI2r: Guiding reinforcement learning using natural language by mapping pixels rewards. In Conference Robot Learning, pages 485-497, PMLR, 2021.
[35] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots real time. IEEE Robotics and Automation Letters, 2023.
[36] Y. Cui, S. Karamcheti, R. Palleti, N. Shivakumar, P. Liang, and D. Sadigh. No, to the right: Online language corrections for robotic manipulation via shared autonomy. In Proceedings of the 2023 ACM/IEEE International Conference Sluman-Robor Interaction, pages 93-101, 2023.
[37] M. Katz, A. Malcki, E. BıyIk, and M. J. Kochenderfer. Preference- learning reward function features. arXiv preprint arXiv:2103.02727, 2021.
[38] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[39] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[40] Y. Zhu, J. Wong, A. Mandlekar, R. Martín-Martin, A. Joshi, S. Nasiriany, and Y. Zhu. robo- suite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 12293, 2020.
[41] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100, PMLR, 2020.
[42] OpenAl. Gpt-3.5, Online. https:/plattformoppeeeo wwwp:r/wwwmooooiiimmmme wtt.:/opan..c.occsooooeeeo httpaaac....////mmmle/pp333355
[43] M. J. Kochenderfer and T. Wheeler. Algorithms for optimization. Mit Press, 2019.
[44] Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and
S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint 1xXiv:2099113999 2021.
[45] M. Culver, D. Kun, and S. Scott. Active learning to maximize area under the roc curve. In Sixth International Conference Data Mining (ICDM '66), pages 149-158. IEEE, 2006.
[46] D. Fuchsgruber, T. Wollschläger, B. Charpentier, A. Oroz, and S. Günnemann. Uncertainty for active learning on graphs. arXiv preprint 4rX1v:2405.01462, 2024.

---

A Dataset Features
Robosuite Features
The height of the robot's end-effector
The speed of the end-effector
The distance between the end-effector and the bottle
The distance between the end-effector and the cube
How well the robot lifts the cube (i.e., level of success of cube-lifting task)
The level of success of the cube-lifting task is quantified by: 1) whether or not the cube lifted
above the height of success threshold or weighted sum of the distance to the cube and whether or not the end-effector is grasping the cube (the level of success in picking the spoon).
Meta-Worid Features
The height of the robot's end-effector.
The velocity of the robot's end-effector.
The distance between the end-effector and the button.
WidowX Features
The velocity of the end-effector
The distance between the end-effector and the pan
How well the robot picks up the spoon (i.e. level of success)
The level of success is quantified by: 1) whether the spoon is grasped from the hook 2) a weighted sum of the distance the spoon and whether the end-effector grasping the spoon.
B User Study Surveys
Here are the questions in our user study surveys.
Pre-Study Survey. To assess the subjects' experience and level of skill with robotics and machine learning, we conducted a short pre-study survey before any of the experiments.
1. "Age"
2. "Gender"
3. "Race"
4. "Highest level of education"
5. "Please deseribe your level of robotics experience, on scale from to 5."
6. "Please deseribe your level of machine learning experience, scale from 1 to 5."
7. "Have you ever interacted with robot before? Please describe:"
Post-Study Survey - Improve Trajectory. Users filled out this post-study survey after completing the Improve Trajectory experiment.
1. "Are you satisfied with the final trajectory?" [1 Completely unsatisfied 5 Completely satisfied]
2. "How many iterations did it take for the robot to adapt to your feedback?"
3. "How fast does the robot adapt to your feedback?" [1 Very slow 5 Very fast]
4. "How did the robot's 's learning capabilities affect your interaction experience?" [] Nega- tively 55 Positively]
5. "Overall, do you think the approach is effective?" [Y/N]
6. "Do you have any other comments?"
12

---

Post-Study Survey Preference Learning. Users filled out this post-study survey after complet- ing two methods of the preference learning experiment.
1. "What did you like about these methods? For each aspect below, please click the circle you found the method satisfactory in that regard:" [Adaptability, Time efficiency, Conve- nience, None of them]
2. "Could you explain the reasons?"
3. "How long did it take for the robot to adapt to your feedback?" [1 quickly]
Very slowly Very
4. "How did the robot's learning capabilities affect your interaction experience?" [1 Nega- tively Positively]
5. "Did you notice the robot learning adapting to your behavior?" [1 Not at all 5 Con- stantly]
6. "How did the robot's learning process affect your level of trust in its capabilities?" [1 Neg- atively 5 Positively]
7. "Overall, which method do you prefer?" [Pair wise comparison/Languuee feedback]
8. "Why do you prefer this method?"
9. "Is there anything you wish the system would do but currently does not?"
10. "Do you have any other comments?"
C Experiments - Additional Figures
C.1 The Effect of Co-finetuning
Co-finetuning both trajectory encoder and language model, versus utilizing frozen language model and only training the trajectory encoder, helps map trajectories and language utterances to the same latent space. This is shown in Table 2.
Table 2: Test Accuracy of training with frozen language model and co- finetuning
Method
Robosuite
Metaworld
Co-finetune
0.8489
0.8288
Freeze T5
0.7625
0.7344
C.2 Response Time to Different Feedback Types
Time Per Query
40
Language
35.
Comparison
30
Count
15-
10
5-
120 140 160
Time (+)
Figure 11: Time per query for both methods and averages, for 10 distinet users, indicated by color.
Figure 11 shows user-level differences of feedback length and time. The average of time of language-based reward learning is less than the comparison preference learning, illustrating that our method outperforms baseline in terms of the time-efficiency. However, the variance of language- based reward learning is larger than the baseline, which means users' thinking time varies a lot based on the trajectory shown.
13

---

D User Study Guidelines
Introduction. Welcome to
study human preference learning! In this study, you will interact with a robot and provide feedback based on your preferences. Please note that there will be no physical contact with the robot.
Task Description. The robot is situated in a kitchen setup and needs to pick up a spoon from the wall. However, there is pan the stove cooking something. For safety reasons, you should aim to have the robot successfully pick up the spoon while avoiding the pan. You can also adjust the robot's speed based on your preference.
Pre-study Survey. Before we begin, please complete pre-study survey about your background and understanding of robotics and Al systems.
Experiment 1 Improve Trajectory. In this experiment, the robot will first execute suboptimal trajectory. Your task is to improve its behavior by providing comparative language feedback, such "avoid the pan better." After each iteration, you will be asked if you are satisfied with the current trajectory. The experiment ends when you are satisfied with the trajectory or reach the maximum number of iterations. post-study survey will be required after completing this experiment.
Experiment Preference Learning. In this experiment, you will be shown series of queries and provide feedback each one. Through this process, will gradually learn your preferences and present you with the best trajectories based on the learned preferences. You will compare two approaches: language preference learning and pairwise preference learning.
1. Language Preference Learning
In each query, you will be shown one trajectory. You need to give comparative lan- guage feedback based on your preferences, such as "move faster." "avoid the pan better," or "be more adept at picking up the spoon." After every queries, you will be shown the best trajectory based on the currently learned preferences and asked rate it. You will complete total of 20 queries.
2. Pairwise Preference Learning
In each query, you will be shown a pair of trajectories. You need to choose your preferred one. After every 5 queries, you will be shown the best trajectory based on the currently learned preferences and asked to rate it. You will complete total of 20 queries.
After completing the experiment, you will be required to fill in post-study survey.
E Original Language Feedback Utterances
All the original language feedback for each of the simulation environments are listed here. These sentences were then augmented with GPT-3.55.
Robosuite:
Distance to the cube
Move farther from the cube.
Move further from the cube.
Move more distant from the cube.
Move less nearby from the cube.
Move nearer to the cube.
Move closer to the cube.
Move more nearby to the cube.
Move less distant to the cube.
Distance to the bottle
Move further from the bottle.
Move farther from the bottle.
14

---

Move more distant from the bottle.
Move less nearby from the bottle.
Move
the bottle.
Move closer to the bottle.
Move more nearby to the bottle.
Move less distant to the bottle.
Height of the robot arm
Move taller.
Move at a greater height.
Move higher.
Move to greater height.
Move lower.
Move lesser height.
Move shorter.
Move to lower height.
Speed of the robot arm
Move quicker.
Move swifter.
Move at higher speed.
Move faster.
Move more quickly.
Move lower speed.
Move more moderate.
Move slower.
Move more sluggish.
Move more slowly.
Proficiency at cube-lifting
Lift the cube better.
Lift the cube more successfully.
Lift the cube more effectively.
Lift the cube worse.
Lift the cube not as well.
Lift the cube less successfully.
Meta-Worts
Height of the robot arm
Move higher.
Move more up.
Move higher up from the table.
Increase the overall height of the trajectory.
Go higher up.
Move your gripper higher.
Don't go down, instead go up.
Stay higher and farther from the table.
Move your hand up you perform the task.
Make sure to stay higher above the table, rather than lower.
Move lower.
Move more down.
Move lower to the table.
15

---

Decrease the overall height of the trajectory.
Go lower down.
Move your gripper lower.
Don't go up, instead go down.
Stay lower and nearer to the table.
Move your hand lower as you perform the task.
Make sure to lower to the table, rather than higher.
Speed of the robot arm
Move faster.
Move at quicker speed.
Increase the pace.
Press the button faster.
Increase your velocity.
Move your gripper faster.
Move to the button faster.
Don't 't go too slowly, instead go quickly.
Move your hand faster you perform the task.
Make sure to go much faster, rather than slower.
Move slower.
Move more sluggish speed.
Decrease the pace.
Press the button slower.
Decrease your velocity.
Move your gripper slower.
Move the button slower.
Don't go fast, instead go more slowly.
Move your hand slower as you perform the task.
Make sure to much slower, rather than faster.
Distance to the button
Move farther from the button.
Increase distance from the button.
Stay farther from the button.
Give wider berth to the button.
Keep larger distance from the button.
Keep your gripper farther away from the button.
Move your gripper away from the button.
Don't 't go towards the button, instead move away from
Move your hand farther from the button on the wall as you perform the task.
Make sure to much farther from the button, rather than closer to the button.
Move closer to the button.
Decrease distance from the button.
Stay closer the button.
Get closer to the button.
Keep smaller distance to the button.
Keep your gripper closer to the button.
Move your gripper so that it is closer to the button.
Don't go away from the button, instead go towards it.
Move your hand closer to the button on the wall as you perform the task.
Make sure to go much closer to the button, rather than farther from the button.
16


Reviews:
[{"review_id": "lmvh6ZW6Aj", "reviewer": "Reviewer_KGNz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper aims to propose a novel way of doing preference based learning by utilizing comparative language as the feedback rather than traditional comparison based approaches. For this purpose authors created a model that leverages learning the latent state representations that align language utterance and changes in robot trajectory. Then this model is used to both iteratively update the robot trajectories as well as learning humans reward model. Authors backed their proposed approach with simulated and real-world experiments.", "summary_of_recommendation": "I recommend this paper to be accepted given the novelty and importance of the subject as well as the good execution and testing done by the authors.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "My main question for the authors are the statistical test results on the user study survey responses. I think that should be reported even if the subject size is quite small. Another point to address would be how would this method work in settings where many language utterances can be mapped to a single trajectory update? How well can this system generalize in terms of the language utterances it can understand? What is the benefit of co-finetunining versus using a frozen language model for getting the embeddings?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TrQLjOuLyR", "reviewer": "Reviewer_e57u", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a method to leverage language feedback to improve robot trajectories to better satisfy human preferences. Instead of seeking a binary comparative feedback, as popularized by RLHF-style methods, this work seeks to obtain natural language feedback, which hypothetically might contain more information than a binary comparative label. The authors propose to learn a mapping in latent space between correction in language space and that in trajectory space. They then use this latent space mapping to learn a reward function, interpreting language correction as a preference between the two corresponding points in the trajectory space. To learn the reward function they use the Bradley-Terry model, commonly used for comparative feedback. They evaluate their model on simulated data on two tasks, and through a user study.", "summary_of_recommendation": "The paper proposes a novel method to incorporate language feedback in iteratively improving trajectories to better fit a human’s preferences. The work proposes a viable formulation and promising method, but evaluations could include language-based baselines and several details regarding the method and evaluation are unclear.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See questions in review, particularly those regarding:\n- Missing details in the evaluation sections\n- Using language-based baselines\n- Explaining results in Figure 10 and justifying the narrow margins in performance improvement\n- The difference between iterative trajectory improvement, and reward learning, and why both aspects are important to include", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U226N2g2zn", "reviewer": "Reviewer_1rvf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper propose a new approach to use the rich information contained in the language feedback to a trajectory and compare that to pairwise preferences.", "summary_of_recommendation": "Paper presents a new and simple solution but the experiments consider a toy setting making it far from real world world deployment.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Overclaim of first method to use language feedback in reward learning: Some examples where language feedback has been used in prior works [1,2].\n\n\n[1]: https://arxiv.org/abs/2310.17555\n[2]: https://cdn.aaai.org/ojs/16749/16749-13-20243-1-2-20210518.pdf", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Paper summary**

This paper presents a method for using language feedback to indicate and learn from preferences between trajectories. The results indicate that language feedback provides more informative data than preferences alone, leading to more efficient and user-preferred learning performance.

**Review summary**


Summary of strengths:
+ Learning effectively from language feedback is a highly challenging, yet important, problem. 
+ The proposed method is novel in that it aims to learn a latent space mapping from language feedback to trajectory modifications.
+ The evaluation involves a user study with non-expert users.

Summary of weaknesses:
- The paper is missing important comparisons to prior work on learning from language. Of the papers that are cited, none are used as baselines in the evaluation. **[Edit: the revised paper includes several new citations to relevant SOTA work. However, none are evaluated as baselines.]**
- The proposed method must be trained on labeled data, yet it is unclear how this would be obtained in a scaleable way. **[Edit: the rebuttal proposes two ways to improve the scaleability of data collection.]**
- The paper does not discuss how the proposed method would be integrated into a policy optimization problem (which would seemingly be the goal of learning from language feedback). **[Edit: the rebuttal argues that its sampling-based method is similar to other SOTA approaches for generating and evaluating trajectories.]**

**Response to rebuttal**

Overall, the reviews are positive and remain so after the rebuttal. The rebuttal addressed the reviewers' requests for clarifications and added citations. 

However, the rebuttal does not resolve the request for a baseline method that uses language feedback. The authors claim that existing approaches are not suitable as baselines due to the lack of published code, but this is not a sufficient reason to exclude otherwise-relevant SOTA baselines.


**Recommendations for improvement**

* Add baselines to contextualize the results with respect to SOTA methods for learning from language.
Decision: Accept