Trajectory Improvement and Reward Learning from Comparative Language Feedback

Authors: zhaojing  yang
miru  jun
jeremy  tien
stuart j russell
anca  dragan
erdem  bıyık
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:48:18

Abstract: Learning from human feedback has gained traction in fields like robotics and natural language processing in recent years. While prior works mostly rely on human feedback in the form of comparisons, language is a preferable modality that provides more informative insights into user preferences. In this work, we aim to incorporate comparative language feedback to iteratively improve robot trajectories and to learn reward functions that encode human preferences. To achieve this goal, we learn a shared latent space that integrates trajectory data and language feedback, and subsequently leverage the learned latent space to improve trajectories and learn human preferences. To the best of our knowledge, we are the first to incorporate comparative language feedback into reward learning. Our simulation experiments demonstrate the effectiveness of the learned latent space and the success of our learning algorithms. We also conduct human subject studies that show our reward learning algorithm achieves a 23.9% higher subjective score on average and is 11.3% more time-efficient compared to preference-based reward learning, underscoring the superior performance of our method. Our website is at https://liralab.usc.edu/comparative-language-feedback/.

Main paper: 
 introduction: 
 Learning from human feedback has gained significant popularity in robotics, leading to the study of different forms of human feedback: demonstrations [1, 2, 3, 4] , preference comparisons [5, 6, 7, 8] , rankings [9] , physical corrections [10] , visual saliency maps [11] , human language [12, 13] , etc. Among these, preference comparisons grew popular for its simplicity and ease of use, especially compared to demonstrations [7] . Preference comparisons often involve users choosing between a pair of choices. Using these selections to learn a reward function and train a policy is known as reinforcement learning from human feedback (RLHF) [6] or more generally preference-based learning [5, 14] . It has proven applicable to a broad range of fields ranging from robotics [7, 15] to natural language processing [16] , from traffic routing [17] to human-computer interaction [18] .Despite their successes, preference comparisons suffer from problems [19] such as the unreliability of human data and the limited information bandwidth, i.e., each pairwise comparison contains at most 1 bit of information [20] . There has been research to provide a better interface [21] , allowing the users to specify their preferences for every feature, but they require features to be designed by hand. As an alternative form of human feedback, comparative language is considerably more informative than preference comparisons, allowing users to prioritize specific aspects. For example, it allows users to naturally indicate their preference about speed by simply saying, "the robot should move faster," making it more intuitive and interpretable.In this work, we aim to leverage comparative language feedback to learn the human preferences, i.e., their reward functions. In pursuit of this objective, we first learn a shared latent space that aligns trajectories and language feedback. This alignment enables the robot to comprehend human language feedback, leverage it for adapting its behavior to learn and better align with the human's preferences. To test the effectiveness of our approach, we conduct experiments in two simulation environments and a human-subject study with a real robot. The results suggest that reward learning from Compared to traditional comparison preference learning, our language preference learning enables users to give more informative feedback, which helps the robot to capture human preferences more efficiently.comparative language feedback outperforms traditional preference comparisons in performance and time-efficiency, and is favored by most of the users.• In each query, you will be shown one trajectory. You need to give comparative language feedback based on your preferences, such as "move faster," "avoid the pan better," or "be more adept at picking up the spoon." After every 5 queries, you will be shown the best trajectory based on the currently learned preferences and asked to rate it. You will complete a total of 20 queries. 
related work: 
 Before formally defining the problem, we will first review existing works in learning from human feedback, preference-based learning, and robot learning from human language feedback.Learning from Human Feedback. Several promising approaches leverage human feedback to train robots [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 22] . However, the most popular methods of learning from demonstrations and rankings involve a trade-off between informativeness and ease of use [6, 19, 23, 24] . Furthermore, none has taken advantage of the expressive nature of language to develop a method of reward learning from comparative language feedback. Addressing this gap will provide more scalable methods for training robots, particularly in complex environments where traditional feedback forms may be insufficient or impractical.Preference-based Learning. Based on preference comparison feedback, preference-based learning is widely used for its logical intuitiveness and ease of use. Learning from rankings is beneficial to deep reinforcement learning [6] , but suffers from several shortcomings [19] such as the reliability of human data and difficulties of choosing from equivalent choices. Works have sought to improve its shortcomings in time-and sample-efficiency by actively generating pairs based on information gain [20] or volume removal [5] , and doing these in batches of pairs [25] . Sikchi et al. [26] and Brown et al. [27] integrate preference feedback into imitation learning, demonstrating superior performance in their respective approaches. However, the fundamental limitation remains that each pairwise comparison provides at most 1 bit of information [20] . In addition, users struggle to choose between two similar trajectories [7] . In contrast, our method of comparative language feedback offers more informative input after observing just one trajectory, while maintaining intuitiveness and ease of use.Robot Learning with Human Language Feedback. Benefiting from advancements in natural language processing, works have leveraged language for adjusting robot trajectories [13, 28, 29, 30, 31, 32] , fine-tuning language models [12] , and reward shaping [33, 34] . For example, Shi et al. [31] use language-conditioned behavior cloning (LCBC) for corrective language commands and improving policies. Lynch et al. [35] describe an approach for real-time guidance from humans using natural language in order to achieve a goal. Cui et al. [36] introduce an approach to use human language feedback to correct robot manipulation in real-time via shared autonomy. Goyal et al. [33] utilize human language combined with past action sequences to generate rewards, and in a separate work, Goyal et al. [34] leverage natural language to map image observations to rewards. However, these works focus on using human language as the instruction or correction to the robot, none of them has explored learning the reward function of humans entirely from comparative language feedback, which offers benefits over policy learning with generalizability and explainability. Additionally, our method iteratively updates the learned reward function for better preference alignment, providing an advantage over one-shot corrections [13, 29] .• In each query, you will be shown a pair of trajectories. You need to choose your preferred one. After every 5 queries, you will be shown the best trajectory based on the currently learned preferences and asked to rate it. You will complete a total of 20 queries.After completing the experiment, you will be required to fill in a post-study survey. 
problem definition: 
 Having reviewed the related literature, we are now ready to formally define the problem. We model the robot as a decision making agent in a standard finite-horizon Markov decision pro- T -1 t=0 R(s t , a t ). For each trajectory, the human may provide language feedback l that attempts to improve the trajectory (based on the reward function) in one aspect, e.g., speed, distance to objects, height of the robot's arm, etc. As an example, the human might say "move farther from the stove" (see Figure 1 ) if the robot's trajectory would be higher-reward in that way. Our goal is to develop a framework where we learn the reward function based on such feedback.Unfortunately, this task is doomed without any information about how language feedback relates to the different aspects of the task. In this work, we learn this relation with an offline pretraining phase where we collect a dataset that consists of pairs of robot trajectories and a language label describing how the two trajectories differ. We use this dataset to learn encoders that map trajectories and language feedback onto a shared latent space. We leverage these encoders to learn the preferences (reward functions) of different users based on their language feedback.In the next section, we detail our approach to learning the latent space and explain how this learned latent space is utilized for iterative trajectory improvements and reward learning. 
our approach: 
 As shown in Figure 2 , our approach is composed of two stages: First, we learn a shared latent space where robot trajectories and human language feedback are aligned. Second, we leverage this learned latent space to (1) improve robot trajectory or (2) learn human preferences.To learn a shared latent space for trajectories and language feedback, we collect a dataset of (τ A , τ B , l) tuples, where τ A and τ B are a pair of trajectories and l is a language utterance that describes the difference between the two trajectories. Note that this language utterance does not necessarily align with the human's preferences about the robot: it just describes a difference between the trajectories -it is possible to have l ="move faster" if the robot moves faster in τ B than τ A even though users want the robot to move slowly.After collecting such a dataset, we propose the model visualized in Figure 2 to learn the shared latent space between trajectories and language utterances. Similar to [37] , we use a neural network to encode each state-action pair (s t , a t ) from the pair of trajectories, τ A and τ B , to embeddings φ A t and φ B t , respectively. We want these embeddings to contain information about aspects of the trajectories that the human may care and give feedback about.To achieve this, we align the difference between the embeddings φ A and φ B with embeddings ψ l of the language feedback l by using the following loss function:EQUATIONwhere the sigmoid can be considered as the probability that the language utterance l is inputted for the pair of trajectories τ A and τ B . Intuitively, when φ B -φ A and ψ l align, i.e., have the same direction for fixed magnitudes, the loss is minimized. This alignment will enable us to acquire the embedding of an improved trajectory as φ A +ψ l when given an initial trajectory τ A and comparative language feedback l.Instead of training a language encoder from scratch, which would require a prohibitively large and diverse dataset of (τ A , τ B , l) tuples, we use a pretrained T5 model [38] . To align embeddings of trajectories with those of language feedback, we first freeze T5 model and train the trajectory encoder. Subsequently, we perform co-finetuning of both components.Additionally, we incorporate a normalization term into the loss function to help balance the magnitude of the embeddings. This term constrains the norms of the trajectory embeddings to remain below 1 and the norm of the language embeddings to be close to 1:L norm (τ A , τ B , l) = a • max{ φ A -1, 0} + max{ φ B -1, 0} + b • ψ l -1 2(2) where a and b are two hyperparameters.Overall, the objective we use in the model training consists of two terms:L(τ A , τ B , l) = L align (τ A , τ B , l) + L norm (τ A , τ B , l),(3) which we use to train the trajectory encoder and finetune T5. Training this architecture gives us the ability to encode any trajectory and language utterance in the same latent space. In the next subsection, we demonstrate how this latent space is useful for iteratively improving robot trajectories and for learning human preferences based on their language feedback.The shared latent space aligns robot trajectories with human language feedback. This alignment enables an intuitive understanding of user preferences. We will now explore two primary ways to leverage this learned latent space: first, to iteratively improve the robot's trajectory, and second, to accurately learn user preferences.Firstly, we leverage the latent space to iteratively improve an initial suboptimal robot trajectory. We start with showing the initial trajectory τ 0 to the user and asking for language feedback l 0 . Upon receiving human's language feedback, we use our trained encoders to compute the trajectory embedding φ 0 and language embedding ψ l 0 . We then find the improved trajectory τ 1 such that its difference with τ 0 best aligns with the human's language feedback based on cosine similarity:EQUATIONWe iteratively continue this process for N iterations to obtain τ 0 , τ 1 , . . . , τ N . In this work, we search over a discrete, predefined set of trajectories to solve the optimization in Eq. ( 4) for computational efficiency. It is, however, possible to use reinforcement learning or model-predictive control algorithms to solve this optimization at the expense of increased computational cost.In addition to improving trajectories, we also utilize the learned latent space to learn the user's preference, i.e., their reward function. Previous approaches ask users to label their preference from a pair of trajectories, which requires the users to watch two trajectories in order to give at most 1 bit of information. In our language-based reward learning approach, for each query i, we only show users one trajectory τ i to collect language feedback l i based on their preferences. Given one trajectory, we construct an improved trajectory τi : φ τi = φ τi + ψ li . Note that τi is just an imaginary trajectory that maps to φ τi +ψ li in the learned latent space. Then following the Bradley-Terry model [39] , which is commonly used in preference-based learning, we model a preference predictor with the reward function r ξ as:EQUATIONwhere r ξ is a neural network parameterized with ξ. The reward function r ξ is trained by minimizing the following negative loglikelihood loss:EQUATIONWe now make an important observation about the comparative language feedback. When the user tells the robot to move farther from the stove, it indeed indicates a preference between the original trajectory and the improved trajectory that moves farther from the stove. However, this feedback contains much more information than this comparison. The user could use any comparative language utterance to teach the robot, but they specifically selected one about the distance to the stove. This means, with high probability, the improvement the robot may get from this feedback is higher than that of any other comparative language feedback.Mathematically, this indicates a preference for τi over τi : φ τi = φ τi + ψ li where li is any language utterance other than l i . Again we apply the Bradley-Terry model to utilize this implicit preference:EQUATIONBased on this, we sample k language feedback from a set of pre-collected language utterances other than l i , and minimize the following loss about the chosen language feedback:EQUATIONOverall, the loss for reward learning from comparative language feedback is as follows:EQUATIONTraining the reward function with this loss enables us to efficiently capture human preferences through comparative language feedback. 
experiments: 
 We conducted simulation experiments and human subject studies to validate our methods with diverse tasks and features that humans may give feedback about.Synthetic humans. For simulation experiments, we synthesize the comparative language feedback from a simulated human with a simplified reward function R: R(s t , a t ) = w θ(s t , a t ) (10) where θ is a function that maps a state-action pair to a vector of high-level features (e.g., speed, distance to the stove). We similarly define θ(τ ) to denote the sum of features over a trajectory τ . w is a vector of weights that maps the features to a scalar reward value. Both θ and w are unknown to our algorithm. Given a true reward function w, let τ * be the optimal trajectory under reward w. Then, the synthetic human gives the noisy language feedback l 0 when shown trajectory τ 0 :EQUATIONwhere θ * and θ 0 denote the true cumulative features of τ * and τ 0 respectively, denotes element-wise multiplication. The function takes a sample from the output of the softmax, which is computed across different features we designed for the simulated humans (see Appendix A), and outputs a language feedback that corresponds to the sampled feature. The language feedback to output is chosen randomly from all language utterances of that feature in the GPT-augmented dataset (see Appendix E).Environments. We experimented in two simulation environments: Robosuite [40] and Meta-World [41] . Robosuite has a Jaco robot arm at a table with a cube and a bottle, and the task is to pick up the cube. We trained RL policies with randomized weights w using stratified sampling [43] over the features, then generated rollout trajectories with timesteps T = 500 for each environment. For Robosuite, we generated 448 unique rollouts and 324 for Meta-World. The splits were 359, 44, 45 and 260, 32, 32 for training, validation, and test sets.Trajectories were paired within each set and matched with a language feedback that describes the change in each feature from τ A to τ B (see Figure 3 ).We train the encoders with the loss in Eq. ( 3), and use accuracy as the metric to evaluate:EQUATION)Training the encoders with the training set of trajectories and language utterances, we observed a test accuracy of 84.9% in Robosuite and 82.9% in Meta-World. This indicates the trajectory and language embeddings are well-aligned. We found co-finetuning both trajectory encoder and language model, versus utilizing a frozen language model and only training the trajectory encoder, helps map trajectories and language utterances to the same latent space (see Table 2 in the Appendix).Iterative Trajectory Improvement. Next, we conducted iterative trajectory improvement experiments based on the method we presented in Section 4.2.1. We set the number of iteration steps N = 15 and repeat the full experiment over 100 random seeds. The true rewards of these trajectories are shown in Fig. 4 . In both environments, we consistently improve the trajectories, which showcases the effectiveness of the learned latent space and the improvement algorithm. However, it can be noted that our algorithm cannot reach the performance of the optimal trajectory. This is because every improvement iteration is completely independent from the previous iterations and the robot may get stuck in a loop between good, but non-optimal, trajectories.Our reward learning algorithm presented in Section 4.2.2, on the other hand, utilizes the entire history of human feedback, so it should not suffer from this problem. We will now demonstrate this.Reward Learning. We again randomly initialize the reward weights w to simulate human feedback.For each environment, we simulate three synthetic humans, and run the experiments with three random seeds for each simulated human. We use the loss in Eq. ( 8) to learn the reward function. We adopt cross-entropy CE(P w , P ξ ) as the evaluation metric, whereEQUATION)Another metric is the true reward value of the optimal trajectory selected from the test set based on the learned reward, reflecting how close the learned reward is to the true reward. All reward values are normalized between 0 and 1.We compare our method against reward learning from comparisons, where for each query, the simulated user chooses a preferred trajectory from pair (τ A , τ B ) and with probability shown in Eq. ( 13).As indicated in Figure 5a , the cross-entropy of our method decreases faster than that of comparison preference learning in both environments, demonstrating that our approach converges more quickly than the baseline. Meanwhile, Figure 5b shows that the true reward of optimal trajectories reaches a value of 1 in a shorter time with our language-based method, especially in Meta-World, further supporting that it learns the reward function more efficiently.Ablation Study. We conducted an ablation study to evaluate the different components of the loss function (Eq. ( 8)). As indicated in Figure 6a , the combination of both components outperforms using each component individually, demonstrating the validity of our loss function design. Also, using both components fits the true reward function better, as shown in Figure 6b . To verify the effectiveness of our approach, we conducted human subject studies by recruiting 10 subjects (4 female, 6 male) from varying backgrounds and observing them to interact with our real robot setup. Our study has been approved by the IRB office of the University of Southern California.Real Robot Setup. We used a WidowX 250 6DOF robot with the setup of [44] in front of a kitchen set (Figure 7 ). The task is to pick up a spoon while avoiding going over a pan on the stove. The states are given as 480×320 images resized to 224×224, and 22-dimensional proprioception. The actions are 7-dimensional (6DOF+gripper). To train the latent space, we collected a dataset of 321 trajectories with varying levels of success at picking up the spoon, avoiding the pan, and speed. These trajectories were divided into 192 for training, 64 for validation, and 65 for testing. 496 GPTaugmented sentences were split into 297 for training, 99 for validation, and 100 for testing. These sentences were paired with the trajectories in the same manner as in the simulation experiments. Another set of 32 trajectories was collected for the user studies following the pretraining phase. Study 1: Trajectory Improvement. Users start with a suboptimal trajectory from the dataset and give language feedback for a maximum of 10 iterations until they are satisfied.Study 2: Preference Learning. For 20 iterations, users give feedback (preference comparison or comparative language) on random trajectories and rate the optimal trajectory from the dataset with respect to the learned reward after every 5 iterations. To avoid bias, 5 users began with the comparison based method and 5 with the comparative language based method.After each study, users were asked to complete post-study surveys (see Appendix B). Results. Our post-study surveys reveal that the first study has consistently positive responses for user experience and speed of adaptation, but has two peaks in iterations to satisfaction (Fig. 8 ). We conjecture that the dataset of 32 trajectories may not have contained trajectories that the users desired. As for the second study, our language-based method scored better than the comparison-based method for all attributes. The average score of the language-based method over all attributes is 23.9% higher than the comparison-based method , illustrating the superior performance of our approach. In terms of time-efficiency, our method takes 11.3% less time for users to answer each query as shown in Fig. 9 , even though it includes the time it took for users to type their feedback. Indeed, users found the language method to be more time efficient and adaptable but found comparison method to be more convenient (Table 1 ). Fig. 10 shows the user rating of optimal trajectories given by the learned reward function. To quantitatively assess the learning efficiency, we follow prior work in active learning [9, 45, 46] by checking the area under curve (AUC) for both lines. We found that the AUC for the comparative language line is statistically significantly higher than that of preference comparison line (p < 0.05). This indicates that our language-based approach more efficiently captures human preferences compared to the comparison-based method. An unexpected observation from Fig. 10 is that humans' ratings decreased from 5 queries to 10 queries while using comparative language feedback. This may be because of the relatively small subject size: it is possible that the average rating after 5 iterations, where the standard deviation is indeed large, is higher than it should be. Another potential explanation is that humans may be more lenient to the robot after 5 iterations than 10 iterations, especially in comparative language feedback where they only watch the robot 5 times to give their feedback as opposed to 10 times in preference comparisons. 
conclusion: 
 In this paper, we presented a robot learning framework to learn human preferences using comparative language feedback. For this, we aligned trajectories with language feedback in a shared latent space, then used it to improve trajectories and learn preferences. Experiments in simulation environments and real-world user studies suggest that our approach consistently outperforms comparison preference learning and is favored by most users for aspects such as time efficiency and adaptability.Limitations and Future Work. Even though our user studies show some generalizability in the comparative language feedback, our model is limited by the feedback about the objects seen in the pretraining data. Additionally, we sample queries uniformly at random during reward learning.Based on these limitations, the future works include: (1) use image annotations to generate language feedback containing all objects, and (2) implement active learning to reduce the number of iterations required to reach the optimal trajectory. 
 
 Appendix: Robosuite Features• The height of the robot's end-effector• The speed of the end-effector• The distance between the end-effector and the bottle• The distance between the end-effector and the cube• How well the robot lifts the cube (i.e., level of success of cube-lifting task)The level of success of the cube-lifting task is quantified by: 1) whether or not the cube is lifted above the height of a success threshold or 2) a weighted sum of the distance to the cube and whether or not the end-effector is grasping the cube (the level of success in picking up the spoon).• The height of the robot's end-effector.• The velocity of the robot's end-effector.• The distance between the end-effector and the button.• The velocity of the end-effector• The distance between the end-effector and the pan• How well the robot picks up the spoon (i.e. level of success)The level of success is quantified by: 1) whether the spoon is grasped from the hook 2) a weighted sum of the distance to the spoon and whether the end-effector is grasping the spoon.Here are the questions in our user study surveys.Pre-Study Survey. To assess the subjects' experience and level of skill with robotics and machine learning, we conducted a short pre-study survey before any of the experiments.Post-Study Survey -Preference Learning. Users filled out this post-study survey after completing two methods of the preference learning experiment.1. "What did you like about these methods? For each aspect below, please click the circle if you found the method satisfactory in that regard:" [Adaptability, Time efficiency, Convenience, None of them] 2. "Could you explain the reasons?" 3. "How long did it take for the robot to adapt to your feedback?" [1 Very slowly -5 Very quickly] 4. "How did the robot's learning capabilities affect your interaction experience?" [1 Negatively -5 Positively] 5. "Did you notice the robot learning or adapting to your behavior?" [1 Not at all -5 Constantly] 6. "How did the robot's learning process affect your level of trust in its capabilities?" [1 Negatively -5 Positively] 7. "Overall, which method do you prefer?" [Pair-wise comparison/Language feedback] 8. "Why do you prefer this method?" 9. "Is there anything you wish the system would do but currently does not?" 10. "Do you have any other comments?"Co-finetuning both trajectory encoder and language model, versus utilizing a frozen language model and only training the trajectory encoder, helps map trajectories and language utterances to the same latent space. This is shown in Table 2 . Figure 11 shows user-level differences of feedback length and time. The average of time of language-based reward learning is less than the comparison preference learning, illustrating that our method outperforms baseline in terms of the time-efficiency. However, the variance of languagebased reward learning is larger than the baseline, which means users' thinking time varies a lot based on the trajectory shown.Introduction. Welcome to our user study on human preference learning! In this study, you will interact with a robot and provide feedback based on your preferences. Please note that there will be no physical contact with the robot.Task Description. The robot is situated in a kitchen setup and needs to pick up a spoon from the wall. However, there is a pan on the stove cooking something. For safety reasons, you should aim to have the robot successfully pick up the spoon while avoiding the pan. You can also adjust the robot's speed based on your preference.Pre-study Survey. Before we begin, please complete a pre-study survey about your background and understanding of robotics and AI systems.Experiment 1 -Improve Trajectory. In this experiment, the robot will first execute a suboptimal trajectory. Your task is to improve its behavior by providing comparative language feedback, such as "avoid the pan better." After each iteration, you will be asked if you are satisfied with the current trajectory. The experiment ends when you are satisfied with the trajectory or reach the maximum number of iterations. A post-study survey will be required after completing this experiment.Experiment 2 -Preference Learning. In this experiment, you will be shown a series of queries and provide feedback on each one. Through this process, we will gradually learn your preferences and present you with the best trajectories based on the learned preferences. You will compare two approaches: language preference learning and pairwise preference learning.All the original language feedback for each of the simulation environments are listed here. These sentences were then augmented with GPT-3.5.• Distance to the cube -Move farther from the cube.-Move further from the cube.-Move more distant from the cube.-Move less nearby from the cube.-Move nearer to the cube.-Move closer to the cube.-Move more nearby to the cube.-Move less distant to the cube.• Distance to the bottle -Move further from the bottle.-Move farther from the bottle.-Move more distant from the bottle.-Move less nearby from the bottle.-Move nearer to the bottle.-Move closer to the bottle.-Move more nearby to the bottle.-Move less distant to the bottle.• Height of the robot arm -Move taller.-Move at a greater height.-Move higher.-Move to a greater height.-Move lower.-Move at a lesser height.-Move shorter.-Move to a lower height.• Speed of the robot arm -Move quicker.-Move swifter.-Move at a higher speed.-Move faster.-Move more quickly.-Move at a lower speed.-Move more moderate.-Move slower.-Move more sluggish.-Move more slowly.• Proficiency at cube-lifting -Lift the cube better.-Lift the cube more successfully.-Lift the cube more effectively.-Lift the cube worse.-Lift the cube not as well.-Lift the cube less successfully.Meta-World:• Height of the robot arm -Move higher.-Move more up.-Move higher up from the table.-Increase the overall height of the trajectory.-Go higher up.-Move your gripper higher.-Don't go down, instead go up.-Stay higher and farther from the table.-Move your hand up as you perform the task.-Make sure to stay higher above the table, rather than lower.-Move lower.-Move more down.-Move lower to the table.-Decrease the overall height of the trajectory.-Go lower down.-Move your gripper lower.-Don't go up, instead go down.-Stay lower and nearer to the table.-Move your hand lower as you perform the task.-Make sure to go lower to the table, rather than higher.• Speed of the robot arm -Move faster.-Move at a quicker speed.-Increase the pace.-Press the button faster.-Increase your velocity.-Move your gripper faster.-Move to the button faster.-Don't go too slowly, instead go quickly.-Move your hand faster as you perform the task.-Make sure to go much faster, rather than slower.-Move slower.-Move at a more sluggish speed.-Decrease the pace.-Press the button slower.-Decrease your velocity.-Move your gripper slower.-Move to the button slower.-Don't go too fast, instead go more slowly.-Move your hand slower as you perform the task.-Make sure to go much slower, rather than faster.• Distance to the button -Move farther from the button.-Increase distance from the button.-Stay farther from the button.-Give wider berth to the button.-Keep a larger distance from the button.-Keep your gripper farther away from the button.-Move your gripper away from the button.-Don't go towards the button, instead move away from it.-Move your hand farther from the button on the wall as you perform the task.-Make sure to go much farther from the button, rather than closer to the button.-Move closer to the button.-Decrease distance from the button.-Stay closer to the button.-Get closer to the button.-Keep a smaller distance to the button.-Keep your gripper closer to the button.-Move your gripper so that it is closer to the button.-Don't go away from the button, instead go towards it.-Move your hand closer to the button on the wall as you perform the task.-Make sure to go much closer to the button, rather than farther from the button.Our work trivially extends to the cases where trajectory length is not fixed. 


Decision: **Paper summary**

This paper presents a method for using language feedback to indicate and learn from preferences between trajectories. The results indicate that language feedback provides more informative data than preferences alone, leading to more efficient and user-preferred learning performance.

**Review summary**


Summary of strengths:
+ Learning effectively from language feedback is a highly challenging, yet important, problem. 
+ The proposed method is novel in that it aims to learn a latent space mapping from language feedback to trajectory modifications.
+ The evaluation involves a user study with non-expert users.

Summary of weaknesses:
- The paper is missing important comparisons to prior work on learning from language. Of the papers that are cited, none are used as baselines in the evaluation. **[Edit: the revised paper includes several new citations to relevant SOTA work. However, none are evaluated as baselines.]**
- The proposed method must be trained on labeled data, yet it is unclear how this would be obtained in a scaleable way. **[Edit: the rebuttal proposes two ways to improve the scaleability of data collection.]**
- The paper does not discuss how the proposed method would be integrated into a policy optimization problem (which would seemingly be the goal of learning from language feedback). **[Edit: the rebuttal argues that its sampling-based method is similar to other SOTA approaches for generating and evaluating trajectories.]**

**Response to rebuttal**

Overall, the reviews are positive and remain so after the rebuttal. The rebuttal addressed the reviewers' requests for clarifications and added citations. 

However, the rebuttal does not resolve the request for a baseline method that uses language feedback. The authors claim that existing approaches are not suitable as baselines due to the lack of published code, but this is not a sufficient reason to exclude otherwise-relevant SOTA baselines.


**Recommendations for improvement**

* Add baselines to contextualize the results with respect to SOTA methods for learning from language.

Reviews: review: title: title: Review of Trajectory Improvement and Reward Learning from Comparative Language Feedback
paper_summary: paper_summary: This paper aims to propose a novel way of doing preference based learning by utilizing comparative language as the feedback rather than traditional comparison based approaches. For this purpose authors created a model that leverages learning the latent state representations that align language utterance and changes in robot trajectory. Then this model is used to both iteratively update the robot trajectories as well as learning humans reward model. Authors backed their proposed approach with simulated and real-world experiments.
main_review: main_review: Overall I think this paper was clearly written and addresses a novel yet challenging problem of understanding human preferences from language in robotics setting. I thank the authors for their clear presentation of their work. In my opinion this paper has the following strengths

Most important strength of this paper was the novelty and the difficulty of the problem space in my opinion. As humans we communicate our preferences to each other by language and I do agree that this would be a much preferred way of communicating our preferences to robots as well. On the other hand learning humans reward functions from these language utterances is a challenging problem. I think authors present an interesting attempt at solving this problem by pairing the trajectory embedding differences with the language embedding to learn an aligned latent space representation. Another strength of this paper showing usability and preference of actual users by running a user study. The user study is important to back the claims of comparative language being more natural to the users. Lastly I think overall authors did a good job on clarity and quality when it comes to writing their work.

In terms of weaknesses I think one major weakness in my point of view was the authors did not report significance of the difference in their user study results. I would like to see statical test results mentioned and explained. Also I think 10 participants might be too little. I think having a bigger and more open user study would have benefitted this work. Another weakness in my view was the use of very simple tasks in the simulated settings I think authors could have shown the method in more complicated domains where language utterance can also significantly vary.
questions: questions: My main question for the authors are the statistical test results on the user study survey responses. I think that should be reported even if the subject size is quite small. Another point to address would be how would this method work in settings where many language utterances can be mapped to a single trajectory update? How well can this system generalize in terms of the language utterances it can understand? What is the benefit of co-finetunining versus using a frozen language model for getting the embeddings?
limitations: limitations: 3
review_summary: review_summary: I recommend this paper to be accepted given the novelty and importance of the subject as well as the good execution and testing done by the authors.
title: title: Refining trajectories by interpreting language feedback as difference between current and ideal trajectories. Lacks strong baselines.
paper_summary: paper_summary: The authors propose a method to leverage language feedback to improve robot trajectories to better satisfy human preferences. Instead of seeking a binary comparative feedback, as popularized by RLHF-style methods, this work seeks to obtain natural language feedback, which hypothetically might contain more information than a binary comparative label. The authors propose to learn a mapping in latent space between correction in language space and that in trajectory space. They then use this latent space mapping to learn a reward function, interpreting language correction as a preference between the two corresponding points in the trajectory space. To learn the reward function they use the Bradley-Terry model, commonly used for comparative feedback. They evaluate their model on simulated data on two tasks, and through a user study.
main_review: main_review: **Strengths**

- The proposed idea of interpreting language as a comparison or difference between the desired and current trajectory, and implementing it through a shared latent space is novel and useful
- The dual interpretation of human feedback, that the improved trajectory is more preferable than both the current trajectory, as well as the trajectories resulting from alternative language feedbacks is innovative, and the ablation results empirically show how each of those two kinds of information help the final model performance
- The attached video clearly outlines major advancements proposed in the work

**Concerns**

- Several key details of the evaluation are unclear. 
     - “Lists of language feedback were created for hand-crafted features  (see Appendix A)” → How are the language corrections created? Appendix A, is not present in the submission. Also, what do the authors mean by “[the language feedback] were augmented with GPT 3.5”? How are the language corrections matched with the trajectory rollouts?
     - For reward learning, what are the features ($\theta$)? How are those generated and what is the dimensionality of \theta? Is it correct that a randomly sampled vector $w$ represents a simulated human? And if so, how many simulated humans is the evaluation done on?
     - In simulated results, can the authors explain why the cross entropy does not drop past 0.4, especially since the function simulating human preferences matches the loss function being learned. Also, why is cross-entropy used to evaluate reward learning accuracy rather than something more direct, such as % error in predicted reward, compared to the true reward, since the true reward is available?
     - For the user study, how were the participants instructed to determine their preferences, especially since it is difficult to evoke preferences in short horizon tasks in a lab setting. Can the authors show what kind of preferences are expressed, and how diverse the preferences and the set of sampled trajectories were?
- No other work utilizing language feedback is used as a baseline. Several methods that utilize natural language as feedback to improve trajectories are mentioned in related work (such as [13,12]). While these methods utilize feedback in different ways, since they fit the same problem formulation of finding a better trajectory given language feedback, why is none of them used as a baseline?
- The problem of improving a robot trajectory and learning a human preference are fundamentally the same, in that some human correction needs to be used to find a better trajectory, and accordingly, both iterative trajectory improvement (4.2.1), and reward learning (4.2.2) serve to ultimately find an optimal trajectory given human corrections. Then why are both methods highlighted, since iterative trajectory improvement clearly performs worse than the latter?
- The ability to find the optimal trajectory in a real setting should be the core empirical result. Figure 10 alludes to this but is not referenced anywhere in the text. Can the authors elaborate on it, especially why the two methods end up with very similar performance at 20 iterations?

**Other Clarifications and Suggestions**

- The related work section can be better structured, and highlight works on language-based robot manipulation better. The ‘Learning from Human feedback’ section talks about robots and seems redundant with ‘Robot Learning with Human Feedback’. It would also help to elaborate on the works that use natural language to improve robot policies, and how exactly the proposed method is positioned in context of those methods. Related works on using language corrections to guide robot control such as the following, amongst others, could be highlighted.                                  
[1] Lynch, Corey, et al. "Interactive language: Talking to robots in real time." IEEE Robotics and Automation Letters (2023).                                  
[2] Cui, Yuchen, et al. "No, to the right: Online language corrections for robotic manipulation via shared autonomy." Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction. 2023.
- In Section 3, it is unclear how language feedback relates to the reward framework discussed before it.
- In Section 4.1. eq(3), how does optimizing the output of the network to be normalized prevent overfitting? Traditionally, regularization constraints are applied to network parameters and not network outputs.
- The authors emphasize how comparative feedback contains 1-bit of information. I wonder if it is possible to quantify statistical information gain from a language feedback, especially in the evaluation setting which has a fixed number of trajectories.
- In Section 4.2.2, the terminology of ‘implicit’ referring to direct comparison against current trajectory, and ‘explicit’ referring to comparison against trajectories resulting from other possible language corrections, seems upside-down intuitively, since the former is a more ‘explicit’ meaning of the correction.
- In Section 5.1, eq(12), what is the softmax being computed across? And how does the function $l$ generate language feedback?

**Edit after rebuttal:** The authors answered questions and clarified the necessary details.
questions: questions: See questions in review, particularly those regarding:
- Missing details in the evaluation sections
- Using language-based baselines
- Explaining results in Figure 10 and justifying the narrow margins in performance improvement
- The difference between iterative trajectory improvement, and reward learning, and why both aspects are important to include
limitations: limitations: 3
review_summary: review_summary: The paper proposes a novel method to incorporate language feedback in iteratively improving trajectories to better fit a human’s preferences. The work proposes a viable formulation and promising method, but evaluations could include language-based baselines and several details regarding the method and evaluation are unclear.
title: title: A practical approach to learn improved trajectories from language feedback
paper_summary: paper_summary: The paper propose a new approach to use the rich information contained in the language feedback to a trajectory and compare that to pairwise preferences.
main_review: main_review: Strengths:
1. The paper takes a step towards answering how language feedback may be used to guide finding optimal trajectories much more efficiently than comparitive feedback. This setting is more practical and extremely relevant to real world robot learning.
2. A novel approach is proposed: Learn trajectory embeddings such that their difference is aligned with a language text that explains that difference. With a learned embedding, whenever an user gives a language feedback on the trajectory, many trajectory comparisons can be obtained by creating artificial trajectories and assuming the humans are giving the most informative feedback.  
3. The algorithm is simple, based on supervised learning: It is tested for trajectory improvement using a pre decided dataset of trajectories and language feedback. Both synthetic and human experiments show that this method indeed achieved better performance compared to using comparitive feedbacks,


Weaknesses:
1. Obtaining supervision labels for training embedding model: Obtaining supervision labels for training the embedding model is a hard task. It is clear the using humans to label is not scalable and using VLM's to label can lead to hallucination. It is not discussed how the authors propose to solve this problem and consider a simple synthetic setting in their experiments.
2. Unable to handle ambiguity and lack of principled grounding: When a human gives a language feedback " move right" they may also want to "increase velocity" as another objective. By constructing fake labels we downweight the "increase velocity" and get stuck in suboptimal policies. The method thus lacks a principled grounding and it is not clear if optimal trajectories can be guaranteed.
3.  Policy optimization vs a toy setting: The authors consider an oversimplified setting of finding the best trajectory within a set of trajectories. While this setting makes the experiments easy it is an impractical one. In practice, it would be used in a policy optimization loop. How does the algorithm work in that setting remains an open question
4.  Situating in literature: The authors claim in the work they are the first to use language feedback for reward learning - this is untrue (See questions). Second, the authors perform a shallow literature review of the work of preference based learning, for example comparisons to various models that explore reward learning in PbRL from first principles are missing [1,2,3]

[1]: https://arxiv.org/abs/2202.03481
[2]: https://arxiv.org/abs/2310.13639
[3]: https://arxiv.org/abs/2206.02231
questions: questions: 1. Overclaim of first method to use language feedback in reward learning: Some examples where language feedback has been used in prior works [1,2].


[1]: https://arxiv.org/abs/2310.17555
[2]: https://cdn.aaai.org/ojs/16749/16749-13-20243-1-2-20210518.pdf
limitations: limitations: 3
review_summary: review_summary: Paper presents a new and simple solution but the experiments consider a toy setting making it far from real world world deployment.