Adaptive Language-Guided Abstraction from Contrastive Explanations

Authors: andi  peng
belinda z li
mit  ilia
sucholutsky  princeton
nishanth  kumar
julie a shah
mit  jacob
andreas  mit
andreea  bobu
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 04:00:54

Abstract: Many approaches to robot learning begin by inferring a reward function from a set of human demonstrations.
To learn a good reward, it is necessary to determine which features of the environment are relevant before determining how these features should be used to compute reward.
In particularly complex, high-dimensional environments, human demonstrators often struggle to fully specify their desired behavior from a small number of demonstrations.
End-to-end reward learning methods (e.g., using deep networks or program synthesis techniques) often yield brittle reward functions that are sensitive to spurious state features.
By contrast, humans can often generalizably learn from a small number of demonstrations by incorporating strong priors about what features of a demonstration are likely meaningful for a task of interest. 
How do we build robots that leverage this kind of background knowledge when learning from new demonstrations?
This paper describes a method named ALGAE which alternates between using language models to iteratively identify human-meaningful features needed to explain demonstrated behavior, then standard inverse reinforcement learning techniques to assign weights to these features.
Experiments across a variety of both simulated and real-world robot environments show that ALGAElearns generalizable reward functions defined on interpretable features using only small numbers of demonstrations.
Importantly, ALGAE can recognize when features are missing, then extract and define those features without any human input -- making it possible to quickly and efficiently acquire rich representations of user behavior.

Main paper: 
 introduction: 
 When training robots to perform complex tasks -like watering plants in cluttered household environments (Fig. 1 ) -it is often useful to begin by specifying a reward function from which optimal robot behavior can be derived. Assigning rewards to trajectories typically requires extracting important state or trajectory-level features (e.g. distance to goal or end effector orientation), then using these features to compute scalar rewards. While it is sometimes possible for humans to directly specify reward functions in code, this process is challenging and prone to error, even for experts [1] . A slightly more general class of approaches uses manual specification of feature functions (either in code or with targeted supervision [2, 3] ) followed by reward learning (e.g. with inverse reinforcement learning, or IRL [4] ). But it can be challenging for users to identify and describe all features relevant to a task of interest (e.g. the distance between the watering can and an optimal pouring height relative to the pot), so reward learning with human-designed feature sets runs the risk of feature under-specification -situations in which important task-relevant information is unavailable to the reward function. An alternative family of end-to-end approaches, such as deep IRL [5, 6, 7] attempt to implicitly extract these features from user demonstrations, but can require Figure 1 : Adaptive Language-Guided Abstraction from Contrastive Explanations (ALGAE) alternates between two main stages: in feature specification, ALGAE expands the current feature set by identifying under-specified features of the current reward; then in reward validation, ALGAE learns an updated reward function defined on top of the new feature set and validates it can explain the user demonstrations. ALGAE results in more generalizable learned reward functions vs. baselines without manual feature specification, and can iteratively improve its own reward estimate given multiple under-specified features.very large numbers of demonstrations to ensure that learned features are robust, generalizable, and insensitive to spurious features in the user demonstrations (e.g., distance to apple, etc. in Fig. 1 ).Reliable, sample-efficient reward learning thus requires both identifying a discrete set of features that are salient to the task and learning how they parameterize a continuous reward function [8] . Fortunately, good features that explain human decision-making do not come from a blank slate. When learning (and planning), humans draw upon a wealth of prior experience to identify which features are meaningful and generalizable for acting in the world [9, 10] . Can we instill these priors into reward learning procedures without requiring human supervision at an impractical scale?In this work, we propose to leverage the expressive human priors embedded in natural language text corpora. We describe an iterative framework for autonomously alternating between feature specification and reward learning. Our approach, called Adaptive Language-Guided Abstraction from contrastive Explanations, (ALGAE), leverages language models (LMs) in combination with user demonstrations to learn human-aligned reward functions informed by semantically-relevant features. ALGAE distills the reward learning problem into two components: 1) the feature specification problem of identifying missing reward features that are salient to the user's reward, and 2) the reward validation problem of finding and verifying a reward function that best explains demonstrated user behavior. By alternating between these two components, ALGAE iteratively builds increasingly rich representations of human decision-making while ensuring that learned rewards are explained by intuitive, semantically-meaningful features.We empirically validate ALGAE's benefits across simulated and real-world robot tasks. ALGAE recovers missing reward features and produces trajectories that better align with users' ground truth reward compared to several baselines. Importantly, ALGAE can iteratively refine its own representation by sequentially identifying missing features and validating them against the demonstrations, thereby mitigating the over-parameterization problem exhibited by end-to-end approaches. 
related work: 
 Our work builds on several fundamental ideas in reinforcement learning and reward learning.Inverse reinforcement learning. Inverse reinforcement learning (IRL) methods propose to learn the unknown objective function from observed actions in the environment, e.g., human demonstrations [4, 5, 11, 12] . Such methods suffer from identifiability issues [12, 13] . That is, given a constrained number of demonstrations, multiple objective functions can explain the same observed behavior. This is due to expressive function approximators overfitting to a few demonstrations, a problem that is exacerbated in high-dimensional and messy state spaces such as in robot learning.Learning features from users. A large body of work has studied how to elicit features (e.g. for use in RL) from human users. In this paradigm, learning good objectives is dependent on a set of carefully, hand-specified features that capture aspects of the environment or task that the user may care about [2, 14, 4, 1] . If selected well, this feature set introduces a well-formed inductive bias for facilitating more generalizable learning from few demonstrations. Such a feature set helps reduce the dimensionality of what is typically higher-dimensional states and trajectories, affording a better-shaped space for IRL to operate. But defining a good feature set is both challenging and laborious for experts and novices alike, and the chosen feature(s) may not be expressive enough to fully capture complex behavior in more ineffable robotic tasks [15] . Some studies explore having humans offer corrections to the agent using spatial labels [16] , teleoperation with joysticks [17, 18] , physical interaction [15] , and natural language [19] , but these approaches are difficult to scale as they involve expensive online human interventions.Learning features from language. At a high level, most of the previous work can be described as designing methods for users to specify their priors about the environment and task to the robot. Recent work on language-guided abstraction ( [20, 21] ) instead aims to learn task-relevant features from language by leveraging the semantic priors embedded in language models [22] to best inform which environment elements are important to include in the state and which elements are not. These approaches are limited by the imitation learning framework-that is, there is no true way to learn an objective that explains the demonstrated behavior. As a result, these approaches reduce the number of human demonstrations required to learn preference-tailored generalizable policies, but may still require expensive, online interventions to correct for feature under-specification.In this work, we extend the feature learning and language-guided frameworks to identify underspecified features using language-guided contrastive explanations between the demonstration and suboptimal trajectory, then iteratively learn reward functions using these features. 
problem definition: 
 We consider the problem of learning reward functions that capture the (unknown) preferences held by a human given a small number of user demonstrations and language.Preliminaries. We model our problem as a Markov Decision Process (MDP) [23] ⟨S, A, T , R⟩, where S ∈ R d is the state space, A the action space, T : S × A × S → [0, 1] the transition probability distribution, and R : S × A → R the reward function. A solution to the MDP is a policy π : S → A that specifies what actions the robot should take in different states. Following prior work [3] , we assume that the robot operates under a parameterized linear reward function defined on the state, R θ (s) = θ ⊤ ϕ(s), where ϕ : S → R p is a feature vector and θ ∈ R p represents the reward weights on the features. The robot executes a task by following a state trajectory τ = {s 0 , ..., s T } given by the policy. To pick the best trajectory for the task, the robot optimizes the cumulative reward R θ (τ ) = s t ∈τ θ ⊤ ϕ(s t ) = θ ⊤ ϕ(τ ), where ϕ(τ ) = s t ∈τ ϕ(s t ) is the total feature count along the trajectory, and solves:EQUATIONMaximum Entropy Inverse Reinforcement Learning (MaxEnt IRL). In practice, the reward function R θ is typically unknown to the robot or very challenging to manually specify. Thus, in IRL, the robot's goal is to learn R θ given human feedback like user demonstrations. Concretely, given human demonstrated trajectories D = {τ i } i=1...N , the robot interprets them as evidence for the human's preferred behavior and attempts to recover the reward parameters θ that explain the human's desired objective. We adopt the maximum-entropy framework for modeling human decisionmaking [12, 5] , and model the human as a noisily-optimal agent that tends to choose demonstrations in proportion to the exponentiated reward:EQUATIONUnder this framework, the human is likely to act optimally and will generate suboptimal trajectories with a probability that decreases exponentially as the trajectories become lower in reward [12] . The unknown reward parameters θ can then be optimized via gradient descent on the objective:EQUATIONTo optimize this objective, we approximate the intractable integral in Eq. ( 2) using importance sampling, as in prior work [5, 3] . The robot then acts according to Eq. ( 1).The features ϕ we choose to represent the reward R θ dramatically impact the reward functions that can be learned [2] . In the limited-data regime, this is true even when R θ belongs to an expressive function class (e.g., a neural network). Motivated by recovering a reward function R θ that incorporates human-like priors, we are interested in explicitly constructing human-meaningful features ϕ.In this sense, we consider a feature set to be under-specified if it does not explicitly represent all the salient features relevant to the task, and consider a reward function to be under-specified if it uses an under-specified feature set as its basis. 
adaptive language-guided abstraction from contrastive explanations: 
 At a high level, ALGAE aims to identify the most likely set of preference weights θ * and preference featurizations ϕ * given the human demonstrations under p(ϕ, θ | τ 1 , ..., τ N ). The idealized joint optimization given demonstration τ thus takes the following form:EQUATIONWe maximize this objective tractably by iteratively optimizing θ and ϕ. Given an initialization of the feature vector ϕ 0 , ALGAE decomposes the learning problem into two subtasks: 1) performing feature specification by expanding each feature set ϕ k into ϕ k+1 by querying for missing features in language, then 2) performing reward validation by finding an optimal reward weight θ k for each ϕ k and comparing its predictions to ground-truth trajectories.In feature specification, we use an LM to identify missing features salient to the user's reward given a demonstration τ and the robot's current best estimate of the (under-specified) reward function R θ k .Once the missing feature has been identified, we query the LM for code to compute that missing feature and append the resulting feature function ϕ i to the set of known features ϕ k to obtain ϕ k+1 .Operationally, by fixing θ = θ k , Eq. ( 4) becomes:EQUATIONwhere p(ϕ | θ k , τ ) may be estimated using an LM as described below.Querying for a missing feature. The key insight of our approach is that language models have strong common-sense priors over salient task features (e.g., ideal watering height above plant) which we can leverage to approximate the probability p(ϕ | θ k , τ ) in Eq. ( 5). Concretely, we approximate the optimization Eq. ( 5) by first computing best estimated trajectory for the current reward τ R θ k using Eq. ( 1) and pair it with the user demonstrated trajectory τ . We then prompt the LM with the contrastive pair of trajectories (full state information concatenated over all timesteps) along with a high level description of the task (e.g. the user wishes to water the plant) and environment context (a description of the state of the environment) and query the LM for the likeliest feature ϕ i that explains this difference 1 . Intuitively, τ R θ k acts as evidence for θ k that is easier for the LM to if ∥ϕ * (τ R θ k ) -ϕ * (τ H )∥ < ϵ // If optimized trajectory explains the demo then break ϕ i = LM(τ R θ k , τ H ) // Query LM for missing feature that explains the contrast. ϕ k+1 ← ϕ k ∪ ϕ i interpret than reward weights, resulting in a sample from the LM's prior on p(ϕ i | τ R θ k , τ ). Thus, our approximate solution ϕ k+1 concatenates the LM sampled ϕ i with the previous vector ϕ k .Querying for feature code. Now that we have identified a missing feature ϕ i , we query the LM to directly generate the feature definition code that can be used to compute a feature value from raw state data over the given demonstrations. The LM is given raw trajectories τ and a natural language feature description of ϕ i , generated from the previous stage, and is asked to generate its code ϕ i . We append ϕ i and its respective function code to the existing list of features ϕ k+1 ← ϕ k ∪ ϕ i .In the second stage, reward validation, we train and validate a reward function using the updated set of features ϕ k+1 as the basis function for the new reward, learning new parameters θ k+1 . Because there may be additional missing features we have yet to identify, we now leverage existing user demonstrations to validate whether reward learning is complete. This iterative update is motivated by the desire to not over-parameterize the reward by specifying non-salient features.Learning a new reward. Given our updated feature set ϕ k+1 , we fix ϕ = ϕ k+1 in Eq. ( 4) to obtain:EQUATIONassuming uniform p(θ | ϕ k+1 ). This equation directly corresponds to the MaxEnt IRL optimization procedure in Section 3 using ϕ k+1 as the feature vector. We now learn updated reward weights θ k+1 from the user demonstrations via Eq. ( 3), then optimize for an updated trajectory τ R θ k+1 via Eq. ( 1).Validating the new reward. To determine whether we have recovered all salient features, we leverage existing user demonstrations for validation. In a real-world deployment scenario, there are many reasonable ways to implement a comparison between trajectories, with the ideal being a human user giving feedback as to whether the robot is executing the desired behavior. However, in our simulated experiments, we approximate this judgement by measuring L2 distance between these trajectories in the ground truth feature space ϕ * . If this is within a threshold ϵ, the learned reward function is sufficient for explaining the desired behavior and the algorithm terminates; otherwise, we iterate through the feature specification stage again. Pseudocode for the full pipeline can be seen in Algorithm 1. 
simulated experiments: 
 We evaluate ALGAE in both simulated and real-world robot environments. To highlight the flexibility of ALGAE across diverse feature types and high-dimensional control, our domains include 2D maze navigation tasks and a 7DoF JACO tabletop manipulation task, along with real-world mobile manipulation tasks with a Spot in Section 6. We use GPT-4o [24] to identify salient missing features in all experiments.Figure 2 : We evaluate on both simulated and real-world domains with a variety of missing features.A: 2D maze navigation, where the robot must navigate to a goal while interacting with other objects. B: 7DoF JACO manipulation, where the JACO arm must manipulate a held coffee mug while respecting features like end effector orientation. C: Spot mobile manipulation tasks, where the robot must complete tasks like WaterPlant while respecting the height of pot.Domain 1: 2D Navigation. Inspired by the AI Safety Gridworlds [25] , where under-specified features can implicitly lead to reward hacking behavior [26] , we evaluate ALGAE on the following 2D maze navigation environments: GridRobot, Lavaland, and Island (Fig. 2 ). Trajectories are sequences of 9 states beginning with the start and ending at the goal. The 18-dimensional input consists of the xy coordinate positions of the robot in the grid at each timestep. The action space is the four coordinate directions. Each scenario is a 5-by-5 2D grid consisting of an agent, a goal the agent must navigate to, and possible objects the agent must interact with. In GridRobot, the agent must avoid the (1) Stove; in Lavaland, the agent must avoid (2) Lava; in Island, the agent needs to drink (move to) (3) Water. To ensure there is the possibility of over-specifying features, we also randomly place non-relevant objects that do not impact the true reward in the environment (e.g., Ball).Domain 2: 7DoF Tabletop Manipulation. To assess a higher-dimensional state and action space, we also evaluate a 7DoF JACO robot arm tasked with manipulating a coffee cup on a tabletop in a PyBullet simulator (see Fig. 2 .) Each scenario is initialized with a starting and goal pose location, a laptop placed on the tabletop, and a human by the table. The robot must manipulate the coffee cup it is carrying to a specific goal location while respecting user preferences for motion. Trajectories are length 21, and each state has 97 dimensions: the xyz positions of all robot joints and angles, and xyz object positions along with rotation matrices. We introduce four scenarios based on the relevant human preference: (1) Laptop: xy-plane distance of the EE to the laptop, to ensure the cup does not pass over the laptop, (2) Table : z-distance of the EE to the table, to keep the mug of height close to the table, (3) Orientation: EE orientation relative to upright, to ensure the cup does not spill, and (4) Human: xy-distance of the EE to the human, to ensure the cup does not collide with the human.Test Scenarios. To study whether our learned reward functions generalize across different scenarios related to spurious environment features, we construct two possible distribution shifts for each demonstrated scenario: (1) different trajectory goals (Goal) (e.g., the robot must move to a different goal location) and ( 2) different feature object locations (Object) (e.g., the laptop changes location in the scene). These are intended to test whether the learned reward captures true salient environment features rather than spurious feature correlations. We report the normalized reward of the optimized trajectory (against the reward of the ground truth best and worst trajectories) on a set of five randomly sampled test scenarios across three seeds.Comparisons. A straightforward way to learn rewards from demonstrations is to learn directly from the state space. We refer to this baseline as (1) Full-State. We would expect such a baseline to fail to generalize to novel environments since the learned reward may be prone to overfitting spurious state correlations in the demonstrations, rather than true salient features in the environment. However, if access to a language prior was given without contrastive demonstrations (such that there is no demonstrated trajectory of the under-specified feature), we could query for all hypothesized features at once as (2) LM-Feature. This baseline is intended to isolate the impact of the language model proposing features given only the task description and environment context, and can be seen as a prompting-only baseline. For an additional prompting-only baseline, we also evaluate (3) LM-Reward, a baseline where the LM also proposes the reward weights in addition to the features.Inspired by work that proposes to construct rewards directly from language [27, 28, 29] , this baseline is intended to test the zero-shot reward specification capability of LMs. Finally, to benchmark these values, we report a (4) Random reward, which randomly samples a trajectory for each test scenario.Single feature recovery. We first assess the ability of ALGAE to recover a single missing feature. Table 1 shows results across test environments and their feature scenarios. As shown, ALGAE outperforms comparisons across all features. In simpler 2D environments such as GridRobot and Lavaland, ALGAE achieves close to perfect feature recovery and generalization on unseen test scenarios. While the two prompting-only baselines (LM-Feature and LM-Reward) perform well above Random, upon inspection of the LM rewards, they tend to produce many irrelevant features which result in over-parameterized reward functions. In the more complex JACO scenarios, ALGAE consistently outperforms baselines. Its performance is close to the prompting-only baselines on the Orientation task, which is unsurprising as there is no explicit penalty for over-parameterizing the reward (i.e., the robot can keep the cup upright while optimizing for additional features).Iterative feature recovery. We next study the case where more than one feature is under-specified in a scenario, and ALGAE must recover them iteratively. In these experiments, the training scenario is instantiated with two missing features (Water and Dinosaur for Island) and (Laptop and Orientation for JACO). As seen in Fig. 3 , ALGAE iteratively improves reward given multiple missing features. 
real-world experiments: 
 To assess the feasibility of our approach on robotic hardware operating in cluttered human environments, we now evaluate ALGAE's performance on mobile manipulation tasks with a Spot robot.Domain 3: Spot Mobile Manipulation. Spot is a legged robot equipped with a manipulator (arm with gripper) and six RGB-D cameras (one in gripper, two in front, one on each side, one in back). Each scenario is initialized with a starting and goal robot pose location, along with the following objects: watering can, plant pot, apple, towel, and textbook. Trajectories are length 5 and each state consists of 10 dimensions: the xyz positions of the robot and its hand, as well as rotation of the gripper. We introduce the following three scenarios and their relevant reward preferences: (1) GetWateringCan (xy-distance of the robot to the watering can), to ensure the robot is positioned to execute a successful grip, ( 2 height above pot), to ensure the hand is adjusting to the pot size, and ( 3) ThrowAwayRag (xy-distance of the hand to the textbook), to ensure the robot does not drip water from the carried rag.Data collection. For each scenario, a human teleoperates the robot from the starting to goal pose while satisfying the hidden preference (e.g. raise the robot hand to a particular height above the pot before performing a watering motion), and generates three demonstrations per scenario. We also collect all detected object identities and positions via image segmentation from the robot cameras [30, 31] and captioning [32] as environment context for the demonstrations.Test scenarios and comparisons. We evaluate on the following test scenarios: for GetWateringCan, we vary the location of the watering can before executing a grip, for WaterPlant, we vary the height of the plant pot before executing a pour, and for ThrowAwayRag, we vary the location of the textbook on the table the robot hand must avoid while carrying the wet rag. To ensure the robot does not execute a drastically wrong action given an incorrectly learned reward, we constrain the sample space of the robot pose at test time around safe zones. We report the normalized reward of the optimized trajectory on a set of three test scenarios across three seeds. We report comparisons with the previous best performing method (LM-Reward) along with a Random benchmark.Results and Analysis. ALGAE outperforms baselines on real-world scenarios (Fig. 3 ). This gap actually increases relative to simulated environments due to the real-world clutter introduced in the scene. Upon visual inspection of the LM-Reward features, we indeed see unnecessary features (e.g. relative position of the robot to all objects in the scene, not just the salient ones) included. This underscores the usefulness of ALGAE's interactive, iterative approach to feature construction. 
discussion and limitations: 
 ALGAE is an iterative reward learning framework that alternates between specifying missing features with a LM and weighting them into a reward via IRL. One limitation is that we used the ground truth feature vector to judge task completion on the contrasting pair of trajectories; future work might evaluate a true human-in-the-loop scenario, and study whether LM-informed priors have additional benefits beyond the performance improvements evaluated here. Moreover, we focus on reward functions that are linear in the feature set; in the future, we can imagine extending our method for nonlinear rewards. Finally, ALGAE is only effective for cases where the LM prior agrees with the desired behavior -which may not always be true in practice.We provide the prompts used for both the feature query in all domains.2D Navigation.An agent is in a 2D gridworld environment of size 5x5. (0,0) is the top left corner and (4,4) is the bottom right corner (note this is not a typical coordinate grid).The ): position(s) of other objects in the environment. This will be empty if there is no object in that slot. objects{1-3} color (str): color of the corresponding object at the position objects{1-3}. This will be empty if there is no object in that slot. objects{1-3} type (str): object type of the corresponding object at the position objects{1-3}. This will be empty if there is no object in that slot.This is an environment config (disregard the extra attributes that are not specified above): {env}The following is a demonstration the agent took in this first environment (each trajectory contains x,y coordinates for the agent, 8 timesteps total). Keep track of where the agent's positions relative to the other objects in the environment. For example, [0,0 0,1 0,2 0,3] means the agent started at [0,0] and moved to [0,3]. If there is an object at [0,3], the agent moved towards that object.The following is an (incorrect) demonstration that the agent took in this environment. The trajectory is the same format as the first demonstration. incorrect trajectory = {best traj} Describe the demonstration and trajectory in the context of the environment they are in, then use this to give the specific feature in that environment that explains what the demonstration did better. Reason step by step and think about why the agent might be moving in the way that it is. If there are multiple of the same object, please give the average distance to all objects of that type. Please specify a feature (only one feature e.g. if there are multiple possible explanations choose the most likely object) that we can write code to compute from the position of the agent to other objects in the scene. This is the current list of features that we already know: {self.feature names}. Please give a different feature in your answer (e.g. if distance to goal feature is already in the list, do not give distance to goal again). Please give your confidence for your answer between 0 and 1. Ground your answer in visible objects in the scene, and give an answer (only 1 specific answer (e.g. distance to book objects)) that we can compute relative to the position of the agent.A 7-DoF Jaco robot arm is carrying a coffee cup on a tabletop environment. The arm is carrying a fragile ceramic coffee cup that is at risk of slipping. The arm is controlled via a Pybullet simulator.The environment config has the following attributes: object centers: {'HUMAN CENTER': [x,y,z], 'LAPTOP CENTER': [x,y,z]} # coordinates of the human and laptop objects in the environment tabletop: z=0 # the z-coordinate plane of the tabletop This is an environment config (disregard the extra attributes that are not specified above): {env}The following is a correct demonstration trajectory the robot took in this first environment. Each trajectory is a 21x97 matrix representing the 21 timesteps the robot took over the course of the demonstration. Each state in the timestep is a 97-dimensional vector representing the x,y,z positions of all robot joints and objects, and their rotation matrices. This is the code that generates each state. The following is an incorrect trajectory that the robot took in this environment. The trajectory is the same format as the first demonstration. incorrect trajectory = {best traj} Describe this trajectory in the context of the environment it is in, then use this to give the specific feature in that environment that explains why this trajectory was incorrect relative to the demonstration, which was correct. Reason step by step and think about how the robot's movements are related to the objects in the environment, as well as how aspects of the robot's position and orientations may be impacting the desired task (bringing coffee). Please specify a feature (only one feature e.g. if there are multiple possible explanations choose the most likely object) that we can write code to compute from the position of the robot to other objects in the scene. This is the current list of features that we already know: {self.feature names}. Please give a different feature in your answer (e.g. if distance to goal feature is already in the list, do not give distance to goal again). Please give your confidence for your answer between 0 and 1. Ground your answer in visible objects in the scene, and give an answer (only 1 specific answer (e.g. distance to book)) that we can compute from the position of the robot.This is an environment configuration of objects and their xyz positions in the room.The following is a successful trajectory (over 5 timesteps) the robot took in this environment (each trajectory contains xyz positions of the robot, along with the xyz positions and rotation quarternion of the robot's hand). Keep track of where the robot's positions and hand are relative to the other objects in the environment.The following is an unsuccessful trajectory (over 5 timesteps) the robot took in this environment.Failed trajectory: {failed traj} Describe the key feature that distinguishes the successful trajectory from the failed trajectory. Make sure the feature is semantically sensible for the goal. Reason step by step. Please specify a feature (only one feature e.g. if there are multiple possible explanations choose the most likely one). Make sure the feature is computable from the state features in the scene (and no other information): plant XYZ, robot XYZ, robot arm XYZ, robot arm rotation. The feature can be relative positions to objects, relative positions to positions around objects (e.g. a force field around the object), relative positions to absolute points in space, relative rotations, etc. The feature can even be non-linear. However, make sure the feature is continuous and not discrete. Please describe the feature in detail.8th Conference on Robot Learning (CoRL 2024), Munich, Germany.If there are multiple missing features, the algorithm will identify them in future iterations. 
 
 Appendix:  


Decision: The paper's strengths include:
The ALGAE method, which leverages language models for feature learning in inverse reinforcement learning (IRL), offers a novel approach by iteratively learning and integrating features relevant to the reward function.
The separation of feature learning and reward learning into distinct subproblems helps in focusing on each component effectively, combining the strengths of data-driven and model-based approaches.
The paper provides empirical evidence of the method's effectiveness across both simulated and real-world environments, demonstrating clear advantages over baseline methods and ablated versions.
The writing is clear and well-organized, making the method and its results easy to understand. The iterative feature addition and evaluation process is a valuable contribution to the field.

The paper's weaknesses include:
The experiments are somewhat engineered with a focus on a small number of features, which may not fully showcase the method's robustness in more complex scenarios. More diverse baselines and experiments are needed.
The experiments utilize highly handcrafted state representations, which could bias the results. More exploration with less engineered state representations or full image inputs would be beneficial.
The paper does not thoroughly address limitations or failure cases, such as generalization to new tasks or environments and reliance on known ground truth feature vectors.
The design of prompts for language models involves significant domain knowledge and tuning, which might not be practical for all applications.

Post-rebuttal meta-review:
The paper introduces ALGAE, a method designed to improve robot learning by leveraging language models to identify and incorporate relevant features into the reward learning process, particularly in scenarios with limited human demonstrations.

ALGAE uses language models to iteratively identify human-meaningful features, which are then used to guide the reward learning process. The method alternates between feature identification and reward optimization, allowing it to refine its understanding and adapt to missing features without human intervention. ALGAE demonstrates its ability to generalize learned reward functions across various simulated and real-world robotic environments.

The integration of language models into feature learning for inverse reinforcement learning (IRL) is a novel approach. The paper is well-written, clearly explaining the method and supporting its claims with solid empirical evidence in both simulation and real-world experiments. The method shows robustness in extracting meaningful features and generalizing across different tasks.

The experiments appear heavily engineered, with a focus on a small number of features, which may not fully showcase the method's robustness in more complex scenarios. The reliance on highly handcrafted state representations could bias the results, limiting the exploration of more diverse or less engineered inputs. The paper lacks a deep comparative analysis with state-of-the-art methods, which could better demonstrate its distinctiveness and impact.

Final Recommendation: Borderline Accept-- The paper presents a novel approach with promising results, but it has some limitations, such as engineered experiments and a lack of comprehensive baselines, which may affect its broader applicability. While it is a valuable contribution, further work could strengthen its claims.

Reviews: review: title: title: Submission422 Review
paper_summary: paper_summary: The authors introduce a method called ALGAE, which aims to improve robot learning by leveraging human-like priors encoded in natural language. The method addresses the challenge of learning robust reward functions from a limited set of human demonstrations by identifying and incorporating relevant features into the reward learning process.
main_review: main_review: The authors introduce a method called ALGAE, which aims to improve robot learning by leveraging human-like priors encoded in natural language. The method addresses the challenge of learning robust reward functions from a limited set of human demonstrations by identifying and incorporating relevant features into the reward learning process.
The authors provide a detailed explanation of the method and support their claims with empirical evidence from experiments conducted in both simulated and real-world environments.
The paper is well written generally and and the use of figures and diagrams effectively show the contribution.  However, certain technical sections, especially those involving equations, could benefit from more intuitive explanations. 
The paper does not thoroughly discuss limitations or failure cases of the method. Addressing these aspects could provide a more deatailed view of the method's applicability and robustness in various scenarios. more detailed comparisons with a wider range of existing method, would help to better highlight the unique advantages of the approach
questions: questions: How well does ALGAE generalize to entirely new tasks (or environments) that were not seen during the training phase?
limitations: limitations: 2
review_summary: review_summary: I recommend an accept
title: title: novel idea for feature learning in IRL
paper_summary: paper_summary: This paper proposes to leverage language models to learn missing features for inverse reinforcement learning. The reward learning problem is decoupled into two sub problems: 1) feature learning with language model reasoning about one missing feature at a time; and 2) reward learning using demonstrations and learned features. Experiments in both sim and real domains show the effectiveness of the proposed method in comparison with baseline and ablated methods.
main_review: main_review: Strengths:
- The idea of leveraging pretrained language models for iterative feature learning in IRL is novel
- The proposed method cleanly decouples the reward learning and feature learning problem
- Incorporating large language model functionality into traditional IRL method combines the strengths of data-driven and model-based methods
- Experiments in this work contain different domains in both sim and real
- The proposed method is compared with ablations
- The writing and presentation of this paper is clear and easy to follow

The major weakness of this work is that the experiments are heavily engineered:
- In all tasks, there are at most 3 features to learn, which favors the proposed method that learns one feature at a time
- One baseline could be randomly sampling a subset of features and pick the one that best explain the demos
- Current experiments assume access to known GT feature vectors for determining if the current set of features are good enough, which the authors acknowledge as a major limitation but do not practically address or demonstrate the performance of the system without such assumption
- The prompt design seems to involve a lot of domain knowledge and tuning effort
- if these tasks are easy enough to hand engineer rewards, why use LLMs to learn?
questions: questions: - [see comments on weakness above]
- Can you show an example input trajectory to the LLM and its response? How do you make sure if LLM can understand the trajectory in the form of numbers/matrices? Does the output of the LLM end up writing code to parse the trajectory information or it only uses text? 
- On a similar note to the question above, why not use visual state as input for LLM? For the LM-feature ablation, I can imagine if the LLM does not see the layout of the environment, it may want to track each and every object for obstacle avoidance; but if it could see the image and understand what is near the robot and target object, it might filter unrelated features
- How does the proposed method compare with zero-shot using language models to plan robot behavior? If it is able to understand trajectories in numbers, seems like it can also be prompted to output the trajectory directly. Similarity, is VoxPoser [1] a fair baseline here?

[1] https://voxposer.github.io/
limitations: limitations: 1
review_summary: review_summary: The method is novel and interesting but the experiments are toy and more baselines should be included.
title: title: Very compelling and generally useful inverse RL algorithm using language model
paper_summary: paper_summary: The main contribution is a method for inverse RL that uses a language model to determine reward features that are relevant for the task.
main_review: main_review: The authors present a very compelling story for reward selection. It's a welcome addition to add more to the reward learning with LM literature.

The most compelling pieces is the iterative feature addition and the evaluation/weighting of the generated reward function. In prior reward learning work [1], weighting was often ignored and reward function generation was basically considered a zero-shot process. There was no iteration once the reward function was generated, so this iterative process lets you craft the best reward given a set of demonstrations. I think this system is very valuable, and likely to be helpful in constructing reward functions from language models.

The experiments are promising, but it would be nice to see some more results here. The core hypothesis is that human-crafted feature selection is hard, and it's prone to reward hacking, so the priors in the language model can propose the most salient features for the task. However, the state representation for all the experiments were highly handcrafted already (the positions of the robot and objects). It is unlikely that the policy will focus on spurious features that are not relevant once there is already a small state space (<50 values). By handcrafting state features, you are already doing some steering of what kind of reward function the language model should generate.

For the purpose of this paper which is proving the ALGAE method, this is alright. But it would be nice to show an example where the state representation might be a full image, and higher level features could be extracted from the VLM (via tool use) when computing the reward function [2,3].

The gridworld experiments are a good unit test, and it's good to prove that ALGAE outperformed baseline methods in that toy scenario. The sim robot experiment and real robot experiment were a bit nicer, but like I mentioned above, it would be nicer if the state representation wasn't already highly reduced.

The method is very general, and the contrastive method of evaluating the reward function is very appealing. You could even have some holdout demonstrations from your dataset if you didn't want to take a demonstration in-the-loop.

[1] Language to Rewards for Robotic Skill Synthesis, Yu et al
[2] ViperGPT: Visual Inference via Python Execution for Reasoning, Suris, Memon, Vondrick
[3] How to Prompt Your Robot: A PromptBook for Manipulation Skills with Code as Policies, M.G. Arenas et al
questions: questions: - The reward function building adds new features at every iteration. If a bad feature is proposed, how could we determine if we should not include it? Or is that implicit in the feature weighting.
- A small detail: was the feature weighting a simple linear combination of weights, or something more complex in your experiments?
- The biggest limitation is most likely the simple state representation. Could you comment on how to remove some of the handcrafting of the state representation when doing reward function creation?
limitations: limitations: 3
review_summary: review_summary: Overall, the work is quite nice. The iterative construction and evaluation of the reward function is a good contribution. Would love to see some more real robot experiments with a bit less state tuning too.