Adaptive Language-Guided Abstraction from Contrastive Explanations
Authors: Andi Peng
Belinda Z. Li
Ilia Sucholutsky
Nishanth Kumar
Julie Shah
Jacob Andreas
Andreea Bobu
Venue: CoRL 2024 (2024)
Keywords: reward learning
language-guided abstraction
reward features
TL;DR: We propose an algorithm to iteratively specify missing features and then update the reward parameters for reward learning.

Abstract: Many approaches to robot learning begin by inferring a reward function from a set of human demonstrations.
To learn a good reward, it is necessary to determine which features of the environment are relevant before determining how these features should be used to compute reward.
In particularly complex, high-dimensional environments, human demonstrators often struggle to fully specify their desired behavior from a small number of demonstrations.
End-to-end reward learning methods (e.g., using deep networks or program synthesis techniques) often yield brittle reward functions that are sensitive to spurious state features.
By contrast, humans can often generalizably learn from a small number of demonstrations by incorporating strong priors about what features of a demonstration are likely meaningful for a task of interest. 
How do we build robots that leverage this kind of background knowledge when learning from new demonstrations?
This paper describes a method named ALGAE which alternates between using language models to iteratively identify human-meaningful features needed to explain demonstrated behavior, then standard inverse reinforcement learning techniques to assign weights to these features.
Experiments across a variety of both simulated and real-world robot environments show that ALGAElearns generalizable reward functions defined on interpretable features using only small numbers of demonstrations.
Importantly, ALGAE can recognize when features are missing, then extract and define those features without any human input -- making it possible to quickly and efficiently acquire rich representations of user behavior.

Adaptive Language-Guided Abstraction from Contrastive Explanations
Andi Peng MIT
Belinda Z. Li MIT
Ilia Sucholutsky Princeton
Nishanth Kumar MIT
Julie A. Shah
Jacob Andreas MIT
Andreea Bobu The Al Institute
MIT
Abstract: Many approaches to robot learning begin by inferring reward fune- tion from a set of human demonstrations. To learn a good reward, it is necessary to determine which features the environment are relevant before determining how these features should be used to compute reward. End-to-end methods for joint feature and reward learning (e.g., using deep networks or program synthesis tech- niques) often yield brittle reward functions that sensitive spurious state fea- tures. By contrast, humans can often generalizably learn from a small number of demonstrations by incorporating strong priors about what features of a demonstra- tion are likely meaningful for a task of interest. How do we build robots that lever- age this kind of background knowledge when learning from new demonstrations? This paper describes a method named ALGAE (Adaptive Language-Guin Guidedd Ab- straction from [Contrastive] Explanations) which alternates between using lan- guage models to iteratively identify human-meaningful features needed explain demonstrated behavior, then standard inverse reinforcement learning techniques assign weights to these features. Experiments across variety of both simulated and real-world robot environments show that ALGAE learns generalizable reward functions defined on interpretable features using only small numbers of demon- strations. Importantly, ALGAE can recognize when features are missing, then extract and define those features without any human input - making it possible quickly and efficiently acquire rich representations user behavior.
Keywords: reward learning, language-guided abstraction, reward features
Introduction
When training robots to perform complex tasks like watering plants in cluttered household envi- ronments (Fig. 1) it is often useful to begin by specifying reward function from which optimal robot behavior can be derived. Assigning rewards to trajectories typically requires extracting impor- lant state or trajectory-level features (e.g. distance to goal or end effector orientation), then using these features to compute scalar rewards. While it is sometimes possible for humans to directly specify reward functions in code, this process is challenging and prone to error, even for experts [1]. A slightly more general class of approaches uses manual specification of feature functions (ci- ther in code or with targeted supervision [2. 3]) followed by reward learning (e.g. with inverse reinforcement learning, IRL [4]). But it can be challenging for users to identify and describe all features relevant to task of interest (e.g. the distance between the watering can and an opti- mal pouring height relative to the pot), so reward learning with human-designed feature sets runs the risk of feature inder-specification situations in which important task-relevant information is unavailable the reward function. An alternative family of end-to-end approaches, such as deep IRL [5, 6, 7] attempt to implicitly extract these features from user demonstrations, but can require
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Reward validation:
validate the updated reward against user demonstration
Demoustration
θ
Best Trojectory under $
distance to jo_plant, height_above..
distance to plantt
Contrastive explanation: height, above_pot
Feature specification:
Feature code:
ideniify missing features
heightitrai postoon: hand, PCS, fral hand poo hand pos. y hand poo
Possible Features
Figure Adaptive Language-Guided Abstraction from Contrastive Explanations (ALGAE) alternates between two main stages: in feature specification. ALGAE expands the current feature set by identifying under-specified features the current reward; then in reward validation. ALGAE learns updated reward function defined top of the feature and validates it explain the user demonstrations. ALGAE results in more generalizable learned reward functions vs. base- lines without manual feature specification, and can iteratively improve its reward estimate given multiple under-specified features.
very large numbers of demonstrations ensure that learned features are robust, generalizable, and insensitive to spurious features in the user demonstrations (e.g., distance apple, etc. in Fig. 1).
Reliable, sample-efficient reward learning thus requires both identifying a discrete set of features that are salient to the task and learning how they parameterize a continuous reward function [8]. Fortunately, good features that explain human decision-making do not come from blank slate. When learning (and planning), humans draw upon wealth of prior experience to identify which features are meaningful and generalizable for acting the world [9, 10]. Can we instill these priors into reward learning procedures without requiring human supervision at an impractical scale?
In this work, we propose to leverage the expressive human priors embedded in natural language text corpora. We describe an iterative framework for autonomously alternating between feature speci- fication and reward learning. Our approach, called Adaptive Language-Guut Guided Abstraction from contrastive Explanations, (ALGAE), leverages language models (LMs) combination with user demonstrations to learn human- human-aligned reward functions informed by semantically-releet
features. ALGAE distills the reward learning problem into components: 1) the feature specification prob- lem of identifying missing reward features that are salient to the user's reward, and 2) the reward validation problem of finding and verifying reward function that best explains demonstrated user behavior. By alternating between these two components, ALGAE iteratively builds increasingly rich representations human decision-making while ensuring that learned rewards are explained by intuitive, semanticalyy-meaninghf features.
We empirically validate ALGAE's benefits across simulated and real-world robot tasks. ALGAE recovers missing reward features and produces trajectories that better align with users' ground truth reward compared to several baselines. Importantly, ALGAE can iteratively refine its own represen- tation by sequentially identifying missing features and validating them against the demonstrations, thereby mitigating the over-parameterization problem exhibited by end-to-end approaches.
2
Related Work
Our work builds on several fundamental ideas reinforcement learning and reward learning.
Inverse reinforcement learning. Inverse reinforcement learning (IRL) methods propose learn the unknown objective function from observed actions in the environment, e.g., human demonstrations [4, 5, 11, 12]. Such methods suffer from identifiability issues [12, 13]. That is, given a constrained

---

number of demonstrations, multiple objective functions can explain the same observed behavior. This is due to expressive function approximators overfitring to a few demonstrations, a problem that is exacerbated in high- nigh-dimensional and messy state spaces such as in robot learning.
Learning features from users. large body of work has studied how to elicit features (e.g- for use in RL) from human users. In this paradigm, learning good objectives is dependent features that capture aspects of the environment or task that the user may care about [2, 14, 4, 1]. If selected well, this feature set introduces a well-formed inductive bias for facilitating more generalizable learning from few demonstrations. Such a feature set helps reduce the dimensionality of what is typically higher-dimensionall states and trajectories, affording a better-shaped space for IRL. to operate. But defining a good feature set is both challenging and laborious for experts and novices alike, and the chosen feature(s) may not be expressive enough to fully capture complex behavior in more ineffable robotic tasks [15]. Some studies explore having humans offer corrections to the agent using spatial labels [16], teleoperation with joysticks [17, 18), physical interaction [15], and natural language [19], but these approaches are difficult to scale as they involve expensive online human interventions.
set
of carefully,
Learning features from language. At a high level, most of the previous work can be described as designing methods for users to specify their priors about the environment and task to the robot. Recent work language-guided abstraction ([20, 21]) instead aims to learn task-relevant features from language by leveraging the semantic priors embedded in language models [22] to best inform which enviromment elements are important to include in the state and which elements are not. These approaches are limited by the imitation learning framework-tha is, there is no true way to learn an objective that explains the demonstrated behavior. As a result, these approaches reduce the number of human demonstrations required to learn sefference-tillrre generalizable policies, but may still require expensive, online interventions correct for feature under-speciffiattio.
In this work, we extend the feature learning and sanguage-guided frameworks to identify under- specified features using anguage-guided contrastive explanations between the demonstration and suboptimal trajectory, then iteratively learn reward functions using these features.
3 Problem Definition
We consider the problem of learning reward functions that capture the (unknown) preferences held by a human given a small number of user demonstrations and language.
Preliminaries. We model our problem a Markov Decision Process (MDP) [23] (S, A, R), where S Rd is the state space, AA the action space, T S A S - [0, the transition probability distribution, and R: S x A + R the reward function. A solution to the MDP is policy :: + A that specifies what actions the robot should take in different states. Following prior work [3], we assume that the robot operates under a parameterized linear reward function defined on the state, (s) = T ((s), where ő S + Iº' is feature vector and E Rp represents the reward weights on the features. The robot executes a task by following a state trajectory = (s"..., 8 given by the policy. To pick the best trajectory for the task, the robot optimizes the cumulative reward (T) = E. A (Td(s")= = T ((T), where ((T) = Ever o(s') is the total feature count along the trajectory, and solves:
TRa arg max Ro(T) arg max 0 ((())
Maximum Entropy Inverse Reinforcement Learning (MaxEnt IRL). In practice, the reward function Rs is typically unknown to the robot or very challenging to manually specify. Thus, in IRL, the robot's goal is to learn R, given human feedback like user demonstrations. Coneretely, given human demonstrated trajectories D = (7.) i==...N, the robot interprets them as evidence for the human's 's preferred behavior and attempts recover the reward parameters 0 that explain the hu- man's 's desired objective. We adopt the maximum-entropy framework for modeling human decision- making [12, 5], and model the human noisily-optine agent that tends to choose demonstrations
3

---

in proportion to the exponentiated reward:
(
p(T (0) - J d
x exp(Ro(7)) = exp(1" x(T))
(2)
Under this framework, the human is likely to act optimally and will generate suboptimal trajectories with probability that decreases exponentially as the trajectories become lower in reward [12]. The unknown reward parameters can then be optimized via gradient descent on the objective:
0- = arg max ((0) = arg max
log((T (0)
TED
To optimize this objective, we approximate the intractable integral in Eq. (2) using importance sam- pling, as in prior work [5, 3]. The robot then acts according to Eq. (I).
The features é we choose to represent the reward Rg dramatically impact the reward functions that can be learned [2]. In the limited-data regime, this true even when Rg belongs to an expressive function class (e.g., neural network). Motivated by recovering reward function Rg that incorpo- rates human-like priors, we are interested in explicitly constructing human-meaninggh features ø. In this sense, we consider a feature set to be under- under-specifedd if it does not explicitly represent all the salient features relevant to the task, and consider a reward function to be under-specified if it uses an inder-specified feature set as its basis.
4 Adaptive Language-Guided Abstraction from Contrastive Explanations
At a high level, ALGAE aims identify the most likely set of preference weights 0* and preference featurizations o" given the human demonstrations under p(ó. A Ti...., TN). The idealized joint optimization given demonstration r thus takes the following form:
0" 60 arg max p(0, 7)
0.6
We maximize this objective tractably by iteratively optimizing 0 and 6. Given an initialization of
the feature vector Φº, ALGAE decomposes the learning problem into two subtasks: 1) performing feature specification by expanding each feature set ø² into &++1 by querying for missing features in language, then 2) performing reward validation by finding an optimal reward weight @h for each @* and comparing its predictions to ground -ruth trajectories.
4.1 Identifying missing features
In feature specification, we use an LM to identify missing features salient to the user's reward given demonstration and the robot's 's current best estimate of the (under-speciffed) reward function Rgg. Once the missing feature has been identified, we query the LM for code to compute that missing feature and append the resulting feature function di to the set of known features d² obtain gx+1,
Operationally, by fixing 0 = 0², Eq. (4) becomes:
+++ arg max p(Ø, 0* 1 = arg max p(ò) 0h, T) p(0* ( = arg max maxp(e (0,T7) (5)
where p( T)) may be estimated using an LM as described below.
Querying for missing feature. The key insight our approach is that language models have strong common-sense priors over salient task features (e.g., ideal watering height above plant) which can leverage to approximate the probability p(e 0+, in Eq. (5). Concretely, approximate the optimization Eq. (5) by first computing best estimated trajectory for the current re- ward TR₂, using Eq. (1) and pair it with the user demonstrated trajectory т. We then prompt the LM with the contrastive pair of trajectories (full state information concatenated over all timesteps) along with a high level description of the task (e.g. the user wishes water the plant) and environment context (a description of the state of the environment) and query the LM for the likeliest feature Φ; that explains this difference Intuitively, TR,s acts evidence for 0* that is easier for the LM to
If there are multiple missing features, algorithm will identify them future iterations.

---

Algorithm 1: ALGAE
Input: Demonstrations = {T; ) ii
Init: Features 60 [distance to goal]. confidence threshold
while true do
0 arg maxg ((D 0, Φ+) // Optimize reward as Eq. (3).
FR = arg m Rge (T) // Optimize trajectory according to this reward as in Eq. (1).
TH ~ D Sample one human demonstration to contrast it against.
if | (TRxe) - 0" (TH) e// optimized trajectory explains the demo then
break
Φ LM(TR I // Query LM for missing feature that explains the contrast.
bl o% U
interpret than reward weights, resulting sample from the LM's prior on p(oe Thyn T). Thus, approximate solution o++1 concatenates the LM sampled d; with the previous vector o².
Querying for feature code. Now that we have identified a missing feature Φi, we query the LM to directly generate the feature definition code that can be used to compute a feature value from raw state data over the given demonstrations. The LM given raw trajectories and natural language feature description of d;, generated from the previous stage, and is asked generate its code o, We append d; and its respective function code to the existing list of features &++1 t 6² U Ué.
4.2 Learning good reward functions
In the second stage, reward validation, train and validate a reward function using the updated set of features @++1 as the basis function for the new reward, learning new parameters 0++1 Because there may be additional missing features we have yet to identify, we now leverage existing user demonstrations to validate whether reward learning is complete. This iterative update motivated by the desire to not over-parumeterize the reward by specifying non-salient features.
+1 in Eq. (4) obtain: Learning new reward. Given our updated feature set dx+11, we fix x
p ()p((1(*** ")((*+1))))) arg max p(T d+ 0) (6)
arg max p 0 = arg max
p(T)
assuming uniform p(0 1 This equation directly corresponds to the MaxEnt IRL optimization
procedure in Section 3 using r the feature vector. We now learn updated reward weights + from the user demonstrations via Eq. (3), then optimize for an updated trajectory TR via Eq. (1).
Validating the new reward. To determine whether we have recovered all salient features, we leverage existing user demonstrations for validation. In real-world deployment scenario, there many reasonable ways to implement comparison between trajectories, with the ideal being human user giving feedback as to whether the robot is executing the desired behavior. However, in our simulated experiments, we approximate this judgement by measuring L2 distance between these trajectories in the ground truth feature space o² If this is within threshold e, the learned reward function is sufficient for explaining the desired behavior and the algorithm terminates; otherwise, we iterate through the feature specification stage again. Pseudocode for the full pipeline can be seen in Algorithm
Simulated Experiments
We evaluate ALGAE both simulated and real-world robot environments. To highlight the flexibil- ity of ALGAE across diverse feature types and high high-dimensional control, our domains include 2D maze navigation tasks and a 7DoF JACO tabletop manipulation task, along with real-world mobile manipulation tasks with Spot in Section 6. We use GPT-4o [24] to identify salient missing features in all experiments.
5

---

2D Navigation
JACO
c
Getaateringoo
ThruwAwayRog
WaterPlant
Figure 2: We evaluate on both simulated and real-world domains with variety of missing features. A: 2D maze navigation, where the robot must navigate to a goal while interacting with other objects. B: 7DoF JACO manipulation, where the JACO must manipulate held coffee mug while re- specting features like end effector orientation. C: Spot mobile manipulation tasks, where the robot must complete tasks like WaterPlant while respecting the height of pot.
5.1 Experiment Setup
Domain 1: 2D Navigation. Inspired by the AI Safety Gridworlds [25], where under-specified fea- tures can implicitly lead to reward hacking behavior [26], we evaluate ALGAE on the following 2D maze navigation environments: GridRobot, Lavaland, and Island (Fig. 2). Trajectories are sequences of 9 states beginning with the start and ending at the goal. The 18-dimensional input consists of the ry coordinate positions of the robot in the grid at each timestep. The action space is the four coor- dinate directions. Each scenario is 5-by-5 5 2D grid consisting of an agent, a goal the agent must navigate to, and possible objects the agent must interact with. In GridRobot, the agent must avoid the (1) Stove; in Lavaland, the agent must avoid (2) Lava; in Island, the agent needs to drink (move (o) (3) Water. To ensure there is the possibility of over-specifying features, we also randomly place non non-relevant objects that do not impact the true reward in the environment (e.g., Ball).
state and action space, we also evaluate 7DoF JACO robot arm tasked with manipulating coffee cup tabletop in PyBullet simulator (see Fig. 2.) Each scenario is initialized with starting and goal pose location, a laptop placed on the tabletop, and a human by the table. The robot must manipulate the coffee cup is carrying to a specific goal location while respecting user preferences for motion. Trajectories are length 21, and each state has 97 dimensions: the zy² positions of all robot joints and angles, and zyz object positions along with rotation matrices. We introduce four scenarios based on the relevant human preference: (1) Laptop: zy-plane distance of the EE to the laptop, ensure the cup does not pass over the laptop, (2) Table: 2-distance of the EE to the table, to keep the mug of height close to the table, (3) Orientation: EE orientation relative to upright, to ensure the cup does not spill, and (4) Human: ry-disttace of the EE to the human, to ensure the cup does not collide with the human.
Domain 2: 7DoF Tabletop Manipulation. To assess higher-dimension
Test Scenarios. To study whether our learned reward functions generalize across different scenar- ios related to spurious environment features, we construct two possible distribution shifts for each demonstrated scenario: (1) different trajectory goals (Goal) (c.g., the robot must move to a different goal location) and (2) different feature object locations (Object) (e.g., the laptop changes location in the scene). These are intended to test whether the learned reward captures true salient environ- ment features rather than spurious feature correlations. We report the normalized reward of the optimized trajectory (against the reward of the ground truth best and worst trajectories) on set of five randomly sampled test scenarios across three seeds.
Comparisons. A straightforward way to learn rewards from demonstrations is to learn directly from the state space. We refer to this baseline as (1) Full-State. We would expect such a baseline to fail to generalize to novel environments since the learned reward may be prone to overfitting spurious state correlations in the demonstrations, rather than true salient features in the environment. However, access to a language prior was given without contrastive demonstrations (such that there is no demonstrated trajectory of the under-specified feature), we could query for all hypothesized features at once as (2) LM-Feature. This baseline is intended to isolate the impact of the language model proposing features given only the task description and environment context, and can be seen as

---

GridRobot
Lavaland
Island
JACO
Stove
Lava
Water
Laptop
Tuble
Orientation
Human
Goal ALGAE
1.00 (0.00)
1.00 (0.00) 0.99 (0.01) 0.90 (0.06) 0.68 (0.12)
0.73 (0.08)
0.66 (0.12)
Full-State
0.49 (0.06)
0.60 (0.06) 0.52 (0.06) 0.40 (0.05) 0.37 (0.07)
0.35 (0.03)
0.39 (0.01)
LM-Feat
0.85 (0.02)
0.86 (0.05) 0.86 (0.08) 0.53 (0.05)
0.51 (0.16)
0.58 (0.10)
0.54 (0.04)
LM-Rew
0.81 (0.04)
0.83 (0.10) 0.72 (0.09) 0.63 (0.08) 0.48 (0.17)
0.58 (0.07)
0.67 (0.01)
Random
0.58 (0.07)
0.62 (0.04) 0.57 (0.03) 0.37 (0.03)
0.39 (0.08)
0.36 (0.03)
0.38 (0.08)
Obj. ALGAE
1.00 (0.00) 1.00 (0.00) 1.00 (0.00) 0.93 (0.06) 0.65 (0.12) 0.67 (0.03)
0.53 (0.03)
Full-State
0.62 (0.01)
0.55 (0.06) 0.44 (0.05) 0.38 (0.07) 0.31 (0.03)
0.31 (0.06)
0.44 (0.07)
LM-Feat
0.77 (0.04)
0.81 (0.05) 0.90 (0.05) 0.54 (0.08)
0.45 (0.
0.59 (0.07)
0.50 (0.01)
LM-Rew
0.76 (0.08)
0.81 (0.10) 0.79 (0.07)
0.61 (0.06) 0.56 (0.15)
0.58 (0.09)
0.53 (0.02)
Random
0.54 (0.02) 0.61 (0.06) 0.53 (0.02) 0.38 (0.02)
0.34 (0.09)
0.36 (0.02)
0.38 (0.08)
Table Normalized ground truth reward of the optimized trajectories produced by different methods across simulated test environments and feature scenarios. Goal represents test scenarios where the goal locations have changed from training and Object represents test scenarios where the relevant feature has changed location from training. We report standard error across three seeds.
a prompting-only baseline. For an additional prompting-only baseline, we also evaluate (3) LM- Reward, a baseline where the LM also proposes the reward weights in addition to the features. Inspired by work that proposes to construct rewards directly from language [27, 28, 29], this baseline is intended to test the zero-shot reward specification capability of LMs. Finally, to benchmark these values, we report a (4) Random reward, which randomly samples a trajectory for each test scenario.
5.2 Results and Analysis
Single feature recovery. We first assess the ability of ALGAE to recover single missing fea- ture. Table shows results across test environments and their feature scenarios. As shown, ALGAE outperforms comparisons across all features. In simpler 2D environments such as GridRobot and Lavaland, ALGAE achieves close to perfect feature recovery and generalization on unseen test sce- narios. While the two prompting-onfe baselines (LM-Feetture and LM-Reward) perform well above Random, upon inspection of the LM rewards, they tend to produce many irrelevant features which result in over-purameterized reward functions. In the more complex JACO scenarios, ALGAE con- sistently outperforms baselines. Its performance is close to the prompting-only baselines on the Orientation task, which is unsurprising as there is no explicit penalty for over-parameterizing the reward (i.c., the robot can keep the cup upright while optimizing for additional features).
Iterative feature recovery. We next study the case where more than one feature is under-specified in scenario, and ALGAE must recover them iteratively. In these experiments, the training scenario is instantiated with two missing features ( (Water and Dinosaur for Island) and (Laptop and Orientation for JACO). As seen in Fig. 3, ALGAE iteratively improves reward given multiple missing features.
6 Real-World Experiments
To assess the feasibility of our approach on robotic hardware operating in cluttered human environ- ments, we now evaluate ALGAE's performance on mobile manipulation tasks with a Spot robot.
Domain 3: Spot Mobile Manipulation. Spot is legged robot equipped with manipulator (arm with gripper) and six RGB-D cameras (one in gripper, two in front, one on each side, one in back). Each scenario is initialized with a starting and goal robot pose location, along with the following objects: watering can, plans pot, apple, towel, and textbook. Trajectories are length 5 and each state consists of 10 dimensions: the zyz positions of the robot and its hand, as well as rotation of the gripper. We introduce the following three scenarios and their relevant reward preferences: (I) GerWateringCan disttiit of the robot to the watering can), to ensure the robot is positioned to execute successful grip, (2) WaterPlant (z (z-itanance of the robot hand to the demonstrated watering
7

---

(Curs)
LM Feature
Island
JACO
Spu
LM Reward
0.5
0.3
0.7
0.6
Rovand
0.5
0.4
0.3
0.2
0.1
0.1
0.0
0.0
0. GaWrastggaa
Weadfaa
Thoow-rrs Rag
Becrtion
Scanann
Figure 3: Normalized reward across multiple iterations simulated domains. ALGAE (pink) improves across each iteration, continuously finding under-specified features and updating its reward estimate. In contrast, prompting-one baselines such as LM-Feature (gray) and LM-Reward (black) do not iteratively improve themselves after instantiation. B: Normalized reward on Spot domains. Real-world clutter introduced the scene leads to over-paramen wver-parameterized LM-only rewards, increasing the gap between ALGAE and baselines. Error bars depict standard error across three seeds.
height above pot), ensure the hand adjusting to the pot size, and (3) ThrowAwayRag (ry-distance of the hand to the textbook), to ensure the robot does not drip water from the carried rag.
Data collection. For each scenario, a human teleoperates the robot from the starting to goal pose while satisfying the hidden preference (e.g. raise the robot hand to particular height above the pot before performing a watering motion), and generates three demonstrations per scenario. We also collect all detected object identities and positions via image segmentation from the robot cameras [30, 31] and captioning [32] as environment context for the demonstrations.
Test scenarios and comparisons. We evaluate on the following test scenarios: for GetWatering we vary the location of the watering can before executing a grip. for WaterPlant, we vary the height of the plant pot before executing a pour, and for ThrowAwayRag, we vary the location of textbook on the table the robot hand must avoid while carrying the wet rag. To ensure the robot does not execute a drastically wrong action given an incorrectly learned reward, we constrain the sample space of the robot pose at test time around safe zones. We report the normalized reward of the optimized trajectory on a set of three test scenarios across three seeds. We report comparisons with the previous best performing method (LM-Reward) along with a Random benchmark.
Results and Analysis. ALGAE outperforms baselines on real-world scenarios (Fig. 3). This gap actually increases relative to simulated environments due the worl-world clutter introduced in the scene. Upon visual inspection of the LM-Reward features, indeed see unnecessary features (e.g- relative position of the robot all objects in the scene, not just the salient ones) included. This underscores the usefulness of ALGAE's s interactive, iterative approach to feature construction.
7 Discussion and Limitations
ALGAE is an iterative reward learning framework that alternates between specifying missing fea- tures with LM and weighting them into a reward via IRL. One limitation is that we used the ground truth feature vector to judge task completion on the contrasting pair of trajectories; future work might evaluate a true human-in-the-loog scenario, and study whether L.M-informed priors have additional benefits beyond the performance improvements evaluated here. Moreover, we focus on reward functions that are linear in the feature set; in the future, for nonlinear rewards. Finally, ALGAE is only effective for cases where the LM prior agrees with the desired behavior which may not always be true in practice.
imagine extending our method

---

Acknowledgements
We thank the MIT Learning and Intelligent Systems Group for providing robotic hardware resources used this project. We thank members the Language and Intelligence and Interactive Robotics Groups for helpful feedback and discussions. Andi Peng supported by the NSF Graduate Re- search Fellowship and Open Philanthropy. Belinda Li is supported by National Defense Science and Engineering Graduate Fellowships. Ilia Sucholutsky is supported by NSERC Fellowship (567554-2022). This work was partially supported by the National Science Foundation under grant IIS-2212310.
References
[1] D. Hadfield-Menell, Milli, P. Abbeel, Russell, and A. Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
[2] A. Bobu, M. Wiggert, C. Tomlin, and A. D. Dragan. Feature expansive reward learning: Rethinking human input. In Proceedings of the 202/ ACM/IEE International Conference on Hoiman-Robor Interaction, pages 216-224, 2021.
[3] A. Bobu, M. Wiggert, C. Tomlin, and A. D. Dragan. Inducing structure in reward learning by learning features. The International Journal of Robotics Reseurch, 41(5):497-518, 2022.
[4] A. Y. Ng and S. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2000.
[5] C. Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference machine learning, pages 49-58. PMLR, 2016.
[6] C. Finn, Christiano, Abbeel, and Levine. A connection between generative adver- sarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852. 2016.
[7] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[8] Levine, Popovic, and V. Koltun. Feature construction for inverse reinforcement learning. Advances in neural information processing systems, 23, 2010.
[9] J. E. Fan, R. D. Hawkins, M. Wu, and N. D. Goodman. Pragmatic inference and visual ab- straction enable contextual flexibility during visual communication. Computational Brain & Behavior, 3:86-101, 2020.
[10] M. K. Ho, D. Abel, T. L. Griffiths, and M. L. Littman. The value of abstraction. Current Opinion in Behavioral Sciences, 2019.
[11] P. Abbeel and A. Y. Ng. Apprenticeship Learning via Inverse Reinforcement Learning. In Proceedings of the International Conference on Machine learning. 2004.
[12] B. D. Ziebart, A. L.. Maas, A. Bagnell, A. K. Dey, at. Maximum entropy inverse reinforce- ment learning. In Aaai, volume 8, pages 1438-1438. Chicago, II., USA, 2008
[13] T. Sumers, R. Hawkins, M. K. Ho. т. Griffiths, and D. Hadfield-Menell. How to talk so ai will learn: Instructions, descriptions, and autonomy. Advances in Neural Information Processing Systems, 35:34762-34775. 2022.
[14] A. Bobu, A. Peng, P. Agrawal, J. Shah, and A. D. Dragan. Aligning robot and human repre- sentations. arXiv preprint 27X.v2230..01928. 2023.

---

[15] A. Bajcsy, D. P. Losey. M. K. O'Malley. and A. D. Dragan. Learning from physical human corrections, one feature at a time. In Proceedings of the 2018 ACM/IEEE International Con- ference on Human-Robee Interaction, pages 141-149, 2018.
[16] S. A. Bowyer, B. L.. Davies, and F. R. Baena. Active constraints/virtual fixtures: A survey. IEEE Transactions Robotics, 30(1):138-13) 2013.
[17] D. Rakita, B. Mutlu, and M. Gleicher. An autonomous dynamic camera method for effective remote telcoperation. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Roii Robot Interaction, pages 325-333, 2018.
[18] J. Spencer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang. P. Ramadge, and S. Srinivasa. Learning from interventions: Human-robot interaction as both explicit and implicit feedback. In 16th Robotics: Science and Systems, RSS 2020. MIT Press Journals, 2020.
[19] Sharma, B. Sundaralingam, V. Blukis, C. Paxton, T. Hermans, A. Torralba, An- dreas, and D. Fox. Correcting robot plans with natural language feedback. arXiv preprint 200452200.005186 2022.
[20] A. Peng, I. Sucholutsky, B. Li, T. Sumers, T. Griffiths, J. Andreas, and J. Shah. Learning with language-guider state abstractions, In International Conference on Learning Representations, 2024.
[21] A. Peng, A. Bobu, B. Z. Li, T. R. Sumers, I. Sucholutsky, N. Kumar, T. L. Griffiths, and J. A. Shah. Preference-conditione hanuaag--gidded abstraction. In Proceedings of the 2024 ACM/IEEE International Conference on Iluman-Robol Interaction, pages 572-581, 2024.
[22] B. Li, W. Chen, P. Sharma, and Andreas. Lampp: language models as probabilistic priors for perception and action. arXiv preprini arXiv:2302.02881, 2023.
[23] M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[24] Achiam, Adler, S. Agarwal. L. Ahmad, Akkaya, F. L. Aleman, D. Almeida,
J. Altenschmidt, S. Altman, S. Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[25] J. Leike, M. Martic, V. Krakovna, P. A. Ortega, T. Everitt, A. Lefraneq. L. Orseau, and S. Legg. Ai safety gridworlds. arXiv preprint urXiv:/711.. 171110883,.. 2017.
[26] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete prob- lems in ai safety. arXiv preprint arXiv: 1606.06555 2016.
[27] Rocamonde, Montesinos, E. Nava, E. Perez, and D. Lindner. Vision-language models are zero-shot reward models for reinforcement learning. arXiv preprint arXiv:2310. 12921, 2023.
[28] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar, Eureka: Human-lever reward design via coding large language models. arXiv preprint arXiv:2310. 12931, 2023.
[29] M. Kwon, M. Xie, K. Bullard, and D. Sadigh. Reward design with language models, arXiv preprint arXiv:2303.00000 2023.
[30] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.- Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
[31] N. Kumar, T. Silver, W. McClinton, L. Zhao, S. Proulx, T. Lozano-Pérez, L. Kaelbling, and
J. Barry. Practice makes perfect: Planning lear skill parameter policies. arXiv preprint 1522 2024.
10

---

[32] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. Detecting twenty-thousany classes using image-level supervision. European Conference on Computer Vision pages 350-368, 2022.

---

A Full Prompts
We provide the prompts used for both the feature query in all domains.
2D Navigation.
An agent is in 2D gridworld environment of size 5x5. (0,0) is the top left corner and (4,4) is the bottom right corner (note this is not typical coordinate grid).
The environment config has the following attributes: X (int): width of the environment (int): height of the environment starts (List(Tuple(int]]) starting coordinate of the agent goals.pos (List[Tuple(intDD): coordinate position of the goal goals color (str): color of the goal goals type (str): object type of the goal objects(1-3)-pos (List(TupleCintI]) objects in the environment. This will be empty if there is no object in that slot. objects(133).colo (str): color of the corresponding object at the position objects(1-3). This will be empty if there is no object in that slot. objects(133)- _type (str): object type of the corresponding object at the position o1-ects(1-3). This will be empty if there is no object in that slot.
position(s) of other
This is environment config (disregard the extra attributes that are not specified above): {env}
The following is a demonstration the agent took in this first environment (each trajectory contains x,y coordinates for the agent, timesteps total). Keep track of where the agent's positions relative to the other objects in the environment. For example, [0,0 0, 0,2 0,3] means the agent started at [0,0] and moved to [0,3]. If there is an object at [0,3], the agent moved towards that object.
demonstration {demo}
The following is (incorrect) demonstration that the agent took in this environment. The trajectory is the same format the first demonstration. incorrect trajectory = [best_traj}
Describe the demonstration and trajectory in the context of the environment they are in, then use this give the specific feature in that environment that explains what the demonstration did better. Reason step by step and think about why the agent might be moving in the way that it is. If there are multiple of the same object, please give the average distance to all objects of that type. Please specify a feature (only one feature e.g. if there are multiple possible explanations choose the most likely object) that we can write code to compute from the position of the agent to other objects the scene.
This is the current list of features that already know: {self. Iself.feature feature .nmeee...m...... Please give different feature in your answer (e.g. if distance to goal feature is already in the list, do not give distance to goal again). Please give your confidence for your answer between 0 and 1. Ground your answer in visible objects in the scene, and give an answer (only specific answer (e.g. distance to book objects)) that we can compute relative to the position the agent.
JACO.
A 7-DoF Jaco robot arm is carrying a coffee cup on a tabletop environment. The arm is carrying fragile ceramic coffee cup that is at risk of slipping. The arm is controlled via a Pybullet simulator.
The environment config has the following attributes: object centers: ['HUMAN_CENTER': [x,y,2], 'LAPTOP_CENTER'. [x,y,z]] # coordinates of the human
12

---

objects and their xyz positions in the and laptop objects in the environment tabletop: z=0 the z-coordinate plane of the tabletop
This is an environment config (disregard the extra attributes that are not specified above): (env}
The following is correct demonstration trajectory the robot took in this first environment. Each trajectory is 21x97 matrix representing the 21 timesteps the robot took over the course of the demonstration. Each state in the timestep is 97-dimensional vector representing the x,y, positions of all robot joints and objects, and their rotation matrices. This is the code that generates each state. Note the last six values represent fixed xyz positions of the human and laptop:
Get relevant objects the environment posH, p. gettseeeesiiiooooooroaaaaaoioiie posL, p. o.gebassessstttooiooieeeettttopt
object_coords = np. arroy( posL])
Get xyz coords and orientations. 'waypt' is single waypoint in the trajectory move_ rooe joint_poses=waypt)
coords robot_ccorrds(oojjcci1 orientations (ebbll..rrienttatttooso(eeeedd""
return np. hes.ap (mp (wwp..cooonttttttttaaatttt orientations. flatten()
coords. flatten(), object_coords flatten))) (-1,))
Reason step by step, and keep track of where the robot's movements relative to important features in the scene. correct trajectory = {demo}
The following is an incorrect trajectory that the robot took in this environment.
The trajectory the same format as the first demonstration. incorrect trajectory (best traj]
Describe this trajectory in the context of the environment it is in, then use this to give the specific feature in that environment that explains why this trajectory was incorrect relative to the demonstration, which was correct. Reason step by step and think about how the robot's movements are related to the objects in the environment, as well as how aspects of the robot's position and orientations may be impacting the desired task (bringing coffee). Please specify feature (only one feature e.g. if there are multiple possible explanations choose the most likely object) that we can write code to compute from the position of the robot to other objects in the scene. This is the current list of features that we already know: (self. feature.names). Please give a different feature in your answer (e.g. if distance to goal feature is already in the list, do not give distance to goal again). Please give your confidence for your answer between and Ground your answer visible objects in the scene, and give an answer (only specific answer (e.g. distance to book)) that we can compute from the position of the robot. Spot.
This an environment configuration of room.
Environment:
[environments
Goal: (goal)
The following is successful trajectory (over 5 timesteps) the robot took in
this environment (each trajectory contains xyz positions of the robot, along with
13

---

the xyz positions and rotation quarternion of the robot's hand). Keep track of where the robot's positions and hand are relative to the other objects in the environment.
Successful trajectory: {demo}
The following is an unsuccessful trajectory (over 5 timesteps) the robot took in this environment.
Failed trajectory: {failed traj]
Describe the key feature that distinguishes the successful trajectory from the failed trajectory. Make sure the feature is semantically sensible for the goal. Reason step by step. Please specify a feature (only one feature e.g. if there are multiple possible explanations choose the most likely one). Make sure the feature is computable from the state features the scene (and no other information): plant_XYZ, robot_XYZ, robot.arm.XYZ, robot.arm.rotation The feature can be relative positions to objects, relative positions to positions around objects (e.g. force field around the object), relative positions to absolute points in space, relative rotations, etc. The feature can even be non-linear. However, make sure the feature is continuous and not discrete. Please describe the feature in detail.
14


Reviews:
[{"review_id": "Q5VdkhEXqj", "reviewer": "Reviewer_sFkR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors introduce a method called ALGAE, which aims to improve robot learning by leveraging human-like priors encoded in natural language. The method addresses the challenge of learning robust reward functions from a limited set of human demonstrations by identifying and incorporating relevant features into the reward learning process.", "summary_of_recommendation": "I recommend an accept", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "How well does ALGAE generalize to entirely new tasks (or environments) that were not seen during the training phase?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JpM9k1dWDr", "reviewer": "Reviewer_vt5Y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes to leverage language models to learn missing features for inverse reinforcement learning. The reward learning problem is decoupled into two sub problems: 1) feature learning with language model reasoning about one missing feature at a time; and 2) reward learning using demonstrations and learned features. Experiments in both sim and real domains show the effectiveness of the proposed method in comparison with baseline and ablated methods.", "summary_of_recommendation": "The method is novel and interesting but the experiments are toy and more baselines should be included.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "- [see comments on weakness above]\n- Can you show an example input trajectory to the LLM and its response? How do you make sure if LLM can understand the trajectory in the form of numbers/matrices? Does the output of the LLM end up writing code to parse the trajectory information or it only uses text? \n- On a similar note to the question above, why not use visual state as input for LLM? For the LM-feature ablation, I can imagine if the LLM does not see the layout of the environment, it may want to track each and every object for obstacle avoidance; but if it could see the image and understand what is near the robot and target object, it might filter unrelated features\n- How does the proposed method compare with zero-shot using language models to plan robot behavior? If it is able to understand trajectories in numbers, seems like it can also be prompted to output the trajectory directly. Similarity, is VoxPoser [1] a fair baseline here?\n\n[1] https://voxposer.github.io/", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p3q4d4SyUf", "reviewer": "Reviewer_cr8T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The main contribution is a method for inverse RL that uses a language model to determine reward features that are relevant for the task.", "summary_of_recommendation": "Overall, the work is quite nice. The iterative construction and evaluation of the reward function is a good contribution. Would love to see some more real robot experiments with a bit less state tuning too.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- The reward function building adds new features at every iteration. If a bad feature is proposed, how could we determine if we should not include it? Or is that implicit in the feature weighting.\n- A small detail: was the feature weighting a simple linear combination of weights, or something more complex in your experiments?\n- The biggest limitation is most likely the simple state representation. Could you comment on how to remove some of the handcrafting of the state representation when doing reward function creation?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper's strengths include:
The ALGAE method, which leverages language models for feature learning in inverse reinforcement learning (IRL), offers a novel approach by iteratively learning and integrating features relevant to the reward function.
The separation of feature learning and reward learning into distinct subproblems helps in focusing on each component effectively, combining the strengths of data-driven and model-based approaches.
The paper provides empirical evidence of the method's effectiveness across both simulated and real-world environments, demonstrating clear advantages over baseline methods and ablated versions.
The writing is clear and well-organized, making the method and its results easy to understand. The iterative feature addition and evaluation process is a valuable contribution to the field.

The paper's weaknesses include:
The experiments are somewhat engineered with a focus on a small number of features, which may not fully showcase the method's robustness in more complex scenarios. More diverse baselines and experiments are needed.
The experiments utilize highly handcrafted state representations, which could bias the results. More exploration with less engineered state representations or full image inputs would be beneficial.
The paper does not thoroughly address limitations or failure cases, such as generalization to new tasks or environments and reliance on known ground truth feature vectors.
The design of prompts for language models involves significant domain knowledge and tuning, which might not be practical for all applications.

Post-rebuttal meta-review:
The paper introduces ALGAE, a method designed to improve robot learning by leveraging language models to identify and incorporate relevant features into the reward learning process, particularly in scenarios with limited human demonstrations.

ALGAE uses language models to iteratively identify human-meaningful features, which are then used to guide the reward learning process. The method alternates between feature identification and reward optimization, allowing it to refine its understanding and adapt to missing features without human intervention. ALGAE demonstrates its ability to generalize learned reward functions across various simulated and real-world robotic environments.

The integration of language models into feature learning for inverse reinforcement learning (IRL) is a novel approach. The paper is well-written, clearly explaining the method and supporting its claims with solid empirical evidence in both simulation and real-world experiments. The method shows robustness in extracting meaningful features and generalizing across different tasks.

The experiments appear heavily engineered, with a focus on a small number of features, which may not fully showcase the method's robustness in more complex scenarios. The reliance on highly handcrafted state representations could bias the results, limiting the exploration of more diverse or less engineered inputs. The paper lacks a deep comparative analysis with state-of-the-art methods, which could better demonstrate its distinctiveness and impact.

Final Recommendation: Borderline Accept-- The paper presents a novel approach with promising results, but it has some limitations, such as engineered experiments and a lack of comprehensive baselines, which may affect its broader applicability. While it is a valuable contribution, further work could strengthen its claims.
Decision: Accept