Learning with Language-Guided State Abstractions
Authors: Andi Peng
Ilia Sucholutsky
Belinda Z. Li
Theodore Sumers
Thomas L. Griffiths
Jacob Andreas
Julie Shah
Venue: ICLR 2024 poster (2024)
Keywords: human-ai interaction
state abstractions
Primary area: applications to robotics, autonomy, planning
TL;DR: We use language models to help construct human-aligned state abstractions for robot learning.

Abstract: We describe a framework for using natural language to design state abstractions for imitation learning.
Generalizable policy learning in high-dimensional observation spaces is facilitated by well-designed state representations, which can surface important features of an environment and hide irrelevant ones.
These state representations are typically manually specified, or derived from other labor-intensive labeling procedures.
Our method, LGA (\textit{language-guided abstraction}), uses a combination of natural language supervision and background knowledge from language models (LMs) to automatically build state representations tailored to unseen tasks.
In LGA, a user first provides a (possibly incomplete) description of a target task in natural language; next, a pre-trained LM translates this task description into a state abstraction function that masks out irrelevant features; finally, an imitation policy is trained using a small number of demonstrations and LGA-generated abstract states. 
Experiments on simulated robotic tasks show that LGA yields state abstractions similar to those designed by humans, but in a fraction of the time, and that these abstractions improve generalization and robustness in the presence of spurious correlations and ambiguous specifications.
We illustrate the utility of the learned abstractions on mobile manipulation tasks with a Spot robot.

Published as a conference paper at ICLR 2024
LEARNING WITH LANGUAGE-GUIDEE STATE ABSTRACTIONS
Andi Peng MIT
Ilia Sucholutsky *
Belinda Z. Li MIT
Theodore R. Sumers
Princeton
Princeton
Thomas L. Griffiths
Jacob Andreas
Julie A. Shah MIT
Princeton
MIT
ABSTRACT
We describe framework for using natural language to design state abstractions for imitation learning. Generalizable policy learning in high-dimensional observa- tion spaces is facilitated by well-designed state representations, which surface important features of an environment and hide irrelevant ones. These state repre- sentations are typically manually specified, or derived from other labor-intensive labeling procedures. Our method, LGA coanuuage-guider abstraction), uses a combination of natural language supervision and background knowledge from language models (LMs) to automatically build state representations tailored to unseen tasks. LGA, a user first provides a (possibly incomplete) description of target task in natural language; next, a pre-trained LM translates this task de- scription into a state abstraction function that masks out irrelevant features; finally, imitation policy is trained using small number of demonstrations and LGA- generated abstract states. Experiments on simulated robotic tasks show that LGA yields state abstractions similar to those designed by humans, but in fraction of the time, and that these abstractions improve generalization and robustness in the presence of spurious correlations and ambiguous specifications. We illustrate the utility of the learned abstractions on mobile manipulation tasks with a Spot robot.
INTRODUCTION
In unstructured environments with many objects, distractors. and possible goals, learning generaliz- able policies from scratch using small number of demonstrations is challenging (Bobu al. [024] Correia & Alexandre 2023). Consider the demonstration in Fig. [TA, which shows Spot robot ex- ecuting specific maneuver to perform a desired task. Which task is demonstrated here-grabbiig
an object, grabbing orange, or giving orange to user? Here, the observations do not provide enough evidence to meaningfully disentangle which features are relevant for downstream learning.
In humans, abstraction is essential for generalizable learning (McCarthy et al. 2021b). When learn- ing (and planning), humans reason over simplified representations of environment states that hide details and distinctions not needed for action prediction (Ho al. 2022). Useful abstractions task deeeedeen,,,,, and a growing body of evidence supports the concluss usion that humans flexibly and dynamically construct such representations learn new tasks (Ho et al., 2023 Huuey et al.) 2023). Importantly, this process of abstraction does not begin with a blank slate instead, experience, common-sense knowledge, and direct instruction provide rich sources of prior knowledge about which features matter for which tasks (Fan et al. 2020). In Fig. ITA learning that the demonstrated skill involves a fruit combined with prior knowledge about which objects are fruits makes clear that the object's identity (orange) is likely important. Meanwhile, the demonstration provides comple- mentary information (about the desired movement speed, goal placement, etc.) that is hard to specify in language and not encapsulated by the utterance bring me fruit alone. In other words, actions also contain valuable information that is necessary to complement the abstraction.
"Equal contribution.

---

Published as a conference paper at ICLR 2024
A
B
demonstration
state abstraction
policy m
language
f:8 C 8
abstract state
"Bring
state
segmentation
Ceater: (216, 38733
Twater 1235, 20113 CO/1LL/C133
caption
abstraction
Figure 1: A: Example demonstration in our environment, showing Spot picking up orange and bringing it to the Our approach, Language Guided Abstraction (I.GA), creates state abstraction with task-relevant features identified by an LMM The policy is learned directly over this abstracted state.
What would take to build autonomous agents that can leverage both demonstrations and back- ground knowledge to reason about tasks and representations? State abstraction has been a major topic of research from the very earliest days of research on sequential decision-matke
with signifi-
cant research devoted to both unsupervised representation learning (Coates & Ng 2012: Bobu et al., 2023 Higgins et al. 2017]] Lee et al. 202T) and human-aided design (Abel et al. 2018,, Cakmak & Thomaz, 2012 Bobu al. 2021 Abel et al. 2016). However, there are currently few tools for autonomously incorporating human prior knowledge for constructing state abstractions unseen,
unstructured environments.
In this paper, we propose to use natural language as source of information for constructing state ab- stractions. Our approach, Language-Guidel Abstraction (LGA) (Fig. (Fig.DB). begins by querying hu- mans for high-level task descriptions, then uses a pre-trained language model (LM) to translate these descriptions into task-relevant state abstractions. Importantly, LGA requires only natural language annotations for state features. Unlike most recent work applying LMs to sequential decision-making tasks, does not depend on pre-trained skills (Ahn et al. [2022] Huang al. 2022b), environment interaction (Du et al. 2023), large multitask datasets (Karamcheti et al. 2023 Shridhar et al. 2022). or even the ability describe behavior in language (Kwon al. 2023). complements traditional
supervised learning methods like behavior cloning (BC), without relying on additional assumptions about the data labeling process. Experiments comparing LGA to BC (and stronger variants of BC) abstractions improve sample efficiency and distributional robustness in
show that LGAA-generan both single- and multi-task settings. They match the performance of human-designed state abstrac- tions while requiring a fraction of the human effort.
In summary, we (1) introduce LGA, a method for using text descriptions and language models to build state abstractions for skill learning: (2) show that LGA produces state abstractions similar (and similarly effective) to those manually designed by human annotators with significantly less time; (3) show that LGA constructe state abstractions enable imitation learning that is more robust to the presence of observational covariate shift and ambiguous linguistic utterances; and (4) demonstrate LGA's utility in real world mobile manipulation tasks with Spot robot.
RELATED WORK
Language-Aides Reward Design. Large language models, trained on large amounts of text data, contain commonsense information about object properties, functions, and their saliencc/relevance to various tasks. Several works (Goyal al. 2019 Kwon et al. 20023] Sumers et al., 2021 Carta al.) 2022) leverage this to shape or learn reward models by training a policy to complete intermediate or higher-level tasks and asking the LM to annotate resulting behavior. Language feedback can then be incorporated as way to guide the policy during training (Mu et al. 2022 Du et al. 2023). However, just as with any reinforcement learning method, these works assume continuous environment access during the training process the policy gradually improves with the updated reward (Shinn et al.) 2023). In contrast, we instead leverage LMs to produce state abstractions for learning skills via imitation learning, which does not require continuous environment interaction access rewards.
2

---

Published as a conference paper at ICLR 2024
Language-Aided Planning. In parallel, great deal of work has leveraged LMs to output plans directly, L... generate primitives or high-level action sequences (Sharma et al. 2022 Ahn et al. 2022 Huang et al. 2022a b).= These approaches use priors embedded in LMs to produce better instruction following models, or in other words, better compose pre-existing skills to generate more complex behavior (Zeng et al. 2023] Li al., 2023; Ahn et al.) 2022] Wang al. 2023). Such methods can be thought of as performing action abstraction and require access to library of pre- trained (often text annotated) skills. In contrast, we use LMs perform state abstraction for learning better skills from scratch, and can therefore also sit upstream of any instruction following method.
State Abstractions in Learning. There substantial evidence to suggest much of the flexibility of human learning and planning can be attributed to information filtering of task-relevant features (McCarthy al. 2021a Huey et al. 2023 Ho et al. [022 Chater Vitányi 2003). This suggests flexibly creating task-conditioned abstractions is important for generalizable downstream learning, particularly low-data regimes (Mu et al., 2022 Boob et al. 2024). While state abstraction has been explored in sequential decision-
(Thomas & Barto 2011 existing methods often as- sume pre-defined hierarchies over the state-action space of the given environment or hand defined primitives (Abel et al. 2016 Diuk al., 2008). In this work, explore method to autonomously construct state abstractions from RGB observations with only a text specification of the task.
work Misra et al. (2018). which conditions on language and raw visual observations to create a binary goal mask, specifying goal location within an observation. However, our approach is much more flexible (our approach can learn any demonstrated behavior, not just goal navigation) and more personalizable (our approach allow humans to interact with the system, enabling them refine the representation based on individual preference).
Perhaps the closest comparison
PROBLEM STATEMENT
Preliminaries. We formulate our tasks Markov Decision Processes (MDPs) (Puterman 2014) defined by tuples (S, AS T, R) where S is the state space, A the action space, T : S x AxS the transition probability distribution, and R S + IR the reward function. policy is denoted as Te : S + A.
[0, 1
In behavioral cloning (BC), we assume access to a set of expert demonstrations Danin = {r = {(6), ab, of, af, s. a)) from which we "clone" a policy कe (Pomerleau, 1988) by minimizing:
LBC = ((s),at) Derain |I| (s) - al1
(1)
In goal-conditioned behavioral cloning (GCBC), policies additionally condition on goals & (Co- Reyes al., 2018). Motivated by the idea that natural language is flexible, intuitive interface for humans to communicate, we specify the goal through language instruction L E L, resulting in:
LGCBC = (f Derson IT (s), (*) afl
Above we have highlighted differences from BC in red.
A (GC)BC policy T(s), E') must generalize to novel commands El and contextualize them against potentially novel states s/. Due to difficulties with sampling representative data and expense of collecting a large number expert demonstrations, systematic biases may be present in both the training demonstrations (s), ...) for goal, or the commands e describing the goal. This can result in brittle policies which fail to generalize (Ross et al. 2011).
We study two sources of covariate shift: (1)
observations s', (2) out-of- distribution utterances t.. Training a policy π to be robust to both types requires a wealth of task- specific training data that can be expensive for humans to annotate and produce (Peng al. 2023). Below, we offer a more efficient way to do so by constructing a task-specific state abstraction.
3

---

Published as a conference paper at ICLR 2024
4 LANGUAGE-GUIDED ABSTRACTION (LGA)
Traditional (GC)BC forces the behavioral policy to learn a joint distribution over language, obser- vations, and actions-effecctiee requiring the robot develop both language and perceptual scene understanding simultaneously, and to ground language in the current observation. Our approach in- stead offloads contextual language understanding to a LM which identifies task-relevant features in the perceptual state. Unlike other recent methods that offload language understanding to LMs (Ahm al., (022). we do not rely on libraries of pre-trained uuguage-annotate skills, which may be insufficient when there are key behaviors indescribable in language. Instead, we introduce state abstraction function that takes the raw perceptual inputs and language-specifiet goal, and outputs a set of task-relevant features. Intuitively, LGA can be seen as a form of language-guide guided attention (Bahdanau et al.] 2015); conditions the robot's 's observations on language, removing the burden of language understanding from the policy. We describe general framework below, which will instantiate in Section
4.1 STATE ABSTRACTION FUNCTION
Formally, we define a state abstraction function f that produces task-relevant state representations: f:Sx L S S, consisting three steps:
Textualization (s - Φ). First, similar to other LM-assisted methods (Huang et al. 2022b; Ahn jet al., 2022) the raw perceptual input is transformed into text-based feature set Φ, representing set of features (described in natural language) that encapsulate the agent's 'ss full perceptual inputs This text representation may include common visual attributes of the observation like objects in the scene, which are extractable via segmentation models (Kirillov al. (023). In Figure TB, for example, textualization transforms observations to a feature set of object names and pixel locations.
Feature abstraction (< -> .). Given a feature set Φ we achieve abstraction by using an LM to select the subset of features from & relevant to the task é: (Ø, () - 3. In Figure 3. abstraction removes distractor objects from the feature set, in this case preserving only the target object (orange).
Instantiation (6- $). As a last step, we transform the abstracted feature set back into an (ab- stracted) perceptual input with only relevant features on display: Φ - This step is required for rendering the abstracted feature set in a form usable by a policy. In Figure [3 for example, this step transforms the abstracted feature set into an observation showing only the relevant object.
4.2
POLICY LEARNING
After receiving abstract state & from the state abstraction function, we learn policy mapping from the abstract state to actions, yielding Th S A T is trained to minimize the loss:
LLGA Beee Deramn I|| (f(st at 1
with the differences from GCBC (Eq, D highlighted red. The LGA policy TH never sees the language input instead, operates over the language-condee conditioned state representation, $. Conse- quently, it can exhibit non-trivial generalization capabilities relative to Eqs. and [ which given single goal seen at training and test, collapse to the same non-generalizable policy.
LGA offers several appealing properties relative to traditional (GC)BC. First, it mitigates spurious correlations during training because all goal information is highlighted semantic maps rather than raw pixels. Second, the LM can help resolve ambiguous goals present at test by converting the language utterance and test observation into an unambiguous abstract state - effective even LGA has only seen single unambiguous task at training. Therefore, using LMs confer valuable generalization capabilities at test-time, as LMs can "intercede" to determine only the contextually- appropriate task-relevan features for policy input.
In the general case, this could be implemented via segmentation (Kirillov et al. 2023) and captioner. ZLGA can also he used to augment the original observation, in which case policy is learned over the concatenated original and abstracted state-spaces, e.g., T.. S S A. This is more conservative approach, allowing the policy observe both the original and abstracted states.
4

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTAL SETUP
ENVIRONMENT AND FEATURE SPACE
We generate robotic control tasks from VIMA (Jang er al., 2022), vision-based manipulation environment. We study three types of tasks: pick-and-place of an ob- ject onto goal location, rotation of an object to a pre- specified degree, and sweep of an object while avoid- ing potential obstacle. The environment consists of large feature space (29 objects, e.g. bowl, and 81 col- ors/textures, e.g. wooden) of various semantic mean- ing. Observations are top-down RGB images and actions continuous pick-and-place or sweep poses each con- sisting of 2D coordinate and rotation expressed as quaternion. Success is determined by executing the cor- rect action on the target object within radius of the goal.
A
B
VIMA is an appealing domain for evaluation due to (1) its combinatorially large feature space for assessing pol- icy generalization and (2) the presence tasks that re- quire behaviors that are difficult to specify in language (e.g. velocity, rotation, etc.). The latter means we cannot deploy instruction- followiinnn method with library of lunguage-annotaan pre-trained skills.
evaluate on three task set- tings in VIMA. A: Pick-and-place B: Ro- tate. C: Sweep while avoid. Red bounding boxes depict task-relevant features which must be accounted for in the abstraction.
Figure 2:
The task-relevant feature set is specified via a JSON containing the set of possible task-relevan objects and colors. For liik-annd-llac/rotall this feature includes the target object; for sweep, this feature set parameterizes both the target object and the object to be avoided. Because the fea- ture space can be combinatorially large, constructing abstractions that preserve only task-relevant features of an ambiguously specified objective, e.g. something that holds water, is challenging.
5.2 LGA IMPLEMENTATION
Textualization. first extract a structured feature set ø by obtaining the ground truth state seg- mentation mask and object descriptions from the simulator.
Feature abstraction. We use two versions of LGA for defining relevant feature sets G. LGA uses GPT4 (OpenAl 2023) to specify a relevant feature set We give GPT4 a description of the VIMA environment (including list of all possible object types and colors), the goal utterance, and target object type or feature to evaluate (full prompt and additional details can be found in appendix (A2).. GPT4 provides binary response indicating whether the object type feature should be included in the abstraction. L.G-HIILL extends this by placing a human in the loop. The human is shown the proposed abstraction and can refine it by adding or removing features (see Section 5.4 for details).
Instantiation. As described in Section the abstract state representation consists of a distribution over object types and object colors, including both target objects and relevant obstacles. Given specified task-relevant object type and color, we use an image editor to identify all salient objects in the observation.
particular, the image editor produces a "goal mask". a pixel mask where the relevant objects are converted to ones and everything else is zeroed out. In our setting, the simulator provides the ground-truth scene segmentation which we use to produce the goal mask
policy proce- dure, described in Section 4.2 our environment. We assume access to ground truth demonstrations generated by oracle. LGA learns policy from scratch via imitation learning these demon- strations, using& 8. We implement a CNN architecture that processes abstract states into embeddings, which we then feed through MLP for action prediction. See Appendix A.3 for details.
EssractiionAugmeentt Policy Learning We instantiate the instraction-augment
3 use gpt-4 061 version GPT4.
'eere, produce simple binary pixel mask highlighting relevant objects. In general, however, it may be fruitful experiment with more general scalar cost maps, such highlighting that should he avoided.
5

---

Published as a conference paper at ICLR 2024
5.3 TRAINING AND TEST TASKS
Training Tasks. We are interested in studying how well different methods perform task-relevant feature selection, i.c specify e given single language description of the task. We evaluate the quality of abstractions by using them to construct a training distribution for each task (on which we perform imitation learning). We want methods which construct a minimal abstraction that contains features necessary for the task. If these abstractions do not contain all features necessary for completion of the task, then the policy will not succeed; if these abstractions are not minimal, then learning will be less data-efficient. Thus, we evaluate both data efficiency and task performance; good abstractions will result in the policy achieving higher success with fewer demonstrations.
We construct this distribution by placing a uniform probability over the set of task-relevant features specified in é. To create a task, we sample an object and texture from this distribution as the "target" object and place it randomly in the environment. For example, for something that can hold water, we sample an object from {pan, bowl, container) and sample a color from the universe of all colors. In the sweep task, we perform this same process, but for possible obstacles. We then generate random distractor object. Distractors, target objects, and obstacles are randomly placed in one of four discretized state locations. Last, for wock-and-llcce we place fixed goal object (pallet) in the state as the "place" goal. We generate corresponding training demonstrations via an oracle.
Test Tasks. In practice, we do not have access ground-truth distributions this specification must come from the end user. However, for our experiments, the first three authors defined "true" distributions from which we sampled starting states for each task. See details in Appendix A.2
5.4 COMPARISONS
We analyze our approach (LGA and LGA-HILL) against three baselines and two ablations.
Human (Baseline). The first baseline explores having users (instead of LMs) manually specify task- relevant features & in the feature abstraction stage, to isolate the effect of LM-aided specification. All other components remain the same in LGA. The features are fed to the same CNN architecture as for LGA and the policy is again learned over the generated state abstractions.
GCBC-DART (Baseline): Our second baseline is (goal-conditioned) behavior cloning, described in Eq. 2 Because imitation learning is notoriously brittle to distribution shift, we implement a stronger variant (DART) (Laskey al. 2017) by injecting Gaussian noise (swept from μ 0.0 to 0.2, = 0.5) into the collected demonstrations, sampling noisy demonstrations per true demonstration. We implement goal-conditioning by concatenating an LM embedding of the goal utterance with CNN embedding of the state and then learning a policy over the joint representation. We generate language embeddings using Sentence-BERT (Reimers & Gurevych 2019).
GCBC-SEG (Baseline): To study the impact of object segmentation, our third baseline implements noal-conditioning on the segmentation mask in addition to the observation. We independently pro- cess the two observations via CNNs, and concatenate the embeddings as a joint representation.
LGA-S (Ablation): We explore the effect of training with both the raw observation in addition to á to study the impact of discarding the original observation. The two observations are independently processed via CNNs, then concatenated a joint representation.
LGA-L (Ablation): We implement variant of abstraction where we condition only on the language embedding of the relevant features & rather than on the state abstraction 8.
For comparisons that require human data (LGA-HILL and Human), we conducted computer-based in-person user study to assess the ability of humans to specify task- relevant features ò in the feature abstraction step, both with and without LM. We recruited 18 participants (67% male, aged 22-44) from the greater MIT community. We paid each participant S30 for their time. Our study received institutional IRB approval and all user data was anonymized.
Human Data Collection.
We first introduce the user to the environment and full feature space Φ. To additionally help partici- pants, we provide both text and visual representation of features (the latter of which the LM does not see). We also walk the user through an example task specification. In the Human condition, we introduce task objectives (detailed in Section 5) sequentially to the user and ask for them to specify

---

Published as a conference paper at ICLR 2024
A
LGA
B
LGA-IEL Harso
0.8
175
GUH, (14R7
Sugges Rate
0.6
OCT SFC
150
125
0.4
75
Usee
OCBC
LGA
LGA.IILL
Human
Number
Figure 3: (Q1) A: Comparing task performance (averaged over all tasks) each method when controlling the number of training demonstrations. B: Comparing the amount of time (averaged over all tasks) that human users spent specifying lask-relevant features for each method. LGA outperforms baselines on task performance while significantly reducing user time spent compared to manual feature specification < 0.001).
by typing their selected features into a task design file (we provide the full feature list for easy access). In the LGA-HILL condition, show the same objectives, but with LM answers prefilled as the "raw" 3. To minimize ordering bias, we randomly assign half of our participants to begin with each condition. For both conditions, we measure the time each user spent specifying the feature set.
6 RESULTS
6.1 Q1: IMPROVING EASE AND POLICY PERFORMANCE TASK SPECIFICATION
We begin by evaluating both LGA and LGA-HILL against the baseline methods on nine scenarios from pick-and-place, two from rotate, and two from sweep. Each scenario demonstrates varied feature complexity, where the ground truth task distributions may be combinatorially large. Each method learns a single-task policy from scratch for each of the tasks.
Metrics. We report two metrics: task performance vs. # of demonstrations and user time, Task performance is assessed as the success rate of the policy on 20 sampled test states. We plot this with respect to the number of training demonstrations an agent sees. User specification time is measured as the time (in seconds) each user spent specifying the task-relevant feature set for each task.
Task Objectives. For pick-and-place, we construct nine scenarios with a wide range of target ob- jects: red heart, hearl, letter, tiger-colored object, letter from the word letter, consonant with warm color, vowel with multiple colors, something drink water out of, and something from a typi- cal kitchen. We chose these tasks to 1) test the ability of the LM in LGA to semantically reason over which target objects and features satisfy human properties, c.g., to drink water out of, and 2) explore how potentially laborious performing manual feature specification from large feature sets can be for human users. For rotate, we construct two scenarios: rotate block, rotate something can put food in, illustrate LGA's ability learn behaviors that are difficult specify in language (e.g., rotate an object 125 degrees), but easy to specify for task-relevance (e.g.. pan). For sweep, we construct two scenarios: sweep the block without touching the pan and sweep the block without touching the line to illustrate LGA's ability to identify task-relevant features that are both target objects and obstacles.
Results. We vary the number of demonstrations that each policy trained on from 10 50. We visualize the resulting policy performance in Figure (A). In Figure 3 (B), we visualize how much time users spent specifying the relk-relevant features for each method From these results, it is clear that the feature- specification methods (i.e. LGA, HIILL, and Human) are more sample- efficietf than baselines with consistently higher performance at each number of training demonstrations. Furthermore, LGA methods require significantly less user time than the Human baseline (p < 0.001 using a paired t-test for all tasks). We note that despite comparable task performance between LGA and LGA-HILL. the latter remains valuable instantiation for tasks that require human input, such as high-stakes piccision-mking scenarios or personalization to individual user preferences.
Summary. LGA requires significantly less human time than manually specifying features, but still leads to more cumple-efficieer performance compared to baselines.
'While we assign zero time spent specifying features for GCBC in our experiments, in practice, user time would be instead spent specifying initial state configurations for generating every demonstration, which only disadvantages naive GCBC more when compared to methods that specify features.
7

---

Published as a conference paper at ICLR 2024
A
Conurbar
B
Andkgaky
10
AGAG
87
62
Talk
Tal..
Figure 4: (Q2) A: Results on state covariate shifts. LGA variants that condition on state abstractions (with or without the original observation) outperform LGA-L (which conditions the language abstraction only) and GCBC+DART (which attempts to use noise injection to handle covariate shift). (Q3) Results on multi-task ambiguity. We observe the same trends, with LGA variants that condition on state abstractions able to resolve previously unseen linguistic commands.
6.2
Q2: IMPROVING POLICY ROBUSTNESS TO OBSERVATIONAL COVARIATE SHIFT
We have shown that LGA enables faster feature specification when compared to hand-engineering features from scratch, and also leads to more sample-efficient policies when compared to traditional imitation learning. We now ask how the different design choices taken in LGA handle state covariate shift compared to GCBC+DART, which is a stronger imitation learning baseline designed to help mitigate covariate shift by synthetically broadening the demonstrations data with injected noise.
Task Objectives. We evaluate on four previously defined scenarios in Q1: heart (pick-and brick-and-llan and-place), letter (pick-and-place), bowl (rotate), and block (sweep while avoid). For each task, now struct state covariate shifts designed to test policy robustness.
For all tasks, we define training distribution where the task-relevant features contain some subset of the feature list, e.g., red and blue. At test time, we evaluate by sampling tasks from a distribution where either 1) task-relevant textures change (e.g., objects are now pink) 2) a sampled distractor is added to the scene. These tasks are intended to evaluate LGA's flexibility in including (and excluding) appropriate task-relevant features relative imitation learning over raw observations.
Results. For each method, we train on 50 demonstrations from the train distribution then evaluate on 20 tasks from the test distribution. As shown in Fig. 4A policies trained with either of the LGA methods are more robust to state covariate shifts than those trained with GCBC+DART. Notably, we observe that LGA-L (our ablation where we provide the language abstraction only) underperforms the variants that operate over state abstractions. This makes sense considering observations (even if masked) contain information that directly grounds task-relevant information for the policy to oper- ate over into the environment. This confirms our hypothesis that providing non-ambiguous state abstraction that already precludes non-relevant task features from the state result in policies that are less brittle to spuriously correlated state features.
Summary. Policies trained with LGA state abstractions (with or without raw observations) are more robust observational shift than GCBC+DART.
6.3
Q3: IMPROVING MULTI-TASK POLICY ROBUSTNESS TO LINGUISTIC AMBIGUITY
The previous experiments have focused on evaluating LGA in single-task settings where each policy corresponds to a singular language specification. We now evaluate how LGA performs in the multi- task case where test time environments allow for multiple task specifications, and conditioning on language is necessary to resolve the ambiguity.
Task Objectives. We now define three new tasks: pick up fruit (pick-and-place), pul food in (rotate), and avoid utensil (sweep). For each task, we train a multi-task policy. We define our training distribution states that contain single object from each specification (e.g., tomato apple). At test time, we present the policy with states that now contain both objects. We condition the policy on both the linguistic utterances it has seen before: e.g., "Bring me tomato." or "Bring an apple", and as well as one it has not: e.g., "Bring me a fruit." We now evaluate LGA's zero-shot generalization performance on both seen and unseen linguistic goal utterances.
Results. For each method, we train on 50 demonstrations sampled from the train distribution for each task, then evaluate on 20 task instances from the test distribution. We visualize the averaged
8

---

Published as a conference paper at ICLR 2024
results in Figure These results demonstrate LGA's ability to flexibly construct state abstractions even if the language utterance was previously unseen.
Summary. Multi-task policies trained with LGA can better resolve task ambiguity compared to GCBC+DART as they can flexibly adapt new language specifications to the observation.
REAL-WORK WORLD ROBOTICS TASKS
We highlight LGA's real world applicability on Spot robot performing mobile manipulation tasks.
Robot Platform. Spot is mobile manipulation legged robot equipped with six RGB-D cameras (one in gripper, two in front, one each side, one in back), each producing observation of size 640. For our demonstrations, we only use observations taken from the robot's 's front camera.
Tasks and Data Collection. We collected demonstrations of human operator teleoperating the robot while performing two mobile manipulation tasks with various household objects: bring fruit and throw away object. The manipulation action space consists of the following three actions along with their parameters: (xy, grasp), (xy, move), (drop) while the navigation action space consists of a SE(3) group denoting navigation waypoints. For bring fruit, the robot is tasked with picking up fruit on a table, bringing it to a user at a specified location, and dropping it in their hand. For throw away drink, the robot is tasked with picking up drink from a table, moving dropping it into recycling bin. Both tasks include possible distractors like brushes and drills on the table. For each task, we collected single demonstration of the robot successfully performing the task (e.g., throwing away soda can). The full action sequence is recorded for training.
specified location, and
Training and Test Procedure. First, extract segmented image using Segment Anything (Kir- illov et al. 2023) and captioner Dedic (Zhou et al. 2022); second, we query the feature list along with the linguistic specification (e.g.. throw away drink) to construct our abstract feature list; last, we map the abstract feature list back into the original observation as an abstracted state. We train policy to map from the abstracted state into the action sequence For test, we then change the scene (e.g., with predict the action sequence from the new observation.
?
target object like water bottle instead of a soda can, along with distractors) and
Takeaway. LGA produced policies capable of successfully completing both tasks consistently. The failures we did observe were largely due to captioning errors (e.g., the segmentation model detected the object but was unable to produce good text deseription).
8 DISCUSSION & CONCLUSION
We proposed LGA to leverage LMs to learn generalizable policies on state abstractions produced for task-relevant features. Our results confirmed that LGA produces state abstractions similarly effective to those manually designed by human annotators while requiring significantly less user time than manual specification. Furthermore, showed that these abstractions yield policies robust to observational covariate shift and ambiguous language in both single-task and multi-task settings.
Limitations. We assume state abstractions for training good policies be captured by visual features present in individual scenes. An exciting direction for future work would be to learn how to build state abstractions for trajectory-relevant features. We also assume features are a) segmentable and b) expressible in language, i.e. a LM (or human) can understand how each feature may or may relate the task. more complex scenarios, many real-world tasks may be dependent on features that are less expressible in text, e.g. relative position, and are difficult to encapsulate in an abstraction. Although there is evidence to suggest even ungrounded LMs can learn grounded concepts such as spatial relations (Patel & Pavlick] 2021), we did not rigorously test this. We are excited to explore additional feature representations (perhaps multimodal visualization interfaces in addition to language representations) that can be jointly used to construct state abstractions.
'Our Spot's name is Moana.
For latency purposes and ease of data generation, we perform imitation learning over the full trajectory rather than each state (i.e. predict sequence of actions from an initial observation only).

---

Published as a conference paper at ICLR 2024
9 ACKNOWLEDGEMENTS
We thank the Boston Dynamics Al Institute for providing robotic hardware resources used this project, well as Nishanth Kumar for help in running (late-night) demonstration videos. We thank members of the Language and Intelligence and Interactive Robotics Groups for helpful feedback and discussions. Andi Peng supported by the NSF Graduate Research Fellowship and Open Philanthropy. Belinda Li and Theodore Sumers are supported by National Defense Science and Engineering Graduate Fellowships. Ilia Sucholutsky is supported by a NSERC Fellowship (567554- 2022). This work was supported by the National Science Foundation under grant IIS-2238240.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
David Abel, David Hershkowitz, and Michael Littman. Near optimal behavior via approximate state abstraction. In International Conference on Machine Learning, pp. 2915-2923, 2016.
David Abel, Dilip Arumugam, Lucas Lehnert, and Michael Littman. State abstractions for lifelong reinforcement learning. In International Conference on Machine Learning, pp. 10-19, 2018.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as I can, not as 1 say: Grounding language in robotic affordances. Conference on Robot Learning, 2022.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015.
Andreca Bobu, Marius Wiggert, Claire Tomlin, and Anca D. Dragan. Feature expansive reward learning: Rethinking human input. In International Conference on Human-Roue Interaction, pp. 216-224, 2021.
Andreea Bobu, Yi Liu, Rohin Shah, Daniel S. Brown, and Anca D. Dragan. STRL: similaitty- implicit representation learning. International Conference on Hluman-Robor Interaction, 2023,
Andreea Bobu, Andi Peng, Pulkit Agrawal, Julie Shah, and Anca D Dragan. Aligning robot and human representations. International Conference on Human-Rober Interaction, 2024.
Maya Cakmak and Andrea L Thomaz. Designing robot learners that ask good questions. In Inter- national conference on Rohoa Interaction, pp. 17-24, 2012.
Thomas Carta, Pierre-Yves Oudeyer, Olivier Sigaud, and Sylvain Lamprier. Eager: Asking and answering questions for automatic reward shaping in language-guided rl. Advances in Neural Information Processing Systems, 35:12478-12490 2022.
Nick Chater and Paul Vitányi. Simplicity: a unifying principle in cognitive science? Trends in cognitive sciences, 1(1):19-22, 2003.
John D Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, Jacob Andreas, John DeNero, Pieter Abbeel, and Sergey Levine. Guiding policies with language via meta-leariig. arXiv preprint 181107888 2018.
Adam Coates and Andrew Y Ng. Learning feature representations with k-means. In Neural Net- works: Tricks of the Trade: Second Edition, pp. 561-580. Springer, 2012.
André Correia and Luis A Alexandre. A survey of demonstration learning. arXiv preprint arXiv:2303.1 11191, 2023.
Carlos Diuk, Andre Cohen, and Michael L Littman. An object- representation for efficient reinforcement learning. International Conference on Machine Learning, pp. 240-247, 2008
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning, 2023.
Judith E Fan, Robert D Hawkins, Mike Wu, and Noah D Goodman. Pragmatic inference and visual abstraction enable contextual flexibility during visual communication. Computational Brain & Behavior, 3:86-101, 2020.
Prasoon Goyal, Scott Niekum, and Raymond Mooney. Using natural language for reward shaping in reinforcement learning. International Joint Conference Artificial Intelligence, 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.

---

Published as a conference paper at ICLR 2024
Mark K Ho, David Abel, Carlos G Correa, Michael L Littman, Jonathan D Cohen, and Thomas L Griffiths. People construct simplified mental representations to plan. Nature, 606(7912):129-136 2022.
Mark Ho, Jonathan Cohen, and Thomas Griffiths. Rational simplification and rigidity in human planning. PsyArXiv, Mar 2023.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint TrXiv:2201.0720,, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprini SiKiv::2077.06000 2022b.
Holly Huey, Xuanchen Lu, Caren M Walker, and Judith E Fan. Visual explanations prioritize func- tional properties at the expense of visual fidelity. Cognition, 236:105414, 2023.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yonggiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. International Conference Machine Learning, 2022.
Siddharth Karamcheti, Suraj Nair, Annie S Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. Robotics: Science and Systems, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint urXiv:2304.026447 2023.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. International Conference on Learning Representations, 2023.
Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on Robot Learning, pp. 143-156, 2017.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive rein- forcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.05091 2021.
Belinda Z Li, William Chen, Pratyusha Sharma, and Jacob Andreas. Lampp: Language models as probabilistic priors for perception and action. arXiv e-prints, pp. arXiv-2302, 2023.
William P McCarthy, Robert D Hawkins, Haoliang Wang, Cameron Holdaway, and Judith E Fan. Learning to communicate about shared procedural abstractions. arXiv preprint arXiv:2107.0000 2021a.
William P McCarthy, Marcelo G Mattar, David Kirsh, and Judith E Fan. Connecting perceptual and procedural abstractions in physical construction. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 43, 2021b.
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi Mapping instructions actions in 3D environments with visual goal prediction. In Conference on Empirical Methods Natural Language Processing, pp. 2667-2678, 2018
Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah Goodman, Tim Rocktäschel, and Edward Grefenstette. Improving intrinsic exploration with language abstractions. Advances in Neural Information Processing Systems, 2022.
OpenAl. GPT-4 technical report, 2023.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In Inter- national Conference Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2024
Andi Peng, Aviv Netanyahu, Mark K Ho, Tianmin Shu, Andreea Bobu, Julie Shah, and Pulkit Agrawal. Diagnosis, feedback, adaptation: human-in-the- in-oh--ooppp framework for test-time policy adaptation. 2023.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in Neural Information Processing Systems, I, 1988.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley Sons, 2014.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: /908. 10084, 2019.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In Conference on Artificial Intelligence and Statis- tics, pp. 627-635. 2011.
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. In Association for Computational Linguistics, pp. 1713-1726, 2022.
Noah Shimn, Beck Labash, and Ashwin Gopinath. Reflexion: autonomous agent with dynamic memory and self-reflection. arXiv preprint arXivv2303. 2023.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pp. 894-906, 2022.
Theodore R Sumers, Mark K Ho, Robert Hawkins, Karthik Narasimhan, and Thomas L. Grif- fiths. Learning rewards from linguistic feedback. In AAAI Conference on Artificial Intelligence, volume 35, pp. 6002-6010, 2021.
Philip S Thomas and Andrew G Barto. Conjugate markov decision processes. In International Conference Machine Learning, pp. 137-144, 2011.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint 2xXv:2305.16221,, 2023.
Andy Zeng, Maria Autarian, brian ichter, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, Vin- cent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. In International Conference Learning Representations, 2023.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European Conference on Computer Vision, pp. 350-368, 2022.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 ENVIRONMENT DETAILS
States are fully-observable images and represent a RGB fixed-camera topdown view of the scene. The action space is continuous of dimension 4 and consists of high-level pick-and-place actions parameterized by the pose of the end effector. There are 4 possible objects that can be spawned: the target (manipulated) object, goal object, and (up to) two possible distractors. There two types of target features (object type and texture) and 29 possible instantiations of object type and 81 instantiations of texture. Visual depictions of the features can be found in the user study below.
A.2 TASK DETAILS
We provide details regarding all tasks, including ground truth task distributions and full LM prompt.
Pick-and-place:
red heart:
Task prompt: Bring me the red heart.
True distribution: {objects: heart), {textures: red, dark red, dark red swirl, red paisley)
heart:
Task prompt: Bring me the heart.
True distribution: {objects: heart), (textures: ALL.)
tiger-colored object:
Task prompt: Bring me the tiger-colored object.
. True distribution: {objects: ALL., (textures: tiger)
letter from the word letter:
Task prompt: Bring me letter from the word lltttr'.
True distribution: {objects: letter E, letter R, letter T}, (textures: ALL]
consonant with warm-color:
Task prompt: Bring me a consonant with a warm color on it. True distribution: {objects: letter G, letter M, letter R, letter letter v), (textures: dark fred-yylloww-iink--oe
and stripe, and polka dot, dark (rod-yyllow--int--oangg and * polka dot, fred-yellow-ping swirl, dark (red-yellow-pinkk swirl, Locd-yellow-pinhe paisley, tiger, magma, wooden, rainbow, tiles. brick)
dark
vowel with multiple colors:
Task prompt: Bring me vowel with multiple colors on
True distribution: {objects: letter A, letter E}, ftextures: polka dot, tiles, checkerboard, plastic, tiger, magma, rainbow, * and * stripe, * and * polka dot, * swirl, paisley]
something to drink water out of:
Task prompt: Bring me something to drink water out of. True distribution: {objects: bowl, pan, container), [textures: ALL}
something from a typical kitchen:
Task prompt: Bring me something from typical kitchen.
True distribution: (objects: bowl, pan, container), (textures: ALL.)
Rotate:
block:
Task prompt: Rotate the block.
14

---

Published as a conference paper at ICLR 2024
True distribution: (objects: block, small block, shorter block, L shaped block), ftextures: }
something to drink water out of:
Task prompt: Rotate something to drink water out of.
True distribution: (objects: container, bowl, pan), ftextures: red, dark red, dark red swirl, red paisley)
Sweep While Avoid:
block, pan:
Task prompt: Sweep the block without touching the pan.
True distribution: {objects: block, small block, shorter block, L shaped block, pan),
{texture: ALL)
block, line:
Task prompt: Sweep the block without touching the line.
True distribution: {objects: block, small block, shorter block, L shaped block, line),
{textures: ALL]
A.2.1 FULL PROMPT
ChatGPT models (including GPT4) can take both system prompts and user prompts. We split our prompt into these two parts as follows.
System prompt where {object_ is replaced by the list of all object types in the environment and {object_colors) by the list of all colors and textures:
You interfacing with robotics environment that has robotic learning to manipulate objects based on some linguistic command (e.g. "pick up the red bowl"). At each interaction, the researcher will specify the command that you need to teach the robot. In order to teach the robot, you will need to help design the training distribution by specifying what properties task-relevant objects can have based on the given command. Objects in this environment have two properties: object type, object color. Any object type can be paired with any color, but an object can only take on exactly one object type and exactly one color. Object types:
{object_list}
Object colors:
{object colors)
User prompt where {rule} is replaced by one of the task prompts listed above, {group) } is replaced by "object color" or "object type", and (candidate) replaced by each candidate object color or type that would like the LM to evaluate:
The command is "(rule}". In instantiation of the environment that contains only some subset of the object types and colors, could the target object have {group) " "(candidate)"" Think step-by-step and then finish with line that says "Final answer:" followed by "yes" or "no".
A.3 ARCHITECTURE AND TRAINING DETAILS
Architecture.
For GCBC, Sentence-BERT processes a goal utterance in natural language into an embedding of size 384, which we additionally process through a linear MLP of output size 100. We process the state using a standard Conv2D block consisting of 3 stacked Conv2D layers of output channel sizes 32, 64, 32, and strides 4, and 1. Each output layer is processed by BatchNorm2D as well as ReLU activation. After the last Conv2D layer, we flatten the output and concatenate with the output of the goal utterance. We feed the concatenated output through last linear layer for action prediction.
15

---

Published as a conference paper at ICLR 2024
For LGA, process both the state and the state abstraction through dual Conv2Ds blocks like above. We concatenate the output and feed through a last linear layer for action prediction. Training.
We train all networks to convergence for a maximum 750 epochs. All computation was done two NVIDIA GeForce RTX 3090 GPUs. Rollouts were rendered locally using PyBullet commercial hardware (a Macbook Pro).
A.4
USER STUDY
In the following pages, we have included the full user study shared with participants. Following stan- dard user study procedure, we initial briefed users by telling them how long the study was (roughly 30 mins) and that they were free to leave at anytime. Demographic information was collected in person. We showed participants the first pdf during the familiarization phase to introduce them to the environment and full feature list. We then randomly chose to begin with either the human-only task (second pdf) or the LM-aided task (third pdf). After the study, users were debriefed and given the email the study designer to contact they had any questions.
16

---

User Study: Instructions and Dictionary
You are helping teach a robot which properties are important to a desired task.
The robot is learning to pick up objects based some linguistic
command (e.g. "pick up red bowl").
At each interaction, we will specify the task that you need to teach the robot. In order to teach the robot, you will need to help design the task by specifying what properties the target object can have based on the given command.
Target objects in this environment have two properties: [object type, object color].
Any object type can be paired with any color, but an object can only take on exactly one object type and exactly one color.
Note:
Object types:
L-shaped block
block
bowl
container
cross
diamond
flower
frame
heart
hexagon
letter A
letter
M
R
V
letter G
letter M
letter R
letter
letter V
line
pallet
pan
pentagon
ring
round
shorter block
small block
square
star
three-sided rectangle
triangle

---

Object colors:
gron
orange
PAN
donc aku
cark opon
da% gren
da% oronge
ctrt piC
CArE purple
derki red
CArT yeflow
granite
chockerbeard
Pger
reagma
red and
rod and green
redland baue
red and pursle
yeflow and green
yellow and Lave
yellow and purple
green and blue
OPean andpurph
Mas ardparpe
red
and
red
and
pellicu and
wellew
dask yellow and
green blice
dark grant
da.l
red and yertow
rod and roo
red and like
and purplo
yeflow and green
ysllw and lave
polk
palka doe
polk
yUllO O0!
grown poRa
UAO polsa DU
purple pola
green dor
DAV poRi dut
purpe poll
TRUE poRio
BURRO palka
perple polko
daN IO SART
CUS yeloo SAiT
daN DAYSA
Example task: "Bring me polka dot geometric shape"
In [3]: task
task_kwargg kwargs
ipossible_dragged.oo
d'iamond'
'triangle"
hexagon
pentagon
square ),
'possible dossible_draggged obt_oobtturreeeeeeeeeeeeeeeeeeee texture':
reed and yellow polka dot', rred and green polka dot 'red blue polka dot',
'red and purple polka dot' 'eellow and green polka dot', 'yellow and blue polka dot',
'yellow and purple polka dot',
'green and blue polka dot',
'green and purple polka dot',
'blue and purple polka dot'
darr red and yellow polka dot',

---

dark red and green polka dot', dark red and blue polka dot',
dark red and purple polka dot',
dark yellow and green polka dot', 'dark yellow and blue polka dot', ' 'dark yellow and purple polka dot', 'dark green and blue polka dot', 'dark green and purple polka dot 'dark blue and purple polka dot', ,}}

---

Experiment
For your reference, here are the object types and colors (feel free to copy + paste):
In [2]:
possible_draggedddee
('L-shaped block',
'block',
'bowl',
containne',
cross'',
diamond
flower'
frame",
heart',
'hexagon"
"letter A
'letter E',
'letter G',
"letter M',
"letter R',
"letter T',
"letter V',
"line',
'pallet',
"pan',
"pentagon',
'ring',
'round',
'shorter block',
"small block', "square', 'star', 'three-sided rectangle', 'triangle)
In [ [: beasible_draggeet t
texture't
'brick',
'tiles'
'wooden',
granitt''
ppaasicc'
polka dot
checkerboard',
ttieer,
"magma',
"rainbow'
'blue',
cyaa',
'green'
'olive'
'orange"
"pink',
purple'',
"red',
'yellow',
'dark blue'
'dark cyan',
'dark green',
'dark olive',
'dark orange',
ddrr pink',

---

"red and blue polka dot', "red and purple polka dot', "yello and green polka dot', "elllo and blue polka dot', yellow and purple polka dot', "grenn and blue polka dot', ggreen and purple polka dot', "blue and purple polka dot', 'dark red and yellow polka dot', dark purple',
dark red',
'dark yellow',
"red and yellow stripe',
'red and green stripe',
"red and blue stripe',
"red and purple stripe',
yellow and green stripe',
yellow and blue stripe', yellow and purple stripe', green and blue stripe', green and purple stripe', "blue and purple stripe',
'dark red and yellow stripe',
'dark red and green stripe',
'dark red and blue stripe',
'dark red and purple stripe',
'dark yellow and green stripe',
'dark yellow and blue stripe',
'dark yellow and purple stripe',
'dark green and blue stripe',
'dark green and purple stripe'
'dark blue and purple stripe',
"red and yellow polka dot',
"red and green polka dot',
'dark red and green polka dot'
'dark red and blue polka dot',
'dark red and purple polka dot',
'dark yellow and green polka dot',
'dark yellow and blue polka dot',
'dark yellow and purple polka dot', 'dark green and blue polka dot', 'dark green and purple polka dot', 'dark blue and purple polka dot',
'red swirl',
"yellow swirl',
"green swirl',
"blue swirl', purplle swirl', 'dark red swirl',
'dark yellow swirl',
'dark green swirl',
'dark blue swirl',
'dark purple swirl',
'red paisley',
"elllo paisley',
'rrenn paisley',
'blue paisley', 'purple paisley')
Task 1: "Bring me the red heart"
In [4]: task laek__wargs_ {
'possible dareeeddradd___ba obj's [),

---

[ dragged obj ttt_teeturr'.
[]}
Task 2: "Bring me the heart"
In [::
task kwargs posssibee daaeeeedrrgddd__e _bjj' (],
possible draggeddgddddgeeee obj texture's []}
Task 3: "Bring me the tiger object"
In [:
task kwargs "possible lelee_ddrgddd_eeee _bji- [], possible__die daaeeeedrrrddd_eeee obj_ texture's []}
Task 4: "Bring me a letter from the word letter"
In []:
task kwargs possible dragged_objj [],
possible Baaigbeeedrggeddeeee
dragged _bapeed________e pooj__txturr''' : (}}
Task 5: "Bring me consonant that has any warm color on it"
In [:
task kwargs =
possible_dragged obj' [],
"possible dragged obj texture's [}}
Task 6: "Bring me vowel that has multiple colors on it" (white is not color)
In []: task kwargs =
"possible dragged_oii : (),
"possible dragged obj texture's []}
Task 7: "Bring me a letter. If there are multiple, bring the one that comes earliest in the alphabet."
In []: task kwargs = {
ppsssible dragged obj' [],
possible draggedddrrgedd_eeee obj_texture" [}}
Task 8: "Bring me something can drink water out of"
In []:
task kwargs
"possible_dragged obj' : (),
haabbblledrrgeeddd_e obj_otetturee'..eoeee []}
Task 9: "Bring me something can find in a typical kitchen"
In []:
task kwargs
possible_dragged
dragged obj's [],
pessible_dragged batrgeddd_j_ttetttt_eee texture's []}

---

Experiment
For your reference, here are the object types and colors (feel free to copy + paste).
A large language model (GPT-4) has given its answers to help you with the task (they are pre-filled in). You may feel free to change/modify the answers, or accept them without change:
In [: [
[ 'L-sshapdd block',
'block',
'bowl',
'containe'',
'cross',
'diamond'
flower":
frame
heart',
'heaagon'
'letter A
"letter E',
"letter G',
"letter M',
"letter R',
"letter T',
"letter V',
"line',
'pallet',
'pan',
"pentagon',
'ring',
'round',
"shorter block',
'small block', "square', 'star', 'three-sided rectangle', trranglee'
In [ []: 'possible cerpgled_rrgee____ee t
texture's
('brick',
'tiles',
'wooden',
granite',
plastic'
polka dot
checkerboard'
'tiger',
"magma',
rainbow',
blue',
'yyan'
ggreen
'olive"
'orange'.
'pink'.
'purple',
"red',
'eellow',
ddark blue',
dark cyan',

---

dark green',
dark olive',
dark orange',
'dark pink',
'dark purple',
'dark red',
'dark yellow',
"red and yellow stripe',
"red and green stripe',
"red and blue stripe', "red and purple stripe', "ylllw and green stripe', yellow and blue stripe', "elllow and purple stripe' green and blue stripe', green and purple stripe', 'blue and purple stripe',
'dark red and yellow stripe',
'dark red and green stripe',
'dark red and blue stripe',
'dark red and purple stripe',
'dark yellow and green stripe',
'dark yellow and blue stripe',
'dark yellow and purple stripe'
'dark green and blue stripe',
'dark green and purple stripe',
'dark blue and purple stripe',
"red and yellow polka dot',
"red and green polka dot',
"red and blue polka dot', reed and purple polka dot', "yellow and green polka dot', "elllw and blue polka dot', "yellow and purple polka dot', green and blue polka dot', greenn and purple polka dot', 'blue and purple polka dot', 'dark red and yellow polka dot', 'dark red and green polka dot',
'dark red and blue polka dot',
'dark red and purple polka dot',
'dark yellow and green polka dot',
'dark yellow and blue polka dot',
'dark yellow and purple polka dot', 'dark green and blue polka dot', 'dark green and purple polka dot', I 'dark blue and purple polka dot',
red swirl',
"yellow swirl',
"green swirl', 'blue swirl', purplle swirl', "dark red swirl',
'dark yellow swirl',
'dark green swirl',
'dark blue swirl',
'dark purple swirl',
"red paisley',
"yellow paisley',
"green paisley',
'blue paisley',
purple paisley') ]

---

Task 1: "Bring me the red heart"
In []: task kwargs
possible_dragged obj': ('heart' ], possible dragged obj "er__texture''
l'red',
dark red',
red swirl', dark red swirl' 'red paisley') }}
Task 2: "Bring me the heart"
In []: task kwargs
"possibleddrageed obj', [ ('heart's ],
laaigggeeddgggeedeeeee _bl__ootttreeeeeeeeeeeeeee texture't ('brick',
'tiles',
wooden',
granite',
plastic',
polka dot',
checkerboard',
tiiee',
maagma',
rainbow',
blue',
cyan
green'
olive',
'orange'.
'pink',
purple''
red''
yeellw',
'dark blue',
'dark cyan',
'dark green',
'dark olive',
'dark orange',
'dark pink',
'dark purple',
'dark red',
'dark yellow', "red and yellow stripe',
"red and green stripe', 'red and blue stripe',
ree and purple stripe', yellow and green stripe', "eellow and blue stripe', "yellow and purple stripe', greenn and blue stripe', "green and purple stripe', 'blue purple stripe',
'dark red and yellow stripe',
'dark red and green stripe',
'dark red and blue stripe',
'dark red and purple stripe',
'dark yellow and green stripe',
'dark yellow and blue stripe',
'dark yellow and purple stripe',
'dark green and blue stripe',
'dark green and purple stripe', 'dar blue and purple stripe',

---

'dark yellow and green polka dot', "red and yellow polka dot', 'red and green polka dot',
"red and blue polka dot', "red and purple polka dot',
"yello and green polka dot', yellow and blue polka dot',
yellow and purple polka dot green and blue polka dot', green and purple polka dot', 'blue and purple polka dot', 'dark red and yellow polka dot', 'dark red and green polka dot',
'dark red and blue polka dot',
'dark red and purple polka dot',
'dark yellow and blue polka dot',
'dark yellow and purple polka dot', 'dark green and blue polka dot', 'dark green and purple polka dot', "drk blue and purple polka dot',
red swirl', yelllw swirl', green swirl', 'blue swirl', purple swirl', 'dark red swirl',
'dark yellow swirl',
'dark green swirl',
'dark blue swirl',
'dark purple swirl',
"red paisley',
'yellow paisley',
"green paisley',
'blue paisley',
purple paisley' }}
Task 3: "Bring me the tiger object"
In []:
task kwargs Begeibbee_dagggd_eee "possible_daes dragged obj': ; ('L-shaped block',
block',
bowl',
ccottinne',,
cross'
diamond'.
flower"
frame"
heart',
hhxaaon
"letter A',
"letter E',
"letter G',
'letter M',
"letter R',
'letter T',
"letter V',
'line',
'pallet',
'pan',
'pentagon',
'ring',
'round',
"shotter block',

---

small block', square",
star',
tthreessided rectangle', "trianglel,,
ppssible_dragged wet_eooottttteeeeeeeeeeeee ['tiger'] }}
Task 4: "Bring me letter from the word letter"
In [ [: task kwkrgswwrrrsss = {
letter E' 'letter R', 'letter T'
p obj texture's ['brick',
'tiles'
wooden',
granite',
plastic',
polka dot',
checkerboard''
tiger',
magma'
rainbow
blue',
'yan',
green'
oolie'
orange',
"pink',
purplee'
red''
"yellow',
'dark blue',
'dark cyan', 'dark green',
'dark olive',
'dark orange',
'dark pink', 'dark purple', 'dark red',
'dark yellow', 'red and yellow stripe',
"red and green stripe',
"red and blue stripe',
"red and purple stripe', "elllow and green stripe',
yellow and blue stripe', "elllw and purple stripe', green and blue stripe', "green and purple stripe',
'blue and purple stripe', 'dark red and yellow stripe',
'dark red and green stripe',
'dark red and blue stripe',
'dark red and purple stripe',
'dark yellow and green stripe',
'dark yellow and blue stripe',
'dark yellow and purple stripe',
'dark green and blue stripe',
'dark green and purple stripe',
'drr blue and purple stripe',
"red and yellow polka dot',
'red and green polka dot',

---

"dark pink', "red and blue polka dot', red and purple polka dot', yellow and green polka dot', yelllw and blue polka dot', "yello and purple polka dot', greenn and blue polka dot', "green and purple polka dot 'blue and purple polka dot',
'dark red and yellow polka dot', 'dark red and green polka dot',
'dark red and blue polka dot
'dark red and purple polka dot',
'dark yellow and green polka dot',
'dark yellow and blue polka dot',
'dark yellow and purple polka dot',
'dark green and blue polka dot', 'dark green and purple polka dot', 'dark blue and purple polka dot', "red swirl',
"yellow swirl',
green swirl', "blue swirl', purple swirl', 'dark red swirl',
'dark yellow swirl',
'dark green swirl',
'dark blue swirl',
'ark purple swirl',
rred paisley',
'yellow paisley',
green paisley',
'blue paisley', 'purple paisley')
Task 5: "Bring me a consonant that has any warm color on it"
In []:
task kwargs letter G'
'letter M'
'letter R', 'letter T', 'letter V'
'possible dragged _at_e_oootrrree_eeeeeeeeee
texture's ('polka dot', 'orange', 'pink',
redd',
yellow',
dark orange',
'dark red',
'dark yellow',
'red and yellow stripe',
'dark red and yellow stripe',
'dark red and green stripe',
'dark red and blue stripe',
'dark yellow and purple stripe', 'red and yellow polka dot', "dark red and yellow polka dot', "red swirl',
"yellow swirl'
dark red swirl', dark yellow swirl',

---

'red paisley', yellow paisley') }}
Task 6: "Bring me a vowel that has multiple colors on it" (white is not a color)
In []:
task kwargs =
ppsssible caceee_dragddddooe obj':
('letter A',
'letter E E'],
"possible dragged obj texture's : I'polka dot',
checkerboard''
tigerr,
mmggm',
rainbow',
yellow',
dark green',
'dark yellow',
'red and yellow stripe',
"red and green stripe',
"red and blue stripe',
red and purple stripe',
yellow and green stripe',
yellow and blue stripe',
yellow and purple stripe',
green and blue stripe', green and purple stripe', 'blue and purple stripe',
'drk red and yellow stripe',
ddark red and green stripe',
'dark red and blue stripe',
'dark red and purple stripe', 'dark yellow and green stripe',
ddark yellow and blue stripe',
'dark yellow and purple stripe', ddrk green and blue stripe',
'dark green and purple stripe',
'dark blue and purple stripe', reed and yellow polka dot',
"red and green polka dot',
rred and blue polka dot',
red and purple polka dot' "elllo and green polka dot', 'yellow and blue polka dot'
'yellow and purple polka dot', green and blue polka dot',
green and purple polka dot', 'blue and purple polka dot', 'dark red and yellow polka dot', ddark red and green polka dot' 'dark red and blue polka dot', 'dark red and purple polka dot', 'dark yellow and green polka dot', 'dark yellow and blue polka dot', 'dark yellow and purple polka dot', dark green and blue polka dot', dark green and purple polka dot', 'dark blue and purple polka dot', 'yellow swirl',
green swirl',
'blue swirl', 'drr yellow swirl',
dark green swirl', dark blue swirl',

---

'dark purple', dark purple swirl',
red paisley',
green paisley
'blue paisley',
'purple paisley') }}
Task 7: "Bring me a letter. If there are multiple, bring the one that comes earliest in the alphabet."
In []: task_ kwargs saceeedrragdd__b____
'letter A',
'letter E',
'letter G',
'letter K", 'letter R', 'letter T' 'possible dragged obj__oojtttteee texture': ['brick', 'tiles', 'wooden',
'gaanit',,
plastic',
polka dot checkerboard'
tiger',
maaga',
rainbow",
blue',
cyan',
green
olive'
orange'
'pink',
'purple',
red''
yeello'',
'dark blue',
'dark cyan',
'dark green',
'dark olive',
'dark orange',
'dark pink',
'dark red',
'dark yellow',
"red and yellow stripe',
"red and green stripe',
"red and blue stripe',
"red and purple stripe', "elllo and green stripe',
yellow and blue stripe', "eellow and purple stripe', greenn and blue stripe', ggrenn and purple stripe',
'blue and purple stripe',
'dark red and yellow stripe',
'dark red and green stripe',
'dark red and blue stripe',
'dark red and purple stripe',
'dark yellow and green stripe',
'dark yellow and blue stripe',
'dark yellow and purple stripe',
dark green and blue stripe',
"dark green and purple stripe',

---

"ylllw and purple polka dot green and blue polka dot', green and purple polka dot', 'blue and purple polka dot', 'dark red and yellow polka dot', 'dark blue and purple stripe',
"red and yellow polka dot',
"red and green polka dot',
"red and blue polka dot',
"red and purple polka dot',
yellow and green polka dot',
yellow and blue polka dot',
'dark red and green polka dot',
'dark red and blue polka dot
'dark red and purple polka dot',
'dark yellow and green polka dot',
'dark yellow and blue polka dot',
'dark yellow and purple polka dot', 'dark green and blue polka dot', 'dark green and purple polka dot', 'drr blue and purple polka dot', red swirl',
yelllw swirl',
green swirl', 'blue swirl', "purple swirl', 'dark red swirl',
'dark yellow swirl',
'dark green swirl',
'dark blue swirl',
'dark purple swirl',
'red paisley',
"yellow paisley',
"green paisley',
'blue paisley',
'purple paisley' }}
Task 8: "Bring me something can drink water out of"
In [
task kwargs
_bjj
bbowl' 'cntainer' },
p obj texture's [ ['brick',
'tiles',
'wooden',
'granite',
plastic',
polka dot
checkerboard'
tiger',
maagma,
rainbow',
blue',
"yaann
'reen''
'oliv''
'orange'
'pink',
'purple',
'red',
'yellow',
dark blue',
ddrk cyan',

---

dark green',
dark olive',
dark orange',
'dark pink',
'dark purple',
'dark red',
'dark yellow',
"red and yellow stripe',
"red and green stripe',
"red and blue stripe', "red and purple stripe', "ylllw and green stripe', yellow and blue stripe', "elllow and purple stripe' green and blue stripe', green and purple stripe', 'blue and purple stripe',
'dark red and yellow stripe',
'dark red and green stripe',
'dark red and blue stripe',
'dark red and purple stripe',
'dark yellow and green stripe',
'dark yellow and blue stripe',
'dark yellow and purple stripe'
'dark green and blue stripe',
'dark green and purple stripe',
'dark blue and purple stripe',
"red and yellow polka dot',
"red and green polka dot',
"red and blue polka dot', reed and purple polka dot', "yellow and green polka dot', "elllw and blue polka dot', "yellow and purple polka dot', green and blue polka dot', greenn and purple polka dot', 'blue and purple polka dot', 'dark red and yellow polka dot', 'dark red and green polka dot',
'dark red and blue polka dot',
'dark red and purple polka dot',
'dark yellow and green polka dot',
'dark yellow and blue polka dot',
'dark yellow and purple polka dot', 'dark green and blue polka dot', 'dark green and purple polka dot', I 'drrk blue and purple polka dot',
red swirl',
"yellow swirl',
"green swirl', 'blue swirl', purplle swirl', "dark red swirl',
'dark yellow swirl',
'dark green swirl',
'dark blue swirl',
'dark purple swirl',
"red paisley',
"yellow paisley',
"green paisley',
'blue paisley',
purple paisley') }}

---

'dark pink', 'dark purple', Task 9: "Bring me something can find in a typical kitchen"
In []: "possible_dee laaeee_drgge_____ _bbj: [ ('bowl', 'container',
task kwargs
paan 1, "possible gossible_draggeedeee obj texture's : [ ('brick', 'tiles',
'wooden',
'gaanit'',
plastic',
polka dot
checkerboard'
tiger',
magma',
rainbow',
blue',
cyan'
green'
oolie'
orange'
'pink',
'purple',
red',
yeello',
'dark blue',
'dark cyan',
'dark green',
'dark olive',
'dark orange',
'dark red',
'dark yellow', "red and yellow stripe',
"red and green stripe', "red and blue stripe',
reed and purple stripe', "yello and green stripe',
"elllw and blue stripe', "elllw and purple stripe' greenn and blue stripe', "green and purple stripe',
'blue and purple stripe',
'dark red and yellow stripe',
'dark red and green stripe',
'dark red and blue stripe',
'dark red and purple stripe',
'dark yellow and green stripe',
'dark yellow and blue stripe',
'dark yellow and purple stripe',
'dark green and blue stripe',
'dark green and purple stripe',
'dark blue and purple stripe',
'red and yellow polka dot',
"red and green polka dot'
I "red and blue polka dot',
"red and purple polka dot', "yellow and green polka dot', "yellow and blue polka dot',
"yellow and purple polka dot', "green and blue polka dot', "green and purple polka dot',

---

bbuu and purple polka dot',
dark red and yellow polka dot',
darr red and green polka dot',
'dark red and blue polka dot',
'dark red and purple polka dot',
'dark yellow and green polka dot',
'dark yellow and blue polka dot',
'dark yellow and purple polka dot',
'dark green and blue polka dot', 'dark green and purple polka dot', dark blue and purple polka dot', "red swirl', "yellow swirl',
green swirl', 'blue swirl', purple swirl', 'dark red swirl',
'dark yellow swirl',
'dark green swirl',
'dark blue swirl',
'ark purple swirl',
red paisley',
"eellow paisley',
'green paisley',
'blue paisley',
'purple paisley') }}


Reviews:
[{"review_id": "8V5E0BQrE6", "reviewer": "Reviewer_rEzX", "summary": "This paper introduces the Language-Guided Abstraction (LGA) framework, which utilizes natural language to construct state abstraction for imitation learning. The method comprises three key steps: First, in the textualization phase, it transforms raw perceptual input into a text-based feature set. Second, during the state abstraction step, a pre-trained language model is employed to filter out irrelevant features from the feature set, creating task-specific state abstractions. Finally, in the instantiation stage, the reduced abstracted feature set is converted into a format understandable by the policy, such as an observation displaying only the pertinent objects.", "questions": "Can you categorize task failures into two groups: those caused by policy quality and those resulting from incorrect state abstraction?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "LGA avoids spurious correlations by highlighting goal information in semantic maps, not raw pixels. LGA converts language and observations into unambiguous states to enhance policy adaptability. This is especially important when only limited training data is available. The integration with Language Models enables contextually appropriate task-relevant feature selection, boosting the overall policy generalization and performance at test time.\n\nThe experiment results demonstrate that LGA reduces the time needed for feature specification compared to manual methods, yet outperforms non-abstraction-based baselines in terms of sample efficiency. Policies trained using LGA's state abstractions exhibit resilience to observational shifts and language variations. In multi-task scenarios, LGA effectively resolves task ambiguities and adapts to new language specifications in observations.", "weaknesses": "There appears to be a gap in the evaluation regarding task failures—whether they stem from policy quality or incorrect state abstraction remains unclear. The experiment results do not specify how frequently the language model predicts insufficient or redundant state abstraction, and whether refining its choices with feedback from policy execution is a feasible solution remains unexplored.\n\nThe paper appears to overlook extensive research on learning state abstraction for reinforcement learning, including notable works such as \"Approximate State Abstraction\" (ICML 2016), \"State Abstraction In Lifelong RL\" (ICML 2018), and \"State Abstraction As Compression\" (AAAI 2019).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bhp6zfd8cP", "reviewer": "Reviewer_PEPS", "summary": "This work presents a method LGA that leverages language models to compose abstract states for few-shot imitation learning. Visual RGB observations are first segmented and textualized into features that a language model is then tasked to filter conditioning on the language instruction. The policy is then trained with such abstract visual state.\n\nExperiments in a simulated benchmark shows the proposed method outperforms naive baselines.", "questions": "It seems LGA relies heavily on the segmentation and captioning module. How well does these systems work? What are some common failure modes? Can the robot arm be successfully segmented?\n\nDoes LGA or the instantiation assume full observability? or does it run segmentation and feature extraction on each and every frame? this seems expensive given the size of SAM \n\nHow good is the language model at guessing relevant objects if they are not explicitly mentioned in the language instruction?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This work presents an intuitive and simple idea that is shown to be powerful for few-shot imitation learning of policies that are specifically designed to generalize across variations of color and texture.\n\nThe proposed method leverages the commonsense reasoning capabilities of large language models for reducing the task complexity for imitation learning, which is an exciting application of pretrained language models in robotics.", "weaknesses": "Conceptually, the idea of using visual masks as attention or part of state representation isn’t novel and has been explored in various prior works including recent ones such as robotmoo[1] and VIOLA[2].\n\nFeature abstraction of LGA takes a filtering approach that relies on segmentation and textualization operates at the desired abstraction level and is complete. For example, language instruction can be about a group of objects, the object as a whole or only part of an object and it is unclear how to segment or group segmentations before we know the task. \nAn alternative approach to filtering would be leveraging open-vocabulary object detectors or VLMs for identifying the target objects like in recent works using LLMs for planning, which the paper didn’t ablate. \n\nUsing binary masks as state representation seems to be limiting and can hurt in tasks where the texture or details of the object matters, maybe the language model should decide if the binary mask or original state should be used or not based on the context. On a similar note, it seems from the results LGA-S performs better anyway?\n\nAt the same time the background might be important landmarks for the robot to understand relative size. For example if the robot is learning to visually navigate to certain objects, this proposed method would fail if the model removes all the background necessary for the robot to localize itself. \n\nThe authors should explicitly discuss assumptions and limitations of the method to specific types of tasks/settings.\n\n[1] https://robot-moo.github.io/ \n[2] https://ut-austin-rpl.github.io/VIOLA/\n\n\n\n---Edit----\n\nThe rebuttal addressed some of my initial concerns and I appreciate the response and explanation from the authors. I do think demonstrating the failure modes of LGA and discussing the limitations is as important as showing the positive results. \n\nI am happy to raise my evaluation but won't argue strongly for acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1muP84b6NR", "reviewer": "Reviewer_4L8o", "summary": "This paper introduces an LLM-based approach to mask out task-irrelevant objects in visual observations to improve imitation learning. The resulting “abstract states” - observations with only task-relevant features, enables training behavior cloning policies with better data efficiency and generalization. The method is straightforward: taking a task specification and the observed object information as input, a pretrained LLM is leveraged to remove irrelevant objects. Optionally, the authors recruited human subjects to refine the LLM output or identify irrelevant objects from scratch as ablations.\nThe authors evaluate the proposed approach on the VIMA[1] benchmark, comparing it with behavior cloning policies trained with various forms of state abstractions. The results indicate that the proposed method: 1) enhances data efficiency and success rate while requiring less human efforts; and 2) can generalize to unseen object textures, distractor objects, and task specifications.\n\n[1] Jiang, Yunfan, et al. \"Vima: General robot manipulation with multimodal prompts.\" ICML 2023.", "questions": "Regarding the observation input to behavior cloning in LGA, do you use the binary mask directly (as visualized in Figure 1) or the masked image? If you use the former one, I wonder how the latter one works.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea to leverage the semantic reasoning ability of LLMs to identify task-relevant features is interesting and innovative. The approach of masking out irrelevant objects does prevent learning robot policies harmed by spurious correlations.\n2. The paper is written clearly and easy to follow.\n3. The paper includes comprehensive experiments to demonstrate the effectiveness of the proposed method.", "weaknesses": "1. The primary weakness of the paper lies in the absence of a lack of substantial technical contribution. While the proposed strategy mitigates the issue of spurious correlations, it appears more as a choice of system design to generate heuristics rather than a fundamental method to identify useful state features. Notably, the importance of state features not only depends on task semantics, but also the low-level geometric constraints imposed by the environment and robot embodiment. For example, some objects not mentioned in the task specification may be important for robot collision avoidance. Consequently, the learning of important state features and motion policies are coupled and should ideally be learned together, such as in VIOLA [2].\n2. Another drawback is that the imitation learning setup is overly simplified . The use of VIMA [1] benchmark, which employs high-level primitive actions like “pick” and “place” with continuous goal poses, significantly simplifies the training of a behavior cloning policy. Given that the major objective is to assess the proposed state abstraction in imitation learning, it would be advisable for the authors to consider a more rigorous robot manipulation benchmark with continuous actions, such as RoboSuite [3].\n3. State abstraction is a fundamental problem in decision making. I think it would be helpful for the readers to understand the context better if the authors refer and discuss the related literature (such as [4,5]).\n\n\n[2] Zhu, Yifeng, et al. \"Viola: Imitation learning for vision-based manipulation with object proposal priors.\" CoRL 2022.\n[3] https://robosuite.ai/\n[4] Li, Lihong, Thomas J. Walsh, and Michael L. Littman. \"Towards a unified theory of state abstraction for MDPs.\" AI&M 2006.\n[5] Tomar, Manan, et al. \"Model-invariant state abstractions for model-based reinforcement learning.\" ICLR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)