Learning Planning Abstractions from Language
Authors: Weiyu Liu
Geng Chen
Joy Hsu
Jiayuan Mao
Jiajun Wu
Venue: ICLR 2024 poster (2024)
Keywords: Planning and Learning
Learning Abstractions
Compositional Generalization
Robotic Manipulation
Primary area: applications to robotics, autonomy, planning
TL;DR: A framework that utilizes language-annotated demonstrations to automatically discover a symbolic and abstract action space and induce a latent state abstraction for planning.

Abstract: This paper presents a framework for learning state and action abstractions in sequential decision-making domains. Our framework, planning abstraction from language (PARL), utilizes language-annotated demonstrations to automatically discover a symbolic and abstract action space and induce a latent state abstraction based on it. PARL consists of three stages: 1) recovering object-level and action concepts, 2) learning state abstractions, abstract action feasibility, and transition models, and 3) applying low-level policies for abstract actions. During inference, given the task description, PARL first makes abstract action plans using the latent transition and feasibility functions, then refines the high-level plan using low-level policies. PARL generalizes across scenarios involving novel object instances and environments, unseen concept compositions, and tasks that require longer planning horizons than settings it is trained on.

Published as a conference paper at ICLR 2024
LEARNING PLANNING ABSTRACTIONS FROM LANGUAGE
Weiyu Liu ", Geng Chen'r, Joy Hsu', Jiayuan ,Jajuun Wu Stanford MIT
ABSTRACT
This paper presents framework for learning state and action abstractions in sequential recision-makingg domains. Our framework, planning abstraction from language (PARL), utilizes language-annotated demonstrations to automatically discover symbolic and abstract action space and induce latent state abstraction based on it. PARI. consists of three stages: 1) recovering object-level and action concepts, 2) learning state abstractions, abstract action feasibility, and transition models, and 3) applying low-level policies for abstract actions. During inference, given the task description, PARL first makes abstract action plans using the latent transition and feasibility functions, then refines the high-level plan using lowelevel policies. PARL generalizes across scenarios involving novel object instances and environments, unseen concept compositions, and tasks that require longer planning horizons than settings it is trained on.
INTRODUCTION
State and action abstractions have been widely studied in classical planning, reinforcement learning, and operations research as a way to make learning and decision making more efficient (Giunchiglia & Walsh, 1992; Ghallab et al., 2004; Sutton et al., 1999; Li et al., 2006; Rogers al., 1991). In particular, a state abstraction maps the agents' raw observation of the states (e.g., images or point clouds) into abstract state descriptions (e.g., symbolic or latent representations). Action abstraction constructs a new library of actions which can be later refined into raw actions (e.g., robot joint commands). "good" state and action representations are beneficial for both learning and planning, as state abstraction extracts relevant information about the actions to be executed, and action abstraction reduces the search space for meaningful actions that an agent could try.
Many prior works have explored methods for learning or discovering such state and action abstractions, for example, through bi-simulation (Givan et al., 2003), by learning latent transition models (Chiappa et al., 2017; Zhang et al., 2021), by inventing symbolic predicates about states (Pasula et al., 2007; Silver et al., 2021), or from language (Andreas & Klein, 2015; Corona et al., 2021). In this paper, we focus the latter, and learn abstractions from language for planning. Fig. la shows the overall policy
learning and planning paradigm of our system. Our goal is to construct a ianguage-condtiinn for solving complex tasks. Given a sufficient amount of demonstration data online experience, reinforcement learning agent could, in theory, learn a language-conditiong However, in scenarios where there is a significant amount of variations in the number of objects, initial object configurations and planning steps, learning such policies is inefficient and at times infeasible. Therefore, we explore the idea of discovering symbolic abstraction of the action space from language and inducing a latent state abstraction for abstract actions. This allows agents to plan at test time in an abstract state and action space, and reduces the horizon for policy learning by decomposing long trajectories into shorter pieces using the action abstraction.
policy to solve all tasks.
In particular, our framework, planning abstraction from language (PARL) takes a learning and planning approach. First, illustrated in Fig. la, given paired demonstration trajectories and language descriptions of the trajectories, we recover symbolic abstract action space, composed of object-level concepts and action concepts. Each object-level concept describes property of an object (c.g., its category or color), and each action concept is a verb schema that takes objects as its argument (c.g.. clean). These object and action concepts can be recombined in compositional manner to form
denotes equal contribution. denotes equal advising. Project page: https//parr2224gihhbbi

---

Published as a conference paper at ICLR 2024
Training Time
(b1) Generalication nuvel envircements
pick the then...
Planning
Learning Algo
Compatible
"place the pan the counter"
Mudel
(b7) Generalization constrained placing
Test Time
"place the grey pan in the sink"
High-Level Planner Low-1 evel Policy
(63) Gemeralication nuvel compositions
"place the red mag
Action
the red pan the sink''
sink"
(x) The overall training and test framework.
(b) Types of generalization supported by our framework.
The overview of our training and testing paradigm, and different types of generalizations supported by our framework. (a) Given paired demonstration trajectories and language descriptions, our framework discovers an abstract action space and latent state abstraction that supports planning for diverse language goals. (b) The example illustrates that our model can generalize to new kitchen environment, generalize to goal that requires reasoning about the geometries of the sink and grey pan, and generalize to the combination of concepts red and pan that unseen during training.
Figure 1:
the entire action space (e.g., clean(blue bowl)). Next, we learn four models based on the abstract action space: (1) state abstraction function that maps observations
latent space, (2) an abstract action transition model in the induced latent space, (3) an abstract action feasibility model that classifies whether the execution of an abstract action will be successful on particular abstract state, and (4) a policy model that maps the current raw state and the abstract action to raw action that the agent can directly execute in the environment. Unlike works that aim to simultaneously learn symbolic action and state abstraction (Konidaris et al., 201 8; Silver et al., 2021), our framework learns latent state abstraction as it allows us to capture important geometric constraints environments, such as whether the sink has enough space to place particular object.
Such state and action abstraction enables us, at test time, to use planning in the abstract action space to solve tasks. In particular, during test time, given the high-level task description (e.g., clean the bowl), we first translate it into a corresponding abstract action to execute (i.e., clean (ccaa)))))), and then search for an abstract action sequence that sets up the preconditions for the target action (e.g.. by clean up other objects in the sink and turn on the faucet), and finally, execute the target action.
PARL enables various types of generalizations, and we highlight some of them in Fig. lb. First, due to the adoption of an object-centric representation for both states and actions, PARL generalizes to scenarios with different number of objects than those during training, and generalizes unseen composition of action concepts and object concepts (e.g., generalizing from cleaning red plates and blue bowls cleaning red bowls). Second, the planning-based execution enables generalization to unseen sequences of abstract actions, and even to tasks that require a longer planning horizon.
In summary, this paper makes the following contributions. First, we propose to automatically discover anguage-amnotated
action abstractions from language- annotated demonstrations, without any manual definition of their names, parameters and preconditions. Second, our object-centric transition and feasibility model for abstract actions enables us to recover a latent state abstraction without the annotation of symbolic predicates, such as object relations or states. Finally, through the combination of our abstraction learning and planning algorithm, we enable generalization to different object numbers, different verb-noun compositions, and different planning steps.
2 RELATED WORK
Many prior works have explored learning abstract actions from imitation learning or reinforcement learning using the guidance of language. Language instructions can be given in the form of sequences of action terms (Corona et al., 2021; Andreas et al., 2017; Andreas & Klein, 2015; Jiang al., 2019; Sharma et al., 2021; Luo et al., 2023), programs (Sun et al., 2020), and linear temporal logic (LTL) formulas (Bradley et al., 2021; Toro Icarte et al., 2018; Tellex et al., 201 A common idea underlying these works is that language introduces a hierarchical abstraction over the action space that can improve the learning data efficiency and performance. However, they either focus on learning hierarchical policies focus on pure instruction following tasks (Sharma et al., 2021; Corona et al., 2021 1). By contrast, our model leverages the action abstraction to plan (via tree search) in the abstract action space for better generalization to more complex problems.
2

---

Published as a conference paper at ICLR 2024
Integrated model learning and planning is promising strategy for building robots that generalize to novel situations and goals. Specifically, Chiappa et al. (2017); Xu et al. (2019); Zhang et al. (2021), Schrittwieser et al. (2020); Mao et al. (2022) learn dynamics from raw pixels; Jetchev et al. (2013); Pasula et al. (2007); Konidaris et al. (2018), Chitnis et al. (2021); Bonet & Geffner (2020); Asai & Muise (2020); Silver et al. (2021); Zellers et al. assume access to the underlying factored states of objects, such as object colors and other physical properties. Our model falls into the first group where we learn dynamics model from perception inputs (point clouds in our case) and the dynamics model operates in a latent space. The primary difference between our work and other method in the first group is that instead of learning a dynamics model at the lowest primitive action level, we learn a transition model at an abstract level, by leveraging language instructions.
Another line of work has focused on learning latent transition models and planning with them (Shah et al., 2022; Wang et al., 2022; Huang et al., 2023; Agia et al., 2023). These models focus on learning latent transition model for a given set of primitive actions (by contrast, we discover these actions from language instructions), and their algorithm involves searching sampling continuous parameters for their actions (e.g., finding grasping and placement poses). By contrast, our goal is to learn abstract transition model in the latent space where the action set is purely discrete. This allows us to use simpler tree search algorithms for planning and furthermore, enables us to solve more geometrically challenging tasks such object placements with learned feasibility.
A general goal of our system is to build agents that can understand language instructions and execute actions in interactive environments achieve goals (Kaelbling, 1993; Tellex et al., 2011; 1; Mei et al., 2016; Misra et al., 2017; Nair et al., 2022). See also recent surveys from Luketina et al. (2019) and Tellex et al. (2020). Furthermore, the abstract action representation can be viewed as options in hierarchical reinforcement learning (HRL; Sutton et al., 1999; Dietterich, 2000; Barto & Mahadevan, 2003; Mehta, 2011), and is related to domain control knowledge (de la Rosa & Mcllraith, 2011), goal-centric policy primitives (Park et al., 2020), and macro learning (Newton et al., 2007). Our problem formulation is largely based on existing work on HRI., but we focus on discovering options from language and learning planaing-compatible abstract transition models.
3 PROBLEM FORMULATION
We study the problem of learning assume a fully observable environment, denoted as a transition model tuple AS, T) where S is the state space", .A is the action space, and : S x A - S is the transition function. Each task in the environment is associated with a natural language instruction L,- The instruction L,, could either be a simple goal description (e.g.. "clean the bowl") or sequence of steps (e.g.. "first pick up the red bowl from the countertop, then put it in the sink, and finally clean it"). The core insight of our approach lies in the assumption that each natural language description L, consists of abstract actions. Each abstract action a' E A' can be factorized as (w1, W2..... wk), where w1 is a verb symbol (e.g., "clean") and each remaining w; is object argument to the verb, represented by set of object-level symbols (e.g.. "red howl"). For example, the instruction "pick up the red bowl from the sink" can be factorized into (pick-up, red, bowl, sink).
-eggutee-onddi goal-reaching policy. In this work, we
Our training data is composed of wagguage-anootate demonstrations and interactions, each of which consists of state-action pairs and the corresponding Lt. We also assume that each trajectory has been segmented, such that the segments can be aligned to abstract actions in the instruction L,, To allow the system to understand the feasibility of different actions, our training dataset also contains failure trajectories. In particular, the execution of some trajectories may lead to failure, for example, in scenarios where the agent can not place the object in the target position, as the target space has been fully occupied by other objects. During test time, we evaluate our system on unseen instructions I., that contain single goal action (e-g.. "place the bowl into the sink"). Often the goal action is not immediately feasible (c.g., there are other objects blocking the placement); therefore the agent need to plan out additional actions before taking the goal action.
From the language-annotatee data, our objective is to learn policy with state and action abstractions. The abstractions essentially correspond to the abstract state space S', the abstract action space A', and the abstract transition function T'. Our state abstraction function á S + S' maps raw states to
We assume that the raw state space is object-centric; that means, each raw state E S will be represented by image a point cloud, together with the instance segmentation of all objects in the seene.
3

---

Published as a conference paper at ICLR 2024
Training Data
CompitibCC
object point clouds
Place the pan, then pick up the bows,
Absiract Model
Object concepts:
eg, shapes, colors.
(recogartion model)
High-Leve Lan-Lavel Policy
Robot Action
Learning
Action: (policy. fcesibility transition)
eg., pick, place
Goal Instruction:
"place the cabbage..."
Figure 2: The overall framework of PARL. PARL takes paired natural language instructions and demonstration trajectories as inputs. It recovers object-level concepts such as shapes and colors, and action concepts from the natural language. then learns Saaning-compatible model for object and action concepts. At test time, given novel instructions, it performs a combined high-level planning and low-level policy unrolling to output the next action to take.
Prompt
LLM Generation
Examples
[('aare:; "pick-up', 'args': [("red", "plate")]>, ('nane": 'place', 'args": ("red', 'plate"), ("sink")
Clean the plate
[[ [('name': "clean', 'angs': [('plate")))]
Clean the plato place
D1
[('name': 'clean", 'args': [('plate")]] )1>,
('nane': 'place', 'angs': (('plate"), ), ('cabinet')))
Extract Symbols
Object concepts: red, plate, sink Action concepts: pick-up, place
Instruction:
"pick up the red plate place the sink."
Figure 3: PARL prompts a pretrained large language model (LLM) parse instructions into symbolic formulas. Next, we extract the object-level and action concepts from the formulas.
abstract states. Similarly, function T' A' S' learns an abstract transition model. To facilitate planning and execution with these abstract actions a' E A', we additionally learn feasibility model fa + {0, J.. which indicates whether an abstract action is feasible at the abstract state, and a lewelevel policy model To S + A that refine each abstract action into primitive actions in A conditioned on the current environmental state.
4 PLANNING ABSTRACTION FROM LANGUAGE
Our proposed framework, planning abstraction from language (PARL), is illustrated Fig. PARL operates in three main stages: symbol discovery, planning-compattie planning and execution. We first discover a set of action and object concepts from the language instructions in the dataset by leveraging pretrained large language models (LLMs). Next, we ground these symbols using the demonstration data and interactions. In particular, we learn planning- compatible model consisting of the state abstraction, transition and feasibility for abstract actions, and low-level policy. During inference, given state and action abstractions, we translate the instructions into symbolic formulas represented using the discovered action and object concepts, and use search algorithm based on the learned transition and feasibility functions to generate plans and execute.
model learning, and test-time
4.1 SYMBOL DISCOVERY
In the first stage, we consider all L, instructions in the training dataset and employ pretrained large language models (LLMs) to discover a set of symbols. Each action in L,, subsequently translated into a symbolic formula (W), wi...., wx), ), which contain the discovered action and object concepts. Illustrated in Fig. 3, given the instruction Lt, we prompt LLMs to parse the instruction into a list of abstract actions, where each action consists of action concept (e.g., pick-up), and a list of arguments, where each argument is represented as a set of object concepts (e.g., red, plate). We provide detailed prompts in Appendix 1. In contrast to representing each action term textual sentence, the decomposed action and object concepts allow us to easily enumerate all possible actions an agent can take. This compositional approach become especially useful when an agent needs to take additional abstract actions to achieve language goal that is not immediately feasible.
4.2 PLANNING-COMPATIIN
MODEL LEARNING
In the second stage, we utilize the language-annotated demonstration data to ground the discovered action and object concepts on the raw environmental states. Specifically, we train a group of four models. First and foremost, state to the abstract state. Second, we learn an abstract transition function T' that models transitions in the abstract state space S'. Third, for each abstract action a', learn feasibility function fa
leam state abstraction function S S' that maps the raw
4

---

Published as a conference paper at ICLR 2024
Per-Ohject State Abstraction Function
(b) Abstract Transition Transformer: Transition and F Feasibility
h
fear,
Vector embedding for object
(++1)
Ag
(E+1)
(ignored)
Max Pooling
MLP
Abstract Transition Transformer
Object-Level PCT
(4) points object!
14
Ay
b49
Og
FEAS
Object Emboddings
Action Embeddings
Positional emboddings and type embeddings (for Abstract Transition Transformers)
FEAS
Feasibility Query Token
Figure 4: Neural network architectures for our planning-compatiby models, composed of (a) an object-level PCT encoder for extracting state abstractions and (b) an abstract transition Transformer for abstract transition and the feasibility prediction.
that determines whether an action a' is feasible in given abstract state. Fourth, for each action a', we learn a low-level policy function Th that maps abstract actions to specific actions in the raw action space A. In this section, we first describe an end-to-end model that unifies the state abstraction function, the abstract transition model, and the feasibility model. We present how the end-to-end model support plannning with abstraction in Section 4.3. We then present the details for the low-level policies in the Section 4.4. Below, we will describe our models for raw state space S represented by 3D point clouds, but similar ideas can be easily extended to other representations such as 2D images, which discuss in Appendix A.2. We present implementation details of our models in Appendix A.3.
State abstraction function. The state abstraction function 6 is an object-centric encoder, which encodes the raw state from a given point cloud. Specifically, a state is represented by list of segmented object point clouds (I, }, where N is the number of objects in the environment. Each object point cloud contains a set of spatial points and their RGB colors, i.e., 2; E R6x M where M is the number of points in each point cloud. We encode each point cloud separately using a Point Cloud Transformer (PCT; Guo et al., 2021) to obtain latent features for the objects (/..... ,hN).
Abstract transition model. The abstract transition function T takes the abstract state sf (h,, Ay (9) and the abstract action df as input, and predicts the next abstract state 6/41 (h...... ANw] (++1), As illustrated in Fig. 4b, this function implemented by an object-centric Transformer. The input to this function contains two parts: the abstract state representation for time step as a sequence of latent embeddings of the objects, and the encoding of the abstract action as sequence of token embeddings. The function predicts the abstract state representation time step + 1, which is the sequence of output tokens from the Transformer that correspond to the objects. Recall that abstract action is composed of verb name (e.g., place), and sequence of object-level concepts for each argument (e.g., red plate and countertop). We encode the factorized action (wi, :2,.... wx) ) as sequence of discrete tokens using learnable token embeddings. We positional embeddings to differentiate each element of the input sequence to the Transformer encoder.
Feasibility function. One important purpose of the abstract transition model is to be able predict the feasibility of future action. For example, we want the agent to understand that it needs to pick up an object before moving the object; and it needs ensure there is enough space in the target region such that it can place the object in the target region. We predict feasibility by integrating binary feasibility prediction module far with the aforementioned object-centric Transformer. Illustrated in Fig. 4b, we append feasibility query token to the combined sequence of object embeddings for sf and action token embeddings for d;- The output of the Transformer encoder at this query position will be used to predict binary label for the fcasibility of executing a; s/-- Specifically, we use another small MultiLayer Perceptron (MLP) module on top of the Transformer for prediction.
Training. Our training data contains paired language instructions and robot interaction trajectories. For some of the demonstrations, the execution of the last action may cause a failure, therefore indicating infeasible action. We use the successful and failed trajectories positive and negative examples, respectively, to jointly train our state abstraction function, the abstract transition model, and the feasibility model.
5

---

Published as a conference paper at ICLR 2024
st), In particular, each data point is in the form of the sequence {su. ".......'a.., where a'ss are the abstract actions parsed by the LLM, So is the initial state, and subsequent 8,-13s 's are the raw environment state after the execution of each abstract action af. For each data sequence, randomly subsample a suffix to obtain (8k, d , se). Next, we use & to encode the new initial state Sk as k = ((8x). Then we recurrently apply the Transformer based abstract transition model 8/41 = (s', af) and feasibility prediction model feas, = [(8,, d(), for all i k,, ,., For this data point, the loss value is defined
first
2
L
|s ((8))l2 + Σ (CE,eas,..eas.)) Σ k=++1
2=
where BCE is the binary cross-entropy loss, feas, is the groundtruth feasibility label. feus, is true for < f, and is true for i é if the trajectory execution is successful (i.e., the last action successful) or false otherwise. During training, we use stochastic gradient decent to optimize Φ T and fa', by randomly sampling a batch of data points.
4.3 PLANNING WITH LEARNED ABSTRACTIONS
Based on the learned abstract transition model and feasibility function, we use simple tree search algorithm to find abstract action sequences that can achieve the language goal. This is required when the given language goal only contains the last abstract action to achieve (e.g.. put the bowl into the sink.) Specifically, the search algorithm acts in a breadth- first search (BFS) manner; it takes the current raw state Su and finds sequence of high-level actions (a'...., ,xx) such that a's is the underlying abstract action in the language goal.
Given so, we first map it to latent abstract state s/1 using the state abstraction model. Then enumerate all possible ag E AA' (the first action) and compute their feasibility. Subsequently, the abstract transition model is applied to predict the subsequent abstract state s for all generated dg By repeating this strategy recursively we can generate the final search tree. For each generated action sequence dg. ... d'x and the corresponding feasibility scoresfeas, we compute the feasibility of the entire sequence as min; feas,, The algorithm expands the search tree for a fixed horizon, and returns the sequence that ends with the specified last abstract action with highest feasibility score.
To accelerate search, the algorithm prunes the search space at every iteration by considering only a subset of abstract action sequences based the feasibility scores of the sequence. This essentially resembles a beam-search strategy. As the computation for different states and actions can be parallelized, this algorithm can be efficiently implemented on GPU. During execution, we use closed-loop execution, where we replan after executing each action using the low-level controller.
4.4 LOW-LEVEL POLICY
Given the high-level plan computed by the search algorithm, our next step is apply low-level policies Ta S + A that conditions on the current raw state to refine an abstract action a' to a primitive action that the agent can directly execute the environment. Our framework is agnostic to the specific implementation of the low-level policy. For experiments in the 2D gridworld with simplified navigation actions and object interactions, we adopt the FiLM method fuse state and language information into single embedding (Perez et al., 2018), then apply an actor-critic method to learn the policy. For experiments in 3D environments that require more precise end-effector controls for object manipulation, instead of outputting robot control commands, we ask the low-level policy to output either an object to pick, or an object to drop the current holding object on (essentially giving the robot pick and place primitive). Both models trained with data sampled from Language-paires demonstrations and can predict actions directly from raw observations.
5
EXPERIMENTS
We evaluate models in two different domains: BabyAl Sidevvaler-Boisveed et al., 2019), 2D grid world environment, and
(Yang et al., 2023), a 3D robotic manipulation benchmark.
5.1 BABYAI
Environment. The BabyAl Al platform features a 2D grid world, in which an agent is tasked to navigate and interact with objects in the environment given language instructions. Each simulated environment

---

Published as a conference paper at ICLR 2024
Longer Steps
Novel Concept Combinations
Key-Door
All-Objects
Two- Cwo-Corridors
All-Objects
End-to-End BC
0.00
0.00
0.00
0.00
End-to-End A2C
0.00
0.00
0.00
0.00
Ours (high low)
0.45
0.39
0.52
0.27
High-Level BC
0.00
0.00
0.00
0.00
High-Level A2C
0.27
0.00
0,44
0.00
Ours (high only)
0.87
0.89
0.94
0.91
Table 1: Success rate of different models on AI We compare with behavior cloning and A2C baselines on two settings: (End-to-End directly predicting low-level actions, and (High-Leeell predicting high-level action tokens with oracle low-level policies.
contains four types of objects (key, door, ball, and box) with six possible colors. We focus on tasks involving two types of subgoals: 1) Pickup, which requires the agent navigate target object specified by its color and type and then pick up the object; and 2) Open, which requires the agent navigate to colored door and open it with a key matching color. The demonstration data are generated with the bot agent native to the platform. All models tested on this environment use factorized observation space, which encodes the objects and their properties in each grid cell.
Setup. We evaluate models on three task settings: Key-Door, All-Objects, and Two- Corridors. In Key-Door, an agent needs to retrieve a key in locked room. We design the environments such that the agent must first find a sequence of other keys to unlock intermediary doors before gaining access to the target key. Shown in Fig. 5a, for instance, to get blue key behind yellow-locked door, the agent first acquires red key to unlock a red door, revealing yellow key. The All-Objects setting is similar Key-Door, except that the goal can be any object types (e.g.. pick up the blue box, Fig. 5b). Compared to Key-Door, All-Objects requires the models to accurately ground more visual concepts and efficiently plan with larger action space. reach a target key that is at the end of a corridor (see Fig. 5c). To traverse each corridor, the agent needs to find keys to unlock intermediary doors. Each environment has two disconnected corridors, therefore requiring the agent to plan multiple steps into the future to find the correct path to the goal. We provide details for the settings in Appendix C.I.
Key-Door
All-Objects
Two-Corridors
Goal: pick blue key
Goal: pick green box
Goal: pick green key
We evaluate models diverse task set- tings created the BabyAl environments.
Figure 5:
the Two-Corridors setting, the agent needs to
Furthermore, evaluate two types of generalizations the aforementioned task settings. To test generalization to longer steps, we train the models on task instances that require at most abstract actions complete and test task instances that be solved with minimum of 5 abstract actions. To test generalization to novel concept combinations, we withhold data involving red key during training and evaluate on goals involving this unseen concept combination. For all experiment, evaluate on 100 task instances and report the average success rate.
Baseline. We compare our method with two learning-based baselines based on behavior cloning (BC) and advantage actor-critic (A2C). Both baselines use similar neural network backbones as our model, which processes the factorized observation with Feature-wiss wise Linear Modulation (FiLM; Perez et al., 2018). The behavior cloning model is trained to predict the action to take from demonstrations with cross-entropy loss, and the model is trained on the same demonstration dataset as our model. The A2C model has the exploration stage, where the agent can act itself in the environment and receive rewards. For a fair comparison with our bi-level planning and acting model, we studied two groups of models: End-to-End and High-Level. The End-to-End BC/A2C model is trained to directly predict low-level actions in the original action space (e.g., turn left, move forward, pick up an object), but compared to the baseline in BabyAI, we train it solve tasks that require significantly larger number of steps (e.g., the hardest benchmark in Chevalier-Boisvert et al. (2019) usually only involves unlocking one intermediate door before reaching the target). The High-Level BC/A2C model is trained to predict
7

---

Published as a conference paper at ICLR 2024
Novel Environments
Novel Concept Combinations
All-Objects
Sink
All-Objects
Goal-Conditioned BC
0.82
0.39
0.14
Ours
0.86
0.61
0.57
Table 2:
Success rate different models on Kitchen- Worlds. We compare Goal-Condi BC with our model generalization to novel concept combinations and environments.
abstract actions and uses manually scripted oracle controller to refine abstract actions to low-level commands. We provide implementation details for the baselines in Appendix B.I.
Result. We present the results on generalization experiments in Table 1. The End-to-End A2C failed solve the tasks due to the large low-level action space and the long-horizon horizon nature of our tasks. Since here we are only testing models on generalization to more complex tasks (they can all achieve nearly perfect performance on tasks seen during training). we found that self-explorations in the A2C model help the agent generalize to unseen scenarios. Our model (high+low) can efficiently explore the environment with learned abstraction. More importantly, we observe that our model (high+low) generalize the learned dependencies between actions (e.g.. to open a locked door, must first pick the corresponding key) to longer planning horizons, which crucial for completing challenging Two-Corridors tasks. We also evaluate model using the oracle low-level controller and observed a significant increase performance after decoupling planning and low-level control. The same trend was observed for the High-Level RL baseline, affirming the importance of learning planning-compatiin model instead of policy. We also observe very minimal performance drop when generalizing our models on new concept combinations. Since our model uses the network based on FiLM. it can decompose a language instruction to individual action and object concepts, therefore supporting compositional generalization to new combinations. We additionally study two variants of our method (high only) on generalization to longer steps for All-Objects. Without replanning, the success rate of the model decreased from 0.89 0.65. After replacing the FiLM network with MLP to fuse information from language and observation, the success rate dropped to zero, suggesting that disentangling concepts are crucial for robust generalization. In Appendix D.I, we show examples of the generated plans for all three task settings.
5.2 KITCHEN- WORLDS
Environment. The Kitchen-Worlds (Yang al., 2023) environments feature diverse 3D scenes and objects in 3D kitchen environment. It requires the models to ground visual concepts on 3D observations and predict action feasibilities that are affected by geometric features of the objects. Each environment consists of objects from six categories (medicine bottle, water bottle, food, bowl, mug, and pan) and six colors (red, green, blue, yellow, grey, and brown). robotic gripper used to move objects around five types of storage units and surfaces (sink, left counter, right counter, top shelf, and cabinet). The simulation supported by the PyBullet physics simulator (Coumans Bai, 2017), which also trajectories. We place six simulated cameras the environment to capture the whole environment. RGB-D images and instance segmentation masks rendered and converted point clouds.
verify the feasibility of the grasps, object placements, and motion
Setup. We test the models on two task settings: All-Objects and Sink. For both settings, instantiate the environments with random scene layouts, furniture and object assets, and initial object placements. In All-Objects, the task involves placing the target object at a desired location. The main challenge of this task is ground visual concepts involving diverse objects and locations, without explicit supervision (i.e., there is no classification label for object colors and categories). The optimal solutions involve one pick and one place action. The second Sink setting is designed to evaluate the models' abilities to reason about object shapes and environmental constraints. The solution could involve one two pick-and- --ddd--lacc actions. For example, shown in Fig. 7, given a sink that is occupied by large object, the agent needs to first remove the object from the sink, then
Figure 6: The kitchen environments
with 3D furniture and objects.
8

---

Published as a conference paper at ICLR 2024
Start
Goal
Action Sequence:
(sick-up, blue, pan, sink) (place, blue, par, left counter) (place, blue, right counter) (place, blue, rug, sinio
Figure 7: Given the language goal of placing blue mug in the sink, our model can reason about the geometry of the environment and perform planning with learned abstract latent space to predict the sequence of abstract actions. Each abstract action is expressed as verb term and object arguments. For example, (pick-up, blue, pan, sink) corresponds to picking up the blue pan from the sink.
place the target object in the sink. Given enough space, the object can be directly placed in the sink. We provide details for the settings in Appendix C.2.
We conduct generalization experiments the two task settings discussed above. We first evaluate whether the models can generalize to new environments with novel object instances. This is particu- larly important because the visual concepts are directly grounded on point clouds. We then evaluate whether the learned models can generalize to new task instances involving withheld object concepts, specifically red bowl. We evaluate 100 task instances and report the average success rate.
Baseline. We compare our method with Goal-Condinn Conditoodd BC, which is trained to predict the sequence of abstract actions from a starting low-level state conditioned on a language goal. The model is implemented as a Transformer Sincoder-Decoder, where the Encoder encodes object point clouds and the goal, and the decoder autoregressively decodes sequence of abstract actions including the verb and object arguments; details are presented in Appendix B.2. We augment the training data by sampling start and end states, analogous to hindsight goal relabeling (Andrychowicz et al., 2017). During inference, the model predicts the next best action and replan after each step.
Result. Both CondiCondi BC and our model perform well on generalizing to new object instances for All-Objects. Because the fraal-Conditioned BC model is trained relabeled goals, it can fail to find the optimal plan for a given goal, therefore leading to long execution sequences or time out. By contrast, our model, instead of imitating the demonstrations, learns the underlying abstract transition models. We observe a more significant performance difference between our model and the baseline on Sink. Our model can implicitly reason about the size and shape the occupying objects in the sink to predict whether directly moving other objects into the sink is feasible. This type of reasoning is performed implicitly in the latent space instead of requiring manually defined logical predicates. This result demonstrates that our model is able to recover state abstraction (e.g., whether an object can placed the sink) from paired language and demonstration data. Appendix D.2, we show that our model is able to accurately predict feasibilities for abstract actions, even 5 steps into the future. Figure 7 illustrates one example where the feasible plan is found. Furthermore, we see strong performance from models in generalization to novel concept combinations. These results demonstrate the advantage of our factorized state representation, where concepts of colors and object categories can be combined flexibly to support compositional generalization. We perform ablation study by replacing our object-centric transformer with FiLM network that operates on a scene-level abstract latent state and observed 6% decrease in success on the Sink setting. This result highlights the benefits of explicit object-centric representations for geometrically challenging tasks.
CONCLUSION
In this paper, we introduce planning abstraction from language (PARL), framework that leverages aunguage-annotater demonstrations to automatically discover state and action abstraction. We learn transition and feasibility models for abstract actions that can recover latent state abstraction without the annotation of symbolic predicates and generalize to unseen action-object compositions, and longer planning horizons. Currently, our method relies segmented trajectories where the actions in language instructions are aligned with the low-level commands. Future work can leverage automatic segmentation of trajectories (Sharma et al., 2021; Luo al., 2023). Another future direction is to integrate pretrained foundation models (Radford et al., 2021; Zhang et al., 2022; Wang et al., 2023) for object recognition with the factorized representation for generalization to unseen object concepts.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is in part supported by NSF RI #221 1258, ONR MURI N00014-2-1127700, ONR N00014- 23-1-2355, ONR YIP N00014-24-1-2117, ASOSR YIP TA9550-23-1-0117,, and Samsung.
REFERENCES
Christopher Agia, Toki Migimatsu, Jiajun Wu, and Jeannette Bohg. TAPS: Task-agnostic policy sequencing. In ICRA, 2023.
Jacob Andreas and Dan Klein. Alignment-base compositional semantics for instruction following. In EMNLP, 2015. 1,2
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In ICML, 2017. 2
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight Experience Replay. In NeurlPS, 2017.
Masataro Asai and Christian Muise. Learning Neural-Symbolie Descriptive Planning Models via Cube-Space Priors: The Voyage Home (To STRIPS). arXiv:2004. /2850, 2020. 3
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning.
Discrete event dynamic systems, 13117:41-77, 2003.
Blai Bonet and Hector Geffner. Learning Order- Symbolic Representations for Planning from the Structure the State Space. ECAI, 2020.
Christopher Bradley, Adam Pacheck, Gregory J. Stein, Sebastian Castro, Hadas Kress Kress-Gazit, and Nicholas Roy. Learning and planning for temporally extended tasks in unknown environments, 2021. 2
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision Transformer: Reinforcement Learning via Sequence Modeling. In NeurlPS, 2021. 16
Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Baby AI: First Steps Towards Grounded Language Learning With a Human In the Loop. In ICLR, 2019. 6,
Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent Environment Simulators. In ICLR, 2017. 3,3
Rohan Chitnis, Tom Silver, Joshua B Tenenbaum, Tomas Lozano-Pered and Leslie Pack Kaelbling. Learning
Relational Transition Models for Bilevel Planning. 1rXiv:2105.14077
2021. 3
Rodolfo Corona, Daniel Fried, Coline Devin, Dan Klein, and Trevor Darrell. Modular networks for compositional instruction following. In NAACL-HLT, pp. 1033-1040, 2021. 1,2
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation robotics, games and machine learning, 2017. 8
Tomás de la Rosa and Sheila Mcllraith. Learning domain control knowledge for tiplan and beyond.
In ICAPS / Workshop on Planning and Learning, 2011.
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposi- tion. JAIR, 13:227-303, 2000.
Malik Ghallab, Dana Nau, and Paolo Traverso. Automated Planning: Theory and Practice, Elsevier, 2004.
10

---

Published as a conference paper at ICLR 2024
Fausto Giunchiglia and Toby Walsh. A Theory of Abstraction, Artif. Intell., 57(2-3)3323-33855 1992.
Robert Givan, Thomas Dean, and Matthew Greig. Equivalence Notions and Model Minimization in Markov Decision Processes. Artif. Intell,, 147(1-2):163-223 2003.
Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pct: Point cloud transformer. Computational Visual Media, 7:187-199, 2021. 15
Yixuan Huang, Nichols Crawford Taylor, Adam Conkey, Weiyu Liu, and Tucker Hermans. Latent space planning for multi-object manipulation with environment-awer relational classifiers. arXiv preprint ar ArXiv:2305. 10857, 2023. 3
Nikolay Jetchev, Tobias Lang, and Marc Toussaint. Learning Grounded Relational Symbols from Continuous Data for Abstract Reasoning. In ICRA Workshop, 2013. 3
Yiding Jiang. Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019. 2
Leslie Pack Kaelbling. Hierarchical Learning in Stochastic Domains: Preliminary Results. In ICML,, 1993.
Siddharth Karamcheti, Suraj Nair, Annie S Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. RSS, 2023. 19
George Konidaris, Leslie Pack Kaclbling, and Tomas Lozano-Peree From Skills to Symbols: Learning Symbolic Representations for Abstract High-Level Planning. JAIR, 61:215-289, 2018.
Lihong Li, Thomas Walsh, and Michael I. Littman. Towards Unified Theory of State Abstraction for MDPs. In AI&M, 2006.
Weiyu Liu, Chris Paxton, Tucker Hermans, and Dieter Fox. Structformer: Learning spatial structure for language-guides semantic rearrangement of novel objects. In International Conference on Robotics and Automation, 2022. 16
Jelena Luketina. Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. In IJCAI, 2019. 3
Zhezheng Luo, Jiayuan Mao, Jiajun Wu, Tomás Lozano-Pérez, Joshua B Tenenbaum, and Leslie Pack Kaclbling. Learning rational subgoals from demonstrations and instructions. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023. 2, 9.9
Yecheng Jason Ma, William Liang, Vaidehi Som. Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. In ICMI., 2023. 19
Jiayuan Mao, Tomás Lozano-Péree Josh Tenenbaum, and Leslie Kaelbling. Pdsketch: Integrated domain programming, learning, and planning. Advances in Neural Information Processing Systems, 35:36972-36988 -669444 2022. 3
Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4): 1205- 11212, 2022. 16
Neville Mehta. Hierarchical structure discovery and transfer in sequential decision problems. Oregon State University, 2011. 3
Hongyuan Mei, Mohit Bansal, and Matthew Walter. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Proceedings of the AAAI Conference on Artificial Intelligence, 2016.
11

---

Published as a conference paper at ICLR 2024
Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. arXiv preprint wwXiv:170..08770 2017.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, 2016. 16
Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pp. 1303-1315. PMLR, 2022. 3
Muhammad Abdul Hakim Newton, John Levine, Maria Fox, and Derek Long, Learning macro-actions for arbitrary planners and domains. ICAPS, 2007. 3
Daehyung Park, Michael Noseworthy, Rohan Paul, Subhro Roy, and Nicholas Roy. Inferring task goals and constraints using bayesian nonparametric inverse reinforcement learning. In JMLR, 2020.
Hanna M Pasula, Luke S Zettlemoyer, and Leslie Pack Kaelbling. Learning Symbolic Models of Stochastic Domains. JAIR, 29:309-352, 350099352,000 2007. 1,3
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual Reasoning with a General Conditioning Layer. AAAI, 2018. 6, 7, 15,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In ICML, 2021. 9
David F Rogers, Robert D Plante, Richard Wong, and James Evans. Aggregation and Disaggre- gation Techniques and Methodology in Optimization. Operations Research, 39(4):553-582, 1991.
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. In Advances in Neural Information Processing Systems, 2022. 19
Julian Schritwiesser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. Nat., 588(7839):604-600) 2020.
Dhruv Shah, Peng Xu, Yao Lu, Ted Xiao, Alexander Toshev, Sergey Levine, and Brian Ichter. Value function spaces: Skill-centric state abstractions for long-horizon reasoning. In ICLR, 2022. 3
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. arXiv preprint arXiv:2
2021. 2,9
Tom Silver, Rohan Chitnis, Joshua Tenenbaum, Leslie Pack Kaelbling, and Tomas Lozano-Perez. Learning Symbolic Operators for Task and Motion Planning. In IROS, 2021. 1, 2,
Shao-Hua Sun, Te-Lin Wu, and Joseph J Lim. Program guided agent. In ICLR, 2020.
Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning. Artif. Intell., 1 112(1-2):181-21111 1999. 1, 3
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI, 2011. 2,3
Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3:25-55, 2020. 3
Rodrigo Toro Icarte. Toryn Q. Klassen, Richard Valenzano, and Sheila A. Mcllraith. Teaching multiple tasks to agent using ltl. In AAMAS, 2018. 2
12

---

Published as a conference paper at ICLR 2024
Chen Wang, Danfei Xu, and Li Fei-Fei. Generalizable task planning through representation pretrain- ing. IEEE Robotics and Automation Letters, 7(3):8299-83000 2022. 3
Renhao Wang, Jiayuan Mao, Joy Hsu, Hang Zhao, Jiajun Wu, and Yang Gao. Programmatically Grounded, Compositionally Generalizable Robotic Manipulation. In ICLR, 2023.
Ruocheng Wang. Jiayuan Mao, Samuel J Gershman, and Jiajun Wu. Language- Language-mediater mediated, object-
centric representation learning. In ACL Findings, 2021. 19
Danfei Xu, Roberto Martin-Martin, De-An Huang, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Regression planning networks. Advances in Neural Information Processing Systems, 32, 2019.
Zhutian Yang, Caelan R Garrett, Tomás Lozano-Pérez, Leslie Kaelbling, and Dieter Fox. Sequence- Based Plan Feasibility Prediction for Efficient Task and Motion Planning. In RSS, 2023.
Rowan Zellers, Ari Holtzman, Matthew Peters, Roozbeh Mottaghi, Aniruddha Kembhavi, Ali Farhadi, and Yejin Choi, Piglet: Language grounding through neuro-symbolic interaction in a 3d world. In ACI. 3
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning Invariant Representations for Reinforcement Learning without Reconstruction. In ICLR, 2021. 1,3
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision- language understanding. In Advances in Neural Information Processing Systems, 2022. 9
13

---

Example: Clean the plate. Published as a conference paper at ICLR 2024
Supplementary Material for Learning Planning Abstractions from Language
The appendix is organized as the following. In Appendix A, we specify the prompts we use for PARL as well as PARL's 's model architecture in 2D and 3D. In Appendix B we discuss the baseline implementations, and in Appendix C we describe the environments and data collection strategy for BabyAl and Kitchen- Worlds. Appendix D, we provide additional experiments. We discuss the limitations of our method and future directions in Appendix E.
A PARL
A.1 PROMPTS FOR LLMS
We provide examples of the prompts we use for the GPT-4 LLM below. Recall that each natural language L, is parsed to one or multiple abstract actions. Each abstract action is represented as sequence of symbols (W1, W2,.... wk). where W1 is verb symbol (e.g., "clean") and each remaining W; is an object argument to the verb, represented by set of object-level symbols (e.g., "red bowl").
Maр each natural language instruction to symbolic actions. Each symbolic action should contain the name and its arguments.
Output: [[ (nnamme: 'clean', 'args': [('paate }11]
Example: Clean the blue plate and place it into the sink
Output: [[ nname' 'clean', 'args': II ['blue', Splate'Hlh
[ ('name': 'place', 'args': II (bluee', 'plate' 1 |'cabinet' 312]
Example: Pick the red plate and place it into the sink. Output: [[ : upick_up', 'args'; [''rd'' , (plate'Hll ( ('name'; 'place', 'args': [('red', 'plate'), }, ('sink'
Example: Pick up the blue mug from the sink, place it in the cabinet. Then pick the brown pan from the counter, and place it into the sink. Output: [[ I'namee: : 'pick_up', 'args': : b'uue'', 'mug'h }, ('sink']]]
I'name : 'place', 'args': [I [blue', 'mug' 1, ('cabinet' }]},
{ Inamee 'pick_up' up', 'args'; ; 11 'brown', 'pan' b, ('counter' 11 ( ('name': : 'place', 'args": [[ [[brown', 'pan'), h, ('sink
Listing 1: Example GPT-4 prompt to map natural language instructions to symbolic formulas that our model consumes. Prompt context is in gray, input language descriptions green, and generated word tokens are purple.
A.2 MODEL ARCHITECTURE FOR BABYAI
We describe the model architecture for the Baby grid world below. The model comprises the state abstraction function, the abstract transition model, and the feasibility model. The model can be trained end-to-end with the objectives described in Section 4.2.
State abstraction function. The state abstraction function encodes the raw state 8, to latent feature map 8/ Hx where H and W are the height and width of the grid world, and C is the dimension of the latent feature. In particular, the raw environment state is represented by 13 13 grid, where each cell stores three integer values representing the indices the contained object, its color, and its state (e.g., locked). A convolutional neural network (CNN) is used to process the raw environment state and map it to the latent feature map.
Abstract transition model. The abstract transition function T' takes the abstract state s/ R Hf x WCC and the abstract action af as input, and predicts the next abstract state $ E R #xxxx For given abstract action, encode its factorized representation (W1, W2, ... ,KK sequence
14

---

Published as a conference paper at ICLR 2024
of discrete tokens using learnable token embeddings. To predict the next latent state s'41, a total of K FiLM layers (Perez et al., 2018) to modulate the latent feature map representing the current abstract state s/. Each FiLM layer is conditioned on the latent token of the symbol w; in the factorized action.
Feasibility function. The feasibility function predicts whether abstract action of can be performed in an abstract state sy. Similar to the abstract transition model, K FiLM layers are used to fuse the latent tokens of the abstract action with the latent state. The resulting feature map is then flattened and passed to Multi-Layer Perceptron (MLP) module with two linear layers to predict the binary feasibility for the abstract action.
A.3 MODEL ARCHITECTURE FOR KITCHEN- WORLDS
Below we provide details for the end-to-end model based on an object-centric Transformer that unifies state abstraction, abstract transition, and feasibility prediction.
State abstraction. For a given raw environment state St, The state abstraction function encodes the corresponding sequence of N segmented object point clouds { ... xx xx1 where each point cloud contains M number of zyz points and their RGB colors, i.e., I; E R6XM We use learned encoder h to encode each object point cloud separately as ha (I;). This encoder is built on the Point Cloud Transformer (PCT; Guo al., 2021). We treat the encoded object latents the latent abstract state, i.e., (x1)
(xx)) [h1,... ..N) )()
Tokenization. The abstract transition model and feasibility function are both conditioned on the given abstract action dy We convert the factorized representation of an abstract action (wi, W2, ... wxx ) to sequence of latent tokens using learnable word embeddings. Specifically, we convert wy to latent token hww (w;), for i = 1,... K.
Auxiliary embeddings. We use learned embedding for the query token h2 We use a learned position embedding pos (1) to indicate the position index of the object point clouds, action tokens, and query token in input sequences to the subsequent transformer. We use learned type embeddings heype (T) to differentiate the object embeddings (T = 0), action embeddings (r 1), and query embedding (r 2).
Object-centric transformer: The object-centric transformer serves two purposes. First, given a latent abstract state s/ = {h1,- ... hx] (() and the abstract action af as input, and the transformer predicts whether the abstract action is feasible at this time step. Second, given the latent abstract state and feasible abstract action, the transformer predicts the next abstract state 4441 (hi,-.. ... ,,x (1+1),
To predict feasibility, the input to the transformer consists of the object part e, the action part e, and query part Specifically, these are
e [hg; hpos (i); hegpe (0)],
Iw (w:); ); hppo (i N); heype (1)],
q = Ihg hpos
K); heype (2)]
where [:] is the concatenation at the feature dimension. The transformer takes as input the concate- nated sequence {#1," CN CI, CK: (0) The output token of the transformer at the query position is fed into small Multi-Layer Perceptron (MLP) predict the binary feasiblity label for the action af.
To predict the next abstract state, the transformer ignores the query part of the input using the attention mask, and only takes in (e., , , CI,... Cx (1), The output tokens of the transformer the position of the objects taken to be the latent abstract state for the next time step (h,.... ,,v) (t-1))
To batch plan sequences with different lengths and with different number of objects, we zero embeddings at the padding positions and attention masks for the transformer to avoid the effect of these padded inputs. We do not compute loss for padded actions in plan sequences.
Batching.
Parameters. We provide network and training parameters in Table 3.
15

---

Published as a conference paper at ICLR 2024
Parameter
Value
Number of points for object point cloud M
512
Max plan lengths
6
Max number of objects Max
11
Length of factorized action K PCT point cloud encoder lt out dim
4
352
Word embedding vocab size Word embedding hew Word embedding lig dim Position embedding Apos Position embedding ppos dim Type embedding heype Type embedding heype dim Transformer number of layers Transformer number of heads Transformer hidden dim Transformer dropout
23
learned embedding
352
learned embedding
16
learned embedding
12
12
384
0.1
Transformer activation
ReLU
Loss
Focal (? 2)
Epochs
2000
Optimizer
Adam
Learning rate
le-4
Gradient clip value
1.0
Batch size
32
Learing rate warmup
Linear (Tend 20)
Learning rate decay
Cosine annealing (T (Tmaz= = 2000)
Table 3: Model Parameters
B BASELINES
B. RL MODELS FOR BABYAI
We compare method with two RL baselines based on Advantage Actor-Critic (A2C; Mnih et al., 2016). The End-to-End RL baseline takes in a partial and egocentric view the environment represented in 7 7x feature map and directly predicts turn left, turn right, move forward, pick up an object, drop an object, toggle, terminate). The model uses convolutional neural network to process the raw environment state and use FiLM layers (Perez et al., 2018) to condition the model on the input language instructions. The High-Level RL bascline predicts high-level abstract actions instead of low-level actions. The predicted high-level abstract actions are executed using oracle controller provided by the BabyAl simulator. This model observes the raw environment state as a 13 13 x 3 feature map and predicts an integer index representing the selected abstract action. For both baselines, a non-zero reward is credited to the agent only when the agent reaches the language goal. The reward adjusted based on number of low-level steps the agent takes to encourage efficient goal-reaching behavior.
of the seven low-level actions (i.e.,
B.2 GOAL-CONDITIONED BEHAVIOR CLONING FOR KITCHEN-WORLDS
This baseline takes in the current raw environment state 8; {=1..... z (9) and the goal abstract action d' 'ygoal (wi, W2, wx) input, and predicts the next abstract action execute af (W), ".. ... wx) (()) The model design this baseline is related to transformer-basee models (Chen et al., 2021; Liu et al., 2022; Mees et al., 2022) but different in the specific implementation choose for our domain. This baseline uses the same object point cloud encoding, action tokenization, position embeddings, and type embeddings
model, which are discussed in Appendix A.3. Different from the Transformer encoder that is used by our model, this bascline uses the Transformer encoder-decoder architecture. The encoder encodes the concatenated sequence of the object latents
16

---

Published as a conference paper at ICLR 2024
for it and action latents for dooa' and the decoder autoregressively decodes each element w; of the abstract action dzz
C ENVIRONMENTS
C.I BABYAI
Below we present details for the data collection procedures and evaluation setups of the task settings Key-Door, All-Objects and Two-Corridors below. The involved concepts are listed in Table 4.
Type
Value
Verb
Pick-up. Open
Object
Key, Door, Ball, Box Red, Green, Blue, Yellow, Grey, Purple
Color
Table 4: BabyAI Concepts
Key-Door. To collect training data for this task setting, we vary the environments by randomizing the placements of the doors for the 9 rooms in the grid world. During initialization, lock each room and place the key of the locked door in another room. As such, to solve the task, the agent must first find sequence of other keys unlock intermediary doors before gaining access to the target key. For each scene, we put in 3 keys for 3 locked doors and 3 distractor objects. For each scene, we collect demonstration trajectory by first searching for the sequence of high-level actions to reach the goal using task planner and then mapping the high-level actions to low-level actions, which can be executed by an oracle controller. In total, we collect 100,000 demonstration trajectories. For evaluation, the same randomized procedure is used to initialize the testing scenes. A trial considered successful only when the agent reaches the language goal within a limited budget of 200 low-level steps.
recursively
The environments are randomized using the same procedure described above. Different from Key-Door, this task setting requires the agent to locate target object that can be any of the following object types: key, ball, and box. Each scene contains keys for 3 locked doors, 3 distractor objects, and 1 target object. In total, collect 100,000 demonstrations using the combination of the high-level task planner and the low-level oracle controller. For evaluation, the same randomized procedure is used to initialize the testing scenes. A trial is considered successful only when the agent reaches the language goal within a limited budget of 200 low-level steps.
All-Objects.
Two-Corridors. To collect training data for this task setting, vary the environments by randomiz- ing the placements the doors for the 9 rooms in the grid world. We initialize each environment by constructing corridors such that the agent can traverse each corridor by unlocking intermediary doors and that the two corridors are disconnected from each other. Each corridor consists of locked doors with matching keys. Each scene contains total of key objects and distractor objects. We collect 100,000 demonstrations using the combination of the high-level task planner and the low-level oracle controller. For evaluation, the same randomized procedure is used to initialize the testing scenes. trial is considered successful only when the agent reaches the language goal within limited budget of 200 low-level steps.
C.2 KITCHEN- WORLDS
Below we present details for the data collection procedures and evaluation setups of the task settings All-Objects and Sink below. For both settings, the involved concepts are listed in Table 5.
All-Objects. To collect training data for this task setting, we randomly initialize the environments with different 3D models and placements of the furniture (e.g., cabinet, counter, and sink). Examples of the randomized environments are shown in Figure 6. For each scene, we randomly place up to objects in the scene with randomized initial positions. For each scene, collect three trajectories each containing 6 abstract actions by randomly choosing a feasible action to execute at each step and simulating with the Pybullet simulator. To collect failure cases, at each step, also randomly
17

---

Published as a conference paper at ICLR 2024
Type
Value
Verb
Pick, Place
Object
Bowl, Mug, Pan, Medicine Bottle, Water Bottle, Food
Location
Left Counter, Right Counter, Top Shelf, Sink, Cabinet Red, Green, Blue, Yellow, Grey, Brown
Color
Table 5: Kitchen- Kitchen-Worlds Concepts
execute action and record whether the action is successful failed. This procedure gives us failed trajectories with different lengths. In total, we collected interaction trajectories in 3920 scenes. For evaluation, the same randomized procedure is used to initialize the testing scenes. If the model fails to complete the goal abstract action in 5 steps, we consider the trial to be unsuccessful. Note that for this task setting, the optimal solution involves abstract actions pick and place action.
Sink. To collect training data for this task setting, we randomly initialize the environments as described above. For each scene, we randomly place up objects in the scene and choose random subset of the the objects to place in the sink. For each scene, we collect three trajectories each containing 6 abstract actions. At each time step, we choose sink-related action with 50% chance (e.g., pick object from the sink or place an object into the sink). In total, we collected interaction trajectories 3360 scenes. For evaluation, the model fails to complete the goal abstract action in steps, consider the trial to be unsuccessful, For this task setting, the optimal solution can involve up to four abstractions.
D ADDITIONAL RESULTS
D.1 BABYAI QUALITATIVE EXAMPLES
We provide generated plans in Figure 8. Our model is able accurately generate multi-step plans for diverse environments and language goals.
Key-Door
(b) All-Objeets
(c) Two-Corridors
pick up red key
pick up purple key open purple
pick up red key open
open
pick yellow key
pick yellow key open yellow pick up yellow box
pick key
open yellow
open grey
pick blue key
pick green key
pick up green key
pick up yellow key open yellow door pick up purple key open purple door pick up
up yellow key
2. open green door pick up key
open yellow door
pick up purple key
4. open blue dooe pick up yellow key
open purple door
pick up grey key
Figure 8: Plan predictions for different task settings. Each figure visualizes the initial state of the environment and the location of the agent (represented by the red arrow). On the right of each figure, sequence of planned abstract actions is listed. The last abstract action each list is the goal abstract action in the given language instruction.
D.2 FEASIBILITY PREDICTION
We present a breakdown of feasibility prediction performance below. In general, we observe that model is able to accurately predict feasibilities for abstract actions, even 5 steps into the future. The model performs well for both the task setting that requires grounding diverse visual concepts (i.e., All-Ohjects) and the one that requires geometric reasoning (i.e., Sink). We observe decrease
18

---

Published as a conference paper at ICLR 2024
in performance when the model generalizes to concept combination that is never seen during training. However, even in this challenging case, the average Fl score is still above 90% on the testing data.
Train
Test
Task Setting
Generalization
Avg.
Avg.
k
k: 1
k: + 2
k + 3
+
k + 5
All-Objects
New Env.
99.70
98.17 17
99.00
98.51
97.79
97.32
96.45
98.89
Sink
New Env.
99.84
98.01
98.13
98.55
98.10
97.38
97.63
97.01
All-Obbects
New Concept Comb.
91.97
90.08
88.74
90.36
87.67
90.24
88.68
99.01
Table 6: Feasibility prediction Fl scores for Kitchen- environments. For each setting, the
scores on the training data and testing data are presented. For the testing data, we further present the feasibility prediction performance for abstract actions different future time step at given only current environment state 8k, for i k. ... 5.
E LIMITATIONS
For the 3D environment, we assume the instance segmentation is provided so that can extract object point clouds. Future work can explore the use of unsupervised object discovery (Sajjadi et al., 2022: Wang et al., 2021).
We assume that the input language instructions can be parsed into symbolic formulas following fixed form. We further assume that the input language instructions are complete. Future work should relax these assumptions and operate on more diverse forms of language instructions.
In this work, we learn state abstraction and action abstract from language language-annotatee annotated demonstrations. In order to scale to large set of action and object concepts, our Mlaniig-compatiabbe models can be integrated with pretrained laiiuagl representations (Ma et al., 2023; Karamcheti al., 2023) to bootstrap the learning of visual groundings and geometrie reasoning.
19


Reviews:
[{"review_id": "m0xDdsfG0S", "reviewer": "Reviewer_Q2SJ", "summary": "This paper presents an RL agent that solves the problem by utilizing symbolic abstraction and object-centric representation learning. Given a problem and a goal description for the environment, the proposed method uses LLM as a parser to translate the goal description (or instruction) as a collection of action predicates (verb and objects combination), where those action predicates are used as action abstraction (options or skills in hierarchical reinforcement learning).\n\nTraining requires human demonstration annotated with an action predicate. Given the demonstration data, the proposed method trains several functions. State abstraction function takes in the segmented output of a point cloud transformer to disentangle pixel input and it provides a latent space state representation. Action transition model takes in the latent abstract state and symbolic encoding of action predicates. Feasibility function predicts whether an abstract action is applicable in the current state.\n\nAfter training necessary functions with annotated demonstration data, the planning stage uses the state abstraction function to get a latent state and utilize the feasibility function to select applicable actions. Planning with symbolic action predicates is done by brute-force search over all actions in the problem. Last,  given a high-level plan, the low-level policy is trained.", "questions": "### General questions\n1. The title is “learning planning abstractions from language.” In the paper, the role of the LLM is parsing an instruction sentence to extract action predicates and objects. The remaining part of the work is independent of language models or language. The parsing could have been done manually or other methods. I cannot see the rationale of using LLM, other than demonstrating that LLM can do the parsing. What is “learned” from language?\n\n2. What if the goal description or instruction did not reveal enough information to extract required high-level actions? Then, should we collect demonstrations following the derived high-level actions?\n\n3. The instructions in the paper are quite simple sentences to parse. What is the longest abstract plan needed to solve the problem? How many abstract actions were needed?\n\n### Section 4\n4. In section 4.2, how the model was trained given annotated demonstrations? Is it learned per each abstract action? How many trajectories were given to the training process? How did you train Point Cloud Transformer for mini-grid environment and kitchen world environment? Can you present the details on the training of models?\n\n5. In section 4.3, planning is done with BFS search. If the feasibility prediction fails, how did you handle the error? Does a set of actions derived from LLM parser always guarantee to solve a problem? How can you ensure the action space can solve all problems in the test set? Or a human demonstrator should create trajectories that solves problem given the action predicates?\n\n6. In section 4.4, low-level policy is trained using an actor-critic algorithm. Can you present the details?\n\n### BabyAI experiments.\n\n7. The report on BabyAI experiments shows that baseline will not completely fail in all problems. The presented paper and the report also used similar low-level policy algorithm and neural network architectures. It also offers imitation learning experiments. Can you make some comparison with those baselines? What are the high-level actions extracted from LLM and what are the plans found for the problems? Can you present the sample efficiency or training/test performance?\n\n[1] Hui, D. Y. T., Chevalier-Boisvert, M., Bahdanau, D., & Bengio, Y. (2020). BabyAI 1.1. arXiv preprint arXiv:2007.12770.\n\n### Kitchen Worlds experiments.\n8. How many high-level actions were annotated and trained to solve this environment?  From the description in the experiment section, the length of the plan is mostly one or two. Could you present details on the high-level plan and the low-level policies?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Originality: The novel aspect of the presented method is combining symbolic planning with action predicates extracted from natural language goal descriptions or instructions and latent space representation learning with point cloud transformers. The abstract planning is done at the symbolic level, and the abstract state transitions are tracked with object-centric representation learned from segmented pixel data.\n* Quality: The overall description of the method is easy to understand and the experiment was conducted on two types of the environments.\n* Clarity: Figures help understanding the overall approach\n* Significance: I think this is an interesting work that integrating many things to work.", "weaknesses": "* Originality: Individual components are existing approaches and the originality is on bring those components and implement an agent to solve mini-grid and kitchen world problems.\n* Quality: Due to missing details, it is difficult to assess the quality.\n* Clarity: There are many missing details in the paper.\n* Significance: The comparison is made only against a simpler baselines (end to end RL and behavior cloning).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "afYcjwCM4p", "reviewer": "Reviewer_B6My", "summary": "The paper proposes a framework for solving problems in sequential-decision making by combining LLM-generated high-level abstract actions, imitation learning and a low-level policy by a framework-agnostic traditional RL agent.\n\nThe pipeline in more detail is that given a prompt in human language which defines “a language goal”, an LLM decomposes to a verb and corresponding nouns and adjectives (e.g. ‘place’, ‘bowl’, ‘green’), with the assumption that these prompts can always be decomposed to this format. After this,\n\n- a state abstraction function is learned that can identify the objects in the environment\n\n- an abstract transition model is learned which predicts the next state given the current abstract state and high-level action. This model also has a feasibility component that predicts whether a future action can accomplish the language goal\n\n- A breadth-first search algorithm selects the shortest sequence of actions that accomplishes the language goal\n\n- Finally, a low-level policy is applied according to the sequence of high-level actions. These policies are learned with traditional RL\n\nThe paper tests the method on two environments: BabyAI (three task setting) and Kitchen-Worlds (two task settings), compare against low-level (regular) and high-level (when the agent has access to the defined ) RL in the former and Goal-Conditioned BC in the latter.", "questions": "In the state abstraction function, how do you know when to stop training to have the right number of point clouds? Or else, do you assume that the number of objects in the environment is known beforehand? Is this end-to-end trained? (I think the paper would benefit if these points were made clearer there.)\n\nWhat kind of pooling operation do you use? Max or average? And what is the dimensionality of the per-object latent feature?\n\nHow is the generalization success rate actually calculated? What does it mean that an agent “fails to solve the tasks”? I am assuming given the allowed number of steps?\n\nFor the loss of the feasibility prediction model, do you use the L_2 norm?\n\nThis is just notation, but in the reinforcement learning literature s’ usually denotes the next state, and $\\hat{s}$ is used for an approximate state. Why did you decide to change this convention?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality: the paper proposes a novel way to solve sequential decision making problems by combining LLM prompting, imitation learning and traditional reinforcement learning.\n\nQuality: the paper places the work in the literature very well, comparing the differences between previous works and mentions future work. The problem formulation is mostly clearly written up.\n\nClarity: The paper is mostly well-written and apart from a few inconsistencies, easy to understand.\n\nSignificance: its originality could be considered significant.", "weaknesses": "I have three main reservations:\n\n- There is no available code, no experiment details (chosen hyperparameters, tuning) about the algorithm or the baselines and as such the results are not reproducible\n\n- The experiments themselves, the results, and the metrics are described in a very high level without details which does not allow the reader to indeed verify how well they support the claims.\n\n- Scalability: as the number of actions, objects and their combinations increase, the necessary training data size increases intractably (combinatorial explosion). I am concerned that this approach might be feasible for simple problems only due to its inherent limitations.\n\nFurthermore, there are a few things that are unclear to me which could be further weaknesses. (I’ve listed the questions in the next section.)\n\nMy initial rating is due to the above reasons. I would be willing to increase the score if the above concerns are addressed adequately.\n\nClarity issues:\nThe notion of “tasks” is not defined in the problem formulation. I understand this is not easy to do, but including it would make the paper stronger. The expression “language goal” is also used a few times throughout the paper, without definition (and I believe tasks = language goals)\n\n4.2 third paragraph third sentence. Did you mean to say something along the lines of “maps the abstract state representation at the current step and an abstract action $a’$ to the next abstract state $s’_{t+1}$” ?\n\nTypos/style issues that did not affect the score:\n\nIntroduction\nFirst paragraph; A “good” state and action representations -> Remove “A”\n“As the state abstraction extracts relevant information about the actions to be executed” no need for the first “the”, and could you back this up with some sort of example in brackets, ideally with a citation (I can sort of guess what you mean, but I find this a bit too vague and unclear) \n\nThird paragraph: “an particular object” -> a particular object\n\n4th paragraph: “that setups” -> that sets up\n\n4.2\n\nSecond paragraph:\n“Which encodes the raw state from a given point cloud”: from -> to\n“By applying a pooling operation for the point cloud” for -> to\n\nThird paragraph:\n\n“The abstract transition function $\\tau’$ takes the following form. It maps [...] -> remove “takes the following form. It”\n\n5th paragraph\n\n“We appends” -> we append\n\n“Of the Transformer encoder at this query possible will be used” -> no need for possible?\n\n4.3\n1st paragraph: last sentence is not needed, or its information content should be moved towards the end of the second paragraph. E.g. “we can generate the final tree, the leaves of which are $a’_K$ which correspond to the underlying abstract action in the language goal”.\n\n2nd paragraph: subsequence -> subsequent\n\nsearchs ->searches\n\n5\nEXPERIMENT -> EXPERIMENTS\n\n5.2\n\n2nd paragraph: “designed to evaluate models’ abilities” -> missing the after evaluate\n\nCitation for the Goal-Conditioned BC baseline is missing.\n\n6\n\n“A framework that leverage” -> leverages", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qBFSdARy55", "reviewer": "Reviewer_V9Cw", "summary": "The paper introduces a framework for learning state and action abstractions from language-annotated demonstrations. The abstract actions and states are use to train a transition model in the latent space to learn the feasilbility of newer latent actions. This allows agents to generalize actions learned from language to longer, unseen tasks.", "questions": "No questions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originalty:** The paper investigates a problem is not tackled in the literature but can realistically exist. The paper is a novel and creative framework for addressing this problem.\n\n**Clarity:** The paper is well-writtten.\n\n**Significane:**  This work has the potential to be impactful in language-based agent interactions. Furthermore, the framework can be adapted to other sequential planning domains. \n\n**Quality:** The problem described is well-motivated. The approach to addressing the problem is laid out clearly and simply and it reads reasonably. The model framework is creative and intuitive. The experimental design is sound and makes sense to test their claims and results support the claims made by the authors.", "weaknesses": "I don't have any major gripes. However, I found the description of the experimental domains lacking. Particularly I am not totally clear on the difference between the key-door and two-corridor environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HRFypKT79y", "reviewer": "Reviewer_NUXY", "summary": "This work proposes a framework that learns state and action abstractions for planning. It does this by leveraging demonstrations with corresponding language annotations. These demonstrations are used to discover actions, which in turn is used to generate state abstractions. Finally, low-level policies are also learned corresponding to the high-level actions. The evaluation is done on two domains and the results show generalization wrt environments and objects.", "questions": "1. Who provides the examples for prompting in Fig. 3 left?\n2. The training is performed on how many tasks? Was the environment structure the same for all of them? Or was it changed in between tasks? If it was changed, was it ensured that the test environment configuration was not present in the training set? \n3. What is the reason for not achieving 100\\% accuracy for novel concept combinations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written. Except for the algorithm description, other parts like motivation, problem formulation, etc., are explained nicely.\n2. The approach seems to be novel.", "weaknesses": "1. Other related approaches: \n* There are related approaches that use language to guide the abstraction process. E.g., Peng et al., the difference is that human input. Here, this paper gets it in the form of language-annotated task descriptions.\n* Approaches like LIV (with PointCLIP instead of CLIP) can learn a latent representation, which can be used for planning. \n\nPeng et al., Learning with Language-Guided State Abstractions.\n\nLIV: Ma et al., LIV: Language-Image Representations and Rewards for Robotic Control.\n\nCLIP: Radford et al., Learning Transferable Visual Models From Natural Language Supervision.\n\nPointCLIP:  Zhang et al., PointCLIP: Point Cloud Understanding by CLIP.\n\n\n2. Reproducibility:\n* I am not sure how reproducible the work is. There are a large number of details that are swept under the rug. And without an algorithm, it gets difficult to follow the paper. The supplementary material is also not submitted. \n* The inputs are not clear. \n\n3. Experimental Evaluation:\n* I would suggest performing experiments for the accuracy of the feasibility function. \n* The experiments from the grid-like BabyAI setup are not convincing of generalization. The paper claims to withhold \"red key\" in training, but they can learn the model agnostic to such properties. So, this is more of a verification that their approach works. But as we can see, the accuracy for novel concept combinations is only 91\\%.\n\nMinor points:\n* Incorrect citation: I do not believe Silver et al. learn (invent) new predicates as stated in the last two lines of page 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)