Compositional Instruction Following with Language Models and Reinforcement Learning
Authors: Vanya Cohen
Geraud Nangue Tasse
Nakul Gopalan
Steven James
Matthew Gombolay
Ray Mooney
Benjamin Rosman
Venue: Accepted by TMLR ()

Abstract: Combining reinforcement learning with language grounding is challenging as the agent needs to explore the environment while simultaneously learning multiple language-conditioned tasks. To address this, we introduce a novel method: the compositionally-enabled reinforcement learning language agent (CERLLA). Our method reduces the sample complexity of tasks specified with language by leveraging compositional policy representations and a semantic parser trained using reinforcement learning and in-context learning. We evaluate our approach in an environment requiring function approximation and demonstrate compositional generalization to novel tasks. Our method significantly outperforms the previous best non-compositional baseline in terms of sample complexity on 162 tasks designed to test compositional generalization. Our model attains a higher success rate and learns in fewer steps than the non-compositional baseline. It reaches a success rate equal to an oracle policy's upper-bound performance of 92%. With the same number of environment steps, the baseline only reaches a success rate of 80%.

Published in Transactions on Machine Learning Research (12/2024)
Compositional Instruction Following with Language Models and Reinforcement Learning
Vanya Cohen
vanyo@uteras.edi
The University of Texas at Austin
Geraud Nangue Tasse*
University of the Witwalersrand
Nakul Gopalan Arizona State University
nabul.goppalannBsss.dd
Steven James University of the Witwatersrand
steven.jamees@@its.ccc..
Matthew Gombolay
mathhe..gmbboley@oe..tt.. edu
Geongia Institute of Technology
Raymond Mooney
mooney@uteras. edu
The University of Texas at Austin
Benjamin Rosman
denjamin.rosman &wwits..cee
University of the Witwatersmand
Reviewed OpenReview: https://openreette net/ forum? id pR3 CCm tD
Abstract
Combining reinforcement learning with language grounding is challenging as the agent needs to explore the environment while simultancously lcarning multiple language-conddtionee rein-
lasks. To address this, introduce novel method: the compositionallly-nnbbb forcement learning language agent (CERLLA). Our method reduces the sample complexity of tasks specified with language by leveraging compositional policy representations and semantic parser trained using reinforcement learning and in-context learning. We evaluate our approach in an environment requiring function approximation and demonstrate com- positional generalization to novel tasks. Our method significantly outperforms the previous best non-compositional bascline in terms of sample complexity on 162 tasks designed to lest. compositional generalization. Our model altains higher success and learns in fewer steps than the non-compositional baseline. It reaches success rate equal to oracle policy's upper-bound performance of 92%. With the same number of environment steps, the baseline only reaches a success rate of 80%.
1 Introduction
An important goal of reinforcement. learning (RL) is the creation of agents capable of generalizing to novel tasks. Natural language provides an intuitive way to specify a variety of tasks, and natural language has important propertics: its structure is compositional and often mirrors the compositional structure of the tasks being described.
"Equal contribution.

---

Published in Transactions on Machine Learning Research (12/2024)
In-Context Examples
Success Rate z 92%?
Composed Value Function
pick up a red object or ball Symbol_01 Symbol_ pick up red ball Symbol_ SymboL_
Environment Rollouts
pick up a green ball + Symbol_o 0 & Symbol_2
Boolean Expressions
Mission
WVFo
pick up a red object that is not a ball
LLM
Symbol_0 0 & ~Symbol_7 _
Symbol_ & ~Symbol_
Image Observation
World Value Functions
Figure Pipeline diagram of the learning process for the CERLLA agent. The agent takes in a BabyAI language mission command and set of 10 in-context examples that selected using the BM25 search retrieval algorithm (Robertson et al. 2009). The agent produces 10 candidate Boolean expressions. These expressions specify the composition of the base compositional value functions. Each compositional value function is instantiated in the environment and the policy it defines is evaluated over 100 rollouts. If the success rate in reaching the goal is greater than 92%, the expression considered a valid parse of the language instruction and added to the set of in-context examples.
While previous works have attempted use natural language specify tasks for RL agents (Ahn et al. 2023 Blukis et al., (2000), here we exploit the compositional nature of language along with compositional policy representations to demonstrate improvements in sample complexity and generalization in solving novel tasks.
Previous approaches to mapping language to behaviors use policies learned using imitation learning Silva et al. 2021 Ahn et al. 2023 Blukis et al. 2020). In this work, we instead focus on the setting where the agent does not have access to supervised demonstrations and instead learns to ground language to specified behaviors with RL. The challenge in this approach the significantly higher sample complexity of RL- based methods when grounding behaviors. Agents must map variety of potential language instructions unknown corrosponding behaviors. Pretraining and transfer learning offers one possible solution. In natural language processing, pretraining language models such as BERT (Devlin et al., 2019) and GPT-4 (OpenAl, 2023) have enabled substantial reductions in sample complexity of solving novel NLP tasks. However, in RL there is a lack of pretrained policy representations that can be fine-tuned using novel examples in few-shot settings.
In CERLLA RT policy learning, "pretraining" instead involves learning representations that can be com- posed solve novel tasks [Todorov] 22099; Nangue Tasse et al.) 2020). For instance, Nangue Tasse et al.) (2020) demonstrate zero-shot task solving using compositional value functions and Boolean task algebra. The value functions are composed using Boolean operators to produce new complex behaviors. But these methods require manual specification of the Boolean expressions that describe value function composition, thus limiting their application to novel tasks. We overcome this and use RL to learn to compose the value functions, given a task description in natural language.
Our method uses these com compositional value functions and pretrained language models solve a large number of tasks using RL while not relying on curricula, demonstrations, or other external aids to solve novel tasks. Leveraging compositionality is essential to solving large numbers of tasks with shared structure. The sample complexity of learning large number of tasks using RI is often prohibitive unless methods leverage compositional structure (Mendez-Mendez & Eaton 2023).
This work builds on the Boolean compositional value function representations Nangue Tasse et al. (0000) construct a system for learning compositional policies for following language instructions. Our insight is that language commands reflect the compositional structure of the environment, but without compositional RI. representations, this structure cannot be used effectively. Language, therefore, unlocks the utility of
2

---

Published in Transactions on Machine Learning Research (12/2024)
compositional RL, allowing us to not only compose base policies but also negate specific behaviors to solve tasks such as "Don't start the oven." These language-condiioone compositional RI. policies can be used as pretrained general-purpose policies, and novel behaviors can be added as needed when solving new tasks. Morcover, the composed policies themselves are interpretable as we can inspect the base policies from which they are composed.
Our primary contributions are as follows:
We present novel approach for solving RL lasks specified using language. The policies for the tasks are represented as conjunctions, disjunctions, and negations of pretrained compositional value functions.
We combine in-context learning with feedback from environment rollouts to improve the semantic parsing capabilities of LLM. To our knowledge, our method is the first to learn semantic parser using only in-context learning with feedback from environment rollouts.
3. We solve 162 unique tasks within augmented MiniGrid-Babye domain (Chevalir-Boisveert et al., 2023
2019) which, to our knowledge, is the largest set of simultamsously-lennnn language- RL tasks.
4. Our method significantly outperforms the previous best non-compositional baseline in terms of sample complexity. Our compositional model attains success rate equal to oracle policy 's upper-bound performance of 92%. With the same number of environment steps, the baseline only reaches a success rate of 80%.
2 Background
BabyAl Domain
Because we build on the compositional value function representations of Nangue Tasse et al. (2020), our method is applicable to any environment with goal-reaching tasks, the ability to learn value functions through RL, and language instructions. To evaluate our method, we select the BabyAI MiniGrid domain (Chevalier- Boisvert al. 2019) an easily extensible test-bed for compositional language-RL tasks used in many recent language-RL RL works including (Carta et al. 2022 Li et al. 2022). It has language commands, imago-state observations, discrete action space, and objects with both color and type attributes. We augment Baby Al with 162 compositional tasks specified using intersection, disjunction, and negation. The appendix provides full list of task attributes available in the environment and the grammar of the Boolean expressions. Table [ provides examples of the types of tasks our method learns. The environment is initialized with one or more goal objects and distractor objects that are randomly placed.
For each episode in the Baby environment, the agent is provided with two forms of input as observations: task instruction formulated in natural language and 56 56 x 3 RGB image representing the state of the environment at each time-step. The objective for the agent is to correctly identify and pick up specific goal object, which is described by the task instruction. The goal objects are defined by their attributes: three shapes and six colors. The combination of these attributes results in 18 unique goal objects that the agent can pick up. In each episode, the environment contains at least one correct goal object and four additional "distractor" objects, which are sampled randomly.
We consider the case of an agent required to solve series of related tasks. Each task is formalized a Markov decision process (MDP) (S,A,p,r), where S is the state space and A is the set of actions available to the agent. The transition dynamics p(s"|s, a) specify the probability of the agent entering state s' after executing action in state 8, while r(s, a, s') is the reward for executing in s. We further assume that r is bounded by [MMIN; rMAx]. We focus here on goal-reaching tasks, where an agent is required to reach a set of terminal goal states g S.
Tasks are related in that they differ only in their reward functions. Specifically, we first define a background MDP Mo (So, Ao, Po, ro). Then, any task is characterized by task-specifie reward function r, that
3

---

Published in Transactions on Machine Learning Research (12/2024)
i
Figure 2: Example of a task in the BabyAI domain Chevalier- Chovalier-Boisvert et al. 2019). The agent (red triangle) needs to complete the mission "pick up the red key". Solving this task with compositional value functions requires using the conjunction of the pickup "red object" and "key" value functions.
is non-zero only for transitions entering g in g. Consequently, the reward function for the resulting MDP is given by ro + Ty.
We implement this reward function with a penalty of ro = -011 for each step taken. the agent chooses the pickup action upon reaching an object, it observes the picked object, and the episode ends. If the selected object matches the correct goal as per the task instruction, the agent receives a reward of r, +2.
The agent aims to learn optimal policy n, which specifies the probability of executing action in given state. The value function of policy æ is given by (*(s)= = E, (-t r(8t, 0z)] and represents the expected return after executing x from Given this, the optimal policy x² is that which obtains the greatest expected return at each state: V* (s) = Vx)) = max, (*) for all E S. Closely related is the action-value function, Q" a), which represents the expected return obtained by executing a from s, and thereafter following т. Similarly, the optimal action-value function is given by Q" a) max, Q (s, a) for all (s, a) ES x A.
Logical Composition of Tasks using World Value Functions (WVFs)
Recent work [Nangue Tasse al. 2020: 2022) has demonstratod how logical operators such as conjunction (A), disjunction (V) and negation (-) can be applied to value functions to solve semantically meaningful tasks compositionally with no further learning. To achieve this, the reward function is extended to penalise the agent for attaining goals it did not intend to:
F(), = EMIN f
(1)
r(s,a) otherwise,
where FMIN is large negative penalty. The agent receives the unmodified reward r(s, a) for all steps except where it reaches different goal state than intended: * # g. Given F, the related value function, termed world value function (WVF), can be written as: ((s,g,a) = F(8, a) + Is V (8', (o'.g)((s'(s) a)ds', where (**g))) Ex 1 g, a.)].
These value functions are intrinsically compositional since if a task can be written as a logical expression over previous tasks, then the optimal value function can be similarly derived by composing the learned WVF's. For example, consider the PickUpObject environment shown in Figure [ Assume the agent has separately learned the task of collecting red objects (task R) and keys (task K). Using these value functions, the agent immediately solve the tasks defined by their union (R V K), intersection (RA K); and nogation (-R) as follows: QRVK= Q V Q% == max max(Q%: @k), QRAK QR A Qi min(Qh. Qi): and Q*R Q3 == (QMAX QMIN) Q where QMAX and QMIN are the world value functions for the mazimum and minimum tasks respectively
Thee maximum task defined by the reward function FMAX for all Similarly, the minimum task has reward function FMIN for all g.
4

---

Published in Transactions on Machine Learning Research (12/2024)
3 Methods
We propose a two-step process for training an RL agent to solve Boolcan compositional tasks with language. During an initial pretraining phase, a set of WVTs are learned which can later be composed to solve new tasks in the environment. This set forms a task basis that can express any task which can be written as a Boolean algebraic expression using the WVFs.
In second phase, an LLM learns to semantically parse language instructions into the Boolean compositions of WVFs using RL and in-context learning. The parser learns this mapping from abstract symbols to WVFs using RL by observing language instructions and interacting with the environment. Notably, our method does not require the semantic parser to have access to any knowledge of the underlying basis tasks that the WVFs represent, and instead regards the WVFs as abstract symbols which can be composed to solve tasks. Since the semantic parser does not have access to any information about what task a WVF represents, our method can be applied in principle to any basis of tasks.
Tasks like "pickup the red key" can be represented by taking the intersection of the WVFs for picking up "red" objects and "key" objects: red A key. Our method also supports negation and disjunction-we we can specify lasks like "pick up a red object that is not a ball": red A -ball. We augmeni this domain with additional tasks. For further examples of tasks, see Table ! which lists the complete set of tasks created using the attributes yellow and key. We implement the model from Chevalier-Boisee non-compositional baseline. This model does not have a pretraining phase for its RL representations, and in our experiments we account for this discrepancy in training steps by penalizing our agent by the number of training steps needed to learn the WVFs.
et al. (2019) as a
3.1 Pretraining World Value Functions
During pretraining, a set of WVFs is learned which can later be composed to solve any task in the Ba environment. Each WVF takes as input a 56 56 3 RGB image observation of the environment and outputs [g] x |A| = 18 x 7 values for accomplishing one of the basis tasks (by maximising over the goal- action values). As there are nine object attributes (three object type attributes and six color attributes as listed in the appendix) we train nine WVFs. Each WVF is value function for the policy of picking up objects that match one of the nine attributes. However, our method does not require knowledge of the underlying semantics of the value functions (i.e. the names of the underlying task attributes). We therefore assign each WVF a random identifier, denoted as Symbol 0 through Symbol 8. While we refer to the WVFs by their color and object type in the paper, our model does not have access to this information and only represents the WVFs by their unique identifiers.
Each WVF is implemented using |9| 18 CNN-DQN (Mnih et al. 2015) architectures. The WVF pre- training takes nineteen million steps. This is done by first training QMIN (s. for one million steps and @Max(s, (8, for eighteen million steps (one million steps per goal in the environment). Each ba- and QMax g,a) by computing Q7 a) = (MAX(s g, a) if rn(g, a) = TMAX else QMIN 9, a). This yielded a 98% success rate for each basis WVF. For more details on WVF pretraining see Sec. 2 and Nangue Tasse et al. (002)), and see the appendix for a full list of hyperparamoters used in WVF training.
sis WVF (n, (s..g.)))) is then generated from Qirrx(s,
3.2 Compositionally Enabled RL Language Agent (CERLLA)
We assume the downstream tasks are distinct from the basis tasks. During downstream task learning, the pretrained WVFss are composed to solve novel tasks specified in language. To solve BabyAl langnage-speeifted tasks, the agent must interpret the input language command and pick up object of allowed type. To accomplish this, the semantic parser maps from language to Boolcan expression specifying the composition of WVFs. These Boolean expressions are then composed using a fixed pipeline that takes as input the set of WVFss and the Boolean expression. This pipeline parses the Boolean expression and returns a composed WVF. The agent then acts the environment under the policy of the WVF by taking the action with the greatest value at each step. If the Boolean expression is not syntactically correct, it cannot be instantiated as WVF and the episode terminates unsuccessfully.
5

---

Published in Transactions on Machine Learning Research (12/2024)
Table 1: Example language instructions and corresponding Boolean expressions for the yellow and box attributes
Language Instruction
Ground Truth Boolcan Expression
pick up yellow box
yellow & box ~ yellow & box
pick up box that is not yellow
pick up ycllow object that is not box
yellow & ~ box
pick up an object that is not yellow and not a box pick up box or a yellow object
yellow & ~ box
yellow box
pick up a box or an object that is not ycllow
yellow box
pick up a yellow object or not box pick up object that is not yellow not box pick up box pick up an object that is not box pick up a yellow object
yellow box
yellow box
box
box
yellow
pick up an object that is not yellow
~ yellow
3.2.1
In-Context Semantic Parsing with RL
Table 2: The prompting strategy for the CERLLA semantic parsing module.
Role
Content
System
"We are going to map sentences to Boolean expressions. The Boolean expression variable Symbols are numbered 0 to 8, e-g. Symbol_O 0, Symbol__ 1... The operators are and : & or I. not I will now give new sentence and you will come up with an expression. Now wait for a new sentence command. Respond with list of 10 candidate Boolean expressions. Respond only with the list of Boolean expressions. Never say anything else."
User (Ezample) Assistant
"pick up a red ball"
"Symbol 0 &SSymbol_7 7"
[Additional in-context examples]
User (Comonand) Assislant
"pick up a red object that is not a ball" " "Symbol 0 & Symbol__ & ~ Symbol 2
" "Symbol
Symbol 4"
"Symbol & Symbol 6 & Symbol 7"
[Additional candidate expressions)
To implement the semantic parser, we utilize state-of-tho-art large language models: GPT OpenAI 2023 and GPT ( Our method builds on the work of (Shin al. (2021) which builds semantic parser using LLMs and few-shot learning, and Toolformer (Schick et al. 2023) which learns LLM semantic parser from weak supervision. Our method is distinct from these approaches in that it utilizes in-context examples combined with an environment rollout RL signal. At the start of learning, the agent has no in-context examples of valid mappings from language to Boolean expressions (see Figure D. At each episode, our agent is prompted with general instructions defining the semantic parsing task, the imput language command, and up to 10 previously-acquired in-context examples selected using the BM25 retrieval algorithm Robertson et al. )
During training, the LLM is sampled with temperature of 1.0 and produces beam of 10 semantic parses (Boolean expressions) of the input language command. Together the temperature and beam width control the exploitation-exxporaatee
trade-off of the semantic parsing model. Each candidate Boolean expression is parsed using a fixed pipeline and instantiated as a WVF. The policy defined by the WVF evaluated in the environment over 100 episode rollouts. If the success rate across these episodes in reaching the specified

---

Published in Transactions on Machine Learning Research (12/2024)
goals is greater than equal to 92%, the language instruction and Boolean expression added to the list of in-context examples. Multiple Boolean expressions may attain high reward for any given task. To counter this we add a form of length-based regularization. If the agent already has an in-context example with the same language instruction, the length of the Boolcan expressions is compared and only the shorter of the two expressions retained as an in-context example. We thereby favor shorter Boolean expressions that attain high reward the environment. For more details of the prompting strategy, see Table 2
3.3 Baselines
The baseline is joint language and vision model which learns a single act iot-on-vl function for all tasks based on the architecture used in the original BabyAl paper Seevalier-Boisvere et al., 2019). We explore two baseline models:
LM Baseline that utilizes pretrained language representations for embedding mission commands from a frozen "all-mpnet-base-r> model from the TransnoorrTnsss library (Reimers & Gurevych 2019) based on the MPNet model [Song et al.) 2020) and an ablated Baseline which does not use pretrained language representations. This pretrained sentence embedding model is trained on diverse corpora of sentence embedding lasks.
4
Results
1.0
Mean Success Rate
0.8
0.6
Oracle Agent
0.4
CERLLA (Ours)
CERLLA GPT-3.5
0.2
LM Baseline
Baseline
0.0
5.00 x 10 1.20 x 10 1.90 107
1.95 x 107 Steps
2.00 x 107 2.05 x 107
2.10 x 107
Figure 3: Results for learning all 162 tasks simultancously. The mean episode success rate is plotted against the number of enviromment steps. Learning curves are presented for CERLLA and the non-compositional bascline agents. The Oracle agent is given the ground-truth Boolcan expressions and upper bounds the atlainable success rate in the enviromment, denoted by the dashed line at 92%. Our method is initialized 19 million steps to reflect the number of training steps used in pretraining the compositional value functions. Note the change in steps scale at 19 million steps. Means and 95% confidence intervals are reported over 10 trials, 5 trials for the LM Bascline.
We conduct experiments across four agent types and two settings. The first experiment evaluates sample complexity (Figure 3. We learn all 162 tasks simultancously and plot the mean success rate against the number of environment steps. The second experiment divides the task set in half, and measures the ability of the agents to generalize to held-out novel tasks while learning from a fixed set of tasks (Figure 5).
We evaluate our method implemented using both GPT-4 and GPT-3.5. We compare our method to the baseline agents, but penalize our method by the number of environment steps required to learn the pretrained WVFs. As an upper limit on the performance of our method, we compare to an Oracle Agent which has perfect semantic parsing module. has access to the ground-truth mappings from language to Boolean expressions and its performance is limited only by the accuracy of the pretrained policies and randomness in the environment.

---

Published in Transactions on Machine Learning Research (12/2024)
4.1 Simultaneous Learning of 162 Tasks
In this experiment, at each episode random task is sampled from the set of 162 language tasks. The baseline agents learn for 21 million steps, and CERLLA learns for 2 million steps. Because our agent pretrains the WVFs, we penalize our agent by starting it at 19 million steps (Figure 3 Note that this disadvantages our method, as the WVF pretraining phase does not include language information and its only exposure to language-task data is over the following two million steps. Our method therefore has access to less information about the tasks structure than the bascline agents during the first million steps. For CERLLA, due to the latency and cost of invoking the LLM, we only evaluate on one randomly selected task every 5, 000 environment steps, computing the average performance over 100 episodes. For the baseline agents evaluate all 162 tasks every 50, 000 timesteps. This results in higher variance for our method in the plots.
We also plot the number of in-context training examples added to CERLLA's set in Figure This is equivalent to the number of training tasks successfully solved at that step. The Oracle Agent solves the overwhelming majority of tasks during their first occurrence and limited only by the small amount of noise in the policies and environment.
4.2 Held-out Task Generalization
This experiment (Figure 5 measures the generalization performance of each method held-out tasks. We compare the performance of CERLLA to the baseline agents. In this setting, the set of tasks is randomly split into two halves at the start training. At each episode, a random task from the first set is selected. During ovaluation of our agent, one random task from each set is selected and the agent evaluated over 100 episodes. The baseline agents are evaluated over all 81 tasks in each set.
162
Mean Training Tasks Solved
140
120
100
80
60
Oracle Agent
40
CERLLA (Ours)
20
CERLLA GPT-3.5
0
5.00 x 105
1.00 x 106 Steps
1.50 x 106
2.00 x 106
Figure 4: The mean number of tasks solved is plotted against the number of environment steps. This quantity is equal to the total number of in-context examples present in CERLLA's in-context example set at that step. Because the Oracle Agent has access to the ground truth Boolcan expressions for each task, solves lasks immediately. The population of tasks remains constant, so the number of unsolved tasks decreases over time, leading a logistic learning curve and an exponential decay in the rate which new tasks are solved for the Oracle Agent. Mcans and 95% confidence intervals reported over 10 trials.
5 Discussion
In both experiments CERLLA allains a significantly higher success rate in fewer lotal samples than the baselines. Figure 3 shows our method attains 92% success rate (matching the performance upper bound of the Oracle Agent) after only 600k environment steps, a small fraction of the steps of the bascline. The bascline agents are not able to generalize to all 162 tasks and only reach success rate of 80% after 21 million steps. Note that while the WVF pretraining for our method requires 19 million steps, the pretraining
8

---

Published in Transactions on Machine Learning Research (12/2024)
1.0
Mean Success Rate
0.8
0.6
0.4
LM Baseline
CERLLA (Ours)
0.2
Train
Test
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Fraction of Training Steps
Figure 5: Generalization performance for learning on 81 randomly selected tasks (train) and evaluating on the held-out 81 tasks (test). Learning curves are presented for method and the baseline agents. The dashed and solid lines represent performance the training and test sets respectively. The z-axis represents the fraction of the total training steps completed: 1 million for the CERLLA and 21 million for the LM Baseline agent. The dashed line denotes the success rate for considering the environment solved at 92%. Means and 95% confidence intervals are reported over 15 trials, 5 trials for the LM Baseline.
objective does not include any language instructions and is distinct from the downstream task objective. The bascline learns the downstream tasks and language for its first 19 million training steps and still does not solve the 162 lasks. Even though the baseline agent. is given access to the language commands and reward structure for the entire duration of its training, the compositional agent is still able to outperform the baseline. These results show the necessity of compositional representations for being able to learn large numbers of compositional tasks in sample efficient manner.
Figure demonstrates that our method is able to generalize well to held-out tasks. The performance of the agent on training tasks and held-out tasks very similar. This is expected given the ability to generalize compositionally in both the policy and language spaces. The LM Baseline agent cannot generalize well between the training and held-out tasks as it lacks the necessary compositional representations. Note that this experiment trains the CERLLA and LM Bascline agent for fewer steps than the 162 task experiment: one million and 21 million sleps respectively. There is significant generalization gap for this agent even after 21 million steps, significantly more steps than our method, even accounting for WVF pretraining steps
In the 162 task learning experiment, CERLLA using GPT-3.5 does not exceed the performance of the baselines even after two million steps indicating poor compositional generalization. Confirming this, Figure shows the mean number of tasks solved, which the same as the number of total number of potential in-context examples that can be selected from during inference. Despite weaker version of CERLLA solving most of the tasks, this does not transfer to high evaluation success rate in Figure 3 The variance of the CERLLA using GPT 3.5 is higher than with GPT-4. This is caused by relatively worse generalization from available in-context examples than GPT-4. This indicates that more powerful language models are likely to improve the performance of CERLLA, up to the limit the oracle agent. Highlighting the interpretability of our language-RI. learning framework, we provide a qualitative analysis of the cause of the GPT-3.5 agent 's lower performance in the technical appendix.
Overall, our method solves challenging set of 162 language-RL tasks, which to knowledge is the largest set of simultaneousll-leennnn and language has inherent compositional structure. Therefore, its important to develop methods which leverage this shared compositionality to reduce the sample complexity of acquiring new behaviors. This is critical for reinforcement learning problems that already have high sample complexity due to the inefficiency of learning from RI. feodback. We address these challenges using a novel semantic parsing method, which uses in-context and environment feedback learn map natural language instructions Boolean expressions specifying the composition of value functions. Our method significantly purporosms non-compositional
language-RL tasks. In the real world, many tasks share compositional structure,
9

---

Published in Transactions on Machine Learning Research (12/2024)
baseline approach, demonstrating the importance of methods that can generalize compositionally in terms of both language and RL representations.
6 Related Work
Our work is situated within the paradigm of RL, where novel tasks are specified using language and the agent is required to solve the task in the fewest possible steps. BabyAI Sntevalier-Boisvor et al. 2019) explores a large number of language-RI, lasks, however it learns far fewer tasks simultaneously and their lasks do not involve negation. Another compositional RL benchmark CompoSuite (Mendez et al., 2022) does not include language, and has fewer tasks than our 162 task benchmark when accounting for the number of unique goal conditions that could be specified in language.
Previous approaches have solved this problem using end-to-end architectures that are learned improved using RL and set of demonstrations (Anderson et al. 2018 Blukis et al. 2020 Chaplot et al. 2018 A problem with such approaches is lack of compositionality lity in the learned representations. For example, learning to navigate to a red ball provides no information to the agent for the task of navigating to a blue ball. Morcover, demonstrations are hard to collect especially when users cannot perform the desired behavior. Some approaches demonstrate compositionaliy, by mapping to a symbolic representation and then planning over the symbols (Dzifcak et al., 2009 Williams et al., 2018 Gopalan et al., 2018). However, these works do not learn the semantics of these symbols the policies to solve the tasks.
Compositional representation learning has been demonstrated in the computer vision and language process- ing tasks using Neural Module Networks (NMN) (Andreas et al.) 2016 Hu et al. 2018), but we explicitly desire compositional representations both for the RL policies and the language command. Kuo et al. (2021)
demonstrate compositional representations for policies, but they depend on a pre-trained parser and demon-
strations to learn this representation. On the other hand, we use large language models (Raffel al.; 2020)
and compositional policy representations to demonstrate compositionality in our representations and the ability to solve novel unseen instruction combinations.
Compositional policy representations have been developed using value function compositions, as first demon- strated by Todorov (0007) using the linearly solvable MDP framework. Moreover, zero-shot disjunc-
tion Van Niekerk et al. 2019) and approximate conjunction (Haarnoja el al. 2018 Van Niekerk el al.
2019 Hunt et al. 2019 have been shown using compositional value functions. Nangue Tasse et al. (2020)
demonstrate zcro-shot optimal composition for all three logical cirrators-ddsjuuctiion conjunction, and negation in the stochastic shortest path problems. These composed value functions are interpretable be- cause we can inspect intermediate Boolean expressions that specify their composition. Our approach extends ideas from Nangue Tasse et al. (2020) to solve novel commands specified using language.
Recent works like SayCan use language models and pretrained iaaguage-conditionee value functions to solve language specified tasks using few-shot and zero-shot learning (Ahn et al.) 2023). Shridhar et al. (2021) use pretrained image-text representations to perform robotic pick-and-place tasks. Other work incorporates learning from demonstration and language with large-scale pretraining to solve robotics tasks (Driess et al., 2023 Brohan et al.) 2022). However, these works learning from demonstration Furthermore, these approaches do not support negations of pre-trained value functions that our method allows. More importantly, their methodology is unsuitable for continual learning setlings where both the RL value functions and language embeddings are improved over time as novel tasks are introduced.
opposed to RT.
Shin et al. [2021 utilize LLMs to Iearn semantic parsers using few-shot learning with in-context examples and |Schick el al. (2023) uses an LLM to learn a semantic parser in a weakly supervised setting. Our method
is distinct as we use policy rollouts in an environment as the supervision with in-context learning.
7 Limitations and Future Work
One limitation of our method is the need for a pretraining phase where a curriculum is required to learn the basis set of WVFs, In future work, we plan on addressing this through experiments that simultancously learn both the underlying WVFs and the language-instruction semantic parser using only environment. rollouts on
10

---

Published in Transactions on Machine Learning Research (12/2024)
randomly selected tasks. This is challenging optimization problem as the WVF models and the semantic parser must be optimized simultaneously to ensure that the WVFs form good basis for the space of language tasks.
Our future work will also investigate our method's performance in simulated and real-world compositional RI. tasks including vision and language navigation (VIN) and robotic pick-and-place tasks. The current environment has discretized action space (although it utilizes images for state information); while this might limit the method's applicability to some real-world RL tasks, both VLN and pick-and-place tasks have been pursued in discretized forms (Anderson al., 2018 Zeng et al., 2020). Both of these tasks could benefit from our method, as they require solving goal-reaching tasks which often have compositional language and task attributes. As one example, pick-and-place tasks are often compositional in terms of object type and locations for placing objects.
8 Conclusion
We introduce a method that integrates pretraining of compositional value functions with large language models to solve language tasks using RL. Our method rapidly solves a large space of RL tasks specified in language more effectively than previous approaches. Demonstrating efficacy across 162 tasks with reduced sample requirements, our findings also further differentiate the capabilities of more powerful language models from weaker ones in the task of semantic parsing using an environment policy rollout signal. The combination of compositional RT. with language models provides a novel framework for reducing the sample complexity of learning RL tasks specified in language.
Acknowledgements
This material is based upon work supported by the DARPA's Perceptuully-enabbell Task Guidance (PTG) program under Contract No. HR001122C007. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the DARPA. This material is also based upon work supported by the Air Force Office of Scientific Research under award number F49550-2411-023. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Air Force. This work was also supported by ONR grant 2NN00014-331122877 and #N00014-19-007..
References
Michael Ahn, Anthony Brohan, Yevgen Chebotar, Chelsca Finn, Karol Hausman, Alexander Herzog, Danicl Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as I can, not say: Grounding language in robotic affordances. In Conference Robot Learning, pp. 287-30 -18..1..77 PMTRR 2023.
Peter Anderson, Qi Wu, Damien Tency, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. aooo---ndlaanguag navigation: Interpreting fisually-groundee navi- gation instructions in real environments. In Proceedings of the TEEE Conference on Computer Vision and Pattern Recognition, pp. 3674 3683, 2018.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEF.F. Conference on Computer Vision and Potfern Recognition, pp. 39-48 39-48, 2016.
Valts Blukis, Yannick Terme, Eyvind Niklasson, Ross A Knepper, and Yoav Artzi Learning to map natural language instructions physical quadcopter control using simulated flight. In Conference on Robot Learning, pp. 1415-1438. PMLR, 2020.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Koerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real- world control at scale. Xix preprint or 241:2222.008111 2022.
11

---

Published in Transactions on Machine Learning Research (12/2024)
Thomas Carta, Pierre- Yves Oudeyer, Olivier Sigand, and Sylvain Lamprier. Eager: Asking and answering questions for automatic reward shaping in language-guided rl. Advances in Neural Informalion Processing Systems, 35:12478 12490, 2022.
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Maxime Chevelier-Boisvert,, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Hun Nguyen, and Yoshua Bengio. BabyAI: First steps towards grounded language learning with human in the loop. In International Conference on Learning Regresenlations, 2019.
Maxime Seeevalie-BBoiveeti Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, (bs/2306.1388) 2023.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidi- rectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (ods.). Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171-4186. Association for Computational Linguistics, 2019. doi: 10.18653/ (0.18653/v1/1133 11l119-1111111 1423. URL (tttps///ei..orr/11.118555//////
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint ar arXit:2303.03378 2023.
Juraj Dzifcak, Matthias Scheutz, Chitta Baral, and Paul Schermerhorn. What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal manage- ment and action execution. In 2009 IEEE International Conference on Roboties and Aulomation, pp- 4163 4168. IEEE, 2009.
Nakul Gopalan, Dilip Arumugam, Lawson Wong, and Stefanic Tellex Sequence-to-scquence language grounding of non-markovian lask specifications. Robolics: Science and Systems XIV, 2018.
Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Com- posable deep reinforcement learning for robotic manipulation. In 2018 IEEE International Conference on Robotics and Automation, pp. 6244-6251. IEEE, 2018.
Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In Proceedings of the European Conference Computer Vision, pp. 53 69, 2018,
Jonathan Hunt, Andre Barreto, Timothy Lillicrap, and Nicolas Heess. Composing entropic policies using divergence correction. In International Conference on Machine Learning, pp. 2911-2920, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Yen-Ling Kuo, Boris Katz, and Andrei Barbu. Compositional RL agents that follow language commands in temporal logic. Frontiers in Robotics and AI, 8:689550, 2021.
Shuang Li, Xavier Puig, Chris Paxton, Yihun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31199-3122,, 2022.
Jorge A. Mendez, Marccl Hussing, Meghna Gummadi, and Eric Eaton. CompoSuite: A compositional reinforcement. learning benchmark. In 1st Conference on Lifelong Learning Agents (CoT.LAs-22)) 2022.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Jorge Mendez-Mendez and Eric Eaton. How to reuse and compose knowledge for lifetime of tasks: A survey on continual learning and functional composition. Transactions on Machine Learning Research (TMLR), 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):5599-55 5333, 2015.
Geraud Nangue Tasse, Steven James, and Benjamin Rosman. A Boolean task algebra for reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
Geraud Nangue Tasse, Steven James, and Benjamin Rosman. World value functions: Knowledge represen- tation for multitask reinforcement learning. In The 5th Multi- colti-disciplinary Conference on Reinforcement Learning and Decision Making (RLDM), 2022.
OpenAl. Gpt-4 technical report, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 2020.
Nils Reimers and Tryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL htapp:://azxii..r/aabb////// 10084
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333-355) 2009.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancodda, and Thomas Scialom. Toolformer: Language models can teach themsclves to use tools. 2023.
Richard Shin, Christopher H. Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil An- tonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Ben Van Durme. Constrainod language models yield few-shot semantic parsers. 2021 Empirioal Methods Nalural Lan- guage Processing, November 2021. URL http:::/ww..iirressssssteteeeeee haapa/lceeo..aaaaaoooooooedeaaae
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. CLiPort: What and where pathways for robotic manipu- lation. In Conference on Robot Learning, 2021.
Andrew Silva, Nina Moorman, William Silva, Zulfiqar Zaidi, Nakul Gopalan, and Matthew Gombolay. Lancon-learn: Learning with language to enable generalization in multi-task manipulation. IEEE Robotics and Automation Letters, 7(2):1635 1642, 2021.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. arXiv preprint 2xXi:2204.09299) 2020.
Emanuel Todorov. Linearly-solvable Markov decision problems. In Advonces in Neural Information Process- ing Systems, pp. 1369-1376 2007.
Emannel Todorov. Compositionality of optimal control laws. In Advances in Neural Information Processing Systems, pp. 1856-1864, 2009.
Benjamin Van Niekerk, Steven James, Adam Farle, and Benjamin Rosman. Composing value functions in reinforcement learning. In International Conference on Machine Learning, pp. 6401-6409, 2019.
Edward Williams, Nakul Gopalan, Mine Rhoe, and Stefanic Tellex. Learning to parse natural language grounded reward functions with weak supervision. In IEEE International Conference on Robotics and Automation, pp. 4430-4433 2018.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Arm- strong, Tvan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Tee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL), 2020.
A Appendix
Language Instruction
Boolean Expression
pick up a ball or & grey object
Symbol 0 Symbol
pick up a box that
~ Symbol & Symbol 2
is not grey pick up a grey ball pick up a ball or an
Symbol 0 & Symmol
(Symbol 0 Symbol 1)
object that not grey pick up ball that is not grey pick up a grey object or (Symbol 0 & Symbol
Symbol
Symbol 0 & Symbol
4)
Symbol
not ball
pick up a grey object
~ Symbol 0 & Symmbol 4
that is not ball
pick up a grey object or
(Symbol O & Symbol 1)
not a key
(~ Symbol 2 & Symbool 4) ~ Symbol 5 (Symbol 0 & Symbol 1) ~ Symbol
pick up an object that
is not grey pick up an object that is not grey and not ball
~ Symbol_ _ & Symbol_
Table 3: Subset of the in-context examples that the GPT-3.5 agent has accumulated at the end of million steps of learning the 162 tasks. As shown, many of these expressions are not consistent or needlessly complicated. This helps to explain the relatively poorer performance of the GPT-3.5 agent which produces much noisier semantic parses and thus has much higher variance and lower performance than the GPT-4 agent. Reducing the sampling temperature during learning leads to better expressions, but at the cost of slower exploration and learning.
Table 4: Task attributes and Boolean grammar. The symbols uniquely identify each learned WVF.
Task Attributes
Boolean Grammar
Colors
Objects
Symbols
Operators
red, purple, grey green, yellow, blue
key, bull, bor Syrabol 0, Symbol 1,... Syymbbl
AND: & OR: NOT:
14

---

Published in Transactions on Machine Learning Research (12/2024)
Table 5: Hyperparameters for the LLM Agent.
LLM Agent Hyperparameters
LLM
GPT-4 and GPT 3.5
Beam Width
10
Rollouts
100 episodes
In-Context Examples
10
Training Temperature
1.0
Evaluation Temperature
0.0
Table 6: Hyperparameters for world value function pretraining. The Adam optimizer was introduced by Kingma & Ba (2015).
WVF Learning Hyperparameters
Optimizer
Adam
Learning rate
le-4
Batch Size
32
Replay Buffer Size
le3
e init
0.5
final
0.1
decay steps
1e6
15


Reviews:
[{"review_id": "0UXvKK53Tf", "reviewer": "Reviewer_Z83Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and easy to understand. However, the main contributions of this work are not clear. The challenge of mapping language to predicates has been extensively explored across various domains [e.g., R1, R2, R3]. Moreover, previous research has also demonstrated that language can be translated into predicates to tackle complex issues, such as vehicle detection in real-world settings [e.g., R4 and R5].\n\n\nIn summary, while the paper introduces some intriguing concepts, it is unclear how it advances the current state-of-the-art in this field. Additional clarification on the contributions and novelty of this work would be beneficial, along with experiments on some real tasks (similar to R4).\n\n[R1] Pereira, F. C. N. (1982). Logic for natural language analysis.\n\n[R2] Purdy, W. C. (1991). A logic for natural language. Notre Dame Journal of Formal Logic, 32(3), 409-425.\n\n[R3] Clark, A. (2013). Learning trees from strings: A strong learning algorithm for some context-free grammars. The Journal of Machine Learning Research, 14(1), 3537-3559.\n\n[R4] Lu, Y., Chowdhery, A., Kandula, S., & Chaudhuri, S. (2018, May). Accelerating machine learning inference with probabilistic predicates. In Proceedings of the 2018 International Conference on Management of Data (pp. 1493-1508).\n\n[R5] Lu, Y., Chowdhery, A., & Kandula, S. VisFlow: A Declarative Platform for Parallelizing Large-Scale Vision Programs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a method for solving reinforcement learning (RL) tasks defined through language. The authors suggest representing policies as predicates, including conjunctions, disjunctions, and negations, of compositional value functions. The effectiveness of this approach is illustrated by solving 162 tasks in an enhanced MiniGrid-BabyAI environment.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See strengths and weaknesses section.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "09D6sgpHVe", "reviewer": "Reviewer_TYpx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis paper shows a practical pipeline for language-based compositional RL. It provides a task set containing 162 tasks unique in compositions. The proposed algorithm also achieves SOTA performance in success rate and sample complexity.\n\nWeakness:\n\n(a) It is not clear how this pipeline can be extended to continuous control tasks, since it only learns value functions and seems cannot be easily extended with an actor-critic framework.\n\n(b) BabyAI is still a relatively easy benchmark. According to Table 1, each task can be represented with 1 or 2 symbols. What would be the difficulty and how the performance would change, for applications in scenarios with more interactive items?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Built on the idea of compositional value functions (Nangue Tasse et al. (2020)), this paper further introduces an LLM-based semantic parser to improve the generality of the pipeline. Surprisingly, this framework can solve 162 unique tasks within an augmented MiniGrid-BabyAI domain.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please response to the weaknesses listed above. A discussion on the applicability of this work and possible extensions on this work would definitely help.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QOd0NNaJlo", "reviewer": "Reviewer_7MdF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The approach of combining in-context learning with reinforcement learning feedback for semantic parser training represents a novel contribution. This method, distinct from traditional supervised or reinforcement learning, actually discovers relevant in-context examples for LLMs through online feedback, guiding the model to generate meaningful Boolean expressions grounded in environment dynamics.\n\n- Compositional generalization is one of the core strengths of this work. CERLLA effectively leverages the compositional structure of both language and environment to generalize to novel instructions, solving previously unseen combinations by combining learned World Value Functions (WVFs). This capability outperforms traditional reinforcement learning methods that typically learn tasks in isolation.\n\n- Empirical results demonstrate the approach's efficacy. By exploiting compositionality, CERLLA exhibits accelerated learning compared to non-compositional baselines. Its high success rates, matching oracle policy performance, underscore the method's effectiveness in the chosen environment.\n\n- The use of Boolean expressions for policy representation enhances interpretability. Examination of the semantic parser's generated Boolean expressions provides insights into the agent's underlying reasoning and the specific WVFs combined to solve tasks.\n\n## Weaknesses\n\n- The paper's presentation could be significantly enhanced for better self-containment. Additional context on the BabyAI environment and World Value Functions (WVFs) would be beneficial. For example, clarification is needed on several points:\n\t- What are the 18 goals exactly?\n\t- In equation 1, how do you get the goal $g$? What does $g\\ne s \\in G$ exactly mean? I cannot relate this condition to the text description \"the reward function is extended to penalise the agent for attaining goals it did not intend to\".\n\t- How is $r_0$ defined?\n\n- The method's reliance on accessible basis tasks and their known combinations to cover test tasks is a strong assumption. Manual pre-training of WVFs also limits applicability, requiring significant domain knowledge and engineering effort. The paper acknowledges this and suggests future work on automatic WVF discovery, but in its current form, the method requires significant domain knowledge and engineering effort. The lack of clarity on how the 18 goals are selected and how WVFs are trained in downstream tasks further adds to this concern.\n\n- The effectiveness of CERLLA depends on having a set of WVFs that can effectively represent the underlying task space.  If the pre-trained WVFs are not good enough, the agent's ability to compose complex behaviors will be greatly limited. \n\n- The assumption that a policy can be directly derived from a Q function is limited to discrete action spaces.\n\n- The experiments are conducted in a relatively simple grid-world environment (MiniGrid-BabyAI) with a limited set of actions and objects. While this environment is suitable for studying compositional generalization, it's unclear how well the approach would transfer to more complex, real-world environments with continuous state and action spaces.\n\n- The semantic parser's performance is closely tied to the capabilities of the employed Large Language Models (LLMs). The observed lower performance and higher variance with GPT-3.5 compared to GPT-4 highlights potential limitations in terms of cost and accessibility.\n\n- The use of LLMs and multiple environment rollouts for each candidate parse may present scalability challenges in more complex scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents CERLLA (Compositionally-Enabled Reinforcement Learning Language Agent), a method combining Large Language Models and compositional value functions for reinforcement learning from language instructions. The core idea is to parse instructions into Boolean combinations of pre-trained World Value Functions (WVFs) using an LLM, enabling compositional generalization to novel tasks. \n\nThe approach consists of two main phases:\n\n1. Pretraining of World Value Functions that can be composed to solve complex tasks.\n2. Learning a semantic parser using an LLM, in-context learning, and environment feedback to map language instructions to Boolean compositions of WVFs.\n\nExperiments in an augmented MiniGrid-BabyAI environment with 162 compositional tasks demonstrate that CERLLA outperforms non-compositional baselines in terms of sample complexity. CERLLA achieves a 92% success rate, matching the oracle policy's performance, with significantly fewer environment steps compared to the baseline. \n\nThe key contributions of this paper are:\n\n- Application of World Value Function (WVF) concepts to language-conditioned reinforcement learning tasks, demonstrating strong compositional generalization.\n- Development of a novel approach combining in-context learning with environmental feedback to enhance the semantic parsing capabilities of large language models (LLMs) for mapping language instructions to boolean expressions.\n- Experimental results indicating significant improvements in sample complexity compared to previous non-compositional baselines on the BabyAI benchmark.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To make the paper more self-contained, authors could add more detailed explanation for BabyAI and World Value Functions (WVFs).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All of the reviewers are in favor of acceptance and the main concerns raised were about the environment's generality (though this was not grounds for rejection) and some language clarifications, which the authors indicate they will fix in the CR (see reply to 7MdF).  I think I can therefore comfortably check for the inclusion of these minor changes but they do not really make or break the decision.
Decision: Accept (poster)