True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning
Authors: Weihao Tan
Wentao Zhang
Shanqi Liu
Longtao Zheng
Xinrun Wang
Bo An
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement Learning
Large Language Models
Parameter-Efficient Fine-Tuning
Primary area: reinforcement learning
TL;DR: Using reinforcement learning to train large language models to align their knowledge with embodied interactive environments.

Abstract: Despite the impressive performance across numerous tasks, large language models (LLMs) often fail in solving simple decision-making tasks due to the misalignment of the knowledge in LLMs with environments. On the contrary, reinforcement learning (RL) agents learn policies from scratch, which makes them always align with environments but difficult to incorporate prior knowledge for efficient explorations. To narrow the gap, we propose TWOSOME, a novel general online framework that deploys LLMs as decision-making agents to efficiently interact and align with embodied environments via RL without requiring any prepared datasets or prior knowledge of the environments. Firstly, we query the joint probabilities of each valid action with LLMs to form behavior policies. Then, to enhance the stability and robustness of the policies, we propose two normalization methods and summarize four prompt design principles. Finally, we design a novel parameter-efficient training architecture where the actor and critic share one frozen LLM equipped with low-rank adapters (LoRA) updated by PPO. We conduct extensive experiments to evaluate TWOSOME. i) TWOSOME exhibits significantly better sample efficiency and performance compared to the conventional RL method, PPO, and prompt tuning method, SayCan, in both classical decision-making environment, Overcooked, and simulated household environment, VirtualHome. ii) Benefiting from LLMs' open-vocabulary feature, TWOSOME shows superior generalization ability to unseen tasks. iii) Under our framework, there is no significant loss of the LLMs' original ability during online PPO finetuning.

Published as a conference paper at ICLR 2024
TRUE KNOWLEDGE COMES FROM PRACTICE: ALIGNING LLMS WITH EMBODIED ENVIRONMENTS VIA REINFORCEMENT LEARNING
Weihao Tan', Wentao Zhang', Shanqi Liu', Longtao Zheng', Xinrun ; Bo An 3* Nanyang Technological University, Singapore Zhejiang University" Skywork Al (weihao001, wt zhang, longtao001, kinrun.wang, boan @nt edu. sg shanqiliu@z en
ABSTRACT
Despite the impressive performance across numerous tasks, large language mod- els (LLMs) often fail in solving simple decision decision-making tasks due to the mis- alignment of the knowledge in L.LMs with environments. forcement learning (RI.) agents learn policies from scratch, which makes them always align with environments but difficult to incorporate prior knowledge for efficient explorations. To narrow the gap, propose TWOSOME, novel gen- eral online framework that deploys LLMs as decision-making agents to efficiently interact and align with embodied environments via RL without requiring any pre- pared datasets or prior knowledge of the environments. Firstly, we query the joint probabilities of each valid action with LLMs form behavior policies. Then, enhance the stability and robustness of the policies, we propose two normaliza- methods and summarize four prompt design principles. Finally, we design a novel parameter-efficee efffccienn training architecture where the actor and critic share one frozen LLM equipped with low-rank adapters (LoRA) updated by PPO. We con- duct extensive experiments to evaluate TWOSOME. i) TWOSOME exhibits sig- nificantly better sample efficiency and performance compared the conventional RL method, PPO, and prompt tuning method, SayCan, in both classical decision- making environment, Overcooked, and simulated household environment, Virtu- alHome. ii) Benefiting from LLMs' open-vocabulary feature, TWOSOME shows superior generalization ability to unseen tasks. iii) Under our framework, there is significant loss of the LLMss original ability during online PPO finetuning. []
the contrary, rein-
INTRODUCTION
LLMs have demonstrated remarkable success natural language generation and understand-
ing (Brown et al.) 2020, OpenAl (023). Re- cent studies show that LLMs can manage other Al models and tools to address complex mul-
X
timodal tasks (Shen et al. 2023 Lu et al.
2023), assist or play sophisticated games, such as TorlWorld (Yao et al.) 2023), Handbi (Hu & Sadigh 2023), and MineCrafi (Wang et al. 2023a), or be deployed on robots for rorl-world interactions (Brohan al., 2023 Driess al. 2023). While T.MMs can provide insightful sug- gestions for complex tasks, they often fail in solving simple decision decision-making tasks due to misalignment issues (Brohan et al. 2023).
(a) Only tomato, lettuce (b) LLMs guide the agent and onion provided in to the lettuce on the the game. LLMs may cutting board, which choose to pick up addi- already contains tomato, without knowing that each cutting hoard can only contain one item at time.
tional ingredients, such as cucumber and pepper to cook the dish.
There are two main misalignment issues lead- ing the failures of LLMs decision-makinn
Figure 1: Two misalignment examples
Co-corresponding Authors
Code available at wtttttss /

---

Published as a conference paper at ICLR 2024
tasks. i) LLMs may generate invalid actions. As the example shown in Figure la LLMs may keep adding cucumber and pepper when asked to make a tomato and lettuce salad, while these ingredients not provided. ii) LLMs may not know accurately the dynamic transitions of the environments, especially when some specific constraints are introduced in the environments. This incorrect esti- mation will make LLMs tend choose actions that fit their learned common sense, resulting in the failure to solve domain -ppecifc tasks. As the example shown in Figure [b] LLMs may keep trying to put both tomato and lettuce on the same cutting board, without knowing that only one item can be placed on the board in this environment. Addressing these two misalignment issues requires careful alignment between LLMs and environments.
On the contrary, reinforcement learning (RL) learns agents' policies from scratch through trial and error in environments (Sutton & Barto 2018), which ensures that RI agents are well aligned with environments. Most RL. methods start from random policies, updated according to the return from the environments, which leads to poor sample efficiency as most policies have poor performance at the early stage of learning. One way to improve the sample efficiency is to incorporate the prior knowledge with the initialization of the policy and the exploration during training (Kumar et al.) 2023). LLMs are ideal sources of prior knowledge for RL agents LLMs are trained with enormous data from the corpus. Therefore, by leveraging RL to align LLMs with embodied environments solve decision-making tasks, can address the misalignment issues in LLMs and the sample efficiency issue in RL simultaneously.
Motivated by this idea, we propose True WnoWledge cOmeS frOM practicE (TWOSOME), a gen- eral online framework that deploys LLMs as embodied agents to efficiently interact and align with environments via RL to solve decision-making tasks without requiring any prepared datasets or prior knowledge of the environments. Instead of letting LLMs directly generate actions, we use the loglikelihood scores of each token provided by LLMs to calculate the joint probabilities of each action and form valid behavior policies. This process eliminates the misaligmment caused by invalid actions. Moreover, the L.LM agents are optimized with proximal policy optimization (PPO) (Schul- man et al. 2017) using rewards from the environments, which eliminates the misalignment caused by dynamic transitions. We observe that the formed behavior policies suffer a severe issue that longer actions tend to have lower joint probabilities, resulting in unreasonable unbalance over the action distribution. To overcome this issue, we propose token normalization and word normalization in terms of the number of tokens and words in actions rectify the unbalance. Furthermore, design a novel architecture for efficient training, where both the actor and critic in RL methods share the same frozen LLaMA-7B model (Touvron et al., 2023), updated by parameter efficient finetuning methods, e.g., LoRA (Hu et al. 2022). During training, we observe that prompts for observations and actions can greatly influence the initial policies of LLM agents, therefore, we also summarize four principles for designing efficient prompts to enhance the reasoning ability of LLMs.
We first evaluate our methods on a classical RL decision-making environment, Overcooked, and simulated physical environment, VirtualHome, with various tasks to show that our proposed word normalization method can remarkably improve stability and accelerate convergence during the train- ing process. TWOSOME with word normalization exhibits significantly better sample efficiency and performance compared to the conventional RL method, PPO, and prompt tuning method, SayCan, in all tasks. Then, we test our trained TWOSOME agents in eight new unseen tasks and find that TWOSOME has superior generalization ability across unseen tasks. Finally, we evaluate our trained TWOSOME agents on traditional NLP benchmarks to demonstrate that under our framework, there is no significant loss of the L.Mss' original ability during online PPO fine-tuning.
2 RELATED WORK
In this section, we present brief overview of related work. More discussions are in Appendix A
Embodied Agents with LLMs.
Recent methods use LLMs to assist planning and reasoning in
robot learning (Brohan et al.] 2023
Liang et al. 2022 Zeng et al. 2022) and simulation environ-
ments (Fan et al., 2022 Wang al. 2023a Yao et al. 2023). LLMs are also applied to help robot
navigation (Parisi et al. 2022; Majumdar et al.) 2020) and manipulation TTang et al.) 2022 Ren
et al. 2023 Khandelwal et al. 2022). Among them, ReAct ( (Yao et al. 2023) uses
prompting by generating both reasoning traces and action plans with LLMs. SayCan (Brohan et al.)
2023) leverages the ability of L.LMs to understand human instructions to make plans for completing
2

---

Published as a conference paper at ICLR 2024
tasks without finetuning LLMs. Voyager (Wang et al. 2023a) leverages GPT-4 to learn and con- tinually discover skills during learning. While these works exhibit promising results, they rely too heavily the inherent capabilities of powerful LLMs, which are difficult to apply to smaller LLMs with weaker reasoning abilities. Concurrent to our work, GLAM (Carta et al. 2023) uses RI to ground L.M.s. However, they focus simple primitive actions in toy environments without rich semantics, resulting in underutilizing the capabilities of LLMs, and failing to observe the impact of prompt design and address the unbalance over action space.
Finetuning LLMs. Parameter-efficient finetuning (PEFT) can significantly reduce the number of parameters to tune LLMs while with minor loss of the performance (Ding et al. 2023). Prompt tuning (Lester et al. (02T) and prefix tuning (Li & Liang 2021) prepend additional tunable tokens to the input or hidden layers, and adapter tuning (Houlsby et al. 2019) introduces the adapter to LLMs, where only the introduced tokens and adapters are finetuned. Recently, Iow-rank adaption (LoRA) (Hu et al. 2022) introduces low-rank matrices to approximate parameter updates. Rein- forcement learning from human feedback (RLHF) is effective finetuning LLMs (Ouyang et al.; 2022 where the reward for RL is learned from human feedback. Another concurrent work (Xiang et al. 2023) leverages embodied environments to provide feedback to finetune LLMs. Different from method, they use supervised learning to finetune L.MSs with pre-collected embodied expe- riences by random sampling in environments instead of doing decision- decision-making tasks from scratch.
PRELIMINARIES
In this section, we provide a brief background on LLM and the RL problem formulation.
LLMs learn from text data using unsupervised learning. LLMs optimize the joint probabili- ties of variable-length symbol sequences as the product of conditional probabilities by P(x) I P(s;(s1,..., 8;-1), where ($,,,2,...,,,, Sn)) is variable length sequence of symbols.
LoRA is parameter- and compute-efficiet finetuning method that incorporates trainable rank de- composition matrices into each layer of an LLM. It allows indirect training of dense layers with weight matrix, Wo Rdxk by optimizing the rank-decomposition matrices by Wo AW = Wo + BA, where B E Rdxr, A E Br and the rank r is much smaller than d and k:
RL formulates decision-making makinn problems as Markov Decision Processes (MDPs). An MDP is defined by the tuple (S, A, T,, R,, 7). where S is the state space, A is the action space, T is the transition dynamics, is the reward function and y is the discount factor. Agents select actions based on observations, aiming maximize the expected discounted accumulative reward.
PPO is a tlate-of-the-aue actor-critic RL method that optimizes the policy based on the accumulative reward with ad advantage function A(st, ae) Q(st, ae) V(s)), where V(s;) is the value function and (st,, at) is the action-value function. The objective of PPO can be expressed as:
Te (a(s)) Thai (als) ATo (s,a), clip(
To(a 8) To (a s)²
J(0)= Es,a [min(
+ c)A as)
where A A represents the advantage function of the policy before updating, and the clip operator controls the size of policy updates.
Aligning LLMs with Embodied Environments. We intend to deploy LLMs as interactive agents in embodied environments, where LLMs receive textual observations and generate textual actions executed in the environments. The textual observations and actions can be transferred from images or vectors by vision-language modes (VI.Ms) or scripts. Compared to primitive actions, high-level actions, such as macro-actions (Theocharous & Kaclbling] 2003) and learned skills (Konidaris et al., 2011) usually have richer semantics, benefiting LLMs to leverage their prior knowledge.
TWOSOME
In this section, we present TWOSOME, a general online framework to align LLMs with embodied environments via RL. We first describe how to deploy LLMs embodied agents to generate valid actions. Secondly, we investigate the influence of lengths of action prompts on the generated policies and propose normalization techniques, i.e., token normalization and word normalization, to al-
3

---

Published as a conference paper at ICLR 2024
Observation Prompt You see a tomato, bood and custing board. you should first
To
buw
*
LLM
the
turting
Action Prompt pock up tomato take the bywe the hoard chop the toratoo
the toon
0.15
0.05
serve the dish
Token probability
Polley
Figure 2: Overview of how TWOSOME generates policy using joint probabilities of actions. The color areas in the token blocks indicate the probabilities of the corresponding token in the actions.
leviate the unbalance over actions. Then, we design parameter-efficiet training method under the PPO framework. Finally, we summarize four principles for efficient prompt design for TWOSOME.
4.1 VALID POLICY GENERATION
Actions directly generated by LLMs could be invalid in environments. This issue be partially resolved by prompt design, i.e., adding the restrictions actions in the prompt. However, most of the current LLMs cannot exactly follow the restrictions, especially for small and medium LLMs, i.c., less than 65B. Therefore, novel methods are required to let LLMs generate valid policies.
Instead of letting LLMs generate actions directly, TWOSOME queries the scores of all available actions from LLMs. These scores are used to determine the probabilities of executing the actions, which is similar SayCan (Brohan al. 2023). The process of the generation of policy is illustrated in Figure 2 Specifically, we associate a unique semantic prompt, e.g., pick up the tomato, to each action in the embodied environments, named action prompt, and observation prompt for the raw observation. The action prompt 0g A is sequence of tokens a = {w ...., w where N₂ is the length of the action prompt k. We note that the action prompts are not necessarily of the same length, At each step, the observation prompt 8 E is concatenated with each of the valid action prompts da. which forms the input of the LLM. We use the probabilities of tokens generated by the LLMs to compute the probability of the action prompt. Note that this probability differs from the probability of executing the action in the environment. For convenience, call this probability token-level probability. The token-level probability of (a is
Ptoken (axls) w
(ssss) II P(wils, w w
(1)
Normally, scores provided by LLMS the loglikelihood of each token, namely the logits, log Ptoken- We use softmax to normalize token-level probabilities over actions to get the policy:
exp(log Peoken (ak)
P(ayls) EnEA exp(log P Ptoken (a(s))
There two main advantages of this method: i) the generated policy is always valid for execution in the embodied environments and ii) leveraging the compositionability, our method can be applied to enormous actions representable by the vocabulary.
4.2 ACTION PROMPT NORMALIZATION
In this subsection, we will illustrate the issue in the method presented in the above section and present two normalization techniques alleviate them.
Issue of Eq. (1) One key issue in Eq. 1 is that the probability of each token P(w |.) is always less than Therefore, longer action prompts tend to have lower token-level probabilities, even though the longer action prompts may be more reasonable in the environment. For example, in Figure 2 pick up the tomato has more tokens than serve the dish but is the optimal action in terms of the given observation. A simple remedy for this issue is forcing all action prompts to have similar lengths, which is the case in GLAM (Carta et al,) 2023) where all primitive actions have similar lengths. However, this will harm the applicability of the methods which makes the design of the
4

---

Published as a conference paper at ICLR 2024
action prompts difficult, even impossible sometimes. Therefore, we propose two normalization techniques, token normalization and word normalization, to address this issue.
Token and Word Normalization. A simple idea to solve this issue is to normalize the token-level probabilities of the actions with the number of tokens, which can be defined as:
log
ptn token (ax [s) log Ptoken ('ooke((xlss)) (auls)/Ns
We note that log Proken (ax 1s) will always be negative, the longer action prompts will have smaller negative values, therefore, dividing by the number of tokens will make the token-level probabilities of longer action prompts fall in the same magnitude with short action prompts. Another option is that instead of dividing the number of tokens, we can normalize the token-level probability by dividing the number of words, i.e.,
log I token (ak|s) = log P Ptoken (aaken(ap)) (axls)/W,
(4)
where Wx is the number of words in the action prompt k. For the example of pick up the tomato, N₂ = while Wx- =4. 4.
Comparing the Two Normalizations. Though token normalization can eliminate the influence brought by the length of action prompts, it is slightly excessive. We observe that if a word is divided into several tokens, the first token usually has relatively low probability, while the probabilities of the rest of the tokens tend to be remarkably high, which are often almost close to 100%. For example, another important action, chop, in the Overcooked environment, is made up of two tokens, ch and op. The probabilities of ch usually in the range of 20%, depending given by the LLM agent according to the observation, however, once ch appears, the probability of op will boost 90 99% in the next word since there are no other words starting with ch in the observation prompts. The same phenomenon is also observed in lomao and dish as shown in Figure [ LLMs discovered and learned this statistical law during the autoregressive training process. Thus, instead of normalizing the probabilities of actions according to the number of tokens, more reasonable regard the several tokens made up of one word as an integrated symbol. Therefore, word normalization is a more suitable normalization, where the log probabilities of action get averaged according to the number of words in the action prompts.
the priority
4.3 CEFMETEEREEFICIENT PPO FINETUNING
In this section, we present the parameter-efficient finetuning method to align LLMs and embodied environments with the generated policies under the PPO framework. We first introduce the network design and then the training procedure.
Architecture. As shown in Figure we add ad- ditional MLP layers to the last transtormer block of LLaMA-7B model as the critic. The critic's MLPs use the last token of the observation prompt as input and output the estimated value of the observation prompt. On the other hand, the actor is formed by the frozen LLaMA-7B model with the augmentation of LoRA pa- rameters. We also note that the dropout layers would bring additional training instabilities, because the ran- domness of the dropout may violate the KL divergence constraint in PPO, therefore, do not use dropout in our LoRA modules. During training, only the MLPs for the critic and the LoRA parameters for the actor are updated, which makes the training efficient. The I.LMA-7B model can also serve as the reference model to regularize the update of the parameters, which is not explored in this work. Though we focus on the decoder-only models, our method can be seamlessly extended to the encoder-decoder architecture, c.g., T5 (Raffel et al. 2020).
value
action
MLP
Env
LLaMA-7B
(frozen)
LoRA
obs prompt & reward
Figure 3: Parameter-efficies architecture
Training. The training procedure generally follows PPO (Schulman et al., 2017). which is demon- strated to be effective in finetuning LLMs with human feedback (Ouyang et al. 2002). We observe the training instability when updating the actor multiple times with the same data, which is also
5

---

Published as a conference paper at ICLR 2024
observed in DeepSpeed Chat blog Therefore, every sampled data is discarded after training once. Given the fact that the newly added MLPs in the critic are initialized randomly, while the LoRA parameters in the actor are initialized as zero, i.c. the output of the actor is exactly the same as the LLaMA-7B model, which is reasonable, therefore, a larger learning rate for the critic and a smaller learning rate for the actor is preferred for stable training and fast convergence.
Inference. During inference, the critic is discarded and only the actor is needed. Furthermore, the alignment of the LLMs and embodied environments fully encoded in the LoRA parameters, which is normally 20 times less than the LLMs, e.g., 4.2M for LLaMA-7B. Therefore, the LoRA parame- ters can be a plug-and -nd--lay module of LLMs for generalizability across different environments.
4.4 PROMPT DESIGN
The prompts of observations and actions will significantly influence the generated policies by LLMs, which a path orthogonal way to the finetuning for the alignment between L.Mss and embodied environments. We summarize four principles for designing efficient prompts:
Prompts of observations and actions should be cohesive for concatenation. Observation prompts end with you should and the next step is to, indicating the start of action prompts.
Articles, i.e., the, a, and an, are important for action prompts. Most action prompts consist of a verb and noun, e.g., pick up the tomato. As LLMs are trained with high-quality corpus, they are sensitive to the articles. Therefore, pick up the tomato is better than pick up tomalo, where the latter leads to the extremely low probability on tomato.
Preferred actions should appear in the observation prompt. As observed in (Xu al.) 2022 Fu et al.) 2021), LLMs tend to assign higher probabilities to the repetitive tokens. Therefore, we can encourage LLMs to assign higher probability on preferred actions by emphasizing the nouns several times in the observation prompts. For example, if the observation prompt is see a tomato. My task is to make a tomato salad. I should, then the tomato will have a relatively high probability.
The same action can have different action prompts under different observations. For example, in Overcooked, when the agent carries bowl in hand, pick up the toamto can be replaced with put the tomato in the plate, where both action prompts have the same function in the environment, the latter one better fits with the context and thus has higher probability.
The objective of the prompt design is to represent the observations and actions in the understandable manner of LLMs, thus improving the alignment between LLMs and embodied environments.
5 EXPERIMENTS
5.1 EXPERIMENTS SETUP
We deploy our methods on the LLaMA-7B model with half-precision and compare the performance among five methods: PPO (adapted from CleanRL (Huang et al. (022aa). TWOSOME without finetuning (similar to SayCan (Brohan al., 2023) with affordance value set to 1), TWOSOME without action prompt normalization (similar to GLAM (Carta et al. (023) under decoder-only ar- chitecture and high-level actions) and TWOSOME with token normalization or word normalization. We mainly evaluate these methods in typical Dacision-making environment, Overcooked, and a simulated physical household environment, VirtualHome.
Overcooked An agent is placed in the 7x7 Overcooked kitchen, aiming to make and serve tomato salad and tomato-lettuce salad with the provided ingredients and tools in two tasks shown in Figure 4a and 4b In the second task, we add an additional ingredient, onion as a disruptor, to show the robustness of our method. The agent needs to explore and learn the correct order to cook the dish with the provided macro-actions, such Chop, Get-Tomato, and Go- Co-Cutting- Board.. The environment is partially observable. The agent only observes the objects within x 5 square centered on the agent. The reward involves +0.2 for chopping correct ingredient, +1 terminal reward for delivering the correct dish, -0.1 for delivering any wrong item, and -0.001 for every time step.
The environment is adapted from Xiao al. (2022) and Wu al. 2021
RKAIMMM md

---

Published as a conference paper at ICLR 2024
(a) Tomato Salad
(b) Tomato-lettuce Salad
(c) Food Preparation
(d) Entertainment
Figure 4: Experimental environments. Figure 4a and 4b show two tasks in Overcooked. Figure 4c and show two tasks in VirtualHome.
VirtualHome An agent is placed in a fully furnished household with various rich objects. Com- pared to Overcooked, this environment is more complex with larger action space. The agent uses macro-actions to interact with the environment such as walk to the living room, turn the TV and sit on the sofa, As shown in Figure 44 and [dd] we design two tasks. For the first task in Figure 4c the agent needs find the cold pancake For the second task shown in Figure 4d the agent plans to have some entertainment so it wants to find something to enjoy while watching TVV. Thus, it needs to pick up chips and milk in the kitchen, bring them to the living room, turn on the TV, sit on the sofa and enjoy. The challenge arises when the agent already holding both the milk and chips, lacks an additional hand turn on the TV. Con- sequently, it needs to learn to put at least one item on the nearby coffee table before operating the TV. Both tasks adopt sparse reward setting, only when the task finished, will the agent receive +1 reward. The environment is also partially observable. The agent can only see the objects in the current room. The environment is adapted from (Puig et al., (2018). more detailed introduction of these two environments can be found in Appendix [C] Our parameter-efficien framework enables us to complete all the experiments in one NVIDIA Tesla A A100 40GB GPU. All the hyperparameters can be found in Appendix D.2. Finally, provide the policy visualization and detailed analysis of prompt design for all tasks in Appendix and
the table and heat it with the microwave in the kitchen.
5.2 THE IMPACT OF DIFFERENT NORMALIZATIONS IN TWOSOME
Figure shows the performance finetuned TWOSOMES. TWOSOME with word normalization succeeds in learning the optimal policy among all tasks, exhibiting great performance, and high sam- ple efficiency over other methods, which is consistent with the analysis in Section [42] Except for the second Overcooked task, which is more difficult, TWOSOME without normalization learns quickly at the beginning but suffers a severe sudden drop in the the other three tasks. Without normaliza- tion, the unbalance over action distribution prompts happen to be reasonable actions but introduce extra training instability. TWOSOME with token normalization alleviates this instability slightly excessively, resulting in less data efficiency and slow convergence, as shown in the two overcooked tasks.
accelerate the training process if the shorter action
5.3 TWOSOME vs. BASELINES
Figure[5b|showe the performance among the typical RL method, PPO, prompt tuning method, TWO- SOME without finetuning, and our best method, TWOSOME with word normalization.
Comparison with Prompt Tuning. TWOSOME without finetuning fully relies on the capabilities of pre-trained LLMss with the designed prompts, demonstrating the L.M''s decision-making ability

---

Published as a conference paper at ICLR 2024
Tomato Salad
Tomato-lettuce Salad
Food Preparation
Entertainment
1.0
1.0
0.8
0.6
0.8-
0.8
Total Return
0.4
0.4-
0.4
0.4
0.2
.0.2
0k
20k
40k
60K
Búk
100k
0.2
ok 100k 200k 300k 400k 500k
0.2
0k
10k
20k
30k
46k
50k
0.0
0k
50k
100k Steps
150k
200k
Without Normaliztion on
Taken Normaliztion
Word Normalization
Optimal
(a) Comparison of performance among finetuned TWOSOME.
Tomato Salad
Tomato-lettuce Salad
Food Preparation
Entertainment
1.0
1.0
0.6
0.4
0.4-
Total
0.0-
-0.2
0.2
0.0
0k
20k
40k
60k Stops
80k
100k
Ok 100K 200K 300K 400k 500k
0k
10k
20k
30k
40k
50k
0k
50k
100K Steps
150k
200k
Steps
Stops
PPO
Without Fintuning
Word Normalization
Optimal
(b) Comparison of performance among TWOSOME with word normalization and baselines.
Figure 5: Comparison of performance among PPO and four TWOSOME variants: i) without fine- tuning, ii) without normalization, iii) with token normalization, and iv) with word normalization. PPO masks unrelated actions in both VirtualHome tasks. The results are averaged over 3 seeds. The non-trained method averaged over 100 episodes.
and the difficulty of the task. For the second task, which is the most difficult task, the non non-fine con-finetmeed LLM agent fails to finish the task completely. For the other three tasks, the low total returns with large variations indicate that the non-finetuned LLM agent can only finish these tasks with low probabilities. Prompt tuning can improve LLMs in solving easy decision makinn tasks but difficult to solve complex and long-horizontal tasks with more interferents in the environment. We conclude that all training methods including PPO and all TWOSOME variants surpass the performance of the prompt tuning method, emphasizing the necessity of aligning LLMs with environments via RL.
Comparison with PPO. TWOSOME with word normalization succeeds in learning the optimal policies in both Overcooked tasks using 10k and 80k sampled data, respectively. In contrast, PPO fails to learn the optimal policy and gets stuck the suboptimal due to partial observability. For the more complex second task, PPO even needs more than 500K steps converge. As for the two tasks in VirtualHome, we find that vanilla PPO cannot deal with the large action space and learns nothing, shown in Appendix C.3.I So mask all the unrelated actions in VirtualHome for PPO. The masked PPO manages to learn the optimal policy in the task of Food Preparation but still fails the task of Entertainment. The results demonstrate that by leveraging the power of LLM, our method can defeat the traditional RL methods in solving decision-making tasks with large action spaces.
All finetuned TWOSOME methods, matter whether they are normalized, manage overcome partial observability and achieve optimal performance in at least two seeds in the two Overcooked tasks. Even though the agent does not see the target object, e.g., tomato, it can still appear in the goal part of the observation prompt, such as your task is to make a salad consisting of tomato, main- taining the high probabilities of tomato related actions, while other unrelated objects like onion, not appeared in neither the observation nor the observation prompt, remain relatively low probabilities. This feature greatly facilitates the exploration process and helps agents to sample good actions.
5.4 OPEN-VOCABULARY TASK GENERALIZATION
We also evaluate the generalization ability of TWOSOME in eight new unseen tasks, LLMs' open- vocabulary feature enables TWOSOME to transfer learned skills to different tasks, while traditional RL agents do not have such ability. We compare the performance between finetuned TWO- SOME and non-tuned TWOSOME. o that for the first four tasks, which are similar to the agent's original trained task, though non-tuned TWOSOME somehow finish the tasks, fine-
8

---

Published as a conference paper at ICLR 2024
Task Ceneralization
1.0
TWOSOME Fintuning TWOSOME with Word Normalization
0.8
Rote
0.6
0.4
Choese
Hamburger
Apple Pie
Pizza
Washing Plate
Laundry
Food Preparation Entertainment
Tasks
Figure 6: Task Generalization Tests. TWOSOME without Finetuning and TWOSOME with word normalization tested in eight unseen tasks. Each task is tested 100 episodes. For the first four tasks, Cheese, Hamburger, Apple Pie and Pizza, we replace the pancake in the original Food Preparation task with the corresponding food see whether the agent can still finish the task. The following Washing Plate and Laundry tasks are more different but still have similar procedures to Food Preparation (agent needs put dishes clothes into the dishwasher or washing machine). For the last two crossover tests, TWOSOME agent trained in Food Preparation is tested in Entertainment and TWOSOME trained in Entertainment tested in Food Preparation.
tuned TWOSOME achieves perfect performance. Replacing objects has little impact on finetuned TWOSOME. For the following two more difffrrent tasks, Washing Plate and Laundry, there an ob- vious drop in the success rate of non-tuned TWOSOME, while finetuned TWOSOME still manages to complete the task. For the last two crossover tasks, where agents are tested in completely differ- ent task, TWOSOME still exhibits remarkable performance. Especially in the Entertainment task, where non-tuned TWOSOME can barely finish the task and finetuned TWOSOME still maintains 97% success rate. Total returns of each task are provided in Appendix B.2
5.5 IMPACT ONLINE FINETUNING
To investigate the impacts of online fine- tuning on the LLM's abilities, we eval- uate the models trained by TWOSOME with word normalization in Virtualhome on widely used NLP benchmarks (Gao jct al.] (02II. The models trained Food Preparation and Entertainment are named TWOSOME-FP and TWOSOME- E. respectively. The four benchmarks are ARC_C, HellaSwag, PIQA and MMLU, which are also reported in Touvron et al. 2023)) Results of the zero-shot performance loss of the ability of the language understanding of LLMs after the aligning with embodied environ- ments, and even sometimes brings minor improvements, The full results are in Appendix B.3
Table 1: Zero-shot performance on Language Model
Evaluation Harness (Gao et al. 2021.
ARR_C HellaSwag PIQA
MMLU
LLaMA-7B-hf
0.42
0.57
0.79
0.33
TWOSOME-FI
0.43
0.58
0.79
0.32
TWOSOME-E
0.43
0.58
0.79
0.30
displayed in Table T which demonstrates that there is significant
DISCUSSION AND CONCLUSION
In this paper, we propose TWOSOME, general online framework for efficiently aligning LLMs with embodied environments via RL solve decision-making tasks without requiring any prepared datasets or prior knowledge of environments and without significant loss of LLMs' original ability. Instead of letting LLMs generate actions directly. TWOSOME more controllable and feasible with better interpretability. since the impact of tuming prompts can be clearly reflected by the change of action probabilities. TWOSOME with word normalization exhibits significantly better sample effi ciency and performance compared to traditional RL. methods and prompt tuning methods. Moreover, TWOSOME exhibits a remarkable generalization ability to transfer learned skills and knowledge to unseen tasks. However, method suffers major limitation that training PPO agent from scratch far faster and cheaper than finetuning an LLM. TWOSOME needs to feed all the valid actions to the LLMs for every action sampling, resulting in multiple times the amount of computation and small batch size. We hope this work can provide a step toward the general autonomous agent, where LLMss self-improve by interacting with the world and harvesting true knowledge from practice.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
This work does not use any dataset. All the TWOSOME experimental code and environment code of Overcooked and VirtualHome are included the Supplementary Materials. We also provide videos for each task recorded by our best agent in the Supplementary Materials. All the hyperpa- rameters and network architecture we use can be found Appendix [.1 and [.2 We provide the
policy visualization and a detailed analysis of prompt design for all tasks in Appendix E and F Our parameter-effiient framework enables us to complete all the experiments in one NVIDIA Tesla A100 40GB GPU. Additional experimental results such as success rate and NL.P benchmarks can be found in Appendix B A detailed introduction to the Overcooked and VirtualHome environments, including observation space, action space, and macro actions use can be found in Appendix C
ACKNOWLEDGMENTS
This research is supported by the National Research Foundation Singapore and DSO National Labo- ratories under the AI Singapore Programme (AISGAward No: AISG2-GC-2023-009), This research is also supported by the National Research Foundation, Singapore under its Industry Alignment Fund - Pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material those of the author(s) and do not reflect the views of National Research Foundation, Singapore. The computational work for this work was partially performed on resources of the National Supercomputing Centre, Singapore (http://www.nscc.sg))
REFERENCES
Josh Abramson, Arun Ahuja, lain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, et al. Imitating interactive intelli- gence. arXiv preprint 2rXiv:2012.05672, 2020.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as I can, not as I say: Grounding language in robotic affordances. In Conference on Robot Learning, pp. 287-318, 2023.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of the 34th International Conference Neural Information Processing Systems, pp. 1877-1901, 2020.
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud. and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. arXiv preprint arXiv:2332.002662 2023.
Maxime Boisserr, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint / 2018.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023// 2022.
Ishita Dasgupta, Christine Kaeser-Chee Kenneth Marino, Arun Ahuja, Sheila Babayan, Felix Hill, and Rob Fergus. Collaborating with language models for embodied reasoning. In Second Work- shop on Language and Reinforcement Learning, 2022.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen. Chi-Min Chan, Weize Chen, al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-22) 2355, 2023.
Danny Driess. Fei Xia, Mehdi SM Sajjadi, Corey Lynch. Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodied multi- modal language model. arXiv preprint 2300 2023.
10

---

Published as a conference paper at ICLR 2024
Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. MineDojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv:2206.08835. 2022.
Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. A theoretical analysis of the repetition problem in text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp- 12848- 2021.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang. and Andy Zou. A framework for few-shot lan- guage model evaluation, September 2021. URL /tttp:://ao..orr/1105221111 / 5371628
Yicong Hong, Qi Wu, Yuankai Qi. Cristian Rodriguez-Opazo, and Stephen Gould. VIn-bert: A recurrent vision-and-language bert for navigation. Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 1643-1653, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, pp. 2790-2799, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022.
Hengyuan Hu and Dorsa Sadigh. Language instructed reinforcement learning for human-Al coor- dination. arXiv preprint 2rXiv:2304.07297, 2023.
Shengyi Huang, Rousslan Fernand JulienDossa Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João GM Araújo. CleanRL: High-quality single-file implementations of deep reinforcement learning algorithms. The Journal of Machine Learning Research, 23(1):12585. 12602, 2022a.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pp. 9118 9118-9147, 2022b.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In Conference Robot Learning, 2022c
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yonggiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.0300 2022.
Siddharth Karamcheti, Megha Srivastava, Percy Liang, and Dorsa Sadigh. LILA: Language- informed latent actions. In Conference on Robot Learning, pp. 1379-1390, 2021.
Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but ef- fective: CLIP embeddings for embodied AL. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 14809-14818, 2022.
Geunwoo Kim, Pierre Baldi, and Stephen McAlcer. Language models can solve computer tasks. arXiv preprint arxiv:230..1749) 2023.
George Konidaris, Scott Kuindersma, Roderie Grupen, and Andrew Barto. Autonomous skill acqui- sition on a mobile manipulator. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1468-1473, 2011.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline Q- learning on diverse multi-task data both scales and generalizes. In The Eleventh International Conference Learning Representations, 2023.

---

Published as a conference paper at ICLR 2024
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp.
2021.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, and Yuke Zhu. Pre-trained language models for interactive decision-making Advances in Neural Infor- mation P Processing Systems, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning Optimizing continuous prompts for generation. In Pro Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers),
pp. 4582-4597 2021.
J. Liang, Wenlong Huang, F. Xia, Peng Xu, Karol Hausman, Brian Ichter, Peter R. Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. ArXiv, abs/2209.07755 2022.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language mod- els. arXiv preprint 2rXiv:2304.09842, 2023.
Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Improving visoonanddlanguage navigation with image-text pairs from the web. In Computer 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16. pp. 259-274, Springer, 2020.
OpenAl. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback. In Ad- vances Neural Information Processing Systems, 2022.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Kumar Gupta. The un- surprising effectiveness of pre-trained vision models for control. International Conference on Machine Learning, 2022.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Tor- ralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8494-8502, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2 21()):5485-5511, 2020.
Allen Z. Ren, Bharat Govil, Tsung-Yer Yen Yang, Karthik R Narasimhan, and Anirudha Majumdar. Leveraging language for accelerated learning of tool manipulation. In Conference Robot Learning, pp. 1531-1541, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXivv1707.066 2017.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hug- gingGPT: Solving AI tasks with ChatGPT and its friends in Huggingface. arXiv preprint arXiv:2303. 17580, 2023.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: An autonomous agent with dynamic memory and self-refiection. arXiv preprint arXiv:2303. 13366, 2023.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference Robot Learning, pp. 894-906. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2024
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXhv:2209.11300 13002 2022.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Georgios Theocharous and Leslie Kaelbling. Approximate planning in pomdps with macro-actions.
Advances in neural information processing systems, 16, 2003.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint ar1iv:3305. 16291, 2023a.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma. and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint 2321v:230..01560. 2023b.
Sarah A Wu, Rose E Wang. James A Evans, Joshua B Tenenbaum, David Parkes, and Max KleimannWeiner. Too many cooks: Bayesian inference for coordinating multi-agent collabora- tion. Topics in Cognitive Science, 13(2):414-42, 2021.
Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. Language models meet world models: Embodied experiences enhance language models. arXiv preprint arXiv:2205. 10626, 2023.
Yuchen Xiao, Weihao Tan, and Christopher Amato. Asynchronous actor-critic for multi-agent rein- forcement learning. In Advances in Neural Information Processing Systems, 2022.
Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. arXiv preprint arXiv:2206.022. 02369, 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
Andy Zeng, Adrian S. Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveck Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lec, Vincent Vanhoucke, and Peter R. Flo- rence. Socratic models: Composing zero-shot multimodal reasoning with language. ArXiv, abs/2204.00598, 2022.
Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exmppl. prompting with memory for computer control. 2023.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A MORE RELATED WORK
Embodied Agent with L.Ms. The successful integration of language as a semantically rich input for interactive decision making highlights the crucial role of LLMss facilitating interaction and
decision-making processes (Abramson et al., 2020; Karamcheti et al. 2021 Li et al.) 2022). LLMs
are also applied in various environments to aid robot navigation (Parisi et al. 2022
Hong et al.,
2021 Majumdar al., 2020) and manipulation (Tiang et al., 2022 Ren et al.
2023
Khandelwal
et al. 2022). Recently, there have been a large number of methods that utilize LLMs to enhance
planning and reasoning capabilities in embodied agents. SayCan (Brohan al., 2023) assesses
the affordance of candidate actions by multiplying their probabilities under L.LMs with a value
function. Zeng al. (2022) combine the LLM with a visual-language model and pre-trained
language-conditioned policy (Shridhar et al. (022) to enable open vocabulary robotic tasks. Huang
ct al. (2022b) demonstrate that LLMs can be employed for planning and executing simple household tasks. They ground JLM--ennrated actions by comparing their embeddings with predefined list of acceptable actions. To incorporate environment feedback, Inner Monologue (Huang et al.] 2022c)
extends SayCan using a closed-loop principle. This principle is also applied in related works such
as (Yao et al.] 2023; Huang et al. [022cc Kim et al., 2023; |Singh et al., 2022] Liang et al.) 2022]
Shinn et al.) 2023 Wang et al. 2023b a) to continuously monitor agent behaviors and refine and adjust plans accordingly for tasks such as computer automation, Minecraft, etc. Furthermore, there approaches that prompt LLMs to generate temporal-abstractee abstracted actions (Zheng et al.) 2023). Dasgupta et al. (2022) employ the L.M as a planner and success detector for an agent with their actor module necessitates pre-training with RL to enable the agent to follow natural language instructions. While these works demonstrate impressive results, they rely too heavily on the inherent capabilities of powerful LLMs, like GPT4 and PaLM (Chowdhery al. (022), which are difficult apply to smaller LLMs with weaker reasoning abilities, like LLaMA-7B.
Concurrent to our work. GLAM (Carta et al., 2023 utilizes RL finetuning to achieve functional grounding of LLMs. However, they focus on simple primitive actions (turn left, turn right, go forward, etc.) evaluated in toy environments, Baby (Chevalier-ooiseert smaller encoder-decoder LLM, Flan-T5-780M. These primitive actions have a similar number of tokens and less meaningful semantics, resulting in underutilizing the capabilities of LLMs, and failing to observe the impact of prompt design and address the unbalance over action space, resulting in additional instability and poor robustness.
al. 2018) with a much
Finetuning LLMs. Parameter-efficient finetuning (PEFT) can significantly reduce the number of parameters to tune LLMS while with minor loss of the performance (Ding et al. 2023). Prompt tuning (Lester et al. 2 and prefix tuning (Li & Liang 2021) prepend additional tunable tokens to the imput or hidden layers, and adapter tuning (Houlsby et al., 2019) introduces the adapter, i.., the bottleneck layers, to LLMs, where only the introduced tokens and adapters finetuned. Recently, low-rank adaption (LoRA) (Hu et al. 2022) introduces low-rank matrices to approximate parameter updates.
Reinforcement learning from human feedback (RLHF) (Ouyang et al. 2022) proves that RL is an effective way make alignments with human preferences and proposes novel perspective regard- ing LLMs as actors under actor-critic framework. Our work is also motivated by this method. While RLHF needs to train a reward model simulate human preferences, these signals are naturally present in most RL environments, which makes it possible to align LLMs with certain environments.
Another concurrent work (Xiang et al., 2023) leverages VirtualHome collect embodied experi- ences by random sampling and use supervised learning to finetune LLMs to these embodied knowl- edge. They finally apply a simple RL method, MCTS, to plan without training, instead of directly generating plans. It is worth pointing out that our method also applies their supervised learning pro- cess to learn the embodied knowledge and get more familiar with the environments. Our online PPO sampling is much more efficient and feasible than their random sampling, e.g. in the overcooked tasks, it is almost impossible to complete the task by random sampling. In addition, an online learn- ing PPO is obviously better than simple MCTS without training. Compared with this work, our method is a simpler end-to-end online framework, which can automatically acquire new knowledge and solve Dacision-making tasks by interacting with environments without any prepared datasets.
14

---

Published as a conference paper at ICLR 2024
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 SUCCESS RATE IN OVERCOOKED AND VIRTUAL.OME
Besides the total reward provided in 5b here also provide the final success rates of all methods in Table 2 which tells almost the same story. Our TWOSOME with word normalization method achieves 100% success rate across all tasks, exhibiting great stability and remarkable performance.
Table 2: Final success Rate different methods Overcooked and VirtualHome. Results are averaged over three seeds. Each seed is run for 100 episodes. Norm is short for normalization.
TWOSOME
Task
PPO
W/O Finetuning
W/O Norm
Token Norm
Word Norm
Tomato Salad
1 0
0.36 + 0.04
0.67 + 0.47
1
1 1 0
Tomato-lettuce Salad
1 0
0 - 0 0.81 + 0.01
1 0
1
1 0
Food Preparation
1
1 0
1
1 0
Entertainment
0.33 + 0.47
0.45 + 0.05
0.67 + 0.47
1
1 0
B.2 PERFORMANCE TWOSOME AND SAYCAN EIGHT UNSEEN TASKS
and and 4 show the final success rate and total return of our best method TWOSOME with word normalization and our baseline. WenCCan(oonfinttuune TWSOME) in eight unseen tasks. Each task is run for 100 episodes. TWOSOME exhibits superior generalization across all tasks.
Table
Table 3: Final success rate of TWOSOME and SayCan in eight unseen tasks.
Task TWOSOME
Cheese
Hamhurger
Applepie
Pizza
Washing Plate
Laundry
Food Preparation
Entertainment
0.77
0.97
SayCan
0.66
0.79
0.85
0.64
0.65
0.35
0.66
0.04
Table 4: Total return of TWOSOME and SayCan in eight unseen tasks.
Task TWOSOME
Cheese 0.77=0.01
Baabbuggrr 0.77+0001 0.21+0.17
Applepie 0.77+0.01
Pizza 0.77+0.01 0.17+0.18
Washing Plate
Laundry 0.76++0.04 0.08++0.13
Food Preparation
Entertainment
0.77+0.02 0.18+0.17
0.26++0.24 0.20
0.43++0.14
SayCan
0.17
0.23+40.77
0.01+005
B.3 EVALUATION ON NLP BENCHMARKS
To investigate the impacts of the PPO online finetuning on the LLM's abilities, we evaluate models trained by TWOSOME with word normalization in the virtual home tasks on widely used NLP benchmasrks (Gao et al. 202T). The models trained in Food Preparation and Entertainment named TWOSOME-FI FP and TWOSOME-E, respectively. The four benchmarks are ARC_ C_, HellaSwag, PIQA and MMLU, which are also reported in (Touvron et al. 2023). The results on ARC_ Bec__Chllenge, HellaSwag and PIQA displayed in Table and the results of MMLU are dis- All results are calculated following the default configurations in (Gao et al.
played in Table 6
2021
Table 5: Zero-shot performance Common Sense Reasoning tasks.
LLaMA
TWOSOME-FP
TWOSOME-E
ARC_
0.42 0.01
0.43 + 0.01
0.43 + 0.01
HellaSwag
0.57 + 0.00
0.58 + 0.00
0.58 + 0.00
PIQA
0.79 0.01
0.79 0.01
0.79 + 0.01
15

---

Published as a conference paper at ICLR 2024
Table 6: Zero-shot performance Massive Multitask Language Understanding (MMLU).
LLaMA
TWOSOME-FP
TWOSOME-E
cestract_algebae
0.33 0.05
0.30 0.05
0.28 0.05
anatomy
0.38 + 0.04
0.39 + 0.04
0.36 + 0.04
astronomy
0.34 0.04
0.32 + 0.04
0.24 0.03
business_ _ethics
sesiness_eehicss
0.37 + 0.05
0.31 0.05
0.32
clinical_knon knowledge
0.37 0.03
0.35 + 0.03
0.31 0.03
saaleeebbooogy
0.37 + 0.04
0.35 + 0.04
0.34
0.28 = 0.05
0,30 + 0.05
0.27 0.04
lergge_computer_SSe
science
0.25 + 0.04
0.26 + 0.04
0.31 + 0.05
gallege__mthemattee
0.34 0.05
0.35 + 0.05
0.31 + 0.05
medicine
0.31 0.04
0.32 + 0.04
0.27 0.03
follege_physttee
0.25 + 0.04
0.25 + 0.04
0.20 + 0.04
Computer_securee
0.38 0.05
0.40 0.05
0.40 +.0
conceptual_physst
0.32 0.03
0.34 0.03
0.03
econometrics
0.23 0.04
0.20 0.04
0.21
electrical
instrrccal_eggnneering
0.26 0.04
0.26
0.24 0.02
0.23
0
Jormal_logic formal
0.34 0.04
0
0.27
global_face facts
0.32
0,29 0.05
0.26
sa__school_bboloee
0.32 + 0.03
0.32 + 0.03
0.27
high_sche sech_school__g -ecemmstrryyy
0,21 0.03
0 0.03
0.19
high_schoop ltt__/omopppppceeeeee science
0
0.27 0.04
0.28 0.05
high_school_eurppe per_uopppppe_hhttrrry
0,40 + 0.04
0,38 + 0.04
gong_schholl_eogrrphh
0.28 0.03
0.29 + 0.03
0.26
high government and politics
0
0.30
0.03
high_soo school highol macroeconomies
0.28 0.02
0.26 0.02
0.24
0.24 0.03
0.25 0.03
0.27 0.03
gagh__choo__merreccoomme
0.26
0.26
0.24
geggh_scholl_pee physics
0.26
0.27 0.04
0.28 0.04
high_school_pp sccool_psycholog..
0.33
0.32 +.02
0.27
high_school_statioe
0.22 0.03
0.19
0.17
high_schoe gigh_school_us us history
+.00
0,41 +
0.40
Seceh_shhoo__wolee history
0.38
0.40
0.42
human_aging
0,43 0.03
+
0,41 +
human_ syeuulitt
0.34 0.04
0.32 0.04
0.04
international law
0,47 0.05
0,43 0.05
0.36
jurisprudence
0.36 0.05
0.35 0.05
0.35
logical fallacies
0.36 0.04
0.35 + 0.04
0.26 0.03
machine _earning
0.27 0.04
0.28 0.04
0.33 0.04
management
0.28 0.04
0.31 0.05
0
marketing Seadical_genetics
0.38 1.8:000 0.03
0.38 0.03
0.39 +.09
0.42 +.05
0.38 + 0.05
0.36
miscellancous
0.45
0.02
0.02
moral disputes
0.35 0.03
0.34
0.29
moral_scente mooaa_sscenario
0.24 0.01
0.01
0.24
nutrition philosophy
0.33 0.03
0.03
0.30+0003
0.37 0.03
0.03
0.27 0.03
prehistory
0.39 + 0.03
0.34
0.31 0.03
professional _acouunniccccngnggggg
0.27 + 0.03
0.28
226
professional law
0.28 + 0.01
0.28
0.29
professional medicine
0,25 + 0.03
0.24
0.25
professional_ psyyccooo
0.34 0.02
0.31 0.02
0.30 0.02
public_relation
0.35 + 0.05
0.33 + 0.04
+
security_studics
0.27 0.03 0.03
0.27
0.23 0.23
sociology us us_forrign__poiie virology
0.41 + 0.03
0.37 + 0.03
0.37 0.03
0.42 0.05
0.41 0.05
0.41 0.05
0.36 + 0.04
0.36
0.33 0.04
world religions
0.46 0.04
0.48 0.04
4.04
16

---

Published as a conference paper at ICLR 2024
C DETAILS OF EMBODIED ENVIRONMENTS
In this section, we present the details of the embodied environments Overcooked and VirtualHome.
C.I OVERCOOKED
(a) Tomato Salad
(b) Tomato-lettuce Salad
Figure 7: Overcooked environments.
Goal. An agent is placed in the Overcooked kitchen and aims to cook certain dish with the provided ingredients and tools and deliver it to the 'star' cell as soon as possible. Agents have to learn the correct procedure in terms of picking up raw vegetables, chopping them, and merging them in bowl before delivering. Figure 7a and 7b show the recipes for making tomato salad and tomato-lettuce
salad.
Observation Space. The environment is a 7x7 grid world involving ingredients, bowls, cutting boards, and delivery counter. For the task of making tomato salad, there is only tomato, one bowl, one cutting board, and one delivery counter available in the environment. For the task of making tomato-letuuce salad, one tomato, and one delivery cell are provided in the environment. The onion serves as an interferent, though does not appear in the recipe. The global state information consists of the positions of the agent and the above objects, and the status of each ingredient: chopped or unchopped. The environment partially observable. For the observation, the entities within 5 5 square centered on the agent. Other unseen objects are masked in the observation. And the initial positions of all the objects are known to agents. The symbolic raw observations are directly fed into PPO as input and converted to prompts with scripts serve as LLMs' input.
lettuce, one onion, one bowl,
cutting boards,
agent only observes the positions and status of
Primitive-Action Space. Agents use five primitive actions: up, down, left, right, and stay to interact with the environment. Agents can move around and achieve picking, placing, chopping, and deliv- ering by standing next to the corresponding cell and moving against it. All the macro-actions based on these primitive actions.
Space. In this work, mainly focus using high-level macro-actions, since they usually have richer semantics. Every macro-action may take several time steps to complete. The main function of each macro-action and the corresponding termination conditions is exactly the same as the setting in (Xiao et al. 2022) despite that we only have one agent. Here we list and briefly describe all the macro-actionn we use: Chop, chops raw ingredient into pieces when the agent stands next to a cutting board with an unchopped ingredient on it. Get-Tomato, Get-Lettuce, Get-Onion,
Macro-Action
Go-Cutting- Boatting-Board1122 and Deliver, which navigate the agent to the location of the corresponding object and execute the corresponding action.
In the first task of making tomato salad, Get-Tomato, Get-Bowl, Go-Cutting- and Deliver are available. In the second task of making tomato-lettuce salad all the macro-actions are valid.
Dynamics: The transition in this task deterministic. If an agent delivers any wrong item, the item will be reset to its initial position.
Reward: +0.2 for chopping correct ingredient, +1 terminal reward for delivering the correct dish, -0.1 for delivering any wrong dish, and -0.001 for every timestep.
Episode Termination: Each episode terminates either when the agent successfully delivers the target dish to the delivery counter or reaches the maximal time steps, 200.
17

---

Published as a conference paper at ICLR 2024
C.2 VIRTUALHOME
(a) Food Preparation
(b) Entertainment
Figure VirtualHome environments.
Goal. An agent, represented as a humanoid avatar, is placed in a fully furnished household with various rich objects to interact with. Figure 8a and [b) show two tasks we design for the agent to complete. For the first task of food preparation, the agent needs to find the pancake on the table in the kitchen and put it in the microwave to heat. For the second task of entertainment, the agent needs to find and bring chips and milk, from the kitchen to the living room and succeed sitting on the sofa with the TV open and the milk and chips are nearby. The challenge arises when the agent, already holding both the milk and chips, lacks additional hand to turn the TV. Consequently, it needs to Iearn to put at least one item on the nearby coffee table before operating the TV.
Observation Space. The environment is also partially observable. The agent can only see the objects the current room and does not the objects in the other room. The observation consists of a set of bool values, representing whether the agent sees the relative object, whether these objects close to the agent and the status of the objects, such as whether the TV is turned on and whether the milk is on the coffee table. The symbolic raw observations are directly fed into PPO input and converted to prompts with scripts to serve as LLMs' input.
C.3 ACTION SPACE
To simulate the daily activities, all the actions in VirtualHome are macro-actions. Every macro- action takes only time step for execution. Here, we list and briefly describe all the macro-actions as Table 7
Table Actions in VirtualHome and their corresponding descriptions
Action
Description
[STANDUP]
Stand up
[SIT] <Object> <Object> (object_ id)
Sit on the Object[object id)
[WALK] <Object> (object id)
Walk the Object id) Grab the Object(object_ id) Open the Object(object __dd Close the Object(object_L) jid_ Switch/Turn on the Object(object, ji)
[GRAB] <Objecr> (object_id)
[OPEN] <Object> (object_id) id)
[CLOSE] <Object> (object_id)
[SWITCHON] <Object> (object_id)
[SWITCHOFF] <Ohject> (object_id)
Put Switch/Tum off the Object[ jidd)
[PUTIN] <Ohjectl> (objectl _id) <Ohject2> (object2_id) ji) [PUTRACK] <Objectl> (objectl jd) <Ohject2> (object2_in _dd
iid) in the Object2(objecr2 id) Put the (object( -iseeettlliiiii_eee id) on the Objec22 id)
For the first task of heating pancake, there are four rooms and two items in the environment and their corresponding object are as follows: livingroom(267), kitchen( I I), bathroom( (72), bed- room(210), p2ncake(62), microwave(100). For the second task of watching TV, there are four rooms and five items in the environment and their corresponding object id are as follows: livin- groom(267), kitchen( [1), bathroom( /72), bedroom 220), chips(61), milk(46), TV(297), sofa(276), coffeetable(268). 2668). For their actions, corresponding action prompts and abbreviated labels, please refer Table 8 and Table
Only when the agent is close to the object, can the agent operate the object. For example, the agent needs to walk to the TV before turning it on.
18

---

Published as a conference paper at ICLR 2024
Table 8: Actions, action prompts and abbreviated labels in the heating pancake task
Action
Action Prompt
Label
[WALK]
<hvingroom>> (267)
walk to the living room
Walk Livingroom
[WALK]
<kitchen> (11)
walk to the kitchen walk to the bathroom walk to the hedroom reach for the pancake move to the microwave grab the pancake open the microwave close the microwave put the pancake the mierowave
Kalk--itchen
[WAKK]
bathhooom> (172)
Balk-BBthrooo
[WALK] < hedroom> (210)
Walk-BBdroom
[WALK] <pancke>> (62)
Walk-Pancake
[WALK] <micrwave> (109)
Walk-Micree Mill-MMce
[GRAB] <pancake> (62)
Grab-Pancake
[OPEN]
<microwave> (109)
Open- Open-MMcrowwve
(CLOSE]
<microwave> (109)
[PUTIN] <pancake> (62) <microwave> (109)
Monti-Panncaee-a Pancaaa-MMeoeowwav
Table 9: Actions, action prompts and abbreviated labels in the watching TV task
Action
Action Prompt
Label
[WALK]
<livingroom> (267)
walk to the living room
[WALK] <kitchen> (11)
walk to the kitchen
[WALK] <hathroom> (172)
walk the hathroom
Walk-Bathroom
[WALK] <bedroom>> (210)
walk to bedroom reach for the chips reach for the milk move to the coffee table
Walk-Bedroom
[WALK] <chips> (61)
Walk-Chips
[WALK]] <milk> (46)
Walk-Milk
[WALK] <coffcetable> (268)
Walk- Coffeetable
[WALK] <TV> (297)
move the TV move to the sofa grab the chips grab the milk the TV turn off the TV take a seat on the sofa stand up from the sofa put the chips on the coffee table put the milk the coffce table
Walk-TV
[WALK] <sofa> (276)
Walk-Sofa
[GRAB] <chips> (61)
Grab-Chips
[GRAB] <milk> (46)
Grab-Milk
[SWITCHON] <TV> (297)
Switchon-TV
[SWITCHOFF] <TV> (297)
Switchoff-TV
[SIT] <sofa> (276)
Sit-Sofa
[STANDUP]
Standup-Sofa
[PUTBACK] <chips> (61) <coffectable> (268) [PUTBACK]
Puthack-Chinn Chhis--Chfeeettte
<milk> (46) <coffeetable> (268)
Dynamics: The transition in this task is deterministie.
Reward: We adopt a sparse reward setting, where only when the task finished will the agent receive +1 reward.
Each episode terminates either when the agent successfully finishes the task reaches the maximal time steps, 50 since every macro-action takes only time step for execution. For the task heating pancake, the agent succeeds when it places the pancake in the microwave and closes the microwave. For the task of watching TV, the agent succeeds when it sits the sofa with TV turned on and chips and milk are on the coffee table or hold in hand.
Episode Termination:
C.3.1 VANILLA PPO VS PPO WITH ACTION MASK
As mentioned Section 5.3, VirtualHome environment, vanilla PPO without action mask cannot learn anything for the two tasks due to the large action space. Figure shows the performance of vanilla PPO without action mask and PPO with action mask.
Watch
0.5
0.4
0.44
0.2
0K
104
20c
304
60%
5(6
24
50c
103k
1524
2004
S0055
Stago
Optimal
Figure 9: Performance of PPO without action mask and PPO with action mask in VirtualHome tasks.
19

---

Published as a conference paper at ICLR 2024
D TRAINING DETAILS
Our results are mainly generated on single NVIDIA Tesla A100 and NVIDIA RTX A6000 GPU. For all domains, we use the same architecture, which is shown in Figure I0 Throughout all the experiments, we conducted training and testing using the half-precision float16.
value
action
MLP
Env
LLaMA-7B-hf
(frozen)
LoRA
prompt reward
Figure 10: Parameter efficient architecture
D.1 NETWORK ARCHITECTURE
The same neural network architecture is applied to both the actor and critic networks across TWO- SOME method. In the LLaMA-7B model. additional MLP layers are incorporated into the last transformer block to serve as the critic. The critic's MLPs take the last token of the observation prompt as input and output the estimated value of the observation prompt. The critic utilizes a 3- layer MLP structure, with the number of neurons in each layer being 1024. 512, and 1, respectively. ReLU employed the activation function. On the other hand, the actor consists of the frozen LLaMA-7B model with the augmentation of LoRA parameters. The policy optimizer uses Adam W with an epsilon value of le-5 and weight decay of The critic optimizer uses Adam with an ep- silon value of le-5 and the default weight decay of O. In our experiments, we found that the default epsilon value of 1c-8 in the policy optimizer's AdamW can lead to unstable training and potential "nan" outputs in Lora.
In the PPO method, the critic network remains a 3-layer MLP architecture with 64 neurons in each layer and single neuron the final output. However, the main distinction from TWOSOME lies in the choice of activation function, which is Tanh. Meanwhile, the actor in the PPO method also adopts 3-layer MLP architecture. The number of neurons in the first two layers is the same the critic, which is 64. The activation function used is Tanh. The final layer's output depends on the number of actions in the task. For example, in the Entertainment task, there are 17 actions, while in the Food Preparation task, there are 10 actions. The optimizer uses Adam with an epsilon value of 1e-5 and a weight decay of 0.
D.2 HYPERPARAMETERS FOR TWOSOME AND PPO
In following subsections. we first list the hyper-parameter candidates used for training TWOSOME via grid search in the corresponding task, and then show the hyper-parameter table with the parameters used by TWOSOME. and PPO achieving the best performance. We choose the best performance of each method depending on its final convergence value and convergence efficiency.
20

---

Published as a conference paper at ICLR 2024
Table 10: Overcooked syper-parameter candidates of TWOSOME for grid search training
Learning rate pair (policy,critic)
(le-6, 5e-5), (Se-7, 5e-5), (Se-7, le-5)
(3c-6, 5e-5), (1e-6, le-5) (200, 0.99), (50, 0.95)
Max episode steps and gamma Total time steps
1e5, 5e5
First group parameters
Second group parameters
Number of environments
4
4
Number of steps per environment policy rollout
32
128
Number of mini-batches of policy
32
128
Number mini-batches critic
4
16
Number of epochs to update policy
1
1
Gradient checkpoint steps
8
32
Surrogate clipping coefficient
0.2
0.2
Coefficient of the entropy Coefficient of the value function
0.01
0,01
0.5
0.5
Maximum norm for the gradient clipping Target KL divergence threshold
0.5
0.5
0.02
0.02
Table 11: Overcooked hyper-parameter used of TWOSOME in Tomato Salad task
Learning rate pair (policy.critic)
(5e-7, le-5)
Max episode steps and gamma
(200, 0.99)
Total time steps
5e5
Number of environments
4
Number of steps per environment policy rollout
32
Number of mini-batches of policy
32
Number of umini-batche of critic
4
Number of epochs to update policy
1
Gradient checkpoint steps
8
Surrogate clipping coefficient Coefficient of the entropy Coefficient the value function
0.2
0.01
0.5
Maximum norm for the gradient clipping
0.5
Target KL divergence threshold
0.02
Table 12: Overcooked pappmeer used of TWOSOME in Tomato Lettuce Salad task
Learning rate pair (policy.critic)
(5e-7, le-5)
Max episode steps and gamma
(200, 0.99)
Total time steps
5e5
Number environments
4
Number of steps per environment policy rollout
32
Number of mini-batches of policy
32
Number mini-batcha of critic
4
Number of epochs to update policy
1
Gradient checkpoint steps Surrogate clipping coefficient
8
0.2
Coefficient of the entropy
0.01
Coefficient of the value function
0.5
Maximum norm for the gradient clipping
0.5
Target KL divergence threshold
0.02
21

---

Published as a conference paper at ICLR 2024
Table 13: Overcooked hyper-parametts candidates of PPO for grid search training
Learning rate
le-4, 2.5e-4, 5e-4, le-3
Max episode steps and gamma
(200, 0.99), (50, 0.95)
Total time steps
le5, 5e5
Number of environments
4
Number of steps per environment policy rollout
128
Number of maii-bathees of policy
4
Number of epochs to update policy
4
Surrogate clipping coefficient Coefficient of the entropy
0.2
0.01
Coefficient of the value function
0.5
Maximum norm for the gradient clipping Target KL divergence threshold
0.5
0.02
Table 14: Overcooked papermparaetter used of PPO in Tomato Salad task
Learning
le-4
Max episode steps and gamma
(200, 0.99)
Total time steps
5e5
Number of environments
4
Number steps per environment policy rollout
128
Number of mini-batches of policy
4
Number epochs to update policy
4
Surrogate clipping coefficient
0.2
Coefficient of the entropy
0.01
Coefficient of the value function
0.5
Maximum norm for the gradient clipping
0.5
Target KL divergence threshold
0.02
Table 15: Overcooked hyper-parameter used of PPO in Tomato Lettuce Salad task
Learning rate
5e-4
Max episode steps and gamma
(200, 0.99)
Total time steps
5e5
Number of environments
4
Number of steps per environment policy rollout
128
Number of mini-batches of policy
4
Number of epochs update policy
4
Surrogate clipping coeflicient
0.2
Coefficient of the entropy Coefficient of the value function Maximum norm for the gradient clipping
0.01
0.5
0.5
Target KL divergence threshold
0.02
22

---

Published as a conference paper at ICLR 2024
Table 16: Virtualhome
candidates of TWOSOME for grid search training
Learning rate pair (policy,critic)
(1e-6, 5e-5), (5e-7, 5e-5), (5e-7, le-5)
(3c-6, 5c-5), (1e-6, 1e-5) (200, 0.99), (50, 0.95)
Max episode steps and gamma Total time steps
5e4, le5, 2e5
First
Second
Third
Forth
Number of environments
4
4
4
Number of steps per environment policy rollout
32
32
128
Number of mini-batches of policy
32
64
64
128
Number mini-batches critic
8
16
Number of epochs to update policy
1
1
1
Gradient checkpoint steps
16
16
32
Surrogate clipping coefficient
0.2
0.2
0.2
0.2
Coefficient of the entropy
0.01
0.01
0.01
0.01
Coefficient of the value function
0.5
0.5
0.5
0.5
Maximum norm for the gradient clipping
0.5
0.5
0.5
0.5
Target KL divergence threshold
0.02
0.02
0.02
0,02
Table 17: Virtualhome hyper-parameter used of TWOSOME in the task of Food Preparation
Learning rate pair (policy.critic)
(1e-6, 5e-5) (50, 0.95)
Max episode steps and gamma
Total time steps
5e4
Number of environments
4
Number of steps per environment policy rollout
32
Number of mini-batches of policy
32
Number of
of critic
4
Number of epochs to update policy
1
Gradient checkpoint steps
8
Surrogate clipping coefficient Coefficient of the entropy Coefficient the value function
0.2
0.01
0.5
Maximum norm for the gradient clipping
0.5
Target KL divergence threshold
0.02
Table 18: Virtualhome hyper-paramend used of TWOSOME in the task of Entertainment
Learning rate pair (policy,critic)
(1e-6, 5e=5)
Max episode steps and gamma
(50, 0.95)
Total time steps
2e5
Number environments
4
Number of steps per environment policy rollout
32
Number of mini-batches of policy
32
Number mini-batcha of critic
4
Number of epochs to update policy
1
Gradient checkpoint steps Surrogate clipping coefficient
8
0.2
Coefficient of the entropy
0.01
Coefficient of the value function
0.5
Maximum norm for the gradient clipping
0.5
Target KL divergence threshold
0.02
23

---

Published as a conference paper at ICLR 2024
Table 19: Virtualhome hyper-parameter candidates of PPO for grid search training
Learning rate
le-4, 2.5e-4, 5e-4, le-3 (200, 0.99), (50, 0.95)
Max episode steps and gamma
Total time steps
5e4, 1e5, 2c5 4
Number of environments
Number of steps per environment policy rollout
128
Number of maii-bathees of policy
4
Number of epochs to update policy
4
Surrogate clipping coefficient
0.2
Coefficient of the entropy
0.01
Coefficient of the value function
0.5
Maximum norm for the gradient clipping Target KL divergence threshold
0.5
0.02
Table 20: Virtualhome syper-paramettr used of PPO in the task of Food Preparation
Learning rate
le-3
Max episode steps and gamma
(50, 0.95)
Total time steps
5e4
Number of environments
4
Number of steps per environment policy rollout
128
Number of mini-batches of policy
4
Number of epochs to update policy
4
Surrogate clipping coefficient
0.2
Coefficient of the entropy
0.01
Coefficient of the value function
0.5
Maximum norm for the gradient clipping
0.5
Target KL divergence threshold
0.02
Table 21: Virtualhome hyper-parameter used of PPO in the task of Entertainment
Learning rate
le-3
Max episode steps and gamma
(50, 0.95)
Total time steps
2e5
Number of environments
4
Number of steps per environment policy rollout
128
Number of mini-batches of policy
4
Number of epochs update policy
4
Surrogate clipping coefficient
0.2
Coefficient of the entropy Coefficient of the value function
0.01
0.5
Maximum norm for the gradient clipping
0.5
Target KL divergence threshold
0.02
24

---

Published as a conference paper at ICLR 2024
E BEHAVIOR VISUALIZATION
In this section, we display the optimal behaviors learned by TWOSOME with word normalization under all considered domains. We also provide corresponding videos for each task in the Supple- mentary Materials.
E.1 VISUALIZATION OF TOMATO SALAD
(a) The initial state of the environ- ment, the agent is in the kitchen Tomato.
(b) The agent
exccutes
Get-
(c) After picking up the tomato, the
agent executes Go-Cutling-Boudd Board-- 1.
without grabbing anything.
(d) After approaching the cutting board, the agent executes the action Chop to chop the tomato.
After chopping the tomato, the agent executes the action Get- Tomato to pick up the chopped tomato.
(f) After picking up the chopped tomato, the agent executes the ac- tion Gel-Bowl to take a bowl and place the chopped tomato inside it.
After placing the chopped tomato inside the bowl, the agent executes action Deliver to com- plete the task.
25

---

Published as a conference paper at ICLR 2024
E.2 VISUALIZATION OF TOMATO LETTUCE SALAD
(a) The initial state of the en- vironment, the agent is in the kitchen without grabbing any- thing.
(b) The agent executes Get- (c) After picking up the tomato, the agent executes Go-Cutting- Board--.
Tomato.
(d) After approaching the cut- ting board 1, the agent executes the action Chop to chop the
(e) After chopping the tomato, the agent executes Get- Lettuce.
(t) After picking up the let- tuce. the agent executes Go- Cutting-Board-2.
tomato.
(g) After approaching the cut- ting board the agent executes the action Chop to chop the let-
(i) After taking bowl, the (h) After chopping the lettuee, the agent executes the action Get-Bowl to take bowl.
agent exccutes the action Get- Tomato to place the chopped tomato inside the bowl.
tuce.
After picking up the (k) After placing the chopped
chopped tomato, the agent lettuce inside the bowl, the executes Get-Lettuce place agent executes the action De- the chopped lettuce inside the bowl.
liver to complete the task.
26

---

Published as a conference paper at ICLR 2024
E.3 VISUALIZATION OF Food Preparation
(a) The initial state the environment, the agent is in the kitchen without grabbing the pancake.
(b) The agent exccutes Walk-Pancakk.
(c) After approaching the pancake, the agent executes Grab-Pancake.
(d) After grabbing the pancake, the agent executes Walk-Microwave.
(f) After opening the microwave, the agent executes (e) After approaching the microwave, the agent Putin-Pancake-Micrron cutes Open-Microwave.
(g) After placing pancake into the microwave, the agent executes
27

---

Published as a conference paper at ICLR 2024
E.4 VISUALIZATION OF THE Entertainment
(a) The initial state of the environment, the agent is in the kitchen without grabbing anything.
(b) The agent executes Walk-Chips.
After approaching the chips, the agent executes Grab-Chips.
(d) After grabbing the chips, the agent executes Walk- Milk.
(e) After approaching the milk, the agent exccutes Grab-Milk.
(f) After grahhing the milk, the agent executes Walk- Livingroom.
28

---

Published as a conference paper at ICLR 2024
(g) The agent leaves the kitchen and enters the living room, then executes Walk-Coffeetable.
(h) After approaching the coffee table, the agent exe- cutes Putback- Cofps-Coffeetable.
(i) With the chips on the coffee table, the agent exe- cutes Walk-7V.
( After approaching the television, the agent exe- cutes Switchon-TV.
(k) After turning the television, the agent executes Walk-Sofa.
(I) After approaching the sofa. the agent executes the action Sit-Safa to complete the task.
29

---

Published as a conference paper at ICLR 2024
F PROMPT DESIGN
In this section, we provide a detailed analysis of the design of the observation prompt and action prompt and their impacts on the probabilities of each action. All the prompts are converted from raw observations and actions by scripts. For each observation shown in the visualization, we first present the observation prompt of the corresponding observation and then show the action prompts, tokens that make up the action prompts, the probability of each token, and the policy of TWOSOME without normalization, TWOSOME with token normalization and TWOSOME with word normalization, sequentially.
F.1 TOMATO SALAD
(a) The initial state of the environ- ment, the agent in the kitchen without grabbing anything.
Observation prompt in step (a): There is a fixed cutting board in the room. You notice a tomato on the table. Currently you don't 't have anything in hand. To serve the dish of bowl only containing chopped tomato, you should first
Table 22: Action Prompt in Step (a)
Action Probability of TWOSOME
Action
Token Probubility
W/O Norm
Token Norm
Word Norm
up the tomata
[pick, the,
[13.73, 77.67, 72.82, 28.54, 98.87]
55.24
48.34
49.61
take the bowl
[ake, the, bow. II
[14.81, 48 24.65, 99.72]
44.66
37.87
33.61
walk the cutting board
[walk, to, the, cutting. board]
[0.06, 42.28, 89,96. 15.79, 96.56]
0.09
13.33
16.56
serve oothing
[serve, nothing]
[4.65, 0.0]
0.16
0.19
chop nothing
[ch, nothing]
[0.53, 98.64, 0.0)
0.00
0.31
0.02
To ensure a high probability of items that are useful for completing the task like tomato and bowl, need to mention them in the observation prompt. However, due to the partial observ- ability, the agent cannot see bowl in the current position. So we need to mention the bowl in the task by mentioning To serve the dish of bowl, which can increase the probability of serve, too. Because currently there is nothing for the agent to serve and chop, so we can use nothing to lower the probability of these two actions. These two actions actually action prompt list, which is what we do in the VirtualHome environment. To show the performance of our method, which can solve typical decision decision-making tasks, we do not mask these actions here, for fair comparison with PPO.
Analysis:
necessary to appear in the
30

---

Published as a conference paper at ICLR 2024
(b) The
agent
executes
Get-
(c)
After picking up the tomato, the agent executes Cutttnnn--oord
Tomato.
1.
Observation prompt step (b): There is fixed cutting board in the room. Currently you are carrying an unchopped tomato in hand. To serve the dish of a bowl only containing chopped tomato, you should first
Table 23: Action Prompt in Step (b)
Action
Tikes
Token Probubiiity
WKO Norm
Action Prabaniliy of TWOSOME
TIoten Nor
Wend Norn
pick the romorn ukke dhe boxl
[pick, the, mm, 201]
(0.31, 72.1, 64 MA, 20 34 MT 16)
3:28
17.67
73
[take, the, bow, I]
(3:46, 37, 5.28, 99.34]
7.73
14.36
1006]
put tomato cutting hamr the dids
(put, the, tom, ens, on, cutting brard] [11.44, (scrvs ieh)
06.81]
06.50
(2.98, 0.01
97.9)
chop
Ich, nothing
Observation prompt step (c): There is fixed cutting board in the room. unchopped tomato is on the cutting board. Currently you are standing in front of the cutting board without anything in hand. To serve the dish bowl only containing chopped tomato, you should first
Table 24: Action Prompt Step (c)
Action Probability of TWOSOME
Action
Token
Token Probubility
Norm
Token Norm
Word
pick up the lomatia
piikk, the, lom, ato]
[11.26, 78.92, 8T.57, 30.9, 98.76
38.12
$2.52
36.21
take bowl
Itake,
II
52.9, 19.33, 99.75
walk to the cutting board
[walk, to, the, cutting, board]
[0.08, 37.29, 89.38, 37.08, 98.33]
0.18
11.11
14,97
nothing
[serve, nothing]
[3.23, 0.01]
0.00
0.10
0.14
chop the tomato
[ch. op, the, tom, ato]
[4.55, 99.57, 57.2, 88.8, 95.28)
37.91
32.45
26.27
Analysis: In Step (b), because the agent has already picked up the tomato, the third action walk to the cutting board can be replaced to put the tomato on the cutting board, which is more contextually appropriate and thus has high probability. And the agent now can serve the tomato, so the fourth action is replaced with serve the dish, though it is not a correct action. LLMs need to learn which dish is the correct dish, namely the alignment. It worth noting that for TWOSOME without normalization, serve the dish also has high probability, if the agent happens to sample this action several times, it may learn avoid entering this state and refuse to pick up the tomato, resulting in the divergence.
In Step (c), the tomato is put on the cutting board, so the final action is replaced with chop the tomato.
31

---

Published as a conference paper at ICLR 2024
(d) After approaching the cutting board, the agent executes the action Chop chop the tomato.
(e) After chopping the tomato, the agent executes the action Get- Tomato pick up the chopped tomato.
Observation prompt step (d): There is a fixed cutting board in the room. chopped tomato is on the cutting board. Currently you standing in front of the cutting board without anything in hand. To serve the dish of a bowl only containing chopped tomato, you should first
Table 25: Action Prompt in Step (d)
Action Probahility of TWOSOME
Action
Token Probubility
W/O Norm
Token Norn
Word Nomn
the
[pick, up,
ato]
[16.68, 73.75, #2.4, 19.69, 98.68
58.27
33.49
37.31
lake the bowl
[take, the, bow,
[15.05, $4.08, 34.69, 99.76]
54.75
30.09
30.31
walk cutting hoard
[walk, the, cutting, hrand]
[0.12, 36,22, 89.51, 39.64, 98.38]
0.30
12.69
17.21
serve mothing
[serve, nothing]
[3.28, 0.01]
0.00
0.11
0.15
chop the tomato
[ch, op, tom, ato] [0.76, 98.79, 54.96, 90.12, 92.33]
6.68
23.62
15,03
Observation prompt step (e): There is a fixed cutting board in the room. Currently you are standing in front of the cutting board, carrying a chopped tomato in hand. To serve the dish of bowl only containing chopped tomato, you should first
Table 26: Action Prompt in Step (e)
Action Prohability TWOSOME
Action
Token Probability
WOO Norm
Token Norm
Word Norm
pick up
[pick. up, tom,
[0.94,
98.13]
20.00
31.87
take the bowl
[take, the, bow. I]
[4.99, 42.45, 13.73, 99,54]
55.44
28.62
25
walk cutting board
the, cutting, board]
[0.19,
95.86]
serve the dish
[serve, the, d. ish)
[1.25, 63.08, 15.9, 98.87]
23.72
23.15
19.04
chop aothing
op, nothing!
32

---

Published as a conference paper at ICLR 2024
(f) After picking up the chopped tomato, the agent executes the ac- tion Ger-Bowl to take bowl and place chopped tomato inside it.
Observation prompt step (f): There is a fixed cutting board the room. Currently you are carrying bowl containing chopped tomato. To serve the dish of bowl only containing chopped tomato, you should first
Table 27: Action Prompt in Step (f)
Autiion
Token
Token Probubillty
WXO Nomn
Actann Probabilit of TWCSSIEE
Trkon Nemn
Wond Nomn
put the tomoio in the bool take the bowl
(ps), the, tom, ain, in, the bow.
[1187 S1.77, 99.66 21.92 7032 61.58 90.5
1.16
24.12
23.41
[talee the. bow,
[5,83,32.13, 51.03, 99,67]
58.43
23.36
21.52
put the bowl the cutting board (puz, the, bow, 1. on, the, coming board]
(1..8
58 39,39, 19,83, 2888, 38.37, 59.99, 97.23
19.37
34,19
00,82
serve chop notling
(serve, the, ish]
[371, 69.00, 21.5 97.50)
21:44
17.99
15.19
(ch, op. nothing!
[0.17, 0.0)
Analysis: Even for the final step, serve the dish still has a not high probability, which reflects the ability of LLaMA 7B. LLMs need to learn what is the correct dish to serve and make alignment with environment.
33

---

Published as a conference paper at ICLR 2024
F.2 TOMATO LETTUCE SALAD
(a) The initial state of the environ- ment, the agent is in the kitchen Tomato. without grabbing anything.
(b) The agent
executes
Get-
Observation prompt step (a): There are two fixed cutting boards in the room. You notice a tomato, lettuce and an onion the different tables. Currently you don't have anything in hand. To serve the dish of a bowl only containing chopped tomato and lettuce, you should first
Table 28: Action Prompt in Step (a)
Action
Token
Token Probubility
W/O Norm
Action Preéubility of TWOSOME
Token Norm
Word Norm
pick up tomato
[pick, up, lom,
61.3566.35, 62.92, 95.21]
pick up the letince
[pick, up, lett, uce]
17.09, 35, 62.92, 99.72]
14.88
pick up the calon
[pick, up, the on, kon]
[7.00, 61. 35, 62.92, 4.68. 92.45]
10.58
20.11
18660
take the empty bowl
[take, the, empty, bow. I]
[15.35, 50.98, 0.14, 75.08, 98.35)
0.70
11.69
9.14
walk to the lind culling bound
[walk, to, the, linsl, cutting. bound]
[0.08, 40.31, 84, 2.55, 39.84, 94,17]
0.02
9.11
11.31
walk the second cutting board
[walk, second, cutting. bound]
[0.08,
84.18. 0.66, 41.69, 94,06]
7.32
serve nothing
nothingl (ch, op, nothing)
0.01]
chop noching
[0.96, 0.0)
Observation prompt step (b): There are two fixed cutting boards in the room. You notice a lettuce and onion on the different tables. Currently you are carrying unchopped tomato in hand. To serve the dish of a bowl only containing chopped tomato and lettuce, you should first
Table 29: Action Prompt in Step (b)
Porbablity
Action iiiliilyy TWWRIIME
pietig pick op the lennna
20] ppice
D7 aKW, REER, TEX 98
16.75
15764
TXXI
[pick, the vit. use]
11.01, n6 76, AR 10 99
31 12
17:47
17.01
pick up vetion voratiin reapes
(pick: up,
[1,07 7A
13:31
18:
engry
hoand
15.3
1X 0411 251
17.13
put dhe tosnto os the sccond cutina board sarve lc
[pel, daz. bxss alss on, lic spcond, estingg board)
(8.52. 62,29, 15., 00 76, 85 SL 0.25, $7.29, (6.58)
0.07
15334
18.00
(erve the ss]
odiing)
high probability of items that useful for completing the task like tomato, lettuce and bowl, we need to mention them in the observation prompt. However, due to the partial observability, the agent cannot see bowl in the current position. So need mention the bowl in the task by mentioning To serve the dish of a bowl, which can increase the probability of serve, too. Compared with onion, tomato and lettuce appear twice in the observation prompt, thus having higher probabilities. We add the status empty to the bowl to lower the probability of holding the plate at the beginning, where the agent can do nothing but serve the plate the delivery counter and receive a negative reward.
Analysis:
In Step (a), to
In Step (b), because the agent has already picked up the tomato, the third action walk to the first cutting board can be replaced to put the tomato on the first cutting board, which is more contex- tually appropriate and thus has high probability. It is worth noting that TWOSOME without the normalization method has an extremely low probability of long action prompts, such as walk to the first cutting board.. and put the tomato on the first cutting board, which is consistent with the disad mentioned in the main paper. Thus, this method has poor performance in this task.
34

---

Published as a conference paper at ICLR 2024
(c) After picking up the tomato, the (d) After approaching the cutting agent executes Cio-Cuting-Boarr 1.
board I, the agent executes the ac- tion Chop to chop the tomato.
Observation prompt step (c): There are two fixed cutting boards in the room. An unchopped tomato is on the first cutting board. Currently you are standing in front of the first cutting board without anything in hand. To serve the dish of a bowl only containing chopped tomato and lettuce, you should first
Table 30: Action Prompt in Step (c)
Probability
TWOSOME
Token
Word
pick up the somato pick up the lettuce pick the onioe take crupty
ppick. up, the, tom, ato
[5.33, 71.33, 71.0 15.17, 97.87|
25.09
19:24
19.46
ppick, the, lett, uce]
[5.33, 77.33, 71.0. 10.77, 99.9]
18.61
15.04
17.95
[pick, the on, los)
[5.33, 77.33, 71.. 0.1,, 56.95)
0.01
4.05
2.78
the, empry,
[13.95, 48.59,
99.64)
cutting
cutting. heard]
(0.9,4
9.1,92088,
sconnd cutting boand
[walk; to, the; socand, cutting, board]
[0.3,
93.27, 92.63, 97.89)
serve nothing
[serve, nothingl
[4.7, 0.01]
0.19
chop the comatio
(chs op, the, tom, sto]
[2:48, 99.61, 63.58, 61.1, 90.68]
$1.26
22.09
15:57
Observation prompt step (d): There are two fixed cutting boards in the room. A chopped tomato on the first cutting board. Currently you are standing in front of the first cutting board without anything in hand. To serve the dish of a bowl only containing chopped tomato and lettuce, you should first
Table 31: Action Prompt in Step (d)
Action
Token
Tokeen Probabiliiy
Action Prebuciliy TWOSOME Tiken
Wont Namn
pick up tomato
pick, up, the; ato)
[759, 75.02, 9,45, 66.66]
pick up the letuce
[pick, sp, (be, see]
[7.99, 75.07 47, 13.72,
50.44
21.34
pick up the onion
[pick, up. the, on, ion)
[7.99, 75.02, 71.47, 0.01, 51.76]
0.02
4.45
3:07
take the emply bowl
(take, the, empty, bos, I]
[15.99, 51 0.13. 77.09. 99.6)
0.68
9.03
7.43
walk the finst cailing board
[walk, the, fins, cutting, board]
(0.56. 41,54, 92.93, 9.12, 92.14 97.87)
0.97
13.10
17.26
walk the second curting board
[walk, second. cutting, board) [serve, nothing]
[0.36. 41.54,
44.85
4.34
17.06
serve
chop tomaio
[clt op. the, tomt ato]
[0.47, 99.09993,, 66 88.77)
Analysis: In Step (c) and (d), TWOSOME without normalization shows high probabilities the optimal actions for both steps, namely chop the tomato and pick up the lettuce, due to the fewer tokens these actions have. So if all the optimal actions happen have very few tokens, this method can achieve very good performance.
35

---

Published as a conference paper at ICLR 2024
(c) After chopping the tomato, the (f) After picking up the lettuce, the agent executes Get-Lettuce.
agent executes Go-Cutting-Boue Boaiid 2.
Observation prompt step (e): There are two fixed cutting boards in the room. You notice an onion on the table. Currently you are carrying unchopped lettuce in hand. To serve the dish of bowl only containing chopped tomato and lettuce, you should first
Table 32: Action Prompt Step (e)
Protabillty
ALDI ESDERSIRD LWLUSMEE
Bicks brabh,
11.43
TEL30
pick up the letuat
[rkkk up, the lei. sce]
BA ot 47, 19,6,, 99.33
8.50
15.30
14.53
up usica
(pick, the, ims)
(0.3,
98.17]
12:56
pat the inmio on the firs parting hoard
[jen the, Iri: lon, the fire caiting beaet! (ger the vel, nos soord,, casting board] outinn!
91 eu 47 9944 16,M6 85 78, 6 W9
0.66
16.70
10 66
pet lamase seced canting beard
(6.91,
chop nething
(10 98.71,
0.51
Observation prompt step (f): There are two fixed cutting boards in the room. A chopped tomato on the first cutting board. An unchopped lettuce on the second cutting board. Currently you are standing in front of the second cutting board without anything in hand. To serve the dish of bowl only containing chopped tomato and lettuce, you should first
Table 33: Action Prompt Step (f)
Action
Token
Token Probubiliyy
W/O Norm
Action Probability of TTOSOME
Token Nerm
Word Norm
[pick, up, the,
[4.9% 36, 67.6, 7,96, 23]
3.06
16.67
16.20
up lemuee
lpick,
sce]
17.27
19.41
the onion
ion] [take, empry,
[pick, op.
30.74)
take empty
I]
walk to the first cuiting bourd
[walk, to, the, fins, cutling. board]
[0.18, 49.84, 94.62, 61.28, 93.27, 98.43]
1,93
16.15
21,43
walk the second cutting bourd
[walk, to, the, sccond, catting, bourd]
[0.18. 49.84, 94.62, 4.99, 93.69, 98.46)
0.16
10.64
14.12
serve nothing
[serve, nothingl
[4.77, 0.011
0.02
0.14
0.18
chop lettuce
Jch, the,
[3.39, 98.94, 69.9. 76.36, 99.83)
72.38
25.85
20.05
36

---

Published as a conference paper at ICLR 2024
(g) After approaching the cutting board 2, the agent executes the ac- tion Chop chop the lettuce.
(h) After chopping the lettuce, the agent executes the action Gel-Bowl take bowl.
Observation prompt step (g): There are two fixed cutting boards the room. chopped tomato is on the first cutting board. A chopped lettuce is on the second cutting board. Currently you are standing front of the second cutting board without anything in hand. To serve the dish bowl only containing chopped tomato and lettuce, you should first
Table 34: Action Prompt in Step (g)
Probability
Nomm
Preesbility TWOSOME
Token
Word
pick up the somato
Jpick. up, the, tom, abo]
[6.34, 71.54, 68.8 10.22 97.71]
31.09
19:38
19.35
pick up the lettuce
jpick, the, lett, uce]
77.54, 68.8 12.52. 99.88)
39.69
20.27
20.50
pick the onion
[pick, the on, los)
[6.34, 77.54, 68.8, 0.01, 31,62)
0.01
3.74
2.48
take crppty
the, empry,
[18.077
0.54
7.01
cutting
cutting. heard]
sccrnd cutting board
[walk; to, the; socand, cutting. bowrd]
[0.27,49.
94.7, 93.91, 98.55]
serve nothing
[serve, nothingl
[3.9. 0.01]
0.03
0.11
chop the leftuce
[ch, op, the, leil, uce]
[0.56, 18,19, 61.88, 66.44, 44.44,99.88 99.82)
21.12
17:87
10.54
Observation prompt step (h): There are two fixed cutting boards in the room. Currently you are carrying a bowl in hand. To serve the dish of bowl only containing chopped tomato and lettuce, you should first
Table 35: Action Prompt in Step (h)
Achice
lobkes
tokes Utmeaaiiiny
WrO Nassi
Actan Profehliy of TROSSOME
Toles Norto
Weed Nena
JM. Intrator to tool the usice in the seel tike the enple bewl
pL lott, DE:
[30.22 5665, 3.30 23.52, MCS5
00.31)
17.80
us, in bos.
[20S2, S6A3, 3413 65 I8, 10.4 73,34 34.36, 99
0.22
3.15
0.50
like, die crppy, low. I]
JP.,, 30
76.57,
1.39
5.03
0.05
cotling
[ovt,
cuttin,, bood)
120,72
$6665, 59.31, $3.51,
17.10
dis chip selling
(iene, fic. Hi))
[2.56, 42. 21,04, 91
99.07
13.25
11.43
[ch, og. acthing
[0.12 95.65 01]
0.00
0.10
Analysis: In Step (g), because the word empry has an extremely low probability, TWOSOME with- out normalization agent struggles with getting the bowl.
In Step (h), because other action prompts happen to be relatively long, though the probability of serve is still low, the probability of serve the dish is extremely high compared with other actions for TWOSOME without the normalization method. This step also explains why this method cannot have good performance in this task.
37

---

Published as a conference paper at ICLR 2024
(i) After taking a bowl, the agent executes the action Get-Tome to place the chopped tomato inside the bowl.
(j) After picking up the chopped tomato, the agent executes Get- Lettuce to place the chopped let- tuce inside bowl.
Observation prompt step (i): There are two fixed cutting boards in the room. A chopped lettuce is on the second cutting board. Currently you standing in front of the first cutting board, carrying a bowl containing chopped tomato in hand. To serve the dish of a bowl only containing chopped tomato and lettuce, you should first
Table 36: Action Prompt in Step (i)
Porbesiitt
WA0 Somn
Token Numm
Wend Noms
pu Tee IEEao Be SCRE
Jea, lhe, litls als ioc S0n, If
TRARS Suai 4a0, DE 2SA0 Siide Bo 10
1-27
14.7
13.30
put letuae the
(oxt dee, lets, is
20.55,
take the bord
(takc bon.
[8.5. Sa 30 99.26)
60.81
14.43
13,36
pat the beod us lins catling loard
loes, lie. bow, iis das link, cutling, bound) (pst,
[10.6G 69,42, 423,, 1U15, 36 90.3. 150, 08
3.00
19.39
12:15
pa acceed cating
dis sconad, callnng boasd]
[BOLSS,
(8.65)
willing
[ch, 1p..0.... solinul
[6,40. 98.79, 0.30
0.00
0.12
Observation prompt step (j): There are two fixed cutting boards in the room. Currently you are standing in front of the second cutting board, carrying bowl containing chopped tomato and lettuce in hand. To serve the dish of bowl only containing chopped tomato and lettuce, you should first
Table 37: Action Prompt in Step (i)
Artian
Thiccs
Tikonn Pruhanilty
Wot.Nom 1nken
Actie TritaFitty IT TWISIMEE
Weed Nom
lemas hexd pal the unaes the butal the
the,
el S., 3.2.
11.4, 32.21
11
14.26
13,35
(pa), the es, ios, is, the, bow (taks, bow;
112.99 6L.S, OUL 602A,, YEL, (1,34 $4.90 v6-41
1110)
3,65
3.77
EXE),
99244
13:25
sccoed cating board
sccand, cattngg board]
[82.0K 61.52, 99.0, 99.1,, 1963, 90.0 $7.65 (8.27)
3,57
20.13
serve fish
(serve, Bh. 1a0)
1..9, 70 33.37. $6.12)
37.80
12.77
11.18
withing
[ch, op. nitiing
Analysis: In both Step (i) and Step (j), short actions prompts, take the bowl and serve the dish still dominates the probabilities, making it difficult for TWOSOME without normalization to sample optimal action and learn good policy.
38

---

Published as a conference paper at ICLR 2024
F.3 FOOD PREPARATION
(a) The initial state of the environment, the agent is in the kitchen without grabbing the pancake.
(b) The agent executes Walk-Pancake.
Observation prompt step (a): There are four rooms: the kitchen, bathroom, bedroom, and living room. You are in the kitchen. You notice pancake and microwave. Currently, you are not grab- bing anything hand. The pancake and the microwave are not within your immediate reach. The microwave is not opened. In order to heat up the pancake in the microwave, your next step is to
Table 38: Action Prompt in Step (a)
Frobabiitt of TWOSOME
Action
Token
Token Probubility
W/ Nerm
Token Norm
Word Norm
walk living room
to, the, living. room
4L87, 87.8, 97.74]
2.06
walk to the hathmom
[walks, to, the, bath, mom]
[0.06, 41,87, 87.8, 1.19, 99.3]
0.54
10.61
9.84
walk the bedroom
[walk, the, bed, rom]
[0.06, 87. 87.8, 0.59, 98.431
0.27
9.20
8.24
reach for the pancake
Ireach. for, the, pan. ca, ke] [move, to, the, mic, row, ave]
11.31,24.77 76.49, 14.72, 99.0. 100.0]
80.76
37.56
34.36
move to the microwave
[0.19, 8.96, 85,27, 50.8, 99.85, 99.04]
16.37
28.79
23.05
Observation prompt step (b): There four rooms: the kitchen, bathroom, bedroom, and living room. You are in the kitchen. You notice pancake and microwave. Currently, you are not grabbing anything in hand. The pancake within your immediate reach. The microwave is not opend. In order to heat up the pancake in the microwave, your next step is to
Table 39: Action Prompt in Step (b)
Action Probubility of TWOSOME:
Token
Probahility
W/O Narm
Token Norm
Word Norm
walk the living walk the buthroom
[walk, the, living,
[0005, 37.46, 86,09, 3.58, 98.02
0.12
12.14
21.74
wwalk, to, the, both, room]
10.05, 46, 86,09, 0.93, 99.31
0.02
8.51
7,80
walk to the bedroom
[walk, to, the, hed, mom]
[0.05, 37 46, 86.09, 0.42, 98.47]
0.01
7.26
6.39
microwave
[move, to,
row, ave]
[0.17, 84.8,
99.12]
25.63
grab the pancake
Igrab, the, ca, kel
12.08. 74.95. 47.42, 99.49, 99.991
98.81
46.46
43.22
Analysis: In the Food Preparation task, to ensure high probability of the items necessary for task completion, we increase the loglikelihood of their occurrence by mentioning task-related objects observed in the current room. For example, in the kitchen, we use You notice a pancake and microwave to raise the chances of the pancake and microwave appearing. To distinguish the prob- ability of each action, our design utilizes different phrases. For approaching a room, we use walk to. For approaching food, we use reach for. And for approaching furniture, we use move to. This differentiation helps convey the varying probabilities associated with each action.
In Step (a), because the agent is currently not next to the pancake, there is no action available in the options for grabbing objects. To increase the probability of approaching the pancake, we emphasize in the observation prompt that it is not within your immediate reach and we also use the word reach the available actions to raise the probability.
In Step (b), as the agent is currently close to the pancake, the action grab the pancake is included, while the action reach for the pancake removed from the options.
39

---

Published as a conference paper at ICLR 2024
(c) After approaching the pancake, the agent executes Grab-Pancake.
(d) After grabbing the pancake, the agent executes Walk-Microwave.
Observation prompt step (c): There four rooms: the kitchen, bathroom, bedroom, and living room. You are in the kitchen. You notice pancake and microwave. Currently, you have grabbed the pancake in hand. The microwave is not within your immediate reach. The microwave is not opend. In order to heat up the pancake in the microwave, your next step is to
Table 40: Action Prompt in Step (c)
Action Frobubility of TWOSOME
Action
Probability
Norm
Token Norm
Word Norm
walk the living room
[walk, the, living, room]
[0.06, 42.72, 89,77, 98.03]
5.99
15,80
28.45
bathroom
[walk, to,
room]
[0.06,
99.35]
walk to the bedroom
[walk, to, the, bed, room]
[0.06, 42 89.77, 0.48, 98.72]
0.51
9.66
8.74
reach for pancake
freach, the pan, [move, to, the, mic. row, ave]
[1.31,
2.52,
26.78
29.56
23.51
move to the microwase
[0.19, 14.16, 87,89, 59.33, 99.87, 99.29]
65.89
34.35
29.45
Observation prompt step (d): There are four rooms: the kitchen, bathroom, bedroom, and living room. You are in the kitchen. You notice pancake and microwave. Currently, you have grabbed the pancake in hand. The microwave is within your immediate reach. The microwave is not opend. In order heat up the pancake in the microwave, your next step
Table 41: Action Prompt in Step (d)
Tokeo
Probubiiity
Pubbablay
2017. V. TTT, 9610)
[TC0
41:11
walk de kalsoon
fwal, beth, roun)
(0.00 26.56 89.09. 0.16 95.3
0.00
2.14
1.31
walk to bedloces
[valk oo, fic. lel, moss)
1003, D8. $0.09,
15.15]
[.00
1.34
1.14
reach
(sosch, lic, pess
24.71,
micmoove
(epon.
[99.1), 82 96.19 98-44 (9,23)
96.20
42.55
$6.14
dhee menossee
cirms the, mic mos we
[0 41.4% 9133), Wilk 96.1)
1.06
17.19
11.06
Analysis: In Step (c) and Step (d), we control the probability of the agent approaching the microwave and opening it by describing the current state of whether the agent grabbing the pancake not.
40

---

Published as a conference paper at ICLR 2024
(e) After approaching the microwave, the agent exe- (f) After opening the microwave, the agent executes cutes Open-Microwave.
Mik-in-Paacake-Minne
Observation prompt step (e): There four rooms: the kitchen, bathroom, bedroom, and living room. You are in the kitchen. You notice pancake and microwave. Currently, you have grabbed the pancake in hand. The microwave is within your immediate reach. The microwave is opened. In order to heat up the pancake in the microwave, your next step is to
Table 42: Action Prompt in Step (c)
Astin
Token
Toliza Probuisiltty
ALTIE Frstallyy TWOOROOEE
Norm
Toiess Nern
Word Nocn
in the living nemm
(walk: roml
wilk hedhwors
Joolk (toohh for thr, pon on ko]
16.14, 53.0 90.78
3.05
1:37
mach Sorthe pareake
[D
11,01,
99, 10 49 10, 1000)
11
10:71
7.51
pur the pancrite micmooooo opes microwove
(pur
gas, mic nox, asa] (oper, exel
(7 83
74, 1901
9%.9 70]
79:75
(9.60)
Observation prompt step (f): There are four rooms: the kitchen, bathroom, bedroom, and living room. You are in the kitchen. You notice pancake and microwave. The microwave is opened. In order to heat up the pancake in the microwave, your next step is to
Table 43: Action Prompt in Step (f)
Action Probubility of TOSOMEE
Action
Token
Token Probubility
W/O Norm
Token Norm
Word Norm
walk to the living room
[walk, to, the, living, room] [walk, the, bath, room]
[0.05, 35.97, 86.86, 7.82, 98.96]
0.01
7.44
10.70
walk bathroom
[0.05, 35.97, 86, 2.31, 99.56]
0.00
5.84
4,49
walk to the bedroom
wwalk, to. the, bed, room]
10.05, 35.97, 86.86, 1.3, 99.071
0.00
5.20
3,89
open the microwave close the microwave
Jopen, the, row, lelose, the, mic, zow, ave]
[14.45, 81.0, 99.9,
60.07
42.43
43.21
16.47, 85.02, 91,19, 99.88, 99.45]
39.91
39.10
37.71
In Step (e), despite the microwave being already open, the action open the microwave still has the highest probability, while the optimal action put the pancake in the microwave just slightly lower. During the training process, the agent will gradually increase the probability of sampling the optimal action. It is worth noting that for TWOSOME without normalization, open the microwave remains excessively high, if the agent happens sample this action several times, it may learn to avoid entering this state and refuse to put the pancake in the microwave, resulting in the divergence.
Analysis:
In Step (f), open the microwave still has the highest probability, while the optimal action close the microwave is the second highest, which reflects the ability of LLaMA 7B. LLMs need to learn what the correct open and close and make alignment with the environment.
41

---

Published as a conference paper at ICLR 2024
F.4 ENTERTAINMENT
(a) The initial state of the environment, the agent is in the kitchen without grabbing anything.
Observation prompt step (a): There are four rooms: the kitchen, bathroom, bedroom, and living room. You are in the kitchen and notice chips and milk. But they are not within your immediate reach. Currently, you are not grabbing anything hand. order to enjoy the chips and the milk while watching TV, your next step to
Table 44: Action Prompt in Step (a)
Action Probability of TWOSOME
Action
Token
Token Probability
W/O Norm
Token Norm
Word Norm
walk to the living room
[walk, to, the, living, room]
[3.79, 48. 38, 81.0, 11.0, 98.36]
4.13
17.78
20.89
walk to the bathroom
[walk, the, bath, room]
[3.79, 48.38, 81.0, 20.1, 99.5]
7.62
20.11
17.66
walk to the hedroom
[walk, to, the, hed, mom]
[3.79. 48 81.0, 5.98, 98.92]
2,26
15.76
13.02
reach for the chips
freach. freach. for, the, milk]
ch. ips]
[16.71,
48.0, 72.95, 99.96]
79.94
32.17
31,77
reach for the milk
[16.7, 48.12, $2.9, 5.54
6.05
14.19
16.66
Ablation Observation prompt step (a): There are four rooms: the kitchen, bathroom, bedroom, and living room. You are in the kitchen and notice chips and milk. But they are not close to you. Currently, you grabbing anything in hand. In order enjoy the chips and the milk while watching TV, your next step is to
Table 45: Ablation Action Prompt in Step (a)
Action Probability of TWOSOME
Action
Token
Probubility
W/O Norm
Token Norm
Word Norm
living room
[walk, to. tbe, living. room] [walk, to, the, bath, room] walk, the, room] freach, for, the, ch, ips] (reach,
13.87, 44.14,
11.26
20.19
24.45
walk to the bathroom
[3.87, 44.14 80.22, 14.11, 99.4]
16.21
21.71
19.24
the bedroom
13,87,
17.97
reach for the chips
[6.48, 23.32, 64.79, 74.24, 99.94]
61.26
28.32
26.82
reach milk
milk]
[6.49, 23.32, 64.82, 5.99)
11.81
Analysis: In the task of Entertainment, to ensure high probability of the items necessary for task completion, we increase the loglikelihood of their occurrence by mentioning task-related objects observed in the current room. For example, in the kitchen, we use notice chips and milk to raise the chances of the chips and milk appearing, TV and a sofa to raise the chances of the coffee table, TV and sofa appearing. To distinguish the probability of each action, our design utilizes different phrases. For approaching a room, we use walk to. For approaching food, we use reach for. And for approaching furniture, use move to. This differentiation helps to convey the varying probabilities associated with each action.
the living room, we use notice coffee table,
In Step (a), we have found that the combination of within your immediate reach and reach for has higher probability of occurrence compared to the combination of close to and reach for.
42

---

Published as a conference paper at ICLR 2024
(b) The agent executes Walk-Chips.
Observation prompt step (b): There are four rooms: the kitchen, bathroom, bedroom, and living room. You are in the kitchen and notice chips and milk. The chips are within your immediate reach. But you have not grabbed the chips. Currently, you not grabbing anything in hand. In order to enjoy the chips and the milk while watching TV, your next step is to
Table 46: Action Prompt in Step (b)
Action
Token
Token Probability
Action Probability of WOSOME
Norm
Token Norm
Norm
walk the living
[walk, the, living, rom] [walk, to, the, bath, room] Iwalk, the, bed. [reach, for, the, milk] Igrab, the, ips]
[0.08, 50.83, 87.12, 29.02, 99.06]
0.50
16.55
21.19
walk to the bathroom walk the bedroom reach for the milk
[0.08, 50.83, 87 12, 20.9, 99.24]
0.36
15,50
12.37
17,12, 97.89]
0.16
13.14
[3.32, 29.07, 17, 7.37]
2.49
15.68
20.07
grab the chips
81,22,
39.13
Analysis: In Step (b), as the agent is currently close to the chips, the action grab the chips is included, while the action reach for the chips is removed from the options. However, despite the presence of the option reach for the milk, grab the chips still obtains the highest probability.
43

---

Published as a conference paper at ICLR 2024
(c) After approaching the chips, the agent executes (d) After grabbing the chips, the agent executes Walk- Grab-Chips
Milk.
Observation prompt step (c): There four rooms: the kitchen, bathroom, bedroom, and living room. You are in the kitchen and notice chips and milk. The milk is not within your immediate reach. Currently, you have grabbed the chips in hand. In order enjoy the chips and the milk while watching TV, your next step is to
Table 47: Action Prompt in Step (c)
Action Probability of TWOSOME
Action
Token
Probability
W/O Norm
Token Norm
Word Norm
living mom walk to the bathroom walk to the bedroom
[walk, to, living, room] [walk, the, bath, room]
[0.3, 44.83,
98.8]
15.45
[0.3, 44,83, 87.01. 17.79, 99 99]
6,90
23.47
18.24
[walk, to. the, bed. room]
[0.3, 44.83, 87.01. 10.79, 98.43)
4.14
21.20
16.05
reach for the milk
freach, for, the, milk]
[1.26. 29.88, 88.35, 66.42]
73.51
27.75
32,95
Observation prompt step (d): There are four rooms: the kitchen, bathroom, bedroom, and living room. You are in the kitchen and notice chips and milk. The milk is within your immediate reach. But you have not grabbed the milk. Currently, you have grabbed the chips in hand. order to enjoy the chips and the milk while watching TV, your next step is to
Table 48: Action Prompt in Step (d)
Action
Token
Token Probubility
Action Probability of TWOSOME
W/O Norm
Token Norm
Word Norm
walk to the living room
waik,, to, the, living, room] [walk, the, bath, room] [walk, to, the, bed, rom]
[4,93, 52.67, 93.82, 46.07, 99.14]
20.09
29.30
33.09
walk to the bathroom
[4.93, 52.67, 93.82, 19.81, 99.4]
8,66
24.76
21.41
walk to the bedroom
[4.93, 52.67, 82, 10.94, 98.6]
4,74
21.96
18.42
grab the milk
[grah, the, milk]
[5.67, 88.74, 73.16)
66.51
23.97
27.07
44

---

Published as a conference paper at ICLR 2024
(e) After approaching the milk, the agent executes (1) After grabbing the milk, the agent executes Walk- Grab-Milk.
Livingroom.
Observation prompt step (e): There four rooms: the kitchen, bathroom, bedroom, and living room. You are in the kitchen and notice chips and milk. Currently, you have grabbed the chips and the milk in hand. In order enjoy the chips and the milk while watching TVV, your step is to
Table 49: Action Prompt in Step (e)
Action Probability of TWOSOME
Action
Token
Token Probability
W/O Norm
Token Norm
Word Norm
walk to the living room
walk, to, the, living. room] [walk, the, bath, room]
12.18, 46.82, 92.81, 63.93, 98.53 12.18, 92.1, 99.4]
79.68
43:25
52.18
walk bathroom
30.34
walk to the hedroom
[wakk, to, the, hed, mom]
[2.18, 46.82, 92.11 5.59, 98.8]
6.77
26.41
21.84
Observation prompt step (f): There are four rooms: the kitchen, bathroom, bedroom, and living room. You are the living room and notice a coffee table, a TV and a sofa. They are not close to you. Currently, you have grabbed the chips and the milk in hand. To enjoy the chips and the milk while watching TV, your next step is to
Table 50: Action Prompt in Step (f)
Action Profability TWOSOME
Action
Token
Token Probability
W/O Norm
Token Norm
Word Norm
[walk,
[3.66, 90.77, 46.41|
walk to the bathroom walk the bedroom
[walk, to, the, bath, room]
[3.67. 47.91, 90.77, 0.7. 99.99 (9.49]
0.62
10.85
7.97
[walk, to, the, hed, rom]
(3.67,47.9)) 90.77, L.82, 98.62]
1.59
13.10
10.09
move to the coffee table
[move, to, the, coffee, table]
[11.49, 28. 89,56, 3,26, 98.59]
5.28
16.65
19.28
to the TV
TV] [move. to, the, sp, fa]
[move,
[11.44, 28.82, 89.53, 21.56]
35.32
18.91
21.89
move to the sofa
[1..9, 28.79, 89.56, 10.04, 99.96]
16.49
20.91
18.10
Analysis: In Step (e), since the agent now has grabbed the chips and milk, its only choice is to leave the kitchen and move to another room.
In Step (f), currently, the agent is in the living room. Since the agent is not close to the coffee table, TV, or sofa, it does not have any actions related to interacting with those furniture pieces. is worth mentioning that the action walk to the kitchen has the highest probability. However, the difference in probabilities between the action move to "furniture" " and walk to the kitchen is not significant.
45

---

Published as a conference paper at ICLR 2024
(g) The agent leaves the kitchen and enters the living room, then executes Walk-Coffeetable.
(h) After approaching the coffee table, the agent exe- cutes Srabbac-Chips-Cffffee Seiiss-ooffeetabl.
Observation prompt step (g): There four rooms: the kitchen, bathroom, bedroom, and living room. You are in the living room and notice a coffee table, TV and a sofa. The coffee table is close you. Currently, you have grabbed the chips and the milk in hand. In order enjoy the chips and the milk while watching TV, your next step is to
Table 51: Action Prompt in Step (g)
Action
Token
Token Probobiliry
WAD Norm
Actien Fradulilty TWOSOME
Tooken Nerm
Weed Norm
walk die Kishhen
[val,, the, Knclien)
[.R,, 4025, 92,74, 13,44133333
4.70
15.1
10:00
walk to the buthnoom
twalk, the. buth. room
[1.9, 40.77, 92.72 0.23, 99.611
0.07
5.59
3,65
walk to the bedroom
[ealk, to, the. bed, room]
9o, 40.77 91.12 0,35 97 82]
0.10
6.00
3.97
move to the TV
[muoee us die, TV]
(9.4), 23.75 90.77 10.13)
16.11
12.57
14:15
mure to the sofa put the chips on the cofize table
[muse, the, ss,
19.36 23.75, 90.54, 26.99, 95.98]
22.60
17.76
15:39
Ipue, the. ch, ips, on, the, codice, sable) (pul, the, milk, on, the codfee, tabiel
[20.56, 78.62, 84.28, 99.95, 11.51, 95.85, 31.33, 99.171
50.42
29.03
30.19
put the milk on the soffee tile
[20.56, 78A33 4.54, 28.58, 95.2, 72.08, 99.16]
6.00
19.80
22.25
Observation prompt step (h): There are four rooms: the kitchen, bathroom, bedroom, and living You notice coffee table, TV and sofa. The TV is not close to
room. You are in the living you. Currently, you have the chips on the coffee table and the milk in your hand. In order to enjoy the chips and the milk while watching TV, your next step is to
Table 52: Action Prompt in Step (h)
Action Prohahility of TWOSOME
Action
Token
Token Probubility
W/O Norm
Token Norm
Word Norm
walk to the kitchen walk to the bathroom walk the bedroom move to the TV
[walk, to, the, kitchen]
[1.68. 44.48, 92.17, 9.36] [1.68. 44,44, 92.2, 0.2, 99.68)
14.16
18.49
22.91
[walk, to, the, bath, room]
0.30
12.34
8.73
[walk, to, the, bed, rom]
[1.68, 44.44, 92.2, 0.22, 98.18)
0.33
12.57
8.93
[move, to. the TV] [move, the, so, fa]
[5.69. 12.39, 89.18, 42.931
59.25
26.45
32.77
move to
[5.73, 89.32,
99.97]
25.97
Analysis: In Step (g), since the agent is next to the coffee table, there now actions available to place chips and milk the coffee table.
In Step (h), we emphasize that the chips are already on the coffee table, reminding the agent to move other furniture nearby.
46

---

Published as a conference paper at ICLR 2024
(i) With the chips on the coffee table, the agent exe- (i) After approaching the television, the agent exe- cutes Walk-TV.
cutes Switchon-TV.
Observation prompt step (i): There are four rooms: the kitchen, bathroom, bedroom, and living room. You are in the living room. You notice a coffee table, TV and a sofa. The TV is close to you. Currently, you have the chips the coffee table and the milk in your hand. In order to enjoy the chips and the milk while watching TV, your next step is to
Table 53: Action Prompt in Step (i)
Action Prohability TWOSOME:
Action
Token
Token Probability
W/O Norm
Token Norm
Word Norm
kitehen
[walk, kitchen|
|0.59, 41.18,
walk to the bathroom
[walk, to, the, bath, romm
[0.59, 41.2, 93. 0,17, 99.72]
0.06
7.32
4.54
bedreom
[walk, [move, to. the, coffle, table]
room]
10.59, 41.2,
0.19,
move to the coffee lable move turn the TV
[3.92, 15.52, 91.54, 13.48, 99.22]
12.11
21.00
24.27
Imove,
[tur,, on, the, TV]
[0.53, 78.05, 94.08, 94.97]
60.27
21.88
25.29
[turn, off, the, TV]
[0.53,
64.39]
Observation prompt step (j): There are four rooms: the kitchen, bathroom, bedroom, and living
room. You are in the living room. You notice coffee table, TV and sofa. The sofa is not close to you. Currently, the TV is turned on, you have the chips on the coffee table and the milk in your hand. In order to enjoy the chips and the milk while watching TV, your next step is
Table 54: Action Prompt in Step (j)
Action
Token
Token Prohahility
W/O Norm
Action Prohubility TWOSOME
Teken Norm
Word Norm
[1,72,
91,41, 6.41]
11.85
walk to the bathroom
[walk, to, the, bath, room]
45.34, 91.44, 0.09, 99.68]
0.08
5.98
4.02
walk the bedroom
[walk, to, the, hed, room]
[1.71, 45.34, 91 44, 0.16, 97.46]
0..44
6.76
4.68
coffee table
[mnov, the, coffee, table]
[5.62. 16.76, 87.12. 8.05, 99.14]
8.34
15.24
18.64
move sofa
[move,
so, fa]
[5.62, 16.76.
99.98]
take scal on the sula
[take, scat, on, the, su. fa]
[7.79.
14.311
72.9%,
85,85, 90.65. 97
24.72
27.06
28.53
stand from the sofa
[stand, up, from,
71.11,
6.97. 56.23, $2.01. 99.971
0.63
12.64
11.11
Analysis: In Step (i), the agent is next to the TV, and it emphasized that the TV is turned on, increasing the probability of turning on the TV.
In Step (i), found that the probability of take seal the sofa is higher than the probability of move to the sofa, which reflects the ability of LLaMA 7B. However, we believe that TWOSOME will gradually learn the optimal action for this step, and experimental evidence supports this belief.
47

---

Published as a conference paper at ICLR 2024
(k) After turning on the television, the agent executes Walk-Sofa.
Observation prompt step (k): There four rooms: the kitchen, bathroom, bedroom, and living room. You are in the living room. You notice a coffee table, a TV and sofa. The sofa is close you. Currently, the TV is turned on, you have the chips the coffee table and the milk in your hand. In order to enjoy the chips and the milk while watching TV, your next step is to
Table 55: Action Prompt in Step (k)
Token Probability
Action Frobability TWOSOME
Token
walk to the kiichen
[walk, to, the, kitchen]
[0,6, 40001, 93.07, 6.71]
2.38
8.60
9.98
walk the bathroom
[walk, to, the, bath, room]
[0.6, 40.03, 93.07, 0.06, 99.691
0.02
5.27
3.12
[walk;
room]
[0.6,
96.73]
3.52
move to the coffee table the TV
[move, to, the, coflee, table]
[3.77, 13.77,21.08 21,04, 90.28, 26, 99.11]
7,09
16.62
19.27
[move, the, TV]
[3.74, 21,02, 90.39. 11.53]
13.08
13.16
15.27
[take,
99.97]
36.30
stand up from the sofa
[stand, up, from, the, so,
[0.16, 72.24, 6.41, 59.92, 85.86, 99.981
0.60
14.24
11.76
48


Reviews:
[{"review_id": "1oHS0eiDAb", "reviewer": "Reviewer_FDRX", "summary": "- TWOSOME is a general online framework that allows LLMs to be used as decision-making agents in embodied environments.\n\n- TWOSOME uses a two-stage approach to align LLMs with environments: first, it queries the joint probabilities of each valid action with LLMs to form behaviour policies; and second, it enhances the stability and robustness of the policies using normalisation methods and prompt design principles.\n\n- TWOSOME exhibits significantly better sample efficiency and performance compared to other methods in both classical decision-making environment, Overcooked, and simulated household environment, VirtualHome.\n\n- TWOSOME shows superior generalization ability to unseen tasks and does not result in a significant loss of the LLMs' original ability.", "questions": "-", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n\n- Proposes a novel framework, TWOSOME, for aligning LLMs with embodied environments.\n\n- TWOSOME is a general framework that can be applied to a wide range of decision-making tasks.\n\n- TWOSOME exhibits significantly better sample efficiency and performance compared to other methods.\n\n- TWOSOME shows superior generalization ability to unseen tasks.\n\n- TWOSOME does not result in a significant loss of the LLMs' original ability.\n\n\nOverall, the paper proposes a promising new framework for aligning LLMs with embodied environments. The framework is well-motivated and the experimental results are good.", "weaknesses": "Weaknesses:\n\n- The paper does not provide a theoretical analysis of the proposed framework.\n\n- The paper does not evaluate the performance of TWOSOME on a wider range of tasks and environments.\n\n- The paper does not discuss the potential limitations of the proposed framework.\n\n- The paper does not introduce anything novel, rather just combines existing components together to generalise to new tasks (apart from word normalisation which is fairly trivial). \n\n- The paper does not compare to other LLM fine tuning baselines which generalise to new unseen tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6IF891np6q", "reviewer": "Reviewer_Xize", "summary": "This paper studies using a large language model (LLM) as a decision-making agent to interact with an embodied environment. Specifically, the authors proposed to train the policy with reinforcement learning (RL). First, the policy is formed by querying the LLM with each feasible action in a given state. Then, token normalization and word normalization approaches are proposed to stable the training. The proposed approach is evaluated on Overcooked and VirtualHome. The experiments show that the proposed approach outperforms baselines in terms of sample efficiency and return.", "questions": "1. As discussed in Weakness, comparing with methods that also leverage LLM for RL agent could make the experimental section more convincing.  \n\n2. In the experiments, only two tasks from virtualHome and Overcooked are considered. Reporting results on more tasks would be helpful. \n\n3. In Section 4.3, it reads “The critic’s MLPs use the last token of the observation prompt as input …”. Could you elaborate why only the last token of the observation prompt is used? Shouldn’t the MLP use the output of the frozen LLM?\n\nMinor:    \n  4. In Eq (2), using $a_k$ in the summation is confusing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality and Significance:   \nThe reviewer appreciates the motivation of the paper: Classical RL agents align well with a given environment but fail to leverage prior knowledge. On the other hand,  LLM has numerous prior knowledge while often fails to align with the given environment. This paper proposed a method to align the LLM policy with the environment and show improvement over classic RL methods.  \n\nQuality:  \nThe paper is technically sound and the claims are supported by experiments. \n\nClarity:    \nThis paper is generally well-organized and easy to follow. Some minor improvement may be required. Please see the Question section for details.", "weaknesses": "1. The idea of querying joint probabilities of each valid action with LLM to form a policy is explored in previous works such as GLAM (Carta et al., 2023). The reviewer found, in the current version, the credit is not clearly given to authors of previous works. \n\n2. The reviewer has some concerns on the baselines used in the experiments. In Figure 5, aside from the ablations, the authors only consider classic PPO as a baseline. It seems unfair to directly compare an agent equipped with LLM with a classic trial-and-error PPO agent which doesn’t have access to the prior knowledge in LLM. There are many existing works that leverage LLM to form RL policies [1, 2, 3]. Particularly,  Li [1] also uses virtualHome as test environments. Comparing approaches that also leverage knowledge in LLM could make the experimental section more convincing. \n\n3. The technical contribution of the paper is somewhat limited. Specifically, the idea of forming valid policy is proposed by previous works and LoRA for fine-tuning is a standard method. The review found the main new method is the proposed normalization approach. That to be said, the reviewer still considers the combination of existing methods might be valuable to our embodied AI community. \n\n[1] Pre-Trained Language Models for Interactive Decision-Making, Li et al., NeurIPS 2022\n\n[2] Do As I Can, Not As I Say: Grounding Language in Robotic Affordances, Ahn, CoRL 2022. \n\n[3] Grounding large language models in interactive environments with online reinforcement learning, Cartra et al, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tf3kjrWanD", "reviewer": "Reviewer_fSyb", "summary": "This paper provides TWOSOME (True knoWledge cOmeS frOM practicE), a Large Language Model (LLM)-based policy learning method that fine-tunes a LLM agent with a Reinforcement Learning (RL) algorithm. To address problems incurred when adopting text actions sampled from a LLM policy, this paper proposes a normalization method that divides the log probability of a text action by the number of words (or tokens). This paper demonstrates the effectiveness of TWOSOME by employing Llama-2-7B as a LLM agent and fine-tuning the LLM agent with Proximal Policy Optimization (PPO). This paper provides experiment results in Overcooked and VirtualHome, and shows that the LLM agent fine-tuned with PPO can learn better policy.", "questions": "- Q1. Regarding W1, how does the word normalization help LLM agents to learn better policy? What is the intuition of the word normalization?\n\n- Q2. Regarding W2, without manual action masking in VirtualHome, how much scores can TWOSOME achieve?\n\n- Q3. Recently, in RLHF, some enhanced RL algorithms such as Direct Policy Optimization (DPO) and Pair-wise Proximal Policy Optimizaition (P3O) have been proposed. Can these enhance RL algorithms be applied to solving decision making tasks? And then, can we expect some performance improvements of TWOSOME?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- S1. The main approach of fine-tuning a LLM agent (e.g., Llama-2-7B) with a RL algorithm (e.g., PPO) proposed in this paper seems interesting and promising. This approach can be seen as applying recent prevalent reinforcement learning with human feedback (RLHF) into decision-making tasks.\n\n- S2. This paper provides promising initial results toward fine-tuning LLM agents with RL methods.", "weaknesses": "- W1. The overall approach of TWOSOME is to fine-tune a LLM agent (e.g., Llama-2-7B) with a RL algorithm (e.g., PPO). And, the unique contribution of TWOSOME can be seen as providing a normalization method for the probability distribution over text-based actions sampled from a LLM policy. This paper proposes two normalization methods: token normalization (Eq. 3) and word normalization (Eq. 4). Even though this paper proposes a promising direction towards fine-tuning a LLM agent with a RL algorithm, the main contribution seems rather marginal. The normalization method is about dividing the log probability of a varying length text-based action by the number of words (or the number of tokens). It seems to compensate a text action with larger number of words, but I am not sure that this kind of simple division is a proper way to normalize varying length text actions sampled from a LLM policy. \n \n- W2. This paper shows that TWOSOME can learn the optimal policy in a simple environment like Overcooked. However, I am not sure that TWOSOME can properly learn optimal policy in a complex environment like VirtualHome. When showing the performance of TWOSOME in VirtualHome, the authors masks out unrelated actions to reduce the complexity of the action space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MsrafnBNRd", "reviewer": "Reviewer_HnZU", "summary": "The paper introduces TWOSOME, a novel finetuning method designed to ground Large Language Models (LLMs) in embodied environments. TWOSOME comprises three main components: action-selection, where the LLM identifies the best action from a list of actions on criteria such as likelihood, token normalization, and word normalization, LoRA updates and prompt design. The authors conducted empirical experiments using TWOSOME on four fine-tuning tasks within Overcooked/VirtualHome environments. Their results demonstrate that TWOSOME's word normalization component outperforms others. Moreover, the fine-tuned policies exhibit superior generalization to unseen tasks in zero-shot scenarios. Remarkably, TWOSOME's capabilities in NLP benchmarks remain unaffected after the fine-tuning process.", "questions": "- Can you elaborate on the difference between (Carta et al, 2019) and unnormalized action selection.\n- Can you explain the PPO baseline: state/action space and architecture and initialization here (and in the main paper)? This would help in understanding the contribution of pretrained initialization.\n- Discussion question (not required to increase my score): \n     - Do you have insights on the use of embodied environments to **improve** LLMs, e.g. dealing with safety concerns?  How to design such environments?\n    - It would be nice to see how multi-modal text generation could be used to remove the assumption that the agent can extract a textual description of its state.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The authors noticed the limitation of considering plain action likelihood (a problem known in NLP when selecting different texts based on likelihood of the full sentences). To my knowledge, the word normalization is new and I really liked the idea when noticing that dividing by the sequence length is not enough to mitigate the end of words being more likely.\n- The authors worked in a compute budgeted setting so the work can be reproduced easily.\n- I really liked the task generalization section.\n- Results on showing little to no catastrophic forgetting on NLP tasks (possibly coming from LoRA?) are really promising.", "weaknesses": "- As for now it is hard to consider LoRA to be part of the proposed method: hard to see the benefits of the LoRA contribution to sample efficiency and no catastrophic forgetting without an ablation on with and without LoRA. \nThe contribution of the method seems to be the action normalization and some prompt design (qualitatively assessed by the authors in 4.4).  Could you add experiments on these points? \n\n- Paper mentions that (Carta et al, 2023) “focus on primitive actions in toy environments without rich semantics“. I am not sure actions are primitive in (Carta et al, 2023) but rather semantically described actions, e.g. “go forward”, “turn left”, “drop and toggle”. What makes them more “primitive” seems to be the environment they chose for experimenting (BabyAI vs Overcooked/VirtualHome). \n\n- Regarding the choice of environments. I am afraid there are too few fine-tuning tasks. I might be wrong, but the task generalization seems almost too-good from 4 tasks only. BabyAI was procedurally generated which would have enabled experimentation on more finetuning tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)