Language Models Meet World Models: Embodied Experiences Enhance Language Models
Authors: Jiannan Xiang
Tianhua Tao
Yi Gu
Tianmin Shu
Zirui Wang
Zichao Yang
Zhiting Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: Language Model
World Model
Embodied Experience

Abstract: While large language models (LMs) have shown remarkable capabilities across numerous tasks, they often struggle with simple reasoning and planning in physical environments, such as understanding object permanence or planning household activities. The limitation arises from the fact that LMs are trained only on written text and miss essential embodied knowledge and skills. In this paper, we propose a new paradigm of enhancing LMs by finetuning them with world models, to gain diverse embodied knowledge while retaining their general language capabilities. Our approach deploys an embodied agent in a world model, particularly a simulator of the physical world (VirtualHome), and acquires a diverse set of embodied experiences through both goal-oriented planning and random exploration. These experiences are then used to finetune LMs to teach diverse abilities of reasoning and acting in the physical world, e.g., planning and completing goals, object permanence and tracking, etc. Moreover, it is desirable to preserve the generality of LMs during finetuning, which facilitates generalizing the embodied knowledge across tasks rather than being tied to specific simulations. We thus further introduce the classical elastic weight consolidation (EWC) for selective weight updates, combined with low-rank adapters (LoRA) for training efficiency. Extensive experiments show our approach substantially improves base LMs on 18 downstream tasks by 64.28% on average. In particular, the small LMs (1.3B, 6B, and 13B) enhanced by our approach match or even outperform much larger LMs (e.g., ChatGPT).

Language Models Meet World Models: Embodied Experiences Enhance Language Models
Jiannan Xiang" Tianhua T Yi Gu Tianmin Shu Sh 94 Zirui Wang*, Zichao Yang", Zhiting Hu UC San Diego, UIIC, MIT, AJHU, CCMU
Abstract
While large language models (LMs) have shown remarkable capabilities across numerous tasks, they often struggle with simple reasoning and planning in physical environments, such as understanding object permanence or planning household ac- tivities. The limitation arises from the fact that LMs are trained only on written text and miss essential embodied knowledge and skills. this paper, we propose new paradigm of enhancing LMs by finetuning them with world models, to gain diverse embodied knowledge while retaining their general language capabilities. Our ap- proach deploys embodied agent in world model, particularly simulator of the physical world (VirtualHome), and acquires a diverse set embodied experiences through both goal-oriented planning and random exploration. These experiences are then used to finetune LMs to teach diverse abilities of reasoning and acting the physical world, e.g., planning and completing goals, object permanence and tracking, etc. Moreover, it desirable to preserve the generality of LMs during finetuning, which facilitates generalizing the embodied knowledge across tasks rather than being tied to specific simulations. We thus further introduce the classical elastic weight consolidation (EWC) for selective weight updates, combined with low-rank adapters (LoRA) for training efficiency. Extensive experiments show our approach substantially improves base LMs 18 downstream tasks by 64.28% on average. In particular, the small LMs (1.3B, 6B, and 13B) enhanced by our approach match outperform much larger LMs (e.g., ChatGPT),
Introduction
Language Models (LMs) have demonstrated impressive performance on wide range of natural (44) In particular, recent studies show that LMs can assist language processing tasks [34 48 4
25 45 19) demonstrating a certain level of understanding of the physical world. However, such understanding is not robust enough for many reasoning and planning tasks in physical environments. As shown in Figure [ even the latest large LMs like ChatGPT can still make mistakes in seemingly simple inquiries, such as counting objects a location. We hypothesize that this is because current LMs trained merely with large-scale text corpora are devoid of embodied experiences such as navigating in an environment, interacting with objects, and sensing as well as tracking the world state. Consequently, they lack robust and comprehensive embodied knowledge necessary for reasoning and planning associated with physical environments. A related line of research finetunes LMs in order to improve specific embodied tasks, resulting in 57.
decision-making for embodied tasks [
sess-specialized models 6 58
* "qqual contribution.
'The code is available at http:///g/ggggttooooo/oo//aaaaaa language-gmedee Based on GPT-3.5-turbo.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Track & Count
Track & Permanence
Plan Generation
placed cell phone on top at it. (trelevant/icchnn rictionsi, putting the time down next the cell phone. fireleventtAccionn Accioms) She finally by put apple the desic.
Emily
John put book on desk. preeovant Accioney Mary took the book, fiinelevent Actions She plocod the safa. Where the aco
Question:
How to watch TV?
How many hams there on the dask??
Answer:
Chindlt There
There are two items.
It was the desk.
X
on
to living room.
LM
Wolk to living room. 2. Sit on sofa.
Chance
Walk living room
3. Watch TV.
Goal-Oriented Afanning
Activity: Watch TV Walk to kingg room. Sit cn sofs. Warch TV
E2WM
Finetune
Language Model GPT-J-88)
training paradigm
Action
Tracking lof book) On bod, budraam
Rendom
Walk room.
Exploration
Walk living room.
bock.
hand, living room.
hand, bedroom
Embodied Experiences
Model
Figure 1: Examples of tasks requiring embodied knowledge (upper), and an overview of our approach (bottom). In the task examples, blue text indicates the useful information for answering the question.
In this paper, we aim to inject diverse fundamental embodied knowledge and skills into pretrained LMs, while retaining the models' generality. We introduce novel training paradigm for LMs fine- tuning with Embodied Experiences from World Models (E2WM). Here, world models are embodied simulators that emulate physical interactions in real-world environments (e.g., VirtualHome (36)). They provide LMs with the opportunity to comprehend object interactions within the environment and to execute actions, thus enabling a level of active engagement previously unattainable. These world models serve as a simplified and cost-effective replica of our real world that can significantly augment the conventional pretraining paradigm. We anticipate that finetuning LMs embodied experiences gathered from world models can enhance their embodied knowledge- and, with the preserved model generality consequentyy strengthen their abilities solve broad range of embodied tasks.
In this work, we consider a diverse range of fundamental knowledge and skills for embodied tasks, including tracking objects, planning to complete given goals, recognizing other agents' behaviors, etc. To this end, we introduce two ways to collect embodied experiences from world models that give rise to the desired knowledge and skills: goal-oriented planning and random exploration (Figure T]. Specifically, goal-oriented planning aims to gather experiences associated with planning and goal-oriented agent behaviors, while random exploration focuses accumulating experiences that involve object and world state tracking. In goal-oriented planning, models are given the goal (e.g.. IN (dust, trash can)) ) for specific activity (e.g.. Clean Floor), and is supposed to generate plan to complete it. To find the plan, we devise Monte Carlo Tree Search (MCTS) [ 44 to explore the world model. Then the process will be stored as embodied experience. In random exploration, one or more agents are deployed in the world model to execute random actions, while the locations and the movements of all the objects are tracked simultaneously.
After collecting the embodied experiences, we use them to construct set of fine-tuning tasks (e.g.. plan generation, activity recognition, and tracking). Crucially, to finetune LMs on the collected embodied experiences while retaining their original general knowledge and capabilities, we propose to incorporate the classical Elastic Weight Consolidation (EWC) 22 into training paradigm. By regularizing the finetuning loss, EWC aims to preserve the important LM parameters from pretraining. We show that EWC is substantially more effective than the popular KL regularization [35] 28 31). We further introduce efficient low-rank updates by harmonizing the recent Low-Rant Adaptation (LoRA) [6] with the EWC regularizer. This results in the new EWC-LoRA update rule that greatly reduces training costs and makes our E2WM paradigm accessible to cheap hardware (GPUs).
We instantiate world model using virtual household simulator, VirtualHome 36 37 and apply our method to GPT-Neo- 1.3B (3) GPT-J-6B [49] OPT-13B (59), and L 48 models. To
test the generalizability of the finetuned LMs, we evaluate them variety of unseen tasks which

---

demand similar embodied knowledge required to solve the training tasks. Additionally, we assess the models' performance on the original pretraining data to determine the extent which their core language modeling abilities are retained. Experiments show that our method significantly improves the baselines on both seen and unseen tasks (e.g., 34.31 - 51.23 Rouge-I. on plan generation task, 30.41% -> 67.01 % accuracy counting task), without suffering performance degradation on the pretraining dataset (3.443 + 3.537 perplexity on Pile test subset (12).. Moreover, the small GPT- J-6B, OPT-13B, and LLaMA-13B models finetuned with our E2WM paradigm even outperforms ChatGPT on many of the tasks. The experimental results demonstrate the effectiveness of E2WM as a promising fine-tuning mechanism to enhance pretrained LMs with generalizable embodied knowledge and skills.
2 Related Work
World Model. The term "world model" generally refer to a computational representation of the physical world, capable of simulating changes in the world's state in response to various actions. For instance, humans possess an internal world model that aids in predicting the outcomes of specific actions during the planning process. Recent research induces world models from large LMs for robust human-like reasoning [.5]. In this work, we employ a simulator equipped with a physics engine to serve as our world model, effectively emulating real-world conditions. In the field of embodied AI, various world models are built to replicate the real world and serve as virtual test environments for assessing robotic agents before real-world deployment. For example, VirtualHome [36] 37) is a simulated 3D household environment implemented by Unity3D game engine. A12-THOR [23 consists of near photo-realistic 3D indoor scenes and has richer object attributes and interaction types. Other indoor household World Models include VRKitchen [33, CHALET [56), MINOS [4], House3D [53]. etc. Besides, MineCraft is a more challenging and open-ended world model, which has a large number of objectives and a large-scale task hicrarchy [4] 27 In this paper, we use world model.
VirtualHome
Language Model Grounding, A significant number of recent works focused on grounding language models to world models [ 24 38
47 (55] Some of them freeze LMs and leverage certain prompting
strategies or specifically-desiinne
modules. For example, Zero-Shot Planner [88 prompts LMs to generate activity plans and translate them into admissible actions. Mind's eye (29] prompts LMs to do simulations with physical engines to answer physical reasoning questions. SayCan I uses learned affordance function to assist LMs in selecting valid actions. DEPS (50) prompts LMs to describe, explain and generate action plans, incorporated with a learned selector module to choose the most efficient path. There also other works finetuning LMs towards better downstream task performance. For example, Li et al. (25) finetune LMs with supervised learning for interactive decision making, and Carta et al. (6] ground LMs with online reinforcement learning. Different from these works aiming to optimize L.Ms for specific tasks in the target environments, our work instead focuses improving the language model itself by acquiring knowledge from world models.
Language Model Regularization. To facilitate the acquisition of new knowledge and skills without losing LMs' language modeling abilities, regularization is often introduced during finetuning. One popular method is adding KL penalty (35] 28 46
33 60). which leverages KI divergence between the output probability of the currently trained model and the original model to regularize the LM in an RL manner, i.e., by computing policy gradients. For example, InstructGPT uses KL penalty to mitigate over-optimization of the reward model [35]. and Liu et al. (28] add KL regularization for training a commonsense knowledge generator. In this work, we instead use elastic weight consolidation (EWC) for regularization. Our empirical results demonstrate that EWC is more effective than applying KL penalty for retaining language modeling abilities and generality of LMs.
52
3 Approach
In this work, we propose new training paradigm, namely finetuning with Embodied Experiences from World Models (E2WM), to inject embodied knowledge into LMs without sacrificing its general- ity and language modeling abilities. The world model we use VirtualHome [36] 37), a multi-agent
simulator for househould activities. In Virtual an executable action step can be simplified as the format of [action] <arg>, e.g. [Grab] <apple> , The world state of VirtualHome consists of objects and their relations (e.g., apple on table). Details about VirtualHome can be found in Appendix
3

---

Goal - -rrientrd Planning
Random Exploration
Activity: Watch TV
Grab pillow
Give pillow to (0
bothessor
WAALE room
Take piloow
+1.9
Grab apple
Walk living
toonhguria
Work teing room
apple
Put apple on rablc
0.1
+1.9
+1.9
0.11
0.1
Walk to bathroom
Tumbon TV
Walk to bedroom
MCTS
41.
Put pillow on table
Question:
Question:
Question:
Quacsion:
How watch TV? and sofa livig
Givan a pan: living room, Sit on sofa. Turn on TV. What is the task?
Tor grabbed pillow: Tom
Tom griboed pillo.. Tom walkeed to kätchen -hat is the order of nooms where pillow appiars?
gave pillow to in How many ocjects are on the table?
Answer:
Walk to living room. Sit
Answer:
Arswer:
TV.
Answer: Watch
Two. They are pillow and apple.
nodroom, kitchenn living room
Plan Generation
Activity Recognition
Counting
Object Path Tracking
Figure 2: The illustration of goal-oriented planning (left) and random exploration (right) in our training paradigm. In MCTS, the path in orange represents the final plan generated by the planner.
A.1 We first describe how to gather embodied experiences in the world model in Section 3.1 Then in Section 33.2 we demonstrate how to finetune LMs by utilizing collected experiences, as well as our proposed method EWC-LoRA for efficient knowledge generalization.
3.1 Collecting Embodied Experiences from World Model
LMs pretrained on large scale human-written text corpus often have difficulties in solving basic reasoning and planning in physical environments. This is because that LMs lack necessary embodied knowledge and experiential understanding of the physical world. To address the problem, we propose to leverage world models to collect diverse embodied experiences for enhancing LMs. Specifically, to inject different types of embodied knowledge into LMs, we introduce two ways gather experiences: goal-oriented planning and random exploration. Figure illustrates the two methods.
Goal-oriented Planning. One important embodied skill is to plan and complete specific goal, e.g., placing tableware properly to set up the table. To endow LMs with this ability, we propose goal-oriented planning. The approach aims to generate experiences that goal-oriented, thus are useful to facilitate the acquisition of skills and task planning abilities for executing a range of activities in the world model. To do that, collect various activities and their corresponding goals. Formally, the goal for an activity in the world model is defined as a set of predicates describing the target world state. For instance, an activity can be set up table, and its goal can be (fork, table) DN (plate, table), which means that fork and plate should be put on the table to fulfill the activity. More details about predicates and goal definitions can be found in Appendix A.2 As shown in Figure in goal-oriented planning, we devise a Monte Carlo Tree Search (MCTS) planner to search through the action space and find a plan, i.e., a sequence of actions, to achieve the goal. The key to successful MCTS is the reward design. At each time step, if at least one goal predicate is satisfied, the MCTS planner will get reward of +2, and the achieved goal predicates will be removed from the goal. This ensures that the planner does not repeatedly execute the same action to receive rewards, but rather focuses on achieving the remaining unfulfilled goals. Besides, it will get a -0. penalty after each time step to discourage the planner from doing actions irrelevant to fulfilling the goals. Finally, we store the planning process as an embodicd experience.
Random Exploration. In real-world scenarios, humans not only acquire new knowledge by finishing tasks, but also learn by just randomly exploring the surroundings, e.g., randomly observing/tracking objects and knowing their properties. To mimic this learning process, we propose another approach, namely random exploration. By simply exploring the world model, embodied experiences emerge that involve ad advanced cognitive abilities including object permanence and tracking, as agents observe and track the consistent existence of objects even when they are out of sight. Then the experiences are gathered for finetuning LMs later. Specifically, the approach deploys one or multiple agents in the world model wandering aimlessly and randomly executing actions. As illustrated in Figure 2 multiple agents are in the same environment, interacting with each other or executing different actions

---

on the same objects, which simulates complex situations. During the exploration, the moving paths and the final locations of all the objects in the world model are recorded. Then the whole process is captured as an embodied experience.
3.2 Finetuning LMs with Embodied Experiences
There are multiple ways utilize collected embodied experiences for LMs finetuning, such as supervised learning and reinforcement learning. In this work, we use them with supervised learning for simplicity and efficiency. Specifically, goal-oriented planning experiences are compiled into data examples in two formats: plan generation and activity recognition. As shown in Figure [2] in plan generation, the model required to generate a stepwise action sequence to fulfill an activity, given the state of some relevant objects as the initial condition. In activity recognition, the model needs to recognize the activity name given its plan. Experiences obtained from random exploration are also transformed into two self-supervision tasks: counting and object path tracking. Examples of the two tasks can be seen in Figure Specifically, for counting, the LM is tasked with identifying the number and name of the objects at specific location after the agents performed relevant and irrelevant actions and arranged objects randomly. In object path tracking, the model tasked to output the moving path of an object that is picked up by different agents and moved to different rooms at different times. All the tasks are trained with cross-entropy loss. Suppose that x is the imput (e.g., the initial condition in plan generation) and y {yI,.... FM } is the label (e.g.. the stepwise action sequence), we finetune LMs by assigning different weights to different tasks:
Lv=
Lv = => VEV
Em M
m=1 log P(ym y [m: x),
where is the loss function; V is the task set; and a, is the weight for task u. Following Flan-T5 [8] x is a prompt formatted to contain task instruction and sampled in-context demonstrations. We
provide all prompts in Appendix A..
Efficient Finetuning with Preserved Generality. However, there are two key problems for simply finetuning LMs. The first one is that LMs will easily overfit to the downstream tasks, leading to performance degradation on other tasks. This deviates from our goal that the model should generalize acquired knowledge across various tasks. Another problem is that finetuning the entire I.M is resource-intensivg and time-consuming especially when the L.M is extremely large. To overcome the problem and facilitate continual and efficient knowledge acquisition with world models, we propose to finetune only a small number of weights using low-rank adaptors (LoRA) [66 with elastic weight consolidation (EWC) [22), which we refer to as EWC-LORA.
EWC is a regularization-based method typically used in the area of continual learning 9 It calculates a fisher matrix [I]] to estimate the importance of each parameter for task and then uses it to regularize the training on a new task. The regularization term helps to constrain the parameter updates for the new task to avoid forgetting the previous knowledge. Let U be the pretraining task set, and V be the finetuning task set. Following (2). we have:
acc 2
Fr, 1 Σ
1 N
N j=1 00%
((0)= Lv + 2 +, Fe. Og,)²
where L is the loss function, the fisher matrix, is the hyperparameter, and j are the indices for parameters and data samples, respectively, and 0 and 07 are currently trained parameters and frozen task U parameters, respectively. Notice that the first term (0) in Equation 3 is calculated in Equation [ and the second term is the EWC regularizer. In Equation 2 the fisher matrix is calculated by averaging the sum of squares of the gradients from the task U, which indicates the significance of each parameter to the task U. Then the matrix is used in Equation! B to weigh the shift of model parameters when training on V. By using EWC, the LM learns to adapt to new tasks without catastrophic forgetting the pretraining task, which forces it to understand and digest new knowledge from the finetuning tasks instead of overfitting to them.
However, EWC is both time- and memory-inefficient. First, it requires finetuning the entire set of large LM's parameters. Moreover, the approach involves creating a frozen original model and a fisher matrix that is the same size as the LM, leading to memory overhead of three times the original size.
5

---

This makes it particularly challenging to apply to larger models. To alleviate the problem, we propose combine EWC with low-rank adaptors (LoRA), a parameter-effiiieee tuning method. LoRA freezes the pretrained model weights and injects two trainable low-rank matrices into each layer of the model. Suppose that W, W E Rrxd are the trained weight matrix and frozen weight matrix, respectively; and B E IR"* A E R are two low-rank matrices with & & min(r: d). Then the formula for LoRA can be written as W = W + BA. Suppose that H is flattened BA. Notably, we found that 0, in Equation is the element of W. and Of,, is that of W Therefore, 0; 0ū is the element of H. We can thus transform Equation P into the final formula of EWC-LoRA method:
(()) (v) + A Σ, Fi,hh
(4)
where h; = 0; B. is the i-th element of H. One of the benefits of this rewriting is that we no longer need to store the trained LM weight matrixes as what vanilla EWC does, which saves plenty of memory space. Besides, we only need to update B and A during the finetuning, which also lowers memory requirements and leads much faster training speed. Surprisingly, as shown later, we empirically found that adding LoRA into EWC can further mitigate the issue of catastrophic forgetting and overfitting. This aligns with the previous conclusion that limiting the dimension of the optimization problem can alleviate catastrophic forgetting 32
4 Experiments
Training Details. For goal-oriented planning, we collected activities and their corresponding target goals with data from RobotHow (36). housework activity knowledge base created in VirtualHome. We applied our method to GPT-Neo- 1.3B [3 GPT-J-6B [49 OPT-13B [59], and LLaMA-133B (48). To save computing resources, we use Int8 technique [IO]. Both of the models were trained with
the AdamW optimizer (30). All the hyperparameters are chosen according to the performance on a
held out set. We used one NVIDIA GeForce RTX 3090 for training. More details can be found in Appendix A.4
4.1 Downstream Evaluation Tasks
We developed various downstream evaluation tasks for each type of embodied knowledge, including both the training tasks as well as novel tasks unseen during training used for generalization evaluation. Additionally, we evaluate our models on bAbl [51, dataset for testing multiple types of knowledge and abilities including embodied knowledge, logic reasoning, linguistic knowledge, etc. We select the bAbl tasks related to embodied knowledge for our evaluation. We evaluate all the unseen tasks including bAbl under few-shot settings, specifically 2- 2-10 shots, by providing a few in-context exemplars in the prompts. We díscuss more details of the tasks below.
Plan Generation. To evaluate planning ability, construct downstream tasks using human-written plans from RobotHow. Specifically, we have three tasks:
Plan Generation Evaluation. In this task, the model needs to generate plan for housework activity. It is similar to the training task but uses human-written plans as the ground truth instead of the collected experiences. We include activities unseen during training to test the generalizability of the model. Inspired by the previous study showing that LMs can easily be distracted by irrelevant context [43), we also created samples having states of unrelated objects in the context (e.g., TV is on for activity Make Coffee) to confuse LMs. In summary, this results in four settings: Vanilla Seen, Vanilla Unseen, Confusing Seen, and Confusing Unseen. We have 175/54/135/43 examples for four settings, respectively. We use Rouge-L. 26 as the metric.
Housework QA. This is multi-choice QA task, which is unseen during training. It asks which choice is the relevant object to finish a household activity, e.g., which object is relevant to making coffee? It has 261 examples in total, and we use accuracy as the metric. When evaluating, we provide 10 in-context exemplars in the prompts, so this task is evaluated as a 10-shot learning task.
Negation Housework QA. This is similar to Housework QA but inquires about the irrelevant object, e.g., which object is irrelevant to making coffee? is more challenging than the vanilla QA because LMs that simply memorize the word co-occurrence in the training data may succeed in the vanilla QA but will fail in the negation QA. This task has 162 examples and uses accuracy as the metric. We provide 10 in-context exemplars in the prompts.

---

Activity Recognition. We developed two multi- cuoti- QA tasks with the human-written plans and the state changes from RobotHow to test the knowledge gained from activity recognition:
Activity Recognition QA. In this task, a human-writtr plan is given and the model needs to choose the correct activity name. An example of the question is "Given plan: Walk to living room. Sit on sofa. Turn on TV. What is the name of this activity?". And the answer should be Watch TV. The task has 549 examples. We use accuracy as the metric.
Activity Inference QA. In this task, we use the final state of the world model as input for the model to infer the activity name. For example, the input can be "Tom is sitting on the sofa and facing the TV. The TV is on. What is a possible activity he is doing?", and the answer is "Warch TV". We have 262 examples for this task and use accuracy measure the performance. We provide 10 in-context exemplars in the prompts.
Counting. We gathered random exploration experiences to construct Counting QA for evaluating skills learned from the counting task. The model is required to answer the number of objects in a specific location. For example, a query can be "Tom put an apple on the table. Tom turned on the TV. Tom put a cup on the table. How many objects are there on the table?". We can see that there will be irrelevant actions like lurn on TV to confuse the model and make the question more challenging. We collected 194 samples for the task and used accuracy as the evaluation metric. We provide in-context exemplars in the input prompts.
Object Path Tracking. We developed two downstream tasks for the object path tracking training task, namely Object Path Tracking Evaluation and Object Location QA.
Object Path Tacking Evaluation. This evaluation task is the same as the training task, where the model is required to generate the full moving path of an object. An example is "Tom walked the kitchen. Tom grabbed the apple. Mary walked the bedroom. Tom walked to the living room. What is the order of the rooms where the apple appears?". This question typically includes multiple agents and many irrelevant actions, which makes it difficult to track the object. This task contains 200 examples. Following Huang et al. [18), we evaluate the performance by calculating the length of the longest common subsequence (LCS) between the ground truth and the generated path, normalized by the maximum length of the two.
Object Location QA. In this task, the model is asked about the location of object before/after it moves to another location, egg., where is the apple before/after the kitchen? This task has 200 examples with accuracy as the metric. We provide in-context exemplars in the prompts.
A previous study on prompting multiple QA questions [39] introduces two prompting methods, multiple choice prompt and cloze prompt, and two normalization methods, length, and unconditioned normalization. For all the multi-choice QA tasks, we choose the combination of prompting and normalization methods which yields the best performance
held out set.
To further verify the effectiveness of our method, we evaluate our finetuned GPT-Neo and GPT-J on the bAbI dataset. Specifically. we select 8 test sets from bAbl that align with the abilities covered in our collected embodied experiences. We include the description of each test set Appendix [A.5] For all the bAbl tasks, we do 2-shot learning by providing 2 in-context exemplars in the input prompts.
Besides downstream tasks, we also want ensure that our approach does not hurt language modeling performance of the models. Therefore, following previous work 42), we evaluate the perplexity on subset of Pile [2] test set, which is the pretraining dataset for GPT-Neo and GPT-J. We sampled 5000 examples from Pile test set for evaluation.
4.2 Results
Constructed Evaluation Tasks. Results for all the downstream evaluation tasks are shown in Figure 3 and Figure For all the models, we compare the results obtained after finetuning with world model against those of the original base models. For GPT-J, we also include finetuned model without EWC-LoRA as a baseline. Detailed numbers of the results can be found in Appendix [A.6] We also conduct human evaluations for GPT-J on the plan generation task, which can be found in Appendix A.7 In general, the models trained with the world model significantly outperform the baselines on various downstream tasks. Our method is not only effective for small 1.3B model, but can also scale to larger 6B and 13B models. Specifically, our finetuned GPT-J and LLaMA-13B
7

---

100
Performance (%)
40
Confusing Seen
Confusing Unseen
Object Tracking
Negation Hourework
Activity Recognition
Activity Inference
Counting
Object Location QA
UnSeen
QA
Plan Generation
(RougeL))
(Acourecy)
GPT-Neo (Ours)
GPT- (Ours)
GPT-Neo (Base)
GPT-J (Base)
GPT-J (FT)
ChatGPT
Figure 3: Experimental results of GPT-Neo and GPT-J on our constructed downstream tasks. GPT-J (FT) refers to the finetuned GPT-J without EWC-LoRA. Our approach surpasses baselines on all of the 1l tasks, and outperforms ChatGPT on of them. For example, our GPT-J model achieves 98.67 LCS on object path tracking, which is significantly better than 33.86 of base GPT-J and 59.53 of ChatGPT.
100
Porformance
Vanila Seen
Vanilia UnSeen
Confusing Seen
Confusing Unseen
Object Path Tracking
Housework QA
Negation
Counting
Object Location
Housework
Recognition
Inference
QA
Generation
(Rouge-))
(LCS)
(Accuracc)
OPT-13B (Ours)
LL-MA-33 (Ours)
OPT-13B (Base)
LLaMA-13B (Base)
ChatGPT
Figure 4: Experimental results of OPT-13B and LLaMA- 13B on constructed downstream tasks. Our approach applied on LLaMA- 13B outperforms ChatGPT on 8 of them.
Accuracy (8%)
60
20
Two Supporting
Counting
Lista/Sets
Simple Negation
Time Reasoning
Positional Reasoning
Supporting Fact
Fact
Supporting Fact
GPT-Neo (Ours)
GPT-J (Ours)
GPT-Neo (Base)
GPT-J (Base)
ChatGPT
Figure 5: Experimental results on bAbl. Our approach outperforms base LMs on all the tasks except for the Two Supporting Fact task.

---

GPT-Neo
GPT-J
OPT-13B
LLaMA- 13B
Base
Ours
Base
Ours
Base
Ours
Base
Ours
4.120
4.193
3.443
3.537
4.077
4.358
3.036
3.069
Table 1: Perplexity Pile test subset, showing the proposed finetuning with world model manages to preserve the LMs' language modeling capability.
GPT-Neo
GPT-J
Task
Base
EWC
LoRA
LoRA KL
EWC-LoRA
Base
LoRA
EWC-LoRA
Plan Gen
21.25
48.56
51.24
45.99
49.70
34.31
51.23
51.23
Act Recog
69.22
89.98
87.98
81.42
85,43
87.98
90.16
88.52
Count
22.68
55.67
27.84
49.48
28.87
30.41
63.92
67.01
Obj PT
30.80
95.96
87.28
63.59
85.91
33.86
97.22
98.67
Perplexity 4.120"
4.995
4.360
5.029
4.193
3.443
3.675
3.537
Table 2: Results of different regularization methods. The abbreviations in the Task column stand for the corresponding evaluation tasks for four training tasks. We use asterisk* to mark the perplexity of base models.
with world model even achieve better performance than ChatGPT as a much larger LM on most of the 11 tasks. Besides, we can see the world model improves LMs on both seen and unseen tasks. This demonstrates that model absorbs the knowledge for goal-oriented planning and random exploration instead of memorizing the seen experiences. Specifically, the better plan generation performance under the "Confusing" setting indicates that the world model improves the ability of LMs to avoid being interfered with by irrelevant contexts. On both Housework QA and Negation Housework QA, our models surpass the baselines, showing that our models also acquire knowledge about the necessary objects for completing a housework activity. Results on other downstream tasks also prove the effectiveness of our method. For example, on both Activity Recognition Evaluation and Activity Inference, our approach improves over the baselines significantly. Moreover, improvements can be observed in the downstream tasks regarding random exploration. On Counting and Object Location QA tasks, our LLaMA-13B trained with the world model even surpasses ChatGPT.
bAbI Tasks. To further verify the effectiveness of method, evaluate our finetuned models on the bAbl dataset. The results are shown in Figure 5 We can see that our approach significantly outperforms the base LMs. Notably, after finetuned with VirtualHome experiences, GPT-J surpasses the much stronger ChatGPT on the most challenging tasks. Specifically, it outperforms ChatGPT on the Three Supporting Fact task, where the model required use three supporting facts from the context to answer a question like "where was the apple before the kitchen?", and Lists/Sets task, which asks the model to give the answers the form of list, e.g., the answer for "What Daniel holding?" ?" is "apple, milk". These results prove that our approach enables LMs to acquire the knowledge and skills inherent embodied experiences, rather than simply overfitting to the training environment.
Language Modeling. In addition to verifying improved performance on the downstream tasks, we also report results on Pile test subset to ensure the preservation of the generality and language modeling abilities of LMs. From the experimental results shown in Table [ we can see that our approach only causes a negligible increase in perplexity over the base models. This demonstrates the effectiveness of EWC-LoRA to preserve the generality and linguistic competence of LMs. To verify the generality on other NLP tasks, we also include the results on SuperGLUE [40 in Appendix A.8
4.3 Comparison of Different Regularization Methods
We compare our proposed EWC-LoRA with EWC and LoRA. Besides, we also include the baseline using KL penalty as regularization. The experimental results are shown in Table We also include the results of four evaluation tasks. Notice that we do not include the results of GPT-J with pure EWC and KI., since they are overly memory-intensive or time-consuming. EWC requires an original model and a fisher matrix other than the trained model, which triples the memory usage, making

---

GPT-Neo -wo Act Recog
Base
Ours
-w/o Plan Gen
-w/o Count
-w/o Obj PT
Plan Gen
21.25
49.70
14.48
49.38
49.85
50.06
Act Recog
69.22
85.43
85.97
48.63
85.25
84.34
Count
22.68
28.87
18.56
25.26
35.05
32.99
Obj PT
30.80
85.91
92.13
84.17
86.46
29.90
Perplexity
4.120
4.193
4.171
4.151
4.162
4.164
Table 3: Ablation experimental results on training tasks. We use the same abbreviations as Table
it hard to be applied to large models like GPT-J-6B. Besides, KL penalty term is computed by LKL E(x,y)>Pp-- log (Pa- (ylx)/Pa PaylPPay/aa [x)]], thus it requires sampling from the model output probability, which is time-consuming. On the contrary, EWC-LoRA is both memory- and time- efficient. In Table 2 we can see that EWC-LoRA achieves the lowest perplexity compared to other methods, while still significantly outperforming the base LMs. Compared with pure EWC, applying pure LoRA greatly decreases perplexity, which is consistent with the previous conclusion that limiting the dimension of the optimization problem mitigate catastrophic forgetting 322 EWC-LoRA further decreases perplexity, making it extremely close to the original perplexity, while achieving comparable performance with LoRA downstream tasks. This demonstrates the effectiveness of EWC-LLRRA. Besides, We can find that combing LoRA with KI. will greatly increase perplexity while not achieving better downstream performance. Overall, our proposed EWC-LoRA achieves the best trade-off between the perplexity and the downstream performance, which outperforms baselines significantly while almost not increasing the perplexity on the pretraining dataset.
4.4 Ablation Studies
To study the contribution of each training task, we conducted an ablation study by removing one training task every time. We GPT-Neo-1.3B the base model. We include the results tasks seen during training in Table 3 Results on the tasks can be found Appendix [A] We can see that the removal of training task with similar ability leads to a notable decrease in the model's performance on downstream tasks. For example, the performance of plan generation drops significantly when plan generation is removed from the training tasks. Similarly, the removal of activity recognition or object path tracking from the training tasks leads to decline in performance in their respective downstream tasks. We conclude that our gathered embodied experience has tremendous contribution to teaching the corresponding reasoning ability by finetuming. Interestingly. Counting QA performance shows an increase when counting omitted from the training tasks, possibly because the ability of counting can be inferred from other training tasks.
5 Conclusion & Future Work
We proposed a new training framework that uses world models to enhance language models. first collects embodied experiences from world models through both goal-oriented planning and random exploration. The experiences are then compiled into appropriate formats for LMs finetuning. We further introduce EWC-LoRA, which not only facilitates parameter-efficient tuning but also alleviates catastrophic forgetting and enables knowledge generalization. We show the strong performance of method large number of downstream evaluation tasks.
This work demonstrates the advantage of panoramic learning with all forms experience [77. the other hand, the present work is limited to a single household environment as the world model. In the future, we intend study how to integrate embodied experiences from different world models and generalize knowledge learned from each world model to different domains.
10

---

Acknowledgements.
This project is partially supported by DARPA ECOLE HR001 12390063.
References
[1] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
[2] Kristjan Arumae, Qing Sun, and Parminder Bhatia. An empirical investigation towards efficient multi-domain language model pre-training. arXiv preprint arXiv:2010.00784. 2020.
[3] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Teiso--wnsrr,ow, March 2021.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] Cameron Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in games, 4(1):1 2012.
[6] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. arXiv preprint arXiv:2302.02665 2023.
[7] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint www.::2244.031111111 2022.
[8] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finttun language models. arXiv preprint arXiv:2210.114:: 2022.
[9] Matthias De Lange. Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting in classification tasks. IEEE transactions pattern analysis and machine intelligence, 44(7):3366- 3385, 2021.
[10] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer, Llm.int80: 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022.
[11] B Roy Frieden. Science from Fisher information, volume 974. Citeseer, 2004.
[12] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
[13] Xiaofeng Gao, Ran Gong, Tianmin Shu, Xu Xie, Shu Wang, and Song-Chun Zhu. Vrkitchen: an interactive 3d virtual environment for task-oriented learning. arXiv, abs/ aabs/1903.00 190.005777 2019.
[14] William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: a large-scale dataset of minecraft demonstrations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 2442-2448, 2019.
[15] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with Language Model is Planning with World Model. NewrIPS, 2023.
[16] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 09685, 2021.

---

[17] Zhiting Hu and Eric P. Xing. Toward a 'Standard Model' of Machine Learning. Harvard Data Science Review, 4(4), oct 27 2022.
[18] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 1118-9147 PMLR, 2022.
[19]
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and brian ichter. Inner monologue: Em- bodied reasoning through planning with language models. In Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning. volume 205 of Proceedings of Machine Learning Research, pages 1769-1782. PMIR, 14-18 Dec 2023.
[20] Anssi Kanervisto, Stephanie Milani, Karolis Ramanauskas, Nicholay Topin, Zichuan Lin, Jun- you Li, Jianing Shi, Deheng Ye, Qiang Fu, Wei Yang, et al. Minerl diamond 2021 competition: Overview, results, and lessons learned. NeurlPS 2021 Competitions and Demonstrations Track, pages 13-28, 2022.
[21] Yash Kant, Arun Ramachandran, Sriram Yenamandra, Igor Gilitschenski, Dhruv Batra, An- drew Szot, and Harsh Agrawal. Housekeep: Tidying virtual households using commonsense reasoning. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXXIX, page 355-373, Berlin, Heidelberg, 2022. Springer-Verlle Verlag.
[22] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barsiinkk,,,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)33521-335 2017.
[23] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474. 2017.
[24] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03553 2019.
[25] Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, al. Pre-trained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31 199-319993122,, 2022.
[26] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[27] Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang. Juewu-me: Playing minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint 122.4:21 2021.
[28] Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannanch Hajishirzi, and Yejin Choi. Rainier: Reinforeed knowledge introspector for commonsense question answering. arXiv preprint arXiv:22/0.03078 2022.
[29] Ruibo Liu, Jason Wei, Shixiang Shane Gu, Te-Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M Dai. Mind's eye: Grounded language model reasoning through simulation. arXiv preprint arXiv:2210.053399 2022.
[30] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[31] Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609 2022.
12

---

[32] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[33] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprini 12.00221 2021.
[34] OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
[35] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[36] Xavier Puig. Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang. Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494-8502, 2018.
[37] Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B. Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help A challenge for social perception and human- | collaboration. In International Conference on Learning Representations, 2021.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[39] Joshua Robinson, Christopher Michael Rytting, and David Wingate. Leveraging large language models for multiple choice question answering. arXiv preprint arXiv:22 10. 2022.
[40] Paul-Fdouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEÉE/CVE conference on computer vision and pattern recognition, pages 4938 4938-4947, 2020.
[41] Manolis Savva, Angel X Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun. Minos: Multimodal indoor simulator for navigation in complex environments. arXiv preprint 17Xiv:1712.03931, 2017.
[42] Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprini arXiv:2302.04761 2023.
[43] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. arXiv preprint arXiv:2302.00099 2023.
[44] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[45] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In Workshop on Language and Robotics CoRL 2022, 2022.
[46] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 0001, 2020.
[47] Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav Sukhatme. Embodied bert: A transformer model for embodied, language-guides visual task completion. arXiv preprint arXiv:2108.0449 2021.
13

---

[48] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Mnne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 139771, 2023.
[49] Ben Wang and Aran Komatsuzaki, GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https:// http:////.oooiiioooo/////////eee wttteatttaassrrereeeeeeeoe
May 2021.
[50] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprini arXiv:2302.01566 2023.
[51] Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: set of prerequisite toy tasks. In 4th International Conference on Learning Representations, ICLR 2016, 2016.
[52] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiemnon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109. 10862, 2021.
[53] Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3d environment. arXiv preprint arXiv: /0t0://20000099 2018.
[54] Jiannan Xiang, Zhengzhong Liu, Yucheng Zhou, Eric Xing, and Zhiting Hu. ASDOT: Any-shot data-to-text generation with pretrained language models. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 1886-1899, Abu Dhabi, United Arab Emirates. December 2022. Association for Computational Linguistics.
[55] Jiannan Xiang, Xin Wang, and William Yang Wang. Learning stop: A simple yet effective approach to urban vision- Saiiuaga navigation. In Findings of the Association for Computational Linguisties: EMNLP 2020, pages 699-707, 2020.
[56] Claudia Yan, Dipendra Misra, Andrew Bennnett, Aaron Walsman, Yonatan Bisk, and Yoav Artzi. Chalet: Coreell house agent learning environment. arXiv preprint 1rXiv:1801.07. 2018.
[57] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
[58] Rowan Zellers, Ari Holtzman, Matthew E Peters, Roozbeh Mottaghi, Aniruddha Kembhavi, Ali Farhadi, and Yejin Choi. PIGLeT: Language grounding through neuro- couro-yymbolic interaction in a 3d world. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference Natural Language Processing (Volume 1: Long Pupers), pages 2040-2050, 2021.
[59] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
[60] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1909v:1900.08800 2019.
14

---

Action
Template
[Find] <Object>
Find Object
[Walk] <Object>
Walk to Object
[Run] <Object>
Run to Object
[Sit] <0bject>
Sit Object
[StandUpp
Stand up
[Grab] <Object>
Grab Object
[0pen] <0bject>
Open Object
[Close] <Object>
Close Object
[Put] <Object_1>> <Object_2>
Put 0bject_1 on Object_2
[PutIn] <Object_1> <0bject_2>
Put 0jbect_ 1 in 0bject_2
[SwitchOn] <Object>
Switch/Turn on Object
[SwitchOff] <Object>
Switch/Turn off Ojbect
[Drink] <Object>
Drink Object
[TurnTo] <Object>
Turn Object
[LookAt] <Object>
Look at Object
[Wipe] <Object>
Wipe Object
[PutOn] <Object>
Put on Object
[PutOff] <Object>
Put off Object
[Greet] <Object>
Greet Object
[Drop] <ūbject>
Drop Object
[Touch] <Object>
Touch Object
[Lie] <Object>
Lie on Object
[Pour] <0bject_1> <0bject_2>
Pour Object_ into Object_2
[Type] <Object>
Type Object
[Watch] <Object>
Watch Object
[Move] <Object>
Move Object
[Wash] <Object>
Wash Object
[Rinse] <Object>
Rinse Object
[Scrub] <Object>
Scrub Object
[Squeeze] <0bject>
Squeeze Object
[PlugIn] <Object>
Plug in Object Plug out Object
[PlugOut] <Object>
[Cut] <0bject>
Cut Object
[Eat] <Object>
Eat Object
[Sleep]
Sleep
[WakeUp]
Wake up
Table 4: Supported actions in VirtualHome and their corresponding text templates.
A Appendix
A.1
VirtualHome
The complete format of an executable action step in VirtualHome is <char(char_id) [Action] <Object> (Object_iae bocct_idd__a Specifically, char id specifies which agent to execute the action when multiple agents are in the world model the same time. Action should be supported atomic action in VirtualHome. Object is the object with which the agent interacts. Each object in the environment is assigned Object_id distinguish from others of the same object class. We designed template for each action to transform them into natural text for LMs finetuing. The full list of executable actions can be found in Table Note that the list, we omit <char(char_ and (Object_id) for simplicity.
A.2 Acitivity Goal And Predicate
The goal an household activity VirtualHome consists of several predicates. Each predicate represents condition of one object relation between two objects. For example, OPEN (coffe
15

---

maker) means the coffee maker is open, and ON (apple, table) means an apple is on the table. The goal is only achieved when all the predicates are achieved, We collected activities and goals from RobotHow.
A.3 Data Format and Prompts
Following Chung et al. [8] use instructions with in-context exemplars prompts. Specifically, the instruction, the question context, and the answer will be provided each exemplar, and the full prompt will contain multiple such exemplars for in-context learning. The format of the data and the exemplar for each task provided below.
A.3.1
Plan Generation
Data Example
Key
Value
activity
watch TV living room, sofa, The sofa TV are the living room. Walk to living room. Sit on sofa. Watch TV.
condition plan
In-context Exemplar
Q: How to {{ activity }}? Given items include {{ condition }} A: {{ plan }}
A.3.2 Housework QA
Data Example
Key
Value
activity
watch TV
choices
[TV, coffee, bed, toothbrush] TV
answer
In-context Exemplar
Question: To {{ activity },, possibly related item could be {{ answer }}
Answer:
A.3.3 Negation Housework QA
Data Example
Key
Value
activity watch TV (TV, sofa, living room, toothbrush] toothbrush
choices
answer
In-context Exemplar
Question: To {{ activity }>, an unrelated item could be Answer: {{ answer }}
16

---

A.3.4
Activity Recognition
Data Example
Key
Value
plan choices activity
Walk living room. Sit on sofa. Watch TV. (watch TV, make coffee, sleep, brush teeth] watch TV
In-context Exemplar
Given task plan: { plan }} Question: what is the name of this task? Answer: {{ answer }}
A.3.5
Activity Inference
Data Example
Key
Value
state choices activity
Tom is sitting on the sofa. Tom is facing the TV. (watch TV, make coffee, sleep, brush teeth] watch TV
In-context Exemplar
{ state }}
Question: given the above state, possible activity could Answer: {{ answer }}
A.3.6 Counting
Data Example
Key
Value
movement
Tom at home. He grabbed apple and put it the bookshelf. He then walked to the kitchen srcub a plate. He went back to bookshelf and put the plate on it.
location
bookshelf
number
2
items
apple, plate
In-context Exemplar
Given sequence of actions house, and question about what items located in specific place. Answer the number items and items.
Q: {{ movement }} How many items are there the {( location }}? A: Ther {{ number }} itmes the {{ location }}. They {{ items }}
17

---

A.3.7 Counting QA
Data Example
Key
Value
movement
Tom at home. He grabbed apple and put it the bookshelf. He then walked to the kitchen srcub a plate. He went back to bookshelf and put the plate on it. bookshelf
location
number
2
In-context Exemplar
Q: {{ movement }} How many items there the {{ location }}?
number
A.3.8
Object Path Tracking
Data Example
Key
Value
movement
Tom went to the kitchen. Mary walked into the dining room. Tom grabbed plate. Tom travelled the living Mary moved Tom put the plate the table. Mary grabbed the plate. Mary journeyed to the bedroom. plate
the living
object
path
kitchen, living room, bedroom
In-context Exemplar
{ movement }}
Question: What the order the where the {{ object appeared? Answer: {{ path }}
A.3.9 Object Location QA
Data Example
Key
Value
movement
Tom went to the kitchen. Mary walked into the dining room. Tom grabbed plate. Tom travelled the living room. Mary moved the living room. Tom put the plate on the table. Mary grabbed the plate. Mary journeyed to the bedroom. plate
object
reference_ rrom
living
preposition
before
answer
kitchen
In-context Exemplar
{{ movement }}
Question:
Where is the {( object }) {( preposition }} the {{ reference_room }}? Answer: {{ answer }}
18

---

A.4 Hyperparameters
For both GPT-Neo- 1.3e and GPT-J-6B, use a learning rate 10-5 and a batch size of 20. The weights for plan generation, activity recognition, counting, and object path tracking are 1.0, 0.7. 1.0, and 1.0, respectively. We trained GPT-Neo-L.. 1.30 for epochs with the EWC coefficient À 0.5
in Equation 4 For GPT-J-6B, we trained it for 5 epochs with À 2. With our approach, it takes 40
minutes train a GPT-Neo and 220 minutes train a GPT-J. We used rank of and coefficient of 32 for LoRA's hyperparameters.
A.5
bAbl Dataset
We include 8 tasks from bAbl that test embodied knowledge. They are: One Supporting Fact, Two Supporting Fact, Three Supporting Fact, Counting, Lists/Sets, Simple Negation, Time Reasoning, Positional Reasoning. Examples for each task are shown in Table 5
Task I: Single Supporting Fact
Task 2: Two Supporting Facts
Mary the bathroom. John moved hallway. Mary travelled to office.
John in playground. John picked up the football. Bob went to the kitchen.
Where Mary? Acoffice
Where football? A:playground
Task 3:
Three Supporting Facts
Task 4: Counting
John picked up the apple.
Daniel picked up football.
John went to office. John went
Daniel dropped the football.
kitchen.
Daniel got the milk.
John dropped apple.
Daniel took the apple. A: office
Where was the apple before the kitchen? A:office
How many objects Daniel holding? A: two
Task 5: Lists/Sets
Task 6: Simple Negation
Daniel picks up the football.
Sandra travelled to the office. Fred is no longer in the office. Is Fred in the office? A:no Is Sandra in the office? A:yes
Daniel drops the newspaper.
Daniel picks up the milk.
What is Daniel holding? milk, football
Task Time Reasoning
Task Positional Reasoning
In afternoon Julie went to park. Yesterday Julie was school. Julie went
The triangle to right of blue square. The square on top of the blue square.
cinema this evening.
The sphere to the right of blue square. the red sphere the right of blue square? A:yes the red square the left the triangle? Acyes
Where did Julie go after the park? Ascinema Where was Julie before park? Asschool
Table 5: Examples for bAbl tasks.
A.6 Results of Main Experiments and Ablation Studies
Experimental results on our constructed downstream tasks are shown in Table 6 and the results on bAbl are shown in Table 7
We also show the results of ablation studies in Table
A.7 Human Evaluations
We conduct human evaluations plan generation for GPT-J model. Following Huang al. [8 asked 3 people to annotate whether each task can be completed using generated plan. We randomly sampled 150 tasks and asked each person annotate 50 of them. The Results show that the base GPT-J model can only achieve 24.0% accuracy, while the finetuned model can achieve 62.4%. The higher planning accuracy demonstrates the superior task planning ability of our model.
A.8 SuperGLUE Results
We evaluate the base GPT-J model and our finetuned model on appropriate SuperGLUE tasks, e.g. that can be formulated as a multi-choice QA task without prompt engineering. Our model's performance matches and even outperforms the baseline, showing our model retains the general language capability.
19

---

GPT Neo
GPT-J
OPT-13B
LLaMA-138 13B
CharGPT
Task
Metric
Base
Ours
Base
FT
Ours
Base
Ours
Base
Ours
[GPT3.5-turbu)
Generation
Vanilli Seen
Rouge
21.25
49.70
34.31
47.98
51.23
36.00
50.15
41.77
$2,05
40.57
Vindlla Unbeen
Rouge-1.
CCogfuring Seen
Rouge
16.86
46.88
34.81
47.59
48.94
31.92
49.87
40.33
51.00
40.41
CCoufanning Unseen
Rouge-1.
17.05
Housework QA
Accuracy
70.11
72.41
71.78
51.34
85,44
81.61
84.29
81.99
86.59
83.91
Negation Housework QA
Accuracy
38.27
41.98
35.19
33.33
39.51
43,21
40.21
43.21
30.25
87.65
Activity Recognition
Accuracy
69.22
85.43
87.98
71.41
88.52
89.07
91.44
90.53
91.80
95.05
Activity Inference
Accuracy
56.49
66.03
69.08
70.99
74.43
67.94
70.61
74.05
68.32
83.59
Counting
Accuracy
22.68
28.87
30.41
16.49
67.01
20.01
62.37
29.38
79.38
66.49
Object Path Tracking
1.CS
30.80
85.91
33.86
46.25
98.67
33.49
96.28
38.82
96.99
59.53
Object Location QA
Accuracy
22.50
33.50
30.00
22.50
34,50
37.00
33.00
28.50
79.00
67.50
Table 6: Experimental results on various downstream evaluation tasks. The best result among baselines and our method is shown in bold, and the best result among all the models is underlined.
GPT-Neo
GPT-J
Task
ChatGPT
Base
Ours
Base
Ours
Single Supporting Fact
51.86
56.29
65.16
68.98
96.27
Two Supporting Fact
33.43
30.82
40.48
26.08
47.33
Three Supporting Fact
7.85
13.49
22.46
30.41
16.82
Counting
34.04
48.84
41.39
69.08
93.96
Lists/Sets
14.80
51.76
34.74
84.99
76.84
Simple Negation
36.05
65.56
42.80
63.95
93.66
Time Reasoning
21.45
23.46
36.96
59.42
61.63
Positional Reasoning
50.51
53.64
49.70
53.23
58.38
Table 7: Experimental results on bAbl test sets.
GPT--eo -w/o Act Recog
Base
Ours
-wo Plan Gen
-wo Count
-w/o Obj PT
Gen
Vanilla Seen
21.25
49.70
14.48
49.38
49.85
50.06
Vanilla/ Unseen
17.64
49.27
14.28
48.96
$1.16
49.02
Confusing / Seen
16.86
46.88
13.63
46.37
48.30
49.14
-Confusing / Unseen
17.05
42.34
9.86
43.79
46,28
44.64
QA
70.11
72.41
73.18
71.26
74.71
70.11
Neg QA
38.27
41.98
32.72
35.80
36.42
38.89
Act Recog
69.22
85.43
85.97
48.63
85.25
84.34
Act Infer
56.49
66.03
58.40
64.89
62.21
Count
22.68
28.87
18.56
25.26
35.05
32.99
Obj PT
30,80
85.91
92.13
84.17
86,46
29.90
Obj QA
22.50
33.50
35.00
49.00
43.50
22.00
Perplexity
4.120"
4.193
4.171
4.151
4.162
4.164
Table 8: Ablation experimental results on training tasks.
Model
BoolQ
CB
RTE
AX-g
AX-b
COPA
GPT-J
Base
45.20
41.07
47.29
50.00
57.50
59.00
Ours
66.00
41.07
58.84
53.37
54.00
62.00
Table 9: Results on SuperGLUE subset.
20

---

A.9 Broader Impact
Like other generation systems, the language model trained by our approach is susceptible to producing unintended output when confronted with harmful input, such as unethical text or input intended for adversarial attacks. Therefore, we strongly advise against utilizing our approach outside of controlled research environments until these risks have been mitigated. It is important to note that thoughtless deployment of our method could potentially enable malicious exploitation of the underlying language models. Thus, precautions, such as implementing filtering mechanism, must be taken.
21


Reviews:
[{"review_id": "lp4YFZ3yw0", "reviewer": "Reviewer_cenq", "summary": "The paper proposes to enhance language models by finetuning them on “embodied experiences”, which are textual data generated by household activity simulator, which is referred to as “world models” in the paper. Through evaluations on several tasks (e.g., planning, object tracking), it is shown that finetuning on these “embodied experiences” leads to better performance compared to larger models (e.g., ChatGPT) that are not trained on these data. To minimize catastrophic forgetting and improve training efficiency, the paper also proposes to combine “elastic weight consolidation” and “low-rank adapters” to finetune the language models.", "questions": "An important baseline is ChatGPT, which is not trained on “embodied experiences”, but it doesn’t seem like the prompt for querying ChatGPT is provided. In contrast, the paper notes that they used few-shot prompting for the smaller LMs. If they are given the same prompt, would the performance differ?", "limitations": "The limitations are not adequately discussed. See the comments in the “weakness” section above.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The premise of the paper is interesting, novel, and promising — because language models are not trained on embodied data, they might be less robust to those scenarios concerning interactions with the environments (i.e., the “embodied settings”).\n- The idea of using household activity simulator as a “world model” is interesting and likely significant in the context of enhancing language models.\n- Moreover, the authors conduct thorough experiments that support the central claim.\n- The writing and presentation of the paper are also clear.", "weaknesses": "Despite the strengths of the paper, below are some concerns for the problem settings and evaluations:\n\n- While it is shown on GPT-Neo and GPT-J (which are relatively small language models in today’s standard) that the proposed approach improves their capabilities on results such as bAbI, as also indicated in the paper evaluations, larger models (i.e. ChatGPT) which are not trained on these embodied data also attain similar performance. Because the constructed tasks only require scene context in the text form, it is unclear whether similar delta will be seen on larger models. Or put this in other words: will this improvement diminish with larger scale training even without embodied experiences? The reviewer would like to note that due to practical reasons, it is understandable that experiments like these may not be done for larger LMs, but it is worth further discussions or experiments in the paper to support the claim.\n- Another weakness is that the training tasks need to be hand-curated for the “embodied experience”, which includes planning, activity recognition, counting, and object path tracking. This is unlike how autoregressive LMs are trained, which only requires one main self-supervised task of next token prediction. This brings to the question of how scalable the proposed approach is, as the broader “embodied experiences” include tasks of much higher diversity, in addition to including other modalities such as vision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and helpful suggestions. We would like to address your concerns as follows:\n\n**(1) Scalability to larger models**\n\nThanks for the suggestion. We apply our approach on two larger LMs: OPT-13B and LLaMA-13B, respectively. The results on the 11 tasks (as in Figure.3 in the paper) are shown below. We can see improvements consistent to the case on smaller LMs (Figure.3). That is, our method substantially outperforms the respective base LMs while retaining a low perplexity, demonstrating the effectiveness of our method when scaling up to larger models. \n\nIn addition, we’d like to note that our approach’s strong performance on **small** LMs (e.g., GPT-Neo is able to compete or even outperform ChatGPT) itself is of practical significance. And the smaller the LMs, the more significant it becomes for practical *cost-efficient* applications.\n\n|       Model      | Act Infer | Act Recog |   Count   | HouseQA   | NegQA     | ObjMoveQA | ObjMove   | PlanGen   | PlanGen Conf | PlanGen Unseen | PlanGen Conf Unseen | PPL     |\n|----------------|---------|---------|---------|-----------|-----------|-----------|-----------|-----------|--------------|----------------|---------------------|---------|\n|      OPT-13B     |   67.94   |   89.07   |   20.10   | 81.61     | **43.21** | **37.00** | 33.49     | 36.00     | 31.92        | 29.34          | 36.98               | 4.0768* |\n|  Ours (OPT-13B)  | **70.61** | **91.44** | **62.37** | **84.29** | 40.21     | 33.00     | **96.28** | **50.15** | **49.87**    | **45.11**      | **47.93**           | 4.3584  |\n| LLaMA-13B        | **74.05** | 90.53     | 29.38     | 81.99     | **43.21** | 28.50     | 38.82     | 41.77     | 40.33        | 38.78          | 41.73               | 3.0359* |\n| Ours (LLaMA-13B) | 68.32     | **91.80** | **79.38** | **86.59** | 30.25     | **79.00** | **96.99** | **52.05** | **51.00**    | **47.44**      | **50.49**           | 3.0690  |\n\n**(2) Hand-Curated Training Tasks**\n\nAs discussed in Introduction and Method, the diverse embodied skills are centered around two core abilities, i.e., planning and object tracking (e.g., Lines.46-50, 127-129). We collect embodied experiences based on the two abilities, and design diverse training tasks to comprehensively digest the embodied experiences. This is indeed akin to some latest LM pretraining work. For example, next-word-prediction (or sequential denoising) can be seen as training LMs to acquire the core ability of denoising; UL2 [1] shows that diversifying the training tasks (e.g., sequential denoising, span denoising, etc.) for the same core ability of denoising can substantially improve the performance, compared to using only one task (e.g., next-word-prediction). Our design of diverse training tasks follows a similar idea. To further scale up, we speculate that there could be a small set of core abilities (like planning and tracking) that facilitate the collection and design of training tasks. We’re excited to study more in the future.\n\n**(3) ChatGPT prompts**\n\nThe prompts we used for ChatGPT also include in-context few-shot exemplars, as well as instructions describing the task. We will include them in the revised version.\n\n\n[1] Tay et al. UL2: Unifying Language Learning Paradigms. 2023."}, {"review_id": "NNU2XU3Hsd", "reviewer": "Reviewer_DfLY", "summary": "This paper proposes a new method to improve the embodied planning capability of large language models (LLM) via adding both goal-oriented planning and random exploration data from a world model/simulator as well as the EWC-LORA regularizer that prevents catastrophic forgetting of pre-training tasks. The new regularizer EWC-LORA proposed by the authors is not only time and memory-efficient compared prior KL constraint-based regularization and also leads to negligible perplexity increase and better downstream task performance. The authors show superior performance of the method over base LLMs in constructed embodied planning and activity description tasks based on the VirtualHome simulator and also the bAbI a dataset for testing multiple types of knowledge and abilities including embodied knowledge, logic reasoning, linguistic knowledge, etc. Note that the method can only outperform much bigger LLMs such as ChatGPT in many of the scenarios.", "questions": "Please address the comments listed in the section above.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The authors presents a simple yet effective way to improve LLM's embodied planning capability without sacrificing the abilities during pre-training under a small/negligible compute/memory cost. Such idea is neat and of great practical importance. I think the method will be quite significant for the future directions of making LLMs better at embodied tasks.\n\n2. The authors have done extensive experiments in both customized and existing benchmarks/datasets to show that the proposed method can outperform base models without fine-tuning with embodied data. It is impressive that the authors show that the small models with embodied planning data fine-tuning can outperform large LLMs such as ChatGPT in many scenarios.\n\n3. The authors also performed detailed ablation studies to show the importance of the proposed EWC-LORA regularizer via comparing to KL divergence and EWC only. The ablation studies on including various data mixture also make the paper more complete.", "weaknesses": "1. I think the authors should compare to stronger baselines that also consider improving language model's embodied decision-making and reasoning capabilities such as [1,2]. Improvement over base models is good but not that surprising and convincing.\n\n2. The authors should consider other embodied decision-making benchmarks such as ALFWorld, which is used in [1. 2]. This would provide a clearer picture of the comparison between the method and prior approaches. It would also add more tasks in the empirical evaluation, which can further validate the method.\n\n[1] Shinn, Noah, Beck Labash, and Ashwin Gopinath. \"Reflexion: an autonomous agent with dynamic memory and self-reflection.\" arXiv preprint arXiv:2303.11366 (2023).\n\n[2] Yao, Shunyu, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. \"React: Synergizing reasoning and acting in language models.\" arXiv preprint arXiv:2210.03629 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback and suggestions! We would like to address your concerns in the following paragraphs:\n\n**(1) Comparison with Reflexion and React**\n\nWe want to clarify that our approach has a different goal and is solving a different problem compared with the mentioned work, and our approach can be combined with these methods. Specifically, methods like Reflexion and React aim to utilize the existing embodied knowledge of LMs (which might be insufficient since they have no embodied experiences) to improve their performance on specific tasks such as navigation in MineCraft, thus they combine off-the-shelf LMs with different components like memory bank, prompting engineering, environment feedback collection, etc. On the contrary, our goal is to enhance **LM itself** by acquiring new embodied knowledge. Our final outcome is a language model with richer embodied knowledge which can still be integrated with existing methods like Reflexion and React. We aim to explore the combination in the future.\n\nIn addition, we think our results of enhancing LMs as small as GPT-Neo (1.3 B) and GPT-J (6B) to compete or even outperform latest large chatGPT are significant and surprising.\n\n\n**(2) Other embodied decision-making benchmarks**\n\nThe goal of our work is to inject fundamental embodied knowledge into LMs, which are not specific to particular embodied environments but are general and needed in common problems like QA and dialogue. Therefore, our evaluation is designed to assess the knowledge in those general settings (such as QA). Please refer to the general response for more details.\n"}, {"review_id": "MKq5Rk5Owv", "reviewer": "Reviewer_ekFP", "summary": "This paper proposes to incorporate world models into large language models to enable understanding of object permanence and planning capabilities that are missing in text-only models. Specifically, the authors utilize embodied environments (VirtualHome) to collect training examples including goal-oriented planning (plan generation, activity recognition) and random exploration (counting and object plan tracking). To avoid overfitting, this paper finetunes GPT-Neo-1 3B and GPT-J-6B models using LoRA and EWC. Evaluated on seen and unseen tasks, results show that the proposed method after finetuning outperforms the corresponding text-only baselines, and match or outperform ChatGPT performance under few-shot setting. ", "questions": "1. Have you done other evaluation on the base language model (i.e., NLP benchmarks such as superglue) apart from perplexity? \n2. Why do you think EWC only outperform LoRA (in table 2)? As a regularization method, EWC constraint parameter updates and thus should understand LoRA by intuition.\n3. Do you have a naive fine-tuning baseline to compare to?\n4. Do you have detailed analysis on when and why the model improves performance on both seen and unseen tasks?", "limitations": "The paper briefly mentioned limitations.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper is motivated by an interesting limitation in text-only language models, which is the lack of embodied knowledge in pre-training. The proposed fine-tuning dataset, along with EWC-LoRA method, improve the model performance on challenging embodied reasoning and understanding tasks, without dramatically increasing the original language model perplexity.\n2. The paper includes some interesting ablation studies to discuss how much improvement is observed from each fine-tuning task.", "weaknesses": "1. The contribution and findings of the paper is limited. Although the limitation of the LLMs is well motivated in the paper, fine-tuning on similar distribution to improve model performance and thus outperform baselines is expected. Furthermore, fine-tuning with LoRA to improve efficiency, as well as EWC to reduce overfitting have been well studied. Given that the (relatively weak) baseline is the original text-only model (and a few-shot prompted ChatGPT model), it is not convincing that the proposed method (either finetuning data or method) is generalizable to broader embodied tasks (e.g., compared to an embodied or multi-modal model).\n2. Some important details are missing and the evaluation is not very convincing. The paper is phrased to incorporate world model into LLMs, but it is confusing (without explicit explanation) that both training and evaluation are not with the embodied environment. Rather, only partial of the data is collected using VirtualHome. More importantly, it is not clear how the data are constructed (for example, how to sample \"irrelevant actions\" for counting QA and statistics of the training and eval sets) and what the held out set is for training the model. For evaluation, I understand the automatic eval is the most convenient metic, but have been widely adapted, metrics such as Rough-L for tasks like plan generation may not reflect model performance on complex text generation tasks. See more details in the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(1) Limited Contribution of Finetuning**\n\nWe clarify that our key technical contribution is not “finetuning”, but rather identifying and formulating the limitations of current LMs (lacking embodied knowledge), developing novel ways to automatically collect embodied experiences of desired distributions, and designing diverse finetuning tasks. To best of knowledge, no previous work has done similar studies.\n\n**(2) Finetuning on similar distribution**\n\nWe clarify that we finetune and evaluate not only on the similar distribution. To test the generality of our model, we develop and collect various *unseen out-of-distribution* tasks (e.g., time reasoning in bAbI and HouseworkQA). Moreover, we believe the results that our small models (GPT-Neo/-J) compete and even outperform the much larger ChatGPT are indeed surprising, providing the first evidence of using world models to improve LMs. \n\n**(3) Novelty of LoRA and EWC**\n\nAlthough the specific finetuning method is not the focus of this work, we want to emphasize that our EWC-LoRA method does provide new insights.\n* **We’re the first to show that EWC for LLM finetuning is better than the previous more popular method of KL regularizor [4-6]**.\n* The recent EWC-based LLM finetuning work has only focused on in-domain training tasks. That is, they show EWC could help LLMs to remember previously trained tasks. In contrast, **we additionally show that EWC even helps with out-of-domain unseen tasks**. That is, we show the LMs finetuned with EWC obtains better performance on unseen tasks compared to finetuning without EWC (Fig.3 and Fig.4). This is because EWC effectively preverses the LM generality.\n* For LoRA, previous works usually use LoRA to improve finetuning efficiency, but **we are the first to show that combining LoRA with EWC can further prevent overfitting and improve generality** (Table 2).\n\n**(4) Comparison with embodied or multimodal model**\n\nWe want to clarify that our approach has a different goal with those models, thus is not comparable to them. Please refer to the general response.\n\n\n**(5) Incorporating world model into LLMs**\n\nAs above, we are not “incorporating world models into language models”, but *using world models to improve language models*. Therefore, what we finally get is still a general-purpose language model. In the training, all the used embodied experiences are collected from VirtualHome. In the evaluation, we evaluate the model on common general tasks (e.g., QA) to see if it acquires new knowledge and can generalize it to unseen tasks. \n\n**(6) Details of Dataset Construction**\n\nWe have already provided the statistics of evaluation set in Section 4.1. For the training, the size of Plan Generation/Activity Recognition/Counting/Object Path Tracking is 1659/1659/1000/1000, respectively, and the held-out validation set is a Plan Generation subset of size 200. We will include all details in the revised version. For the counting QA task, as we demonstrate in line 151, all the actions (including irrelevant actions) are sampled randomly from the action space of VirtualHome.\n\n**(7) RougeL not reflecting performance**\n\nWe additionally conduct human evaluations on plan generation. We follow [3] to ask 3 people to annotate whether each task can be completed using a generated plan. We randomly sample 150 tasks and ask each person to annotate 50 of them. Results are below. The higher planning accuracy demonstrates the superior task planning ability of our model.\n\n|Model|Accuracy|\n|-|-|\n|GPT-J|24.0|\n|Ours (GPT-J)|**62.4**|\n\n**(8) SuperGLUE besides perplexity**\n\nWe evaluate the base GPT-J-6B and our model on appropriate SuperGLUE tasks (that can be formulated as a multi-choice QA task without prompt engineering).\n\n|Model|BoolQ|CB|RTE|AX-g|AX-b|COPA|\n|-|-|-|-|-|-|-|\n|GPT-J|45.20|**41.07**|47.29|50.00|**57.50**|59.00|\n|Ours|**66.00**|**41.07**|**58.84**|**53.37**|54.00|**62.00**|\n\nOur model’s performance matches and even outperforms the baseline, showing our model retains the general language capability.\n\n\n**(9) Why EWC outperforms LoRA**\n\nIf we understand correctly, your question was: EWC adds constraints to the parameter updating, then why does EWC outperform LoRA in Table 2? (Please correct us if our understanding is not correct.) LoRA freezes the base language model and only updates a small number of parameters in adapters. We hypothesize that this can also be seen as adding constraints to the parameter update, resulting in a slightly lower performance than EWC. \n\n**(10) Naive finetuning baseline**\n\nWe finetune GPT-J (GPTJ-FT) and compare with our method (GPTJ-E2WM). Our method outperforms the baseline significantly.    \n\n|Model|Act-Infer|Act-Recog|Count|HouseQA|NegQA|ObjMove-QA|ObjMove|PlanGen|PlanGen-Conf|PlanGen-Unseen|PlanGen-Conf-Unseen|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|GPTJ-FT|70.99|71.41|16.49|51.34|33.33|22.50|46.25|47.98|47.59|47.86|44.43|\n|GPTJ-E2WM|**74.43**|**88.52**|**67.01**|**85.44**|**39.51**|**34.50**|**98.67**|**51.23**|**48.94**|**49.58**|**45.60**|\n\n**(11) Improvements on both seen and unseen task**\n\nWhen using EWC, the LM learns new knowledge from finetuning while preserving its generality. Thus, on new unseen tasks requiring the same knowledge as in the finetuning tasks, the model can utilize acquired knowledge. We’re happy to do more analysis if you have more specific questions.\n\n[1] Mu et al. EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought. 2023.\n\n[2] Wang et al. Voyager: An Open-Ended Embodied Agent with Large Language Models. 2023.\n\n[3] Huang et al. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. ICML 2023.\n\n[4] Lu et al. Quark: Controllable Text Generation with Reinforced Unlearning. NeurIPS 2022.\n\n[5] Ouyang et al. Training language models to follow instructions with human feedback. 2022.\n\n[6] Liu et al. Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering. EMNLP 2022."}, {"review_id": "UtEBIPRJMU", "reviewer": "Reviewer_7bDY", "summary": "The paper introduces a training paradigm termed \"finetuning with Embodied Experiences from World Models (E2WM)\" to enhance the abilities of language models (LMs) in reasoning and planning tasks associated with physical environments. The authors argue that LMs trained solely on large-scale text corpora lack the embodied knowledge necessary for robust performance in such tasks. To address this, they propose leveraging world models, specifically the VirtualHome simulator, to collect diverse embodied experiences and use them to construct fine-tuning tasks, such as plan generation, activity recognition, counting, and object path tracking. The proposed fine-tuning method incorporates Elastic Weight Consolidation (EWC) regularization with low-rank adaptation (LoRA) to preserve the models' generality and avoid catastrophic forgetting and make the training process more efficient.", "questions": "Have the authors considered an evaluation on actual embodied benchmarks? \nWhat is the motivation for choosing Babi as the benchmark dataset? \n", "limitations": "There is a very brief discussion on limitations as part of the conclusion section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces an approach to enhance off-the-shelf LLMs with embodied knowledge. The experiments involve both goal-oriented planning and random exploration. in collecting embodied experiences. While EWC and LoRA are existing methods (hence the paper may be perceived as with limited technical novelty), this work shows that their incorporation into the fine-tuning process helps retain the models' general knowledge and capabilities while adapting them to new tasks. \n- The related work seems sufficient, albeit a section on catastrophic forgetting, e.g., [1] would be a plus.\n- The paper provides an evaluation of the finetuned LMs on both seen and unseen tasks, and several ablation studies. The generalizability of the models is also tested, as well as their performance on the original pretraining data to ensure core language modeling abilities are retained. \n\n[1] Korbak, Tomasz, Hady Elsahar, German Kruszewski, and Marc Dymetman. \"Controlling conditional language models without catastrophic forgetting.\" In International Conference on Machine Learning, pp. 11499-11528. PMLR, 2022.", "weaknesses": "- Lack of comparison: Only two GPT-based LLMs are used. Would be valuable to consider a more diverse set of recent LLMs. The paper also does not compare the proposed E2WM paradigm with existing methods for enhancing LMs with embodied knowledge, such as [1].\n- Scalability to larger models: Although the EWC-LoRA approach is designed to improve efficiency, it is not clear how well it scales to larger LM architectures. Further investigation into the scalability of the proposed method would be beneficial.\n- Limitations on benchmarks: Evaluation is performed on bAbI but there are several embodied benchmarks and simulators that could be considered. Doing so would allow evaluation in realistic embodied settings, and compare other embodied models proposed (such as the Episodic Transformer [2] or [1]).\n- Experimental Results: The number of tasks and models evaluated is relatively small and the presentation of the results could be clearer, e.g., adding numbers to bars in Figs. 3 and4. It seems that ChatGPT outperforms the fine-tuned models, so overall the model capacity appears to be an important factor, and the proposed fine-tuning shows marginal improvements in some of the tasks. \n\n[1] Lin, B.Y., Huang, C., Liu, Q., Gu, W., Sommerer, S. and Ren, X., 2023, June. On grounded planning for embodied tasks with language models. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 11, pp. 13192-13200).\n[2] Pashevich, Alexander, Cordelia Schmid, and Chen Sun. \"Episodic transformer for vision-and-language navigation.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15942-15952. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the helpful suggestions. The responses to your comments are shown below.\n\n**(1) More diverse LMs; Scalability to larger LMs**\n\nThanks for the suggestion. We apply our approach on two larger LMs: OPT-13B and LLaMA-13B, respectively. The results on the 11 tasks (as in Figure.3 in the paper) are shown below. We can see improvements consistent to the case on smaller LMs (Figure.3). That is, our method substantially outperforms the respective base LMs while retaining a low perplexity, demonstrating the effectiveness of our method when scaling up to larger models. \n\nIn addition, we’d like to note that our approach’s strong performance on **small** LMs (e.g., GPT-Neo is able to compete or even outperform ChatGPT) itself is of practical significance. And the smaller the LMs, the more significant it becomes for practical cost-efficient applications.\n\n|       Model      | Act Infer | Act Recog |   Count   | HouseQA   | NegQA     | ObjMoveQA | ObjMove   | PlanGen   | PlanGen Conf | PlanGen Unseen | PlanGen Conf Unseen | PPL     |\n|----------------|---------|---------|---------|-----------|-----------|-----------|-----------|-----------|--------------|----------------|---------------------|---------|\n|      OPT-13B     |   67.94   |   89.07   |   20.10   | 81.61     | **43.21** | **37.00** | 33.49     | 36.00     | 31.92        | 29.34          | 36.98               | 4.0768* |\n|  Ours (OPT-13B)  | **70.61** | **91.44** | **62.37** | **84.29** | 40.21     | 33.00     | **96.28** | **50.15** | **49.87**    | **45.11**      | **47.93**           | 4.3584  |\n| LLaMA-13B        | **74.05** | 90.53     | 29.38     | 81.99     | **43.21** | 28.50     | 38.82     | 41.77     | 40.33        | 38.78          | 41.73               | 3.0359* |\n| Ours (LLaMA-13B) | 68.32     | **91.80** | **79.38** | **86.59** | 30.25     | **79.00** | **96.99** | **52.05** | **51.00**    | **47.44**      | **50.49**           | 3.0690  |\n\n\n**(2) Comparison with prior LM work for embodied tasks (e.g., Lin et al.)**\n\nWe want to point out the method in Lin et al. has a different goal with our work at all and thus is not comparable to our method. They aim to utilize LMs to enhance the performance on specific tasks in specific environments, thus they finetune LM to read the symbolic state and generate executable plans. However, after finetuning the model becomes a task-specialized model and loses its generality to solve various seen or unseen tasks. On the contrary, our goal is to enhance the LM itself, so the outcome of our method is still a general LM that can solve various tasks and can generalize newly acquired embodied knowledge to unseen tasks. Please refer to the general response for a more detailed explanation.\n\n**(3) Limitations on benchmarks**\n\nAs we mentioned above, Lin et al. and Pashevich et al. are not comparable to our work. Evaluation in embodied environments not only evaluates the general embodied knowledge, but also tests how well the knowledge is utilized by special modules on top of the language model and how well the model is adapted to the specific environments. A strong LLM that possesses rich embodied knowledge can still fail in a specific environment since the generated action is not executable in the environment (e.g., LLM generates “Next, you should go to the kitchen” while the executable action is “<Walk> [kitchen]”)  Therefore, previous works seldom evaluate an off-the-shelf LLM in an embodied environment. They either finetune it to get a task-specialized model (e.g., Lin et al. and Pashevich et al.) or build special modules on top of the LLM [1][2]. On the contrary, we aim to evaluate the LM as a general-purpose model that can generalize new acquired knowledge to common general tasks (like QA). Please refer to the general response for more detailed reply.\n\n**(4) Experimental Numbers**\n\nTo make the figure clearer, we put all the numbers in Table 6. Please refer to the appendix for them.\n\n**(4) Comparison with ChatGPT**\n\nWe think our results of enhancing LMs as small as GPT-Neo (1.3B) and GPT-J (6B) to compete or even outperform the latest large ChatGPT model is surprising and important, and of practical significance. The smaller the LMs, the more significant it becomes for practical cost-efficient applications. Besides, the experiments in (1) demonstrate that our method can be scaled up to larger LMs with better performance, which further highlights the model-agnostic advantage of our method.\n\n[1] Mu et al. EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought. 2023.\n\n[2] Wang et al. Voyager: An Open-Ended Embodied Agent with Large Language Models. 2023."}, {"review_id": "gPhqMsivlw", "reviewer": "Reviewer_311c", "summary": "This paper proposes a new collection of data and tasks from an embodied environment to enhance the embodied reasoning ability of pre-trained language models. The paper also designs a fine-tuning strategy that combines the advantages of EWC and LoRA for stable fine-tuning. The experiments show that the proposed method can efficiently fine-tune GPT-J and outperform much larger language models.", "questions": "1. Related to weakness: It will be good to see whether there are any other benefits gained from EWC.\n\n2. How do you ensure that the answers of the negation Housework QA are truly irrelevant?\n\n3. The activity recognition performance of all methods is quite similar. Does this indicate that the activity types are easily distinguishable without requiring a full description of the experience, for example, just keywords at the end are sufficient for the task? Have you tried to include confusing activities in the evaluation set?\n\n4. How do you introduce the confusion term (counting task, confusing unseen of plan generation)? Do you have any guideline or do you just randomly inject phrases?", "limitations": "The author mentioned the limitations of only collecting embodied experience from one world model.\n\nsocietal impact: did not find negative impacts, probably if the data collection is extended to the real world in the future, it should avoid collecting personal or sensitive experiences.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper designs two ways of data collection in the embodied environment that address goal-specific and general cases in real-world applications.\n\nThis paper designs a comprehensive reasoning task set based on the collected data to evaluate the reasoning performance from different aspects (plan generation, object tracking, etc.)\n\nThis paper adopts a suitable parameter updating strategy (EWC_LoRA) for language model fine-tuning, which demonstrates time and memory efficiency.", "weaknesses": "The paper lacks experiments on why EWC is needed. The motivation for including EWC is to avoid overfitting downstream tasks or catastrophic forgetting on the pretraining task, but LoRA has a similar purpose. From the experiments in Table 2, EWC_LoRA has a lower performance than LoRA except for a slight improvement in perplexity. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback. We would like to respond to your questions as below.\n\n**(1) Why EWC is needed**\n\nWe want to clarify that EWC-LoRA has similar performance with LoRA. From Table 2, we can see EWC-LoRA matches or even outperforms its LoRA counterpart for GPT-J (e.g., 3.09 improvements on Counting QA, 1.45 on Object Path Tracking) and GPT-Neo (e.g., 1.03 on Count), while EWC-LoRA has a lower perplexity. Both methods outperform the base language model and have close performances on the specific tasks, in this case we focus more on the perplexity since we want its language modeling capability and generality to be preserved as much as possible.\n\n**(2) Irrelevant items in Negation QA**\n\nWe use simple heuristics to sample irrelevant items, e.g., we sample items in the environment that are not directly mentioned in the ground truth plan to form the questions. \n\n**(3) Performance on Activity Recognition**\n\nFor now, we just randomly sample different activities. The performances on this task are relatively higher than other tasks, but we can still observe significant performance gap of different models (69.22 by base GPT-Neo, compared to 87.98 by base GPT-J; see Table 6 in the appendix), and our method outperforms the base language model for both GPT-Neo (69.22 → 85.43) and GPT-J (87.98 → 88.52).\n\n**(4) Confusing terms**\n\nFor the counting task, we construct samples from collected Random Exploration experiences, where an agent randomly executes actions in the environment. As a result, some of the actions are naturally irrelevant to object counting. For confusing unseen plan generation, we adopted the similar heuristics in (2).\n"}]

Author rebuttal:
We thank all the reviewers for their insightful and encouraging comments. We are encouraged by the reviewer’s appreciation that the motivation and idea of the paper are novel, interesting and promising (Reviewers cenq, DfLY, ekFP); the proposed method is neat, effective, and of great practical importance  (cenq,DfLY, 311c), and the experiments and ablation studies are thorough and detailed (cenq, DfLY, 311c), showing strong improvements (cenq, DfLY, ekFP, 331c).

We’d like to highlight the unique focuses of our work that differ from the previous work of  embodied LMs: 
* **Our goal** is to inject *fundamental* embodied knowledge and skills into LMs, such as object permanence, action planning, spatial knowledge, etc. Those fundamental embodied knowledge and skills are **not** specific to particular embodied environments (e.g., VirtualHome, ALFWorld). Instead, they are general and needed in common problems such as various question answering (QA) and dialogue. Accordingly, **our approach** aims to finetune LMs while keeping them as general-purpose models, capable of handling the common general problems (e.g., QA) and generalizing acquired knowledge to unseen tasks. Similarly, **our evaluation** assesses the knowledge/skills in those common general settings, including the novel 11 constructed tasks (mostly in QA forms) and the well-known bAbI tasks designed for assessing the fundamental knowledge/skills of models. Those evaluations are independent of specific embodied environments.
* In contrast, **prior work** on LMs for embodied tasks aims to apply the LMs to handle **specific** embodied environments, through either finetuning or prompting. Such work includes (mentioned by the reviewers) [1, 2] that finetune LMs to specific embodied environments, [3,4] that keep LMs frozen but design specialized prompts for respective environments, as well as those already discussed in our Related Work. Accordingly, their evaluation is to deploy the specialized LMs to complete specific tasks in the respective embodied environments.
* In addition, in prior work, to deploy LMs to the specific embodied environments, there are additional components needed, such as mapping the LM-generated free-form text (e.g., “*Next, you should go to the kitchen*”) into the action space (“<Walk> [kitchen]”) of the specific environment. Our work does not involve those components as we focus on the fundamental knowledge/skills and general common settings such as QA.

[1] Lin et al. On grounded planning for embodied tasks with language models. AAAI 2023.

[2] Pashevich et al. Episodic transformer for vision-and-language navigation. ICCV 2021.

[3] Shinn et al. Reflexion: an autonomous agent with dynamic memory and self-reflection. 2023.

[4] Yao et al. React: Synergizing reasoning and acting in language models. 2022.

Meta-review: The authors propose an approach to finetuning LLMs with embodied experience (textual data from a simulator). They do so via EWC and LoRA for efficient training and less catastrophic forgetting. They demonstrate that the approach outperforms larger, untrained models over a variety of physical tasks in VirtualHome.

The reviewers found the paper simple yet effective and show promising experimental results, as well as detailed ablations. The approach is also timely in showing how to improve LLMs for embodied tasks, an area they have performed relatively poorly. During the rebuttal they have also included a larger model (13B). Most reviewers indicated that the paper should be accepted, and though the negative reviewer, ekFP, has some reasonable points that should be addressed, I believe the core issues are well addressed in that the evaluated tasks are out of distribution.

Overall, I think the paper is interesting and timely, and I thus recommend acceptance.

The authors should consider two additions for a final version as mentioned by the reviewers, along with the changes included in the rebuttal. (1) Adding ablations over LLM sizes (e.g., LLaMA 7B, 13B, 33B, and 65B) could establish a scaling curve and provide a significant answer to such questions. (2) More experiences or demonstrations in a second, varied environment (perhaps a manipulation environment, but generally additional embodied environments) would strongly strength the work.
Decision: Accept (poster)