SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Plan Generation Evaluation. In this task, the model needs to generate a plan for a housework activity. It is similar to the training task but uses human-written plans as the ground truth instead of the collected experiences. We include activities unseen during training to test the generalizability of the model. Inspired by the previous study showing that LMs can easily be distracted by irrelevant context [43], we also created samples having states of unrelated objects in the context (e.g., TV is on for activity Make Coffee) to confuse LMs. In summary, this results in four settings: Vanilla Seen, Vanilla Unseen, Confusing Seen, and Confusing Unseen. We have 175/54/135/43 examples for four settings, respectively. We use Rouge-L [26] as the metric.
- Housework QA. This is a multi-choice QA task, which is unseen during training. It asks which choice is the relevant object to finish a household activity, e.g., which object is relevant to making coffee? It has 261 examples in total, and we use accuracy as the metric. When evaluating, we provide 10 in-context exemplars in the prompts, so this task is evaluated as a 10-shot learning task.
- Negation Housework QA. This is similar to Housework QA but inquires about the irrelevant object, e.g., which object is irrelevant to making coffee? It is more challenging than the vanilla QA because LMs that simply memorize the word co-occurrence in the training data may succeed in the vanilla QA but will fail in the negation QA. This task has 162 examples and uses accuracy as the metric. We provide 10 in-context exemplars in the prompts.
Activity Recognition. We developed two multi-choice QA tasks with the human-written plans and the state changes from RobotHow to test the knowledge gained from activity recognition:
- Activity Recognition QA. In this task, a human-written plan is given and the model needs to choose the correct activity name. An example of the question is "Given a plan: Walk to living room. Sit on sofa. Turn on TV. What is the name of this activity?" And the answer should be Watch TV. The task has 549 examples. We use accuracy as the metric.
- Activity Inference QA. In this task, we use the final state of the world model as input for the model to infer the activity name. For example, the input can be "Tom is sitting on the sofa and facing the TV. The TV is on. What is a possible activity he is doing?", and the answer is "Watch TV". We have 262 examples for this task and use accuracy to measure the performance. We provide 10 in-context exemplars in the prompts.
Counting. We gathered random exploration experiences to construct Counting QA for evaluating skills learned from the counting task. The model is required to answer the number of objects in a specific location. For example, a query can be "Tom put an apple on the table. Tom turned on the TV. Tom put a cup on the table. How many objects are there on the table?" We can see that there will be irrelevant actions like turn on TV to confuse the model and make the question more challenging. We collected 194 samples for the task and used accuracy as the evaluation metric. We provide 5 in-context exemplars in the input prompts.
Object Path Tracking. We developed two downstream tasks for the object path tracking training task, namely Object Path Tracking Evaluation and Object Location QA.
- Object Path Tacking Evaluation. This evaluation task is the same as the training task, where the model is required to generate the full moving path of an object. An example is "Tom walked to the kitchen. Tom grabbed the apple. Mary walked to the bedroom. Tom walked to the living room. What is the order of the rooms where the apple appears?" This question typically includes multiple agents and many irrelevant actions, which makes it difficult to track the object. This task contains 200 examples. Following Huang et al. [18], we evaluate the performance by calculating the length of the longest common subsequence (LCS) between the ground truth and the generated path, normalized by the maximum length of the two.
- Object Location QA. In this task, the model is asked about the location of an object before/after it moves to another location, e.g., where is the apple before/after the kitchen? This task has 200 examples with accuracy as the metric. We provide 2 in-context exemplars in the prompts.
A previous study on prompting multiple QA questions [39] introduces two prompting methods, multiple choice prompt and cloze prompt, and two normalization methods, length, and unconditioned normalization. For all the multi-choice QA tasks, we choose the combination of prompting and normalization methods which yields the best performance on a held out set.
To further verify the effectiveness of our method, we evaluate our finetuned GPT-Neo and GPT-J on the bAbI dataset. Specifically, we select 8 test sets from bAbI that align with the abilities covered in our collected embodied experiences. We include the description of each test set in Appendix A.5. For all the bAbI tasks, we do 2-shot learning by providing 2 in-context exemplars in the input prompts.
Besides downstream tasks, we also want to ensure that our approach does not hurt language modeling performance of the models. Therefore, following previous work [42], we evaluate the perplexity on a subset of Pile [12] test set, which is the pretraining dataset for GPT-Neo and GPT-J. We sampled 5000 examples from Pile test set for evaluation.
Constructed Evaluation Tasks. Results for all the downstream evaluation tasks are shown in Figure 3 and Figure 4. For all the models, we compare the results obtained after finetuning with world model against those of the original base models. For GPT-J, we also include a finetuned model without EWC-LoRA as a baseline. Detailed numbers of the results can be found in Appendix A.6. We also conduct human evaluations for GPT-J on the plan generation task, which can be found in Appendix A.7. In general, the models trained with the world model significantly outperform the baselines on various downstream tasks. Our method is not only effective for small 1.3B model, but can also scale to larger 6B and 13B models. Specifically, our finetuned GPT-J and LLaMA-13B
Figure 3: Experimental results of GPT-Neo and GPT-J on our constructed downstream tasks. GPT-J (FT) refers to the finetuned GPT-J without EWC-LoRA. Our approach surpasses baselines on all of the 11 tasks, and outperforms ChatGPT on 7 of them. For example, our GPT-J model achieves 98.67 LCS on object path tracking, which is significantly better than 33.86 of base GPT-J and 59.53 of ChatGPT.
Figure 4: Experimental results of OPT-13B and LLaMA-13B on our constructed downstream tasks. Our approach applied on LLaMA-13B outperforms ChatGPT on 8 of them.
Figure 5: Experimental results on bAbI. Our approach outperforms base LMs on all the tasks except for the Two Supporting Fact task.
GPT-Neo	GPT-J	OPT-13B	LLaMA-13B
Base	Ours	Base	Ours	Base	Ours	Base	Ours
4.120	4.193	3.443	3.537	4.077	4.358	3.036	3.069
Table 1: Perplexity on Pile test subset, showing the proposed finetuning with world model manages to preserve the LMs' language modeling capability.
Task	GPT-Neo	GPT-J
Base	EWC	LoRA	LoRA & KL	EWC-LoRA	Base	LoRA	EWC-LoRA
Plan Gen	21.25	48.56	51.24	45.99	49.70	34.31	51.23	51.23
Act Recog	69.22	89.98	87.98	81.42	85.43	87.98	90.16	88.52
Count	22.68	55.67	27.84	49.48	28.87	30.41	63.92	67.01
Obj PT	30.80	95.96	87.28	63.59	85.91	33.86	97.22	98.67
Perplexity	4.120*	4.995	4.360	5.029	4.193	3.443*	3.675	3.537
Table 2: Results of different regularization methods. The abbreviations in the Task column stand for the corresponding evaluation tasks for four training tasks. We use asterisk* to mark the perplexity of base models.
with world model even achieve better performance than ChatGPT as a much larger LM on most of the 11 tasks. Besides, we can see the world model improves LMs on both seen and unseen tasks. This demonstrates that our model absorbs the knowledge for goal-oriented planning and random exploration instead of memorizing the seen experiences. Specifically, the better plan generation performance under the "Confusing" setting indicates that the world model improves the ability of LMs to avoid being interfered with by irrelevant contexts. On both Housework QA and Negation Housework QA, our models surpass the baselines, showing that our models also acquire knowledge about the necessary objects for completing a housework activity. Results on other downstream tasks also prove the effectiveness of our method. For example, on both Activity Recognition Evaluation and Activity Inference, our approach improves over the baselines significantly. Moreover, improvements can be observed in the downstream tasks regarding random exploration. On Counting and Object Location QA tasks, our LLaMA-13B trained with the world model even surpasses ChatGPT.
bAbI Tasks. To further verify the effectiveness of our method, we evaluate our finetuned models on the bAbI dataset. The results are shown in Figure 5. We can see that our approach significantly outperforms the base LMs. Notably, after finetuned with VirtualHome experiences, GPT-J surpasses the much stronger ChatGPT on the most challenging tasks. Specifically, it outperforms ChatGPT on the Three Supporting Fact task, where the model is required to use three supporting facts from the context to answer a question like "where was the apple before the kitchen?", and Lists/Sets task, which asks the model to give the answers in the form of a list, e.g., the answer for "What is Daniel holding?" is "apple, milk". These results prove that our approach enables LMs to acquire the knowledge and skills inherent in embodied experiences, rather than simply overfitting to the training environment.
Language Modeling. In addition to verifying improved performance on the downstream tasks, we also report results on Pile test subset to ensure the preservation of the generality and language modeling abilities of LMs. From the experimental results shown in Table 1, we can see that our approach only causes a negligible increase in perplexity over the base models. This demonstrates the effectiveness of EWC-LoRA to preserve the generality and linguistic competence of LMs. To verify the generality on other NLP tasks, we also include the results on SuperGLUE [40] in Appendix A.8.
4.3 Comparison of Different Regularization Methods
We compare our proposed EWC-LoRA with EWC and LoRA. Besides, we also include the baseline using KL penalty as regularization. The experimental results are shown in Table 2. We also include the results of four evaluation tasks. Notice that we do not include the results of GPT-J with pure EWC and KL, since they are overly memory-intensive or time-consuming. EWC requires an original model and a fisher matrix other than the trained model, which triples the memory usage, making
Base	Ours	- w/o Plan Gen	GPT-Neo -w/o Act Recog	- w/o Count	- w/o Obj PT
Plan Gen	21.25	49.70	14.48	49.38	49.85	50.06
Act Recog	69.22	85.43	85.97	48.63	85.25	84.34
Count	22.68	28.87	18.56	25.26	35.05	32.99
Obj PT	30.80	85.91	92.13	84.17	86.46	29.90
Perplexity	4.120*	4.193	4.171	4.151	4.162	4.164
Table 3: Ablation experimental results on training tasks. We use the same abbreviations as Table 2.
it hard to be applied to large models like GPT-J-6B. Besides, KL penalty term is computed by  $\mathcal{L}_{KL} = E_{(\mathbf{x},\mathbf{y})\sim P_{\theta^{*}}}[-\log (P_{\theta^{*}}(\mathbf{y}|\mathbf{x}) / P_{\theta}(\mathbf{y}|\mathbf{x}))]$ , thus it requires sampling from the model output probability, which is time-consuming. On the contrary, EWC-LoRA is both memory- and time-efficient. In Table 2, we can see that EWC-LoRA achieves the lowest perplexity compared to other methods, while still significantly outperforming the base LMs. Compared with pure EWC, applying pure LoRA greatly decreases perplexity, which is consistent with the previous conclusion that limiting the dimension of the optimization problem can mitigate catastrophic forgetting [32]. EWC-LoRA further decreases perplexity, making it extremely close to the original perplexity, while achieving comparable performance with LoRA on downstream tasks. This demonstrates the effectiveness of EWC-LoRA. Besides, We can find that combing LoRA with KL will greatly increase perplexity while not achieving better downstream performance. Overall, our proposed EWC-LoRA achieves the best trade-off between the perplexity and the downstream performance, which outperforms baselines significantly while almost not increasing the perplexity on the pretraining dataset.
To study the contribution of each training task, we conducted an ablation study by removing one training task every time. We use GPT-Neo-1.3B as the base model. We include the results on tasks seen during training in Table 3. Results on all the tasks can be found in Appendix A. We can see that the removal of a training task with similar ability leads to a notable decrease in the model's performance on downstream tasks. For example, the performance of plan generation drops significantly when plan generation is removed from the training tasks. Similarly, the removal of activity recognition or object path tracking from the training tasks leads to a decline in performance in their respective downstream tasks. We conclude that our gathered embodied experience has a tremendous contribution to teaching the corresponding reasoning ability by finetuning. Interestingly, Counting QA performance shows an increase when counting is omitted from the training tasks, possibly because the ability of counting can be inferred from other training tasks.
We proposed a new training framework that uses world models to enhance language models. It first collects embodied experiences from world models through both goal-oriented planning and random exploration. The experiences are then compiled into appropriate formats for LMs finetuning. We further introduce EWC-LoRA, which not only facilitates parameter-efficient tuning but also alleviates catastrophic forgetting and enables knowledge generalization. We show the strong performance of our method on a large number of downstream evaluation tasks.
This work demonstrates the advantage of panoramic learning with all forms of experience [17]. On the other hand, the present work is limited to a single household environment as the world model. In the future, we intend to study how to integrate embodied experiences from different world models and generalize knowledge learned from each world model to different domains.
Acknowledgements. This project is partially supported by DARPA ECOLE HR00112390063.
[48] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[49] Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021.
[50] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560, 2023.
[51] Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merrienboer, Armand Joulin, and Tomas Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. In 4th International Conference on Learning Representations, ICLR 2016, 2016.
[52] Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862, 2021.
[53] Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3d environment. arXiv preprint arXiv:1801.02209, 2018.
[54] Jiannan Xiang, Zhengzhong Liu, Yucheng Zhou, Eric Xing, and Zhiting Hu. ASDOT: Any-shot data-to-text generation with pretrained language models. In *Findings of the Association for Computational Linguistics: EMNLP* 2022, pages 1886–1899, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[55] Jiannan Xiang, Xin Wang, and William Yang Wang. Learning to stop: A simple yet effective approach to urban vision-language navigation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 699–707, 2020.
[56] Claudia Yan, Dipendra Misra, Andrew Bennnett, Aaron Walsman, Yonatan Bisk, and Yoav Artzi. Chalet: Cornell house agent learning environment. arXiv preprint arXiv:1801.07357, 2018.
[57] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
[58] Rowan Zellers, Ari Holtzman, Matthew E Peters, Roozbeh Mottaghi, Aniruddha Kembhavi, Ali Farhadi, and Yejin Choi. PIGLeT: Language grounding through neuro-symbolic interaction in a 3d world. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2040–2050, 2021.
[59] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[60] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
Action	Template
[Find] <Object>	Find Object
[Walk] <Object>	Walk to Object
[Run] <Object>	Run to Object
[Sit] <Object>	Sit on Object
[StandUp]	Stand up
[Grab] <Object>	Grab Object
[Open] <Object>	Open Object
[Close] <Object>	Close Object
[Put] <Object_1> <Object_2>	Put Object_1 on Object_2
[PutIn] <Object_1> <Object_2>	Put Object_1 in Object_2
[SwitchOn] <Object>	Switch/Turn on Object
[SwitchOff] <Object>	Switch/Turn off Object
[Drink] <Object>	Drink Object
[TurnTo] <Object>	Turn to Object
[LookAt] <Object>	Look at Object
[Wipe] <Object>	Wipe Object
[PutOn] <Object>	Put on Object
[PutOff] <Object>	Put off Object
[Greet] <Object>	Greet Object
[Drop] <Object>	Drop Object
[Touch] <Object>	Touch Object
[Lie] <Object>	Lie on Object
[Pour] <Object_1> <Object_2>	Pour Object_1 into Object_2
[Type] <Object>	Type Object
[Watch] <Object>	Watch Object
[Move] <Object>	Move Object
[Wash] <Object>	Wash Object
[Rinse] <Object>	Rinse Object
[Scrub] <Object>	Scrub Object
[Squeeze] <Object>	Squeeze Object
[PlugIn] <Object>	Plug in Object
[PlugOut] <Object>	Plug out Object
[Cut] <Object>	Cut Object
[Eat] <Object>	Eat Object
[Sleep]	Sleep
[WakeUp]	Wake up
Table 4: Supported actions in VirtualHome and their corresponding text templates.
A.1 VirtualHome
The complete format of an executable action step in VirtualHome is <char{char_id}> [Action] <Object> (Object_id). Specifically, char_id specifies which agent to execute the action when multiple agents are in the world model at the same time. Action should be a supported atomic action in VirtualHome. Object is the object with which the agent interacts. Each object in the environment is assigned an Object_id to distinguish it from others of the same object class. We designed a template for each action to transform them into natural text for LMs finetuing. The full list of executable actions can be found in Table 4. Note that in the list, we omit {char_id} and {(object_id) for simplicity.
A.2 Acitivity Goal And Predicate
The goal of an household activity in VirtualHome consists of several predicates. Each predicate represents a condition of one object or a relation between two objects. For example, OPEN(coffe
maker) means the coffee maker is open, and ON(apple, table) means an apple is on the table. The goal is only achieved when all the predicates are achieved. We collected activities and goals from RobotHow.
A.3 Data Format and Prompts
Following Chung et al. [8], we use instructions with in-context exemplars as prompts. Specifically, the instruction, the question context, and the answer will be provided in each exemplar, and the full prompt will contain multiple such exemplars for in-context learning. The format of the data and the exemplar for each task is provided below.
A.3.1 Plan Generation
Data Example
Key	Value
activity	watch TV
condition	living room, sofa, TV. The sofa and TV are in the living room.
plan	Walk to living room. Sit on sofa. Watch TV.
In-context Exemplar
Q: How to {{activity}}? Given items include {{condition}} A: {plan}
A.3.2 Housework QA
Key	Value
activity	watch TV
choices	[TV, coffee, bed, toothbrush]
answer	TV
Question: To {{activity}}, a possibly related item could be Answer: {{answer}}
A.3.3 Negation Housework QA
Key	Value
activity	watch TV
choices	[TV, sofa, living room, toothbrush]
answer	toothbrush
Question: To {{activity}}, an unrelated item could be Answer: {{answer}}
A.3.4 Activity Recognition
Key	Value
plan choices activity	Walk to living room. Sit on sofa. Watch TV.
[watch TV, make coffee, sleep, brush teeth]
watch TV
Given a task plan: {{plan}}
Question: what is the name of this task?
Answer: {{answer}}
A.3.5 Activity Inference
Key	Value
state	Tom is sitting on the sofa. Tom is facing the TV.
choices	[watch TV, make coffee, sleep, brush teeth]
activity	watch TV
{{state}}
Question: given the above state, a possible activity could be
A.3.6 Counting
Key	Value
movement	Tom was at home. He grabbed an apple and put it on the bookshelf.
He then walked to the kitchen and srcub a plate. He went back to
bookshelf and put the plate on it.
location	bookshelf
number	2
items	apple, plate
Given a sequence of actions in a house, and a question about what items are located in a specific place. Answer the number of items and list the items.
Q: {{ movement }} How many items are there on the {{ location }}?
A: Ther are {{number}} itmes on the {{location}}. They are {{items}}
A.3.7 Counting QA
Key	Value
movement	Tom was at home. He grabbed an apple and put it on the bookshelf.
He then walked to the kitchen and srcub a plate. He went back to bookshelf and put the plate on it.
location	bookshelf
number	2
Q: { movement } How many items are there on the {{ location }}? A: { number }
A.3.8 Object Path Tracking
Key	Value
movement	Tom went to the kitchen. Mary walked into the dining room. Tom grabbed a plate. Tom travelled to the living room. Mary moved to the living room. Tom put the plate on the table. Mary grabbed the plate. Mary journeyed to the bedroom.
object	plate
path	kitchen, living room, bedroom
{ movement }   
Question: What is the order of the rooms where the { object } appeared?   
Answer: { path }
A.3.9 Object Location QA
Key	Value
movement	Tom went to the kitchen. Mary walked into the dining room. Tom grabbed a plate. Tom travelled to the living room. Mary moved to the living room. Tom put the plate on the table. Mary grabbed the plate. Mary journeyed to the bedroom.
object	plate
reference_room	living room
preposition	before
answer	kitchen
{ movement }   
Question: Where is the { object } {preposition } the { reference_room }?   
Answer: { answer }
A.4 Hyperparameters
For both GPT-Neo-1.3B and GPT-J-6B, we use a learning rate of  $8 \times 10^{-5}$  and a batch size of 20. The weights for plan generation, activity recognition, counting, and object path tracking are 1.0, 0.7, 1.0, and 1.0, respectively. We trained GPT-Neo-1.3B for 3 epochs with the EWC coefficient  $\lambda = 0.5$  in Equation 4. For GPT-J-6B, we trained it for 5 epochs with  $\lambda = 2$ . With our approach, it takes 40 minutes to train a GPT-Neo and 220 minutes to train a GPT-J. We used a rank of 8 and coefficient of 32 for LoRA's hyperparameters.
A.5 bAbI Dataset
We include 8 tasks from bAbI that test embodied knowledge. They are: One Supporting Fact, Two Supporting Fact, Three Supporting Fact, Counting, Lists/Sets, Simple Negation, Time Reasoning, Positional Reasoning. Examples for each task are shown in Table 5.
Task 1: Single Supporting Fact
Mary went to the bathroom.
John moved to the hallway.
Mary travelled to the office.
Where is Mary? A:office
Task 2: Two Supporting Facts
John is in the playground.