SlowGuess/NAACL_2022_OCR huggingface dataset:
Re-ranking hypotheses The top-k recall performance reported in Table 2 is based on  $\log p(h|\tau)$ . We confirmed that the biases reported in the literature such as predicting content from the prompt are present in model predictions (Zhao et al., 2021). Consider the query example Place a martini glass with a fork on it on the table. The following two are among the top generated hypotheses:
a) pick up fork, put in cup, pick up cup, put in sink. b) pick up fork, put in cup, pick up cup, put in table.
When the prompt contains training examples that mention 'sink', the model assigns the following log  $p(h|\tau)$  to these hypotheses: a) -2.4 and b) -4.3. However, when all training instances in the prompt involving 'sink' are removed, the log probabilities now become, a) -13.7 and b) -9.1 The incorrect hypotheses involving 'sink' is now ranked below the correct hypothesis involving table. While language
models can retrieve strong hypotheses as indicated by the high top-10 recall, this observation shows that the ranking of these hypotheses, as determined by  $p(h|\tau)$ , may not be accurate. We thus consider mutual information based ranking approaches. Table 1 shows top-1 recall when model generated hypotheses are ranked according to different criteria. We also vary the number of training examples  $N$  by randomly choosing respectively 1, 2 and 3 instances per fine-grained task type.
We first observe that  $p(\tau|h)$  ranks hypotheses better than  $p(h|\tau)$  with very limited supervision ( $N = 11$ ). However, it is often worse when more supervision is available. In contrast, combining the two log probabilities with  $\lambda = \frac{1}{2}$  yields consistently better performance across models and number of training examples. This shows that generating a large number of hypotheses with a language model, followed by more accurate re-ranking using Mutual Information inspired metrics can be an effective paradigm for few-shot generation tasks with in-context learning.
Comparison with prior work We compare our prediction performance against prior work in Table 3. Jansen (2020) fine-tunes a GPT2-Medium model (325M parameters) to predict subgoals from instructions and report prediction results when the model is trained on varying amounts of training data:  $1\%$ ,  $10\%$ ,  $25\%$ ,  $100\%$  of the training set, which has 7793 instances. We ignore the navigation subgoals in this evaluation and only compare the sequence of object interactions. We report prediction performance of GPT-J using our approach on the same test set. The results show that large language models encode useful knowledge that can help plan from instructions effectively when supervision is limited. However, fine-tuning can be effective when more supervision is available due to the fixed context length limitation of in-context
4 https://github.com/cognitiveailab/alfred-gpt2
Model	Top-1 recall	Training instances
Jansen (2020)	5.07	77
41.92	779
(Fine-tuned GPT2-Medium)	53.80	1948
61.00	7793
Ours	44.02	11
(In-context GPT-J)	47.80	22
49.63	33
learning. See Section 5 for ablations and more discussion about fine-tuning.
Prediction errors We examine prediction errors in identifying task type and object type. Key sources of model errors include annotation issues and ambiguity in object types. Table 4 shows the object types that have the least prediction accuracy, along with the object categories the model is confused about. Annotations can fail to correctly identify the target object - identifying a butter knife as a knife or a pepper shaker as salt shaker. Ambiguity can also arise from identifying an object with different names, depending on the context. For instance, depending on the scene layout, the argument for a look at object in light task can be a floor lamp or a desk lamp. Unless the type of lamp is identified precisely in the instruction, it is not possible to correctly predict the type of lamp. Correctly identifying these objects requires feedback from interaction with the environment.
The experiments so far evaluate the ability of a language model to retrieve ground truth subgoal sequences. Next we examine embodied agents that make use of these predictions and collect more supervision in order to improve subgoal predictions.
4.3 Agent policy and incorporating environment feedback
We now use subgoal predictions to construct an agent policy that acts in a simulated environment to complete tasks. The agent state representation and pre-trained low-level subgoal policy are borrowed from the HLSM model proposed in Blukis et al. (2022). HLSM represents the agent state as a spatial persistent voxel representation of the room which models the location and category of objects present. The representation is constructed using modules that estimate segmentation and depth maps and other visual reasoning components and is updated as the agent gathers new observations. We
Table 3: Comparison against subgoal prediction performance of Jansen (2020).
Obj category	Confusion categories
wateringcan	pencil, kettle
glassbottle	vase
cart	shelf, sidetable, microwave, cart, fridge
butterknife	knife, butterknife
floorlamp	desklamp, floorlamp
vase	pencil, vase, bowl, winebottle, pot
ladle	spoon, ladle
pot	pot, pan
soapbottle	soapbottle, winebottle
Table 4: Object categories the model makes most errors on and the top object categories it confuses with.
use pre-trained models made available by the authors for state estimation and the low-level policy in our experiments.
We combine subgoal predictions with the pretrained HLSM low-level policy and evaluate the overall agent policy on the ALFRED task in Table 5. Unlike the results reported in Section 4.2 which were based on the static dataset, these results are based on subgoals executed against the AI2-Thor simulator. In addition to task success rate, we also report the percentage of goal conditions satisfied, which rewards the model for partial task completions.
We compare against the following baselines on the ALFRED task. Seq2seq (Shridhar et al., 2020) is a simple sequence-to-sequence baseline trained to map text instructions to low-level actions. MOCA (Singh et al., 2020) improves on Seq2seq using subgoal supervision and pre-trained visual reasoning components. Recent work such as HLSM (Blukis et al., 2022) and FiLM (Min et al., 2021b) build and use spatial semantic state representations and achieve stronger performance on the task. Note that, unlike these prior methods (MOCA, HLSM, FiLM) that rely on full subgoal supervision (20k instances), our approach is based on a small amount of subgoal supervision and additional supervision collected using active interaction with the environment. In addition, our approach does not require task-level expert trajectories and only assumes that a subgoal execution policy is provided.
Using the top language model prediction as is without using any information from the environment leads to  $20\%$  success rate. Next, we collect plan execution feedback for 1000 text instructions to train the ranking model described in Section 3.2. Re-ranking language model predictions using the trained ranking model improves the performance
<sup>5</sup>https://hlsm-alfred.github.io
Model	Success rate
Task	Goal-Cond
Seq2seq (Shridhar et al., 2020)	3.7	10.0
MOCA (Singh et al., 2020)	19.2	28.5
FiLM (Min et al., 2021b)	24.6	37.2
HLSM (Blukis et al., 2022)	29.6	38.8
HLSM	Predicted subgoals	19.8	31.4
low-level policy	Re-ranked subgoals	23.9	35.0
Oracle subgoals	37.2	48.2
Table 5: Task completion and goal condition success rates of models on the ALFRED validation seen split (results are based on task executions in the AI2-Thor simulator). The performance of our subgoal predictions combined with the HLSM low-level policy are shown at the bottom. We show the performance before and after re-ranking language model predictions based on agent state. Oracle subgoals show the performance upper bound.
to  $24\%$ , which shows the importance of incorporating feedback from environment interaction. In comparison, the HLSM model with full subgoal supervision has success rate  $30\%$ . Although our predictions fall short of HLSM, they are competitive with the other baselines with subgoal supervision. The performance upper bound estimated using oracle subgoals is  $37\%$ , which shows the room for improvement over our predictions. These results show that accurate subgoal inference coupled with pre-trained low-level components leads to agents that perform well in embodied environments.
Figure 1 shows an example where the ranking model uses environment information to identify better plans. In this example, the instruction ambiguously specifies the receptacle as 'white table with shelving'. The language model's top two predictions for the target receptacle are 'side table' and 'shelf', neither of which are present in the environment. The agent state captures this information and helps identify 'dining table' as the correct receptacle.
We perform a series of ablations to identify the robustness of model predictions. We compare the performance of in-context learning to a sequence-to-sequence model fine-tuned to translate instructions to subgoal sequences. In addition, we observe the effect of varying the number of training examples and choice of training examples.
Number of training examples Figure 3 shows model recall for varying number of training examples. For zero training examples, the prompt consists of just the query instruction and the model
Figure 3: Comparison between GPT-J with in-context learning and a fine-tuned T5-large model for varying number of training examples. See text for details.
decodes subgoals. Beyond 44 examples (4 examples per task type), the model prompt no longer fits the sequence length restriction (1024 tokens) of GPT models. A steady increase in performance can be initially observed when increasing the number of training examples and the performance saturates towards the end. In-context learning further has the limitation of not being able to accommodate a larger number of training examples due to the length restriction. It would be interesting to explore how to make effective use of large numbers of training examples in future work.
Choice of training examples We also estimate performance variance by varying the random seed for choosing examples randomly from the training set and compute standard deviation based on five random seeds for each setting. The plot shows that top-1 predictions from in-context learning have lower variance compared to fine-tuning.
Comparison with fine-tuning In order to understand how well the in-context learning approach compares to fine-tuned models, we fine-tune a T5-large model (Raffel et al., 2019) with 770M parameters on varying amounts of training data (this was the largest model we could fine-tune on our compute infrastructure). Note that this is not a head-to-head comparison between in-context learning and fine-tuning due to the difference in model size. Furthermore, there are other fine-tuning mechanisms such as prompt tuning and head tuning (Min et al., 2021a) which are not considered here. However, the result suggests that in-context learning with large pre-trained models can be favorable when computational constraints do not allow full fine-tuning of large models.
These ablations show that the in-context learning ability of large language models leads to pre
dictions that are accurate, robust and stable in the presence of a small amount of training data.
This work explores the use of pre-trained language models for planning in real-world tasks. We showed that language models have strong capability to reason about subgoal sequences given a small number of training examples. We further demonstrated some simple mechanisms to incorporate feedback from interaction with the environment and show that this leads to more usable predictions. Finally, we show that combining subgoal predictions with a pre-trained low-level policy yields a strong baseline for embodied agent learning.
Our ablations demonstrate that in-context learning with a small amount of subgoal demonstrations has robust generalization properties. However, we also point out that in-context learning has the limitation of not being able to incorporate a large number of training examples due to the fixed context length restriction. It would further be beneficial to perform end-to-end learning with language model based subgoal prediction and a low-level policy, which would be interesting to explore in future work.
Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. 2022. A persistent spatial semantic representation for high-level natural language instruction execution. In Conference on Robot Learning, pages 706-717. PMLR.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosse-lut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
S.R.K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. 2009. Reinforcement learning for mapping instructions to actions. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 82-90, Suntec, Singapore. Association for Computational Linguistics.
SRK Branavan, David Silver, and Regina Barzilay. 2012. Learning to win by reading manuals in a monte-carlo framework. Journal of Artificial Intelligence Research, 43:661-704.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
Haonan Chen, Hao Tan, Alan Kuntz, Mohit Bansal, and Ron Alterovitz. 2020. Enabling robots to understand incomplete natural language instructions using commonsense reasoning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1963-1969. IEEE.
Rodolfo Corona, Daniel Fried, Coline Devin, Dan Klein, and Trevor Darrell. 2021. Modular networks for compositional instruction following. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1033-1040, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-SQL evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 351-360, Melbourne, Australia. Association for Computational Linguistics.
Daniel Fried, Jacob Andreas, and Dan Klein. 2018. Unified pragmatic models for generating and following instructions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1951-1963, New Orleans, Louisiana. Association for Computational Linguistics.
Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Scharli. 2020. Compositional generalization in semantic parsing: Pre-training vs. specialized architectures. arXiv preprint arXiv:2007.08970.
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. 2020. Interactive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7903-7910.
Jonathan Herzig, Peter Shaw, Ming-Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. 2021. Unlocking compositional generalization in pre-trained models using intermediate representations. arXiv preprint arXiv:2104.07478.
Peter Jansen. 2020. Visually-grounded planning without vision: Language models infer detailed plans from high-level instructions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4412-4417, Online. Association for Computational Linguistics.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474.
Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning, pages 2873-2882. PMLR.
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1813-1827, Online. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Jiwei Li and Dan Jurafsky. 2016. Mutual information and diverse decoding improve neural machine translation. arXiv preprint arXiv:1601.00372.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online. Association for Computational Linguistics.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktaschel. 2019. A survey of reinforcement learning informed by natural language. arXiv preprint arXiv:1906.03926.
Hongyuan Mei, Mohit Bansal, and Matthew R Walter. 2016. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Thirtieth AAAI Conference on Artificial Intelligence.
Vincent Micheli and Francois Fleuret. 2021. Language models are few-shot butlers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9312-9318, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2021a. Noisy channel language model prompting for few-shot text classification arXiv preprint arXiv:2108.04106.
So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and Ruslan Salakhutdinov. 2021b. Film: Following instructions in language with modular methods. arXiv preprint arXiv:2110.07342.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguistics.
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7699-7715, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740-10749.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning In Proceedings of the International Conference on Learning Representations (ICLR).
Kunal Pratap Singh, Suvaansh Bhambi, Byeonghwi Kim, Roozbeh Mottaghi, and Jonghyun Choi. 2020. Moca: A modular object-centric approach for interactive instruction following. arXiv preprint arXiv:2012.03208.
Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav Sukhatme. 2021. Embodied bert: A transformer model for embodied, language-guided visual task completion. arXiv preprint arXiv:2108.04927.
Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. 2019. Executing instructions in situated collaborative interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2119-2130, Hong Kong, China. Association for Computational Linguistics.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. 2020. Keep CALM and explore: Language models for action generation in text-based games. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8736-8754, Online. Association for Computational Linguistics.
Yichi Zhang and Joyce Chai. 2021. Hierarchical task learning from language instructions with unified transformers and self-monitoring. In *Findings of the Association for Computational Linguistics: ACLIJCNLP* 2021, pages 4202-4213, Online. Association for Computational Linguistics.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR.
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. 2020. Rtfm: Generalising to new environment dynamics via reading. In International Conference on Learning Representations.
A Mutual Information based scoring