SlowGuess/NAACL_2021_OCR huggingface dataset:
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. 2017. Learning modular neural network policies for multi-task and multirobot transfer. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 2169-2176.
Chi Han, Jiayuan Mao, Chuang Gan, Josh Tenenbaum, and Jiajun Wu. 2019. Visual concept-metaconcept learning. In Advances in Neural Information Processing Systems (NIPS), pages 5002-5013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright, Chris Apps, Demis Hassabis, and Phil Blunsom. 2017. Grounded language learning in a simulated 3d world. CoRR, abs/1706.06551.
Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L McClelland, and Adam Santoro. 2020. Environmental drivers of systematicity and generalization in a situated agent. In Proceedings of the International Conference on Learning Representations (ICLR).
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 804-813.
Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, and Kate Saenko. 2019. Are you looking? grounding to multiple modalities in vision-and-language navigation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 6551-6557. Association for Computational Linguistics.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR).
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017. AI2-THOR: An interactive 3D environment for visual AI. arXiv preprint arXiv:1712.05474.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the International Conference on Machine Learning (ICML).
Brenden M Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. Proceedings of the International Conference on Machine Learning (ICML).
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. 2006. Walk the talk: Connecting language, knowledge, and action in route instructions. In Proceedings of the Conference on Artificial Intelligence (AAAI).
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. 2019. The neurosymbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In Proceedings of the International Conference on Learning Representations (ICLR).
Hongyuan Mei, Mohit Bansal, and Matthew Walter. 2016. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Proceedings of the Conference on Artificial Intelligence (AAAI).
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. Mapping instructions to actions in 3D environments with visual goal prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2667-2678.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. 2017. Zero-shot task generalization with multi-task deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 2661-2670. JMLR.org.
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc'Aurelio Ranzato. 2019. Taskdriven modular networks for zero-shot compositional learning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3593-3602.
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. 2020. A benchmark for systematic generalization in grounded language understanding. arXiv preprint arXiv:2003.05161.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In Computer Vision and Pattern Recognition (CVPR).
Kunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi, and Jonghyun Choi. 2020. Moca: A modular object-centric approach for interactive instruction following. arXiv preprint arXiv:2012.03208.
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth J Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the Conference on Artificial Intelligence (AAAI), volume 1, page 2.
Adam Vogel and Dan Jurafsky. 2010. Learning to follow navigational directions. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 806-814. Association for Computational Linguistics.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Hyperparameter	Value
Optimizer	Adam
Learning Rate	1e-4
Batch Size	8
Hidden State Dim	512
Word/Action Embedding Dim	100
Zero-Goal	True
Zero-Instr	False
Lang. Dropout	0.0
Vision Dropout	0.0
Input Dropout	0.0
Attn. Dropout	0.0
Actor Dropout	0.0
LSTM Dropout	0.3
Mask Loss Wt.	1.0
Action Loss Wt.	1.0
Table 3: Model hyperparameters. These settings largely follow the default parameters used by Shridhar et al. (2020).
A.1 Model and Training Hyperparameters
We list the hyperparameters used for all models in Table 3, we refer the reader to (Shridhar et al., 2020) for more details on the usage of each hyperparameter. Submodules are each structured identically to the monolithic baseline (e.g. each one had a 512 dimensional hidden state).
A.2 Hardware and Training Times
Models were trained on a Quadro RTX 6000 24GB GPU running on a machine with a 14 core Intel Xeon Gold 5120 CPU, with a runtime of approximately 14 hours. Evaluation was done on a V100 16GB GPU on a machine with a 4-core CPU. Subgoal evaluation took approximately 8 hours per split, and full trajectory evaluation approximately 1 hour.
We evaluate our model using the evaluation code provided by Shridhar et al. (2020).<sup>4</sup>
B ALFRED Dataset Details
In ALFRED, the agent observes a first person view, navigates with discrete grid movement, and uses objects by outputting a segmentation mask over its
image observation. The dataset contains approximately 25K expert instruction-trajectory pairs, pertaining to about 8K unique trajectories.
B.1 Task Types
The dataset contains demonstrations for 7 different kinds of tasks.
Pick & Place The agent must pickup a specified object, bring it to a destination, and place it. For example, "Pick up a vase, place it on the coffee table."
Examine in Light The agent must pickup an object and bring it to a light source. For example, "Examine the remote control under the light of the floor lamp."
Heat & Place The agent must pickup an object, put it in the microwave, toggle the microwave, take the object out of the microwave, and finally place the heated object at a specified location. For example: "Put a heated apple next to the lettuce on the middle shelf in the refrigerator."
Cool & Place This is the same as above, but with a refrigerator instead of a microwave. For example, "Drop a cold potato slice in the sink."
Clean & Place The agent must put an object into the sink and turn on the water to clean the object. Then, it must be placed at a specified location. For example, "Put a washed piece of lettuce on the counter by the sink."
Stack & Place The agent must pick up an object, place it into a receptacle, and then bring the stacked objects to a specified location and place them. For example, "Move the pan on the stove with a slice of tomato in it to the table."
Pick Two & Place The agent must pickup an object, place it somewhere, then pick up another instance of that object and put it in the same place. For example, "Place two CDs in top drawer of black cabinet."
These last two task types, Stack & Place and Pick Two & Place, are the ones held out in the Novel Tasks generalization experiments.
<sup>4</sup>https://github.com/askforalfred/alfred