SlowGuess/ACL_2020_OCR huggingface dataset:
Proc. of the 13th Int'l Symposium on Experimental Robotics (ISER).
Dipendra K. Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. 2016. Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions. The International Journal of Robotics Research, 35(1-3):281-300.
Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. 2019. Collaborative dialogue in Minecraft. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5405-5415, Florence, Italy. Association for Computational Linguistics.
Ramakanth Pasunuru and Mohit Bansal. 2018. Game-based video-context dialogue. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 125-136, Brussels, Belgium. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of Twitter conversations. In *Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics*, pages 172-180, Los Angeles, California. Association for Computational Linguistics.
Deb Roy and Ehud Reiter. 2005. Connecting language to the world. Artificial Intelligence, 167(1-2):1-12.
Nicolas Schrading, Cecilia Ovesdotter Alm, Ray Ptucha, and Christopher Homan. 2015. An analysis of domestic abuse discourse on Reddit. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2577-2583, Lisbon, Portugal. Association for Computational Linguistics.
Alane Suhr and Yoav Artzi. 2018. Situated mapping of sequential instructions to actions with single-step reward observation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2072-2082, Melbourne, Australia. Association for Computational Linguistics.
Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. 2019. Executing instructions in situated collaborative interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2119-2130, Hong Kong, China. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Arthur Szlam, Jonathan Gray, Kavya Srinet, Yacine Jernite, Armand Joulin, Gabriel Synnaeve, Douwe Kiela, Haonan Yu, Zhuoyuan Chen, Siddharth Goyal, Demi Guo, Danielle Rothermel, C. Lawrence Zitnick, and Jason Weston. 2019. Why build an assistant in Minecraft? arXiv preprint arXiv:1907.09273.
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, pages 1507-1514.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2019. Vision-and-dialog navigation. arXiv preprint arXiv:1907.04957.
Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Justin Hart, Peter Stone, and Raymond J. Mooney. 2017. Opportunistic active learning for grounding natural language descriptions. In Proceedings of the 1st Annual Conference on Robot Learning (CoRL-17), pages 67-76, Mountain View, California. PMLR.
Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Peter Stone, and Raymond J. Mooney. 2016. Learning multi-modal grounded linguistic semantics by playing "I Spy". In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI-16), pages 3477-3483, New York City.
Jesse Thomason, Shiqi Zhang, Raymond J Mooney, and Peter Stone. 2015. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI 2015), pages 1923-1929.
Sida I. Wang, Samuel Ginn, Percy Liang, and Christopher D. Manning. 2017. Naturalizing a programming language via interactive learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 929-938, Vancouver, Canada. Association for Computational Linguistics.
Terry Winograd. 1971. Procedures as a representation for data in a computer program for understanding natural language. Technical report, MIT. Cent. Space Res.
H	cnn_type	l	j
BAP-baseH1	300	cnn_small	3	1
BAPH3,4x	300	cnn_small	4	1
Table 3: Hyperparameter values for the baseline and full BAP models.
A Model Hyperparameters
We use Gated Recurrent Units (GRUs) (Chung et al., 2014) for all RNN modules and use 300-dimensional pretrained GloVe word embeddings (Pennington et al., 2014). All linear layers were initialized using Xavier initialization (Glorot and Bengio, 2010). All non-linearities in the model are ReLU. All  $3 \times 3 \times 3$  3d-conv layers have stride 1 and padding 1. All  $1 \times 1 \times 1$  3d-conv layers have stride 1 and no padding.
For each model, we perform a grid search over the following hyperparameters:
- The size of the GRU hidden state  $H \in \{200, 300\}$
- The number of 3d-conv layers and channels in the world state encoder and action sequence decoder CNNs. We define a 3-tuple  $(e_{\text{channels}}, m, n)$  where  $e_{\text{channels}}$  defines the number of output channels for the first encoder-CNN 3d-conv layer (which then determines the number of output channels for subsequent encoder-CNN 3d-conv layers);  $m$  is the number of  $3 \times 3 \times 3$  3d-conv layers in the world state encoder; and  $n$  is the number of  $1 \times 1 \times 1$  3d-conv layers in the action sequence decoder.
We choose between 2 hyperparameter configurations:  $cnn_{type} \in \{cnn_{small} = (200,2,3), cnn_{big} = (300,3,2)\}$ .
- The number of dense linear layers in the STOP prediction model  $l \in \{3, 4\}$
- The number of dense linear layers used to embed the action vectors before being fed to the decoder's GRU  $j \in \{1, 2\}$
Table 3 shows values of these hyperparameters for our baseline and best models.
B Qualitative Examples
Here, we provide more examples of action sequences generated by our model, along with the initial game state context and the human B's actions as ground truth, in order to better highlight
Figure 5: Example 3.
Figure 6: Example 4.
Figure 7: Example 5.
Figure 8: Example 6.
Figure 9: Example 7.
the strengths and shortcomings of the full BAP model. Examples 5, 6 and 7 also examine the net actions F1 evaluation metric in context.
Example 3 can be found in Figure 5. Over the course of some back-and-forth dialogue with A, B has just built the leftmost 2 yellow blocks of the left yellow row. From here, our model interprets "do the same on the other side" as placing another 2 yellow blocks, but places them in the wrong location. The human B is able to understand that A means to place the blocks on the other end of the row-in-progress.
Example 4 can be found in Figure 6. This example occurs near the end of a game.  $\mathbf{B}$  has just finished building a  $3\times 3$  ring of blue blocks, while facing the structure from the back side (i.e., facing the camera in the figure). Following the description "the middle block of the ring's right side", our model incorrectly predicts placing a blue block adjacent to one of the middle blocks of the ring, while the human  $\mathbf{B}$  grounds this easily. Clearly, higher-level information needed to help ground the instruction is lost in context: earlier in the dialogue history (yet still within the window of utterances in the  $H_{3}$  history scheme),  $\mathbf{B}$  has clarified with  $\mathbf{A}$  that the structure is entirely 2D, which contradicts the model's prediction.
Example 5 can be found in Figure 7. B has built a  $V$  using blue blocks as placeholders to support the red blocks. Our model interprets "get rid of the blue blocks" partially correctly, and removes one blue block, but does not go all the way as the human B does, who removes all existing blue blocks. While both the model's and human B's action sequences are correct, the model's actions are incomplete, and it is penalized according to net actions F1.
Example 6 can be found in Figure 8. This example occurs at the beginning of a game. Here,  $\mathbf{A}$  does not specify a specific location for the green blocks to be placed, just that they should be "towards the middle." In this instance, both our model's prediction and the human B's actions are valid interpretations. However, our model's output is penalized for not predicting the exact positions of the human B's blocks. This highlights the net actions F1 metric's inflexibility to ambiguous scenarios.
Example 7 can be found in Figure 9. This example is similar to Example 8 in that the model
predicts a sequence of actions that results in a structure that is rotationally equivalent to the human B's resulting structure. However, in this case, A's instruction to place the green blocks "towards the middle of the board" (a suggestion our model does not follow) is extremely important in the larger context of task completion: the model's actions would result in a final structure that cannot fit within the grid boundaries. Here, the strictness of net action F1's exact match requirement works as intended, to our benefit.