SlowGuess/ACL_2020_OCR huggingface dataset:
Our full model represents each cell as an 11-dim vector (consisting of the 7-dim cell state, 1-dim action history weight and 3-dim perspective coordinates), and the entire grid (which serves as input to a CNN-based encoder) as a  $11 \times 11 \times 9 \times 11$  tensor. We refer to the grid at time step  $t$  as  $W_{raw}^{(t)}$ .
Output: a CNN-based encoding To obtain a representation of each grid cell, we feed the raw world state tensor  $W_{raw}^{(t)}$  of Section 3.3 through a multi-layer CNN that embeds each grid cell conditioned on its neighborhood and recent actions (if using action history weights). The model consists of  $m$  3d-conv layers with kernel size 3 ( $\mathrm{CNN}_3$ ), stride 1 and padding 1, followed by a ReLU activation function. Between every successive pair of these layers is a  $1 \times 1 \times 1$  3d-conv layer ( $\mathrm{CNN}_1$ ) with stride 1 and no padding, for dimensionality reduction purposes, again followed by ReLU. With  $W_0^{(t)} = W_{raw}^{(t)}$ , the first  $m - 1$  blocks of this model can be expressed as  $W_i^{(t)} = \mathrm{relu}(\mathrm{CNN}_1^i (\mathrm{relu}(\mathrm{CNN}_3^i (W_{i - 1}^{(t)})))$  The  $m$ 'th  $3 \times 3 \times 3$  3d-conv layer  $\mathrm{CNN}_3^m$  computes the final world state representation  $W_m^{(t)} = \mathrm{relu}(\mathrm{CNN}_3^m (W_{m - 1}^{(t)}))$  that is used to predict the next action.
3.4 Action Sequence Decoder
The GRU backbone The GRU backbone of the decoder captures information about the current action sequence and the game history. We initialize its hidden state with the final hidden state of the game history encoder RNN of Section 3.2. Since the tensor representation of the grid is too unwieldy to be used as input to a recurrent net, we instead compute an explicit 11-dim representation  $\mathbf{a}^{(t - 1)}$  of the action taken at the last time step, consisting of three components: a 2-dim one-hot vector for the action type (placement or removal), a 6-dim one-hot vector for the block color (all zero for removals), and a 3-dim block location vector containing the absolute  $\langle x,y,z\rangle$  coordinates of the cell where the action took place. At the start of decoding, we use a zero vector as a start token. These action vectors get passed through  $j$  dense linear layers with ReLU before being fed to the GRU.
Output: Next action prediction With seven possible actions per cell, there are 7623 possible actions (although only a small subset of these will be feasible at any point in time, a point that we will return to below). Since our models need to
predict a variable length sequence of actions, we also need a special STOP action that is not associated with a single cell, but terminates the sequence. Our action prediction classifier has therefore two sub-components: a block action prediction model, and a stop prediction model. The stop prediction model returns a single element, which we append to the vector returned by the block action prediction model before feeding it through a softmax layer to return the most likely next action.
Block actions scores: We use a CNN-based architecture with parameter sharing across cells to score each of the seven possible actions for every grid cell. The input to this model consists of the CNN-based world state representation  $W_{m}^{(t)}$  (Section 3.3), as well as the decoder GRU's hidden state  $\mathbf{h}^{(t)}$ , concatenated to each cell's representation in  $W_{m}^{(t)}$  as additional channels. This model consists of  $n - 1 \times 1 \times 1$  3d-conv layers followed by ReLU  $(W_{i}^{\prime(t)} = \mathrm{relu}(\mathrm{CNN}_{1}^{i}(W_{i - 1}^{\prime(t)}))$  and with the  $n$ th such 3d-conv layer with 7 output channels (and no ReLU):  $W_{n}^{\prime(t)} = \mathrm{relu}(\mathrm{CNN}_{1}^{n}(W_{n - 1}^{\prime(t)}))$ , which is flattened into a 7623-dim vector of action scores.
STOP score: We also need to predict when an action sequence is complete. While this decision needs access to the same information as the block action scorer, it also needs access to a (compact) global representation of the grid, since the STOP action is not cell-specific. It also needs to know the uncertainty in the block action scorer, since STOP is more likely when it is less clear which block action should be performed, and vice versa. We take the output of the penultimate layer in the block action scorer and apply max-pooling to every cell's vector representation, thus obtaining a single number for each of the 1089 cells. We concatenate these numbers into a single vector and use that as input to the STOP prediction model, which consists of  $l$  dense linear layers (with ReLU after each layer except the last), where the  $l$ th layer has a single output  $W_{l}^{\prime \prime (t)}$ , the score for STOP.
Final action prediction scores: Finally, we concatenate the block action and STOP scores and apply a softmax to obtain the final prediction  $a^{(t)}$ :
$$
a _ {t} = \arg \max (\mathrm {s o f t m a x} (v e c (W _ {n} ^ {\prime (t)}) \oplus W _ {l} ^ {\prime \prime (t)}))
$$
4 Data Augmentation
The small size of the training set (3,709 examples) is a major limiting factor for training complex models. Here, we explore ways of generating synthetic
data to augment the size and variety of our data. For each game log in the original training data, we generate twenty new game logs by combining the following data augmentation techniques:
Utterance paraphrases: We generate paraphrases of the utterances in the dialogue by randomly substituting tokens with any of their synonyms in the hand-engineered synonym lexicon of Narayan-Chen et al. (2019).
Color substitutions: We permute block colors by applying one of the 6! possible permutations, chosen at random, to the entire game log. These substitutions also change the language in the synthetic dialogues to reflect the updated colors.
Spatial transformations: Since the world contains no landmarks besides the built region, absolute coordinates are somewhat arbitrary. We sample one (0, 90, -90, 180) rotation in the ground plane (affecting all  $\langle x, z \rangle$  coordinates, plus  $\mathbf{B}$ 's yaw and position) per synthetic log (subject to the constraint that the target still fit in the built region).
We evaluate our world state encoders, game history and data augmentation schemes.
Experimental Setup Our training, test and development splits contain 3709, 1616, and 1331 Builder action sequences respectively. We increase the training data to 7418 (2x), 14836 (4x) and 22254 (6x) items by sampling items from the synthetic data of Section 4. The average sequence length (in the development set) is 4.3 (with a std. deviation of 4.5). Target structures in the test data do not appear in the training or development data. We train models with AdamW (Loshchilov and Hutter, 2019) and weight decay regularization with a weight decay factor of 0.1. We use a learning rate of 0.001 for the original data and a slightly lower learning rate of 0.0001 in the case of augmented data. We use a batch size of 1. During training, we use teacher forcing and minimize the sum of the cross entropy losses between each predicted and ground truth action sequence (the action sequence performed by the human). We stop training early when loss on the held-out development set has increased monotonically for ten epochs. We use greedy decoding (max. sequence length of 10) to generate action sequences, which seems to work better than beam search decoding (for fixed beam sizes between 5 and 20). We report net action F1 (Section 2.3) on the test set.
H1	H2	H3
BAP-base	11.8	12.4	14.6
+ action history	14.6	18.2	19.7
+ perspective	15.7	18.7	18.8
Table 1: The effect of varying game history and world state representations on test set performance.
2x	4x	6x
BAP-baseH3	15.6	16.1	17.0
+ action historyH3	16.9	20.0	18.4
+ perspectiveH3	19.5	21.2	20.8
Table 2: The effect of data augmentation at  ${2x},{4x}$  and  $6\mathrm{x}$  training data on test set performance.
Model Variants The world state representation of the baseline model (BAP-base) consists of block colors at absolute  $\langle x,y,z\rangle$  coordinates. We examine the effect of augmenting BAP-base first with action history weights, and then also with relative perspective coordinates (both described in Section 3.3). For model hyperparameters, see Appendix A.
Game History We experiment with three schemes for how much game history to provide to the models:  $H_{1}$  includes A's last utterance and any following B utterances.  $H_{2}$  includes all utterances after B's penultimate action sequence.  $H_{3}$  includes all utterances after B's penultimate action sequence interleaved with a token representation of B's last action sequence. If A's last utterance was a standalone instruction,  $H_{1}$  should be sufficient. But prior discourse is often required: A instructions may span multiple utterances and can be interrupted by back-and-forth clarification dialogues. At the same time, B's next action sequence is often directly related to (or a continuation of) their previous actions. This motivates  $H_{2}$  and  $H_{3}$ : by including utterances that sandwich B's previous action sequence, we include additional A history and B context. Finally, to investigate the degree to which previous B actions should be represented,  $H_{3}$  augments  $H_{2}$  with explicit representations of B's actions (as described in Section 3.2).
6.1 Quantitative Evaluation
For each cell in Tables 1 and 2, we first perform a grid search over model hyperparameters and select the best performing model on the development set, then report its performance on the test set.
Table 1 shows how the different game history and world state representations affect model per
formance. We see that performance increases as action weights are added and as the amount of history is increased.  $H_{3}$  consistently performs well across all model variants.
Table 2 shows how different amounts of data augmentation affect performance. We train each model variant with  $H_{3}$  history on  $2\mathrm{x},4\mathrm{x}$  and  $6\mathrm{x}$  augmented training data. This increases BAP-base  $H_{3}$ 's performance from 14.6 to 17.0 (with 6x data). With action history, performance increases from 19.7 to 20.0. With perspective coordinates, performance increases from 18.8 to 21.2 (both with 4x data). Perspective coordinates, thus, help with more training data (although it is unclear why performance drops again for the more complex models at 6x).
Our best model is the full BAP model with action weights, perspective coordinates, history  $H_{3}$  and 4x augmented data  $(\mathrm{BAP}_{H_3,4\mathrm{x}})$  with an F1 of 21.2. This is significantly better than the 11.8 F1 of our baseline BAP model with history  $H_{1}$  and without action history weights, perspective coordinates, or data augmentation  $(\mathrm{BAP - base}_{H_1})$ . We also see an improvement in mean sequence length from 2.23 to 2.66, even if the latter is still much smaller than the mean gold sequence length of 4.3.
Infeasible Actions and Constrained Decoding In any given world state, only a small fraction of the 7623 actions are feasible: blocks can only be placed in locations that are currently empty and adjacent to existing blocks or on the ground, and blocks can only be removed from locations that are currently occupied. Surprisingly, less than  $1\%$  of action sequences generated by any of our models contain one or more infeasible actions. We can force our models to predict only feasible actions by multiplying the output of the block action prediction model (post softmax) with a bit mask over block actions that identifies which of the possible actions are feasible in the current world state, but this does not affect the F1 scores of either the baseline model or our best model.
6.2 Qualitative Evaluation
We return to the development set to illustrate different aspects of  $\mathrm{BAP}_{H_3,4\mathrm{x}}$  's generated action sequences. Figures 3 and 4 provide a few examples; more examples can be found in Appendix B.
Colors: Our model is generally able to correctly identify colors of blocks to be placed. While in many cases continuing the color from the previous
Figure 3: Example 1: After  $\mathbf{B}$  places the rightmost purple block,  $\mathbf{A}$  directs  $\mathbf{B}$  to place another red block on top of it. This occurs after a long back-and-forth clarification dialogue in which  $\mathbf{B}$  struggles to understand  $\mathbf{A}$ 's instructions; but the human  $\mathbf{B}$  now completes the intended substructure by placing two red blocks and removing the purple. The model does not have access to the preceding dialogue, but interprets the most recent instruction correctly.
Figure 4: Example 2: Here,  $\mathbf{B}$  had just placed the two blocks atop the ends of the row of 3 blocks to create a  $U$ . Now, the model can interpret "do it one more time" and extends the  $U$  upwards by placing two more blocks.
action sequence is sufficient, the model is also able to switch colors as needed based on  $\mathbf{A}$  instructions.
Numbers: Our model can sometimes identify the number of blocks to be placed when instructions mention them. But with vague instructions, the model struggles, stopping early or erroneously continuing long sequences of the same color.
Spatial relations: Our model usually predicts a reasonable ballpark of locations for the next action sequence. While predicting correct locations exactly is still difficult, the model is usually able to distinguish "below" from "on top of", and places blocks in the neighborhood of the true sequence.
Placements vs. removals: Finally, our model is able to both place and remove blocks somewhat appropriately based on dialogue context. For instance, corrective utterances in the history ("sorry, my mistake") usually trigger the model to undo previous actions. However, the model sometimes goes overboard: not knowing how much of the penultimate action sequence to remove, an entire sequence of correct blocks can be erroneously erased.
In the Minecraft Collaborative Building Task, Builders must be able to comprehend complex instructions in order to achieve their primary goal of building 3D structures. To this end, we define the challenging subtask of Builder Action Prediction, tasking models with generating appropriate
action sequences learned from the actions of human Builders. Our models process the game history along with a 3D representation of the evolving world to predict actions in a sequence-to-sequence fashion. We show that these models, especially when conditioned on a suitable amount of game history and trained on larger amounts of synthetically generated data, improve over naive baselines. In the future, richer representations of the dialogue history (e.g. by using BERT (Devlin et al., 2019) or of past Builder actions) combined with de-noising of the human data and perhaps more exhaustive data augmentation should produce better output sequences. For true interactivity, the Builder must be augmented with the capability to determine when and how to respond when it is too uncertain to act. And, finally, an approach like the Speaker-Follower Models of Fried et al. (2018) could be used to train our Builder model and the Architect model of Narayan-Chen et al. (2019) jointly.
We would like to thank the reviewers for their valuable comments. This work was supported by Contract W911NF-15-1-0461 with the US Defense Advanced Research Projects Agency (DARPA) Communicating with Computers Program and the Army Research Office (ARO). Approved for Public Release, Distribution Unlimited. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 3674-3683. IEEE Computer Society.
Jacob Andreas and Dan Klein. 2015. Alignment-based compositional semantics for instruction following. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1165-1174, Lisbon, Portugal. Association for Computational Linguistics.
Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49-62.
Yonatan Bisk, Daniel Marcu, and William Wong. 2016a. Towards a dataset for human computer communication via grounded language acquisition. In AAAI Workshop: Symbiotic Cognitive Systems.
Yonatan Bisk, Kevin Shih, Yejin Choi, and Daniel Marcu. 2018. Learning interpretable spatial operations in a rich 3D Blocks World. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 5028-5036.
Yonatan Bisk, Deniz Yuret, and Daniel Marcu. 2016b. Natural language communication with robots. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 751-761, San Diego, California. Association for Computational Linguistics.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Joyce Y. Chai, Qiaozi Gao, Lanbo She, Shaohua Yang, Sari Saba-Sadiya, and Guangyue Xu. 2018. Language to action: Towards interactive task learning with physical agents. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18), pages 2-9. International Joint Conferences on Artificial Intelligence Organization.
David Chen and Raymond Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In Proceedings of the
Twenty-Fifth AAAI Conference on Artificial Intelligence, pages 859-865.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734, Doha, Qatar. Association for Computational Linguistics.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning, December 2014.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 326-335.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. 2018. Speaker-follower models for vision-and-language navigation. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montreal, Canada, pages 3318-3329.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249-256, Chia Laguna Resort, Sardinia, Italy. PMLR.
Jonathan Gray, Kavya Srinet, Yacine Jernite, Haonan Yu, Zhuoyuan Chen, Demi Guo, Siddharth Goyal, C. Lawrence Zitnick, and Arthur Szlam. 2019. CraftAssist: A framework for dialogue-enabled interactive agents. arXiv preprint arXiv:1907.08584.
Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. 2019. Hierarchical decision making by generating and following natural language instructions. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems
2598
2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 10025-10034.
Hsin-Yuan Huang, Eunsol Choi, and Wen-tau Yih. 2019. FlowQA: Grasping flow in history for conversational machine comprehension. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Nikolai Ilinykh, Sina Zarrieß, and David Schlangen. 2019. Meet Up! A corpus of joint activity dialogues in a visual environment. In Proceedings of the 23rd Workshop on the Semantics and Pragmatics of Dialogue - Full Papers, London, United Kingdom. SEMDIAL.
Yacine Jernite, Kavya Srinet, Jonathan Gray, and Arthur Szlam. 2019. CraftAssist instruction parsing: Semantic parsing for a Minecraft assistant. arXiv preprint arXiv:1905.01978.
Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh. 2019. CoDraw: Collaborative drawing as a testbed for grounded goal-driven communication. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6495-6513, Florence, Italy. Association for Computational Linguistics.
Seokhwan Kim, Luis Fernando D'Haro, Rafael E. Banchs, Jason D. Williams, and Matthew Henderson. 2016a. The fourth dialog state tracking challenge. In Dialogues with Social Robots - Enablements, Analyses, and Evaluation, Seventh International Workshop on Spoken Dialogue Systems, IWSDS 2016, Saariselka, Finland, January 13-16, 2016, volume 427 of Lecture Notes in Electrical Engineering, pages 435-449. Springer.
Seokhwan Kim, Luis Fernando D'Haro, Rafael E. Banchs, Jason D. Williams, Matthew Henderson, and Koichiro Yoshino. 2016b. The fifth dialog state tracking challenge. In 2016 IEEE Spoken Language Technology Workshop, SLT 2016, San Diego, CA, USA, December 13-16, 2016, pages 511-517. IEEE.
Reginald Long, Panupong Pasupat, and Percy Liang. 2016. Simpler context-dependent logical forms via model projections. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1456-1465, Berlin, Germany. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. 2013. Learning to parse natural language commands to a robot control system. In