SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\mathbf {g} _ {n t} = f _ {r e c} \left(\mathbf {g} _ {n t - 1}, \left(\mathbf {v} ^ {\prime} _ {n t} + \mathbf {r} _ {n t}\right)\right) \tag {2}
$$
Where attn is multi-head attention,  $\mathbf{M}^{\sigma}\in \mathbb{R}^{d\times d\times K}$  is a tree-wise parameter,  $f_{rec}$  is the GRU recurrence function, and  $\mathbf{v}'_{n_t}$  is a node parameter (one per category for categorical nodes), and  $n_{t - 1}$  denotes either the last predicted left sibling if there is one or the parent node otherwise.
Prediction Heads: Finally, the decoder uses the computed node representations to predict the state of each of the internal, categorical, and span nodes in the grammar. We denote each of these sets by  $\mathcal{I}$ ,  $\mathcal{C}$  and  $\mathcal{S}$  respectively, and the full set of nodes as  $\mathcal{N} = \mathcal{I} \cup \mathcal{C} \cup \mathcal{S}$ .
First, each node in  $\mathcal{N}$  is either active or inactive in a specific logical form. We denote the state of a node  $n$  by  $a_{n} \in \{0,1\}$ . All the descendants of an inactive internal node  $n \in \mathcal{I}$  are considered to be inactive. Additionally, each categorical node  $n \in \mathcal{C}$  has a set of possible values  $C^{n}$ ; its value in a specific logical form is denoted by the category label  $c_{n} \in \{1,\ldots ,|C^{n}|\}$ . Finally, active span nodes  $n \in S$  for a sentence of length  $T$  have a start and end index  $(s_n,e_n) \in \{1,\dots,T\}^2$ . We compute, the representations  $\mathbf{r}_n$  of the nodes as outlined above, then obtain the probabilities of each of the labels by:
$$
\forall n \in \mathcal {N}, \quad p \left(a _ {n}\right) = \sigma \left(\left\langle \mathbf {r} _ {n}, \mathbf {p} _ {n} \right\rangle\right) \tag {3}
$$
$$
\forall n \in \mathcal {C}, \quad p \left(c _ {n}\right) = \operatorname {s o f t m a x} \left(M _ {n} ^ {c} \mathbf {r} _ {n}\right) \tag {4}
$$
$$
\forall n \in \mathcal {S}, \quad p (s _ {n}) = \operatorname {s o f t m a x} \left(\mathbf {r} _ {n} ^ {\mathrm {T}} M _ {n} ^ {s} \left(\mathbf {h} _ {1}, \dots , \mathbf {h} _ {T}\right)\right)
$$
$$
p \left(e _ {n}\right) = \operatorname {s o f t m a x} \left(\mathbf {r} _ {n} ^ {\mathrm {T}} M _ {n} ^ {e} \left(\mathbf {h} _ {1}, \dots , \mathbf {h} _ {T}\right)\right) \tag {5}
$$
where the following are model parameters:
$$
\forall n \in \mathcal {N}, \quad \mathbf {p} _ {n} \in \mathbb {R} ^ {d}
$$
$$
\forall n \in \mathcal {C}, \quad M _ {n} ^ {c} \in \mathbb {R} ^ {d \times d}
$$
$$
\forall n \in \mathcal {S}, (M _ {n} ^ {s}, M _ {n} ^ {e}) _ {n} \in \mathbb {R} ^ {d \times d \times 2}
$$
Let us note the parent of a node  $n$  as  $\pi(n)$ . Given Equations 3 to 5, the log-likelihood of a tree with states  $(\mathbf{a}, \mathbf{c}, \mathbf{s}, \mathbf{e})$  given a sentence  $\mathbf{s}$  is then:
$$
\begin{array}{l} \mathcal {L} = \sum_ {n \in \mathcal {N}} a _ {\pi (n)} \log (p (a _ {n})) + \sum_ {n \in \mathcal {C}} a _ {n} \log (p (c _ {n})) \\ + \sum_ {n \in \mathcal {S}} a _ {n} \left(\log \left(p \left(s _ {n}\right)\right) + \log \left(p \left(e _ {n}\right)\right)\right) \tag {6} \\ \end{array}
$$
Overall, our implementation differs from the original Seq2Tree in three ways, which we found lead to better performance in our setting. First, we replace single-head with multi-head attention. Secondly, the cross-attention between the decoder and attention is conditioned on both the node embedding and previous recurrent state. Finally, we replace the categorical prediction of the next node by a binary prediction problem: since we know which nodes are eligible as the children of a specific node (see Figures 1 and 2), we find that this enforces a stronger prior. We refer to this modified implementation as SentenceRec.
5.2 Sequence to Sequence Model
Our second approach treats the problem of predicting the logical form as a general sequence-to-sequence (Seq2Seq) task; such approaches have been used in semantic parsing in e.g. (Jia and Liang, 2016; Wang et al., 2018). We take the approach of (Jia and Liang, 2016) and linearize the output trees: the target sequence corresponds to a Depth First Search walk through the tree representation of the logical form. More specifically the model needs to predict, in DFS order, a sequence of tokens corresponding to opening and closing internal nodes, categorical leaves and their value, and span leaves with start and end sequences. In practice, we let the model predict span nodes in two steps: first predict the presence of the node, then predict the span value, using the same prediction heads as for the SentenceRec model (see Equation 5 above). With this formalism, the logical form for e.g. "build a large blue dome on top of the walls" will be:
We train a BERT encoder-decoder architecture on this sequence transduction task, where the training loss is a convex combination of the output sequence log-likelihood and the span cross-entropy loss.
Pre-trained Sentence Encoder: Finally, recent work has shown that using sentence encoder that has been pre-trained on large-scale language modeling tasks can lead to substantial performance
Acc. (std)	Inter.	Prompts
SentRec	50.08 (2.97)	64.17	42.49
DistBERT+SentRec	59.58 (3.49)	76.0	50.74
DistBERT+Seq2Seq	60.74 (3.58)	76.06	52.49
improvements (Song et al., 2019). We use the pretrained DistilBERT model of (Sanh et al., 2019) as the encoder of our sequence-to-sequence model, and also propose a version of the SentenceRec which uses it to replace the bidirectional RNN.
In this Section, we evaluate the performance of our baseline models on the proposed dataset.
Training Data: The CAIP datasets consists in a total of 6693 annotated instruction-parse pairs. In order for our models to make the most of this data while keeping the evaluation statistically significant, we create 5 different train/test splits of the data and report the average performance of models trained and evaluated on each of them. In each case, we hold out 650 examples from Prompts and 350 from Interactive for testing, and use the remaining 5693 as the training set.
Modeling Choices: For the end-to-end trained SentenceRec model, we use a 2-layer GRU sentence encoder and all hidden layers have dimension  $d = 256$ . We use pre-trained word embeddings computed with FastText with subword information (Bojanowski et al., 2017). The decoder uses a GRU recurrent cell and 4-headed attention. The Seq2Seq model uses a variant of the bert-base-uncased provided in the Transformer library with 6 encoding and decoding layers. For the Seq2Seq model and the SentenceRec with pre-trained encoder, we use the distilbert-base-uncased encoder from the same library. The Seq2Seq model uses beam search decoding with 15 beams. All models are trained with the Adam optimizer with quadratic learning rate decay. We provide our model and training code along with the dataset for reproducibility purposes.
Overview of Results: Table 2 provides the average accuracy (computed as the proportion of logical forms that are entirely accurately predicted) and standard deviation across all five splits, as well as the contributions of the Interactive and Prompts
Table 2: Average accuracy over a test set of 650 Prompts + 350 Interactive.
N=2	N=5	N=15
Joint	67.7	72.76	75.7
Interactive	83.83	88.34	90.63
Prompts	59.02	64.37	67.66
Table 3: Recall at  $\mathbf{N}$  for the Seq2Seq model beam search.
Figure 6: We show nodes in the grammar which are most often wrongly predicted, with false positive (+) and false negative counts (-).
data. The first observation is that using a pretrained encoder leads to a significant improvement, with a 10 point boost in accuracy. On the other hand, while the Seq2Seq model is more general and makes less use of our prior knowledge of the structure of logical forms, it does marginally better than the recursive prediction model (although within one standard deviation).
Secondly, although the models are trained on more data provided from the Prompts setting than from Interactive play, they all do better on the latter. This is consistent with previous observations on the dataset statistics in Section 3.2.3 which find that players tend to give shorter instructions with simpler execution. Finally, we note that one of the advantages of having the parser be part of an interactive agent is that it can ask the player for clarification and adapt its behavior when it is made aware of a mistake (Yao et al., 2019). In that spirit, Table 3 provides Recall at  $N$  numbers, which represent how often the true parse is within the  $N$  first elements of the beam after beam search. Recall at 2 does provide a consistent boost over the accuracy of a single prediction, but even the full size 15 beam does not always contain the right logical form.
Error Analysis: We further investigate the errors of the Seq2seq models on one of the data splits. We find that the model still struggles with span predictions: out of 363 errors, 125 only make mistakes on spans (and 199 get the tree structure right
but make mistakes on leaves). Figure 6 shows the nodes which are most commonly mistaken, with the number of false positive and false negatives out of these 363 mistakes. Unsurprisingly, the most commonly confused span leaf is "has_tag", which we use as a miscellaneous marker. Aside from that "has_tag" however, the span mistakes are evenly spread over all other leaves. The next most common source of mistakes comes from the model struggling between identifying whether a provided location corresponds to the target of the action or to the reference object, and to identify instructions which imply a repetition. The former indicates a lack of compositionality in the input representation: the model correctly identifies that a location is mentioned, but fails to identify its context. Repeat conditions on the other hand challenge the model due to the wide variety of possible stop condition, a problem we suggest future work pay special attention to.
In this work, we have described a grammar over a mid-level interface for a Minecraft assistant. We then discussed the creation of a dataset of natural language utterances with associated logical forms over this grammar that can be executed in-game. Finally, we showed the results of using this new dataset to train several neural models for parsing natural language instructions. Consistent with recent works, we find that BERT pre-trained models do better than models trained from scratch, but there is much space for improvement. We believe this data will be useful to researchers studying semantic parsing, especially interactive semantic parsing, human-robot interaction, and even imitation and reinforcement learning. The code, dataset and annotation tools described in the paper have been open-sourced<sup>5</sup>.
$^{5}$ https://github.com/facebookresearch/craftassist/tree/master/acl2020submission
Stephan Alaniz. 2018. Deep reinforcement learning with model learning and monte carlo tree search in mycraf. arXiv preprint arXiv:1803.08456.
Fraser Allison, Ewa Luger, and Katja Hofmann. 2018. How players speak to an intelligent game character using natural language messages. Transactions of the Digital Games Research Association, 4(2).
Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49-62.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL, 5:135-146.
Johan Bos and Tetsushi Oka. 2007. A spoken language interface with a mobile robot. Artificial Life and Robotics, 11(1):42-47.
Johan Boye, Joakim Gustafson, and Mats Wiren. 2006. Robust spoken language understanding in a computer game. Speech Commun., 48:335-353.
Qingqing Cai and Alexander Yates. 2013. Large-scale semantic parsing via schema matching and lexicon extension. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 423-433.
Giovanni Campagna, Rakesh Ramesh, Silei Xu, Michael Fischer, and Monica S Lam. 2017. Almond: The architecture of an open, crowdsourced, privacy-preserving, programmable virtual assistant. In Proceedings of the 26th International Conference on World Wide Web, pages 341-350.
Giovanni Campagna, Silei Xu, Mehrad Moradshahi, Richard Socher, and Monica S Lam. 2019. Genie: A generator of natural language semantic parsers for virtual assistant commands. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 394-410.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar. A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1724-1734.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. arXiv preprint arXiv:1601.01280.
Jonathan Gray, Kavya Srinet, Yacine Jernite, Haonan Yu, Zhuoyuan Chen, Demi Guo, Siddharth Goyal, C Lawrence Zitnick, and Arthur Szlam. 2019. Craftassist: A framework for dialogue-enabled interactive agents. arXiv preprint arXiv:1907.08584.
William H Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanie Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, et al. 2019a. The minerl competition on sample efficient reinforcement learning using human priors. arXiv preprint arXiv:1904.10079.
William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. 2019b. Minerl: a large-scale dataset of apache demonstrations. arXiv preprint arXiv:1907.13440.
Kelvin Guu, Panupong Pasupat, Evan Zheran Liu, and Percy Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. arXiv preprint arXiv:1704.07926.
Wonseok Hwang, Jinyeung Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on wikisql with table-aware word contextualization. arXiv preprint arXiv:1902.01069.
Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. arXiv preprint arXiv:1606.03622.
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. 2016. The malmo platform for artificial intelligence experimentation. In *IJCAI*, pages 4246-4247.
Nikita Kitaev and Dan Klein. 2017. Where is misty? interpreting spatial descriptors by modeling regions in space. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 157-166.
Thomas Kollar, Danielle Berry, Lauren Stuart, Karolina Owczarzak, Tagyoung Chung, Lambert Mathias, Michael Kayser, Bradford Snow, and Spyros Matsoukas. 2018. The alexa meaning representation language. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), volume 3, pages 177-184.
Thomas Kollar, Jayant Krishnamurthy, and Grant P Strimel. 2013. Toward interactive grounded language acquisition. In Robotics: Science and systems, volume 1, pages 721-732.
Stanislao Lauria, Guido Bugmann, Theocharis Kyriacou, Johan Bos, and A Klein. 2001. Training personal robots using natural language instruction. IEEE Intelligent systems, 16(5):38-45.
Chen Liang, Jonathan Berant, Quoc Le, Kenneth D Forbus, and Ni Lao. 2016. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. arXiv preprint arXiv:1611.00020.
Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. 2013. Learning to parse natural language commands to a robot control system. In *Experimental Robotics*, pages 403-415. Springer.
Arvind Neelakantan, Quoc V Le, Martin Abadi, Andrew McCallum, and Dario Amodei. 2016. Learning a natural language interface with neural programmer. arXiv preprint arXiv:1611.08945.
Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. 2016. Control of memory, active perception, and action in mycraf. arXiv preprint arXiv:1605.09128.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. 2017. Zero-shot task generalization with multi-task deep reinforcement learning. arXiv preprint arXiv:1706.05064.
Patti J Price. 1990. Evaluation of spoken language systems: The atis domain. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Sebastian Riedel, Matko Bosnjak, and Tim Rocktäschel. 2016. Programming with a differentiable forth interpreter. CoRR, abs/1605.06640.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Tianmin Shu, Caiming Xiong, and Richard Socher. 2017. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. arXiv preprint arXiv:1712.07294.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. MASS: masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 5926-5936.
Lappoon R Tang and Raymond J Mooney. 2001. Using multiple clause constructors in inductive logic programming for semantic parsing. In European Conference on Machine Learning, pages 466-477. Springer.
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and
mobile manipulation. In Twenty-Fifth AAAI Conference on Artificial Intelligence.
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J Mankowitz, and Shie Mannor. 2017. A deep hierarchical approach to lifelong learning in apacheft. In AAAI, volume 3, page 6.
Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidson, Justin Hart, Peter Stone, and Raymond J Mooney. 2019. Improving grounded natural language understanding through human-robot dialog. arXiv preprint arXiv:1903.00122.
Jesse Thomason, Shiqi Zhang, Raymond J Mooney, and Peter Stone. 2015. Learning to interpret natural language commands through human-robot dialog. In Twenty-Fourth International Joint Conference on Artificial Intelligence.
Hiroto Udagawa, Tarun Narasimhan, and Shim-Young Lee. 2016. Fighting zombies in mycrafth with deep reinforcement learning. Technical report, Technical report, Stanford University.
Sida I Wang, Samuel Ginn, Percy Liang, and Christopher D Manning. 2017. Naturalizing a programming language via interactive learning. pages 929-938.
Sida I Wang, Percy Liang, and Christopher D Manning. 2016. Learning language games through interaction. arXiv preprint arXiv:1606.02447.
Wenlu Wang, Yingtao Tian, Hongyu Xiong, Haixun Wang, and Wei-Shinn Ku. 2018. A transferlearnable natural language interface for databases. arXiv preprint arXiv:1809.02649.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1332-1342.
Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. 2019. Model-based interactive semantic parsing: A unified framework and A text-to-sql case study. CoRR, abs/1910.05389.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. 2018. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. In Advances in Neural Information Processing Systems, pages 1039-1050.
Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201-206.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103.
A Basic Data Cleanup
We threw away all duplicate commands in the dataset and only got annotations for unique commands from each data source.
We performed post-processing on the text by first inserting spaces between any special character (brackets, “;”, “x”) followed by alphanumeric character. For example “make a 5x5 hole” was post-processed to “make a 5 x 5 hole” and “go to  $(1,2,3)$ ” to “go to  $(1,2,3)$ ". We then used the tokenizer from spaCy to tokenize every word in the sentence.
When constructing logical forms: we threw away any keys with values: 'None', 'Other' or 'Not Specified'. Our tool allows workers to select these options when annotating. We skipped stopwords and articles like 'a', 'an' etc when constructing spans of children. We reordered the indices of words in spans to always be from left to right (regardless of which order the words were selected in the sentence when annotating).
For commands annotated as "composite" (meaning a command that requires multiple actions), we set up another tool where we asked crowd-sourced workers to split the composite command into individual commands. Each of these commands were then sent to our web-based tool described in 3.1.3 and the results were combined together under the key: "action_sequence" by preserving the order. So in the sentence: "jump twice and then come to me", we first have the sentence split into commands: "jump twice" and "come to me" and then combine their logical forms together under "action_sequence" so we first have the "Dance" action followed by "Move" action. This tool is described in Section B.4.
B Crowd-sourced task and tools instructions
This section covers details of each crowd sourced task we've described in the paper along with screenshots of the web-based annotation tool described in 3.1.
B.1 Image and Text Prompts
In this task we showed a screenshot of the bot and environment to the crowd-sourced workers and asked them to give us free-form commands for the assistant. The instructions shown to workers are shown in 7.
Figure 7: The task instructions shown to crowdsourced workers for the Image and text prompts task
To Proceed
1. Please make sure you have downloaded Minecraft and have a valid license. Follow the instructions here to change your version to 1.12.  
2. Press the green Launch button to access the loading screen.
3. After 1-2 minutes, you will see an IP address, like 123.123.123.123. Open Minecraft, navigate to Multiplayer > Direct Connect, enter the address, and click Join Server.
4. Play with the bot for 10 minutes. When you are done, exit Minecraft or press Escape > Disconnect.
5. Please fill out the short survey on the next page.
Please know that
Facebook AI Research will process the log of your game actions and in-game chats in accordance with our Data Policy. The log of your in-game actions and chats with the bot will be recorded for research purposes, and may be used by and/or shared with third parties in connection with this research. This may involve public disclosure of the log as part of a research paper or data set. We will take measures to remove any information that directly identifies you before doing so, but cannot guarantee that the logs will be completely anonymous. Do not send sensitive or personally identifiable Information (for example, the name, address, phone number) in chats to the bot, and do not build structures with personally identifiable information (for example, writing your name in blocks). Facebook's Community Standards apply and you may not use any racist, sexist, or otherwise offensive language. If you violate our policies you may be blocked.
Figure 8: The task instructions shown to crowdsourced workers for the interactive game play
B.2 Interactive Gameplay