SlowGuess/ACL_2020_OCR huggingface dataset:
Kavya Srinet*
ksrinet@fb.com
Yacine Jernite*†
yacine@
Jonathan Gray
jsgray@fb.com
aszlam@fb.com
We propose a semantic parsing dataset focused on instruction-driven communication with an agent in the game Minecraft<sup>1</sup>. The dataset consists of 7K human utterances and their corresponding parses. Given proper world state, the parses can be interpreted and executed in game. We report the performance of baseline models, and analyze their successes and failures.
Semantic parsing is used as a component for natural language understanding in human-robot interaction systems (Lauria et al., 2001; Bos and Oka, 2007; Tellex et al., 2011; Matuszek et al., 2013; Thomason et al., 2019), and for virtual assistants (Campaigna et al., 2017; Kollar et al., 2018; Campaigna et al., 2019). We would like to be able to apply deep learning methods in this space, as recently researchers have shown success with these methods for semantic parsing more generally, e.g. (Dong and Lapata, 2016; Jia and Liang, 2016; Zhong et al., 2017). However, to fully utilize powerful neural network approaches, it is necessary to have large numbers of training examples. In the space of human-robot (or human-assistant) interaction, the publicly available semantic parsing datasets are small. Furthermore, it can be difficult to reproduce the end-to-end results (from utterance to action in the environment) because of the wide variety of robot setups and proprietary nature of personal assistants.
In this work, we introduce a new semantic parsing dataset for human-bot interactions. Our "robot" or "assistant" is embodied in the sandbox construc
tion game Minecraft², a popular multiplayer openworld voxel-based crafting game. We also provide the associated platform for executing the logical forms in game.
Situating the assistant in Minecraft has several benefits for studying task oriented natural language understanding (NLU). Compared to physical robots, Minecraft allows less technical overhead irrelevant to NLU, such as difficulties with hardware and large scale data collection. On the other hand, our bot has all the basic in-game capabilities of a player, including movement and placing or removing voxels. Thus Minecraft preserves many of the NLU elements of physical robots, such as discussions of navigation and spatial object reference.
Working in Minecraft may enable large scale human interaction because of its large player base, in the tens of millions. Furthermore, although Minecraft's simulation of physics is simplified, the task space is complex. While there are many atomic objects in the game, such as animals and block-types, that require no perceptual modeling, the player also interacts with complex structures made up of collections of voxels such as a "house" or a "hill". The assistant cannot apprehend them without a perceptual system, creating an ideal test bed for researchers interested in the interactions between perception and language.
Our contributions in the paper are as follows:
Grammar: We develop a grammar over a set of primitives that comprise a mid-level interface to Minecraft for machine learning agents.
Data: We collect 7K crowd-sourced annotations of commands generated independent of our grammar. In addition to the natural language commands and the associated logical forms, we release the tools used to collect these, which allow
$^{2}$ https://minecraft.net/en-us/. We limit ourselves to creative mode for this work
† Work done while at Facebook AI Research
<sup>1</sup>Minecraft features: ©Mojang Synergies AB included courtesy of Mojang AB
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4693-4714 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: The basic structure of the ACTIONSequence branch of the assistant's grammar. The gold octagon is an internal node whose children are ordered, blue rectangles are regular internal nodes, and green rectangles are categorical leaf nodes. Not all combinations of children of ACTION are possible, see the full list of possible productions (and the productions for PUT MEMORY and GET MEMORY) in the Appendix C.
crowd-workers to efficiently and accurately annotate parses.
Models: We show the results of several neural semantic parsing models trained on our data.
Execution: Finally, we also make available the code to execute logical forms in the game, allowing the reproduction of end-to-end results. This also opens the door to using the data for reinforcement and imitation learning with language. We also provide access to an interactive bot using these models for parsing<sup>3</sup>.
2 The Assistant Grammar
In this section we summarize a grammar for generating logical forms that can be interpreted into programs for the agent architecture described in (Gray et al., 2019).
2.1 Agent Action Space
The assistant's basic functions include moving, and placing and destroying blocks. Supporting these basic functions are methods for control flow and memory manipulation.
Basic action commands: The assistant can MOVE to a specified location; or DANCE with a specified sequence of steps. It can BUILD an object from a known schematic (or by making a copy of a block-object in the world) at a given location, or DESTROY an existing object. It can DIG a hole of a given shape at a specified location, or FILL one up. The agent can also be asked to complete a partially built structure however it sees fit by FREEBUILD.
Figure 2: The basic structure of internal nodes in the assistant's grammar. Blue rectangles are internal nodes, green rectangles are categorical leaf nodes, and red ovals are span nodes.
Finally, it can SPAWN a mob (an animate NPC in Minecraft).
Control commands: Additionally, the agent can STOP or RESUME an action, or UNDO the result of a recent command. Furthermore, the assistant can LOOP given a task and a stop-condition. Finally, it needs to be able to understand when a sentence does not correspond to any of the above mentioned actions, and map it to a NOOP.
Memory interface: Finally, the assistant can interact with its SQL based memory. It can place or update rows or cells, for example for tagging objects. This can be considered a basic version of the self-improvement capabilities in (Kollar et al., 2013; Thomason et al., 2015; Wang et al., 2016, 2017). It can retrieve information for question answering similar to the VQA in (Yi et al., 2018).
2.2 Logical Forms
The focus of this paper is an intermediate representation that allows natural language to be interpreted into programs over the basic actions from the previous section. The logical forms (represented as trees) making up this representation consist of three basic types of nodes: "internal nodes" that can have children, "categorical" (leaf) nodes that belong to a fixed set of possibilities, and "span" nodes that point to a region of text in the natural language utterance. The full grammar is shown in the Appendix C; and a partial schematic representation is shown in Figures 1 and 2. In the paragraphs below, we give more detail about some of the kinds of nodes in the grammar.
3 Instructions can be found at http://craftassist.io/acl2020demo, requires a Minecraft license and client.
Input: dig a circular hole over there until you find gold
Tool 1
Input: Dig a circular hole over there until you find gold
Q 1. What action is being requested? If multiple separate actions are being requested (e.g. "do X and then do Y"), select "Multiple separate actions"
Answer: Dig (tooltip: Digging of some kind needs to be done)
Q 2. How many times should this action be performed? Answer: Repeated until a certain condition is met
Q 2.1: Until the assistant reaches a specific object(s) / area Answer: Dig a circular hole over there until you find gold
Tool 1: Action type DIG has been chosen
Q 1.1: Select words specifying what needs to be dug Answer: Dig a circular hole over there until you find gold
Q 1.2 Select words specifying where the thing will be dug Answer: Dig a circular hole over there until you find gold
Tool 2
Please specify details of the location at which the digging will happen. Q 1. Where should the digging happen?
Answer: The location is represented using an indefinite noun like 'there' or 'over here'
Please specify details of the thing that needs to be dug.
Q1. Click on all properties of the thing to be dug mentioned in the highlighted text.
Answer 1.1: Name
1.1: Select all words that indicate the name of the thing to be dug Answer: Dig a circular hole over there until you find gold
Answer 1.2: Some other property not mentioned above
1.2: Select all words for this property
Answer: Dig a circular hole over there until you find gold.
Figure 3: A representation of the annotation process using the web-based annotation tool described in Section 3.1.3. The colors of the boxes correspond to annotation tasks. The highlighting on the text in the header of the later tasks is provided by a previous annotator. We show more detailed screenshots of how the tool works in Appendix B.3
We emphasize that this is an intermediate representation. The logical forms do not come with any mechanism for generating language, and nodes do not correspond in any simple way with words. On the other hand, the logical forms do not encode all of the information necessary for execution without the use of an interpreter that can access the assistant's memory and the Minecraft world state.
Internal nodes: Internal nodes are nodes that allow recursion; although most do not require it. They can correspond to top-level actions, for example BUILD; in which case they would just be an "action" node with "action_type" build; see Figure 1. They can also correspond to arguments to top-level actions, for example a "reference_object", which specifies an object that has a spatial location. Internal nodes are not generally required to have children; it is the job of the interpreter to deal with under-specified programs like a BUILD with no arguments.
In addition to the various LOCATION, REFERENCE OBJECT, SCHEMAIC, and REPEAT nodes which can be found at various levels, another notable sub-tree is the action's STOP CONDITION, which essentially allows the agent to understand "while" loops (for example: "dig down until you hit the bedrock" or "follow me").
Leaf nodes: Eventually, arguments have to be specified in terms of values which correspond to (fixed) agent primitives. We call these nodes categorical leaves (green rectangles in Figures 1 and 2). As mentioned above, an "action" internal node
has a categorical leaf child which specifies the action type. There are also repeat type nodes similarly specifying a kind of loop for example in the REPEAT sub-tree corresponding to "make three houses" the repeat type for specifies a "for" loop). There are also location type nodes specifying if a location is determined by a reference object, a set of coordinates, etc.; relative direction nodes that have values like "left" or "right". The complete list of categorical nodes is given in the Appendix C.
However, there are limits to what we can represent with a pre-specified set of hard-coded primitives, especially if we want our agent to be able to learn new concepts or new values. Additionally, even when there is a pre-specified agent primitive, mapping some parts of the command to a specific value might be better left to an external module (e.g. mapping a number string to an integer value). For these reasons, we also have span leaves (red ovals in Figure 2). For example, in the parse for the command "Make three oak wood houses to the left of the dark grey church.", the SCHEMAIC (an internal node) might be specified by the command sub-string corresponding to its name by the span "houses" and the requested block type by the span "oak wood". The range of the for loop is specified by the REPEAT's for value ("three"), and the REFERENCE OBJECT for the location is denoted in the command by its generic name and specific color with spans "church" and "dark grey".
The root: The root of the tree has three productions: PUT MEMORY, and GET MEMORY, corre
Figure 4: Frequency of each action type in the different data collection schemes described in Section 3.1.
sponding to writing to memory and reading from memory; and HUMAN_GIVE_COMMAND which also produces an ACTIONSequence, which is a special internal node whose children are ordered; multiple children correspond to an ordered sequence of commands ("build a house and then a tower"). In Figures 1 and 2 we show a schematic representation for an ACTION_SEQUENCE.
3 The CAIP Dataset
This paper introduces the CraftAssist Instruction Parsing (CAIP) dataset of English-language commands and their associated logical forms (see Appendix D for examples and Appendix C for a full grammar specification).
3.1 Collected Data
We collected natural language commands written by crowd-sourced workers in a variety of settings. The complete list of instructions given to crowd-workers in different settings, as well as step-by-step screen-shot of the annotation tool, are provided in the Appendix B. The basic data cleanup is described in Appendix A.
3.1.1 Image and Text Prompts
We presented crowd-sourced workers with a description of the capabilities of an assistant bot in
Figure 5: Histograms showing distribution over number of nodes in a logical form (top) and utterance length in words (bottom) for each data type. Prompts averages 6.74 nodes per logical form, 7.32 words per utterance, and interactive averages 4.89, 3.42 respectively
a creative virtual environment (which matches the set of allowed actions in the grammar), and (optionally) some images of a bot in a game environment. They were then asked to provide examples of commands that they might issue to an in-game assistant. We refer to these instructions as "prompts" in the rest of this paper.
3.1.2 Interactive Gameplay
We asked crowd-workers to play creative-mode Minecraft with our assistant bot, and they were instructed to use the in-game chat to direct the bot as they chose. The game sessions were capped at 10 minutes and players in this setting had no prior knowledge of the bot's capabilities or the grammar. We refer to these instructions as "Interactive" in the rest of this paper. The instructions of this setting are included in Appendix B.2.
3.1.3 Annotation Tool
Both prompts and interactive instructions come without a reference logical form and need to be annotated. To facilitate this process, we designed a multi-step web-based tool which asks users a series of multiple-choice questions to determine the semantic content of a sentence. The responses to some questions will prompt other more specific questions, in a process that mirrors the hierarchical structure of the grammar. The responses are then processed to produce the complete logical form. This allows crowd-workers to provide annotations with no knowledge of the specifics of the grammar described above. A pictorial representation of the annotation process is shown in Figure 3 and
a more detailed explanation of the process along with screen-shots of the tool is given in Appendix B.3.
We used a small set of tasks that were representative of the actual annotations to select skilled crowd-sourced workers by manually verifying the accuracy of responses on these.
Each utterance in our collection of prompts and interactive chats was shown to three different qualified annotators and we included the utterance and logical form in the dataset only if at least 2 out of 3 qualified annotators agreed on the logical form output. The total number of utterances sent to turkers was 6,775. Out of these, 6,693 had at least 2/3 agreements on the logical form and were kept. Of these, 2,872 had 3/3 agreements.
The final dataset has 4,532 annotated instructions from the prompts setting (Section 3.1.1), and 2,161 from interactive play (Section 3.1.2). The exact instructions shown to Turkers in the annotation tools are reproduced in Figures 9 and 11 in supplementary.
As in (Yih et al., 2016), we have found that careful design of the annotation tool leads to significant improvements in efficiency and accuracy. In particular, we re-affirm the conclusion from (Yih et al., 2016) that having each worker do one task (e.g. labeling a single node in the tree) makes annotation easier for workers.
3.2.1 Action Frequencies
Since the different data collection settings described in Section 3.1 imposed different constraints and biases on the crowd-sourced workers, the distribution of actions in each subset of data is therefore different. The action frequencies of each subset are shown in Figure 4.
3.2.2 Grammar coverage
Some crowd-sourced commands describe an action that is outside the scope of the grammar. To account for this, users of the annotation tool are able to mark that a sentence is a command to perform an action that is not covered by our grammar yet. The resulting trees are labeled as OTHERACTION, and their frequency in each dataset is shown in Figure 4. Annotators still have the option to label other nodes in the tree, such as the action's LOCATION or REFERENCE OBJECT. In both the prompts and interactive data, OTHERACTION amounted to approximately  $14\%$  of the data.
3.2.3 Quantitative analysis
For each of our data types, Figure 5 show a histogram of sentence length and number of nodes. On an average interactive data has shorter sentences and smaller trees.
3.2.4 Qualitative Linguistic Style
We show the linguistic styles and choice of words of the data sources by displaying the surface forms of a set of trees. We randomly picked trees of size (number of nodes) 7 that appear in both data sources, and then for the same tree structure, we looked at the utterances corresponding to that tree. We show some representative examples in table 1. We show more examples of the data in the Appendix D
There have been a number of datasets of natural language paired with logical forms to evaluate semantic parsing approaches, e.g. (Price, 1990; Tang and Mooney, 2001; Cai and Yates, 2013; Wang et al., 2015; Zhong et al., 2017). The dataset presented in this work is an order of magnitude larger than those in (Price, 1990; Tang and Mooney, 2001; Cai and Yates, 2013) and is similar in scale to the datasets in (Wang et al., 2015), but smaller than (Zhong et al., 2017).
In addition to mapping natural language to logical forms, our dataset connects both of these to a dynamic environment. In (Lauria et al., 2001; Bos and Oka, 2007; Tellex et al., 2011; Matuszek et al., 2013; Thomason et al., 2019) semantic parsing has been used for interpreting natural language commands for robots. In our paper, the "robot" is embodied in the Minecraft game instead of in the physical world. In (Boye et al., 2006) semantic parsing has been used for spoken dialogue with an embodied character in a 3-D world with pattern matching and rewriting phases. In our work, the user along with the assistant is embodied in game and instructs using language. We go from language to logical forms end-to-end with no pattern match necessary. Semantic parsing in a voxel-world recalls (Wang et al., 2017), where the authors describe a method for building up a programming language from a small core via interactions with players.
We demonstrate the results of several neural parsing models on our dataset. In particular, we show the results of a re-implementation of (Dong
Prompts	bot move to where the tree is	dig a large size hole to put these waste particles into the hole	please build a sphere on that location	hey bot can you dig a 5 by 5 hole for me
Interactive	find tree	dig large hole	build a sphere over here	dig a 5 x 5 hole
Table 1: Choice of words across different data sources for the same logical form (per column).
and Lapata, 2016) adapted to our grammar, and a straightforward fine-tuned BERT model (Devlin et al., 2018). There have been several other papers proposing neural architectures for semantic parsing, for example (Jia and Liang, 2016; Zhong et al., 2017; Wang et al., 2018; Hwang et al., 2019); in particular (Hwang et al., 2019) uses a BERT based model. In those papers, as in this one, the models are trained with full supervision of the mapping from natural language to logical forms, without considering the results of executing the logical form (in this case, the effect on the environment of executing the actions denoted by the logical form). There has been progress towards "weakly supervised" semantic parsing (Artzi and Zettlemoyer, 2013; Liang et al., 2016; Guu et al., 2017) where the logical forms are hidden variables, and the only supervision given is the result of executing the logical form. There are now approaches that have shown promise without even passing through (discrete) logical forms at all (Riedel et al., 2016; Neelakantan et al., 2016). We hope that the dataset introduced here, which has supervision at the level of the logical forms, but whose underlying grammar and environment can be used to generate essentially infinite weakly supervised or execution rewards, will also be useful for studying these models.
Minecraft, especially via the MALMO project (Johnson et al., 2016) has been used as a base environment for several machine learning papers. It is often used as a testbed for reinforcement learning (RL) (Shu et al., 2017; Udagawa et al., 2016; Alaniz, 2018; Oh et al., 2016; Tessler et al., 2017). In these works, the agent is trained to complete tasks by issuing low level actions (as opposed to our higher level primitives) and receiving a reward on success. Others have collected large-scale datasets for RL and imitation learning (Guss et al., 2019a,b). Some of these works (e.g. (Oh et al., 2017)) do consider simplified, templated language as a method for compositably specifying tasks, but training an RL agent to execute the scripted primitives in our grammar is already nontrivial, and so the task space and language in those works is more constrained
than what we use here. Nevertheless, our work may be useful to researchers interested in RL (or imitation): using our grammar and executing in game can supply (hard) tasks and descriptions, and demonstrations. Another set of works (Kitaev and Klein, 2017; Yi et al., 2018) have used Minecraft for visual question answering with logical forms. Our work extends these to interactions with the environment. Finally, (Allison et al., 2018) is a more focused study on how a human might interact with a Minecraft agent; our collection of free generations (see 3.1.1) includes annotated examples from similar studies of players interacting with a player pretending to be a bot.
In order to assess the challenges of the dataset, we implement two models which learn to read a sentence and output a logical form by formulating the problem as a sequence-to-tree and a sequence-to-sequence prediction task respectively.
5.1 Sequence to Tree Model
Our first model adapts the Seq2Tree approach of (Dong and Lapata, 2016) to our grammar. In short, a bidirectional RNN encodes the input sentence into a sequence of vectors, and a decoder recursively predicts the tree representation of the logical form, starting at the root and predicting all of the children of each node based on its parent and left siblings and input representation.
Sentence Encoder and Attention: We use a bidirectional GRU encoder (Cho et al., 2014) which encodes a sentence of length  $T$  s =  $(w_{1},\dots w_{T})$  into a sequence of  $T$  dimension  $d$  vectors:
$$
f _ {G R U} (\mathbf {s}) = \left(\mathbf {h} _ {1}, \dots , \mathbf {h} _ {T}\right) \in \mathbb {R} ^ {d \times T}
$$
Tree Decoder: The decoder starts at the root, computes its node representation and predicts the state of its children, then recursively computes the representations of the predicted descendants. Similarly to Seq2Tree, a node representation  $\mathbf{r}_n$  is computed based on its ancestors and left siblings. We also found it useful to condition each of the node
4698
representation on the encoder output explicitly for each node. Thus, we compute the representation  $\mathbf{r}_{n_t}$  and recurrent hidden state  $\mathbf{g}_{n_t}$  for node  $n_t$  as:
$$
\mathbf {r} _ {n _ {t}} = \operatorname {a t t n} \left(\mathbf {v} _ {n _ {t}} + \mathbf {g} _ {n _ {t - 1}}, \left(\mathbf {h} _ {1}, \dots , \mathbf {h} _ {T}\right); \mathbf {M} ^ {\sigma}\right) \tag {1}
$$