SlowGuess/ACL_2020_OCR huggingface dataset:
Prashant Jayannavar Anjali Narayan-Chen Julia Hockenmaier
{paj3, nrynchn2, juliahmr}@illinois.edu
The Minecraft Collaborative Building Task is a two-player game in which an Architect A instructs a Builder B to construct a target structure out of 3D blocks. We consider the task of predicting B's action sequences (block placements and removals) in a given game context, and show that capturing B's past actions as well as B's perspective leads to a significant improvement in performance on this challenging language understanding problem.
There is a long-standing interest in building interactive agents that can communicate with humans about and operate within the physical world (e.g. Winograd (1971)). The goal for agents in this scenario is to not only be able to engage in rich natural language discourse with their human conversation partners, but also to ground that discourse to physical objects, and execute instructions in the real world. Traditional dialogue scenarios are either completely ungrounded (Ritter et al., 2010; Schrading et al., 2015), focus on slot-value filling tasks (Kim et al., 2016b,a; Budzianowski et al., 2018) which instead require grounding to entities in a knowledge base, or operate within static environments, such as images (Das et al., 2017) or videos (Pasunuru and Bansal, 2018). Relevant efforts in robotics have largely focused on single-shot instruction following, and are mostly constrained to simple language (Roy and Reiter, 2005; Tellex et al., 2011) with limited resources (Thomason et al., 2015; Misra et al., 2016; Chai et al., 2018).
The recently introduced Minecraft Collaborative Building Task and the corresponding Minecraft Dialogue Corpus (Narayan-Chen et al., 2019) is one attempt to bridge this gap within the simulated game world of Minecraft. In this task, two players, an Architect (A) instructs a Builder (B) to con
struct a target structure out of multi-colored building blocks. The corpus consists of 509 game logs between humans that perform this task. Narayan-Chen et al. (2019) focus on generating Architect utterances. In this paper, we explore models for building an automated Builder agent. $^{1}$  We focus on the subtask of predicting the Builder's block placements, and leave the back-and-forth dialogue aspect of the overall task required of a fully interactive Builder agent to future work. We define the Builder Action Prediction (BAP) task in Section 2, describe our models in Section 3, an approach to augment the training data in Section 4, and our experiments in Section 5. We analyze results and highlight challenges of the BAP task in Section 6.
2 Dataset and Task
2.1 The Minecraft Dialogue Corpus
The Minecraft Dialogue Corpus (Narayan-Chen et al., 2019) consists of 509 human-human dialogues and game logs for the Minecraft Collaborative Building Task, a two-player game in a simulated Blocks World environment between an Architect (A) and a Builder (B). A is given a target structure (Target) and has to instruct B via a text chat interface to build a copy of Target on a given build region. A and B communicate back and forth via chat throughout the game (e.g. to resolve confusions or to correct B's mistakes), but only B can move blocks, while A observes B operating in the world. B is given access to an inventory of 120 blocks of six given colors that it can place and remove. The resulting dialogues consist mainly of A providing instructions, often involving multiple actions to be taken, and grounded in the Builder's perspective, while B executes those instructions and resolves
<sup>1</sup>For models and code see http://juliahmr.cs.ILLINOIS.edu/Minecraft
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2589-2602 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
A: go to the middle and place an orange block two spaces to the left
B: (puts down 1 orange)  
A: now make a staircase with 2 stairs left and 2 right with orange
B: (puts down 1 orange) A: so it will look like a v
B: (puts down 4 orange)
(a)
B: (removes 1 orange, creating a floating section)
(e)
(b)
B: (places 1 orange)
(f)
Figure 1: A sample sequence of human-human game states. The game starts with an empty grid and an initial  $\mathbf{A}$  instruction (a), which  $\mathbf{B}$  executes in the first action sequence (b) by placing a single block. In (c),  $\mathbf{B}$  begins to execute the next  $\mathbf{A}$  instruction given in (b). However,  $\mathbf{A}$  interrupts  $\mathbf{B}$  in (c), leading to two distinct  $\mathbf{B}$  action sequences: (b)-(c) (single block placement), and (c)-(h) (multiple placements and removals).
(c)
B: (removes 1 orange, creating a floating block)
(g)
(d)
$\pmb{B}$  (follows similar pattern to complete the shape)
(h)
any confusion through further dialogue. The task is complete when the structure built by  $\mathbf{B}$  (Built) matches Target (allowing for translations within the horizontal plane and rotations about the vertical axis) and lies completely within the boundaries of the predefined build region. Games in this corpus are based on 150 distinct target structures, split into disjoint test, training, and development sets such that training targets do not appear during test or development. Game logs record all utterances and  $\mathbf{B}$ 's actions (placements and removals), as well as the state of the world (i.e. the  $(\mathrm{x},\mathrm{y},\mathrm{z})$ -coordinates and colors of all blocks in the build region), and  $\mathbf{B}$ 's  $(\mathrm{x},\mathrm{y},\mathrm{z})$  position, vertical rotation (pitch) and horizontal orientation (yaw) at the points in time when an utterance was recorded or an action performed. Since there are six block colors to be placed, we distinguish seven possible types of actions  $A \in \{\text{BLUE}, \text{GREEN}, \dots, \text{YELLOW}, \text{REMOVE}\}$ .  $\mathbf{B}$  actions are 4-tuples  $\langle A, x, y, z \rangle$  consisting of an action type and cell coordinates. A block placement is feasible as long as an adjacent grid location is occupied, while REMOvE is feasible as long as that location is currently occupied by a block. These ac
tions do not include B's movement. B can assume any (continuous) 3D position and orientation, and the dataset records B's position and orientation for each individual action. But since there are many positions and orientations from which blocks in a cell can be placed, B's movement is secondary to the main task of constructing the target configuration.
2.2 The Builder Action Prediction Task
Narayan-Chen et al. (2019) focused on creating models that can generate A utterances, whereas we aim to develop models that can perform B's role. Although back-and-forth dialogue between the two players is a clear hallmark of this task, we leave the question of how to develop B agents that can decide when to speak and what to contribute to the conversation (either by way of chit-chat, verifications or clarification questions to A) to future work, and focus here on the subtask of predicting correct sequences of block placements and removals. Executing A instructions is B's primary role, and a crucial component to overall task completion.
Figure 1 shows an example from the Minecraft Dialogue Corpus that highlights some challenges
of performing this task. A can move around freely, but remains invisible to B and views the structure from behind B when giving instructions. As a result, A instructions frequently include spatial relations, both between pairs of blocks or substructures ("put ... on top of.."), and relative to B's current position and perspective ("left", "right"). A also often uses higher-level descriptions involving complex shapes (e.g. "staircase", "v"). Due to the asynchronous nature of the dialogue, A often interrupts during B action sequences. A may also provide corrections and clarifications to fix B mistakes. Producing feasible sequences of B actions requires a certain amount of planning, since blocks can only be placed in grid cells that are adjacent to other blocks or the ground, and floating structures (a common occurrence among the target structures in this corpus) can only be built if supporting blocks that are not part of the target structure are present when the floating blocks are being placed. Despite these challenges, we show below that training models that use a rich representation of the world (Section 3) on sufficient amounts of diversified data (Section 4) produces promising initial results.
To generate items for this task, we follow a similar strategy to Narayan-Chen et al. (2019), who, as a first step towards designing a fully interactive Architect, define an Architect Utterance Generation Task, where models are presented with a particular human-human game context in which a human Architect produced an utterance and are evaluated based on how well they can generate an appropriate utterance. Conversely, we define the Builder Action Prediction (BAP) Task as the task of predicting the sequence of actions (block placements and/or removals) that a human Builder performed at a particular point in a human-human game.
2.3 Evaluating Builder Action Predictions
To evaluate models for the BAP task, we compare each model's predicted action sequence  $A_{m}$  against the corresponding action sequence  $A_{h}$  that the human builder performed at that point in the game. Specifically, for each pair of model and human action sequences  $(A_{m}, A_{h})$ , where  $A_{h} = \langle a_{h}^{(1)}, \ldots a_{h}^{(k)} \rangle$  led from a world state  $W_{before}$  to a world state  $W_{h}$  and  $A_{m} = \langle a_{m}^{(1)}, \ldots a_{m}^{(l)} \rangle$  led from the same  $W_{before}$  to  $W_{m}$ , we compute an F1 score over the net actions in  $A_{h}$  and  $A_{m}$ , and report a micro-average over all sequences in the test (or development) data.
Net actions ignore actions that were undone within the same sequence, e.g. if a block was placed and then removed. We consider any  $a_{m}$  action correct if the same action (involving the same grid cell and block color) occurs among the net actions in  $A_{h}$ . There are two reasons why we evaluate net rather than all actions: first, many structures contain floating blocks which require the placement of temporary "placeholder" blocks that are later removed. placeholders' colors are arbitrary, and there are often multiple possible locations where placeholders can be put; placeholder predictions should not be penalized, as long as they enable the correct target to be built. Human Builders are also prone to making small mistakes that are immediately resolved (e.g. by removing blocks that were accidentally placed). Evaluation should be robust to this noise in the ground truth sequences.
The F1 metric ignores sequence information because it is either implicit in cases where it matters (e.g. building a vertical stack of blocks from the ground up), or irrelevant (e.g. building a line of blocks on the ground). Other metrics may also be suited for this task, but obvious choices such as an edit distance between  $W_{m}$  and  $W_{h}$  suffer from the problem that they favor models that place fewer blocks, since incorrect placements would incur twice the cost of no placements. However, our current definition of when an action is correct is relatively harsh, and could be relaxed in a number of ways. First, since it only considers an action correct if it matches a human action at the same grid cell, it penalizes cases where there are rotational equivalences between the built and the target structures (as may arise when the target has rotational symmetry). It also ignores any translational equivalences (which are very common at the beginning of a dialogue when the initial structure is empty, and may also need to be taken into account when the action sequence passes through an intermediate state in which all blocks have been removed). Second, looser F1 scores that evaluate actions only with regard to block locations (ignoring color) or colors (ignoring locations) might yield insight into how well models understand spatial relations, colors, or the number of blocks to be placed or removed. We leave exploring such variants to future work.
While our evaluation allows us to compare models directly and automatically against a common gold standard, it is important to keep in mind that such direct comparisons to human action sequences pro
vide only a lower bound on performance because they are based on the assumption that a) the human executed the instructions completely and correctly, and that b) there is only one way to execute the instructions correctly. But instructions are often vague or ambiguous: "Place a red block on the ground next to the blue block" may be resolved to any of four equally correct cells adjoining that block, and ideally, the evaluation metric should score them the same. And human action sequences do not always correspond to a complete execution of the previous instruction, e.g. when B is interrupted by A or stops to ask a question:
A: now it will be a diagonal staircase with 4 steps angling towards the middle
A: if that makes sense
$\pmb{B}$  puts down a red block
B: diagonal staircase with this orientation?
$\pmb{B}$  puts down a red block
A: towards where the yellow blocks are pointing
$\pmb{B}$  picks up 2 red blocks, puts down a red block
There is growing interest in situated collaborative scenarios involving instruction givers/followers with one-way (Hu et al., 2019; Suhr et al., 2019) and two-way (Kim et al., 2019; Ilinykh et al., 2019) communication. Here, we compare our task to related work on instruction following, both generally and within Blocks World and Minecraft.
Instruction following: Prior approaches to instruction comprehension typically take a semantic parsing approach (Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Andreas and Klein, 2015). Semantic parsing components enable human-robot understanding (Tellex et al., 2011; Matuszek et al., 2013); some approaches to interactive robot design combine these architectures with physical robot exploration to enable online learning (Thomason et al., 2015, 2016, 2017). The SCONE corpus (Long et al., 2016) features tasks in three domains requiring context-dependent sequential instruction understanding, in which a system is given a world containing several predefined objects and properties and has to predict the final world state by parsing instructions to intermediate logical forms. Some papers have also applied neural action prediction models (Suhr and Artzi, 2018; Huang et al., 2019) to SCONE. More recently, Vision-and-Language Navigation (VLN), (Anderson et al., 2018), and its dialog counterpart, Cooperative Vision-and-Dialog Navigation
(CVDN) (Thomason et al., 2019), focus on instruction following and cooperative interactions in photorealistic navigation settings.
Since our dataset does not contain any logical forms, we also cannot use semantic parsing approaches, and have to resort to neural action prediction models. However, Minecraft instructions are more challenging than the SCONE tasks because our action space is significantly larger and our utterances are more complex. Minecraft dialogues are also more complex than the sequences of instructions in SCONE because we cannot assume that actions to be executed are described in the last utterance. Minecraft dialogues are also more complex than those in CVDN, because they contain more turns, and because communication is asynchronous. Moreover, construction differs fundamentally from navigation in that construction dynamically changes the environment. While referring expressions in navigation can be safely assumed to refer to objects that exist in the world, construction instructions frequently refer to objects that need to be built by the agent. And although more recent navigation tasks require real vision, their underlying world state space (as defined by fixed viewpoints and the underlying navigation graph) is just as highly discretized. Our task does not require vision, but poses an arguably more challenging planning problem, since its action space is much larger (7623 possible actions vs. six actions in the vision-language navigation work).
Blocks World: There is a renewed interest in instruction comprehension in Blocks World scenarios. Voxelurn (Wang et al., 2017) interfaces with human users and learns to understand descriptions of voxel structures of increasing complexity, but does so by mapping them down to a core programmatic language. Bisk et al. (2016a,b, 2018) build models for understanding single-shot instructions that transform one world state to another using simulated 3D blocks. Blocks are viewed from a fixed bird's-eye perspective, initialized randomly in the initial world state, and uniquely identifiable. The varying Builder perspective and lack of easily identifiable referents, along with the need to understand utterances in a dialogue context, make our task a much more challenging problem. Unlike traditional Blocks World, Minecraft allows blocks to float (requiring nonmonotonic action sequences where placement is followed by removal), or attach to any side of an existing block.
Minecraft: Combining semantic parsing with simulated human-robot interaction, Facebook CraftAssist is a dialogue-enabled framework with an associated dataset for semantic parsing of instructions in Minecraft (Gray et al., 2019; Jernite et al., 2019; Szlam et al., 2019). Their setup enables two-way human-bot interactions in which a human architect can direct an automated builder using natural language to build complex structures. To bootstrap a semantic parser, they synthetically generate (using a hand-defined grammar) and crowdsourced natural language instructions paired with logical tree structures consisting of action primitives. In addition to lacking such annotations, our work differs fundamentally in that our data is sourced from human-human dialogues; instructions are more ambiguous, dialogues have larger variety and Builder action sequences are noisier.
Figure 2: The Builder Action Prediction model.
3 Builder Action Prediction Models
3.1 Overall architecture
Similar to e.g. the models of Suhr and Artzi (2018) for the SCONE tasks, models for the Builder Action Prediction task need to predict an appropriate, variable-length, sequence of actions (block placements and removals) in a given discourse and game context and world state. All our models (Figure 2) are based on a recurrent encoder-decoder architecture (Sutskever et al., 2014; Cho et al., 2014) in which a GRU-based encoder (bottom left box)
captures the game context (dialogue and action history), and a CNN-based encoder (top left box) captures the world state at each time step. The decoder (right box) predicts one action per time step, based on the game history, the world state at that time, and the last action taken. It consists of another GRU backbone over action sequences (bottom right), and a multi-class classifier that reads in the output of the GRU backbone as well as the world state encoding produced by the CNN to predict either the next action (block placement or removal) to be taken, or a special STOP token that terminates the action sequence. The world state representation gets updated and re-encoded after each predicted action. We now describe these components in more detail.
3.2 Game history encoder
Since  $\mathbf{B}$  only knows what blocks to place after receiving an instruction from  $\mathbf{A}$ , we can view the game history as a non-empty sequence of previous utterances (by both players), possibly interleaved with sequences of actions that were taken by  $\mathbf{B}$  in earlier turns of the game. Our experiments examine the question of how much of this history should be given to our model, but all models examined in this paper treat the game history as a single sequence of tokens. Similar to Narayan-Chen et al. (2019), we encode the dialogue history as a sequence of tokens in which each player's utterances are contained within speaker-specific start and end tokens ( $\langle A\rangle \ldots \langle \backslash A\rangle$  or  $(\langle B\rangle \ldots \langle \backslash B\rangle)$ . We also represent  $\mathbf{B}$ 's prior actions naively as tokens that capture the action type (placement or removal) and block color (e.g. as "builderPutdown_red"). The  $2\times 6 = 12$  action tokens as well as the speaker tokens are encoded using 300-dimensional random vectors, while all other tokens are encoded as 300-dimensional pre-trained GloVe word embeddings (Pennington et al., 2014). The token embeddings are passed through a GRU to produce a  $H$ -dim embedding ( $H\in \{200,300\}$ ) of the dialogue history in the GRU's final hidden state.
3.3 World state encoder
The world state is the current grid configuration that is fed into the action prediction model at each time step. We first describe how we represent the raw world state, before we explain how this representation is then encoded via a CNN-based architecture.
Input: the raw world state Minecraft blocks are unit cubes that can be placed at integer-valued
$\langle x, y, z \rangle$  locations in a 3D grid; the Collaborative Building Task restricts these to a build region of size  $11 \times 9 \times 11$ . Since we found it beneficial to explicitly capture empty grid cells, our baseline model represents each cell state as a 7-dim one-hot vector, yielding a  $11 \times 9 \times 11 \times 7$  minimal world state representation encoding the presence (or absence) of blocks at any grid cell. We also found it useful to capture the relative position of each cell with respect to  $\mathbf{B}$ 's current position and orientation, as well as which cells were affected by  $\mathbf{B}$ 's most recent actions, and augment this model in two ways:
Action history weights: Each action affects a single grid cell. Actions that follow each other often affect adjacent grid cells. We encode information about the most recent actions in our world state representation as follows: Given the chronological sequence of all actions  $A = a^{(1)}, a^{(2)} \ldots a^{(t-1)}$  that took place before the  $t$ -th action to be predicted, we assign a real-valued weight  $\alpha^{(i)}$  to each action  $a^{(i)}$  (where  $\alpha^{(i)} \leq \alpha^{(i+1)}$ ), and include these action weights in the world state representation of the corresponding cells. We truncate the action history to the last five elements, assign integer weights 1...5 to  $a^{(t-5)}, \ldots, a^{(t-1)}$  (and 0 to all  $a^{(i < t-5)}$ ), and then include these weights as a separate input feature in each cell. If a cell was affected more than once by the last five actions, we only use the weight of the most recent action. Our action weights do not distinguish between actions taken in the preceding action sequence and those in the current sequence.
Perspective coordinates: B needs to understand the spatial relations in A's instructions. Many of these relations (e.g. "left" in Figure 1) depend on B's current position  $\langle x_B, y_B, z_B \rangle$  and orientation (pitch  $\phi_B \in [-90, \dots, +90]$ , or vertical rotation, and yaw  $\gamma_B \in [-180, \dots, +180]$ , horizontal orientation). Our models assume that spatial relations in an instruction are relative to B's position at that time, and use that information to compute perspective coordinates. We calculate the relative perspective coordinates  $\langle x_c', y_c', z_c' \rangle$  of a cell  $c$  with absolute coordinates  $\langle x_c, y_c, z_c \rangle$  by moving the frame of reference from  $\langle 0, 0, 0 \rangle$  to  $\langle x_B, y_B, z_B \rangle$ , and rotating it to account for B's yaw and pitch:
$$
\langle x _ {c} ^ {\prime}, y _ {c} ^ {\prime}, z _ {c} ^ {\prime} \rangle = P \cdot Y \cdot \langle x _ {c} - x _ {B}, y _ {c} - y _ {B}, z _ {c} - z _ {B} \rangle
$$
We scale these perspective coordinates by a factor of .1 to keep their range closer to that of the cell
$$
{ } ^ { 2 } P = \left( \begin{array} { c c c } 1 & 0 & 0 \\ 0 & \cos \phi _ { B } & \sin \phi _ { B } \\ 0 & - \sin \phi _ { B } & \cos \phi _ { B } \end{array} \right) \text { ~ a n d ~ } Y = \left( \begin{array} { c c c } \cos \gamma _ { B } & 0 & - \sin \gamma _ { B } \\ 0 & 1 & 0 \\ \sin \gamma _ { B } & 0 & \cos \gamma _ { B } \end{array} \right)
$$
state and action history weights.