SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhengxiang Shi and Yue Feng and Aldo Lipani  
University College London, London, UK
{zhengxiang.shi.19,yue.feng.20,aldo.lipani}@ucl.ac.uk
Collaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. We wish to develop an intelligent builder agent in a simulated building environment (Minecraft) that can build whatever users wish to build by just talking to the agent. In order to achieve this goal, such agents need to be able to take the initiative by asking clarification questions when further information is needed. Existing works on Minecraft Corpus Dataset only learn to execute instructions neglecting the importance of asking for clarifications. In this paper, we extend the Minecraft Corpus Dataset by annotating all builder utterances into eight types, including clarification questions, and propose a new builder agent model capable of determining when to ask or execute instructions. Experimental results show that our model achieves state-of-the-art performance on the collaborative building task with a substantial improvement. We also define two new tasks, the learning to ask task and the joint learning task. The latter consists of solving both collaborating building and learning to ask tasks jointly.
Following instructions in natural language by intelligent agents to achieve a shared goal with the instructors in a pre-defined environment is a ubiquitous task in many scenarios, e.g., finding a target object in an environment (Nguyen and Daumé III, 2019; Roman et al., 2020), drawing a picture (Lachmy et al., 2021), or building a target structure (Narayan-Chen et al., 2019). A number of machine learning (ML) research projects about following instructions tasks have been initiated by making use of the video game Minecraft (Johnson et al., 2016; Shu et al., 2018; Narayan-Chen et al., 2019; Guss et al., 2019; Jayannavar et al., 2020). Building such agents requires to make
Figure 1: A simple example of builder task: The builder can observe the world state and dialogue context. For the sake of space, only a part of the dialogue history is displayed. The utterance in green displays understanding and the utterance in yellow asks a clarification question.
" Architect : Aparently, this first structure is a warm up
2 Architect : It's just 3 blocks arranged in an L shape
Builder : Oh cool
2 Architect : So two blocks next to each other on the ground
Builder : Any col
Architect : And then one more block on top of either
" Architect : They are blue!"
progress in grounded natural language understanding – understanding complex instructions, for example, with spatial relations in natural language – self-improvement – studying how to flexibly learn from human interactions – synergies of ML components – exploring the integration of several ML and non-ML components to make them work together (Szlam et al., 2019).
The recently introduced Minecraft Corpus dataset (Narayan-Chen et al., 2019) proposes a collaborative building task, in which an architect and a builder can communicate via a textual chat. Architects are provided with a target structure they want to have built, and the builders are the only ones who can control the Minecraft avatar in the virtual environment. The task consists in building 3D structures in a block world-like scenario collaboratively, as shown in the Figure 1. Earlier works in Minecraft collaborative building tasks (Jayannavar et al., 2020) attempted to build an automated builder agent with a large action space but failed to allow the builder to take the initiative in the conversation. However, an intelligent agent should
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2060 - 2070
not only understand and execute the instructor's requests but also be able to take initiatives, e.g., asking clarification questions, in case the instructions are ambiguous. In the task defined by this dataset, builders may encounter ambiguous situations that are hard to interpret by just relying on the world state information and instructions. For example, in Figure 1, we provide a simple case where the architect fails to provide sufficient information to the builder, such as the color of the blocks. In this situation, it is clearly difficult for the builder to know exactly which action should be taken. If, however, the builder is able to clarify the situation with the architect, this ambiguity can be resolved. Therefore, builders, besides following architects' instructions, should take the initiative in the conversation and ask questions when necessary.
To this end, in this paper we annotate all builder utterances in the Minecraft Corpus dataset by categorizing them in the dataset into eight dialogue utterance types as shown in Table 2, allowing the intelligent agents to learn when and what to ask given the world state and dialogue context. Particularly, a builder would ask task-level questions or instruction-level questions for further clarifications. Experimental results in the Sec. 5.2 show that determining when to ask clarification questions remains a challenging task. However, it is worth noting that the clarification questions in the Minecraft Corpus dataset are more complex and diverse than those in navigation tasks (Roman et al., 2020; Nguyen and Daumé III, 2019; Thomason et al., 2020) whose questions are relatively simpler and mainly about where to go.
Also, we propose a new automated builder agent that learns to map instructions to actions and decide when to ask questions. Our model utilizes three dialogue slots, the action type slot, the location slot, and the color slot. This solution has the benefit of making the learning easier with respect to those models that work using a large action space (Jayannavar et al., 2020). To solve the collaborative building task, both the dialogue context and the world state need to be considered. Therefore, to endow our model with the ability to better learn the representations between the world state and language, our model implements a cross-modality module, which is based on the cross attention mechanism. Experimental results on our extended Minecraft Corpus dataset show that our model achieves state-of-the-art performance with a substantial improve
ment for the collaborative building task. We also provide new baselines for learning to ask task and the joint learning of these two tasks.
Dialogue Tasks. As virtual personal assistants have now penetrated the consumer market, with products such as Siri and Alexa, the research community has produced several works on task-oriented dialogue tasks such as: hotel booking, restaurant booking, movie recommendation, etc. (Budzianowski et al., 2018; Wei et al., 2018; Wu et al., 2019; Feng et al., 2021, 2022; Kim and Lipani, 2022). These task-oriented dialogues have been modelled as slot filling tasks. These tasks consist of correctly identifying and extracting information (slots) useful to solve the task. However, most of these slot filling tasks (Coope et al., 2020; Ni et al., 2020) are considered as semantic tagging or parsing of natural language and do not normally consider visual information. Moreover, these tasks focus only on two of the many components needed by conversational systems: the Natural Language Understanding (NLU) and Dialogue State Tracking (DST) ones (Budzianowski et al., 2018; Williams et al., 2014). Beside these task-oriented dialogue tasks, the research community has also focused on instruction following dialogue tasks, such as target completion tasks (de Vries et al., 2017), object finding tasks (Roman et al., 2020), and navigation tasks (Thomason et al., 2020). Narayan-Chen et al. (2019) proposed the Minecraft Corpus dataset, where the task consists in a cooperative asymmetric task involving an architect and a builder that have to build a target structure collaboratively. Jayannavar et al. (2020) then built a builder model to follow the sequential instructions from the architect.
Multi-Modal. Almost all instruction following dialogue tasks need to consider both contextual information and actions as well as the state of the world (Suhr and Artzi, 2018; Suhr et al., 2019; Chen et al., 2019; Lachmy et al., 2021), which remains a key challenge for instruction following dialogue tasks. In particular, the Vision-and-Dialog Navigation (VDN) task (Chen et al., 2019; Thomason et al., 2020; Roman et al., 2020; Zhu et al., 2021) where the question-answering dialogue and visual contexts are leveraged to facilitate navigation, has attracted increasing research attention. Other tasks, such as moving blocks tasks (Misra et al., 2017) and object finding tasks (Janner et al.,
2018), also require the modelling of both contextual information in natural language as well as the world state representation to be solved.
Spatial Reasoning. Many instruction following dialogue tasks contain texts with spatial-temporal concepts (Chen et al., 2019; Yang et al., 2020). For instance, the Minecraft Corpus dataset contains utterances with spatial relations, e.g., "go to the middle and place an orange block two spaces to the left". Although pre-trained language models have been used successfully in a wide array of downstream tasks, interpreting and grounding abstractions stated in natural language, such as spatial relations, have not been systematically studied and remain still challenging. Therefore, another challenge for an agent is to follow instructions which require the learning and understanding of spatiotemporal linguistic concepts in natural language. To train models able to understand and reason about spatial references in natural language, Shi et al. (2022) proposed a benchmark for robust multi-hop spatial reasoning over texts.
Learning by Asking Questions. Determining whether to ask clarification questions and what to ask is critical for instruction followers to complete the tasks. Several recent studies have focused on learning a dialogue agent with the ability to interact with users by both responding to questions and by asking questions to accomplish their task interactively (Li et al., 2017; de Vries et al., 2017; Misra et al., 2018; Roman et al., 2020). For instance, de Vries et al. (2017) introduced a game to locate an unknown object via asking questions about objects in a given image. A decision-maker is introduced to learn when to ask questions by implicitly reasoning about the uncertainty of the agent. Different from earlier works (Kitaev and Klein, 2017; Suhr et al., 2019), recent works on VDN tasks propose agents that learn to ask a question when the certainty of the next action is low (Thomason et al., 2020; Roman et al., 2020; Chi et al., 2020). Roman et al. (2020) proposed a two models-based agent with a navigator model and a questioner model. The former model was responsible for moving towards the goal object, while the latter model was used to ask questions. Zhu et al. (2021) proposed an agent that learned to adaptively decide whether and what to communicate with users in order to acquire instructive information to help the navigation.
3 Dataset and Tasks
3.1 The Minecraft Dialogue Corpus
The Minecraft Dialogue Corpus (Narayan-Chen et al., 2019) is built upon a simulated block-world environment with dialogues between an architect and a builder. This consists of 509 human-human dialogues (15,926 utterances, 113,116 tokens) playing the role of an architect and a builder, and game logs for 150 target structures of varying complexity (min. 6 blocks, max. 68 blocks, avg. 23.5 blocks). For each target structure at least three dialogues are collected where each dialogue contains 30.7 utterances (22.5 architect utterances and 8.2 builder utterances) and 49.5 builder blocks movements on average.
The architect instructs about a target structure the builder to build it via a dialogue. Although the architect observes the builder operating in the world, only the builder can move blocks. The builder has access to an inventory of 120 blocks of six given colors that he or she can place or remove. The collaborative building task restricts the structures to a build region of size  $11 \times 9 \times 11$ , and contains 3709, 1331, and 1616 samples for training, validation, and test sets.
3.2 Builder Dialogue Annotation
Builders need to be able to decide their actions at any time point rather than only execute actions with the information about when to execute. Thus, we annotate all builders' utterances in the Minecraft Corpus dataset (Narayan-Chen et al., 2019) and categorize all 4,904 builder utterances into 8 utterance types. These types are partially inspired by the work of Lambert et al. (2019). Each utterance falls into exactly one category. These categories are defined as follows: (1) Instruction-level Questions: used to request that the architect clarifies previous instructions; (2) Task-level Questions: used to request the architect to give a description about the whole picture of the building task, e.g., ask for the next instruction or ask to describe how the target structure should look like; (3) Verification Questions: used to request the architect to give feedback on previous actions; (4) Greetings: used to greet each other; (5) Suggestions: used to provide suggestions about building; (6) Display Understanding: used to express whether previous instructions have been understood; (7) Status Update: used to describe the current status, e.g., tell the architect where they are, their current block stock status, or
Table 1: The taxonomy of builder utterances: we categorize them into eight types where instruction-level questions and task-level questions are both a sub-type of clarification questions. There are 4,904 builder utterances in total.
Category	Example	Amount	Percentage
Instruction-level Questions	1. What color?	914	18.64%
2. Is it flat?
Task-level Questions	1. What are we building?	252	5.14%
2. What's next?
Verification Questions	1. Like that or othwr way?	1021	20.82%
2. Is this the cross you wanted?
Greeting	1. Ready?	808	16.48%
2. Hello!
Suggestions	1. In the future we can call these donuts or something	59	1.23%
2. No problem, if it's hard to describe we can just go step by step
Display Understanding	1. No problem.	1296	26.43%
2. Knew what you meant
Status Update	1. I don't have enough green to continue.	101	2.06%
2. I'll stay with this perspective
Others	1. I got my first job from Minecraft.	453	9.24%
2. Oh, wwo, sorry!
Table 2: Statistics of the extended Minecraft Dialogue Corpus: "Execution (Original)" represents that the builder should predict a sequence of building actions given the dialogue context and the world state in a sample; "Ask for clarifications" indicates that the builder should ask for more information in order to execute building actions; "Others" stands for remaining dialogue acts for the builder such as greetings, chit-chat, and display understanding.
Train	Valid	Test
Execution (Original)	3709	1331	1616
Ask for clarifications	437	151	163
Others	837	267	366
Total	4983	1749	2145
whether they have finished a given instruction; (8) Others: not relevant to the task itself (e.g. chit-chat, expressing gratification, or apologies).
Among these 8 utterance types, the instruction-level questions and the task-level questions are a sub-type of clarification questions used to further clarify instructions or the task itself when the information from the architect is not clear or ambiguous. Based on these annotations, we extend the original dataset (the first row in Table 2) with two other dialogue acts, 'Ask for clarifications' and 'Others', as shown in the second and third rows of Table 2. Each 'Ask for clarifications' sample includes a world state and a dialogue context followed by a builder utterance labelled as instruction-level questions or task-level questions; each sample tagged as 'Others' includes a world state and a dialogue context followed by other builder utterance types.
3.3 Task Definition
Let  $\mathcal{H}$  be the set of all dialogue contexts,  $\mathcal{W}$  the set of all world states, and  $\mathcal{A}$  the set of actions, including building actions (placing a block, removing a block, or a special stop action that terminates the task) and utterance actions (ask clarification questions or other utterance categories). Given a dialogue context  $h\in \mathcal{H}$ , an initial grid-based world state  $w_{0}\in \mathcal{W}$ , the target is to predict the next action type as Execution, Ask for clarifications and Others. When the prediction of the action type is Execution, a sequence of actions  $\{a_i\}_{i = 1}^n$ , where  $a_{i}\in \mathcal{A}$  and  $a_{n}$  is the Stop action, is needed to be generated to reach the final target structure  $w_{n}$ . The action type Execution will update the world state via a deterministic transition function  $T$  such that  $w_{i} = T(w_{i - 1},a_{i})$  where  $w_{i}\in \mathcal{W}$ .
In this section we introduce the proposed builder model, as shown in Figure 2. The model comprises four major components: the utterance encoder, the world state encoder, the fusion module, and the slot decoder. The utterance encoder (in Sec. 4.1) and world state encoder (in Sec. 4.2) learn to represent the dialogue context and the world state. These encoded representations are then fed into the fusion module (in Sec. 4.3) that learns contextualized embeddings for the grid world and textual tokens through the single and cross modality modules. Finally, the learned world and text representations are mapped into the pre-defined slot-values in the slot decoder (in Sec. 4.4).
Figure 2: The model architecture. The  $\oplus$  sign represents the concatenation operation. This illustration uses the plate notation. There are a total of  $N_{T} + 1$  text single modality modules,  $N_{G} + 1$  grid single modality modules,  $N_{T}$  text cross modality modules, and  $N_{T}$  grid cross modality modules. Arrows indicate the flow of information.
4.1 Dialogue Context Encoder
We add "architect" and "builder" annotations before each architect utterance  $A_{t}$  and each builder utterance  $B_{t}$  respectively. Then, the dialogue utterances are represented as
$$
D _ {t} = \text {" a c h i c t e c t "} A _ {t} \oplus \text {" b u i l d e r "} B _ {t}
$$
at the turn  $t$ , where  $\oplus$  is the operation of sequence concatenation. The entire dialogue context is defined as:
$$
H = D _ {1} \oplus D _ {2} \oplus \dots \oplus D _ {t} \tag {1}
$$
Given the dialogue context  $H$ , we truncate the tokens from the end of the dialogue context or pad them to a fixed length as inputs and then use the dialogue context encoder to encode utterance history into  $U \in \mathbb{R}^{s \times d_w}$ , where  $d_w$  is the dimension of the word embedding and  $s$  is the maximum number of tokens for a dialogue context. The dialogue context encoder can be word embeddings like Glove Pennington et al. (2014) or contextual word embeddings Devlin et al. (2019), which are both widely used in the literature (Ni et al., 2021a,b; Wang et al., 2020, 2021).
4.2 Grid World State Encoder
The world state is represented by a voxel-based grid. We first represent each grid state as a 7-dimensional one-hot vector that stands for empty or a block having one of six colors, yielding a  $7 \times 11 \times 9 \times 11$  world state representation. Additionally, we truncate the action history to the last five
ones, assign an integer weight in  $1, \ldots, 5$  and then include these weights as a separate input feature in each grid, resulting in a raw world state input of  $W_0 \in \mathbb{R}^{8 \times 11 \times 9 \times 11}$ . We also represent the last action as an 11-dimensional vector  $a$  where the first two dimensions represent the placement or removal actions, the next six dimensions represent the color, and the last three dimensions represent the location of the last action.
The structure of the world state encoder is similar to Jayannavar et al. (2020)'s, i.e., consisting of  $k$  3D-convolutional layers  $(f_{1})$  with kernel size 3, stride 1 and padding 1, followed by a ReLU activation function. Between every successive pair of these layers there is a  $1\times 1\times 1$  3D-convolutional layer  $(f_{2})$  with stride 1 and no padding followed by ReLU:
$$
W _ {i} = \operatorname {R e L U} \left(f _ {2} ^ {i} \left(\operatorname {R e L U} \left(f _ {1} ^ {i} \left(W _ {i - 1}\right)\right)\right)\right), \tag {2}
$$
$$
W _ {k} = \operatorname {R e L U} \left(f _ {1} ^ {i} \left(W _ {k - 1}\right)\right), \tag {3}
$$
where  $i = 1,2,\ldots ,k - 1$ .  $W_{k}\in \mathbb{R}^{d_{c}\times 11\times 9\times 11}$  is the learned world grid-based representation where  $d_{c}$  is the dimension of each grid representation. Then we concatenate the last action representation  $a\in \mathbb{R}^{11}$  to each grid vectors in  $W_{k}$  and reshape them into  $W^{\prime}\in \mathbb{R}^{d_c'\times 1089}$ , where  $d_c^\prime = d_c + 11$
4.3 Fusion Module
The fusion module comprises four major components: two single modality modules and two cross-modality modules. The former modules are based
on self-attention layers and the latter on cross-attention layers. These take as input the world state representation and dialogue history representation. Between every successive pair of grid single-modality modules or text single-modality modules there is a cross modality module. We take  $N_G$  and  $N_T$  layers for the grid cross modality module and the text cross modality module. We first revisit the definition and notations about the attention mechanism (Bahdanau et al., 2015) and then introduce how they are integrated into our single modality modules and cross-modality modules.
Attention Mechanism. Given a query vector  $x$  and a sequence of context vectors  $\{y_j\}_{j=1}^K$ , the attention mechanism first computes the matching score  $s_j$  between the query vector  $x$  and each context vector  $y_j$ . Then, the attention weights are calculated by normalizing the matching score:  $a_j = \frac{\exp(s_j)}{\sum_{j=1}^K \exp(s_j)}$ . The output of an attention layer is the attention weighted sum of the context vectors:  $Attention(x, y_j) = \sum_j a_j \cdot y_j$ . Particularly, the attention mechanism is called self-attention when the query vector itself is in the context vectors  $\{y_j\}$ . We use the multi-head attention following Devlin et al. (2019); Tan and Bansal (2019).
Single-Modality Module. Each layer in a single-modality module contains a self-attention sub-layer and a feed-forward sub-layer, where the feedforward sub-layer is further composed of a linear transformation layer, a dropout layer and a normalization layer. We take  $N_G + 1$  and  $N_T + 1$  layers for the grid single-modality modules and the text single-modality modules respectively, interspersed with cross-modality module as shown in Figure 2. Since new blocks can only be feasibly placed if one of their faces touches the ground or another block in the Minecraft world, we add masks to all infeasible grids in the grid single-modality modules. For a set of text vectors  $\{u_i^n\}_{i=1}^s$  and a set of grid vectors  $\{w_j^m\}_{j=1}^{1089}$  as inputs of  $n$ -th text single-modality layer and  $m$ -th grid single-modality layer, where  $n \in \{1, \dots, N_T + 1\}$  and  $m \in \{1, \dots, N_G + 1\}$ , we first feed them into two self attention sub-layers:
$$
u _ {i} ^ {n} = \operatorname {S e l f A t t n} _ {u} ^ {n} \left(u _ {i} ^ {n}, \left\{u _ {i} ^ {n} \right\}\right), \tag {4}
$$
$$
w _ {j} ^ {m} = \operatorname {S e l f A t t n} _ {w} ^ {m} \left(w _ {j} ^ {m}, \left\{w _ {j} ^ {m} \right\}, \text {m a s k}\right) \tag {5}
$$
Lastly, the outputs of self attention modules,  $u_{i}^{n}$  and  $w_{j}^{m}$ , are followed by feed-forward sub-layers to obtain  $\hat{u}_i^n$  and  $\hat{w}_j^m$ .
Cross-Modality Module. Each layer in the cross-modality module consists of one cross-attention sub-layer and one feed-forward sub-layer, where the feed-forward sub-layers follow the same setting as the single-modality module. Given the outputs of  $n$ -th text single-modality layer,  $\{ \hat{u}_i^n \}_{i=1}^s$ , and the  $m$ -th grid single-modality layer,  $\{ \hat{w}_j^m \}_{j=1}^{1089}$  as the query and context vectors, we pass them through cross-attention sub-layers, respectively: