SlowGuess/NAACL_2022_OCR huggingface dataset:
$$
\hat {u} _ {i} ^ {n + 1} = \operatorname {C r o s s A t t n} _ {u} ^ {n} \left(\hat {u} _ {i} ^ {n}, \left\{\hat {w} _ {j} ^ {m} \right\}\right), \tag {6}
$$
$$
\hat {w} _ {j} ^ {m + 1} = \operatorname {C r o s s A t t n} _ {w} ^ {m} \left(\hat {w} _ {j} ^ {m}, \left\{\hat {u} _ {i} ^ {n} \right\}\right), \tag {7}
$$
The cross-attention sub-layer is used to exchange the information and align the entities between the two modalities in order to learn joint cross-modality representations. Then the output of the cross-attention sub-layer is processed by one feed-forward sub-layer to obtain  $\{u_i^{n + 1}\}_{i = 1}^s$  and  $\{w_j^{m + 1}\}_{j = 1}^{1089}$ , which will be passed to the following single-modality modules.
Finally, we obtain a set of word vectors,  $\{\hat{u}_i^{N_T + 1}\}_{i = 1}^s$ , and a set of grid vectors,  $\{\hat{w}_j^{N_G + 1}\}_{j = 1}^{1089}$ , that is,  $U^{N_T}$  and  $W^{N_G}$ . Since the value of  $N_G$  and  $N_T$  could be different, the modality with more layers would keep using the last single modality module's output of another modality as the input of its cross modality modules, as shown in the Figure 2.
4.4 Slot Decoder
The Slot Decoder contains three linear projection layers of trainable parameters,  $M_L \in \mathbb{R}^{d_c'}$ ,  $M_C \in \mathbb{R}^{6 \times d_w}$ ,  $M_T \in \mathbb{R}^{d_a \times d_w}$  where  $d_a$  is the number of action types to predict. We compute the average of  $U^{N_T} \in \mathbb{R}^{s \times d_w}$  alongside the  $s$ -dimension to obtain  $u \in \mathbb{R}^{d_w}$ . Then we compute location logits, color logits, and action type logits:
$$
\hat {l} = \operatorname {s o f t m a x} \left(M _ {L} \cdot W ^ {N _ {G}}\right), \tag {8}
$$
$$
\hat {c} = \operatorname {s o f t m a x} \left(M _ {C} \cdot u\right), \tag {9}
$$
$$
\hat {t} = \operatorname {s o f t m a x} \left(M _ {T} \cdot u\right), \tag {10}
$$
where softmax functions are used to map the extracted information into  $\hat{l} \in \mathbb{R}^{1089}$ ,  $\hat{c} \in \mathbb{R}^6$ , and  $\hat{t} \in \mathbb{R}^{d_a}$ .
In this section, we first compare our model against the baseline for the collaborative building task where models only need to learn the instruction
following task (in Sec. 5.1). Then, we train our model to learn when to ask and evaluate on our extended Minecraft Dialogue Corpus (in Sec. 5.2). Finally, we evaluate our model's ability on the combination of the two above-mentioned tasks (in the Sec. 5.3). All training details are reported in the Appendix. The software and data are available at: https://github.com/ZhengxiangShi/LearnToAsk.
Table 3: Evaluation on the collaborative building task.
Model	Metric	Augmentation
None	2x	4x	6x
BAP model	F1	16.1	18.7	18.4	18.2
Recall	12.6	15.2	14.3	15.0
Precision	22.4	24.5	25.7	23.3
Ours (GloVe)	F1	35.0	36.5	37.8	39.4
Recall	28.3	30.1	31.4	33.4
Precision	45.8	46.2	47.6	48.1
Ours (BERT)	F1	34.5	30.1	30.4	35.4
Recall	26.7	23.6	23.4	27.9
Precision	48.7	42.6	43.5	48.5
5.1 Collaborative Building Task
Settings. In this task, the models are only trained to generate a sequence of actions without the need of considering Ask for clarifications and Others. We first compare our model against the only baseline<sup>1</sup> named BAP (Jayannavar et al., 2020) by only using the "Execution (Original)" dataset as in Table 2. Then, we conduct the experiments by using the augmented data from Jayannavar et al. (2020): the models are trained and evaluated with the augmented training data: 5,563 (indicated as 2x), 9,272 (4x), and 12,981 (6x) training samples. We provide the ground-truth previous actions and the world state for the next action prediction. For the sake of fairness, we retrain the BAP model under the same setting.
For our models, we present the performance of two different dialogue context encoders: we use the pre-trained GloVe word embeddings with 300 dimensions (Pennington et al., 2014) as the initial word embeddings followed by a GRU (Chung et al., 2014) and contextual word embeddings using the pre-trained BERT base model (Devlin et al., 2019). For the action type slot, we pre-define three potential values: placement, removal, and stop. The value of the location slot can be one of 1,089 candidate voxels and the value of the color slot can
be one of six candidate colors. During training we minimize the sum of the cross entropy losses of the location slot, the color slot, and the action type slot. The F1-score metric on the test set is used to evaluate model performance by comparing the model predictions against the action sequence performed by the human builder.
Results. In Table 3 we present the results of our model and the baselines for the collaborative building task on the Minecraft Corpus Dataset. Experimental results show that our model outperforms the baseline model with a large margin. Results on the augmented dataset show that the advantage of the data augmentation is not obvious. The performance using contextualized word embeddings is poorer. This could be due to the size of the builder model with the BERT encoder which makes it more difficult to train.
5.2 Learning to Ask Task
Settings. In this task, all the models are trained only to predict one type of actions, Execution, Ask for clarifications and Others, without the need to generate a sequence of actions. All datasets in the Table 2 are used. In our model, for the action type slot, we define three potential slot values: Execution, Ask, and Others. In this experiment, the slots for location and color are not used. We use the pretrained GloVe embeddings in the dialogue context encoder. During the training, the cross entropy loss of the action type is minimized.
Table 4: Test Accuracy of the learning to ask task.
Test Accuracy(%)	Prediction	Size
Execute	Ask	Others
Oracle	Execution	93.81	4.33	1.86	1616
Ask	22.09	63.80	14.11	163
Others	35.79	36.89	27.32	366
Overall Test Acc	80.05	2145
Results. In Table 4, we present the results of our model. Although our model achieves around  $80\%$  overall test accuracy, the correct answers mainly come from the execution type while the model struggles with the ask and other types. These two types have in fact a joint test accuracy of  $38.6\%$ . Experimental results demonstrate that the difficulty of the learning to ask task and that there is still a large room for improvement.
'https://github.com/prashant-jayan21/  
minecraft-bap-models
Figure 3: Case study of the collaborative building task in Sec. 5.1:  $A$  and  $B$  represents the architect and the builder.
5.3 Joint Learning Task
Settings. In this task, the models are trained to not only predict one type of actions from Execution, Ask for clarifications and Others but also to generate a sequence of actions. All the datasets in Table 2 are used. In our model, for the action type slot, we pre-define five potential values: Placement, Removal, Stop, Ask, and Others. The value of the location slot can be one of 1,089 candidate grid and the value of the color slot is one of six candidate colors. We still use pre-train GloVe embedding in the dialogue context encoder. During the training we minimize the sum of the cross entropy losses of the location slot, the color slot, and the action type slot with weights equal to 0.1, 0.1, 0.8. We provide the ground-truth previous actions and the world state for the next action prediction.
Table 5: Test accuracy of the joint task.
Test Accuracy(%)	Prediction
Execution	Ask	Others
Oracle	Execution	82.64	8.64	8.72
Ask	8.61	60.93	30.46
Others	25.47	47.07	31.46
Overall Test Acc	72.26
Table 6: The evaluation of the joint learning task.
F1	Recall	Precision
Ours	28.4	20.9	43.9
Results. In Table 5, we present the results of our model's test accuracy for each action type. The model has an  $72.3\%$  test accuracy. However, if the execution of building actions is excluded, its joint test accuracy of ask and other action types is about  $40.5\%$ , indicating that deciding when to take the initiative remains challenging. In Table 6, we also report the results for the building task. Not surprisingly, the performance of our model drops slightly compared to those in Table 3, reflecting the difficulty of jointly learning.
Although our model can predict the actions more accurately than the baseline, for example our model can usually predict the color of the blocks correctly with about  $60\%$  test accuracy rate, it is still nontrivial for our model to predict the whole action sequence correctly. In Figure 3, the architect instructed the builder to build a  $3 \times 3$  square and then our model generated only parts of the structure successfully.
The dataset noise makes the learning process more challenging: the builder action sequences are noisy due to, for example, the builder miss-clicking in the construction process (Narayan-Chen, 2020). Also, builder action sequences are often fragmented between utterances due to the frequent interruptions of the architect. In order to solve these issues a good model should be capable to learn better representations for higher-level abstractions in natural language like spatial relation concepts and be more robust to noisy actions (Shi et al., 2022). However, existing models including pre-trained ones (Devlin et al., 2019) fail to learn such representations for spatial reasoning, which translates into poor performance in these instruction following tasks.
In this paper, we extend the Minecraft Corpus dataset by labelling builder utterances into eight types, in which two of them are relevant to clarification questions. This allows builder models to learn to take initiatives in the instruction following tasks. Also, we have proposed a new model that achieves state-of-the-art performance on the Minecraft collaborative building task with a large improvement. Besides these contributions, we introduce a new learning to ask task for clarification questions and a jointly learning task with the collaborative building task. We leave the generation of clarification questions to future work.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. TOUCHDOWN: natural language navigation and spatial reasoning in visual street environments. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019.
Ta-Chung Chi, Minmin Shen, Mihail Eric, Seokhwan Kim, and Dilek Hakkani-Tür. 2020. Just ask: An interactive learning framework for vision and language navigation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.
Samuel Coope, Tyler Farghly, Daniela Gerz, Ivan Vulic, and Matthew Henderson. 2020. Span-ConveRT: Few-shot span extraction for dialog with pretrained conversational representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron C. Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Yue Feng, Aldo Lipani, Fanghua Ye, Qiang Zhang, and Emine Yilmaz. 2022. Dynamic schema graph fusion network for multi-domain dialogue state tracking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Yue Feng, Yang Wang, and Hang Li. 2021. A sequence-to-sequence approach to dialogue state tracking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. 2019. Minerl: A large-scale dataset of apache demonstrations. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019.
Michael Janner, Karthik Narasimhan, and Regina Barzilay. 2018. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguistics, 6.
Prashant Jayannavar, Anjali Narayan-Chen, and Julia Hockenmaier. 2020. Learning to execute instructions in a Minecraft dialogue. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. 2016. The malmo platform for artificial intelligence experimentation. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016.
To Eun Kim and Aldo Lipani. 2022. A Multi-Task Based Neural Model to Simulate Users in Goal-Oriented Dialogue Systems. Association for Computing Machinery, New York, NY, USA.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Nikita Kitaev and Dan Klein. 2017. Where is misty? interpreting spatial descriptors by modeling regions in space. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
Royi Lachmy, Valentina Pyatkin, and Reut Tsarfaty. 2021. Draw me a flower: Grounding formal abstract structures stated in informal natural language. arXiv preprint arXiv:2106.14321.
Charlotte Lambert, Ariel Cordes, Eli Kaplan, Prashant Jayannavar, and Julia Hockenmaier. 2019. Virtual world context encoding for grounded dialogue in apache.
Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. 2017. Learning through dialogue interactions by asking questions. In ICLR.
Dipendra Misra, John Langford, and Yoav Artzi. 2017. Mapping instructions and visual observations to actions with reinforcement learning. In EMNLP.
Ishan Misra, Ross B. Girshick, Rob Fergus, Martial Hebert, Abhinav Gupta, and Laurens van der Maaten. 2018. Learning by asking questions. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018.
Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. 2019. Collaborative dialogue in Minecraft. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Anjali Yuan Narayan-Chen. 2020. Towards collaborative dialogue in Minecraft. Ph.D. thesis, University of Illinois at Urbana-Champaign.
Khanh Nguyen and Hal Daume III. 2019. Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. In EMNLP-IJCNLP.
Pin Ni, Gangmin Li, Patrick CK Hung, and Victor Chang. 2021a. Staresgru-cnn with cmedlms: A stacked residual gru-cnn with pre-trained biomedical language models for predictive intelligence. Applied Soft Computing.
Pin Ni, Yuming Li, Gangmin Li, and Victor Chang. 2020. Natural language understanding approaches based on joint task of intent detection and slot filling for iot voice interaction. Neural Computing and Applications, 32(20).
Pin Ni, Yuming Li, Gangmin Li, and Victor Chang. 2021b. A hybrid siamese neural network for natural language inference in cyber-physical systems. ACM Transactions on Internet Technology (TOIT), 21(2).
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In EMNLP.
Homero Roman Roman, Yonatan Bisk, Jesse Thomason, Asli Celikyilmaz, and Jianfeng Gao. 2020. Rmm: A recursive mental model for dialogue navigation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1732-1745.
Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. 2022. Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts. In Association for the Advancement of Artificial Intelligence.
Tianmin Shu, Caiming Xiong, and Richard Socher. 2018. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. In ICLR.
Alane Suhr and Yoav Artzi. 2018. Situated mapping of sequential instructions to actions with single-step reward observation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Alane Suhr, Claudia Yan, Jack Schluger, Stanley Yu, Hadi Khader, Marwa Mouallem, Iris Zhang, and Yoav Artzi. 2019. Executing instructions in situated collaborative interactions. In EMNLP-IJCNLP.
Arthur Szlam, Jonathan Gray, Kavya Srinet, Yacine Jernite, Armand Joulin, Gabriel Synnaeve, Douwe Kiela, Haonan Yu, Zhuoyuan Chen, Siddharth Goyal, et al. 2019. Why build an assistant in apache? arXiv preprint arXiv:1907.09273.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In EMNLP-IJCNLP.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2020. Vision-and-dialog navigation. In Conference on Robot Learning. PMLR.
Xi Wang, Iadh Ounis, and Craig Macdonald. 2020. Negative confidence-aware weakly supervised binary classification for effective review helpfulness classification. CIKM '20, page 1565-1574, New York, NY, USA. Association for Computing Machinery.
Xi Wang, Iadh Ounis, and Craig Macdonald. 2021. Leveraging review properties for effective recommendation. In Proceedings of the Web Conference 2021, WWW '21, page 2209-2219, New York, NY, USA. Association for Computing Machinery.
Wei Wei, Quoc Le, Andrew Dai, and Jia Li. 2018. AirDialogue: An environment for goal-oriented dialogue research. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Jason D Williams, Matthew Henderson, Antoine Raux, Blaise Thomson, Alan Black, and Deepak Ramachandran. 2014. The dialog state tracking challenge series. AI Magazine, 35(4).
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Tsung-Yen Yang, Andrew Lan, and Karthik Narasimhan. 2020. Robust and interpretable grounding of spatial references with relation networks. In Findings of the Association for Computational Linguistics: EMNLP 2020.
Yi Zhu, Yue Weng, Fengda Zhu, Xiaodan Liang, Qixiang Ye, Yutong Lu, and Jianbin Jiao. 2021. Self-motivated communication agent for real-world visiondialog navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
Training Details. We examine our model on the extend Minecraft Dialogue Corpus dataset. Our model's hyper-parameters are fixed for all three experiments in the Sec 5 as follows. The number of 3D-conv layers  $k$  is 3, the dimension of each grid representation  $d_{c}$  is 300, the number of layers of the grid cross-modality modules  $N_{G}$  is 4, and the number of layers of the text cross-modality modules  $N_{T}$  is 2. The max length of the dialogue context  $s$  is selected as 100 and the dropout rates are all set to 0.2. The number of heads for the attention mechanism in the text single and cross modality modules is set to 2, while the number of heads is set to 1 for the attention mechanism in the grid single and cross modality modules. The cross entropy loss from the location slot is not counted if the ground truth label of the action type is not 'placement' or 'removal', and the cross entropy loss from the color slot is not counted if the ground truth label of the action type is not 'placement'. The ground-truth of previous actions and the world state is provided to the models during the training and testing. For the experiment in the Sec 5.2 and 5.3, we randomly sample from 'Ask' and 'Others' sets in the training set to make training samples of different action types ('Execution', 'Ask', and 'Others') in the training set balanced. We train our model with cross entropy loss functions of all slots and a batch size of 50, using Adam optimizer (Kingma and Ba, 2015) with a learning rate of 1e-6,  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.99$ . We train our model with 50 epochs and select the model with the highest F1-score on the valid set.