SlowGuess/NAACL_2022_OCR huggingface dataset:
Training. We trained VDTN by jointly minimizing  $\mathcal{L}_{dst}$  and  $\mathcal{L}_{bb/seg}$ . We trained all models using the Adam optimizer (Kingma and Ba, 2015) with a warm-up learning rate period of 1 epoch and the learning rate decays up to 160 epochs. Models are selected based on the average  $\mathcal{L}_{dst}$  on the validation set. To standardize model sizes, we selected embedding dimension  $d = 128$  for all models, and experimented with both shallow ( $N = 1$ ) and deep networks ( $N = 3$ ) (by stacking attention or RNN blocks), and 8 attention heads in Transformer backbones. We implemented models in Pytorch and released the code and model checkpoints<sup>1</sup>. Refer to Appendix C for more training details.
Evaluation. We followed the unimodal DST task (Budzianowski et al., 2018; Henderson et al., 2014a) and used the state accuracy metric. The prediction is counted as correct only when all the component values exactly match the oracle values. In multimodal states, there are both discrete slots (object attributes) as well as continuous slots (temporal start and end time). For continuous slots, we followed (Hu et al., 2016; Gao et al., 2017) by using Intersection-over-Union (IoU) between predicted temporal segment and ground-truth segment. The predicted segment is counted as correct if its IoU with the oracle is more than  $p$ , where we chose  $p = \{0.5, 0.7\}$ . We reported the joint state accuracy of discrete slots only ("Joint Acc") as well as all slot values ("Joint Acc IoU@p"). We also reported the performance of component state predictions, including predictions of object identities  $o_j$ , object slot tuples  $(o_j, s_i, v_{i,j})$ , and object state tuples  $(o_j, s_i, v_{i,j}) \forall s_i \in S$ . Since a model may simply output all possible object identities and slot values and achieve 100% component accuracies, we reported the F1 metric for each of these component predictions.
Overall results. From Table 2, we have the following observations:
- we noted that simply using naive retrieval models such as  $Q$ -retrieval achieved zero joint state accuracy only. State prior achieved only about  $15\%$  and  $8\%$  F1 on object identities and object slots, showing that a model cannot simply rely on distribution bias of dialogue states.
Model	Dial.	Vid.	Obj Ident. F1	Obj Slot F1	Obj State F1	Joint Acc	Acc IoU @.5	Acc IoU @.7
Q-retrieval	✓	-	6.7	3.3	2.7	1.0	0.8	0.7
State prior	-	-	14.9	7.7	0.1	0.0	0.0	0.0
Object (rand.)	-	✓	19.8	14.1	0.4	0.0	0.0	0.0
Object (all)	-	✓	60.5	27.2	1.5	0.0	0.0	0.0
RNN(V)	-	✓	21.2	10.8	8.3	1.0	0.1	0.1
RNN(D)	✓	-	57.8	43.3	38.0	4.8	1.1	0.6
RNN(V+D)	✓	✓	63.2	48.5	42.8	6.8	2.6	2.3
RNN(V+D)+Att	✓	✓	73.4	59.0	46.8	8.5	3.3	2.0
TRADE (N=1)	✓	-	75.3	63.2	47.8	8.7	2.2	1.1
TRADE (N=1)	✓	✓	75.8	63.8	48.0	9.2	3.3	2.5
TRADE (N=3)	✓	-	74.2	62.6	47.2	8.3	2.1	1.1
TRADE (N=3)	✓	✓	76.1	64.5	48.2	8.9	3.2	2.4
UniConv (N=1)	✓	-	70.6	58.0	44.7	11.1	4.5	3.2
UniConv (N=1)	✓	✓	73.6	60.5	46.2	11.6	6.1	5.4
UniConv (N=3)	✓	-	76.4	62.7	52.5	15.0	6.4	4.6
UniConv (N=3)	✓	✓	76.4	62.7	50.5	14.5	7.8	7.0
NADST (N=1)	✓	-	78.0	63.8	44.9	11.6	4.6	3.2
NADST (N=1)	✓	✓	78.4	64.0	47.7	12.7	6.1	5.5
NADST (N=3)	✓	-	80.6	67.3	50.2	15.3	6.3	4.3
NADST (N=3)	✓	✓	79.0	65.1	49.2	13.3	6.3	5.5
VDTN	✓	✓	84.5	72.8	60.4	28.0	15.3	13.1
VDTN+GPT2	✓	✓	85.2	76.4	63.7	30.4	16.8	14.3
Table 2: Performance (in %) of VDTN vs. baselines on the test split of DVD-DST.  $\checkmark$  on "Dial" or "Vid" column indicates whether we use dialogue context or video context respectively.
- The results of Object (random/all) show that in DVD-DST, dialogues often focus on a subset of visual objects and an object perception model alone cannot perform well.
- The performance gains of RNN models show the benefits of neural network models compared to retrieval models. The higher results of  $RNN(D)$  against  $RNN(V)$  showed the dialogue context is essential and reinforced the above observation.
- Comparing TRADE and UniConv, we noted that TRADE performed slightly better in component predictions, but was outperformed in joint state prediction metrics. This showed the benefits of UniConv which avoids the assumptions of conditionally independent slots and learns to extract the dependencies between slot values.
- Results of TRADE, UniConv, and NADST all displayed minor improvement when adding video inputs to dialogue inputs, displaying their weakness when exposed to cross-modality learning.
- VDTN achieves significant performance gains and achieves the SOTA results in all component or joint prediction metrics.
We also experimented with a version of VDTN in which the transformer network (Section 3.3) was
'https://github.com/henryhungle/mm.dst
Video self-supervision	Loss	Acc	IoU @0.5	IoU @0.7
None	N/A	24.8	13.8	11.8
Lobj	L1	26.0	14.4	12.4
Lobj	L2	24.1	13.3	11.4
Lobj (tracking)	L1	27.2	14.7	12.6
Lobj (tracking)	L2	22.9	12.7	10.9
Lseg	L1	28.0	15.3	13.1
Lseg	L2	27.4	14.7	12.7
Lobj + Lseg	L1	23.7	13.0	11.2
Lobj + Lseg	L2	24.3	13.4	11.6
initialized from a GPT2-base model (Radford et al., 2019) with a pretrained checkpoint released by HuggingFace. Aside from using BPE to encode text sequences to match GPT2 embedding indices, we keep other components of the model the same. VDTN+GPT2 is about  $36 \times$  bigger than our default VDTN model. As shown in Table 2, the performance gains of VDTN+GPT2 indicate the benefits of large-scale language models (LMs). Another benefit of using pretrained GPT2 is faster training time as we observed the VDTN+GPT2 converged much earlier than training it from scratch. From these observations, we are excited to see more future extension of SOTA unimodal DST models (Lin et al., 2021; Dai et al., 2021) and large pretrained LMs (Brown et al., 2020; Raffel et al., 2020), especially ones with multimodal learning such as (Lu et al., 2019; Zhou et al., 2020), to MM-DST task.
Impacts of self-supervised video representation learning. From Table 3, we noted that compared to a model trained only with the DST objective  $\mathcal{L}_{dst}$ , models enhanced with self-supervised video understanding objectives can improve the results. However, we observe that L1 loss works more consistently than L2 loss in most cases. Since L2 loss minimizes the squared differences between predicted and ground-truth values, it may be susceptible to outliers (of segment features or bounding boxes) in the dataset. Since we could not control these outliers, an L1 loss is more suitable.
We also tested with  $\mathcal{L}_{obj}$  (tracking), in which we used oracle bounding box labels during training, and simply passed the features of all objects to VDTN. This modification treats the self-supervised
Table 3: Accuracy (in %) by self-supervised objectives.  $\mathcal{L}_{obj}$  (tracking) assumes access to oracle bounding box labels and treats the self-supervised learning task as an object tracking task.
Video Features	Dialogue State	Video loss	Acc	IoU @0.5	IoU @0.7
Xbb	B\time	-	17.9	N/A	N/A
Xbb + Xcnn	B\time	-	22.4	N/A	N/A
Xbb	B	-	19.3	11.0	9.5
Xbb + Xcnn	B	-	24.8	13.8	11.8
Xbb	B	Lobj	24.0	12.9	11.0
Xbb + Xcnn	B	Lobj	26.0	14.4	12.4
Xbb + Xcnn	B	Lseg	28.0	15.3	13.1
Table 4: Accuracy (in %) by video features and state formulations
learning task as an object tracking task in which all output representations are used to predict the ground-truth bounding box coordinates of all objects. Interestingly, we found  $\mathcal{L}_{obj}$  (tracking) only improves the results insignificantly, as compared to the self-supervised learning objective  $\mathcal{L}_{obj}$ . This indicates that our self-supervised learning tasks do not strongly depend on the availability of object boundary labels.
Finally, we found combining both segment-level and object-level self-supervision is not useful. This is possibly due to our current masking strategy that masks object and segment features independently. Therefore, the resulting context features might not be sufficient for recovering masked representations. Future work can be extended by studying a codependent masking technique to combine segment-based and object-based representation learning.
Impacts of video features and time-based slots.
Table 4 shows the results of different variants of VDTN models. We observed that:
- Segment-based learning is marginally more powerful than object-based learning.
- By considering the temporal placement of objects and defining time-based slots, we noted the performance gains by "Joint Obj State Acc" ( $\mathcal{B}$  vs.  $\mathcal{B} \backslash \text{time}$ ). The performance gains show the interesting relationships between temporal slots and discrete-only slots and the benefits of modelling both in dialogue states.
- Finally, even with only object-level features  $X_{bb}$ , we still observed performance gains from using self-supervised loss  $\mathcal{L}_{obj}$ , confirming the benefits of better visual representation learning.
<sup>2</sup>https://huggingface.co/gpt2
Figure 3: Ablation results of VDTN and baselines by dialogue turn positions ( $x$  axis)
Ablation analysis by turn positions. Figure 3 reported the results of VDTN predictions of states that are separated by the corresponding dialogue positions. The results are from the VDTN model trained with both  $\mathcal{L}_{dst}$  and  $\mathcal{L}_{seg}$ . As expected, we observed a downward trend of results as the turn position increases. We noted that state accuracy reduces more dramatically (as shown by "Joint Acc") than the F1 metrics of component predictions. For instance, "Object Identity F1" shows almost stable performance lines through dialogue turns. Interestingly, we noted that the prediction performance of dialogue states with temporal slots only deteriorates dramatically after turn 2 onward. We expected that VDTN is able to learn short-term dependencies (1-turn distance) between temporal slots, but failed to deal with long-term dependencies ( $>1$ -turn distance) between temporal slots. In all metrics, we observed VDTN outperforms both RNN baseline and UniConv (Le et al., 2020b), across all turn positions. However, future work is needed to close the performance gaps between lower and higher turn positions.
Impacts on downstream response prediction task. Finally, we tested the benefits of studying multimodal DST for a response prediction task. Specifically, we used the best VDTN model to predict dialogue states across all samples in DVD-DST. We then used the predicted slots, including object identities and temporal slots, to select the video features. The features are the visual objects and segments that are parts of the predicted dialogue states. We then used these selected features as input to train new Transformer decoder models which are added with an MLP as the response prediction layer. Note that these models are trained only with a cross-entropy loss to predict answer candidates.
Dialogue State	Response Accuracy
No state	43.0
B\time	46.8/47.1
B	48.7/48.9
Table 5: Accuracy (in %) of response predictions (by greedy/beam search states)
From Table 5, we observed the benefits of filtering visual inputs by predicted states, with up to  $5.9\%$  accuracy score improvement<sup>3</sup>. Note that there are more sophisticated approaches such as neural module networks (Hu et al., 2018) and symbolic reasoning (Chen et al., 2020) to fully exploit the decoded dialogue states. We leave these extensions for future research.
For more experiment results, analysis, and qualitative examples, please refer to Appendix D.
Compared to conventional DST (Mrkšić et al., 2017; Lei et al., 2018b; Gao et al., 2019; Le et al., 2020c), we show that the scope of DST can be further extended to a multimodal world. Compared to prior work in multimodal dialogues (Das et al., 2017a; Hori et al., 2019; Thomason et al., 2019) which focuses more on vision-language interactions, our work was inspired from a dialogue-based strategy with a formulation of a dialogue state tracking task. For more comparison to related work, please refer to Appendix A.
We noted the current work are limited to a synthetic benchmark with a limited video domain (3D objects). However, we expect that MM-DST task is still applicable and can be extended to other video domains (e.g. videos of humans). We expect that MM-DST is useful in dialogues centered around a "focus group" of objects. For further discussion of limitations, please refer to Appendix E.
In summary, in this work, we introduced a novel MM-DST task that tracks visual objects and their attributes mentioned in dialogues. For this task, we experimented on a synthetic benchmark with videos simulated in a 3D environment and dialogues grounded on these objects. Finally we proposed VDTN, a Transformer-based model with self-supervised learning objectives on object and segment-level visual representations.
3The response prediction performance is lower than the results reported by DVD (Le et al., 2021b) as the training splits are not the same; DVD-DST has about  $10\mathrm{x}$  smaller training data than (Le et al., 2021b).
During the research of this work, there is no human subject involved and hence, no ethical concerns regarding the experimental procedures and results. The data is used from a synthetically developed dataset, in which all videos are simulated in a 3D environment with synthetic non-human visual objects. We intentionally chose this dataset to minimize any distribution bias and make fair comparisons between all baseline models.
However, we wanted to emphasize on ethical usage of any potential adaptation of our methods in real applications. Considering the development of AI in various industries, the technology introduced in this paper may be used in practical applications, such as dialogue agents with human users. In these cases, the adoption of the MM-DST task or VDTN should be strictly used to improve the model performance and only for legitimate and authorized purposes. It is crucial that any plan to apply or extend MM-DST in real systems should consider carefully all potential stakeholders as well as the risk profiles of application domains. For instance, in case a dialogue state is extended to human subjects, any information used as slots should be clearly informed and approved by the human subjects before the slots are tracked.
Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Stefan Lee, Peter Anderson, Irfan Essa, Devi Parikh, Dhruv Batra, Anoop Cherian, Tim K. Marks, and Chiori Hori. 2019. Audio-visual scene-aware dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Moshe Bar. 2004. Visual objects in context. Nature Reviews Neuroscience, 5(8):617-629.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Prithvijit Chattopadhyay, Deshraj Yadav, Viraj Prabhu, Arjun Chandrasekaran, Abhishek Das, Stefan Lee, Dhruv Batra, and Devi Parikh. 2017. Evaluating visual conversational agents via cooperative human-air games. In Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP).
Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoc V. Le. 2020. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In International Conference on Learning Representations.
Yinpei Dai, Hangyu Li, Yongbin Li, Jian Sun, Fei Huang, Luo Si, and Xiaodan Zhu. 2021. Preview, attend and review: Schema-aware curriculum learning for multi-domain dialogue state tracking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 879-885, Online. Association for Computational Linguistics.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017a. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 326-335.
Abhishek Das, Satwik Kottur, José M. F. Moura, Stefan Lee, and Dhruv Batra. 2017b. Learning cooperative visual dialog agents with deep reinforcement learning. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2970-2979.
Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5503-5512.
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. 2017. Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 37-49, Saarbrücken, Germany. Association for Computational Linguistics.
Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010. Every picture tells a story: Generating sentences from images. In European conference on computer vision, pages 15-29. Springer.
Tharindu Fernando, Simon Denman, Sridha Sridharan, and Clinton Fookes. 2018. Tracking by prediction: A deep generative model for multi-person localisation and tracking. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1122-1132. IEEE.
Cora Fischer, Stefan Czoschke, Benjamin Peters, Benjamin Rahm, Jochen Kaiser, and Christoph Bledowski. 2020. Context information supports serial dependence of multiple visual objects across memory episodes. Nature communications, 11(1):1-11.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. 2017. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pages 5267-5275.
Shuyang Gao, Abhishek Sethi, Sanchit Aggarwal, Tagyoung Chung, and Dilek Hakkani-Tur. 2019. Dialog state tracking: A neural reading comprehension approach. arXiv preprint arXiv:1908.01946.
Rohit Girdhar and Deva Ramanan. 2020. Cater: A diagnostic dataset for compositional actions and temporal reasoning. In International Conference on Learning Representations.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014a. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263-272.
Matthew Henderson, Blaise Thomson, and Steve J. Young. 2014b. Robust dialog state tracking using delexicalised recurrent neural networks and unsupervised adaptation. 2014 IEEE Spoken Language Technology Workshop (SLT), pages 360-365.
C. Hori, H. Alamri, J. Wang, G. Wichern, T. Hori, A. Cherian, T. K. Marks, V. Cartillier, R. G. Lopes, A. Das, I. Essa, D. Batra, and D. Parikh. 2019. End-to-end audio visual scene-aware dialog using multimodal attention-based video features. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2352-2356.
Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. 2018. Explainable neural computation via stack neural module networks. In Proceedings of the European conference on computer vision (ECCV), pages 53-69.
Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. 2016. Natural language object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4555-4564.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatiotemporal reasoning in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2758-2766.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798.
Hyounghun Kim, Hao Tan, and Mohit Bansal. 2020. Modality-balanced models for visual dialogue. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8091-8098.
Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR).
Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi. 2021. SIMMC 2.0: A task-oriented dialog dataset for immersive multimodal conversations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4903-4912, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2018. Visual coreference resolution in visual dialog using neural module networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 153-169.
Gakuto Kurata, Bing Xiang, Bowen Zhou, and Mo Yu. 2016. Leveraging sentence-level information with encoder LSTM for semantic slot filling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2077-2083, Austin, Texas. Association for Computational Linguistics.