SlowGuess/NAACL_2022_OCR huggingface dataset:
Hung Le, Nancy F. Chen, and Steven Hoi. 2021a. Learning reasoning paths over semantic graphs for videogrounded dialogues. In International Conference on Learning Representations.
Hung Le, Doyen Sahoo, Nancy Chen, and Steven Hoi. 2019. Multimodal transformer networks for end-to-end video-grounded dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5612-5623, Florence, Italy. Association for Computational Linguistics.
Hung Le, Doyen Sahoo, Nancy Chen, and Steven C.H. Hoi. 2020a. BiST: Bi-directional spatio-temporal reasoning for video-grounded dialogues. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1846-1859, Online. Association for Computational Linguistics.
Hung Le, Doyen Sahoo, Chenghao Liu, Nancy Chen, and Steven C.H. Hoi. 2020b. UniConv: A unified conversational neural architecture for multi-domain task-oriented dialogues. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1860-1877, Online. Association for Computational Linguistics.
Hung Le, Chinnadhurai Sankar, Seungwhan Moon, Ahmad Beirami, Alborz Geramifard, and Satwik Kottur. 2021b. DVD: A diagnostic dataset for multi-step reasoning in video grounded dialogue. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5651-5665, Online. Association for Computational Linguistics.
Hung Le, Richard Socher, and Steven C.H. Hoi. 2020c. Non-autoregressive dialog state tracking. In International Conference on Learning Representations.
Hwaran Lee, Jinsik Lee, and Tae yoon Kim. 2019. Sumbt: Slot-utterance matching for universal and scalable belief tracking. In ACL.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018a. TVQA: Localized, compositional video question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1369-1379, Brussels, Belgium. Association for Computational Linguistics.
Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018b. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1437-1447.
Zekang Li, Zongjia Li, Jinchao Zhang, Yang Feng, and Jie Zhou. 2021. Bridging text and video: A universal multimodal transformer for video-audio scene-aware
dialog. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pages 1-1.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer.
Weizhe Lin, Bo-Hsiang Tseng, and Bill Byrne. 2021. Knowledge-aware graph-enhanced GPT-2 for dialogue state tracking. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7871-7881, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Xiangyang Mou, Brandyn Sigouin, Ian Steenstra, and Hui Su. 2020. Multimodal dialogue state tracking by qa approach with data augmentation. arXiv preprint arXiv:2007.09903.
Nikola Mrkšić, Diarmuid Řeaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1777-1788. Association for Computational Linguistics.
Wei Pang and Xiaojie Wang. 2020. Visual dialogue state tracking for question generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11831-11838.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer imaged-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649.
Jiaxin Qi, Yulei Niu, Jianqiang Huang, and Hanwang Zhang. 2020. Two causal principles for improving visual dialog. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10860-10869.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Osman Ramadan, Paweł Budzianowski, and Milica Gasic. 2018. Large-scale multi-domain belief tracking with knowledge sharing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, volume 2, pages 432-437.
Abhinav Rastogi, Dilek Z. Hakkani-Tür, and Larry P. Heck. 2017. Scalable multi-domain dialogue state tracking. 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 561-568.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28:91-99.
Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. 2015. A dataset for movie description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3202-3212.
Idan Schwartz, Seunghak Yu, Tamir Hazan, and Alexander G Schwing. 2019. Factor graph attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2039-2048.
Paul Hongsuck Seo, Andreas Lehrmann, Bohyung Han, and Leonid Sigal. 2017. Visual reference resolution using attention memory for visual dialog. In Advances in neural information processing systems, pages 3719-3729.
Aviv Shamsian, Ofri Kleinfeld, Amir Globerson, and Gal Chechik. 2020. Learning object permanence from video. In European Conference on Computer Vision, pages 35-50. Springer.
Yangyang Shi, Kaisheng Yao, Hu Chen, Dong Yu, Yi-Cheng Pan, and Mei-Yuh Hwang. 2016. Recurrent support vector machines for slot tagging in spoken language understanding. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 393-399, San Diego, California. Association for Computational Linguistics.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2019. Vision-and-dialog navigation. In Conference on Robot Learning (CoRL).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Tana Wang, Yaqing Hou, Dongsheng Zhou, and Qiang Zhang. 2021. A contextual attention network for multimodal emotion recognition in conversation. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-7. IEEE.
Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438-449, Valencia, Spain. Association for Computational Linguistics.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 808-819, Florence, Italy. Association for Computational Linguistics.
Saining Xie, Ross Girshick, Piotr Dólar, Zhuowen Tu, and Kaiming He. 2017. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500.
Puyang Xu and Qi Hu. 2018. An end-to-end approach for handling unknown slot values in dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1448-1457. Association for Computational Linguistics.
Yaoda Xu and Marvin M Chun. 2009. Selecting and perceiving multiple visual objects. Trends in cognitive sciences, 13(4):167-174.
Steve Young, Milica Gašić, Simon Keizer, François Mairesse, Jost Schatzmann, Blaise Thomson, and Kai Yu. 2010. The hidden information state model: A practical framework for pomdp-based spoken dialogue management. Computer Speech and Language, 24(2):150-174.
Jianguo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wang, Philip Yu, Richard Socher, and Caiming Xiong. 2020. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 154-167, Barcelona, Spain (Online). Association for Computational Linguistics.
Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive encoder for dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1458-1467, Melbourne, Australia. Association for Computational Linguistics.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13041-13049.
A Details of Related Work
Our work is related to the following domains:
A.1 Dialogue State Tracking
Dialogue State Tracking (DST) research aims to develop models that can track essential information conveyed in dialogues between a dialogue agent and human (defined as hidden information state by (Young et al., 2010) or belief state by (Mrkšić et al., 2017)). DST research has evolved largely within the domain of task-oriented dialogue systems. DST is conventionally designed in a modular dialogue system (Wen et al., 2017; Budzianowski et al., 2018; Le et al., 2020b) and preceded by a Natural Language Understanding (NLU) component. NLU learns to label sequences of dialogue utterances and provides a tag for each word token (often in the form of In-Out-Begin representations) (Kurata et al., 2016; Shi et al., 2016; Rastogi et al., 2017). To avoid credit assignment problems and streamline the modular designs, NLU and DST have been integrated into a single module (Mrkšić et al., 2017; Xu and Hu, 2018; Zhong et al., 2018). These DST approaches can be roughly categorized into two types: fixed-vocabulary or open-vocabulary. Fixed-vocabulary approaches are designed for classification tasks which assume a fixed set of (slot, value) candidates and directly retrieve items from this set to form dialogue states during test time (Henderson et al., 2014b; Ramadan et al., 2018; Lee et al., 2019). More recently, we saw more approaches toward open-vocabulary strategies which learn to generate candidates based on input dialogue context (Lei et al., 2018b; Gao et al., 2019; Wu et al., 2019; Le et al., 2020c). Our work is more related to open-vocabulary DST, but we essentially redefined the DST task with multimodality. Based on our literature review, we are the first to formally extend DST and bridge the gap between traditional task-oriented dialogues and multimodal dialogues.
A.2 Visually-grounded Dialogues
A novel challenge to machine intelligence, the intersection of vision and language research has expanded considerably in the past few years. Earlier benchmarks test machines to perceive visual inputs, and learn to generate captions (Farhadi et al., 2010; Lin et al., 2014; Rohrbach et al., 2015), ground text phrases and objects (Kazemzadeh et al., 2014; Plummer et al., 2015), and answer questions about the visual contents (Antol et al., 2015; Zhu et al., 2016; Jang et al., 2017; Lei et al., 2018a). As an orthogonal development from Visual Question Answering problems, we noted recent work that targets vision-language in dialogue context, in which an image or video is given and the dialogue utterances are centered around its visual contents (De Vries et al., 2017; Das et al., 2017a; Chattopadhyay et al., 2017; Hori et al., 2019; Thomason et al., 2019; Le et al., 2021b). Recent work has addressed different challenges in visually-grounded dialogues, including multimodal integration (Hori et al., 2019; Le et al., 2019; Li et al., 2021), cross-turn dependencies (Das et al., 2017b; Schwartz et al., 2019; Le et al., 2021a), visual understanding (Le et al., 2020a), and data distribution bias (Qi et al., 2020). Our work is more related to the challenge of visual object reasoning (Seo et al., 2017; Kottur et al., 2018), but focused on a multi-turn tracking task over multiple turns of dialogue context. The prior approaches are not well designed to track objects and maintain a recurring memory or state of these objects from turn to turn. This challenge becomes more obvious when a dialogue involves multiple objects of similar characters or appearance. We directly tackle this challenge as we formulated a novel multimodal state tracking task and leveraged the research development from DST in task-oriented dialogue systems. As shown in our experiments, baseline models that use attention strategies similar to (Seo et al., 2017; Kottur et al., 2018) did not perform well in MM-DST.
A.3 Multimodal DST
We noted a few studies have attempted to integrate some forms of state tracking in multimodal dialogues. In (Mou et al., 2020), however, we are not convinced that a dialogue state tracking task is a major focus, or correctly defined. In (Pang and Wang, 2020), we noted that some form of object tracking is introduced throughout dialogue turns. The tracking module is used to decide which object
Split	# Videos	# Dialogues	# Turns	# Slots
DVD-DST-Train	9300	9295	92950	6
DVD-DST-Val	3327	3326	33260	6
DVD-DST-Test	1371	1371	13710	6
DVD-DST-All	13998	13992	139920	6
MultiWOZ (Budzianowski et al., 2018)	N/A	8438	115424	25
CarAssistant (Eric et al., 2017)	N/A	2425	12732	13
WOZ2 (Wen et al., 2017)	N/A	600	4472	4
DSTC2 (Henderson et al., 2014a)	N/A	1612	23354	8
Table 6: Dataset summary: statistics of related benchmarks are from (Budzianowski et al., 2018)
3 Object Sizes	9 Object Color	2 Object Materials	5 Object Shapes	Small golden metal snitch	Large red metal cylinder	Medium gray rubber cone
Small Large Medium	Gold Brown Gray Green Blue Cyan Red Purple Yellow	Rubber Metal	Cube Sphere Cylinder Cone Snitch
Figure 4: Synthetic visual objects in the CATER universe
the dialogue centers around. This method extends to multi-object tracking but the objects are only limited within static images, and there is no recurring information state (object attributes) maintained at each turn. Compared to our work, their tracking module only requires object identity as a single-slot state from turn to turn. Almost concurrent to our work, we noted (Kottur et al., 2021) which formally, though very briefly, focuses on multimodal DST. However, the work is limited to the task-oriented domain, and each dialogue is only limited to a single goal-driven object in a synthetic image. While this definition is useful in the task-oriented dialogue domain, it does not account for the DST of multiple visual objects as defined in our work.
B DVD-DST Dataset Details
For each of CATER videos from the extended split (Shamsian et al., 2020), we generated up to 10 turns for each CATER video. In total, DVD-DST contains more than  $13k$  dialogues, resulting in more  $130k$  (human, system) utterance pairs and corresponding dialogue states. A comparison of statistics of DVD-DST and prior DST benchmarks can be seen in Table 6. We observed that DVD-DST contains a larger scale data than the related DST benchmark. Even though the number of slots in DVD-DST is only 6, lower than prior state tracking datasets, our experiments indicate that most current conventional DST models perform poorly on
DVD-DST.
CATER universe. Figure 4 displays the configuration of visual objects in the CATER universe. In total, there are 3 object sizes, 9 colors, 2 materials, and 5 shapes. These attributes are combined randomly to synthesize objects in each CATER video. We directly adopted these attributes as slots in dialogue states, and each dialogue utterance frequently refers to these objects by one or more attributes. In total, there are 193 (size, color, material, shape) valid combinations, each of which corresponds to an object class in our models.
Sample dialogues. Please refer to Figure 5, Table 14 and Table 15.
Usage. We want to highlight that the DVD-DST dataset should only be used for its intended purpose, i.e. to diagnose dialogue systems on their tracking abilities. Any derivatives of the data should be limited within the research contexts of MM-DST.
In practice, we applied label smoothing (Szegedy et al., 2016) on state sequence labels to regularize the training. As the segment-level representations are stacked by the number of objects, we randomly selected only one vector per masked segment to apply  $\mathcal{L}_{seg}$ . We tested both L1 and L2 losses on  $\mathcal{L}_{bb/seg}$ . All model parameters, except pretrained visual perception models, are initialized by a uniform distribution (Glorot and Bengio, 2010).
Video Features	Dialogue State	Video loss	Greedy	Beam Search
Joint Obj State Acc	Joint State IoU@0.5	Joint State IoU@0.7	Joint Obj State Acc	Joint State IoU@0.5	Joint State IoU@0.7
Xbb	B\time	-	17.3%	N/A	N/A	17.9%	N/A	N/A
Xbb + Xcnn	B\time	-	20.0%	N/A	N/A	22.4%	N/A	N/A
Xbb	B	-	16.6%	9.6%	8.3%	19.3%	11.0%	9.5%
Xbb + Xcnn	B	-	22.4%	12.7%	10.8%	24.8%	13.8%	11.8%
Xbb	B	Lobj	21.7%	11.7%	10.0%	24.0%	12.9%	11.0%
Xbb + Xcnn	B	Lobj	23.1%	13.2%	11.3%	26.0%	14.4%	12.4%
Xbb + Xcnn	B	Lseg	24.3%	13.4%	11.4%	28.0%	15.3%	13.1%
Table 7: Ablation results by joint state predictions, using greedy or beam search decoding styles
Video Features	Dialogue State	Video self-supervision	Obj Identity F1	Obj Slot F1	Obj State F1	Size F1	Color F1	Material F1	Shape F1
\(X_{bb}\)	\(\mathcal{B}\backslash\text{time}\)	-	79.4%	64.2%	48.5%	55.9%	76.6%	41.4%	63.5%
\(X_{bb}+X_{cnn}\)	\(\mathcal{B}\backslash\text{time}\)	-	81.4%	66.9%	52.5%	58.0%	79.4%	39.5%	66.6%
\(X_{bb}\)	\(\mathcal{B}\)	-	78.5%	63.6%	49.8%	56.5%	76.4%	38.8%	63.1%
\(X_{bb}+X_{cnn}\)	\(\mathcal{B}\)	-	83.3%	69.4%	55.1%	56.7%	81.8%	47.0%	69.8%
\(X_{bb}\)	\(\mathcal{B}\)	\(\mathcal{L}_{obj}\)	82.2%	69.5%	56.2%	61.4%	81.0%	44.9%	69.9%
\(X_{bb}+X_{cnn}\)	\(\mathcal{B}\)	\(\mathcal{L}_{obj}\)	84.7%	72.0%	58.6%	59.7%	83.5%	52.3%	71.7%
\(X_{bb}+X_{cnn}\)	\(\mathcal{B}\)	\(\mathcal{L}_{seg}\)	84.5%	72.8%	60.4%	64.1%	84.2%	50.9%	71.9%
Table 8: Ablation results by component predictions of object identities, slots, and object states
For fair comparison among baselines, all models use both object-level and segment-level feature representations, encoded by the same method as Describe in Section 3.1. In TRADE, the video representations are passed to an RNN encoder, and the output hidden states are concatenated to the dialogue hidden states. Both are passed to the original pointer-based decoder. In UniConv and NADAST, we stacked another Transformer attention layer to attend on video representations before the original state-to-dialogue attention layer. We all baseline models, we replaced the original (domain, slot) embeddings as (object class, slot) embeddings and kept the original model designs.
Note that in our visual perception model, we adopted the finetuned Faster R-CNN model used by (Shamsian et al., 2020). The model was finetuned to predict object bounding boxes and object classes. The object classes are derived based on object appearance, based on the four attributes of size, color, material, and shape. In total, there are 193 object classes. For segment embeddings, we adopted the ResNeXt-101 model (Xie et al., 2017) finetuned on Kinetics dataset (Kay et al., 2017). For all models (except for VDTN ablation analysis), we standardized  $N_{obj} = 10$  and  $N_{stride} = 12$  to sub-sample object and segment-level embeddings.
Resources. Note that all experiments did not require particularly large computing resources as we limited all model training to a single GPU, specifically on a Tesla V100 GPU of 16G configuration.
Greedy vs. Beam Search Decoding. Table 7 shows the results of different variants of VDTN models. We observed that compared to greedy decoding, beam search decoding improves the performance in all models. As beam search decoding selects the best decoded state by the joint probabilities of tokens, this observation indicates the benefits of considering slot values to be co-dependent and their relationships should be modelled. This is consistent with similar observations in later work of unimodal DST (Lei et al., 2018b; Le et al., 2020c).
Ablation analysis by component predictions. From Table 8, we have the following observations: (1) In ablation results by component predictions, we noted that models can generally detect object identities well with F1 about  $80\%$ . However, when considering object and slot tuples, F1 reduces to  $48 - 60\%$ , indicating the gaps are caused by slot value predictions. (2) By individual slots, we noted "color" and "shape" slots are easier to track than
Video Features	Dialogue State	Video self-supervision	Obj Identity Recall	Obj Identity Precision	Obj Slot Recall	Obj Slot Precision	Obj State Recall	Obj State Precision
Xbb	B\time	-	77.2%	81.8%	65.0%	63.4%	47.1%	50.0%
Xbb+Xcnn	B\time	-	75.1%	88.8%	63.1%	71.3%	48.5%	57.3%
Xbb	B	-	73.6%	84.1%	61.7%	65.7%	46.7%	53.4%
Xbb+Xcnn	B	-	78.2%	89.1%	66.2%	73.0%	51.7%	58.9%
Xbb	B	Lobj	76.4%	88.9%	67.4%	71.7%	52.2%	60.8%
Xbb+Xcnn	B	Lobj	80.1%	90.0%	69.1%	75.2%	55.4%	62.2%
Xbb+Xcnn	B	Lseg	80.5%	89.0%	70.2%	75.6%	57.6%	63.6%
Table 9: Ablation results by individual object identity/slot/state
Video Features	Dialogue State	Video self-supervision	Size Recall	Size Precision	Color Recall	Color Precision	Material Recall	Material Precision	Shape Recall	Shape Precision
\(X_{bb}\)	\(\mathcal{B}\backslash\text{time}\)	-	60.1%	52.2%	76.8%	76.4%	43.2%	39.7%	61.4%	65.6%
\(X_{bb}+X_{cnn}\)	\(\mathcal{B}\backslash\text{time}\)	-	52.0%	65.6%	76.2%	82.9%	34.8%	45.8%	65.5%	67.8%
\(X_{bb}\)	\(\mathcal{B}\)	-	52.0%	61.9%	72.0%	81.2%	40.8%	37.1%	63.3%	63.0%
\(X_{bb}+X_{cnn}\)	\(\mathcal{B}\)	-	49.4%	66.5%	79.2%	84.6%	45.0%	49.2%	68.9%	70.6%
\(X_{bb}\)	\(\mathcal{B}\)	\(\mathcal{L}_{obj}\)	59.6%	63.4%	79.3%	82.9%	43.8%	46.0%	66.6%	73.5%
\(X_{bb}+X_{cnn}\)	\(\mathcal{B}\)	\(\mathcal{L}_{obj}\)	54.1%	66.6%	82.4%	84.7%	48.8%	56.3%	69.3%	74.3%
\(X_{bb}+X_{cnn}\)	\(\mathcal{B}\)	\(\mathcal{L}_{seg}\)	60.9%	67.7%	83.2%	85.4%	48.6%	53.4%	67.9%	76.5%
Table 10: Ablation results by individual slot type
“size” and “material” slots. We noted that in the CATER universe, the latter two slots have lower visual variances (less possible values) than the others. As a result, objects are more likely to share the same size or material and hence, discerning objects by those slots and tracking them in dialogues become more challenging.