SlowGuess/NAACL_2022_OCR huggingface dataset:
correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308.
Yun-Wei Chu, Kuan-Yen Lin, Chao-Chun Hsu, and Lun-Wei Ku. 2020. Multi-step joint-modality attention network for scene-aware dialogue system. DSTC Workshop @ AAAI.
Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256-2262, Austin, Texas. Association for Computational Linguistics.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019. Universal transformers. In International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. 2019. Heterogeneous memory enhanced multimodal attention model for video question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1999-2007.
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719–3728, Brussels, Belgium. Association for Computational Linguistics.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. 2016. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 457-468, Austin, Texas. Association for Computational Linguistics.
Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevata. 2018. Motion-appearance co-memory networks for video question answering. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, pages 6576-6585.
Lianli Gao, Pengpeng Zeng, Jingkuan Song, Yuan-Fang Li, Wu Liu, Tao Mei, and Heng Tao Shen. 2019. Structured two-stream attention network for video question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6391-6398.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, volume 1, page 3.
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. 2020. Neural module networks for reasoning over text. In International Conference on Learning Representations.
Chi Han, Jiayuan Mao, Chuang Gan, Josh Tenenbaum, and Jiajun Wu. 2019. Visual concept-metaconcept learning. In Advances in Neural Information Processing Systems, pages 5002-5013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. CNN architectures for largescale audio classification. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pages 131-135. IEEE.
C. Hori, H. Alamri, J. Wang, G. Wichern, T. Hori, A. Cherian, T. K. Marks, V. Cartillier, R. G. Lopes, A. Das, I. Essa, D. Batra, and D. Parikh. 2019. End-to-end audio visual scene-aware dialog using multimodal attention-based video features. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2352-2356.
Chiori Hori, Anoop Cherian, Tim K Marks, and Takaaki Hori. 2019. Joint student-teacher learning for audio-visual scene-aware dialog. Proc. Interspeech 2019, pages 1886-1890.
Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. 2018. Explainable neural computation via stack neural module networks. In Proceedings of the European conference on computer vision (ECCV), pages 53-69.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 804-813.
Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. 2016. Natural language object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4555-4564.
Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, and Chuang Gan. 2020. Location-aware graph convolutional networks for video question answering. In The AAAI Conference on Artificial Intelligence (AAAI), volume 1.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatiotemporal reasoning in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2758-2766.
Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao. 2020. Divide and conquer: Question-guided spatio-temporal contextual attention for video question answering. In The AAAI Conference on Artificial Intelligence (AAAI).
Pin Jiang and Yahong Han. 2020. Reasoning with heterogeneous graph alignment for video question answering. In The AAAI Conference on Artificial Intelligence (AAAI).
Yichen Jiang and Mohit Bansal. 2019. Self-assembling modular networks for interpretable multi-hop reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4474-4484, Hong Kong, China. Association for Computational Linguistics.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017a. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901-2910.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017b. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2989-2998.
Seonhoon Kim, Seohyeong Jeong, Eunbyul Kim, Inho Kang, and Nojun Kwak. 2021. Self-supervised pretraining and contrastive representation learning for multiple-choice video qa. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):13171-13179.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR).
Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2018. Visual coreference resolution in visual dialog using neural module networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 153-169.
Shachi H Kumar, Eda Okur, Saurav Sahay, Jonathan Huang, and Lama Nachman. 2019. Leveraging topics and audio features with multimodal attention for audio visual scene-aware dialog. arXiv preprint arXiv:1912.10131.
Hung Le, S Hoi, Doyen Sahoo, and N Chen. 2019a. End-to-end multimodal dialog systems with hierarchical multimodal attention on video features. In DSTC7 at AAAI2019 workshop.
Hung Le and Steven C.H. Hoi. 2020. Video-grounded dialogues with pretrained generation language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5842-5848, Online. Association for Computational Linguistics.
Hung Le, Doyen Sahoo, Nancy Chen, and Steven Hoi. 2019b. Multimodal transformer networks for end-to-end video-grounded dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5612-5623, Florence, Italy. Association for Computational Linguistics.
Hung Le, Doyen Sahoo, Nancy Chen, and Steven C.H. Hoi. 2020a. BiST: Bi-directional spatio-temporal reasoning for video-grounded dialogues. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1846-1859, Online. Association for Computational Linguistics.
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. 2019c. Learning to reason with relational video representation for question answering. arXiv preprint arXiv:1907.04553.
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. 2020b. Hierarchical conditional relation networks for video question answering. arXiv preprint arXiv:2002.10698.
Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. 2020. Dstc8-avsd: Multimodal semantic transformer network with retrieval style word generator. DSTC Workshop @ AAAI 2020.
Chenyi Lei, Lei Wu, Dong Liu, Zhao Li, Guoxin Wang, Haihong Tang, and Houqiang Li. 2020. Multiquestion learning for visual question answering. In The AAAI Conference on Artificial Intelligence (AAAAI).
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. 2021. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7331-7341.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. TVQA: Localized, compositional video question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1369-1379, Brussels, Belgium. Association for Computational Linguistics.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020a. HERO: Hierarchical encoder for Video+Language omni-representation pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2046-2065, Online. Association for Computational Linguistics.
Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, and Chuang Gan. 2019. Beyond rnns: Positional self-attention with co-attention for video question answering. In The 33rd AAAI Conference on Artificial Intelligence, volume 8.
Zekang Li, Zongjia Li, Jinchao Zhang, Yang Feng, Cheng Niu, and Jie Zhou. 2020b. Bridging text and video: A universal multimodal transformer for video-audio scene-aware dialog. arXiv preprint arXiv:2002.00163.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. 2019. The neurosymbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In International Conference on Learning Representations.
Dat Tien Nguyen, Shikhar Sharma, Hannes Schulz, and Layla El Asri. 2018. From film to video: Multi-turn question answering with multi-modal context. In AAAI 2019 Dialog System Technology Challenge (DSTC7) Workshop.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Mengye Ren, Ryan Kiros, and Richard Zemel. 2015a. Exploring models and data for image question answering. In Advances in neural information processing systems, pages 2953-2961.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015b. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NIPS).
Ramon Sanabria, Shruti Palaskar, and Florian Metze. 2019. Cmu sinbad's submission for the dstd7 avsd challenge. In DSTC7 at AAAI2019 workshop, volume 6.
Idan Schwartz, Seunghak Yu, Tamir Hazan, and Alexander G Schwing. 2019. Factor graph attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2039-2048.
Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence.
Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, pages 510-526. Springer.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826.
Zineng Tang, Jie Lei, and Mohit Bansal. 2021. DeCEMBERT: Learning from noisy instructional videos via dense captions and entropy minimization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2415-2426, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio,
H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575.
Zekun Yang, Noa Garcia, Chenhui Chu, Mayu Otani, Yuta Nakashima, and Haruo Takemura. 2020. Bert representations for video question answering. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1556-1565.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. 2018. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. In Advances in Neural Information Processing Systems, pages 1031-1042.
Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. 2017. End-to-end concept word detection for video captioning, retrieval, and question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3165-3173.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. Merlot: Multimodal neural script knowledge models. arXiv preprint arXiv:2106.02636.
A.1 Question Parsers
To learn compositional programs, we follow (Johnson et al., 2017a; Hu et al., 2017) and consider program generation as a sequence-to-sequence task. We adopt a simple template  $\langle \mathrm{param}_1\rangle \langle \mathrm{module}_1\rangle \langle \mathrm{param}_2\rangle \langle \mathrm{module}_2\rangle \ldots$  as the target sequence. The resulting target sequences for dialogue and video understanding programs are sequences  $\mathcal{P}_{\mathrm{dial}}$  and  $\mathcal{P}_{\mathrm{vid}}$  respectively.
The parsers decompose questions into subsequences to construct compositional reasoning programs for dialogue and video understanding. Each parser is an attention-based Transformer decoder. The Transformer attention is a multi-head attention on query  $q$ , key  $k$ , and value  $v$  tensors, denoted as Attention(q, k, v). For each token in the q sequence, the distribution over tokens in the k sequence is used to obtain the weighted sum of the corresponding representations in the v sequence.
$\operatorname{Attention}(q, k, v) = \operatorname{softmax}(\frac{qk^T}{\sqrt{d_k}})v \in \mathbb{R}^{L_{\mathrm{q}} \times d_{\mathrm{q}}}$
Each attention is followed by a feed-forward network applied to each position identically. We exploit the multi-head and feed-forward architecture, which show good performance in NLP tasks such as NMT and QA (Vaswani et al., 2017; Dehghani et al., 2019), to efficiently incorporate contextual cues from dialogue components to parse question into reasoning programs. At decoding step 0, we simply use a special token _sos as the input to the parser. In each subsequent decoding step, we concatenate the prior input sequence with the generated token to decode in an auto-regressive manner. We share the vocabulary sets of input and output components and thus, use the same embedding matrix. Given the encoded question  $Q$ , to decode the program for dialogue understanding, the contextual signals are integrated through 2 attention layers: one attention on previously generated tokens, and the other on question tokens. At time step  $j$ , we denote the output from an attention layer as  $A_{\mathrm{dial},j}$ .
$$
\begin{array}{l} A _ {\mathrm {d i a l}} ^ {(1)} = \mathrm {A t t e n t i o n} (P _ {\mathrm {d i a l}} | _ {0} ^ {j - 1}, P _ {\mathrm {d i a l}} | _ {0} ^ {j - 1}, P _ {\mathrm {d i a l}} | _ {0} ^ {j - 1}) \\ A _ {\mathrm {d i a l}} ^ {(2)} = \mathrm {A t t e n t i o n} (A _ {\mathrm {d i a l}} ^ {(1)}, Q, Q) \in \mathbb {R} ^ {j \times d} \\ \end{array}
$$
To generate programs for video understanding, the contextual signals are learned and incorporated in a similar manner. However, to exploit dialogue contextual cues, the execution output of dialogue understanding neural modules  $Q_{\mathrm{ctx}}$  is incorporated to each vector in  $P_{\mathrm{dial}}$  through an additional attention layer. This layer integrates the resolved entity information to decode the original entities for video understanding. It is equivalent to a reasoning process that converts the question from its original multi-turn semantics to single-turn semantics.
$$
\begin{array}{l} A _ {\mathrm {v i d}} ^ {(1)} = \mathrm {A t t e n t i o n} (P _ {\mathrm {v i d}} | _ {0} ^ {j - 1}, P _ {\mathrm {v i d}} | _ {0} ^ {j - 1}, P _ {\mathrm {v i d}} | _ {0} ^ {j - 1}) \\ A _ {\mathrm {v i d}} ^ {(2)} = \operatorname {A t t e n t i o n} \left(A _ {\mathrm {v i d}} ^ {(1)}, Q, Q\right) \in \mathbb {R} ^ {j \times d} \\ A _ {\mathrm {v i d}} ^ {(3)} = \operatorname {A t t e n t i o n} (A _ {\mathrm {v i d}} ^ {(2)}, Q _ {\mathrm {c t x}}, Q _ {\mathrm {c t x}}) \in \mathbb {R} ^ {j \times d} \\ \end{array}
$$
A.2 How to locate entities?
Noted that in the neural modules described in Section 3.3, during training, we simply feed the ground-truth programs to optimize these modules. For instance, the neural module where received the ground truth entities  $P$  which is then used to instantiate the neural network and retrieve from video  $V$ . During test time, we decode the programs token by token through the question parsers, and feed the predicted entities  $\hat{P}$  to neural modules. Note that we do not assume, and hence not train model to
retrieve ground-truth locations of visual entities in videos. This strategy enables the applicability of VGNMN as we consider these entity annotations mostly unavailable in real-world systems.
B.1 Non-NMN Models
We experiment with several Non-NMN based variants of our models. As can be seen in Table 7, our approach to video and dialogue understanding through compositional reasoning programs exhibits better performance than non-compositional approaches. Compared to the approaches that directly process frame-level features in videos (Row B) or token-level features in dialogues (Row C, D), our full VGNMN (Row A) considers entity-level and action-level information extraction and thus, avoids unnecessary and possibly noisy extraction. Compared to the approaches that obtain dialogue contextual cues through a hierarchical encoding architecture (Row E, F) such as (Serban et al., 2016; Hori et al., 2019), VGNMN directly addresses the challenge of entity references in dialogues. As mentioned, we hypothesize that the hierarchical encoding architecture is more appropriate for less entity-sensitive dialogues such as chit-chat and open-domain dialogues.
B.2 Dialogue context integration
Experimenting with different ways to integrate dialogue context representations, we observe that adding an attention layer attending to question during response decoding (Row G) is not necessary. This can be explained as the representation  $Q_{\mathrm{ctx}}$  obtained from dialogue understanding program already contains contextual information of both dialogue history and question and question input is no longer needed in the decoding phase. Furthermore, we investigate the model sensitivity to natural language generation through its ability to construct linguistically correct programs and responses. To generate responses that are linguistically appropriate, VGNMN needs dialogue context representation  $Q_{\mathrm{ctx}}$  as input to the response decoder (Row H). The model also needs encoded question  $Q$  as input to the video understanding program parser to be able to decompose this sequence to entity and action module parameters (Row I).
#	Model Variant	generated programs	ground-truth programs
BLEU4	CIDEr	BLEU4	CIDEr
A	Full VGNMN	0.421	1.171	0.423	1.167
B	← video NMNs ~ response-decoder-to-video attn.	0.415	1.159	-	-
C	← dial. NMNs ~ res-decoder-to-(history→question) attn.	0.412	1.151	-	-
:q!D	← dial. NMNs ~ res-decoder-to-concat(history+question) attn.	0.411	1.133	-	-
E	← dial. NMNs ~ HREDLSTM(history) + question attn.	0.414	1.153	-	-
F	← dial. NMNs ~ HREDGRU(history) + question attn.	0.415	1.138	-	-
G	← + response-decoder-to-question attn.	0.424	1.166	0.426	1.164
H	← - response-decoder-to-dialogue-context attn.	0.405	1.124	0.404	1.123
I	← - video-understanding-prog-parser-to-question attn.	0.414	1.146	0.424	1.166
Table 7: Ablation analysis of VGNMN with different model variants on the test split of the AVSD benchmark
C Interpretability
We extract the predicted programs and responses for some example dialogues in Figure 6, 7, 8, and 9 and report our observations: