Venue: NAACL 2022

# VGNMN: Video-grounded Neural Module Networks for Video-Grounded Dialogue Systems

Hung Le\({}^{{\dagger}{\ddagger}}\), Nancy F. Chen\({}^{\lx@sectionsign}\), Steven C.H. Hoi\({}^{{\ddagger}}\)

\({}^{{\dagger}}\)Salesforce Research Asia

\({}^{{\ddagger}}\)Singapore Management University

\({}^{\lx@sectionsign}\)Agency for Science, Technology and Research (A*STAR)

{hungle,shoi}@salesforce.com, nfychen@i2r.a-star.edu.sg

###### Abstract

Neural module networks (NMN) have achieved success in image-grounded tasks such as Visual Question Answering (VQA) on synthetic images. However, very limited work on NMN has been studied in the video-grounded dialogue tasks. These tasks extend the complexity of traditional visual tasks with the additional visual temporal variance and language cross-turn dependencies. Motivated by recent NMN approaches on image-grounded tasks, we introduce Video-grounded Neural Module Network (VGNMN) to model the information retrieval process in video-grounded language tasks as a pipeline of neural modules. VGNMN first decomposes all language components in dialogues to explicitly resolve any entity references and detect corresponding action-based inputs from the question. The detected entities and actions are used as parameters to instantiate neural module networks and extract visual cues from the video. Our experiments show that VGNMN can achieve promising performance on a challenging video-grounded dialogue benchmark as well as a video QA benchmark.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Vision-language tasks have been studied to build intelligent systems that can perceive information from multiple modalities, such as images, videos, and text. Extended from image-grounded tasks, e.g. Antol et al. (2015), recently Jang et al. (2017); Lei et al. (2018) propose to use video as the grounding features. This modification poses a significant challenge to previous image-based models with the additional temporal variance through video frames. Recently Alamri et al. (2019) further develop video-grounded language research into the dialogue domain. In the proposed task, _video-grounded dialogues_, the dialogue agent is required to answer questions about a video over multiple dialogue turns. Using Figure 1 as an example, to answer questions correctly, a dialogue agent has to resolve references in dialogue context, e.g. "he" and "it", and identify the original entity, e.g. "a boy" and "a backpack". Besides, the agent also needs to identify the actions of these entities, e.g. "carrying a backpack" to retrieve information from the video.

Current state-of-the-art approaches to video-grounded dialogue tasks, e.g. Le et al. (2019b); Fan et al. (2019) have achieved remarkable performance through the use of deep neural networks to retrieve grounding video signals based on language inputs. However, these approaches often assume the reasoning structure, including resolving references of entities and detecting the corresponding actions to retrieve visual cues, is implicitly learned. An explicit reasoning structure becomes more beneficial as the tasks complicate in two scenarios: video with complex spatial and temporal dynamics, and language inputs with sophisticated semantic dependencies, e.g. questions positioned in a dialogue context. These scenarios often challenge researchers to interpret model hidden layers, identify errors, and assess model reasoning capability.

Similar challenges have been observed in image-grounded tasks in which deep neural networks exhibit shallow understanding capability as they exploit superficial visual cues Agrawal et al. (2016); Goyal et al. (2017); Feng et al. (2018); Serrano and

Figure 1: A sample video-grounded dialogue with a demonstration of a reasoning process

Smith, 2019). Andreas et al. (2016b) propose neural module networks (NMNs) by decomposing a question into sub-sequences called _program_ and assembling a network of neural operations. Motivated by this line of research, we propose a new approach, VGNMN, to video-grounded language tasks. Our approach benefits from integrating neural networks with a compositional reasoning structure to exploit low-level information signals in video. An example of the reasoning structure can be seen on the right side of Figure 1.

Video-grounded Neural Module Network (VGNMN) tackles video understanding through action and entity-paramterized NMNs to retrieve video features. We first decompose question into a set of entities and extract video features related to these entities. VGNMN then extracts the temporal steps by focusing on relevant actions that are associated with these entities. VGNMN is analogous to how human processes information by gradually retrieving signals from input modalities using a set of discrete subjects and their actions.

To tackle dialogue understanding, VGNMN is trained to resolve any co-reference in language inputs, e.g. questions in a dialogue context, to identify the unique entities in each dialogue. Previous approaches to video-grounded dialogues often obtain question global representations in relation to dialogue context. These approaches might be suitable to represent general semantics in open-domain dialogues Serban et al. (2016). However, they are not ideal to detect fine-grained information in a video-grounded dialogue which frequently entails dependencies between questions and past dialogue turns in the form of entity references.

In summary, our contributions include:

* VGNMN, a neural module network-based approach for video-grounded dialogues.
* The approach includes a modularized system that creates a reasoning pipeline parameterized by entity and action-based representations from both dialogue and video contexts.
* Our experiments are conducted on the challenging benchmark for video-grounded dialogues, Audio-visual Scene-Aware Dialogues (AVSD) Alamri et al. (2019) as well as TGIF-QA Jang et al. (2017) for video QA task.
* Our results indicate strong performance of VGNMN as well as improved model interpretability and robustness to difficult scenarios of dialogues, videos, and question structures.

## 2 Related Work

### Video-Language Understanding

The research of video-language understanding aims to develop a model's joint understanding capability of language, video, and their interactions. Jang et al. (2017); Gao et al. (2018); Jiang et al. (2020) propose to learn attention guided by question global representation to retrieve spatial-level and temporal-level visual features. Li et al. (2019); Fan et al. (2019); Jiang and Han (2020) model interaction between all pairs of question token-level representations and temporal-level features of the input video through similarity matrix, memory networks, and graph networks respectively. Gao et al. (2019); Le et al. (2019c, 2020b); Lei et al. (2020); Huang et al. (2020) extends the previous approach by dividing a video into equal segments, sub-sampling video frames, or considering object-level representations of input video. We propose to replace token-level and global question representations with question representations composed of specific entities and actions.

Recently, we have witnessed emerging techniques in video-language systems that exploit deep transformer-based architectures such as BERT Devlin et al. (2019) for pretraining multimodal representations Li et al. (2020a); Yang et al. (2020); Kim et al. (2021); Tang et al. (2021); Lei et al. (2021); Zellers et al. (2021) in very large-scale video-language datasets. While these systems can achieve impressive performance, they are not straightforward to apply in domains with limited data such as video-grounded dialogues. Moreover, as we shown in our qualitative examples, our approach facilitates better interpretability through the output of decoded functional programs.

### Video-grounded Dialogues

Extended from video QA, video-grounded dialogue is an emerging task that combines dialogue response generation and video-language understanding research. This task entails a novel requirement for models to learn dialogue semantics and decode entity co-references in questions. Nguyen et al. (2018); Hori et al. (2019); Hori et al. (2019); Sanabria et al. (2019); Le et al. (2019a, b) extend traditional QA models by adding dialogue history neural encoders. Kumar et al. (2019) en 

hances dialogue features with topic-level representations to express the general topic in each dialogue. Schwartz et al. (2019) treats each dialogue turn as an independent sequence and allows interaction between questions and each dialogue turn. Le et al. (2019b) encodes dialogue history as a sequence with embedding and positional representations. Different from prior work, we dissect the question sequence and explicitly detect and decode any entities and their references. Our approach also enables insights on how models extract deductive bias from dialogues to extract video information.

### Neural Module Network

Neural Module Network (NMN) Andreas et al. (2016, 2016) is introduced to address visual QA by decomposing questions into linguistic sub-structures, known as programs, to instantiate a network of neural modules. NMN models have achieved success in synthetic image domains where a multi-step reasoning process is required Johnson et al. (2017); Hu et al. (2018); Han et al. (2019). Yi et al. (2018); Han et al. (2019); Mao et al. (2019) improve NMN models by decoupling visual-language understanding and visual concept learning. Our work is related to the recent work Kottur et al. (2018); Jiang and Bansal (2019); Gupta et al. (2020) that extended NMNs to image reasoning in dialogues and reading comprehension reasoning. Our approach follows the previous approaches that learn to generate program structure and require no parser at evaluation time. Compared to prior work, we use NMN to learn dependencies between the composition in language inputs and the spatio-temporal dynamics in videos. Specifically, we propose to construct a reasoning structure from text, from which detected entities are used to extract visual information in the spatial space and detected actions are used to find visual information in the temporal space.

## 3 Method

In this section, we present the design of our model. An overview of the model can be seen in Figure 2.

### Task Definition

The input to the model consists of a dialogue \(\mathcal{D}\) which is grounded on a video \(\mathcal{V}\). The input components include the question of current dialogue turn \(\mathcal{Q}\), dialogue history \(\mathcal{H}\), and the features of the input video, including visual and audio input. The output is a dialogue response, denoted as \(\mathcal{R}\). Each text input component is a sequence of words \(w_{1},...,w_{m}\in\mathbb{V}^{in}\), the input vocabulary. Similarly, the output response \(\mathcal{R}\) is a sequence of tokens \(w_{1},...,w_{n}\in\mathbb{V}^{out}\), the output vocabulary. The objective of the task is the generation objective that output answers of the current dialogue turn \(t\):

\[\hat{\mathcal{R}}_{t} =\operatorname*{arg\,max}_{\mathcal{R}_{t}}P(\mathcal{R}_{t}| \mathcal{V},\mathcal{H}_{t},\mathcal{Q}_{t};\theta)\] \[=\operatorname*{arg\,max}_{\mathcal{R}_{t}}\prod_{n=1}^{L_{ \mathcal{R}}}P_{m}(w_{n}|\mathcal{R}_{t,1:n-1},\mathcal{V},\mathcal{H}_{t}, \mathcal{Q}_{t};\theta)\]

where \(L_{\mathcal{R}}\) is the length of the sequence \(\mathcal{R}\). In a Video-QA task, the dialogue history \(\mathcal{H}\) is simply absent and the output response is typically collapsed to a single-token response.

### Encoders

**Text Encoder.** A text encoder is shared to encode text inputs, including dialogue history, questions, and captions. The text encoder converts each text sequence \(\mathcal{X}=w_{1},...,w_{m}\) into a sequence of embeddings \(X\in\mathbb{R}^{m\times d}\). We use a trainable embedding matrix to map token indices to vector representations of \(d\) dimensions through a mapping function \(\phi\). These vectors are then integrated with ordering information of tokens through a positional encoding function with layer normalization Ba et al. (2016); Vaswani et al. (2017). The embedding and positional representations are combined through element-wise summation. The encoded dialogue history and question of the current turn are defined as \(H=\mathrm{Norm}(\phi(\mathcal{H})+\mathrm{PE}(\mathcal{H}))\in\mathbb{R}^{L_{ \text{I}}\times d}\) and \(Q=\mathrm{Norm}(\phi(\mathcal{Q})+\mathrm{PE}(\mathcal{Q}))\in\mathbb{R}^{L_{ \text{Q}}\times d}\).

**Video Encoder.** To encode video, we use pretrained models to extract visual and audio features. We denote \(F\) as the sampled video frames or video clips. For object-level visual features, we denote \(O\) as the maximum number of objects considered in each frame. The resulting output from a pretrained object detection model is \(Z_{\rm obj}\in\mathbb{R}^{F\times O\times d_{\rm vis}}\). We concatenate each object representation with the

Figure 2: An overview of the VGNMN approach. Colo-erful boxes are network components and the rest are input/output. Dotted lines are for optional components.

 

corresponding coordinates projected to \(d_{\rm vis}\) dimensions. We also make use of a CNN-based pre-trained model to obtain features of temporal dimension \(Z_{\rm cnn}\in\mathbb{R}^{F\times d_{\rm vis}}\). The audio feature is obtained through a pretrained audio model, \(Z_{\rm aud}\in\mathbb{R}^{F\times d_{\rm aud}}\). We passed all video features through a linear transformation layer with ReLU activation to the same embedding dimension \(d\).

### Neural Modules

We introduce neural modules that are used to assemble an executable program constructed by the generated sequence from question parsers. We provide an overview of neural modules in Table 1 and demonstrate dialogue understanding and video understanding modules in Figure 3 and 4 respectively. Each module parameter, e.g. "a backpack", is extracted from the parsed program (See Section 3.4). For each parameter, we denote \(P\in\mathbb{R}^{d}\) as the average pooling of component token embeddings.

**find(P,H)**\(\rightarrow\)**h\({}_{\rm ent}\). This module handles entity tracing by obtaining a distribution over tokens in the dialogue history. We use an entity-to-dialogue-history attention mechanism applied from an entity \(P_{i}\) to all tokens in the dialogue history. Any neural network that learn to generate attention between two tensors is applicable e.g. Bahdanau et al. (2015); Vaswani et al. (2017). The attention matrix normalized by softmax, \(A_{\rm find,i}\in\mathbb{R}^{L_{\text{H}}}\), is used to compute the weighted sum of dialogue history token representations. The output is combined with entity embedding \(P_{i}\) to obtain contextual entity representation \(H_{\rm ent,i}\in\mathbb{R}^{d}\).

**summarize**(H\({}_{\rm ent},Q\))\(\rightarrow\)**Q\({}_{\rm ctx}\)**. For each contextual entity representation \(H_{\rm ent,i}\), \(i=1,...,N_{\rm ent}\), it is projected to \(L_{\text{Q}}\) dimensions and is combined with question token embeddings through elementwise summation to obtain entity-aware question representation \(Q_{\rm ent,i}\in\mathbb{R}^{L_{\text{Q}}\times d}\). It is fed to a one-dimensional CNN with max-pooling layer Kim (2014) to obtain a contextual entity-aware question representation. We denote the final output as \(Q_{\rmctx}\in\mathbb{R}^{N_{\rm ent}\times d}\).

While previous models usually focus on global or token-level dependencies Hori et al. (2019); Le et al. (2019b) to encode question features, our modules compress fine-grained question representations at the entity level. Specifically, find and summarize modules can generate entity-dependent local and global representations of question semantics. We show that our modularized approach can achieve better performance and transparency than traditional approaches to encode dialogue context Serban et al. (2016); Vaswani et al. (2017) (Section 4).

**where(P,V)**\(\rightarrow\)**V\({}_{\rm ent}\)**. Similar to the find module, this module handles entity-based attention to the video input. However, the entity representation \(P\), in this case, is parameterized by the original entity in dialogue rather than in question (See Section 3.4 for more description). Each entity \(P_{i}\) is stacked to match the number of sampled video frames/clips \(F\). An attention network is used to obtain entity-to-object attention matrix \(A_{\rm where,j}\in\mathbb{R}^{F\times O}\). The attended feature are compressed through weighted sum pooling along the spatial dimension, resulting in \(V_{\rm ent,i}\in\mathbb{R}^{F\times d}\), \(i=1,...,N_{\rm ent}\).

**when(P,Vent)**\(\rightarrow\)**V\({}_{\rm ent+act}\)**. This module follows a similar architecture as the where module. However, the action parameter \(P_{i}\) is stacked to match \(N_{\rm ent}\) dimensions. The attention matrix \(A_{\rm when,i}\in\mathbb{R}^{F}\) is then used to compute the visual entity-action representations through weighted sum along the temporal dimension. We denote the output for all actions \(P_{i}\) as \(V_{\rm ent+act}\in\mathbb{R}^{N_{\rm Ent}\times N_{\rm act}\times d}\)

**describe(P,Vent\({}_{\rm ant+act}\))\(\rightarrow\)V\({}_{\rm ctx}\)**. This module is a linear transformation to compute \(V_{\rm ctx}=W_{\rm desc}^{T}[V_{\rm ent+act};P_{\rm stack}]\in\mathbb{R}^{N_{ \rm ent}\times N_{\rm act}\times d}\) where \(W_{\rm desc}\in\mathbb{R}^{2d\times d}\), \(P_{\rm stack}\) is the stacked representations of parameter embedding \(P\) to \(N_{\rm ent}\times N_{\rm act}\) dimensions, and \([;]\) is the concatenation operation. Note that the parameter \(P\) here is extracted from questions, often as the type of questions e.g. "what"

\begin{table}
\begin{tabular}{c l l l} \hline
**Module** & **Input** & **Output** \\ \hline find & P,H & \({}_{\rm cont}\) & For related entities in question, select the relevant tokens from dialogue history \\ summarize & H\({}_{\rm ent},Q\) & Q\({}_{\rm ctx}\) & Based on contextual entity representations, summarise the question semantics \\ where & P,V & Vent & Select the relevant spatial position corresponding to original (resolved) entities \\ when & P,Vent & Vent\({}_{\rm ext+act}\) & Select the relevant entity-aware temporal steps corresponding to the action parameter \\ describe & P,Vent\({}_{\rm art+act}\) & Vets & Select visual entity-action features based on non-binary question types \\ exist & Q,Vent\({}_{\rm art+act}\) & Vets & Select visual entity-action features based on binary (yes/no) question types \\ \hline \end{tabular}
\end{table}
Table 1: Description of the modules and their functionalities. We denote \(P\) as the parameter to instantiate each module, \(H\) as the dialogue history, \(Q\) as the question of the current dialogue turn, and \(V\) as video input.

 

and "how". This eliminates the need to have different modules for different question types. However, we noted the current design may be challenged in rare cases in which an utterance contain numerous questions (refer to Figure 5).

The exist module is used when the questions are "yes/no" questions. This module is a special case of describe module where the parameter \(P\) is simply the average pooled question embeddings. The above where module is applied to object-level features. For temporal-based features such as CNN-based and audio features, the same neural operation is applied along the temporal dimension. Each resulting entity-aware output is then incorporated to frame-level features through element-wise summation.

An advantage of our architecture is that it separates dialogue and video understanding. We adopt a transparent approach to solve linguistic entity references during the dialogue understanding phase. The resolved entities are fed to the video understanding phase to learn entity-action dynamics in the video. We show that our approach is robust when dialogue evolves to many turns and video extends over time (Please refer to Section 4).

### Question Parsers

To learn compositional programs, we follow Johnson et al. (2017a); Hu et al. (2017) and consider program generation as a sequence-to-sequence task. We adopt a simple template "\(\langle{\rm param}_{1}\rangle\langle{\rm module}_{1}\rangle\langle{\rm param}_{2 }\rangle\langle{\rm module}_{2}\rangle...\)" as the target sequence. The resulting target sequences for dialogue and video understanding programs are sequences \(\mathcal{P}_{\mathrm{dial}}\) and \(\mathcal{P}_{\mathrm{vid}}\) respectively.

The parsers decompose questions into subsequences to construct compositional reasoning programs for dialogue and video understanding. Each parser is a vanilla Transformer decoder, including multi-head attention layers on questions and past dialogue turns (Please refer to Appendix A.1 for more technical details).

### Response Decoder

System response is decoded by incorporating the dialogue context and video context outputs from the corresponding reasoning programs to target token representations. We follows a vanilla Transformer decoder architecture Le et al. (2019b), which consists of 3 attention layers: self-attention to attend on existing tokens, attention to \(Q_{\mathrm{ctx}}\) from dialogue understanding program execution, and attention to \(V_{\mathrm{ctx}}\) from video understanding program execution.

\[A^{(1)}_{\mathrm{res}}={\rm Attention}(R|_{0}^{j-1},R|_{0}^{ j-1},R|_{0}^{j-1})\in\mathbb{R}^{j\times d}\] \[A^{(2)}_{\mathrm{res}}={\rm Attention}(A^{(1)}_{\mathrm{ res}},Q_{\mathrm{ctx}},Q_{\mathrm{ctx}})\in\mathbb{R}^{j\times d}\] \[A^{(3)}_{\mathrm{res}}={\rm Attention}(A^{(2)}_{\mathrm{ res}},V_{\mathrm{ctx}},V_{\mathrm{ctx}})\in\mathbb{R}^{j\times d}\]

Multimodal FusionFor video features come from multiple modalities, visual and audio, the contextual features, denoted \(V_{\mathrm{ctx}}\), is obtained through a weighted sum of component modalities, e.g. contextual visual features \(V^{\mathrm{vis}}_{\mathrm{ctx}}\) and contextual audio features \(V^{\mathrm{aud}}_{\mathrm{ctx}}\) . The scores \(S_{\mathrm{fusion}}\) to compute the weighted sum is defined as:

\[S_{\mathrm{fusion}}={\rm Softmax}(W_{\mathrm{fusion}}^{T}[Q_{ \mathrm{stack}};V^{\mathrm{vis}}_{\mathrm{ctx}},V^{\mathrm{aud}}_{\mathrm{ ctx}}])\]

where \(Q_{\mathrm{stack}}\) is the mean pooling output of question embeddings \(Q\) which is then stacked to \(N_{\mathrm{ent}}+N_{\mathrm{act}}\) dimensions, and \(W_{\mathrm{fusion}}\in\mathbb{R}^{3d\times 2}\) are trainable model parameters. The resulting \(S_{\mathrm{fusion}}\) has a dimension of \(\in\mathbb{R}^{\left<N_{\mathrm{ent}}+N_{\mathrm{act}}\right>\times 2}\).

Figure 4: where and when neural modules for video understanding

Figure 3: find and summarize neural modules for dialogue understanding





**Response Generation.** To generate response sequences, a special token "_sos_" is concatenated as the first token \(w_{0}\). The decoded token \(w_{1}\) is then appended to \(w_{0}\) as input to decode \(w_{2}\) and so on. Similarly to input source sequences, at decoding time step \(j\), the input target sequence is encoded to obtain representations of system response \(R|^{j-1}_{0}\). We combine vocabulary of input and output sequences and share the embedding matrix \(E\in\mathbb{R}^{[\![V]\!]\times d}\) where \(\mathbb{V}=\mathbb{V}^{in}\cap\mathbb{V}^{out}\). During training time, we directly use the ground-truth responses as input to the decoder and optimize VGNMN with a cross-entropy loss to decode the next ground-truth tokens. During test time, responses are generated auto-regressively through beam search with beam size \(5\). Note that we apply the same procedure to generate reasoning programs from question parsers.

## 4 Experiments

Datasets.We use the AVSD benchmark from the Dialogue System Technology Challenge 7 (DSTC7) Hori et al. (2019). The benchmark consists of dialogues grounded on the Charades videos Sigurdsson et al. (2016). Each dialogue contains up to 10 dialogue turns, each turn consists of a question and expected response about a given video. For visual features, we use the 3D CNN-based features from a pretrained I3D model Carreira and Zisserman (2017) and object-level features from a pretrained FasterRNN model Ren et al. (2015b). The audio features are obtained from a pretrained VGGish model Hershey et al. (2017). In the experiments with AVSD, we consider two settings: one with video summary and one without video summary as input. In the setting with video summary, the summary is concatenated to the dialogue history before the first dialogue turn. We also adapt VGNMN to the video QA benchmark TGIF-QA Jang et al. (2017). Different from AVSD, TGIF-QA contains a diverse set of QA tasks:

* _Count_: an open-ended task which counts the number of repetitions of an action
* _Action_: a multiple-choice (MC) task which asks about a certain action occurring for a fixed number of times
* _Transition_: an MC task which emphasizes temporal transition in video
* _Frame_: an open-ended QA about visual contents of one of the video frames

For the TGIF-QA benchmark, we use the extracted features from a pretrained ResNet model He et al. (2016). Table 2 shows a summary of the AVSD and TGIF-QA benchmarks.

Training Details.We follow prior approaches Hu et al. (2017, 2018); Kottur et al. (2018) by obtaining the annotations of the programs through a language parser Hu et al. (2016) and a reference resolution model Clark and Manning (2016). During training, we directly use these as ground-truth labels of programs to train our models. The ground-truth responses are augmented with label smoothing technique Szegedy et al. (2016). During inference time, we generate all programs and responses from given dialogues and videos. We run beam search to enumerate programs for dialogue and video understanding and dialogue responses.

We use a training batch size of 32 and embedding dimension \(d=128\) in all experiments. Where Transformer attention is used, we fix the number of attention heads to 8 in all attention layers. In neural modules with MLP layers, the MLP network is fixed to 2 linear layers with a ReLU activation in between. In neural modules with CNN, we adopt a vanilla CNN architecture for text classification (without the last MLP layer) where the number of input channels is 1, the kernel sizes are \(\{3,4,5\}\), and the number of output channels is \(d\). We initialize models with uniform distribution Glorot and Bengio (2010). During training, we adopt the Adam optimizer Kingma and Ba (2015) and a decaying learning rate Vaswani et al. (2017) where we fix the warm-up steps to 15K training steps. We employ dropout Srivastava et al. (2014) of 0.2 at all networks except the last linear layers of question parsers and response decoder. We train models up to 50 epochs and select the best models based on the average loss per epoch in the validation set.

All models are trained in a V100 GPU with a capacity of 16GB. We approximated each training epoch took about 20 minutes to run. For each

\begin{table}
\begin{tabular}{c l c c c} \hline  & \# & **Train** & **Val.** & **Test** \\ \hline \multirow{3}{*}{**AVSD**} & Dialogs & 7,659 & 1,787 & 1,170 \\  & Turns & 153,180 & 35,740 & 13,490 \\  & Words & 1,450,754 & 339,006 & 110,252 \\ \hline \multirow{3}{*}{**TGIFQA**} & Count QA & 24,159 & 2,684 & 3,554 \\  & Action QA & 18,428 & 2,047 & 2,274 \\  & Trans. QA & 47,434 & 5,270 & 6,232 \\  & Frame QA & 35,453 & 3,939 & 13,691 \\ \hline \end{tabular}
\end{table}
Table 2: Summary of DSTC7 AVSD and TGIF-QA benchmark

model experiment with VGNMN, we obtained at least 2 runs and reported the average results. We implemented models in Pytorch and released the code and model checkpoints 1.

Footnote 1: [https://github.com/henryhungle/vgnnn](https://github.com/henryhungle/vgnnn)

Optimization.We optimize models by joint training to minimize the cross-entropy losses to generate responses and functional programs.

\[{\mathcal{L}} =\alpha{\mathcal{L}_{\mathrm{dial}}}+\beta{\mathcal{L}_{\mathrm{ vid}}}+{\mathcal{L}_{\text{res}}}\] \[=\alpha\sum_{j}-\log(\mathbf{P}_{\mathrm{dial}}({\mathcal{P}_{ \mathrm{dial,j}}}))\] \[+\beta\sum_{l}-\log(\mathbf{P}_{\mathrm{video}}({\mathcal{P}_{ \mathrm{video,l}}}))\] \[+\sum_{n}-\log(\mathbf{P}_{\mathrm{res}}({\mathcal{R}_{n}}))\]

where \(\mathbf{P}\) is the probability distribution of an output token. The probability is computed by passing output representations from the parsers and decoder to a linear layer \(W\in\mathbb{R}^{d\times V}\) with softmax activation. We share the parameters between \(W\) and embedding matrix \(E\).

AVSD Results.We evaluate model performance by the objective metrics, including BLEU Papinieni et al. (2002), METEOR Banerjee and Lavie (2005), ROUGE-L Lin (2004), and CIDEr Vedantam et al. (2015), between each generated response and 6 reference gold responses. As seen in Table 3, our models outperform most of existing approaches. We observed that our approach did not outperform the GPT-based baselines Li et al. (2020b); Le and Hoi (2020) in the setting that allows video summary/caption input. However, the performance of our model in the setting without video summary/caption input is on par with the GPT-based baseline Li et al. (2020b), even though our model did not rely on deep pretrained representations on large-scale text data. These observations imply that GPT-based models can better capture video context from video caption/summary through rich pretrained representations. However, without access to video caption/summary, these models may fail to understand video from visual-only representations. In this setting, GPT-based models may be inferior to VGNMN, which explicitly exploits the compositional structures from textual inputs to integrate visual features. We also found that VGNMN applied to object-level features is competitive to the model applied to CNN-based features. The flexibility of VGNMN neural programs show when we treat the caption as an input equally to visual or audio inputs and execute entity-action level neural operations on the encoded caption sequence.

Robustness.To evaluate model robustness, we report BLEU4 and CIDEr of model variants in various experimental settings. Specifically, we compare against performance of output responses in the first dialogue turn position (i.e. 2\({}^{nd}\)-10\({}^{th}\) turn vs. the 1\({}^{st}\) turn), or responses grounded on the shortest video length range (video ranges are intervals of 0-10\({}^{th}\), 10-20\({}^{th}\) percentile and so on). We report results of the following model variants: (1) _w/o video NMN_: VGNMN without using video-based modules, e.g. when and where. Video features are retrieved through a token-level representation of questions Le et al. (2019b). (2) _no NMN_: (1) + without dialogue-based modules, e.g. find and summarize. Dialogue history is encoded by a hierarchical LSTM encoder Hori et al. (2019).

_Robustness to video length_: In Table 3(a), we noted that the performance gap between VGNMN and (1) is quite distinct, with 7/10 cases of video ranges in which VGNMN outperforms. However, in lower ranges (i.e. 1-23 seconds) and higher ranges (37-75 seconds), VGNMN performs not

\begin{table}
\begin{tabular}{c c|c c c c c} Model & PF & Vic. & And. & B-4 & M & R & C \\ \hline
* & 0.30€ & 0.217 & 0.481 & 0.733 \\ Baseline Hou et al. (2019) & & I & V & 0.390 € & 0.217 & 0.487 & 0.746 \\ Lee et al. (2019) & & I & V & 0.315 & 0.239 & 0.590 & 0.848 \\ FDA Schwartz et al. (2019) & & I & V & – & – & – & 0.806 \\ MAN Chu et al. (2020) & & I & V & 0.309 & 0.240 & 0.530 & 0.890 \\ Girl et al. (2019) & & I & V & 0.371 & 0.240 & 0.552 & 0.966 \\ MTN et al. (2019b) & & I & V & 0.343 & 0.247 & 0.520 & 0.936 \\ MTN et al. (2019b) & & I & V & 0.368 & 0.295 & 0.537 & 0.964 \\ MSTN Lee et al. (2020) & & I & V & 0.379 & 0.261 & 0.548 & 1.028 \\ BTST Lee et al. (2020) & & K & V & 0.390 & 0.290 & 0.552 & 1.030 \\ OTT Jütmann et al. (2020) & ✓ & I & V & **0.402** & 0.254 & 0.544 & 1.052 \\ VGNMN & & I & V & 0.397 & 0.264 & 0.548 & **0.549** \\ VGNMN & & & I & V & 0.388 & 0.295 & 0.549 & 1.040 \\ VGNMN & & & I & V & 0.381 & 0.250 & 0.534 & 1.044 \\ VGNMN & & & i & V & 0.366 & **0.263** & 0.549 & **1.059** \\ \hline
* & 0.421 & 0.277 & 0.574 & 1.717 \\ VGNMN & & & I &
- & 0.421 & 0.277 & 0.571 & 1.148 \\ VGNMN & & & I & V & 0.421 & 0.277 & 0.573 & 1.167 \\ VGNMN & & & I & V & 0.429 & 0.278 & 0.575 & 1.188 \\ \hline  & & & & & & & \\ \end{tabular}
\end{table}
Table 3: AVSD test results: Metrics are: BLEU-4 (B-4), METEOR (M), ROUGE-L (R), and CIDEr (C). The visual features are: I3D (I), ResNeXt-101 (RX), Faster-RCNN (FR), caption as a video input (C). The audio features are: VGGish (V), AdNet (A). ✓on PT denotes models using pretrained weights and/or additional finetuning. The best/second-best results are bold/underlined respectively.
 

as well as model (1). We observed that related factors might affect the discrepancy, such as the complexity of the questions for these short and long-range videos. Potentially, our question parser for the video understanding program needs to be improved (e.g. for tree-based programs) to retrieve information in these ranges.

_Robustness to dialogue turn_: In Table 3(b), we observed that model (1) performs better than model (2) overall, especially in higher turn positions, i.e. from the 4\({}^{th}\) turn to 8\({}^{th}\) turn. Interestingly, we noted some mixed results in very low turn position, i.e. the 2\({}^{nd}\) and 3\({}^{rd}\) turn, and very high turn position, i.e. the 10\({}^{th}\) turn. Potentially, with a large dialogue turn position, the neural-based approach such as hierarchical RNN can better capture the global dependencies within dialogue context than the entity-based compositional NMN method.

_Robustness to question structure_: Finally, we compared performance of VGNMN with the no-NMN variant (1) in different cases of question structures: single-question vs. multiple-part structure. In single-question structures, we examined by the question types (e.g. yes/no, wh-questions). In multi-part structures, we further classified whether there are sentences preceding the question (e.g. "1Sent+Que") or there are smaller (sub-)questions (e.g. "2SubQue") within the question. In Table 3(c), we observed that VGNMN has clearer performance gains in multi-part structures than single-question structures. In multi-part structures, we observed higher gaps between VGNMN and model (1) in highly complex cases e.g. "2Sent+Que" vs. "1Sent+Que". These observations indicate the robustness of VGNMN and the underlying compositionality principle to deal with complex question structures. We also noted that VGNMN is still susceptible to extremely long questions (">2Sent+Que") and future work is needed to address these scenarios.

Interpretability.In Figure 5, we show both success and failure cases of generated responses and corresponding generated functional programs. In each example, we marked predicted outputs as incorrect if they do not match the ground-truth completely (even though the outputs might be partially correct). From Figure 5, we observe that in cases where generated dialogue programs and video programs match or are close to the gold labels, the

Figure 5: Interpretability of model outputs on AVSD: Example A (Top) and Example B (Bottom).

 

model can generate generally correct responses. For cases where some module parameters do not exactly match but are closed to the gold labels, the model can still generate responses with the correct visual information (e.g. the \(4^{th}\) turn in example B). In cases of wrong predicted responses, we can further look at how the model understands the questions based on predicted programs. In the 3\({}^{rd}\) turn of example A, the output response is missing a minor detail as compared to the label response because the video program fails to capture "rooftop" as a where parameter. These subtle yet important details can determine whether output responses can fully address user queries. In the 3\({}^{rd}\) turn of example B, the model wrongly identifies "what room" as a where parameter and subsequently generates a wrong response that it is "a living room".

TGIF-QA Results.We report the result using the L2 loss in _Count_ task and accuracy in other tasks. From Table 5, VGNMN outperforms the majority of the baseline models in all tasks by a large margin. Compared to AVSD experiments, the TGIF-QA experiments emphasize the video understanding ability of the models, removing the requirement for dialogue understanding and natural language generation. Since TGIF-QA questions follow a very specific question type distribution (count, action, transition, and frameQA), the question structures are simpler and easier to learn than AVSD. Using exact-match accuracy of parsed programs vs. label programs as a metric, our question parser can achieve a performance 81% to 94% accuracy in TGIF-QA vs. 41-45% in AVSD. The higher accuracy in decoding a reasoning structure translates to better adaptation between training and test time, resulting in higher performance gains.

Cascading Errors.Compared to prior approaches, we noted that VGNMN is a modularized system which may result in cascading errors to downstream modules. One major error is the error of generated programs which is used as parameters in neural modules. To gauge this error, we compare the performance of VGNMN between 2 cases: with generated programs and with ground-truth programs. From Table 6, we noticed some performance gaps between these cases. These observations imply that: (1) program generations and response generations are positively correlated and more accurate programs can lead to better responses; and (2) current question parsers are not perfect, resulting in wrong parameters to instantiate neural modules. Future work may focus on learning better question parsers or directly deploying a better off-the-shelf parser tool.

For additional experiment results, qualitative samples, and analysis between model variants, refer to Appendix B and C.

## 5 Conclusion

In this work, we introduce Video-grounded Neural Module Network (VGNMN). VGNMN consists of dialogue and video understanding neural modules, each of which performs entity and action-level operations on language and video components. Our comprehensive experiments on AVSD and TGIF-QA benchmarks show that our models can achieve competitive performance while promoting a compositional and interpretable learning approach.

## 6 Broader Impacts

During the duration of this work, there have been no ethical concerns regarding the model implementation, training, and testing. The data used in this work has been carefully reviewed and accordingly to the description from the original authors, we did not find any concerns on any significant biases. For

\begin{table}
\begin{tabular}{c c c c c} \hline AVSD & BLEU4 & METEOR & ROUGE-L & CIDEr \\ \hline Gen. & 0.396 & 0.263 & 0.549 & 1.059 \\ GT & **0.408** & **0.272** & **0.560** & **1.115** \\ \hline TGIF-QA & Count & Action & Transition & FrameQA \\  & (Loss) & (Acc) & (Acc) & (Acc) \\ \hline Gen. & 2.65 & 0.845 & 0.887 & 0.747 \\ GT & **1.90** & **0.857** & **0.898** & **0.780** \\ \hline \end{tabular}
\end{table}
Table 6: Comparison of VGNMN on AVSD (top) and TGIF-QA (bottom) when using generated (“Gen.”) vs. ground-truth (“GT”) programs.

\begin{table}
\begin{tabular}{l l r r r r} \hline \multicolumn{1}{c}{**Model**} & \multirow{2}{*}{**Vis.**} & \multicolumn{2}{c}{**Count**} & \multicolumn{1}{r}{**Action**} & \multicolumn{1}{r}{**Tran.**} & \multicolumn{1}{r}{**FFOA} \\  & & **(Loss)** & **(Acc)** & **(Acc)** & **(Acc)** \\ \hline VIS [18] & R & 4.80 & 0.488 & 0.348 & 0.350 \\ MCB [16] & R & 5.17 & 0.589 & 0.243 & 0.257 \\ Yu et al. [17] & R & 5.13 & 0.561 & 0.640 & 0.396 \\ ST-YQA [18] & RaF & 4.32 & 0.629 & 0.694 & 0.495 \\ CG-Mem [19] & ReF+ & 4.10 & 0.682 & 0.743 & 0.515 \\ PSAC [19] & R & 4.27 & 0.704 & 0.769 & 0.557 \\ HME [18] & R+C & 4.02 & 0.739 & 0.778 & 0.538 \\ STA [19] & R & 4.25 & 0.723 & 0.790 & 0.566 \\ CRN-MMC [19] & R & 4.23 & 0.713 & 0.787 & 0.592 \\ MQI. [18] & V & - & - & - & 0.598 \\ Quest? [17] & R & 4.19 & 0.759 & 0.810 & 0.597 \\ HG [19] & RC+ & 4.09 & 0.754 & 0.810 & 0.551 \\ GCN [19] & R+C & 3.95 & 0.743 & 0.811 & 0.563 \\ HCN [18] & R+RX & 3.82 & 0.750 & 0.841 & 0.559 \\ HST [18] & RX & **2.14** & **0.847** & 0.819 & 0.648 \\ \hline VGNMN & R & 2.65 & 0.845 & **0.887** & **0.747** \\ \hline \end{tabular}
\end{table}
Table 5: Experiment results on the TGIF-QA benchmark. The visual features are: ResNet-152 (R), C3D (C), Flow CNN from two-stream model (F), VGG (V), ResNeXt-101 (RX).

 

any potential application or extension of this work, we would like to highlight some specific concerns. First, as the work is developed to build an intelligent dialogue agents, models should not be used with the intention to create fake human profiles for any harmful purposes (e.g. fishing or spreading fake news). For wider use of dialogue systems, the application of work might result in certain impacts to some stakeholders whose jobs may be affected by this application (e.g. customer service call agents). We hope any application should be carefully considered against these potential risks.

## References

* Agrawal et al. (2016) Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016. Analyzing the behavior of visual question answering models. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1955-1960, Austin, Texas. Association for Computational Linguistics.
* Alamri et al. (2019) Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Stefan Lee, Peter Anderson, Irfan Essa, Devi Parikh, Dhruv Batra, Anoop Cherian, Tim K. Marks, and Chiori Hori. 2019. Audio-visual scene-aware dialog. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
* Andreas et al. (2016a) Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016a. Learning to compose neural networks for question answering. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1545-1554, San Diego, California. Association for Computational Linguistics.
* Andreas et al. (2016b) Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016b. Neural module networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 39-48.
* Antol et al. (2015) Stanislaw Antol, Aishway Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In _Proceedings of the IEEE international conference on computer vision_, pages 2425-2433.
* Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. _arXiv preprint arXiv:1607.06450_.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_, pages 65-72.
* Carreira and Zisserman (2017) Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 6299-6308.
* Chu et al. (2020) Yun-Wei Chu, Kuan-Yen Lin, Chao-Chun Hsu, and Lun-Wei Ku. 2020. Multi-step joint-modality attention network for scene-aware dialogue system. _DSTC Workshop @ AAAI_.
* Clark and Manning (2016) Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2256-2262, Austin, Texas. Association for Computational Linguistics.
* Dehghani et al. (2019) Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019. Universal transformers. In _International Conference on Learning Representations_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Fan et al. (2019) Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. 2019. Heterogeneous memory enhanced multimodal attention model for video question answering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 1999-2007.
* Feng et al. (2018) Shi Feng, Eric Wallace, Alvin Grisson II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. Pathologies of neural models make interpretations difficult. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3719-3728, Brussels, Belgium. Association for Computational Linguistics.
* Fukui et al. (2016) Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. 2016. Multimodal compact bilinear pooling for visual question answering and visual grounding. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 457-468, Austin, Texas. Association for Computational Linguistics.
* Gao et al. (2018) Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. 2018. Motion-appearance co-memory networks for video question answering. In _Proceedings of the 

_IEEE Conference on Computer Vision and Pattern Recognition_, pages 6576-6585.
* Gao et al. (2019) Lianli Gao, Pengpeng Zeng, Jingkuan Song, Yuan-Fang Li, Wu Liu, Tao Mei, and Heng Tao Shen. 2019. Structured two-stream attention network for video question answering. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 6391-6398.
* Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_, pages 249-256.
* Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In _CVPR_, volume 1, page 3.
* Gupta et al. (2020) Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. 2020. Neural module networks for reasoning over text. In _International Conference on Learning Representations_.
* Han et al. (2019) Chi Han, Jiayuan Mao, Chuang Gan, Josh Tenenbaum, and Jiajun Wu. 2019. Visual concept-metaconcept learning. In _Advances in Neural Information Processing Systems_, pages 5002-5013.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 770-778.
* Hershey et al. (2017) Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. Cnn architectures for large-scale audio classification. In _Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on_, pages 131-135. IEEE.
* 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 2352-2356.
* Hori et al. (2019) Chiori Hori, Anoop Cherian, Tim K Marks, and Takaaki Hori. 2019. Joint student-teacher learning for audio-visual scene-aware dialog. _Proc. Interspeech 2019_, pages 1886-1890.
* Hu et al. (2018) Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. 2018. Explainable neural computation via stack neural module networks. In _Proceedings of the European conference on computer vision (ECCV)_, pages 53-69.
* Hu et al. (2017) Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In _Proceedings of the IEEE International Conference on Computer Vision_, pages 804-813.
* Hu et al. (2016) Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jaishi Feng, Kate Saenko, and Trevor Darrell. 2016. Natural language object retrieval. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 4555-4564.
* Huang et al. (2020) Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkuti Tan, and Chuang Gan. 2020. Location-aware graph convolutional networks for video question answering. In _The AAAI Conference on Artificial Intelligence (AAAI)_, volume 1.
* Jang et al. (2017) Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 2758-2766.
* Jiang et al. (2020) Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao. 2020. Divide and conquer: Question-guided spatio-temporal contextual attention for video question answering. In _The AAAI Conference on Artificial Intelligence (AAAI)_.
* Jiang and Han (2020) Pin Jiang and Yahong Han. 2020. Reasoning with heterogeneous graph alignment for video question answering. In _The AAAI Conference on Artificial Intelligence (AAAI)_.
* Jiang and Bansal (2019) Yichen Jiang and Mohit Bansal. 2019. Self-assembling modular networks for interpretable multi-hop reasoning. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4474-4484, Hong Kong, China. Association for Computational Linguistics.
* Johnson et al. (2017a) Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017a. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 2901-2910.
* Johnson et al. (2017b) Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017b. Inferring and executing programs for visual reasoning. In _Proceedings of the IEEE International Conference on Computer Vision_, pages 2989-2998.
* Kim et al. (2021) Seonhoom Kim, Seohyeong Jeong, Eunbyul Kim, Inho Kang, and Nojun Kwak. 2021. Self-supervised pretraining and contrastive representation learning for multiple-choice video qa. _Proceedings of the AAAI Conference on Artificial Intelligence_, 35(14):13171-13179 

Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar. Association for Computational Linguistics, pp. 1746-1751. Cited by: SS3.
* Diederick P Kingma and Jimmy Ba (2015)Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR), Cited by: SS3.
* S. Kottur, J. M. Moura, D. Parikh, D. Batra, and M. Rohrbach (2018)Visual coreference resolution in visual dialog using neural module networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 153-169. Cited by: SS3.
* S. H. Kumar, E. Okur, S. Sahay, J. Huang, and L. Nachman (2019)Leveraging topics and audio features with multimodal attention for audio visual scene-aware dialog. arXiv preprint arXiv:1912.10131. Cited by: SS3.
* H. Le, S. Hoi, D. Sahoo, and N. Chen (2019)End-to-end multimodal dialog systems with hierarchical multimodal attention on video features. In DSTC' at AAAI2019 workshop, Cited by: SS3.
* H. Le and S. C.H. Hoi (2020)Video-grounded dialogues with pretrained generation language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 5842-5848. External Links: Link, Document Cited by: SS3.
* H. Le, D. Sahoo, N. Chen, and S. Hoi (2020)BialST: bi-directional spatio-temporal reasoning for video-grounded dialogues. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 1846-1859. External Links: Link, Document Cited by: SS3.
* H. Le, D. Sahoo, N. Chen, and S. Hoi (2020)Multimodal transformer networks for end-to-end video-grounded dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 5612-5623. External Links: Link, Document Cited by: SS3.
* T. Lin (2004)Rouge: a package for automatic evaluation of summaries. Text Summarization Branches Out. Cited by: SS3.
* T. Minh Le, V. Le, S. Venkatesh, and T. Tran (2019)Learning to reason with relational video representation for question answering. arXiv preprint arXiv:1907.04553. Cited by: SS3.
* T. Minh Le, V. Le, S. Venkatesh, and T. Tran (2020)Hierarchical conditional relation networks for video question answering. arXiv preprint arXiv:2002.10698. Cited by: SS3.
* H. Lee, S. Yoon, F. Dernoncourt, D. Soon Kim, T. Bui, and Kyomin Jung (2020)Dustc8-avsd: multimodal semantic transformer network with retrieval style word generator. DSTC Workshop @AAAI 2020. Cited by: SS3.
* C. Lei, L. Wu, D. Liu, Z. Li, G. Wang, H. Tang, and H. Li (2020)Multiquestion learning for visual question answering. In The AAAI Conference on Artificial Intelligence (AAAI), Cited by: SS3.
* J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu (2021)Less is more: clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7331-7341. Cited by: SS3.
* J. Lei, L. Yu, M. Bansal, and T. Berg (2018)TVQA: localized, compositional video question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 1369-1379. External Links: Link, Document Cited by: SS3.
* L. Li, Y. Chen, Y. Cheng, Z. Gan, L. Yu, and J. Liu (2020)HERO: hierarchical encoder for Video+Language omni-representation pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 2046-2065. External Links: Link, Document Cited by: SS3.
* X. Li, Z. Song, L. Gao, X. Liu, W. Huang, X. He, and C. Gan (2019)Beyond rnns: positional self-attention with co-attention for video question answering. In The 33rd AAAI Conference on Artificial Intelligence, Vol. 8, pp. . Cited by: SS3.
* Z. Li, Z. Li, J. Zhang, Y. Feng, C. Niu, and J. Zhou (2020)BRIGING text and video: a universal multimodal transformer for video-audio scene-aware dialog. arXiv preprint arXiv:2002.00163. Cited by: SS3.
* C. Lin (2004)Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out. Cited by: SS3.
* J. Mao, C. Gan, P. Kohli, J. B. Tenenbaum, and J. Wu (2019)The neuro-symbolic concept learner: interpreting scenes, words, and sentences from natural supervision. In International Conference on Learning Representations, Cited by: SS3.
* D. T. Nguyen, S. Sharma, H. Schulz, and L. El Asri (2018)From film to video: multi-turn question answering with multi-modal context. In AAAI 2019 Dialog System Technology Challenge (DICT7) Workshop, Cited by: SS3.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311-318.

 

Mengye Ren, Ryan Kiros, and Richard Zemel. 2015a. Exploring models and data for image question answering. In _Advances in neural information processing systems_, pages 2953-2961.
* Ren et al. (2015b) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015b. Faster R-CNN: Towards real-time object detection with region proposal networks. In _Advances in Neural Information Processing Systems (NIPS)_.
* Sanabria et al. (2019) Ramon Sanabria, Shruti Palaskar, and Florian Metze. 2019. Cmu sinbad's submission for the dstc7 avsd challenge. In _DSTC7 at AAAI2019 workshop_, volume 6.
* Schwartz et al. (2019) Idan Schwartz, Seunghak Yu, Tamir Hazan, and Alexander G Schwing. 2019. Factor graph attention. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 2039-2048.
* Serban et al. (2016) Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In _Thirtieth AAAI Conference on Artificial Intelligence_.
* Serrano and Smith (2019) Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2931-2951, Florence, Italy. Association for Computational Linguistics.
* Sigurdsson et al. (2016) Gunnar A Sigurdsson, Gul Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In _European Conference on Computer Vision_, pages 510-526. Springer.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. _The Journal of Machine Learning Research_, 15(1):1929-1958.
* Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 2818-2826.
* Tang et al. (2021)Zineng Tang, Jie Lei, and Mohit Bansal. 2021. DeCEMBERT: Learning from noisy instructional videos via dense captions and entropy minimization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2415-2426, Online. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, pages 5998-6008. Curran Associates, Inc.
* Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 4566-4575.
* Yang et al. (2020) Zekun Yang, Noa Garcia, Chenhui Chu, Mayu Otani, Yuta Nakashima, and Haruo Takemura. 2020. Bert representations for video question answering. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_, pages 1556-1565.
* Yi et al. (2018) Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. 2018. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. In _Advances in Neural Information Processing Systems_, pages 1031-1042.
* Yu et al. (2017) Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. 2017. End-to-end concept word detection for video captioning, retrieval, and question answering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 3165-3173.
* Zellers et al. (2021) Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. Merlot: Multimodal neural script knowledge models. _arXiv preprint arXiv:2106.02636_.

## Appendix A Additional Model Details

### Question Parsers

To learn compositional programs, we follow Johnson et al. (2017a); Hu et al. (2017) and consider program generation as a sequence-to-sequence task. We adopt a simple template "\({\langle{\rm param}_{1}\rangle}{\langle{\rm module}_{1}\rangle}{\langle{\rm param}_{2 }\rangle}{\langle{\rm module}_{2}\rangle}...\)" as the target sequence. The resulting target sequences for dialogue and video understanding programs are sequences \({\cal P}_{\mathrm{dial}}\) and \({\cal P}_{\mathrm{vid}}\) respectively.

The parsers decompose questions into subsequences to construct compositional reasoning programs for dialogue and video understanding. Each parser is an attention-based Transformer decoder. The Transformer attention is a multi-head attention on query \(q\), key \(k\), and value \(v\) tensors, denoted as \({\rm Attention(q,k,v)}\). For each token in the q sequence , the distribution over tokens in the k sequence is used to obtain the weighted sum of the corresponding representations in the v sequence.

\[{\rm Attention($q,k,v$)=softmax(\frac{qk^{T}}{\sqrt{d_{k}}})v\in\mathbb{R}^{L _{\text{q}}\times d_{\text{q}}}}\] 

Each attention is followed by a feed-forward network applied to each position identically. We exploit the multi-head and feed-forward architecture, which show good performance in NLP tasks such as NMT and QA [2017]; Dehghani et al. (2019), to efficiently incorporate contextual cues from dialogue components to parse question into reasoning programs. At decoding step \(0\), we simply use a special token _sos_ as the input to the parser. In each subsequent decoding step, we concatenate the prior input sequence with the generated token to decode in an auto-regressive manner. We share the vocabulary sets of input and output components and thus, use the same embedding matrix. Given the encoded question \(Q\), to decode the program for dialogue understanding, the contextual signals are integrated through 2 attention layers: one attention on previously generated tokens, and the other on question tokens. At time step \(j\), we denote the output from an attention layer as \(A_{\rm dial,j}\).

\[A^{(1)}_{\rm dial}=\mathrm{Attention}(P_{\rm dial}|_{0}^{j-1}, P_{\rm dial}|_{0}^{j-1},P_{\rm dial}|_{0}^{j-1})\] \[A^{(2)}_{\rm dial}=\mathrm{Attention}(A^{(1)}_{\rm dial},Q,Q) \in\mathbb{R}^{j\times d}\]

To generate programs for video understanding, the contextual signals are learned and incorporated in a similar manner. However, to exploit dialogue contextual cues, the execution output of dialogue understanding neural modules \(Q_{\rm ctx}\) is incorporated to each vector in \(P_{\rm dial}\) through an additional attention layer. This layer integrates the resolved entity information to decode the original entities for video understanding. It is equivalent to a reasoning process that converts the question from its original multi-turn semantics to single-turn semantics,

\[A^{(1)}_{\rm vid}=\mathrm{Attention}(P_{\rm vid}|_{0}^{j-1},P_{ \rmvid}|_{0}^{j-1},P_{\rm vid}|_{0}^{j-1})\] \[A^{(2)}_{\rm vid}=\mathrm{Attention}(A^{(1)}_{\rm vid},Q,Q) \in\mathbb{R}^{j\times d}\] \[A^{(3)}_{\rm vid}=\mathrm{Attention}(A^{(2)}_{\rm vid},Q_{\rm ctx },Q_{\rmctx})\in\mathbb{R}^{j\times d}\]

### How to locate entities?

Noted that in the neural modules described in Section 3.3, during training, we simply feed the ground-truth programs to optimize these modules. For instance, the neural module where received the ground truth entities \(P\) which is then used to instantiate the neural network and retrieve from video \(V\). During test time, we decode the programs token by token through the question parsers, and feed the predicted entities \(\hat{P}\) to neural modules. Note that we do not assume, and hence not train model to retrieve ground-truth locations of visual entities in videos. This strategy enables the applicability of VGNMN as we consider these entity annotations mostly unavailable in real-world systems.

## Appendix B Additional Experimental Results

### Non-NMN Models

We experiment with several Non-NMN based variants of our models. As can be seen in Table 7, our approach to video and dialogue understanding through compositional reasoning programs exhibits better performance than non-compositional approaches. Compared to the approaches that directly process frame-level features in videos (Row B) or token-level features in dialogues (Row C, D), our full VGNMN (Row A) considers entity-level and action-level information extraction and thus, avoids unnecessary and possibly noisy extraction. Compared to the approaches that obtain dialogue contextual cues through a hierarchical encoding architecture (Row E, F) such as [56, 48], VGNMN directly addresses the challenge of entity references in dialogues. As mentioned, we hypothesize that the hierarchical encoding architecture is more appropriate for less entity-sensitive dialogues such as chit-chat and open-domain dialogues.

### Dialogue context integration

Experimenting with different ways to integrate dialogue context representations, we observe that adding an attention layer attending to question during response decoding (Row G) is not necessary. This can be explained as the representation \(Q_{\rm ctx}\) obtained from dialogue understanding program already contains contextual information of both dialogue history and question and question input is no longer needed in the decoding phase. Furthermore, we investigate the model sensitivity to natural language generation through its ability to construct linguistically correct programs and responses. To generate responses that are linguistically appropriate, VGNMN needs dialogue context representation \(Q_{\rm ctx}\) as input to the response decoder (Row H). The model also needs encoded question \(Q\) as input to the video understanding program parser to be able to decompose this sequence to entity and action module parameters (Row I) 



## Appendix C Interpretability

We extract the predicted programs and responses for some example dialogues in Figure 6, 7, 8, and 9 and report our observations:

* We observe that when the predicted programs are correct, the output responses generally match the ground-truth (See the 1\({}^{st}\) and 2\({}^{nd}\) turn in Figure 6, and the 1\({}^{st}\) and 4\({}^{th}\) turn in Figure 8) or close to the ground-truth responses (1\({}^{st}\) turn in Figure 7).
* When the output responses do not match the ground truth, we can understand the model mistakes by interpreting the predicted programs. For example, in the 3\({}^{rd}\) turn in Figure 6, the output response describes a room because the predicted video program focuses on the entity "what room" instead of the entity "an object" in the question. Another example is the 3\({}^{rd}\) turn in Figure 8 where the entity "rooftop" is missing in the video program. These mismatches can deviate the information retrieved from the video during video program execution, leading to wrong output responses with wrong visual contents.
* We also note that in some cases, one or both of the predicted programs are incorrect, but the predicted responses still match the ground-truth responses. This might be explained as the predicted module parameters are still close enough to the "gold" labels (e.g. 4\({}^{th}\) turn in Figure 6). Sometimes, our model predicted programs that are more appropriate than the ground truth. For example, in the 2\({}^{nd}\) turn in Figure 7, the program is added with a where module parameterized by the entity "the shopping bag" which was solved from the reference "them" mentioned in the question.
* We observe that for complex questions that involve more than one queries (e.g. the 3\({}^{rd}\) turn in Figure 8), it becomes more challenging to decode an appropriate video understanding program and generate responses that can address all queries.
* In Figure 9, we demonstrate some output examples of VGNMN and compare with two baselines: Baseline Hori et al. (2019) and MTN Le et al. (2019b). We noted that VGNMN can include important entities relevant to the current dialogue turn to construct output responses while other models might miss some entity details, e.g. "them/dishes" in example A and "the magazine" in example B. These small yet important details can determine the correctness of dialogue responses.



* [26] M. C. Bader, A. K. Goyal, and R. J. Brox. The role of the web-based system for a large number of applications in social networks. In _Proceedings of the 10th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 357-384, 2009.

[MISSING_PAGE_POST]

 

* [16] M. C. B. Krizhevsky, A. Vedaldi, and R. Girshick. _Deep Learning for Deep Representation_. MIT Press, Cambridge University Press, 2015.
* [17] D. L. Kingma and J. Ba. Adam: a method for stochastic optimization. In _Advances in neural information processing systems_, pages 349-358, 2014.
* [18] S. Kullback and E. T. Newman. _Elements of Information Theory_. Princeton university press, 1984.
* [19] G. Hinton and P. Hariharan. _Neural machine learning_. MIT Press, New York, NY, USA, 2012.
* [20] I. Goodfellow, I. Guyon, Y. Bengio, and J. Zisserman. Generative adversarial nets. In _NIPS_, volume 2. MIT press, 2015.
* [21] N. Mirza, F. Ozair, and X. Liu. _Generative Adversarial Networks with Applications to Machine Learning_. Springer, Berlin, Heidelberg, 2015.
* [22] O. Russakovsky et al., "A tutorial on deep learning for image recognition." _arXiv preprint arXiv:1412.6098_, 2014.
* [23] W. Tehgal, "Learning Representations from the Image Processing Systems." _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 2015.

[MISSING_PAGE_POST]