SlowGuess/NAACL_2022_OCR huggingface dataset:
Hung Le†‡§, Nancy F. Chen§, Steven C.H. Hoit‡
Neural module networks (NMN) have achieved success in image-grounded tasks such as Visual Question Answering (VQA) on synthetic images. However, very limited work on NMN has been studied in the video-grounded dialogue tasks. These tasks extend the complexity of traditional visual tasks with the additional visual temporal variance and language cross-turn dependencies. Motivated by recent NMN approaches on image-grounded tasks, we introduce Video-grounded Neural Module Network (VGNMN) to model the information retrieval process in video-grounded language tasks as a pipeline of neural modules. VGNMN first decomposes all language components in dialogues to explicitly resolve any entity references and detect corresponding action-based inputs from the question. The detected entities and actions are used as parameters to instantiate neural module networks and extract visual cues from the video. Our experiments show that VGNMN can achieve promising performance on a challenging video-grounded dialogue benchmark as well as a video QA benchmark.
Vision-language tasks have been studied to build intelligent systems that can perceive information from multiple modalities, such as images, videos, and text. Extended from image-grounded tasks, e.g. (Antol et al., 2015), recently Jang et al. (2017); Lei et al. (2018) propose to use video as the grounding features. This modification poses a significant challenge to previous image-based models with the additional temporal variance through video frames. Recently Alamri et al. (2019) further develop videogrounded language research into the dialogue domain. In the proposed task, video-grounded dialogues, the dialogue agent is required to answer questions about a video over multiple dialogue turns. Using Figure 1 as an example, to answer
Figure 1: A sample video-grounded dialogue with a demonstration of a reasoning process
questions correctly, a dialogue agent has to resolve references in dialogue context, e.g. "he" and "it", and identify the original entity, e.g. "a boy" and "a backpack". Besides, the agent also needs to identify the actions of these entities, e.g. "carrying a backpack" to retrieve information from the video.
Current state-of-the-art approaches to video-grounded dialogue tasks, e.g. (Le et al., 2019b; Fan et al., 2019) have achieved remarkable performance through the use of deep neural networks to retrieve grounding video signals based on language inputs. However, these approaches often assume the reasoning structure, including resolving references of entities and detecting the corresponding actions to retrieve visual cues, is implicitly learned. An explicit reasoning structure becomes more beneficial as the tasks complicate in two scenarios: video with complex spatial and temporal dynamics, and language inputs with sophisticated semantic dependencies, e.g. questions positioned in a dialogue context. These scenarios often challenge researchers to interpret model hidden layers, identify errors, and assess model reasoning capability.
Similar challenges have been observed in image-grounded tasks in which deep neural networks exhibit shallow understanding capability as they exploit superficial visual cues (Agrawal et al., 2016; Goyal et al., 2017; Feng et al., 2018; Serrano and
Human Language Technologies, pages 3377 - 3393
Smith, 2019). Andreas et al. (2016b) propose neural module networks (NMNs) by decomposing a question into sub-sequences called program and assembling a network of neural operations. Motivated by this line of research, we propose a new approach, VGNMN, to video-grounded language tasks. Our approach benefits from integrating neural networks with a compositional reasoning structure to exploit low-level information signals in video. An example of the reasoning structure can be seen on the right side of Figure 1.
Video-grounded Neural Module Network (VGNMN) tackles video understanding through action and entity-paramterized NMNs to retrieve video features. We first decompose question into a set of entities and extract video features related to these entities. VGNMN then extracts the temporal steps by focusing on relevant actions that are associated with these entities. VGNMN is analogous to how human processes information by gradually retrieving signals from input modalities using a set of discrete subjects and their actions.
To tackle dialogue understanding, VGNMN is trained to resolve any co-reference in language inputs, e.g. questions in a dialogue context, to identify the unique entities in each dialogue. Previous approaches to video-grounded dialogues often obtain question global representations in relation to dialogue context. These approaches might be suitable to represent general semantics in open-domain dialogues (Serban et al., 2016). However, they are not ideal to detect fine-grained information in a video-grounded dialogue which frequently entails dependencies between questions and past dialogue turns in the form of entity references.
- VGNMN, a neural module network-based approach for video-grounded dialogues.
- The approach includes a modularized system that creates a reasoning pipeline parameterized by entity and action-based representations from both dialogue and video contexts.
- Our experiments are conducted on the challenging benchmark for video-grounded dialogues, Audio-visual Scene-Aware Dialogues (AVSD) (Alamri et al., 2019) as well as TGIFQA (Jang et al., 2017) for video QA task.
- Our results indicate strong performance of VGNMN as well as improved model inter
pretability and robustness to difficult scenarios of dialogues, videos, and question structures.
2.1 Video-Language Understanding
The research of video-language understanding aims to develop a model's joint understanding capability of language, video, and their interactions. Jang et al. (2017); Gao et al. (2018); Jiang et al. (2020) propose to learn attention guided by question global representation to retrieve spatial-level and temporal-level visual features. Li et al. (2019); Fan et al. (2019); Jiang and Han (2020) model interaction between all pairs of question token-level representations and temporal-level features of the input video through similarity matrix, memory networks, and graph networks respectively. Gao et al. (2019); Le et al. (2019c, 2020b); Lei et al. (2020); Huang et al. (2020) extends the previous approach by dividing a video into equal segments, sub-sampling video frames, or considering object-level representations of input video. We propose to replace token-level and global question representations with question representations composed of specific entities and actions.
Recently, we have witnessed emerging techniques in video-language systems that exploit deep transformer-based architectures such as BERT (Devlin et al., 2019) for pretraining multimodal representations (Li et al., 2020a; Yang et al., 2020; Kim et al., 2021; Tang et al., 2021; Lei et al., 2021; Zellers et al., 2021) in very large-scale video-language datasets. While these systems can achieve impressive performance, they are not straightforward to apply in domains with limited data such as video-grounded dialogues. Moreover, as we shown in our qualitative examples, our approach facilitates better interpretability through the output of decoded functional programs.
2.2 Video-grounded Dialogues
Extended from video QA, video-grounded dialogue is an emerging task that combines dialogue response generation and video-language understanding research. This task entails a novel requirement for models to learn dialogue semantics and decode entity co-references in questions. Nguyen et al. (2018); Hori et al. (2019); Hori et al. (2019); Sanabria et al. (2019); Le et al. (2019a,b) extend traditional QA models by adding dialogue history neural encoders. Kumar et al. (2019) en
hances dialogue features with topic-level representations to express the general topic in each dialogue. Schwartz et al. (2019) treats each dialogue turn as an independent sequence and allows interaction between questions and each dialogue turn. Le et al. (2019b) encodes dialogue history as a sequence with embedding and positional representations. Different from prior work, we dissect the question sequence and explicitly detect and decode any entities and their references. Our approach also enables insights on how models extract deductive bias from dialogues to extract video information.
2.3 Neural Module Network
Neural Module Network (NMN) (Andreas et al., 2016b,a) is introduced to address visual QA by decomposing questions into linguistic sub-structures, known as programs, to instantiate a network of neural modules. NMN models have achieved success in synthetic image domains where a multi-step reasoning process is required (Johnson et al., 2017b; Hu et al., 2018; Han et al., 2019). Yi et al. (2018); Han et al. (2019); Mao et al. (2019) improve NMN models by decoupling visual-language understanding and visual concept learning. Our work is related to the recent work (Kottur et al., 2018; Jiang and Bansal, 2019; Gupta et al., 2020) that extended NMNs to image reasoning in dialogues and reading comprehension reasoning. Our approach follows the previous approaches that learn to generate program structure and require no parser at evaluation time. Compared to prior work, we use NMN to learn dependencies between the composition in language inputs and the spatio-temporal dynamics in videos. Specifically, we propose to construct a reasoning structure from text, from which detected entities are used to extract visual information in the spatial space and detected actions are used to find visual information in the temporal space.
In this section, we present the design of our model. An overview of the model can be seen in Figure 2.
The input to the model consists of a dialogue  $\mathcal{D}$  which is grounded on a video  $\nu$ . The input components include the question of current dialogue turn  $\mathcal{Q}$ , dialogue history  $\mathcal{H}$ , and the features of the input video, including visual and audio input. The output is a dialogue response, denoted as  $\mathcal{R}$ .
Figure 2: An overview of the VGNMN approach. Colorful boxes are network components and the rest are input/output. Dotted lines are for optional components.
Each text input component is a sequence of words  $w_{1},\ldots ,w_{m}\in \mathbb{V}^{in}$ , the input vocabulary. Similarly, the output response  $\mathcal{R}$  is a sequence of tokens  $w_{1},\ldots ,w_{n}\in \mathbb{V}^{out}$ , the output vocabulary. The objective of the task is the generation objective that output answers of the current dialogue turn  $t$ :
$$
\begin{array}{l} \hat {\mathcal {R}} _ {t} = \underset {\mathcal {R} _ {t}} {\arg \max} P (\mathcal {R} _ {t} | \mathcal {V}, \mathcal {H} _ {t}, \mathcal {Q} _ {t}; \theta) \\ = \underset {\mathcal {R} _ {t}} {\arg \max } \prod_ {n = 1} ^ {L _ {\mathcal {R}}} P _ {m} (w _ {n} | \mathcal {R} _ {t, 1: n - 1}, \mathcal {V}, \mathcal {H} _ {t}, \mathcal {Q} _ {t}; \theta) \\ \end{array}
$$
where  $L_{\mathcal{R}}$  is the length of the sequence  $\mathcal{R}$ . In a Video-QA task, the dialogue history  $\mathcal{H}$  is simply absent and the output response is typically collapsed to a single-token response.
3.2 Encoders
Text Encoder. A text encoder is shared to encode text inputs, including dialogue history, questions, and captions. The text encoder converts each text sequence  $\mathcal{X} = w_{1},\dots,w_{m}$  into a sequence of embeddings  $X\in \mathbb{R}^{m\times d}$ . We use a trainable embedding matrix to map token indices to vector representations of  $d$  dimensions through a mapping function  $\phi$ . These vectors are then integrated with ordering information of tokens through a positional encoding function with layer normalization (Ba et al., 2016; Vaswani et al., 2017). The embedding and positional representations are combined through element-wise summation. The encoded dialogue history and question of the current turn are defined as  $H = \mathrm{Norm}(\phi (\mathcal{H}) + \mathrm{PE}(\mathcal{H}))\in \mathbb{R}^{L_{\mathrm{H}}\times d}$  and  $Q = \mathrm{Norm}(\phi (\mathcal{Q}) + \mathrm{PE}(\mathcal{Q}))\in \mathbb{R}^{L_{\mathrm{Q}}\times d}$ .
Video Encoder. To encode video, we use pretrained models to extract visual and audio features. We denote  $F$  as the sampled video frames or video clips. For object-level visual features, we denote  $O$  as the maximum number of objects considered in each frame. The resulting output from a pretrained object detection model is  $Z_{\mathrm{obj}} \in \mathbb{R}^{F \times O \times d_{\mathrm{vis}}}$ . We concatenate each object representation with the
Module	Input	Output	Description
find	P,H	Hent	For related entities in question, select the relevant tokens from dialogue history
summarize	Hent,Q	Qctx	Based on contextual entity representations, summarise the question semantics
where	P,V	Vent	Select the relevant spatial position corresponding to original (resolved) entities
when	P,Vent	Vent+act	Select the relevant entity-aware temporal steps corresponding to the action parameter
describe	P,Vent+act	Vctx	Select visual entity-action features based on non-binary question types
exist	Q,Vent+act	Vctx	Select visual entity-action features based on binary (yes/no) question types
Table 1: Description of the modules and their functionalities. We denote  $P$  as the parameter to instantiate each module,  $H$  as the dialogue history,  $Q$  as the question of the current dialogue turn, and  $V$  as video input.
corresponding coordinates projected to  $d_{\mathrm{vis}}$  dimensions. We also make use of a CNN-based pretrained model to obtain features of temporal dimension  $Z_{\mathrm{cnn}} \in \mathbb{R}^{F \times d_{\mathrm{vis}}}$ . The audio feature is obtained through a pretrained audio model,  $Z_{\mathrm{aud}} \in \mathbb{R}^{F \times d_{\mathrm{aud}}}$ . We passed all video features through a linear transformation layer with ReLU activation to the same embedding dimension  $d$ .
3.3 Neural Modules
We introduce neural modules that are used to assemble an executable program constructed by the generated sequence from question parsers. We provide an overview of neural modules in Table 1 and demonstrate dialogue understanding and video understanding modules in Figure 3 and 4 respectively. Each module parameter, e.g. "a backpack", is extracted from the parsed program (See Section 3.4). For each parameter, we denote  $P \in \mathbb{R}^d$  as the average pooling of component token embeddings.
find  $(\mathbb{P},\mathbb{H})\to \mathbb{H}_{\mathrm{ent}}$  . This module handles entity tracing by obtaining a distribution over tokens in the dialogue history. We use an entity-to-dialogue-history attention mechanism applied from an entity  $P_{i}$  to all tokens in the dialogue history. Any neural network that learn to generate attention between two tensors is applicable .e.g. (Bahdanau et al., 2015; Vaswani et al., 2017). The attention matrix normalized by softmax,  $A_{\mathrm{find,i}}\in \mathbb{R}^{L_{\mathrm{H}}}$  , is used to compute the weighted sum of dialogue history token representations. The output is combined with entity embedding  $P_{i}$  to obtain contextual entity representation  $H_{\mathrm{ent,i}}\in \mathbb{R}^d$
summarize  $(\mathsf{H}_{\mathrm{ent}},\mathsf{Q})\to \mathsf{Q}_{\mathrm{ctx}}$  . For each contextual entity representation  $H_{\mathrm{ent,i}}$ $i = 1,\dots ,N_{\mathrm{ent}}$  it is projected to  $L_{\mathbb{Q}}$  dimensions and is combined with question token embeddings through elementwise summation to obtain entity-aware question representation  $Q_{\mathrm{ent,i}}\in \mathbb{R}^{L_{\mathbb{Q}}\times d}$  . It is fed to a one-dimensional CNN with max-pooling layer (Kim, 2014) to obtain a contextual entity-aware question representation. We denote the final output
as  $Q_{\mathrm{ctx}} \in \mathbb{R}^{N_{\mathrm{ent}} \times d}$ .
While previous models usually focus on global or token-level dependencies (Hori et al., 2019; Le et al., 2019b) to encode question features, our modules compress fine-grained question representations at the entity level. Specifically, find and summarize modules can generate entity-dependent local and global representations of question semantics. We show that our modularized approach can achieve better performance and transparency than traditional approaches to encode dialogue context (Serban et al., 2016; Vaswani et al., 2017) (Section 4).
where  $(\mathbf{P},\mathbf{V})\to \mathbf{V}_{\mathrm{ent}}$  Similar to the find module, this module handles entity-based attention to the video input. However, the entity representation  $P$  in this case, is parameterized by the original entity in dialogue rather than in question (See Section 3.4 for more description). Each entity  $P_{i}$  is stacked to match the number of sampled video frames/clips  $F$  . An attention network is used to obtain entity-to-object attention matrix  $A_{\mathrm{where,i}}\in \mathbb{R}^{F\times O}$  . The attended feature are compressed through weighted sum pooling along the spatial dimension, resulting in  $V_{\mathrm{ent,i}}\in \mathbb{R}^{F\times d}$ $i = 1,\dots,N_{\mathrm{ent}}$
when  $(\mathbb{P},\mathbf{V}_{\mathrm{ent}})\to \mathbf{V}_{\mathrm{ent + act}}$  . This module follows a similar architecture as the where module. However, the action parameter  $P_{i}$  is stacked to match  $N_{\mathrm{ent}}$  dimensions. The attention matrix  $A_{\mathrm{when,i}}\in \mathbb{R}^{F}$  is then used to compute the visual entity-action representations through weighted sum along the temporal dimension. We denote the output for all actions  $P_{i}$  as  $V_{\mathrm{ent + act}}\in \mathbb{R}^{N_{\mathrm{ent}}\times N_{\mathrm{act}}\times d}$
describe  $(\mathbb{P},\mathbf{V}_{\mathrm{ent + act}})\to \mathbf{V}_{\mathrm{ctx}}$  . This module is a linear transformation to compute  $V_{\mathrm{ctx}} = W_{\mathrm{desc}}^{T}[V_{\mathrm{ent + act}};P_{\mathrm{stack}}]\in \mathbb{R}^{N_{\mathrm{ent}}\times N_{\mathrm{act}}\times d}$  where  $W_{\mathrm{desc}}\in \mathbb{R}^{2d\times d}$ $P_{\mathrm{stack}}$  is the stacked representations of parameter embedding  $P$  to  $N_{\mathrm{ent}}\times N_{\mathrm{act}}$  dimensions, and  $[\cdot ]$  is the concatenation operation. Note that the parameter  $P$  here is extracted from questions, often as the type of questions e.g. "what"
Figure 3: find and summarize neural modules for dialogue understanding
Figure 4: where and when neural modules for video understanding
and "how". This eliminates the need to have different modules for different question types. However, we noted the current design may be challenged in rare cases in which an utterance contain numerous questions (refer to Figure 5).
The exist module is used when the questions are "yes/no" questions. This module is a special case of describe module where the parameter  $P$  is simply the average pooled question embeddings. The above where module is applied to object-level features. For temporal-based features such as CNN-based and audio features, the same neural operation is applied along the temporal dimension. Each resulting entity-aware output is then incorporated to frame-level features through element-wise summation.
An advantage of our architecture is that it separates dialogue and video understanding. We adopt a transparent approach to solve linguistic entity references during the dialogue understanding phase. The resolved entities are fed to the video understanding phase to learn entity-action dynamics in the video. We show that our approach is robust when dialogue evolves to many turns and video extends over time (Please refer to Section 4).
3.4 Question Parsers
To learn compositional programs, we follow (Johnson et al., 2017a; Hu et al., 2017) and consider program generation as a sequence-to-sequence task. We adopt a simple template “ $\langle \mathrm{param}_1\rangle \langle \mathrm{module}_1\rangle \langle \mathrm{param}_2\rangle \langle \mathrm{module}_2\rangle \ldots$ ” as the target sequence. The resulting target sequences for dialogue and video understanding programs are sequences  $\mathcal{P}_{\mathrm{dial}}$  and  $\mathcal{P}_{\mathrm{vid}}$  respectively.
The parsers decompose questions into sub-
sequences to construct compositional reasoning programs for dialogue and video understanding. Each parser is a vanilla Transformer decoder, including multi-head attention layers on questions and past dialogue turns (Please refer to Appendix A.1 for more technical details).
3.5 Response Decoder
System response is decoded by incorporating the dialogue context and video context outputs from the corresponding reasoning programs to target token representations. We follows a vanilla Transformer decoder architecture (Le et al., 2019b), which consists of 3 attention layers: self-attention to attend on existing tokens, attention to  $Q_{\mathrm{ctx}}$  from dialogue understanding program execution, and attention to  $V_{\mathrm{ctx}}$  from video understanding program execution.
$$
A _ {\mathrm {r e s}} ^ {(1)} = \mathrm {A t t e n t i o n} (R | _ {0} ^ {j - 1}, R | _ {0} ^ {j - 1}, R | _ {0} ^ {j - 1}) \in \mathbb {R} ^ {j \times d}
$$
$$
A _ {\mathrm {r e s}} ^ {(2)} = \operatorname {A t t e n t i o n} \left(A _ {\mathrm {r e s}} ^ {(1)}, Q _ {\mathrm {c t x}}, Q _ {\mathrm {c t x}}\right) \in \mathbb {R} ^ {j \times d}
$$
$$
A _ {\mathrm {r e s}} ^ {(3)} = \operatorname {A t t e n t i o n} \left(A _ {\mathrm {r e s}} ^ {(2)}, V _ {\mathrm {c t x}}, V _ {\mathrm {c t x}}\right) \in \mathbb {R} ^ {j \times d}
$$
Multimodal Fusion. For video features come from multiple modalities, visual and audio, the contextual features, denoted  $V_{\mathrm{ctx}}$ , is obtained through a weighted sum of component modalities, e.g. contextual visual features  $V_{\mathrm{ctx}}^{\mathrm{vis}}$  and contextual audio features  $V_{\mathrm{ctx}}^{\mathrm{aud}}$ . The scores  $S_{\mathrm{fusion}}$  to compute the weighted sum is defined as:
$$
S _ {\mathrm {f u s i o n}} = \operatorname {S o f t m a x} \left(W _ {\mathrm {f u s i o n}} ^ {T} \left[ Q _ {\mathrm {s t a c k}}; V _ {\mathrm {c t x}} ^ {\mathrm {v i s}}; V _ {\mathrm {c t x}} ^ {\mathrm {a u d}} \right]\right)
$$
where  $Q_{\mathrm{stack}}$  is the mean pooling output of question embeddings  $Q$  which is then stacked to  $N_{\mathrm{ent}} + N_{\mathrm{act}}$  dimensions, and  $W_{\mathrm{fusion}} \in \mathbb{R}^{3d \times 2}$  are trainable model parameters. The resulting  $S_{\mathrm{fusion}}$  has a dimension of  $\in \mathbb{R}^{(N_{\mathrm{ent}} + N_{\mathrm{act}}) \times 2}$ .
Response Generation. To generate response sequences, a special token "\_sos" is concatenated as the first token  $w_{0}$ . The decoded token  $w_{1}$  is then appended to  $w_{0}$  as input to decode  $w_{2}$  and so on. Similarly to input source sequences, at decoding time step  $j$ , the input target sequence is encoded to obtain representations of system response  $R|_{0}^{j-1}$ . We combine vocabulary of input and output sequences and share the embedding matrix  $E \in \mathbb{R}^{|V| \times d}$  where  $\mathbb{V} = \mathbb{V}^{in} \cap \mathbb{V}^{out}$ . During training time, we directly use the ground-truth responses as input to the decoder and optimize VGNMN with a cross-entropy loss to decode the next ground-truth tokens. During test time, responses are generated auto-regressively through beam search with beam size 5. Note that we apply the same procedure to generate reasoning programs from question parsers.