VilNMN: A Neural Module Network approach to Video-Grounded Language Tasks

Neural module networks ( NMN ) have achieved success in image-grounded tasks such as Visual Question Answering ( VQA ) on synthetic images . However , very limited work on NMN has been studied in the video-grounded language tasks . These tasks extend the complexity of traditional visual tasks with the additional visual temporal variance . Motivated by recent NMN approaches on image-grounded tasks , we introduce Visio-Linguistic Neural Module Network ( VilNMN ) to model the information retrieval process in video-grounded language tasks as a pipeline of neural modules . VilNMN first decomposes all language components to explicitly resolve any entity references and detect corresponding action-based inputs from the question . The detected entities and actions are used as parameters to instantiate neural module networks and extract visual cues from the video . Our experiments show that VilNMN can achieve promising performance on two video-grounded language tasks : video QA and video-grounded dialogues . 1 INTRODUCTION . Vision-language tasks have been studied to build intelligent systems that can perceive information from multiple modalities , such as images , videos , and text . Extended from imaged-grounded tasks , e.g . ( Antol et al. , 2015 ) , recently Jang et al . ( 2017 ) ; Lei et al . ( 2018 ) propose to use video as the grounding features . This modification poses a significant challenge to previous image-based models with the additional temporal variance through video frames . Recently Alamri et al . ( 2019 ) further develop video-grounded language research into the dialogue domain . In the proposed task , videogrounded dialogues , the dialogue agent is required to answer questions about a video over multiple dialogue turns . Using Figure 1 as an example , to answer questions correctly , a dialogue agent has to resolve references in dialogue context , e.g . “ he ” and “ it ” , and identify the original entity , e.g . “ a boy '' and “ a backpack '' . In addition , the dialogue agent also needs to identify the actions of these entities , e.g . “ carrying a backpack ” to retrieve information along the temporal dimension of the video . Current state-of-the-art approaches to video-grounded language tasks , e.g . ( Le et al. , 2019b ; Fan et al. , 2019 ) have achieved remarkable performance through the use of deep neural networks to retrieve grounding video signals based on language inputs . However , these approaches often assume the reasoning structure , including resolving references of entities and detecting the corresponding actions to retrieve visual cues , is implicitly learned . An explicit reasoning structure becomes more beneficial as the tasks complicates in two scenarios : video with complex spatial and temporal dynamics , and language inputs with sophisticated semantic dependencies , e.g . questions positioned in a dialogue context . In these cases , it becomes challenging to interpret model outputs , assess model reasoning capability , and identify errors in neural network models . Similar challenges have been observed in image-grounded tasks in which deep neural networks often exhibit shallow understanding capability as they simply exploit superficial visual cues ( Agrawal et al. , 2016 ; Goyal et al. , 2017 ; Feng et al. , 2018 ; Serrano & Smith , 2019 ) . Andreas et al . ( 2016b ) propose neural model networks ( NMNs ) by decomposing a question into sub-sequences called program and assembling a network of neural operations . Motivated by this line of research , we propose an NMN approach to video-grounded language tasks . Our approach benefits from integrating neural networks with a compositional reasoning structure to exploit low-level information signals in video . An example of the reasoning structure can be seen on the right side of Figure 1 . We propose Visio-Linguistic Neural Module Network ( VilNMN ) for video-grounded language tasks . VilNMN leverages entity-based dialogue representations as inputs to neural operations on spatial and temporal-level visual features . Previous approaches exploit question-level and token-level representations to extract question-dependent information from video ( Jang et al. , 2017 ; Fan et al. , 2019 ; Le et al. , 2019b ) . In complex videos with many entities or actions , these approaches might not be optimal to locate the right features . To exploit object-level features , VilNMN is trained to identify relevant entities first , and then to extract the temporal steps using detected actions of these entities . VilNMN is also trained to resolve any co-references in language inputs , e.g . questions in a dialogue context , to identify the original entities . Previous approaches to video-grounded dialogues often obtain question global representations in relation to dialogue context . These approaches might be suitable to represent general semantics in open-domain or chit-chat dialogues ( Serban et al. , 2016 ; Li et al. , 2016 ) but they are not ideal to detect fine-grained entity-based information as the dialogue context evolves over time . In summary , we introduce a neural module network approach to video-grounded language tasks through a reasoning pipeline with entity and action representations applied on the spatio-temporal dynamics of video . To cater to complex semantic inputs in language inputs , e.g . dialogues , our approach also allows models to resolve entity references to incorporate question representations with fine-grained entity information . In our evaluation , we achieve competitive performance on the large-scale benchmark Audio-visual Scene-aware Dialogues ( AVSD ) ( Alamri et al. , 2019 ) . We also adapt VilNMN for video QA and obtain the state-of-the-art on the TGIF-QA benchmark ( Jang et al. , 2017 ) across all tasks . Our experiments and ablation analysis indicate a potential direction to develop compositional and interpretable neural models for video-grounded language tasks . 2 RELATED WORK . Video QA has been a proxy for evaluating a model ’ s understanding capability of language and video and the task is treated as a visual information retrieval task . Jang et al . ( 2017 ) ; Gao et al . ( 2018 ) ; Jiang et al . ( 2020 ) propose to learn attention guided by question global representation to retrieve spatial-level and temporal-level visual features . Li et al . ( 2019 ) ; Fan et al . ( 2019 ) ; Jiang & Han ( 2020 ) model interaction between all pairs of question token-level representations and temporal-level features of input video . Extended from video QA , video-grounded dialogue is an emerging task that combines dialogue response generation and video-language understanding research . Nguyen et al . ( 2018 ) ; Hori et al . ( 2019 ) ; Hori et al . ( 2019 ) ; Sanabria et al . ( 2019 ) ; Le et al . ( 2019a ; b ) extend traditional QA models by adding dialogue history neural encoders . Kumar et al . ( 2019 ) enhances dialogue features with topic-level representations to express the general topic in each dialogue . Sanabria et al . ( 2019 ) considers the task as a video summary task and concatenates question and dialogue history into a single sequence and proposes to transfer parameter weights from a large-scale video summary model . Different from prior work , we dissect the question sequence and explicitly detect and decode any entities and their references . Our models also benefit from the additional insights on how models learn to use component linguistic inputs for extraction of visual information . Extending from the line of research on neural semantic parsing ( Jia & Liang , 2016 ; Liang et al. , 2017 ) , Andreas et al . ( 2016b ; a ) introduce NMNs to address visual QA by decomposing questions into linguistic sub-structures , known as programs , to instantiate a network of neural modules . NMN models have achieved significant success in synthetic image domains where complex reasoning process is required ( Johnson et al. , 2017b ; Hu et al. , 2018 ; Han et al. , 2019 ) . Our work is related to the recent work that extends NMN models to real data domains . For instance , Kottur et al . ( 2018 ) ; Jiang & Bansal ( 2019 ) ; Gupta et al . ( 2020 ) extend NMNs to visual dialogues and reading comprehension tasks . In this paper , we introduce a new approach that exploits NMN to learn dependencies between the lexical composition in language inputs and the spatio-temporal dynamics in videos . This is not present in prior NMN models which are designed to apply on a two-dimensional image input without temporal variance . In video represented as sequence of images , each represented by object-level features , applying prior NMN models require aggregating frame-level features , e.g . through average pooling , resulting in potential loss of information . An alternative solution is a late-fusion method in which an NMN model performs reasoning structure programs on sampled video frames only . An object tracking mechanism or attention mechanism is then used to fuse the output representations . Instead , we propose to construct a reasoning structure with multi-step interaction between the space-time information in video with entity-action detected in text . 3 METHOD . In this section , we present the design of our model , called Visio-Linguistic Neural Module Networks ( VilNMN ) . An overview of the model can be seen in Figure 2 . The input to the model consists of a dialogue D which is grounded on a video V . The input components include the question of current dialogue turnQ , dialogue historyH , and the features of input video , including visual and audio input . The output is a dialogue response , denoted asR . Each text input component is a sequence of words w1 , ... , wm ∈ Vin , the input vocabulary . Similarly , the output response R is a sequence of tokens w1 , ... , wn ∈ Vout , the output vocabulary . To learn compositional programs , we follow Johnson et al . ( 2017a ) ; Hu et al . ( 2017 ) and consider program generation as a sequence-to-sequence task . Different from prior approaches , our models are trained to fully generate the parameters of component modules in text . This approach is appropriate as reasoning programs in real data domains such as current video-grounded dialogues are usually shorter than those for synthetic data ( Johnson et al. , 2017a ) and thus , program generation takes less computational cost . However , module parameters , i.e . entities and actions , contain much higher semantic variance than synthetic data , and our approach facilitates better transparency and interpretability . We adopt a simple template “ 〈param1〉〈module1〉〈param2〉〈module2〉 ... ” as the target sequence . The resulting target sequences for dialogue and video understanding programs are sequences Pdial and Pvid respectively . 3.1 ENCODERS . Text Encoder . A text encoder is shared to encode text inputs , including dialogue history , questions , and captions . The text encoder converts each text sequence X = w1 , ... , wm into a sequence of embeddings X ∈ Rm×d . We use a trainable embedding matrix to map token indices to vector representations of d dimensions through a mapping function φ . These vectors are then integrated with ordering information of tokens through a positional encoding function with layer normalization ( Ba et al. , 2016 ; Vaswani et al. , 2017 ) . The embedding and positional representations are combined through element-wise summation . The encoded dialogue history and question of the current turn are defined as H = Norm ( φ ( H ) + PE ( H ) ) ∈ RLH×d and Q = Norm ( φ ( Q ) + PE ( Q ) ) ∈ RLQ×d . To decode program and response sequences auto-repressively , a special token “ _sos ” is concatenated as the first token w0 . The decoded token w1 is then appended to w0 as input to decode w2 and so on . Similarly to input source sequences , at decoding time step j , the input target sequence is encoded to obtain representations for dialogue understanding program Pdial|j−10 , video understanding program Pvid|j−10 , and system response R| j−1 0 . We combine vocabulary of input and output sequences and share the embedding matrix E ∈ R|V|×d where V = Vin ∩ Vout . Video Encoder . To encode video input , we use pre-trained models to extract visual features and audio features . We denote F as the sampled video frames or video clips . For object-level visual features , we denote O as the maximum number of objects considered in each frame . The resulting output from a pretrained object detection model is Zobj ∈ RF×O×dvis . We concatenate each object representation with the corresponding coordinates projected to dvis dimensions . We also make use of a CNN-based pre-trained model to obtain features of temporal dimension Zcnn ∈ RF×dvis . The audio feature is obtained through a pretrained audio model , Zaud ∈ RF×daud . We passed all video features through a linear transformation layer with ReLU activation to the same embedding dimension d .

Summary: The paper studies the application of neural module network to video-grounded language tasks. They propose a method dubbed Visio-Linguistic Neural Module Network (VilNMN) to retrieve spatio-temporal information in a video through a linguistic-based parsed program. In particular, VilNMN first extracts entity references and their corresponding actions in linguistic cues. This information is then being used to locate relevant information in the visual cue to arrive at the correct answer. The proposed method is evaluated on two large scales benchmarks AVSD and TGIF-QA, demonstrating competitive performance with state-of-the-art methods.