VilNMN: A Neural Module Network approach to Video-Grounded Language Tasks
Authors: Hung Le
Nancy F. Chen
Steven Hoi
Venue:  (2021)
Keywords: neural modular networks
video-grounded dialogues
dialogue understanding
video understanding
video QA
video-grounded language tasks

Abstract: Neural module networks (NMN) have achieved success in image-grounded tasks such as question answering (QA) on synthetic images. However, very limited work on NMN has been studied in the video-grounded language tasks. These tasks extend the complexity of traditional visual tasks with the additional visual temporal variance.  Motivated by recent NMN approaches on image-grounded tasks, we introduce Visio-Linguistic Neural Module Network (VilNMN) to model the information retrieval process in video-grounded language tasks as a pipeline of neural modules. VilNMN first decomposes all language components to explicitly resolves entity references and detect corresponding action-based inputs from the question. Detected entities and actions are used as parameters to instantiate neural module networks and extract visual cues from the video. Our experiments show that VilNMN can achieve promising performance on two video-grounded language tasks: video QA and video-grounded dialogues. 

Under review as a conference paper at ICLR 2021
VILNMN: A NEURAL MODULE NETWORK APPROACH TO VIDEO-GROUNDED LANGUAGE TASKS
Anonymous authors Paper under double-bling review
ABSTRACT
Neural module networks (NMN) have achieved success in image-grounded tasks such Visual Question Answering (VQA) synthetic images. However, very limited work on NMN has been studied in the video-grounded language tasks. These tasks extend the complexity of traditional visual tasks with the additional vi- sual temporal variance. Motivated by recent NMN approaches on image-grounded tasks, we introduce Visio-Linguistic Neural Module Network (ViINMN) to model the information retrieval process in video-grounde language tasks as pipeline of neural modules. VilNMN first decomposes all language components to explicitly resolve any entity references and detect corresponding action-based inputs from the question. The detected entities and actions are used as parameters to instantiate neural module networks and extract visual cues from the video. Our experiments show that VilNMN can achieve promising performance on two video-grounded language tasks: video QA and video-grounde grounded dialogues.
INTRODUCTION
Vision-language tasks have been studied to build intelligent systems that can perceive information from multiple modalities, such images, videos, and text. Extended from imaged tasks, e.g. (Antol et al.] 2015),, recently Jang et al. (2017); Lei et al. (2018) propose to use video as the grounding features. This modification poses significant challenge previous image-based models with the additional temporal variance through video frames. Recently AAlami et al. (2019) further develop video- giieud-groddedd language research into the dialogue domain. In the proposed task, video- grounded dialogues, the dialogue agent is required to answer questions about video over multiple dialogue turns. Using Figure [ as an example, to answer questions correctly, a dialogue agent has to resolve references in dialogue context, e.g. "he" and "it", and identify the original entity, e.g. "a boy" and "a backpack". In addition, the dialogue agent also needs to identify the actions of these entities, e.g. "carrying backpack" to retrieve information along the temporal dimension of the video.
Current tlate-of-thh-as approaches to groeuned language tasks, c.g. (Le et al, 2019b Fan et al., 2019) have achieved remarkable performance through the of deep neural networks retrieve grounding video signals based on language inputs. However, these approaches often assume the reasoning structure, including resolving references of entities and detecting the corresponding actions to retrieve visual cues, is implicitly learned. An explicit reasoning structure becomes more beneficial as the tasks complicates in two scenarios: video with complex spatial and temporal dynamics, and language inputs with sophisticated semantic dependencies, e.g. questions positioned in dialogue context. In these cases, becomes challenging interpret model outputs, assess model reasoning capability, and identify errors in neural network models.
Similar challenges have been observed image-grounded tasks which deep neural networks often exhibit shallow understanding capability as they simply exploit superficial visual cues (Agrawal et al. 2016; Goyal et al. 2017 Feng et al., 2018 Serrano & Smith 2019). AAndres et al. (2016b) propose neural model networks (NMNs) by decomposing a question into sub- sub-seeuences called program and assembling a network of neural operations. Motivated by this line of research, we propose an NMN approach to video-groe grounded language tasks. Our approach benefits from integrating neural networks with a compositional reasoning structure to exploit low-level information signals in video. An example of the reasoning structure be seen the right side of Figure |

---

Under review as a conference paper at ICLR 2021
Video
Predicted Answer: he is cleaning a mirror
Caption: a boy and a man walk to the room. The boy carries backpack the man...
Visual:
Khon pinild
deacsribe (sbai)
shoze
Video Understanding
Audio:
Dialog History
Question: how many people are in the video?
Answer: there are a boy and man
T
Find Iite:
Question: what the boy doing?
Surmeard f
Answer: the boy walks downstairs and carries a backpack
Find
Question: what he doing while carrying
Dialogue Understanding
Figure 1: A sample rideo-grounde dialogue: Inputs question, dialogue history, video with caption, visual and audio input, and the output is the answer the question. On the right side, we demonstrate an example symbolic reasoning process a dialogue agent can perform to extract textual and visual clues for the answer.
We propose Lisio-Linguistic Neural Module Network (ViINMN) for video-grounded language tasks. VilNMN leverages entity-based dialogue representations as inputs to neural operations on spatial and temporal-level visual features. Previous approaches exploit question-level and token-level representations to extract question-dependent information from video (Jang et al.) [0117] Fan et al.) 2019; Le et al., 2019b). complex videos with many entities or actions, these approaches might not be optimal to locate the right features. To exploit object-level features, VilNMN is trained to identify
relevant entities first, and then to extract the temporal steps using detected actions of these entities.
VilNMN is also trained to resolve any co-references in language inputs, e.g. questions in dialogue context, to identify the original entities. Previous approaches to grooudgr dialogues often obtain question global representations in relation to dialogue context. These approaches might be suitable to represent general semanties in open-domain or chit-chat dialogues (Serban et al., 2016 Li al. 2016 but they not ideal to detect fine-grained entity- based information the dialogue context evolves over time.
In summary, we introduce neural module network approach to video-grae deloo-grounde language tasks through a reasoning pipeline with entity and action representations applied on the spatio-temporal dynamics of video. To cater complex semantic inputs in language inputs, e.g. dialogues, our approach also allows models to resolve entity references to incorporate question representations with fine-grained entity information. In our evaluation, we achieve competitive performance on the large-scale benchmark Audio-visual Scene-aware Dialogues (AVSD) (Alamri et al., 2019). We also adapt ViINMN for video QA and obtain the state-of-thee the-art on the TGIF- QA benchmark (Jang et al., 2017) across all tasks. Our experiments and ablation analysis indicate a potential direction to develop compositional and interpretable neural models for groundee language tasks.
2 RELATED WORK
Video QA has been a proxy for evaluating a model's understanding capability of language and video
and the task is treated visual information retrieval task. Dang al. (2017); Gao et al. (2018);
Jiang et al. (2020) propose to learn attention guided by question global representation to retrieve
spatial-level and temporal-level visual features. Li al. (2019); Fan et al. 2019); Diang & Han (2020)
model interaction between all pairs of question token-level representations and temporal-level features of input video. Extended from video QA. video-grounded dialogue an emerging task that combines
dialogue response generation and video-language understanding research. Nguyen et al. (2018): Hori
jet al. (2099): HHor et al. (2019); Sanabria et al. (201)); Le et al. (2019a (b) extend traditional QA
models by adding dialogue history neural encoders. Kumar et al. 2019) enhances dialogue features with topic-level representations to express the general topic in each dialogue, Sanabria et al. (2019)
considers the task as a video summary task and concatenates question and dialogue history into single sequence and proposes to transfer parameter weights from a large-scale video summary model. Different from prior work, we dissect the question sequence and explicitly detect and decode any entities and their references. Our models also benefit from the additional insights on how models leam to use component linguistic inputs for extraction of visual information.
2

---

Under review as a conference paper at ICLR 2021
Dial. History
Queslion Parser
Dialog Understanding
Entity-level
Question
Neural Modules
Dialogue Context
Response
Response
Concat.
Question Parser
Video Underslanding Neural Modules
Entiy/Actiion-evell Video Context
Caption
Video
VisualiAudio
Video Encoder
Figure 2: ViINMN includes 4 major components: (1) encoders that encode dialogue and video components into continuous representations; (2) question parsers that parse question of the current dialogue into compositional programs for dialogue and vidco understanding: (3) an inventory neural modules that operate on dialogue and video imput components; and (4) a response decoder that generates natural language sequence using maalloue--bsed and video-based execution outputs.
Extending from the line of research on neural semantic parsing (Jia Liang 2016 Liang et al. 2017), Andreas al.) (2016b a) introduce NMNs address visual QA by decomposing questions into linguistic sub-structures, known as programs, to instantiate a network of neural modules. NMN models have achieved significant success synthetic image domains where complex reasoning process is required (Johnson et al. 201 7b Hu et al. 2018 Han et al. 2019). Our work is related to the recent work that extends NMN models real data domains. For instance, Kottur al. (2188) Diang Bansal 2019); Gupta et al. 2020) extend NMNs to visual dialogues and reading comprehension tasks. In this paper, we introduce a new approach that exploits NMN learn dependencies between the lexical composition in language inputs and the spatio-tempore dynamics in videos. This is not present prior NMN models which are designed to apply on a two-dimensional image input without temporal sequence of images, each represented by object-level features, applying prior NMN models require aggregating frame-level features, c.g. through average pooling, resulting in potential loss of information. An alternative solution is late-fusion method in which an NMN model performs reasoning structure programs on sampled video frames only. An object tracking mechanism attention mechanism is then used fuse the output representations. Instead, we propose to construct reasoning structure with multi-step interaction between the space-time information in video with entity-action detected in text.
variance. In video represented
3 METHOD
In this section, we present the design our model, called Visio-Linguistic Neural Module Networks (VilNMN). An overview of the model be seen in Figure The input the model consists of dialogue D which is grounded on a video V. The input components include the question of current dialogue turn Q, dialogue history H. and the features of input video, including visual and audio input, The output is a dialogue response, denoted as R. Each text input component is a sequence of words W1,..., Wm yrán the input vocabulary. Similarly, the output response R. is sequence of tokens Wi,..., E grout, the output vocabulary.
To learn compositional programs, we follow Johnson et al. (2017a); Hu et al. (2017) and consider sequence-to-sequees
program generation
task. Different from prior approaches, our models are trained fully generate the parameters of component modules in text. This approach is appropriate as reasoning programs in real data domains such as current indeo-groundes dialogues are usually shorter than those for synthetic data (Johnson et al. 2017a) and thus, program generation takes less computational cost. However, module parameters, i.e. entities and actions, contain much higher semantic variance than synthetic data, and our approach facilitates better transparency and interpretability. We adopt simple template "(param,) (module;) (param,) (modulez)..." as the target sequence. The resulting target sequences for dialogue and video understanding programs are sequences Paial and Peid respectively.
3

---

Under review as a conference paper at ICLR 2021
Table 1: Description of the modules and their functionalities. We denote P as the parameter instantiate each module, H as the dialogue history, Q as the question of the current dialogue turn, and V as video input.
Module
Input
Output
Deseription
find
F, H
Hent
For related entities in question, select the relevant tokens from dialogue history
summari
Hent.
Oetx
Based contextual entity representations, summarise the question semantics
where
P.V
Vent
Select the relevant spatial position corresponding to original (resolved) entities
when
P, Vant
Van++ouu
Select the relevant entity-aware temporal steps corresponding to the action parameter
describe
Veut act
Veta
Select visual entity features based non-binary question types
exist
O. Ventstact
Vets
Select visual entity- action features based on binary (yes/no) question types
3.1 ENCODERS
Text Encoder. A text encoder is shared to encode text inputs, including dialogue history, questions, and captions. The text encoder converts each text sequence X' Wi;,.., Wm into a sequence of embeddings X E IRmxxd We use a trainable embedding matrix to map token indices to vector representations of d dimensions through a mapping function Φ. These vectors are then integrated with ordering information of tokens through a positional encoding function with layer normalization (Ba et al., 2016; Vaswani et al., 2017). The embedding and positional representations are combined through element- summation. The encoded dialogue history and question of the current turn are defined as H Norm(6(H)) + PE())) E R Andd and Q = Norm(o( + PE()) E Rdaaxd
To decode program and response sequences auto-repressively, special token "_sos" is concatenated as the first token wop The decoded token wi then appended to wi as input to decode w2 and so on. Similarly to input source sequences, at decoding time step j. the input target sequence is encoded to obtain representations for dialogue understanding program Pa video understanding program Pvialo-', and system response Rlg We combine vocabulary of input and output sequences and share the embedding matrix E R [V] where V =vn yout
Video Encoder. To encode video input, we use pre-trained models to extract visual features and audio features. We denote F as the sampled video frames or video clips. For object-level visual features, we denote O as the maximum number of objects considered in each frame. The resulting output from a pretrained object detection model is Zobj E FxOxdi,, We concatenate each object representation with the corresponding coordinates projected to dyis dimensions. We also make use of a CNN-based pre-trained model to obtain features of temporal dimension Zenn E TR. Fxdeis The audio feature is obtained through a pretrained audio model, Zaud E IR Fxdaud, We passed all video features through a linear transformation layer with ReLU activation to the same embedding dimension d.
3.2 NEURAL MODULES
We introduce neural modules that used assemble an executable program constructed by the generated sequence from question parsers. We provide an overview of neural modules in Table [[ andd demonstrate dialogue understanding and video understanding modules in Figure B and 4 respectively. Each module parameter, e.g. "a backpack", " is extracted from the parsed program. For each parameter, we denote P E Edd as the average pooling of component token embeddings.
find (P, H) Hent- This module handles entity tracing by obtaining a distribution over tokens in dialogue history. We use an moiit--o-diallouu-hhie
attention mechanism applied from an entity P; to all tokens in dialogue history. Any neural network that learn to generate attention between two tensors is applicable .e.g. (Bahdanau et al. 2015 Vaswani et al., 2017). The attention matrix normalized by softmax, Aind,i E tm, is used to compute the weighted sum of dialogue history token representations. The output is combined with entity embedding P. to obtain contextual entity representation Hent,; Rdd
summarize (Heat 2) Retx- For each contextual entity representation Henti = 1,..., Nent. is projected to La dimensions and is combined with question token embeddings through element- wise summation obtain entity-aware question representation Qent.i R xd It is fed to one-dimensional CNN with max pooling layer (Kim) 2014) to obtain a contextual entity-aware question representation. We denote the final output Qatx
IR Xd.
While previous models usually focus on global or token-level dependencies (Hori et al. 2019 Le
al. 2019b) to encode question features, our modules compress fine-grained question representations
4

---

Under review as a conference paper at ICLR 2021
Find (he)
cunnarize
"to"
MLP
max pocling
MLP
there bcy
Allention
there are
Sum paoling
Dialog History
what is he doing while carrying it? Question
stacking
(it))
Dialogue Context
Figure 3: find and suruaa ze neural modules for dialogue understanding
at entity level. Specifically, find and summariz ze modules can generate entity-depende local and global representations of question semantics. We show that our modularized approach can achieve better performance and transparency than traditional approaches encode dialogue context (Serban) et al. 2016, Vaswani et al. 2017) (Section
where (P, v) Veent- Similar to the find module, this module handle entity-based attention to the video input. However, the entity representation P in this case is parameterized by the original entity in dialogue rather than in question (See Section 3.3 for more description). Each entity P. is stacked to match the number of sampled video frames/clips F. An attention network is used to obtain entity-to-object attention matrix Awhere, E IR FxO. The attended feature compressed through weighted sum pooling along the spatial dimension, resulting in Vent,1 E R Fxd i L,..,,ennnt
when (P, Vent) -Vent-actt This module follows a similar architecture as the where module. However, the action parameter P; is stacked match Nent dimensions. The attention matrix Awhen.i E FRF then used to compute the visual entity-action representations through weighted sum along the temporal dimension. We denote the output for all actions P, as Vent+act E Nea x Nact xd
describe (P, Veut-tact) etxx This module is linear transformation to compute Vetx Waese T [Veut+act; ( Pstack] R Nens Naa xd where Wdese Rddxd Pstack is the stacked representa- tions of parameter embedding P to Neat x Nact dimensions, and [: the concatenation operation.
The exist module is special
of describe module where the parameter P is the average pooled question embeddings. The above where module is applied to object-level features. For temporal-based features such CNN-based and audio features, the along the temporal dimension. Each resulting entity-aware output is then incorporated to frame-level features through element-wiss summation (Please refer Appendix A.
neural operation is applied
boy!
whan (while carcying 18)
Stacking
carrytng
Stacking
Spatial
Athention
pooling
Temporal pooling
Attantion
backpack"
Visual Entiy-Action
whare (a backpack)
Figure 4: where and when neural modules for vidco understanding
An advantage our architecture is that separates dialogue and video understanding. We adopt a transparent approach to solve linguistic entity references during the dialogue understanding phase. The resolved entities are fed to the video understanding phase to learn entity-action dynamics in video. We show that approach is robust when dialogue evolves to many turns and video extends over time (Please see Section and Appendix
3.3 DECODERS
Question parsers. The parsers decompose questions into sub-sequences to construct compositional reasoning programs for dialogue and video understanding, Each parser is an iitention-based Trans- former decoder. Given the encoded question Q, to decode program for dialogue understanding, the contextual signals are integrated through attention layers: one attention on previously generated tokens, and the other question tokens.
To generate programs for video understanding, the contextual signals are learned and incorporated in similar manner. However, exploit dialogue contextual cues, the execution output of dialogue
5

---

Under review as a conference paper at ICLR 2021
understanding neural modules Qetx (See Section 3.2)
is incorporated to each vector in Pvid through an additional attention layer. This layer integrates the antity-dependent contextual representations from Qetx to explicitly decode the original entities for video understanding programs.
Response Decoder. System response is decoded by incorporating the dialogue context and video context outputs from the corresponding reasoning programs target token representations. We follows a vanilla Transformer decoder architecture (Le et al. 2019b), which consists attention layers: self-attention to attend on existing tokens, attention to Qetx from dialogue understanding program execution, and attention to Vetx from video understanding program execution.
Optimization. We the standard cross-entropy losses for prediction of dialogue and video understanding programs and output responses. We optimize models by joint training to minimize:
L aCaial (Lri fres D3 log(Pasa((Piau)) + 8 vosee ( (Prideo.)) + >nn (ee
where P is the probability distribution of an output token. The probability is computed by passing output representations from the parsers and decoder activation. We share the parameters between W and embedding matrix E. The hyper-parameters a 2 and 3 2 are fine-tuned during training.
linear layer W Rd with softmax
4 EXPERIMENTS
Datasets. We use the AVSD benchmark from the 7th Dialogue System Technology Challenge (DSTC7) (Hori et al. 2019). In the experiments with AVSD, we consider two settings: one with video summary and one without video summary as input. In the setting with video summary, the summary is concatenated the dialogue history before the first dialogue turn. We also adapt VilNMN to the video QA benchmark TGIF-QA -BIF-QA (Jang et al., 2017). Different from AVSD, TGIF-QA contains a diverse set of tasks, which address different visual aspects in video.
Training Procedure. We follow prior approaches (Hu et al., 2017 2018 Kottur et al. 2018) by obtaining the annotations of the programs through language parser (u al.) (016) and reference resolution model (Clark & Manning, 2016). During training, we directly use these soft labels of programs and the given ground-truth responses train the models. The labels are augmented with label smoothing technique (Szegedy et al., (2016). During inference time, we generate all programs and responses from given dialogues and videos. We run beam search enumerate programs for dialogue and video understanding and dialogue responses. (Please see Appendix B for more details).
AVSD Results. We evaluate model performance by the objective metrics based on word overlapping, including BLEU (Papineni et al., 2002). METEOR (Banerjee & Lavie, (205)), ROUGE-L (Lin) 2004), and CIDEr (Vedantam et al.] 2015), between each generated response and 6 reference gold responses. As seen in Table 2 our models outperform most of existing approaches. In particular, the performance of our model in the setting without video summary input is comparable to the
GaT-based RLM (Li et al. 2020) with much smaller model size. The Student-Teacher baseline (Hori ct al. 2019) specifically focuses on the performance gap between models with and without textual signals from video summary through a dual network of expert and student models. Instead, VINNMN reduces this performance gap through efficiently extracting relevant visual/audio information based on fine-grained entity and action signals. We also found that VilNMN applied on object-level features is competitive to the model applied on CNN-based features. The flexibility of VilNMN neural programs can also be seen in the experiment when the video understanding program is applied on the caption input as a visual feature.
Ablation Analysis. We experiment with several variants of VilMNM (either NMN or non-NMN- based) in the setting with CNN based features and video summary input As can be seen in Table 3, our approach video and dialogue understanding through compositional reasoning programs exhibits better performance than non-compositional approaches. Compared to the approaches that directly process frame-level features in videos (Row B) or token-level features in dialogues (Row C, D). our full VilNMN (Row A) considers entity-level and action-level information extraction and thus, avoids unnecessary and possibly noisy extraction. Compared to the approaches that obtain dialogue contextual cues through hierarchical encoding architecture (Row E, F) such as (Serban et al. 2016)) Hori et al. 2019). VilNMN directly addresses the challenge of entity references in dialogues. As mentioned, we hypothesize that the hierarchical encoding architecture is more appropriate for less

---

Under review as a conference paper at ICLR 2021
Table AVSD test results: The visual features are: I (I3D), ResNeXt-101 (RX), Faster- -CNN (FR). (caption video imput). The audio features are: VGGish (V), AclNet (A), on PT denotes models using pretrained weights and/or additional finetuning. Best and second best results are bold and underlined respectively.
Model
PT
Visual
Audio
BLEU4
METEOR
ROUGE-L
CIDEr
Audio/Visual only (without Video Summary/Caption)
Baseline (Hori al.
2019
0.305
0.217
0,481
0.733
Buseline (Hon ct al.
2019
0,309
0.215
0.487
0.746
Buxeline-GRU+Aiin Le al. 2019a
0.315
0.239
0.509
0.848
FGA Schwwrtzz
2019
0.806
JMAN
2020
0,309
0,520
0.890
Student-Teacher
2019
0.371
0.248
0.527
0.966
MTN [L
20196
0.343
0.247
0.520
0.936
MTN LC
2019h
V
0.368
0.259
0,537
0.964
MSIN Lee et
2020
V
0.379
0.261
0,548
1.028
RLM-GPTZ (Li al. 2020
0.402 V
0.254
0,544
1.052
VINNMN
0.397
0.262
0,550
1,059
VIINMN
FR
0,388
0.259
0.549
1.040
VINNMN
0.381
0.252
0.534
1.004
VIINMN
0.396
0.263
0,549
1.059
Audio/ Visual only (with Video Sunmary/Caption)
TopicEmb (Kumar et al. 2019
0.329
0.488
0.762
BaselineGGR++Atm (Le
2019a
V
0.310
0.242
0.515
0.856
JMAN (Chu et al. 2020
0.334
0.239
0.533
0.941
FA+HRED (Nguyen et al.
2018
V
0.360
0.249
0.544
0.997
VideoSum (Sanabria et al 2019
RX
0.394
0.267
0,563
1.094
VideoSum=Howw Sanahria et al.
2019
RX
0.387
0,266
0.564
1.087
MSTN Loc ct
2020
I
V
0.377
0.275
0.566
1.115
Student- Teacher
[on al. 2019
0.405
0.273
0,566
1.118
MIN
al.
20196
0,392
0,269
0,559
1,066
MTN (L
2019h
0.410
0.274
0.569
1.129
VGD-GPT2
Hoi
2020
I
0.436
0.282
0.579
L.194
RLM-GPT2
[L
V
0.459
0,294
0,606
1.308
ViNNMN
0.421
0.277
0.574
1.171
ViINMN
FR
0.421
0.275
0,571
1.148
VIINMN
I
V 0.421
0.277
0,573
1.167
VINNMN
T+C
V
0.429
0.278
0.578
1.188
Table Ablation analysis VilNMN with different model variants on the test split the AVSD benchmark
Model Variant
BLEU4
CIDEr
A
Full VINNMN
0.421
1.171
< No video NMNs; vamilla video atlention
0.415
1.159
dial. NMNs; + response-hissory attention
0.412
1.151
papponee-comccthhiitt-----sussss No dial. NMNs; LIREDisra((bstorr question attn.
0.411
1.133
No dial. NMNs:
0.414
1.153
F
No dial. NMNs; HREDGu, (history) question attn.
0.415
1.138
entity-sensitive dialogues such chit-chat and open-domain dialogues. Please Appendix Cor additional analysis of performance breakdown by dialogue turns and video lengths.
Intepretability. difference of VilNMN from previous approaches to video-grounded dialogues is the model interpretability based on the predicted dialogue and video programs. From Figure 5 observe that in cases where predicted dialogue programs and video program match are close to the gold labels, the model can generate generally correct responses. In cases of wrong predicted responses, we can further look at how the model understands the questions based on predicted programs. In the 3rd turn of example 1, the output response is missing minor detail as compared to the lahel response because the video program fails capture the parameter "rooftop". These subtle yet important details can determine whether output responses can fully address user queries. Similarly, in example 2, the model answers the question "what room" instcad of question about "an object". For additional qualitative analysis, please Appendix D
TGIF-QA Results. In TGIF-QA experiments, we report the result using the L.2 loss in Count task and accuracy in other tasks. From Table 4 ViINMN outperforms all baseline models in all tasks by a large margin. Compared to AVSD experiments, the TGIF-QA experiments emphasize video understanding ability of the models, removing the requirement for dialogue understanding and natural
7

---

Under review as a conference paper at ICLR 2021
Turn
Question
Dialogue NMN
Video NMN
Response
what are they
Predicted:
Predicted: where (IMO men in the videot where (the scene)- when (doing the scenel- describe (wbat) X Gold: where towo men),
Predicted: one is washing
skzing are they on
(chey),
and other
find scene!
taking pictures x Gold: yes,
3
mollogp?
-eurma
Gold: rind (they)
tooeEoop) wween (doiing iig in the scene)
floor roef deck, one man washing another man either filming or
idescribe
toking pictures
does be sit the chaic after washing at?
Predicted: rind (he),
Predicted: where rone man the video),
Predleted: no does not in chair
find chairl -Gunmar i (VV
where (a cheir) when fait the chair
after washinng ie)-exiat / Gold: where fone man), where la chairt- when (asit in the chaii arser washing (e)aaxiatt
Gold: no he does not, there a pipe with water running all over
Gold: (hel,
rind (the chafrt- summarize ()
Turn
Question
Dialogue Understanding Program
Video Understanding Program
Response
way next room, there is an objoct, what is that?
Predicted: find tthe room) fand (the
room] -aheen Iwhat in that]-exi
Predicted:
Predicted: looks he is in a living room, x went to
3
dooz) -sunmarizee zi ()
Gold: describe (what)
Gold:
Gold: summarize
doorway for vacuum.
when he gots does he have anything his hand?
Predicted: Find the),
Predieted: where tone pezaon in
Predicted: be has vacuum in hands. Gold: goes for the vacuum.
find thist
the video! -when (get up, have anything his hands) desczibe (wben) x
()
Gold:Find (bel,
4
rind 1h
Gold: where (one person in the video) wwhen (get. upl
summa. semneallle
when (have anything his hands) desor lbee (when)
Figure 5; Intepretability of model outputs from dialogue in the test split of the AVSD benchmark.
Table 4:
Experiment results on the TGIF-QA benchmark. The visual features are: ResNet-152 (R), C3D (C). Flow CNN from two-stream model (F), VGG (V), ResNeXt-101 (RX).
Count
Action
Transition
FrameQA
Model
Visual
(Loss)
(Ace)
(Ace)
(Ace)
VIS(avg) Ren et al.
2015a
4.80
0.488
0.348
0.350
MCB (ager) Fukur al. 2016
5.17
0.589
0.243
0.257
et al.
Yu cl 2017
5.13
0.640
ST-VQA (U (Gao ci 2018
R+F
4.32
0.629
0.694
0.495
Co-Mem Gao ct al. 2018
R+F
4.10
0.682
0.743
0.515
PSAC (Li
2019
4.27
0.704
0.769
HME [Fan et al. 2019
R+C
4.02
0.739
0.778
STA (Gao et al.
4.25
0.723
0.790
0.566
CRN+MAC eL
2019c
4.23
0.713
0.787
0.592
MQL (Lei
0.598
QueST (Jang al. 2020
4.19
0.759
0.810
0.597
HGA Urang
R+C
4.09
0.754
0.810
0,551
GCN
Huang
2020
R+C
3.95
0.743
0.811
0,563
HCRN
[Le
2020
R+RX
3.82
0.750
0.814
0.559
ViNNMN
2.65
0.845
0.887
0.747
soft label programs
1.90
0.857
0.898
0.780
-rssto--question attn.
3.28
0.801
0.776
0.679
language generation. This is demonstrated through higher performance gaps between VilNMN with generated programs and soft label programs compared to ones in AVSD experiments. We also observe that an attention layer attending to question is important during the response decoding phase in TGIF-QA there is dialogue context Qetx in Video QA tasks.
8

---

Under review as a conference paper at ICLR 2021
5 CONCLUSION
While conventional neural network approaches have achieved notable successes in video-grounde dialogues and video QA, they often rely on superficial pattern learning principles between contextual cues from questions/dialogues and videos. In this work, we introduce Visio-Linguistic Neural Module Network (VilNMN). VilNMN consists of dialogue and video understanding neural modules, each of which performs entity and action-level operations on language and video components. Our comprehensive experiments on AVSD and TGIF- IGIF-QA benchmarks show that our models can achieve competitive performance while promoting compositional and interpretable learning approach.
REFERENCES
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior visual question answer- ing models. In Proceedings the 2016 Conference Empirical Methods in Natural Language Pro- cessing. pp. 1955-19999 1996,,,,04000 Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 18653vv I/D16- 1203. URL https::////// www. ac www.wwawwwe..ooo http://www.....oor////aaaoooooo.
Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang. Stefan Lee, Peter Anderson, Irfan Essa, Devi Parikh, Dhruv Batra, Anoop Cherian, Tim K. Marks, and Chiori Hori. Audio-visual scene- aware dialog, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1545-1554, San Diego, California, June 201 Association for Computational Linguistics. doi: 10.18653/v1/N16-- 1181. URthttts::/ wwwp///wwwww...o//////////w/wwww c web. org/ wwwp//nthooogg//1.....
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 39. 2016b.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 1422-2443,, 2015.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064500 2016.
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA. May 7-9, 2015, Conference Track Proceedings, 2015. htt.:tttp///xw....tte
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Kun-Wei Chu, Kuan- Yen Lin, Chao-Chun Hsu, and Lun-Wei Ku. Multi-step jodal-my attention network for scene-awas aware dialogue system. DSTC Workshop @ AAAI, 2020.
Kevin Clark and Christopher D. Manning. Deep reinforcement learning for mention-ranking corefer- ence models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2256 2262. Austin, Texas, November 2016. Association for Computational Lin- guistics. doi: 10.18653/11/D1--111 1245. D1
URL https: www. ac ceweb. org/ hothol

---

Under review as a conference paper at ICLR 2021
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations, 2019. URL https
net/ // Ltte://ooommmmmnhdd.aaee
Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. Heteroge- neous memory enhanced multimodal attention model for video question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1999-20077, 2019.
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit lyyer, Pedro Rodrigue7, and Jordan Boyd-Graber. Pathologies of neural models make interpretations difficull. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 3719-3728, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18833/11/118-444
URL httph: www. acl www.wwebbbe..o wwt.w/ewww......e...o.ooooe..e.a wttp./////aaooo///..ee.e...
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/
pp. 1/41--104,, BtLLhttps:/ www acl web http:/ww......ttoooooo///////-/.
Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. Motion-appearance co-memory networks for video question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6576-6585 2018
Lianli Gao, Pengpeng Zeng, Jingkuan Song, Yuan-Fang Li, Wu Liu, Tao Mei, and Heng Tao Shen. Structured two-stream attention network for vidco question answering. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 6391-6398, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. CVPR, volume 1, pp. 3, 2017.
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. Neural modulc networks for reasoning over text. In International Conference on Learning Representations, 2020. URL
https::/cpenrev/
net fentt/ooruii http:/orrmmmiisyy/a/a/rrr
Chi Han, Jiayuan Mao, Chuang Gan, Josh Tenenbaum, and Jiajun Wu. Visual concept-metaconncep learning. In Advances in Neural Information Processing Systems, pp. 5002-5013, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pp. 131-135. IEEE, 2017.
C. Hori, H. Alamri, J. Wang, G. Wichern, T. Hori, A. Cherian, T. K. Marks, V. Cartillier, R. G. Lopes, A. Das, I. Essa, D. Batra, and D. Parikh. End-to-end audio visual scene-aware dialog using multimodal attention-based video features. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2352-2356, May 2019. doi: 10.1 AtA.AASSSSP.11.88825555
Chiori Hori, Anoop Cherian, Tim K Marks, and Takaaki Hori. Joint student-teacher learning for audio-visual scene-aware dialog. Proc. Interspeech 2019, pp. 1886-1890, 2019.
Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4555 4564, 2016.
10

---

Under review as a conference paper at ICLR 2021
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference Computer Vision, pp. 804-814 2017.
Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. Proceedings of the European conference on computer vision (ECCV), pp. 53-69, 2018.
Deng Huang, Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, and Chuang Gan. Location-aweer aware graph convolutional networks for video question answering. The AAAI Conference on Artificial Intelligence (AAAI), volume 1, 2020.
Yunseok Jang. Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning visual question answering. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2758-2766, 2017.
Robin Jia and Percy Liang. Data recombination for neural semantic parsing, In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 12-22, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v 1/P16- 1002. URL httts::: www. aclweb. http///ooo//aattoooo//// 1002
Jianwen Jiang. Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao. Divide and conquer: Question- guided
contextual attention for video question answering, In The AAAI Conference on Artificial Intelligence (AAAI), 2020.
Pin Jiang and Yahong Han. Reasoning with heterogeneous graph alignment for video question answering. In The AAAI Conference on Artificial Intelligence (AAAI), 2020.
Yichen Jiang and Mohit Bansal. Self-assembling modular networks for interpretable multi-hop reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4474-4484, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10. 10.18653/w//D19-1453 1455. 119-1455
URL https: www www.wllwwwbbc.. org/ anthologhhhh/
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2901-2916 2017a.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zimick, and Ross Girshick. Inferring and executing programs for visual reason- ing. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2989 2998, 2017b.
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. Doha. Qatar, October 2014. Association for Computational Linguistics. doi: 1/D3415/
URL wtw.tttp:///ec..m //www./w/www.. www. a c web. org/ wwwpw/wwwwww//..oooooooooooooooo 4-111111111111111111111
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. Visual coreference resolution in visual dialog using neural module networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 153-169, 2018.
Shachi H Kumar, Eda Okur, Saurav Sahay, Jonathan Huang, and Lama Nachman. Leveraging topics and audio features with multimodal attention for audio visual scene-aware dialog. arXiv preprint arXiv: 1013, 2019.
11

---

Under review as a conference paper at ICLR 2021
Hung Le and Steven C.H. Hoi. Video-grounded dialogues with pretrained generation language models. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 5842-5848, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/1//2220. acl-main.518. URL https: www. ac web. http:///aatttoooooooooo...o
ogy wtwgg/22o.... acl 1n.518
Hung Le. S Hoi, Doyen Sahoo, and N Chen. End-to-end multimodal dialog systems with hierarchical multimodal attention on video features. In DSTC7 at AAA12019 workshop, 2019a.
Hung Le, Doyen Sahoo, Nancy Chen, and Steven Hoi. Multimodal transformer networks for end-to-end aadeo-grounded dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5612-5623, Florence, Italy, July 2019b. Association for Computational Linguistics. doi: 10.18653/v1/P19- 1564.1166 URL https: / www aclweb
orc/antthelogy/P19-113
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Learning to reason with relational
video representation for question answering. arXiv preprint arXiv:1907.0453 2019c.
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for video question answering, arXiv preprini arXiv:2002. 10698, 2020.
Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. Dstc8-avsd: Multimodal semantic transformer network with retrieval style word generator, DSTC Workshop @ AAAI 2020, 2020.
Chenyi Lei, Lei Wu, Dong Liu, Zhao Li, Guoxin Wang, Haihong Tang, and Houqiang Li. Multi- question learning for visual question answering. In The AAAI Conference on Artificial Intelligence (AAAI), 2020.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. TVQA: Localized, compositional video question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1369-1379, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.11865/1/DD1---. 1167. URL htttp:://www/ ac web
htgg/anthclogyyDoo 10
Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. A persona-base based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 994-1003, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.18865/11/16-- 1094.
URL https: www a C 1 web. org. http//nnhoooogg.. /P1 anthologgy/P16-10944
Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang. Xiangnan He, and Chuang Gan. Beyond mns: Positional self-attention with co-attention for video question answering. In The 33rd AAAI Conference Artificial Intelligence, volume 8, 2019.
Zekang Li, Zongjia Li, Jinchao Zhang, Yang Feng, Cheng Niu, and Jie Zhou. Bridging text and video: A universal multimodal transformer for video-audio scene-aware dialog. arXiv preprint arXiv:2002.00163, 2020.
Chen Liang, Jonathan Berant, Quoe Le, Kenneth D. Forbus, and Ni Lao. Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23-33, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi:
10.18653/v1/P17-111 1003. UtL/ttps: www www.wa.www.......... org/ wttp://////.tttoooo//-----000000
Chin Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004.
Dat Tien Nguyen, Shikhar Sharma, Hannes Schulz, and Layla El Asri. From film video: Multi-turn question answering with multi-modal context. In AAAI 2019 Dialog System Technology Challenge (DSTC7) Workshop, December 2018.
Kishore Papineni, Salim Roukos, Todd Ward. and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th anmual meeting on association for computational linguistics, pp. 311-318, Association for Computational Linguisties, 2002.
12

---

Under review as a conference paper at ICLR 2021
Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. In Advances in neural information processing systems, pp. 2953-2961, 2015a.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NIPS), 2015b.
Ramon Sanabria, Shruti Palaskar, and Florian Metze. Cmu sinbad's submission for the dstc7 avsd challenge. In DSTC7 at AAA12019 workshop, volume 6, 2019.
Idan Schwartz, Seunghak Yu, Tamir Hazan, and Alexander Schwing. Factor graph attention. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2039-2048, 2019.
Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Sofia Serrano and Noah A. Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2931 -2911, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/v 1/P//3//1/11911/// 1282. URL https
wwwww www..aclweb.oo www.:eww.rgggatotttoooeo hetrg//nthoooggyyeeoo
1282
Gunnar Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.
Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, pp. 510-526. Springer, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017. URL http: //papers n ps.
Ramakrishna Vedantam, C Lawrence Zimick, and Devi Parikh. Cider: Consensus-base based image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4566-4575, 2015.
Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. End-to-end concept word detection for video captioning, retrieval, and question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3165-3173, 2017.
13

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL MODEL DETAILS
A.1 NEURAL MODULES ON TEMPORAL FEATURES
To adapt neural modules to temporal features, apply the same neural architectures in all modules except for the where module. In object-level features, this module operates on object-based or spatial-based level. We can apply this module to temporal-based features similarly simply by not stacking the parameter and pooling the attended features along the temporal dimension. For an entity parameter P. the attention matrix in this case an entity-tor to-temmpoal--tee step matrix Awhere, E R when module,
and the resulting pooled feature Vent.1 E Rd. Before feeding this representation we incorporate each Vent, into feature of each temporal step through an MLP layer and element-wiss summation, resulting in entt.i E IR F xd where I' is the number of sampled video frames/clips. An overview of the where module with temporal features can be seen in Figure 6
where la boy!
"d boy"
MLP
Temporal
Stacking
Attention
Visual Input
Figure 6: Adaptation where module to temporal-based features
We adapt this module in a similar manner to other temporal-level features such as audio and textual features such video caption. We keep the same architecture in the when module. We denote the resulting output from the when module for all actions P; is Vact E Nae We concatenate this to This is used
the output from the previous where module Veut to obtain Vent-act E IR input the describe exist module.
A.2 QUESTION PARSER
The parsers decompose questions into sub-sequences to construct compositional reasoning programs for dialogue and video understanding. Fach parser is attention-based Transformer decoder. The Transformer attention is a multi-head attention on query, key, and value tensors, denoted as Attention(Query, Key, Value). For each token in the Query sequence, the distribution over tokens in the Key sequence is used to obtain the weighted sum of the corresponding representations in the Value sequence.
QueryKey T Value E IR Iiquery Xdquary
Dnrention(Queryy Key, Value) = softmax
ddkey
Each attention is followed by feed-forward network applied each position identically. We exploit the multi-head and feed-forward architecture, which show good performance in NLP tasks such as NMT and QA (Vaswani al. 2017 Dehghani al., 2019), to efficiently incorporate contextual cues from dialogue components to parse question into reasoning programs. Given the encoded question Q. to decode program for dialogue understanding, the contextual signals are integrated through attention layers: one attention previously generated tokens, and the other on question tokens. At time step j. we denote the output from an attention layer as Adial.j-
A
(1)
Iitentioo(Patalll 1-11 .Paial 1/-1 Patalll E Rixd
(2) dial
Attention(A(1) Q,Q) Rixd
Similarly, to generate programs for video understanding, the contextual signals are learned and incorporated similar manner. However, to exploit dialogue contextual cues, the execution output of dialogue understanding neural modules Qetx is incorporated to each vector in Paial through an additional attention layer. This layer integrates the resolved entity information to decode the original entities for video understanding. It is equivalent to reasoning process that converts the question from its original multi-turn semantics to single-turn semantics.
A (3) vid Attention A Qetx: Qaxx) Riixd
14

---

Under review as a conference paper at ICLR 2021
A.3 NON-NMN MODELS
For ablation analysis, we evaluate several variants of VilNMN, based on the following categories:
To test the contribution of our NMN approach for video understanding. video understanding program and related neural modules and replace them with pure neural network architecture (Model B). Specifically, we remove neural modules where, when, describe, and exist. We then directly use video feature embeddings V as Veex as input to the original attention layer in response decoder similarly (Hori al. 2019 Sanabria al. 2019).
remove the parser for
Attention (A V.V) E Rixd
res
To further test the contribution of NMN architecture for dialogue understanding, similarly remove the question parser for dialogue understanding program and neural modules find and descr ibe. We then directly use the dialogue history embeddings H and question embeddings Q inputs the response decoder in two different ways. First, we replace the original attention on dialogue context Qetx with two attention layers to attend on dialogue history and question sequentially (Model C). As noted by Le et al. (2019b), question imput contains much more relevant signals than dialogue history and attention operation should be separated from the one on dialogue history.
A (2n)
Attention ( (A H, H) Rixd
(26) = Attention (A(2s) Q,Q)= E Rixd Tes
Alternatively, we simply concatenate dialogue and question embeddings similarly to (Hori et al. 2019 Sanabria et al. 2019) and use it as input to the original attention layer (Model D).
( Attention
A (1) [H; Q) [H; Q)) E /
rest
To use more sophisticated neural models for dialogue understanding, we further adopt the hierarchical encoding architecture with question attention (Li al. 2016 /erban et al., 2016 Hori et al.) 2019). Each dialogue turn Hr, including a pair of human utterance and system response, is processed separately by a word-level RNN such as LSTM (Model E)) or GRU (Model F). A sentence-level RNN is used to sequentially process the last hidden states obtained previously turn by turn. The output in each recurrent step is fed to an attention layer such as (Bahdanau et al. 2015 Vaswani et al.) 2017) to obtain question-aware representations of dialogue history.
H, RNN(H,) E IRd
H. sent RNN H,woodd E RRd
H = [H T-1 E 11)
Qctx Attention QQ, H, H) E Rood
where T is the current dialogue turn. The output is treated as Qetx and fed to the corresponding attention layer in the response decoder.
B ADDITIONAL EXPERIMENT DETAILS
B.1 DATASETS
We use the AVSD benchmark from DSTC7 (Hori et al. 2019 which consists of dialogues grounded on the Charades videos (Sigurdsson et al. 2016). 6). Fach dialogue contains up to 10 dialogue turns, each turn consists of question and expected response about given video. For visual features, we use the 3D CNN based features from a pretrained I3D model (Carreira & Zisserman, 2017) and object-level features from a pretrained FasterRNN model (Ren et al. 2015b). The audio features are obtained from pretrained VGGish model (Hershey et al.) 2017). In the experiments with AVSD, we consider two settings: one with video summary and one without video summary as input. In the the dialogue history before the first
setting with video summary, the summary is concatenated dialogue turn. We also adapt VilNMN to the video QA benchmark TGIF-QA (Jang et al. 2017). Different from AVSD, TGIF-QA contains a diverse set of QA tasks:
Count: open-ended task which counts the number of repetitions of an action
15

---

Under review as a conference paper at ICLR 2021
Action: multiple-choice (MC) task which asks about certain action occurring for fixed number of times
Transition: MC task which emphasizes temporal transition in video
Frame: open-ended task which can be answered from visual contents of one of video frames
For the TGIF-QA benchmark, we use the extracted features from a pretrained ResNet model (He
al. 2016).
Table 5: Summary of DSTC7 AVSD and TGIF-QA benchmark
Train
Val.
Test
Dialogs
7,659
1,787
1,710
AVSD
Turns
153,180
35,740
13,490
Words
1,450,754
339,006
110,252
Count QA
24,159
2,684
3,554
Action QA
18,428
2,047
2,274
TGIFQA
Trans. QA
47,434
5,270
6,232
Frame QA
35,453
3,939
13,691
B.2 TRAINING DETAILS
We use training batch size of 32 and embedding dimension d d= 128 in all experiments. Where Transformer attention is used, we fix the number of attention heads to 8 in all attention layers. In neural modules with MLP layers, the MLP network is fixed to 2 linear layers with a ReLU activation in between. neural modules with CNN, we adopt vanilla CNN architecture for text classification (without the last MLP layer) where the number of input channels is 1, the kernel sizes are f3, 4, 5]. and the number of output channels is d. We initialize models with uniform distribution (Glorot & Bengio 2010). During training, we adopt the Adam optimizer Kingma & Ba 2015) and decaying learning rate Vaswani et al. (2017) where we fix the warm-up steps to 15K training steps. We employ dropout (Srivastava et al. 2014) of 0.2 at all networks except the last linear layers of question parsers and response decoder. We train models up to 50 epochs and select the best models based on the average loss per epoch in the validation set.
C ADDITIONAL RESULTS
To evaluate model robustness, we report the relative performance by calculating the difference of CIDEr experimental settings against the most basic setting. Specifically, we compare against performance of output responses in the first dialogue turn position (i.e. 2nd-10 turn vs. the 1st tum), or responses grounded on the shortest video length range (video ranges are intervals 0- 0-10th, 10-20th percentile and on). We report the results of the model variants A, B, and E (See the Ablation Analysis section in the main paper and Appendix AA.3 for model description). First, as can be seen in Figure 7 for various dialogue turn positions, observe that the original ViINMN (model A) suffers less than model E when dialogues extend over time the 8th turn. This explains the contribution of dialogue understanding modules solving entities even when the dialogues grow longer. Secondly, as compared to model B, we observe that the Full VilNMN (model A) is less affected as the videos grounding the dialogues grow longer. The difference clear when the video length increases up to 33 seconds.
We also report the absolute scores and compare model variants. In Table 6a we compare model variants B and F. We observe that model B generally performs better than model E in overall, especially in higher turn positions, i.e. from the 4th turn to 8th turn. Interestingly, we note some mixed results in very low turn position, i.c. the nnd and 3rd turn, and very high turn position, i.c. the 1hth turn. Potentially, in very high turn position, the neural based approach such as hierarchical RNN can better capture the global dependencies within dialogue context than the entity-based compositional NMN method.
In Table 6b compare model variants A and B. We note that the performance gap between model and B is quite distinct, with 7/10 cases of video ranges in which model A outperforms. However, similarly our prior observations in experiments by dialogue turn, in lower ranges (i.e. 1-23
16

---

Under review as a conference paper at ICLR 2021
relative CIDEr per dialogue turn vs. 1st turn
relative CIDEr per video range vs. range
-0.4
0.5
-0.1
0.6
-0.
-0.3
0.7
-0.4
23-28
18:30
30:30.8
308-31
31-31.8
31,6-32
32-33
35-37
37-75
BB
turn position
*A
video length range (seconds)
Figure Performance model variants A. B, and E, by dialogue turn position video length. The performance is calculated relatively to performance the most basic setting, i.c. responses of the first dialogue turn ACIDEraum.i = CIDEruum,i CIDEr or responses grounding on the lowest video range (0 to 23 scconds) ACIDErrangei CIDErrange. CIDEro-23.
Table Performance breakdown BLEU4 and CIDEr
(a) by dialoguc turn between model variants B and E.
BLEU4
CIDEr
turn position
Model B
Model E
Model B
Model E
0.579
0.587
1.623
1.650
2
0.429
0.430
1.155
1.142
3
0.275
0.289
0.867
0,846
4
0.309
0.305
0.859
0.855
5
0.355
0.335
1.088
1,023
6
0.357
0.329
1.044
0,950
7
0.342
0.325
0.896
0.847
8
0.361
0.332
1.025
0.973
9
0.383
0.431
1.043
1.182
10
0.395
0.371
0.931
0.977
(b) by video length range (in seconds) between model variants A and
BLEU4
CIDEr
video range (seconds)
Model
Model
Model A
Model
1-23
0.432
0,447
1.298
1.355
23-28
0.436
0.433
1.264
1.165
28-30
0.398
0.376
1.203
1.164
30-30.6
0.441
0.418
1.220
1.202
30.6-31
0.413
0.411
1.250
1.166
31-31.6
0.439
0.451
1.249
1.295
31.6-32
0.430
0.419
1.217
1.192
32-33
0.468
0.445
1.343
1.237
33-37
0.388
0.381
1.149
1.124
37-75
0.356
0.365
0.910
0.962
seconds) and higher ranges (37-75 seconds), model A performs not as well as model B. There are additional factors that we will need to examine further to explain the results, such as the complexity of the questions for these short and long-rau range videos. Potentially, question parser for video understanding program needs more sophisticated composition method to retrieve information from these video ranges.
17

---

Under review as a conference paper at ICLR 2021
D QUALITATIVE ANALYSIS
We extract the predicted programs and responses for some example dialogues in Figure
10 and
11 and report our observations:
We observe that when the predicted programs are correct, the output responses generally match the ground-truth (See the !st and 2nd turn Figure 8 and the 1* and 4th turn in Figure 10 close the ground-truti truth responses (1#t turn in Figure 9).
When the output responses do not match the ground truth, we can understand the model mistakes by interpreting the predicted programs. For example, the 3rd turn in Figure 8 the output response describes room because the predicted video program focuses on the entity "what room" instead of the entity "an object" the question. Another example is the 3rd turn in Figure 10 where the entity "rooftop" is missing in the video program. These mismatches can deviate the information retrieved from the video during video program execution, leading wrong output responses with wrong visual contents.
We also note that in some cases, one or both of the predicted programs are incorrect, but the predicted responses still match the ground-trutt responses. This might be explained as the predicted module parameters are not exactly the same the ground truth but they are close enough (e.g. 4th turn in Figure 8). Sometimes, our model predicted programs that are more appropriate than the ground truth. For example, in the 2nd turn in Figure 9 the program is added with a where module parameterized by the entity "the shopping bag" which was solved from the reference "them" mentioned in the question.
We observe that for complex questions that involve more than queries (e.g. the 3rd turn in Figure 10), becomes more challenging to decode an appropriate video understanding program and generate responses that can address all queries.
In Figure we demonstrate some output examples of VilNMN and compare with two baselines: Baseline (Hori et al.) 2019) and MTN (Le et al. 2019b). We noted that VilNMN include important entities relevant to the current dialogue turn to construct output responses while other models might miss some entity details, c.g. "them/dishes" example A and "the magazine" in example B. These small yet important details can determine the correctness of dialogue responses.
18

---

Under review as a conference paper at ICLR 2021
Turn
Question
Dialogue Understanding Program
Video Understanding Program
Response
how many people
Predicted: summarized
Predicted: describe (how many people)
Predicted: there only one person video. /
video
Gold: summarize ()
Gold: describe ibe (how many people)
Gold: there is one
person the video.
does this person ever the field of view
Predicted: finnd (indithis
Predicted: where tone person the video) -when (leave the
Predicted: no, he does not leave the
peraon)-Bee -Bummari ()
V
Gold: is in the video the whole time.
2
Gold: find (this
field of view)-exist(f () / Gold: where (one person in video) when (leave field of view) -exist
person) sumeeris
in the door way the next romm there is an object, what that?
Predieted: (tide
room) -when (what is shat)- ()
Predicted: where
Predicted: it looks like he is in living room. x
(oom), Sind((ha
3
door) (bor)-summo -summa xe () *
Gold:
()
Gold: describe (what)
Gold: be
doerway for vacuum.
when he does he have unything kand??
Predicted: Eind (hel,
Predicted: where lone person in
Predicted: has vacuum in his hands. / Gold: he goes for the
nd (hi
the videa) -ihee) (got
-when
summarized ( /
have anything in his hands) describe twhnn)
Gold:find tha),
find (his)
Gold: where (one person in the
summarize ()
video) whhen (gez up), when (have anything in his hands) describe (when)
Figure Example output programs and responses from a dialogue in the test split of AVSD benchmark.
Turn
Question
Dialogue Understanding Program
Video Understanding Program
Response
what is is holding in
Predicted: find (his)
Predicted: where man) when (holding in his hand in the beginning: describe (what)
Predieted: he holding shopping bag in his x
Juis hand in the
.summazize ze ()
beginning?
ii thiss)
Gold: looks like bag of beans
Gold: where (a mar) whon (holding in his hand in the beginning)- describe (what)
does he put them
Predicted: find (he),
Predicted: where (a man), where (a shopping bag) (put them down) -exlst x
Predieted: yes puts down the table
down
find (:3em)
-ssumaar:
Gold: find the)
Gold: where mar) when (out then down) (ewia -)xi==()
Gold: yes he puts it
2
surmarize ()
down after looking at
Figure 9: Example output programs and responses from dialogue in the test split of the AVSD benchmark.
19

---

Under review as a conference paper at ICLR 2021
Turn
Question
Dialogue Understanding Program
Video Understanding Program
Response
hi many peuple
Predicted: summar
Predicted: describo (how many people)
Predicted: are people the video // Gold: hello, there are two men
the video from start to finish?
Gold: summaria ze ()
Gold: describe (how many people)
are they carrving on
Predicted: Eind (theyl
Predicted: where (two mon in the deo) wwhen ryaingg conversation between thensel vves) -oxist () /
Predicted: yes, they are carrying conversation
conversation
-oumma
between chemservee
Gold: find (they),
x
sind (themsel
Gold: no background
2
- suumarrize ()
Gold: where (:wo men) when (carrying converaation between themselves) -exist
noise is birdsong shuffling noises
they doing in the seene they on ruollup? ?
Predicted: Find (theyl, Find (the scene)
Predicted: where (two mon video) where (the scene)- when (doing in the szene)- describe (what) Gold: where (zwo men), where (rooftop) when (doing in the acene] -describe (what)
Predicted: one washing chair and the
-summari ze () x Gold: find (they)
other is taking pictures
Gold: yes second floor roof dock, one man is washing a chair, another man either filming taking tures
3
does he ait the chair nfter washing
Predicted: Find (hel,
Predicted: where (one in the video), where (a chair)-
Predicted: he does not sit in the chair / Gold: no he not, there pipe with water running all over
find (the chair)
aurmarize () /
(hin in the chair aftez washing (x))sxxst() Gold: where (one man),, where cha r)- when (sit in the chair after washing it) -oxist()
Gold: tind (he),
sind chair)-
summarizze
Figure 10: Example output programs and responses from dialogue in the test split of the AVSD benchmark.
Distog Hartoy and Question
Predicted Distogue Video
and Ground-tuth Responass
Dialog Histery: whet is going video starts
Dielages Progran (HINMN)) Flnd thai, find (thaes
Predicted (ViINMN) he doesn't them with towel.
louking out doorwayy..whh what is he doing there then starts wash
Video (MISMS) whereta nan) shere the diches) petpe then with towel) sxisk!!
Predicted (Basellnee) no he does not take picturu of the
Predieted (MTN)) hands. X
do anything
Question: does ba wije thea witha
Gold. looks like he
his hands.
DMalog wh.t he reading?
Ittllggs Fragram (VINMNN
Predicted (VHINMN): yes, laughing the
rending magazine hughing...
cind that, Line eugasinel surmacieet)
Predicted (Raseline); yes talking camem. Predicted (MTN): no he not lughing at the end of the
Progren (MISMS))
Question: lunohing becusee the magorine?
siace
boy
videc), sbare raçazinet vhen. claughing becense of neganinet
Gold: maybe, because then he throws the magazine aside..
Dialog Historry: ho. many people one penson
Dialgges Program (VINANN
Predicted (VIINMN): looks
living
riwetche reomt, Tirdithe
Predieted (Basellne): not Predicted (MFNN) he walks into the mom x Gold. he the doorway for
in the video...
Ballgge Frogram (MINMNY:
Question: the poces, there ofject. What is thu!?
viace Edbat. visun Estat.
La Liali- seivic)
Figure 11: Intepretability of example outputs from ViINMN and baselines models (Hori et al. 2019 Le et al. 2019h)
20


Reviews:
[{"review_id": "gIovmw9vbz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper studies the application of neural module network to video-grounded language tasks. They propose a method dubbed Visio-Linguistic Neural Module Network (VilNMN) to retrieve spatio-temporal information in a video through a linguistic-based parsed program. In particular, VilNMN first extracts entity references and their corresponding actions in linguistic cues. This information is then being used to locate relevant information in the visual cue to arrive at the correct answer. The proposed method is evaluated on two large scales benchmarks AVSD and TGIF-QA, demonstrating competitive performance with state-of-the-art methods.\n\nComments (Technical, Major Flaws of this paper): \n\n(1) Overall, it is interesting to see neural module network works in such a complex setting as in video-grounded language tasks and I appreciate the efforts of the authors trying to explain the method as detailed as possible. \n\n(2) The use of the mathematical symbols in the paper is very confusing. Normally, upper cases (capital letters) are used to denote matrices/sets of vectors while lower cases (non-capital letters) are often used to denote vectors.\n\n(3) How did the authors subsample F frames/clips from a video? In my understanding, there are a lot of frames in a video are blurry or distorted so if you sample them in a random manner, it would greatly affect the performance of object detection (Faster RCNN in this case). Please elaborate more on this.\n\n(4) Subsection 3.1, in the description of \"when\" module: A_when,i is a vector, not a matrix. Same in the supplementary document. In addition, I am wondering since there are many objects/entities in a video, how would the \"when\" module be able to localize the same object/entity over time given no additional supervision? I think object tracking would greatly be beneficial in this case.\n\n(5) In the experiments on TGIF-QA:\n- Why the authors only used ResNet features instead of using similar features as in AVSD? Spatial-based level features are fine in terms of computations but are less intuitive as I expect the object-based features counterpart and the linguistic entity references represent things at the same level of abstraction. \n- Honestly, I am skeptical about the results on the TGIF-QA datasets as the gap between VilNMN and the existing methods is very significant. From Table 4, it looks like motion features (optical flow, C3D, and the likes) play a role in tasks containing repetition of actions (such as count and action). I am not sure the reason why VilNMN without the use of any of those motion features could manage to outperform the existing methods with large margins?\n- Please provide more analysis on the results on the TGIF-QA as in my understanding even when a model correctly links entities in a question with their visual representations, it does not guarantee that it can arrive at correct answers. For example, in counting task, I would say most of the questions have the same parsed program. How does your model work in this scenario?\n\n\nSome other concerns:\n\n(6) At the end of the related works, the authors wrote \"In video represented as sequence of images,...., e.g though average pooling, resulting in potential loss of information\". This is a big assumption as using average pooling over object proposals is a bad idea as object appearance may vary very little over time in a video and the average pooling would smash the temporal information in the video. If one can properly model object tubelets via object tracking, what written in the paper wouldn't make sense.\n\n(7) Section 3.2: The sentence \"...calculated as softmax scores between an entity P_i and each token in dialogue history\" is mathematically incorrect. What drives the attention weights? Softmax is a function applying over a set of elements.\n\nI would be okay to raise the rating if the authors sufficiently address my concerns during the rebuttal phase.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MuS5HYVnPwx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the language grounding aspect of video-language problems. It proposes a Neural Module Network (NMN) for explicit reasoning of visually-grounded object/action entities and their relationships. The proposed method is demonstrated to be somewhat effective in the audio-visual dialogue task and has been shown superior to existing works on video QA. Overall, the paper is motivated clearly and is delivered with good clarity. The followings need to be clarified.\n\ni) The proposed model demonstrates impressive results on TGIF-QA but without any insightful justification. Since the questions in TGIF-QA are short and usually do not involve complicated reasoning, intuitively, a heavy reasoning scheme might not necessarily pay off. Please clarify the performance gain and possible reasons. Also, \"soft label programs\" lack the necessary context (and should be in bold instead in Tab. 4).\n\nii) Including intense model variants in the main result table (Tab. 2) gives this paper a somewhat unfair advantage, especially when the best performing method on each metric comes from different model variants. The validation set (from both AVSD and TGIF-QA) is supposed to serve the purpose of model architecture search and ablation studies. Besides, the underlines in the lower part of Tab. 2 should go to method VGD-GPT2.\n\n========== Post-Rebuttal ==========\n\nConcerns on paper/results clarity still persist. Lowering my rating to 5.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HkG_DfE0r24", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The author present a novel neural moudalar network  for video grounding tasks, which can provide interpretable intermediate reasoing outcomes and show the model robustness. \nThis model achieves competitive results on AVSD datasets and state-of-the-art performance on TGIF-QA datasets, which demonstrates the effectiveness of the model design.\n\nDetailed comments are listed in the following\n• The novelity of the NMN is limited in this paper. The similari idea have been used in many previous literatures. I am wondering that how you define the modular space? Is there any prinpicle guidelines to design module like \"find, summarize, when, describe\"?\n• The reasoning struture in this papar is simple. The module \"find, when, where\" are more like signal detectors. There is no reasoning structure for how to get the final response. (in this paper, just fuse the detected information to get the final answer by a response decoder). So this methods cannot reveal the inner correlation between final response and detected visual/language entities.\n• [Question] How do you train the program generation tasks from language? Is there any groundtruth program structure annotation to supervise this? How do you determine the hyper-parameters \\alpha and \\beta?\n• The paper is written pooly and some expressions are confusing, like \"Different from..., our model are trained to fully genrate the parameters of components in the text\". The parameter here refer the input of each module, which is different from model parameters.", "extra_scores": {}, "extra_text": {}}, {"review_id": "D_mav22ZWd-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Description:\n\nThis paper introduces the Visio-Linguistic Neural Module Network (VilNMN) consisting of a pipeline of dialogue and video understanding neural modules. Motivated by Hu et al. (2017), Kottur et al (2017), this paper extends the NMNs on video tasks for interpretable neural models. The model explicitly resolves entity references (dialog understanding) and detects actions from videos (video understanding) for response generation. Experiments show that NMNs achieve competitive results on AVSD (video-dialog) and TGIF-QA (video-QA) benchmarks. \n\nStrengths:\n- New modules for video understanding (“where”, “when”) have been proposed. A step towards interpretability of compositional neural networks \n- Ablation studies have been provided to understand the importance of each module in the VILNMN model.  \n- The breakdown of relative CIDEr/BLEU (Figure 7 in supplementary) for different context and video length is interesting. \n- SOTA results on TGIF-QA (Video QA) while competitive results on AVSD (dialog task). \n\nWeaknesses:\n- Availability of code is not discussed which is essential for reproducibility. \n- It would help to specify whether sentence vs corpus level BLEU was used for evaluation\n- Human evaluation is not provided. Limitations of automatic metrics and their reliability in language generation have been discussed repeatedly. See (Reiter and Belz, 2009; Novikova et al., 2017; Reiter, 2018). \n- Table 6 denotes that the evaluation results decrease with longer video or larger context modeling which is the main focus of the paper.   \n- It might be argued that this approach would not generalize. How would this model scale when the dialog becomes challenging (in terms of disfluencies, ellipses or alignment, topic switch, etc apart from co-reference; see Haber et al 2019 Photobook dataset for brief summary of dialog phenomena)? Similarly when the videos become more complex, would action recognition suffice? \n- CorefNMN (Kottur et al 2017) was designed specifically for co-reference resolution in the dialog. The paper could be similarly improved by explicitly motivating the specific utility of the NMN modules compared to high-level description- eg. to capture co-reference (dialog) and action recognition (video). \n- An analysis of the AVSD dataset would help understand the importance of the dialog context in the dataset - focus of one of the modules in this paper. (See Agarwal et al 2020 study for Visual dialog)\n\nQuestions:\n- Since AVSD is also posed as a retrieval task (Alamri et al. 2019), have the authors evaluated the system on ranking based metrics?  \n- Could the authors clarify why accuracy is also not reported for CountQA in Table 4? \n- It would help to explicitly mention the neural modules previously defined and the novel modules, eg how the “find” module differs from Kottur et al. 2018. \n- Have the authors experimented with the dialog-based modules from Kottur et al. 2018 - eg. “refer” module? \n- In Fig 2 (as well as the main text), it would help to clarify if the underlying text encoder is shared for the dialog history, question, and caption. \n- How are the audio signals incorporated in the VilNMN mentioned in Table 2? Pardon if I missed this. \n- Have the authors experimented with pre-trained weights (decoders)? \n\nSuggestions/Comments:\n- Previously Johnson et al. (2017a); Hu et al. (2017), Kottur et al. (2018) have all explored NMN for visio-linguistic tasks (such as VQA, Visual dialog), the nomenclature “Visio-Linguistic Neural Module Network (VLNMN)” seems too broad. Something on the grounds of “ActionNMN/ActNMN” would do justice to the work. \n- Model descriptions of ablations in Table 3 could be improved for clarity. \n- Implementation details could be further specified - the framework, all other hyperparameters to ease reproducibility. \n\n--------------------------------------------------------------------------------------------------------------------------------------------------------\nPost Rebuttal update:\n\nI would like to thank the authors for answering the questions. I believe that an updated version addressing all the concerns in detail will find its place in other future conferences. Original rating is maintained.\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a neural module based approach for reasoning about video grounding.  The goal is to provide performance and interpretability.  Unfortunately, the reviewers found the paper opaque, the results confusing, and expressed repeated concerns about the novelty, fairness of comparisons and concerns that the surprising results were not sufficiently well justified by the paper (or the author's response).
Decision: Reject