SlowGuess/NAACL_2022_OCR huggingface dataset:
Datasets. We use the AVSD benchmark from the Dialogue System Technology Challenge 7 (DSTC7) (Hori et al., 2019). The benchmark consists of dialogues grounded on the Charades videos (Sigurdsson et al., 2016). Each dialogue contains up to 10 dialogue turns, each turn consists of a question and expected response about a given video. For visual features, we use the 3D CNN-based features from a pretrained I3D model (Carreira and Zisserman, 2017) and object-level features from a pretrained FasterRNN model (Ren et al., 2015b). The audio features are obtained from a pretrained VGGish model (Hershey et al., 2017). In the experiments with AVSD, we consider two settings: one with video summary and one without video summary as input. In the setting with video summary, the summary is concatenated to the dialogue history before the first dialogue turn. We also adapt VGNMN to the video QA benchmark TGIF-QA (Jang et al., 2017). Different from AVSD, TGIF-QA contains a diverse set of QA tasks:
- Count: an open-ended task which counts the number of repetitions of an action
- Action: a multiple-choice (MC) task which asks about a certain action occurring for a fixed number of times
Transition: an MC task which emphasizes temporal transition in video
- Frame: an open-ended QA about visual contents of one of the video frames
#	Train	Val.	Test
AVSD	Dialogs	7,659	1,787	1,710
Turns	153,180	35,740	13,490
Words	1,450,754	339,006	110,252
TGIFQA	Count QA	24,159	2,684	3,554
Action QA	18,428	2,047	2,274
Trans. QA	47,434	5,270	6,232
Frame QA	35,453	3,939	13,691
Table 2: Summary of DSTC7 AVSD and TGIF-QA benchmark
For the TGIF-QA benchmark, we use the extracted features from a pretrained ResNet model (He et al., 2016). Table 2 shows a summary of the AVSD and TGIF-QA benchmarks.
Training Details. We follow prior approaches (Hu et al., 2017, 2018; Kottur et al., 2018) by obtaining the annotations of the programs through a language parser (Hu et al., 2016) and a reference resolution model (Clark and Manning, 2016). During training, we directly use these as ground-truth labels of programs to train our models. The ground-truth responses are augmented with label smoothing technique (Szegedy et al., 2016). During inference time, we generate all programs and responses from given dialogues and videos. We run beam search to enumerate programs for dialogue and video understanding and dialogue responses.
We use a training batch size of 32 and embedding dimension  $d = 128$  in all experiments. Where Transformer attention is used, we fix the number of attention heads to 8 in all attention layers. In neural modules with MLP layers, the MLP network is fixed to 2 linear layers with a ReLU activation in between. In neural modules with CNN, we adopt a vanilla CNN architecture for text classification (without the last MLP layer) where the number of input channels is 1, the kernel sizes are  $\{3,4,5\}$ , and the number of output channels is  $d$ . We initialize models with uniform distribution (Glorot and Bengio, 2010). During training, we adopt the Adam optimizer (Kingma and Ba, 2015) and a decaying learning rate (Vaswani et al., 2017) where we fix the warm-up steps to 15K training steps. We employ dropout (Srivastava et al., 2014) of 0.2 at all networks except the last linear layers of question parsers and response decoder. We train models up to 50 epochs and select the best models based on the average loss per epoch in the validation set.
All models are trained in a V100 GPU with a capacity of 16GB. We approximated each training epoch took about 20 minutes to run. For each
model experiment with VGNMN, we obtained at least 2 runs and reported the average results. We implemented models in Pytorch and released the code and model checkpoints<sup>1</sup>.
Optimization. We optimize models by joint training to minimize the cross-entropy losses to generate responses and functional programs.
$$
\begin{array}{l} \mathcal {L} = \alpha \mathcal {L} _ {\text {d i a l}} + \beta \mathcal {L} _ {\text {v i d}} + \mathcal {L} _ {\text {r e s}} \\ = \alpha \sum_ {j} - \log (\mathbf {P} _ {\mathrm {d i a l}} (\mathcal {P} _ {\mathrm {d i a l}, j})) \\ + \beta \sum_ {l} - \log \left(\mathbf {P} _ {\text {v i d e o}} \left(\mathcal {P} _ {\text {v i d e o}, 1}\right)\right) \\ + \sum_ {n} - \log \left(\mathbf {P} _ {\mathrm {r e s}} \left(\mathcal {R} _ {\mathrm {n}}\right)\right) \\ \end{array}
$$
where  $\mathbf{P}$  is the probability distribution of an output token. The probability is computed by passing output representations from the parsers and decoder to a linear layer  $W\in \mathbb{R}^{d\times V}$  with softmax activation. We share the parameters between  $W$  and embedding matrix  $E$ .
AVSD Results. We evaluate model performance by the objective metrics, including BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), ROUGE-L (Lin, 2004), and CIDEr (Vedantam et al., 2015), between each generated response and 6 reference gold responses. As seen in Table 3, our models outperform most of existing approaches. We observed that our approach did not outperform the GPT-based baselines (Li et al., 2020b; Le and Hoi, 2020) in the setting that allows video summary/caption input. However, the performance of our model in the setting without video summary/caption input is on par with the GPT-based baseline (Li et al., 2020b), even though our model did not rely on deep pretrained representations on large-scale text data. These observations imply that GPT-based models can better capture video context from video caption/summary through rich pretrained representations. However, without access to video caption/summary, these models may fail to understand video from visual-only representations. In this setting, GPT-based models may be inferior to VGNMN, which explicitly exploits the compositional structures from textual inputs to integrate visual features. We also found that VGNMN applied to object-level features is competitive to the model applied to CNN-based features. The
Model	PT	Vis.	Aud.	B-4	M	R	C
Without Video Summary/Caption
Baseline (Hori et al., 2019)	-	I	-	0.305	0.217	0.481	0.733
Baseline (Hori et al., 2019)	-	I	V	0.309	0.215	0.487	0.746
(Le et al., 2019a)	-	I	V	0.315	0.239	0.509	0.848
FGA (Schwartz et al., 2019)	-	I	V	-	-	-	0.806
JMAN (Chu et al., 2020)	-	I	-	0.309	0.240	0.520	0.890
(Hori et al., 2019)	-	I	V	0.371	0.248	0.527	0.966
MTN (Le et al., 2019b)	-	I	-	0.343	0.247	0.520	0.936
MTN (Le et al., 2019b)	-	I	V	0.368	0.259	0.537	0.964
MSTN (Lee et al., 2020)	-	I	V	0.379	0.261	0.548	1.028
BiST (Le et al., 2020a)	RX	V	0.390	0.259	0.552	1.030
GPT2 (Li et al., 2020b)	✓	I	V	0.402	0.254	0.544	1.052
VGNMN	-	I	-	0.397	0.262	0.550	1.059
VGNMN	-	FR	-	0.388	0.259	0.549	1.040
VGNMN	-	-	V	0.381	0.252	0.534	1.004
VGNMN	-	I	V	0.396	0.263	0.549	1.059
With Video Summary/Caption
TopicEmb (Kumar et al., 2019)	-	I	A	0.329	0.223	0.488	0.762
(Le et al., 2019a)	-	I	V	0.310	0.242	0.515	0.856
JMAN (Chu et al., 2020)	-	I	-	0.334	0.239	0.533	0.941
(Nguyen et al., 2018)	-	I	V	0.360	0.249	0.544	0.997
(Sanabria et al., 2019)	✓	RX	-	0.387	0.266	0.564	1.087
MSTN (Lee et al., 2020)	-	I	V	0.377	0.275	0.566	1.115
(Hori et al., 2019)	-	I	V	0.405	0.273	0.566	1.118
MTN (Le et al., 2019b)	-	I	-	0.392	0.269	0.559	1.066
MTN (Le et al., 2019b)	-	I	V	0.410	0.274	0.569	1.129
BiST (Le et al., 2020a)	RX	V	0.429	0.284	0.581	1.192
GPT2 (Le and Hoi, 2020)	✓	I	V	0.436	0.282	0.579	1.194
GPT2 (Li et al., 2020b)	✓	I	V	0.459	0.294	0.606	1.308
VGNMN	-	I	-	0.421	0.277	0.574	1.171
VGNMN	-	FR	-	0.421	0.275	0.571	1.148
VGNMN	-	I	V	0.421	0.277	0.573	1.167
VGNMN	-	I+C	V	0.429	0.278	0.578	1.188
Table 3: AVSD test results: Metrics are: BLEU-4 (B-4), METEOR (M), ROUGE-L (R), and CIDEr (C). The visual features are: I3D (I), ResNeXt-101 (RX), Faster-RCNN (FR), caption as a video input (C). The audio features are: VGGish (V), AclNet (A).  $\sqrt{}$  on PT denotes models using pretrained weights and/or additional finetuning. The best/second-best results are bold/underlined respectively.
flexibility of VGNMN neural programs show when we treat the caption as an input equally to visual or audio inputs and execute entity-action level neural operations on the encoded caption sequence.
Robustness. To evaluate model robustness, we report BLEU4 and CIDEr of model variants in various experimental settings. Specifically, we compare against performance of output responses in the first dialogue turn position (i.e.  $2^{nd} - 10^{th}$  turn vs. the  $1^{st}$  turn), or responses grounded on the shortest video length range (video ranges are intervals of  $0 - 10^{th}$ ,  $10 - 20^{th}$  percentile and so on). We report results of the following model variants: (1) w/o video NMN: VGNMN without using video-based modules, e.g. when and where. Video features are retrieved through a token-level representation of questions (Le et al., 2019b). (2) no NMN: (1) + without dialogue-based modules, e.g. find and summarize. Dialogue history is encoded by a hierarchical LSTM encoder (Hori et al., 2019).
Robustness to video length: In Table 4a, we noted that the performance gap between VGNMN and (1) is quite distinct, with 7/10 cases of video ranges in which VGNMN outperforms. However, in lower ranges (i.e. 1-23 seconds) and higher ranges (37-75 seconds), VGNMN performs not
'https://github.com/henryhungle/vgnmn
Video (seconds)	BLEU4	CIDEr
VGNMN	(1)	VGNMN	(1)
1-23	0.432	0.447	1.298	1.355
23-28	0.436	0.433	1.264	1.165
28-30	0.398	0.376	1.203	1.164
30-30.6	0.441	0.418	1.220	1.202
30.6-31	0.413	0.411	1.250	1.166
31-31.6	0.439	0.451	1.249	1.295
31.6-32	0.430	0.419	1.217	1.192
32-33	0.468	0.445	1.343	1.237
33-37	0.388	0.381	1.149	1.124
37-75	0.356	0.365	0.910	0.962
(a) Performance by video length between VGNMN and variant (1) (w/o video NMN).
Dial. Turn	BLEU4	CIDEr
(1)	(2)	(1)	(2)
1	0.579	0.587	1.623	1.650
2	0.429	0.430	1.155	1.142
3	0.275	0.289	0.867	0.846
4	0.309	0.305	0.859	0.855
5	0.355	0.335	1.088	1.023
6	0.357	0.329	1.044	0.950
7	0.342	0.325	0.896	0.847
8	0.361	0.332	1.025	0.973
9	0.383	0.431	1.043	1.182
10	0.395	0.371	0.931	0.977
(b) Performance by dialogue turn between variants (1)  $(w / o$  video NMN) and (2) (no NMN)
Question structure	BLEU-4	CIDEr
VGNMN	(2)	VGNMN	(2)
Yes/No	0.474	0.447	1.306	1.244
Wh-	0.266	0.265	0.706	0.699
How	0.636	0.663	1.817	1.878
Others	0.287	0.318	0.701	0.768
1Sent+Que	0.374	0.357	0.854	0.822
2Sent+Que	0.303	0.225	0.554	0.487
>2Sent+Que	0.000	0.000	0.000	0.000
2SubQue	0.196	0.180	0.489	0.460
3SubQue	0.332	0.000	0.653	0.112
(c) Performance by utterance structures between VGNMN and variant (2) (no NMN): Single-question utterances (Top) vs. Multi-part utterances (Bottom).
Table 4: VGNMN and model variants by configurations of dialogues, videos, and question structures
as well as model (1). We observed that related factors might affect the discrepancy, such as the complexity of the questions for these short and long-range videos. Potentially, our question parser for the video understanding program needs to be improved (e.g. for tree-based programs) to retrieve information in these ranges.
Robustness to dialogue turn: In Table 4b, we observed that model (1) performs better than model (2) overall, especially in higher turn positions, i.e. from the  $4^{th}$  turn to  $8^{th}$  turn. Interestingly, we noted some mixed results in very low turn position, i.e. the  $2^{nd}$  and  $3^{rd}$  turn, and very high turn position, i.e. the  $10^{th}$  turn. Potentially, with a large dialogue turn position, the neural-based approach such as hierarchical RNN can better capture the global dependencies within dialogue context than the entity-based compositional NMN method.
Robustness to question structure: Finally, we compared performance of VGNMN with the non-NMN variant (1) in different cases of question structures: single-question vs. multiple-part structure. In single-question structures, we examined by the question types (e.g. yes/no, wh-questions). In multi-part structures, we further classified whether there are sentences preceding the question (e.g. "1Sent+Que") or there are smaller (sub-)questions (e.g. "2SubQue") within the question. In Table 4c, we observed that VGNMN has clearer performance gains in multi-part structures than single-question structures. In multi-part structures, we observed higher gaps between VGNMN and model (1) in highly complex cases e.g. "2Sent+Que" vs. "1Sent+Que". These observations indicate the robustness of VGNMN and the underlying compositionality principle to deal with complex question structures. We also noted that VGNMN is still susceptible to extremely long questions (">2Sent+Que") and future work is needed to ad
dress these scenarios.
1
Turn	Question	Dialogue NMN	Video NMN	Response
3	...what are they doing in the scene? are they on a rooftop?	Predicted: find (they), find (the scene) →summarize() Gold: find (they) →summarize()	Predicted: where (two men in the video), where (the scene) → when (doing in the scene) → describe (what) × Gold: where (two men), where (rooftop) → when (doing in the scene) →describe (what)	Predicted: one is washing a chair and the other is taking pictures × Gold: yes, on a second floor roof deck, one man is washing a chair, another man is either filming or taking pictures
4	does he sit in the chair after washing it?	Predicted: find (he), find (the chair) →summarize() Gold: find (he), find (the chair) →summarize()	Predicted: where (one man in the video), where (a chair) → when (sit in the chair after washing it) → exist() Gold: where (one man), where (a chair) → when (sit in the chair after washing it) → exist()	Predicted: no , he does not sit in the chair × Gold: no he does not, there is a pipe with water running all over
Turn	Question	Dialogue Understanding Program	Video Understanding Program	Response
3	in the door way to the next room, there is an object. what is that?	Predicted: find (the room), find (the door)
→summarize()
Gold: summarize ()	Predicted: where (what room)
→when (what is that)
→exist()
Gold: describe (what)	Predicted: it looks like he is in a living room.
Gold: he went to the doorway for a vacuum.
4	when he gets up, does he have anything in his hand?	Predicted: find (he), find (his) → summarize()
Gold: find (he), find (his) → summarize ()	Predicted: where (one person in the video)
→when (get up, have anything in his hands) →describe (when)
Gold: where (one person in the video) →when (get up), when (have anything in his hands) →describe (when)	Predicted: he has a vacuum in his hands.
Gold: he goes for the vacuum.
Figure 5: Interpretability of model outputs on AVSD: Example A (Top) and Example B (Bottom).
Interpretability. In Figure 5, we show both success and failure cases of generated responses and corresponding generated functional programs. In each example, we marked predicted outputs as incorrect if they do not match the ground-truth completely (even though the outputs might be partially correct). From Figure 5, we observe that in cases where generated dialogue programs and video programs match or are close to the gold labels, the
model can generate generally correct responses. For cases where some module parameters do not exactly match but are closed to the gold labels, the model can still generate responses with the correct visual information (e.g. the  $4^{th}$  turn in example B). In cases of wrong predicted responses, we can further look at how the model understands the questions based on predicted programs. In the  $3^{rd}$  turn of example A, the output response is missing a minor detail as compared to the label response because the video program fails to capture "rooftop" as a where parameter. These subtle yet important details can determine whether output responses can fully address user queries. In the  $3^{rd}$  turn of example B, the model wrongly identifies "what room" as a where parameter and subsequently generates a wrong response that it is "a living room".
TGIF-QA Results. We report the result using the L2 loss in Count task and accuracy in other tasks. From Table 5, VGNMN outperforms the majority of the baseline models in all tasks by a large margin. Compared to AVSD experiments, the TGIF-QA experiments emphasize the video understanding ability of the models, removing the requirement for dialogue understanding and natural language generation. Since TGIF-QA questions follow a very specific question type distribution (count, action, transition, and frameQA), the question structures are simpler and easier to learn than AVSD. Using exact-match accuracy of parsed programs vs. label programs as a metric, our question parser can achieve a performance  $81\%$  to  $94\%$  accuracy in TGIF-QA vs.  $41 - 45\%$  in AVSD. The higher accuracy in decoding a reasoning structure translates to better adaptation between training and test time, resulting in higher performance gains.
Model	Vis.	Count (Loss)	Action (Acc)	Tran. (Acc)	FrQA (Acc)
VIS (Ren et al., 2015a)	R	4.80	0.488	0.348	0.350
MCB (Fukui et al., 2016)	R	5.17	0.589	0.243	0.257
Yu et al. (Yu et al., 2017)	R	5.13	0.561	0.640	0.396
ST-VQA (Gao et al., 2018)	R+F	4.32	0.629	0.694	0.495
Co-Mem (Gao et al., 2018)	R+F	4.10	0.682	0.743	0.515
PSAC (Li et al., 2019)	R	4.27	0.704	0.769	0.557
HME (Fan et al., 2019)	R+C	4.02	0.739	0.778	0.538
STA (Gao et al., 2019)	R	4.25	0.723	0.790	0.566
CRN+MAC (Le et al., 2019c)	R	4.23	0.713	0.787	0.592
MQL (Lei et al., 2020)	V	-	-	-	0.598
QueST (Jiang et al., 2020)	R	4.19	0.759	0.810	0.597
HGA (Jiang and Han, 2020)	R+C	4.09	0.754	0.810	0.551
GCN (Huang et al., 2020)	R+C	3.95	0.743	0.811	0.563
HCRN (Le et al., 2020b)	R+RX	3.82	0.750	0.814	0.559
BiST (Le et al., 2020a)	RX	2.14	0.847	0.819	0.648
VGNMN	R	2.65	0.845	0.887	0.747
Cascading Errors. Compared to prior approaches, we noted that VGNMN is a modularized system which may result in cascading errors to downstream modules. One major error is the error of generated programs which is used as parameters in neural modules. To gauge this error, we compare the performance of VGNMN between 2 cases: with generated programs and with groundtruth programs. From Table 6, we noticed some performance gaps between these cases. These observations imply that: (1) program generations and response generations are positively correlated and more accurate programs can lead to better responses; and (2) current question parsers are not perfect, resulting in wrong parameters to instantiate neural modules. Future work may focus on learning better question parsers or directly deploying a better off-the-shelf parser tool.
Table 5: Experiment results on the TGIF-QA benchmark. The visual features are: ResNet-152 (R), C3D (C), Flow CNN from two-stream model (F), VGG (V), ResNeXt-101 (RX).
AVSD	BLEU4	METEOR	ROUGE-L	CIDEr
Gen.	0.396	0.263	0.549	1.059
GT	0.408	0.272	0.560	1.115
TGIF-QA	Count (Loss)	Action (Acc)	Transition (Acc)	FrameQA (Acc)
Gen.	2.65	0.845	0.887	0.747
GT	1.90	0.857	0.898	0.780
Table 6: Comparison of VGNMN on AVSD (top) and TGIF-QA (bottom) when using generated ("Gen.") vs. ground-truth ("GT") programs.
For additional experiment results, qualitative samples, and analysis between model variants, refer to Appendix B and C.
In this work, we introduce Video-grounded Neural Module Network (VGNMN). VGNMN consists of dialogue and video understanding neural modules, each of which performs entity and action-level operations on language and video components. Our comprehensive experiments on AVSD and TGIFQA benchmarks show that our models can achieve competitive performance while promoting a compositional and interpretable learning approach.
During the duration of this work, there have been no ethical concerns regarding the model implementation, training, and testing. The data used in this work has been carefully reviewed and accordingly to the description from the original authors, we did not find any concerns on any significant biases. For
any potential application or extension of this work, we would like to highlight some specific concerns. First, as the work is developed to build an intelligent dialogue agents, models should not be used with the intention to create fake human profiles for any harmful purposes (e.g. fishing or spreading fake news). For wider use of dialogue systems, the application of work might result in certain impacts to some stakeholders whose jobs may be affected by this application (e.g. customer service call agents). We hope any application should be carefully considered against these potential risks.
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016. Analyzing the behavior of visual question answering models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1955-1960, Austin, Texas. Association for Computational Linguistics.
Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Stefan Lee, Peter Anderson, Irfan Essa, Devi Parikh, Dhruv Batra, Anoop Cherian, Tim K. Marks, and Chiori Hori. 2019. Audio-visual scene-aware dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016a. Learning to compose neural networks for question answering. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1545-1554, San Diego, California. Association for Computational Linguistics.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016b. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 39-48.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved