SlowGuess/NAACL_2022_OCR huggingface dataset:
Comparison of Different Architectures We compare the three architectures shown in Figure 2, namely, the widely-adopted architecture based on Cross-Modal Encoding, ClipBERT, and the architecture proposed in this paper. For all architectures, we use RoBERTa (Liu et al., 2019) as a language encoder and Swin Transformer (Liu et al., 2021) as a vision encoder. We sparsely sample 8 clips from each video, then uniformly sample a single frame within each clip. In addition, we remove the video scene graph from our model and only conduct single-step reasoning to ensure fairness. Other hyper-parameters are the same for all models. The results are given in Table 2, which show that our architecture achieves the best performance, in spite of the removing of the video scene graph and only conducting single-step reasoning. This may because that the proposed architecture makes decisions based on the global video information, unlike ClipBERT (Lei et al., 2021) and most other methods, which integrates the decision made by each clip to get the final decision. Furthermore, unlike previous works, we make a more apparent distinction between the process of question understanding, video understanding, reasoning, and answer deci
Method	Pre-training data	MSRVTT	TGIF-QA
QA	MC	Action	Transition	FrameQA
SNUVL (Yu et al., 2016) (by Yu et al.)	-	-	65.4	-	-	-
ST-VQA (Jang et al., 2017) (by Fan et al.)	-	30.9	66.1	60.8	67.1	49.3
CT-SAN (Yu et al., 2017) (by Yu et al.)	-	-	66.4	-	-	-
MLB (Kim et al., 2016) (by Yu et al.)	-	-	76.1	-	-	-
JSFusion (Yu et al., 2018b)	-	-	83.4	-	-	-
ActBERT (Zhu and Yang, 2020)	HowTo100M	-	85.7	-	-	-
Co-Memory (Gao et al., 2018) (by Fan et al.)	-	32.0	-	68.2	67.1	49.3
AMU (Xu et al., 2017b)	-	32.5	-	-	-	-
PSAC (Li et al., 2019)	-	-	-	70.4	76.9	55.7
Heterogenous Memory (Fan et al., 2019)	-	33.0	-	73.9	77.8	53.8
QuesST (Jiang et al., 2020)	-	-	-	75.9	81.0	59.7
HCRN (Le et al., 2020)	-	35.6	-	75.0	81.4	55.9
MASN (Seo et al., 2021)	-	35.2	-	84.4	87.4	59.5
VQA-T (Yang et al., 2021)	-	39.6	-	-	-	-
VQA-T (Yang et al., 2021)	HowTo100M	40.4	-	-	-	-
VQA-T (Yang et al., 2021)	HowToVQA69M	41.5	-	-	-	-
HGQA (Xiao et al., 2021)	-	38.6	-	76.9	85.6	61.3
CLIPBERT (Lei et al., 2021)	COCO & VG	37.4	88.2	82.8	87.8	60.3
Ours	-	41.6	91.4	84.6	90.1	62.5
Table 1: Comparison with state-of-the-art methods on video question answering. We verified performance on standard test sets of three datasets. The evaluation metric is accuracy. It is worth specifying that ActBERT, VQA-T, and CLIPBERT use additional large-scale data for pre-training. The results show that our method achieves the best performance without the use of additional pre-training data.
Architecture	#params	MSRVTT (Acc.)
QA	MC
Existing methods	257M	35.2 ± 0.32	85.4 ± 0.36
ClipBERT	215M	37.8 ± 0.28	88.7 ± 0.29
Ours*	222M	38.5 ± 0.14	89.1 ± 0.16
sion, providing a basis for introducing video scene graphs and dynamic multistep reasoning.
Analysis of Video Scene Graph We introduce the video scene graph to get a structural visual semantic representation which is better for reasoning. We use all architectures shown in Figure 2 as the benchmarks to evaluate the effect of the video scene graph. We use an image captioning model to extract captions for each clip. Then we use the scene parser (Schuster et al., 2015) to convert each caption sentence into a semantic sub-graph and integrate the same nodes of each sub-graph to obtain a video scene graph. We use a 2-layer Graph Attention Network (Veličković et al., 2017) with 12 heads to learn the representations of the scene
Table 2: Our proposed architecture vs. Previous mainstream architectures. All architectures use the same language and vision encoder. For the architecture of the existing methods, we use standard Transformer Encoder as the module of Cross-Modal Encoding. For fairness, we remove the video scene graph from our model and only do single-step reasoning (Ours*). The rest of the hyper-parameters are the same as each other.
Architecture	Visual Feat.	MSRVTT
(Acc.)
QA	MC
Existing methods	Image	35.2 ± 0.32	85.1 ± 0.36
Image + Caption	36.3 ± 0.26	85.9 ± 0.24
Image + Scene Graph	37.1 ± 0.16	86.4 ± 0.18
ClipBERT	Image	37.8 ± 0.28	88.7 ± 0.29
Image + Caption	38.4 ± 0.22	89.2 ± 0.19
Image + Scene Graph	39.2 ± 0.21	89.8 ± 0.18
Ours	Image	38.5 ± 0.14	89.1 ± 0.16
Image + Caption	39.0 ± 0.12	89.4 ± 0.11
Image + Scene Graph	39.5 ± 0.14	90.2 ± 0.11
Table 3: Impact of the video scene graph.
graph. As shown in Table 3, adding the clip-level captions improves performance. And the video scene graph brings further improvement. The video scene graph contains video-level semantic information in a structural form. Therefore, it can better represent key semantic information and reduce redundant information.
Analysis of Dynamic Reasoning As Table 4 shows. We first evaluate the simple static reasoning mechanism adopted by most previous works. Then we evaluate the performance of the 3.3 Evaluator and 3.3 Integrator. Row 0 shows the result of the single-step reasoning, achieving  $39.5\%$  accuracy on MSRVTT-QA. As expected, the static multistep reasoning mechanism (row 1-4) performs better and achieves the best performance around Step = 3. But the performance does not increase when setting a larger number of steps. It means that the average complexity of all questions is moderate, which can be handled well with three-step reason
Step Strategy	Integration Strategy	Step	MSRVTT (Acc.)
QA	MC
0	Single-Step	-	1	39.5 ± 0.14	89.9 ± 0.11
1	Static Multistep	-	2	40.2 ± 0.12	90.4 ± 0.09
2	3	40.6 ± 0.12	90.6 ± 0.12
3	4	40.4 ± 0.13	90.4 ± 0.10
4	5	40.2 ± 0.11	90.2 ± 0.10
5	Evaluator	-	≤ 5	40.8 ± 0.05	90.7 ± 0.04
6	Static Multistep	MeanPooling	2	39.8 ± 0.11	90.3 ± 0.10
7	3	40.2 ± 0.10	90.4 ± 0.12
8	4	40.1 ± 0.13	90.4 ± 0.09
9	5	39.8 ± 0.12	90.2 ± 0.08
10	MaxPooling	2	40.3 ± 0.08	90.4 ± 0.07
11	3	40.9 ± 0.09	90.5 ± 0.08
12	4	40.6 ± 0.07	90.7 ± 0.09
13	5	40.8 ± 0.08	90.6 ± 0.07
14	Integrator	2	40.4 ± 0.05	90.5 ± 0.04
15	3	41.0 ± 0.04	90.7 ± 0.04
16	4	40.8 ± 0.06	90.8 ± 0.06
17	5	40.7 ± 0.04	90.6 ± 0.05
18	Evaluator	Integrator	≤ 5	41.6 ± 0.02	91.4 ± 0.03
Table 4: Impact of dynamic reasoning strategy. When using a static reasoning strategy, the model will execute multistep reasoning with a fixed number of steps. When using a dynamic reasoning strategy, the model will dynamically determine the number of steps of multistep reasoning according to the problem's difficulty (Evaluator) and dynamically get the weight sum of intermediate reasoning results as the input of answer decision (Integrator).
ing. The Evaluator can dynamically decide the number of reasoning steps  $(\leq 5)$  for each question. Row 5 shows that using different numbers of reasoning steps for questions of different complexity can perform better than a static multistep reasoning mechanism. At the same time, we believe that each step of reasoning pays attention to different parts of the video content and gradually deepens the understanding of the question. Therefore, the intermediate reasoning results are also helpful in choosing an answer. Row 6-17 show that the Integrator can further improves the performance and has better integration ability than the traditional pooling method. Row 18 shows that the Evaluator and Integrator can promote each other and achieve the best performance.
5.3 Interpretability
Our model is interpretable by visualizing the attention weight of the Reader. As Figure 4 shows, we conclude with the following two conclusions: (1) Video scene graph can better represent the visual semantic information. The Video Scene Graph contains the global semantic information of the video, and the structured form can not only highlight the critical semantic elements but also reduce the interference of redundant information. (2) Dynamic multistep reasoning mechanism can deepen the understanding of the question iteratively. With the reasoning process, the semantic
Question: what does the video show old footage of a man and also of? Answer: beach
Reasoning: step1/3
step 2/3
step 3/3
Figure 4: Visualization for Dynamic Multistep Reasoning. We trace the attention weight distribution of the Reader to visualize the dynamic multistep reasoning process. For this case, the output of the Evaluator is  $S = 3$ , which means that three steps of reasoning are required. With the reasoning process, the attention weight of 'beach' is increasing. On the contrary, the attention weight of the irrelevant nodes reduced gradually. It indicates that the reasoning mechanism is gradually deepening the understanding of the question.
nodes related to the question obtain more and more attention. On the contrary, the attention weight of the irrelevant nodes reduced gradually. It shows that the dynamic multistep reasoning mechanism can deepen the understanding of the question.
We propose a dynamic reasoning mechanism based on video scene graph for video QA to alleviate the drawback of existing methods, that is, lack of deep understanding and multistep reasoning. Experiments show that our method significantly surpasses previous methods on multiple video QA datasets due to better understanding and reasoning mechanisms and achieves much better interpretability by backtracing along with the attention mechanism to the video scene graph. In addition, different from the conventional manners that perform classification after crossmodal feature encoding, the model realizes the decoupling of question understanding and video understanding and the decoupling of understanding and decision-making, thus providing more possibilities for improvement. On the one hand, we can optimize video QA by introducing external knowledge, designing more effective reasoning mechanisms, defining and constructing better video scene graphs. On the other hand, we can also jointly model multimodal QA, such as QA based on videos, tables, texts, and graphs.
This work is supported by the Beijing Natural Science Foundation (Z190020). This work is also supported by Baidu and CASICT Joint Project. We would like to thank the anonymous reviewers for their valuable feedback.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39-48.
Qingxing Cao, Xiaodan Liang, Bailing Li, Guanbin Li, and Liang Lin. 2018. Visual question reasoning on general dependency tree. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7249-7257.
Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. 2021. Lgesql: Line graph enhanced text-to-sql model with mixed local and non-local relations. arXiv preprint arXiv:2106.01093.
Ronald P Carver and James V Hoffman. 1981. The effect of practice through repeated reading on gain in reading ability using a computer-based instructional system. Reading Research Quarterly, pages 374-390.
Aman Chadha, Gurneet Arora, and Navpreet Kaloty. 2020. iperceive: Applying common-sense reasoning to multi-modal dense video captioning and video question answering. arXiv preprint arXiv:2011.07735.
Anna Chang and Sonia Millett. 2013. Improving reading rates and comprehension through timed repeated reading.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2019. Multi-step retriever-reader interaction for scalable open-domain question answering. arXiv preprint arXiv:1905.05733.
Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. 2019. Heterogeneous memory enhanced multimodal attention model for video question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1999-2007.
Jiyang Gao, Runzhou Ge, Kan Chen, and Ram Nevatia. 2018. Motion-appearance co-memory networks for video question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6576-6585.
Noa Garcia and Yuta Nakashima. 2020. Knowledge-based video question answering with unsupervised scene descriptions. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pages 581-598. Springer.
Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima. 2020. Knowit vqa: Answering knowledge-based questions about videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10826-10834.
Greta Gorsuch and Etsuo Taguchi. 2008. Repeated reading for developing reading fluency and reading comprehension: The case of efl learners in vietnam. System, 36(2):253-278.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, JianGuang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-sql in cross-domain database with intermediate representation. arXiv preprint arXiv:1905.08205.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 804-813.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatiotemporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2758-2766.
Jianwen Jiang, Ziqiang Chen, Haojie Lin, Xibin Zhao, and Yue Gao. 2020. Divide and conquer: Question-guided spatio-temporal contextual attention for video question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11101-11108.
Justin Johnson, Andrej Karpathy, and Li Fei-Fei. 2016. Densecap: Fully convolutional localization networks for dense captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4565-4574.
Hyounghun Kim, Zineng Tang, and Mohit Bansal. 2020a. Dense-caption matching and frame-selection gating for temporal localization in videoqa. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4812-4822.
Jin-Hwa Kim, Kyoung-Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. 2016. Hadamard product for low-rank bilinear pooling. arXiv preprint arXiv:1610.04325.
Seonhoon Kim, Seohyeong Jeong, Eunbyul Kim, Inho Kang, and Nojun Kwak. 2020b. Self-supervised pre-training and contrastive representation learning for multiple-choice video qa. arXiv preprint arXiv:2009.08043.
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. 2020. Hierarchical conditional relation networks for video question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9972-9981.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. 2021. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7331-7341.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. Tvqa: Localized, compositional video question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1369-1379.
Jie Lei, Licheng Yu, Tamara Berg, and Mohit Bansal. 2020. Tvqa+: Spatio-temporal grounding for video question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8211-8225.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020. Hero: Hierarchical encoder for video+ language omni-representation pretraining. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2046-2065.
Tianhao Li and Limin Wang. 2020. Learning spatiotemporal features via video and text pair discrimination. arXiv preprint arXiv:2001.05691.
Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, and Chuang Gan. 2019. Beyond rnns: Positional self-attention with co-attention for video question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8658-8665.
Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1736-1745.
Jiangming Liu, Matt Gardner, Shay B Cohen, and Mirella Lapata. 2020a. Multi-step inference for reasoning over paragraphs. arXiv preprint arXiv:2004.02995.
Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, and Jingjing Liu. 2020b. Violin: A large-scale dataset for video-and-language inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10900-10910.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030.
Kwabena Nuamah. 2021. Deep algorithmic question answering: Towards a compositionally hybrid ai for algorithmic reasoning. arXiv preprint arXiv:2109.08006.
Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2021. Unik-qa: Unified representations of structured and unstructured knowledge for open-domain question answering. arXiv preprint arXiv:2012.14610, 54:57-60.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2016. Faster r-cnn: towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6):1137-1149.
Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7008-7024.
Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D. Manning. 2015. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Workshop on Vision and Language (VL15), Lisbon, Portugal. Association for Computational Linguistics.
Ahjeong Seo, Gi-Cheon Kang, Joonhan Park, and Byoung-Tak Zhang. 2021. Attend what you need: Motion-appearance synergistic networks for video question answering. arXiv preprint arXiv:2106.10446.
Dandan Song, Siyi Ma, Zhanchen Sun, Sicheng Yang, and Lejian Liao. 2021. Kvl-bert: Knowledge enhanced visual-and-linguistic bert for visual commonsense reasoning. Knowledge-Based Systems, 230:107408.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464-7473.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2019. Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. arXiv preprint arXiv:1911.04942.
Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, and Tat-Seng Chua. 2021. Video as conditional graph hierarchy for multi-granular question answering. arXiv preprint arXiv:2112.06197.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017a. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645-1653.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017b. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645-1653.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288-5296.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2021. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1686-1697.
Youngjae Yu, Jongseok Kim, and Gunhee Kim. 2018a. A joint sequence fusion model for video question answering and retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pages 471-487.
Youngjae Yu, Jongseok Kim, and Gunhee Kim. 2018b. A joint sequence fusion model for video question answering and retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pages 471-487.
Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. 2016. Video captioning and retrieval models with semantic attention. arXiv preprint arXiv:1610.02947, 6(7).
Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. 2017. End-to-end concept word detection for video captioning, retrieval, and question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3165-3173.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. Merlot: Multimodal neural script knowledge models. arXiv preprint arXiv:2106.02636.
Mantong Zhou, Minlie Huang, and Xiaoyan Zhu. 2018. An interpretable reasoning network for multi-relation question answering. arXiv preprint arXiv:1801.04726.
Linchao Zhu and Yi Yang. 2020. Actbert: Learning global-local video-text representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8746-8755.