Venue: NAACL 2022

# Dynamic Multistep Reasoning based on Video Scene Graph

for Video Question Answering

Jianguo Mao\({}^{1,2}\)

This is joint work of CASICT and Baidu. 
\begin{tabular}{}

Wenbin Jiang\({}^{3}\)

Xiangdong Wang\({}^{1}\)

Zhifan Feng\({}^{3}\)

Yajuan Lyu\({}^{3}\)

Hong Liu\({}^{1}\)

Yong Zhu\({}^{3}\)

###### Abstract

Existing video question answering (video QA) models lack the capacity for deep video understanding and flexible multistep reasoning. We propose for video QA a novel model which performs dynamic multistep reasoning between questions and videos. It creates video semantic representation based on the video scene graph composed of semantic elements of the video and semantic relations among these elements. Then, it performs multistep reasoning for better answer decision between the representations of the question and the video, and dynamically integrate the reasoning results. Experiments show the significant advantage of the proposed model against previous methods in accuracy and interpretability. Against the existing state-of-the-art model, the proposed model dramatically improves more than \(4\%/3.1\%/2\%\) on the three widely used video QA datasets, MSRVTT-QA, MSRVTT multi-choice, and TGIF-QA, and displays better interpretability by backtracking along with the attention mechanisms to the video scene graphs.

+
Footnote †: journal: Computer Vision and Pattern Recognition

0
Footnote 0: This is joint work of CASICT and Baidu. _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3894 - 3904

[FOOTNOTE:5]Footnote 5: footnotetext: This is joint work of CASICT and Baidu. _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3894 - 3904

## 1 Introduction

Video question answering (video QA) aims to answer questions according to the given videos. It is usually defined as a classification task, where the most appropriate answer is chosen from a candidate list for the given question and video. Existing methods for video QA conduct direct answering selection based on the multimodal encoding of questions and videos Jang et al. (2017); Lei et al. (2018, 2020). In recent years, researchers have proposed many optimization strategies for better performance in video question answering, e.g., designing delicate encoding mechanisms Kim et al. (2020a); Nuamah (2021); Gao et al. (2018); Li et al. (2019); Fan et al. (2019); Le et al. (2020); Jiang et al. (2020); Kim et al. (2020b); Seo et al. (2021), introducing video scene graphs Garcia and Nakashima (2020), adopting video pre-trained language models Li et al. (2020); Zellers et al. (2021); Li and Wang (2020); Lei et al. (2021); Sun et al. (2019), and leveraging external knowledge or resources Chadha et al. (2020); Garcia et al. (2020); Liu et al. (2020); Song et al. (2021); Garcia and Nakashima (2020). Compared with conventional monomodal question answering tasks such as text QA Oguz et al. (2021); Zhou et al. (2018); Lin et al. (2018) and table QA Cao et al. (2021); Wang et al. (2019). Video QA is more difficult due to the need for crossmodal understanding and reasoning of the video and the question. Existing methods are mainly concerned with how to encode the crossmodal features better. When faced with a complex question, they usually lack the abilities of deep understanding and complex reasoning.

Similar to the situations in text QA and table QA, it is also necessary for video QA to deeply understand the semantics of the context, namely, the video, and the reasoning on the context and the question. Statistical analysis on several datasets

Figure 1: Error cases of the previous state-of-the-art method Lei et al. (2021) that needed multistep reasoning. We demonstrate the multistep reasoning process based on question and video(left).



reveals that a significant percentage of failed cases of the state-of-the-art (SOTA) model ClipBERT Lei et al. (2021) is caused by the lack of deep understanding and reasoning. Figure 1 shows some error cases of the SOTA model on MSRVTT-QA. We randomly select one hundred error cases and find that about \(24\%\) of the error cases need multiple steps of reasoning on the question and the video, and about \(15\%\) of the error cases need a deep understanding of the semantic of the video. These cases could be solved by a model emphasizing deep understanding and reasoning. The questions in these three datasets are relatively simple according to the building procedures Xu et al. (2017); Yu et al. (2018); Jang et al. (2017). It will be more valuable for video QA to enable deep understanding and reasoning on the question and the video in realistic application scenarios.

In this work, we propose a novel dynamic reasoning model for video QA to overcome the weakness of previous models in deep understanding and reasoning. It first creates the video semantic representation from the video scene graph, which is composed of the semantic elements of the video and the semantic relations between these elements. Then it conducts multistep reasoning of the question based on the video semantic representation to generate a series of video-aware question representations. Finally, it generates the most appropriate question representation for the final answering decision by dynamically integrating these video-aware question representations according to the reasoning complicity prediction. Figure 2 shows the overall architecture of the proposed model and the comparison with previous methods. It simulates the reasoning procedure of human beings, while previous methods follow the pipeline of multimodal encoding and answering selection. In addition, the proposed model enables the decomposition of question understanding and video understanding, thus leading to more opportunities for future optimization. On the one hand, more external knowledge resources and better reasoning architectures can be introduced for better video QA performance. On the other hand, it can act as a unified framework for different QA tasks such as video QA, table QA, and text QA.

We verify the proposed model on three well-known datasets, MSRVTT-QA Xu et al. (2017), MSRVTT multi-choice Yu et al. (2018), and TGIF-QA Jing et al. (2017), widely used in recent video QA works Jiang et al. (2017); Gao et al. (2018); Li et al. (2019); Fan et al. (2019); Le et al. (2020); Zhu and Yang (2020); Lei et al. (2021); Seo et al. (2021). Experiments show that our model achieves dramatic improvement over the powerful state-of-the-art model ClipBERT Lei et al. (2021), with an average accuracy increment of more than 3 percentage points. Ablation studies show that the dynamic reasoning strategy significantly outperforms previous implicit simple reasoning strategies. The video semantic representation based on the video scene graph makes the dynamic reasoning strategy work better. Backtracing along with the at

Figure 2: Comparison of our model architecture with (a). popular video-and-language learning paradigm and (b).ClipBERT. In contrast to most previous methods that adopt simple and implicit reasoning mechanisms, Our method uses video scene graphs to represent the video semantic information and adopt a dynamic multistep reasoning mechanism.

 

tention mechanism to the video scene graph clearly shows the semantic elements that answer decision relies on each reasoning step, thus giving better interpretability than most of the previous methods.

## 2 Background

Given the question \(q\), video question answering requires choose the correct answer \(\hat{a}\) from the candidates set \(\Omega_{a}\) according to the video content \(V\).

\[\hat{a}=\underset{a\in\Omega_{a}}{\mathrm{argmax}}p(a|q,V;\theta) \tag{1}\]

As Figure 2 (a) shows, most existing works utilize offline (stop gradient) extracted dense video features and text features. As Figure 2 (b) shows, ClipBERT Lei et al. (2021) achieves the state-of-the-art by using sparsely sampled clips and raw text for end-to-end training, yet suffer from two main drawbacks: (i) Lack of a deep understanding of the video content and reasoning on the question based on the video. (ii) Strong coupling between video and question modeling process, which needs additional image-text pair data for pre-training to enhance the ability of the text encoder to model multimodal features and leads to poor scalability and repeated computation. As shown in Figure 2 (c), we propose a simple but effective architecture to solve the weakness of previous works. It first decouples the question understanding module and the video understanding module. Then, we introduce the video scene graph to get a better representations of the video semantic information, which can enhance the understanding of the video content and its graph structure is also better for reasoning. At last, we design a dynamic multistep reasoning mechanism to iteratively deepen the understanding of the question according to the video content.

## 3 Method

### Overall Architecture

Figure 3 gives an overview of the model architecture. For the visual representation, we use both video scene graphs and image features. On the one hand, we construct a video scene graph to represent semantic information in a structural form. On the other hand, we use Swin Transformer Liu et al. (2021) to extract image features to make up for the missing information of the scene graph. The reasoning module (_Reasoner_) will iteratively updates the understanding of the question according to the video content. The _Evaluator_ will decide the number of reasoning steps according to the complexity of the question. The _Integrator_ will integrate all intermediate reasoning results to get the final reasoning results. The answer decision

Figure 3: Overview of our model architecture. It contains two part: 3.2. Video Representation Learning (right) and 3.3. Dynamic Multistep Reasoning (left)



module chooses an answer according to the final comprehensive understanding of the question.

### Video Representation Learning

We chose a structured video scene graph to describe video semantics which is better for reasoning. We also extract image features by Swin Transformer Liu et al. (2021) to make up for the missing information in the scene graph. The video scene graph and the image features constitute the _Video Representation Memory_ shown in Figure 3, a memory for the _Reader_ module to access.

#### 3.2.1 Video Scene Graph

The video scene graph is the basis for conducting dynamic reasoning, it is a graph-based semantic representation of video content, representing the objects in the video, their attributes, and their relationships in a structured form. Unlike the image scene graph commonly used in visual question answering, our video scene graph is semantically richer and contains spatio-temporal information of the video. Specifically, We first use an image captioning model to generate captions for each clip. Then we use the scene parser Schuster et al. (2015) to convert each caption sentence into a semantic sub-graph and integrate the same nodes of each sub-graph to obtain a video scene graph. Compared to caption sentences, the video scene graph represents the video-level semantic information in a structured form, better modeling the visual semantic information.

Graph Representation LearningWe first obtain the embeddings of nodes \({\bf n}=\left\{n_{1},n_{2},...,n_{m}\right\}\) and edges in the video scene graph via a parameter-sharing language encoder. Then, we use graph attention neural network Velickovic et al. (2017) iteratively to update the representation of the scene graph. Each node updates its representation based on the correlation with its neighbor nodes.

\[{\bf n^{{}^{\prime}}_{i}}=a_{ii}{\bf W}{\bf n}_{i}+\sum_{j\in{\cal N}_{i}}a_{ij }{\bf W}{\bf n}_{j} \tag{2}\]

where \({\bf W}\) is a weight matrix, \(a_{ij}\) is the attention weight of node \({\bf n}_{i}\) and \({\bf n}_{j}\), and \({\cal N}_{i}\) is the neighbors of the node \({\bf n_{i}}\) in the graph. In our experiments, we use standard graph attention neural network setting, applying the LeakyReLU nonlinearity (with negative input slope \(\alpha=0.2\)). At the same time, the edges have explicit meanings of relations between nodes, so we also consider edges features \({\bf e}_{ij}\) when calculating attention weight. The attention weight \(a_{ij}\) are computed as

\[a_{ij}=\frac{\exp(\mathrm{LeakyReLU(A({\bf n}_{i},{\bf n}_{j},{\bf e}_{ij})))} }{\sum_{k\in{\cal N}_{i}}\exp(\mathrm{LeakyReLU(A({\bf n}_{i},{\bf n}_{k},{ \bf e}_{ik}))})} \tag{3}\]

\[\mathrm{A({\bf n}_{i},{\bf n}_{j},{\bf e}_{ij})}={{\bf W_{a}}^{\top}}[{\bf W}{ \bf n}_{i}||{\bf W}{\bf n}_{j}||{\bf W}_{e}{\bf e}_{ij}] \tag{4}\]

where \(.^{\top}\) represents transposition, \(||\) is the concatenation operation and the \({\bf W_{a}}\) and \({\bf W_{e}}\) are weight matrices.

#### 3.2.2 Image Features

We extract image features to make up for the missing information in the scene graph. First, we use Lei et al.'s sparse sampling method to sparsely and randomly sample \(N_{train}\) clips \(\left\{c_{i}\right\}_{i=1}^{N_{train}}\) from video. \(N_{train}\) is typically much smaller than the entire video length \(N\). This sampling method can reduce the computation cost and obtain better performance than dense sampling. For inference, we uniformly sample \(N_{test}\) clips of the same duration. Swin Transformer Liu et al. (2021) is one of the mainstream visual backbone networks. It alleviates the problem of large variations in the scale of visual entities and the high resolution of pixels in images. We use it as a vision encoder \({\mathcal{E}}_{v}\) to extract clip features \(\left\{F_{i}\right\}_{i=1}^{N_{train}},F_{i}={\mathcal{E}}_{v}(c_{i})\in \mathbbm{R}^{w\times w\times d}\), where \(w\) is the window size and \(d\) is the feature dimension.

### Dynamic Multistep Reasoning

Humans will deepen their understanding of a complex question through repeated reading the context information. The more complex the question, the more repetitions are required Chang and Millett (2013); Gorsuch and Taguchi (2008); Carver and Hoffman (1981). At each step of reading, people will focus on different parts of the context information. Inspired by it, we designed the dynamic multistep reasoning mechanism. It will iteratively update the understanding of the question based on the video representations. We first extract the question representation by language model RoBERTa Liu et al. (2019). Specifically, we concatenate the question text with a special token \([\mathrm{CLS}]\) as the input and take the \([\mathrm{CLS}]\)'s hidden state \(R^{0}\) as the representation of the question. At the first reasoning step, we select \(R^{0}\) as the input. Then, we get question-related information from Video Representation Memory 

through the _Reader_, which is an attention mechanism. Then, we use this retrieved video information to update the understanding of the question and get the first reasoning step result \(R_{1}\) through the _Updater_. At the next reasoning step, we select \(R_{1}\) as the input. After \(S\) steps of reasoning, we obtain all results \(\mathbf{R}=\{R^{0},R^{1},R^{2},...,R^{S}\}\).

\[R^{s}=\mathrm{Updater}(R^{s-1},V^{s-1}) \tag{5}\]

Updater consists of two linear transformations with a ReLU activation in between.

\[\mathrm{Updater}(Q,K)=Q+\mathrm{ReLU}(\mathbf{W_{1}}K+b_{1})\mathbf{W_{2}}+ b_{2} \tag{6}\]

where \(\mathbf{W_{1}}\) and \(\mathbf{W_{2}}\) are weight matrices and \(b_{1}\) and \(b_{2}\) are biases. \(V_{s-1}\) represents question-related video information.

\[V^{s-1}=\mathrm{Reader}(R^{s-1},\mathbf{Vid}) \tag{7}\]

where \(\mathbf{Vid}\) consists of the node features \(\{n_{1},n_{2},...,n_{m}\}\) of the video scene graph and image features \(\{F_{1},F_{2},...,F_{N_{train}}\}\). We use Scaled Dot-Product Attention Vaswani et al. (2017) as the _Reader_. The input consists of query \(Q\), and context \(K\) of dimension \(d_{k}\).

\[\mathrm{Reader}(Q,K)=\mathrm{softmax}(\frac{QK^{\top}}{\sqrt{d_{k}}})K \tag{8}\]

The word _dynamic_ has two meanings: (i). dynamically decide the number of reasoning steps according to the question's complexity. _(3.3 Evaluator)_. (ii). dynamically integrate the results of all reasoning steps as the final result. _(3.3 Integrator)_.

EvaluatorWhen humans begin faced with different complexity questions, they will dynamically adjust the number of times to read relevant information Chang and Millett (2013); Gorsuch and Taguchi (2008); Carver and Hoffman (1981). We propose the first dynamic reasoning strategy by imitating the human reading and understanding mechanisms. It will decides the number of reasoning steps according to the complexity of the question. Specifically, we perform a nonlinear transformation with GumbelSoftmax as activation function on the question representation \(R^{0}\), and output an S-dimensional vector \(\mathrm{D}(R_{0})\in\mathbb{R}^{1\times S}\) to represents the distribution probability of the number of reasoning steps from 1 to \(\mathrm{S}\).

\[\mathrm{D}(R_{0})=\mathrm{GumbelSoftmax}(\mathbf{W_{d}}R^{0}+b) \tag{9}\]

Where \(\mathbf{W_{d}}\) is a weight matrix, and \(b\) is a bias. We choose the one with the greatest probability as the number of the reasoning steps \(S\).

IntegratorIn the reasoning process, the _Reader_ pays attention to the different parts of the video content at each step to gradually deepen the question's understanding. Therefore, we think the intermediate reasoning results are also helpful in choosing an answer. After \(S\) steps of reasoning, we select all intermediate reasoning results as input and perform a nonlinear transformation with softmax function to calculate the distribution of the weight of \(\mathbf{R}\) and get the weighted sum as the final reasoning results \(R^{f}\).

\[R^{f}=\mathrm{softmax}(\mathbf{W_{f}R}+\mathbf{b})\mathbf{R} \tag{10}\]

where \(\mathbf{W_{f}}\) is a weight matrix and \(b\) is a bias.

### Answer Decision

Given the final reasoning results \(R^{f}\), we use two fully-connected layers as a classifier to obtain the logits \(l_{f}\) for the answer options. Then we use a softmax function to obtain the probability distribution of each answer option and apply cross-entropy loss as our model loss \(\mathcal{L}\).

\[l_{f}=\mathrm{classifier}(R^{f}) \tag{11}\]

\[\hat{y}=\mathrm{softmax}(l_{f}),~{}\mathcal{L}=-\sum_{i=1}^{M}y_{i}log\hat{ y_{i}} \tag{12}\]

## 4 Related Work

Video QA requires fine-grained modeling of multimodal features. We have witnessed many efforts devoted to video understanding for video QA. Some methods use visual techniques such as object detection Ren et al. (2016) and image captioning Johnson et al. (2016); Rennie et al. (2017) to extract additional visual information Kim et al. (2020). In recent years, visual pre-training based on large-scale data has become a popular method to improve video applications including video QA Li et al. (2020); Zellers et al. (2021); Li and Wang (2020); Sun et al. (2019). In addition, several advanced techniques such as contrastive self-supervised learning Kim et al. (2020) and symbolized video scene graph Garcia and Nakashima (2020) are proposed to improve the performance of video understanding 

for video QA. We extract video semantic representations based on both the visual pre-training model Liu et al. (2021) and the video scene graphs. Experiments show the amazing complementarity of the two kinds of information.

Video QA also requires flexible reasoning for complicated questions and videos. Although the reasoning capability is rarely emphasized in previous work for video QA, it is broadly investigated in other QA tasks Clark et al. (2018). For example, text QA resorts to iterative update of the question and the context Das et al. (2019); Liu et al. (2020a), table QA generates a structural query such as SQL which is then executed on the tabular data Guo et al. (2019), and visual QA builds a specific module network according to the question and runs it on the image Andreas et al. (2016); Cao et al. (2018); Hu et al. (2017). The overall architecture of the proposed method is similar to the iterative reasoning strategies for text QA but with significant innovation. Our model can dynamically determine the best integration strategy for the intermediate reasoning results according to the given question and video, leading to better interpretability and much better performance.

## 5 Experiments

In this section, we validate our method on three mainstream video QA datasets. We conduct comparison experiments with previous works and perform ablation experiments to analyze the critical improvement in our proposed method. We use standard train/val/test splits for all datasets and use accuracy to measure the performance. All experimental results are the mean and standard deviation of ten replicate experiments.

### Datasets

Msrvtt-QASR-VTT Xu et al. (2016) is a large video description dataset. It provides 10k web video clips with 41.1 hours and 200k clip-sentence pairs in total. MSRVTT-QA Xu et al. (2017) is created based on clip-sentence pairs in MSR-VTT automatically through a program. It contains 243k open-ended questions with 1500 answers.

MsRvtt-McMSRVTT-MC multiple-choice Yu et al. (2018) is a dataset for video-text matching tasks built on MSR-VTT with videos are used as queries, captions as answers. Each video contains five captions. Only one is correct.

Toif-QATGIF-QA Jang et al. (2017) dataset contains 165K QA pairs for the animated GIFs from the TGIF dataset. We experiment on 3 TGFIF-QA tasks: Repeating Action and State Transition for multiple-choice QA and Frame QA for open-ended QA. We follow most previous works and ClipBERT's Lei et al. (2021) approach to leave the Count task as future work as it requires directly modeling full-length videos.

### Results and Analysis

#### 5.2.1 Comparison with existing approaches

As shown in Table 1, our method reaches the new state-of-the-art and achieves \({\bf 41.6}\%/{\bf 91.4}\%/{\bf 84.6}\%,{\bf 90.1}\%,{\bf 62.5}\%\) accuracy on MSRVTT-QA/MSRVTT-MC (multi-choice) _/_TGIUF-QA_ (Action, Transition, FrameQA), with \({\bf 4.2}\%/{\bf 3.2}\%/{\bf 1.8}\%,{\bf 2.3}\%,{\bf 2.2}\%\) improvement over the previous state-of-the-art method ClipBERT Lei et al. (2021). The _Pre-training data_ column represents that the model was pre-trained with additional data. The results show that our method achieves the best performance on all three video question answering datasets without using additional data.

#### 5.2.2 Ablations Analysis

Comparison of Different ArchitecturesWe compare the three architectures shown in Figure 2, namely, the widely-adopted architecture based on Cross-Modal Encoding, ClipBERT, and the architecture proposed in this paper. For all architectures, we use RoBERTa Liu et al. (2019) as a language encoder and Swin Transformer Liu et al. (2021) as a vision encoder. We sparsely sample 8 clips from each video, then uniformly sample a single frame within each clip. In addition, we remove the video scene graph from our model and only conduct single-step reasoning to ensure fairness. Other hyper-parameters are the same for all models. The results are given in Table 2, which show that our architecture achieves the best performance, in spite of the removing of the video scene graph and only conducting single-step reasoning. This may because that the proposed architecture makes decisions based on the global video information, unlike ClipBERT Lei et al. (2021) and most other methods, which integrates the decision made by each clip to get the final decision. Furthermore, unlike previous works, we make a more apparent distinction between the process of question understanding, video understanding, reasoning, and answer deci 

sion, providing a basis for introducing video scene graphs and dynamic multistep reasoning.

Analysis of Video Scene GraphWe introduce the video scene graph to get a structural visual semantic representation which is better for reasoning. We use all architectures shown in Figure 2 as the benchmarks to evaluate the effect of the video scene graph. We use an image captioning model to extract captions for each clip. Then we use the scene parser Schuster et al. (2015) to convert each caption sentence into a semantic sub-graph and integrate the same nodes of each sub-graph to obtain a video scene graph. We use a 2-layer Graph Attention Network Velickovic et al. (2017) with 12 heads to learn the representations of the scene graph. As shown in Table 3, adding the clip-level captions improves performance. And the video scene graph brings further improvement. The video scene graph contains video-level semantic information in a structural form. Therefore, it can better represent key semantic information and reduce redundant information.

Analysis of Dynamic ReasoningAs Table 4 shows. We first evaluate the simple static reasoning mechanism adopted by most previous works. Then we evaluate the performance of the \(3.3~{}\textbf{Evaluator}\) and \(3.3~{}\textbf{Integrator}\). Row 0 shows the result of the single-step reasoning, achieving 39.5% accuracy on MSRVTT-QA. As expected, the static multistep reasoning mechanism (row 1-4) performs better and achieves the best performance around \(Step=3\). But the performance does not increase when setting a larger number of steps. It means that the average complexity of all questions is moderate, which can be handled well with three-step reason

\begin{table}
\begin{tabular}{l|c|c|c c c c} \hline \multirow{2}{*}{Method} & \multicolumn{2}{c|}{Pre-training data} & \multicolumn{2}{c|}{MSRVTT} & \multicolumn{2}{c}{TIGF-QA} \\ \cline{3-8}  & & QA & MC & Action & Transition & FrameQA \\ \hline SNUVL Yu et al. & - & - & 65.4 & - & - & - & - \\ ST-VQA Jang et al. (2017) (by Fan et al.) & - & 30.9 & 66.1 & 60.8 & 67.1 & 49.3 \\ CT-SAN Yu et al. (2017) (by Yu et al.) & - & - & 66.4 & - & - & - \\ MLB Kim et al. (2016) (by Yu et al.) & - & - & 76.1 & - & - & - \\ JSFusion Yu et al. (2018b) & - & - & 83.4 & - & - & - \\ AGBERT Zhu and Yang (2020) & HowTo100M & – & 85.7 & - & - & - \\ Co-Memory Gao et al. (2018) (by Fan et al.) & - & 32.0 & - & 68.2 & 67.1 & 49.3 \\ AMU Xu et al. (2017b) & - & 32.5 & - & - & - & - \\ PSAC Li et al. (2019) & - & - & - & 70.4 & 76.9 & 55.7 \\ Heterogeneous Memory Fan et al. (2019) & - & 33.0 & - & 73.9 & 77.8 & 53.8 \\ QuesST Jiang et al. (2020) & - & - & - & 75.9 & 81.0 & 59.7 \\ HCRN Le et al. (2020) & - & 35.6 & - & 75.0 & 81.4 & 55.9 \\ MASN Seo et al. (2021) & - & 35.2 & - & 84.4 & 87.4 & 59.5 \\ VQA-T Wang et al. (2021) & - & 39.6 & - & - & - & - \\ VQA-T Yang et al. (2021) & HowTo100M & 40.4 & - & - & - & - \\ VQA-T Yang et al. (2021) & HowToVQA69M & 41.5 & - & - & - & - \\ HGQA Xiao et al. (2021) & - & 38.6 & - & 76.9 & 85.6 & 61.3 \\ CLIPBERT Lei et al. (2021) & COCO \& VG & 37.4 & 88.2 & 82.8 & 87.8 & 60.3 \\ \hline Ours & - & **41.6** & **91.4** & **84.6** & **90.1** & **62.5** \\ \hline \end{tabular}
\end{table}
Table 1: Comparison with state-of-the-art methods on video question answering. We verified performance on standard test sets of three datasets. The evaluation metric is accuracy. It is worth specifying that ActBERT, VQA-T, and CLIPBERT use additional large-scale data for pre-training. The results show that our method achieves the best performance without the use of additional pre-training data.

\begin{table}
\begin{tabular}{c|c|c|c} \hline \multirow{2}{*}{Architecture} & \multirow{2}{*}{\#params} & \multicolumn{2}{c}{MSRVTT (Acc.)} \\ \cline{3-4}  & & QA & MC \\ \hline Existing & Image + Caption & \(35.2\pm 0.32\) & \(85.4\pm 0.36\) \\ methods & Image + Scene Graph & \(37.1\pm 0.16\) & \(86.4\pm 0.18\) \\ \hline ClipBERT & Image + Caption & \(37.8\pm 0.28\) & \(88.7\pm 0.29\) \\ \hline Ours* & 222M & \(\mathbf{38.5\pm 0.14}\) & \(\mathbf{89.1\pm 0.16}\) \\ \hline \end{tabular}
\end{table}
Table 2: **Our proposed architecture vs. Previous mainstream architectures. All architectures use the same language and vision encoder. For the architecture of the existing methods, we use standard Transformer Encoder as the module of Cross-Modal Encoding. For fairness, we remove the video scene graph from our model and only do single-step reasoning (Ours*). The rest of the hyper-parameters are the same as each other.**

\begin{table}
\begin{tabular}{c|c|c} \hline \multirow{2}{*}{Acchitecture} & \multirow{2}{*}{Visual Feaf.} & \multicolumn{2}{c}{MSRVTT} \\ \cline{3-4}  & & QA & MC \\ \hline Existing & Image + Caption & \(36.3\pm 0.26\) & \(85.9\pm 0.24\) \\ methods & Image + Scene Graph & \(37.1\pm 0.16\) & \(86.4\pm 0.18\) \\ \hline ClipBERT & Image + Caption & \(38.7\pm 0.28\) & \(88.7\pm 0.29\) \\ \hline City-BERT & Image + Caption & \(38.4\pm 0.22\) & \(89.2\pm 0.19\) \\ \hline Ours & Image + Scene Graph & \(39.2\pm 0.21\) & \(89.8\pm 0.18\) \\ \hline Ours & Image + Caption & \(38.5\pm 0.14\) & \(89.1\pm 0.16\) \\  & Image + Scene Graph & \(\mathbf{39.5\pm 0.14}\) & \(\mathbf{90.2\pm 0.11}\) \\ \hline \end{tabular}
\end{table}
Table 3: **Impact of the video scene graph.** 

ing. The _Evaluator_ can dynamically decide the number of reasoning steps (\(\leq 5\)) for each question. Row 5 shows that using different numbers of reasoning steps for questions of different complexity can perform better than a static multistep reasoning mechanism. At the same time, we believe that each step of reasoning pays attention to different parts of the video content and gradually deepens the understanding of the question. Therefore, the intermediate reasoning results are also helpful in choosing an answer. Row 6-17 show that the _Integrator_ can further improves the performance and has better integration ability than the traditional pooling method. Row 18 shows that the Evaluator and Integrator can promote each other and achieve the best performance.

### Interpretability

Our model is interpretable by visualizing the attention weight of the _Reader_. As Figure 4 shows, we conclude with the following two conclusions: (1) **Video scene graph can better represents the visual semantic information.** The Video Scene Graph contains the global semantic information of the video, and the structured form can not only highlight the critical semantic elements but also reduce the interference of redundant information. (2) **Dynamic multistep reasoning mechanism can deeper the understanding of the question iteratively.** With the reasoning process, the semantic nodes related to the question obtain more and more attention. On the contrary, the attention weight of the irrelevant nodes reduced gradually. It shows that the dynamic multistep reasoning mechanism can deepen the understanding of the question.

## 6 Conclusion

We propose a dynamic reasoning mechanism based on video scene graph for video QA to alleviate the drawback of existing methods, that is, lack of deep understanding and multistep reasoning. Experiments show that our method significantly surpasses previous methods on multiple video QA datasets due to better understanding and reasoning mechanisms and achieves much better interpretability by backtracing along with the attention mechanism to the video scene graph. In addition, different from the conventional manners that perform classification after crossmodal feature encoding, the model realizes the decoupling of question understanding and video understanding and the decoupling of understanding and decision-making, thus providing more possibilities for improvement. On the one hand, we can optimize video QA by introducing external knowledge, designing more effective reasoning mechanisms, defining and constructing better video scene graphs. On the other hand, we can also jointly model multimodal QA, such as QA based on videos, tables, texts, and graphs.

Figure 4: **Visualization for Dynamic Multistep Reasoning. We trace the attention weight distribution of the _Reader_ to visualize the dynamic multistep reasoning process. For this case, the output of the _Evaluation_ is \(S=3\), which means that three steps of reasoning are required. With the reasoning process, the attention weight of ‘beach’ is increasing. On the contrary, the attention weight of the irrelevant nodes reduced gradually. It indicates that the reasoning mechanism is gradually deepening the understanding of the question.**





## Acknowledgements

This work is supported by the Beijing Natural Science Foundation (Z190020). This work is also supported by Baidu and CASICT Joint Project. We would like to thank the anonymous reviewers for their valuable feedback.

## References

* J. Andreas, M. Rohrbach, T. Darrell, and D. Klein (2016-05)Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 39-48. Cited by: SS1.
* Q. Cao, X. Liang, B. Li, G. Li, and L. Lin (2018-07)Visual question reasoning on general dependency tree. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7249-7257. External Links: Document, ISSN 1095-9209 Cited by: SS1.
* R. Cao, L. Chen, Z. Chen, Y. Zhao, S. Zhu, and K. Yu (2021-07)Lgesql: line graph enhanced text-to-sql model with mixed local and non-local relations. arXiv preprint arXiv:2106.01093. Cited by: SS1.
* R. P. Carver and J. V. Hoffman (1981)The effect of practice through repeated reading on gain in reading ability using a computer-based instructional system. Reading Research Quarterly, pp. 374-390. Cited by: SS1.
* A. Chadha, G. Arora, and N. Kaloty (2020-07)ipercive: applying common-sense reasoning to multi-modal dense video captioning and video question answering. arXiv preprint arXiv:2011.07735. Cited by: SS1.
* A. Chang and S. Millett (2013-07)Improving reading rates and comprehension through timed repeated reading. Cited by: SS1.
* P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord (2018-07)Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: SS1.
* R. Das, S. Dhuliawala, M. Zaheer, and A. McCallum (2019-07)Multi-step retriever-reader interaction for scalable open-domain question answering. arXiv preprint arXiv:1905.05733. Cited by: SS1.
* C. Fan, X. Zhang, S. Zhang, W. Wang, C. Zhang, and H. Huang (2019-07)Heterogeneous memory enhanced multimodal attention model for video question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1999-2007. Cited by: SS1.
* J. Gao, R. Ge, K. Chen, and R. Nevatia (2018-07)Motion-appearance co-memory networks for video question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6576-6585. Cited by: SS1.
* N. Garcia and Y. Nakashima (2020)Knowledge-based video question answering with unsupervised scene descriptions. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII '16, pp. 581-598. Cited by: SS1.
* N. Garcia, M. Otani, C. Chu, and Y. Nakashima (2020-07)Knowit vqa: answering knowledge-based questions about videos. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, pp. 10826-10834. Cited by: SS1.
* G. Gorsuch and E. Taguchi (2008-07)Repeated reading for developing reading fluency and reading comprehension: The case of efl learners in vietnam. System36 (2), pp. 253-278. Cited by: SS1.
* J. Guo, Z. Zhan, Y. Gao, Y. Xiao, J. Lou, T. Liu, and D. Zhang (2019-07)Towards complex text-to-sql in cross-domain database with intermediate representation. arXiv preprint arXiv:1905.08205. Cited by: SS1.
* R. Hu, J. Andreas, M. Rohrbach, T. Darrell, and K. Saenko (2017-07)Learning to reason: end-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 804-813. Cited by: SS1.
* Y. Jang, Y. Song, Y. Yu, Y. Kim, and G. Kim (2017-07)Tgif-qa: toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2758-2766. Cited by: SS1.
* J. Jiang, Z. Chen, H. Lin, X. Zhao, and Y. Gao (2020-07)Divide and conquer: question-guided spatio-temporal contextual attention for video question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, pp. 11101-11108. Cited by: SS1.
* J. Johnson, A. Karpathy, and L. Fei-Fei (2016)Densecap: fully convolutional localization networks for dense captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4565-4574. Cited by: SS1.
* H. Kim, Z. Tang, and M. Bansal (2020-07)Dense-caption matching and frame-selection gating for temporal localization in videoqa. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4812-4822. Cited by: SS1.
* J. Kim, K. On, W. Lim, J. Kim, J. Ha, and B. Zhang (2016-07)Hadamard product for low-rank bilinear pooling. arXiv preprint arXiv:1610.04325. Cited by: SS1.
* Seonhoom Kim, S. Jeong, E. Kim, I. Kang, and N. Kwak (2020-07)Self-supervised pre-training and contrastive representation learning for multiple-choice video qa. arXiv preprint arXiv:2009.08043.



Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. 2020. Hierarchical conditional relation networks for video question answering. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 9972-9981.
* Lei et al. (2021) Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. 2021. Less is more: Clipbert for video-and-language learning via sparse sampling. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 7331-7341.
* Lei et al. (2018) Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. Tvqa: Localized, compositional video question answering. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1369-1379.
* Lei et al. (2020) Jie Lei, Licheng Yu, Tamara Berg, and Mohit Bansal. 2020. Tvqa+: Spatio-temporal grounding for video question answering. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8211-8225.
* Li et al. (2020) Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. 2020. Hero: Hierarchical encoder for video+ language omni-representation pretraining. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2046-2065.
* Li and Wang (2020) Tianhao Li and Limin Wang. 2020. Learning spatiotemporal features via video and text pair discrimination. _arXiv preprint arXiv:2001.05691_.
* Li et al. (2019) Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, and Chuang Gan. 2019. Beyond rnns: Positional self-attention with co-attention for video question answering. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 8658-8665.
* Lin et al. (2018) Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1736-1745.
* Liu et al. (2020a) Jiangming Liu, Matt Gardner, Shay B Cohen, and Mirella Lapata. 2020a. Multi-step inference for reasoning over paragraphs. _arXiv preprint arXiv:2004.02995_.
* Liu et al. (2020b) Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, and Jingjing Liu. 2020b. Violin: A large-scale dataset for video-and-language inference. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 10900-10910.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. _arXiv preprint arXiv:2103.14030_.
* Nuamah (2021) Kwabena Nuamah. 2021. Deep algorithmic question answering: Towards a compositionally hybrid ai for algorithmic reasoning. _arXiv preprint arXiv:2109.08006_.
* Oguz et al. (2021) Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliey, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2021. Unifeq-qa: Unified representations of structured and unstructured knowledge for open-domain question answering. _arXiv preprint arXiv:2012.14610_, 54:57-60.
* Ren et al. (2016) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2016. Faster r-cnn: towards real-time object detection with region proposal networks. _IEEE transactions on pattern analysis and machine intelligence_, 39(6):1137-1149.
* Rennie et al. (2017) Steven J Rennie, Etienne Marcheret, Youssef Morueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 7008-7024.
* Schuster et al. (2015) Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D. Manning. 2015. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In _Workshop on Vision and Language (VL15)_, Lisbon, Portugal. Association for Computational Linguistics.
* Seo et al. (2021) Ahjeong Seo, Gi-Cheon Kang, Joonhan Park, and Byoung-Tak Zhang. 2021. Attend what you need: Motion-appearance synergistic networks for video question answering. _arXiv preprint arXiv:2106.10446_.
* Song et al. (2021)Dandan Song, Siyi Ma, Zhanchen Sun, Sicheng Yang, and Lejian Liao. 2021. Kvl-bert: Knowledge enhanced visual-and-linguistic bert for visual commonsense reasoning. _Knowledge-Based Systems_, 230:107408.
* Sun et al. (2019) Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learning. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 7464-7473.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Velickovic et al. (2017) Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. _arXiv preprint arXiv:1710.10903_ 

Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2019. Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. _arXiv preprint arXiv:1911.04942_.
* Xiao et al. (2021) Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, and Tat-Seng Chua. 2021. Video as conditional graph hierarchy for multi-granular question answering. _arXiv preprint arXiv:2112.06197_.
* Xu et al. (2017a) Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017a. Video question answering via gradually refined attention over appearance and motion. In _Proceedings of the 25th ACM international conference on Multimedia_, pages 1645-1653.
* Xu et al. (2017b) Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017b. Video question answering via gradually refined attention over appearance and motion. In _Proceedings of the 25th ACM international conference on Multimedia_, pages 1645-1653.
* Xu et al. (2016) Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msrvtt: A large video description dataset for bridging video and language. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 5288-5296.
* Yang et al. (2021) Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2021. Just ask: Learning to answer questions from millions of narrated videos. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 1686-1697.
* Yu et al. (2018a) Youngjae Yu, Jongseok Kim, and Gunhee Kim. 2018a. A joint sequence fusion model for video question answering and retrieval. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pages 471-487.
* Yu et al. (2018b)Youngjae Yu, Jongsoek Kim, and Gunhee Kim. 2018b. A joint sequence fusion model for video question answering and retrieval. In _Proceedings of the European Conference on Computer Vision (ECCV)_, pages 471-487.
* Yu et al. (2016) Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. 2016. Video captioning and retrieval models with semantic attention. _arXiv preprint arXiv:1610.02947_, 6(7).
* Yu et al. (2017) Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. 2017. End-to-end concept word detection for video captioning, retrieval, and question answering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 3165-3173.
* Zellers et al. (2021) Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. 2021. Merlot: Multimodal neural script knowledge models. _arXiv preprint arXiv:2106.02636_.
* Zhou et al. (2018) Mantong Zhou, Minlie Huang, and Xiaoyan Zhu. 2018. An interpretable reasoning network for multi-relation question answering. _arXiv preprint arXiv:1801.04726_.
* Zhu and Yang (2020) Linchao Zhu and Yi Yang. 2020. Actbert: Learning global-local video-text representations. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 8746-8755.