SlowGuess/NAACL_2022_OCR huggingface dataset:
Jianguo Mao $^{1,2*}$ , Wenbin Jiang $^{3}$ , Xiangdong Wang $^{1}$ , Zhifan Feng $^{3}$ , Yajuan Lyu $^{3}$ , Hong Liu $^{1}$ , Yong Zhu $^{3}$
$^{1}$ Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China  $^{2}$  University of Chinese Academy of Sciences, Beijing, China  $^{3}$  Baidu Inc., Beijing, China {maojianguo20s, xdwang, hliu}@ict.ac.cn {jiangwenbin, fengzhifan, lvyajuan, zhuyong}@baidu.com
Existing video question answering (video QA) models lack the capacity for deep video understanding and flexible multistep reasoning. We propose for video QA a novel model which performs dynamic multistep reasoning between questions and videos. It creates video semantic representation based on the video scene graph composed of semantic elements of the video and semantic relations among these elements. Then, it performs multistep reasoning for better answer decision between the representations of the question and the video, and dynamically integrate the reasoning results. Experiments show the significant advantage of the proposed model against previous methods in accuracy and interpretability. Against the existing state-of-the-art model, the proposed model dramatically improves more than  $4\% / 3.1\% / 2\%$  on the three widely used video QA datasets, MSRVTTQA, MSRVTT multi-choice, and TGIF-QA, and displays better interpretability by backtracing along with the attention mechanisms to the video scene graphs.
Video question answering (video QA) aims to answer questions according to the given videos. It is usually defined as a classification task, where the most appropriate answer is chosen from a candidate list for the given question and video. Existing methods for video QA conduct direct answering selection based on the multimodal encoding of questions and videos (Jang et al., 2017; Lei et al., 2018, 2020). In recent years, researchers have proposed many optimization strategies for better performance in video question answering, e.g., designing delicate encoding mechanisms (Kim et al., 2020a; Nuamah, 2021; Gao et al., 2018; Li et al., 2019; Fan et al., 2019; Le et al., 2020; Jiang et al., 2020; Kim et al., 2020b; Seo et al., 2021), introducing video scene
Complexity: Easy Reasoning Process:  $\succ$  Q: Identify answer type: Object (what)  $\succ$  Q: Identify action: throwing around  $\succ$  Q: Identify person: people  $\succ$  11-14: Identify scene: a group of people  $\succ$  11-14: Identify scene: people playing ball  $\succ$  Choose the answer: ball
Complexity: Medium   
Reasoning Process:   
 $\succsim$  Q: Identify answer type: Object (what)   
 $\succsim$  Q:Identify person: a man (in a old footage)   
 $\succsim$  Q:Identify condition: also of   
 $\succsim$  Exclude old footage and man   
 $\succsim$  11-12: Identify scene: beach with a lot of people   
 $\succsim$  13:Identify scene: sea and beach   
 $\succsim$  Choose the answer: beach
Figure 1: Error cases of the previous state-of-the-art method (Lei et al., 2021) that needed multistep reasoning. We demonstrate the multistep reasoning process based on question and video(left).
Complexity: Hard   
Reasoning Process:   
 $\Rightarrow$  Q: Identify answer type: Object (what)   
 $\Rightarrow$  Q: Identify person: a teenager   
 $\Rightarrow$  Q: Identify action: throws basketball   
 $\Rightarrow$  Q: Identify person: his friends   
 $\Rightarrow$  Q: Identify action: watch   
 $\Rightarrow$  Q: Identify location: by a gray house   
 $\Rightarrow$  Q: Identify condition: after throws miss the hoop   
 $\Rightarrow$  11-13: Identify scene: a teenager playing basketball   
 $\Rightarrow$  14: Identify scene: basketball on the ground   
 $\Rightarrow$  Choose the answer: ground
graphs (Garcia and Nakashima, 2020), adopting video pre-trained language models (Li et al., 2020; Zellers et al., 2021; Li and Wang, 2020; Lei et al., 2021; Sun et al., 2019), and leveraging external knowledge or resources (Chadha et al., 2020; Garcia et al., 2020; Liu et al., 2020b; Song et al., 2021; Garcia and Nakashima, 2020). Compared with conventional monomodal question answering tasks such as text QA (Oguz et al., 2021; Zhou et al., 2018; Lin et al., 2018) and table QA (Cao et al., 2021; Wang et al., 2019). Video QA is more difficult due to the need for crossmodal understanding and reasoning of the video and the question. Existing methods are mainly concerned with how to encode the crossmodal features better. When faced with a complex question, they usually lack the abilities of deep understanding and complex reasoning.
Similar to the situations in text QA and table QA, it is also necessary for video QA to deeply understand the semantics of the context, namely, the video, and the reasoning on the context and the question. Statistical analysis on several datasets
*This is joint work of CASICT and Baidu.
Human Language Technologies, pages 3894 - 3904
(a). Existing methods
(b).ClipBERT (2021)
(c). Ours
Figure 2: Comparison of our model architecture with (a). popular video-and-language learning paradigm and (b).ClipBERT. In contrast to most previous methods that adopt simple and implicit reasoning mechanisms, Our method uses video scene graphs to represent the video semantic information and adopt a dynamic multistep reasoning mechanism.
reveals that a significant percentage of failed cases of the state-of-the-art (SOTA) model ClipBERT (Lei et al., 2021) is caused by the lack of deep understanding and reasoning. Figure 1 shows some error cases of the SOTA model on MSRVTT-QA. We randomly select one hundred error cases and find that about  $24\%$  of the error cases need multiple steps of reasoning on the question and the video, and about  $15\%$  of the error cases need a deep understanding of the semantic of the video. These cases could be solved by a model emphasizing deep understanding and reasoning. The questions in these three datasets are relatively simple according to the building procedures (Xu et al., 2017a; Yu et al., 2018a; Jang et al., 2017). It will be more valuable for video QA to enable deep understanding and reasoning on the question and the video in realistic application scenarios.
In this work, we propose a novel dynamic reasoning model for video QA to overcome the weakness of previous models in deep understanding and reasoning. It first creates the video semantic representation from the video scene graph, which is composed of the semantic elements of the video and the semantic relations between these elements. Then it conducts multistep reasoning of the question based on the video semantic representation to generate a series of video-aware question representations. Finally, it generates the most appropriate question representation for the final answering decision by dynamically integrating these video-aware question representations according to the reasoning
complicity prediction. Figure 2 shows the overall architecture of the proposed model and the comparison with previous methods. It simulates the reasoning procedure of human beings, while previous methods follow the pipeline of multimodal encoding and answering selection. In addition, the proposed model enables the decomposition of question understanding and video understanding, thus leading to more opportunities for future optimization. On the one hand, more external knowledge resources and better reasoning architectures can be introduced for better video QA performance. On the other hand, it can act as a unified framework for different QA tasks such as video QA, table QA, and text QA.
We verify the proposed model on three well-known datasets, MSRVTT-QA (Xu et al., 2017a), MSRVTT multi-choice (Yu et al., 2018a), and TGIF-QA (Jang et al., 2017), widely used in recent video QA works (Jang et al., 2017; Gao et al., 2018; Li et al., 2019; Fan et al., 2019; Le et al., 2020; Zhu and Yang, 2020; Lei et al., 2021; Seo et al., 2021). Experiments show that our model achieves dramatic improvement over the powerful state-of-the-art model ClipBERT (Lei et al., 2021), with an average accuracy increment of more than 3 percentage points. Ablation studies show that the dynamic reasoning strategy significantly outperforms previous implicit simple reasoning strategies. The video semantic representation based on the video scene graph makes the dynamic reasoning strategy work better. Backtracing along with the at
Figure 3: Overview of our model architecture. It contains two parts: 3.2. Video Representation Learning (right) and 3.3. Dynamic Multistep Reasoning (left)
tention mechanism to the video scene graph clearly shows the semantic elements that answer decision relies on each reasoning step, thus giving better interpretability than most of the previous methods.
Given the question  $q$ , video question answering requires choose the correct answer  $\hat{a}$  from the candidates set  $\Omega_{a}$  according to the video content  $V$ .
$$
\hat {a} = \underset {a \in \Omega_ {a}} {\operatorname {a r g m a x}} p (a | q, V; \theta) \tag {1}
$$
As Figure 2 (a) shows, most existing works utilize offline (stop gradient) extracted dense video features and text features. As Figure 2 (b) shows, ClipBERT (Lei et al., 2021) achieves the state-of-the-art by using sparsely sampled clips and raw text for end-to-end training, yet suffer from two main drawbacks: (i) Lack of a deep understanding of the video content and reasoning on the question based on the video. (ii) Strong coupling between video and question modeling process, which needs additional image-text pair data for pre-training to enhance the ability of the text encoder to model multimodal features and leads to poor scalability and repeated computation. As shown in Figure 2 (c), we propose a simple but effective architecture to solve the weakness of previous works. It first de
couples the question understanding module and the video understanding module. Then, we introduce the video scene graph to get a better representations of the video semantic information, which can enhance the understanding of the video content and its graph structure is also better for reasoning. At last, we design a dynamic multistep reasoning mechanism to iteratively deepen the understanding of the question according to the video content.
Figure 3 gives an overview of the model architecture. For the visual representation, we use both video scene graphs and image features. On the one hand, we construct a video scene graph to represent semantic information in a structural form. On the other hand, we use Swin Transformer (Liu et al., 2021) to extract image features to make up for the missing information of the scene graph. The reasoning module (Reasoner) will iteratively updates the understanding of the question according to the video content. The Evaluator will decide the number of reasoning steps according to the complexity of the question. The Integrator will integrate all intermediate reasoning results to get the final reasoning results. The answer decision
module chooses an answer according to the final comprehensive understanding of the question.
3.2 Video Representation Learning
We chose a structured video scene graph to describe video semantics which is better for reasoning. We also extract image features by Swin Transformer (Liu et al., 2021) to make up for the missing information in the scene graph. The video scene graph and the image features constitute the Video Representation Memory shown in Figure 3, a memory for the Reader module to access.
3.2.1 Video Scene Graph
The video scene graph is the basis for conducting dynamic reasoning, it is a graph-based semantic representation of video content, representing the objects in the video, their attributes, and their relationships in a structured form. Unlike the image scene graph commonly used in visual question answering, our video scene graph is semantically richer and contains spatio-temporal information of the video. Specifically, We first use an image captioning model to generate captions for each clip. Then we use the scene parser (Schuster et al., 2015) to convert each caption sentence into a semantic sub-graph and integrate the same nodes of each sub-graph to obtain a video scene graph. Compared to caption sentences, the video scene graph represents the video-level semantic information in a structured form, better modeling the visual semantic information.
Graph Representation Learning We first obtain the embeddings of nodes  $\mathbf{n} = \{n_1, n_2, \dots, n_m\}$  and edges in the video scene graph via a parameter-sharing language encoder. Then, we use graph attention neural network (Velicković et al., 2017) iteratively to update the representation of the scene graph. Each node updates its representation based on the correlation with its neighbor nodes.
$$
\mathbf {n} _ {i} ^ {\prime} = a _ {i i} \mathbf {W n} _ {i} + \sum_ {j \in \mathcal {N} _ {i}} a _ {i j} \mathbf {W n} _ {j} \tag {2}
$$
where  $\mathbf{W}$  is a weight matrix,  $a_{ij}$  is the attention weight of node  $\mathbf{n}_i$  and  $\mathbf{n}_j$ , and  $\mathcal{N}_i$  is the neighbors of the node  $\mathbf{n}_i$  in the graph. In our experiments, we use standard graph attention neural network setting, applying the LeakyReLU nonlinearity (with negative input slope  $\alpha = 0.2$ ). At the same time, the edges have explicit meanings of relations between nodes, so we also consider edges features  $\mathbf{e}_{ij}$  when
calculating attention weight. The attention weight  $a_{ij}$  are computed as
$$
a _ {i j} = \frac {\exp (\text {L e a k y R e L U} (\mathrm {A} (\mathbf {n} _ {i} , \mathbf {n} _ {j} , \mathbf {e} _ {i j})))}{\sum_ {k \in \mathcal {N} _ {i}} \exp (\text {L e a k y R e L U} (\mathrm {A} (\mathbf {n} _ {i} , \mathbf {n} _ {k} , \mathbf {e} _ {i k})))} \tag {3}
$$
$$
\mathrm {A} (\mathbf {n} _ {i}, \mathbf {n} _ {j}, \mathbf {e} _ {i j}) = \mathbf {W} _ {\mathbf {a}} ^ {\top} [ \mathbf {W} \mathbf {n} _ {i} | | \mathbf {W} \mathbf {n} _ {j} | | \mathbf {W} _ {e} \mathbf {e} _ {i j} ] \tag {4}
$$
where  $\mathbf{\tau}^{\top}$  represents transposition,  $||$  is the concatenation operation and the  $\mathbf{W}_{\mathbf{a}}$  and  $\mathbf{W}_{\mathbf{e}}$  are weight matrices.
3.2.2 Image Features
We extract image features to make up for the missing information in the scene graph. First, we use Lei et al.'s sparse sampling method to sparsely and randomly sample  $N_{train}$  clips  $\{c_i\}_{i=1}^{N_{train}}$  from video.  $N_{train}$  is typically much smaller than the entire video length  $N$ . This sampling method can reduce the computation cost and obtain better performance than dense sampling. For inference, we uniformly sample  $N_{test}$  clips of the same duration. Swin Transformer (Liu et al., 2021) is one of the mainstream visual backbone networks. It alleviates the problem of large variations in the scale of visual entities and the high resolution of pixels in images. We use it as a vision encoder  $\mathcal{E}_v$  to extract clip features  $\{F_i\}_{i=1}^{N_{train}}, F_i = \mathcal{E}_v(c_i) \in \mathbb{R}^{w \times w \times d}$ , where  $w$  is the window size and  $d$  is the feature dimension.
3.3 Dynamic Multistep Reasoning
Humans will deepen their understanding of a complex question through repeated reading the context information. The more complex the question, the more repetitions are required (Chang and Millett, 2013; Gorsuch and Taguchi, 2008; Carver and Hoffman, 1981). At each step of reading, people will focus on different parts of the context information. Inspired by it, we designed the dynamic multistep reasoning mechanism. It will iteratively update the understanding of the question based on the video representations. We first extract the question representation by language model RoBERTa (Liu et al., 2019). Specifically, we concatenate the question text with a special token [CLS] as the input and take the [CLS]'s hidden state  $R^0$  as the representation of the question. At the first reasoning step, we select  $R^0$  as the input. Then, we get question-related information from Video Representation Memory
through the Reader, which is an attention mechanism. Then, we use this retrieved video information to update the understanding of the question and get the first reasoning step result  $R_{1}$  through the Up-dater. At the next reasoning step, we select  $R_{1}$  as the input. After  $S$  steps of reasoning, we obtain all results  $\mathbf{R} = \{R^0,R^1,R^2,\dots,R^S\}$ .
$$
R ^ {s} = \operatorname {U p d a t e r} \left(R ^ {s - 1}, V ^ {s - 1}\right) \tag {5}
$$
Updater consists of two linear transformations with a ReLU activation in between.
$$
\operatorname {U p d a t e r} (Q, K) = Q + \operatorname {R e L U} \left(\mathbf {W} _ {\mathbf {1}} K + b _ {1}\right) \mathbf {W} _ {\mathbf {2}} + b _ {2} \tag {6}
$$
where  $\mathbf{W}_1$  and  $\mathbf{W}_2$  are weight matrices and  $b_{1}$  and  $b_{2}$  are biases.  $V_{s - 1}$  represents question-related video information.
$$
V ^ {s - 1} = \operatorname {R e a d e r} \left(R ^ {s - 1}, \mathbf {V i d}\right) \tag {7}
$$
where Vid consists of the node features  $\{n_1, n_2, \dots, n_m\}$  of the video scene graph and image features  $\{F_1, F_2, \dots, F_{N_{train}}\}$ . We use Scaled Dot-Product Attention (Vaswani et al., 2017) as the Reader. The input consists of query  $Q$ , and context  $K$  of dimension  $d_k$ .
$$
\operatorname {R e a d e r} (Q, K) = \operatorname {s o f t m a x} \left(\frac {Q K ^ {\top}}{\sqrt {d _ {k}}}\right) K \tag {8}
$$
The word dynamic has two meanings: (i). dynamically decide the number of reasoning steps according to the question's complexity. (3.3 Evaluator). (ii). dynamically integrate the results of all reasoning steps as the final result. (3.3 Integrator).
Evaluator When humans begin faced with different complexity questions, they will dynamically adjust the number of times to read relevant information (Chang and Millett, 2013; Gorsuch and Taguchi, 2008; Carver and Hoffman, 1981). We propose the first dynamic reasoning strategy by imitating the human reading and understanding mechanisms. It will decide the number of reasoning steps according to the complexity of the question. Specifically, we perform a nonlinear transformation with GumbelSoftmax as activation function on the question representation  $R^0$ , and output an S-dimensional vector  $\mathrm{D}(R_0) \in \mathbb{R}^{1 \times S}$  to represent the distribution probability of the number of reasoning steps from 1 to S.
$$
\mathrm {D} \left(R _ {0}\right) = \operatorname {G u m b e l S o f t m a x} \left(\mathbf {W} _ {\mathbf {d}} R ^ {0} + b\right) \quad (9)
$$
Where  $\mathbf{W}_{\mathbf{d}}$  is a weight matrix, and  $b$  is a bias. We choose the one with the greatest probability as the number of the reasoning steps  $S$ .
Integrator In the reasoning process, the Reader pays attention to the different parts of the video content at each step to gradually deepen the question's understanding. Therefore, we think the intermediate reasoning results are also helpful in choosing an answer. After  $S$  steps of reasoning, we select all intermediate reasoning results as input and perform a nonlinear transformation with softmax function to calculate the distribution of the weight of  $\mathbf{R}$  and get the weighted sum as the final reasoning results  $R^{f}$ .
$$
R ^ {f} = \operatorname {s o f t m a x} \left(\mathbf {W} _ {\mathbf {f}} \mathbf {R} + \mathbf {b}\right) \mathbf {R} \tag {10}
$$
where  $\mathbf{W}_{\mathrm{f}}$  is a weight matrix and  $b$  is a bias.
3.4 Answer Decision
Given the final reasoning results  $R^f$ , we use two fully-connected layers as a classifier to obtain the logits  $l_f$  for the answer options. Then we use a softmax function to obtain the probability distribution of each answer option and apply cross-entropy loss as our model loss  $\mathcal{L}$ .
$$
l _ {f} = \operatorname {c l a s s i f i e r} \left(R ^ {f}\right) \tag {11}
$$
$$
\hat {y} = \operatorname {s o f t m a x} \left(l _ {f}\right), \mathcal {L} = - \sum_ {i = 1} ^ {M} y _ {i} \log \hat {y _ {i}} \tag {12}
$$
Video QA requires fine-grained modeling of multimodal features. We have witnessed many efforts devoted to video understanding for video QA. Some methods use visual techniques such as object detection (Ren et al., 2016) and image captioning (Johnson et al., 2016; Rennie et al., 2017) to extract additional visual information (Kim et al., 2020a). In recent years, visual pre-training based on large-scale data has become a popular method to improve video applications including video QA (Li et al., 2020; Zellers et al., 2021; Li and Wang, 2020; Sun et al., 2019). In addition, several advanced techniques such as contrastive self-supervised learning (Kim et al., 2020b) and symbolized video scene graph (Garcia and Nakashima, 2020) are proposed to improve the performance of video understanding
for video QA. We extract video semantic representations based on both the visual pre-training model (Liu et al., 2021) and the video scene graphs. Experiments show the amazing complementarity of the two kinds of information.
Video QA also requires flexible reasoning for complicated questions and videos. Although the reasoning capability is rarely emphasized in previous work for video QA, it is broadly investigated in other QA tasks (Clark et al., 2018). For example, text QA resorts to iterative update of the question and the context (Das et al., 2019; Liu et al., 2020a), table QA generates a structural query such as SQL which is then executed on the tabular data (Guo et al., 2019), and visual QA builds a specific module network according to the question and runs it on the image (Andreas et al., 2016; Cao et al., 2018; Hu et al., 2017). The overall architecture of the proposed method is similar to the iterative reasoning strategies for text QA but with significant innovation. Our model can dynamically determine the best integration strategy for the intermediate reasoning results according to the given question and video, leading to better interpretability and much better performance.
In this section, we validate our method on three mainstream video QA datasets. We conduct comparison experiments with previous works and perform ablation experiments to analyze the critical improvement in our proposed method. We use standard train/val/test splits for all datasets and use accuracy to measure the performance. All experimental results are the mean and standard deviation of ten replicate experiments.
MSRVTT-QA MSR-VTT (Xu et al., 2016) is a large video description dataset. It provides 10k web video clips with 41.1 hours and 200k clip-sentence pairs in total. MSRVTT-QA (Xu et al., 2017a) is created based on clip-sentence pairs in MSR-VTT automatically through a program. It contains 243k open-ended questions with 1500 answers.
MSRVTT-MC MSRVTT-MC (multiple-choice) (Yu et al., 2018a) is a dataset for video-text matching tasks built on MSR-VTT with videos are used as queries, captions as answers. Each video contains five captions. Only one is correct.
TGIF-QA TGIF-QA (Jang et al., 2017) dataset contains 165K QA pairs for the animated GIFs from the TGIF dataset. We experiment on 3 TGIF-QA tasks: Repeating Action and State Transition for multiple-choice QA and Frame QA for open-ended QA. We follow most previous works and ClipBERT's (Lei et al., 2021) approach to leave the Count task as future work as it requires directly modeling full-length videos.
5.2.1 Comparison with existing approaches
As shown in Table 1, our method reaches the new state-of-the-art and achieves  $41.6\% /91.4\% /84.6\% ,90.1\% ,62.5\%$  accuracy on MSRVTT-QA/MSRVTT-MC (multi-choice) /TGIF-QA (Action, Transition, FrameQA), with  $4.2\% /3.2\% /1.8\% ,2.3\% ,2.2\%$  improvement over the previous state-of-the-art method ClipBERT (Lei et al., 2021). The Pre-training data column represents that the model was pre-trained with additional data. The results show that our method achieves the best performance on all three video question answering datasets without using additional data.
5.2.2 Ablations Analysis