SlowGuess/ACL_2020_OCR huggingface dataset:
Jie Lei Licheng Yu Tamara L. Berg Mohit Bansal
{jielei, licheng, tlberg, mbansal}@cs.unc.edu
We present the task of Spatio-Temporal Video Question Answering, which requires intelligent systems to simultaneously retrieve relevant moments and detect referenced visual concepts (people and objects) to answer natural language questions about videos. We first augment the TVQA dataset with 310.8K bounding boxes, linking depicted objects to visual concepts in questions and answers. We name this augmented version as  $\mathrm{TVQA + }$ . We then propose Spatio-Temporal Answerer with Grounded Evidence (STAGE), a unified framework that grounds evidence in both spatial and temporal domains to answer questions about videos. Comprehensive experiments and analyses demonstrate the effectiveness of our framework and how the rich annotations in our  $\mathrm{TVQA + }$  dataset can contribute to the question answering task. Moreover, by performing this joint task, our model is able to produce insightful and interpretable spatio-temporal attention visualizations. $^1$
We have witnessed great progress in recent years on image-based visual question answering (QA) tasks (Antol et al., 2015; Yu et al., 2015; Zhu et al., 2016b). One key to this success has been spatial attention (Anderson et al., 2018; Shih et al., 2016; Lu et al., 2016), where neural models learn to attend to relevant regions for predicting the correct answer. Compared to image-based QA, there has been less progress on the performance of video-based QA tasks. One possible reason is that attention techniques are hard to generalize to the temporal nature of videos. Moreover, due to the high cost of annotation, most existing video QA datasets only contain QA pairs, without providing labels for the
Question: What is Sheldon's holding when he is talking to Howard about the sword? Correct Answer: A computer.
Figure 1: Samples from TVQA+. Questions and correct answers are temporally localized to clips, and spatially localized to frame-level bounding boxes. Colors indicate corresponding box-object pairs. Subtitles are shown in dashed blocks. Wrong answers are omitted.
Question: Who is talking to Howard when he is in the kitchen upset?  
Correct Answer: Raj is talking to Howard.
key clips or regions needed to answer the question. Inspired by previous work on grounded image and video captioning (Lu et al., 2018; Zhou et al., 2019), we propose methods that explicitly localize video clips as well as spatial regions for answering videobased questions. Such methods are useful in many scenarios, such as natural language guided spatiotemporal localization, and adding explainability to video question answering, which is potentially useful for decision making and model debugging. To enable this line of research, we also collect new joint spatio-temporal annotations for an existing video QA dataset.
In the past few years, several video QA datasets have been proposed, e.g., MovieFIB (Maharaj et al., 2017), MovieQA (Tapaswi et al., 2016), TGIF-QA (Jang et al., 2017), PororoQA (Kim et al., 2017), MarioQA (Mun et al., 2017), and TVQA (Lei et al., 2018). TVQA is one of the largest video QA datasets, providing a large video QA dataset built on top of 6 famous TV series. Be
$^{1}$ Dataset and code are publicly available: http://tvqa.cs.unc.edu, https://github.com/jayleicn/TVQAplus
8211
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8211-8225 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
cause TVQA was collected on television shows, it is built on natural video content with rich dynamics and complex social interactions, where question-answer pairs are written by people observing both videos and their accompanying dialogues, encouraging the questions to require both vision and language understanding to answer. Movie (Tapaswi et al., 2016; Maharaj et al., 2017) and television show (Lei et al., 2018) videos come with the limitation of being scripted and edited, but they are still more realistic than cartoon/animation (Kim et al., 2017) and game (Mun et al., 2017) videos, and they also come with richer, real-world-inspired inter-human interactions and span across diverse domains (e.g., medical, crime, sitcom, etc.), making them a useful testbed to study complex video understanding by machine learning models.
One key property of TVQA is that it provides temporal annotations denoting which parts of a video clip are necessary for answering a proposed question. However, none of the existing video QA datasets (including TVQA) provide spatial annotation for the answers. Actually, grounding spatial regions correctly could be as important as grounding temporal moments for answering a given question. For example, in Fig. 1, to answer the question of "What is Sheldon holding when he is talking to Howard about the sword?", we need to localize the moment when "he is talking to Howard about the sword?", as well as look at the region of "What is Sheldon holding".
Hence, in this paper, we first augment a subset of the TVQA dataset with grounded bounding boxes, resulting in a spatio-temporally grounded video QA dataset,  $\mathrm{TVQA + }$ . It consists of 29.4K multiple-choice questions grounded in both the temporal and the spatial domains. To collect spatial groundings, we start by identifying a set of visual concept words, i.e., objects and people, mentioned in the question or correct answer. Next, we associate the referenced concepts with object regions in individual frames, if there are any, by annotating bounding boxes for each referred concept (see examples in Fig. 1). Our  $\mathrm{TVQA + }$  dataset has a total of 310.8K bounding boxes linked with referred objects and people, spanning across 2.5K categories (more details in Sec. 3).
With such richly annotated data, we then propose the task of spatio-temporal video question answering, which requires intelligent systems to localize relevant moments, detect referred objects
and people, and answer questions. We further design several metrics to evaluate the performance of the proposed task, including QA accuracy, object grounding precision, temporal localization accuracy, and a joint temporal localization and QA accuracy. To address spatio-temporal video question answering, we propose a novel end-to-end trainable model, Spatio-Temporal Answerer with Grounded Evidence (STAGE), which effectively combines moment localization, object grounding, and question answering in a unified framework. We find that the QA performance benefits from both temporal moment and spatial region supervision. Additionally, we provide visualization of temporal and spatial localization, which is helpful for understanding what our model has learned. Comprehensive ablation studies demonstrate how each of our annotations and model components helps to improve the performance of the tasks.
- We collect TVQA+, a large-scale spatiotemporal video question answering dataset, which augments the original TVQA dataset with frame-level bounding box annotations. To our knowledge, this is the first dataset that combines moment localization, object grounding, and question answering.
- We design a novel video question answering framework, Spatio-Temporal Answerer with Grounded Evidence (STAGE), to jointly localize moments, ground objects, and answer questions. By performing all three sub-tasks together, our model achieves significant performance gains over the baselines, as well as presents insightful, interpretable visualizations.
Question Answering In recent years, multiple question answering datasets and tasks have been proposed to facilitate research towards this goal, in both vision and language communities, in the form of visual question answering (Antol et al., 2015; Yu et al., 2015; Jang et al., 2017) and textual question answering (Rajpurkar et al., 2016; Weston et al., 2016), respectively. Video question answering (Lei et al., 2018; Tapaswi et al., 2016; Kim et al., 2017) with naturally occurring subtitles are particularly interesting, as it combines both visual and textual information for question answering. Different from
8212
Dataset	Origin	Task	#Clips/#QAs (#Sentences)	#Boxes	Temporal Annotation
MovieFIB (Maharaj et al., 2017)	Movie	QA	118.5K/349K	-	X
MovieQA (Tapaswi et al., 2016)	Movie	QA	6.8K/6.5K	-	✓
TGIF-QA (Jang et al., 2017)	Tumblr	QA	71.7K/165.2K	-	X
PororoQA (Kim et al., 2017)	Cartoon	QA	16.1K/8.9K	-	X
DiDeMo (Hendricks et al., 2017)	Flickr	TL	10.5K/40.5K	-	✓
Charades-STA (Gao et al., 2017)	Home	TL	- /19.5K	-	✓
TVQA (Lei et al., 2018)	TV Show	QA/TL	21.8K/152.5K	-	✓
ANet-Entities (Zhou et al., 2019)	Youtube	CAP/TL/SL	15K/52K	158K	✓
TVQA+	TV Show	QA/TL/SL	4.2K/29.4K	310.8K	✓
existing video QA tasks, where a system is only required to predict an answer, we propose a novel task that additionally grounds the answer in both spatial and temporal domains.
Language-Guided Retrieval Grounding language in images/videos is an interesting problem that requires jointly understanding both text and visual modalities. Earlier works (Kazemzadeh et al., 2014; Yu et al., 2017, 2018b; Rohrbach et al., 2016) focused on identifying the referred object in an image. Recently, there has been a growing interest in moment retrieval tasks (Hendricks et al., 2017, 2018; Gao et al., 2017), where the goal is to localize a short clip from a long video via a natural language query. Our work integrates the goals of both tasks, requiring a system to ground the referred moments and objects simultaneously.
Temporal and Spatial Attention Attention has shown great success on many vision and language tasks, such as image captioning (Anderson et al., 2018; Xu et al., 2015), visual question answering (Anderson et al., 2018; Trott et al., 2018), language grounding (Yu et al., 2018b), etc. However, sometimes the attention learned by the model itself may not agree with human expectations (Liu et al., 2016; Das et al., 2016). Recent works on grounded image captioning and video captioning (Lu et al., 2018; Zhou et al., 2019) show better performance can be achieved by explicitly supervising the attention. In this work, we use annotated frame-wise bounding box annotations to supervise both temporal and spatial attention. Experimental results demonstrate the effectiveness of supervising both domains in video QA.
Table 1: Comparison of TVQA+ with other video-language datasets. TL=Temporal Localization, SL=Spatial Localization, CAP=Captioning.
Split	#Clips/#QAs	#Annotated Images	#Boxes	#Categories
Train	3,364/23,545	118,930	249,236	2,281
Val	431/3,017	15,350	32,682	769
Test	403/2,821	14,188	28,908	680
Total	4,198/29,383	148,468	310,826	2,527
Table 2: Data Statistics for TVQA+ dataset.
In this section, we describe the TVQA+ Dataset, the first video question answering dataset with both spatial and temporal annotations. TVQA+ is built on the TVQA dataset introduced by Lei et al.. TVQA is a large-scale video QA dataset based on 6 popular TV shows, containing 152.5K multiple choice questions from 21.8K, 60-90 second long video clips. The questions in the TVQA dataset are compositional, where each question is comprised of two parts, a question part ("where was Sheldon sitting"), joined via a link word, ("before", "when", "after"), to a localization part that temporally locates when the question occurs ("he spilled the milk"). Models should answer questions using both visual information from the video, as well as language information from the naturally associated dialog (subtitles). Since the video clips on which the questions were collected are usually much longer than the context needed for answering the questions, the TVQA dataset also provides a temporal timestamp annotation indicating the minimum span (context) needed to answer each question. While the TVQA dataset provides a novel question format and temporal annotations, it lacks spatial grounding information, i.e., bounding boxes of the concepts (objects and people) mentioned in the QA pair. We hypothesize that object annotations could provide an additional useful training signal for models to learn a deeper understanding
Figure 2: Box distributions for top 60 categories in TVQA+ train set.
of visual information. Therefore, to complement the original TVQA dataset, we collect frame-wise bounding boxes for visual concepts mentioned in the questions and correct answers. Since the full TVQA dataset is very large, we start by collecting bounding box annotations for QA pairs associated with The Big Bang Theory. This subset contains 29,383 QA pairs from 4,198 clips.
Identify Visual Concepts To annotate the visual concepts in video frames, the first step is to identify them in the QA pairs. We use the Stanford CoreNLP part-of-speech tagger (Manning et al., 2014) to extract all nouns in the questions and correct answers. This gives us a total of 152,722 words from a vocabulary of 9,690 words. We manually label the non-visual nouns (e.g., "plan", "time", etc.) in the top 600 nouns, removing 165 frequent non-visual nouns from the vocabulary.
Bounding Box Annotation For the selected The Big Bang Theory videos from TVQA, we first ask Amazon Mechanical Turk workers to adjust the start and end timestamps to refine the temporal annotation, as we found the original temporal annotation were not ideally tight. We then sample one frame every two seconds from each span for spatial annotation. For each frame, we collect the bounding boxes for the visual concepts in each QA pair. We also experimented with semi-automated annotation for people with face detection (Zhang et al., 2016) and recognition model (Liu et al., 2017), but they do not work well mainly due to many partial occlusion of faces (e.g., side faces) in the frames. During annotation, we provide the original videos (with subtitles) to help the workers understand the context for the given QA pair. More annotation details (including quality check) are presented in
Figure 3: Box/image area ratios (left) and span length distributions (right) in TVQA+.
the appendix.
TVQA+ contains 29,383 QA pairs from 4,198 videos, with 148,468 images annotated with 310,826 bounding boxes. Statistics of TVQA+ are shown in Table 2. Note that we follow the same data splits as the original TVQA dataset, supporting future research on both TVQA and TVQA+. Table 1 compares TVQA+ dataset with other video-language datasets. TVQA+ is unique as it supports three tasks: question answering, temporal localization, and spatial localization.
It is also of reasonable size compared to the grounded video captioning dataset ANet-Entities (Zhou et al., 2019). On average, we obtain 2.09 boxes per image and 10.58 boxes per question. The annotated boxes cover 2,527 categories. We show the number of boxes (in log scale) for each of the top 60 categories in Fig. 2. The distribution has a long tail, e.g., the number of boxes for the most frequent category "sheldon" is around 2 orders of magnitude larger than the 60th category "glasses". We also show the distribution of bounding box area over image area ratio in Fig. 3 (left). The majority of boxes are fairly small compared to the image, which makes object grounding challenging. Fig. 3 (right) shows the distribution of localized span length. While most spans are
8214
Figure 4: Overview of the proposed STAGE framework.
less than 10 seconds, the largest spans are up to 20 seconds. The average span length is 7.2 seconds, which is short compared to the average length of the full video clips (61.49 seconds).
Our proposed method, Spatio-Temporal Answerer with Grounded Evidence (STAGE), is a unified framework for moment localization, object grounding and video QA. First, STAGE encodes the video and text (subtitle, QA) via frame-wise regional visual representations and neural language representations, respectively. The encoded video and text representations are then contextualized using a Convolutional Encoder. Second, STAGE computes attention scores from each QA word to object regions and subtitle words. Leveraging the attention scores, STAGE is able to generate QA-aware representations, as well as automatically detecting the referred objects/people. The attended QA-aware video and subtitle representation are then fused together to obtain a joint frame-wise representation. Third, taking the frame-wise representation as input, STAGE learns to predict QA relevant temporal spans, then combines the global and local (span localized) video information to answer the questions. In the following, we describe STAGE in detail.
In our tasks, the inputs are: (1) a question with 5 candidate answers; (2) a 60-second long video; (3) a set of subtitle sentences. Our goal is to predict the answer and ground it both spatially and temporally. Given the question,  $q$ , and the answers,  $\{a_k\}_{k=1}^5$ , we first formulate them as 5 hypotheses (QA-pair)
$h_k = [q, a_k]$  and predict their correctness scores based on the video and subtitle context (Onishi et al., 2016). We denote the ground-truth (GT) answer index as  $y^{ans}$  and thus the GT hypothesis as  $h_{y^{ans}}$ . We then extract video frames  $\{v_t\}_{t=1}^T$  at 0.5 FPS ( $T$  is the number of frames for each video). Subtitle sentences are then temporally aligned with the video frames. Specifically, for each frame  $v_t$ , we pair it with two neighboring sentences based on the subtitle timestamps. We choose two neighbors since this keeps most of the sentences at our current frame rate, and also avoids severe misalignment between the frames and the sentences. The set of aligned subtitle sentences are denoted as  $\{s_t\}_{t=1}^T$ . We denote the number of words in each hypothesis and subtitle as  $L_h$ ,  $L_s$ , respectively. We use  $N_o$  to denote the number of object regions in a frame, and  $d = 128$  as the hidden size.
4.2 STAGE Architecture
Input Embedding Layer For each frame  $v_{t}$ , we use Faster R-CNN (Ren et al., 2015) pre-trained on Visual Genome (Krishna et al., 2017) to detect objects and extract their regional representation as our visual features (Anderson et al., 2018). We keep the top-20 object proposals and use PCA to reduce the feature dimension from 2048 to 300, to save GPU memory and computation. We denote  $o_{t,r} \in \mathbb{R}^{300}$  as the r-th object embedding in the t-th frame. To encode the text input, we use BERT (Devlin et al., 2019), a transformer-based language model (Vaswani et al., 2017) that achieves state-of-the-art performance on various NLP tasks. Specifically, we first fine-tune the BERT-base model using the masked language model and next sentence pre
diction objectives on the subtitles and QA pairs from TVQA+ train set. Then, we fix its parameters and use it to extract 768D word-level embeddings from the second-to-last layer for the subtitles and each hypothesis. Both embeddings are projected into a 128D space using a linear layer with ReLU.
Convolutional Encoder Inspired by the recent trend of replacing recurrent networks with CNNs (Dauphin et al., 2016; Yu et al., 2018a) and Transformers (Vaswani et al., 2017; Devlin et al., 2019) for sequence modeling, we use positional encoding (PE), CNNs, and layer normalization (Ba et al., 2016) to build our basic encoding block. As shown in the bottom-right corner of Fig. 4, it is comprised of a PE layer and multiple convolutional layers, each with a residual connection (He et al., 2016) and layer normalization. We use Layernorm(ReLU(Conv(x)) + x) to denote a single Conv unit and stack  $N_{\mathrm{conv}}$  of such units as the convolutional encoder.  $x$  is the input after PE, Conv is a depthwise separable convolution (Chollet, 2017). We use two convolutional encoders at two different levels of STAGE, one with kernel size 7 to encode the raw inputs, and another with kernel size 5 to encode the fused video-text representation. For both encoders, we set  $N_{\mathrm{conv}} = 2$ .
QA-Guided Attention For each hypothesis  $h_k = [q, a_k]$ , we compute its attention scores w.r.t. the object embeddings in each frame and the words in each subtitle sentence, respectively. Given the encoded hypothesis  $H_k \in \mathbb{R}^{L_h \times d}$  for the hypothesis  $h_k$  with  $L_h$  words, and encoded visual feature  $V_t \in \mathbb{R}^{N_o \times d}$  for the frame  $v_t$  with  $N_o$  objects, we compute their matching scores  $M_{k,t} \in \mathbb{R}^{L_h \times N_o} = H_k V_t^T$ . We then apply softmax at the second dimension of  $M_{k,t}$  to get the normalized scores  $\bar{M}_{k,t}$ . Finally, we compute the QA-aware visual representation  $V_{k,t}^{att} \in \mathbb{R}^{L_h \times d} = \bar{M}_{k,t} V_t$ . Similarly, we compute QA-aware subtitle representation  $S_{k,t}^{att}$ .
Video-Text Fusion The above two QA-aware representations are then fused together as:
$$
F _ {k, t} = [ S _ {k, t} ^ {a t t}; V _ {k, t} ^ {a t t}; S _ {k, t} ^ {a t t} \odot V _ {k, t} ^ {a t t} ] W _ {F} + b _ {F},
$$
where  $\odot$  denotes hadamard product,  $W_{F}\in \mathbb{R}^{3d\times d}$  and  $b_{F}\in \mathbb{R}^{d}$  are trainable weights and bias,  $F_{k,t}\in$ $\mathbb{R}^{L_h\times d}$  is the fused video-text representation. After collecting  $F_{k,t}^{att}$  from all time steps, we get  $F_{k}^{att}\in$ $\mathbb{R}^{T\times L_h\times d}$ . We then apply another convolutional encoder with a max-pooling layer to obtain the output  $A_{k}\in \mathbb{R}^{T\times d}$ .
Span Predictor To predict temporal spans, we predict the probability of each position being the start or end of the span. Given the fused input  $A_{k} \in \mathbb{R}^{T \times d}$ , we produce start probabilities  $\mathbf{p}_{\mathbf{k}}^{\mathbf{1}} \in \mathbb{R}^{T}$  and end probabilities  $\mathbf{p}_{\mathbf{k}}^{\mathbf{2}} \in \mathbb{R}^{T}$  using two linear layers with softmax, as shown in the top-right corner of Fig. 4. Different from existing works (Seo et al., 2017; Yu et al., 2018a) that used the span predictor for text only, we use it for a joint localization of both video and text, which requires properly-aligned joint embeddings.
Span Proposal and Answer Prediction Given the max-pooled video-text representation  $A_{k}$ , we use a linear layer to further encode it. We run max-pool across all the time steps to get a global hypothesis representation  $G_{k}^{g} \in \mathbb{R}^{d}$ . With the start and end probabilities from the span predictor, we generate span proposals using dynamic programming (Seo et al., 2017). At training time, we combine the set of proposals with  $IoU \geq 0.5$  with the GT spans, as well as the GT spans to form the final proposals  $\{st_{p}, ed_{p}\}$  (Ren et al., 2015). At inference time, we take the proposals with the highest confidence scores for each hypothesis. For each proposal, we generate a local representation  $G_{k}^{l} \in \mathbb{R}^{d}$  by max-pooling  $A_{k, st_{p}:ed_{p}}$ . The local and global representations are concatenated to obtain  $G_{k} \in \mathbb{R}^{2d}$ . We then forward  $\{G_{k}\}_{k=1}^{5}$  through softmax to get the answer scores  $\mathbf{p}^{ans} \in \mathbb{R}^{5}$ . Compared with existing works (Jang et al., 2017; Zhao et al., 2017) that use soft temporal attention, we use more interpretable hard attention, extracting local features (together with global features) for question answering.
4.3 Training and Inference
In this section, we describe the objective functions used in the STAGE framework. Since our spatial and temporal annotations are collected based on the question and GT answer, we only apply the attention loss and span loss on the targets associated with the GT hypothesis (question + GT answer), i.e.,  $M_{k = y^{ans},t}$ ,  $\mathbf{p}_{k = y^{ans}}^{1}$  and  $\mathbf{p}_{k = y^{ans}}^{2}$ . For brevity, we omit the subscript  $k = y^{ans}$  in the following.
Spatial Supervision While the attention described in Sec. 4.2 can be learned in a weakly supervised end-to-end manner, we can also train it with supervision from GT boxes. We define a box as positive if it has an  $IoU \geq 0.5$  with the GT box. Consider the attention scores  $M_{t,j} \in \mathbb{R}^{N_o}$  from a concept word  $w_j$  in GT hypothesis  $h_{y^{ans}}$  to the set of proposal boxes' representations  $\{o_{t,r}\}_{r=1}^{N_o}$  at
8216
frame  $v_{t}$ . We expect the attention on positive boxes to be higher than the negative ones, and therefore use LSE (Li et al., 2017) loss for the supervision: