SlowGuess/NAACL_2021_OCR huggingface dataset:
Arka Sadhu<sup>1</sup> Kan Chen<sup>2</sup>
<sup>1</sup>University of Southern California
Ram Nevatia
$^{2}$ Facebook Inc.
{asadhu|nevatia}@usc.edu kanchen18@fb.com
Video Question Answering (VidQA) evaluation metrics have been limited to a single-word answer or selecting a phrase from a fixed set of phrases. These metrics limit the VidQA models' application scenario. In this work, we leverage semantic roles derived from video descriptions to mask out certain phrases, to introduce VidQAP which poses VidQA as a fill-in-the-phrase task. To enable evaluation of answer phrases, we compute the relative improvement of the predicted answer compared to an empty string. To reduce the influence of language-bias in VidQA datasets, we retrieve a video having a different answer for the same question. To facilitate research, we construct ActivityNet-SRL-QA and CharadesSRL-QA and benchmark them by extending three vision-language models. We perform extensive analysis and ablative studies to guide future work. Code and data are public.
Given a video, Video Question Answering (VidQA) requires a model to provide an answer to a video related question. However, existing works treat VidQA as an N-way  $(N\sim 1k)$  classification task across a fixed set of phrases. Models trained under such formulations are strictly restricted in their recall rate, generalize poorly, and have severe limitations for end-user applications.
In this work, we introduce Video Question Answering with Phrases (VidQAP) which treats VidQA as a fill-in-the-phrase task. Instead of a question, the input to VidQAP consists of a query expression with a query-token. Then, given a video, VidQAP requires replacing query-token with a sequence of generated words. To generate a query, we leverage video descriptions and assign semantic roles to each phrase in these descriptions. Replacing a particular semantic-role with a query token produces a query-answer pair. We illustrate this in Figure 1 (details in Section 3.1).
Video description: A man on top of a building throws a bowling ball towards the pins
Semantic Roles:
ARGO
V ARG1 ARG2
Q1: Who throws a bowling ball towards the pins?
Model's Top Predictions:
A:A man B:A man under the tree
C:A person D:This boy
Correct Answer: A man on top of a building
Q2: Does a man on top of a building throw a bowling ball towards to pins? Model's Top Predictions:
A:Yes B:No C:Maybe
Correct Answer: Yes
Q3: A man on top of a building throws a bowling ball towards the. Model's Top Predictions:
A: field B:pins C:basket D:man
Correct Answer: pins
(a) N-way Classification of Phrases
iQ4: <Q-ARG> throws a bowling ball towards the pins.
Model's generated answer: A man standing on a house
iCorrect answer: A man on top of a building
Q5: A man on top of a building  $< \mathbf{Q} - \mathbf{V}>$  a bowling ball towards the pins.
!Model's generated answer: throws  
!Correct answer: throws
iQ6:A man on top of a building throws  $\langle \mathbf{Q}\cdot \mathbf{ARG1}\rangle$  towards the pins.
Model's generated answer: a ball  
Correct answer: a bowling ball
Q7: A man on top of a building throws a bowling ball  $\langle \mathrm{Q - ARG2}\rangle$
Model's generated answer: towards some bottles
!Correct answer: towards the pins
(b) Free-form Answer Generation
Figure 1: Previous methods formulate VidQA as a N-way classification task. The questions are converted via question generation tool (Q1, Q2) or masking-out strategy (Q3). However, such QA has a theoretical recall upper bound when the correct answer is not among the choice list. In comparison, we propose a free-form text generation task which do not suffer such limitation (Q4-Q7)
While free-form answer generation is highly desirable, evaluating them is non-trivial due to two main challenges. First, existing language generation metrics like BLEU (Papineni et al., 2002) or BERTScore (Zhang* et al., 2020) operate on sentences rather than phrases. When applied to short phrases, in the absence of context, even close matches like "A person" and "The man" would be falsely rejected due to no n-gram overlap or poor contextual embeddings. Second, natural language questions often have strong language priors making it difficult to ascertain if the model retrieved information from the video.
To propose a reasonable evaluation metric, we
Association for Computational Linguistics: Human Language Technologies, pages 2460-2478
Dataset	Source	#Clips	Clip Duration(s)	#QA-Pairs	# QA / Clip	Task Type	Scripts	Box	QA Pair Creation
Movie-QA	Movies	6771	202.7	6462	0.95	MC	✓	×	Human
Movie-FIB	Movies	128,085	4.8	348,998	2.72	OE	×	×	Automatic
VideoQA*	Internet videos	18100	45	174,775	9.66	OE	×	×	Automatic
MSVD-QA	Internet videos	1,970	9.7	50,505	25.64	OE	×	×	Automatic
MSR-VTT-QA	Internet videos	10,000	14.8	243,680	24.37	OE	×	×	Automatic
TGIF-QA	Tumblr GIFs	62,846	3.1	139,414	2.22	OE+MC	×	×	Human+Automatic
TVQA	TV Show	21,793	76	152,545	7	MC	✓	×	Human
TVQA+	TV Show	4200	61.5	29,383	7	MC	✓	✓	Human
ActivityNet-QA*	Internet videos	5800	180	58000	10	OE	×	×	Human
ASRL-QA	Internet videos	35805	36.2	162091	5.54	OE + Phrase	×	✓	Automatic
Charades-SRL-QA	Crowd-Sourced	9513	29.85	71735	7.54	OE + Phrase	×	×	Automatic
Table 1: Comparison of Existing datasets for VidQA with our proposed ASRL-QA and Charades-SRL-QA. Here,  $\mathrm{{OE}} =$  Open-Ended,  $\mathrm{{MC}} =$  Multiple Choice. "Scripts": if answering questions requires access to scripts or subtitles. "Box": if dataset provides bounding box annotations. *: Includes Yes/No questions
revisit our fill-in-the-phrase formulation. Since we know where exactly the generated answer fits in the original query, we can create a complete sentence. With this key insight, we propose relative scoring: using the description as reference sentence, we compute the metrics once by replacing the query-token once with the predicted answer phrase and once with an empty-string. The model's performance is measured by the relative improvement from the predicted answer compared to the empty string. In particular, substituting the answer phrase in the query expression allows the computing the contextual embeddings required by BERTScore.
To mitigate the language-bias issue, we emulate the procedure proposed by (Goyal et al., 2017) where for a given question, another image (or video in our case) is retrieved which has a different answer for the same question. To retrieve such a video, we use a contrastive sampling method (Sadhu et al., 2020) over the dataset by comparing only the lemmatized nouns and verbs within the semantic roles (SRLs). We then propose contrastive scoring to combine the scores of the two answer phrases obtained from the contrastive samples (details on evaluation in Section 3.2).
To investigate VidQAP, we extend three vision-language models namely, Bottom-Up-Top-Down (Anderson et al., 2018), VOGNet (Sadhu et al., 2020) and a Multi-Modal Transformer by replacing their classification heads with a Transformer (Vaswani et al., 2017) based language decoder. To facilitate research on VidQAP we construct two datasets ActivityNet-SRL-QA (ASRL-QA) and Charades-SRL-QA and provide a thorough analysis of extended models to serve as a benchmark for future research (details on model framework in Section 3.3 and dataset creation in Section 4.1).
Our experiments validate the merits of mov-
ing away from N-way classification, and further show even among sequence generation models there exists a large disparity in performance across semantic-roles (i.e. queries for some roles can be answered very easily compared to other roles). Moreover, certain roles hardly benefit from vision-language models suggesting room for improvement. Finally, we investigate the effects of relative scoring and contrastive scoring for VidQAP with respect to BertScore.
Our contributions in this work are two-fold: (i) we introduce VidQAP and propose a systematic evaluation protocol to leverage state-of-art language generation metrics and reduce language bias (ii) we provide extensive analysis and contribute a benchmark on two datasets evaluated using three vision-language models. Our code and dataset are publicly available.
Question Answering in Images has received extensive attention in part due to its end-user applicability. Key to its success has been the availability of large-scale curated datasets like VQA v2.0 (Goyal et al., 2017) for visual question answering and GQA (Hudson and Manning, 2019) for relational reasoning. To address the strong language priors, the datasets are balanced by retrieving images which given the same question lead to a different answer. However, these procedures cannot be extended for VidQA since crowd-sourcing to retrieve videos is expensive and there exists no scene-graph annotations for videos. In this work, we perform the retrieval using lemmatized nouns and verbs of the semantic roles labels obtained from video descriptions to balance the dataset.
1 https://github.com/TheShadow29/Video-QAP
Question Answering in Videos: has garnered less attention compared to ImageQA. A major bottleneck is that there is no principled approach to curating a VidQA dataset which reflects the diversity observed in ImageQA datasets. For instance, naively crowd-sourcing video datasets leads to questions about color, number which is same as ImageQA datasets and doesn't reflect any spatial-temporal structure. To address this issue, TGIF-QA (Jang et al., 2017) and ActivityNet-QA (Yu et al., 2019) use a question-template to enforce questions requiring spatio-temporal reasoning but forgo the question diversity. An orthogonal approach is to combine VidQA with movie scripts (Tapaswi et al., 2016) or subtitles (Lei et al., 2018). However, this severely restricts the domain of videos. Moreover, recent works have noted that language-only baselines often outperform vision-language baselines (Jasani et al., 2019; Yang et al., 2020; Zellers et al., 2019). A separate line of related research has focused on scene-aware dialogue (Alamri et al., 2019). Instead of a single annotator providing both questions and answers, the annotation procedure follows a two-player game setup with one player asking a question and the other player answering with the roles switching after each turn. However, the evaluation method utilizes recall metrics which require the set of phrases to be known apriori. As a result, it doesn't strictly measure the performance of free-form generation but rather how well the ground-truth answer is ranked given a competing set of phrases which is analogous to multiple-choice questions.
Automatic Question Generation: Due to the above limitations, the dominant approach to create large-scale VidQA dataset has been automatic question generation from existing video descriptions which can be easily crowd-sourced. Our proposed formulation of using SRLs to generate query-expressions falls in this category. Prior works include VideoQA (Zeng et al., 2017), MSR-VTTQA and MSVD-QA (Xu et al., 2017) which use a rule based question generator (Heilman and Smith, 2009) to convert descriptions to questions and Movie-Fill-in-the-Blanks (Maharaj et al., 2017) which mask outs at most one word which could be a noun, adjective or verb in a sentence. In comparison, our method poses VidQAP as fill-in-blanks but with phrases, explicitly asks questions about actions, and the answer phrases are not constrained to a fixed set. As a result of this increased space
of phrases, methods on existing datasets cannot be directly applied to VidQAP. To enable further research, we contribute two datasets ASRL-QA and Charades-SRL-QA. In Table 1 we compare these with existing VidQA datasets.
SRL in Vision: has been explored in the context of human object interaction (Gupta and Malik, 2015), situation recognition (Yatskar et al., 2016), and multi-media extraction (Li et al., 2020). Most related to ours is the usage of SRLs for grounding (Silberer and Pinkal, 2018) in images and videos (Sadhu et al., 2020). Our work builds on (Sadhu et al., 2020) in using SRLs on video descriptions, however, our focus is not on grounding. Instead, we use SRLs primarily as a query generation tool and use the argument as a question directive.
3 Design Considerations for VidQAP
The VidQAP task is conceptually simple: given a video and a query expression with a query-token, a model should output an answer phrase that best replaces the query-token. This leads to three main design considerations: (i) How to generate a query-expression from existing resources (Section 3.1) (ii) How to evaluate the answer phrases returned by a model (Section 3.2) (iii) What modeling framework choices enable VidQAP (Section 3.3).
3.1 Using SRLs to Generate Queries for VidQAP
We first briefly describe semantic-role labels  $(\mathrm{SRLs})^2$ . Then we detail how SRLs are used to create VidQAP queries.
Query Generation Using SRLs: Semantic Role Labels (SRLs) provide a high-level label to entities extracted from a sentence in the form of who (ARG0), did what (V) to whom (ARG1) (Strubell et al., 2018). Other roles such as to whom / using what (ARG2) and where (LOC) are also common. As a pre-processing step, we assign SRLs to video descriptions using a state-of-art SRL labeler (Shi and Lin, 2019). A particular description could consist of multiple verbs, in which case, we consider each verb and its associated SRLs independently. For a particular semantic-role, we substitute the corresponding phrase with a query token to generate the query expression. The replaced phrase is the corresponding answer. Using this method we
${}^{2}$  Detailed discussion is provided in supplementary. A demo is available here: https://demo.allennlp.org/semanticrole-labeling
ARG0	V	ARG1	DIR	LOC
A person moves exercise equipment around in the office
Query-Expressions	Answers
<Q-ARG0> moves exercise equipment in the office	A person
A person <Q-V> exercise equipment in the office	moves
A person moves <Q-ARG1> in the office	exercise equipment
A person moves exercise equipment <Q-LOC>	in the office
(a) Following SRLs are considered: ARG0, ARG1, ARG2, V, LOC to generate query-expressions and answers. Here, the phrase corresponding to the semantic-role DIR is removed from both query-expressions and answers.
ARGO	V	DIR	MNR
A person climbs down with his hands folded
(b) Query-expressions would have less than 3 semantic-roles and hence ignored.
are able to generate multiple queries from a single description. An added merit of using SRLs is that query phrases are centered around "verb-phrases" which are highly relevant to the video content.
Generating queries using every SRL is not beneficial as some SRLs are more concerned with phrasing of the language rather than the video. For instance, in the phrase "Players are running around on the field", if we mask out the word "around" (DIR), it can be answered without looking at the video. To address the above issue, we confine our description phrases to a fixed set of semantic-roles namely: ARG0, ARG1, V, ARG2, ARGM-LOC. Only those phrases which belong to the above set of SRLs may appear in the query-expression or as an answer phrase. We further remove phrases which have only two arguments as these are too ambiguous to fill. Figure 2 illustrates these steps.
While using a slot for each slot could potentially limit the vocabulary used in each slot (for instance, the vocabulary set for  $<\mathsf{Q}-\mathsf{ARG1}>$  could be limited to a small number of objects), empirically we don't find this to be the case (see Appendix A.3 for detailed statistics). As a result, VidQAP is no simpler than VidQA task.
We also remark that generating queries need not be strictly limited to masking out a single SRL and one could easily mask multiple SRLs in the same description. However, we find two problems: first, for many cases, the output of masking multiple SRLs becomes exceedingly similar to video description task; second, using contrastive scoring (described in Section 3.2) for multiple SRLs be
Query Expression:	A person <Q-V> exercise equipment.
Reference (Ground Truth):	A person moves exercise equipment.
Hypothesis (Prediction):	A person lifts exercise equipment.
Baseline (Empty String):	A person exercise equipment.
α = B(Ref, Base), β = B(Ref, Hyp), γ = B(Ref, Ref)
Relative Metric Score	Br(Ref, Hyp) = β - α / γ - α
Figure 2: Illustration of our query generation process. In (a) DIR is ignored from both Query and Answers. In (b) the question is removed from validation set since at most two arguments from considered set are present.
Figure 4: Illustration of Contrastive Sampling Process. For the same query-expression, we retrieve two videos with different answers. The model is required to correctly answer both the original and contrastive sample query.
A person holding  $<  \mathrm{Q - ARG1}>$  in their hands
Figure 3: Illustration of the Relative Metric Computation. "moves" is the ground-truth answer and "lifts" is a model's prediction. Relative Metric compares the relative improvement from using the model's prediction as compared to an empty string.
comes considerably more involved. As a result, in this work, we focus on using a single SRL and keep the generalization to include multiple SRL queries for future work.
3.2 Evaluating Answer Phrases
A key challenge in VidQAP is the lack of any standard protocol to evaluate free-form generated phrases. A simple way is to adopt metrics like BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), and CIDER (Vedantam et al., 2015) which are already used for captioning in images and videos. However, these metrics suffer from limited generalization: BLEU, ROUGE, and CIDER require exact n-gram matches. While this is fine for captioning where longer phrases average out errors, answers phrases are typically much smaller than a complete sentence. This leads to many near-correct answers receiving very low scores.
This issue is resolved to a certain extent for captioning by learned metrics like BERTScore (Zhang* et al., 2020) which utilize contextual embeddings obtained from large pretrained models like BERT (Devlin et al., 2019) and RoBerta (Liu et al., 2019). However, answer phrases are usually short and don't provide meaningful contextual embeddings. In the extreme case when the answer is a single word, for instance when the query is about a
Verb, these embeddings turn out to be very noisy leading to large number of false-positives.
Relative Scoring: To enable usage of contextual embeddings, we propose evaluating the relative improvement of the generated answer phrase compared to the ground-truth phrase. We denote the input query expression as  $Q$ , the ground-truth answer is  $A_{gt}$ , and the predicted answer is  $A_{pred}$ . Let  $Q(X)$  denote  $Q$  with the question tokens replaced by  $X$ . Then for a given metric  $B$ , we compute the relative metric  $B_r$  as (see Figure 3 for illustration)
$$
R e f = Q (A _ {g t}), H y p = Q (A _ {p r e d}), B a s e = Q (^ {\prime \prime \prime})
$$
$$
B _ {r} \left(A _ {g t}, A _ {p r e d}\right) = \frac {B (R e f , H y p) - B (R e f , B a s e)}{B (R e f , R e f) - B (R e f , B a s e)} \tag {1}
$$
Note that  $B(Ref, Ref) = 1$  for BLEU, METEOR, ROUGE, BERTScore but not for CIDEr.
The empty-string baseline in Eqn 1 could be replaced with predictions from any model trained for this task. In this work, we restrict to only empty-string baseline due to two desirable properties: its computational simplicity and it being agnostic to models and datasets.
We further observe that Eqn 1 is very similar to the re-scaling proposed in BERTScore. However, in BertScore re-scaling aims at making the score more readable and doesn't change the relative ranking of the hypothesis. In our case, Eqn 1 plays two roles: first, it allows computing the contextual embeddings because the answers are now embedded inside a complete phrase, second while the ranking is not affected for a particular query, the score would be different across queries and hence affect the overall relative metric.
Contrastive Scoring: Visual Question Answering suffers from heavy language priors, and as a result, it is often difficult to attribute whether the image or video played a role in the success. For images, (Goyal et al., 2017) resolved this by balancing the dataset where they crowd-sourced the task of collecting an image that has a different answer for the same question. However, such a crowdsourcing method is difficult to extend to videos since searching for videos requires a much longer time. This is further complicated by accepting answer phrases compared to single word.
We simulate the balancing process using the contrastive sampling method used in (Sadhu et al., 2020). Specifically, for a given video-query-answer  $(V_{1},Q_{1},A_{1})$  tuple we retrieve another video-query-answer  $(V_{2},Q_{2},A_{2})$  tuple which share the same
semantic role structure as well as lemmatized noun and verbs for the question, but a different lemmatized noun for the answer. At test time, the model evaluates the question separately, but the evaluation function requires both answers to be correct. Since our answers comprise of phrases, the notion of correctness is not absolute (unlike say accuracy metric). Thus, we put a threshold  $t$  below which the answer is deemed incorrect.
Mathematically, let  $S_{i} = B_{r}(A_{gt_{i}}, A_{pred_{i}})$  be the relative score for sample  $i$ , and we are given sample  $j$  is a contrastive example for sample  $i$ . Then the contrastive score  $(CS_{i})$  for sample  $i$  at a threshold  $T_{CS}$  would be
$$
C S _ {i} = \max  \left(S _ {i} \mathbb {1} \left[ S _ {j} > T _ {C S} * B \left(R e f _ {j}, R e f _ {j}\right) \right], 0\right) \tag {2}
$$
Here  $\mathbb{1}[]$  is the indicator variable which is 1 if the expression within brackets is True, otherwise 0. The max operator ensures the scores don't become negative. For our experiments, we use  $T_{CS} = 0$  which requires that the answer for the contrastive sample should be better than an empty string.
We further use the contrastive samples to compute a consistency metric. For sample  $i$ , the consistency  $C o n s_{i}$  for a threshold  $T_{c o n s}$  is given by
$$
C o n s _ {i} = \mathbb {1} \left[ \left(S _ {i} - T _ {\text {c o n s}}\right) * \left(S _ {j} - T _ {\text {c o n s}}\right) > 0 \right] \tag {3}
$$
As such, Consistency requires the model to be either correct or incorrect for both the original and the contrastive sample.
Combined Metric at a Glance: Given metric  $B$ , for a given sample  $i$  and contrastive sample  $j$
1. Compute relative metric (Eqn 1) for  $i,j$
2. Compute contrastive score (Eqn 2)
3. Optionally compute Consistency (Eqn 3)
We use the prefix "R-" such as in R-B to denote both relative scoring and contrastive scoring is being computed. We report Consistency for BertScore with  $T_{cons} = 0.1$
We note that, by construction, the relative scoring (Eqn 1) is positively correlated with human judgment, as the closer, the hypothesis is to the reference, the higher would the score be. The contrastive scoring is a metric used to prevent the model from guessing the correct answer by exploiting language biases and instead use the video to give a suitable prediction. Since humans don't have the ability to exploit such biases, it is difficult to relate to human evaluation.
Figure 5: Schematic of the various models used to benchmark VidQAP. Input Query: "A person picks up  $<\mathsf{Q}-\mathsf{ARG1}>$ ". Ground-Truth Answer: "a pair of shoes". (a) Lang-QAP is a language-only model which encodes the query input and passes to a decoder. (b) BUTD-QAP uses the pooled feature representation from language encoder and attends over the visual features. (c) VOG-QAP uses an additional phrase encoder and applies a Transformer over the multi-modal features (d) MTX-QAP consumes both the language and visual features with a multi-modal transformer.
3.3 Model Framework
Models for VidQAP require a language encoder to encode the question, a visual encoder to extract video features, a multi-modal module to jointly learn over vision-language space and a decoder to generate a sequence of words.