Venue: NAACL 2021

# Video Question Answering with Phrases via Semantic Roles

Arka Sadhu\({}^{1}\)  Kan Chen\({}^{2}\)  Ram Nevatia\({}^{1}\)

\({}^{1}\)University of Southern California  \({}^{2}\)Facebook Inc.

{asadhu|nevatia}@usc.edu    kanchenl8@fb.com

###### Abstract

Video Question Answering (VidQA) evaluation metrics have been limited to a single-word answer or selecting a phrase from a fixed set of phrases. These metrics limit the VidQA models' application scenario. In this work, we leverage semantic roles derived from video descriptions to mask out certain phrases, to introduce VidQAP which poses VidQA as a fill-in-the-phrase task. To enable evaluation of answer phrases, we compute the relative improvement of the predicted answer compared to an empty string. To reduce the influence of language-bias in VidQA datasets, we retrieve a video having a different answer for the same question. To facilitate research, we construct ActivityNet-SRL-QA and Charades-SRL-QA and benchmark them by extending three vision-language models. We perform extensive analysis and ablative studies to guide future work. Code and data are public.

+
Footnote †: _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2460–2478 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Given a video, Video Question Answering (VidQA) requires a model to provide an answer to a video related question. However, existing works treat VidQA as an N-way (\(N{\sim}1k\)) classification task across a fixed set of phrases. Models trained under such formulations are strictly restricted in their recall rate, generalize poorly, and have severe limitations for end-user applications.

In this work, we introduce Video Question Answering with Phrases (VidQAP) which treats VidQA as a _fill-in-the-phrase_ task. Instead of a question, the input to VidQP consists of a query expression with a query-token. Then, given a video, VidQAP requires replacing query-token with a sequence of generated words. To generate a query, we leverage video descriptions and assign semantic roles to each phrase in these descriptions. Replacing a particular semantic-role with a query token produces a query-answer pair. We illustrate this in Figure 1 (details in Section 3.1).

While free-form answer generation is highly desirable, evaluating them is non-trivial due to two main challenges. First, existing language generation metrics like BLEU Papineni et al. (2002) or BERTScore Zhang* et al. (2020) operate on sentences rather than phrases. When applied to short phrases, in the absence of context, even close matches like "A person" and "The man" would be falsely rejected due to no n-gram overlap or poor contextual embeddings. Second, natural language questions often have strong language priors making it difficult to ascertain if the model retrieved information from the video.

To propose a reasonable evaluation metric, we

Figure 1: Previous methods formulate VidQA as a N-way classification task. The questions are converted via question generation tool (Q1, Q2) or masking-out strategy (Q3). However, such QA has a theoretical recall upper bound when the correct answer is not among the choice list. In comparison, we propose a free-form text generation task which do not suffer such limitation (Q4-Q7)

revisit our _fill-in-the-phrase_ formulation. Since we know where exactly the generated answer fits in the original query, we can create a complete sentence. With this key insight, we propose **relative scoring**: using the description as reference sentence, we compute the metrics once by replacing the query-token once with the predicted answer phrase and once with an empty-string. The model's performance is measured by the relative improvement from the predicted answer compared to the empty string. In particular, substituting the answer phrase in the query expression allows the computing the contextual embeddings required by BERTScore.

To mitigate the language-bias issue, we emulate the procedure proposed by Goyal et al. (2017) where for a given question, another image (or video in our case) is retrieved which has a different answer for the same question. To retrieve such a video, we use a contrastive sampling method Sadhu et al. (2020) over the dataset by comparing only the lemmatized nouns and verbs within the semantic roles (SRLs). We then propose **contrastive scoring** to combine the scores of the two answer phrases obtained from the contrastive samples (details on evaluation in Section 3.2).

To investigate VidQAP, we extend three vision-language models namely, Bottom-Up-Top-Down Anderson et al. (2018), VOGNet Sadhu et al. (2020) and a Multi-Modal Transformer by replacing their classification heads with a Transformer Vaswani et al. (2017) based language decoder. To facilitate research on VidQAP we construct two datasets ActivityNet-SRL-QA (ASRL-QA) and Charades-SRL-QA and provide a thorough analysis of extended models to serve as a benchmark for future research (details on model framework in Section 3.3 and dataset creation in Section 4.1).

Our experiments validate the merits of moving away from N-way classification, and further show even among sequence generation models there exists a large disparity in performance across semantic-roles (i.e. queries for some roles can be answered very easily compared to other roles). Moreover, certain roles hardly benefit from vision-language models suggesting room for improvement. Finally, we investigate the effects of relative scoring and contrastive scoring for VidQAP with respect to BertScore.

Our contributions in this work are two-fold: (i) we introduce VidQAP and propose a systematic evaluation protocol to leverage state-of-art language generation metrics and reduce language bias (ii) we provide extensive analysis and contribute a benchmark on two datasets evaluated using three vision-language models. Our code and dataset are publicly available. 1

Footnote 1: [https://github.com/TheShadow29Video-QP](https://github.com/TheShadow29Video-QP)

## 2 Related Works

**Question Answering in Images** has received extensive attention in part due to its end-user applicability. Key to its success has been the availability of large-scale curated datasets like VQA v2.0 Goyal et al. (2017) for visual question answering and QQA Hudson and Manning (2019) for relational reasoning. To address the strong language priors, the datasets are balanced by retrieving images which given the same question lead to a different answer. However, these procedures cannot be extended for VidQA since crowd-sourcing to retrieve videos is expensive and there exists no scene-graph annotations for videos. In this work, we perform the retrieval using lemmatized nouns and verbs of the semantic roles labels obtained from video descriptions to balance the dataset.

\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline Dataset & Source & \#Clips & Clip Duration(s) & \#QA-Pairs & \# QA / Clip & Task Type & Scripts & Box & QA Pair Creation \\ \hline Movie-QA & Movies & 6771 & 202.7 & 6462 & 0.95 & MC & ✓ & ✗ & Human \\ Movie-FIB & Movies & 128,085 & 4.8 & 348,998 & 2.72 & OE & ✗ & ✗ & Automatic \\ VideoQA* & Internet videos & 18100 & 45 & 174,775 & 9.66 & OE & ✗ & ✗ & Automatic \\ MSVD-QA & Internet videos & 1,970 & 9.7 & 50,505 & 25.64 & OE & ✗ & ✗ & Automatic \\ MSR-VTT-QA & Internet videos & 10,000 & 14.8 & 243,680 & 24.37 & OE & ✗ & ✗ & Automatic \\ TGIF-QA & Tumblef Citys & 62,846 & 3.1 & 139,414 & 2.22 & OE+MC & ✗ & ✗ & Human=Automatic \\ TVQA & TV Show & 21,793 & 76 & 152,545 & 7 & MC & ✓ & ✗ & Human \\ TVQA+ & TV Show & 4200 & 61.5 & 290,833 & 7 & MC & ✓ & ✓ & Human \\ ActivityNet-QA* & Internet videos & 5800 & 180 & 58000 & 10 & OE & ✗ & ✗ & Human \\ \hline ASRL-QA & Internet videos & 35805 & 36.2 & 162091 & 5.54 & OE + Phrase & ✗ & ✓ & Automatic \\ Charades-SRL-QA & Crowd-Sourced & 9513 & 29.85 & 71735 & 7.54 & OE + Phrase & ✗ & ✗ & Automatic \\ \hline \end{tabular}
\end{table}
Table 1: Comparison of Existing datasets for VidQA with our proposed ASRL-QA and Charades-SRL-QA. Here, OE = Open-Ended, MC = Multiple Choice. “Scripts”: if answering questions requires access to scripts or subtitles. ‘Box”: if dataset provides bounding box annotations. *: Includes Yes/No questions 



**Question Answering in Videos:** has garnered less attention compared to ImageQA. A major bottleneck is that there is no principled approach to curating a VidQA dataset which reflects the diversity observed in ImageQA datasets. For instance, naively crowd-sourcing video datasets leads to questions about color, number which is same as ImageQA datasets and doesn't reflect any spatial-temporal structure. To address this issue, TGIF-QA (Jang et al., 2017) and ActivityNet-QA (Yu et al., 2019) use a question-template to enforce questions requiring spatio-temporal reasoning but forgo the question diversity. An orthogonal approach is to combine VidQA with movie scripts (Tapaswi et al., 2016) or subtitles (Lei et al., 2018). However, this severely restricts the domain of videos. Moreover, recent works have noted that language-only baselines often outperform vision-language baselines (Jasani et al., 2019; Yang et al., 2020; Zellers et al., 2019). A separate line of related research has focused on scene-aware dialogue (Alamri et al., 2019). Instead of a single annotator providing both questions and answers, the annotation procedure follows a two-player game setup with one player asking a question and the other player answering with the roles switching after each turn. However, the evaluation method utilizes recall metrics which require the set of phrases to be known apriori. As a result, it doesn't strictly measure the performance of free-form generation but rather how well the ground-truth answer is ranked given a competing set of phrases which is analogous to multiple-choice questions.

**Automatic Question Generation:** Due to the above limitations, the dominant approach to create large-scale VidQA dataset has been automatic question generation from existing video descriptions which can be easily crowd-sourced. Our proposed formulation of using SRLs to generate query-expressions falls in this category. Prior works include VideoQA (Zeng et al., 2017), MSR-VTT-QA and MSVD-QA (Xu et al., 2017) which use a rule based question generator (Heilman and Smith, 2009) to convert descriptions to questions and Movie-Fill-in-the-Blanks (Maharaj et al., 2017) which mask outs at most one word which could be a noun, adjective or verb in a sentence. In comparison, our method poses VidQAP as fill-in-blanks but with phrases, explicitly asks questions about actions, and the answer phrases are not constrained to a fixed set. As a result of this increased space of phrases, methods on existing datasets cannot be directly applied to VidQAP. To enable further research, we contribute two datasets ASRL-QA and Charades-SRL-QA. In Table 1 we compare these with existing VidQA datasets.

**SRL in Vision:** has been explored in the context of human object interaction (Gupta and Malik, 2015), situation recognition (Yatskar et al., 2016), and multi-media extraction (Li et al., 2020). Most related to ours is the usage of SRLs for grounding Silberer and Pinkal, 2018) in images and videos (Sadhut et al., 2020). Our work builds on (Sadhu et al., 2020) in using SRLs on video descriptions, however, our focus is not on grounding. Instead, we use SRLs primarily as a query generation tool and use the argument as a question directive.

## 3 Design Considerations for VidQAP

The VidQAP task is conceptually simple: given a video and a query expression with a query-token, a model should output an answer phrase that best replaces the query-token. This leads to three main design considerations: (i) How to generate a query-expression from existing resources (Section 3.1) (ii) How to evaluate the answer phrases returned by a model (Section 3.2) (iii) What modeling framework choices enable VidQAP (Section 3.3).

### Using SRLs to Generate Queries for VidQAP

We first briefly describe semantic-role labels (SRLs)2. Then we detail how SRLs are used to create VidQAP queries.

Footnote 2: Detailed discussion is provided in supplementary. A demo is available here: [https://demo.allennlp.org/semantic-role-labeling](https://demo.allennlp.org/semantic-role-labeling)

**Query Generation Using SRLs:** Semantic Role Labels (SRLs) provide a high-level label to entities extracted from a sentence in the form of who (ARG0), did what (V) to whom (ARG1) (Strubell et al., 2018). Other roles such as to whom / using what (ARG2) and where (LOC) are also common. As a pre-processing step, we assign SRLs to video descriptions using a state-of-art SRL labeler (Shi and Lin, 2019). A particular description could consist of multiple verbs, in which case, we consider each verb and its associated SRLs independently. For a particular semantic-role, we substitute the corresponding phrase with a query token to generate the query expression. The replaced phrase is the corresponding answer.



are able to generate multiple queries from a single description. An added merit of using SRLs is that query phrases are centered around "verb-phrases" which are highly relevant to the video content.

Generating queries using every SRL is not beneficial as some SRLs are more concerned with phrasing of the language rather than the video. For instance, in the phrase "Players are running around on the field", if we mask out the word "around" (DIR), it can be answered without looking at the video. To address the above issue, we confine our description phrases to a fixed set of semantic-roles namely: ARG0, ARG1, V, ARG2, ARGM-LOC. Only those phrases which belong to the above set of SRLs may appear in the query-expression or as an answer phrase. We further remove phrases which have only two arguments as these are too ambiguous to fill. Figure 2 illustrates these steps.

While using a slot for each slot could potentially limit the vocabulary used in each slot (for instance, the vocabulary set for \(<\!\!Q\)-ARG1\(>\) could be limited to a small number of objects), empirically we don't find this to be the case (see Appendix A.3 for detailed statistics). As a result, VidQAP is no simpler than VidQA task.

We also remark that generating queries need not be strictly limited to masking out a single SRL and one could easily mask multiple SRLs in the same description. However, we find two problems: first, for many cases, the output of masking multiple SRLs becomes exceedingly similar to video description task; second, using contrastive scoring (described in Section 3.2) for multiple SRLs becomes considerably more involved. As a result, in this work, we focus on using a single SRL and keep the generalization to include multiple SRL queries for future work.

### Evaluating Answer Phrases

A key challenge in VidQP is the lack of any standard protocol to evaluate free-form generated phrases. A simple way is to adopt metrics like BLEU Papineni et al. (2002), ROUGE Lin (2004), METEOR Banerjee and Lavie (2005), and CIDER Vedantam et al. (2015) which are already used for captioning in images and videos. However, these metrics suffer from limited generalization: BLEU, ROUGE, and CIDER require exact n-gram matches. While this is fine for captioning where longer phrases average out errors, answers phrases are typically much smaller than a complete sentence. This leads to many near-correct answers receiving very low scores.

This issue is resolved to a certain extent for captioning by learned metrics like BERTScore Zhang* et al. (2020) which utilize contextual embeddings obtained from large pretrained models like BERT Devlin et al. (2019) and RoBerta Liu et al. (2019). However, answer phrases are usually short and don't provide meaningful contextual embeddings. In the extreme case when the answer is a single word, for instance when the query is about a

Figure 4: Illustration of Contrastive Sampling Process. For the same query-expression, we retrieve two videos with different answers. The model is required to correctly answer both the original and contrastive sample query.

Figure 3: Illustration of the Relative Metric Computation. “moves” is the ground-truth answer and “lifts” is a model’s prediction. Relative Metric compares the relative improvement from using the model’s prediction as compared to an empty string.

 

Verb, these embeddings turn out to be very noisy leading to large number of false-positives.

**Relative Scoring:** To enable usage of contextual embeddings, we propose evaluating the relative improvement of the generated answer phrase compared to the ground-truth phrase. We denote the input query expression as \(Q\), the ground-truth answer is \(A_{gt}\) ,and the predicted answer is \(A_{pred}\). Let \(Q(X)\) denote \(Q\) with the question tokens replaced by \(X\). Then for a given metric \(B\), we compute the relative metric \(B_{r}\) as (see Figure 3 for illustration)

\[Ref{=}Q(A_{gt}),Hyp{=}Q(A_{pred}),Base{=}Q(``")\]

\[B_{r}(A_{gt},A_{pred})=\frac{B(Ref,Hyp)-B(Ref,Base)}{B(Ref,Ref)-B(Ref,Base)} \tag{1}\]

Note that \(B(Ref,Ref){=}1\) for BLEU, METEOR, ROUGE, BERTScore but not for CIDEr.

The empty-string baseline in Eqn 1 could be replaced with predictions from any model trained for this task. In this work, we restrict to only emptying baseline due to two desirable properties: its computational simplicity and it being agnostic to models and datasets.

We further observe that Eqn 1 is very similar to the re-scaling proposed in BERTScore. However, in BertScore re-scaling aims at making the score more readable and doesn't change the relative ranking of the hypothesis. In our case, Eqn 1 plays two roles: first, it allows computing the contextual embeddings because the answers are now embedded inside a complete phrase, second while the ranking is not affected for a particular query, the score would be different across queries and hence affect the overall relative metric.

**Contrastive Scoring:** Visual Question Answering suffers from heavy language priors, and as a result, it is often difficult to attribute whether the image or video played a role in the success. For images, Goyal et al. (2017) resolved this by balancing the dataset where they crowd-sourced the task of collecting an image that has a different answer for the same question. However, such a crowdsourcing method is difficult to extend to videos since searching for videos requires a much longer time. This is further complicated by accepting answer phrases compared to single word.

We simulate the balancing process using the contrastive sampling method used in Sadhu et al. (2020). Specifically, for a given video-query-answer \((V_{1},Q_{1},A_{1})\) tuple we retrieve another video-query-answer \((V_{2},Q_{2},A_{2})\) tuple which share the same semantic role structure as well as lemmatized noun and verbs for the question, but a different lemmatized noun for the answer. At test time, the model evaluates the question separately, but the evaluation function requires both answers to be correct. Since our answers comprise of phrases, the notion of correctness is not absolute (unlike say accuracy metric). Thus, we put a threshold \(t\) below which the answer is deemed incorrect.

Mathematically, let \(S_{i}{=}B_{r}(A_{gt_{i}},A_{pred_{i}})\) be the relative score for sample \(i\), and we are given sample \(j\) is a contrastive example for sample \(i\). Then the contrastive score (\(CS_{i}\)) for sample \(i\) at a threshold \(T_{CS}\) would be

\[CS_{i}=max(S_{i}\mathbbm{1}[S_{j}>T_{CS}*B(Ref_{j},Ref_{j})],0) \tag{2}\]

Here \(\mathbbm{1}[]\) is the indicator variable which is \(1\) if the expression within brackets is True, otherwise \(0\). The \(max\) operator ensures the scores don't become negative. For our experiments, we use \(T_{CS}{=}0\) which requires that the answer for the contrastive sample should be better than an empty string.

We further use the contrastive samples to compute a consistency metric. For sample \(i\), the consistency \(Cons_{i}\) for a threshold \(T_{cons}\) is given by

\[Cons_{i}=\mathbbm{1}[(S_{i}-T_{cons})*(S_{j}-T_{cons})>0] \tag{3}\]

As such, Consistency requires the model to be either correct or incorrect for both the original and the contrastive sample.

**Combined Metric at a Glance:** Given metric \(B\), for a given sample \(i\) and contrastive sample \(j\)

1. Compute relative metric (Eqn 1) for \(i,j\)
2. Compute contrastive score (Eqn 2)
3. Optionally compute Consistency (Eqn 3)

We use the prefix "R-" such as in R-B to denote both relative scoring and contrastive scoring is being computed. We report Consistency for BertScore with \(T_{cons}{=}0.1\)

We note that, by construction, the relative scoring (Eqn 1) is positively correlated with human judgment, as the closer, the hypothesis is to the reference, the higher would the score be. The contrastive scoring is a metric used to prevent the model from guessing the correct answer by exploiting language biases and instead use the video to give a suitable prediction. Since humans don't have the ability to exploit such biases, it is difficult to relate to human evaluation 



### Model Framework

Models for VidQAP require a language encoder to encode the question, a visual encoder to extract video features, a multi-modal module to jointly learn over vision-language space and a decoder to generate a sequence of words.

**Inputs** include query expression \(\{w\}_{i=1}^{L}\) (\(L\) is number of words), video segment features for \(F_{1}\) frames and optionally \(k\) RCNN features for \(F_{2}\) frames. In either case, frames are sampled uniformly from the video segment time-span. While the models differ in their encoding scheme, our language decoder model (Transformer based) used to generate the output answer phrase is kept same across all models with QAP suffix.

**Lang-QAP**: is a language-only (video-blind) model using only the query input. It uses Transformer based encoder to encode the query into \(\hat{q}\in\mathbb{R}^{L\times d}\). The decoder subsequently uses the last layer output of the encoder (Figure5-(a)).

**BUTD-QAP**: Bottom-up-Top-Down Anderson et al. (2018) is a popular approach for image question answering as well as captioning. It first computes attention between the question and the RCNN visual features to generate an attended visual feature, which is then used with the question to produce an output answer. Here, we replace the RCNN features with the segment features (\(\hat{v}\in\mathbb{R}^{F_{1}\times d}\)). We can also include RCNN features by projecting them to same dimension as segment features and then concatenate them along the frame-axis (\(\hat{v}\in\mathbb{R}^{(F_{1}+F_{2}*k)\times d}\)). For language features, we use the [CLS] token representation from the last layer of the language encoder used in Lang-QAP. The output using the language and visual features is (\(\hat{m}\in\mathbb{R}^{d}\)) passed to the decoder (Figure 5(b)).

**VOG-QAP**: VGGNet Sadhu et al. (2020) has been proposed for grounding objects in videos given a natural language query. Following the architecture, we first derive phrase encoding which corresponds to a single SRL i.e. \(\hat{q}\in\mathbb{R}^{S\times d}\) (\(S\) is number of semantic roles). These phrase features are concatenated with the visual features (same as those used in BUTD-QAP (i.e. \(\hat{v}\))) to get multimodal features \(m[l,i]{=}[\hat{v}_{i}||\hat{q}_{l}]\) and then reshaped to get \(m\in\mathbb{R}^{SsK\times d}\). These multi-modal features are subsequently passed to decoder to generate the output sequence (Figure 5 (c)).

**MTX-QAP**: Recently, transformer models pre-trained on large-scale paired image-text data have become popular. Even in the absence of pretraining, such architectures can achieve competitive performance Lu et al. (2019). In the context of videos, ActBert Zhu and Yang (2020) has been proposed. We create a similar architecture to ActBert but we replace their proposed Tangled-Transformer with a vanilla Transformer 3. Specifically, we jointly encode the language and visual features in a single transformer and feed the output to the decoder (Figure 5 (d)).

Footnote 3: The code for ActBert is not publicly available.

**LangCL and MTxCL:** Apart from QAP models, we also consider their phrase classification counterparts where the decoder is replaced with a N-way classifier (two-layered MLP in our case) across a fixed set of phrases. For our experiments, we used \(N{=}1k\) phrases for LangCL and \(N{\in}\{1k,10k\}\) for MTxCL.

Figure 5: Schematic of the various models used to benchmark VidQAP. Input Query: “A person picks up ¡Q–ARG1¿”. Ground-Truth Answer: “a pair of shoes”. (a) Lang-QAP is a language-only model which encodes the query input and passes to a decoder. (b) BUTD-QAP uses the pooled feature representation from language encoder and attends over the visual features. (c) VGG-QAP uses an additional phrase encoder and applies a Transformer over the multi-modal features (d) MTX-QAP consumes both the language and visual features with a multi-modal transformer.



Experiments

We briefly discuss the dataset creation process (Section 4.1), followed by experimental setup (Section 4.2). We then summarize our results (Section 4.3) and discuss key-findings. We provide implementation details, qualitative visualizations of our dataset, metrics and trained models in the appendix.

### Dataset Creation

We create two datasets ASRL-QA and CharadesSRL-QA derived from ActivityNet-Captions Krishna et al. (2017) and Charades Sigurdsson et al. (2016) respectively.

There are three key steps to create QA datasets from descriptions: (i) assign semantic-roles to the descriptions (ii) perform co-reference resolution so that the questions are self-contained (iii) obtain lemmatized nouns and verbs to perform contrastive sampling. For semantic-role labeling, we use Shi and Lin (2019). For co-reference resolution, we use the co-reference resolution model provided by allennlp library Gardner et al. (2017) which uses the model by Lee et al. (2017) but replaces the GloVe Pennington et al. (2014) embeddings with SpanBERT embeddings Joshi et al. (2019)4.

Footnote 4: [https://demo.allennlp.org/coreference-resolution](https://demo.allennlp.org/coreference-resolution)

Since Charades primarily involves videos with a single person, we discard questions involving ARG0. We limit to using a single description per video to avoid repetitive questions. We re-use the same train split for both datasets. For ASRL-QA, test set of ActivityNet is not public and Charades only has a test set but no official validation set. Thus, we split the existing validation set by video names and create the validation and test sets. For both validation and test splits, we remove those questions for which no contrastive sample was found as it indicates data-biases.

### Experimental Setup

**Dataset Statistics:**ASRL-QA has 35.7k videos and 162k queries split into train, validation and test sets with 30.3k, 2.7k, 2.7k videos and 147k, 7.5k, 7.5k queries. We observe that the size of validation and test sets are proportionately smaller compared to their respective train sets. This is because only queries with corresponding contrastive sample are included while no such filtering is done for the train set (\({\sim}95k\) queries in train set have a contrastive pair). Charades-SRL-QA contains 9.4k videos and 71.7k queries split across train, validation and test sets with 7.7k, 0.8k, 0.8k videos and 59.3k, 6.1k, 6.2k queries. Despite its smaller size, the size of validation, test sets of Charades-SRL-QA is comparable to ASRL-QA as Charades is curated with the goal of diversifying subject, verb, object tuples. Supplementary material provides further details on the dataset statistics and visualizations.

**Evaluation Metrics:** As discussed in Section 3.2, we report the combined metric (i.e. metrics prefixed with "R-") for the commonly used generation metrics: BLEU, METEOR, ROUGE, CIDEr and BertScore implementations from Chen et al. (2015); Zhang* et al. (2020)). For BLEU, we report the sentence level BLEU-2. All reported results are test set results using the model which performs best on validation set.

### Results and Discussions

Table 2 compares performance of the proposed VidQAP models with N-way classification baselines (denoted with suffix "CL") on ASRL-QA and Charades-SRL-QA.

**Comparing Metrics:** It is evident that compared to other metrics, R-BertScore shows a higher relative improvement. This is because BertScore allows soft-matches by utilizing contextual embeddings obtained from a pre-trained BERT Devlin et al. (2019) or Roberta Liu et al. (2019) model.

**Comparison Across Datasets:** We find that performance on both datasets follow very similar trends across all metrics. Charades-SRL-QA has slightly higher scores compared to ASRL-QA likely because it has lesser data variations (Charades is mostly confined indoor videos) suggesting findings on either dataset would transfer.

**Comparison within N-way Classification:** We notice that when \(1k\) fixed set of phrases are used classification models show very limited performance. Allowing \(10k\) phrases gives a significant improvement in performance on Charades-SRL-QA (\(12\) points on R-BS) however this doesn't translate to ASRL-QA. This is because ASRL-QA contains many more probable phrases (\(29K\) compared to \(8K\)) in their respective training sets. We also notice that increasing the number of phrases vocabulary coincides with decreasing consistency.

**Comparing Free-from Answer Generation (QP) with N-way Classification (CL):** We investigate the advantages of using a decoder network to generate phrases compared to an N-way classification over a fixed set of phrases (denoted 

with the suffix "CL" and number of phrases used in parenthesis). Table 2 shows that both Lang-QAP and MTX-QAP outperform their classification counterparts, namely Lang-CL and MTX-CL on both datasets. This implies the free-form generation are not limited to simply generating the most frequently appearing phrases in the training set, thereby showing its effectiveness.

**Comparison Across Models:** We find that multi-modal models outperform language-only baseline. However, the improvement over language baseline is small. The reason for the small gap is elucidate in Table 3 where we report R-BertScore for every considered SRL.

We find a large disparity in performance depending on the SRL. Most strikingly, multi-modal models perform worse than language-only model on ARG0 and V. For ARGO, the strong performance of the Lang-QAP arises because most of the time the agent who causes an action is a human. Therefore answer phrases having simply "A man" or "A woman" or "A person" leads to reasonable performance. This additionally suggests that grounding "who" is performing the action remains non-trivial.

The more surprising result is the strong performance of Lang-QAP on V which is consistent across both datasets despite using contrastive sampling. There are two likely causes. First, the distinction between verbs is not as strict as object nouns, i.e. even similar verbs are classified as a separate verb diminishing the returns of contrastive sampling. For instance, "jumping" and "hoping" have different lemma and thus considered distinct verbs but R-BS would treat them as similar even if the specific action would be classified "jumping" rather than "hoping". Second, SRLs such as ARG1 confines the set of possible verbs. For instance, if the object is "glass", only limited verbs such as "drink", "hold" are probable.

On the remaining arguments namely ARG1, ARG2, and LOC, multi-modal models show a steady improvement over language-only baseline ranging from \(1{-}10\%\). However, the performance

\begin{table}
\begin{tabular}{l l c c c c c} \hline  & & ARG0 & V & ARG1 & ARG2 & LOC \\ \cline{5-6}  & Direct & 0.552 & 0.9268 & 0.234 & 0.302 & 0.216 \\  & Rel Score & 0.7 & 0.534 & 0.332 & 0.237 & 0.1 \\  & CS@0 & 0.697 & 0.519 & 0.325 & 0.322 & 0.145 \\  & CS@0.1 & 0.69 & 0.492 & 0.295 & 0.28 & 0.132 \\  & CS@0.2 & 0.68 & 0.459 & 0.262 & 0.212 & 0.106 \\  & CS@0.3 & 0.657 & 0.423 & 0.219 & 0.149 & 0.085 \\ \hline \hline \multirow{5}{*}{
\begin{tabular}{c} ✓ \\ \end{tabular} } & Direct & 0.566 & 0.929 & 0.269 & 0.321 & 0.258 \\  & Rel Score & 0.706 & 0.488 & 0.366 & 0.25 & 0.14 \\  & CS@0 & 0.702 & 0.478 & 0.374 & 0.344 & 0.17 \\  & CS@0.1 & 0.693 & 0.45 & 0.343 & 0.305 & 0.145 \\  & CS@0.2 & 0.681 & 0.413 & 0.306 & 0.239 & 0.117 \\  & CS@0.3 & 0.659 & 0.376 & 0.27 & 0.17 & 0.08 \\ \hline \hline \end{tabular}
\end{table}
Table 4: BertScore Metrics computed Directly on answer phrases. Rel Score: After Relative Scoring. CS@T: Contrastive scoring with threshold T.

\begin{table}
\begin{tabular}{l|r r r r r r|r r r r r} \hline  & \multicolumn{6}{c|}{ASRL-QA} & \multicolumn{6}{c}{Charades-SRL-QA} \\  & R-BS & Cons & R-B@2 & R-R & R-M & R-C & R-BS & Cons & R-B@2 & R-R & R-M & R-C \\ \hline LangCL (\(1k\)) & 0.253 & **0.889** & 0.120 & 0.098 & 0.071 & 0.044 & 0.293 & 0.697 & 0.224 & 0.209 & 0.114 & 0.077 \\ MTxCL (\(1k\)) & 0.255 & **0.869** & 0.130 & 0.114 & 0.080 & 0.050 & 0.288 & 0.707 & 0.215 & 0.208 & 0.116 & 0.075 \\ MTxCL (\(10k\)) & 0.286 & 0.788 & 0.157 & 0.133 & 0.100 & 0.061 & 0.408 & 0.695 & 0.286 & 0.261 & 0.142 & 0.108 \\ \hline Lang-QAP & 0.402 & 0.728 & 0.228 & 0.182 & 0.125 & 0.095 & 0.406 & 0.719 & 0.277 & 0.253 & 0.147 & 0.121 \\ BUTD-QAP & 0.413 & 0.716 & 0.237 & 0.203 & **0.147** & 0.105 & 0.399 & 0.714 & 0.271 & 0.231 & 0.115 & 0.105 \\ VOC-QAP & **0.414** & 0.717 & **0.239** & **0.204** & 0.142 & **0.108** & **0.442** & **0.739** & **0.297** & **0.274** & **0.165** & **0.136** \\ MTX-QAP & **0.414** & 0.715 & **0.247** & **0.206** & **0.149** & **0.113** & **0.439** & **0.757** & **0.294** & **0.267** & **0.157** & **0.139** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of our extended models for VidGAP and Classification based (CL) models across two datasets on our proposed Metric. Here, “R-” prefix implies it is the final metric computed after relative scoring and contrastive scoring with threshold 0. ”BS”: BertScore, “Cons’: Consistency on BertScore, B@2: Sentence BLEU-2, R: ROUGE, M: METEOR, C: CIDEr. Reported numbers are on the test set. For classification models, the number within the parenthesis denotes the size of fixed vocabulary of phrases. **Best result. Second Best result.**

\begin{table}
\begin{tabular}{l|l l l l l l l l} \hline  & & \multicolumn{6}{c|}{ASRL-QA} & \multicolumn{6}{c}{Chandrase-SRL-QA} \\  & ARG0 & V & ARG1 & ARG2 & LOC & V & ARG1 & ARG2 & LOC \\ \hline LungCL (\(1k\)) & 0.598 & 0.423 & 0.102 & 0.125 & 0.018 & 0.564 & 0.291 & 0.146 & 0.173 \\ MTLXL (\(0k\)) & 0.687 & 0.599 & 0.106 & 0.142 & 0.019 & 0.549 & 0.346 & 0.152 & 0.166 \\ MTXL (\(0k\)) & 0.697 & 0.396 & 0.161 & 0.044 & 0.019 & 0.014 & 0.315 & 0.152 & 0.272 \\ \hline Liang-QAP & **0.697** & **0.519** & 0.355 & 0.322 & 0.145 & **0.631** & 0.583 & 0.326 \\ BUTD-QAP & _0.681_ & 0.518 & **0.372** & **0.334** & 0.145 & 0.583 & 0.413 & 0.316 & 0.299 \\ VOO-QAP & _0.671_ & 0.513 & 0.366 & 0.332 & **0.185** & 0.63 & **0.647** & **0.365** & **0.305** \\ MTX-QAP & **0.729** & 0.478 & **0.374** & **0.344** & **0.17** & **0.632** & **0.458** & **0.364** & **0.304** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of our extended models per SRL. All reported scores are R-BS: BertScore computed after relative scoring and contrastive scoring with threshold 0.

 

in absolute terms remains very low. As such, our proposed task VidQAP remains extremely challenging for current multi-modal models.

**Evaluation Metric Scores:** In Table 4 we record the BertScore computation in three parts: directly computing over the answer phrases, performing relative scoring, finally performing contrastive scoring with different thresholds.

We observe that for V, naive computation leads to absurdly high scores. This is because verbs consist of a single word which means the embeddings are not contextual. This is remedied by relative scoring and is further controlled by combining with contrastive sampling.

Further note that relative scoring operates differently based on the SRLs. For instance, it increases the score for ARG0 and ARG1 where the answers more often paraphrased the ground-truth questions while for ARG2 and LOC, it decreases the score due to incorrect matches. While contrastive scoring is aimed at reducing language-only bias and as such should always reduce the relative score, we observe increased score in ARG2 for both Lang-QAP and MTX-QAP. This is caused by the \(max\) function which restricts the lower-limit to be \(0\).

**Effect of Region Boxes:** As noted earlier, the visual features can also include region features extracted from an object detector like FasterRCNN Ren et al. (2015). In Table 5 we record the effect of including regional features. In particular, we use the GT5 setting used in Sadhu et al. (2020) where \(5\) region proposals are used from \(10\) frames uniformly sampled from the video segment. Interestingly, MTX-QAP under-performs than both BUTD-QAP and VOG-QAP on ARGO. A possible reason is that the transformer is unable to effectively reason over both language and vision over such a large range of inputs.

## 5 Conclusion

In this work, we introduce Video Question Answering with Phrases (VidQAP) where we pose VidQA as a fill-in-the-phrase task. Given a video and query expression, a model needs to compose a sequence of words to answer. We then propose a method to leverage semantic roles from video descriptions to generate query expressions and outline a robust evaluation protocol. This involves computing the relative improvement of the prediction answer compared to an empty string followed by a contrastive sampling stage which reduces language-only biases. We then contribute two datasets ASRL-QA and Charades-SRL-QA to facilitate further on VidQAP and benchmark them with three vision-language models extended for our proposed task.

### Acknowledgement

We thank the anonymous reviewers for their suggestions and feedback. This research was supported, in part, by the Office of Naval Research under grant #N00014-18-1-2050.

### Ethics Statement

In this work, we propose an extension to the existing video question answering framework to include free-form answers and suggest how to evaluate such a task.

**Direct Application (Positive):** A direct application of our task would be to enrich existing descriptions obtained from video captioning models which could lead to better video retrieval results. For instance, one could query about what tool to use in order to cut a piece of cardboard by querying "A person cutting a piece of cardboard <Q-ARG2>".

**Direct Application (Negative):** Caution must be taken in directly applying models trained on descriptions without properly balancing the data-distributions as it is possible that hidden data-biases are amplified. As an example, ASRL-QA has many videos involving men throwing shot puts. As a result, a model could learn this biased correlation and whenever queried "who" (<Q-ARG0> throws a shot put) it would always produce the answer "man" even if the video clearly shows a "woman".

**Broader Societal Impacts (Positive):** Question answering is an excellent tool for diagnosing a model's understanding due to its high interactivity. Our proposed formulation takes this a step forward with answer phrases and can in-turn facilitate human-computer interactions. Our proposed model can be extended to down-stream tasks such as retrieving a video or retrieving a part of the video given a question or query.

**Broader Societal Impacts (Negative):** Since our method is agnostic to the end user case, it can be re-purposed to extract out sensitive information and be a threat to privacy 



## References

* Alamri et al. (2019) Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. 2019. Audio visual scene-aware dialog. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 7558-7567.
* Anderson et al. (2018) Peter Anderson, X. He, C. Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 6077-6086.
* Baker et al. (1998) Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In _36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1_, pages 86-90.
* Banerjee and Lavie (2005) S. Banerjee and A. Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In _IEEvaluation@ACL_.
* Bonial et al. (2012) Claire Bonial, Jena Hwang, Julia Bonn, Kathryn Conger, Olga Babko-Malaya, and Martha Palmer. 2012. English propbank annotation guidelines. _Center for Computational Language and Education Research Institute of Cognitive Science University of Colorado at Boulder_, 48.
* Chen et al. (2015) Xinlei Chen, H. Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, P. Dollar, and C. L. Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. _ArXiv_, abs/1504.00325.
* Denkowski and Lavie (2014) Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In _Proceedings of the EACL 2014 Workshop on Statistical Machine Translation_.
* Devlin et al. (2019) J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _NAACL-HLT_.
* Gardner et al. (2017) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: A deep semantic natural language processing platform. In _Arxiv_.
* Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and D. Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 6325-6334.
* Gupta and Malik (2015) Saurabh Gupta and Jitendra Malik. 2015. Visual semantic role labeling. _ArXiv_, abs/1505.04474.
* Heilman and Smith (2009) Michael Heilman and Noah A Smith. 2009. Question generation via overgenerating transformations and ranking. Technical report, Carnegie-Mellon Univ Pittsburgh pa language technologies insT.
* Hudson and Manning (2019) Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. _Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Jang et al. (2017) Y. Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1359-1367.
* Jasani et al. (2019) Bhavan Jasani, Rohit Girdhar, and D. Ramanan. 2019. Are we asking the right questions in movieqa? _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_, pages 1879-1882.
* Joshi et al. (2019) Mandar Joshi, Danqi Chen, Y. Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. _Transactions of the Association for Computational Linguistics_, 8:64-77.
* Kingsbury and Palmer (2002) Paul R Kingsbury and Martha Palmer. 2002. From treebank to propbank. In _LREC_, pages 1989-1993. Citeseer.
* Krishna et al. (2017) R. Krishna, Kenji Hata, F. Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-captioning events in videos. _2017 IEEE International Conference on Computer Vision (ICCV)_, pages 706-715.
* Krishna et al. (2016) R. Krishna, Yuke Zhu, O. Groth, J. Johnson, Kenji Hata, J. Kravitz, Stephanie Chen, Yannis Kalantidis, L. Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2016. Visual genome: Connecting language and vision using crowdsourced dense image annotations. _International Journal of Computer Vision_, 123:32-73.
* Lee et al. (2017) Kenton Lee, Luheng He, M. Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In _EMNLP_.
* Lei et al. (2018) Jie Lei, Licheng Yu, Mohit Bansal, and T. Berg. 2018. TVqa: Localized, compositional video question answering. In _EMNLP_.
* Li et al. (2020) Manling Li, Alireza Zareian, Q. Zeng, Spencer Whitehead, Di Lu, Huai zhong Ji, and Shih-Fu Chang. 2020. Cross-media structured common space for multimedia event extraction. In _ACL_.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _ACL 2004_.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In _Advances in Neural Information Processing Systems_, volume 32. Curran Associates, Inc.
* Maharaj et al. (2017) Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron C. Courville, and C. Pal. 2017. A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering. _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 7359-7368.
* Miech et al. (2020) Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, I. Laptev, Josef Sivic, and Andrew Zisserman. 2020. End-to-end learning of visual representations from uncurated instructional videos. _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 9876-9886.
* Micch et al. (2019) Antoine Michel, D. Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, I. Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_, pages 2630-2640.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of NAACL-HLT 2019: Demonstrations_.
* Papineni et al. (2002) Kishore Papineni, S. Roukos, T. Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _ACL_.
* Paszke et al. (2017) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alche-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_, pages 8024-8035. Curran Associates, Inc.
* Pennington et al. (2014) Jeffrey Pennington, R. Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_.
* Pradhan et al. (2013) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Bjorkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning_, pages 143-152.
* Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross B. Girshick, and J. Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 39:1137-1149.
* Sadhu et al. (2020) Arka Sadhu, K. Chen, and R. Nevatia. 2020. Video object grounding using semantic roles in language description. _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 10414-10424.
* Shi and Lin (2019) Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling. _arXiv preprint arXiv:1904.05255_.
* Sigurdsson et al. (2016) Gunnar A. Sigurdsson, Gul Varol, X. Wang, Ali Farhadi, I. Laptev, and A. Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In _ECCV_.
* Silberer and Pinkal (2018) Carina Silberer and Manfred Pinkal. 2018. Grounding semantic roles in images. In _EMNLP_.
* Strubell et al. (2018) Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 5027-5038, Brussels, Belgium. Association for Computational Linguistics.
* Tapaswi et al. (2016) Makarand Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler. 2016. Movieqa: Understanding stories in movies through question-answering. _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 4631-4640.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Vedantam et al. (2015) Ramakrishna Vedantam, C. L. Zitnick, and D. Parikh. 2015. Clider: Consensus-based image description evaluation. _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 4566-4575.
* Wang et al. (2016) L. Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, D. Lin, X. Tang, and L. Gool. 2016. Temporal segment networks: Towards good practices for deep action recognition. In _ECCV_.
* Xie et al. (2018) Saining Xie, C. Sun, J. Huang, Zhuowen Tu, and Kevin Murphy. 2018. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In _ECCV_.
* Xu et al. (2017) Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined attention over appearance and motion. In _ACM Multimedia_ 

Jianing Yang, Yuying Zhu, Yongxin Wang, Ruitao Yi, Amir Zadeh, and Louis-Philippe Morency. 2020. What gives the answer away? question answering bias analysis on video qa datasets. _arXiv:2007.03626 [cs, stat]_.
* Yatskar et al. (2016) Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual semantic role labeling for image understanding. In _Conference on Computer Vision and Pattern Recognition_.
* Yu et al. (2019) Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. In _AAAI_, pages 9127-9134.
* Zellers et al. (2019) Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 6713-6724.
* Zeng et al. (2017) Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and Min Sun. 2017. Leveraging video descriptions to learn video question answering. In _AAAI_.
* Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In _International Conference on Learning Representations_.
* Zhu and Yang (2020) Linchao Zhu and Yi Yang. 2020. Actbert: Learning global-local video-text representations. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.

 



## Appendix A Dataset Construction

We first discuss semantic-role labeling used in natural language processing. Then, we detail the dataset construction process used for ASRL-QA and Charades-SRL-QA (Section A.2) and then provide the dataset statistics (Section A.3).

### Semantic Role Labeling

Semantic-Role Labels extract out high-level meanings from a natural language description. Two widely used SRL annotations are PropBank Kingsbury and Palmer (2002) and FrameNet Baker et al. (1998). Here we use SRLs which follow PropBank annotation guidelines (see Bonial et al. (2012) for complete guideline).

Most commonly used argument roles are

* V: the verb. All remaining roles are dependent on this verb. While the numbered arguments differ slightly based on the verb used, they share common themes across verbs as listed below (see Bonial et al. (2012) for full details). For instance, "cut" is a Verb.
* ARG0: the agent, or the one causing the verb. For most action verbs, this is usually a human or an animal. For instance, "A person cuts a vegetable", "A person" is ARGO.
* ARG1: the object, on which the action is being performed. In "A person cuts a vegetable", "a vegetable" is ARG1.
* ARG2: the tool being used for the verb, or someone who benefits from the verb. For instance, in "A person is cutting a vegetable with a knife", "with a knife" denotes the tool and is ARG2. In "A person throws a basketball to the basket", "to the basket" denotes the benefactor and is ARG2. Footnote 5: [https://github.com/TheShadow29/Video-QAP](https://github.com/TheShadow29/Video-QAP)
* ARGM-LOC or simply LOC denotes the place or location where the verb takes place. For instance, in "A person is cutting a vegetable on a plate", "on a plate" is the LOC.

To assign SRLs to language descriptions we use allennlp library Gardner et al. (2017) which provides an implementation of a BERT Devlin et al. (2019) based semantic-role labeler Shi and Lin (2019). The system achieves \(86.49\) F1 score on OntoNotes Pradhan et al. (2013) 5.0 dataset.

### Construction Process

Both ASRL-QA and Charades-SRL-QA follow the same process with few subtle differences:

1. Pre-Process Data: * Assign semantic role labels (SRLs) to video descriptions using SRL labeller Shi and Lin (2019). * Remove stopword verbs with lemmas; "be", "start", "end", "begin", "stop", "lead", "demonstrate", "do". * For the original descriptions spread across multiple video segments, combine the sentences into a document. Use a coherence resolution model on this model (we use Lee et al. (2017) with SpanBERT embeddings Joshi et al. (2019) provided in allennlp library Gardner et al. (2017)). * Replace the following pronouns: "they", "he", "she", "his", "her", "it" with the relevant noun-phrase obtained from the co-reference resolution output.
2. Query-Generation: * For each verb-role set within a description (each description can have multiple verbs), consider the role set ARG0, ARG1, V, ARG2, LOC for ASRL-QA and ARG1, V, ARG2, LOC for Charades-SRL-QA. * If there are at least \(3\) verb-roles for the given verb, for each SRL replace it with a query token (with <Q-{R}> where \(R\) is the role). This forms one query. Repeat for all SRLs in the considered set. * The minimum of \(3\) verb-roles is present to avoid ambiguity in the query. Limiting the argument role-set helps in generating queries less likely to have strong

language-priors (though as seen in qualitative examples, some priors are still present).
* After the queries are generated, create lemmatized verbs, and nouns set for each query, and store the video segment ids in a dictionary. This is similar to the process used in Sadhu et al. (2020), with the difference that we additionally have query-tokens.
* For each query, use the dictionary to sample set of video segment ids which share the same semantic role structure, but for the query-token have a different answer. These are used for matching when computing the scores for the validation and testing set using the contrastive score.
3. Creating Train/Test Splits: * Keep the training set for each dataset the same. * For validation and testing, we split the dataset based on the video ids (half video ids are set as validation, and half as testing). The queries are then split based on the video ids. * Note that while contrastive sampling is done before validation test split. So validation and test ids are used for computing the other's score for contrastive sampling. This is similar to the setting used in Sadhu et al. (2020) as the total number of videos available for validation, and testing are insufficient for contrastive sampling.

### Dataset Statistics

Dataset statistics can be found in Table 1. Lemma distributions are visualized in Figure 1 Overall, we find slightly skewed distribution of Argument roles across the datasets. For instance, ARG0, ARG1 are much more frequent than ARG2 and LOC. Also, since every SRL needs to have a verb (V), the distribution of the videos is the same as the overall.

As shown in Table 1, vocabularies in both the train and validation/test sets for each argument role (slot) are reasonably large compared (eg. 60% for ARG1) to the total vocabulary and not too limited. This results is further consistent across both datasets.

## Appendix B Implementation Details

We first report the implementation details for the metrics (Section B.1). Then, we detail the model implementation details (Section B.2).

### Metric Implementation

For Bleu Papineni et al. (2002), Rouge Lin (2004), Meteor Banerjee and Lavie (2005), and CIDEr Vedantam et al. (2015) we use the implementations

Figure 1: Lemma Distribution for both ASRL-QA and Charades-SRL-QA. The number of instances across the whole dataset are given in the parenthesis of each lemmatized noun or verb.

 

provided in coco-captions repository6[Chen et al., 2015].

Footnote 6:ithub url: [https://github.com/tylin/coco-caption](https://github.com/tylin/coco-caption)

For BERTScore we use the official implementation7

Footnote 7:ithub url: [https://github.com/Tiiiger/bert_score](https://github.com/Tiiiger/bert_score)

**BLEU-2:** computes Bleu with n-gram with \(n{=}2\). We use sentence-bleu score instead of the more commonly used corpus bleu score. This is further used for contrastive sampling.

**ROUGE:** we use ROUGE-L which computes the longest common sub-sequence.

**METEOR:** we use Meteor 1.5 version [Denkowski and Lavie, 2014].

**CIDEr:** we use CIDEr-D implementation which includes idf-weighting.

**BertScore:** we use BertScore with hash "roberta

\begin{table}
\begin{tabular}{l|r r r r|r r r} \hline \hline  & & \multicolumn{3}{c|}{ASRL-QA} & \multicolumn{3}{c}{Charades-SRL-QA} \\ \hline \multirow{4}{*}{Overall} & \multirow{2}{*}{Videos} & Train & Val & Test & Train & Val & Test \\  & & 30337 & 2729 & 2739 & 7733 & 860 & 876 \\  & Queries & 147439 & 7414 & 7238 & 59329 & 4431 & 4520 \\  & Query Length & 8.03 & 6.03 & 6 & 7.11 & 5.6 & 5.62 \\  & Answer Length & 2.2 & 2.33 & 2.33 & 1.83 & 1.96 & 1.94 \\  & Vocabulary & 4597 & 3261 & 3261 & 1479 & 884 & 884 \\ ARG0 & Videos & 24483 & 1372 & 1419 & & & \\  & Queries & 37218 & 1603 & 1643 & & & \\  & Query Length & 7.31 & 5.73 & 5.65 & & & \\  & Answer Length & 2.51 & 2.37 & 2.48 & & & \\  & Vocabulary & 1763 & 840 & 840 & & & \\ V & Videos & 29922 & 1737 & 1733 & 7733 & 802 & 811 \\  & Queries & 52447 & 2247 & 2187 & 27745 & 1824 & 1829 \\  & Query Length & 9.2 & 7.26 & 7.18 & 7.7 & 6.37 & 6.44 \\  & Answer Length & 1 & 1 & 1 & 1 & 1 & 1 \\  & Vocabulary & 1860 & 1167 & 1167 & 678 & 377 & 377 \\ ARG1 & Videos & 24863 & 1810 & 1793 & 7600 & 808 & 828 \\  & Queries & 36787 & 2250 & 2179 & 21557 & 1857 & 1874 \\  & Query Length & 7.4 & 5.4 & 5.43 & 6.43 & 5.07 & 5.04 \\  & Answer Length & 2.8 & 2.82 & 2.83 & 2.31 & 2.39 & 2.39 \\  & Vocabulary & 3560 & 2124 & 2124 & 935 & 527 & 527 \\ ARG2 & Videos & 12048 & 850 & 805 & 5433 & 490 & 522 \\  & Queries & 14321 & 941 & 886 & 8279 & 651 & 699 \\  & Query Length & 7.49 & 5.45 & 5.36 & 6.94 & 5.13 & 5.13 \\  & Answer Length & 3.55 & 3.69 & 3.62 & 3.11 & 3.22 & 3.04 \\  & Vocabulary & 2607 & 1326 & 1326 & 556 & 365 & 365 \\ LOC & Videos & 6025 & 340 & 319 & 1578 & 87 & 112 \\  & Queries & 6666 & 373 & 343 & 1748 & 99 & 118 \\  & Query Length & 7.57 & 5.17 & 5.35 & 6.93 & 4.75 & 5.06 \\  & Answer Length & 3.61 & 3.87 & 3.63 & 3.22 & 3.19 & 3.08 \\  & Vocabulary & 1390 & 669 & 669 & 265 & 138 & 138 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Detailed dataset statistics for both ASRL-QA and Charades-SRL-QA with respect to different argument roles. Recall that ARGO is not present in Charades-SRL-QA, and hence the corresponding rows are kept blank.

 

large_L17_idf_version=0.3.5(hug_trans=3.0.2)-rescaled"

We show examples of computing the metrics.

### Model Implementation

We report all model implementation details.

**General Settings:** Our code is implemented using Pytorch [46]. For Transformer, we use the implementation provided in FairSeq [9]. The vocabulary consists of \(5\)k words for ASRL-QA and \(3\)k words for Charades-SRL-QA. The segment features are of dimension \(3072\) and \(512\) for ASRL-QA and Charades-SRL-QA respectively obtained from TSN [48] and S3D [21] trained on HowTo100M [11] using the loss function presented in [11]8. The proposal features are of dimension \(1024\) and only used for ASRL-QA extracted using FasterRCNN [10] trained on Visual Genome [12].

Footnote 8: [https://github.com/antoine77340/S3D_HowTo100M](https://github.com/antoine77340/S3D_HowTo100M)

For all cases, we report the output dimension of MLP. Unless otherwise stated, MLP is followed by ReLU activation.

**Decoder:** The decoder uses an input of \(T\times 512\) (where \(T\) refers to the length of the input embedding). Note that for Lang-QAP, \(T\) is same as sequence length of the query, for BUTD-QAP \(T{=}1\), for VGG-QAP, \(T\) is number of SRLs \(*\) number of segment features. For MTX-QAP, \(T\) is sequence length of query \(+\) number of segment features. To generate output sequences, we use the usual beam-search with a beam-size of \(2\), with a temperature of \(1.0\).

**Encoder:** Encoder differs based on the specific model. All encoders are transformer based using \(8\) attention heads and \(3\) layers unless otherwise mentioned.

**Lang-QAP**: The language encoder uses \(3\) encoding layers, with \(8\) attention heads each. The embedding layer uses a dimension of \(512\).

**BUTD-QAP**: We use the same language query, with and pre-pend a \([CLS]\) token. The embedding of the \([CLS]\) token serves as the language embedding, and is passed through a MLP of dimension \(512\). The language encoder is the same as Lang-QAP. The segment features are passed through MLP of dimension \(512\). If proposal features are used, they are passed through a separate MLP of dimension \(512\). The language embedding (also of dimension \(512\)) is used to compute attention score with the visual features, and finally obtain an attended visual feature. These attended visual features are concatenated with the language embedding along the last axis, and then passed to the decoder.

**VOG-QAP**: We use the same language encoder, but further use the SRL phrase start and endpoints for the phrase encoder. The phrase encoder uses these start and end points to gather the language embeddings corresponding to these start and end points, concatenate them (dimension \(512{+}512{=}1024\)) and use MLP with dimension \(512\). This gives an output of the phrase encoder of size number of SRLs \(*s512\). The phrase encoded query is then concatenated with all the segment features and passed through a MLP. Finally a multi-modal transformer encoder is applied over the phrase encoded input, and is passed to the language decoder.

**MTX-QAP**: We collate all the language tokens (passed through embedding layer) as well as segment features passed through MLP, to get all features of dimension \(512\). A transformer based encoder is applied on these features, and the output is passed to the decoder.

**Training**: We train using standard cross-entropy loss. The decoder is trained using teacher forcing. All models are trained for \(10\) epochs with batch size of \(32\). On a TitanX, for ASRL-QA each epoch takes around \(30-40\) mins. Our training infrastructure included a 8 GPU Titan X machine

## Appendix C Visualization

We visualize the model outputs on ASRL-QA in Figure 2 (a), (b), Figure 3 (a), (b) and Figure 4. For each case, we show the considered input in the first row, and the contrastive sample in the second row. Each row contains \(5\) frames uniformly sampled from the video segment to be representative of the content observed by the model. For every query, we show the ground-truth answer and the outputs from Lang-QAP, BUTD-QAP, VOG-QAP and MTX-QAP.

Overall, we often find Lang-QAP suggesting very probable answers, but as expected they are not grounded in the video. As a result, in either of the original sample or the contrastive sample, it performs poorly.

 

* [258] M. Cacciari, G. P. Salambi, and A. Zisserman. _The Large Hadron Collider: The LHC Run II Detector at the CERN_. PhD thesis, Stanford University, 1973.
* [259] J. Alwall et al., "MadGraph -- a simulation toolkit for hadronic physics", _JHEP_**06** (2014) 040, doi:10.1088/1126-6708/2006/06/040, arXiv:1406.0270.
* [260] S. Alioli, R. Frederix, F. Piccinini, and E. Salvati. _A Brief Introduction to PYTHIA 8.1_. Comput. Phys. Commun. **188** (2015) 159, doi:10.1016/j.cpc.2015.01.024, arXiv:1406.0270.
* [261] T. Sjostrand, S. Mrenna, and P. Skands. _PYTHIA 6.1_ (2006) P0602, doi:10.1088/1126-6708/2006/06/P0602, arXiv:hep-ph/0603175.
* [262] CMS Collaboration, "Event generator tunes obtained from underlying event and multiparton scattering measurements", _Eur. Phys. J. C_ **76** (2016) 155, doi:10.1140/epjc/s10052-016-3988-x, arXiv:1512.0015.
* [263] GEANT4 Collaboration, "GEANT4--a simulation toolkit", _Nucl. Instrum. Meth. A_ **506** (2003) 250, doi:10.1016/S0168-9002(03)01368-8.
* [264] I. Lokhtinen, V. Khozei, and B. Wattler. _Pythia 6.4: A Monte Carlo program for parton shower simulations_. Nucl. Instrum. Meth. A **506** (2003) 250, doi:10.1016/S0168-9002(03)01368-X.

[MISSING_PAGE_POST]

 

* [25] M. Cacciari, G. P. Salambi, and A. Zisserman. _The automated computation of tree-level programs in the MSSM: a general framework for implementing the Higgs sector_. PhD thesis, University of California, Berkeley, CA (1987)
* [26] J. Alwall et al., "Software for the MSSM: 

* [25] M. Cacciari, G. P. Salambi, and A. Zisserman. _The Large Hadron Collider: The LHC Run II: the Tevatron Experiment at CERN_, (2016). arXiv:1703.04985v1 [hep-ex].
* [26] J. Alwall et al., "MadGraph 5: going beyond", _JHEP_**04** (2014) 040, doi:10.1088/1126-6708/2014/04/040, arXiv:1405.02701 [hep-ph].
* [27] S. Alioli, F. Maltoni, and R. Rattazzi. _Nucl. Phys. B_**506** (1998) 475, doi:10.1016/S0550-3213(98)90045-X.
* [28] T. Sjostrand, S. Mrenna, and P. Skands. _PYTHIA 6.4 Physics and Manual_, _JHEP_**05** (2006) 026, doi:10.1088/1126-6708/2006/05/026, arXiv:hep-ph/0603175.
* [29] CMS Collaboration, "Event generator tunes obtained from underlying event and multiparton scattering measurements", _Eur. Phys. J. C_**76** (2017) 155, doi:10.1140/epjc/s10052-017-1559-z, arXiv:1602.00921 [hep-ex].

[MISSING_PAGE_POST]