SlowGuess/NAACL_2021_OCR huggingface dataset:
Inputs include query expression  $\{w\}_{i = 1}^{L}$  ( $L$  is number of words), video segment features for  $F_{1}$  frames and optionally  $k$  RCNN features for  $F_{2}$  frames. In either case, frames are sampled uniformly from the video segment time-span. While the models differ in their encoding scheme, our language decoder model (Transformer based) used to generate the output answer phrase is kept same across all models with QAP suffix.
Lang-QAP: is a language-only (video-blind) model using only the query input. It uses Transformer based encoder to encode the query into  $\hat{q} \in \mathbb{R}^{L \times d}$ . The decoder subsequently uses the last layer output of the encoder (Figure5-(a)).
BUTD-QAP: Bottom-up-Top-Down (Anderson et al., 2018) is a popular approach for image question answering as well as captioning. It first computes attention between the question and the RCNN visual features to generate an attended visual feature, which is then used with the question to produce an output answer. Here, we replace the RCNN features with the segment features  $(\hat{v} \in \mathbb{R}^{F_1 \times d})$ . We can also include RCNN features by projecting them to same dimension as segment features and then concatenate them along the frame-axis  $(\hat{v} \in \mathbb{R}^{(F_1 + F_2*k) \times d})$ . For language features, we use the [CLS] token representation from the last layer of the language encoder used in Lang-QAP.
The output using the language and visual features is  $(\hat{m}\in \mathbb{R}^{d})$  passed to the decoder (Figure 5(b)).
VOG-QAP: VOGNet (Sadhu et al., 2020) has been proposed for grounding objects in videos given a natural language query. Following the architecture, we first derive phrase encoding which corresponds to a single SRL i.e.  $\hat{q} \in \mathbb{R}^{S \times d}$  ( $S$  is number of semantic roles). These phrase features are concatenated with the visual features (same as those used in BUTD-QAP (i.e.  $\hat{v}$ ) to get multi-modal features  $m[l,i] = [\hat{v}_i||\hat{q}_l]$  and then reshaped to get  $m \in \mathbb{R}^{S * F \times d}$ . These multi-modal features are subsequently passed to decoder to generate the output sequence (Figure 5 (c)).
MTX-QAP: Recently, transformer models pretrained on large-scale paired image-text data have become popular. Even in the absence of pretraining, such architectures can achieve competitive performance (Lu et al., 2019). In the context of videos, ActBert (Zhu and Yang, 2020) has been proposed. We create a similar architecture to ActBert but we replace their proposed TangledTransformer with a vanilla Transformer<sup>3</sup>. Specifically, we jointly encode the language and visual features in a single transformer and feed the output to the decoder (Figure 5 (d)).
LangCL and MTxCL: Apart from QAP models, we also consider their phrase classification counterparts where the decoder is replaced with a N-way classifier (two-layered MLP in our case) across a fixed set of phrases. For our experiments, we used  $N = 1k$  phrases for LangCL and  $N \in \{1k, 10k\}$  for MTxCL.
3The code for ActBert is not publicly available.
We briefly discuss the dataset creation process (Section 4.1), followed by experimental setup (Section 4.2). We then summarize our results (Section 4.3) and discuss key-findings. We provide implementation details, qualitative visualizations of our dataset, metrics and trained models in the appendix.
4.1 Dataset Creation
We create two datasets ASRL-QA and Charades-SRL-QA derived from ActivityNet-Captions (Krishna et al., 2017) and Charades (Sigurdsson et al., 2016) respectively.
There are three key steps to create QA datasets from descriptions: (i) assign semantic-roles to the descriptions (ii) perform co-reference resolution so that the questions are self-contained (iii) obtain lemmatized nouns and verbs to perform contrastive sampling. For semantic-role labeling, we use (Shi and Lin, 2019). For co-reference resolution, we use the co-reference resolution model provided by allennlp library (Gardner et al., 2017) which uses the model by (Lee et al., 2017) but replaces the GloVe (Pennington et al., 2014) embeddings with SpanBERT embeddings (Joshi et al., 2019) $^{4}$ .
Since Charades primarily involves videos with a single person, we discard questions involving ARG0. We limit to using a single description per video to avoid repetitive questions. We re-use the same train split for both datasets. For ASRL-QA, test set of ActivityNet is not public and Charades only has a test set but no official validation set. Thus, we split the existing validation set by video names and create the validation and test sets. For both validation and test splits, we remove those questions for which no contrastive sample was found as it indicates data-biases.
Dataset Statistics: ASRL-QA has 35.7k videos and 162k queries split into train, validation and test sets with 30.3k, 2.7k, 2.7k videos and 147k, 7.5k, 7.5k queries. We observe that the size of validation and test sets are proportionately smaller compared to their respective train sets. This is because only queries with corresponding contrastive sample are included while no such filtering is done for the train set ( $\sim$ 95k queries in train set have a contrastive pair). Charades-SRL-QA contains 9.4k videos and 71.7k queries split across train, validation and test
sets with 7.7k, 0.8k, 0.8k videos and 59.3k, 6.1k, 6.2k queries. Despite its smaller size, the size of validation, test sets of Charades-SRL-QA is comparable to ASRL-QA as Charades is curated with the goal of diversifying subject, verb, object tuples. Supplementary material provides further details on the dataset statistics and visualizations.
Evaluation Metrics: As discussed in Section 3.2, we report the combined metric (i.e. metrics prefixed with "R-") for the commonly used generation metrics: BLEU, METEOR, ROUGE, CIDEr and BertScore (implementations from (Chen et al., 2015; Zhang* et al., 2020)). For BLEU, we report the sentence level BLEU-2. All reported results are test set results using the model which performs best on validation set.
Table 2 compares performance of the proposed VidQAP models with N-way classification baselines (denoted with suffix "CL") on ASRL-QA and Charades-SRL-QA.
Comparing Metrics: It is evident that compared to other metrics, R-BertScore shows a higher relative improvement. This is because BertScore allows soft-matches by utilizing contextual embeddings obtained from a pre-trained BERT (Devlin et al., 2019) or Roberta (Liu et al., 2019) model.
Comparison Across Datasets: We find that performance on both datasets follow very similar trends across all metrics. Charades-SRL-QA has slightly higher scores compared to ASRL-QA likely because it has lesser data variations (Charades is mostly confined indoor videos) suggesting findings on either dataset would transfer.
Comparison within N-way Classification: We notice that when  $1k$  fixed set of phrases are used classification models show very limited performance. Allowing  $10k$  phrases gives a significant improvement in performance on Charades-SRL-QA (12 points on R-BS) however this doesn't translate to ASRL-QA. This is because ASRL-QA contains many more probable phrases  $(29K$  compared to  $8K)$  in their respective training sets. We also notice that increasing the number of phrases vocabulary coincides with decreasing consistency.
Comparing Free-from Answer Generation (QAP) with N-way Classification (CL): We investigate the advantages of using a decoder network to generate phrases compared to an N-way classification over a fixed set of phrases (denoted
<sup>4</sup>https://demo.allennlp.org/coreference-resolution
ASRL-QA	Charades-SRL-QA
R-BS	Cons	R-B@2	R-R	R-M	R-C	R-BS	Cons	R-B@2	R-R	R-M	R-C
LangCL (1k)	0.253	0.889	0.120	0.098	0.071	0.044	0.293	0.697	0.224	0.209	0.114	0.077
MTxCL (1k)	0.255	0.869	0.130	0.114	0.080	0.050	0.288	0.707	0.215	0.208	0.116	0.075
MTxCL (10k)	0.286	0.788	0.157	0.133	0.100	0.061	0.408	0.695	0.286	0.261	0.142	0.108
Lang-QAP	0.402	0.728	0.228	0.182	0.125	0.095	0.406	0.719	0.277	0.253	0.147	0.121
BUTD-QAP	0.413	0.716	0.237	0.203	0.147	0.105	0.399	0.714	0.271	0.231	0.115	0.105
VOG-QAP	0.414	0.717	0.239	0.204	0.142	0.108	0.442	0.739	0.297	0.274	0.165	0.136
MTX-QAP	0.414	0.715	0.247	0.206	0.149	0.113	0.439	0.757	0.294	0.267	0.157	0.139
Table 2: Comparison of our extended models for VidQAP and Classification based (CL) models across two datasets on our proposed Metric. Here, "R-" prefix implies it is the final metric computed after relative scoring and contrastive scoring with threshold 0. "BS": BertScore, "Cons": Consistency on BertScore, B@2: Sentence BLEU-2, R: ROUGE, M: METEOR, C: CIDEr. Reported numbers are on the test set. For classification models, the number within the parenthesis denotes the size of fixed vocabulary of phrases. Best result, Second Best result.
ASRL-QA	Charades-SRL-QA
ARG0	V	ARG1	ARG2	LOC	V	ARG1	ARG2	LOC
LangCL (1k)	0.598	0.423	0.102	0.125	0.018	0.564	0.291	0.146	0.173
MTxCL (1k)	0.607	0.399	0.106	0.142	0.019	0.549	0.346	0.152	0.106
MTxCL (10k)	0.697	0.379	0.161	0.144	0.049	0.601	0.445	0.315	0.272
Lang-QAP	0.697	0.519	0.325	0.322	0.145	0.631	0.458	0.33	0.206
BUTD-QAP	0.681	0.515	0.372	0.334	0.162	0.568	0.413	0.316	0.299
VOG-QAP	0.671	0.513	0.366	0.332	0.188	0.63	0.467	0.365	0.305
MTX-QAP	0.702	0.478	0.374	0.344	0.17	0.633	0.455	0.364	0.304
Table 3: Comparison of our extended models per SRL. All reported scores are R-BS: BertScore computed after relative scoring and contrastive scoring with threshold 0.
ARG0	V	ARG1	ARG2	LOC
Lang-QAP	Direct	0.552	0.9268	0.234	0.302	0.216
Rel Score	0.7	0.534	0.332	0.237	0.1
CS@0	0.697	0.519	0.325	0.322	0.145
CS@0.1	0.69	0.492	0.295	0.28	0.132
CS@0.2	0.68	0.459	0.262	0.212	0.106
CS@0.3	0.657	0.423	0.219	0.149	0.085
MTX-QAP	Direct	0.566	0.929	0.269	0.321	0.258
Rel Score	0.706	0.488	0.366	0.25	0.14
CS@0	0.702	0.478	0.374	0.344	0.17
CS@0.1	0.693	0.45	0.343	0.305	0.145
CS@0.2	0.681	0.413	0.306	0.239	0.117
CS@0.3	0.659	0.376	0.27	0.17	0.08
with the suffix "CL" and number of phrases used in parenthesis). Table 2 shows that both Lang-QAP and MTX-QAP outperform their classification counterparts, namely Lang-CL and MTX-CL on both datasets. This implies the free-form generation are not limited to simply generating the most frequently appearing phrases in the training set, thereby showing its effectiveness.
Comparison Across Models: We find that multi-modal models outperform language-only baseline. However, the improvement over language baseline is small. The reason for the small gap is elucidate in Table 3 where we report R-BertScore
Table 4: BertScore Metrics computed Directly on answer phrases. Rel Score: After Relative Scoring. CS@T: Contrastive scoring with threshold T.
ARG0	V	ARG1	ARG2	LOC	Overall
BUTD-QAP	0.706	0.506	0.388	0.36	0.196	0.431
VOG-QAP	0.704	0.516	0.366	0.352	0.202	0.429
MTX-QAP	0.685	0.465	0.378	0.355	0.19	0.416
Table 5: Effect of Adding Region Proposals. All reported scores are R-BS. Best result, Second Best result.
for every considered SRL.
We find a large disparity in performance depending on the SRL. Most strikingly, multi-modal models perform worse than language-only model on ARG0 and V. For ARG0, the strong performance of the Lang-QAP arises because most of the time the agent who causes an action is a human. Therefore answer phrases having simply "A man" or "A woman" or "A person" leads to reasonable performance. This additionally suggests that grounding "who" is performing the action remains non-trivial.
The more surprising result is the strong performance of Lang-QAP on V which is consistent across both datasets despite using contrastive sampling. There are two likely causes. First, the distinction between verbs is not as strict as object nouns, i.e. even similar verbs are classified as a separate verb diminishing the returns of contrastive sampling. For instance, "jumping" and "hoping" have different lemma and thus considered distinct verbs but R-BS would treat them as similar even if the specific action would be classified "jumping" rather than "hoping". Second, SRLs such as ARG1 confines the set of possible verbs. For instance, if the object is "glass", only limited verbs such as "drink", "hold" are probable.
On the remaining arguments namely ARG1, ARG2, and LOC, multi-modal models show a steady improvement over language-only baseline ranging from  $1 - 10\%$ . However, the performance
in absolute terms remains very low. As such, our proposed task VidQAP remains extremely challenging for current multi-modal models.
Evaluation Metric Scores: In Table 4 we record the BertScore computation in three parts: directly computing over the answer phrases, performing relative scoring, finally performing contrastive scoring with different thresholds.
We observe that for  $\mathrm{v}$ , naive computation leads to absurdly high scores. This is because verbs consist of a single word which means the embeddings are not contextual. This is remedied by relative scoring and is further controlled by combining with contrastive sampling.
Further note that relative scoring operates differently based on the SRLs. For instance, it increases the score for ARG0 and ARG1 where the answers more often paraphrased the ground-truth questions while for ARG2 and LOC, it decreases the score due to incorrect matches. While contrastive scoring is aimed at reducing language-only bias and as such should always reduce the relative score, we observe increased score in ARG2 for both Lang-QAP and MTX-QAP. This is caused by the max function which restricts the lower-limit to be 0.
Effect of Region Boxes: As noted earlier, the visual features can also include region features extracted from an object detector like FasterRCNN (Ren et al., 2015). In Table 5 we record the effect of including regional features. In particular, we use the GT5 setting used in (Sadhu et al., 2020) where 5 region proposals are used from 10 frames uniformly sampled from the video segment. Interestingly, MTX-QAP under-performs than both BUTD-QAP and VOG-QAP on ARG0. A possible reason is that the transformer is unable to effectively reason over both language and vision over such a large range of inputs.
In this work, we introduce Video Question Answering with Phrases (VidQAP) where we pose VidQA as a fill-in-the-phrase task. Given a video and query expression, a model needs to compose a sequence of words to answer. We then propose a method to leverage semantic roles from video descriptions to generate query expressions and outline a robust evaluation protocol. This involves computing the relative improvement of the prediction answer compared to an empty string followed by a contrastive sampling stage which reduces language
only biases. We then contribute two datasets ASRL-QA and Charades-SRL-QA to facilitate further on VidQAP and benchmark them with three vision-language models extended for our proposed task.
We thank the anonymous reviewers for their suggestions and feedback. This research was supported, in part, by the Office of Naval Research under grant #N00014-18-1-2050.
In this work, we propose an extension to the existing video question answering framework to include free-form answers and suggest how to evaluate such a task.
Direct Application (Positive): A direct application of our task would be to enrich existing descriptions obtained from video captioning models which could lead to better video retrieval results. For instance, one could query about what tool to use in order to cut a piece of cardboard by querying "A person cutting a piece of cardboard  $< \mathrm{Q - ARG2}>$ ".
Direct Application (Negative): Caution must be taken in directly applying models trained on descriptions without properly balancing the data-distributions as it is possible that hidden data-biases are amplified. As an example, ASRL-QA has many videos involving men throwing shot puts. As a result, a model could learn this biased correlation and whenever queried "who" (<Q-ARG0> throws a shot put) it would always produce the answer "man" even if the video clearly shows a "woman".
Broader Societal Impacts (Positive): Question answering is an excellent tool for diagnosing a model's understanding due to its high interactivity. Our proposed formulation takes this a step forward with answer phrases and can in-turn facilitate human-computer interactions. Our proposed model can be extended to down-stream tasks such as retrieving a video or retrieving a part of the video given a question or query.
Broader Societal Impacts (Negative): Since our method is agnostic to the end user case, it can be re-purposed to extract out sensitive information and be a threat to privacy.
Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. 2019. Audio visual scene-aware dialog. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7558-7567.
Peter Anderson, X. He, C. Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6077-6086.
Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 86-90.
S. Banerjee and A. Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In IEEvaluation@ACL.
Claire Bonial, Jena Hwang, Julia Bonn, Kathryn Conger, Olga Babko-Malaya, and Martha Palmer. 2012. English propbank annotation guidelines. Center for Computational Language and Education Research Institute of Cognitive Science University of Colorado at Boulder, 48.
Xinlei Chen, H. Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, P. Dollar, and C. L. Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. ArXiv, abs/1504.00325.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation.
J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: A deep semantic natural language processing platform. In *Arxiv*.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and D. Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6325-6334.
Saurabh Gupta and Jitendra Malik. 2015. Visual semantic role labeling. ArXiv, abs/1505.04474.
Michael Heilman and Noah A Smith. 2009. Question generation via overgenerating transformations and
ranking. Technical report, Carnegie-Mellon Univ Pittsburgh pa language technologies instT.
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR).
Y. Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatiotemporal reasoning in visual question answering. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1359-1367.
Bhavan Jasani, Rohit Girdhar, and D. Ramanan. 2019. Are we asking the right questions in movieqa? 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 1879-1882.
Mandar Joshi, Danqi Chen, Y. Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Paul R Kingsbury and Martha Palmer. 2002. From treebank to propbank. In LREC, pages 1989-1993. Cite-seer.
R. Krishna, Kenji Hata, F. Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-c captioning events in videos. 2017 IEEE International Conference on Computer Vision (ICCV), pages 706-715.
R. Krishna, Yuke Zhu, O. Groth, J. Johnson, Kenji Hata, J. Kravitz, Stephanie Chen, Yannis Kalantidis, L. Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2016. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123:32-73.
Kenton Lee, Luheng He, M. Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In EMNLP.
Jie Lei, Licheng Yu, Mohit Bansal, and T. Berg. 2018. Tvqa: Localized, compositional video question answering. In EMNLP.
Manling Li, Alireza Zareian, Q. Zeng, Spencer Whitehead, Di Lu, Huai zhong Ji, and Shih-Fu Chang. 2020. Cross-media structured common space for multimedia event extraction. In ACL.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In ACL 2004.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron C. Courville, and C. Pal. 2017. A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7359-7368.
Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, I. Laptev, Josef Sivic, and Andrew Zisserman. 2020. End-to-end learning of visual representations from uncurated instructional videos. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9876-9886.
Antoine Miech, D. Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, I. Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2630-2640.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Kishore Papineni, S. Roukos, T. Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Jeffrey Pennington, R. Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143-152.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and J. Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE