SlowGuess/NAACL_2021_OCR huggingface dataset:
Transactions on Pattern Analysis and Machine Intelligence, 39:1137-1149.
Arka Sadhu, K. Chen, and R. Nevatia. 2020. Video object grounding using semantic roles in language description. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10414-10424.
Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255.
Gunnar A. Sigurdsson, Gül Varol, X. Wang, Ali Farhadi, I. Laptev, and A. Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In ECCV.
Carina Silberer and Manfred Pinkal. 2018. Grounding semantic roles in images. In EMNLP.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5027-5038, Brussels, Belgium. Association for Computational Linguistics.
Makarand Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler. 2016. Movieqa: Understanding stories in movies through question-answering. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4631-4640.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Ramakrishna Vedantam, C. L. Zitnick, and D. Parikh. 2015. Cider: Consensus-based image description evaluation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566-4575.
L. Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, D. Lin, X. Tang, and L. Gool. 2016. Temporal segment networks: Towards good practices for deep action recognition. In ECCV.
Saining Xie, C. Sun, J. Huang, Zhuowen Tu, and Kevin Murphy. 2018. Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In ECCV.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia.
Jianing Yang, Yuying Zhu, Yongxin Wang, Ruitao Yi, Amir Zadeh, and Louis-Philippe Morency. 2020. What gives the answer away? question answering bias analysis on video qa datasets. arXiv:2007.03626 [cs, stat].
Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual semantic role labeling for image understanding. In Conference on Computer Vision and Pattern Recognition.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, pages 9127-9134.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6713-6724.
Kuo-Hao Zeng, Tseng-Hung Chen, Ching-Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and Min Sun. 2017. Leveraging video descriptions to learn video question answering. In AAAI.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
Linchao Zhu and Yi Yang. 2020. Actbert: Learning global-local video-text representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
This is the appendix for the paper "Video Question Answering with Phrases via Semantic Roles". The appendix provides details on
1. Dataset construction and Dataset statistics (Section A)
2. Implementation Details for both the Metrics as well as the Models (Section B).
3. Visualization of Model Outputs (Section C)
4. Code and Data are publicly available  ${}^{5}$  .
We first discuss semantic-role labeling used in natural language processing. Then, we detail the dataset construction process used for ASRL-QA and Charades-SRL-QA (Section A.2) and then provide the dataset statistics (Section A.3).
A.1 Semantic Role Labeling
Semantic-Role Labels extract out high-level meanings from a natural language description. Two widely used SRL annotations are PropBank (Kingsbury and Palmer, 2002) and FrameNet (Baker et al., 1998). Here we use SRLs which follow PropBank annotation guidelines (see (Bonial et al., 2012) for complete guideline).
Most commonly used argument roles are
- V: the verb. All remaining roles are dependent on this verb. While the numbered arguments differ slightly based on the verb used, they share common themes across verbs as listed below (see (Bonial et al., 2012) for full details). For instance, "cut" is a Verb.
- ARG0: the agent, or the one causing the verb. For most action verbs, this is usually a human or an animal. For instance, “A person cuts a vegetable”, “A person” is ARG0.
- ARG1: the object, on which the action is being performed. In “A person cuts a vegetable”, “a vegetable” is ARG1.
- ARG2: the tool being used for the verb, or someone who benefits from the verb. For instance, in "A person is cutting a vegetable with a knife", "with a knife" denotes the tool and is ARG2. In "A person throws a basketball to the basket", "to the basket" denotes the benefactor and is ARG2.
- ARGM-LOC or simply LOC denotes the place or location where the verb takes place. For instance, in "A person is cutting a vegetable on a plate", "on a plate" is the LOC.
To assign SRLs to language descriptions we use allennlp library (Gardner et al., 2017) which provides an implementation of a BERT (Devlin et al., 2019) based semantic-role labeler (Shi and Lin, 2019). The system achieves 86.49 F1 score on OntoNotes (Pradhan et al., 2013) 5.0 dataset.
A.2 Construction Process
Both ASRL-QA and Charades-SRL-QA follow the same process with few subtle differences:
1. Pre-Process Data:
- Assign semantic role labels (SRLs) to video descriptions using SRL labeller (Shi and Lin, 2019).
- Remove stopwords verbs with lemmas: "be", "start", "end", "begin", "stop", "lead", "demonstrate", "do".
- For the original descriptions spread across multiple video segments, combine the sentences into a document. Use a co-reference resolution model on this model (we use (Lee et al., 2017) with SpanBERT embeddings (Joshi et al., 2019) provided in allennlp library (Gardner et al., 2017)).
- Replace the following pronouns: "they", "he", "she", "his", "her", "it" with the relevant noun-phrase obtained from the co-reference resolution output.
2. Query-Generation:
- For each verb-role set within a description (each description can have multiple verbs), consider the role set ARG0, ARG1, V, ARG2, LOC for ASRL-QA and ARG1, V, ARG2, LOC for Charades-SRL-QA.
- If there are at least 3 verb-roles for the given verb, for each SRL replace it with a query token (with  $< \mathbb{Q} - \{\mathbb{R}\} >$  where  $R$  is the role). This forms one query. Repeat for all SRLs in the considered set.
- The minimum of 3 verb-roles is present to avoid ambiguity in the query. Limiting the argument role-set helps in generating queries less likely to have strong
$^{5}$ https://github.com/TheShadow29/Video-QAP
language-priors (though as seen in qualitative examples, some priors are still present).
- After the queries are generated, create lemmatized verbs, and nouns set for each query, and store the video segment ids in a dictionary. This is similar to the process used in (Sadhu et al., 2020), with the difference that we additionally have query-tokens.
- For each query, use the dictionary to sample set of video segment ids which share the same semantic role structure, but for the query-token have a different answer. These are used for matching when computing the scores for the validation and testing set using the contrastive score.
3. Creating Train/Test Splits:
- Keep the training set for each dataset the same.
- For validation and testing, we split the dataset based on the video ids (half video ids are set as validation, and half as testing). The queries are then split based on the video ids.
- Note that while contrastive sampling is done before validation test split. So validation and test ids are used for computing the other's score for contrastive sampling. This is similar to the setting used in (Sadhu et al., 2020) as the total number of videos available for validation, and testing are insufficient for contrastive sampling.
A.3 Dataset Statistics
Dataset statistics can be found in Table 1. Lemma distributions are visualized in Figure 1 Overall, we find slightly skewed distribution of Argument roles across the datasets. For instance, ARG0, ARG1 are much more frequent than ARG2 and LOC. Also, since every SRL needs to have a verb (V), the distribution of the videos is the same as the overall.
As shown in Table 1, vocabularies in both the train and validation/test sets for each argument role (slot) are reasonably large compared (eg.  $60\%$  for ARG1) to the total vocabulary and not too limited. This results is further consistent across both datasets.
(a) Top-5 lemmatized nouns or verbs for the considered semantic roles in ASRL-QA
(b) Top-5 lemmatized nouns or verbs for the considered semantic roles in Charades-SRL-QA
Figure 1: Lemma Distribution for both ASRL-QA and Charades-SRL-QA. The number of instances across the whole dataset are given in the parenthesis of each lemmatized noun or verb.
We first report the implementation details for the metrics (Section B.1). Then, we detail the model implementation details (Section B.2).
B.1 Metric Implementation
For Bleu (Papineni et al., 2002), Rouge (Lin, 2004), Meteor (Banerjee and Lavie, 2005), and CIDEr (Vedantam et al., 2015) we use the implementations
ASRL-QA	Charades-SRL-QA
Overall	Videos	Train	Val	Test	Train	Val	Test
Queries	30337	2729	2739	7733	860	876
Query Length	147439	7414	7238	59329	4431	4520
Answer Length	8.03	6.03	6	7.11	5.6	5.62
Vocabulary	2.2	2.33	2.33	1.83	1.96	1.94
ARG0	Videos	24483	1372	1419
Queries	37218	1603	1643
Query Length	7.31	5.73	5.65
Answer Length	2.51	2.37	2.48
Vocabulary	1763	840	840
V	Videos	29922	1737	1733	7733	802	811
Queries	52447	2247	2187	27745	1824	1829
Query Length	9.2	7.26	7.18	7.7	6.37	6.44
Answer Length	1	1	1	1	1	1
Vocabulary	1860	1167	1167	678	377	377
ARG1	Videos	24863	1810	1793	7600	808	828
Queries	36787	2250	2179	21557	1857	1874
Query Length	7.4	5.4	5.43	6.43	5.07	5.04
Answer Length	2.8	2.82	2.83	2.31	2.39	2.39
Vocabulary	3560	2124	2124	935	527	527
ARG2	Videos	12048	850	805	5433	490	522
Queries	14321	941	886	8279	651	699
Query Length	7.49	5.45	5.36	6.94	5.13	5.13
Answer Length	3.55	3.69	3.62	3.11	3.22	3.04
Vocabulary	2607	1326	1326	556	365	365
LOC	Videos	6025	340	319	1578	87	112
Queries	6666	373	343	1748	99	118
Query Length	7.57	5.17	5.35	6.93	4.75	5.06
Answer Length	3.61	3.87	3.63	3.22	3.19	3.08
Vocabulary	1390	669	669	265	138	138
provided in coco-captions repository $^6$  (Chen et al., 2015).
For BERTScore we use the official implementation
BLEU-2: computes Bleu with n-gram with  $n = 2$ . We use sentence-bleu score instead of the more commonly used corpus bleu score. This is further
used for contrastive sampling.
ROUGE: we use ROUGE-L which computes the longest common sub-sequence.
METEOR: we use Meteor 1.5 version (Denkowski and Lavie, 2014).
CIDEr: we use CIDEr-D implementation which includesidf-weighting.
Table 1: Detailed dataset statistics for both ASRL-QA and Charades-SRL-QA with respect to different argument roles. Recall that ARG0 is not present in Charades-SRL-QA, and hence the corresponding rows are kept blank.
BertScore:	we	use
BertScore	with	hash	“roberta-
$^{6}$ github url: https://github.com/tylin/coco-caption
$^{7}$ github url: https://github.com/Tiiger/bert_score
large_L17_idf_version=0.3.5(hug_trans=3.0.2)-rescaled"
We show examples of computing the metrics.
B.2 Model Implementation
We report all model implementation details.
General Settings: Our code is implemented using Pytorch (Paszke et al., 2019). For Transformer, we use the implementation provided in FairSeq (Ott et al., 2019). The vocabulary consists of 5k words for ASRL-QA and 3k words for Charades-SRL-QA. The segment features are of dimension 3072 and 512 for ASRL-QA and Charades-SRL-QA respectively obtained from TSN (Wang et al., 2016) and S3D (Xie et al., 2018) trained on HowTo100M (Miech et al., 2019) using the loss function presented in (Miech et al., 2020)  $^{8}$ . The proposal features are of dimension 1024 and only used for ASRL-QA extracted using FasterRCNN (Ren et al., 2015) trained on Visual Genome (Krishna et al., 2016).
For all cases, we report the output dimension of MLP. Unless otherwise stated, MLP is followed by ReLU activation.
Decoder: The decoder uses an input of  $T \times 512$  (where  $T$  refers to the length of the input embedding). Note that for Lang-QAP,  $T$  is same as sequence length of the query, for BUTD-QAP  $T = 1$ , for VOG-QAP,  $T$  is number of SRLs * number of segment features. For MTX-QAP,  $T$  is sequence length of query + number of segment features. To generate output sequences, we use the usual beam-search with a beam-size of 2, with a temperature of 1.0.
Encoder: Encoder differs based on the specific model. All encoders are transformer based using 8 attention heads and 3 layers unless otherwise mentioned.
Lang-QAP: The language encoder uses 3 encoding layers, with 8 attention heads each. The embedding layer uses a dimension of 512.
BUTD-QAP: We use the same language query, with and pre-pend a [CLS] token. The embedding of the [CLS] token serves as the language embedding, and is passed through a MLP of dimension 512. The language encoder is the same as Lang-QAP. The segment features are passed through MLP of dimension 512. If proposal features are used, they are passed through a separate MLP of dimension 512. The language embedding
(also of dimension 512) is used to compute attention score with the visual features, and finally obtain an attended visual feature. These attended visual features are concatenated with the language embedding along the last axis, and then passed to the decoder.
VOG-QAP: We use the same language encoder, but further use the SRL phrase start and endpoints for the phrase encoder. The phrase encoder uses these start and end points to gather the language embeddings corresponding to these start and end points, concatenate them (dimension  $512 + 512 = 1024$ ) and use MLP with dimension 512. This gives an output of the phrase encoder of size number of SRLs *s512. The phrase encoded query is then concatenated with all the segment features and passed through a MLP. Finally a multi-modal transformer encoder is applied over the phrase encoded input, and is passed to the language decoder.
MTX-QAP: We collate all the language tokens (passed through embedding layer) as well as segment features passed through MLP, to get all features of dimension 512. A transformer based encoder is applied on these features, and the output is passed to the decoder.
Training: We train using standard cross-entropy loss. The decoder is trained using teacher forcing. All models are trained for 10 epochs with batch size of 32. On a TitanX, for ASRL-QA each epoch takes around  $30 - 40$  mins. Our training infrastructure included a 8 GPU Titan X machine
We visualize the model outputs on ASRL-QA in Figure 2 (a), (b), Figure 3 (a), (b) and Figure 4. For each case, we show the considered input in the first row, and the contrastive sample in the second row. Each row contains 5 frames uniformly sampled from the video segment to be representative of the content observed by the model. For every query, we show the ground-truth answer and the outputs from Lang-QAP, BUTD-QAP, VOG-QAP and MTX-QAP.
Overall, we often find Lang-QAP suggesting very probable answers, but as expected they are not grounded in the video. As a result, in either of the original sample or the contrastive sample, it performs poorly.
$^{8}$ https://github.com/antoine77340/S3D_HowTo100M
Query:  $<  \mathrm{Q - ARG0}>$  play the song on the piano
Target Answer: A little girl
Lang-QAP: The man
BUTD-QAP: A young child
VOG-QAP: A woman
MTX-QAP: The woman
Query:  $<  \mathrm{Q - ARG0 >}$  playing a song
Target Answer: A man wearing a hat
Lang-QAP: A woman
BUTD-QAP: A man
VOG-QAP: A man wearing a hat
MTX-QAP: A man
(a) Query of type ARG0
Query: A man  $<  \mathrm{Q - V > }$  a skateboard
Target Answer: holding
Lang-QAP: riding
BUTD-QAP: picks
VOG-QAP: holding
MTX-QAP: holding
(b) Query of type V
Figure 2: Queries of Type ARG0 and V on ASRL-QA
Query: Men  $<  \mathrm{Q - V >}$  skateboards
Target Answer: riding
Lang-QAP: riding
BUTD-QAP: riding
VOG-QAP: riding
MTX-QAP: riding
Query: People hit  $<  \mathrm{Q - ARG1}>$
Target Answer: a pinata
Lang-QAP: the ball
BUTD-QAP: the pinata
VOG-QAP: the pinata
MTX-QAP: the pinata
(a) Query of type ARG1
Query: The people hit  $<  \mathrm{Q - ARG1}>$
Target Answer: the ball
BUTD-QAP: the ball
VOG-QAP: the ball
MTX-QAP: the ball
(b) Query of type ARG2
Query: A man sitting  $<  \mathrm{Q - ARG2 > }$
Target Answer: behind a drum kit
Lang-QAP: on a bed
BUTD-QAP: on a drum set
VOG-QAP: behind a drum set
MTX-QAP: in front of a drum set
Figure 3: Queries of Type ARG1 and ARG2 on ASRL-QA
Query: A man sits  $<  \mathrm{Q - ARG2 >}$  next to a baby
Target Answer: on a playground swing
BUTD-QAP: on the ground
VOG-QAP: on a swing
MTX-QAP: on a swing
Figure 4: Queries of Type ARGM-LOC on ASRL-QA
Query: A lady washing clothes  $<$  Q-ARGM-LOC>
Lang-QAP: in a sink
Target Answer: in a bucket
BUTD-QAP: in a bowl
VOG-QAP: in a bucket
MTX-QAP: in the water
Query: People washing their clothes <Q-ARGM-LOC>
Target Answer: in a river
Lang-QAP: in a sink
BUTD-QAP: in a lake
VOG-QAP: on a river
MTX-QAP: in the water