SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\mathcal {L} _ {t, j} = \sum_ {r _ {p} \in \Omega_ {p}, r _ {n} \in \Omega_ {n}} \log \left(1 + \exp \left(M _ {t, j, r _ {n}} - M _ {t, j, r _ {p}}\right)\right),
$$
where  $M_{t,j,r_p}$  is the  $\mathbf{r}_p$ -th element of the vector  $M_{t,j}$ .  $\Omega_p$  and  $\Omega_n$  denote the set of positive and negative box indices, respectively. LSE loss is a smoothed alternative to the widely used hinge loss, it is easier to optimize than the original hinge loss (Li et al., 2017). During training, we randomly sample two negatives for each positive box. We use  $\mathcal{L}_i^{att}$  to denote the attention loss for the i-th example, which is obtained by summing over all the annotated frames  $\{v_t\}$  and concepts  $\{w_j\}$  for  $\mathcal{L}_{t,j}^{att}$ . We define the overall attention loss  $\mathcal{L}_{t,j}^{att} = \frac{1}{N}\sum_{i=1}^{N}\mathcal{L}_i^{att}$ . At inference time, we choose the boxes with scores higher than 0.2 as the predictions.
Temporal Supervision Given softmax normalized start and end probabilities  $\mathbf{p}^1$  and  $\mathbf{p}^2$ , we apply cross-entropy loss:
$$
\mathcal {L} ^ {s p a n} = - \frac {1}{2 N} \sum_ {i = 1} ^ {N} \left(\log {\bf p} _ {y _ {i} ^ {1}} ^ {1} + \log {\bf p} _ {y _ {i} ^ {2}} ^ {2}\right),
$$
where  $y_{i}^{1}$  and  $y_{i}^{2}$  are the GT start and end indices.
Answer Prediction Similarly, given answer probabilities  $\mathbf{p}^{ans}$ , our answer prediction loss is:
$$
\mathcal {L} ^ {a n s} = - \frac {1}{N} \sum_ {i = 1} ^ {N} \log {\bf p} _ {y _ {i} ^ {a n s}} ^ {a n s},
$$
where  $y_{i}^{ans}$  is the index of the GT answer.
Finally, the overall loss is a weighted combination of the three objectives above:  $\mathcal{L} = \mathcal{L}^{ans} + w_{att}\mathcal{L}^{att} + w_{span}\mathcal{L}^{span}$ , where  $w_{att}$  and  $w_{span}$  are set as 0.1 and 0.5 based on validation set tuning.
As introduced, our task is spatio-temporal video question answering, requiring systems to temporally localize relevant moments, spatially detect referred objects and people, and answer questions. In this section, we first define the evaluation metrics, then compare STAGE against several baselines, and finally provide a comprehensive analysis of our model. Additionally, we also evaluate STAGE on the full TVQA dataset.
Model	QA Acc.	Grd. mAP	Temp. mIoU	ASA
ST-VQA (Jang et al., 2017)	48.28	-	-	-
two-stream (Lei et al., 2018)	68.13	-	-	-
STAGE (video)	52.75	26.28	10.90	2.76
STAGE (sub)	67.99	-	30.16	20.13
STAGE	74.83	27.34	32.49	22.23
Human (Lei et al., 2018)	90.46	-	-	-
Table 3: TVQA+ test set results.
To measure QA performance, we use classification accuracy (QA Acc.). We evaluate span prediction using temporal mean Intersection-over-Union (Temp. mIoU) following previous work (Hendricks et al., 2017) on language-guided video moment retrieval. Since the span depends on the hypothesis (QA pair), each QA pair provides a predicted span, but we only evaluate the span of the predicted answer. Additionally, we propose Answer-Span joint Accuracy (ASA), that jointly evaluates both answer prediction and span prediction. For this metric, we define a prediction to be correct if the predicted span has an  $IoU \geq 0.5$  with the GT span, provided that the answer prediction is correct. Finally, to evaluate object grounding performance, we follow the standard metric from the PASCAL VOC challenge (Everingham et al., 2015) and report the mean Average Precision (Grd. mAP) at  $IoU$  threshold 0.5. We only consider the annotated words and frames when calculating the mAP.
5.2 Comparison with Baseline Methods
We consider the two-stream model (Lei et al., 2018) as our main baseline. In this model, two streams are used to predict answer scores from subtitles and videos respectively and final answer scores are produced by summing scores from both streams. We retrain the model using the official code on TVQA+ data, with the same feature as STAGE. We also consider ST-VQA (Jang et al., 2017) model, which is primarily designed for question answering on short videos (GIFs). We also provide STAGE variants that use only video or subtitle to study the effect of using only one of the modalities. Table 3 shows the test results of STAGE and the baselines. STAGE outperforms the baseline model (two-stream) by a large margin in QA Acc.,<sup>3</sup> with  $9.83\%$  relative gains. Additionally, STAGE also lo
2https://github.com/jayleicn/TVQA
This also holds true when considering mean (standard-deviation) of 5 runs: 74.20 (0.42).
8217
Model	QA Acc.	Grd. mAP	Temp. mIoU	ASA
baseline	65.79	2.74	-	-
+ CNN	67.25	3.16	-	-
+ Aligned Fusion (backbone)	68.31	7.31	-	-
+ Temp. Sup.	71.40	10.86	30.77	20.09
+ Spat. Sup.	71.99	24.10	31.16	20.42
+ Local Feature (STAGE)	72.56	25.22	31.67	20.78
STAGE with GT Span	73.28	-	-	-
Table 4: Ablation study of STAGE on TVQA+ val set. Each row adds an extra component to the row above it.
Model	baseline	+CNN	+AF	+TS	+SS	+LF
what (60.52%)	65.66	66.43	67.58	70.76	71.25	72.34
who (10.24%)	65.37	64.08	64.72	72.17	73.14	74.11
where (9.68%)	65.41	64.38	68.49	71.58	71.58	74.32
why (9.55%)	74.31	78.82	77.43	79.86	78.12	76.39
how (9.05%)	60.81	67.03	69.23	66.30	69.96	67.03
total (100%)	65.79	67.25	68.31	71.40	71.99	72.56
calizes the relevant moments with temporal mIoU of  $32.49\%$  and detects referred objects and people with mAP of  $27.34\%$ . However, a large gap is still observed between STAGE and human, showing space for further improvement.
Backbone Model Given the full STAGE model defined in Sec. 4, we define the backbone model as the ablated version of it, where we remove the span predictor along with the span proposal module, as well as the explicit attention supervision. We further replace the CNN encoders with RNN encoders, and remove the aligned fusion from the backbone model. This baseline model uses RNN to encode input sequences and interacts QA pairs with subtitles and videos separately. The final confidence score is the sum of the confidence scores from the two modalities. In the backbone model, we align subtitles with video frames from the start, fusing their representation conditioned on the input QA pair, as in Fig. 4. We believe this aligned fusion is essential for improving QA performance, as the latter part of STAGE has a joint understanding of both video and subtitles. With both changes, our backbone model obtains  $68.31\%$  on QA Acc., significantly higher than the baseline's  $65.79\%$ . The results are shown in Table 4.
Table 5: QA Acc. by question type on TVQA+ val set. For brevity, we only show top-5 question types (percentage in brackets). AF=Aligned Fusion, TS=Temp. Sup., SS=Spat. Sup., LF=Local Feature. Each column adds an extra component to the column before it.
Model	Temp. Sup.	val	test-public
two-stream (Lei et al., 2018)	✘	65.85	66.46
PAMN (Kim et al., 2019b)	✘	66.38	66.77
multi-task (Kim et al., 2019a)	✓	66.22	67.05
STAGE backbone (GloVe)	✘	66.46	-
STAGE backbone + Temp. Sup. (GloVe)	✓	66.92	-
STAGE backbone	✘	68.56	69.67
STAGE backbone + Temp. Sup.	✓	70.50	70.23
Table 6: QA Acc. on the full TVQA dataset.
Temporal and Spatial Supervision In Table 4, we also show the results when using temporal and spatial supervision. After adding temporal supervision, the model is able to ground on the temporal axis, which also improves the model's performance on other tasks. Adding spatial supervision gives additional improvements, particularly for Grd. mAP, with  $121.92\%$  relative gain.
Span Proposal and Local Feature In the second-to-last row of Table 4, we show our full STAGE model, which is augmented with local features  $G^{l}$  for question answering. Local features are obtained by max-pooling the span proposal regions, which contain more relevant cues for answering the questions. With  $G^{l}$ , we achieve the best performance across all metrics, indicating the benefit of using local features.
Inference with GT Span The last row of Table 4 shows our model uses GT spans instead of predicted spans at inference time. We observe better QA Acc. with GT spans.
Accuracy by Question Type In Table 5, we show a breakdown of QA Acc. by question type. We observe a clear increasing trend on "what", "who", and "where" questions after using the backbone net and adding attention/ span modules in each column. Interestingly, for "why" and "how" questions, our full model fails to present overwhelming performance, indicating some reasoning (textual) module to be incorporated as future work.
Qualitative Examples We show two correct predictions in Fig. 5, where Fig. 5(a) uses grounded objects to answer the question, and Fig. 5(b) uses text. More examples (including failure cases) are provided in the appendix.
TVQA Results We also conduct experiments on the full TVQA dataset (Table 6), without relying on the bounding boxes and refined timestamps in  $\mathrm{TVQA + }$ . Without temporal supervision, STAGE backbone is able to achieve  $3.91\%$  relative gain from the best published result (multi-task) on
8218
Figure 5: Example predictions from STAGE. Span predictions are shown on the top, each block represents a frame, the color indicates the model's confidence for the spans. For each QA, we show grounding examples and scores for one frame in GT span. GT boxes are in green. Predicted and GT answers are labeled by Pred and GT, respectively.
TVQA test-public set. Adding temporal supervision, performance is improved to  $70.23\%$ . For a fair comparison, we also provided STAGE variants using GloVe (Pennington et al., 2014) instead of BERT (Devlin et al., 2019) as text feature. Using GloVe, STAGE models still achieve better results.
We collected the TVQA+ dataset and proposed the spatio-temporal video QA task. This task requires systems to jointly localize relevant moments, detect referred objects/people, and answer questions. We further introduced STAGE, an end-to-end trainable framework to jointly perform all three tasks. Comprehensive experiments show that temporal and spatial predictions help improve QA performance, as well as providing explainable results. Though our STAGE achieves state-of-the-art performance, there is still a large gap compared with human performance, leaving space for further improvement.
We thank the reviewers for their helpful feedback. This research is supported by NSF Awards #1633295, 1562098, 1405822, DARPA MCS Grant #N66001-19-2-4031, DARPA KAIROS Grant #FA8750-19-2-1004, Google Focused Research Award, and ARO-YIP Award #W911NF-18-1-0336.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, and Stephen Gould. 2018. Bottom-up and top-down attention for image captioning and vqa. In CVPR.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In ICCV.
Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450.
François Chollet. 2017. Xception: Deep learning with depthwise separable convolutions. In CVPR.
Abhishek Das, Harsh Agrawal, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. 2016. Human attention in visual question answering: Do humans and deep networks look at the same regions? In EMNLP.
Yann Dauphin, Angela Fan, Michael Auli, and David Grangier. 2016. Language modeling with gated convolutional networks. In ICML.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. 2015. The pascal visual object classes challenge: A retrospective. IJCV.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ramakant Nevatia. 2017. Tall: Temporal activity localization via language query. In ICCV.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR.
Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. 2018. Ava: A video dataset of spatio-temporally localized atomic visual actions. In CVPR.
8219
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2018. Localizing moments in video with temporal language. In EMNLP.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan C. Russell. 2017. Localizing moments in video with natural language. In ICCV.
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatiotemporal reasoning in visual question answering. In CVPR.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP.
Junyeong Kim, Minuk Ma, Kyungsu Kim, Sungjin Kim, and Chang Dong Yoo. 2019a. Gaining extra supervision via multi-task learning for multi-modal video question answering. IJCNN.
Junyeong Kim, Minuk Ma, Kyungsu Kim, Sungjin Kim, and Chang Dong Yoo. 2019b. Progressive attention memory network for movie story question answering. In CVPR.
Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, and Byoung-Tak Zhang. 2017. Deepstory: Video story qa by deep embedded memory networks. In *IJCAI*.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. 2018. Tvqa: Localized, compositional video question answering. In EMNLP.
Yuncheng Li, Yale Song, and Joebo Luo. 2017. Improving pairwise ranking for multi-label image classification. In CVPR.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV.
Chenxi Liu, Junhua Mao, Fei Sha, and Alan Loddon Yuille. 2016. Attention correctness in neural image captioning. In AAAI.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. 2017. Spherical face: Deep hypersphere embedding for face recognition. In CVPR.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical question-image co-attention for visual question answering. In NeurIPS.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2018. Neural baby talk. In CVPR.
Tegan Maharaj, Nicolas Ballas, Aaron C. Courville, and Christopher Joseph Pal. 2017. A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering. In CVPR.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL.
Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung, and Bohyung Han. 2017. Marioqa: Answering questions by watching gameplay videos. In ICCV.
Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David A. McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In EMNLP.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy S. Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. TPAMI.
Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. 2016. Grounding of textual phrases in images by reconstruction. In ECCV.
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR.
Kevin J. Shih, Saurabh Singh, and Derek Hoiem. 2016. Where to look: Focus regions for visual question answering. In CVPR.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. Movieqa: Understanding stories in movies through question-answering. In CVPR.
8220
Alexander Trott, Caiming Xiong, and Richard Socher. 2018. Interpretable counting for visual question answering. In ICLR.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. CoRR, abs/1502.05698.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan R. Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In ICML.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018a. Janet: Combining local convolution with global self-attention for reading comprehension. In ICLR.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L. Berg. 2018b. Mattnet: Modular attention network for referring expression comprehension. In CVPR.
Licheng Yu, Eunbyung Park, Alexander C. Berg, and Tamara L. Berg. 2015. Visual madlabs: Fill in the blank description generation and question answering. In ICCV.
Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. 2017. A joint speaker-listener-reinforcer model for referring expressions. In CVPR.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In CVPR.
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters.
Zhou Zhao, Qifan Yang, Deng Cai, Xiaofei He, and Yueting Zhuang. 2017. Video question answering via hierarchical spatio-temporal attention networks. In *IJCAI*.
Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J. Corso, and Marcus Rohrbach. 2019. Grounded video description. In CVPR.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016a. Visual7W: Grounded Question Answering in Images. In CVPR.
Yuke Zhu, Oliver Groth, Michael S. Bernstein, and Li Fei-Fei. 2016b. Visual7w: Grounded question answering in images. In CVPR.
A.1 Timestamp Annotation
During our initial analysis, we find the original timestamp annotations from the TVQA (Lei et al., 2018) dataset to be somewhat loose, i.e., around  $8.7\%$  of 150 randomly sampled training questions had a span that was at least 5 seconds longer than what is needed. To have better timestamps, we asked a set of Amazon Mechanical Turk (AMT) workers to refine the original timestamps. Specifically, we take the questions that have a localized span length of more than 10 seconds ( $41.33\%$  of the questions) for refinement while leaving the rest unchanged. During annotation, we show a question, its correct answer, its associated video (with subtitle), as well as the original timestamp to the AMT workers (illustrated in Fig. 6, with instructions omitted). The workers are asked to adjust the start and end timestamps to make the span as small as possible, but need to contain all the information mentioned in the QA pair.
Figure 6: Timestamp refinement interface.
We show span length distributions of the original and the refined timestamps from TVQA+ train set in Fig. 7. The average span length of the original timestamps is 14.41 secs, while the average for the refined timestamps is 7.2 secs.
In Table 7 we show STAGE performance on TVQA+ val set using the original timestamps and the refined timestamps. Models with the refined timestamps perform consistently better than the ones with the original timestamps.
A.2 Bounding Box Annotation
At each step, we show a question, its correct answer, and the sampled video frames to an AMT
8221
Figure 7: Comparison between the original and the refined timestamps in the TVQA+ train set. The refined timestamps are generally tighter than the original ones.
Model	QA Acc.
Original	Refined
STAGE backbone	68.31	68.31
+ Temp. Sup.	70.87	71.40
+ Spat. Sup.	71.23	71.99
+ Local Feature (STAGE)	70.63	72.56
Table 7: STAGE performance comparison between the original timestamps and the refined timestamps on TVQA+ val set. Each row adds an extra component to the row above it.
worker. (illustrated in Fig. 8). We do not annotate the wrong answers as most of them cannot be grounded in the video. We checked 200 sampled QAs - only  $3.13\%$  of the wrong answers could be grounded, while  $46\%$  of the correct answers could be grounded. As each QA pair has multiple visual concepts as well as multiple frames, each task shows one pair of a concept word and a sampled frame. For example, in Fig. 8, the word "laptop" is highlighted, and workers are instructed to draw a box around it. In our MTurk instructions, we required workers to draw boxes for each instance of a plural word. E.g., for the word "everyone", the worker need to draw a box for each person in the frame. Note, it is possible that the highlighted word will be a non-visual word or a visual word that is not present in the frame being shown. In that case, the workers are allowed to check the box indicating the object is not present. Recent works (Zellers et al., 2019; Gu et al., 2018) suggest the use of pre-trained detectors for semi-automated annotation. However, since TVQA+ has a wide range of categories (see Fig. 2 and Table 1), it is challenging to use off-the-shelf detectors in the annotation process. As face detection and recognition might be easier than recognizing open set objects, we initially also tried using strong face detection (Zhang et al., 2016) and recognition (Liu
Figure 8: Bounding box annotation interface. Here, the worker is asked to draw a box around the highlighted word "laptop".
et al., 2017) model for character face annotation, but the quality was much poorer than expected. Thus, we decided to invest the required funds to collect boxes manually and ensure their accuracy. After the collection, with the GT labels, we again used the above models to test face retrieval performance for 12 most frequently appeared characters in TVQA+. To allow (Liu et al., 2017) to work, we manually collected 5 GT faces for each character as our gallery set. At test time, we assign each test face the label of its closest neighbor from the gallery set in the learned embedding space. This method achieves 55.6 F1/74.4 Precision/44.4 Recall. Such performance is not strong enough to support further research. We found the main reason is due to many partial occlusion of faces (e.g., side faces) in TV shows.
A.3 Quality
To ensure the quality of the collected bounding boxes, we only allow workers from English-speaking countries to participate the task. Besides, we set high requirements for workers – they needed to have at least 3000 accepted HITs and  $95\%$  accept rate. Qualified workers were well paid. We also kept track of the quality of the data during collection - workers with poor annotations were disqualified to work on our task. After collection, we further conducted an in-house check,  $95.5\%$  of 200 sampled QAs are correctly labeled, indicating the high quality of our data.
A.4 Training Details
We optimize our model using Adam with an initial learning rate of 1e-3, weight decay 3e-7. A mini
8222
Model	QA Acc.	Grd. mAP	Temp. mIoU	ASA
STAGE-LXMERT	71.46	21.01	26.31	18.04
STAGE	74.83	27.34	32.49	22.23
Table 8: TVQA+ test set results with LXMERT.
batch contains 16 questions. We train the model for maximum 100 epochs with early stop – if QA Acc. is not improving for consecutive 5 epochs, the training is stopped. CNN hidden size is set to 128.
A.5 Vision-Language Pretrained Features