SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4.4 Ablation Studies on SEViLA Framework
We conduct ablation studies on our SEViLA framework about the effectiveness of LOCALIZER and ANSWERER. We show the results in Table 3. We summarize our findings as follows:
(a) Sparse frames outperform dense frames: We observe performance declines when the ANSWERER adopts more frames (A v.s. B), confirming that sparse frames work better due to the original Image-LM model's limited temporal modeling ability, while too dense frames may distract the model.
(b) Keyframes outperform uniformly sampled frames: We compare ANSWER with LOCALIZER (SEViLA framework) and ANSWER that takes uniformly sampled frames. We observed significant performance gains in the zero-shot ANSWER setting when utilizing the zero-shot LOCALIZER (B v.s. C), with improvements on NExT-QA (+1.0%), STAR (+2.4%), How2QA (+1.5%), TVQA (+1.6%), and VLEP (+0.4%). And pseudo-label refinement for LOCALIZER further increased performance by an average of 2.1% across all tasks (B v.s. D). In the fine-tuned ANSWER setting, the benefits of the LOCALIZER remained evident. Our SEViLA framework, which utilized keyframes from the
Table 3: Ablation studies on SEViLA framework. 'uniform' refers to the uniform sampling of video frames. LOCALIZER† refers to the zero-shot LOCALIZER without refining on pseudo-labels.
ANSWERER	Keyframe	NExT-QA	STAR	How2QA	TVQA	VLEP
# frame finetuned?	Tem.	Cau.	Des.	Avg.	Int.	Seq.	Pre.	Fea.	Avg.
A.	32	X	uniform	54.7	56.7	67.8	57.7	46.2	43.6	40.7	41.0	42.8	67.0
B.	4	X	uniform	59.7	60.8	73.8	62.4	45.5	41.8	41.8	40.0	42.2	70.8
C.	4	X	LOCALIZER†	61.3	61.5	75.6	63.6	48.3	45.0	44.4	40.8	44.6	72.3
D.	4	X	LOCALIZER	62.3	63.1	74.9	64.6	49.0	46.4	45.2	41.6	45.5	72.9
E.	4	✓	uniform	68.1	72.9	81.2	72.6	65.4	69.0	59.7	54.2	62.0	82.2
F.	4	✓	LOCALIZER†	68.8	73.4	83.5	73.4	63.2	66.6	61.3	60.0	62.7	83.7
G.	4	✓	LOCALIZER	69.4	74.2	81.3	73.8	63.7	70.4	63.1	62.4	64.9	83.6
Table 4: Comparison on QVHighlights test split. We aggregate frame-level results of our LOCALIZER for video-level evaluation (see Appendix).
Model	R1@0.5	R1@0.7	mAP
CAL [13]	25.4	11.5	9.8
XML [29]	41.8	30.3	32.1
Moment-DETR [30]	52.8	33.0	30.7
QD-DETR [51]	62.4	44.9	39.8
LOCALIZER (Ours)	54.5	36.5	32.3
Table 5: The impact of QVHighlights Pre-Training (PT) and Self-Refinement (SR) for our LOCALIZER in Sec. 3.3.
PT	SR	NExT-QA	How2QA
Tem.	Cau.	Des.	Avg.
-	-	60.4	61.0	74.6	62.9	70.7
✓	-	61.3	61.5	75.6	63.6	72.3
-	✓	62.1	62.6	75.1	64.3	72.8
✓	✓	62.3	63.1	74.9	64.6	72.9
LOCALIZER, outperformed the 4-frame ANSWER by an average of  $0.7\%$  across tasks (E v.s. F). Pseudo-label refinement continues to be effective in this setting, providing an average boost of  $1.5\%$  across all tasks (E v.s. G). These results demonstrate that keyframe selection contributes to non-trivial improvements in video-language tasks for zero-shot and fine-tuning settings.
4.5 Comparison to State-of-the-Art on Video Moment Retrieval
In this section, we evaluate our LOCALIZER on the video moment retrieval task. We pre-train the LOCALIZER on the QVHighlights [30] dataset, as discussed in Sec. 3.3, and then assess its performance on the same dataset. To test on QVHighlights, we first extract video frames at 0.5 fps, following Moment-DETR [30] and pass them through our LOCALIZER to obtain binary frame-level predictions that indicate whether a frame matches the query sentence. Next, we combine these predictions into video-level temporal span predictions. We aggregate frame-level predictions into video-level spans by merging adjacent positive predictions with intervals not exceeding a threshold. These merged results are then consolidated into a single video-level span. More information on the aggregation process can be found in Appendix. Interestingly, as shown in Table 4, our LOCALIZER, which has no temporal modeling/training and operates on frame-level, outperforms many previous methods [13, 30, 29] with complex temporal modeling and video data training. It demonstrates our LOCALIZER can further work as a standalone model for certain tasks. It also suggests large image-LM with temporal designs may be a promising study for video moment retrieval. This is evidenced by our LOCALIZER's superior performance compared to Moment-DETR, despite the absence of temporal modeling.
4.6 Detailed Analysis on the Localizer
In this section, we first analyze the impact of pre-training and self-refinement on our LOCALIZER. Then, we compare our LOCALIZER with other keyframe selection methods in both zero-shot and fine-tuning settings. Next, we experiment with different keyframe selection ranges and quantities in our LOCALIZER to assess the impact of temporal localization on the overall performance. We further show the impact of LOCALIZER in ANSWERER fine-tuning. We also present upper-bound analysis based on BLIP-2 and oracle keyframe localization. Lastly, we provide visualization results of our LOCALIZER. Additional experiments are in Appendix.
Ablation on LOCALIZER pre-training and self-refinement. We performed these ablation studies with the zero-shot 4-frame ANSWER. As shown in Table 5, the untrained BLIP-2 LOCALIZER
results in only a minor improvement to the ANSWERER. Moreover, both QVHighlights pre-training and self-refinement via the reverse chain independently provide significant performance boosts. The optimal results are achieved when both pre-training and self-refinement are applied. It further demonstrates our method is label-efficient for keyframe temporal localization.
Comparison with other keyframe selection methods. In Table 6, we compare our LOCALIZER with different keyframe localization methods, including CLIP [55], Moment-DETR [30] which are zero-shot, and ATP [3], Differentiable Top-K [8] which are fine-tuned with answer-label. We combine those keyframe localization methods with our zero-shot ANSWERER. Those methods select 4 keyframes from 32 uniformly sampled frames. We find that keyframes from neither CLIP nor Moment-DETR can not help ANSWERER. It may be due to their CLIP pre-training on images and short declarative sentences, which fail to produce question-aware visual features, and distract ANSWERER with irrelevant features. In contrast, our zero-shot LOCALIZER improves on NExT-QA by averaging  $1.2\%$ . Furthermore, our LOCALIZER refined on pseudo-labels outperforms fine-tuned ATP and Differentiable top-K, with a  $2.2\%$  average improvement
Table 6: Comparison of our LOCALIZER with other keyframe localization methods.
Method	NExT-QA
Tem.	Cau.	Des.	Avg.
ANSWERER	59.7	60.8	73.7	62.4
(zero-shot)
+ CLIP [55]	59.2	60.0	72.5	61.8
+ Moment-DETR [30]	59.5	60.6	72.1	62.0
+ LOCALIZER†	61.3	61.5	75.6	63.6
(fine-tuning)
+ ATP [3]	60.4	61.3	73.4	62.8
+ Differentiable Top-K [8]	59.5	59.7	72.7	61.6
+ LOCALIZER	62.3	63.1	74.9	64.6
across all question types. Overall, our LOCALIZER shows superior effectiveness in both settings.
Impact of keyframe selection ranges and quantities. In Table 7, we evaluate temporal keyframe localization in a zero-shot setting using various keyframe selection ranges and quantities. Even with one keyframe selected, our LOCALIZER shows significant improvements over the BLIP-2 $^{\text{voting}}$  based on majority vote on 8 frame-level answers: NExT-QA-Causal  $(+2.4\%)$ , NExT-QA-Description  $(+3.6\%)$ , and How2QA  $(+2.6\%)$ . This highlights our LOCALIZER's effectiveness in localizing selective keyframes. We also note that multiple keyframes benefit NExT-QA-Temporal questions, but denser frames result in worse perfor
Table 7: Ablation of different numbers of input frames and output keyframes.
Settings	NExT-QA	How2QA
Tem.	Cau.	Des.	Avg.
BLIP-2voting (8)	59.9	60.2	72.4	62.0	69.8
8→1	59.8	61.1	76.0	62.9	72.4
16→1	59.2	62.6	74.9	63.4	73.2
16→4	60.7	61.5	75.8	63.4	72.4
32→4	61.3	61.5	75.6	63.6	72.3
32→8	59.4	60.9	74.7	62.5	71.3
64→8	58.9	60.9	74.0	62.2	71.8
mance. It supports our finding in Sec. 4.4, that using too dense frames may distract image-LM.
Impact of different frame sampling during ANSWERER fine-tuning. In Sec. 3.3, we discussed how the ANSWERER in the SEVILA framework can be further fine-tuned in the forward chain using keyframes from the LOCALIZER. Table 8 compares fine-tuning in different frame sampling strategies, and indicates SEVILA framework performs optimally when utilizing LOCALIZER in both ANSWERER training and evaluation. This is likely due to the provision of more informative keyframes and milder domain shifts between the training and evaluation.
Upper-bound performance analysis on oracle keyframes. We further explore the upper-bound performance with the assumption of a 'perfect' localizer, capable of always providing the right keyframes for the ANSWER. To achieve this, we uniformly sample four frames from each video, input them into BLIP-2 individually, and generate four frame-level answers. The upper-bound performance is represented by the oracle accuracy, which considers a question correctly answered if at least one of the four frames yields the right answer. As shown in Table 9, significant gaps exist between BLIP-2 majority voting and oracle accuracy in both settings. These gaps emphasize the need for more future work in temporal localization to effectively harness image-LM for video-language tasks.
Qualitative analysis on LOCALIZER. In Fig. 4, we present an example from NExT-QA (more in Appendix), showcasing the QA pairs, our LOCALIZER results, and the ground truth task-related video clips that we manually annotated. Our LOCALIZER more accurately matches human annotations than
Frame Sampling	NExT-QA
Training	Inference	Temp.	Cau.	Des.	Avg.
Random	Uniform	68.1	72.9	81.2	72.6
Random	LOCALIZER†	67.6	73.4	84.0	73.1
LOCALIZER†	Uniform	68.2	72.7	80.0	72.3
LOCALIZER†	LOCALIZER†	68.8	73.4	83.5	73.4
Table 8: Comparing different frame sampling during ANSWERER fine-tuning. The LOCALIZER† is frozen during fine-tuning. We use 4 frames for ANSWERER training, while the LOCALIZER† is the default  $32 \rightarrow 4$ .
Datasets	BLIP-2voting (Oracle)
Zero-Shot	Fine-tuned
NExT-QA (Avg.)	62.7 (70.1)	70.1 (79.7)
STAR (Avg.)	40.3 (52.9)	51.8 (72.2)
How2QA	69.8 (77.8)	79.6 (86.4)
TVQA	35.7 (45.4)	54.5 (69.0)
VLEP	63.8 (70.5)	67.0 (79.1)
Table 9: BLIP-2 $^{\text{voting}}$  and oracle (in brackets) performance analysis across datasets. We use 4 frames for each video question. Oracle: at least 1 of 4 frames can give the right answer.
Question: why did the two ladies put their hands above their eyes while staring out?
A: practicing cheer. B: posing for photo. C: to see better. D: dancing. E: wiping their face.
[Uniform Selection]
Question: What did both of them do after completing skiing?
A: jump and pose. B: bend down. C: raised their hands. D: turn around. E: take off clothes.
Figure 4: Visualization of our LOCALIZER. We use zero-shot ANSWERER with different frame sampling (uniform v.s. LOCALIZER) to answer the question. Red options are answered wrongly with uniformly sampled frames. Green options are answered correctly with our LOCALIZER. Best viewed in color.
uniform selection. This accurate localization enables the ANSWERER to correctly answer questions, while uniform selection leads to incorrect responses. These results demonstrate that our LOCALIZER can effectively locate task-related keyframes in a video, thus benefiting downstream tasks.
In this paper, we introduced SEViLA, a novel video-language framework. SEViLA adapts an image-language model to obtain two modules: (1) LOCALIZER for language-aware temporal localization and (2) ANSWERER for question answering on keyframes. SEViLA tackles video-language tasks by chaining the output of LOCALIZER to the input of ANSWERER (forward chain), while ANSWERER can give feedback to refine the LOCALIZER (backward chain) via pseudo labeling. The proposed temporal localization allows a more focused understanding of videos and improves the accuracy of video-language tasks, while the pseudo-labeling process alleviates the need for expensive language-aware keyframe annotations. Compared with state-of-the-art baselines, SEViLA achieves competitive or better performance on five video questions answering and event prediction benchmarks. We also provide a comprehensive analysis elaborating on the design of the proposed two-stage self-chaining. Our research encourages future work to improve temporal localization in video understanding.
We thank the reviewers and Archiki Prasad, Hao Tan, Yi-Lin Sung, and Jie Lei for their valuable feedback and input for the paper. This work was supported by ARO Award W911NF2110220, DARPA KAIROS Grant FA8750-19-2-1004, DARPA MCS Grant N66001-19-2-4031, and NSF-AI Engage Institute DRL211263. The views, opinions, and/or findings contained in this article are those of the authors and not of the funding agency.
[1] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736, 2022.
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[3] S. Buch, C. Eyzaguirre, A. Gaidon, J. Wu, L. Fei-Fei, and J. C. Niebles. Revisiting the" video" in video-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2917-2927, 2022.
[4] M. Cao, T. Yang, J. Weng, C. Zhang, J. Wang, and Y. Zou. Locvtp: Video-text pre-training for temporal localization. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, pages 38-56. Springer, 2022.
[5] Y.-C. Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.
[6] J. Cho, J. Lei, H. Tan, and M. Bansal. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning, pages 1931-1942. PMLR, 2021.
[7] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.
[8] J.-B. Cordonnier, A. Mahendran, A. Dosovitskiy, D. Weissenborn, J. Uszkoreit, and T. Unterthiner. Differentiable patch selection for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2351–2360, 2021.
[9] R. Cui, T. Qian, P. Peng, E. Daskalaki, J. Chen, X. Guo, H. Sun, and Y.-G. Jiang. Video moment retrieval from text queries via single frame annotation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1033-1043, 2022.
[10] A. Deng, T. Yang, and C. Chen. A large-scale study of spatiotemporal representation learning with a new benchmark on action recognition. arXiv preprint arXiv:2303.13505, 2023.
[11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[12] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Thompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
[13] V. Escorcia, M. Soldan, J. Sivic, B. Ghanem, and B. Russell. Temporal localization of moments in video collections with natural language, 2019.
[14] H. Fang, P. Xiong, L. Xu, and Y. Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097, 2021.
[86] L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu. Filip: fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021.
[87] Q. Ye, G. Xu, M. Yan, H. Xu, Q. Qian, J. Zhang, and F. Huang. Hitea: Hierarchical temporal-aware video-language pre-training. arXiv preprint arXiv:2212.14546, 2022.
[88] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[89] L. Yuan, D. Chen, Y.-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
[90] R. Zellers, J. Lu, X. Lu, Y. Yu, Y. Zhao, M. Salehi, A. Kusupati, J. Hessel, A. Farhadi, and Y. Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16375-16387, 2022.
[91] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[92] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
In this Appendix, we present the following:
- Additional comparison between image-language models (image-LMs) and video language models (video-LMs) in terms of model size and pre-training data size (Sec. A).
- Additional information about our experimental setup (Sec. B), including benchmarks details and task definitions (Sec. B.1), metrics (Sec. B.2), baseline implementation details (Sec. B.3), and SEViLA implementation details (Sec. B.4).
- Additional experiments (Sec. C), including the single-frame v.s. multi-frame LOCALIZER (Sec. C.1), iterative self-refinement results (Sec. C.2), different LOCALIZER pre-training settings (Sec. C.3), SEViLA framework with another Image-LM (MiniGPT4) (Sec. C.4), computational cost of SEViLA (Sec. C.5), the impact of prompt designs (Sec. C.6) and more qualitative visualization (Sec. C.7).
- Limitation and broader impact of this work (Sec. D), as well as license information (Sec. E) for the datasets, codes, and models that we used in this paper.
A Comparison Between Image-LMs and Video-LMs
In this section, we further provide a more visualized comparison between Image-LMs [55, 35, 43, 61, 70] and Video-LMs [71, 87, 37, 67, 17], in terms of model size and pre-training data size. As illustrated in Fig. 5, recent video-LMs generally have smaller scales in model and pre-training data sizes. This is because video-level annotations are more costly than image-level annotations, as discussed in the main paper Sec. 2 The gap between image-LMs and video-LMs has driven extensive research into image-to-video transfer learning to effectively leverage powerful image-LMs for video-language tasks. In this paper, we adopt this image-to-video transfer strategy by employing the state-of-the-art pre-trained image-LM, BLIP-2 [35].
Figure 5: Comparison between image-LMs and video-LMs, in terms of model size and pre-training data size.
In this section, we present additional information on the used datasets/benchmarks and task definitions (Sec. B.1), metrics (Sec. B.2), baseline implementations (Sec. B.3), and our SEViLA framework implementations (Sec. B.4).
Dataset	# frames	Batch Size per GPU	Learning Rate	Epoch	Gradient Accumulation Step
NExT-QA	8	4	1e-5	10	32
STAR	8	4	1e-5	10	32
How2QA	8	4	1e-5	10	32
TVQA	8	4	1e-5	10	32
VLEP	8	4	1e-5	10	32
Table 10: InternVideo fine-tuning hyperparameters.
B.1 Benchmarks
Video Question Answering (Video QA). We test models on four challenging long-form multi-choice benchmarks: (1) NExT-QA [77], a benchmark for causal and temporal reasoning. It contains 5440 videos with an average length of 44s and approximately 52K questions. NExT-QA contains 3 different question types: Temporal (Tem.), Causal (Cau.), and Descriptive (Des.). (2) STAR [75], a benchmark for situated reasoning, which contains 22K video clips with an average length of 12s, along with 60K questions. STAR contains 4 different question types: Interaction (Int.), Sequence (Seq.), Prediction (Pre.), and Feasibility (Fea.). (3) How2QA [36], a benchmark containing 44k questions along with 22 thousand 60-second clips selected from 9035 videos. (4) TVQA [27], another benchmark contains 152K questions along with 21k video clips (76 seconds on average) from TV shows.
Video Event Prediction (Video EP). We test models on VLEP [28], a benchmark that requires the model to predict two future events based on the video premise. Thus, we can formulate this task as a multi-choice QA. VLEP contains 28,726 future event prediction cases from 10,234 diverse TV Shows and YouTube Lifestyle Vlog video clips.
Video Moment Retrieval. We test our LOCALIZER performance on QVHighlights [30], where a model predicts the temporal span in a video corresponding to a human-generated language description. QVHighlights consists of 10,148 videos with a duration of 150s, 18,367 moments, and 10,310 queries.
For video question answering and video event prediction, we adopt the standard answer accuracy. For video moment retrieval, we follow Lei et al. [30] and report the standard mean average precision (mAP) over multiple IoU thresholds [0.5: 0.05: 0.95], and the standard Recall@1 (R@1) where we define a prediction to be positive if it has a high IoU ( $>= 0.5$  or  $>= 0.7$ ) with one of the ground truth moments. For NExT-QA, STAR, How2QA, TVQA, and VLEP we report the performance on the validation set whereas QVHighlights we report on the hidden test set.
B.3 Baseline Implementations
Baseline Architecture. For InternVideo [71], we adopt the released largest MM-L-14 variant that initialized from CLIP-L/14 [55] with 1B parameters and follow its default 8-frame setting for comparison. We fine-tuned InternVideo on downstream tasks by ourselves. For BLIP-2 $^{\text{voting}}$  and BLIP-2 $^{\text{concat}}$  (ANSWERER), we use the same BLIP-2 ViT-G Flan-T5 XL to our SEViLA framework.
Baseline Training. In Table 10, we show the setup of InternVideo [71] fine-tuning on downstream tasks. we adopt the released largest MM-L-14 variant with 1B parameters and follow its default 8-frame setting. We conduct experiments with  $4\times 48\mathrm{GB}$  A6000 GPUs. It takes about 4h on NExT-QA, 8h on STAR, 4h on How2QA, 12h on TVQA, and 2h on VLEP.
Implementation Details of Other Keyframe Selection Methods.. In the main paper Sec. 4.6, we compare our LOCALIZER with other keyframe selection methods in both zero-shot and fine-tuning settings. Specifically, in zero-shot setting, we compare our LOCALIZER with CLIP [55] and Moment-DETR [30], utilizing the same ANSWERER  $(32\rightarrow 4)$  for each. For CLIP one, we calculate the pre-trained CLIP-ViT-B/32 image-language similarity between each frame's visual feature and the corresponding question and option features. We then select the top-4 similarity keyframes from 32 uniformly sampled frames for the ANSWERER. In the case of Moment-DETR, we apply the Moment-DETR pre-trained on QVHighlights to first detect a temporal span corresponding to the question and option
Dataset	Batch Size per GPU	Learning Rate	Warmup Step	Epoch	Gradient Accumulation Step
LOCALIZER Pre-Training
QVHighlights	64	3e-5	1000	80	1
AnswerER Fine-tuning in Forward Chain
NExT-QA	8	3e-5	1000	10	2
STAR	8	3e-5	1000	10	2
How2QA	4	3e-5	3000	10	4
TVQA	4	3e-5	8000	10	4
VLEP	4	1e-5	1200	10	4
LOCALIZER Self-Refinement in Reverse Chain
NExT-QA	64	3e-5	500	10	1
STAR	64	3e-5	500	10	1
How2QA	64	3e-5	500	10	1
TVQA	64	3e-5	2000	10	1
VLEP	64	3e-5	500	10	1
Table 11: SEViLA framework training hyperparameters.
Pre-training Localization on QVHighlights
Aggregating frame predictions to spans
Figure 6: Left: For LOCALIZER pre-training, we utilize the video moment retrieval labels for the keyframe localization task. Right: we aggregate LOCALIZER's frame-level predictions into video-level span predictions.
sentence. Subsequently, we uniformly sample 4 frames within this span for the ANSWERER to predict answers. Next, we compare our LOCALIZER with the fine-tuned ATP and the Differentiable Top-K plugin. Both ATP [3] and Differentiable Top-K [8] utilize the BERT-base encoder [66], which is of the same size as the Q-former in our LOCALIZER. Both ATP and Differentiable Top-K modules are inserted after the Q-former to learn salient frame feature selection in an end-to-end manner.
B.4 SEVILA Implementation Details