Self-Chained Image-Language Model for Video Localization and Question Answering
Authors: Shoubin Yu
Jaemin Cho
Prateek Yadav
Mohit Bansal
Venue: NeurIPS 2023 poster (2023)
Keywords: Video Question Answering
Video Localization
Image-Language Model

Abstract: Recent studies have shown promising results on utilizing large pre-trained image-language models for video question answering. While these image-language models can efficiently bootstrap the representation learning of video-language models, they typically concatenate uniformly sampled video frames as visual inputs without explicit language-aware, temporal modeling. When only a portion of a video input is relevant to the language query, such uniform frame sampling can often lead to missing important visual cues. Although humans often find a video moment to focus on and rewind the moment to answer questions, training a query-aware video moment localizer often requires expensive annotations and high computational costs. To address this issue, we propose Self-Chained Video Localization-Answering (SeViLA), a novel framework that leverages a single image-language model (BLIP- 2) to tackle both temporal keyframe localization and question answering on videos. SeViLA framework consists of two modules: Localizer and Answerer, where both are parameter-efficiently fine-tuned from BLIP-2. We propose two ways of chaining these modules for cascaded inference and self-refinement. First, in the forward chain, the Localizer finds multiple language-aware keyframes in a video, which the Answerer uses to predict the answer. Second, in the reverse chain, the Answerer generates keyframe pseudo-labels to refine the Localizer, alleviating the need for expensive video moment localization annotations. Our SeViLA framework outperforms several strong baselines/previous works on five challenging video question answering and event prediction benchmarks, and achieves the state-of-the-art in both fine-tuning (NExT-QA and STAR) and zero-shot (NExT-QA, STAR, How2QA, and VLEP) settings. We show a comprehensive analysis of our framework, including the impact of Localizer, comparisons of Localizer with other temporal localization models, pre-training/self-refinement of Localizer, and varying the number of keyframes.

Self-Chained Image-Language Model for Video Localization and Question Answering
Shoubin Yu Jaemin Cho
Prateek Yadav Mohit Bansal
UNC Chapel Hill
(shoubin, jmincho, praty, mbansal] unc. edu
Abstract
Recent studies have shown promising results on utilizing large pre-trained image- language models for video question answering. While these image-language models can efficiently bootstrap the representation learning of vidoo-language models, they typically concatenate uniformly sampled video frames visual inputs without explicit language-aware, temporal modeling. When only a portion of a video input is relevant the language query, such uniform frame sampling can often lead missing important visual cues. Although humans often find video moment to focus on and rewind the moment answer questions. training a query-aware video moment localizer often requires expensive annotations and high computational costs. To address this issue, we propose Self-Chained Video DocalizationnAnsweer (SeViLA), a novel framework that leverages a single image-language model (BLIP- 2) tackle both temporal keyframe localization and question answering on videos. SEVILA framework consists of two modules: LOCALIZER and ANSWERER, where both are parameter-efficienty fine-tuned from BLIP-2. We propose two ways chaining these modules for cascaded inference and self-refinement. First, in the forward chain, the Localizer finds multiple language-aware keyframes in a video, which the ANSWERER uses to predict the answer. Second, in the reverse chain, the ANSWERER generates keyframe pseudo-labels to refine the LOCALIZER, alleviating the need for expensive video moment localization annotations. Our SEVILA framework outperforms several strong baselines/previous works on five challenging video question answering and event prediction benchmarks. and achieves the state- of-the-art in both fine-tuning (NExT-QA and STAR) and zero-shot (NExT-Qr Q-QA, STAR, How2QA, and VLEP) settings. We show comprehensive analysis of our framework, including the impact of LoCaLIZeR, comparisons of LOCaLIZER with other temporal localization models, pre-taiiningssel--eeenmmee of LOCALIZER, and varying the number of keyframes.'
1 Introduction
The recent success of large-scale pre-trained language models [2, 7, 65] has led burst of multimodal vision-and-languaage models that can jointly understand visual (image/video) and language data [64, 6, 60]. However, due to higher computational and annotation costs, lideollngguage models (video-LMs) are more challenging to scale in terms of model and data size than image-langusde models (image-LMs). Hence, recent studies have explored efficient training of video-LMs by leveraging pre-trained image-IMss [44, 14, 23, 85, 70, 31, 19, 82). While such a warm-start strategy facilitates visual representation learning of video-I.Ms, they typically concatenate uniformly/randomyy sampled video frames as visual inputs without explicit language- simple uniform/random sampling of frames can lead losing important visual cues, resulting the video-LMs focusing on frames that are
temporal modeling. However, such
to language queries [42].
OOur code and checkpoints are available at: http:://itthu..o https:/gituub.oo///e wwwpittuu..om/uuiiiiai Ywt.//000020////ii....
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

Forward Chain
Reverse Chain
Language-aware Temporal Localization
Refine Localizer with Pseudo-labels
(1) Localize Keyframes
(2) with Keyframes
(1) Generate Pseudo-labels
(2) Refine Localizer
Question
Question
Question
Question
Frames
Frameo
Localizer
Answerer
Answerer
Localizer
BLIP-2
BLIP-2
BLIP-2
BLIP-2 +
Answer
Answer
Answer
Neyframes
Incorrect X
Correct
X
X
Frozen
Finetuned
Answer
Psaudo-Labeling
for Keyframes
Figure 1: Self-Chiiie Chained Video Localization Answering (SEVILA) consists of a LOCALIZER and an ANswERER. Left: Forward chain for language-aye aware temporal keyframe localization and question answering. Right: Reverse chain for LOCALIZER self-refinement with keyframe pseudo-labels.
To address this, we introduce
Video Localization- Antiriiing (SEVILA), novel video-language framework where we adopt a single image-LM to handle both temporal local- ization and question answering on videos, while avoiding expensive language-aware, temporal grounding annotations (plus self-refinement [48] between the two modules). Our SEVILA frame- work obtains two modules, a LocaLIzeR and an ANSWERER through parameter-efficient fine- tuning [62] of BLIP-2 [35], recent ootte-of-the-att image-LM. SEVILA tackles video-language tasks by chaining the output of LoCALIZER to the input ANSWERER (forward chain, Fig. left), while the ANSWERER gives feedback to refine the LOCALIZER (backward chain, Fig. I right). In the forward chain, Localizer leverages the original image-language understanding of the BLIP-2 backbone and chooses the important language- localization prompt "Does the information within the frame provide the necessary details to accurately answer the given question?" for each video frame. Then AN- SWERER takes the concatenation of selected keyframes as visual input to predict video-level answers. In the backward chain, generate keyframe pseudo-labels [26] to refine the LOCALIZER, where label a video frame as a keyframe if ANSwERER can output the correct answer using that frame. This self-refinement improves the language-awar temporal localization accuracy and alleviates the need for expensive keyframe annotations.
video keyframes via the
We demonstrate the effectiveness of SEVILA framework five challenging video question answering and event prediction benchmarks (NEXT-QA, STAR, HOw2QA, TVQA, and VLEP) [75, 77, 36, 27, 28], where SEVILA outperforms several strong baselines/previous works, and achieves the state-of-the-arr in both fine-tuning (NExT-QA and STAR) and zero-shot (NEXT-QA, STAR, How2QA, and VLEP) settings. We also show that LOCALIZER be used strong stand-alone moment retrieval model. We present a comprehensive analysis to elaborate the design choices of the proposed framework, including the impact of temporal localization, the impact of the self-refinement process (backward chain), and varying the number of keyframes. Our contributions are summarized as follows:
A novel lineu-language framework SEVILA, where the LOCALIZER and ANswERER are initialized from a single image-languee model to handle temporal localization and question answering on videos, respectively.
A new self-refinement method for language-aware temporal keyframe localization, where the ANswERER generates keyframe pseudo-labels to refine the LoCaLIZeR, without expensive temporal grounding annotation.
Strong empirical performance with state-of-the of-tth--rrttt on multiple video-language benchmarks. Comprehensive analysis elaborating the design choices of the proposed framework.
2 Related Work
Image-Languaae Pre-trained Models. As the demand for cross-modal applications continues to grow, image-language pre-training studies have received tremendous attention and success. Image-

---

language pre-trained models [55, 12, 69, 5, 36, 64, 31, 34, 86] have advanced more rapidly than video-language pre-trained models [71, 90, 45, 17, 80, 67, 81, 83, 87, 37, 88], both in terms of model [61, 38, 35, 1, 70, 43] and pre-training data scale [61, 1, 89, 21] (more detailed model size and pre-training data scale comparisons are in Appendix). This can be attributed to the increased accessibility of image data and the comparatively simpler data structures, which makes scaling up image-language learning easier [58]. In our paper, SEVILA built on the recent state-of-the-at image-LM BLIP-2 2 [35] and extends it to adopt video input for video-language tasks. We also compare our S&VILA framework with the current state-of-the-at video-LM, Intern Video [71], to demonstrate the superiority of large image-LM that incorporates keyframe localization.
Image-to-Video Transfer Learning. The gap between image- and video-language models has inspired numerous useful methods focusing on imaee-tovvideo transfer learning, which leverage limited number of video frames to enhance learning efficiency [80, 32, 23, 46, 15, 44, 14, 31,4,, 4, 82, 68). Luo et al. [44] adapt pre-trained CLIP [55] backhone to solve video clip retrieval. Yang et al. [85] extend frozen bidirectional language models [66] to incorporate multiple images and apply additional
video-level pre-training to facilitate model adaptation [62]. Wang et al. [72] convert multiple images into hierarchical captions, arranged with a temporal order prompt to help language models comprehend video-level events. However, these works employ uniform sampling strategy that is not language-awand aware. This can lead to the loss of key visual cues for temporal modeling and even burden the models with irrelevant information [31, 76]. In this paper, we propose LOCALIZER provide language-awan aware visual information to rideo-language tasks.
Language-awarr Keyframe Localization. Many methods [42, 3, 18, 54, 24, 42, 41, 73, 9] have been proposed to address the challenge of language-aware keyframe localization. Buch et al. [3] optimized an end-to-end pipeline using answer labels to select single keyframe for downstream tasks. Lu et al. [42] selects frames with separate image and language models, and answers questions by QA model with multiple training objectives. Qian et al. [54] designed video clip proposal model with predefined ranges, iteratively training it with a QA model. Kim et al. [24] utilized a semi-parametria retriever to obtain keyframes based on frame and language feature similarity. We adopt a large image-LM as our LOCALIZER and chain it with an ANSWERER. Our LocaLIZER can help to fine-tune ANSWERER in the forward chain and be refined with pseudo-labela in the reverse chain.
3 Method: VILA
In this section, we introduce the method details of our Self-Chained Video Localization (SEVIL.A) framework. First, we provide BL.IP- preliminaries, which serve as the foundation for our framework. Then we elaborate our design of the BLIP-2 LocalIZer and BLIP-2 ANSWERER for temporal localization and question answering on videos. Finally, we present the SEVILA framework's training and inference processes in the forward and reverse chain.
3.1 Preliminaries: BLIP-2
We adopt BLIP-2 [35] as the backbone of our SEVILA framework. BLIP-2 is recent state-of-the-art pre-trained
model (image-L.M) comprising of: (1) a frozen image encoder [11, 16]; (2) frozen large language model (LLM) [7, 91]; and (3) Q-former, which is trainable transformer [66] module that bridges the image encoder and LLM, similar to acting as an adapter [62, 20). It takes as input visual features h from the image encoder and learnable query embeddings g, and outputs fixed-length visual features v. The BL.P-- Q-Former undergoes a two-stage pre-training. First, connects the image encoder perform image-to-text pre-training. This stage enables the Q-Former to extract the most informative visual information for the text and remove any irrelevant details in v. Subsequently, the Q-Former is connected to the LLM leverage its generative language capabilities. This is achieved using fully-connected layer to project query embeddings into the LLM's 's dimension with image-to- to-text pre- pre-training. As result, these query features serve as soft visual prompts [22] for the LLM. With the two-stage pre-trained Q-rormer and LLM, BLIP-2 shows advanced performance on various image-language tasks. our SEVILA framework, we adopt BLIP-2 as the basic building block for both video temporal localization and question answering modules. We retain the visual encoder and the L.LM from BLIP-2 by keeping them frozen during training. this case, only these two Q formeers are updated during the LOCALIZER and ANSWERER training (see Sec. 3.3).
3

---

Localizer
visual features
learnable query g
Topk Selection
frame scores
[0.2. 0.3]
+ VIT
G-Former Qloe
* FlanT5
frame features
[Qusttio]] [Options] [Loc Prompt)
Frames
Answerer
learnable visual features query
answer:
language-awarr keyframe features
FlanT5
O-Former Qans
[Question] [Options] [GA Prompt]
[Question] what does the boy Show to City unerapping
[Opptions]
[Loc Prompt)
(OA Prompt] Consicering informtion select arrever from the options.
Gres information within the frsme proside heik-
Reba fingers.
alone.
soary detalls sccurstily ariswwr the givwn queedion??
Figure 2: In SEVILA framework, LOCALIZER (top) selects top-K video frames, which guides ANSWERER (bottom) focus important language-awan video moments and predict answers. Both LOCALIZER and ANSWERER are initialized from a single pre-trained BLIP-2 model, where only Q-formers and linear layer (2.5% of total parameters) are tuned for each module. We omit the linear layer after the Q-former for simplicity.
3.2 Self-Chained Video Localization
Adapting BLIP-2 to Temporal Localization and Question Answering Videos. As illustrated in Fig. 2, our SEVILA framework adopts BLIP-2 to tackle both video temporal localization and question-answering. We assign BLIP-2 two roles of being LOCALIZER and an ANsweRER by using different Q-formers. We first elaborate our LOCALIZER and ANSWERER in detail as follows:
LOCALIZER. We first extract frame features via the frozen image-encoder VIT [16], referred to E... Given the video, we uniformly sample n frames {fi...., fn We then obtain éu frame feature h4 h, (fi).. Finally, we represent the video as set of frame features V (hi,..,,nh) h. These features are extracted once and then saved to be subsequently reused by the LoCaLiZER and the ANSWERER. The primary objective of the Localizer is select k: language-aware keyframe features from V, where & is typically much smaller than n. As illustrated in Fig. (top), then independently extract visual query features o; from original frame features in V via a Q-Former Qioe- Next, visual query features vú and language contexts L are concatenated and fed into the LLM (Flan-T5 [7]), where we create L by combining question, options, and localization prompt "Does the information within the frame provide the necessary details to accurately answer the given question?", The LOCALIZER outputs the score for each frame st. which is the probability of generating a word 'yes', given the visual features Us and language context L: s; LLM (v,, L)). We can localize kunuuage-award keyframes K (utt ..., v based on the top-k frame scores. Our LOCAlIZER can be formulated as:
K = LOCALIZER (V, L), |K] = En
ANSWERER. With the keyframe set obtained from the LOCALIZER, as illustrated in Fig. (bottom), we can proceed to generate answers using the ANSWERER. We first obtain keyframe query features v; by processing them through Qans, following the same procedure used the LOCALIZER. Next, we feed the LLM with all query features and language contexts by concatenating them and obtain the video-level answer a = LLM (concat(v)" .... uxx 2))². Then, the frozen LLM can conduct temporal modeling with multiple frame inputs. Our ANswERER be formulated as:
ANSWERER K, L)
(2)
We also insert a frame ID token before each frame, where we omit the notations from the equation for simplicity.

---

Forward Chain Inference Fine-tune
[Question] [Options] Prompi]
[Options] [Loc Prompt)
[Quaation]
Localizer
Answerer
Option
Reverse Chain
Refine Localizer
[Question]
[Options] [QA Prompt]
Option
correct
Pseudo
Option
Incomect
[Question] [Options] [QA Prompt]
Answerer
Optinn
Ground Truth
Pseudo Labels: no
Localizer
[Question]
[Options]
Prompt)
Figure 3: Top: In the forward chain, the LOCALIZER finds multiple language-awe aware keyframes, then the ANswERER utilizes these keyframes to predict answers. We use the forward chain for both inference and ANSWERER fine-tuning. Bottom: In the reverse chain, we generate keyframe pseudo-labels by using the ANswERER to refine the LOCALIZER.
3.3 Training ANSWERER and LOCALIZER via Self-Chaining
Fine-tuning ANSWERER in forward chain. As illustrated in Fig. 3 (top), we fine-tune the ANswERER on downstream tasks using keyframes from LoCaLIZER via the forward chain. ANSWERER takes the keyframes generated by LOCALIZER. We compare the default setting to other settings (e.g., input frames are uniformly selected) in Appendix.
Refining LOCaLIZER reverse chain. We adopt pseudo-labeling [26] in our reverse chain to address the costly frame-level localization annotations. We use binary pseudo-labels, where label video frame as a keyframe if ANSWERER can output the correct answer using that frame. As shown in Fig. (bottom), The frozen ANswERER is first prompted by a QA task prompt and generates frame-level answer, then we obtain pseudo labels by comparing this prediction with the ground-truth answer. The LOCALIZER trained locate language-aware pseudo-label keyframes.
Pre-training LOCALIZER with moment retrieval label. To enhance our LOCALIZER, we conduct transfer learning from a video moment retrieval/grounding task [56. 30] via pre-training. We use videos, queries, and video-level temporal span labels from QVHighlights [30], and assign a binary localization label each frame by comparing its timestamp the span annotations. We provide more details pre-training in Appendix.
4 Experiments
In this section, we first outline our experimental setup (Sec. 4.1). Then, we demonstrate the superiority of SEVILA framework 5 challenging long-form video question answering and event prediction benchmarks in both fine-tuning (Sec. 4.2) and zero-shot (Sec. 4.3) settings. we also conduct ablation studies SE VEAA framework show the effectiveness of each of its components downstream tasks (Sec. 4.4). Next, We report the performance of our LOCALIZER on video moment retrieval (Sec. 4.5). Lastly, perform in-depth quantitative, and qualitative analyses show the effect of our design for temporal keyframe localization (Sec. 4.6 and Appendix). More results on single multi-frame LOCAL IZEER, pre-training strategies, iterative self-refinement, computational cost, and extension to another Image-LM model are in Appendix.
LocaliZer
4.1 Experimental setup
Benchmarks. We evaluate our L framework on 3 video-language tasks, including multi-choice Video Question Answering (NEXT-QA [77], STAR [75), How2QA [36], TVQA [27]), Video Event Prediction (VLEP [28]), and Moment Retrieval (QVHighlights [30]). See Appendix for details.

---

Table I: Fine-tuning results on video question answering (NEXT-QA, STAR, How2QA, TVQA) and video event prediction (VLEP). We gray out the methods take extra speech input or use dense frames. We bold the best numbers, and underlined the second-best numbers. dense/Ifps: the model takes dense (Ifps) video frames instead of a fixed number of frames. 32 - 4: our LOCALIZER selects keyframes from 32 frames. represents the results tested by ourselves. SEVILAT uses the zero-shot LocalIZER without refining on pseudo-labels via the reverse chain.
NEXT-QA
STAR
Model (# Frames)
How2QA TVQA VLEP
Tem. Cau. Des. Avg. Int. Seq. Pre. Fea. Avg.
(W/ speech input or use dense frames)
HERO (dense/ Lfps) [36]
73.8
73.6
JustAsk (20) [84]
51.4 49.6 63.1 52.3
84.4
FrozenBil M (10) [85]
86.7
82.0
VidII. 4-shot (12) [72]
72.0
T+T (dense/ Ifp)) [40]
92.4
T+T (+ASR, dense/Ifps [40]
93.2
Flamingo-80B 32-shot (30) [1]
42.2
FrozenBiLM (10) [85]
81.5
57.5
All-in-One (32) [67]
48.6
48.0
63.2
50.6
47.5
50.8
47.7
44.0
47.5
Temp[ATP] (32) [3]
49.3
48.6
65.0 51.5 50.6
52.8
49.3
40.6
48.3
VGT (32) [78]
55.0
52.2 64.0 55.0
44.2
MIST (32) [18]
56.6
54.6
66.9 57.1 55.5 54.2
542
44.4
51.1
VFC (32) [50]
53.3
57.6
72.8 58.6
CoVGT (32) [79]
57.4
58.8
69.3 60.0
45.9
SeViT (10) [24]
60.6
HiTeA (16) [87]
58.3
62.4
75.6 63.1
Intern Video" (8) [71]
58.5
62.5
75.8 63.2 62.7 65.6
54.9
51.9 58.7
79.0
57.2
63.9
BLIP-2 (4)
65.2
70.1
80.1 70.1 52.3
54.8
49.0
51.2 51.8
79.6
54.5
67.0
BLIP-2 (ANSWFRER)
68.1
72.9
81.2
72.6 65.4 69.0
59.7
54.2
62.0
82.2
59.8
68.6
I 1 (32 4)
68.8
73.4
83.5
73.4
63.2
66.6
61.3
60.0
62.7
83.7
59.7
69.0
SEVILA (32 4)
69.4
74,2
81.3
73.8 63.7 70.4 63.1 62.4
64.9
83.6
61.6
68.9
Baselines. We compare our SEVILA framework with the state-of-the-art video-language pre-trained model, Intern Video [71] as well as our backbone BLIP-2 [35]. We extend BL.IP-2 to adapt videos in two settings: (1) BLIP-2votin which processes each uniformly sampled frame independently and obtains the final answer by majority voting on all frame-level answers, and (2) BLIP-2 comeatt where Q-former processes each frame and Flan-T5 takes the concatenation of visual features prefix. BLIP-2 is equivalent to our ANSWERER with uniformly sampled frames. See Appendix for details.
Implementation Details. SEVILA framework adopts BI.IP-2 [35], an image-language model with 4.1B parameters and pre-trained on 129M images in total, including COCO [39], Visual Genome [25], CC12M [59], SBU [52], and 1 15M images from LAION400M [57]. See Appendix for details.
4.2 Fine-tuning Comparison to SOTA on Video QA and Event Prediction
We compare our SEVILA framework to recent state-of-the-ars models on 4 video QA benchmarks and 1 video event prediction dataset. We show results in Table 1, and summarize our findings as follows.
(a) Temporal modeling matters. BI.IP- underperforms our BI.IP- (ANSWERER) and other video-LM models on STAR, How2QA, TVQA, and VLEP. Especially on STAR-Scquence, a task demanding heavy temporal understanding, our BLIP- 2 (AnSwereR) outperforms BLIP-2 woting significantly by 13.1% (69.0% vs. 54.8%). As 2 voting processes frames independently and lacks temporal modeling among frames, the result indicates that temporal modeling is essential to tackle Manou-la tasks and the effectiveness of our temporal modeling design.
(b) Keyframe selection helps. Our SEVILA framework, featuring a zero-shot LOCALIZER, leads on all tasks with an average advantage of 5.3% over the top video-LM (Intern Video). also surpasses BLIP-2concar (ANSWERER) that uses uniform frame sampling on NeXT-QQ (+1 1.2%)) STAR (+0.7%) How2QA (+ (+1,5%) and VLEP (+0.4%). This highlights the importance of keyframe selection in video-language tasks, even when using a zero-shot LOCALIZER.

---

Table 2: Zero-shot results video question answering and video event prediction.
NEXT-QA
STAR
Model (# Frames)
How2QA TVQA VLEP
Tem. Cau. Des. Avg. Int. Seq. Pre. Fea. Avg.
(w/ speech input or use dense frames)
JustAsk (20) [84]
51.1
FrozenBiLM (10) [85]
58.4
59.2
ViperGPT (dense/ (ifess/Ifps) [63]
60.0
Flamingo-80B (30) [1]
39.7
Fro (10) [85]
41.9
29.7
VFC (32) [50]
45.4
51.6
64.1
51.5
Intern Video (8) [71]
43.4
48.0
65.1
49.1
43.8
43.2
42.3
37.4
41.6
62.2
35.9
58.7
BLIP-2 (4)
59.1
61.3
74.9
62.7
41.8
39.7
40.2
39.5
40.3
69.8
35.7
63.8
BLIP-2 2 (ANSWERER) (4)
59.7
60.8
73.8
62.4
45.5
41.8
41.8
40.0
42.2
70.8
36.6
64.0
SEVILA (32-44)
61.3
61.5
75.6
63.6
48.3
45.0 44.4
40.8
44.6
72.3
38.2
64.4
(c) Self-refinement improves temporal localization. For SEVILA, we refine the LOCALIZER with pseudo-labei, (see Sec. 3.3). Compared with SEVILA! SEVILA further increases performance on NExT-QA (0.4%), STAR (+2.2%) and TVQA (+1. 1.9%). SEVILA framework achieves new state-of-the-art fine-tuning performance on NEXT-QA, STAR, TVQA, and VLEP, using only visual and language modalities. This illustrates the significance of temporal localization and the efficacy of our self-refinement method for keyframe localization.
4.3 Zero-shot Comparison to SOTA on Video QA and Event Prediction
We further compare our SEVILA framework to recent state-of-the-art models in the zero-shot setting. We show the zero-shot results in Table 2, then discuss the findings in the following.
(a) Image-IM outperforms Video-LM, without video pretraining. Surprisingly, BI.IP-2voing without inter-framee temporal modeling, outperforms the previous state-of-the-art video-LM.. Intern- Video on several datasets that require temporal reasoning. BLIP-2 outperforms Intern on NEXT-QA (+ 13.6%) How2QA (+7.6%), and VLEP (+5.1%). On How2QA, BLIP-2 even surpasses FrozenBiLM which performs extra speech and video pre- pre-training by 11.4%. It highlights the potential of image-LM for video-language tasks due to its model size and sufficient pre-trale pre-training.
(b) Keyframe selection is more effective than uniform sampling. Our SEVILA framework. combining the zero-shot LOCALIZER and the zero-shot ANSWERER, outperforms the BLIP-22eacal (ANSwEreR) with uniformly sampled frames on NEXT-QA (+1.2%), STAR (+2.4%) How2QA (+1.5%), TVQA (+1.6%) and VLEE (+0.4%)) achieving new state-of-the-are zero-shot performance on NEXT-QA, STAR, How2QA, and VLEP, and new state-of-the-ar on TVQA with only visual and language modalities. On STAR, our SEVILA framework even outperforms zero-shot Flamingo [1] with 8OB parameters by 4.9%. The result demonstrates the effectiveness of our SEVILA framework to adapt video-language tasks and the importance of language-aware keyframe selection.
4.4 Ablation Studies on SEVILA Framework
We conduct ablation studies on our VEVI framework about the effectiveness of LOCALIZER and ANSWERER. We show the results in Table 3. We summarize our findings follows:
We observe performance declines when the ANSWERER adopts more frames (A v.s. B), confirming that sparse frames work better due to the original Image-LM model's ' limited temporal modeling ability, while too dense frames may distract the model.
(a) Sparse frames outperform dense frames:
(b) Keyframes outperform uniformly sampled frames:
We compare ANSWERER with LOCALIZER VILV framework) and ANSWERER that takes uniformly sampled frames. We observed significant performance gains in the zero-shot ANSWERER setting when utilizing the zero-shot LOCALIZER (B v.s. C), with improvements on NEXT-QA (+1.0%), STAR (+2.4%) How2QA (+ (1,5%) TVQA (+ (+1.6%), and VLEP (+0.4%) And pseudo-label refinement for LOCALIZER further increased performance by an average of 2. % across all tasks (B v.s. D). the fine-tuned ANSWERER setting, the benefits of the LOCALIZER remained evident. Our SEVILA framework, which utilized keyframes from the
7

---

Table 3: Ablation studies on SEVILA framework. 'uniform' refers the uniform sampling of video frames. LOCALIZER refers to the zero-shot LOCALIZER without refining on pseudo-labels.
ANSWERER
NEXT-QA
STAR
Keyframe
How2QA TVQA VLEP
# frame finetuned?
Tem. Cau. Des. Avg. Int. Seq. Pre. Fea.Avg.
A.
32
X
uniform
54.7 56.7 67.8 57.7 4.2.4444444...744100
67.0
33.2
54.0
x
uniform
59.7 60.8 73.8 62.4 45.544844888880000
70.8
36.6
64.0
X
LOCALIZER
61.3 75.6
48.3455.0444444884444
72.3
38.2
64.4
4
X
LOCALIZER 62.3 63.1 74.9 64.6 49.046445.241.64555
72.9
39.1
64.6
E
4
V
uniform 68.1 72.9 81.2 72.6 59.7742.
82.2
59.8
68.6
V
F
LOCALIZER 68.8 73.4 83.5 73.4 60.
83.7
59.7
69.0
V LOCALIZER 74.2 81.3 63.7 70.4 63.1 62.4 64.9
83.6
61.6
Table 4: Comparison on QVHighlights test split. We aggregate frame-level results of our LoCALIZER for video-level evaluation (see Appendix).
Table 5: The impact of QVHighlights Pre- Training (PT) and Self- Seff-Refinement (SR) for our LOCALIZER Sec. 3.3.
Model
R1@0.5
R1 @077
mAP
NEXT-QA
PT SR
HowQQA
CAL [13]
25,4
11.5
9.8
Tem.
Cau.
Des.
Avg.
XML [29]
41.8
30.3
32.1
60.4
61.0
74.6
62.9
70.7
Moment-DETR [30]
52.8
33.0
30.7
61.3
61.5
75.6
63.6
72.3
QD-DETR [51]
62.4
44.9
39.8
62.1
62.6
75.1
64.3
72.8
LOCALIZER (Ours)
54.5
36.5
32.3
62.3
63.1
74.9
72.9
LOCALIZER, outperformed the 4-frame ANSWERER by an average of 0.7% across tasks (E v.s. F). Pseudo-label refinement continues to be effective in this setting, providing average boost of 1.5% across all tasks (E v.s. G). These results demonstrate that keyframe selection contributes to non-trivial improvements in video-language tasks for zero-shot and fine-tuning settings.
4.5 Comparison to State-of-the-Art Video Moment Retrieval
In this section, we evaluate our LOCALIZER on the video moment retrieval task. We pre-train the LocaliZeR the QVHighlights [30] dataset, discussed in Sec. 3.3, and then assess its performance on the same dataset. To test on QVHighlights, we first extract video frames at 0.5 fps, following Moment-DETT -ETR [30] and pass them through our LOCALIZER to obtain binary frame-level predictions that indicate whether a frame matches the query sentence. Next, we combine these predictions into video-level temporal span predictions. We aggregate frame-level predictions into video-level spans by merging adjacent positive predictions with intervals not exceeding a threshold. These merged results then consolidated into single video-level span. More information the aggregation process be found in Appendix. Interestingly, as shown in Table 4, our LOCALIZER, which has no temporal modeling/training and operates on frame-level, outperforms many previous methods [13, 30, 29] with complex temporal modeling and video data training. demonstrates our LocaLiZER can further work as standalone model for certain tasks. It also suggests large image-LM with temporal designs may be a promising study for video moment retrieval. This is evidenced by our LOCALIZER* 's superior performance compared Moment-DETR, despite the absence of temporal modeling.
4.6 Detailed Analysis on the Localizer
In this section, we first analyze the impact of pre-training and self-refinement Then, we compare our LoCaLIZeR with other keyframe selection methods in both zero-shot and fine-tuning settings. Next, experiment with different keyframe selection ranges and quantities in our LOCalIZER to assess the impact of temporal localization on the overall performance. We further show the impact of LoCALIZER in ANSWERER fine-tuning. We also present upper-bound analysis based on BLIP-2 and oracle keyframe localization. Lastly, we provide visualization results of our LOCALIZER. Additional experiments
LOCALIZER.
Appendix.
Ablation on LOCALIZER pre-training and self-refinement. We performed these ablation studies with the zero-shot 4-frame ANsWERER. As shown in Table 5, the untrained BLIP-2 LOCALIZER

---

results in only minor improvement to the ANSWERER. Moreover, both QVHighlights pre-training and self-refinement via the reverse chain independently provide significant performance boosts. The optimal results are achieved when both pre-training and self-refinement are applied. It further demonstrates our method is label-efficient for keyframe temporal localization.
Comparison with other keyframe selection meth- ods. In Table 6, we compare our LOCALIZER with different keyframe localization methods, including CLIP [55]. Moment-DETR [30] which are zero-shot, and ATP [3], Differentiable Top-K K [8] which are fine-tuned with answer-label. We combine those keyframe localization methods with our zero-shot ANSWERER. Those methods select keyframes from 32 uniformly sampled frames. We find that keyframes from neither CLIP nor Moment-DETT can not help ANSWERER. It may be due to their CLIP pre-training on images and short declarative sentences, which fail to produce question-aware visual features, and distract ANSWERER with irrelevant features. In contrast, our zero-shot LOCALIZER improves NExT-QA by aver- aging 1.2%. Furthermore, our LOCALIZER refined on pscudo-labels outperforms fine-tuned ATP and Dif- ferentiable top-K, with a 2.2% average improvement across all question types. Overall, our LocalizeR shows superior effectiveness in both settings.
Table 6: Comparison of our LocaliZeR with other keyframe localization methods.
NExT-QA
Method
Tem. Cau. Des. Avg.
ANSWERER
59.7 60.8 73.7 62.4
(zero-shot)
+ CLJIP [55]
59.2 60.0 72.5 61.8
+ Moment-DETR [30]
59.5
60.6 72.1 62.0
+ LOCALIZER
61.3 61.5
75.6 63.6
(fine-tuning)
+ ATP [3]
60.4 61.3 73.4 62.8
+ Differentiable Top-K [8] 59.5 59.7 72.7 61.6
LOCALIZER
62.3 63.1 74.9 64.6
Impact of keyframe selection ranges and quantities. In Table 7, we evalu- ate temporal keyframe localization zero-shot setting using various keyframe selection ranges and quantities. Even with one keyframe selected, our LOCAL- IZER shows significant improvements over the BLIP-22oting based on majority on 8 frame-level answers: NEXT-QA-
Table 7: Ablation of different numbers of input frames and output keyframes.
NEXT-QA
Settings
How2QA
Tem.
Cau.
Des.
Avg.
BLIP-2 (8)
59.9
60.2
72.4
62.0
69.8
Causal (+2.4%)) NEXT-OA--Dsscription (+3.6%), and How2QA (+2.6%). This
8-+1
59.8
61.1
76.0
62.9
72.4
16-+1
59.2
62.6
74.9
63.4
73.2
16
60.7
61.5
75.8
63.4
72.4
highlights our LOCALIZER's 's effectiveness
32
61.3
61.5
75.6
63.6
72.3
in localizing selective keyframes. We also note that multiple keyframes ben- efit WEXT-QA-Tempores questions, but denser frames result worse perfor- mance. It supports our finding in Sec. 4.4, that using too dense frames may distract image-LM.
32-+8
59.4
60.9
74.7
62.5
71.3
64-18
58.9
60.9
74.0
62.2
71.8
Impact of different frame sampling during ANSWERER fine-tuning. In Sec. 3.3, we discussed how the ANSWERER in the SEVL framework can be further fine-tuned in the forward chain using keyframes from the LOCALIZER. Table 8 compares fine-tuning different frame sampling strategies, and indicates SEVILA framework performs optimally when utilizing LocaLIZeR in both ANSWERER training and evaluation. This is likely due the provision of more informative keyframes and milder domain shifts between the training and evaluation.
Upper-bound performance analysis oracle keyframes. We further explore the upper-bound performance with the assumption of a 'perfect' localizer, capable of always providing the right keyframes for the ANSWERER. To achieve this, uniformly sample four frames from each video, input them into BLIP-2 individually, and generate four frame-level answers. The upper- bound performance represented by the oracle accuracy, which considers question correctly answered if at least one of the four frames yields the right answer. As shown in Table 9, significant gaps exist between BLIP-2 majority voting and oracle accuracy in both settings. These gaps emphasize the need for more future work in temporal localization to effectively harness image-LM for video-language tasks.
Qualitative analysis LOCALIZER. In Fig. 4, present example from NExT-QA (more in Appendix), showcasing the QA pairs, our LocaliZeR results, and the ground truth task-related video clips that manually annotated. Our LOCALIZER accurately matches human annotations than

---

BLIP-2 (Oracle)
Frame Sampling
NEXT-QA
Datasets
Zero-Shot Fine-tuned
Training
Inference
Temp.
Cau.
Des.
Avg.
NEXT-QA (Avg.)
62.7 (70.1) 70.1 (79.7)
Random Uniform
68.1
72.9
81.2
72.6
STAR (Avg.)
40.3 (52.9)
51.8 (72.2)
Random
LOCALIZER
67.6
73.4
84.0
73.1
How2QA
69.8 (77.8)
79.6 (86.4)
LOCALIZER Uniform
68.2
72.7
80.0
72.3
TVQA
35.7 (45.4)
54.5 (69.0)
LOCALIZER
LOCALIZER
68.8
73.4
83.5
73.4
VLEP
63.8 (70.5)
67.0 (79.1)
Table 8: Comparing different frame sampling during ANSWERER fine-tuning. The LOCALIZER is frozen during fine-tuning. We use 4 frames for ANSWERER training, while the LOCALIZE is the default 32->4.
Table BLIP-2 and oracle (in brackets) performance analysis across datasets. We use frames for each video question. Oracle: at least 1 of 4 frames can give the right answer.
Question: why did the two ladies put their hands above their while staring out? A: pracficing cheer. B: posing for photo. C: to see better. D: dancing. E: wiping their face.
[Uniform Selection]
X
X
X
X
X
[Our Localizer Solection]
X
X
X
X
X
[Human Localization Annotation]
19s-30s
Question:
What did both of them do after completing skinng
A: jump pose.
raised around.
[Uniform Selection]
X
X
X
X
Selection)
X
X
X
(Human Temporal Localization Annotation]
36s-55s
Figure 4: Visualization of our LoCALIZER. We use zero-shot ANswereR with different frame sampling (uniform LOCALIZER) to answer the question. Red options answered wrongly with uniformly sampled frames. Green options are answered correctly with our LOCALIZER. Best viewed in color.
uniform selection. This accurate localization enables the ANSWERER correctly answer questions, while uniform selection leads to incorrect responses. These results demonstrate that our LOCALIZER effectively locate task-related keyframes in video, thus benefiting downstream tasks.
Conclusion and Future Work
In this paper, we introduced SEVILA, a novel video-language framework. SEVILA adapts an image- language model to obtain two modules: (1) LOCALIZER for language-aware temporal localization and (2) ANSWERER for question answering on keyframes. SEVILA tackles vidco-language tasks by chaining the output of LOCALIZER to the input of ANSWERER (forward chain), while ANSWERER can give feedback to refine the LocalIZer (backward chain) via pseudo labeling. The proposed temporal localization allows more focused understanding of videos and improves the accuracy of video- language tasks, while the pseudo-labeling process alleviates the need for expensive language-awan aware keyframe annotations. Compared with oftte-oftthe---e art baselines, SEVILA achieves competitive or better performance on five video questions answering and event prediction benchmarks. We also provide comprehensive analysis elaborating the design of the proposed two-stage self-chaining. Our research encourages future work to improve temporal localization in video understanding.
Limitations & Broader Impacts. See Appendix for limitations and broader impacts discussion.
10

---

Acknowledgement
We thank the reviewers and Archiki Prasad, Hao Tan, Yi-Lin Sung, and Jie Lei for their valuable feedback and input for the paper. This work was supported by ARO Award W91 10220, DARPA KAIROS Grant 148750-19-2-1004, DARPA MCS Grant N66001-19-2-4031, and NSF-AI Engage Institute DRL21 1263. The views, opinions, and/or findings contained in this article are those of the authors and not the funding agency.
References
[1] J.-B. Alayrac,, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc. A. Mensch, K. Millican, M. Reynolds, al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 15.23716-23755 2022.
[2] Brown, B. Mann, Ryder, M. Subbiah. J. D. Kaplan, Dhariwal, A. Neelakantan, Shyam. G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[3] S. Buch, C. Eyzaguirre, A. Gaidon, J. Wu, I. Fei-Fei, and J. Niebles Revisiting the" video" in video-language understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2917-2927, 2022.
[4] M. Cao, T. Yang. J. Weng, Zhang. Wang, and Y. Zou. Locvtp: Video-text pre-training for temporal localization. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, pages 38-56. Springer, 2022.
[5] Y.-C. Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.
[6] J. Cho, J. Lei, H. Tan, and M. Bansal. Unifying vision-and laiion-and-angguaee tasks via text generation. In International Conference on Machine Learning, pages 1931-1942. PMLR, 2021.
[7] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. De- hghani, S. Brahma, et al. Scaling instructionfiieturne language models. arXiv preprint 2XXi:2210.11416: 2022.
[8] J.-B. Cordonnier, A. Mahendran, A. Dosovitskiy, D. Weissenborn, J. Uszkoreit, and Unterthiner. Differentiable patch selection for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2351-2360, 2021.
[9] R. Cui, T. Qian, Peng, Daskalaki, J. Chen, X. Guo, H. Sun, and Y.-G. Jiang. Video moment retrieval from text queries via single frame annotation. In Proceedings of the 45th International ACM SIGIR Conference Research and Development in Information Retrieval, pages 1043, 2022.
[10] A. Deng, T. Yang, and Chen. large-scale study of spatiotemporal represertttonnnn learning with new benchmark on action recognition. arXiv preprint
2023.
[11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11922 2020.
[12] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, Yu, al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 2023.
[13] V. Escorcia, M. Soldan, J. Sivic, Ghanem, and B. Russell. Temporal localization of moments in video collections with natural language, 2019.
[14] H. Fang, Xiong, I. Xu, and Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint 2106. 1097, 2021.

---

[15] H. Fang, P. Xiong, L. Xu, and W. Luo. Transferring image-clip to video-text retrieval via temporal relations. IEEE Transactions on Multimedia, 2022.
[16] Y. Fang, W. Wang, B. Xie, Q Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint 2022.
arXiv:2211.074
[17] T.-J. Li, Z. Gan, K. Lin, W. Y. Wang, L. Wang, and Z. Liu. Violet: End-to-end video- language transformers with masked visual-token modeling, arXiv preprint arXiv:2111.1268! 2021.
[18] D. Gao, L.. Zhou, L.. Ji, L. Zhu, Y. Yang, and M. Z. Shou. Mist: Multi-modal iterative spatial- temporal transformer for long-form video question answering. arXiv preprint arXiv:2212.09522, 2022.
[19] S. K. Gorti, N. Vouitsis, J. Ma, K. Golestan, M. Volkovs, A. Garg, and G. Yu. X-pool: Cross-modal language-video attention for text-video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5006-5015, 2022.
[20] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nip. In International Conference Machine Learning, pages 2790-279999 2799. PMLR, 2019.
[21] C. Jia Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung. Z. Li, and T. Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904 4916. PMLR, 2021.
[22] M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim. Visual prompt tuning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII, pages 709-727. Springer, 2022.
[23] C. Ju, T. Han, K. Zheng, Y. Zhang, and W. Xie. Prompting visual-language models for efficient video understanding. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV, pages 105-124. Springer, 2022.
[24] S. Kim, J.-H. Kim, J. Lee, and M. Seo. Semi-parametric video-grounded text generation. arXiv preprint arXiv: 1100, 2023.
[25] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li.
D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-753 73, 2017.
[26] D.-H. Lee al. Pseudo-label: The simple and efficient semi- sumi-supee learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, page 896, 2013.
[27] J. Lei, L. Yu, M. Bansal, and T. L. Berg. Tvqa: Localized, compositional video question answering, arXiv preprint 1rXiv:1809.0160 2018.
[28) Lei, L. Yu, L. Berg, and M. Bansal. What is more likely to happen next? video-and lannuuuage future event prediction. arXiv preprint 1xXiv20010007999 2020.
[29] Lei, L. Yu, T. L. Berg, and M. Bansal. Tvr: A large-scale dataset for video-subtitle moment retrieval. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pages 447-463. Springer, 2020.
[30] J. Lei, T. L. Berg, and M. Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34:1 1846-11858, 2021.
[31] J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu. Less is more: Clipbert for aade--and-language learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7331-7341, 2021.
[32] Lei, T. I.. Berg, and M. Bansal Revealing single frame bias for hideo-and-language learning. Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics, 2023,
12

---

[33] D. Li. H. Le. G. Wang, S. Savarese, and S. C. Hoi. Lavis: A one library for language-viste vision intelligence. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 31-41, 2023.
Li, [34] Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-inaa -mnge pre-training for
unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[35] J. Li, Li, Savarese, and Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.15597 2023,
[36] L. Li, Y.-C. Chen, Y. Cheng. Z. Gan, L. Yu, and J. Liu. Hero: Hierarchical encoder for video+ language omni-representation pre-training. arXiv preprint arXiv:2005.00200 2020.
[37] L. Li, Z. Gan, K. Lin, C.-C. Lin, Z. Liu, C. Liu, and Wang. Lavender: Unifying rideo-language understanding masked language modeling. arXiv preprint arXhv:2206.071660 2022.
[38] Y. Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He. Scaling language-image pre-training via masking. arXiv preprint wrXiv:2212.0079. 2022.
[39] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Eisin-EECT 2014: 13th European Conference, Zurich, Switzerland, September 6- 2014, Proceedings. Part V 13, pages 740-755. Springer, 2014.
[40] X. Lin, S. Tiwari, S. Huang, M. Li, M. Z. Shou, H. Ji, and S.-F. Chang, Towards fast adaptation of pretrained contrastive models for multi-channel video-language retrieval. arXiv preprint arXiv:2206.. 02082, 2022.
[41] Y. Liu, P. Xiong, L. Xu, S. Cao, and Q. Jin. Ts2-net: Token shift and selection transformer for text-video retrieval. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV, pages 319-335. Springer, 2022.
[42] H. Lu, M. Ding, N. Fei, Y. Huo, and Z. Lu. Lgdn: Language-guided denoising network for video-language modeling. arXiv preprint 2..Kv:2209.113866 2022.
[43] J. Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint (XXi:2206.089166 2022.
[44] H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing. 508:293-3000 2022.
[45] Ma, X. Jin, H. Wang, Huang, L. Zhu, Feng, and Y. Yang. Temporal perceiving lideo-language pre-training. arXiv preprint arXiv:2301.0744 2023.
[46] Y. Ma. G. Xu, X. Sun, M. Yan, J. Zhang, and R. Ji. X-elip: End-to-end learning for video-text retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2022.
contrastive
[47] Z. Ma. J. Pan, and J. Chai. World-to-words: Grounded open vocabulary acquisition through fast mapping in vision-language models. arXiv preprint arXiv: 08685, 2023.
[48] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, Prabhumoye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint
2023.
[49] S. Marcel and Y. Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference on Multimedia, pages 1485-1488, 2010.
[50] L. Momeni, M. Caron, A. Nagrani, A. Zisserman, and C. Schmid. Verbs in action: Improving verb understanding in video-language models. arXiv preprint 2304. 067000000........0 2023.
[51] W. Moon, S. Hyun, Park, D. Park, and J.-P. Heo. Query-dependent video representation for moment retricval and highlight detection. arXiv preprint 2r31v:2303.13874, 2023.
13

---

[52] V. Ordonez, G. Kulkarni, and T. Berg. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24, 2011.
[53] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L.. Antiga, et al. Pytorch: An imperative style, iigh-ppeffrrmncce deep learning library. Advances in neural information processing systems, 32, 2019.
[54] T. Qian, R. Cui, J. Chen, P. Peng, X. Guo, and Y-G.. Jiang. Locate before answering: Answer guided question localization for video question answering. arXiv preprint arXiv:2210.02081, 2022.
[55] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages
PMLR, 2021.
[56] M. Regneri, M. Rohrbach, D. Wetzel, S. Thater, B. Schiele, and M. Pinkal. Grounding action descriptions in videos. Transactions of the Association for Computational Linguistics, 1:25-36, 36, 2013.
[57] C. Schuhmann, R. Vencu, R. Beaamont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint wrXiv:211..02111 2021.
[58] C. Schuhmann, R. Beaumont, R. Veneu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
[59] P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 2018.
[60] C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7464-7473, 2019.
[61] Q Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2333. (53000 2023.
[62] Y.-L. Sung. J. Cho, and M. Bansal. Vl-adapter: Parameter-efficient transfer learning for lision-and-langunn
tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5227-5237, 2022.
[63] D. Surís, S. Menon, and C. Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08 2023.
[64] H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv: 07490, 2019.
[65] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro. F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023.
[66] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez. L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[67] A. J. Wang, Y. Ge, R. Yan, Y. Ge, X. Lin, G. Cai, J. Wu, Y. Shan, X. Qie, and M. 7. Shou. All in one: Exploring unified video-language pre-training. arXiv preprint arXiv:2203.07233 00300333,, 2022.
[68] J. Wang, Y. Ge, G. Cai, R. Yan, X. Lin, Y. Shan, X. Qie, and M. Z. Shou. Object-aware video-language pre-training for retrieval. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 33 3313-3322, 2022.
14

---

[69] P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to-seee -e-uueence learning framework. In International Conference on Machine Learning, pages 23318-2831 2338--3344. PMIR, 2022.
[70] W. Wang. H. Bao, L. Dong, Bjorek, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022.
[71] Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022.
[72] Z. Wang, M. Li, R. Xu, L.. Zhou, J. Lei, X. Lin, S. Wang, Z. Yang, C. Zhu, D. Hoiem, et al. Language models with image descriptors are strong few-shot video-language learners. arXiv preprint arXiv: 10747, 2022.
[73] Z. Wang, Y. Zhong, Y. Miao, L. Ma, and L. Specia. Contrastive video-language learning with fine-grained frame sampling. arXiv preprint 22X0002200.00339 2022.
[74] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Huggingface's s transformers: State-of-the-art natural language processing. arXiv preprint arXiv: 19xiv:9900.00 1910.03771, 2019.
[75] B. Wu, S. Yu, Z. Chen, J. B. Tenenbaum, and C. Gan. Star: benchmark for situated reasoning in real-world videos. In Thirty-fiith Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[76] W. Wu, D. He, X. Tan, S. Chen, and S. Wen. Multi-agent reinforcement learning based frame sampling for effective untrimmed video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6222-62226222 2019.
[77] J. Xiao, X. Shang, A. Yao, and T.-S. Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9777-9786, 2021.
[78] J. Xiao, P. Zhou, T.-S. Chua, and S. Yan. Video graph transformer for video question answering. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 39-58. Springer, 2022.
[79] J. Xiao, P. Zhou, A. Yao, Y. Li, R. Hong, S. Yan, and T.-S. Chua. Contrastive video question answering via video graph transformer. arXiv preprint arXiv:2302. 2023.
[80] H. Xu, G. Ghosh, P.-Y. Huang, D. Okhonko, A. Aghajanyan, F. Metze, L.. Zettlemoyer, and C. Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint 1iX.::1109.14088 2021.
[81] H. Xu. Q. Ye, M. Yan, Y. Shi, J. Ye, Y. Xu, C. Li, B. Bi, Q. Qian, W. Wang, et al. mplug-2: A modularized mudti-modal foundation model across text, image and video. arXiv preprint 2023.
arXiv:2302.0044
[82] H. Xue, Y. Sun, B. Liu, J. Fu, R. Song, H. Li, and J. Luo. Clip-vip: Adapting pre-trained image-text model to video-language alignment. In The Eleventh International Conference on Learning Representations, 2023.
[83] S. Yan, T. Zhu, Z. Wang, Y. Cao, M. Zhang, S. Ghosh, Y. Wu, and J. Yu. Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint 2xXi:22222004979 2022.
[84] A. Yang, A. Miech, Sivic, Laptev, and Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1686-1697, 2021.
[85] A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid. Zero-shot video question answering via frozen bidirectional language models. arXiv preprint arXiv:2206.00 22010 2022.
15

---

[86] L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li. X. Jiang, and C. Xu. Filip: fine-grained interactive aanguage-imagg pre-training. arXiv preprint arXiv:2111.0778. 2021.
[87] Q. Ye, G. Xu, Yan, H. Xu, Q. Qian, Zhang, and F. Huang. Hitea: Hierarchical temporal-aware video-language pre-training. arXiv preprint arXiv:2212. 2022.
[88] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 2022.
[89] I.. Yuan, D. Chen, Y.-... Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, et al. Florence: new foundation model for computer vision. arXiv preprint arXiv:2111. 11112,,112,,,, 2021.
[90] R. Zellers, J. Lu, X. Lu, Y. Yu, Y. Zhao, M. Salehi, A. Kusupati, J. Hessel, A. Farhadi, and Y. Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16375-16387, 2022.
[91] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 00066, 2022.
[92] D. Zhu, Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
16

---

Appendix
In this Appendix, present the following:
Additional comparison between image-language models (image-LMMs) and video language models (video-IMss) terms of model size and pre-training data size (Sec. A).
Additional information about experimental setup (Sec. B), including benchmarks details and task definitions (Sec. B.1), metrics (Sec. B.2), baseline implementation details (Sec. B.3), and SEVILA implementation details (Sec. B.4).
Additional experiments (Sec. C), including the single-frame v.s. multi-frame LOCALIZER (Sec. C.I), iterative self-refinement results (Sec. C.2), different Localizer pre-training settings (Sec. C.3), SEVILA framework with another Image-LM (MiniGPT4) (Sec. C.4), computational cost of SEVILA (Sec. C.5), the impact prompt designs (See. C.6) and more qualitative visualization (Sec. C.7).
Limitation and broader impact of this work (Sec. D), as well as license information (Sec. E) for the datasets, codes, and models that we used in this paper.
A Comparison Between Image-LMs and Video-LMs
In this section, we further provide more visualized comparison between Image-LMMs [55, 35, 43, 61, 70] and Video-LMs [71, 87, 37, 67, 17]. in terms of model size and pre-training data size. As illustrated in Fig. 5, recent video-LMM generally have smaller scales in model and pre-training data sizes. This is because video-level annotations as discussed in the main paper Sec. The gap between image-LMs and video-LMs has driven extensive research into image-to-vidd videe transfer learning effectively leverage powerful image-LMs for video-language tasks. In this paper, we adopt this amage-to-vided transfer strategy by employing the sate-offthheaa pre-trained image-LM, BLIP-2 [35].
more costly than image-level annotations,
Image-Language Models
Video-Language Models
BET-3
EVA-CLIP
Unliid-IO
BLIP2
Ali-in-One
VIOLET
IntornViddo
12.5
Pre-training Model Size (B)
10.0
7.5
5.0
2.5
0
10
100
1000
Pre-training Data Size (M)
Figure 5: Comparison between image-LMs and video-LMs, in terms of model size and pre-training data size.
B Experimental Setup
In this section, we present additional information on the used datasets/benchmarks and task definitions (Sec. B))), metrics (Sec. B.2), baseline implementations (Sec. B.3), and our SEVILA framework implementations (Sec. B.4).
17

---

Batch Size
Learning Rate
Gradient Accumulation Step
Dataset
frames
Epoch
per GPU
NEXT-QA
8
4
le-5
10
32
STAR
le-5
10
32
How2QA
8
4
le-5
10
32
TVQA
le-5
10
32
VLEP
8
le-5
10
32
Table 10: Videro fine-tuning hyperparameters.
B.1 Benchmarks
Video Question Answering (Video QA). We test models on four challenging long-form multi-choice benchmarks: (1) NEXT-QA [77]. a benchmark for causal and temporal reasoning. contains 5440
videos with an average length of 44s and approximately 52K questions. NExT-QA contains 3 different question types: Temporal (Tem.), Causal (Cau.), and Descriptive (Des.). (2) STAR [75], a benchmark for situated reasoning. which contains 22K video clips with an average length of 12s, along with 60K questions. STAR contains different question types: Interaction (Int.), Sequence (Seq.). Prediction (Pre.), and Feasibility (Fea.). (3) How2QA [36], benchmark containing 44k questions along with 22 thousand 60-second clips selected from 9035 videos. (4) TVQA [27], another benchmark contains
152K questions along with 21k video clips (76 seconds average) from TV shows.
Video Event Prediction (Video EP). We test models on VLEP [28], a benchmark that requires the model to predict two future events based on the video premise. Thus, we can formulate this task as multi-choice QA. VLEP contains 28, 726 future event prediction cases from 10,234 diverse TV Shows and YouTube Lifestyle Vlog video clips.
Video Moment Retrieval. We test LOCaLiZER performance QVHighlights [30], where model predicts the temporal span in video corresponding to a numan-generated language description. QVHighlights consists of 10,148 videos with duration of 150s, 18,367 moments, and 10,310 queries.
B.2 Metrics
For video question answering and video event prediction, we adopt the standard answer accuracy. For video moment retrieval, follow Lei al. [30] and report the standard mean average precision (mAP) over multiple IoU thresholds [0.5: 0.05: 0.95], and the standard Recall @ (R@ 1) where we define prediction be positive if it has high IoU (>= 0.5 or 0.7) with one of the ground truth moments. For NExT-QA, STAR, How2QA, TVQA, and VLEE we report the performance on the validation set whereas QVHighlights report the hidden test set.
B.3 Baseline Implementations
Baseline Architecture. For Intern Video [71], we adopt the released largest MM-L-14 variant that initialized from CLIP-L/14 [55] with 1B parameters and follow its default 8-frame setting for comparison. We tine-tuned Intern Video on downstream tasks by ourselves. For BLIP-2votngg and the same BLIP-2 VIT-G Flan-T5 XL to SEVILA framework.
BLIP-25omear (ANSWERER),
Baseline Training. In Table 10, we show the setup of Intern Video [71] fine-tuning on downstream tasks. we adopt the released largest MM-L-14 variant with 1B parameters and follow its default 8-frame setting. We conduct experiments with 4 x 48GB A6000 GPUs. It takes about 4h on NExT-QA, 8h on STAR, 4h on How2QA, 12h TVQA, and 2h VLEP.
Implementation Details of Other Keyframe Selection Methods.. In the main paper Sec. 4.6, we compare our LOCALIZER with other keyframe selection methods in both zero-shot and fine-tuning settings. Specifically, in zero-shi shot setting, we compare our LOCALIZER with CLIP [55] and Moment- DETR [30], utilizing the same ANSWIRER (32 (3) for each. For CLIP one, we calculate the pre-traee mre-trined CLIP-ViT-B/33 image-language similarity between each frame's s visual feature and corresponding question and option features. We then select the top-4 similarity keyframes from 32 uniformly sampled frames for the ANSWERER. In the case of Moment-DETR we apply the Moment-DETR pre-trained on QVHighlights to first detect a temporal span corresponding to the question and option
18

---

Batch Size
Learning Rate
Warmup Step
Gradient
Dataset
Epoch
per GPU
Accumulation Step
LOCALIZER Pre-Training
QVHighlights
64
3e-5
1000
80
ANSWERER Fine-tuning in Forward Chain
NExT-QA
8
3e-5
1000
10
STAR
8
3e-5
1000
10
2
How2QA
4
3e-5
3000
10
TVQA
4
3e-5
8000
10
4
VLEP
4
le-5
1200
10
LOCALIZER Self- Rett-eeeinemeett Reverse Chain
NEXT-QA
64
3e-5
500
10
STAR
64
3e-5
500
10
How2QA
64
3e-5
500
10
TVQA
64
3e-5
2000
10
VLEP
64
3e-5
500
10
Table 11: SEVILL framework training hyperparameters.
Pre-training Localizer QVHighlights
Aggregating frame predictions spans
[Query Sentence]: shark swimming underwater
Rebieval
Video-Level
threshold = (typerparameter))
Localization Label
Frame-Lovel
yes
yes
predicted span
predicted span 2
longth span
Localizer
10011101111111000000000000001110
[Queey Santance) (Pasudo Options] [Loc Prompt)
Frame-level Predictions
Time
[Loc Prompt]
[Pseudo Options] no
Does the information within the frame provide neceesary sccuralely
Figure 6: Left: For LocaLIzeR pre-training, we utilize the video moment retrieval labels for the keyframe localization task. Right: we aggregate LOCALIZER'S 's frame-level predictions into video-level span predictions.
sentence. Subsequently, we uniformly sample 4 frames within this span for the ANSWIRER to predict answers. Next, we compare our LOcalIZer with the fine-tuned ATP and the Differentiable Top-K plugin. Both ATP [3] and Differentiable Top-K [8] utilize the BERT-base encoder [66]. which is of the same size as the Q-former our LocalIzER. Both ATP and Differentiable Top-K modules are inserted after the Q-former to learn salient frame feature selection in an end-to-end manner.
B.4 SEVILA Implementation Details
SEVILA Architecture. SEVILA framework adopts BLIP-2 [35], an image-language model with 4.1B parameters. We freeze its visual encoder (ViT-G [16]; IB) and large language model (Flan-T5 XL [7]: 3B), and only fine-tune me-tune the Q-former (including its learnable queries), and a fully connected layer after Q-former for each of both LOCALIZER and ANSWERER. There 106M trained parameters, which is 2.5% of the total parameters BLIP-2.
SEVILA Framework Training, We conduct experiments with 4 x 48GB A6000 GPUs. For LoCALIZE per-training, we pre-train LoCaliZer on the QVHighlights for epochs, taking approximately 12 hours with 4 29GB on A6000 GPUs. For LOCALIZER self-refinement, train LOCALIZER
19

---

ANSWERER
# frames of LOCALIZER
NExT-QA (Average)
1
64.6
zero-shot
4
63.6
1
73.4
fine-tuned
4
71.3
Table 12: Comparison between single-frame and multi-frame LOCALIZER.
each downstream dataset with pseudo-labels for an additional 10 epochs with 4 x 29GB on A6000 GPUs. It takes about 7h NExT-QA, 8h on STAR, 6h How2QA, 17h on TVQA, and 3h on VLEP. For ANswERER fine-tuning, we fine-tune ANSWERER on each downstream dataset with answer labels and frozen pre-trained LOCALIZER for 10 epochs with 4 48GB on A6000 GPUs. It takes about 13h on NEXT-QA, 15h on STAR, 13h on How2QA, 48h on TVQA, and 8h on VLEP. We employ standard cross-entropy loss between the model outputs and the target values. We report SEVILA framework training hyperparameters in LOCALIZER pre-training, ANSWERER fine-tuning, and LOCALIZER self-refimment, Table 11.
Prompt Engineering, We write multiple QA and Loe prompts shown in the main paper and choose the prompt yielding the best zero-shot performance the downstream task (see Sec. C.6).
LoCALIZER Pre-training Details. As illustrated in Fig. (left), we provide the visualization on our LoCALIZER pre-training. As discussed in the main paper Sec. 3.3, we adopt QVHighlighs [30] to pre-train our LOCALIZER, we transfer its temporal span annotations to keyframe localization labels by comparing frame time-stamp with spans. We further design a prompt template and fill in it with query sentences in QVHighlighs. makes the pre-training and downstream video QA and event prediction tasks with similar input format and can further benefit downstream tasks.
Details of Aggregation for Video Moment Retrieval. We elaborate on the aggregation of frame-level predictions from our LOCALIZER, as discussed in the main paper Sec. 4.5. As illustrated in Fig. 6 (right), we use hyperparameter called 'span threshold', maximum number of continuing 'O's (continuing 'no" prediction for level localization) inside single span. If there are more '0's than the value in row, we split the frames into separate spans. use the span threshold=6. We determined this value based on statistics of QVHighlights training data. In QVHighlights, query sentence has multiple grounding spans video. We calculate the average interval among those spans our span threshold.
C Experiments
In this section, we present extra in-depth analysis on ANSWERER and LOCALIZER, including the single-frame v.s. multi-frame LOCALIZER (Sec. C.I), iterative self-refinement results (Sec. C.2), different LoCaLIZeR pre-training settings (Sec. C.3), SEVILA framework with another Image-LM (MiniGPT4) (Sec. C.4), computational cost of SEVILA (Sec. C.5), the impact of prompt designs (Sec. C.6) and more qualitative visualization (Sec. C.7)
C.1 Single-frame v.s. Multi-frame LOCALIZER
Even though our single-frame LOCALIZER in SEVILA can locate language-aware keyframes, temporal modeling for video-level tasks cannot overlooked. In this section, we've ve expanded our LoCaLIZer to multi-frame mode by concatenating frames into a long image before Q-Former. This allows for full attention across all frames, enabling temporal modeling. As shown Table 12 we find that the 4-famme LOCALIZER performs worse than the single- frame LOCALIZER in both zero-shot and fine-tuning settings. We suspect that this is because our backbone model (BLIP-2) has not seen video data during its pre-training. We think that multi-frame LOCALIZER could be more powerful once we have enough temporal grounding annotations or enough pre-training on large-scale video dataset [10]. We leave the improvement of the multi-frame LOCALIZER to future work.
20

---

LOCALIZER
NeXT-QA (Average)
w/o Localizer
62.4
+ Moment-DETR
62.0
+ Our Localizer (without pre-training)
62.9
+ Our Localizer (weakly pre-trained with QVH ASR)
63.2
+ Our Localizer (pre-trained with QVH)
63.6
Table 13: Comparison among different pre-training settings of LOCALIZER.
Iteration
NeXT-QA (Average)
73.8
74.2
73.7
Table 14: Iterative self-refinement results of SEVILA framework.
C.2 Iterative Self-refinement on LOCALIZER and ANSWERER
Madaan al. [48] introduces the 'Self-Refine' framework, where LLM iteratively generates initial output, provides feedback, and refines the outputs. We experiment with iterative self-refinement, where the ANswERER gives pseudo labels train LOCALIZER, and then the LOCALIZER provides more accurate language-aware frames to fine-tune ANSWERER, iteratively. As shown in Table 14, we find that two iterations of self-refinement slightly increase performance compared the single self-refinement, but the performance saturates from three iterations. We leave further analysis with self-refinement to future work.
C.3 Different Pre-training Settings of LOCALIZER
We explored weakly-supervised pre- tre-training using ASR similar to Moment-DETR [30]. As shown in Table 13, our Localizer performance improves with the weakly supervised pre-training, closing the gap with the pre-training with manual annotations (QVHighlights [30]).
C.4 SEVILA Framework with Another Image-LM (MiniGPT4)
We experimented with extending the SEVILA framework with another recent Image- Language model (MiniGPT4 [92]). On NeXT-QA, zero-shot MiniGPT4 ANswERER achieves 52.7% average accuracy and obtains 0.7% boost with zero-shot MiniGPT4 LOCALIZER. We find proposed self-chaining scheme can also improve the performance of zero-shot MiniGPT4, and we expect that tine-tuning the model will even improve the performance.
Model
Memory (GB)
Running Time (sec//sample)
Parameter (B)
Answerer (4)
7.56
1.79
4.1
SeViLA (32->4)
7.98
3.28
4.2
Table 15: Computational cost of SEVILA framework.
C,5 Analysis on Computational Cost of SEVILA Framework
In Table 15, show memory, running time, and parameter size (ViT FlanT55 Qformer) comparison between our ANSWERER and our SEViLA (LocALIZi+ANSWERR share the most parameters, adding Localizer has very small additional memory footprint.
As the LoCalIZEr and ANSWERER
C.6 Impact of Prompt Design
We write several localization prompts and choose the one giving the best performance. We examine the impact of various localization prompts shown in Table 16. For this experiment, we employ
21

---

NEXT-QA
Localization Prompt
Temporal Casual Descriptive Average
Does the frame have the information needed to answer the question correctly?
59.9
61.1
74.2
62.7
Does the provided frame contain the necessary
59.9
60.8
75.0
62.7
information accurately answer the given question?
Does the information within the frame provide the necessary details accurately answer the given question?
60.4
61.0
74.6
62.9
Table 16: Impact of different localization prompts on the zero-shot Video QA performance
zero-shot ANSWERER and untrained LocalIZER to choose the most effective localization prompt based on performance. It demonstrates that our model is insensitive the changes the prompt.
Question: what the black the do after dropped the red toy? A: jump on man's leg. B: jump onto couch. C: push dog away. D: put on boy in red. E: jumpj down.
[Uniform Selection]
X
X
X
[Our Localizer Solection]
X
X
X
X
X
[Human Temporal Localization Annotation]
85- 19s
Question:
What room was Wilson breaking into when House found him?
A: the bedroom. B: the bathroom. the kitchen. D: the living room. E: the dining room.
[Uniform Selection]
X
X
X
X
X
Localizer
X
X
Temporal Annotation]
28s-956
Figure 7: Visualization of LOCALIZER. We show various keyframe amounts in those examples. We use zero-shot ANSWERER with different frame sampling (uniform v.s. LOCALIZER) to answer the question. Red options answered wrongly with uniformly sampled frames. Green options answered correctly with our LOCALIZER. Best viewed in color.
C.7 Visualization
As illustrated in Fig. provide visualization examples from different datasets, and with various selected keyframe amounts. Compared with uniform sampling, our Localizer can locate relevant frames that match human annotations well regardless of keyframe amounts. And such good keyframe localization also brings the correct answers to corresponding questions.
D Limitations and Broader Impacts
Limitations. Although our SEVILA framework can help locate language-aware keyframes in many real-world video tasks, e.g., video content analysis and event detection, our LoCaLiZeR uses frame-level keyframe localization and might not handle well some complex, fine-grained temporal events (e.g., open the door vs. close the door). Future work can explore structured prediction for temporal localization beyond the frame level.
22

---

Broader impacts. SEVILA framework leverages large image-language model trained on massive internet-scale data. Similar to most such large models, this might occasionally yield unexpected or inappropriate responses, potentially reflecting societal biases related to gender, race, or sexuality [47]. More studies in the large image-language model are needed evaluate and mitigate these negative biases, and toxic output.
E License
We will make our code and models publicly accessible. We use standard licenses from the community and provide the following links to the licenses for the datasets, codes, and models that used in this paper. For further information, please refer to the specific link.
NEXT-QA [77]: MIT
STAR [75]: Apache
How2QA [36]; MIT
TVQA [27]: MIT
VLEP [28]: MIT
QVHighlights [30]: CC BY-NC-SA 4.0
LAVIS [33]: BSD 3-Clause
PyTorch [53]: BSD-style
Huggingface Transformers [74]] Apache
Torchvision [49]: BSD 3-Clause
23


Reviews:
[{"review_id": "eINOKTVUGi", "reviewer": "Reviewer_TABX", "summary": "This paper proposes a novel framework called Self-Chained Video Localization-Answering (SEVILA) that leverages a single image-language model (BLIP-2) for both temporal keyframe localization and question answering in videos. The framework consists of two modules, Localizer and Answerer, which are parameter-fine-tuned (LoRA finetuning) from BLIP-2. The paper demonstrates that the SEVILA framework outperforms several strong previous works in video question answering and event prediction benchmarks.", "questions": "see above.", "limitations": "see above.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "(1) The paper proposes a novel framework that addresses the issue of missing important visual cues in video question answering tasks by leveraging a single image-language model for keyframe localization and question answering.\n(2) The self-chaining mechanism in the framework allows for cascaded inference and self-refinement, improving the accuracy of temporal localization without the need for expensive annotations.\n(3) The paper provides a comprehensive analysis of the proposed framework, including the impact of temporal localization, the self-refinement process, and the number of keyframes.", "weaknesses": "1. In the backward chain, the pseudo-tags are frame-level, and if a single frame of information can not give the correct answer, the Localizer is considered to have given the wrong keyframe. This is obviously unreasonable and ignores the important time dependencies provided by the intermediate frame. This is probably why there was no significant change in their performance when the self-refine part was removed in the ablation study. \n2. the proposal is not good at dealing with temporal causality. Because Blip-2 is a image-text MLLM model, there is no space-time inference, and their answer is to directly add the q-former query generated by each frame to the word token of LLM. This is equivalent to letting q-former himself to realize the time level of information understanding, and q-former does not have this ability.\nCould you make me clear?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Weakness 1**\n\n> In the backward chain, the pseudo-tags are frame-level, and if a single frame of information can not give the correct answer, the Localizer is considered to have given the wrong keyframe. \n\n&nbsp; We note that even allowing multi-frame in Localizer can also cause the misleading issue in the single-frame Localizer, since both one frame and short video clips have a chance to give incorrect pseudo labels. Because there is no frame-level annotation, this is a general issue for such a weakly supervised pseudo-label training scheme despite single or multiple frame(s).\n\n> This is obviously unreasonable and ignores the important time dependencies provided by the intermediate frame. \n\n&nbsp; **We still acknowledge the importance of temporal modeling** for video-level tasks.\nIn Section 4.2.a (Line 190-193), we highlighted that temporal modeling matters for video-language tasks. In Section 4.5 (Line 248-253) and Table 4 (results on video moment retrieval), we suggest more temporal designs for the Localizer to improve language-aware localization. In Section 4.6 (Line 294-299) and Table 7 (oracle performance analysis), we show that better localizers further improve the video question answering accuracy, emphasizing the need for more future work in such temporal localization. In the appendix Sec. D Limitation (appendix Line 136-139), we also discuss the failure cases of the current single-frame Localizer. \n\n> This is probably why there was no significant change in their performance when the self-refine part was removed in the ablation study. \n\n&nbsp; Our single-frame Localizer can improve video QA performance even without self-refinement. \nAs shown in Table 3 rows B, C, and D, our single-frame Localizer demonstrates non-trivial performance boosts on a multitude of long video QA benchmarks. \n\n| **Model** | **NeXT-QA (Avg.)** | **STAR (Avg.)** | **How2QA** | **TVQA** |\n|-------------------------------|----------------|-------------|-------------|-------------|\n| B - Answerer-only | 62.4 | 42.2 | 70.8 | 36.6 |\n| C - SeViLA+ (Localizer + Answerer) | 63.6 (+1.2) | 44.6 (+2.4) | 72.3 (+1.5) | 38.2 (+1.6) |\n| D - SeViLA (Localizer + Answerer + self-refinement) | 64.6 (+**2.2**) | 45.5 (+**3.3**) | 72.9 (+**2.1**) | 39.1 (+**2.5**) |\n\n&nbsp; Inspired by your suggestion, we also explored using a multi-frame localizer, but found that the single-frame Localizer works better than the multi-frame Localizer. As shown in the table below, we find that the 4-frame localizer performs worse than the single-frame localizer in both zero-shot and finetuning settings. We suspect that this is because our backbone model (BLIP-2) has not seen video data during its pretraining. We think that multi-frame Localizer could be more powerful once we have enough temporal grounding annotations. We leave the improvement of the multi-frame Localizer to future work.\n\n| **Answerer** | **# frames of Localizer** | **NExT-QA (Avg.)** |\n|----|------------------------|--------------------|\n| zero-shot | 1 | 64.6 |\n| zero-shot | 4 | 63.6 (-1.0) |\n| fine-tuned | 1 | 73.4 |\n| fine-tuned | 4 | 71.3 (-2.1) |\n\n### **Weakness 2**\n> The proposal is not good at dealing with temporal causality. Because Blip-2 is an image-text MLLM model, there is no space-time inference, and their answer is to directly add the q-former query generated by each frame to the word token of LLM. This is equivalent to letting q-former himself to realize the time level of information understanding, and q-former does not have this ability. Could you make me clear?\n\nPlease see the general response.\n"}, {"review_id": "OxsiN1JUf0", "reviewer": "Reviewer_b5LJ", "summary": "The paper proposes a simple yet effective framework for video localization and question answering using pretrained image-language models. The key idea is to introduce a LOCALIZER module which localizes the keyframes in a video in order to ignore irrelevant frames and better answer the question. The LOCALIZER module is realized with a separate image-language model and the module can be further improved by fine-tuning on the pseudo labels generated by the QA results from the ANSWER module. The proposed framework, SEVILA, achieves state-of-the-art VQA results on multiple challenging benchmarks under both fine-tuned and zero-shot setups.", "questions": "1. Is it necessary to use two different Q-former for the LOCALIZER and ANSWERER? Since both models are image based, what if we share the same Q-former for both modules?\n\n2. What if we make the self-refinement an iterative process? In other words, we refine the ANSWERER module after the self-refinement of LOCALIZER, and vice versa. It's interesting to see when the performance would saturate with such interactive refinement process.\n\nA minor suggestion: I don't think Figure 4 is a good qualitative example because (1) the action of the ladies is barely visible in the small figure, let alone the intention of the action; (2) the question can be answered by only checking the question itself without knowing the visual content, since \"staring out\" and \"hands above their eyes\" are explicated mentioned.", "limitations": "The authors have discussed the limitations of the work and its potential social impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed framework is simple yet effective in extending image-language models to tackle video-related tasks such as video question answering and moment localization. The idea of using the LOCALIZER module to detect keyframes in a long video is technically sound and the results are convincing.\n\n2. The paper provides extensive ablation study to analyze different components of the framework under different setups. E.g., the impact of selecting keyframes for VQA, the impact of self-refinement for the LOCALIZER module, and how the LOCALIZER compared with some existing keyframe localization methods. While there exists many keyframe localization methods, it's interesting to see the language-model-based method used in this paper gets comparable or even better results in the ablation study.\n\n3. The paper is well organized, clearly written and the description of technical details is clear and easy to follow.", "weaknesses": "1. The technical contribution of the work is relatively weak. The idea of selecting keyframes in a video for VQA is straightforward and widely explored in prior work. The proposed self-refinement strategy is also a standard semi-supervised learning approach and there's no new training strategies / techniques proposed for the fine-tuning of image-language models.\n\n2. While I understand the main goal of this work is to utilize image-language models for VQA, the limitation of image-based models for video-related tasks should not be ignored. First, image-based models may not be sufficient to accurately localize the keyframes in a long video. Second, temporal modeling is also missing in the ANSWERER module, which makes the model less capable of modeling temporal dependencies compared with video-based models. These limitations may not be significant in the current VQA benchmarks since the questions are less sensitive to temporal dependencies and reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Weakness 1**\n> The technical contribution of the work is relatively weak. The idea of selecting keyframes in a video for VQA is straightforward and widely explored in prior work. The proposed self-refinement strategy is also a standard semi-supervised learning approach and there's no new training strategies / techniques proposed for the fine-tuning of image-language models.\n\n&nbsp; We agree that keyframe selection in VideoQA is a popular topic as we mentioned in the related works (Line 94-102). \nDifferent from previous works [23, 40, 49] which link an image model (e.g., ViT) and a language model (e.g., BERT) to first select keyframes by vision & text similarity and then answer questions with a third QA model, we self-chained a single image-langue model to tackle both keyframe localization and question answering by prompting LLM. Our SeViLA shows the pre-trained large image-language model with Parameter Efficient Fine-Tuning (PEFT) is able to handle both video localization and question answering in a self-chaining style. \n\n&nbsp; As Reviewer mWtS pointed out, the image-LM for video-language tasks is an important direction since the image-LMs are improving rapidly, and our work shows an efficient way to leverage those image LMs for video-level tasks. Besides, as you mentioned, our work shows that image LM has a strong potential for video localization (Table 4 video moment retrieval result & Table 6 comparison with other keyframe localization methods).\n\n### **Weakness 2**\n\n> First, image-based models may not be sufficient to accurately localize the keyframes in a long video.\n\n&nbsp; We’d like to note that multiple recent works [3,31] have also found the effectiveness and efficiency of Single-frame localizer in various long video QA benchmarks. In our experiment, as listed in Table 3 (and the below table), our single-frame Localizer demonstrates non-trivial performance boosts on a multitude of long video QA benchmarks. \n\n| **Model** | **NeXT-QA (Avg.)** | **STAR (Avg.)** | **How2QA** | **TVQA** |\n|---|---|----|---|---|\n| (zero-shot) | | | | |\n| Answerer-only | 62.4 | 42.2 | 70.8 | 36.6 |\n| SeViLA | 64.6 (+2.2) | 45.5 (+3.3) | 72.9 (+2.1) | 39.1 (+2.5) |\n| (fine-tuning) | | | | |\n| Answerer-only | 72.6 | 62 | 82.2 | 59.8 |\n| SeViLA | 73.8 (+1.2) | 64.9 (+2.9) | 83.6 (+1.4) | 61.6 (+1.8) |\n\n> the limitation of image-based models for video-related tasks should not be ignored.\n\n&nbsp; We acknowledge that the significance of temporal modeling for video-level tasks cannot be overlooked. In various sections, we emphasize its role in video-language tasks and suggest enhancements in temporal designs for the Localizer. In the Limitation (supp Line 136-139), we also point out the shortcoming and failure cases of the current single-frame Localizer. In Section 4.5 (Line 248-253) and Table 4 (results on video moment retrieval), we suggest more temporal designs for the Localizer to improve language-aware localization. Moreover, our analysis in Section 4.6 (Line 294-299), Table 7 (oracle performance analysis) and the Limitation (supp Line 136-139) indicates potential areas of improvement and underscores the importance of future research in temporal localization.\n\n&nbsp; Furthermore, in this rebuttal, we've expanded our Localizer to a multi-frame mode by concatenating frames into a long image **before** Q-Former. This allows for full attention across all frames, enabling temporal modeling. The results are the following, we find that the 4-frame localizer performs worse than the single-frame localizer in both zero-shot and finetuning settings.\n\n| **Answerer** | **# frames of Localizer** | **NExT-QA (Avg.)** |\n|----|----|----|\n| zero-shot | 1 | 64.6 |\n| zero-shot | 4 | 63.6 (-1.0) |\n| fine-tuned | 1 | 73.4 |\n| fine-tuned | 4 | 71.3 (-2.1) |\n\n> Second, temporal modeling is also missing in the ANSWERER module, which makes the model less capable of modeling temporal dependencies compared with video-based models. \n\nPlease see the general response.\n\n### **Question 1**\n> What if we share the same Q-former for both modules?\n\n&nbsp; Please note that Table 5 row 1 shows the setting where both Localizer and Answerer share the entire parameters, including Q-former. The model achieves 62.9% on NeXT-QA and 70.7% on How2QA, outperforming strong InternVideo and BLIP-2 voting baselines. We will also experiment with multi-task finetuning and discuss the results in the paper.\n\n### **Question 2**\n> What if we make the self-refinement an iterative process? \n\n&nbsp; This is an interesting idea, thanks for your suggestion! We conducted experiments of iterative self-refinement and show the results in the table below. We found that self-refinement helps until two iterations and saturates afterward. We will incorporate this result in the paper.\n\n| **Iteration** | **NExT-QA (Avg.)** |\n|-----------|--------------------|\n| 1 (current) | 73.8 |\n| 2 | 74.2 |\n| 3 | 73.7 |\n\n### **Minor Suggestion**\n> Qualitative visualization example\n\n&nbsp; Thanks for your suggestion on the qualitative example, we argue that even in this example visual content is necessary, because the correct answer \"to see better\" and not \"to pose for photo\" is only clear once the video shows that they are playing golf or there is no photographer. \n\n&nbsp; Please note that we have additional visualization examples in the supplementary Figure 3, where option events are all possible to happen without seeing videos and our model localizes the correct moment and gives the right answer.\n```\nQ: What did both of them do after completing skill?\nA1: jump A2: bend down A3: raise their hands A4: turn around A5: take off clothes\n```\n\nWe will update the qualitative example in the main paper with the new example in the rebuttal PDF to show the effectiveness of our method better. \n"}, {"review_id": "UjG9aYpPsh", "reviewer": "Reviewer_wzBa", "summary": "The paper presents the SeViLA framework, which addresses the limitations of existing image-language models for video question answering. It introduces two modules, Localizer and Answerer, that are fine-tuned from a pre-trained image-language model. SeViLA utilizes chaining for cascaded inference and self-refinement. The Localizer identifies language-aware keyframes, and the Answerer predicts the answer based on these keyframes. Additionally, the Answerer generates keyframe pseudo-labels to refine the Localizer, eliminating the need for costly video moment annotations. ", "questions": "Please see weaknesses.", "limitations": "Please see weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe paper is well-written and easy to understand.\n2.\tThe concept of localizing and then answering is intuitive and effective.\n3.\tExtensive experiments demonstrate that SeViLA achieves state-of-the-art performance on challenging video question answering benchmarks. The paper also provides a comprehensive analysis, including comparisons with other localization models and the impact of keyframe variations.\n", "weaknesses": "1.\tThe motivation and LGDN (which also addresses VideoQA tasks) are quite similar. While the paper mentions LGDN in the introduction and related work sections (which I appreciate), I suggest the authors further clarify the differences between the two approaches in the related work section.\n2.\tThe self-chaining approach requires additional computational resources (key frames need to be recomputed). The paper should provide information on the additional computational costs involved.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Weakness 1**\n\n> The motivation and LGDN are quite similar. While the paper mentions LGDN in the introduction and related work sections (which I appreciate), I suggest the authors further clarify the differences between the two approaches.\n\n&nbsp; While LDGN selects salient frames with two different modules an image model (ViT) and language model (BERT), and answers questions by a QA model (Transformer), with multiple training objectives, our SeViLA proposes a simple and unified self-chaining of a single image-language model (BLIP-2) to tackle both (1) keyframe localization task and (2) question answering task. We will add more clarification of this in the paper.\n\n&nbsp; As pointed out by Reviewer mWtS, leveraging of image-language model for video-language tasks is an important further direction, and as recognized by Reviewer TABX and b5LJ, our work shows a novel self-chained way to effectively use a single large image-language model for video-language tasks. \n\n### **Weakness 2**\n> The self-chaining approach requires additional computational resources (key frames need to be recomputed). The paper should provide information on the additional computational costs involved.\n\n&nbsp; In the following table, we show memory, running time, and parameter size (ViT + FlanT5 + Qformer) comparison between our Answerer and our Localizer+Answerer. As the Localizer and Answerer share the most parameters, adding Localizer has a very small additional memory footprint.\n\n| **Model** | **Memory (GB)** | **Running Time (sec./sample)** | **Parameter (B)** |\n|----|--|--------------|-------------------|\n| Answerer (4) | 7.56 | 1.79 | 4.1 |\n| Localizer + Answerer (SeViLA, 32->4) | 7.98 |  3.28 | 4.2 |\n"}, {"review_id": "pCQZcpE1iC", "reviewer": "Reviewer_mWtS", "summary": "This work builds a joint model for temporal language grounding and video question answering. The model is built on a state-of-the-art image-language model, BLIP-2, and finetunes it in a parameter-efficient way to derive a localizer and an answerer. The localizer finds language-aware keyframes in a video and the answerer uses these frames to predict the answer. The answerer is also used to generate pseudo-labels to refine the localizer. Experiments on multiple video question answering datasets in both finetuning and zero-shot settings show that the proposed method achieves state-of-the-art results and improves slightly over the BLIP-2 baseline.\n", "questions": "See weaknesses.", "limitations": "Limitations and potential negative societal impact are discussed in the Supplementary Material.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Leveraging image-language models for video-language tasks is an important direction, and localizing language-aware keyframes seems a good idea to achieve this. \n- The method achieves strong results on a wide variety of benchmarks in both finetuning and zero-shot mode.\n", "weaknesses": "- Most of the improvement compared to state-of-the-art methods come from the use of the BLIP-2 model. The proposed method only show small improvement over BLIP-2+concat in finetuning setting (see Table 1).\n- Can the method generalize with other image-language models, e.g. BLIP, and if so how does it perform?\n- The localizer relies on manual temporal language grounding annotations from QVHighlights. Could it benefit from weak supervision from e.g. HowTo100M like Moment-DETR?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Weakness 1**\n> Most of the improvement compared to state-of-the-art methods comes from the use of the BLIP-2 model. The proposed method only shows small improvement over BLIP-2+concat in fine-tuning setting (see Table 1).\n\n&nbsp; Our SeViLA consists of Localizer + Answerer, where both have BLIP-2 (visual encoder + Q-former + LLM) architecture, with separate Q-former parameters. Note that the ‘**BLIP2 concat**’ in Tables 1&2 is equivalent to our '**Answerer**' as a contribution that extends BLIP-2 to adapt to video. \n\n&nbsp; Our Answerer (BLIP-2 concat) performs better than the original BLIP-2 with single frame inputs (BLIP-2 voting). And our SeViLA which incorporates our Localizer with our Answerer further boosts the performance. In both fine-tuning and zero-shot settings (Tables 1&2, and as follows), our SeViLA outperforms the original BLIP-2 (BLIP-2-voting) with non-trivial improvements.\n\n| **Model** | **NeXT-QA (Avg.)** | **STAR (Avg.)** | **How2QA** | **TVQA** |\n|----|----|----|----|----|\n| (Zero-shot) | | | | |\n| BLIP-2 voting | 62.7 | 40.3 | 69.8 | 35.7 |\n| SeViLA | 63.6 (+0.9) | 44.6 (+4.3) | 72.3 (+2.5) | 38.2 (+2.5) |\n| (Fine-tuning) | | | | |\n| BLIP-2 voting | 70.1 | 51.8 | 79.6 | 54.5 |\n| SeViLA | 73.8 (+3.7) | 64.9 (+13.1) | 83.6 (+4.0) | 61.6 (+7.1) |\n\n\n### **Weakness 2**\n> Can the method generalize with other image-language models, e.g. BLIP, and if so how does it perform?\n\n\n&nbsp; As you suggested, we experimented with extending SeViLA with another recent Image-Language model (MiniGPT4 [a]), and show the results as follows. We find our proposed self-chaining scheme can also improve the performance of zero-shot MiniGPT4, and we expect that finetuning the model will even improve the performance.\n\n| **Model** | **NExT-QA (Avg.)** |\n|----------------------------------------|--------------------|\n| MiniGPT4 Answerer | 52.7 |\n| MiniGPT4 Localizer + MiniGPT4 Answerer | 53.5 |\n\nRef:\n[a] D. Zhu et.al. MiniGPT-4: Enhancing Vision-language Understanding with Advanced Large Language Models. ArXiv pre-print, 2023.\n\n### **Weakness 3**\n> The localizer relies on manual temporal language grounding annotations from QVHighlights. Could it benefit from weak supervision from e.g. HowTo100M like Moment-DETR?\n\n&nbsp; We’d like to note that our localizer brings 0.5% (62.4 in Table 3 row B v.s. 62.9 in Table 5) boost on NeXT-QA, without pretraining. Our localizer even outperforms the Moment-DETR pre-trained on HowTo100M + QVH in Table 6 (comparison with other localization methods). \nThis shows that our temporal grounding method is already effective even without manual annotations.\n\n&nbsp; Following your suggestion, we also explored weakly-supervised pretraining using ASR similar to Moment-DETR. As shown in the following table, our Localizer performance improves with the weakly supervised pretraining, closing the gap with the pretraining with manual annotations (QVH).\n\n| **Localizer** | **NeXT-QA (Avg.)** |\n|--------------------------------------|--------------------|\n| w/o Localizer | 62.4 |\n| Moment-DETR | 62.0 |\n| Our Localizer (without pre-training) | 62.9 |\n| Our Localizer (weakly pre-trained with QVH ASR) | 63.2 |\n| Our Localizer (pre-trained with QVH) | 63.6 |"}]

Author rebuttal:
We thank the reviewers for their time and valuable comments. We appreciate that reviewers recognized:
- motivation of our localization+answering design (b5LJ, wzBa)
- novelty of our self-chaining framework design (TABX)
- strong experimental results (TABX, b5LJ, wzBa, mWtS)
- extensive ablation studies and analysis (TABX, b5LJ, wzBa)
- well-organized presentation, clear description (b5LJ, wzBa)
- contribution in extending the image-language model for video-language tasks (b5LJ, mWtS).

In the responses, we include the following discussion/experiments.
- temporal modeling in both Localizer and Answerer (TABX, b5LJ)
- our novelty compared with prior work (wzBa)
- exp1: single-frame v.s. multi-frame Localizers (TABX)
- exp2: Single model version - Localizer and Answerer sharing Q-former (b5LJ)
- exp3: iterative self-refinement (b5LJ)
- exp4: SeViLA with another Image-LM (MiniGPT4) (mWtS)
- exp5: weakly supervised pre-training of Localizer (mWtS)

**Please also find the attached PDF for an additional qualitative example (Reviewer b5Lj).**

Below we answer to a common question about **Temporal modeling in Answerer** by Reviewer TABX W2 and Reviewer b5LJ W2

Reviewer TABX W2:
> The proposal is not good at dealing with temporal causality. Because Blip-2 is an image-text MLLM model, there is no space-time inference, and their answer is to directly add the q-former query generated by each frame to the word token of LLM. This is equivalent to letting q-former himself to realize the time level of information understanding, and q-former does not have this ability. Could you make me clear?

Reviewer b5LJ W2:
> Second, temporal modeling is also missing in the Answerer module, which makes the model less capable of modeling temporal dependencies compared with video-based models. These limitations may not be significant in the current VQA benchmarks since the questions are less sensitive to temporal dependencies and reasoning.

&nbsp; Our Answerer conducts temporal modeling with multiple frames via LLM rather than Q-former. As shown in Figure 2 and Line 147-149, Answerer Q-former does not take multiple frames at once but processes them into query features independently, and then those features are concatenated in a temporal order to feed into LLM with Frame ID hints. (Note that the ‘**BLIP2 concat**’ in Tables 1&2 is equivalent to our '**Answerer**' as a contribution that extends BLIP-2 to adapt to video). 

&nbsp; According to fine-tuning results in Table 1 (and in the table below), we find that LLM in our Answerer has the capabilities of temporal understanding with multiple frame inputs, since our Answerer performs better than the BLIP-2 voting, which outputs answers by majority voting with the original BLIP-2 taking a single frame input: 

| **Model** | **NeXT-QA (Avg.)** | **STAR (Avg.)** | **How2QA** | **TVQA** |
|----|----|----|----|----|
| BLIP-2 voting | 70.1 | 51.8 | 79.6 | 54.5 |
| Our Answerer | 72.6 (+**2.5**) | 62.0 (+**10.2**) | 82.2 (+**2.6**) | 59.8 (+**5.3**) |

&nbsp; Such a temporal understanding ability via LLM is evident in the case of the STAR-Sequence, a task demanding heavy temporal understanding, our Answerer outperforms BLIP-2 voting by a significant **13.1%** (fine-tuning Table 1) and **5.3%** (zero-shot Table 2). Those observations are also highlighted in Section 4.2.a where we discussed the importance of temporal modeling (Line 190-193). 


Meta-review: This paper has 4 positive ratings: 2 Borderline Accepts, and 2 Week Accepts. Before the rebuttal, the main concerns about this work are: 1) The novelty is limited compared to previous work. 2) The temporal modeling ability of the proposed method is limited. 3) More comprehensive experimental results are needed. After the rebuttal, all reviewers think most of their concerns have been addressed, and keep a positive rating for this paper. In summary, I recommend accepting this paper, and hope the authors can incorporate all these discussions into the final camera-ready version.
Decision: Accept (poster)