SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shoubin Yu Jaemin Cho Prateek Yadav Mohit Bansal
{shoubin, jmincho, praty, mbansal}@cs.unc.edu
Recent studies have shown promising results on utilizing large pre-trained image-language models for video question answering. While these image-language models can efficiently bootstrap the representation learning of video-language models, they typically concatenate uniformly sampled video frames as visual inputs without explicit language-aware, temporal modeling. When only a portion of a video input is relevant to the language query, such uniform frame sampling can often lead to missing important visual cues. Although humans often find a video moment to focus on and rewind the moment to answer questions, training a query-aware video moment localizer often requires expensive annotations and high computational costs. To address this issue, we propose Self-Chained Video Localization-Answering (SEViLA), a novel framework that leverages a single image-language model (BLIP-2) to tackle both temporal keyframe localization and question answering on videos. SEViLA framework consists of two modules: LOCALIZER and ANSWERER, where both are parameter-efficient fine-tuned from BLIP-2. We propose two ways of chaining these modules for cascaded inference and self-refinement. First, in the forward chain, the LOCALIZER finds multiple language-aware keyframes in a video, which the ANSWERER uses to predict the answer. Second, in the reverse chain, the ANSWERER generates keyframe pseudo-labels to refine the LOCALIZER, alleviating the need for expensive video moment localization annotations. Our SEViLA framework outperforms several strong baselines/previous works on five challenging video question answering and event prediction benchmarks, and achieves the state-of-the-art in both fine-tuning (NExT-QA and STAR) and zero-shot (NExT-QA, STAR, How2QA, and VLEP) settings. We show a comprehensive analysis of our framework, including the impact of LOCALIZER, comparisons of LOCALIZER with other temporal localization models, pre-training/self-refinement of LOCALIZER, and varying the number of keyframes. $^1$
The recent success of large-scale pre-trained language models [2, 7, 65] has led to a burst of multimodal vision-and-language models that can jointly understand visual (image/video) and language data [64, 6, 60]. However, due to higher computational and annotation costs, video-language models (video-LMs) are more challenging to scale in terms of model and data size than image-language models (image-LMs). Hence, recent studies have explored efficient training of video-LMs by leveraging pre-trained image-LMs [44, 14, 23, 85, 70, 31, 19, 82]. While such a warm-start strategy facilitates visual representation learning of video-LMs, they typically concatenate uniformly/randomly sampled video frames as visual inputs without explicit language-aware, temporal modeling. However, such a simple uniform/random sampling of frames can lead to losing important visual cues, resulting in the video-LMs focusing on frames that are unimportant/irrelevant to language queries [42].
1Our code and checkpoints are available at: https://github.com/Yui010206/SeViLA
Forward Chain Language-aware Temporal Localization
Reverse Chain Refine Localizer with Pseudo-labels
Figure 1: Self-Chained Video Localization-Answering (SEVILA) consists of a LOCALIZER and an ANSWER. Left: Forward chain for language-aware temporal keyframe localization and question answering. Right: Reverse chain for LOCALIZER self-refinement with keyframe pseudo-labels.
To address this, we introduce Self-Chained Video Localization-Answering (SEViLA), a novel video-language framework where we adopt a single image-LM to handle both temporal localization and question answering on videos, while avoiding expensive language-aware, temporal grounding annotations (plus self-refinement [48] between the two modules). Our SEViLA framework obtains two modules, a LOCALIZER and an ANSWERER through parameter-efficient fine-tuning [62] of BLIP-2 [35], a recent state-of-the-art image-LM. SEViLA tackles video-language tasks by chaining the output of LOCALIZER to the input of ANSWERER (forward chain, Fig. 1 left), while the ANSWERER gives feedback to refine the LOCALIZER (backward chain, Fig. 1 right). In the forward chain, LOCALIZER leverages the original image-language understanding of the BLIP-2 backbone and chooses the important language-aware video keyframes via the localization prompt "Does the information within the frame provide the necessary details to accurately answer the given question?" for each video frame. Then ANSWERER takes the concatenation of selected keyframes as visual input to predict video-level answers. In the backward chain, we generate keyframe pseudo-labels [26] to refine the LOCALIZER, where we label a video frame as a keyframe if ANSWERER can output the correct answer using that frame. This self-refinement improves the language-aware temporal localization accuracy and alleviates the need for expensive keyframe annotations.
We demonstrate the effectiveness of SEViLA framework on five challenging video question answering and event prediction benchmarks (NExT-QA, STAR, How2QA, TVQA, and VLEP) [75, 77, 36, 27, 28], where SEViLA outperforms several strong baselines/previous works, and achieves the state-of-the-art in both fine-tuning (NExT-QA and STAR) and zero-shot (NExT-QA, STAR, How2QA, and VLEP) settings. We also show that our LOCALIZER can be used as a strong stand-alone moment retrieval model. We present a comprehensive analysis to elaborate the design choices of the proposed framework, including the impact of temporal localization, the impact of the self-refinement process (backward chain), and varying the number of keyframes. Our contributions are summarized as follows:
- A novel video-language framework SEViLA, where the LOCALIZER and ANSWERER are initialized from a single image-language model to handle temporal localization and question answering on videos, respectively.
- A new self-refinement method for language-aware temporal keyframe localization, where the ANSWERER generates keyframe pseudo-labels to refine the LOCALIZER, without expensive temporal grounding annotation.
- Strong empirical performance with state-of-the-art on multiple video-language benchmarks.
- Comprehensive analysis elaborating the design choices of the proposed framework.
Image-Language Pre-trained Models. As the demand for cross-modal applications continues to grow, image-language pre-training studies have received tremendous attention and success. Image-
language pre-trained models [55, 12, 69, 5, 36, 64, 31, 34, 86] have advanced more rapidly than video-language pre-trained models [71, 90, 45, 17, 80, 67, 81, 83, 87, 37, 88], both in terms of model [61, 38, 35, 1, 70, 43] and pre-training data scale [61, 1, 89, 21] (more detailed model size and pre-training data scale comparisons are in Appendix). This can be attributed to the increased accessibility of image data and the comparatively simpler data structures, which makes scaling up image-language learning easier [58]. In our paper, SEViLA is built on the recent state-of-the-art image-LM BLIP-2 [35] and extends it to adopt video input for video-language tasks. We also compare our SEViLA framework with the current state-of-the-art video-LM, InternVideo [71], to demonstrate the superiority of a large image-LM that incorporates keyframe localization.
Image-to-Video Transfer Learning. The gap between image- and video-language models has inspired numerous useful methods focusing on image-to-video transfer learning, which leverage a limited number of video frames to enhance learning efficiency [80, 32, 23, 46, 15, 44, 14, 31, 4, 82, 68]. Luo et al. [44] adapt pre-trained CLIP [55] backbone to solve video clip retrieval. Yang et al. [85] extend frozen bidirectional language models [66] to incorporate multiple images and apply additional video-level pre-training to facilitate model adaptation [62]. Wang et al. [72] convert multiple images into hierarchical captions, arranged with a temporal order prompt to help language models comprehend video-level events. However, these works employ a uniform sampling strategy that is not language-aware. This can lead to the loss of key visual cues for temporal modeling and even burden the models with irrelevant information [31, 76]. In this paper, we propose a LOCALIZER to provide language-aware visual information to video-language tasks.
Language-aware Keyframe Localization. Many methods [42, 3, 18, 54, 24, 42, 41, 73, 9] have been proposed to address the challenge of language-aware keyframe localization. Buch et al. [3] optimized an end-to-end pipeline using answer labels to select a single keyframe for downstream tasks. Lu et al. [42] selects frames with separate image and language models, and answers questions by a QA model with multiple training objectives. Qian et al. [54] designed a video clip proposal model with predefined ranges, iteratively training it with a QA model. Kim et al. [24] utilized a semi-parametric retriever to obtain keyframes based on frame and language feature similarity. We adopt a large image-LM as our LOCALIZER and chain it with an ANSWERER. Our LOCALIZER can help to fine-tune ANSWERER in the forward chain and be refined with pseudo-labels in the reverse chain.
3 Method: SEViLA
In this section, we introduce the method details of our Self-Chained Video Localization-Answering (SEViLA) framework. First, we provide BLIP-2 preliminaries, which serve as the foundation for our framework. Then we elaborate our design of the BLIP-2 LOCALIZER and BLIP-2 ANSWERER for temporal localization and question answering on videos. Finally, we present the SEViLA framework's training and inference processes in the forward and reverse chain.
3.1 Preliminaries: BLIP-2
We adopt BLIP-2 [35] as the backbone of our SEViLA framework. BLIP-2 is a recent state-of-the-art pre-trained image-language model (image-LM) comprising of: (1) a frozen image encoder [11, 16]; (2) a frozen large language model (LLM) [7, 91]; and (3) a Q-former, which is a trainable transformer [66] module that bridges the image encoder and LLM, similar to acting as an adapter [62, 20]. It takes as input visual features  $h$  from the image encoder and learnable query embeddings  $q$ , and outputs fixed-length visual features  $v$ . The BLIP-2 Q-Former undergoes a two-stage pre-training. First, it connects to the image encoder to perform image-to-text pre-training. This stage enables the Q-Former to extract the most informative visual information for the text and remove any irrelevant details in  $v$ . Subsequently, the Q-Former is connected to the LLM to leverage its generative language capabilities. This is achieved using a fully-connected layer to project query embeddings into the LLM's dimension with image-to-text pre-training. As a result, these query features serve as soft visual prompts [22] for the LLM. With the two-stage pre-trained Q-former and LLM, BLIP-2 shows advanced performance on various image-language tasks. In our SEViLA framework, we adopt BLIP-2 as the basic building block for both video temporal localization and question answering modules. We retain the visual encoder and the LLM from BLIP-2 by keeping them frozen during training. In this case, only these two Q-formers are updated during the LOCALIZER and ANSWERER training (see Sec. 3.3).
Figure 2: In SEViLA framework, LOCALIZER (top) selects top-K video frames, which guides ANSWERER (bottom) to focus on important language-aware video moments and predict answers. Both LOCALIZER and ANSWERER are initialized from a single pre-trained BLIP-2 model, where only Q-formers and a linear layer (2.5% of total parameters) are tuned for each module. We omit the linear layer after the Q-former for simplicity.
3.2 Self-Chained Video Localization-Answering
Adapting BLIP-2 to Temporal Localization and Question Answering on Videos. As illustrated in Fig. 2, our SEViLA framework adopts BLIP-2 to tackle both video temporal localization and question-answering. We assign BLIP-2 two roles of being a LOCALIZER and an ANSWERER by using different Q-formers. We first elaborate our LOCALIZER and ANSWERER in detail as follows:
LOCALIZER. We first extract frame features via the frozen image-encoder ViT [16], referred to  $E_{v}$ . Given the video, we uniformly sample  $n$  frames  $\{f_1,\dots,f_n\}$ . We then obtain  $i_{th}$  frame feature  $h_i$  as  $h_i = E_v(f_i)$ . Finally, we represent the video as a set of frame features  $V = \{h_1,\dots,h_n\}$ . These features are extracted once and then saved to be subsequently reused by the LOCALIZER and the ANSWER. The primary objective of the LOCALIZER is to select  $k$  language-aware keyframe features from  $V$ , where  $k$  is typically much smaller than  $n$ . As illustrated in Fig. 2 (top), we then independently extract visual query features  $v_{i}$  from original frame features in  $V$  via a Q-Former  $Q_{loc}$ . Next, visual query features  $v_{i}$  and language contexts  $L$  are concatenated and fed into the LLM (Flan-T5 [7]), where we create  $L$  by combining question, options, and localization prompt "Does the information within the frame provide the necessary details to accurately answer the given question?" The LOCALIZER outputs the score for each frame  $s_i$ , which is the probability of generating a word 'yes', given the visual features  $v_{i}$  and language context  $L$ :  $s_i = LLM(\text{concat}(v_i,L))$ . We can localize language-aware keyframes  $K = \{v_1^k,\dots,v_K^k\}$ , based on the top-k frame scores. Our LOCALIZER can be formulated as:
$$
K = \operatorname {L o c a l i z e r} (V, L), \quad | K | = k \ll n \tag {1}
$$
ANSWERER. With the keyframe set  $K$  obtained from the LOCALIZER, as illustrated in Fig. 2 (bottom), we can proceed to generate answers using the ANSWERER. We first obtain keyframe query features  $v_{i}$  by processing them through  $Q_{ans}$ , following the same procedure used in the LOCALIZER. Next, we feed the LLM with all query features and language contexts by concatenating them and obtain the video-level answer  $a = LLM(\text{concat}(v_{1}^{k}, \dots, v_{K}^{k}, L))^{2}$ . Then, the frozen LLM can conduct temporal modeling with multiple frame inputs. Our ANSWERER can be formulated as:
$$
a = \operatorname {A N S W E R E R} (K, L) \tag {2}
$$
2We also insert a frame ID token before each frame, where we omit the notations from the equation for simplicity.
Forward Chain Inference & Fine-tune Answerer
Figure 3: Top: In the forward chain, the LOCALIZER finds multiple language-aware keyframes, then the ANSWERER utilizes these keyframes to predict answers. We use the forward chain for both inference and ANSWERER fine-tuning. Bottom: In the reverse chain, we generate keyframe pseudo-labels by using the ANSWERER to refine the LOCALIZER.
3.3 Training ANSWERER and LOCALIZER via Self-Chaining
Fine-tuning ANSWERER in forward chain. As illustrated in Fig. 3 (top), we fine-tune the ANSWERER on downstream tasks using keyframes from LOCALIZER via the forward chain. ANSWERER takes the keyframes generated by LOCALIZER. We compare the default setting to other settings (e.g., input frames are uniformly selected) in Appendix.
Refining LOCALIZER in reverse chain. We adopt pseudo-labeling [26] in our reverse chain to address the costly frame-level localization annotations. We use binary pseudo-labels, where we label a video frame as a keyframe if ANSWERER can output the correct answer using that frame. As shown in Fig. 3 (bottom), The frozen ANSWERER is first prompted by a QA task prompt and generates a frame-level answer, then we obtain pseudo labels by comparing this prediction with the ground-truth answer. The LOCALIZER is trained to locate language-aware pseudo-label keyframes.
Pre-training LOCALIZER with moment retrieval label. To enhance our LOCALIZER, we conduct transfer learning from a video moment retrieval/grounding task [56, 30] via pre-training. We use videos, queries, and video-level temporal span labels from QVHighlights [30], and assign a binary localization label to each frame by comparing its timestamp to the span annotations. We provide more details of pre-training in Appendix.
In this section, we first outline our experimental setup (Sec. 4.1). Then, we demonstrate the superiority of SEViLA framework on 5 challenging long-form video question answering and event prediction benchmarks in both fine-tuning (Sec. 4.2) and zero-shot (Sec. 4.3) settings. We also conduct ablation studies on SEViLA framework to show the effectiveness of each of its components on downstream tasks (Sec. 4.4). Next, We report the performance of our LOCALIZER on video moment retrieval (Sec. 4.5). Lastly, we perform in-depth quantitative, and qualitative analyses on our LOCALIZER to show the effect of our design for temporal keyframe localization (Sec. 4.6 and Appendix). More results on single v.s. multi-frame LOCALIZER, pre-training strategies, iterative self-refinement, computational cost, and extension to another Image-LM model are in Appendix.
**Benchmarks.** We evaluate our SEViLA framework on 3 video-language tasks, including multi-choice Video Question Answering (NExT-QA [77], STAR [75], How2QA [36], TVQA [27]), Video Event Prediction (VLEP [28]), and Moment Retrieval (QVHighlights [30]). See Appendix for details.
Table 1: Fine-tuning results on video question answering (NExT-QA, STAR, How2QA, TVQA) and video event prediction (VLEP). We gray out the methods take extra speech input or use dense frames. We bold the best numbers, and underlined the second-best numbers. dense/1fps: the model takes dense (1fps) video frames instead of a fixed number of frames.  $32 \rightarrow 4$ : our LOCALIZER selects 4 keyframes from 32 frames. * represents the results tested by ourselves. SEViLA† uses the zero-shot LOCALIZER without refining on pseudo-labels via the reverse chain.
Model (# Frames)	NExT-QA	STAR	How2QA	TVQA	VLEP
Tem.	Cau.	Des.	Avg.	Int.	Seq.	Pre.	Fea.	Avg.
(w/ speech input or use dense frames)
HERO (dense/1fps) [36]	-	-	-	-	-	-	-	-	73.8	73.6	-
JustAsk (20) [84]	51.4	49.6	63.1	52.3	-	-	-	-	84.4	-	-
FrozenBiLM (10) [85]	-	-	-	-	-	-	-	-	86.7	82.0	-
VidIL 4-shot (12) [72]	-	-	-	-	-	-	-	-	-	-	72.0
T+T (dense/1fps) [40]	-	-	-	-	-	-	-	-	92.4	-	-
T+T (+ASR, dense/1fps) [40]	-	-	-	-	-	-	-	-	93.2	-	-
Flamingo-80B 32-shot (30) [1]	-	-	-	-	-	-	-	42.2	-	-	-
FrozenBiLM (10) [85]	-	-	-	-	-	-	-	-	81.5	57.5	-
All-in-One (32) [67]	48.6	48.0	63.2	50.6	47.5	50.8	47.7	44.0	47.5	-	-
Temp[ATP] (32) [3]	49.3	48.6	65.0	51.5	50.6	52.8	49.3	40.6	48.3	-	-
VGT (32) [78]	55.0	52.2	64.0	55.0	-	-	-	-	44.2	-	-
MIST (32) [18]	56.6	54.6	66.9	57.1	55.5	54.2	54.2	44.4	51.1	-	-
VFC (32) [50]	53.3	57.6	72.8	58.6	-	-	-	-	-	-	-
CoVGT (32) [79]	57.4	58.8	69.3	60.0	-	-	-	-	45.9	-	-
SeViTFiD (10) [24]	-	-	-	60.6	-	-	-	-	-	-	-
HiTeA (16) [87]	58.3	62.4	75.6	63.1	-	-	-	-	-	-	-
InternVideo* (8) [71]	58.5	62.5	75.8	63.2	62.7	65.6	54.9	51.9	58.7	79.0	57.2
BLIP-2voting (4)	65.2	70.1	80.1	70.1	52.3	54.8	49.0	51.2	51.8	79.6	54.5
BLIP-2concat (ANSWERER) (4)	68.1	72.9	81.2	72.6	65.4	69.0	59.7	54.2	62.0	82.2	59.8
SEViLAd(32→4)	68.8	73.4	83.5	73.4	63.2	66.6	61.3	60.0	62.7	83.7	59.7
SEViLA (32→4)	69.4	74.2	81.3	73.8	63.7	70.4	63.1	62.4	64.9	83.6	61.6
Baselines. We compare our SEViLA framework with the state-of-the-art video-language pre-trained model, InternVideo [71] as well as our backbone BLIP-2 [35]. We extend BLIP-2 to adapt videos in two settings: (1) BLIP-2 $^{\text{voting}}$ , which processes each uniformly sampled frame independently and obtains the final answer by majority voting on all frame-level answers, and (2) BLIP-2 $^{\text{concat}}$ , where Q-former processes each frame and Flan-T5 takes the concatenation of visual features as a prefix. BLIP-2 $^{\text{concat}}$  is equivalent to our ANSWERER with uniformly sampled frames. See Appendix for details.
Implementation Details. SEVILA framework adopts BLIP-2 [35], an image-language model with 4.1B parameters and pre-trained on 129M images in total, including COCO [39], Visual Genome [25], CC12M [59], SBU [52], and 115M images from LAION400M [57]. See Appendix for details.
4.2 Fine-tuning Comparison to SOTA on Video QA and Event Prediction
We compare our SEViLA framework to recent state-of-the-art models on 4 video QA benchmarks and 1 video event prediction dataset. We show results in Table 1, and summarize our findings as follows.
(a) Temporal modeling matters. BLIP-2 $^{\text{voting}}$  underperforms our BLIP-2 $^{\text{concat}}$  (ANSWERER) and other video-LM models on STAR, How2QA, TVQA, and VLEP. Especially on STAR-Sequence, a task demanding heavy temporal understanding, our BLIP-2 $^{\text{concat}}$  (ANSWERER) outperforms BLIP-2 $^{\text{voting}}$  significantly by  $13.1\%$  ( $69.0\%$  vs.  $54.8\%$ ). As BLIP-2 $^{\text{voting}}$  processes frames independently and lacks temporal modeling among frames, the result indicates that temporal modeling is essential to tackle video-language tasks and the effectiveness of our temporal modeling design.
(b) Keyframe selection helps. Our SEViLA† framework, featuring a zero-shot LOCALIZER, leads on all tasks with an average advantage of  $5.3\%$  over the top video-LM (InternVideo). It also surpasses BLIP-2concat (ANSWERER) that uses uniform frame sampling on NeXT-QA  $(+1.2\%)$ , STAR  $(+0.7\%)$ , How2QA  $(+1.5\%)$ , and VLEP  $(+0.4\%)$ . This highlights the importance of keyframe selection in video-language tasks, even when using a zero-shot LOCALIZER.
Table 2: Zero-shot results on video question answering and video event prediction.
Model (# Frames)	NExT-QA	STAR	How2QA	TVQA	VLEP
Tem.	Cau.	Des.	Avg.	Int.	Seq.	Pre.	Fea.	Avg.
(w/ speech input or use dense frames)
JustAsk (20) [84]	-	-	-	-	-	-	-	-	51.1	-	-
FrozenBiLM (10) [85]	-	-	-	-	-	-	-	-	58.4	59.2	-
ViperGPT (dense/1fps) [63]	-	-	-	60.0	-	-	-	-	-	-	-
Flamingo-80B (30) [1]	-	-	-	-	-	-	-	39.7	-	-	-
FrozenBiLM (10) [85]	-	-	-	-	-	-	-	-	41.9	29.7	-
VFC (32) [50]	45.4	51.6	64.1	51.5	-	-	-	-	-	-	-
InternVideo* (8) [71]	43.4	48.0	65.1	49.1	43.8	43.2	42.3	37.4	41.6	62.2	35.9
BLIP-2voting (4)	59.1	61.3	74.9	62.7	41.8	39.7	40.2	39.5	40.3	69.8	35.7
BLIP-2concat (ANSWERER) (4)	59.7	60.8	73.8	62.4	45.5	41.8	41.8	40.0	42.2	70.8	36.6
SEViLA† (32→4)	61.3	61.5	75.6	63.6	48.3	45.0	44.4	40.8	44.6	72.3	38.2
(c) Self-refinement improves temporal localization. For SEViLA, we refine the LOCALIZER with pseudo-labels (see Sec. 3.3). Compared with SEViLA†, SEViLA further increases performance on NExT-QA (0.4%), STAR (+2.2%), and TVQA (+1.9%). SEViLA framework achieves new state-of-the-art fine-tuning performance on NExT-QA, STAR, TVQA, and VLEP, using only visual and language modalities. This illustrates the significance of temporal localization and the efficacy of our self-refinement method for keyframe localization.
4.3 Zero-shot Comparison to SOTA on Video QA and Event Prediction
We further compare our SEViLA framework to recent state-of-the-art models in the zero-shot setting. We show the zero-shot results in Table 2, then discuss the findings in the following.
(a) Image-LM outperforms Video-LM, without video pretraining. Surprisingly, BLIP-2 $^{\text{voting}}$ , without inter-frame temporal modeling, outperforms the previous state-of-the-art video-LM, InternVideo on several datasets that require temporal reasoning. BLIP-2 $^{\text{voting}}$  outperforms InternVideo on NExT-QA (+13.6%), How2QA (+7.6%), and VLEP (+5.1%). On How2QA, BLIP-2 $^{\text{voting}}$  even surpasses FrozenBiLM which performs extra speech and video pre-training by 11.4%. It highlights the potential of image-LM for video-language tasks due to its model size and sufficient pre-training.
(b) Keyframe selection is more effective than uniform sampling. Our SEViLA† framework, combining the zero-shot LOCALIZER and the zero-shot ANSWERER, outperforms the BLIP-2concat (ANSWERER) with uniformly sampled frames on NExT-QA (+1.2%), STAR (+2.4%), How2QA (+1.5%), TVQA (+1.6%), and VLEP (+0.4%), achieving new state-of-the-art zero-shot performance on NExT-QA, STAR, How2QA, and VLEP, and new state-of-the-art on TVQAs with only visual and language modalities. On STAR, our SEViLA† framework even outperforms zero-shot Flamingo [1] with 80B parameters by 4.9%. The result demonstrates the effectiveness of our SEViLA framework to adapt video-language tasks and the importance of language-aware keyframe selection.