Title: Self-Chained Image-Language Model for Video Localization and Question Answering
Venue: NeurIPS 2023
Accepted: yes
Keywords: Video Question Answering

Video Localization

Image-Language Model
Abstract: Recent studies have shown promising results on utilizing large pre-trained image-language models for video question answering. While these image-language models can efficiently bootstrap the representation learning of video-language models, they typically concatenate uniformly sampled video frames as visual inputs without explicit language-aware, temporal modeling. When only a portion of a video input is relevant to the language query, such uniform frame sampling can often lead to missing important visual cues. Although humans often find a video moment to focus on and rewind the moment to answer questions, training a query-aware video moment localizer often requires expensive annotations and high computational costs. To address this issue, we propose Self-Chained Video Localization-Answering (SeViLA), a novel framework that leverages a single image-language model (BLIP- 2) to tackle both temporal keyframe localization and question answering on videos. SeViLA framework consists of two modules: Localizer and Answerer, where both are parameter-efficiently fine-tuned from BLIP-2. We propose two ways of chaining these modules for cascaded inference and self-refinement. First, in the forward chain, the Localizer finds multiple language-aware keyframes in a video, which the Answerer uses to predict the answer. Second, in the reverse chain, the Answerer generates keyframe pseudo-labels to refine the Localizer, alleviating the need for expensive video moment localization annotations. Our SeViLA framework outperforms several strong baselines/previous works on five challenging video question answering and event prediction benchmarks, and achieves the state-of-the-art in both fine-tuning (NExT-QA and STAR) and zero-shot (NExT-QA, STAR, How2QA, and VLEP) settings. We show a comprehensive analysis of our framework, including the impact of Localizer, comparisons of Localizer with other temporal localization models, pre-training/self-refinement of Localizer, and varying the number of keyframes.

Self-Chained Image-Language Model for Video Localization and Question Answering

Abstract

Recent studies have shown promising results on utilizing large pre-trained imagelanguage models for video question answering. While these image-language models can efficiently bootstrap the representation learning of video-language models, they typically concatenate uniformly sampled video frames as visual inputs without explicit language-aware, temporal modeling. When only a portion of a video input is relevant to the language query, such uniform frame sampling can often lead to missing important visual cues. Although humans often find a video moment to focus on and rewind the moment to answer questions, training a query-aware video moment localizer often requires expensive annotations and high computational costs. To address this issue, we propose Se lf-Chained Vi deo L ocalizationA nswering (SeViLA), a novel framework that leverages a single image-language model (BLIP2) to tackle both temporal keyframe localization and question answering on videos. SeViLA framework consists of two modules: Localizer and Answerer, where both are parameter-efficiently fine-tuned from BLIP-2. We propose two ways of chaining these modules for cascaded inference and self-refinement. First, in the forward chain, the Localizer finds multiple language-aware keyframes in a video, which the Answerer uses to predict the answer. Second, in the reverse chain, the Answerer generates keyframe pseudo-labels to refine the Localizer, alleviating the need for expensive video moment localization annotations. Our SeViLA framework outperforms several strong baselines/previous works on five challenging video question answering and event prediction benchmarks, and achieves the stateof-the-art in both fine-tuning (NExT-QA and STAR) and zero-shot (NExT-QA, STAR, How2QA, and VLEP) settings. We show a comprehensive analysis of our framework, including the impact of Localizer, comparisons of Localizer with other temporal localization models, pre-training/self-refinement of Localizer, and varying the number of keyframes. 1

1 Introduction

The recent success of large-scale pre-trained language models [2, 7, 65] has led to a burst of multimodal vision-and-language models that can jointly understand visual (image/video) and language data [64, 6, 60]. However, due to higher computational and annotation costs, video-language models (video-LMs) are more challenging to scale in terms of model and data size than image-language models (image-LMs). Hence, recent studies have explored efficient training of video-LMs by leveraging pre-trained image-LMs [44, 14, 23, 85, 70, 31, 19, 82]. While such a warm-start strategy facilitates visual representation learning of video-LMs, they typically concatenate uniformly/randomly sampled video frames as visual inputs without explicit language-aware, temporal modeling. However, such a simple uniform/random sampling of frames can lead to losing important visual cues, resulting in the video-LMs focusing on frames that are unimportant/irrelevant to language queries [42].

GLYPH<c=14,font=/AAAAAB+font000000002a0d573a>

GLYPH<c=14,font=/AAAAAB+font000000002a0d573a>

GLYPH<c=14,font=/AAAAAB+font000000002a0d573a>

,QFRUUHFW

¡

&RUUHFW

GLYPH<c=29,font=/AAAAAB+font000000002a0d573a>GLYPH<c=3,font=/AAAAAB+font000000002a0d573a>)UR]HQ

GLYPH<c=29,font=/AAAAAB+font000000002a0d573a>GLYPH<c=3,font=/AAAAAB+font000000002a0d573a>)LQHWXQHG

/DQJXDJHGLYPH<c=16,font=/AAAAAB+font000000002a0d573a>DZDUHGLYPH<c=3,font=/AAAAAB+font000000002a0d573a>7 HPSRUDOGLYPH<c=3,font=/AAAAAB+font000000002a0d573a>/RFDOL]DWLRQ

/RFDOL]HU

%/,3GLYPH<c=16,font=/AAAAAB+font000000002a0d573a>GLYPH<c=21,font=/AAAAAB+font000000002a0d573a>

$QVZHUHU

%/,3GLYPH<c=16,font=/AAAAAB+font000000002a0d573a>GLYPH<c=21,font=/AAAAAB+font000000002a0d573a>

GLYPH<c=11,font=/AAAAAB+font000000002a0d573a>GLYPH<c=20,font=/AAAAAB+font000000002a0d573a>GLYPH<c=12,font=/AAAAAB+font000000002a0d573a>GLYPH<c=3,font=/AAAAAB+font000000002a0d573a>/RFDOL]HGLYPH<c=3,font=/AAAAAB+font000000002a0d573a>.H\IUDPHV

GLYPH<c=11,font=/AAAAAB+font000000002a0d573a>GLYPH<c=21,font=/AAAAAB+font000000002a0d573a>GLYPH<c=12,font=/AAAAAB+font000000002a0d573a>GLYPH<c=3,font=/AAAAAB+font000000002a0d573a>4$ GLYPH<c=3,font=/AAAAAB+font000000002a0d573a>ZLWKGLYPH<c=3,font=/AAAAAB+font000000002a0d573a>.H\IUDPHV

$QVZHU

4XHVWLRQ

4XHVWLRQ

5HILQHGLYPH<c=3,font=/AAAAAB+font000000002a0d573a>/RFDOL]HUGLYPH<c=3,font=/AAAAAB+font000000002a0d573a>ZLWKGLYPH<c=3,font=/AAAAAB+font000000002a0d573a>3VHXGRGLYPH<c=16,font=/AAAAAB+font000000002a0d573a>ODEHOV

$QVZHUHU

%/,3GLYPH<c=16,font=/AAAAAB+font000000002a0d573a>GLYPH<c=21,font=/AAAAAB+font000000002a0d573a>

*7GLYPH<c=3,font=/AAAAAB+font000000002a0d573a>$QVZHU

GLYPH<c=14,font=/AAAAAB+font000000002a0d573a>

GLYPH<c=14,font=/AAAAAB+font000000002a0d573a>

/RFDOL]HU

%/,3GLYPH<c=16,font=/AAAAAB+font000000002a0d573a>GLYPH<c=21,font=/AAAAAB+font000000002a0d573a>

¡



4XHVWLRQ

$QVZHU

$QVZHU



GLYPH<c=11,font=/AAAAAB+font000000002a0d573a>GLYPH<c=20,font=/AAAAAB+font000000002a0d573a>GLYPH<c=12,font=/AAAAAB+font000000002a0d573a>GLYPH<c=3,font=/AAAAAB+font000000002a0d573a>*HQHUDWHGLYPH<c=3,font=/AAAAAB+font000000002a0d573a>GLYPH<c=3,font=/AAAAAB+font000000002a0d573a>3VHXGRGLYPH<c=16,font=/AAAAAB+font000000002a0d573a>ODEHOV

GLYPH<c=11,font=/AAAAAB+font000000002a0d573a>GLYPH<c=21,font=/AAAAAB+font000000002a0d573a>GLYPH<c=12,font=/AAAAAB+font000000002a0d573a>GLYPH<c=3,font=/AAAAAB+font000000002a0d573a>5HILQHGLYPH<c=3,font=/AAAAAB+font000000002a0d573a>/RFDOL]HU

3VHXGRGLYPH<c=16,font=/AAAAAB+font000000002a0d573a>/DEHOLQJ

IRUGLYPH<c=3,font=/AAAAAB+font000000002a0d573a>.H\IUDPHV

)RUZDUGGLYPH<c=3,font=/AAAAAB+font000000002a0d573a>&KDLQ

5HYHUVHGLYPH<c=3,font=/AAAAAB+font000000002a0d573a>&KDLQ

4XHVWLRQ

)UDPHV

)UDPHV

.H\IUDPHV

To address this, we introduce Se lf-Chained Vi deo L ocalizationA nswering ( SeViLA ), a novel video-language framework where we adopt a single image-LM to handle both temporal localization and question answering on videos, while avoiding expensive language-aware, temporal grounding annotations (plus self-refinement [48] between the two modules). Our SeViLA framework obtains two modules, a Localizer and an Answerer through parameter-efficient finetuning [62] of BLIP-2 [35], a recent state-of-the-art image-LM. SeViLA tackles video-language tasks by chaining the output of Localizer to the input of Answerer ( forward chain , Fig. 1 left), while the Answerer gives feedback to refine the Localizer ( backward chain , Fig. 1 right). In the forward chain , Localizer leverages the original image-language understanding of the BLIP-2 backbone and chooses the important language-aware video keyframes via the localization prompt 'Does the information within the frame provide the necessary details to accurately answer the given question?' for each video frame. Then Answerer takes the concatenation of selected keyframes as visual input to predict video-level answers. In the backward chain , we generate keyframe pseudo-labels [26] to refine the Localizer, where we label a video frame as a keyframe if Answerer can output the correct answer using that frame. This self-refinement improves the language-aware temporal localization accuracy and alleviates the need for expensive keyframe annotations.

We demonstrate the effectiveness of SeViLA framework on five challenging video question answering and event prediction benchmarks (NExT-QA, STAR, How2QA, TVQA, and VLEP) [75, 77, 36, 27, 28], where SeViLA outperforms several strong baselines/previous works, and achieves the state-of-the-art in both fine-tuning (NExT-QA and STAR) and zero-shot (NExT-QA, STAR, How2QA, and VLEP) settings. We also show that our Localizer can be used as a strong stand-alone moment retrieval model. We present a comprehensive analysis to elaborate the design choices of the proposed framework, including the impact of temporal localization, the impact of the self-refinement process (backward chain), and varying the number of keyframes. Our contributions are summarized as follows:

A novel video-language framework SeViLA, where the Localizer and Answerer are initialized from a single image-language model to handle temporal localization and question answering on videos, respectively.

A new self-refinement method for language-aware temporal keyframe localization, where the Answerer generates keyframe pseudo-labels to refine the Localizer, without expensive temporal grounding annotation.

Strong empirical performance with state-of-the-art on multiple video-language benchmarks.

Comprehensive analysis elaborating the design choices of the proposed framework.

2 Related Work

Image-Language Pre-trained Models. As the demand for cross-modal applications continues to grow, image-language pre-training studies have received tremendous attention and success. Image-

language pre-trained models [55, 12, 69, 5, 36, 64, 31, 34, 86] have advanced more rapidly than video-language pre-trained models [71, 90, 45, 17, 80, 67, 81, 83, 87, 37, 88], both in terms of model [61, 38, 35, 1, 70, 43] and pre-training data scale [61, 1, 89, 21] (more detailed model size and pre-training data scale comparisons are in Appendix). This can be attributed to the increased accessibility of image data and the comparatively simpler data structures, which makes scaling up image-language learning easier [58]. In our paper, SeViLA is built on the recent state-of-the-art image-LM BLIP-2 [35] and extends it to adopt video input for video-language tasks. We also compare our SeViLA framework with the current state-of-the-art video-LM, InternVideo [71], to demonstrate the superiority of a large image-LM that incorporates keyframe localization.

Image-to-Video Transfer Learning. The gap between image- and video-language models has inspired numerous useful methods focusing on image-to-video transfer learning, which leverage a limited number of video frames to enhance learning efficiency [80, 32, 23, 46, 15, 44, 14, 31, 4, 82, 68]. Luo et al. [44] adapt pre-trained CLIP [55] backbone to solve video clip retrieval. Yang et al. [85] extend frozen bidirectional language models [66] to incorporate multiple images and apply additional video-level pre-training to facilitate model adaptation [62]. Wang et al. [72] convert multiple images into hierarchical captions, arranged with a temporal order prompt to help language models comprehend video-level events. However, these works employ a uniform sampling strategy that is not language-aware. This can lead to the loss of key visual cues for temporal modeling and even burden the models with irrelevant information [31, 76]. In this paper, we propose a Localizer to provide language-aware visual information to video-language tasks.

Language-aware Keyframe Localization. Many methods [42, 3, 18, 54, 24, 42, 41, 73, 9] have been proposed to address the challenge of language-aware keyframe localization. Buch et al. [3] optimized an end-to-end pipeline using answer labels to select a single keyframe for downstream tasks. Lu et al. [42] selects frames with separate image and language models, and answers questions by a QA model with multiple training objectives. Qian et al. [54] designed a video clip proposal model with predefined ranges, iteratively training it with a QA model. Kim et al. [24] utilized a semi-parametric retriever to obtain keyframes based on frame and language feature similarity. We adopt a large image-LM as our Localizer and chain it with an Answerer. Our Localizer can help to fine-tune Answerer in the forward chain and be refined with pseudo-labels in the reverse chain.

3 Method: SeViLA

In this section, we introduce the method details of our Self-Chained Video Localization-Answering (SeViLA) framework. First, we provide BLIP-2 preliminaries, which serve as the foundation for our framework. Then we elaborate our design of the BLIP-2 Localizer and BLIP-2 Answerer for temporal localization and question answering on videos. Finally, we present the SeViLA framework's training and inference processes in the forward and reverse chain.

3.1 Preliminaries: BLIP-2

We adopt BLIP-2 [35] as the backbone of our SeViLA framework. BLIP-2 is a recent state-of-the-art pre-trained image-language model (image-LM) comprising of: (1) a frozen image encoder [11, 16]; (2) a frozen large language model (LLM) [7, 91]; and (3) a Q-former, which is a trainable transformer [66] module that bridges the image encoder and LLM, similar to acting as an adapter [62, 20]. It takes as input visual features h from the image encoder and learnable query embeddings q , and outputs fixed-length visual features v . The BLIP-2 Q-Former undergoes a two-stage pre-training. First, it connects to the image encoder to perform image-to-text pre-training. This stage enables the Q-Former to extract the most informative visual information for the text and remove any irrelevant details in v . Subsequently, the Q-Former is connected to the LLM to leverage its generative language capabilities. This is achieved using a fully-connected layer to project query embeddings into the LLM's dimension with image-to-text pre-training. As a result, these query features serve as soft visual prompts [22] for the LLM. With the two-stage pre-trained Q-former and LLM, BLIP-2 shows advanced performance on various image-language tasks. In our SeViLA framework, we adopt BLIP-2 as the basic building block for both video temporal localization and question answering modules. We retain the visual encoder and the LLM from BLIP-2 by keeping them frozen during training. In this case, only these two Q-formers are updated during the Localizer and Answerer training (see Sec. 3.3).

Localizer

Answerer

[Question] [Options] [Prompt 2]

[Question] [Options] [Prompt 2]

[Question] [Options] [Prompt 2]

Q-Former

Qloc

[Question] [Options] [Loc Prompt]

frame scores

[0.2,

0.9

,

1.2

, 0.3]

FlanT5

Q-Former

Qans

Frames

ViT

FlanT5

language-aware

keyframe

features

frame

features

[Question] [Options] [QA Prompt]

[Question]

what does the boy do after

unwrapping the present?

[Options]

A:

Show to the camera.  B: Cry.

C: Relax his fingers.       D: Walk alone.

[QA Prompt]

Considering information in frames, select

the correct answer from the options.

[Loc Prompt]

Does the information within the frame provide nece-

ssary details to accurately answer the given question?

TopK Selection

answer: A

learnable

query

q

visual features

v

visual features

v

learnable

query

q

3.2 Self-Chained Video Localization-Answering

Adapting BLIP-2 to Temporal Localization and Question Answering on Videos. As illustrated in Fig. 2, our SeViLA framework adopts BLIP-2 to tackle both video temporal localization and question-answering. We assign BLIP-2 two roles of being a Localizer and an Answerer by using different Q-formers. We first elaborate our Localizer and Answerer in detail as follows:

Localizer. We first extract frame features via the frozen image-encoder ViT [16], referred to E v . Given the video, we uniformly sample n frames { f 1 , ..., f n } . We then obtain i th frame feature h i as h i = E v ( f i ) . Finally, we represent the video as a set of frame features V = { h 1 , ..., h n } . These features are extracted once and then saved to be subsequently reused by the Localizer and the Answerer. The primary objective of the Localizer is to select k language-aware keyframe features from V , where k is typically much smaller than n . As illustrated in Fig. 2 (top), we then independently extract visual query features v i from original frame features in V via a Q-Former Q loc . Next, visual query features v i and language contexts L are concatenated and fed into the LLM (Flan-T5 [7]), where we create L by combining question, options, and localization prompt 'Does the information within the frame provide the necessary details to accurately answer the given question?' . The Localizer outputs the score for each frame s i , which is the probability of generating a word 'yes', given the visual features v i and language context L : s i = LLM ( concat ( v i , L )) . We can localize language-aware keyframes K = { v k 1 , ..., v k K } , based on the top-k frame scores. Our Localizer can be formulated as:

Answerer. With the keyframe set K obtained from the Localizer, as illustrated in Fig. 2 (bottom), we can proceed to generate answers using the Answerer. We first obtain keyframe query features v i by processing them through Q ans , following the same procedure used in the Localizer. Next, we feed the LLM with all query features and language contexts by concatenating them and obtain the video-level answer a = LLM ( concat ( v k 1 , ..., v k K , L )) 2 . Then, the frozen LLM can conduct temporal modeling with multiple frame inputs. Our Answerer can be formulated as:

2SWLRQGLYPH<c=3,font=/AAAAAB+font000000002a0d591b> $

2SWLRQGLYPH<c=3,font=/AAAAAB+font000000002a0d591b> $

*URXQGGLYPH<c=3,font=/AAAAAB+font000000002a0d591b>7 UXWK

)RUZDUGGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>&KDLQ

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

,QIHUHQFHGLYPH<c=3,font=/AAAAAB+font000000002a0d591b>GLYPH<c=9,font=/AAAAAB+font000000002a0d591b>

GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

)LQHGLYPH<c=16,font=/AAAAAB+font000000002a0d591b>WXQHGLYPH<c=3,font=/AAAAAB+font000000002a0d591b> $QVZHUHUGLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

5HYHUVHGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>&KDLQ

5HILQHGLYPH<c=3,font=/AAAAAB+font000000002a0d591b>/RFDOL]HUGLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

,QFRUUHFW

$QVZHUHU

$QVZHUHU

FRUUHFW

/RFDOL]HU

$QVZHUHU

2SWLRQGLYPH<c=3,font=/AAAAAB+font000000002a0d591b> $

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>>4$GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

/RFDOL]HU

3VHXGRGLYPH<c=16,font=/AAAAAB+font000000002a0d591b>/DEHOGLYPH<c=29,font=/AAAAAB+font000000002a0d591b>GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>\HV

3VHXGRGLYPH<c=16,font=/AAAAAB+font000000002a0d591b>/DEHOVGLYPH<c=29,font=/AAAAAB+font000000002a0d591b>GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>QR

2SWLRQGLYPH<c=3,font=/AAAAAB+font000000002a0d591b> $

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>4$GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>4$GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

>4XHVWLRQ@

GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>

>2SWLRQV@GLYPH<c=3,font=/AAAAAC+font000000002a0d591b>

>/RFGLYPH<c=3,font=/AAAAAC+font000000002a0d591b>3URPSW@

2SWLRQGLYPH<c=3,font=/AAAAAB+font000000002a0d591b> $

2SWLRQGLYPH<c=3,font=/AAAAAB+font000000002a0d591b> $

2SWLRQGLYPH<c=3,font=/AAAAAB+font000000002a0d591b> $

2SWLRQGLYPH<c=3,font=/AAAAAB+font000000002a0d591b>%

3VHXGRGLYPH<c=16,font=/AAAAAB+font000000002a0d591b>/DEHOGLYPH<c=29,font=/AAAAAB+font000000002a0d591b>GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>\HV

3VHXGRGLYPH<c=16,font=/AAAAAB+font000000002a0d591b>/DEHOGLYPH<c=29,font=/AAAAAB+font000000002a0d591b>GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>\HV

3VHXGRGLYPH<c=16,font=/AAAAAB+font000000002a0d591b>/DEHOVGLYPH<c=29,font=/AAAAAB+font000000002a0d591b>GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>QR

3VHXGRGLYPH<c=16,font=/AAAAAB+font000000002a0d591b>/DEHOVGLYPH<c=29,font=/AAAAAB+font000000002a0d591b>GLYPH<c=3,font=/AAAAAB+font000000002a0d591b>QR

3.3 Training Answerer and Localizer via Self-Chaining

Fine-tuning Answerer in forward chain. As illustrated in Fig. 3 (top), we fine-tune the Answerer on downstream tasks using keyframes from Localizer via the forward chain. Answerer takes the keyframes generated by Localizer. We compare the default setting to other settings ( e . g ., input frames are uniformly selected) in Appendix.

Refining Localizer in reverse chain. We adopt pseudo-labeling [26] in our reverse chain to address the costly frame-level localization annotations. We use binary pseudo-labels, where we label a video frame as a keyframe if Answerer can output the correct answer using that frame. As shown in Fig. 3 (bottom), The frozen Answerer is first prompted by a QA task prompt and generates a frame-level answer, then we obtain pseudo labels by comparing this prediction with the ground-truth answer. The Localizer is trained to locate language-aware pseudo-label keyframes.

Pre-training Localizer with moment retrieval label. To enhance our Localizer, we conduct transfer learning from a video moment retrieval/grounding task [56, 30] via pre-training. We use videos, queries, and video-level temporal span labels from QVHighlights [30], and assign a binary localization label to each frame by comparing its timestamp to the span annotations. We provide more details of pre-training in Appendix.

4 Experiments

In this section, we first outline our experimental setup (Sec. 4.1). Then, we demonstrate the superiority of SeViLA framework on 5 challenging long-form video question answering and event prediction benchmarks in both fine-tuning (Sec. 4.2) and zero-shot (Sec. 4.3) settings. we also conduct ablation studies on SeViLA framework to show the effectiveness of each of its components on downstream tasks (Sec. 4.4). Next, We report the performance of our Localizer on video moment retrieval (Sec. 4.5). Lastly, we perform in-depth quantitative, and qualitative analyses on our Localizer to show the effect of our design for temporal keyframe localization (Sec. 4.6 and Appendix). More results on single v.s. multi-frame Localizer, pre-training strategies, iterative self-refinement, computational cost, and extension to another Image-LM model are in Appendix.

4.1 Experimental setup

Benchmarks. We evaluate our SeViLA framework on 3 video-language tasks, including multi-choice Video Question Answering ( NExT-QA [77], STAR [75], How2QA [36], TVQA [27]), Video Event Prediction ( VLEP [28]), and Moment Retrieval ( QVHighlights [30]). See Appendix for details.

Baselines. We compare our SeViLA framework with the state-of-the-art video-language pre-trained model, InternVideo [71] as well as our backbone BLIP-2 [35]. We extend BLIP-2 to adapt videos in two settings: (1) BLIP-2 voting , which processes each uniformly sampled frame independently and obtains the final answer by majority voting on all frame-level answers, and (2) BLIP-2 concat , where Q-former processes each frame and Flan-T5 takes the concatenation of visual features as a prefix. BLIP-2 concat is equivalent to our Answerer with uniformly sampled frames. See Appendix for details.

Implementation Details. SeViLA framework adopts BLIP-2 [35], an image-language model with 4.1B parameters and pre-trained on 129M images in total, including COCO [39], Visual Genome [25], CC12M [59], SBU [52], and 115M images from LAION400M [57]. See Appendix for details.

4.2 Fine-tuning Comparison to SOTA on Video QA and Event Prediction

We compare our SeViLA framework to recent state-of-the-art models on 4 video QA benchmarks and 1 video event prediction dataset. We show results in Table 1, and summarize our findings as follows. (a) Temporal modeling matters. BLIP-2 voting underperforms our BLIP-2 concat (Answerer) and other video-LM models on STAR, How2QA, TVQA, and VLEP. Especially on STAR-Sequence, a task demanding heavy temporal understanding, our BLIP-2 concat (Answerer) outperforms BLIP-2 voting significantly by 13.1% (69.0% vs. 54.8%). As BLIP-2 voting processes frames independently and lacks temporal modeling among frames, the result indicates that temporal modeling is essential to tackle video-language tasks and the effectiveness of our temporal modeling design.

(b) Keyframe selection helps. Our SeViLA † framework, featuring a zero-shot Localizer, leads on all tasks with an average advantage of 5.3% over the top video-LM (InternVideo). It also surpasses BLIP-2 concat (Answerer) that uses uniform frame sampling on NeXT-QA (+1.2%), STAR (+0.7%), How2QA (+1.5%), and VLEP (+0.4%). This highlights the importance of keyframe selection in video-language tasks, even when using a zero-shot Localizer.

(c) Self-refinement improves temporal localization. For SeViLA, we refine the Localizer with pseudo-labels (see Sec. 3.3). Compared with SeViLA † , SeViLA further increases performance on NExT-QA (0.4%), STAR (+2.2%), and TVQA (+1.9%). SeViLA framework achieves new state-of-the-art fine-tuning performance on NExT-QA, STAR, TVQA, and VLEP, using only visual and language modalities. This illustrates the significance of temporal localization and the efficacy of our self-refinement method for keyframe localization.

4.3 Zero-shot Comparison to SOTA on Video QA and Event Prediction

We further compare our SeViLA framework to recent state-of-the-art models in the zero-shot setting. We show the zero-shot results in Table 2, then discuss the findings in the following.

(a) Image-LM outperforms Video-LM, without video pretraining. Surprisingly, BLIP-2 voting , without inter-frame temporal modeling, outperforms the previous state-of-the-art video-LM, InternVideo on several datasets that require temporal reasoning. BLIP-2 voting outperforms InternVideo on NExT-QA (+13.6%), How2QA (+7.6%), and VLEP (+5.1%). On How2QA, BLIP-2 voting even surpasses FrozenBiLM which performs extra speech and video pre-training by 11.4%. It highlights the potential of image-LM for video-language tasks due to its model size and sufficient pre-training.

(b) Keyframe selection is more effective than uniform sampling. Our SeViLA † framework, combining the zero-shot Localizer and the zero-shot Answerer, outperforms the BLIP-2 concat (Answerer) with uniformly sampled frames on NExT-QA (+1.2%), STAR (+2.4%), How2QA (+1.5%), TVQA (+1.6%), and VLEP (+0.4%), achieving new state-of-the-art zero-shot performance on NExT-QA, STAR, How2QA, and VLEP, and new state-of-the-art on TVQA with only visual and language modalities. On STAR, our SeViLA † framework even outperforms zero-shot Flamingo [1] with 80B parameters by 4.9%. The result demonstrates the effectiveness of our SeViLA framework to adapt video-language tasks and the importance of language-aware keyframe selection.

4.4 Ablation Studies on SeViLA Framework

We conduct ablation studies on our SeViLA framework about the effectiveness of Localizer and Answerer. We show the results in Table 3. We summarize our findings as follows:

(a) Sparse frames outperform dense frames: Weobserve performance declines when the Answerer adopts more frames (A v.s. B), confirming that sparse frames work better due to the original Image-LM model's limited temporal modeling ability, while too dense frames may distract the model.

(b) Keyframes outperform uniformly sampled frames: We compare Answerer with Localizer (SeViLA framework) and Answerer that takes uniformly sampled frames. We observed significant performance gains in the zero-shot Answerer setting when utilizing the zero-shot Localizer (B v.s. C), with improvements on NExT-QA (+1.0%), STAR (+2.4%), How2QA (+1.5%), TVQA (+1.6%), and VLEP (+0.4%) . And pseudo-label refinement for Localizer further increased performance by an average of 2.1% across all tasks (B v.s. D). In the fine-tuned Answerer setting, the benefits of the Localizer remained evident. Our SeViLA framework, which utilized keyframes from the

Localizer, outperformed the 4-frame Answerer by an average of 0.7% across tasks (E v.s. F). Pseudo-label refinement continues to be effective in this setting, providing an average boost of 1.5% across all tasks (E v.s. G). These results demonstrate that keyframe selection contributes to non-trivial improvements in video-language tasks for zero-shot and fine-tuning settings.

4.5 Comparison to State-of-the-Art on Video Moment Retrieval

In this section, we evaluate our Localizer on the video moment retrieval task. We pre-train the Localizer on the QVHighlights [30] dataset, as discussed in Sec. 3.3, and then assess its performance on the same dataset. To test on QVHighlights, we first extract video frames at 0.5 fps, following Moment-DETR [30] and pass them through our Localizer to obtain binary frame-level predictions that indicate whether a frame matches the query sentence. Next, we combine these predictions into video-level temporal span predictions. We aggregate frame-level predictions into video-level spans by merging adjacent positive predictions with intervals not exceeding a threshold. These merged results are then consolidated into a single video-level span. More information on the aggregation process can be found in Appendix. Interestingly, as shown in Table 4, our Localizer, which has no temporal modeling/training and operates on frame-level, outperforms many previous methods [13, 30, 29] with complex temporal modeling and video data training. It demonstrates our Localizer can further work as a standalone model for certain tasks. It also suggests large image-LM with temporal designs may be a promising study for video moment retrieval. This is evidenced by our Localizer's superior performance compared to Moment-DETR, despite the absence of temporal modeling.

4.6 Detailed Analysis on the Localizer

In this section, we first analyze the impact of pre-training and self-refinement on our Localizer. Then, we compare our Localizer with other keyframe selection methods in both zero-shot and fine-tuning settings. Next, we experiment with different keyframe selection ranges and quantities in our Localizer to assess the impact of temporal localization on the overall performance. We further show the impact of Localizer in Answerer fine-tuning. We also present upper-bound analysis based on BLIP-2 and oracle keyframe localization. Lastly, we provide visualization results of our Localizer. Additional experiments are in Appendix.

Ablation on Localizer pre-training and self-refinement. We performed these ablation studies with the zero-shot 4-frame Answerer. As shown in Table 5, the untrained BLIP-2 Localizer

results in only a minor improvement to the Answerer. Moreover, both QVHighlights pre-training and self-refinement via the reverse chain independently provide significant performance boosts. The optimal results are achieved when both pre-training and self-refinement are applied. It further demonstrates our method is label-efficient for keyframe temporal localization.

Comparison with other keyframe selection methods. In Table 6, we compare our Localizer with different keyframe localization methods, including CLIP [55], Moment-DETR [30] which are zero-shot, and ATP [3], Differentiable Top-K [8] which are fine-tuned with answer-label. We combine those keyframe localization methods with our zero-shot Answerer. Those methods select 4 keyframes from 32 uniformly sampled frames. We find that keyframes from neither CLIP nor Moment-DETR can not help Answerer. It may be due to their CLIP pre-training on images and short declarative sentences, which fail to produce question-aware visual features, and distract Answerer with irrelevant features. In contrast, our zero-shot Localizer † improves on NExT-QA by averaging 1.2%. Furthermore, our Localizer refined on pseudo-labels outperforms fine-tuned ATP and Differentiable top-K, with a 2.2% average improvement

across all question types. Overall, our Localizer shows superior effectiveness in both settings.

Impact of keyframe selection ranges and quantities. In Table 7, we evaluate temporal keyframe localization in a zero-shot setting using various keyframe selection ranges and quantities. Even with one keyframe selected, our Localizer shows significant improvements over the BLIP-2 voting based on majority vote on 8 frame-level answers: NExT-QACausal (+2.4%), NExT-QA-Description (+3.6%), and How2QA (+2.6%). This highlights our Localizer's effectiveness in localizing selective keyframes. We also note that multiple keyframes benefit NExT-QA-Temporal questions, but denser frames result in worse perfor-

mance. It supports our finding in Sec. 4.4, that using too dense frames may distract image-LM.

Impact of different frame sampling during Answerer fine-tuning. In Sec. 3.3, we discussed how the Answerer in the SeViLA framework can be further fine-tuned in the forward chain using keyframes from the Localizer. Table 8 compares fine-tuning in different frame sampling strategies, and indicates SeViLA framework performs optimally when utilizing Localizer in both Answerer training and evaluation. This is likely due to the provision of more informative keyframes and milder domain shifts between the training and evaluation.

Upper-bound performance analysis on oracle keyframes. We further explore the upper-bound performance with the assumption of a 'perfect' localizer, capable of always providing the right keyframes for the Answerer. To achieve this, we uniformly sample four frames from each video, input them into BLIP-2 individually, and generate four frame-level answers. The upper-bound performance is represented by the oracle accuracy, which considers a question correctly answered if at least one of the four frames yields the right answer. As shown in Table 9, significant gaps exist between BLIP-2 majority voting and oracle accuracy in both settings. These gaps emphasize the need for more future work in temporal localization to effectively harness image-LM for video-language tasks.

Qualitative analysis on Localizer. In Fig. 4, we present an example from NExT-QA (more in Appendix), showcasing the QA pairs, our Localizer results, and the ground truth task-related video clips that we manually annotated. Our Localizer more accurately matches human annotations than

Question:

why did the two ladies put their hands above their eyes while staring out?

A:

practicing cheer.

B:

posing for photo.

C:

to see better.

D:

dancing.

E:

wiping their face.

19s-30s

✅

✅

✅

✅

❌

❌

❌

❌

❌

❌

[Human Temporal Localization Annotation]

[Our Localizer Selection]

✅

❌

❌

✅

✅

❌

❌

✅

❌

❌

[Uniform Selection]

Question:

What did both of them do after completing skiing?

A:

jump and pose.

B:

bend down.

C:

raised their hands.

D:

turn around.

E:

take off clothes.

[Human Temporal Localization Annotation]

36s-55s

[Our Localizer Selection]

❌

✅

✅

✅

❌

❌

❌

❌

✅

❌

❌

❌

✅

❌

❌

✅

❌

[Uniform Selection]

❌

❌

❌

uniform selection. This accurate localization enables the Answerer to correctly answer questions, while uniform selection leads to incorrect responses. These results demonstrate that our Localizer can effectively locate task-related keyframes in a video, thus benefiting downstream tasks.

5 Conclusion and Future Work

In this paper, we introduced SeViLA, a novel video-language framework. SeViLA adapts an imagelanguage model to obtain two modules: (1) Localizer for language-aware temporal localization and (2) Answerer for question answering on keyframes. SeViLA tackles video-language tasks by chaining the output of Localizer to the input of Answerer ( forward chain ), while Answerer can give feedback to refine the Localizer ( backward chain ) via pseudo labeling. The proposed temporal localization allows a more focused understanding of videos and improves the accuracy of videolanguage tasks, while the pseudo-labeling process alleviates the need for expensive language-aware keyframe annotations. Compared with state-of-the-art baselines, SeViLA achieves competitive or better performance on five video questions answering and event prediction benchmarks. We also provide a comprehensive analysis elaborating on the design of the proposed two-stage self-chaining. Our research encourages future work to improve temporal localization in video understanding.

Limitations & Broader Impacts. See Appendix for limitations and broader impacts discussion.

References

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716-23736, 2022.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

S. Buch, C. Eyzaguirre, A. Gaidon, J. Wu, L. Fei-Fei, and J. C. Niebles. Revisiting the" video" in video-language understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2917-2927, 2022.

M. Cao, T. Yang, J. Weng, C. Zhang, J. Wang, and Y. Zou. Locvtp: Video-text pre-training for temporal localization. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI , pages 38-56. Springer, 2022.

Y.-C. Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In ECCV , 2020.

J. Cho, J. Lei, H. Tan, and M. Bansal. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning , pages 1931-1942. PMLR, 2021.

H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022.

J.-B. Cordonnier, A. Mahendran, A. Dosovitskiy, D. Weissenborn, J. Uszkoreit, and T. Unterthiner. Differentiable patch selection for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2351-2360, 2021.

R. Cui, T. Qian, P. Peng, E. Daskalaki, J. Chen, X. Guo, H. Sun, and Y.-G. Jiang. Video moment retrieval from text queries via single frame annotation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 1033-1043, 2022.

A. Deng, T. Yang, and C. Chen. A large-scale study of spatiotemporal representation learning with a new benchmark on action recognition. arXiv preprint arXiv:2303.13505 , 2023.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023.

V. Escorcia, M. Soldan, J. Sivic, B. Ghanem, and B. Russell. Temporal localization of moments in video collections with natural language, 2019.

H. Fang, P. Xiong, L. Xu, and Y. Chen. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097 , 2021.

H. Fang, P. Xiong, L. Xu, and W. Luo. Transferring image-clip to video-text retrieval via temporal relations. IEEE Transactions on Multimedia , 2022.

Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636 , 2022.

T.-J. Fu, L. Li, Z. Gan, K. Lin, W. Y. Wang, L. Wang, and Z. Liu. Violet: End-to-end videolanguage transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681 , 2021.

D. Gao, L. Zhou, L. Ji, L. Zhu, Y. Yang, and M. Z. Shou. Mist: Multi-modal iterative spatialtemporal transformer for long-form video question answering. arXiv preprint arXiv:2212.09522 , 2022.

S. K. Gorti, N. Vouitsis, J. Ma, K. Golestan, M. Volkovs, A. Garg, and G. Yu. X-pool: Cross-modal language-video attention for text-video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5006-5015, 2022.

N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning , pages 2790-2799. PMLR, 2019.

C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pages 4904-4916. PMLR, 2021.

M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim. Visual prompt tuning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII , pages 709-727. Springer, 2022.

C. Ju, T. Han, K. Zheng, Y. Zhang, and W. Xie. Prompting visual-language models for efficient video understanding. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV , pages 105-124. Springer, 2022.

S. Kim, J.-H. Kim, J. Lee, and M. Seo. Semi-parametric video-grounded text generation. arXiv preprint arXiv:2301.11507 , 2023.

R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision , 123:32-73, 2017.

D.-H. Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , page 896, 2013.

J. Lei, L. Yu, M. Bansal, and T. L. Berg. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01696 , 2018.

J. Lei, L. Yu, T. L. Berg, and M. Bansal. What is more likely to happen next? video-and-language future event prediction. arXiv preprint arXiv:2010.07999 , 2020.

J. Lei, L. Yu, T. L. Berg, and M. Bansal. Tvr: A large-scale dataset for video-subtitle moment retrieval. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16 , pages 447-463. Springer, 2020.

J. Lei, T. L. Berg, and M. Bansal. Detecting moments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems , 34:11846-11858, 2021.

J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7331-7341, 2021.

J. Lei, T. L. Berg, and M. Bansal. Revealing single frame bias for video-and-language learning. Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics , 2023.

D. Li, J. Li, H. Le, G. Wang, S. Savarese, and S. C. Hoi. Lavis: A one-stop library for language-vision intelligence. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , pages 31-41, 2023.

J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023.

L. Li, Y.-C. Chen, Y. Cheng, Z. Gan, L. Yu, and J. Liu. Hero: Hierarchical encoder for video+ language omni-representation pre-training. arXiv preprint arXiv:2005.00200 , 2020.

L. Li, Z. Gan, K. Lin, C.-C. Lin, Z. Liu, C. Liu, and L. Wang. Lavender: Unifying video-language understanding as masked language modeling. arXiv preprint arXiv:2206.07160 , 2022.

Y. Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He. Scaling language-image pre-training via masking. arXiv preprint arXiv:2212.00794 , 2022.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

X. Lin, S. Tiwari, S. Huang, M. Li, M. Z. Shou, H. Ji, and S.-F. Chang. Towards fast adaptation of pretrained contrastive models for multi-channel video-language retrieval. arXiv preprint arXiv:2206.02082 , 2022.

Y. Liu, P. Xiong, L. Xu, S. Cao, and Q. Jin. Ts2-net: Token shift and selection transformer for text-video retrieval. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV , pages 319-335. Springer, 2022.

H. Lu, M. Ding, N. Fei, Y. Huo, and Z. Lu. Lgdn: Language-guided denoising network for video-language modeling. arXiv preprint arXiv:2209.11388 , 2022.

J. Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916 , 2022.

H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing , 508:293-304, 2022.

F. Ma, X. Jin, H. Wang, J. Huang, L. Zhu, J. Feng, and Y. Yang. Temporal perceiving video-language pre-training. arXiv preprint arXiv:2301.07463 , 2023.

Y. Ma, G. Xu, X. Sun, M. Yan, J. Zhang, and R. Ji. X-clip: End-to-end multi-grained contrastive learning for video-text retrieval. In Proceedings of the 30th ACM International Conference on Multimedia , pages 638-647, 2022.

Z. Ma, J. Pan, and J. Chai. World-to-words: Grounded open vocabulary acquisition through fast mapping in vision-language models. arXiv preprint arXiv:2306.08685 , 2023.

A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651 , 2023.

S. Marcel and Y. Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference on Multimedia , pages 1485-1488, 2010.

L. Momeni, M. Caron, A. Nagrani, A. Zisserman, and C. Schmid. Verbs in action: Improving verb understanding in video-language models. arXiv preprint arXiv:2304.06708 , 2023.

W. Moon, S. Hyun, S. Park, D. Park, and J.-P. Heo. Query-dependent video representation for moment retrieval and highlight detection. arXiv preprint arXiv:2303.13874 , 2023.

V. Ordonez, G. Kulkarni, and T. Berg. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems , 24, 2011.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

T. Qian, R. Cui, J. Chen, P. Peng, X. Guo, and Y.-G. Jiang. Locate before answering: Answer guided question localization for video question answering. arXiv preprint arXiv:2210.02081 , 2022.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

M. Regneri, M. Rohrbach, D. Wetzel, S. Thater, B. Schiele, and M. Pinkal. Grounding action descriptions in videos. Transactions of the Association for Computational Linguistics , 1:25-36, 2013.

C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2556-2565, 2018.

C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision , pages 7464-7473, 2019.

Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 , 2023.

Y.-L. Sung, J. Cho, and M. Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5227-5237, 2022.

D. Surís, S. Menon, and C. Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128 , 2023.

H. Tan and M. Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490 , 2019.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

A. J. Wang, Y. Ge, R. Yan, Y. Ge, X. Lin, G. Cai, J. Wu, Y. Shan, X. Qie, and M. Z. Shou. All in one: Exploring unified video-language pre-training. arXiv preprint arXiv:2203.07303 , 2022.

J. Wang, Y. Ge, G. Cai, R. Yan, X. Lin, Y. Shan, X. Qie, and M. Z. Shou. Object-aware video-language pre-training for retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3313-3322, 2022.

P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning , pages 23318-23340. PMLR, 2022.

W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 , 2022.

Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191 , 2022.

Z. Wang, M. Li, R. Xu, L. Zhou, J. Lei, X. Lin, S. Wang, Z. Yang, C. Zhu, D. Hoiem, et al. Language models with image descriptors are strong few-shot video-language learners. arXiv preprint arXiv:2205.10747 , 2022.

Z. Wang, Y. Zhong, Y. Miao, L. Ma, and L. Specia. Contrastive video-language learning with fine-grained frame sampling. arXiv preprint arXiv:2210.05039 , 2022.

T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 , 2019.

B. Wu, S. Yu, Z. Chen, J. B. Tenenbaum, and C. Gan. Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021.

W. Wu, D. He, X. Tan, S. Chen, and S. Wen. Multi-agent reinforcement learning based frame sampling for effective untrimmed video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6222-6231, 2019.

J. Xiao, X. Shang, A. Yao, and T.-S. Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9777-9786, 2021.

J. Xiao, P. Zhou, T.-S. Chua, and S. Yan. Video graph transformer for video question answering. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI , pages 39-58. Springer, 2022.

J. Xiao, P. Zhou, A. Yao, Y. Li, R. Hong, S. Yan, and T.-S. Chua. Contrastive video question answering via video graph transformer. arXiv preprint arXiv:2302.13668 , 2023.

H. Xu, G. Ghosh, P.-Y. Huang, D. Okhonko, A. Aghajanyan, F. Metze, L. Zettlemoyer, and C. Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084 , 2021.

H. Xu, Q. Ye, M. Yan, Y. Shi, J. Ye, Y. Xu, C. Li, B. Bi, Q. Qian, W. Wang, et al. mplug-2: A modularized multi-modal foundation model across text, image and video. arXiv preprint arXiv:2302.00402 , 2023.

H. Xue, Y. Sun, B. Liu, J. Fu, R. Song, H. Li, and J. Luo. Clip-vip: Adapting pre-trained image-text model to video-language alignment. In The Eleventh International Conference on Learning Representations , 2023.

S. Yan, T. Zhu, Z. Wang, Y. Cao, M. Zhang, S. Ghosh, Y. Wu, and J. Yu. Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:2212.04979 , 2022.

A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1686-1697, 2021.

A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid. Zero-shot video question answering via frozen bidirectional language models. arXiv preprint arXiv:2206.08155 , 2022.

L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu. Filip: fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783 , 2021.

Q. Ye, G. Xu, M. Yan, H. Xu, Q. Qian, J. Zhang, and F. Huang. Hitea: Hierarchical temporal-aware video-language pre-training. arXiv preprint arXiv:2212.14546 , 2022.

J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 , 2022.

L. Yuan, D. Chen, Y.-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 , 2021.

R. Zellers, J. Lu, X. Lu, Y. Yu, Y. Zhao, M. Salehi, A. Kusupati, J. Hessel, A. Farhadi, and Y. Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16375-16387, 2022.

S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023.

Appendix

In this Appendix, we present the following:

Additional comparison between image-language models (image-LMs) and video language models (video-LMs) in terms of model size and pre-training data size (Sec. A).

Additional information about our experimental setup (Sec. B), including benchmarks details and task definitions (Sec. B.1), metrics (Sec. B.2), baseline implementation details (Sec. B.3), and SeViLA implementation details (Sec. B.4).

Additional experiments (Sec. C), including the single-frame v.s. multi-frame Localizer (Sec. C.1), iterative self-refinement results (Sec. C.2), different Localizer pre-training settings (Sec. C.3), SeViLA framework with another Image-LM (MiniGPT4) (Sec. C.4), computational cost of SeViLA (Sec. C.5), the impact of prompt designs (Sec. C.6) and more qualitative visualization (Sec. C.7).

Limitation and broader impact of this work (Sec. D), as well as license information (Sec. E) for the datasets, codes, and models that we used in this paper.

A Comparison Between Image-LMs and Video-LMs

In this section, we further provide a more visualized comparison between Image-LMs [55, 35, 43, 61, 70] and Video-LMs [71, 87, 37, 67, 17], in terms of model size and pre-training data size. As illustrated in Fig. 5, recent video-LMs generally have smaller scales in model and pre-training data sizes. This is because video-level annotations are more costly than image-level annotations, as discussed in the main paper Sec. 2 The gap between image-LMs and video-LMs has driven extensive research into image-to-video transfer learning to effectively leverage powerful image-LMs for video-language tasks. In this paper, we adopt this image-to-video transfer strategy by employing the state-of-the-art pre-trained image-LM, BLIP-2 [35].

$OOGLYPH<c=16,font=/AAAAAB+font000000002a0bf836>LQGLYPH<c=16,font=/AAAAAB+font000000002a0bf836>2QH

/$9(1'(5

9,2/(7

+L7H$

,QWHUQ9LGHR

&/,3

%(,7GLYPH<c=16,font=/AAAAAB+font000000002a0bf836>GLYPH<c=22,font=/AAAAAB+font000000002a0bf836>

8QLILHGGLYPH<c=16,font=/AAAAAB+font000000002a0bf836>,2

(9$GLYPH<c=16,font=/AAAAAB+font000000002a0bf836>&/,3

%/,3GLYPH<c=21,font=/AAAAAB+font000000002a0bf836>

9LGHRGLYPH<c=16,font=/AAAAAB+font000000002a0bf836>/DQJXDJHGLYPH<c=3,font=/AAAAAB+font000000002a0bf836>0RGHOV

,PDJHGLYPH<c=16,font=/AAAAAB+font000000002a0bf836>/DQJXDJHGLYPH<c=3,font=/AAAAAB+font000000002a0bf836>0RGHOV

GLYPH<c=20,font=/AAAAAB+font000000002a0bf836>

GLYPH<c=20,font=/AAAAAB+font000000002a0bf836>GLYPH<c=19,font=/AAAAAB+font000000002a0bf836>

3UHGLYPH<c=16,font=/AAAAAB+font000000002a0bf836>WUDLQLQJGLYPH<c=3,font=/AAAAAB+font000000002a0bf836>0RGHOGLYPH<c=3,font=/AAAAAB+font000000002a0bf836>6L]HGLYPH<c=3,font=/AAAAAB+font000000002a0bf836>GLYPH<c=11,font=/AAAAAB+font000000002a0bf836>%GLYPH<c=12,font=/AAAAAB+font000000002a0bf836>

GLYPH<c=20,font=/AAAAAB+font000000002a0bf836>GLYPH<c=19,font=/AAAAAB+font000000002a0bf836>GLYPH<c=19,font=/AAAAAB+font000000002a0bf836>

GLYPH<c=20,font=/AAAAAB+font000000002a0bf836>GLYPH<c=19,font=/AAAAAB+font000000002a0bf836>GLYPH<c=19,font=/AAAAAB+font000000002a0bf836>GLYPH<c=19,font=/AAAAAB+font000000002a0bf836>

GLYPH<c=19,font=/AAAAAB+font000000002a0bf836>

GLYPH<c=21,font=/AAAAAB+font000000002a0bf836>GLYPH<c=17,font=/AAAAAB+font000000002a0bf836>GLYPH<c=24,font=/AAAAAB+font000000002a0bf836>

GLYPH<c=24,font=/AAAAAB+font000000002a0bf836>GLYPH<c=17,font=/AAAAAB+font000000002a0bf836>GLYPH<c=19,font=/AAAAAB+font000000002a0bf836>

GLYPH<c=26,font=/AAAAAB+font000000002a0bf836>GLYPH<c=17,font=/AAAAAB+font000000002a0bf836>GLYPH<c=24,font=/AAAAAB+font000000002a0bf836>

GLYPH<c=20,font=/AAAAAB+font000000002a0bf836>GLYPH<c=21,font=/AAAAAB+font000000002a0bf836>GLYPH<c=17,font=/AAAAAB+font000000002a0bf836>GLYPH<c=24,font=/AAAAAB+font000000002a0bf836>

GLYPH<c=20,font=/AAAAAB+font000000002a0bf836>GLYPH<c=19,font=/AAAAAB+font000000002a0bf836>GLYPH<c=17,font=/AAAAAB+font000000002a0bf836>GLYPH<c=19,font=/AAAAAB+font000000002a0bf836>

3UHGLYPH<c=16,font=/AAAAAB+font000000002a0bf836>WUDLQLQJGLYPH<c=3,font=/AAAAAB+font000000002a0bf836>'DWDGLYPH<c=3,font=/AAAAAB+font000000002a0bf836>6L]HGLYPH<c=3,font=/AAAAAB+font000000002a0bf836>GLYPH<c=11,font=/AAAAAB+font000000002a0bf836>0GLYPH<c=12,font=/AAAAAB+font000000002a0bf836>GLYPH<c=3,font=/AAAAAB+font000000002a0bf836>

B Experimental Setup

In this section, we present additional information on the used datasets/benchmarks and task definitions (Sec. B.1), metrics (Sec. B.2), baseline implementations (Sec. B.3), and our SeViLA framework implementations (Sec. B.4).

B.1 Benchmarks

Video Question Answering (Video QA). We test models on four challenging long-form multi-choice benchmarks: (1) NExT-QA [77], a benchmark for causal and temporal reasoning. It contains 5440 videos with an average length of 44s and approximately 52K questions. NExT-QA contains 3 different question types: Temporal (Tem.), Causal (Cau.), and Descriptive (Des.). (2) STAR [75], a benchmark for situated reasoning, which contains 22K video clips with an average length of 12s, along with 60K questions. STAR contains 4 different question types: Interaction (Int.), Sequence (Seq.), Prediction (Pre.), and Feasibility (Fea.). (3) How2QA [36], a benchmark containing 44k questions along with 22 thousand 60-second clips selected from 9035 videos. (4) TVQA [27], another benchmark contains 152K questions along with 21k video clips (76 seconds on average) from TV shows.

Video Event Prediction (Video EP). We test models on VLEP [28], a benchmark that requires the model to predict two future events based on the video premise. Thus, we can formulate this task as a multi-choice QA. VLEP contains 28,726 future event prediction cases from 10,234 diverse TV Shows and YouTube Lifestyle Vlog video clips.

Video Moment Retrieval. We test our Localizer performance on QVHighlights [30], where a model predicts the temporal span in a video corresponding to a human-generated language description. QVHighlights consists of 10,148 videos with a duration of 150s, 18,367 moments, and 10,310 queries.

B.2 Metrics

For video question answering and video event prediction, we adopt the standard answer accuracy. For video moment retrieval, we follow Lei et al. [30] and report the standard mean average precision (mAP) over multiple IoU thresholds [0.5: 0.05: 0.95], and the standard Recall@1 (R@1) where we define a prediction to be positive if it has a high IoU (>= 0.5 or >= 0.7) with one of the ground truth moments. For NExT-QA, STAR, How2QA, TVQA, and VLEP we report the performance on the validation set whereas QVHighlights we report on the hidden test set.

B.3 Baseline Implementations

Baseline Architecture . For InternVideo [71], we adopt the released largest MM-L-14 variant that initialized from CLIP-L/14 [55] with 1B parameters and follow its default 8-frame setting for comparison. We fine-tuned InternVideo on downstream tasks by ourselves. For BLIP-2 voting and BLIP-2 concat (Answerer), we use the same BLIP-2 ViT-G Flan-T5 XL to our SeViLA framework.

Baseline Training . In Table 10, we show the setup of InternVideo [71] fine-tuning on downstream tasks. we adopt the released largest MM-L-14 variant with 1B parameters and follow its default 8-frame setting. We conduct experiments with 4 × 48GB A6000 GPUs. It takes about 4h on NExT-QA, 8h on STAR, 4h on How2QA, 12h on TVQA, and 2h on VLEP.

Implementation Details of Other Keyframe Selection Methods. . In the main paper Sec. 4.6, we compare our Localizer with other keyframe selection methods in both zero-shot and fine-tuning settings. Specifically, in zero-shot setting, we compare our Localizer with CLIP [55] and MomentDETR[30], utilizing the same Answerer (32 → 4) for each. For CLIP one, we calculate the pre-trained CLIP-ViT-B/32 image-language similarity between each frame's visual feature and the corresponding question and option features. We then select the top-4 similarity keyframes from 32 uniformly sampled frames for the Answerer. In the case of Moment-DETR, we apply the Moment-DETR pre-trained on QVHighlights to first detect a temporal span corresponding to the question and option

>/RFGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>3URPSW@

'RHVGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>WKHGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>LQIRUPDWLRQGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>ZLWKLQGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>WKHGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>IUDPHGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>SURYLGHGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>

QHFHVVDU\GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GHWDLOVGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>WRGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>DFFXUDWHO\GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>DQVZHUGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>WKHGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>TXHVWLRQ"

>3VHXGRGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>2SWLRQV@

$GLYPH<c=29,font=/AAAAAC+font000000002a0d58dd>

GLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>

\HVGLYPH<c=17,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>%GLYPH<c=29,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>QRGLYPH<c=17,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd> GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>

3UHGLYPH<c=16,font=/AAAAAB+font000000002a0d58dd>WUDLQLQJGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>/RFDOL]HUGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>RQGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>49+LJKOLJKWVGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>

>4XHU\GLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>6HQWHQFH@GLYPH<c=29,font=/AAAAAB+font000000002a0d58dd>

GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd> $GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>VKDUNGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>LVGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>VZLPPLQJGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>XQGHUZDWHUGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>

/RFDOL]HU

\HV

\HV

QR

QR

9LGHRGLYPH<c=16,font=/AAAAAB+font000000002a0d58dd>/HYHOGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>

5HWULHYDOGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>/DEHOGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>

)UDPHGLYPH<c=16,font=/AAAAAB+font000000002a0d58dd>/HYHO

/RFDOL]DWLRQGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>/DEHO

3UHGLFWHGGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>

$QVZHU

/RVV

>4XHU\GLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>6HQWHQFH@GLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>

>3VHXGRGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>2SWLRQV@

GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>

>/RFGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>3URPSW@

GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=19,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=19,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=19,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=19,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=19,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=19,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=19,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=19,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=19,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAE+font000000002a0d58dd>GLYPH<c=3,font=/AAAAAE+font000000002a0d58dd>

SUHGLFWHGGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>

VSDQGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=20,font=/AAAAAC+font000000002a0d58dd>

VSDQGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>WKUHVKROGGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd> GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=25,font=/AAAAAC+font000000002a0d58dd>

GLYPH<c=11,font=/AAAAAC+font000000002a0d58dd>K\SHUSDUDPHWHUGLYPH<c=12,font=/AAAAAC+font000000002a0d58dd>

)UDPHGLYPH<c=16,font=/AAAAAC+font000000002a0d58dd>OHYHOGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>3UHGLFWLRQV

SUHGLFWHGGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>

VSDQGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>GLYPH<c=21,font=/AAAAAC+font000000002a0d58dd>

OHQJWKGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>!GLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>VSDQGLYPH<c=3,font=/AAAAAC+font000000002a0d58dd>WKUHVKROG

7LPH

$JJUHJDWLQJGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>IUDPHGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>SUHGLFWLRQVGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>WRGLYPH<c=3,font=/AAAAAB+font000000002a0d58dd>VSDQV

sentence. Subsequently, we uniformly sample 4 frames within this span for the Answerer to predict answers. Next, we compare our Localizer with the fine-tuned ATP and the Differentiable Top-K plugin. Both ATP [3] and Differentiable Top-K [8] utilize the BERT-base encoder [66], which is of the same size as the Q-former in our Localizer. Both ATP and Differentiable Top-K modules are inserted after the Q-former to learn salient frame feature selection in an end-to-end manner.

B.4 SeViLA Implementation Details

SeViLA Architecture. SeViLA framework adopts BLIP-2 [35], an image-language model with 4.1B parameters. We freeze its visual encoder (ViT-G [16]; 1B) and large language model (Flan-T5 XL [7]; 3B), and only fine-tune the Q-former (including its learnable queries), and a fully connected layer after Q-former for each of both Localizer and Answerer. There are 106M trained parameters, which is 2.5% of the total parameters of BLIP-2.

SeViLAFrameworkTraining. Weconductexperimentswith4 × 48GBA6000GPUs. ForLocalizer per-training, we pre-train Localizer on the QVHighlights for 80 epochs, taking approximately 12 hours with 4 × 29GB on A6000 GPUs. For Localizer self-refinement, we train Localizer on

each downstream dataset with pseudo-labels for an additional 10 epochs with 4 × 29GB on A6000 GPUs. It takes about 7h on NExT-QA, 8h on STAR, 6h on How2QA, 17h on TVQA, and 3h on VLEP. For Answerer fine-tuning, we fine-tune Answerer on each downstream dataset with answer labels and frozen pre-trained Localizer for 10 epochs with 4 × 48GB on A6000 GPUs. It takes about 13h on NExT-QA, 15h on STAR, 13h on How2QA, 48h on TVQA, and 8h on VLEP. We employ standard cross-entropy loss between the model outputs and the target values. We report SeViLA framework training hyperparameters in Localizer pre-training, Answerer fine-tuning, and Localizer self-refinment, in Table 11.

Prompt Engineering. We write multiple QA and Loc prompts shown in the main paper and choose the prompt yielding the best zero-shot performance on the downstream task (see Sec. C.6).

Localizer Pre-training Details. As illustrated in Fig. 6 (left), we provide the visualization on our Localizer pre-training. As discussed in the main paper Sec. 3.3, we adopt QVHighlighs [30] to pre-train our Localizer, we transfer its temporal span annotations to keyframe localization labels by comparing frame time-stamp with spans. We further design a prompt template and fill in it with query sentences in QVHighlighs. It makes the pre-training and downstream video QA and event prediction tasks with a similar input format and can further benefit downstream tasks.

Details of Aggregation for Video Moment Retrieval. Weelaborate on the aggregation of frame-level predictions from our Localizer, as discussed in the main paper Sec. 4.5. As illustrated in Fig. 6 (right), we use a hyperparameter called 'span threshold', a maximum number of continuing '0's (continuing 'no' prediction for frame-level localization) inside a single span. If there are more '0's than the value in a row, we split the frames into separate spans. We use the span threshold=6. We determined this value based on statistics of QVHighlights training data. In QVHighlights, a query sentence has multiple grounding spans in a video. We calculate the average interval among those spans as our span threshold.

C Experiments

In this section, we present extra in-depth analysis on our Answerer and Localizer, including the single-frame v.s. multi-frame Localizer (Sec. C.1), iterative self-refinement results (Sec. C.2), different Localizer pre-training settings (Sec. C.3), SeViLA framework with another Image-LM (MiniGPT4) (Sec. C.4), computational cost of SeViLA (Sec. C.5), the impact of prompt designs (Sec. C.6) and more qualitative visualization (Sec. C.7)

C.1 Single-frame v.s. Multi-frame Localizer

Even though our single-frame Localizer in SeViLA can locate language-aware keyframes, temporal modeling for video-level tasks cannot be overlooked. In this section, we've expanded our Localizer to a multi-frame mode by concatenating frames into a long image before Q-Former. This allows for full attention across all frames, enabling temporal modeling. As shown in Table 12 we find that the 4-frame Localizer performs worse than the single-frame Localizer in both zero-shot and fine-tuning settings. We suspect that this is because our backbone model (BLIP-2) has not seen video data during its pre-training. We think that a multi-frame Localizer could be more powerful once we have enough temporal grounding annotations or enough pre-training on large-scale video dataset [10]. We leave the improvement of the multi-frame Localizer to future work.

C.2 Iterative Self-refinement on Localizer and Answerer

Madaan et al. [48] introduces the 'Self-Refine' framework, where an LLM iteratively generates initial output, provides feedback, and refines the outputs. We experiment with iterative self-refinement, where the Answerer gives pseudo labels to train Localizer, and then the Localizer provides more accurate language-aware frames to fine-tune Answerer, iteratively. As shown in Table 14, we find that two iterations of self-refinement slightly increase performance compared to the single self-refinement, but the performance saturates from three iterations. We leave further analysis with self-refinement to future work.

C.3 Different Pre-training Settings of Localizer

We explored weakly-supervised pre-training using ASR similar to Moment-DETR [30]. As shown in Table 13, our Localizer performance improves with the weakly supervised pre-training, closing the gap with the pre-training with manual annotations (QVHighlights [30]).

C.4 SeViLA Framework with Another Image-LM (MiniGPT4)

We experimented with extending the SeViLA framework with another recent Image-Language model (MiniGPT4 [92]). On NeXT-QA, zero-shot MiniGPT4 Answerer achieves 52.7% average accuracy and obtains 0.7% boost with zero-shot MiniGPT4 Localizer. We find our proposed self-chaining scheme can also improve the performance of zero-shot MiniGPT4, and we expect that fine-tuning the model will even improve the performance.

C.5 Analysis on Computational Cost of SeViLA Framework

In Table 15, we show memory, running time, and parameter size (ViT + FlanT5 + Qformer) comparison between our Answerer and our SeViLA (Localizer+Answerer). As the Localizer and Answerer share the most parameters, adding Localizer has a very small additional memory footprint.

C.6 Impact of Prompt Design

We write several localization prompts and choose the one giving the best performance. We examine the impact of various localization prompts as shown in Table 16. For this experiment, we employ a

zero-shot Answerer and an untrained Localizer to choose the most effective localization prompt based on performance. It demonstrates that our model is insensitive to the changes in the prompt.

Question:

what did the black dog on the sofa do after it dropped the red toy?

A:

jump on man's leg.

B:

jump onto couch.

C:

push dog away.

D:

put it on boy in red.

E:

jumps down.

[Human Temporal Localization Annotation]

8s-19s

[Our Localizer Selection]

✅

❌

✅

❌

❌

❌

❌

✅

❌

❌

✅

✅

❌

❌

✅

❌

❌

[Uniform Selection]

✅

❌

✅

Question:

What room was Wilson breaking into when House found him?

A:

the bedroom.

B:

the bathroom.

C:

the kitchen.

D:

the living room.

E:

the dining room.

[Human Temporal Localization Annotation]

28s-95s

[Our Localizer Selection]

❌

❌

✅

❌

❌

❌

❌

❌

❌

❌

❌

✅

❌

❌

❌

❌

✅

[Uniform Selection]

❌

✅

❌

C.7 Visualization

As illustrated in Fig. 7, we provide more visualization examples from different datasets, and with various selected keyframe amounts. Compared with uniform sampling, our Localizer can locate relevant frames that match human annotations well regardless of keyframe amounts. And such a good keyframe localization also brings the correct answers to corresponding questions.

D Limitations and Broader Impacts

Limitations. Although our SeViLA framework can help locate language-aware keyframes in many real-world video tasks, e . g ., video content analysis and event detection, our Localizer uses frame-level keyframe localization and might not handle well some complex, fine-grained temporal events ( e . g ., open the door vs. close the door). Future work can explore structured prediction for temporal localization beyond the frame level.

Broader impacts. SeViLA framework leverages a large image-language model trained on massive internet-scale data. Similar to most such large models, this might occasionally yield unexpected or inappropriate responses, potentially reflecting societal biases related to gender, race, or sexuality [47]. More studies in the large image-language model are needed to evaluate and mitigate these negative biases, and toxic output.

E License

We will make our code and models publicly accessible. We use standard licenses from the community and provide the following links to the licenses for the datasets, codes, and models that we used in this paper. For further information, please refer to the specific link.

Peer reviews:
comments: SUMMARY:
This work builds a joint model for temporal language grounding and video question answering. The model is built on a state-of-the-art image-language model, BLIP-2, and finetunes it in a parameter-efficient way to derive a localizer and an answerer. The localizer finds language-aware keyframes in a video and the answerer uses these frames to predict the answer. The answerer is also used to generate pseudo-labels to refine the localizer. Experiments on multiple video question answering datasets in both finetuning and zero-shot settings show that the proposed method achieves state-of-the-art results and improves slightly over the BLIP-2 baseline.

STRENGTHS:
- Leveraging image-language models for video-language tasks is an important direction, and localizing language-aware keyframes seems a good idea to achieve this. 
- The method achieves strong results on a wide variety of benchmarks in both finetuning and zero-shot mode.

WEAKNESSES:
- Most of the improvement compared to state-of-the-art methods come from the use of the BLIP-2 model. The proposed method only show small improvement over BLIP-2+concat in finetuning setting (see Table 1).
- Can the method generalize with other image-language models, e.g. BLIP, and if so how does it perform?
- The localizer relies on manual temporal language grounding annotations from QVHighlights. Could it benefit from weak supervision from e.g. HowTo100M like Moment-DETR?

QUESTIONS:
See weaknesses.

LIMITATIONS:
Limitations and potential negative societal impact are discussed in the Supplementary Material.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-17

comments: SUMMARY:
The paper presents the SeViLA framework, which addresses the limitations of existing image-language models for video question answering. It introduces two modules, Localizer and Answerer, that are fine-tuned from a pre-trained image-language model. SeViLA utilizes chaining for cascaded inference and self-refinement. The Localizer identifies language-aware keyframes, and the Answerer predicts the answer based on these keyframes. Additionally, the Answerer generates keyframe pseudo-labels to refine the Localizer, eliminating the need for costly video moment annotations.

STRENGTHS:
1.	The paper is well-written and easy to understand.
2.	The concept of localizing and then answering is intuitive and effective.
3.	Extensive experiments demonstrate that SeViLA achieves state-of-the-art performance on challenging video question answering benchmarks. The paper also provides a comprehensive analysis, including comparisons with other localization models and the impact of keyframe variations.

WEAKNESSES:
1.	The motivation and LGDN (which also addresses VideoQA tasks) are quite similar. While the paper mentions LGDN in the introduction and related work sections (which I appreciate), I suggest the authors further clarify the differences between the two approaches in the related work section.
2.	The self-chaining approach requires additional computational resources (key frames need to be recomputed). The paper should provide information on the additional computational costs involved.

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
Please see weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a simple yet effective framework for video localization and question answering using pretrained image-language models. The key idea is to introduce a LOCALIZER module which localizes the keyframes in a video in order to ignore irrelevant frames and better answer the question. The LOCALIZER module is realized with a separate image-language model and the module can be further improved by fine-tuning on the pseudo labels generated by the QA results from the ANSWER module. The proposed framework, SEVILA, achieves state-of-the-art VQA results on multiple challenging benchmarks under both fine-tuned and zero-shot setups.\n\nSTRENGTHS:\n1. The proposed framework is simple yet effective in extending image-language models to tackle video-related tasks such as video question answering and moment localization. The idea of using the LOCALIZER module to detect keyframes in a long video is technically sound and the results are convincing.\n\n2. The paper provides extensive ablation study to analyze different components of the framework under different setups. E.g., the impact of selecting keyframes for VQA, the impact of self-refinement for the LOCALIZER module, and how the LOCALIZER compared with some existing keyframe localization methods. While there exists many keyframe localization methods, it's interesting to see the language-model-based method used in this paper gets comparable or even better results in the ablation study.\n\n3. The paper is well organized, clearly written and the description of technical details is clear and easy to follow.\n\nWEAKNESSES:\n1. The technical contribution of the work is relatively weak. The idea of selecting keyframes in a video for VQA is straightforward and widely explored in prior work. The proposed self-refinement strategy is also a standard semi-supervised learning approach and there's no new training strategies / techniques proposed for the fine-tuning of image-language models.\n\n2. While I understand the main goal of this work is to utilize image-language models for VQA, the limitation of image-based models for video-related tasks should not be ignored. First, image-based models may not be sufficient to accurately localize the keyframes in a long video. Second, temporal modeling is also missing in the ANSWERER module, which makes the model less capable of modeling temporal dependencies compared with video-based models. These limitations may not be significant in the current VQA benchmarks since the questions are less sensitive to temporal dependencies and reasoning.\n\nQUESTIONS:\n1. Is it necessary to use two different Q-former for the LOCALIZER and ANSWERER? Since both models are image based, what if we share the same Q-former for both modules?\n\n2. What if we make the self-refinement an iterative process? In other words, we refine the ANSWERER module after the self-refinement of LOCALIZER, and vice versa. It's interesting to see when the performance would saturate with such interactive refinement process.\n\nA minor suggestion: I don't think Figure 4 is a good qualitative example because (1) the action of the ladies is barely visible in the small figure, let alone the intention of the action; (2) the question can be answered by only checking the question itself without knowing the visual content, since \"staring out\" and \"hands above their eyes\" are explicated mentioned.\n\nLIMITATIONS:\nThe authors have discussed the limitations of the work and its potential social impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework called Self-Chained Video Localization-Answering (SEVILA) that leverages a single image-language model (BLIP-2) for both temporal keyframe localization and question answering in videos. The framework consists of two modules, Localizer and Answerer, which are parameter-fine-tuned (LoRA finetuning) from BLIP-2. The paper demonstrates that the SEVILA framework outperforms several strong previous works in video question answering and event prediction benchmarks.\n\nSTRENGTHS:\n(1) The paper proposes a novel framework that addresses the issue of missing important visual cues in video question answering tasks by leveraging a single image-language model for keyframe localization and question answering.\n(2) The self-chaining mechanism in the framework allows for cascaded inference and self-refinement, improving the accuracy of temporal localization without the need for expensive annotations.\n(3) The paper provides a comprehensive analysis of the proposed framework, including the impact of temporal localization, the self-refinement process, and the number of keyframes.\n\nWEAKNESSES:\n1. In the backward chain, the pseudo-tags are frame-level, and if a single frame of information can not give the correct answer, the Localizer is considered to have given the wrong keyframe. This is obviously unreasonable and ignores the important time dependencies provided by the intermediate frame. This is probably why there was no significant change in their performance when the self-refine part was removed in the ablation study. \n2. the proposal is not good at dealing with temporal causality. Because Blip-2 is a image-text MLLM model, there is no space-time inference, and their answer is to directly add the q-former query generated by each frame to the word token of LLM. This is equivalent to letting q-former himself to realize the time level of information understanding, and q-former does not have this ability.\nCould you make me clear?\n\nQUESTIONS:\nsee above.\n\nLIMITATIONS:\nsee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}