SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SeVILA Architecture. SeVILA framework adopts BLIP-2 [35], an image-language model with 4.1B parameters. We freeze its visual encoder (ViT-G [16]; 1B) and large language model (Flan-T5 XL [7]; 3B), and only fine-tune the Q-former (including its learnable queries), and a fully connected layer after Q-former for each of both LOCALIZER and ANSWERER. There are 106M trained parameters, which is  $2.5\%$  of the total parameters of BLIP-2.
SEVILA Framework Training. We conduct experiments with  $4 \times 48$  GB A6000 GPUs. For LOCALIZER per-training, we pre-train LOCALIZER on the QVHighlights for 80 epochs, taking approximately 12 hours with  $4 \times 29$  GB on A6000 GPUs. For LOCALIZER self-refinement, we train LOCALIZER on
ANSWERER	# frames of LOCALIZER	NExT-QA (Average)
zero-shot	1	64.6
4	63.6
fine-tuned	1	73.4
4	71.3
Table 12: Comparison between single-frame and multi-frame LOCALIZER.
each downstream dataset with pseudo-labels for an additional 10 epochs with  $4 \times 29\mathrm{GB}$  on A6000 GPUs. It takes about 7h on NExT-QA, 8h on STAR, 6h on How2QA, 17h on TVQA, and 3h on VLEP. For ANSWERER fine-tuning, we fine-tune ANSWERER on each downstream dataset with answer labels and frozen pre-trained LOCALIZER for 10 epochs with  $4 \times 48\mathrm{GB}$  on A6000 GPUs. It takes about 13h on NExT-QA, 15h on STAR, 13h on How2QA, 48h on TVQA, and 8h on VLEP. We employ standard cross-entropy loss between the model outputs and the target values. We report SEViLA framework training hyperparameters in LOCALIZER pre-training, ANSWERER fine-tuning, and LOCALIZER self-refinement, in Table 11.
Prompt Engineering. We write multiple QA and Loc prompts shown in the main paper and choose the prompt yielding the best zero-shot performance on the downstream task (see Sec. C.6).
LOCALIZER Pre-training Details. As illustrated in Fig. 6 (left), we provide the visualization on our LOCALIZER pre-training. As discussed in the main paper Sec. 3.3, we adopt QVHighlights [30] to pre-train our LOCALIZER, we transfer its temporal span annotations to keyframe localization labels by comparing frame time-stamp with spans. We further design a prompt template and fill in it with query sentences in QVHighlights. It makes the pre-training and downstream video QA and event prediction tasks with a similar input format and can further benefit downstream tasks.
Details of Aggregation for Video Moment Retrieval. We elaborate on the aggregation of frame-level predictions from our LOCALIZER, as discussed in the main paper Sec. 4.5. As illustrated in Fig. 6 (right), we use a hyperparameter called 'span threshold', a maximum number of continuing '0's (continuing 'no' prediction for frame-level localization) inside a single span. If there are more '0's than the value in a row, we split the frames into separate spans. We use the span threshold=6. We determined this value based on statistics of QVHighlights training data. In QVHighlights, a query sentence has multiple grounding spans in a video. We calculate the average interval among those spans as our span threshold.
In this section, we present extra in-depth analysis on our ANSWERER and LOCALIZER, including the single-frame v.s. multi-frame LOCALIZER (Sec. C.1), iterative self-refinement results (Sec. C.2), different LOCALIZER pre-training settings (Sec. C.3), SEViLA framework with another Image-LM (MiniGPT4) (Sec. C.4), computational cost of SEViLA (Sec. C.5), the impact of prompt designs (Sec. C.6) and more qualitative visualization (Sec. C.7)
C.1 Single-frame v.s. Multi-frame LOCALIZER
Even though our single-frame LOCALIZER in SEViLA can locate language-aware keyframes, temporal modeling for video-level tasks cannot be overlooked. In this section, we've expanded our LOCALIZER to a multi-frame mode by concatenating frames into a long image before Q-Former. This allows for full attention across all frames, enabling temporal modeling. As shown in Table 12 we find that the 4-frame LOCALIZER performs worse than the single-frame LOCALIZER in both zero-shot and fine-tuning settings. We suspect that this is because our backbone model (BLIP-2) has not seen video data during its pre-training. We think that a multi-frame LOCALIZER could be more powerful once we have enough temporal grounding annotations or enough pre-training on large-scale video dataset [10]. We leave the improvement of the multi-frame LOCALIZER to future work.
LOCALIZER	NeXT-QA (Average)
w/o Localizer	62.4
+ Moment-DETR	62.0
+ Our Localizer (without pre-training)	62.9
+ Our Localizer (weakly pre-trained with QVH ASR)	63.2
+ Our Localizer (pre-trained with QVH)	63.6
Table 13: Comparison among different pre-training settings of LOCALIZER.
Iteration	NeXT-QA (Average)
1	73.8
2	74.2
3	73.7
C.2 Iterative Self-refinement on LOCALIZER and ANSWERER
Madaan et al. [48] introduces the 'Self-Refine' framework, where an LLM iteratively generates initial output, provides feedback, and refines the outputs. We experiment with iterative self-refinement, where the ANSWERER gives pseudo labels to train LOCALIZER, and then the LOCALIZER provides more accurate language-aware frames to fine-tune ANSWERER, iteratively. As shown in Table 14, we find that two iterations of self-refinement slightly increase performance compared to the single self-refinement, but the performance saturates from three iterations. We leave further analysis with self-refinement to future work.
C.3 Different Pre-training Settings of LOCALIZER
We explored weakly-supervised pre-training using ASR similar to Moment-DETR [30]. As shown in Table 13, our LOCALIZER performance improves with the weakly supervised pre-training, closing the gap with the pre-training with manual annotations (QVHighlights [30]).
C.4 SEViLA Framework with Another Image-LM (MiniGPT4)
We experimented with extending the SEViLA framework with another recent Image-Language model (MiniGPT4 [92]). On NeXT-QA, zero-shot MiniGPT4 ANSWERER achieves  $52.7\%$  average accuracy and obtains  $0.7\%$  boost with zero-shot MiniGPT4 LOCALIZER. We find our proposed self-chaining scheme can also improve the performance of zero-shot MiniGPT4, and we expect that fine-tuning the model will even improve the performance.
Table 14: Iterative self-refinement results of SEViLA framework.
Model	Memory (GB)	Running Time (sec./sample)	Parameter (B)
Answerer (4)	7.56	1.79	4.1
SeViLA (32->4)	7.98	3.28	4.2
Table 15: Computational cost of SeViLA framework.
C.5 Analysis on Computational Cost of SEViLA Framework
In Table 15, we show memory, running time, and parameter size (ViT + FlanT5 + Qformer) comparison between our ANSWERER and our SEVILA (LOCALIZER+ANSWERER). As the LOCALIZER and ANSWERER share the most parameters, adding Localizer has a very small additional memory footprint.
C.6 Impact of Prompt Design
We write several localization prompts and choose the one giving the best performance. We examine the impact of various localization prompts as shown in Table 16. For this experiment, we employ a
Localization Prompt	NExT-QA
Temporal	Casual	Descriptive	Average
Does the frame have the information needed to answer the question correctly?	59.9	61.1	74.2	62.7
Does the provided frame contain the necessary information to accurately answer the given question?	59.9	60.8	75.0	62.7
Does the information within the frame provide the necessary details to accurately answer the given question?	60.4	61.0	74.6	62.9
Table 16: Impact of different localization prompts on the zero-shot Video QA performance
zero-shot ANSWERER and an untrained LOCALIZER to choose the most effective localization prompt based on performance. It demonstrates that our model is insensitive to the changes in the prompt.
Question: what did the black dog on the sofa do after it dropped the red toy?
A: jump on man's leg. B: jump onto couch. C: push dog away. D: put it on boy in red. E: jumps down.
Question: What room was Wilson breaking into when House found him?
A: the bedroom. B: the bathroom. C: the kitchen. D: the living room. E: the dining room.
Figure 7: Visualization of our LOCALIZER. We show various keyframe amounts in those examples. We use zero-shot ANSWERER with different frame sampling (uniform v.s. LOCALIZER) to answer the question. Red options are answered wrongly with uniformly sampled frames. Green options are answered correctly with our LOCALIZER. Best viewed in color.
C.7 Visualization
As illustrated in Fig. 7, we provide more visualization examples from different datasets, and with various selected keyframe amounts. Compared with uniform sampling, our LOCALIZER can locate relevant frames that match human annotations well regardless of keyframe amounts. And such a good keyframe localization also brings the correct answers to corresponding questions.
D Limitations and Broader Impacts
Limitations. Although our SEViLA framework can help locate language-aware keyframes in many real-world video tasks, e.g., video content analysis and event detection, our LOCALIZER uses frame-level keyframe localization and might not handle well some complex, fine-grained temporal events (e.g., open the door vs. close the door). Future work can explore structured prediction for temporal localization beyond the frame level.
Broader impacts. SEViLA framework leverages a large image-language model trained on massive internet-scale data. Similar to most such large models, this might occasionally yield unexpected or inappropriate responses, potentially reflecting societal biases related to gender, race, or sexuality [47]. More studies in the large image-language model are needed to evaluate and mitigate these negative biases, and toxic output.
We will make our code and models publicly accessible. We use standard licenses from the community and provide the following links to the licenses for the datasets, codes, and models that we used in this paper. For further information, please refer to the specific link.
NExT-QA [77]: MIT
STAR [75]: Apache
How2QA [36]: MIT
TVQA [27]: MIT
VLEP [28]: MIT
QVHighlights [30]: CC BY-NC-SA 4.0
LAVIS [33]: BSD 3-Clause
PyTorch [53]: BSD-style
Huggingface Transformers [74]: Apache
Torchvision [49]: BSD 3-Clause