TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment
Authors: Wei Li
Hehe Fan
Yongkang Wong
Mohan Kankanhalli
Yi Yang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: video understanding
Large language model
vision-language
text-only alignment
Primary area: machine_vision
TL;DR: Text-Only Pre-Alignment for video understanding

Abstract: Recent advancements in image understanding have benefited from the extensive use of web image-text pairs. However, video understanding remains a challenge despite the availability of substantial web video-text data. This difficulty primarily arises from the inherent complexity of videos and the inefficient language supervision in recent web-collected video-text datasets. In this paper, we introduce Text-Only Pre-Alignment (TOPA), a novel approach to extend large language models (LLMs) for video understanding, without the need for pre-training on real video data. Specifically, we first employ an advanced LLM to automatically generate Textual Videos comprising continuous textual frames, along with corresponding annotations to simulate real video-text data. Then, these annotated textual videos are used to pre-align a language-only LLM with the video modality. To bridge the gap between textual and real videos, we employ the CLIP model as the feature extractor to align image and text modalities. During text-only pre-alignment, the continuous textual frames, encoded as a sequence of CLIP text features, are analogous to continuous CLIP image features, thus aligning the LLM with real video representation. Extensive experiments, including zero-shot evaluation and finetuning on various video understanding tasks, demonstrate that TOPA is an effective and efficient framework for aligning video content with LLMs. In particular, without training on any video data, the TOPA-Llama2-13B model achieves a Top-1 accuracy of 51.0% on the challenging long-form video understanding benchmark, Egoschema. This performance surpasses previous video-text pre-training approaches and proves competitive with recent GPT-3.5 based video agents.

TOPА: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment
Wei Li
Hehe Fan
Yongkang Wong' Mohan Kankanhalli
Yi Yang 1.2
ReLER Lab, CCAI, Zhejiang University, China
The State Key Laboratory of Brain-Machine Intelligence, Zhejiang University, China 3 School of Computing, National University of Singapore, Singapore Aweilif,hehefaat han aathefan,yangyccssss yangyics edu. cn
yongkang, wong@nus. edu. sg mohan@comp. nus edu. sg https: thub womn/dhgwwei//eee
Abstract
Recent advancements in image understanding have benefited from the extensive use of web image-text pairs. However, video understanding remains challenge despite the availability of substantial web video-text data. This difficulty primarily arises from the inherent complexity of videos and the inefficient language supervision in recent web-collected video-text datasets. In this paper, we introduce Text-Onl) Pre-Alignment (TOPA), a novel approach extend large language models (LLMs) for video understanding, without the need for pre-training Specifically, we first employ an advanced LLM to automatically generate Textual Videos comprising continuous textual frames, along with corresponding annotations to simulate real video-text pairs. Then, these annotated textual videos are used pre-align language-only LLMs with the video modality. To bridge the gap between textual and real videos, we employ the CLIP model as the feature extractor align image and text modalities. During text-only pre-alignment, the continuous textual frames, encoded as a sequence of CLIP text features, are analogous to continuous CLIP image features, thus aligning the LLM with real video representation. Extensive experiments, including zero-shot evaluation and finetuning on various video understanding tasks, demonstrate that TOPA is effective and efficient framework for aligning vidco modality with LLMs. In particular, without training any video data, the TOPA-Llama2-11. 13B model achieves Top-1 accuracy of 51.0% on the challenging long-form video understanding benchmark, EgoSchema. This performance surpasses previous video-text pre-training approaches and is competitive with recent GPT-3.5-bass video agents.
real video data.
Introduction
Image-language understanding has made large advancements in both image-language alignment [26, 48] and Multimodal Large Language Models (MLLMs) [1, 27, 34, 89]. aided by pre-training on large-scale noise- caisee image-text data collected from the web [6, 19, 51, 53, 52]. This raises question: Can we mirror this success in video- aideo-languaee understanding? Research [45, 64, 77, 86] has explored pretraining video-language models millions of web video-text data [3, 40, 65], achieving promising results in basic video tasks such as video-text retrieval, video captioning, and video question answering across conventional video benchmarks. However, recent research reveals that these models struggle with a challenging long-form video understanding benchmark, i.e., EgoSchema [39], which requires intrinsic temporal understanding capabilities. This highlights the gap in adapting web video-text pretrained models to more comprehensive video understanding tasks.
*Hehe Fan is the corresponding author.
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

We attribute this gap to two primary factors: 1) The intrinsic complexity of the video modality. Videos introduce intrinsic complexities in both spatial and temporal dimensions, which are not present in static images. These complexities require extensive training on larger-scale data to effectively capture video dynamics. Furthermore, representing videos typically involves processing multiple frames, significantly increasing computational demands compared to image modeling. The dual challenges of large-scale training and increased computational requirements make video-language modeling particularly challenging. 2) The limitations of web langnage supervision. The language supervision in recent web video-text datasets primarily comes from subtitles or descriptions associated with the videos [3, 40]. However, subtitles often suffer from the issues of visual-textur misalignment [33, 17]. Moreover, the form of descriptive supervision is inefficient in building robust video reasoning capabilities, especially in terms of temporal reasoning. This mismatch between the complex video content and the limited supervision hinders effective video-language modeling.
In this paper, we propose an innovative approach to develop video understanding capabilities by using L.MSs to simulate and understand video dynamics. Instead of directly aligning LLMs with real video representation, we first introduce textual video representation designed to mimic real visual dynamics. This textual video can be readily generated by advanced LLMs and effectively simulates various video dynamics by describing them in text. Specifically, we present a Textual Video (TextVid) dataset, automatically generated by LLMs. TextVid includes: 1) Textual videos (hereinafter referred to "Tideo"), which consist of sequence of textual frames crafted to mimic the keyframes of real videos, and 2) Tideo annotations, including comprehensive Tideo-level dense descriptions and varied question-am answer (QA) pairs. These annotations are of high quality and closely align with the Tideo content, by virtue of the powerful capability of L.LM in language generation.
sequence of textual frames
Building on the proposed TextVid dataset, we introduce the Text-Only Pre-Alignment (TOPA) framework, to effectively and efficiently pre-align LLMs with the video modality, reducing the need for costly video-text pre-training. We introduce three tasks for video-LLM pre-alignment: Tideo summarization, Tideo QA and multi-choice Tideo QA. To bridge the gap between textual Tidcos and visual videos, we leverage the CLIP [48] model for feature extraction. Specifically, employ the CLIP text encoder to extract level representations for Tideos, and the CLIP visual encoder for real videos. During the text-only pre-aligmment phase, the LLM learns to process continuous CLIP text features of Tideos. In the real video inference phase, it transitions to handling continuous CLIP image features of real video. Due to the aligned CLIP image-text feature space, the LLM can adapt real video inputs despite being trained on textual representations. Our main contributions include:
(1) We propose novel Text-Only Pre-Alignment (TOPA) framework to extend Large Language Models (LLMs) for video understanding. TOPA aligns LLMs with the video modality efficiently and effectively without the need for training real vidcos, reducing the costs for video-text pre-training.
(2) We introduce Tix,Vid, a textual video dataset automatically generated by advanced LLMs. TextVid dataset comprises 721K diverse Tideos along with associated high-quality annotations, which include detailed Tideo descriptions and variety of question-answw -nnswer pairs.
(3) Extensive experiments demonstrate TOPA's effectiveness across various video understanding tasks. Particularly, the TOPA-Llama2-13B model achieves 51.0% Top-1 accuracy in the challenging EgoSchema benchmark, outperforming previous video-text pretraining methods and competitive with recent GPT-3.5-based video agents.
2 Related Work
Vision-language alignment. CLIP [48] aligns the vision and language modalities in common feature space via contrastive learning with large-scale web image-text data. MLLMs [1, 27, 34, 89] align the visual model with LLM via training image-caption pairs and interleaved image-text data. Video-LLMs [7, 23, 32, 83] explore modeling video sequences within LLM spaces, leveraging LLM for video-language understanding. In this paper, we focus on video-LLM, alignment. Rather than using multimodal data for vision-languee alignment, we introduce a novel text-only pre-alignment framework extend L.Mss for video understanding without pre-training on real video-text data.
L.Ms for multimodal data augmentation. Recent research explores the use of L.LMs to enhance the multimodal data. A line of work [5, 12, 34] use LLMs for refining captions extending the

---

User:
CLIP text festure CLIP image feature
Frowen
Can you assist me with urvaling textual video about "Register House Cleaning Business"
Learnbble
Answer: Blande.
A dapter
LLM A ssistant:
Large Language Model
Frame 1:
Caption: "A mid-age
long
blonde hair is sitting in the living room." oomam: mimd-age woman with long hlonde rdd dress,
Q Qestion:: the women's hiir?
hairis sitting on couch. reading book."
Linear Projector
"The touch brown with leather:
Several pill oes are placed on the couch."
Cross modal Project (Eq.
Frame
Text-Only Pigmmenttmnnt
Zero-Shot Inference
Supervised Finetuning
Delailed Video Descriplion:
demonsrastes steps starting house cleaning business
Pairs:
Text Encoder
ligned
CLIP Image
Question:
Encoder
Options
A: Red. B: Black. C: Wlonde. D. Gree.
Frame
Frame
Answen:
A mid age (emaae
The woman stanss the computer and
Blonde. (C)
with long
Monde hail
1..
TextVid Generation
Text-Only Prgmmlitm
Real Videos Inference
Figure! 1: Overview the proposed Text-Only Pre-Alignment (TOPA) framework. Left: The pipeline used for generating the VextVid dataset. Right: The video-LLM alignment framework. During text-only pre-alignment, the LLM learns to process continuous CLIP text features. In zero-shot inference, the LLM uses projected CLIP visual features input. Additionally, TOPA supports supervised fine-tuning on downstream video datasets to further improve the performance.
image caption pairs to diverse visual tasks like visual conversation and image editing. Another line of work [28, 37, 38, 47] further employ advanced LLM to enrich web video supervision for video instruction tuning. In this paper, rather than enhancing multimodal datasets, we propose generating text-only data consisting of "textual videos" and diverse language supervision, which aims simulate real videos and their corresponding annotations.
Long-form video understanding. Long-form video understanding [39, 61, 71] presents significant challenges due to the intricate spatial and temporal dynamicc. Conventional video-text pretraining approaches [4, 45, 65, 66, 77, 90] utilize extensive web video-caption data for video-language alignment. Recent research [28, 66, 83, 88] employ video instruction-tuning for video-LLMM alignment to enhance video-language understanding. Another line of research [22, 54, 81, 49] seeks adapt recent image MILMMS to video understanding. A parallel line of research [9, 42, 55, 60, 63, 13, 78, 82, 67, 21] combine the LLM with various VLM tools video agents perform video-understanding tasks. In this paper, we propose novel text-only pre-alignment framework to efficiently and effectively align LLMs with videos without pre-training on real videos.
Method
In this section, detail the TOPA framework. We first introduce the data generation pipeline of TextVid Dataset (Section 3.1). Next, we describe how to align the Tideo representation with LLM (Section 3.2). Finally, diseuss adapting the text-only aligned video-LLM model for real video inference (Section 3.3). An overview is illustrated in Figure I.
3.1 TextVid Dataset
This dataset, comprising textual videos (Tideos) and associated annotations, is generated by an advanced L.MM (i.e., Gemini Pro 1.0 [56]). The data generation pipeline is detailed in Appendix D. Each Tideo presented textual format and contains 5-15 sequential frames. Each frame includes frame caption that describes the scene and multiple object captions. To enhance understanding and

---

interaction with these Tideos, the dataset features a dense description summarizing the Tideo, as well as set of multiple-chocce questions and answers related to the Tideo content. The structure of each element is as follows:
Dataset Element:
Tideo: Sequence of textual frames (Ti, T2...., Tn 5 = n 2 15
For each frame T;:
Frame caption: C;
Object captions: Di,, for main objects in T;
Annotations:
Global Dense Description of the Tideo: Dy {(Qr: O2. A²)]
Set of Questions-Optioon--e
There are two major advantages of the TextVid dataset. (1) The large-scale and diverse Tideos. As the dataset is text-only and fully generated by an LLM, the size of Vex is practically unlimited. Moreover, the Tideos can cover broad range of domains by simply prompting the language model with appropriate conditions. It is distinctly different from previous web video-text dataset like Howtol00M [40] that are limited to specific human-centric instructional videos. In practice, we enhance the diversity of TextVid by randomly sampling video captions from WebVid-2M [3], video titles from Howto 100m [40], video tasks from Ego4D [15] and object names with descriptions from WordNet [41] as condition of prompts. These varied prompts enable the language model to generate a diverse dataset. (2) The high-quality, consistent and free-form language supervision. The language supervisions are generated along with Tideos. The advanced capabilities of LLM ensure the quality of these supervisions, making them less noisy than web video-text data. Moreover, both the Tideo and the supervision are in textual format, making the supervision elosely aligned with the Tideo's s content. Additionally, the format of the language supervision is unrestricted. For example, we prompt the LLM to generate dense descriptions and multi-choice QA pairs as language supervision.
3.2 Text-Only Pre-Alignment
Preliminary:
Video-LLM alignment. The goal of video-LLM, alignment is to extend pre-trained
LLMs for processing video inputs. Given video sampled with frames { I2.... I, ). Recent work [23, 77] uses a frozen CLIP model to extract the frame-level visual feature, formulated as f.' magg (I.), where Eimage denotes CLIP image encoder. The CLIP features are then projected into the L.M space via a simple linear layer, denoted as G ((f ((ff))........ ...., P(fP)), where G denotes a language model and P denotes a projection layer that projects the CLIP feature to LLM space.
Tideo representation. In this work, we leverage Tideos (c.f. Section 3.1) for video-LLM pre-alignment instead of training on real videos. Specifically, given the textual frame T;, we employ CLIP text encoder to extract the frame representation from frame caption C, and detailed object captions D,, represented as f Fiusion (Eusa(C,), (Dau(D.,),.... Emsu (D.,)), )), where Fiusion is a fusion function such as simple average pooling, and Ejext denotes the CLIP text encoder. A Tideo with n textual frames is represented as V [f] .f.
Text-only pre-alignment. Given the Tideo T, dense Tideo-level description Dy, and QA pairs with multiple choices {[Qx, Ok, Ax)). we introduce the following tasks for Tideo-ILM alignment: (1) Tideo Summarization: Given the Tideo, generate detailed description to summarize the Tideo; (2) Tideo QA: Given the Tideo and question, predict the answer; (3) Multi-choice Tideo QA: Given the Tideo, question and multiple choices, choose the correct answer from the candidates. We employ unified auto-regressive Language Modeling (LM) objective for these three tasks:
[t|
1 |
LLM(OG. 0p)
log G(t)P((V))
(1)
where Vth denotes the Tideo representation, and V [fi. .. during the text-only training, denotes the task specific condition tokens and f; denotes the ith target token. fG and 0p denote the learnable parameters of the L.M adapter and the projection layer P, respectively. In practice, we use the following format as the LLM input: (Task Instruction). Video: (f], .... r.). (Task Conditions). Answer: (Predict Targets). ). For the Tideo summarization task, the target detailed Tideo descriptions. For Tideo QA task, the target is the answer and the condition is the question. For multi-choice Tideo

---

QA task, the target is the correct option and the condition consists with question and options. The details of the specifc prompts are included in Appendix F.1.
3.3 Adapting to Real Video Understanding
Section 3.2 introduces the text-only pre-alignment using the TextVid dataset. this section, we detail how to adapt this text-only pre-aligned LLM for real video understanding. We introduce two approaches: one is zero-shot inference, which directly infers with real video data. And the other is supervised finetuning, where the pre-aligned model is further finetuned downstream video data.
Zero-shot inference. During pre-alignment, we leverage the textual representation V V ='= fx as the Tideo representation. During inference, we take real videos features input, i.e., V where r Fimage(I.), These two modality features f and f that come from CLIP image encoder and CLIP text encoder are aligned via CLIP pre-training. This aligned image-text representation makes possible to perform zero-shot inference without additional finetuning. However, the modality gap phenomenon [16. 30, 31, 44, 85], i.e., CLIP image feature and CLIP text feature are located two completely separate regions of the feature space, prevents us from directly taking the visual feature f" the textual feature ft. To bridge this modality gap, follow DeCap [30] to employ a support memory to project the CLIP visual feature into the CLIP text feature space. This training-fo frree projection process is formulated as:
N
N
exp((mm, ff')]
m,;
(2)
k= exp((m,
where m, denotes CLIP text feature from pre-constructed memory of size N, f" denotes input frame feature of real video and fu-t denotes the projected feature. During zero-shot inference, we take the V (fr- ..., f. ft as the real video's representation.
Supervised finetuning, On the other hand, the text-only pre-alignment can be viewed as a pretraining stage. Following the ceeraiiinnn-inettniii paradigm, the pre-aligned LLMs can then be fine-tuned on real video data for improved downstream task performance. The finetuning process is similar to the text-only pre-al alignmenttn detailed in Section 3.2, except that the LLM receives sequence of CLIP visual features as input instead of CLIP textual features.
3.4 Implementation Details
We leverage Llama2-7B, Llama2-138 13B [57] and Llama3-8B as the LLM backbone, Additionally, we employ the Klama-adapte [84] with an adaptation embedding length of 50. We utilize CLIP-ViT-L as the multimodal encoder. We employ simple linear layer to project the CLIP feature into the LLM feature space. During training, the CL.PP model and LLM backbone are frozen. The projection layer and additional Llama-adapter are trainable. For text-only pre-alignment, we uniformly sample the Tideos into 10 frames. We train the model on mixture of tasks comprising Tideo summarization, Tideo QA. multi-choice Tideo QA with the ratio of 1:1:2. For zero-shot inference, memory for cross-modal projection, consisting 2M CLIP text features sampled from captions in the VextVid dataset. TOPA-Llama2-77 and TOPA-Llama3-SB are trained on four 40G-A 100 GPUs in one day. TOPA-Llama2- 13B is trained in two days. More training details of TOPA and basclines included in Appendix E.2.
construct
Experiments
TOPA enables the LLM to perform various video understanding tasks as shown in Figure 2. In this section, we evaluate TOPA on multi-choice video QA and video captioning tasks. Section 4. evaluates TOPA on NeXT-QA [72], STAR [70], TVQA [24], recent challenging EgoSchema [39] and MVBench[29] benchmarks with the zero-shot setting. We further evaluate TOPA on multi-choice video QA with the finetuning setting (Section 4.2) and zero-shot video captioning task (Section 4.3). In Section 4.4, we conduct ablation study on the LLM prior and input video frames. We report Top-1 accuracy on multi-choice video QA benchmarks and CIDEr [58] score on video captioning benchmarks. We mainly compare TOPA with the following categories of video understanding approaches:
5

---

Describe the video detail.
The video first shows person holding bird of prey her left arm. The bird of prey is perched her fist. The person then holds the bird of prey up her face and shows to the camern. Thee bird of prey is white goshawk, has a black beak and black eyes. The person then shows the bird of prey to the camera again. bird prey is perched
fist looking around.
What is the relationship between the woman and the bird in the video?
The woman is the owner of the bird.
What is the final scene of the video?
The bird is perched the handler's fist. A close-up of the bird's face is shown. The bird is looking around.
Figure 2: Examples of TOPA-LLama2-133 for video-language understanding. Given video, TOPA
able to summarize the video content and answer the questions.
(1) Web video pre-training approaches [4, 45, 64, 66, 77]. This line of work aims to develop general video-language models by leveraging extensive web videos, using associated video captions or audio as weak supervision signals.
(2) Adapting image MILMMs for video understanding [22, 81, 49]. These approaches aim to extend the image understanding capabilities of recent vision-language models (VLMs) to video understanding. Specifically, SeViLa [81] utilizes BLIP-- for localizing and understanding key frames of video. IG-VLM [22] converts video into a composite image by arranging the video frames into grid layout.
(3) LLM-based video agents [42, 59, 60, 63, 82, 13, 21, 67]. This line of work leverages LLMs like GPT-3.5 and GPT-4 as agent to understand video by designing and executing a series of actions. The language-onle agents perceive visual information via recent foundation VLMs (e.g., CLIP [48], BLIP-2 [27], La [87] and PALI [8].
(4) Our text-only pre-alignment. Different from the above works, TOPA leverages the proposed TixtVid dataset for video-ILLM pre-alignment, enabling the LLM to process continuous features. Thus, it can enable performing video understanding tasks.
4.1 Zero-Shot Evaluation on Multi-Choic Video QA
4.1.1 Zero-shot Results on EgoSchema
Table shows the results on EgoSchema full set. We compare our method against range of recent approaches in video understanding. Our proposed text-only pre-alignment framework. despite training without real videos, shows impressive results on the EgoSchema benchmark. TOPA outperforms previous image-based adaptation approach IG-VLM and video agents LLoVi and Vamos with the same scale LLM (Llama2-7B and Llama2- 13B). Moreover, TOPA shows consistent improvements when scaled with a larger LLM backbone, indicating the effectiveness of LIMs in complex video understanding tasks.
The necessity of high-quality language supervision for video understanding, Recent video pre-training approaches like Long [45] and Intern Video [64], despite training on million-level web video-text data, show inferior performance on EgoSchema evaluation. These results highlight the inefficacy and inefficiency of conventional contrastive pre-training understanding long-form videos. primarily due to noisy and simplistic language supervision. In contrast, our TOPA, trained on 721K Tideoswith high-quality language supervision, shows impressive results on EggSchema. It indicates that, unlike image understanding which significantly benefits from leveraging web language as supervision, video understanding may require more precise and accurate language supervision to better capture the complex visual dynamics.
Discussion 1:

---

Table 1: Zero-shot results on EgoSchema [39] full set. Methods that leverage closed-source LLMs marked in gray. denotes the model is trained with in-domain egocentric videos from Ego4D [15]. denotes results EgoSchema subset. Results of Intern Video and MM sourced from
[39]. Results of SEVILA are sourced from [45].
Core VLMs
Core LLMs
Acc@1
Human Eval NeurlPS 2023]
75.0
Gemini-l.5-Pai (aXiv 2024.2
Gemini-1.5-PH
63.2
(Pre-train on web video-text data)
FrozenBiLM [7r][N 2022]
26.9
Intern Video [64][arXiv 2022.12]
32.1
LongViViT [45][CVPR 2024
33.3
MC-VIT-L' [4][CML 2024
44.4
Intern Videov.ee2...3B [66][arXiv 2024.3
(Adapt image MILMs for video understanding)
SEViLA [81 NeeuulPS 2023]
BLIP-2
FLAN-TS-X XL [10]
22.7
MVU [ariv 2024.3
LLaVA-v1.5-138
Vicuna- 13B
37.6
IG-VLM (ariva 2024.3
LLaVA v1.6-7B
Vicuna-7B
35.8'
IG-VLM [22 (arXiv 2024.3
LLaVA v1.6-13B
Vicuna- 13B
47.0"
(LLM-baxed video agents)
RengRepo [ar]va
2024.3
CLIRViT-L
Mixtral-12B [20]
41.2
Vamos [60][arXiv 2023.11]
BLIP-2
Llama2-13B
36.7"
Vamos [60][arXiv 2023.11]
BLIP-2
GPT-3.5
41.2"
Vamos [60][arXiv 2023.11]
BLIP-2
GPT-4
48.3"
MoReVQA [42][CVPR 2024
PALI-3-SB [8]
PaLM-2 [2]
51.7
LLoVi [82][arXiv 2024.3
LaViLa'
GPT-3.5
50,3
VidcoAgent [63]|ECCV 2024
LaViLa'
GPT-4
54.1
LifelongMemory
[67] (arXiv 2024.3
LaViLa"
GPT-4
62.4
VideoAgent [13]
ECCV 2024
Video-LLave [32]
GPT-4
TOPA
CLIP-VIT-L
Llama2-7B
41.2
(Our Text-Only Pre-Alignment)
TOPA
CLIP-VT-LL
Llama3-8B
44.2
TOPA
CLIP-VI-LL
Llama2-13B
51.0
Discussion 2: Video agents versus end-to-end video-LLM modeling. Video agents have shown impressive results on the EgoSchema benchmark, aided by advanced LLMs and VLMs. However, significant limitation of these approaches is their heavy reliance on the powerful LLMs. For example, accuracy Vamos drops by -11.6% when the GPT-4 is replaced with Llama2-13B, largely falling behind the performance of the TOPA-Llama2- LLMs restricts application fields and introduces external overheads. Moreover, video agents make decisions based on the language format clues collected by VLMs. Converting the vidco content into language clues may lead to a limited upper bound compared to end-to-end modeling. Additionally, the inference speed of these approaches is another concern, since it involves multiple interactions with both VLMs and LLMs. contrast, end-to-end video-LLM models, which condense the video into sequence of embeddings the input of LLM, more efficient.
model. The reliance powerful closed-source
4.1.2 Zero-shot Results NEXT-QA, STAR and TVQA
Table shows the multi-choice video QA results impressive performance on the TVQQ and EgoSchema benchmarks, significantly outperforming previous video pre-training models and image-to-video adaptation approaches. This indicates that our TOPA framework effectively enables LLMs to handle video input, despite not being pre-trained on real videos. However, for the NeXT-QA and STAR benchmarks, TOPA underperforms compared to SeViLA and IG-VLM. A major reason is that these benchmarks involve many fine- aineggrinee visual questions, including those about object locations and relationships. SeViLA and IG-VLM, benefiting from the advanced undgrstadddss capabilities of pre- pre-trainne trained VLMs such as LLaVA, excel in answering these fine-grained visual questions. In contrast, our TOPA framework primarily focuses on high-level semantic alignment. Moreover, during zero-shot inference, we project the visual features into the text feature space to bridge the modality gap, as described Eq. This cross-modal semantic projection process tends to overlook fine-graines visual details, such object locations, which leads
various benchmarks. TOPA achieves
7

---

to inferior performance on the STAR benchmark. We provide extensive qualitative results to illustrate TOPA's advantages and limitations across various video understanding tasks in Appendix A.3.
Table 2: Zero-shot results on multi-choice video QA benchmarks.
NExT-QA
STAR
Model (# Frames)
TVQA
EgoSchema
Tem.
Cau.
Des.
Avg.
Int. Seq.
Pre.
Fea.
Avg.
FrozenBil.M (10) [77]
29.7
26.9
VnternVideo (8) [64]
43.4
48.0
65.1
49.1
43.8
43.2
42.3
37.4
41.6
35.9
32.1
SEVILA (32 - 4) [81]
61.3
61.5
75.6
63.6
48.3
45.0
44.4
40.8
44.6
38.2
22.7
IG-VIM--Llava7B (6) [22]
63.1
57.3
74.9
63.1
49.3
50.1
48.4
48.8
49.6
42.1
35.8
IG-VLMM-LIavaI.3BB (6) [22]
61.6
55.7
70.8
61.2
51.5
52.0
51.0
51.8
51.7
44.5
47.0
TOPA-Llama2-78 (10)
53,4
61.3
68.3
59,9
36.4
45.6
39.3
36,3
41.3
48.2
41.2
TOPA-LIama3-SB (10)
53.0
61.9
64.5
59.5
40.8
43.1
39.4
34.5
41.4
48.5
44.2
TOPA-Llama2-13B (10)
57.2
63.6
68.9
62.1
41.6
46.2
44.2
36.7
43.0
50.2
51.0
4.1.3 Results MVBench
MVBench [29] is a recent video-language understanding benchmark that covers 20 challenging video tasks, regrouped from existing video-language benchmarks. Table 3 shows the results. TOPA demonstrates impressive results compared to previous image MLLM and video MLLM. It excels particularly in tasks requiring bigh-level video-language understanding, such as Scene Transition (ST), Episodic Reasoning (ER), and Unexpected Action (UA). TOPA Surprisingly excels in the Action Localization (AL) task, which requires identifying the moment an action occurs. This indicates that the text-only pre-alignment enables the L.LM to understand temporal visual sequences. However, TOPA struggles with tasks that demand fine-graiedd visual understanding, such as Moving Direction (MR), Action Antonym (AA), and Object Shuffle (OS). A common challenge in these tasks the requirement for detailed visual understanding. For example, Action Antonym involves identifying the direction of an action, while Object Shuffle involves locating objects. TOPA struggles these fine-grained visual tasks since it is trained with CLIP text features. The modality gap between CLIP text features and image features hinders TOPA from capturing visual details. Further video instruction tuning might address this limitation, which we leave for future work. We provide qualitative results in Appendix A.3 to illustrate TOPA's advantages and limitations on various vidco understanding tasks.
Table 3: Evaluation results on MVBench. The results of other approaches are sourced from [29]. We gray out the results of VideoChat2 since it utilizes extensive annotated downstream video data.
Model
LLM
Avg
AS
AP
AA
EA
UA
OE
0f OS
MD
AL
ST
AC
MC
MA SC
FP
CO
EN
ER
CT
Random
27.3
25.0
25.0
25.0
25.0
25.0
25.0
25.0
33.3
33.3
33.3
25.0
33.3
20.0
Image ML.MMM Fillowing [if],
modele soke 4.
0 Inpun, with the ougjut emboddings
cowcniznated heforr Sveding Juto the 11M.
mPLUG.Oww-I [74]
LLaMA-7B
29.4|25.0|2.00)
44.5
23.0
24.0
34.0
23.0 24.0
34.5
31.5
40.0
24.0
(1.00
37.0
BIJP2 (27)
FlanT--XI.
24.5
29.0
33,5
17.0
42.0
51.5
26.0
11.0
25.5 26.0
12.5
25.5
30.0
40.0
42.0
27.0
50.0
26.0
37.00
31.0
LLMMA-Adapter [84]
LLaMA-7B
31.7
28.0
51.0
33.0
32.5
33.5
21.5
29.0
41.5
39.5
31.5
28.0
Otter1 [25]
MPT-7B
33.5
33.5
32.0
39.5
30.5
38.5
48.5
44.0
29.5
19.0
25.5
55.0
20.0
32.5
28.5
39.0
28.0
27.0
32.0
29.0
36.5
MiniGPT-4 [89]
Vicuna-TB
18.8
16.0
26.0
16.0
13.0
12.0
32.5 15.5
8.0
34.0
29.5
9.9
InstructBLIP [11]
Vicuna-7B 7B
32.5
20.0
16.5
46.0
24.5
46.0
$1.0
26.0
37.5
22.0
23:0
46.5
42,5
26.5
40.5
32.0
25.5
30.0
25.5
30.5
38.0
LLaVA [34]
Vieuna-78 7B
36.0
28.0
39.5
39.0
41.0
20.5
45.0
34.0
20.5
38.5
47.0
25.0
36.0
26.5
42.0
Video MLLMss All models take 16 frames az Inpur
Onee--V [25]
LLaMA-7B
26.8
23.0
27.5
27.0
29.5
28.0
33.0
23.5
26.0
28.5
18.0
38.5
22.0
22.0
19.0
19.5
mPLUG- mPLUG-OOw.V [79]
LLaMA-7B
29.7
22.0
26.0
34.0
29.0
29.0
40.5
27.0
31.5
27.0
23:0
29.0
31.5
27.0
40.0
44.0
24.0
31.0
26.0
20.5
29.5
Instructed Video MM: models zake
as isppt wini
enceptionn
VidiooCuzzGP)
whtch uoves
fromeez.
VideoChanGPT [38]
Vicuna 7B
32.7
23.5
26.0)
62.0
22.5
26.5
54.0
28.0
40.0
23.0 20.0
31.0
30.5
25.5
39.5
48.5
29.0
33.0
29.5
26.0
35.5
VadeaL.LMMA [83]
Vicuna-7B
34.
27.5
25.5
51.0
29.0
39.0
40.5
38.0
22.5
34.0
22.5
32.5
45.5
40.0
21.0
VideoChat [28]
Vicuna- 7B
35.5
33.5
26.5
56.0
33.5
40.5
53.0 40.5
30.0
25.5
27.0
48,5
35.0
20.5
42.5
46.0
26.5
41.0
23.5
23.5
36.0
VideoCha22 [29]
Vicuna-7B
$1.1
66.0
17.5
60.0
71.5
23.0
23.0
39:0
58.5
11.0
19.0
10.5
Tou-Only Pre-Aligument Video-MLLLM: TOPA
zeno-ahor inference wish 10 frames.
Tero-ZerShoo
LLama2-7B
39.8
42.00
38.5
35.00
66.0
47.5 28.0
37.5
38.0
24.0
42.5
41.5
28.5
34.0
44.00
30.5
TOPA ZeroSbot
LLama2- 130
42.5
38.0
40.0
42.5
35.0
69.0
52.5
58.5
29.5
22.5 43.5
80.5 38.0
25.5
43.0
43.0
29.5
37.5
38.5
50.0
32.5
AS
AA
UA
O8
OS Object Shutte
Aution Sequence ST
Action
Action
anined Notion
Unoupestad Action
Olgest CLEVEER
Object Lnteraction
Mosing Disction
Action
Prediction
Autcrgon
Localization
STAR
FurQA 173]
STAR
Pesception
CLEVVER [t]] Chander
SC
CO
EN
Scrme Tranaiion
Artion
Mextng
Miving
State Change
Broegnired
Chenater
Pgsezatric Navigation
Eglaodic Rosoning
Commetirtral
Counting
Counting
Onker
Inieeees
MaxicNe: [18]
Prooption Teal (10)
CLEVEER [xn
(00)
Perappiin Texl [46]
NTU RQB-D (15)
Peseption Toai (46)
(EN-CE
TVQA [24]
CLEVRER (60]
8

---

4.2 Supervised Finetuning
In this section, we further finetune the pre-aligned TOPA models to study the benefits of TOPA for downstream supervised learning. During finetuning, TOPA directly takes the video feature as input without the cross-modal projection. More finetuning details for each dataset are provided in Appendix E.2. Table 4 shows the finetuning results on multi-choice video QA dataset. For comparison, we include baseline models without text-only pretraining. Our text-only pre-alignment consistently improves the performance across three benchmarks. Notably, TOPA-Llama2-77 achieves 67. % accuracy TVQA, outperforming other approaches by a large margin. These results suggest that our text-only pre-alignment, even without training on real videos, has a similar effect to conventional video-language pre-training.
Table 4: Finetuning results on NEXT-QA, STAR and TVQA.
NExT-QA
STAR
Model (# Frames)
TVQA
Tem.
Cau.
Des.
Avg.
Int.
Seq.
Pre.
Fea.
Avg.
FrozenBiLM (10) [77]
57.5
InterViidoo (8) [64]
58.5
62.5
75.8
63.2
62.7
65.6
54.9
51.9
58.7
57.2
BLIP-2 (4) [81]
70.1
80.1
70.1
52.3
54.8
49.0
51.2
54.5
SEVILA (32 4) [81]
69.4
74.2
81.3
73.8
63.7
70.4
63.1
62.4
64.9
61.6
Llama-VQA-7B (10) [23]
69.2
72.7
75.8
72.0
66.2
67.9
57.2
52.7
65.4
Baseline (10)
65.3
69.0
72.6
68.4
60.8
61.5
49.2
49.8
59.4
63.8
TOPA-Llama2-78 (10)
71.3
74.2
78.5
73.9
66.8
68.9
59.1
55.5
66.4
67.1
+6.0
+5.2
+5.9
+5.5
+6.0
+7.4
+9.9
+5.7
+7.0
+3.3
Baseline (10)
66.0
69.7
73.7
69.1
61.4
62.4
50.6
51.8
60.3
66.2
TOPA-Llama3 8B (10)
70.1
74.5
74.6
73.1
66.3
67.0
59.1
56.5
65.4
68.1
+4.1
+4.8
+0.9
+4.0
+4.9
+4.6
+8.5
+4.7
+5.1
+1.9
Baseline (10)
67.8
71.6
75.2
70.5
58.7
39.5
554.
51.8
38.2
66.6
TOPA-LJama2-13B (10)
72.1
75.8
79.3
75.1
66.8
68.3
61.0
55.1
66.3
69.0
+4.3
+4.2
44.1
+4.2
+8.1
+8.8
+6.7
43.3
+8.1
+2.4
Data-efficient finetuning. Figure 3 shows the results of finetuning LLMs with various ratios of training data. TOPA trained with 10% data achieves 64. 64.7% Top accuracy on NeXT-QA benchmark, significantly outperforming the baseline that without text-only pre-alignment. Besides, when trained with less than 20% data, the baseline model even performs worse than TOPA-zeroshot on NeXT-QA and TVQA, clearly demonstrating the effectiveness of TOPA in limited annotated data scenarios.
75.0
65.0
Acc (90
65.0
62.0
55.0
55.0
35.0
50.0
30.0
Baseline
52.0
Baseline
45.0
Baseling
TOPA
TOPA
TOPA
40.0
TOPA-Z5
TOPA-Z5
ZS
20%
100%
20%
100%
20%
training
training
training
(a) NextQA
(b) STAR
(c) TVQA
Figure 3: Results of finetuning TOPA with various ratios of training data.
4.3 Video Captioning
Results on zero-shot video captioning. We further perform zero-shot video captioning on MSR- VTT [75] and VATEX [62]. As shown Table 5, TOPA largely outperforms previous text-only approaches like Decap which is trained on captions sourced from CC3M [53]. TOPA even outperforms the video-text pre-training approaches like VideoCoCa, which is pre-trained on millions videos-text data, demonstrating that TOPA is efficient and effective framework for video-LLM alignment.

---

Table 5: shro-shot video captioning results. We report CIDEr score for all benchmarks. VT denotes (video clip, text) pairs, IT' denotes (image, text) pairs, and WP denotes webpages consisting of interleaved image and text data.
Training data
MSR-VTT
VATEX
(Web video-text Pre-training)
VideoCoCa-g [76]
144M VT
27.1
22.8
Flamingo-3B [1]
27M VT & 2.IB IT & 43M WP
40.1
Flamingo-9B [1]
27M VT
43M WP
39.5
VideoPrism-B B [86] w/ BBB
618M VT
40.3
24.2
VideoPrism-B [86] w/ PaLM-2-8B
618M VT
38.5
31.7
(Text-only Pre-training)
DeCap [30]
3M Captions
18.6
18.7
TOPA-Llama2-7B
721K TexVVid
32.9
31.0
TOPA-Llama22-
721K TextVid
33.4
32.0
4.4 Ablations
LLM prior in video-language understanding. To investigate the impact of LLM prior in multi- choice video QA, we conduct experiments on EgoSchema with the blind setting, where only the questions and choices are provided to the LLM. Table 6 shows the results. Bard and GPT-4-Turbo achieve 33.2% and 30.8% accuracy, respectively. Gemini-Pro- 1.0 reaches 38.2% accuracy. These blind results of advanced LLMs suggest that in some video QA cases, LLMs accurately choose the correct answer solely based on the question and choices, without visual input. However, the blind performance of Llama2-7B and Ilama2-133 is inferior, potentially due to their smaller model size. After training on the TextVid dataset, TOPA-Llama2-13 133 achieves a blind accuracy of 37.5% (or + +11.7%), closely approaching that of Gemini-Pro-1 model. These results suggest that text-only pre-alignment can effectively prepare LLMs for downstream video-language tasks by leveraging specialized text-only tasks, even in complex scenarios where the original LLMs limited.
Table 6: Blind results on EgoSchema. t denotes
results sourced from [4].
Table 7: Ablation on video frames.
Visual Input
ES Full
TOPA
#Frame
NexIQA
Full
Random Selection
20.0
1
56.1
39.4
GPT-4-Turbo
30.8
Llama2-7B
58.9 (+2.8) 41.0 (+1.6)
X
Bard
33.2
10
59.9 (43.8)
41.2 (+1.8)
Gemini-Pro-1 Pron--
38.2
57.3
47.6
Llama2-7B
X
20.1
Llama2- 13B
60.8 (+3.5)
50.5 (+2.9)
LJama2-138
25.8
10
62.1 (44.8)
51.0 (+3.4)
'OOPA-llama2-78
29.3
TOPA-Llama2-11 13B
37.5
TOPA-Llama2-7B
41.2
TOPA-Llama2-133B
51.0
The impact of video frames. To better investigate TOPA's capability in understanding temporal dynamics of real videos, conduct experiments with different number of frames. Table 7 shows the results. Multiple frames input consistently enhances performance on NeXT-QA and EgoSchema for both TOPA-Llama2-7B and TOPA-Llama2-11 13BB. This indicates that the text-only pre-alignment effectively enables the LLM to handle multiple video frames, despite not being trained on real videos.
Conclusions
In this paper, we introduce TOPA, text-only pre-alignment framework designed for aligning LLMs with video modality without requiring training on real videos. TOPA has demonstrated remarkable performance on the recent, challenging long-form video understanding benchmark, i.e., EgoSchema, showcasing that a text-only approach is effective in capturing the dynamics of long-form videos. Our approach, which includes data generation and text-only pre-alignment, has potential applications across various vision- lisoun-language tasks where obtaining paired vision -anguage data is difficult.
10

---

Acknowledgements
This work was supported by National Key R&D Program of China (No. 2023YFC3305000, the National Natural Science Foundation of China (U2336212), the Fundamental Research Funds for the Zhejiang Provincial Universities (226-2024-00208), Lu's Graduate Education International Exchange Foundation and the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material those of the author(s) and do not reflect the views of National Research Foundation, Singapore. The computational work was partially performed on resources of the National Supercomputing Centre, Singapore httpss//www.nscc.sg))
References
[1] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, al. Flamingo: a visual language model for few-shot learning. Advances in 2022.
Neural Information Processing Systems,
[2] R. Anil, M. Dai, O. Firat, M. Johnson, Lepikhin, A. Passos, Shakeri, E. Taropa, P. Bailey,
Z. Chen, et al. PaLM 2 technical report. arXiv preprint 1.Xii:2335.104000 2023.
[3] M. Bain, A. Nagrani, G. Varol, and A. Zisserman. Frozen in time: A joint video and image encoder for and-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728-1738, 2021.
[4] Balazevic, Y. Shi, P. Papalampidi, R. Chaabouni, Koppula, and O. J. Henaff. Memory consolidation enables long-context video understanding. In Forty-first International Conference on Machine Learning, 2024.
[5] T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
[6] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail tail visual concepts. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pages 3558-3568, 2021.
[7] G. Chen, Y.-D. Zheng, J. Wang, Xu, Y. Huang, J. Pan, Y. Wang, Y. Wang, Y. Qiao, T. Lu, al. VideoLLM: Modeling video sequence with large language models. arXiv preprint arXiv:2305. 13292, 2023.
[8] X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, Salz, S. Goodman, A. Gryener, B. Mustafa, I.. Beyer, A. Kolesnikov, J. Puigcerver, N. Ding, K. Rong, H. Akbari,
G. Mishra, L. Xue, A. V. Thapliyal, J. Bradbury, W. Kuo, M. Seyedhosseini, C. Jia, B. K. Ayan, C. R. Ruiz, A. P. Steiner, A. Angelova, X. Zhai, N. Houlshy, and R. Soricut, PaLI: A jointly-scaled multilingual language-image model. In The Eleventh International Conference on Learning Representations, 2023.
[9] R. Choudhury, K. Niinuma, K. M. Kitani, and L. A. Jeni. Zero-shot video question answering with procedural programs. arXiv preprint XXXx:2312..00337 2023.
[10] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Li, X. Wang, M. Dehghani, Brahma, et al. Scaling instruction-finetuue language models. Journal of Machine Learning Reseurch, 25(70): 2024.
[11] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P.N. Fung, and S. Hoi. Instructblip: Towards general-purpose vasion-language models with instruction tuning. Advances Neural Information Processing Systems, 36, 2023.
[12] I. Fan, D. Krishnan, Isola, D. Katabi, and Y. Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36, 2023.
11

---

[13] Y. Fan, X. Ma, R. Wu, Y. Du, J. Li. Z. Gao, and Q. Li. Videoagent: A memory-augmenn multimodal agent for video understanding. European Conference on Computer Vision, volume 15080 of Lecture Notes in Computer Science, pages 75-92. Springer, 2025.
[14] J. Gao, C. Sun, Z. Yang, and R. Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pages 5267-5275, 2017.
[15] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19011 2022.
[16] S. Gu, C. Clark, and A. Kembhavi. can't believe there's no images! learning visual tasks using only language supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2672-2683, 2023.
[17] T. Han, W. Xie, and A. Zisserman. Temporal alignment networks for long-term video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2906-2916 2022.
[18] Q Huang. Y. Xiong, A. Rao. J. Wang, and D. Lin. MovieNet: A holistic dataset for movie understanding. In European Conference Computer Vision, volume 12357 of Lecture Notes in Computer Science, pages 709-727. Springer, 2020.
[19] C. Jia, Yang. Y. Xia, Y.TT. Chen, Parekh, Pham. Q. Le. Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-languas representation learning with noisy text supervision. In International conference on machine learning. pages 4904-4916. PMLR, 2021.
[20] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. I. Casas, E, B. Hanna, F. Bressand, al. Mixtral of experts. arXiv preprint arXiv:2401 000088 2024.
[21] K. Kahatapitiya, K. Ranasinghe, J. Park, and M. S. Ryoo. Language repository for long video understanding. arXiv preprint arXiv:2403. 14622, 2024.
[22] W. Kim, C. Choi, W. Lee, and W. Rhee. An image grid can be worth a video: Zero-shot video question answering using vlm. arXiv preprint arXiv:2403. 18406, 2024.
[23] D. Ko, J. S. Lee, W.- Kang, B. Roh, and H. Kim. Large language models are temporal and causal reasoners for video question answering. In The 2023 Conference Empirical Methods in Natural Language Processing. 2023.
[24] J. Lei, L. Yu, M. Bansal, and T. L. Berg. Tvqa: Localized, compositional video question answering. In EMNLP, 2018
[25] B. Li, Y. Zhang, L. Chen, J. Wang, F. Pu, J. Yang, C. Li, and Z. Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint 2xXiv:2306005222 2023.
[26] J. Li, D. Li, C. Xiong, and S. Hoi. BLIP: Bootstrapping lunguage-imase pre-training for unified vision-language understanding and generation. In International conference machine learning, pages 12888-12000 PMLR, 2022.
[27] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language -magg pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742, PMLR, 2023.
(28) K. Li, Y. He, Y. Wang, Y. Li, W. Wang. P. Luo, Y. Wang. L. Wang, and Y. Qiao. VideoChat: Chat-centric video understanding. arXiv preprint 2rXiv.2305.06355 2023.
[29] K. Li, Y. Wang, Y. He, Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Lou, L. Wang, and Y. Qiao. MVBench: A comprehensive multi-modal video understanding benchmark. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22195-22200 2024.
12

---

[30] W. Li, L. Zhu, L. Wen, and Y. Yang. Decap: Decoding clip latents for zero-shot captioning via text-only training. The Eleventh International Conference on Learning Representations. 2023.
[31] V. W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 15:17612-17625, 2022.
[32] B. Lin, B. Zhu, Y. Ye, M. Ning, P. Jin, and L. Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:231..10122, 2023.
[33] Y. Lin, J. Zhang, 7. Huang, J. Liu, Z. Wen, and X. Peng. Multi-granularity correspondence learning from long-term noisy videos. In Proceedings of the International Conference on Learning Representations, May 2024.
[34] H. Liu, C. Li, Q. Wu, and Y. Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2023.
[35] J. Liu, A. Shahroudy, M. Perez, G. Wang, L.-Y. Duan, and A. C. Kot. Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding. IEEE transactions pattern analysis and machine intelligence, 42(10):2684-200. 2019.
[36] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[37] R. Luo, Z. Zhao, M. Yang. J. Dong. M. Qiu, P. Lu, T. Wang, and Z. Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:300.00220) 2023.
[38] M. Maaz, H. Rasheed, S. Khan, and F. S. Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05224, 2023.
[39] K. Mangalam, R. Akshulakov, and J. Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36, 2023.
[40] A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic. Howto100m: Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2630-2640, 2019.
[41] G. A. Miller. WordNet: lexical database for english. Communications of the ACM, 1995.
[42] J. Min, S. Buch, A. Nagrani, M. Cho, and C. Schmid. Morevqa: Exploring modular reasoning models for video question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[43] M. Monfort, Andonian, B. Zhou, K. Ramakrishnan, A. Bargal, T. Yan, L. Brown, Q. Fan, D. Gutfreund, C. Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE transactions on pattern analysis and machine intelligence, 42(2))502-500: 2019.
[44] D. Nukrai, Mokady, and Globerson. Text-only training for image captioning using noise- injjse- clip. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4055 4063, 2022.
[45] P. Papalampidi, S. Koppula, S. Pathak, J. Chiu, J. Heyward, V. Patraucean, J. Shen, A. Miech, A. Zisserman, and A. Nematzdeh. A simple recipe for contrastively pre-training video-first encoders beyond 16 frames. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14386-14397, 2024.
[46] V. Patraucean, L. Smaira, A. Gupta, A. Recasens, L. Markeeva, D. Banarse, S. Koppula, M. Malinowski, Y. Yang, C. Doersch, et al. Perception test: A diagnostic benchmark for multimodal vidco models. Advances in Neural Information Processing Systems, 36, 2023.
13

---

[47] L. Q. Li, Y. Wu, Y. Ye. H. Fei, T.-S. Chua, Y. Zhuang, and S. Tang. Momentor: Advancing video large language model with fine-grained temporal reasoning. In Forty-firt International Conference on Machine Learning, 2024.
[48] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8888 8763, PMLR, 2021.
[49] K. Ranasinghe, X. Li, K. Kahatapitiya, and M. S. Ryoo. Understanding long videos in one multimodal language model pass. arXiv preprint (rxXi:2403./6990 2024.
[50] M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut,
A. Lazaridou, O. Firat, J. Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2433005530 2024.
[51] C. Schuhmann, R. Kaczmarezyk, A. Komatsuzaki, A. Katta, R. Vencu, R. Beaumont, J. Jitsev, T. Coombes, and C. Mullis. Laion-400m. Open dataset of clip-filtered 400 million image-text pairs. In NeurlPS Workshop Datacentric A1, 2021.
[52] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278 25278-25294, 2022.
[53] P. Sharma, N. Ding, S. Goodman, and R. Sorieut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2018.
[54] E. Song. W. Chai, G. Wang, Y. Zhang, H. Zhou, F. Wu, X. Guo, T. Ye, Y. Lu, J.-N. Hwang, et al. MovieChat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18221-18232, 2024.
[55] D. Surís, S. Menon, and C. Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888-1 1 1898, 2023.
[56] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[57] H. Touvron, L.. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint ar3.v:2307.09288 2023.
[58] R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-basse image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4566 4575,,55, 2015.
[59] Wang, D. Chen, C. Luo, X. Dai, Yuan, Z. Wu, and Y.-G. Jiang. Chatvideo: trackict-centrri multimodal and versatile video understanding system. arXiv preprint arXiv:2304.14407 2023.
[60] S. Wang, Q Zhao, M. Q. Do, N. Agarwal, K. Lee, and C. Sun. Vamos: Versatile action models for video understanding. arXiv preprint arXiv:2311.13627, 2023.
[61] W. Wang, Z. He, W. Hong, Y. Cheng, X. Zhang, J. Qi, S. Huang, B. Xu, Y. Dong, M. Ding, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.080335 2024.
[62] X. Wang, J. Wu, J. Chen, L. Li, Y.-F. Wang. and W. Y. Wang. Vatex: A large-scale, high- quality multilingual dataset for lad-uuuugg research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4581-4591, 2019.
[63] x. Wang, Y. Zhang, O. Zohar, and S. Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision (ECCV), 2024.
14

---

[64] Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint
2022.
[65] Y. Wang, Y. He, Li, K. Li, J. Yu, X. Ma, X. Li, G. Chen, X. Chen, Y. Wang, P. Luo, Z. Liu, Y. Wang, L. Wang, and Y. Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation. In The Twelfih International Conference on Learning Representations, 2024.
[66] Y. Wang, K. Li, X. J. Yu, Y. He. G. Chen, Pei, R. Zheng, Xu, Z. Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding. CoRR, 2024.
[67] Y. Wang, Y. Yang, and M. Ren. Lifelongmemory: Leveraging llms for answering queries in long-fore form egocentric videos. arXiv preprint arXiv:2312.05266 2024.
[68] Z. Wang, A. Blume, S. Li, G. Liu, Cho, Z. Tang, M. Bansal, and H. Ji. Paxion: Patching action knowledge in video-language foundation models. Advances in Neural Information Processing Systems, 36, 2023.
[69] Z. Wang. J. Li, Y. Hong, Y. Wang, Q Wu, M. Bansal, S. Gould, H. Tan, and Y. Qiao. Scaling data generation in vision-and-languagge navigation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 12009-12020, 2023.
[70] B. Wu, S. Yu, Z. Chen, B. Tenenbaum, and C. Gan. Star: benchmark for situated reasoning in real-world videos. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (Round 2), 2021.
[71] H. Wu, D. Li, B. Chen, and J. Li. Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024.
[72] Xiao, X. Shang, A. Yao, and T.-S. Chua. NEXT-QA: Next phase of question-anse answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777-9786, 2021.
[73] B. Xie, S. Zhang, Z. Zhou, B. Li, Y. Zhang, J. Hessel, J. Yang, and Z. Liu. Funqa: Towards surprising video comprehension. In European Conference on Computer Vision, pages 39-57. Springer, 2025.
[74] H. Xu, Q. Ye, M. Yan, Y. Shi, J. Ye, Y. Xu, Li, B. Bi, Q. Qian, W. Wang, et al. mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video. In ICML, 2023.
[75] J. Xu, T. Mei, T. Yao, and Y. Rui. MSR-VTT: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2016.
[76] S. Yan, T. Zhu, Z. Wang, Y. Cao, M. Zhang, S. Ghosh, Y. Wu, and J. Yu. VideoCoCa: Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arxiv:2212.04997, 2022.
[77] A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid. Zero-shot video question answering via frozen bidirectional language models. In NeurIPS, 2022.
[78] Z. Yang, G. Chen, X. Li, W. Wang, and Y. Yang. Doraemongpt: Toward understanding dynamic scenes with large language models (cxemplified as video agent). In Forty-first International Conference on Machine Learning, 2024.
[79] Q. Ye, H. Xu, G. Xu, J. Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, P. Shi, Y. Shi, et al. mplug- owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[80] K. Yi, C. Gan, Li, P. Kohli, J. Wu, A. Torralba, J. B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations, 2019.
15

---

[81] S. Yu, J. Cho. P. Yaday, and M. Bansal. Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems, 36, 2023.
[82] C. Zhang, T. Lu, M. M. Islam, Z. Wang. S. Yu, M. Bansal, and G. Bertasius, A simple Ilm framework for long-range video question-answering arXiv preprint arXiv:2312.17233 2023.
[83] H. Zhang, X. Li, and L. Bing. Video-Ilama: An instruction-tune audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 543-555 2023.
[84] R. Zhang, Han, C. Liu, A. Zhou, P. Lu, Qiao, H. Li, and P. Gao. LLaMA-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, 2024.
[85] Y. Zhang, E. Sui, and S. Yeung-Levy. Connect, collapse, corrupt: Learning cross-modal tasks with uni-modal data. In International Conference on Learning Representations (ICIR), 2024.
[86] L. Zhao, N. B. Gundavarapu, L. Yuan, H. Zhou, S. Yan, J. J. Sun, L. Friedman, R. Qian, Weyand, Y. Zhao, et al. Videoprism: A foundational visual encoder for video understanding. In Forty-first International Conference Machine Learning. 2024.
[87] Y. Zhao, I. Mísra, Krähenbühl, and R. Girdhar. Learning video representations from large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6586-6597, 2023.
[88] Y. Zhao, L. Zhao, X. Zhou, Wu, C.-T. Chu, H. Miao, F. Schroff, Adam, Liu, Gong, et al. Distilling vision-languee models on millions of videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13106-133 16, 2024.
[89] D. Zhu, Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-languag understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2023.
[90] L. Zhu and Y. Yang. Actbert: Learning global-local video-text representations. Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 8746-8755. 2020.
16

---

Appendix
In Appendix A, we provide additional experiments and analysis.
In Appendix A.1, we further discuss the Multi-choice QA task and study the impact of the multi-choice Tideo QA pre-training.
In Appendix A.2, we study the impact of cross-modal projection (Eq. 2).
In Appendix A.3, provide extensive gualitative results to illustrate TOPA's advantages and limitations across various video understanding tasks.
Appendix B: The limitations of TOPA. Appendix C: The broader impact of TOPA.
Appendix D: The details of proposed Text dataset.
Appendix E, The details of benchmarks.
Appendix E.2, The training details of TOPA.
Appendix F: The prompts used this paper.
Appendix G: The licenses of datasets, codes and models used in this paper.
Appendix H: Examples from TextVid.
17

---

A Additional Experiments
A.1 Further Discussion on Multi-Choice Video QA Task
Table 8: Multi-choice video QA on EgoSchema subset and full set. "Gap" refers to the difference in
performance between the subset and the full set
Method
Eval Mode
ES Subset
ES Full
Gap
LongViViT [45]
Similarity
56.8
33.3
-23.5
MC-ViT-L
Similarity
62.6
44.0
-18.6
MVU [49]
LLM logits
60.3
37.6
-22.7
Miteeppo-Mixtrra------BB--- active) [21]
LLM logits
66.2
41.2
-25.0
VideoAgent (GPT-4) [63]
LLM Selection
60.2
54.1
-6.1
TOPA-IIlama2-13B
LLM Logits
67.5
41.6
-25.9
TOPA-I..ama2-133.
L.M Selection
51.2
51.0
-0.2
TOPA-I.lama2-78
LLM Logits
64.5
41.7
-22.8
TOPA-LI.ama2-7B
LLM Selection
40.4
41.2
+0.8
TOPA-LLama2-7B (w/o multi-choice training)
LLM Logits
65.1
40,5
-24.6
TOPA-LLama2-7B (w/ multi-choice training)
LLM Selection
24.3
24.7
+0.4
A significant advantage of the text-only framework is that we can utilize the LLM to automatically generate diverse language-basee based supervisions as needed, such as the multi-choice QA pairs. To explore the impact of the multi-choice QA training tasks, we conduct ablation study as shown in Table 8.
We would like to first introduce the different evaluation modes for multi-choice video QA tasks: (1)
LLM Selection: Asking the LLM to predict the correct answer given the video-question-choices
(2) LLM Logits: Given video and question as LLM context, we calculate the logits for each choice by averaging the logits of all words within the choice. The choice with higher logit tends to match the video-question context better and is thus selected as the predicted answer. (3) Similarity Comparison [45, 4]: Mapping the multiple crestton-choice pairs and video to common feature space and calculating the similarity between the video and each question-choiie
The performance gap between the EgoSchema subset and full set. Previous work [4, 45] highlights a huge performance gap between the subset and the full set of EgoSchema as shown in Table While concurrent work [21, 49] introduces log- ligeliie based approaches for LLM inference, which significantly improve the performance on EgoSchema subset, the issue of the performance gap still persists. In this paper, observe that such a performance gap phenomenon also occurs in approaches based on LLM logits. However, but diminishes or even disappears selection. We find that this phenomenon may be attributed to differences in the linguistic structures of the choices, as shown below. The choices in the subset often differ in several key works like "create", "repair" and "clean". The similarity logit can effectively identify this keyword- difference to select a more appropriate choice. Conversely, the choices in the full set display more substantial linguistic differences. These variations introduce significant language biases, i.e., some sentences naturally receive higher logits in LLM,, complicating the reliance on similarity or logit for choice selection. In contrast, LLM selection methods take all the choices within the context, allowing the LLM to leverage its robust contextual understanding to select the correct choice.
methods employing LLM
Question- Qnestion-Choices examples from subset:
Q: Can you sunmarize the primary objective and the steps the person took throughou the video to achieve u?? ensure your answer cuptures the essence of the video without listing all actions.
A: The main aim the person's primary objective
create and build new, sturdy wooden bench.
The primary objective for the person was to thoroughly repair restore the wooden bench.
C: The person's 's primary objective was to thoroughly clean and sanitize the wooden bench's 's surface,
Question- Choices examples from full
Q: Considering the entire video, what would you identify the most crucial moments in the person's
shopping experience and why?
A: Following a strict shopping list as guideline and rejecting unfit produce.
B: Conducting taste tests and checking for the freshness of each vegetable.
C: Using math algorithm for optimal vegetable selection.
18

---

LLM for multi-choice QA. In Table 8, we observe a notable phenomenon where the TOPA models achieve impressive results on the subset with the logits evaluation mode. TOPA-LLama2-15 achieves 67.5% topl accuracy, surpassing GPT-4-based video agents. However, when evaluated with the multi-choice selection mode, the performance of the subset declines to 51.2%, but the performance of the full set increases from 41.6% to 51.0%. These results suggest that while the LLM is capable of selecting the answer from multiple choices, it is less sensitive to the keywords within those choices. In contrast, the logit-based approach is sensitive to the keywords but has difficulty with complex sentence understanding.
The impact of the Multi-Choice Tideo QA pre-training. In Table 8, we report the results of TOPA without the multi-choice Tideo QA task, i.e., trained with Tideo summarization and Tideo QA tasks. In this case, we find that ROPA-LLama2-73 maintains similar performance when evaluated with the logit mode. However, there significant performance drop when evaluated with the multi-choice selection mode. This result suggests that while the LLM is adapted to process video inputs, its capability is somewhat constrained and can not extend to more complex video-language tasks beyond the pre-training tasks. This finding highlights the advantage of our text-only data generation and text-only pre-alignment framework, which enable us to develop variety of pre-alignment tasks to better equip the LLM for general video-language tasks such as dense captioning, multi-choice video QA, and video chat.
A.2 The CLIP Modality Gap
TOPA is pretrained with CLIP text features while inferenced with CLIP image features. We employ modality projection approach, i.e., Eq. 2, to bridge this CLIP modality gap during zero-shot inference. Table 9 shows the impact of Eq. 2. TOPA shows inferior results when directly taking the visual feature as input due to the modality gap problem. The projection approach effectively alleviates such modality gap problem without additional training.
Table 9; Ablation on the modality projection (Eq. 2). Results on EgoSchema full set.
Model
without Eq.
with Eq.
POPA-LLamm2-77 7B
30.6
41.2
2oppa-Lamma--11 13B
38.3
51.0
19

---

A.3 Qualitative Results and Analysis
We present qualitative results to illustrate the capabilities and limitations of TOPA across various video understanding tasks. Figure shows qualitative results on the NExT-QA validation set. Figure shows qualitative results the EgoSchema subset. Figure 6 shows qualitative results on 20 video understanding tasks from MVBench.
Question: How did the childrea fill up the bucket?
Choices: (A) tum the tap.
(B) through straw,
person bringing food to baby mouth.
(D) touch top bottle.
(E) sit around
Prediction:
Ground Truth: A
Question: What does the girl do the dog next her the start?
Choices: (A) look table. (B) wave
(C) shave the dog.
(D) play with toy.
(E) caress dog.
Prediction: Ground Truth: E
Question: Why did the man lying down roll the floor the end?
Choices: (A) move to the camera,
(B) fall over. (C) laughing.
(D) buby crying.
(E) let kids walk through.
Prediction:
Ground Truth:
Question: many people are stage!
Choices: (A) one. (B) six. (C) three. (D) five.
E seven.
Prediction: Ground Truth: B
Figure 4: Qualitative results on NeXT-QA. TOPA effectively performs complex video understanding tasks. Additionally, a failure case is also shown in the figure, i.e., in the last sample, TOPA failed to accurately count the number of people.
20

---

Question:
What the overall process and purpose of the actions performed by the person the video?
Choices:
(A) The person molding bricks with mortar and wet clay.
(B) Currently, the person is diligently working constructing wall.
(C) Currentl,, artist the person is diligently working creating unique sculpture.
Currentl, young the person is actively playing and having fun with mud.
(E) The person doing science experiment.
Prediction: A
Ground Truth: A
How does the person interact with the camera during the video and what might be the reason behind these adjustments?
Question:
Choices:
(A) Casually, the adjusts the camera angle skillfully achieve visually sppealing view of herself.
The person skillfully adjusts
angle achieve better, enhanced view of the entire improved, better view the entrance door. The person adjusts the camera to get better view the window.
(C) Casually, the person adjusts the
angle
(E) The person adjusts the camera to better view of the bed and the cloths.
Prediction: E
Ground Truth: E
Determine primary purpose the person's actions the video, and explain the importance repetitive involved process.
Question:
Choices:
(A) In the kitchen, the person diligently making cup of coffee. the repetitive actions, steps, necessary to ensure
that the coffee concoction is mixed evenly throughout.
(B) The person stirring pot of soup. the repetitive actions are necessary to ensure that the soup is cooked evenly.
(C) The person is painting piece of fumiture. the repetitive actions are necessary to ensure that the paint is applied evenly.
(D) The person is diligently washing dishes. the repetitive actions performed are extremely necessary to ensure that every
single dish is thoroughly clean.
(E) Currently, the person is diligently brushing his teeth. these repetitive actions are crucial and necessary to effectively
ensure that his teeth remain ultimately clean und healthy.
Prediction:
Ground Truth:
Figure 5: EgoSchema presents unique challenges compared to previous video benchmarks. The questions in EgoSchema are complex and demand advanced video capabilities, encompassing both recognition and reasoning skills.
21

---

Question:
What huppened after the person held the blanket?
Choices:
(A) Tidied up the table.
(B) Took the dish.
(C) Opened the window.
(D) Took the pillow.
Task: AS
Prediction: Ground Truth:
Action Sequence
Question:
What will the person do next?
Choices:
(A) Eat the medicine.
(B) Take the shoe.
(C) Take the clothes.
()) Throw the food.
Task: AP Action Prediction
Prediction:
Ground Truth:
Question: What activity does the video depiet?
Choices:
(A) Catching something air preventing it ascend
(B) Not sure.
(C) Throwing something air letting
Task: AA
Prediction:
Ground Truth:
Action Antonym
Question:
Which one of these descriptions correctly matches the actions in the video?
Choices:
(A) Descending
B stacking
(C) Flipping
loading
Task: FA
Prediction:
Ground Truth:
Fine-grained Action
Question: What unusual event takes place in the video during the magical segment?
Choices:
(A) The basketball transforms into football field midway through game.
(B) small basketball transforms person when thrown hoop.
(C) man playing baskethall suddenly grows wings and starts flying. gigantic basketball appears after throws nermal-sized the hoop.
Task: UA
Prediction: D
Ground Truth: D
Unexpected Action
Figure 6: Qualitative results MVBench (Task 1-5). The tasks where TOPA performs well, average, or poorly are marked in green, blue, and red colors respectively.
22

---

Question:
Are there any moving brown objects?
Choices:
(A) not sure
(B) no
(C) yes.
Task: OE
Prediction: C
Ground Truth:
Object Existence
Question: Which object down by the person?
Choices:
Task: OI Object Interaction
The laptop.
The dish.
(C) The picture.
The food.
Prediction: C
Ground Truth:
Question: The
multiple similar objects play occlusion game. Where is the hidden object at the end of the game from the person's point of view?
Choices:
(A) Under the second object from the left.
(B) Under the third object from the left.
(C) Under fourth object from the left.
Task: OS
Prediction: B
Ground Truth:
Object Shuffle
Question: you identify direction of sphere's movement video?
Choices:
(A) Down and to the right.
(B) Up and the left.
(C) Up and to the right.
(D) The object is stationary.
Task: MD
Prediction: Ground Truth:
Moving Direction
Question: During which part video does action 'person drinking from cup'
Choices:
(A) In the middle the video.
(B) the of the video.
(C) At the beginning of the video.
(D) Throughout the eatire video.
Task: AL Action Localization
Prediction: A
Ground Truth:
Figure 7: Qualitative results on MVBench (Task 6- 10).
23

---

Question:
Task: MA Which choice matches the scene changes video?
Choices:
(A) From the grocery store to the park.
(B) From the classroom to the library.
(C) From the barbershop to the front of the glass window.
(D) From the office to the beach.
Task: ST
Prediction: C
Ground Truth:
Scene Transition
Question: How miny times did the person launch objects the table?
Choices: (A)
Task: AC
Prediction:
Ground Truth:
Action Count
Question: How many green spheres enter the scene?
Choices: (A)
(B)
(C)
Task: MC
Prediction: Ground Truth:
Moving Count
Question: What shape is the metal object that stationary when the video ends?
Choices: (A) cube
cylinder
sphere
Prediction: Ground Truth:
Moving Attribute
Question:
The person interacts with lighting device among other objects. Is the lighting device at any point?
Choices: (A) I don't know
(B) yes
(C) no
Task: SC State Change
Prediction: C
Ground Truth:
Figure 8: Qualitative results MVBench (Task 11-15).
24

---

Question: What is the action performed by the person in the video?
Choices:
(A) up
(B) drop
(C) (D) jump up
Task: FP
Prediction: A
Ground Truth:
Fine-grained Pose
BAG
Question:
What order letters at the end?
Choices:
(A) kag
(B) kav
(C) bag
Task: CO
Prediction:
A
Ground Truth:
Character Order
This navigation video of agent following instruction: "Walk the front of the fireplace." "hht is the next action it should take?
Question:
Choices:
(A) Move forward
BB) left and move forward
(C) Stop
(D) right move forward
Task: EN
Prediction: B
Ground Truth: B
Egocentric Navigation
Home Baly
Question: Why did Jecy make sign before Monica and Chandler meant to come home?
Choices:
(A) sundwich euting
To welcome new puppy
(C) welcome the baby
To welcome Phocbe
Task: ER
(E) welcome Rachel
Episodic Reasoning
Prediction: Ground Truth:
Question: Without the gray object, which of the following will happen?
Choices:
(A) The yellow cylinder collides with the blue object
Task: CI
(B) The sphere collides with the blue object
Counterfactual Inference
(C) The sphere and the rubber object collide
Prediction:
C
Ground Truth: C
Figure 9: Qualitative results MVBench (Task 15-20).
25

---

B Limitations.
Modality gap in CLIP. Despite the fact that TOPA achieves impressive results, a significant limitation TOPA the gap between the CLIP text feature and CLIP image feature. On the one hand, we use the CLIP text feature for pre-alignment, while inference is with the CLIP visual feature. The modality gap makes the performance degrade, despite employing modality projection mechanism to mitigate it. On the other hand, the CLIP text features cannot fully capture the fine-grained visual details present actual images, such as object locations and relationships. This limitation causes TOPA to struggle in scenarios where questions involve detailed visual information, as shown in Appendix A.3.
Struggles in fine-grained visual understanding. In TOPA, we propose textual videos mimic real videos. However, this approach primarily focuses on keyframes understanding, which is insufficient for scenarios requiring the model to process hundreds of frames high fps, such as action counting tasks. Besides, for the fin-grainne action understanding scenarios, TOPA is unable to capture the fineggriined visual information. For example, in a scene where person elimbs ladder, it is difficult for TOPA to identify whether the person is going up or down due to the limited capability to capture detailed visual dynamics. Further enhancing TOPA with video instruction tuning might address these limitations which we leave for future work.
C Broader Impact
Academic Impact. TOPA's methodology, which frees the need for costly video-text data collection and large-scale pre-training, lowers the barriers to entry for research and development in video-languar understanding technologies. The text-only learning framework of TOPA may inspire researchers with limited resources to engage cutting-edge multi-modal research, providing a more diverse range of perspectives and contributions to this field.
Social Impact. The ultimate objective of TOPA is to develop a general video-language understanding model. Its primary application enables users to extract information from long-form videos without the need for detailed viewing. Moreover, these capabilities for interpreting and managing video content could significantly enhance content moderation systems. Platforms hosting user geregeeneeted content could employ sophisticated video- Lalguaaa models efficiently detect and mitigate the effects of inappropriate or harmful video content.
D The details of proposed TixtVid dataset
We utilize Gemini Pro 1.0 API for our data generation process. We prompt the LLM to create textual videos along with associated annotations. To ensure diverse dataset that covers a wide of domains, we add condition prompts including different themes, video captions, video events, and the names of main objects. Specifically, we leverage video titles from Howto 100M [40], video captions from Web Vid2M [3]. video events from Ego4D) [15], and object from Wordnet [41] conditions to generate diverse textual videos. For Ego4D condition, we ask the LLM mimic an ego-centric video to further improve the diversity of the dataset. Table 10 compares vocabulary sizes. Figure 11 shows that Tideos generated under different conditions have different distributions. For each data generation, we prompt the LLM with the task prompt and one of the condition prompts shown in Figure 13. The statistics TextVid are shown in Table 10. Additionally. we provide Wordcloud of TextVid in Figure 10. The examples of VextVid are shown in Appendix H.
Table 10: Statistics of TextVid.
TextVid
Generated by
Gemini Pro 1.0
# textual videos each condition all QA pairs each question types
721K
Video Title-213;; Video Caption-188: Video Scenarios-2088 205K; Object-120K
3.5M
What-2,5M, Why-410K: 2lw-287K:0tthers-222
all frames
4.4M
Avg. frame
6.13
26

---

Table 11: Vocabulary size of Tideos generated under different prompts. We randomly sampled 20,000
global captions from each type of Tideos for comparison.
Howtol 100m
Ego4D
WebVid
WordNet
Vocab Size
17492
7320
15095
26486
ground tree
worker Fantral vidable
young woman
field
wall tallnng ClOS6
facs Scraen
pan
made
bottle RELASO her
blue Shirt
person
woman
smiling
frame
hanthand
barth
dets
work est
Dold
emile lield
fower
click
gaine
water food Next
back
pick
video
video end
floor
Keople gonding young
femove
camera
bed ronw
ground -
ladg
open
leDut wharing
Irounaan
silling
pioure
vidog
chock
snoying
mest
use
SS
wearing
right hand
ands
god
video Snow
Pair
computer
blaek vesble
butkling
outing bogut
mounaa
using
arge
break
panite eool Aier
hodding
Continue.
park
gana window &iinng
house bazgroumi
An paint
atating
wood
middle
Hplop
pade
back
sitting
tonn foreground
BOWL huld
boy
calor
card
door
R Clugan
man
ED
work
lood
background
Skknng
holding
Biik
werie
knte
bed
girl
grane
Com Pure Schpon
Beard
cleer
book
walking
Begin
showlng
Toblo
small
kichen
make in
forest
talk
person cooking
view
piece
Jcn sky Rgair look quiar
laundry man video start
solr compulet - show group Jeave ceeees
place
book hand dog glake finshhed
ven be
stand
close
strect top
phone room
walks away vdeo thowcase
Ea
Finally
add
bue
nver.
front
couch
placed filed covered d pot p0on brchil
stil
side
sky
Feed
some Roar agr strest
dPg
brown hälr 6 dB5K .rt
jeen
elstance
abot
window tw
standing
geople
variety
vegetable aomire
video Tegin
oook
head
pour
Iron clean
begio
person
Sit
Smiling
bouch
video provide
shot
woman
washing machine
facs
city
paper
stop
shirt
Jana room Plece
table
peinting table..
video demonstrate
start-
plrt
Figure 10: Wordeloud of VixtVid. The frame caption (left) and the dense video caption (right).
Howtol00m
WebVid
Ego4D
WordNet
Figure 11: Visualization of Tideo features generated from different type of prompts.
E Experimental Setup
E.1 Benchmarks
EgoSchema [39] is challenging long-form video understanding benchmark with 5000 multi-choice
questions. The videos EgoSchema are sourced from Ego-4D [15], with an average length of
3 minutes, distinct from previous benchmarks that focused shorter, seconds-long videos. The
27

---

questions in EgoSchema are manually curated to demand long temporal reasoning. We report results on EgoSchema full set.
NEXT-QA [72] is a multi-choice video QA benchmark for causal and temporal reasoning, including 5,440 natural videos. The average length of video is 44 seconds. We report results NExT-QA validation set, which contains 570 videos and 5,000 multiple-choice questions.
STAR [70] is benchmark for situated reasoning. It contains 22K video clips with average length of 44 seconds. There are different question types STAR: Interaction (Int.), Sequence (Seq.), Prediction (Pre.), and Feasibility (Fea.). We report results on STAR validation set.
TVQA [24] is benchmark containing 21k video clips with an average length of 76 seconds. We report results on TVQA validation set without subtitles.
MVbench [29] is reorganized benchmark containing 20 video understanding tasks. These tasks are sourced from STAR [70], PAXION [68], MîT Vi [43], FunQA [73], Perception Test [46],
Charades-STA [14], MovieNet [18], NTU RGB+D [35], VLN-CE [69] and TVQA [24].
E.2 The details of training and evaluation.
We leverage Llama2-7B. Llama2-13B [57] and Llama3-8B the LLM backbone. Additionally, employ the Llama-adapter [84] with an adaptation embedding length 50 for efficient finetuning. We utilize CLIP-ViT-L as the multimodal encoder. We employ simple linear layer to project the CLIP feature into the L.MM feature space. The CLIP model and LLM backbone are frozen. The projection trainable. For text-only pre-alignment, we uniformly sample
layer and additional Llama-adapter the Tideos into 10 frames. We train the model on a mixture of tasks comprising Tideo summarization, Tideo QA, multi-choice Tideo QA with the ratio of 1:1:2. TOPA-Llama2-71 and TOPA-Llama3-8B are trained on four 40G-A 100 GPUs in one day. TOPA-Llama2-13B is trained in two days. For construct memory for cross-modal projection, consisting 2M CLIP text
zero-shot inference, features sampled from the frame captions in the TextVid dataset. We include the training details in Table 12. The actual learning rate is calculated by base Ir Effective Batchsize/2260
Table 12: Training hyper-parameters.
Effective Batchsize (bs, #GPUs, grad accu)
base
Model
Training Dataset
Epoch
Optimizer
Ir
TOPA-LLama2-1 7B
18x4x4
5e-3
AdamW
Pre-training
SOOPA-LLama2-133
TextVid 721K
20
4x4x8
8e-3
weight decay 0.1
TOPA-LLLama3-8B
14x4x8
Se-3
warm up epoch
NextQA
5c-3
TOPA-LLLma2-77
STAR
5
20x4x4
1c-2
TVQA
5c-3
NexIQA
2c-3
Adam A [36]
Fine-tuning TOPA-LLama2-13B
STAR
5
6x4x16
2e-3
weight decay
TVQA
5e-3
warm up 1 epochs
NextQA
le-2
TOPA-LJaam33-88
STAR
5
20x4x4
le-2
TVQA
5e-3
NexIQA
le-2
LLama2-7B
STAR
10
20x4x4
2c-2
TVQA
2c-2
NextQA
1c-2
AdamW
Baseline
LLama2-138
STAR
10
6x4x16
2c-2
weight decay 0.1
TVQA
2c-2
warm up 2 epachs
NexIQA
2c-
LLama3-8B
STAR
10
20x4x4
2e-2
TVQA
le-2
28

---

Prompts
F.1 Text-only Training Prompts.
We use the following prompts for Text-only training. The prompts are partially based on [23].
Tideo Multi-choice QA:
Instruction: Choose the correct answer based on the video and question.
Video: f......(..
Question: [Question].
Choices:
(A): [Option A). (B): (Option B). (C): (Option CJ. (D): (Option D). (E): [Option E).
Answer: The correct choice is [Correct Choice).
Tideo QA:
Instruction: Predict the answer based on the video and question.
Video: (ff.....f.)
Question: [Question).
Answer: (Answer).
Tideo Description:
Instruction: Generate dense description for the video.
Video: [ff,...,ff
Description: (Tideo Description).
F.2
Prompt for Gemini Blind Evaluation
In Table 6, we use the following prompt for the blind evaluation of Gemini-Pro-I on EgoSchema. The prompt is based on [4].
You helpful assistant, expert in answering questions about videos. You will be given question about a video and five possible answer options. You will not be provided frames from the video, but still do your best to guess an answer. You are very capable, think step-by stsep when answering the question.
Question: <question
Possible answer choices:
(A): <Choice 1>
(B): CCoice
(C): <Coice
(D): <Choice 4>
(E): <Choice S=
response should ("Answer": "(X)")
Figure 12: The multi-choice QA prompts used for the blind evaluation of Cemmini-1.0-Pae
F.3 Prompts for Data Generation
The prompts for TextVid generation are shown in Figure 13.
29

---

Task Prompt
Please in creating textual dataset that simulates video with 5-15 frame -ovel deseriptions. The output should structured in JSON fomut.
For each frame, you should describe as following:
Frame Caption: Start with a Framme Caption' each frame, providing comprehensive overview;
Detailed Captions: Follow the frame caption with 1-3 detailed captions focusing on specific objects within the frame.
Each detailed caption should highlight the main object at the beginning.
describing frames, provide:
Global Video Description: Provide summany that synihesies infommation (rom all deseribed frames, ollening
overarching narrative of the vidco.
video. Questions can very, including "when," "wlty"" "whoi," "how. "duesido." "is islar," and "where." Emphasis should be on visual reasoning. Each question will come with five options, loding preferred.
QA Pairs: sei quesiion-mnwer
Questions require integration multiple frimes deduce the
Output should structured following:
"Viden Tisle"
"Frames::
"Prame Caption":""
"Object Name":
Condition Prompts
"Ohject Name
"Object Name":
title (Howto 100M)
Now, please generate textual video about (title).
"Global Video Description":""
Video caption (WehVid):
"QA Paint:
Now, plense generate textual vidco about [caption).
"Question"""
"Options":
Video scenarios (Ego4d):
'A"
Now, please generate ego-centric video about [scenarios]. j. This video is taken by [name).
"g"."
"C"
"D"
Object (Wordnet):
"p
Now, please generate textual video about (object name]. (object description).
"Answer": "TY"
Figure 13: The prompts used in TextVid generation.
30

---

G License
The code, model and proposed dataset will be publicly accessible. We use standard licenses from the community. We include the following licenses for the codes, datasets and models we used in this paper.
1. Benchmarks
NEXT-QA [72]: MIT
STAR [70]: Apache
TVQA [24]: MIT
EgoSchema [39]: MIT
MVBench [29]: MIT
2. Codes
Ldapeerdapter [84] GNU General Public License v3.0
Flipped-VQA [23]: MIT
3. Models
CLIP [48]: MIT
I.aam22 [57]] Llama 2 Community License Agreement
LLama3: Meta Llama 3 Community License Agreement
Gimini-API [56]: Gemini API Additional Terms of Service.
31

---

H Examples of TextVid
Textual Video
Annotations
Detailed Video Descriptions: The video begins with the racecars speeding around track, for the load. Suddenly, one of the racecars suffers a blawout and crashes, causing the drivere to exit the vehiole and surrender the race.
Video Tirle: Win the Budwaiser Shootout Nascar 07
navigates the turns and battles
Frame Caption car mice taking place on racetrack.
Racecar: The mecear speading along the track.
large gutheeee around the watch.
QA Pairt:
Question: What coused the racecar to crash?
Frame Caption 2: The racecar approaching turn in the track.
Options::
Rucecur: The proparing tumn the corner.
mechanical
Driver::
collision with another cor.
Stands: The grandsunds are filled with spectatoos.
C: flat tire.
driver
The
fierce bottle lad. trying
weather conditions.
Rececarl: rocccar
Answer: C
Racecar2: The mecear back pushing hard to overake."
Track: The trak &s lined with sdvertising hillbaards"
happened the
Options:
Frame Caption di The racccar running at the mectrack.
He was injured and taken to the hospital. away He was disqualified from the race. He was armated for micless drriing.
Racecar: The racecar speeding along the straghhaway.
driver focused
ahead.
ceew: The pii crew proparing for pil stop
He the
Frame Caption S: The
blowing damaged smoking
Answer:
The
Driver: The drivar is trying to control the ear.
Question What the name the track where the race taking placa?
Track: The trak is covered debeis.
Options:
Daytona Intrinaional Spaadway.
Caption The got
our and give up the gamte.
B: Indianapolis Motor Speedwwy
Racecar: The rececar stopped on the track."
Tatladegaa Superspeecivay
Driver: The driver walking from the
Charlotte Speedwway
rimning towards
Bristal Motor Speedway
Answerr A
Annotations
Textual Video
Detailed Video Descriptions: This video depicts the Life Croesus, a wealthy and powarful king who was known for his wirdom and intelligance. Tha video Croesus
Title:
throne shown
study, public scuare. Throughout the video, interacting with verious people. including scholars, advisors, and commoners. The video enda with Croesus standing elone in desolata wasteland, contomplating the consequences
Frame Caption Croeuu sits un the throne grandeur with golden
crown and scepter in hand, exuding wealth and power.
Croesus: wealthy
golden
scopter. sitting
throne.
Throne: large, omate chair made of gold and jewels.
Question: Croesus holding
frome
Scepter: thin staff made gold, topped with decorative finial.
Options:
book.
Caption Cresus holds acroll muthematical equationn
seroil.
written en revealing his intelligence and passion for knowledge. golden crown, holding scroll
scepter.
Croesus: wealthy
A sword.
A long, this place poper with writing on Equation: A nuthemntical expression consisting of variables and operators.
Caplion
stands before group people, gesturing
Question:
standing descilate wasteland
arms delivers speech confidene: authornty.
Croesust A wealthy man wearing golden crowa, standing in froms of crowd.
Options:
He
large group people pathored tugether.
has been defeated in battle.
He contemplating the consequences of his actions.
Frame Caption d: Croesus sits with wise man, seeking guidance and knowlelge from older's experience wisdom.
D: He seeking solitude.
hiding
Crocmns wealthy man wearing a golden crows, siting
man.
Answer:
Wise man: An elderly man with loog heard, dresed simple robes.
expression change throughout
wasteland, expressios one of despair and regret, as he costemplates the consequences
Caption
Options:
He becomes increasingly confident.
his actions.
He becomes incressingy worried.
Crocaut: wealthy
standing wasteland.
He becomes incressingl angry.
Wasteland A burm,, umppyy area hmn..
He becomes increasingly sad.
His expression does not
32

---

Textual Video
Annotations
Video Title: Make Dough Without Yeast
Descriptions:
dough
yeast. The weman stans by adding olive to bowl and She then stiss the ingredients together and adds water. The woman continues stir ingredients umill form dough. then kmeads the dough suriace shapes in baking pan and bakes in the oven, Once the becad is baked, the wocman removes it from the oven and lets it cool.
Frame Caption kiichen counter with flous, salt, baking powdes, and
oil bowls
side counter
of salt next the bowl flour.
woman places
dough
Baking powder: A bowl of baking powder nest to the bowl of salt.
Sult: bowl
the bowl flour.
Frume Caption 2: A woman is adding olive oil to the florr and salt
QA What is the woman adding the flour and salt?
bund
Question:
Optioos:
standing behind
adding
florr
Olive
Olive cup of olive oil is in the woman's left hand.
C. Raking powder
Bowdi bowl floor and sall is en the counter frumt the woman.
Sugar
Frume Caption The woman stimrin the ingredients in buwd,
Woman: The womun is standing behind the counter, stimiin the
Answer: B
ingredients bowl.
Question: What
doing tiume
Spoon: spoon
right Bowdi bowl of ingredients on the counter front of the woman.
Optioes:
A: Stiriing the ingredients
H: Kaeading the dough
Caption The adding
ingredients
Shaping Baking the bread
bawl.
basi of ingredieuts.
Woman:: The woman standing behind the counter, adding water to
E: Removing the bread from the even Answer: B
cup Bowl: of ingredients the counter front of the woman.
Question: Where the woman placing the loof dough frome 87
Options:
Frame Cuption The
the ingredients bond with
bowl
baking
The woman standing behind the counter, stirring the ingredients a bowl with fork.
Woman:
C: In baking pat
D: On cutting board
Forkz
plastic heg
Bowd: ingredients the counter
wermen.
Answer:
Frame Caption The woman linedding the dough floured surfacce,
standing behind the
deugh
Question:
Why is the weetnn stirring the ingredients frome
Optioes:
floured surface.
A:To combine them
Suar faor: The counter covered flour.
Doasht A ball of dough on the counter front the woman.
B. To dissolve the salt
activate yeast
dough B: To prevent the dought from sticking Asswer: A
The woman shaping the dough into a loaf. Wamsn:: The woman standing behind the counter, shoping dough into
Frame Caption T:
loaf.
dough counter The woman's hands are on the dough.
Question: What the final product this video?
Hands:
Ogeions:
Dough
woman placing dough baking pan. Womun: The womun standing behind the counter, placing loaf of
Cuption
dough in baking pun.
Pizza
Doughi A lozf of dough
hands.
E: Cake
D: Pasta
baking
33

---

Textual Video
Annotations
Detailed
shauldered
Video Object Name: Shouldered Arch
The arch la made of stone and has a slightly pointad shape. It supported by two corbels that project into the aperture. The arch doorway
Frame Caption 1: A close up view of shouldered arch. The arch made
countyard. The courtyard surrouncie by a wall and has a fourrtain in the center. The arch is alsu decorated with carving of lion's head.
slightly pointed shape. supponed by muject sperture.
Shouldered urch: The shouldered arch made of stone and has slightly
shape
QA Pairs:
Corbell:: The corbels made stome and project into aperture.
Question: What arch shown the videu?
Lintel: The listel is horizontal stone that spans the opening of the arch.
Oytions:
A. Round arch.
pans out context a langer building. The arch is part of'a doorway that leads into
Cuption
shouldered
B: Pointed arch.
Shouldared
coumyyard
D: Horseshae arch.
Shouldezed arch:
part doorway
B: Segmentai arch.
courtyard.
Courtyard: The courtyard is surrounded by wall and has fountain in
What
of corbets the arch?
Buildiing:
The building made of stone and has tiled roof.
Options:
A: To support the lintol.
To decorane the erch
person through doceway wearing long robe and corrying a book.
courtyard.
ventilation
Person: Jose wearing long rehe and canying a book.
Courtvard: courtyad is ounoundedd by wall
fountain
D: To strengthen the urch
To reduce the weight the arch
the center.
Shouldered arch: The shouldered anch part of dourway that leads into courtyard
Question:
What carved into the arch?
Oytions:
A: A lion's head
Frame Caption 4: The camern zooms in on the shouldered arch. The arch decorated with curving lion's head. Shouldered arch:
design
carving
D: A goometric pattem
lion's head.
Nothing
Lion's bend: An elderly with beard. dressed robes.
Frame Caption S: The camer puna out show the shoukkered arch the contest a larger building. The arch is of'a window that overlaokss the
Question: Where the arch located?
Options:
A: In doarway
Shouldered arch: The shouldered urch part a window that overlooks the courtyard.
B: In window
courlyard: The courtyard surourded
fountain
D: cailing
B: In bridge
Annwer
34

---

Textual Video
Video Deseription: Aerial view of young man by the mountain
lake looking at trail map for directions.
Frame Cuption
sitting couch,
temple
Man: with serious expression sitting oouch, his head
eyes closed.
Sota: beige fabric sofa the center the living room, and man
Annotations
sitting
Hands: The right hand forchesd, rubbing temple.
ceuch rubbing his temple to relieve migraine headsche
Detailed Video Descriptions:
Background: Behind sofs, a large window white curtain the right sade.
Question: What the man doing help reliave his pain?
Caption 2: massages temple.
Options:
Mon: The man is still sitting on the couch, his expression is still scrious
Rubbing
eyes still closed.
D: Takkigg medication.
man sitting
beige
living room.
C. Applying cold compress.
Hands: The mun's right hand is still on his forehusd, applying more
Drinking plenty
E: Exenciaing.
Background:
large
behind
Answer
sofa.
Question What the men's axpression?
pain. Man: The man's expression is getting worse, he wrinkles his brows and his mouth slighly
Cuption
Options:
Hoppy
D: Sad
beige fabric forshead, applying lots
Handis: The right preseare.
pain
Surprised
Background:: large with white behind and
Anower:
Question: What does the man have?
Frame Cuption holding his and grimacing.
Options:
up paint, eyes closed tightly.
A: headache
Sufa: An olderly mun with long beard, dresed in simple robes.
A toothscha
Hands: The man's right hand is still his firshedd applying pressure.
A sttoaahaache
Backstround: A luge
sofa.
Auu
Answem
slightly
Man: The man's face looks little better, starting release the
preseare on his forehead.
beige fabric
living
Hands: The man's right hand is still his forebead, but he not applying
preseare anymsore.
Baclgroond: A large window with white behind and
sofa.
35

---

Annotations
Detailed Video Descriptions: The video shows Jose wooking an omelet. He first choos vagetables cutting board, then cracks then pours the mixture into pan. flips omeelet the pan untä cooked through.
whisks egg other ingradients the bowl,
Textual Video
QA Pairs:
Question: What Joee's primary this video?
Video Recorder: Jose
Oytions:
A: To wash dishes.
Frame Caption firat-peraon perspective standing the
omnllt.
kitchen holding a knite and chopping vogotables on a cutting baard.
Cutting board: rectangular wooden board placed
D: est dinmer.
To chop vagetables.
Knife: Jose holding a sharp, stainless steel kniffe in right hend.
Question: What
does Jose cook the omelet?
Frame
Jose
Man: The man is still sitting on the couch. his expression still scrious
Oytions:
and his eyes are still closed.
A: A cast fron skillet.
bowt placed the kitchen Ees: Jose is holding brown egs in his left hand.
A nonstiic
ovan
D: A saucepan.
Caption
whisking egg
ingredients
baking dish.
Whisk:: lose holding sliver whisk his right hand.
What ingredients
the mexture?
Bowt: The white ceramic bowl Eilled with
other ingredients
Options:
A: Solt and pepper
Frame Caption d: Jose is pouring the misture into pum.
Milk and cheese.
Pan: A black nonetick pan placed the stovetop.
ccramic Spatula: Jose is holding a black spotula in his right hand.
holding
D: Mest and vegetables.
Stugar and butter
Frame Caption
flipping Omelet: The omelet is cooking the pan, browning on one side.
pan.
Question: What does Jese the ornelet after cooked?
Sputulu: Jose using the black spatula to flip the omelet.
serves
8: Ho eats with a tork. lunch
D: He gives to dog.
Di Her throwws erway.
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
This paper aim extend Large Language Models for video understanding via Text-Only Pre-Alignment, which does not need real videos for pre-training. Extensive experiments on video understanding benchmarks demonstrate the effectiveness of TOPA.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
The limitations of TOPA are detailed in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or a speech-to-tex. system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
37

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The training details of TOPA are included in Appendix E.2. The dataset generation pipeline detailed in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but should possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
38

---

Question: Does the paper provide open access to the data and code. with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
The code and dataset is publicly accessible (https: trub.hu...c. htw.tttu....m........ caan/dhg-vve wei/
TOPA).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
www.uuiiiisiooooipcooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines httts::/mips.cccppbbic///////
for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ys]]
Justification:
The details of training and evaluation are included in Appendix E.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: TOPA is large-scale pre-training framework. Similar to previous related work [45, 81, 29], error bars are not reported because would be computationally too expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
39

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors). should be clear whether the error the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The details of compute resources are included in Section 3.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the Code of Ethics and it conforms with the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The details of broader impacts are included Appendix C.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
40

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Our model and data focus on video- ladoulanguaage understanding, with minimal risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The licenses mentioned in Appendix G.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
41

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
The details of the new dataset and model are detailed in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
42


Reviews:
[{"review_id": "4Nh8msHCap", "reviewer": "Reviewer_kuEy", "summary": "The authors introduce Text-Only Pre-Alignment (TOPA), a novel approach that extends large language models (LLMs) for video understanding without pre-training on real video data. TOPA generates Textual Videos, comprising continuous textual frames and annotations to simulate video-text data, and uses these to pre-align a language-only LLM with the video modality. The CLIP model aligns image and text modalities, bridging the gap between textual and real videos. TOPA encodes continuous textual frames as CLIP text features, analogous to CLIP image features, thus aligning the LLM with real video representations.", "questions": "1. Can you add the performance of the proposed models on the subset of EgoSchema?\n2. Can you record the inference time of your proposed method on the entire dataset and compare it with the inference times of existing works, including those listed in this review? There is a concern that the visual-to-text feature conversion in zero-shot mode may slow down the entire model.\n3. Can you attempt to use the largest open-source models and show the zero-shot performance on the EgoSchema dataset? It is important to assess whether the proposed model can scale up to larger LLMs and if its performance can be comparable to existing works using closed-source LLMs.\n\nI will reevaluate the proposed model based on the answers to the questions.", "limitations": "I mentioned some concerns about the work in the weakness section.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors present a novel strategy for approaching video understanding.\n\nThe proposed models perform well on Egoschema, NextQA, STAR, and TVQA using the Llama2 and Llama3 LLMs.", "weaknesses": "In Table 1, the author did not include key papers such as LifelongMemory[1], Video-Agent: A Memory-Augmented Multimodal Agent for Video Understanding[2], LangRepo[3], and MVU[4]. LifelongMemory achieves 68% and 62.1% on the subset and the full set of EgoSchema, respectively, and should be included in the table to allow readers and reviewers to understand the relative performance, considering the size of the LLM. Additionally, LangRepo and MVU use open-source LLMs (Mixtral7B, Mixtral-8×7B, Llama-2-7b-Chat, Gemma-7b-IT, and Mistral-7B-Instruct) and should also be included to provide a comprehensive comparison. Furthermore, the table is missing the performance on the subset, whereas other papers report performance on both the subset and the full set. Including this information is crucial to observe how accuracy changes from the subset to the full set. Similarly, in Table 2, the author did not include key papers such as LangRepo, MVU, VideoAgent, and LLoVi, making it difficult for readers and reviewers to compare the proposed models with existing works.\n\nhttps://arxiv.org/abs/2312.05269v1\n[1] Ying Wang, Yanlai Yang, and Mengye Ren. Lifelongmemory: Leveraging llms for answering queries in long-form egocentric videos, 2023. \n\nhttps://arxiv.org/abs/2403.11481\n[2] Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Video-agent: A memory-augmented multimodal agent for video understanding, 2024\n\nhttps://arxiv.org/abs/2403.14622\n[3] Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo. Language repository for long video understanding, 2024.\n\nhttps://arxiv.org/abs/2403.16998v1\n[4] Kanchana Ranasinghe, Xiang Li, Kumara Kahatapitiya, Michael S. Ryoo. Understanding Long Videos in One Multimodal Language Model Pass, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Weaknesses:** In Table 1, the author did not include key papers such as LifelongMemory[1], Video-Agent: A Memory-Augmented Multimodal Agent for Video Understanding[2], LangRepo[3], and MVU[4]. LifelongMemory achieves 68% and 62.1% on the subset and the full set of EgoSchema, respectively, and should be included in the table to allow readers and reviewers to understand the relative performance, considering the size of the LLM. Additionally, LangRepo and MVU use open-source LLMs (Mixtral7B, Mixtral-8×7B, Llama-2-7b-Chat, Gemma-7b-IT, and Mistral-7B-Instruct) and should also be included to provide a comprehensive comparison. Furthermore, the table is missing the performance on the subset, whereas other papers report performance on both the subset and the full set. Including this information is crucial to observe how accuracy changes from the subset to the full set. Similarly, in Table 2, the author did not include key papers such as LangRepo, MVU, VideoAgent, and LLoVi, making it difficult for readers and reviewers to compare the proposed models with existing works.\n\nAns: Thank you for your valuable suggestions and for highlighting these related papers. We will include and discuss them in the final version of our paper for a more comprehensive comparison. We kindly note that these papers are appeared/updated on arXiv in **late March 2024** and should be considered as **contemporaneous work**.\n\n**Discussion on LifelongMemory, Video-Agent, LangRepo, and MVU.** We have categoried existing video understanding approaches into 4 categories in Section 4, including *Web video pre-training approaches*, *Adapt image MLLMs for video understanding*, *LLM-based video agents*, and our *Text-only Pre-alignment*. We should note that our *Text-only Pre-alignment* different from previous work that levearges text-only data for video-LLM alignment. We'd like to categorize **LifelongMemory, Video-Agent and LangRepo** into *LLM-based video agents* approaches that leverage VLMs tools convert visual information into textual, and then process video understanding task via LLMs. **MVU** belongs to *Adapt image MLLMs for video understanding*, focusing on adapting image-MLLMs for video understanding. We will update the tables to include results from these papers for a more comprehensive comparison.\n\n**Performance on EgoSchema subset & Evaluation approaches for multi-choices QA**. We report results for the EgoSchema subset in Appendix A.2. We analyze the performance gap between the subset and the full set and discuss the impact of various evaluation methods, including similarity-based, logits-based, and LLM-selection approaches. Appendix A.2 reveals that logits-based and similarity-based approaches perform well on the EgoSchema subset. However, these approaches show limited effectiveness on the full set, resulting in a significant performance gap. This gap may be due to the differing linguistic structures of the two sets: the subset features more similar sentence structures with slight variations, making it easier to distinguish using these methods. In contrast, the full set contains a wider variety of choices, which poses challenges for logits and similarity-based approaches.\n\nWe notice that LangRepo and MVU use log-likelihood-based selection for EgoSchema evaluation, which aligns with our discussion in Appendix A.2. We'll include a further discussion about these two contemporaneous work on this section.\n\n>Question1: Can you add the performance of the proposed models on the subset of EgoSchema?\n\nAnswered in Weakness.\n\n>  Question 2: Can you record the inference time of your proposed method on the entire dataset and compare it with the inference times of existing works, including those listed in this review? There is a concern that the visual-to-text feature conversion in zero-shot mode may slow down the entire model.\n\nAns: Thank you for your suggestion. We provide inference time on the following table. We conduct the experiment on a single A100 GPU and report the inference time per sample. The main computational cost of TOPA comes from LLM inference, while the conversion of visual-to-text features adds negligible overhead. Notably, TOPA is more efficient than video-agent approaches, which require multiple additional calls of VLMs.\n\n| Method                      | Core LLM/VLM   | Visual Encoding | Visual-to-Text | LLM inference | All    |\n| --------------------------- | -------------- | --------------- | -------------- | ------------- | ------ |\n| TOPA-LLama2-13B             | LLama2-13B     | 0.0241          | 0.0003         | 0.4998        | 0.5242 |\n| SF-VLM [4]                  | LLaVA-v1.5-13B | -               | -              | -             | 0.9794 |\n| LLoVi (results from MVU[4]) | -              | -               | -              | -             | 207    |\n\n\n\n> Question 3: Can you attempt to use the largest open-source models and show the zero-shot performance on the EgoSchema dataset? \n\nWe provide the results of TOPA using Llama2-7B, Llama2-13B, and Llama3-8B across various experiments, showing that TOPA benefits from a more powerful and larger-scale LLM backbone. Exploring the performance of TOPA with even more capable models, such as Llama3.1-70B, would be valuable. However, the training overhead for such large models is currently prohibitive for our research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3tv3XMNKyu", "reviewer": "Reviewer_nFok", "summary": "The paper presents Text-Only Pre-Alignment (TOPA), a method to extend LLMs for video understanding without training on real video data. TOPA generates \"Textual Videos\" using LLMs to simulate real video data, then pre-aligns the LLMs with video modalities using the CLIP model for feature extraction. This approach achieves impressive results across various benchmarks, surpassing previous methods and competing well with recent video understanding models.", "questions": "**Questions:**\n\n1. Have you considered using combined pretraining with both Tideo and multi-modal data?\n2. Does the Video-LLM alignment include temporal aggregation of frames?\n3. Could you provide ablation studies on how the prompts and different parts of the dataset affect downstream performance, such as the inclusion of condition prompts, the influence of global descriptions, and various QA pairs? It would also be interesting to see how complex the Tideos are and the consistency of frame descriptions. Could the proposed adaptation of LLMs be trained using only text data from available video datasets?\n4. Have you considered creating textual descriptions of frames from the original videos (for evaluation) and comparing zero-shot evaluation text vs. videos?\n\nMain Concern:\nThe quality of the generated video descriptions (Question 3).", "limitations": "The limitations are discussed and cover the main disadvantages of the current model.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The term \"Tideo\" for textual videos effectively captures the concept.\n- The motivation of the work is well-formulated, addressing both the high cost of video training and the challenges of aligning different modalities.\n- The authors discuss the modality gap, the misalignment between visual and text modalities, and its impact on performance.\n- The related work section is concise and clearly positions the study within the context of previous research.\n- The novel idea of creating textual descriptions of videos without having the actual videos, thereby skipping the alignment part between video and text modalities, is interesting. It shows that simulating other modality via text and  tuning LLMs solely on language can be more efficient than combining multiple modalities.\n- The proposed method is thoroughly evaluated across a range of tasks and models.", "weaknesses": "1. L42: The authors describe subtitles as \"frame-level descriptions,\" but subtitles are actually spoken speech extracted from videos and have intrinsic context and alignment issues [HowTo100M].\n2. The description of the training process and stages is confusing. It needs to be clearer what is trained when and how. The subsection \"Video-LLM alignment\" is confusing given the title \"text-only pre-alignment.\" The presentation would benefit from an overview of high-level steps for training/inference, followed by detailed descriptions. The dataset section lacks references to the appendix for exact prompt details.\n\n**Minor:**\n\n- The references should be clickable.\n- Correct “Egoschema” to “EgoSchema.”\n- The statements “1. Intrinsic complexity of the video modality” and “2. The limitations of web language supervision” should include references to support these claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Overall, thank you for your careful reading, detailed feedback, and valuable suggestions. We will revise our paper based on the discussion.\n\n> Weakness1: L42: The authors describe subtitles as \"frame-level descriptions,\" but subtitles are actually spoken speech extracted from videos and have intrinsic context and alignment issues [HowTo100M].\n\nAns: We agree that \"frame-level descriptions\" is not an accurate term in this context. Our intention was to highlight that subtitles primarily provide local context and lack the long-term temporal supervision necessary for understanding longer videos, such as those in EgoSchema. We will revise this section for clarity, address [Minor3], and include a discussion on the challenges of visual-textual misalignment [1] [2].\n\n[1]  Lin, Yijie, et al. \"Multi-granularity correspondence learning from long-term noisy videos.\"  ICLR 2024\n\n[2] Han, Tengda, Weidi Xie, and Andrew Zisserman. \"Temporal alignment networks for long-term video.\" CVPR 2022\n\n> Weakness2:The description of the training process and stages is confusing. The subsection \"Video-LLM alignment\" is confusing given the title \"text-only pre-alignment.\" \n\nAns: We appreciate the feedback and will revise the paper for clarity. Here is a brief response.\n\n1. **Training and Inference**.  At \"text-only pre-alignment\" stage, the model is trained on the TextVid dataset, where the LLM learns to process continuous CLIP text features. \n\n   Then, the text-only pre-aligned model can be applied for real videos in two ways:\n\n   (a) Zero-shot inference (Sec. 4.1). The visual features of test videos are projected into textual features space, and then processed by LLM. \n\n   (b) Finetuning on downstream video datasets (Sec. 4.2). The model is further fine-tuned and evaluated using video features.\n\n2. **Video-LLM alignment and text-only pre-alignment.** Video-LLM alignment involves aligning LLMs with video modalities, enabling the LLM to process video inputs. Our text-only pre-alignment is a specialized approach within this framework, leveraging text-only data for training.\n\n\n\n> **Q1**: Have you considered using combined pretraining with both Tideo and multi-modal data?\n\nAns: We attempted pretraining with both Tideos and the WebVid dataset but did not observe performance improvements. We attribute this to the limited supervision signals from the short video captions in WebVid. We believe the use of Tideos in this paper effectively highlights its features and advantages. Integrating Tideos with multimodal data will be explored in future work.\n\n> **Q2**: Temporal aggregation of frames.\n\nAns: We haven’t designed extra temporal aggregation modules in this work. We assume the LLM can handle temporal aggregation since frames are represented as a sequence of embeddings within the LLM. Table 6 shows TOPA benefits from using more frames, suggesting that the LLM can achieve temporal aggregation.\n\n> **Q3**: Ablation studies on TextVid ... Consistency of frame descriptions ...Trained using text data from video datasets\n\nAns:\n\n1. **Condition Prompts Enhance Tideo Diversity.** We aim to enhance the diversity of Tideos through varied condition prompts. We provide visualizations in the submitted PDF. (a) Figure 1 shows that Tideos generated under different conditions have different distributions, with Tideos-WordNet being the most diverse and widely dispersed. (b) Figure 2 illustrates that Tideos-Howto100m and Tideos-Ego4D focus primarily on human-centric activities, while Tideos-WebVid and Tideos-WordNet cover more scenarios. (c) Table 3 compares vocabulary sizes, revealing that Tideos-WordNet encompasses more objects.\n\n   **Diversity and performance**. Our early experiments found that Tideos-WebVid and Tideos-WordNet do not improve downstream performance. It's because the video understanding benchmarks, such as EgoSchema, primarily focus on human-centric activities, which are more closely aligned with Tideos-Ego4D and Tideos-Howto100m. Consequently, Tideos-WebVid and Tideos-WordNet offer limited additional benefits. We look forward to future open-domain video understanding benchmarks to better assess the impact of the our diverse Tideos.\n\n2. **Global descriptions and QA pairs.** In submitted PDF, an ablation study (Table 1) shows that multi-choice Tideo-QA and Tideo Summarization tasks enhance the performance. For further results and detailed analysis of the multi-choice QA tasks, please see Appendix A.2.\n\n3. **Tideos Consistency:** In Appendix H, we provide examples of Tideos. The frame descriptions, including frame captions and object captions, show strong consistency.\n\n4. **Text data from available video datasets may not be suitable for text-only pre-alignment.**  Our text-only pre-alignment relies on Tideos and corresponding supervision. Text data from video datasets, such as Howto100M, provide insufficient supervision for effective training. Additionally, the textual data associated with these videos, such as narrations, is often less detailed compared to our Tideos.\n\n> **Q4:** Have you considered creating textual descriptions of frames from the original videos (for evaluation) and comparing zero-shot evaluation text vs. videos?\n\nThank you for your insightful suggestion. We add an experiment to study this. We used LaViLa-xl [1] as the video captioner. We divided each test video into 10 clips, each containing 4 frames, and generated clip-level captions for them. These clip-level captions are then processed by the CLIP text encoder to produce 10 textual features, which serve as input for the LLMs. \n\nResults in Table 2 reveal that both project-based and caption-based inference approaches help to mitigate the domain gap issue. Using textual features derived from video captions achieves better results, benefiting from clip-level captions. This demonstrates the flexibility of TOPA and its applicability to various scenarios.\n\nRef: [1] Zhao, Yue, et al. \"Learning video representations from large language models.\" CVPR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "beO4YyytFn", "reviewer": "Reviewer_oFSV", "summary": "This paper introduces Text-Only Pre-Alignment (TOPA), a framework designed to extend LLMs for video understanding, without the need for real video data. TOPA leverages textual videos (Tideo), generated by capable LLMs, to mimic real videos. Aided by CLIP's aligned cross-modal space, this  framework despite training on text-only data,  can effectively handle real video input. The effectiveness of TOPA is demonstrated through extensive experiments on video understanding benchmarks.", "questions": "1. The proposed approach regards video as a few key frames. Can TOPA handle scenarios with more frames?\n\n2. Could you provide more details about the finetuning stage? Are both the projector and adapter optimized during finetuning?\n\n3. How scalable is the TOPA framework when applied to larger datasets or more complex video understanding tasks?", "limitations": "As stated by the authors, the text-only learning framework is restricted by the imperfectly aligned CLIP model. The projected visual representations are limited in capturing fine-grained visual details.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper propose a novel text-only pre-alignment framework, TOPA, to extend LLM for video understanding. This method leverages LLM-generated text-only data for training, reducing the reliance on video data and human annotations. The text-only learning pipeline introduced in this paper has potential implications for multimodal learning.\n\n2. The introduced TextVid dataset is innovative. The idea of using LLM-generated \"Tideo\" to mimic real videos and employing CLIP to bridge the Tideo representation with real video representation is particularly interesting\n\n3. The method is evaluated across various video understanding benchmarks, including the challenging EgoSchema and MVbench. The effectiveness of TOPA is demonstrated in multiple settings, including zero-shot evaluation, pretrain-finetune schemes, and data-efficient finetuning. The paper also performs ablation studies to analyze the impact of proposed method.\n\n4. Many different approaches have compared and discussed in this paper, including vidoe-text pretraining approaches, image MLLM-based approaches and LLM-based video agents. This comprehensive comparison could serve as a valuable reference for future research on video understanding.\n\n5. The paper is well written and easy to follow.", "weaknesses": "1. The paper could benefit from providing a more detailed comparison between Tideo and real video. For instance, the TextVid dataset likely covers more diverse domains, due to the use of varied prompts during Tideo generation. It could be analyzed in depth.\n\n2. The proposed Tideos are primarily key-frame level representations, which may not fully capture the temporal dynamics of real videos.\n\n3. The experiment results on MVBench should be incorporated into Section 4.1.2 for improved understanding of TOPA. The results on MVBench clearly demonstrate TOPA's strengths and limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness1: The paper could benefit from providing a more detailed comparison between Tideo and real video. For instance, the TextVid dataset likely covers more diverse domains, due to the use of varied prompts during Tideo generation. It could be analyzed in depth.\n\nAns: Thank you for your suggestion. TextVid is notably diverse compared to real video datasets. For instance, previous datasets like Howto100m and Ego4D focus on specific domains, such as instructional videos from YouTube. By using varied condition prompts, we enhance the diversity of Tideos to cover a broader range of domains.\n\nWe provide visualizations in the submitted PDF. (a) Figure 1 shows that Tideos generated under different conditions exhibit varying distributions. Tideos-WordNet, in particular, displays the most diverse distribution, scattered across the space. (b) Figure 2 illustrates that Tideos-Howto100m and Tideos-Ego4D focus primarily on human activities, while Tideos-WebVid and Tideos-WordNet cover a broader range of scenarios. (c) Table 3 compares vocabulary sizes, showing that Tideos-WordNet tends to cover more objects.\n\n> Weakness2: The proposed Tideos are primarily key-frame level representations, which may not fully capture the temporal dynamics of real videos.\n\nAns: Sampling videos into several key frames is a widely adopted approach in vision-language models [1,2,3] due to its efficiency and considerable performance. Understanding videos at high frame rates is indeed challenging and essential for tasks, such as action counting. Recent work has explored this area [4,5]. However, this is not the primary focus of our paper. TOPA may be extended to handle more frames scenarios by incorporating additional local aggregation modules and fine-tuning with a larger number of frames, which we leave it as future work.\n\n\n\n> Weakness3: The experiment results on MVBench should be incorporated into Section 4.1.2 for improved understanding of TOPA. The results on MVBench clearly demonstrate TOPA's strengths and limitations.\n\nAns: Thank you for your suggestion. Due to page limitations, we could not include the MVBench results in the submitted version. We will try to incorporate the MVBench experiment into the main paper for improved clarity and understanding of TOPA in final version.\n\n\n\n> Q1: The proposed approach regards video as a few key frames. Can TOPA handle scenarios with more frames?\n\nAns: In this paper, we fix the input frames to 10 for simplicity. There are several approaches to extend TOPA to handle more frames like (a) Frame selection: Choosing a subset of frames based on criteria such as importance or relevance. (b) Additional local aggregation module: Adding a module that aggregates information from local regions to obtain clip-level representation. (c) Fine-tuning with more frames: Finetuning the model with larger number of frames.\n\n> Q2: Could you provide more details about the finetuning stage? Are both the projector and adapter optimized during finetuning?\n\nAns: We provide detailed finetuing hyper-parameters for each dataset on Table 11. During supervised finetuning, both of the projector and adapter are optimized. We will enhance the presentation of training, fine-tuning, and inference details in the final version of our paper.\n\n> Q3: How scalable is the TOPA framework when applied to larger datasets or more complex video understanding tasks?\n\nAns: The TOPA framework exhibits considerable potential for scalability across larger datasets and video understanding tasks, due to the innovative data generation method and text-only training. TOPA framework leverages LLMs for data generation, providing several key advantages: (1) the potential size of the dataset, TextVid, is virtually unlimited; (2) the diversity of domains covered by Tideos can be easily expanded by prompting LLMs with specific conditions; (3) the supervision signals can be dynamically generated to meet video understanding task requirements, such as video summarization or video chat.\n\n\n\nRef: \n\n[1] Yang, Antoine, et al. \"Zero-shot video question answering via frozen bidirectional language models.\"  NeurIPS 2022\n\n[2] Yu, Shoubin, et al. \"Self-chained image-language model for video localization and question answering.\"  NeurIPS 2023\n\n[3] Ko, Dohwan, et al. \"Large language models are temporal and causal reasoners for video question answering.\" EMNLP 2023\n\n[4] Papalampidi, Pinelopi, et al. \"A simple recipe for contrastively pre-training video-first encoders beyond 16 frames.\" CVPR 2024\n\n[5] Balažević, Ivana, et al. \"Memory consolidation enables long-context video understanding.\" ICML 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response:

We greatly appreciate the reviewers' careful reading and insightful feedback on our work. It's encouraging to receive comments recognizing our work's **novel idea, good motivation, thorough experiments and promising results**. We have respond the concerns raised and are open to further discussions.

We received many valuable suggestions to further improve our paper, and we'd like to share some revision plans here.

(1) **Including and discussing related papers (Reviewer kuEy).** Video understanding is a rapidly evolving field, with numerous new works emerging, such as video-agent approaches. Although we have tried to make comprehensive comparisons with various methods in our submitted version, Reviewer kuEy pointed out some contemporaneous work that we overlooked. We appreciate this feedback and will further discuss and incorporate these references to provide a more comprehensive comparison.

(2) **Providing more analysis on Tideos (Reviewers nFok and oFSV).** We'll include the visualizations to illustrate the diversity of Tideos and the impact of the prompts used in Tideo generation.

(3) **Improving the presentation on pretraining/finetuning/inference (Reviewers nFok and oFSV).**  We will provide a clearer overview of the high-level steps, including text-only pre-alignment, finetuning, and various zero-shot inference strategies, and offer more details.





# Brief Introduction of the PDF:

(1) **Table 1 (suggested by Reviewer nFok):** An ablation study shows that multi-choice Tideo-QA and Tideo Summarization tasks enhance the performance. For further results and detailed analysis of the multi-choice QA tasks, please see Appendix A.2.

(2) **Table 2 (suggested by Reviewer nFok):** We add an experiment to study inference with textual features of videos. We used LaViLa-xl [1] as the video captioner. Results in Table 2 reveal that both project-based and caption-based inference approaches help to mitigate the domain gap issue. Using textual features derived from video captions achieves better results, benefiting from clip-level captions. This demonstrates the flexibility of TOPA and its applicability to various scenarios.

(3) **Figure 1, Figure 2, and Table 3 (suggested by Reviewer nFok and oFSV):** Visualizations of diversity of Tideos. (a) Figure 1 shows that Tideos generated under different conditions have different distributions, with Tideos-WordNet being the most diverse and widely dispersed. (b) Figure 2 illustrates that Tideos-Howto100m and Tideos-Ego4D focus primarily on human-centric activities, while Tideos-WebVid and Tideos-WordNet cover more scenarios. (c) Table 3 compares vocabulary sizes, revealing that Tideos-WordNet encompasses more objects.

Meta-review: The paper initially received mixed scores, but during the AC-reviewers discussion phase, the reviewers reached a consensus that the work holds significant value due to its novelty, effectiveness, and potential impact. However, it is important to note that all three reviewers agree that incorporating the missing works identified by Reviewer kuEy would enhance the paper. Therefore, the AC also strongly recommends that the authors include these references in the camera-ready version.
Decision: Accept (spotlight)