TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment

Authors: wei  li
hehe  fan
yongkang  wong
mohan  kankanhalli
yi  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:20:16

Abstract: Recent advancements in image understanding have benefited from the extensive use of web image-text pairs. However, video understanding remains a challenge despite the availability of substantial web video-text data. This difficulty primarily arises from the inherent complexity of videos and the inefficient language supervision in recent web-collected video-text datasets. In this paper, we introduce Text-Only Pre-Alignment (TOPA), a novel approach to extend large language models (LLMs) for video understanding, without the need for pre-training on real video data. Specifically, we first employ an advanced LLM to automatically generate Textual Videos comprising continuous textual frames, along with corresponding annotations to simulate real video-text data. Then, these annotated textual videos are used to pre-align a language-only LLM with the video modality. To bridge the gap between textual and real videos, we employ the CLIP model as the feature extractor to align image and text modalities. During text-only pre-alignment, the continuous textual frames, encoded as a sequence of CLIP text features, are analogous to continuous CLIP image features, thus aligning the LLM with real video representation. Extensive experiments, including zero-shot evaluation and finetuning on various video understanding tasks, demonstrate that TOPA is an effective and efficient framework for aligning video content with LLMs. In particular, without training on any video data, the TOPA-Llama2-13B model achieves a Top-1 accuracy of 51.0% on the challenging long-form video understanding benchmark, Egoschema. This performance surpasses previous video-text pre-training approaches and proves competitive with recent GPT-3.5 based video agents.

Main paper: 
 introduction: 
 Image-language understanding has made large advancements in both image-language alignment [26, 48] and Multimodal Large Language Models (MLLMs) [1, 27, 34, 89] , aided by pre-training on large-scale noise-paired image-text data collected from the web [6, 19, 51, 53, 52] . This raises a question: Can we mirror this success in video-language understanding? Research [45, 64, 77, 86] has explored pretraining video-language models on millions of web video-text data [3, 40, 65] , achieving promising results in basic video tasks such as video-text retrieval, video captioning, and video question answering across conventional video benchmarks. However, recent research reveals that these models struggle with a challenging long-form video understanding benchmark, i.e., EgoSchema [39] , which requires intrinsic temporal understanding capabilities. This highlights the gap in adapting web video-text pretrained models to more comprehensive video understanding tasks. *Hehe Fan is the corresponding author.38th Conference on Neural Information Processing Systems (NeurIPS 2024).We attribute this gap to two primary factors: 1) The intrinsic complexity of the video modality. Videos introduce intrinsic complexities in both spatial and temporal dimensions, which are not present in static images. These complexities require extensive training on larger-scale data to effectively capture video dynamics. Furthermore, representing videos typically involves processing multiple frames, significantly increasing computational demands compared to image modeling. The dual challenges of large-scale training and increased computational requirements make video-language modeling particularly challenging. 2) The limitations of web language supervision. The language supervision in recent web video-text datasets primarily comes from subtitles or descriptions associated with the videos [3, 40] . However, subtitles often suffer from the issues of visual-textual misalignment [33, 17] . Moreover, the form of descriptive supervision is inefficient in building robust video reasoning capabilities, especially in terms of temporal reasoning. This mismatch between the complex video content and the limited supervision hinders effective video-language modeling.In this paper, we propose an innovative approach to develop video understanding capabilities by using LLMs to simulate and understand video dynamics. Instead of directly aligning LLMs with real video representation, we first introduce a textual video representation -a sequence of textual frames designed to mimic real visual dynamics. This textual video can be readily generated by advanced LLMs and effectively simulates various video dynamics by describing them in text. Specifically, we present a Textual Video (TextVid) dataset, automatically generated by LLMs. TextVid includes: 1) Textual videos (hereinafter referred to as "Tideo"), which consist of a sequence of textual frames crafted to mimic the keyframes of real videos, and 2) Tideo annotations, including comprehensive Tideo-level dense descriptions and varied question-answer (QA) pairs. These annotations are of high quality and closely align with the Tideo content, by virtue of the powerful capability of LLM in language generation. Building on the proposed TextVid dataset, we introduce the Text-Only Pre-Alignment (TOPA) framework, to effectively and efficiently pre-align LLMs with the video modality, reducing the need for costly video-text pre-training. We introduce three tasks for video-LLM pre-alignment: Tideo summarization, Tideo QA and multi-choice Tideo QA. To bridge the gap between textual Tideos and visual videos, we leverage the CLIP [48] model for feature extraction. Specifically, we employ the CLIP text encoder to extract frame-level representations for Tideos, and the CLIP visual encoder for real videos. During the text-only pre-alignment phase, the LLM learns to process continuous CLIP text features of Tideos. In the real video inference phase, it transitions to handling continuous CLIP image features of real video. Due to the aligned CLIP image-text feature space, the LLM can adapt to real video inputs despite being trained on textual representations. Our main contributions include:(1) We propose a novel Text-Only Pre-Alignment (TOPA) framework to extend Large Language Models (LLMs) for video understanding. TOPA aligns LLMs with the video modality efficiently and effectively without the need for training on real videos, reducing the costs for video-text pre-training.(2) We introduce TextVid, a textual video dataset automatically generated by advanced LLMs. TextVid dataset comprises 721K diverse Tideos along with associated high-quality annotations, which include detailed Tideo descriptions and a variety of question-answer pairs.(3) Extensive experiments demonstrate TOPA's effectiveness across various video understanding tasks. Particularly, the TOPA-Llama2-13B model achieves 51.0% Top-1 accuracy in the challenging EgoSchema benchmark, outperforming previous video-text pretraining methods and competitive with recent GPT-3.5-based video agents. interaction with these Tideos, the dataset features a dense description summarizing the Tideo, as well as a set of multiple-choice questions and answers related to the Tideo content. The structure of each element is as follows:Tideo: Sequence of textual frames {T 1 , T 2 , . . . , T n }, 5 ≤ n ≤ 15For each frame T i : Frame caption: C i Object captions: D i,j for main objects in T i Annotations:Global Dense Description of the Tideo: D V Set of Questions-Options-Answers:{(Q k , O k , A k )}There are two major advantages of the TextVid dataset.(1) The large-scale and diverse Tideos. As the dataset is text-only and fully generated by an LLM, the size of TextVid is practically unlimited. Moreover, the Tideos can cover a broad range of domains by simply prompting the language model with appropriate conditions. It is distinctly different from previous web video-text dataset like Howto100M [40] that are limited to specific human-centric instructional videos. In practice, we enhance the diversity of TextVid by randomly sampling video captions from WebVid-2M [3] , video titles from Howto100m [40] , video tasks from Ego4D [15] and object names with descriptions from WordNet [41] as a condition of prompts. These varied prompts enable the language model to generate a diverse dataset.(2) The high-quality, consistent and free-form language supervision. The language supervisions are generated along with Tideos. The advanced capabilities of LLM ensure the quality of these supervisions, making them less noisy than web video-text data. Moreover, both the Tideo and the supervision are in textual format, making the supervision closely aligned with the Tideo's content. Additionally, the format of the language supervision is unrestricted. For example, we prompt the LLM to generate dense descriptions and multi-choice QA pairs as language supervision.NExT-QA [72] Justification: The paper does not include theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
text-only pre-alignment: 
 Preliminary: Video-LLM alignment. The goal of video-LLM alignment is to extend pre-trained LLMs for processing video inputs. Given a video sampled with n frames {I 1 , I 2 , . . . , I n }, Recent work [23, 77] uses a frozen CLIP model to extract the frame-level visual feature, formulated asf v i = E image (I i ), where E image denotes CLIP image encoder. The CLIP features are then projected into the LLM space via a simple linear layer, denoted as G(P (f v 1 ), ..., P (f v n )), where G denotes a language model and P denotes a projection layer that projects the CLIP feature to LLM space. Tideo representation. In this work, we leverage Tideos (c.f. Section 3.1) for video-LLM pre-alignment instead of training on real videos. Specifically, given the textual frame T i , we employ CLIP text encoder to extract the frame representation from frame caption C i and detailed object captions D i , represented asf t i = F fusion (E text (C i ), E text (D i,1 ), ..., E text (D i,j )), where F fusion is a fusion function such as simple average pooling, and E text denotes the CLIP text encoder. A Tideo with n textual frames is represented as V t = {f t 1 , ..., f t n }. Text-only pre-alignment. Given the Tideo T , dense Tideo-level description D V , and QA pairs with multiple choices {(Q k , O k , A k )}, we introduce the following tasks for Tideo-LLM alignment:(1) Tideo Summarization: Given the Tideo, generate a detailed description to summarize the Tideo;(2) Tideo QA: Given the Tideo and question, predict the answer; (3) Multi-choice Tideo QA: Given the Tideo, question and multiple choices, choose the correct answer from the candidates. We employ a unified auto-regressive Language Modeling (LM) objective for these three tasks:EQUATIONwhere V t denotes the Tideo representation, and V t = {f t 1 , ..., f t n } during the text-only training, Z denotes the task specific condition tokens and t i denotes the i th target token. θ G and θ P denote the learnable parameters of the LLM adapter and the projection layer P , respectively. In practice, we use the following format as the LLM input: {Task Instruction}. Video:{f t 1 , ..., f t n }. {Task Conditions}. Answer: {Predict Targets}. For the Tideo summarization task, the target is detailed Tideo descriptions. For Tideo QA task, the target is the answer and the condition is the question. For multi-choice Tideo QA task, the target is the correct option and the condition consists with question and options. The details of the task-specific prompts are included in Appendix F.1.Section 3.2 introduces the text-only pre-alignment using the TextVid dataset. In this section, we detail how to adapt this text-only pre-aligned LLM for real video understanding. We introduce two approaches: one is zero-shot inference, which directly infers with real video data. And the other is supervised finetuning, where the pre-aligned model is further finetuned on downstream video data. Zero-shot inference. During pre-alignment, we leverage the textual representation V t = {f t 1 , ..., f t n } as the Tideo representation. During inference, we take real videos features as input, i.e.,V v = {f v 1 , ..., f v n }, where f v i = E image (I i ).These two modality features f t and f v that come from CLIP image encoder and CLIP text encoder are aligned via CLIP pre-training. This aligned image-text representation makes it possible to perform zero-shot inference without additional finetuning. However, the modality gap phenomenon [16, 30, 31, 44, 85] , i.e., CLIP image feature and CLIP text feature are located in two completely separate regions of the feature space, prevents us from directly taking the visual feature f v as the textual feature f t . To bridge this modality gap, we follow DeCap [30] to employ a support memory to project the CLIP visual feature into the CLIP text feature space. This training-free projection process is formulated as:EQUATIONwhere m i denotes CLIP text feature from a pre-constructed memory of size N , f v denotes input frame feature of real video and f v→t denotes the projected feature. During zero-shot inference, we take the V v→t = {f v→t 1 , ..., f v→t n } as the real video's representation.Supervised finetuning. On the other hand, the text-only pre-alignment can be viewed as a pretraining stage. Following the pretraining-finetuning paradigm, the pre-aligned LLMs can then be fine-tuned on real video data for improved downstream task performance. The finetuning process is similar to the text-only pre-alignment as detailed in Section 3.2, except that the LLM receives a sequence of CLIP visual features as input instead of CLIP textual features.We leverage Llama2-7B, Llama2-13B [57] and Llama3-8B as the LLM backbone. Additionally, we employ the Llama-adapter [84] with an adaptation embedding length of 50. We utilize CLIP-ViT-L as the multimodal encoder. We employ a simple linear layer to project the CLIP feature into the LLM feature space. During training, the CLIP model and LLM backbone are frozen. The projection layer and additional Llama-adapter are trainable. For text-only pre-alignment, we uniformly sample the Tideos into 10 frames. We train the model on a mixture of tasks comprising Tideo summarization, Tideo QA, multi-choice Tideo QA with the ratio of 1:1:2. For zero-shot inference, we construct a memory for cross-modal projection, consisting of 2M CLIP text features sampled from captions in the TextVid dataset. TOPA-Llama2-7B and TOPA-Llama3-8B are trained on four 40G-A100 GPUs in one day. TOPA-Llama2-13B is trained in two days. More training details of TOPA and baselines are included in Appendix E.2. 
experiments: 
 TOPA enables the LLM to perform various video understanding tasks as shown in Figure 2 . In this section, we evaluate TOPA on multi-choice video QA and video captioning tasks. Section 4.1 evaluates TOPA on NeXT-QA [72] , STAR [70] , TVQA [24] , recent challenging EgoSchema [39] and MVBench [29] benchmarks with the zero-shot setting. We further evaluate TOPA on multi-choice video QA with the finetuning setting (Section 4.2) and zero-shot video captioning task (Section 4.3). In Section 4.4, we conduct ablation study on the LLM prior and input video frames. We report Top-1 accuracy on multi-choice video QA benchmarks and CIDEr [58] score on video captioning benchmarks. We mainly compare TOPA with the following categories of video understanding approaches:The video first shows a person holding a bird of prey on her left arm. The bird of prey is perched on her fist. The person then holds the bird of prey up to her face and shows it to the camera. The bird of prey is a white goshawk. It has a black beak and black eyes. The person then shows the bird of prey to the camera again. The bird of prey is perched on her fist and is looking around.Describe the video in detail.The bird is perched on the handler's fist. A close-up of the bird's face is shown. The bird is looking around.What is the final scene of the video?What is the relationship between the woman and the bird in the video?The woman is the owner of the bird.Figure 2 : Examples of TOPA-LLama2-13B for video-language understanding. Given a video, TOPA is able to summarize the video content and answer the questions.(1) Web video pre-training approaches [4, 45, 64, 66, 77] . This line of work aims to develop general video-language models by leveraging extensive web videos, using associated video captions or audio as weak supervision signals.(2) Adapting image MLLMs for video understanding [22, 81, 49] . These approaches aim to extend the image understanding capabilities of recent vision-language models (VLMs) to video understanding. Specifically, SeViLa [81] utilizes BLIP-2 for localizing and understanding key frames of a video. IG-VLM [22] converts video into a composite image by arranging the video frames into a grid layout.(3) LLM-based video agents [42, 59, 60, 63, 82, 13, 21, 67] . This line of work leverages LLMs like GPT-3.5 and GPT-4 as an agent to understand a video by designing and executing a series of actions.The language-only agents perceive visual information via recent foundation VLMs (e.g., CLIP [48] , BLIP-2 [27] , LaViLa [87] and PALI [8] ).(4) Our text-only pre-alignment. Different from the above works, TOPA leverages the proposed TextVid dataset for video-LLM pre-alignment, enabling the LLM to process continuous features. Thus, it can enable performing video understanding tasks.Table 1 shows the results on EgoSchema full set. We compare our method against a range of recent approaches in video understanding. Our proposed text-only pre-alignment framework, despite training without real videos, shows impressive results on the EgoSchema benchmark. TOPA outperforms previous image-based adaptation approach IG-VLM and video agents LLoVi and Vamos with the same scale LLM (Llama2-7B and Llama2-13B). Moreover, TOPA shows consistent improvements when scaled up with a larger LLM backbone, indicating the effectiveness of LLMs in complex video understanding tasks. Discussion 1: The necessity of high-quality language supervision for video understanding.Recent video pre-training approaches like LongViVit [45] and InternVideo [64] , despite training on million-level web video-text data, show inferior performance on EgoSchema evaluation. These results highlight the inefficacy and inefficiency of conventional contrastive pre-training in understanding long-form videos, primarily due to noisy and simplistic language supervision. In contrast, our TOPA, trained on 721K Tideoswith high-quality language supervision, shows impressive results on EgoSchema. It indicates that, unlike image understanding which significantly benefits from leveraging web language as supervision, video understanding may require more precise and accurate language supervision to better capture the complex visual dynamics.Table 1 : Zero-shot results on EgoSchema [39] full set. Methods that leverage closed-source LLMs are marked in gray. † denotes the model is trained with in-domain egocentric videos from Ego4D [15] . * denotes results on EgoSchema subset. Results of InternVideo and FrozenBiLM are sourced from [39] . Results of SeViLA are sourced from [45] .Acc@1 Human Eval [39] Video-LLava [32] GPT-4 60.2(Our Text-Only Pre-Alignment) TOPA CLIP-ViT-L Llama2-7B 41.2 TOPA CLIP-ViT-L Llama3-8B 44.2 TOPA CLIP-ViT-L Llama2-13B 51.0Discussion 2: Video agents versus end-to-end video-LLM modeling. Video agents have shown impressive results on the EgoSchema benchmark, aided by advanced LLMs and VLMs. However, a significant limitation of these approaches is their heavy reliance on the powerful LLMs. For example, the accuracy of Vamos drops by -11.6% when the GPT-4 is replaced with Llama2-13B, largely falling behind the performance of the TOPA-Llama2-13B model. The reliance on powerful closed-source LLMs restricts its application fields and introduces external overheads. Moreover, video agents make decisions based on the language format clues collected by VLMs. Converting the video content into language clues may lead to a limited upper bound compared to end-to-end modeling. Additionally, the inference speed of these approaches is another concern, since it involves multiple interactions with both VLMs and LLMs. In contrast, end-to-end video-LLM models, which condense the video into a sequence of embeddings as the input of LLM, are more efficient.Table 2 shows the multi-choice video QA results across various benchmarks. TOPA achieves impressive performance on the TVQA and EgoSchema benchmarks, significantly outperforming previous video pre-training models and image-to-video adaptation approaches. This indicates that our TOPA framework effectively enables LLMs to handle video input, despite not being pre-trained on real videos. However, for the NeXT-QA and STAR benchmarks, TOPA underperforms compared to SeViLA and IG-VLM. A major reason is that these benchmarks involve many fine-grained visual questions, including those about object locations and relationships. SeViLA and IG-VLM, benefiting from the advanced image-understanding capabilities of pre-trained VLMs such as LLaVA, excel in answering these fine-grained visual questions. In contrast, our TOPA framework primarily focuses on high-level semantic alignment. Moreover, during zero-shot inference, we project the visual features into the text feature space to bridge the modality gap, as described in Eq. 2. This cross-modal semantic projection process tends to overlook fine-grained visual details, such as object locations, which leads to inferior performance on the STAR benchmark. We provide extensive qualitative results to illustrate TOPA's advantages and limitations across various video understanding tasks in Appendix A.3. MVBench [29] is a recent video-language understanding benchmark that covers 20 challenging video tasks, regrouped from existing video-language benchmarks. Table 3 shows the results. TOPA demonstrates impressive results compared to previous image MLLM and video MLLM. It excels particularly in tasks requiring high-level video-language understanding, such as Scene Transition (ST), Episodic Reasoning (ER), and Unexpected Action (UA). TOPA Surprisingly excels in the Action Localization (AL) task, which requires identifying the moment an action occurs. This indicates that the text-only pre-alignment enables the LLM to understand temporal visual sequences. However, TOPA struggles with tasks that demand fine-grained visual understanding, such as Moving Direction (MR), Action Antonym (AA), and Object Shuffle (OS). A common challenge in these tasks is the requirement for detailed visual understanding. For example, Action Antonym involves identifying the direction of an action, while Object Shuffle involves locating objects. TOPA struggles in these fine-grained visual tasks since it is trained with CLIP text features. The modality gap between CLIP text features and image features hinders TOPA from capturing visual details. Further video instruction tuning might address this limitation, which we leave for future work. We provide qualitative results in Appendix A.3 to illustrate TOPA's advantages and limitations on various video understanding tasks.Table 3 : Evaluation results on MVBench. The results of other approaches are sourced from [29] . We gray out the results of VideoChat2 since it utilizes extensive annotated downstream video data. [70] STAR [70] PAXION [68] MiT V1 [43] FunQA [73] CLEVRER [80] STAR [70] Perception Test [46] CLEVRER [80] Charades-STA [ [18] Perception Test [46] CLEVRER [80] CLEVRER [80] Perception Test [46] NTU RGB+D [35] Perception Test [46] VLN-CE [69] TVQA [24] CLEVRER [80] In this section, we further finetune the pre-aligned TOPA models to study the benefits of TOPA for downstream supervised learning. During finetuning, TOPA directly takes the video feature as input without the cross-modal projection. More finetuning details for each dataset are provided in Appendix E.2. Table 4 shows the finetuning results on multi-choice video QA dataset. For comparison, we include baseline models without text-only pretraining. Our text-only pre-alignment consistently improves the performance across three benchmarks. Notably, TOPA-Llama2-7B achieves 67.1% accuracy on TVQA, outperforming other approaches by a large margin. These results suggest that our text-only pre-alignment, even without training on real videos, has a similar effect to conventional video-language pre-training. Results on zero-shot video captioning. We further perform zero-shot video captioning on MSR-VTT [75] and VATEX [62] . As shown in Table 5 , TOPA largely outperforms previous text-only approaches like Decap which is trained on captions sourced from CC3M [53] . TOPA even outperforms the video-text pre-training approaches like VideoCoCa, which is pre-trained on millions of videos-text data, demonstrating that TOPA is an efficient and effective framework for video-LLM alignment. LLM prior in video-language understanding. To investigate the impact of LLM prior in multichoice video QA, we conduct experiments on EgoSchema with the blind setting, where only the questions and choices are provided to the LLM. Table 6 shows the results. Bard and GPT-4-Turbo achieve 33.2% and 30.8% accuracy, respectively. Gemini-Pro-1.0 reaches 38.2% accuracy. These blind results of advanced LLMs suggest that in some video QA cases, LLMs can accurately choose the correct answer solely based on the question and choices, without visual input. However, the blind performance of Llama2-7B and Llama2-13B is inferior, potentially due to their smaller model size.After training on the TextVid dataset, TOPA-Llama2-13B achieves a blind accuracy of 37.5% (or +11.7%), closely approaching that of Gemini-Pro-1.0 model. These results suggest that text-only pre-alignment can effectively prepare LLMs for downstream video-language tasks by leveraging specialized text-only tasks, even in complex scenarios where the original LLMs are limited. The impact of video frames. To better investigate TOPA's capability in understanding temporal dynamics of real videos, we conduct experiments with different number of frames. Table 7 shows the results. Multiple frames input consistently enhances performance on NeXT-QA and EgoSchema for both TOPA-Llama2-7B and TOPA-Llama2-13B. This indicates that the text-only pre-alignment effectively enables the LLM to handle multiple video frames, despite not being trained on real videos.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The training details of TOPA are included in Appendix E.2. The dataset generation pipeline is detailed in Appendix D. 
conclusions: 
 In this paper, we introduce TOPA, a text-only pre-alignment framework designed for aligning LLMs with video modality without requiring training on real videos. TOPA has demonstrated remarkable performance on the recent, challenging long-form video understanding benchmark, i.e., EgoSchema, showcasing that a text-only approach is effective in capturing the dynamics of long-form videos. Our approach, which includes data generation and text-only pre-alignment, has potential applications across various vision-language tasks where obtaining paired vision-language data is difficult.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: The code and dataset is publicly accessible (https://github.com/dhg-wei/ TOPA). 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: TOPA is a large-scale pre-training framework. Similar to previous related work [45, 81, 29] , error bars are not reported because it would be computationally too expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our model and data focus on video-language understanding, with minimal risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The licenses are mentioned in Appendix G.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 Further Discussion on Multi-Choice Video QA Task A significant advantage of the text-only framework is that we can utilize the LLM to automatically generate diverse language-based supervisions as needed, such as the multi-choice QA pairs. To explore the impact of the multi-choice QA training tasks, we conduct an ablation study as shown in Table 8 . We would like to first introduce the different evaluation modes for multi-choice video QA tasks: (1) LLM Selection: Asking the LLM to predict the correct answer given the video-question-choices.(2) LLM Logits: Given the video and question as LLM context, we calculate the logits for each choice by averaging the logits of all words within the choice. The choice with higher logit tends to match the video-question context better and is thus selected as the predicted answer. (3) Similarity Comparison [45, 4] : Mapping the multiple question-choice pairs and video to a common feature space and calculating the similarity between the video and each question-choice.The performance gap between the EgoSchema subset and full set. Previous work [4, 45] highlights a huge performance gap between the subset and the full set of EgoSchema as shown in Table 8 . While concurrent work [21, 49] introduces log-likelihood based approaches for LLM inference, which significantly improve the performance on EgoSchema subset, the issue of the performance gap still persists. In this paper, we observe that such a performance gap phenomenon also occurs in approaches based on LLM logits. However, but it diminishes or even disappears in methods employing LLM selection. We find that this phenomenon may be attributed to differences in the linguistic structures of the choices, as shown below. The choices in the subset often differ in several key works like "create", "repair" and "clean". The similarity or logit can effectively identify this keyword-level difference to select a more appropriate choice. Conversely, the choices in the full set display more substantial linguistic differences. These variations introduce significant language biases, i.e., some sentences naturally receive higher logits in LLM, complicating the reliance on similarity or logit for choice selection. In contrast, LLM selection methods take all the choices within the context, allowing the LLM to leverage its robust contextual understanding to select the correct choice. LLM for multi-choice QA. In Table 8 , we observe a notable phenomenon where the TOPA models achieve impressive results on the subset with the logits evaluation mode. TOPA-LLama2-13B achieves 67.5% top1 accuracy, surpassing GPT-4-based video agents. However, when evaluated with the multi-choice selection mode, the performance of the subset declines to 51.2%, but the performance of the full set increases from 41.6% to 51.0%. These results suggest that while the LLM is capable of selecting the answer from multiple choices, it is less sensitive to the keywords within those choices.In contrast, the logit-based approach is sensitive to the keywords but has difficulty with complex sentence understanding.The impact of the Multi-Choice Tideo QA pre-training. In Table 8 , we report the results of TOPA without the multi-choice Tideo QA task, i.e., trained with Tideo summarization and Tideo QA tasks.In this case, we find that TOPA-LLama2-7B maintains similar performance when evaluated with the logit mode. However, there is a significant performance drop when evaluated with the multi-choice selection mode. This result suggests that while the LLM is adapted to process video inputs, its capability is somewhat constrained and can not extend to more complex video-language tasks beyond the pre-training tasks. This finding highlights the advantage of our text-only data generation and text-only pre-alignment framework, which enable us to develop a variety of pre-alignment tasks to better equip the LLM for general video-language tasks such as dense captioning, multi-choice video QA, and video chat.TOPA is pretrained with CLIP text features while inferenced with CLIP image features. We employ a modality projection approach, i.e., Eq. 2, to bridge this CLIP modality gap during zero-shot inference. Table 9 shows the impact of Eq. 2. TOPA shows inferior results when directly taking the visual feature as input due to the modality gap problem. The projection approach effectively alleviates such a modality gap problem without additional training. We present qualitative results to illustrate the capabilities and limitations of TOPA across various video understanding tasks. Figure 4 shows qualitative results on the NExT-QA validation set. Figure 5 shows qualitative results on the EgoSchema subset. Modality gap in CLIP. Despite the fact that TOPA achieves impressive results, a significant limitation in TOPA is the gap between the CLIP text feature and CLIP image feature. On the one hand, we use the CLIP text feature for pre-alignment, while inference is with the CLIP visual feature. The modality gap makes the performance degrade, despite employing a modality projection mechanism to mitigate it. On the other hand, the CLIP text features cannot fully capture the fine-grained visual details present in actual images, such as object locations and relationships. This limitation causes TOPA to struggle in scenarios where questions involve detailed visual information, as shown in Appendix A. 3 .Struggles in fine-grained visual understanding. In TOPA, we propose textual videos to mimic real videos. However, this approach primarily focuses on keyframes understanding, which is insufficient for scenarios requiring the model to process hundreds of frames at high fps, such as action counting tasks. Besides, for the fine-grained action understanding scenarios, TOPA is unable to capture the fine-grained visual information. For example, in a scene where a person climbs a ladder, it is difficult for TOPA to identify whether the person is going up or down due to the limited capability to capture detailed visual dynamics. Further enhancing TOPA with video instruction tuning might address these limitations which we leave for future work.Academic Impact. TOPA's methodology, which frees the need for costly video-text data collection and large-scale pre-training, lowers the barriers to entry for research and development in video-language understanding technologies. The text-only learning framework of TOPA may inspire researchers with limited resources to engage in cutting-edge multi-modal research, providing a more diverse range of perspectives and contributions to this field. Social Impact. The ultimate objective of TOPA is to develop a general video-language understanding model. Its primary application enables users to extract information from long-form videos without the need for detailed viewing. Moreover, these capabilities for interpreting and managing video content could significantly enhance content moderation systems. Platforms hosting user-generated content could employ sophisticated video-language models to efficiently detect and mitigate the effects of inappropriate or harmful video content.We utilize Gemini Pro 1.0 API for our data generation process. We prompt the LLM to create textual videos along with associated annotations. To ensure a diverse dataset that covers a wide of domains, we add condition prompts including different themes, video captions, video events, and the names of main objects. Specifically, we leverage video titles from Howto100M [40] , video captions from WebVid2M [3] , video events from Ego4D [15] , and object from Wordnet [41] as conditions to generate diverse textual videos. For Ego4D condition, we ask the LLM to mimic an ego-centric video to further improve the diversity of the dataset. Table 10 compares vocabulary sizes. Figure 11 shows that Tideos generated under different conditions have different distributions. For each data generation, we prompt the LLM with the task prompt and one of the condition prompts as shown in Figure 13 . The statistics TextVid are shown in Table 10 . Additionally, we provide Wordcloud of TextVid in Figure 10 . The examples of TextVid are shown in Appendix H. EgoSchema [39] is a challenging long-form video understanding benchmark with 5000 multi-choice questions. The videos in EgoSchema are sourced from Ego-4D [15] , with an average length of 3 minutes, distinct from previous benchmarks that focused on shorter, seconds-long videos. The questions in EgoSchema are manually curated to demand long temporal reasoning. We report results on EgoSchema full set. NExT-QA [72] is a multi-choice video QA benchmark for causal and temporal reasoning, including 5,440 natural videos. The average length of video is 44 seconds. We report results on NExT-QA validation set, which contains 570 videos and 5,000 multiple-choice questions. STAR [70] is a benchmark for situated reasoning. It contains 22K video clips with an average length of 44 seconds. There are 4 different question types in STAR: Interaction (Int.), Sequence (Seq.), Prediction (Pre.), and Feasibility (Fea.). We report results on STAR validation set. TVQA [24] is a benchmark containing 21k video clips with an average length of 76 seconds. We report results on TVQA validation set without subtitles.MVbench [29] is a reorganized benchmark containing 20 video understanding tasks. These tasks are sourced from STAR [70] , PAXION [68] , MiT Vi [43] , FunQA [73] , Perception Test [46] , Charades-STA [14] , MovieNet [18] , NTU RGB+D [35] , VLN-CE [69] and TVQA [24] .We leverage Llama2-7B, Llama2-13B [57] and Llama3-8B as the LLM backbone. Additionally, we employ the Llama-adapter [84] with an adaptation embedding length of 50 for efficient finetuning. We utilize CLIP-ViT-L as the multimodal encoder. We employ a simple linear layer to project the CLIP feature into the LLM feature space. The CLIP model and LLM backbone are frozen. The projection layer and additional Llama-adapter are trainable. For text-only pre-alignment, we uniformly sample the Tideos into 10 frames. We train the model on a mixture of tasks comprising Tideo summarization, Tideo QA, multi-choice Tideo QA with the ratio of 1:1:2. TOPA-Llama2-7B and TOPA-Llama3-8B are trained on four 40G-A100 GPUs in one day. TOPA-Llama2-13B is trained in two days. For zero-shot inference, we construct a memory for cross-modal projection, consisting of 2M CLIP text features sampled from the frame captions in the TextVid dataset. We include the training details in Table 12 . The actual learning rate is calculated by base lr * Effective Batchsize/256. Please assist in creating a textual dataset that simulates a video with 5-15 frame-level descriptions. The output should be structured in JSON format.For each frame, you should describe it as following:Frame Caption: Start with a 'Frame Caption' for each frame, providing a comprehensive overview. Detailed Captions: Follow the frame caption with 1-3 detailed captions focusing on specific objects within the frame.Each detailed caption should highlight the main object at the beginning.After describing the frames, provide:Global Video Description: Provide a summary that synthesizes information from all described frames, offering an overarching narrative of the video.A set of question-answer pairs related to the video. Questions can vary, including "when," "why," "what,""how," "does/do," "is/are," and "where." Emphasis should be on visual reasoning. Each question will come with five options, including one correct answer. Questions that require integration of multiple frames to deduce the answer are preferred.Output should structured as following:``{ "Video_Title": " ", "Frames": [ { "Frame Caption": " ", "Object Name": " ", "Object Name": " ", "Object Name": " " }, ... ], "Global_Video_Description": " ", "QA Pairs": [ { "Question": " ", "Options": { "A": " ", "B": " ", "C": " ", "D": " ", "E": " " }, "Answer": "D" }, . Now, please generate a textual video about {caption}.Now, please generate an ego-centric video about {scenarios}. This video is taken by {name}.Now, please generate a textual video about {object name}, {object description}. The code, model and proposed dataset will be publicly accessible. We use standard licenses from the community. We include the following licenses for the codes, datasets and models we used in this paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 


Decision: The paper initially received mixed scores, but during the AC-reviewers discussion phase, the reviewers reached a consensus that the work holds significant value due to its novelty, effectiveness, and potential impact. However, it is important to note that all three reviewers agree that incorporating the missing works identified by Reviewer kuEy would enhance the paper. Therefore, the AC also strongly recommends that the authors include these references in the camera-ready version.

Reviews: review: paper_summary: paper_summary: The authors introduce Text-Only Pre-Alignment (TOPA), a novel approach that extends large language models (LLMs) for video understanding without pre-training on real video data. TOPA generates Textual Videos, comprising continuous textual frames and annotations to simulate video-text data, and uses these to pre-align a language-only LLM with the video modality. The CLIP model aligns image and text modalities, bridging the gap between textual and real videos. TOPA encodes continuous textual frames as CLIP text features, analogous to CLIP image features, thus aligning the LLM with real video representations.
strength_weakness: strength_weakness: In Table 1, the author did not include key papers such as LifelongMemory[1], Video-Agent: A Memory-Augmented Multimodal Agent for Video Understanding[2], LangRepo[3], and MVU[4]. LifelongMemory achieves 68% and 62.1% on the subset and the full set of EgoSchema, respectively, and should be included in the table to allow readers and reviewers to understand the relative performance, considering the size of the LLM. Additionally, LangRepo and MVU use open-source LLMs (Mixtral7B, Mixtral-8×7B, Llama-2-7b-Chat, Gemma-7b-IT, and Mistral-7B-Instruct) and should also be included to provide a comprehensive comparison. Furthermore, the table is missing the performance on the subset, whereas other papers report performance on both the subset and the full set. Including this information is crucial to observe how accuracy changes from the subset to the full set. Similarly, in Table 2, the author did not include key papers such as LangRepo, MVU, VideoAgent, and LLoVi, making it difficult for readers and reviewers to compare the proposed models with existing works.

https://arxiv.org/abs/2312.05269v1
[1] Ying Wang, Yanlai Yang, and Mengye Ren. Lifelongmemory: Leveraging llms for answering queries in long-form egocentric videos, 2023. 

https://arxiv.org/abs/2403.11481
[2] Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Video-agent: A memory-augmented multimodal agent for video understanding, 2024

https://arxiv.org/abs/2403.14622
[3] Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo. Language repository for long video understanding, 2024.

https://arxiv.org/abs/2403.16998v1
[4] Kanchana Ranasinghe, Xiang Li, Kumara Kahatapitiya, Michael S. Ryoo. Understanding Long Videos in One Multimodal Language Model Pass, 2024
questions: questions: 1. Can you add the performance of the proposed models on the subset of EgoSchema?
2. Can you record the inference time of your proposed method on the entire dataset and compare it with the inference times of existing works, including those listed in this review? There is a concern that the visual-to-text feature conversion in zero-shot mode may slow down the entire model.
3. Can you attempt to use the largest open-source models and show the zero-shot performance on the EgoSchema dataset? It is important to assess whether the proposed model can scale up to larger LLMs and if its performance can be comparable to existing works using closed-source LLMs.

I will reevaluate the proposed model based on the answers to the questions.
limitations: limitations: I mentioned some concerns about the work in the weakness section.
paper_summary: paper_summary: The paper presents Text-Only Pre-Alignment (TOPA), a method to extend LLMs for video understanding without training on real video data. TOPA generates "Textual Videos" using LLMs to simulate real video data, then pre-aligns the LLMs with video modalities using the CLIP model for feature extraction. This approach achieves impressive results across various benchmarks, surpassing previous methods and competing well with recent video understanding models.
strength_weakness: strength_weakness: 1. L42: The authors describe subtitles as "frame-level descriptions," but subtitles are actually spoken speech extracted from videos and have intrinsic context and alignment issues [HowTo100M].
2. The description of the training process and stages is confusing. It needs to be clearer what is trained when and how. The subsection "Video-LLM alignment" is confusing given the title "text-only pre-alignment." The presentation would benefit from an overview of high-level steps for training/inference, followed by detailed descriptions. The dataset section lacks references to the appendix for exact prompt details.

**Minor:**

- The references should be clickable.
- Correct “Egoschema” to “EgoSchema.”
- The statements “1. Intrinsic complexity of the video modality” and “2. The limitations of web language supervision” should include references to support these claims.
questions: questions: **Questions:**

1. Have you considered using combined pretraining with both Tideo and multi-modal data?
2. Does the Video-LLM alignment include temporal aggregation of frames?
3. Could you provide ablation studies on how the prompts and different parts of the dataset affect downstream performance, such as the inclusion of condition prompts, the influence of global descriptions, and various QA pairs? It would also be interesting to see how complex the Tideos are and the consistency of frame descriptions. Could the proposed adaptation of LLMs be trained using only text data from available video datasets?
4. Have you considered creating textual descriptions of frames from the original videos (for evaluation) and comparing zero-shot evaluation text vs. videos?

Main Concern:
The quality of the generated video descriptions (Question 3).
limitations: limitations: The limitations are discussed and cover the main disadvantages of the current model.
paper_summary: paper_summary: This paper introduces Text-Only Pre-Alignment (TOPA), a framework designed to extend LLMs for video understanding, without the need for real video data. TOPA leverages textual videos (Tideo), generated by capable LLMs, to mimic real videos. Aided by CLIP's aligned cross-modal space, this  framework despite training on text-only data,  can effectively handle real video input. The effectiveness of TOPA is demonstrated through extensive experiments on video understanding benchmarks.
strength_weakness: strength_weakness: 1. The paper could benefit from providing a more detailed comparison between Tideo and real video. For instance, the TextVid dataset likely covers more diverse domains, due to the use of varied prompts during Tideo generation. It could be analyzed in depth.

2. The proposed Tideos are primarily key-frame level representations, which may not fully capture the temporal dynamics of real videos.

3. The experiment results on MVBench should be incorporated into Section 4.1.2 for improved understanding of TOPA. The results on MVBench clearly demonstrate TOPA's strengths and limitations.
questions: questions: 1. The proposed approach regards video as a few key frames. Can TOPA handle scenarios with more frames?

2. Could you provide more details about the finetuning stage? Are both the projector and adapter optimized during finetuning?

3. How scalable is the TOPA framework when applied to larger datasets or more complex video understanding tasks?
limitations: limitations: As stated by the authors, the text-only learning framework is restricted by the imperfectly aligned CLIP model. The projected visual representations are limited in capturing fine-grained visual details.

Comments: title: Regarding the use of Ego4D "Scenarios"
Clarification3: Performance on EgoSchema fullset and subset
Clarification2: Comparison with video agents approaches
Clarification1: Our approach——Text-only Pre-alignment
Benchmark Comparison and Training Datasets
Looking Forward to Further Discussion
comment: Thank you for your feedback.

We are currently conducting an ablation study on the use of "scenarios" prompts from Ego4D. The current results are as follows:

|                 | Epoch | Training data                                 | EgoScheme SubSet (500) | EgoSchema Fullset (5031) |
| --------------- | :---: | :-------------------------------------------- | :--------------: | :---------------: |
| TOPA-LLama2-13B |  18   | All Tideos (721K)                             |       51.2       |       51.0        |
| TOPA-LLama2-13B |   8   | Tideos excluding the Ego4D "scenairos" (516K) |       48.4       |       51.3        |

These results indicate that Tideos-Ego4D does not significantly impact performance on the full EgoSchema dataset, although there is some effect on the subset. We believe this outcome demonstrates that the use of high-level "scenarios" does not lead to special information leakage within the EgoSchema benchmark.

In fact, many of the "scenarios" in the Ego4D metadata represent common human activities. Considering the scale and diversity of Tideos, we believe that "scenarios" in Ego4D are well-covered by the other three types of Tideos, as shown in the PDF. Thus, excluding Tideo-Ego4D does not substantially affect performance on EgoSchema.

Furthermore, we disagree with the notion that our use of high-level "scenarios" is not a zero-shot approach, while the use of same-domain videos and narrations (used in captioner of LLoVi and VideoAgent) is considered a zero-shot approach. From any perspective, the information contained in high-level, common "scenarios" is far less than that provided by videos and narrations from the same dataset.

In our final version, we will clearly state our position on the zero-shot setting and specify the exact data we used.

## **About "Scenarios" in Ego4D meta:**

Some "Scenarios" in Ego4D meta (Without seletion): ```"Watching tv","Cleaning / laundry","Potting plants (indoor),","Scooter mechanic","Eating","Walking on street","Working out outside","Playing with pets","Bike mechanic","Doing yardwork / shoveling snow","Eating","Farmer","Baker","Cleaning / laundry","Crafting/knitting/sewing/drawing/painting","jobs related to construction/renovation company","Carpenter"```

We use these common scenarios as prompts to generate Tideo, and we don't believe these common scenarios  would involve the leakage of specific video information.

Additionally, most of "Scenarios" are repeated multiple times in Ego4D meta. 

For examples, ```Carpenter: 332, Eating:497, Cleaning:1068, Playing with pets:105, Watching tv:154```

It means that even if we exclude all the video information included in EgoSchema, we can still obtain a similar 'Scenarios' corpus for Tideo generation.
Thank you once again for the detailed response. The captioners used by VideoAgent and LLoVi were pre-trained on Ego4D while excluding the EgoSchema videos. However, I remain concerned that TOPA was trained using "scenarios" from the Ego4D metadata, which includes EgoSchema scenarios. It would be clearer to claim zero-shot if the authors could demonstrate the performance of TOPA when trained on Ego4D excluding the EgoSchema scenarios. 

I will raise the score to weak reject, as the authors have addressed some concerns regarding performance against VideoAgent.
# Performance on EgoSchema fullset and subset.

> The authors also need to address why the accuracy improves when evaluating from the EgoSchema subset to the fullest, while all existing models have shown a performance drop, a pattern generally observed in other datasets. This raises the question of whether this improvement is because TOPA was trained on Ego4D and it is not a zero-shot performance.

### **(6) EgoSchema subset (500) is a part of EgoShema fullset (5031).**

EgoSchema is focused entirely on evaluation: the hidden full set (test set) consists of 5031 videos via an evaluation server, of which 500 were publicly released for validation. **Therefore, we believe that reporting and comparing results on the full set is valid and convinced, as presented in the EgoSchema paper [8].**

### **(7) We don't leverage Ego4D videos or data for training.**

TOPA don't train on Ego4D videos or narrations. We only use "scenarios" from Ego4D meta data like ```"Watching tv","Cleaning / laundry","Potting plants (indoor),","Scooter mechanic","Eating","Walking on street","Working out outside","Playing with pets"``` as prompts for Tideo generation.

In contrast, most video agents, such as VideoAgent [1], LLoVi [3], and LangRepo [4] use LaViLa[7] as the video captioner, which is **pre-trained on video-narration pairs from Ego4D**. MC-ViT-L [6] finetuned their model on Ego4D.

### **(8) The performance gap largely related to evaluation approach.**

As shown in Appendix A.2, we found that most approaches, which evaluate with logits and video-text similarity, share a huge performance gap between subset and full set. **TOPA, when evaluated using the logits method, also experiences this common performance drop.** While for LLM selection approaches, like VideoAgent [1] and TOPA-LLama2-13B, exhibit a smaller gap or no gap. In Appendix A.2, we suggest that this discrepancy may be due to differences in the linguistic structure of the choices.

**Why does TOPA-LLama2-13B show no performance gap?** TOPA uses different training strategies, different inference strategies, different training data and different LLM backbone, compared to other approaches. It's challenging to figure out "why our model performs consistently". Besides, the performance gap phenomenon and evaluation approach is not the focus of this paper. Instead, it would be more appropriate for other approaches, especially those focus on evaluation methods, to investigate why they underperform on the EgoSchema full set.




|                                        | Eval mode         | EgoSchema Subset (500) | EgoSchema Fullset (5031) |  Gap  |
| -------------------------------------- | ----------------- | :--------------------: | :----------------------: | :---: |
| LongViViT [5]                          | **Similarity**    |          56.8          |           33.3           | -23.5 |
| MC-ViT-L [6]                           | **Similarity**    |          62.6          |           44.0           | -18.6 |
| LangRepo-Mixtral-8×7B-(12B active) [4] | **LLM logits**    |          66.2          |           41.2           | -25.0 |
| **TOPA-LLama2-13B**                    | **LLM logits**    |          67.5          |           41.6           | -25.9 |
| VideoAgent (GPT-4) [1]                 | **LLM selection** |          60.2          |           54.1           | -6.1  |
| **TOPA-LLama2-13B**                    | **LLM selection** |          51.2          |           51.0           | -0.2  |

Ref:

[1] **(ECCV 2024)** Wang, Xiaohan, et al. "Videoagent: Long-form video understanding with large language model as agent."

[2] **(CVPR 2024)** Min, Juhong, et al. "MoReVQA: Exploring Modular Reasoning Models for Video Question Answering."

[3] **(arXiv 2023.12)** Zhang, Ce, et al. "LLoVi: A simple llm framework for long-range video question-answering."

[4] **(arXiv 2024.3)** Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo. Language repository for long video understanding, 2024.

[5] **(CVPR 2024)** Papalampidi, Pinelopi, et al. "A simple recipe for contrastively pre-training video-first encoders beyond 16 frames."

[6] **(ICML 2024)** Balažević, Ivana, et al. "Memory consolidation enables long-context video understanding."

[7] **(CVPR 2023)** Zhao, Yue, et al. "Learning video representations from large language models." 

[8] **(NeurIPS)** Mangalam, Karttikeya, Raiymbek Akshulakov, and Jitendra Malik. "Egoschema: A diagnostic benchmark for very long-form video language understanding."
# Comparison with video agents approaches

In this paper, we discussed various video understanding approaches, including video agents. Our goal is to present diverse perspectives on video understanding, rather than to assert that TOPA is the ultimate solution.

### **(3) The final version will include all the related works we discussed.**

> LangRepo as archived on March 21, 2024, which is two months earlier than the submission deadline, so it should have been included in the authors' Table 1 based on the NeurIPS submission guidelines.

We are glad to include and discuss more related works for comprehensive understanding in our final version. 

However, it is important to emphasize once again that **TOPA is not a video agent**. The comparion of these new video agents doesn't affect the main contributions of this paper. Besides,  We have already discussed and compared several representative video agents [1,2,3].

### **(4) We don't claim that "TOPA outperforms VideoAgent [1]"**

> Additionally, the authors claim that their model outperforms VideoAgnet in lines 213-214 or sentence was miswritten.

Line 213-214 write: *"TOPA outperforms previous image-based adaptation approach IG-VLM and video agents LLoVi and Vamos with the same scale LLM (Llama2-7B and Llama2-13B)"*. Here, "video agents" specifically refer to LLoVi and Vamos, not VideoAgent [1].

### **(5) Comparison with VideoAgent [1].**

> At a minimum, the authors should have compared performance by either applying Llama2-13B to VideoAgent or applying GPT-4 to their model.

TOPA need train on TextVid dataset, which involves LLM backforward, making it challenging for us to use GPT-4 as the LLM backbone.
In the table below, we observed a significant performance drop in VideoAgent when using Llama2-70B as the LLM backbone. Notably, TOPA with Llama2-13B outperforms VideoAgent with Llama2-70B, highlighting TOPA's effectiveness.


|                | LLM            | EgoSchema Subset |
| :------------- | :------------- | :--------------: |
| VideoAgent [1] | **GPT-4**      |       60.2       |
| VideoAgent [1] | **LLama2-70B** |       45.4       |
| TOPA           | **LLama2-13B** |       51.2       |
> LangRepo as archived on March 21, 2024, which is two months earlier than the submission deadline, so it should have been included in the authors' Table 1 based on the NeurIPS submission guidelines.

>Additionally, the authors claim that their model outperforms VideoAgnet in lines 213-214 or sentence was miswritten. TOPA is not shown to outperform VideoAgent in Table 1 and table lacks existing models with the similar model size, making it difficult to assess where TOPA actually stands in the context of very long-form video. At a minimum, the authors should have compared performance by either applying Llama2-13B to VideoAgent or applying GPT-4 to their model and explaining the performance differences. Outperforming LLoVi is not sufficient. Furthermore, since TOPA was trained on the Ego4D datase, which includes the EgoSchema videos, the authors cannot claim that the TOPA accuracy in Table 1 is a zero-shot result.

> The authors also need to address why the accuracy improves when evaluating from the EgoSchema subset to the fullest, while all existing models have shown a performance drop, a pattern generally observed in other datasets. This raises the question of whether this improvement is because TOPA was trained on Ego4D and it is not a zero-shot performance.

> Based on this reason, I lowered my rating to reject

-----

###  Thank you for the detailed feedback. However, we believe there are many misunderstandings regarding our paper.

# Our approach——Text-only Pre-alignment:

### **(1) TOPA is NOT a video agent approach.**

While both TOPA and video agents [1,2,3,4] utilize LLMs for video understanding, they are based on fundamentally different core concepts and technical frameworks, representing distinct lines of research.

In this paper, we propose text-only pre-alignment to extend LLMs for video understanding. We propose *Tideo*, consisting of sequential textual frames, to mimic real video. During text-only pre-alignment, the LLM learns to process *Tideo*, which is represented as **sequential CLIP textual features**. During zero-shot inference, real videos are first representated as sequential CLIP visual features. These **visual features** are then projected into the CLIP textual space and processed by the pre-aligned LLM.

Video agents typically employ an LLM as a central agent to iteratively identify and compile key information to answer questions, using VLMs as tools to convert video content into **textual descriptions**.

The major and essential difference is that TOPA takes **video features** as LLM's input, while video agents take **textual descriptions** as LLM's input.

### **(2) For zero-shot evaluation, TOPA doesn't train on any videos, let along Ego4D videos.**

>  Furthermore, since TOPA was trained on the Ego4D datase, which includes the EgoSchema videos, the authors cannot claim that the TOPA accuracy in Table 1 is a zero-shot result.

> This raises the question of whether this improvement is because TOPA was trained on Ego4D and it is not a zero-shot performance.

For zero-shot evaluation, the TOPA model is trained on generated **text-only** TextVid dataset. There are **no videos** involved in our text-only pre-alignment process. 

During the data generation process, we use four types of prompts for diverse Tideo generation, one of which is based on "scenarios" from Ego4D metadata. These "scenarios" are very high-level descriptions like: ```"Watching tv","Cleaning / laundry","Potting plants (indoor),","Scooter mechanic","Eating","Walking on street","Working out outside","Playing with pets"```. We do not believe that the use of such general scenarios compromises the zero-shot setting.

Moreover, most video agents, such as VideoAgent [1], LLoVi [3], and LangRepo [4] use LaViLa[7] as the video captioner, which is **pre-trained on video-narration pairs from Ego4D**. This involves far more detailed information than the high-level scenarios we employ.
LangRepo as archived on March 21, 2024, which is two months earlier than the submission deadline, so it should have been included in the authors' Table 1  based on the NeurIPS submission guidelines.

Additionally, the authors claim that their model outperforms VideoAgnet in lines 213-214 or sentence was miswritten. TOPA is not shown to outperform VideoAgent in Table 1 and table lacks existing models with the similar model size, making it difficult to assess where TOPA actually stands in the context of very long-form video. At a minimum, the authors should have compared performance by either applying Llama2-13B to VideoAgent or applying GPT-4 to their model and explaining the performance differences. Outperforming LLoVi is not sufficient. Furthermore, since TOPA was trained on the Ego4D datase, which includes the EgoSchema videos, the authors cannot claim that the TOPA accuracy in Table 1 is a zero-shot result.

The authors also need to address why the accuracy improves when evaluating from the EgoSchema subset to the fullest, while all existing models have shown a performance drop, a pattern generally observed in other datasets. This raises the question of whether this improvement is because TOPA was trained on Ego4D and it is not a zero-shot performance.

Based on this reason, I lowered my rating to reject
Dear Reviewer kuEy,

The reviews of this paper are diverging and thus your further input is needed. Can you please carefully read the other reviews and the responses from the authors and share any updated thoughts?

Do you still think the paper needs improvements to be accepted even after reading the other two reviews that are strongly supporting the paper?

AC
We're glad that we addressed your concerns, and we appreciate your recognition of our work. We would like to thank you once again for your valuable suggestions, which helped improve our paper. We will ensure that our discussions are reflected in the final version.
We're glad that we addressed your concerns, and we appreciate your recognition of our work. We would like to thank you once again for your valuable suggestions, which helped improve our paper. We will ensure that our discussions are reflected in the final version.
I have carefully reviewed the authors' responses as well as the other reviews. I appreciate the authors' thorough comments, which have addressed all of my concerns. I'm looking forward to the revised version of the paper. I am happy with the current evaluation and discussions.
Dear Reviewer,

Thank you once again for your valuable suggestions. We hope that our response has adequately addressed your concerns. If you have any further questions or comments, please do not hesitate to reach out. We look forward to any further discussion.

We would like to bring to your attention that the discussion period is scheduled to close on August 13 at 11:59 PM AoE.

Thank you for your time and consideration.

Best regards,

The Authors
Thank the authors for the detailed response. Most of my concerns have been addressed. Overall, the proposed text-only pre-alignment framework, including the Tideo dataset and the text-only learning process, is both novel and effective. It offers a new perspective and solution for multimodal alignment, which may inspire future research.