LongVILA: Scaling Long-Context Visual Language Models for Long Videos
Authors: Yukang Chen
Fuzhao Xue
Dacheng Li
Qinghao Hu
Ligeng Zhu
Xiuyu Li
Yunhao Fang
Haotian Tang
Shang Yang
Zhijian Liu
Yihui He
Hongxu Yin
Pavlo Molchanov
Jan Kautz
Linxi Fan
Yuke Zhu
Yao Lu
Song Han
Venue: ICLR 2025 Poster (2025)
Keywords: Large language models
Long context
Multi-modality
Video understanding
Primary area: generative models

Abstract: Long-context capability is critical for multi-modal foundation models, especially for long video understanding. We introduce LongVILA, a full-stack solution for long-context visual-language models by co-designing the algorithm and system. For model training, we upgrade existing VLMs to support long video understanding by incorporating two additional stages, i.e., long context extension and long video supervised fine-tuning. However, training on long video is computationally and memory intensive. We introduce the long-context Multi-Modal Sequence Parallelism (MM-SP) system that efficiently parallelizes long video training and inference, enabling 2M context length training on 256 GPUs without any gradient checkpointing. LongVILA efficiently extends the number of video frames of VILA from 8 to 2048, achieving 99.8% accuracy in 6,000-frame (more than 1 million tokens) video needle-in-a-haystack. LongVILA-7B demonstrates strong accuracy on 9 popular video benchmarks, e.g., 65.1% VideoMME with subtitle. Besides, MM-SP is 2.1x - 5.7x faster than ring style sequence parallelism and 1.1x - 1.4x faster than Megatron with a hybrid context and tensor parallelism. Moreover, it seamlessly integrates with Hugging Face Transformers.

Published as a conference paper at ICLR 2025
LONGVILA: SCALING LONG-CONTEXT VISUAL LAN- GUAGE MODELS FOR LONG VIDEOS
Yukang Chen
Fuzhao Xue!
Dacheng Li
Qinghao Hu²
Ligeng Zhu
Xiuyu Li
Yunhao Fang*
Haotian Tang!
Shang Yang²
Zhijian Liu
Ethan He
Hongxu Yin
Pavlo Molchanov
Jan Kautz!
Linxi Fan-
Yuke Zhu
Yao Lu
Song Han
NVIDIA
MIIT
SUC Berkeley
UT Austin
ABSTRACT
Long-context capability is critical multi-modal foundation models, especially for long video understanding. We introduce LongVILA, a full-stack solution for long-context visual-language models by co-designing the algorithm and system. For model training, upgrade existing VLMs to support long video understand- ing by incorporating two additional stages, i.e., long context extension and long video supervised fine-tuning. However, training long video is computation- ally and memory intensive. We introduce the long-context Multi-Modal Sequence Parallelism (MM-SP) system that efficiently parallelizes long video training and inference, enabling 2M context length training on 256 GPUs without any gradi- ent checkpointing. LongVILA efficiently extends the number video frames of VILA from 8 to 2048, achieving 99.8% accuracy in 6,000-frame (more than I LongVILA-
million tokens) video gredle-in-a-haystaee accuracy on 9 popular video benchmarks, e.g., 65.1% VideoMME with subtitle. Besides, MM-SP 2.1x 2.1x-5.74 5.7x faster than ring style sequence parallelism and 1.1x 1.4x faster than Megatron with hybrid context and tensor parallelism. Moreover, seamlessly integrates with Hugging Face Transformers. Our code and models available at Batiabbb.oo///llab/////L//
demonstrates strong
INTRODUCTION
Integrating multi-modal understanding with long-context capability important. A foundation model supporting more modalities take more flexible input signals that people can inter- act with the model more diverse manners, e.g., GPT-4o-like multi-modal chatbot, molti-modal web agent (Koh et al. 2024), and real-world roboties foundation model (Brohan et al.) 2022 2023) Padalkar et al. 2023). Longer context enables models to process more information, e.g., long doc- uments, level codebase, and hour-length video, which similarly provides required features to more real-world applications.
While works have enabled long-context Vision-Language Models (VLMs) (Lin al. 2023b Weng et al.) 2024), they employ simplified approaches rather than offering a comprehensive solution. For instance, LongVA VA (Zhang et al.] (024b) relies on long-context LLMs and trains models on short-context data. LongVLM VLM (Weng al.) (0224) utilizes token compression to circumvent context extension. These approaches sidestep more challenging issues, such as the development of robust long-context multi-mode training framework and corresponding dataset design.
full-stack design is crucial for long-context Vision-Language Models (VLMs). Training large models is typically complex, systematic endeavor that demands both data engineering (Betker) al. 2023 Ouyang al. 22022] /hou et al.] (024)) and system-software co-design (Lepikhin al.
2020
Chowdhery et al. 2023 Shocybi et al. 2019:; Brown et al.) 2020 Dehghani al.) (023). Unlike text-only LLMs. VLMs (e.g.. LLaVA (Liu et al. (023c)) often require distinct model archi-
tectures and flexible distributed training strategies. Additionall, long-context modeling necessitates
Algorithm Lead. System Lead. The first four authors have equal contributions.

---

Published as a conference paper at ICLR 2025
1500
DOM
Language Model
Language Model
Language Model
1286
MM Projector
MM Projector d
MM Projector
900
MM-52
MM
Vision Tower
Vision Tower
Vision Tower
256
faancs
LLM
ViLA
I.ONG
Stage 2 &
Stage Mid-training
Stage 5: Long Video SFT
Align & Pre-train & SFT
Context Extension
w/ MM-SP
Figure 1: The Vong training pipeline. In Stages through 3, the process starts with alignment, pre-training, and supervised fine-tuning. In Stage 4, the model undergoes mid-training context ex- tension. Finally, in Stage 5, the model is fine-tuned for long video understanding with Multi-Modal Sequence Parallelism (MM-SP).
Baseline
LongVA
LongVILA
500
Figure 2: Comparison of Needle in the Long Video Haystack Experiment. The 32-frame baseline model (left) can not retrieve right needles after 32 frames. Long (middle) achieves 87.6% accu- racy in 3,000 frames. In contrast, the LongVILA model (right), trained on 2048 frames, presents 99.8% accuracy on 6,000 frames (more than 1 million context length).
not only long-contens data to fully utilize the model's 's capabilities (Fu et al. 2024c Chen 2023
but also infrastructure capable of supporting memory veetory-itensive long-context training (Li et al. 2021
Jacobs et al. 2023 Li et al. 2023a). Therefore, full-stack design, encompassing training pipeline and system, indispensable for long-context VLMs.
In this work, we introduce LongVILA, a comprehensive solution for long-context VLMs. For train- ing pipeline, we implement five-stage training curriculum Figure D (1) multi-modal align- ment, (2) large-scale pre-training, (3) short supervised fine-tuning, (4) context extension for LLMs, and (5) long supervised fine-tuning. For system, establish an efficient and user-friendB frame- work, namely Multi-Moda Sequence Parallelism (MM-SP), which supports training and inferencing memory-intensive long-context VLMs.
LongVILA-7B presents strong performance on 9 popular benchmarks, egg., 65.1% on VideoMME (Fu al.) 2024a) with subtitle. The LongVILA model, trained on 2048 frames, achieves 99.8% accuracy the needle-in-a-haystacc experiments with 6,000 frames, with a con- text length of more than I million tokens. In ablations, by increasing the number of video frames using LongVILA, the performance on VideoMME in long videos consistently improves (Figure Our MM-SP system can efficiently scale the context length up to 2 million tokens without gradient checkpointing, achieving 2.1 x to 5.7 x speedup compared to ring style sequence parallelism, and 1.1x to 1.4x compared to Megatron with hy context parallelism and tensor parallelism.
2 RELATED WORKS
Visual language model architecture. There are two predominant designs for VLMs: the encoder- decoder architecture (e.g.., LLaVA (Liu et al. 2023c), PaI--EE (Driess et al. (003)) and the decoder- only architecture (e.g.. Fuyu (Bavishi et al. 2023). Chameleon (Team) (024)). Encoder-Decode VLMs connect the vision encoder to the LLM decoder through a modll projector. Certain multi-modal projectors. such spatial pooling and Q-former, significantly reduce the number of tokens per image or video frame, thereby lowering the computational burden on the LLM decoder. In contrast, decoder-only LLMs typically process patches as input without hierarchical token pooling, making it more challenging to reduce the token count for each image or frame. In this work, build VILA (Lin al. 2023b) foundation. It is worth noting that enhanced variants
2

---

Published as a conference paper at ICLR 2025
subtitle w/ subtiile
57.1
57.8
58
Training Stages
VideoMME
Average
Short
Medium
Long
54.7
1-2-3-4-5
57.5
69.3
56.1
47.0
53.4
53.7
1-2-4-33855)
55.9
67.4
54.1
46.1
48
50.6
51
51.3
4-1-2-3-5
56.0
69.2
54.1
44.5
4-1-2-(385)
55.3
67.2
53.6
45.1
46.8
Number of Frames
32
64
128
256
512
1024
Table 1: Ablations on various training stage settings on VideoMME (without subtitle). (3&5) means joint training of the stage 3 and 5.
Figure 3: Scaling video frames improves
VideoMME accuracy in long category.
of VILA exist, such as VILA et al. 2024) for improved performance and X- X-VILA Ye et al. 2024) for cross-modality understanding, reasoning, and generation. For our model architecture and training pipeline, we adhere to the standard VII.A-1.5 version.
Sequence parallelism and hybrid strategy. Long-context training examples often exceed the memory capacity of single device. To address this issue, the sequence parallelism paradigm has been widely adopted in the text-only L.MM community, distributing a single sequence across multiple devices. Specifically, Ring-style systems[Li et al. (202T) [203a [ (2.23a) use Point-to-Point (P2P) communication primitives to collectively compute the attention module, while DeepSpeed- Ulysses [aacbb et al. (2023) employs an All-to-All (A2A) primitive alternate between sharding the sequence dimension and the attention head dimension during attention computation. Ulysses generally achieves higher throughput than Ring-style SP due its more efficient A2A communi- cation primitive and larger, unsegmented computation blocks. However, its scalability is limited by number of attention heads. Recently, USP (Fang Zhao 2024) was introduced as the first to integrate Ring-style SP and Ulysses SP, combining the strengths of both approaches. LoongTran (uu et al. 2024) further optimizes communication and placement strategies to enhance training ef- ficiency. Following (Fang& & Zhao 2024 Gu al., (024), extend the system to multi-modal scenarios to accommodate complex attention masks and variable-length input sequences. Our work the first to design and implement sequence parallelism system for visual language models.
LONGVILA TRAINING PIPELINE
As shown in Figure [ in our pipeline, there are five training stages, i.e., Stage 1: multi-modal align- ment, Stage 2: large-scale pre-training, Stage 3: supervised fine-tuning, Stage 4: context extension for LLM, Stage 5: long supervised fine-tuning. Stage 2, and follow VILA (Lin al. 2023b), to firstly bridge the gap between LLM and vision encoder, and then pre-training In Stage 1, only the multi modti- projector is trainable with others frozen. Stage 2, we freeze the vision encoder and training L.MM and the multi-modal projector. In Stage 3, we fully fine-tune the model for short data instruction following, egg., image and short video datasets. Afterwards, we extend the context length of LLM with text-only dataset in continued pre-training manner in Stage 4. In Stage 5, we adopt our MM-SP system ($4) to enhance the instruction following abilities by long video supervised fine-tuning. is noted that all parameters are trainable in the final stage.
larger datasets.
3.1 STAGE1 &283: ALIGNMENT, PRE- PRE-TRAINING, AND SHORT SUPERVISED TINNIGGNING
We first use open-sourced image and video caption datasets to train the multi modal projector in stage (1) conduct the multi-modal alignment. Note that, following (Lin al. 2023b). both vision encoder and LLM decoder are frozen at this stage. After that, we conduct large-scale pre-training to learn general multi-modal capacity at scale. To improve the quality of large open-sourced datasets, we follow VILA² (Fang et al. 2024) to relabel COYO-25M (Lin et al. 2023b Byeon et al. 2022 with VILA-115-40B (Lin et al. 2023b). The supervised fine-tuning process incorporates mixed data types, including both images and videos. For short video comprehension, video instruction-followiin datasets, e.g., YouCook2 Zhou et al. (2018) and ShareGPTVideo Zhang et al. (002c).. In experiments, model is based Qwen2-1.5h and Qwen2-7B (qwe 2024).
utilize open-source
3

---

Published as a conference paper at ICLR 2025
8%
Proportion of video categories
Question calegories
8%
10%
Travel & Events
Sports
8%
13%
%%
Spatial
Education
Pets & Animals
Attribute
12%
5%
People & Blogs
News & Politics
8%
24%
Action
3%
13%
13%
6%
Music
Science & Technology
9%
Object OCR
Comedy
Entertainment
29%
Synopsis
5%
Film
Gaming
Temporal
Figure 4: The proportion question and video categories our Long VoLA dataset. have 15,292 videos in total. For each video, there are sample for captioning and the other question.
Clip
Clip 4
Clip
ViLA
Caption
Caption
LLM
Summarizations
Clip 3
Clip
Question-Answen
Caption
Caption
Caption
Caption
Long video
Long video
10-sccond short video clips
Captions each clip
Long SFT data
Figure 5: The pipeline for generating instruction-followin data from long videos. The process be- gins by segmenting the long video into short clips, each approximately 10 seconds in length. These clips individually annotated with captions using the VILA-1.5 model. Subsequently, an LLM employed to generate question-ane and-anndwae pairs based on the captions of these clips. Generated questions include summarization and other inquiries pertinent to the content of long videos.
3.2 STAGE4: CONTEXT EXTENSION FOR LLMS
Our empirical research indicates that extending the context length of LLMs is essential prior to en- gaging in supervised fine-tuning with long video datasets. Following Stage 2 of our methodology, we execute a continuation of pre-training on the LLM to enhance its context length to 262, 144, utilizing total of 17B tokens. We employ progressive training schedule, incrementally increas- ing the context length from 8, 8,192 to 65,536, and ultimately to 262, 144, utilizing the SlimPajama dataset (Soboleva et al.. 2023) in accordance with the methodology outlined by (Fu al. 2024d).
Furthermore, we augment the base frequency of the Rotary Position Embeddings (RoPE) as de- scribed in (Su al. 20211) during the fine-tuning phase. Sequence parallelism is implemented for the training at the 262, 144 context length. We use low-rank adaptation for context extension fine-
tuning (Chen al., 2024b). These processes collectively require approximately 336 GPU hours
machines equipped with 80GB A100 GPUs.
3.3 STAGE5: LONG SUPERVISED FINE-TUNING
Long video instruction following To facilitate the fine-tuning of long videos, we constructed a new, dedicated dataset for long video training, each consisting 15,292 videos. We use the original long videos from the Shot2Story dataset (Han et al., 2023). Each video includes different questions and answers: one for generating captions and another for answering questions, enabling diverse appli- cations in video understanding. Figure illustratts the process for generating instruction-folloiing datasets from long videos. Initially, the long video segmented into shorter clips, each approxi- mately 10 seconds in duration. These clips are then independently annotated with descriptive cap- tions utilizing the VIL.A-155 model. Subsequently, an L.M is employed to generate question-and. answer pairs derived from the captions of these clips. The generated questions encompass summa- rization and other queries relevant to the comprchensive understanding of long video content.
As in Figure 4 the left chart categorizes videos into several domains, including Travel & Events, Sports, Education, Pets & Animals, People Blogs, News & Politics, Music. Science & Technol- ogy, Comedy, Entertainment, Film, and Gaming, ensuring wide-ranging representation of video content. The right chart breaks down the categories of questions into Spatial, Attribute, Action, Object, OCR, Synopsis, and Temporal, reflecting the variety of inquiries and cognitive tasks that
4

---

Published as a conference paper at ICLR 2025
Send KV P2P
# Image tokens
GPU
GPU
GPU
GPU
Aggregate KV via AlIAAII
Text tokens
200
Ring
GPU
GPU
GPU
GPU
GPU
GPU
MM-SP
<img
<img
<img>
300
Two-Stage Sharding Strategy
(Ours)
Stage I: Shard by ji images
100
100
100
100
300
GPU
GPU
50 GRN
GPU
Stage Shard by tokens
082
GPU
GPU
GPU
GPU
100
100
100
50
300
Figure 6: Sharding strategy and communication pattern of MM-SP. For sharding strategy, Ring SP is designed for text-only modalities, without optimization for the workload of an image encoder. Our MM-SP implements a novel sharding strategy that balances the computational load between the image encoder and the language modeling stages. For communication pattern, Ring SP (Liu al., [223a] Li et al., 2023a) (top) relies on P2P communication for both intra-node and inter- node settings, resulting in underutilization of intra-node bandwidth. MM-SP (bottom) adopts 2D- Attention (Fang & Zhao 2024, Gu al. (2024) mechanism which utilizes intra-node All-to-All (All2All) and inter-node Point-to-Poin (P2P) communication to transfer keys and values (KV),
enhancing the efficiency of intra-node NVLink utilization. The bandwidth is for H100.
the dataset can address. This dataset provides rich resource for advancing the understanding and processing of long video formats in supervised fine-tuning.
Once we acquired the long video dataset, applying it for supervised fine-tuning introduced new challenges, primarily due to the substantial number of frames in each sample-often ranging in the hundreds or even thousands. For instance, single sequence from 1400 video frames can encom- pass around 274k tokens. Existing data- aata-parallel training systems struggle to handle such extensive contexts. We developed the MM-SP system (Section 4 to efficiently train long-context VLMs.
4 MULTI-MODAL SEQUENCE PARALLELISM
Training long-context Vision-Language Models (VLMs) results in substantial memory demands. The most widely used open-source solution, fully sharded data parallelism, does not distribute the activations generated by single sequence, making it unsuitable for needs. Consequently. developed a custom system based on sequence parallelism (Li et al.) 2021 2023a [iu et al., (2023a] Jacobs et al., 2023), technique commonly employed in existing foundation model systems opti- mize text-only LLM training. However, we discovered that existing systems are neither efficient nor scalable enough to handle our foog-cooted VLM workloads.
4.1 LIMITATIONS OF EXISTING SYSTEMS
Modality heterogeneity. text-only LLMs, sequences are processed by single tokenizer into tokens, allowing for straightforward distribution of tokens across multiple GPUs. However, VLMs incorporate an encoder architecture where non-text data initially represented by placeholder to- ken (e.g., <img>> and subsequently encoded into multiple real tokens during training. For instance, a single video frame typically requires around 256 tokens (Lin et al. 2023b). Due to the differ- ing processing requirements of visual and text modalities, simplistic implementation that treats placeholder tokens the same as text tokens leads to an imbalance GPU workloads (Figure
Networking heterogeneity. Our multi moduti-m comprises extremely long videos (Figure I which requires employing sequence parallelism in a molti-node setting. In multi-node setting, inter- node and intra-node network bandwidth differs significantly. For example, the NVIDIA DGX H100
5

---

Published as a conference paper at ICLR 2025
Stage I: Shard Images
Stage Shard Tokens
Stage LLM Execution Training Serving with MM-SP
Balancing Vision Tower Load
Balancing LLM Load
(1) Gathering Global lapul Embedding
GPU
Viius
LIM
Aspuneer
(2) Padding Sequence
GPU
Visus
GPU2
Tower
LLM
(3) Sharding to Local GPU
GPE
GPU
GPU
LLM
Tower
5
3
Video Frames
Pad Token Enable P2P Communicntion
Local Inguts
Captions
Figure 7: Workflow of Multi-Moda Sequence Parallelism with Batch Size (bs) of 4 and Sequence Parallel Size (SP Size) of 3. To accomodate multi-modal inputs, we developed a customized shard- ing strategy that ensures balanced workload distribution and compatibility with SP communucation.
utilizes NVLink at 900 GB/s for intra-node GPU communication and InfiniBand at 50 GB/s for inter-node GPU communication (single path), resulting in an 18x difference in bandwidth. Previous work, Ring-Style sequence parallelism (Li al. 2021 2023a Liu al.. [203a Zhu 2023) ignores the heterogeneous networking feature on GPUs and utilizes P2P communication in both inter-node and intra-node settings. This design induces excessive communication costs where they usually attempt to overlap them into computation. However, we found that this design cannot always hide the overhead, and even slows down the computation kernel (Table 2
Limited maximal sequence length. DeepSpeed-Ulyseee (Jacobs et al. 2023 presents potential solution to the communication challenges in ring-style sequence parallelism by employing All-to- All communication primitives, which reduce the overall communication volume. However, this approach has its limitations. The design relies on parallelizing along the attention head dimension rather than the sequence dimension during attention computation. As result, DeepSpeed-Ulyseeee cannot scale effectively beyond the number attention heads. For instance, the Llama-3 model uses Grouped Query Attention (GQA) with Key-Vat Key-Value heads, which restricts the maximum quence parallelism degree to 8. Even when using replication for Key-Value heads, which introduces additional communication overhead (Li et al.) 2023a), the highest achievable sequence parallelism degree is still limited to 32 (the number of Query heads). This constraint insufficient for handling extremely long sequences, such full-length movies.
4.2 MULTI-MODAL SEQUENCE PARALLELISM TRAINING MODE
After identifying the limitations in existing systems, we conclude that an ideal multi-modal se- quence parallelism approach should prioritize efficiency and scalability by addressing both modality and network heterogeneity. and should also be capable of scaling beyond the number of attention
heads. To achieve this, we adopt 2D-ateention (Fang Zhao 2024 Gu et 2024) mechanism for sequence parallelism. For instance, as illustrated on the left in Figure 6 enable 8-degree
sequence parallelism across 2 nodes, we construct 4x2 communication mesh using 2D-SP. In this setup, the A2A process group, with size of 4, distributes the QKV tensors according to the head dimension and re-partitions them along the sequence dimension within each node. Simultane- ously, the P2P process group, with size of 2, transfers the partitioned KV chunks between nodes. Additionally, to further explain how the 2D-attention mechanism operates, we depict the attention computation schedule using different methods in Figure
MM-SP workflow. address the challenge of modality heterogeneity, we propose a two-stage sharding strategy that optimizes the compute workload for both image encoding and language mod-

---

Published as a conference paper at ICLR 2025
Table 2: The forward and backward attention kernel wall-clock time with or without the overlapping design (Unit: μs). The communication overlap design in Ring-style SP slows down the attention kernel by occupying streaming multiprocessor (SM) resources.
Seq. length
4K
SK
16 K
24K
32K
forw. w/o 29.5
49.3
122.1
239.2
402.9
forw.
35.0 (+18.6%)
54.6 (+)0.7%)
131.2 (+7.5%)
250.9 (+4.8%)
420. (+4.2%)
backw. w/o
77.7
123.3
362.9
730.0
1218.9
backw, 82.2 (+5.8%)
129.8 (+5.3%)
367.0 (+1.1%)
743.2 (+1.8%)
1225.3 (+0.5%)
eling stages. As illustrated in Figure the process begins by evenly distributing images (e.g., video frames) across devices within the sequence parallelism (SP) process group, thereby achieving load balancing during the image encoding stage. the second stage, we aggregate global vision and text inputs for token-level sharding. To support ring-based attention, sequences extended with arbi- trary dummy tokens, ensuring that each sequence can be evenly divided according to the ring-based SP degree. This adjustment maintains consistency with the original approach by modifying label inputs to ignore padded tokens during loss calculation. We implement balanced sharding strategy that distributes the context to each rank from both ends, ensuring equal computation across ranks. The effectiveness of this strategy will be demonstrated later (Table (5). Since this redistribution is performed only once during training, the overhead is minimal. Finally, the balanced local inputs are processed by the LLM backbone, utilizing 2D-Attent AD-Attention to achieve efficient sequence parallelism.
4.3 MULTI-MODAL SEQUENCE PARALLELISM INFERENCE MODE
The model we developed through sequence parallelism training is capable of handling long-context multi-modal downstream tasks. However, the most commonly used inference system, built on HuggingFace Transformers, typically operates on a single GPU. This lack of distributed imple- mentation limits the maximum sequence length that can be processed during inference. The most straightforward solution within HuggingFacc Transformers is to use its pipeline parallelism inference feature, which shards single model across multiple devices on a layer-by-layer ba- sis (Huang et al.] 2019, Narayanan et al. 2019). However, this approach is inefficient, as it only activates one GPU at a time. Additionally, it struggles support long sequence because the first device must store large input embeddings and images, creating a memory bottleneck.
To address these limitations, we implemented sequence parallelism for distributed inference in VLMs. Unlike the training mode, the inference system additionally man- ages tensors, such as input tokens and position encodings, that progressively change during the decoding phase (Yu jet al. 2022). It detects signals from the machine with the last token terminate the distributed process appropri- ately. Compared to HuggingFace' pipeline parallelism strategy, our inference mode is more efficient, as all de- vices participate in computation simultancously, acceler- ating the process by a factor proportional to the num- ber of machines (Figure R). Furthermore, it is scalable, with memory evenly distributed across devices, enabling longer sequences with additional machines.
Hluggingacee Pipeline
MM-SP (Ouns)
Active
Tdlle
Idle
Idll
Active
GPU
Idle
Activve
Idll
Edlla
Active
Idle
Idle
Activve
Idile
Active
GPU
Idle
Tdle
Idle
Active
Active
T-0
T-1
T-2
Th
T-O
Figure 8: Inference scheduling compar- ison between HuggingFace Pipeline and MM-SP, illustrated with 4 GPUs. MM- SP utilizes all GPUs concurrently.
5 EXPERIMENTAL RESULTS
5.1 TRAINING AND INFERENCE SYSTEM
Our training and inference systems can be integrated with HuggingFace Transformers through straightforward monkey patching, in line with the popular open- source approach outlined in (Zheng ct al,, 2023). In this section, we present a quantitative evaluation of the training system's 's throughput, the inference system's latency, and the maximum supported sequence length.
7

---

Published as a conference paper at ICLR 2025
Iteration Per Second
Max Sequence Length (K)
Number of leration
2.15
Megatron (2)
Megatron (CP 4-14--77 TP-43
Ulyses
Only
(Ouns) ZigZag RingArts
Sog. length
ZagZag-Kingan
1.5
(Ouna Ulyases
(Oung 21 Allantion
1500
Atlentioa (ouns)
0:75
750
8
64
128
356
Sequence Length
Number of GPUs
Figure 9: Performance comparison of training systems on 32 H100 GPUs. MM-SP is as scalable as ZigZag- Attn, and efficient as Ulysses.
Time to First Token (s) 26.3
300
Max Sequence Length (K)
276
Well Clock Time (4)
Hegp Pipeline
(K3
HupcingFace Pipeline
Max Seq, length
14
150
138
3.5
9.2
58
96
4.2
1.8
2.5
0.5
1.9
2.9
16K
128K
2
4
Sequence Length
Number of GPUs
Figure 10: Performance comparison of inference systems on 8 H100 GPUs.
5.1.1 TRAINING SYSTEM
Baselines and hardware setup For training efficiency, compare system with ZigZag ring- style sequence parallelism, which incorporates load balancing and GPU optimization (ZIGZAG-
RINGATTN for consistency)(L. li et al.) 2023a Zhu 2023 Liu al. 2024al Korthikanti al. 2023
We use a widely adopted open-soure implementation (Zhu) 2023) To reduce the memory footprint of models, gradients, and optimizer states, we employ Fully-Sharded Sharded Data Parallelism (FSDP) (Zhao (Zhao et al. 2023) instead of 7ero-33 3earo-3(Rabbhandar et al., 2020) (Table Additionally, we compare our sys- tem with the expert-designed and highly optimized Megatron-T.M
et al. 2019 Korthikanti et al., 2023) system, focusing on their implementation of sequence parallelism, termed "context parallelism" (CP). We also evaluate hybrid strategy that combines tensor model parallelism (TP) within a node and CP across nodes, recommended by the Megatron-LM team for advanced usage.
We conduct most experiments on H100 nodes, each equipped with 8x (80GB) GPUs intercon- nected via intra-node NVLink and 400 Gbps inter-node InfiniBand. For experiments involving the maximum supported sequence length during training, we extend the setup to 32 A100 nodes, each with Ax 100 (80GB) GPUs, where the conclusions equivalent total memory. Our evaluations are based on an 8B model with a batch size of Since the Megatron- LM baseline does not natively support VLM training and the visual encoder is typically orders of magnitude smaller than LLMs, we report the main results for the LLM backbone without the visual encoder. An ablation study of the visual encoder is provided in $ 5.1.3
consistent with those for H100 due to the
Throughput Figure / presents throughput results measured as iteration per second over 32 H100 GPUs. These results were obtained after 10 warmup iterations and averaged over 5 iterations min- imize variance. Our system achieves a speedup of 2.1x x to 5.7x compared to ZIGZAG-RINGArTI and performs on par with DeepSpeed-Ulysses When compared to the more optimized ring-style se- quence parallelism in Megatron-LMM CP, our method shows 3. to 4.3x speedup. This highlights that our system design effectively addresses the issues inherent in ring-style sequence parallelism, as in 8 4.2 Furthermore, our system achieves 1.1x to 1.4x speedup compared to Megatron- LM's bridid strategy. Note that our system is currently implemented in Triton (Tillet et al., 2019), and further porting it to C++ could yield even greater speedup. Additionally, Megatron- LM system supports significantly lower maximum sequence length, which why its results not in the next section. We observe similar observations using 8 H100 nodes (Table 8).
observed that the
8

---

Published as a conference paper at ICLR 2025
Table 3: Performance comparisons on video benchmarks, including ActivityNet-QA Yuu et al, 2019 EgoSchema (Mangalam et al. ( EventBench (Du et al., 2024). Long VongeBeeeenchh (Wu
al. 2024). Perception Test (Patraucean et al. 2023). MVBench (Li al. (0024). NExT-QA (Xiao
al.
2021 VNBench (Zhao et al. 2024), and VideoMME (Fu al. 2024a).
LVideoBench
ActNet-QA
EgoSchema
EventBench
LLM Size
PercepTest
MVBench
NEXT-QA
VNBench
VideoMME
Model
test
test
val
val
val
test
me
val
w/o sub.
w/ sub.
GPT-4V
57.0
32.6
61.3
43.5
59.9
63.3
GPT-4o
53.3
66.7
64.4
71.9
77.2
Gemini-1.55 1.5-Pro
57.5
72.2
43.2
64.0
66.7
75.0
81.3
Video-LLaVA
7B
45.3
38.4
5.9
37.6
43.5
12.4
39.9
41.6
Flash- VStream
7B
51.9
61.6
ShareGPT4Video
8B
50.8
41.8
51.2
39.9
43.6
VideoLLaMA2
51.7
6.9
51.4
54.6
4.5
47.9
Vid20LLaMA2.1
7B
53.0
53.1
54,9
57.3
54.9
56.4
Kangaroo
8B
62.7
54.8
61.1
56.0
57.6
PLLaVA
7B
56.3
28.2
39.2
46.6
LLaVA-OV
56.7
60.1
56.4
57.1
56.7
79.4
51.8
58.2
61.5
LongVILA
7B
59.5
67.7
58.0
57.1
58.1
67.1
80.7
63.0
60.1
65.1
We evaluate the maximum sequence length supported by fixed number of GPUs by progressively increasing the per-GPU sequence length from I to 10k until an out-of-memory error occurs. The re- sults summarized in Figure To ensure fair comparison, activation checkpointing is disabled. Vanilla data parallelism fails to scale for long videos at larger cluster sizes. DeepSpeed-US peed-Ulysses par- titions based on attention heads, which limits its ability scale to higher context lengths, the 8B model has only 32 attention heads. Consequently, our approach supports approximately 8x higher context lengths when scaled to 256 GPUs. Additionally, our system achieves similar context length scaling as ZIGZAG-BR ARAAAATT,, with support for over 2 million context length on 256 GPUs.
In summary, our training system combines the best of both worlds-it achieves scalability compara- ble to ZIGZAG-RINGATI while maintaining the throughput of CrepSpeed-Ulyseeeee Additionally, delivers a 1.3x speedup and supports 2.5x longer context lengths compared to the highly opti- mized Megatron-LM.
5.1.2 INFERENCE SYSTEM
We evaluated our inference system against HuggingFace Pipeline parallelism using single node with 8 H100 GPUs and the 8B model (Figure 10). Our system achieves an 8.2x speedup com- pared to HuggingFace Pipeline on 8xH100 GPUs. This significant improvement primarily due to HuggingFace Pipeline inference activating only one GPU at time, whereas our method lever- ages all GPUs to compute jointly. Figure TO compares the maximum supported sequence length, where method supports sequences that are 2.9x longer than those supported by HuggingFace Pipeline. Specifically, during 96K sequence length inference, HuggingFace Pipeline stores 80GB of activations the first GPU and only 18GB the remaining GPUs. This imbalanced allocation of activations limits the maximum supported sequence length.
5.1.3 EFFECT OF -WO-STAGE SHARDING
We evaluate the impact of our two-stage sharding strategy using video captioning dataset (Chen et al. 2024a). We compare our two-stage sharding to a one-stage baseline that only distributes work- load based on the number of images. We measure the time per iteration across different numbers of H100 GPUs. For & GPUs, we use & images per video and a batch size of The results, shown in Table 5 indicate speedup ranging from 1% to 7%. This improvement is primarily observed

---

Published as conference paper at ICLR 2025
Table 4: Performance comparison on VideoMME (Fu et al.
2024a) benchmark in details.
LLM
w/o subtitle
subiitte
Model
Frames
Size
Overall
Short
Medium
Long
Overall
Short
Medium
Long
Video-LLaVA
7B
39.9
45.3
38.0
36.2
41.6
46.1
40.7
SliME
8B
45.3
53.3
55.4
39.8
47.2
44.4
41.7
ShareGPT4Video
8B
39.9
48.3
36.3
35.0
43.6
39.3
37.9
VideoChat2
7B
39.5
48.3
37.0
33.2
43.8
52.8
39.4
39.2
LaM2
7B
47.9
56.0
45.4
50.3
47.6
43.8
Chat-Univi-v1.5
7B
40.6
45.7
40.3
35.8
45.9
44.6
41.8
Kangaroo
8B
56.0
66.1
55.3
46.7
57.6
55.4
49.3
ShareGemini
7B
43.2
49.1
41.3
47.9
49.1
47.3
43.4
Long
7B
52.6
61.1
50.4
46.2
54.3
61.1
53.6
47.6
20B
10
50.7
60.2
46.4
45.6
52.4
61.7
49.1
46.6
VITA
8x7B
20
55.0
64.2
53.3
47.6
57.6
67.9
55.3
49.6
Video-CCAM
14B
96
53,9
62.1
52.8
47.0
56.1
63.9
55.9
48.3
1.5B
53.6
66.2
49.3
45.3
57.5
70.2
54.1
48.2
VIng
256
7B
60.1
69.0
58.3
53.0
65.1
72.9
64.9
57.4
in longer captioning tasks, where the baseline suffers from workload imbalance due to the lack of sharding based on the number of text tokens.
5.2
GENERAL VIDEO UNDERSTANDING
Table presents the performance of VILA, comparing state-of-the-arr models (Lin al.
2023a Zhang et al.
2024d;
Chen et al. 2024a
Li et al.
2023c Cheng et al. 2024 Jin et al. 2023
Zhang et al. Share
Fei al. 2024 Liu al. 2024c Xu et 2024 Li et al. 2024a; Zhang et al. 2024a) on popular video benchmarks. Long VILA-
2024 Chen al. 2024c
Fu et al.
2024b
7B achieves strong performance across all these benchmarks. Table compares their effective- ness across short, medium, and long video lengths, as well as overall performance VideoMMEE LongVILA, utilizing 256 frames, achieves an overall score of 60.1 65.1 without / with subtitle, which are competitive results. We include LongVILA-I.55 (starting from Qwen2-1.55 (qwe 2024)) for evaluation, which is also competitive. We provide a detailed model complexity Long VILA among various model size, number of frames, context length, latency and FL.PSs in Table 10 in the appendix. We also do the ablation on training schedules in Table
5.3
In Figure we present the results of the Needle Haystack experiment for long videos. Following the methodology established in the existing literature (Zhang et al.), we prepared long video and sampled a fixed number of frames. We inserted specifically designed images at various depths and tasked the model with answering corresponding questions. The 32-frame baseline model (left) was unable to accurately retrieve the correct images beyond 32 frames. In contrast, the Long VILA model (right) demonstrated 99.8% accuracy across 6,000 frames, which contains more than million tokens. To our best knowledge, this is the first VLM which can handle million context length.
6 CONCLUSION
We introduce VILA, a comprehensive full-stack solution for long-context visual language mod- els, encompassing model training pipeline and distributed system. Based on curated long video datasets and five-stage training pipeline, our Long VILA model extends the feasible frame count from 2048, precisely capturing fine-grained information from 2-hour peedle-in-a-haystaee with more than million tokens Our LongVILA-7B model achieves strong performance across pop- ular video benchmarks, especially on VideoMME, e.g., 65.1% accuracy with subtitle. Our system efficiently scales context length up to 2 million tokens, achieving speedups of 2. to 5.7x com- pared to ring sequence parallelism and 1.1x to 1.4x compared to a hybrid Megatron context and tensor parallelism.
videos,
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Qwen2 technical report. 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 230.::2208.66 /2966, 2023.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dallle-3, pdf. 2(3):8, 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale, arXiv preprint arXiv:2212.0681/ 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. R1-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818. 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sachoon Kim. Coyo-700m: Image-text pair dataset, 2022.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong. Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and genera- tion with better captions. arXiv preprint vr.Xv:2246.04332 2024a.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309. 12307, 2023.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. In The International Conference on Learning Representations, 2024b.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo. Zheng Ma. Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui Hc, Botian Shi. Zhenjiang Jin, Chao Xu, Bin Wang. Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far are we to gpt-4v? clos- ing the gap to commercial multimodal models with open-source suites. CoRR, abs/2404.16821. 2024c.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. CoRR, abbs/2406.07476, 2024.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Wenliang Dai. Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiv, abs/2305.06500, 2023.

---

Published as a conference paper at ICLR 2025
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In ICLR, 2024.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp. 7480-7512. PMLR, 2023.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint arXiv:2303.0m 03378, 2023.
Yifan Du, Kun Zhou, Yuqi Huo, Yifan Li, Wayne Xin Zhao, Haoyu Lu, Zijia Zhao, Bingning Wang, Weipeng Chen, and Ji-Rong Wen. Towards event-oriented long video understanding. CoRR, (oss24006.1119, 2024.
Jiarui Fang and Shangchun Zhao. Usp: A unified sequence parallelism approach for long context generative ai. arXiv preprint arXiv:2405.07719, 2024.
Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang. Pavlo Molchanov, Jang Hyun Cho, Marco Pavone, Song Han, and Hongxu Yin. Vila2: Vila augmented vila. arXiv preprini arXiv:2407.1744 2024.
Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. CoRR, abs/2408. 14022, 2024.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. CoRR, abs/2405.21075, 2024a.
Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, Ran He, Rongrong Ji, Yunsheng Wu, Caifeng Shan, and Xing Sun. VITA: towards open-source interactive omni multimodal LLM. CoRR, abs/2408.05211, 2024b.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for sealing language models 128k context. arXiv preprint arXiv:2402. 10171, 2024c.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. CoRR, aabs/2402.101777 2024d.
Diandian Gu, Peng Sun, Qinghao Hu, Ting Huang, Xun Chen, Yingtong Xiong, Guoteng Wang, Qiaoling Chen, Shangchun Zhao, Jiarui Fang, Yonggang Wen, Tianwei Zhang, Xin Jin, and Xu- anzhe Liu. Loongtrain: Efficient training long-sequence Ilms with head-context parallelism. CoRR, pdf/2406. 18485, 184455 2024.
Mingfei Han, Linjie Yang, Xiaojun Chang, and Heng Wang. Shot2story20k: A new benchmark for comprehensive understanding of multi-shot videos. arXiv preprint arXiv:2311.17043, 2023.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Leon Song, Samyam Ra- jbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309. 14509, 2023.
Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. CoRR, Abs/2311.08044 2023.
12

---

Published as a conference paper at ICLR 2025
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXin:2401.13649, 2024.
Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mo- hammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5:341-353, 2023.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. CoRR, 00s//4008.0332, 2024a.
Dacheng Li, Rulin Shao, Anze Xie, Eric Xing, Joseph Gonzalez. Ion Stoica, Xuezhe Ma, and Hao Zhang. Lightseq: Sequence level parallelism for distributed training of long context transform- ers. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arX1v:2301.12597. 2023b.
Kunchang Li. Yinan He. Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo. Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. CoRR, abs/2305.06355, 2023c.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. Mybench: A comprehensive multi-modal video under- standing benchmark. In CVPR, pp. 22195-22206, 2024b.
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective. arXiv preprins arXiv:2105.13120, 2021.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models, 2024c.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by aligmment before projection. CoRR, abs/2311.10011 2023a.
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre- preitnain for visual language models, 2023b.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context. arXiv preprint arXiv:2310.01889 2023a.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. arXiv preprint arXiv:2402.08268, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved basclines with visual instruction tuning. arXiv preprint a0x07:2310.03744, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2023c.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge. January 2024b.
Jiajun Liu, Yibing Wang, Hanghang Ma. Xiaoping Wu, Xiaogi Ma, xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-Language model supporting long-context video input. arXiv preprint a40i 15542, 2024c.
13

---

Published as a conference paper at ICLR 2025
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very long-form video language understanding. In NeurIPS, 2023.
Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gre- gory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline par- allelism for dnn training. In Proceedings of the 27th ACM symposium on operating systems principles, pp. 1-15, 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08866 2023.
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adrià Recasens, Larisa Markeeva, Dylan Ba- narse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alexandre Fréchette, Hanna Klimezak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: diagnostic benchmark for multimodal video models. In NeurlPS, 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Share. Sharegemini: Scaling up video caption data for multimodal large language models, June 2024.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training billio parameter language models using model par- allelism. arXiv preprint 1iXiv:1909900053. 2019.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama, 2023.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. CoRR, abs/2104.09886) 09964, 2021.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.098/8, 2024.
Philippe Tillet, Hsiang-Tsung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10-19, 2019.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: fully open, vision-centric exploration of multimodal 1lms, 2024.
Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longylm: Efficient long video understanding via large language models. CoRR, abs/2404.03384. 2024.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. CoRR, abs/2407.15754 2024.
14

---

Published as a conference paper at ICLR 2025
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In CVPR, pp. 9777-9786, 2021.
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See-Kiong Ng, and Jiashi Feng. Pllava: : Parameter- free llava extension from images videos for video dense captioning. CoRR, abs/2404. 2024.
Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, and Hongxu Yin. X- X-VILA: cross-modality alignment for large language model. CoRR, Abs/2405.19333 2024.
Gyeong-In In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Chun. Orca: A distributed serving system for (Transformer-Bassee generative models. In 16th USENIX Sympo- 2022.
sium on Operating Systems Design and Implementation (OSDI 22), pp.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet- qa: A dataset for understanding complex web videos via question answering. In AAAI, pp. 9127 9134, 2019.
Haoji Zhang, Yiqin Wang, Yansong Tang. Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash- vstream: Memory-basch real-time understanding for long video streams. CoRR, abs/2406.08085, 2024a.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language vision. CoRR.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language vision.
arXiv preprint arXiv:2406. 106006666855 /6852, 2024b.
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chun- yuan Li, Alexander Hauptmann, Yonatan Bisk, and Yiming Yang. Direct preference optimization of video large multimodal models from language model reward, 2024c.
Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond llava-hd: Diving into high-resolution large multimodal models. CoRR, abs/2400.08887, 2024d.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences scaling fully sharded data parallel. arXiv preprint arXiv:2304.1123 11277 2023.
Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle in video haystack: A scalable synthetic framework for benchmark- ing video milms. arXiv preprint, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional vidcos. In AAAI, pp. 7590-7598, 2018.
Zilin Zhu. Ring flash attention, 2023. Accessed: 2024-07-28.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 LONGVILA-CAPTION
We have developed long video captioning benchmark, videos, with captions generated as detailed in Section and verified through human examination. In line with the methodology of VideoChatGPT (Maaz al. 2024), we evaluate the predictions of each model based on their correctness, detailed orientation, and contextual understanding. For instance, we assess correctness by employing GPT-4 to predict scores using a specific prompt. Ad- ditionally, present two examples in Figures 1 and 14 featuring long videos in sports and tech- nology. These examples demonstrate that Long VILA, with its capability to process more frames, offers more comprchensive understanding of videos compared to its short-framm counterpart.
Long WalioonggVL-Caatiiii consisting of 100 long
the performance metrics for the LongVILA models being trained and evaluated varying numbers of frames: 8. 128, and 256. As the number of frames increases, the model's 's per- formance improves significantly. Specifically, the average scores rise from 2.00 to 3.26, highlighting the model's enhanced capability in generating accurate and rich captions with frames.
The
Iteration time (seconds) the dataset (Chen et al. (2024)) with and with- our two-stage sharding strategy.
Table 5:
Table Evaluation of LongVVIIA- performance across different frame counts.
Frames
Correctness
Detailed
Contextual
Average
2 GPUs
4 GPUs
8 GPUs
8
1.87
1.85
2.27
2.00
one-stage
0.78
0.89
1.20
128
2.36
2.79
2.53
two-stage
0.77
0.86
1.12
256
3.23
3.11
3.43
3.26
P2P
A2A
Head Parallel
00
90
90
Rack 0
Rank 2
47
97
Rack
97
96
Rank
43
96
92
Rank 2
Rank
Rank
45
Rerok
Rank 3
93
Rank
99
47
94
2v0 Avy Rv2 2vd 1od AvS kvó lw/
AvO Rv? Jur Avó kv2 kus kvd Avst
AvO kvl Juó kv7 kv2 kud Ju4 Avi
(a) RingAttention
(b) Zigzag RingAttention
(c) 2D-Attention
Figure 11: Comparison of RingAttention Liu et al. (2023a),
Zhu 2023) and
2D-Attention (Fang & Zhao (024).. The blue block indicates communication between QKV, while the black frame represents local attention computation within each SP group rank. The sequence length 8 and the global SP degree is Due the triangular structure causal attention computa- tions, RingAttention experiences computation imbalance, where rank 0 becomes idle after the first round while rank 3 continues computing through all stages. ZIGZAG-RINGATTH addresses this by reordering input sequence tokens along the sequence dimension to achieve load balance. The 2D- Attention mechanism uses a ring parallel degree effective global sequence parallel degree of 4. This approach also incorporates workload balanc- ing strategy within the ring- based process group and uses the All-to-All operation to distribute QKV tensors across devices based on the head dimension, ensuring efficient and balanced computation.
and a head parallel degree of 2, resulting in an
16

---

Published as a conference paper at ICLR 2025
7
LongVILA (DP=8)
L (DP=2, SP=4(2U+22)
5.25
Loss
1.75
0
100
200
300
400
500
600
Train Step
Figure 12: Convergence evaluation. compare the training loss curves for LongVILA VILA with and without sequence parallelism. This figure illustrates the convergence of the VILA model on 8 H100 GPUs, with sequence parallelism degree of 4, compared to pure data parallelism training. 2U+2Z indicates that 2D-Attention mechanism is enabled, where both the Ulysses and Zigzag- Ringttn degrees are The training dataset is shot2story. The two curves align closely, indicating that MM-SP system does not negatively impact training quality.
Table 7: Our training system, used in conjunction with FSDP (Zhao et al. 2023) Zero-3 Ra-- jbhandari et al., 2020)). on 32 H100 GPUs. We found that FSDP offers more efficient memory management, which led us to select it as our default configuration. (Time iteration, seconds).
Sequence Length
Zero-3
FSDP Ulysses
Ulysses
Attention
2D Attention
320 K
0OM
OOM
OOM
23.57
10.70
11.12
288 K
0OM
OOM
OOM
20.24
8.68
8.65
256 K
0OM
OOM
OOM
17.54
6.98
7.04
224 K
19.04
7.06
5.73
15.22
5,47
5.53
192 K
13.01
4.24
4,38
12.97
4.15
4.24
160 K
10.73
3.09
3,23
10.83
3.02
3.11
128 K
8.63
2.16
2.30
8.38
2.07
2.17
96 K
6.49
1.43
1.53
6.35
1.33
1.41
64 K
4.40
1.01
1.08
4.25
0.76
0.80
32 K
2.06
1.58
1.04
2.26
0.39
0.40
Table Training system throughput comparison on 64 H100 GPUs, measured in time per iteration (seconds). Ulysses is not included in this comparison as it supports only up to 32 GPUs.
Sequence length
Megatron-LNN
Ours
CP
CP-8+TP-8
ZIGZAG-RINGATTH
Ulysses
2D Attention
640 K
OOM
OOM
88.4
OOM
578 K
OOM
OOM
77.2
16.9
512 K
OOM
00M
66.1
13.31
448 K
OOM
OOM
57.5
10.39
384 K
OOM
OОM
48.6
7.80
320 K
OOM
OOM
40.5
5.63
256 K
OOM
5.31
32.2
3.93
192 K
8.81
3.10
24.1
2.49
128 K
7.10
1.57
16.0
1.36
64 K
3.09
0.61
8.04
0.57
32
1.86
0.44
4.24
0.33
17

---

Published as a conference paper at ICLR 2025
Table 9: Comparison with state-of-the-ar methods (Li ct al, 2023b Dai et al.
2023 Bai et al,
2023
Liu et al. 2023b Lin et al. 2023b Liu al. 2024b; Tong et al.
2024 Li ct al.
(024c) on 10
image based VLM benchmarks. S3 refers to the stage 3 model in Long VILA training pipeline.
Method
LLM
Rex.
VQA
GQA
VizWiz
SQA!
VQA"
MMB
MMB
SEED
LLaVA
MM-V
BLIP2
Vieume 13B
224
41.0
41
19.6
61
12.5
46.4
38.1
22.1
IndructRLIP
Vicuna-71
224
49.2
34.5
60.5
50.1
23.7
33,4
60.9
26.2
13B
Qwen-V VL
Qwen-7B
448
78.8
59.3
35.2
67.1
63.8
38.2
1.4
56.3
Owen Chat
Owen 7B
448
78.2
57.5
38.9
68.2
61.5
60.6
56.7
58.2
LLAVA-1.5
Vicuna-1.5-78
336
78.5
62.0
50.0
64.3
58.1
58.6
VILA
Laana 2-7H
Lama 2-138
336
336
80.8
79.9
63.3
62.3
$7.8
60.6
68.2
73.7
66.6
64.4
68.9
70.3
64.3
61.7
61.1
62.8
69.7
73.0
34.9
38.8
3-88
Cambrian-1-88 1-8B
64.6
71.7
75.9
Mini-Gemini-HH HD-8B
Laana 3-88
1536
64.5
75.1
70.2
72.7
LongVIL.A-3 7-B7 (83)
Qwen2-7B
dynamic
85.4
65.4
65.0
98.5
77,8
83.4
80.9
70.6
77.6
51.7
Table 10: Detailed model complexity analysis LongVILA among various model size, number of frames, parameters, latency (ms), and TFLOPs. We profile LongVILA models into 4 types of components. Image Encoder includes vision tower and mm projector. I.IM Linears include k/q/v/o projectors and linears in decoder layers. LLM Attention is the attention computation. LLM Others include other components, like embeddings, output heads, and normalization layers. We use fp16 data type, Flash-Atiention2 (Dao 2024) on A100 GPU for latency measurement. As the number of frames increases, the FLOPs and latency of LLM Attention grow quadratically, whereas other components increase linearly. LLM Attention represents the predominant computational cost in long video understanding, highlighting the MM-SP system.
Long V-I.----..B
LongVII.A-77
Frames
Context
Metric
Image Encoder
LLM
LLM Attention
LLM Others
Image Encoder
LLM Linears
LLM Attention
LLM Others
Linears
Params
0.44B
1.31B
0.23B
0.46B
6.53B
1.09B
Latency
196.0
109.3
27.0
28.9
201.0
426.2
51.1
42.1
32
6415
TFLOPs
5.13
4.20
1.77
0.75
5.22
20.93
4.13
1.74
Latency
375.2
208.1
80.7
50.9
375.8
831.1
173.8
80.8
64
12719
TFLOPs
10.26
8.33
6.96
1.48
10.45
41.50
16.24
3.46
Latency
740.4
403.2
288.1
97.1
755.1
1642.4
644.9
158.3
128
25327
TFLOPs
20.52
16.60
27.59
2.95
20.89
82.64
64.38
6.89
Latency
1456.0
811.2
1087.7
192.5
1476.3
3308.2
2529.6
325.1
256
50543
TFLOPs
41.05
33.12
109.89
5.88
41.78
164.92
256.40
13.74
Latency
2921.0
1660.9
4359.2
389.4
2980.2
6675.1
10149.3
653.4
512
100975
TFLOPs
82.09
66.17
438.59
11.75
83.57
329.48
1023.37
27.45
18

---

Published as a conference paper at ICLR 2025
Elaborate on the visual and narrative elements of the video detail. <Video>
The video opens with with one of them holding a soccer ball. The next scene shows goalkeeper green jersey diving to save a goal. Following this, there is a clip of player in red jersey falling to the ground, possibly after tackle. The subsequent scene captures player in red jersey running with the ball towards the goal. The video then transitions white jerseys,
of two players in blue jerseys standing green field,
8 frames
wide shot of soccer field with players in crowd the stands. final scene displays logo with the text "BUNDESLIGA" and a call to action to "CLICK HERE TO SUBSCRIBE!"
The video begins with soccer jerseys seen, the ball being passed around occasionally kicked. The camera focuses on different players, including one in blue jersey who scores goal. The scene shifts to celebration with player in blue jersey holding his hands up, followed by shot of number 10. The game continues with players in various moments, including goals being scored and players running the field. the players are seen hugging each other. The video transitions to series shots that include numbers against white background, followed by more scenes of the soccer game. The players wearing red and white jerseys, and the crowd is cheering enthusiastically. The video also features close-ups individual players moments of celebration, such player jersey being congratulated by teammates. Towards the end, the camera shows a man in black jacket walking on the field, and then cuts to three men standing together. Finally, the video concludes with man in black T-shirt talking to the camera, with red and white logo appearing the
being played field, where players in blue and red
256 frames
and white jerseys, and the camera captures point,
Figure 13: Examples of sports long video caption with LongVILA. For the gameplay opening, the 8-frame baseline describes only static image, two players in only blue jerseys. contrast, 256- frame LongVILA describes players in blue and red jerseys passing and kicking the ball. In addition, the 256-frame version also include the detail players hugging emphasizes the celebratory aspects, which is missing in the 8-frame bascline.
19

---

Published as a conference paper at ICLR 2025
Elaborate the visual and narrative elements the video detail. <Video>
The video begins dark screen, followed by series images that highlight different features of Mini Cooper. The first image shows two Mini Coopers side by side, with arrows pointing to specific features such as "Side Mirror Lens," "New Bumper Fascia," "Larger Fog Lights," " and "Functional Brake Ducts." " The cars blue, and the text is white with red arrows. The next image blurred motion shot sides. The following image is close-up of a car's air vent with a blurred background. Another image shows man in black coat and glasses, presumably car expert, speaking inside Mini Cooper, with the interior visible in the background. The subsequent image is blurred shot of white Mini Cooper parked garage. final image is dark screen with the text FULL REVIEW, VISIT: AutoGuide.com
8 frames
Mini Cooper driving on road with snow on
red and white.
The video begins with white car driving on snowy road, with close-ups coat is then introduced, standing next to silver car in garage, talking to the camera. The scene shifts to showroom setting, showcasing two blue followed by discussing features front of laptop. the car's interior and controls, including the dashboard, gear shift, and steering wheel. The video emphasizes the driving experience, with repeated shots of the man talking and gesturing while driving snowy road, with the camera occasionally focusing the car's dashboard, gauges, controls. The sequence includes close-ups detailed view the dashboard. Towards end, the video features man black coat discussing the car's features in front of white car parked in garage, highlighting the headlights, wheels, and the car's interior. The video concludes with the man standing next to the white dimly lit garage, continuing his dialogue with the
silver car's ignition button and transitions
shots of front, rear, and side. A man in black
256 frames
in black coat narrative continues with man driving, detailing
car's ignition button, gear shift, and
Figure 14: Examples of technology long video caption with LongVILA. At the beginning of cap- tions, the 8-frame baseline only describes static image and two cars. In contrast, the 256-frame LongVILA describes the car snowy road, covering front, rear, and side views. For details, the 256-frame LongVILA describes close-ups of ignition button, gear shift, and dashboard elements, which are missing in the 8-frame baseline.
20


Reviews:
[{"review_id": "FuVa3dcOds", "reviewer": "Reviewer_DDGN", "summary": "The paper proposes a novel long-context Multi-Modal Sequence Parallelism (MM-SP) system specifically designed to enhance the performance of Vision-Language Models (VLMs) when processing extended contexts, increasing the window size from 8 to 1024 frames. The authors demonstrate how their approach can effectively manage and leverage longer input sequences, aiming to address limitations in current VLMs regarding video processing capabilities.", "questions": "- If the authors were to mix the stage 3 with stage 5 at the end, which is stage 1-2-4-(3&5), what anticipated effects on performance or learning dynamics might arise? Would the integration lead to any synergies or drawbacks?\n- How would the performance change if context extension were prioritized by using a long-window LLM first, followed by stages 4-1-2-3-5?\n- How would the performance change if you conbine these two stratgies, which is 4-1-2-(3&5)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The introduction of MM-SP is a significant contribution to the field, as it creatively combines techniques for managing long-context sequences in a multi-modal setting. This approach addresses a gap in existing models that struggle with extensive video inputs.", "weaknesses": "- The evaluation primarily utilizes a single benchmark (VideoMME), which may not sufficiently demonstrate the capabilities of the proposed system in handling long videos. It would be beneficial to incorporate additional benchmarks focused on long video analysis, such as those introduced in recent works (e.g., arXiv:2406.09367 and arXiv:2406.14129) to validate the robustness of the findings.\n- While the authors mention the ability to process 1024 frames, the results are only provided for 256 frames (Figure 2(a)). It would strengthen the paper to include performance metrics for 512 and 1024 frames across the discussed benchmarks to fully substantiate the claims made.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wgDm9qQMHL", "reviewer": "Reviewer_7Lyg", "summary": "The paper introduces LongVILA, a comprehensive framework aimed at enhancing long-context video understanding capabilities in multi-modal foundation models. To optimize model training for long videos, LongVILA introduces two pivotal stages: long context extension and long video supervised fine-tuning. Considering the computational and memory challenges associated with training on long videos, the authors present the long-context Multi-Modal Sequence Parallelism (MM-SP) system. This system significantly improves training efficiency, enabling the processing of context lengths up to 2 million on 256 GPUs without relying on gradient checkpointing. The results demonstrate a remarkable increase in performance metrics, with the long video captioning score rising from 2.00 to 3.26 and achieving 99.5% accuracy in video needle-in-a-haystack tasks. Additionally, LongVILA shows strong performance on the VideoMME benchmark, with scores of 57.5% and 60.6% without and with subtitles, respectively. Overall, LongVILA presents a significant advancement in the realm of long-context visual-language models, providing both theoretical contributions and practical solutions to the challenges posed by long video understanding.", "questions": "Refer to weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written and easy to understand. The figure illustration and captions are informative.\n* The authors provide a full-stack design for long-context Vision-Language Models, including both training curriculum and system implementation. These contributions are significant in the multimodal foundation model community.\n* The proposed model, LongVILA, presents strong performance on VideoMME and other long video understanding tasks.\n* The Multi-Modal Sequence Parallelism design can greatly reduce the memory cost in both training and inference.", "weaknesses": "* The authors do not seem to provide performance metrics on general video understanding benchmarks, such as the Perception Test [A] or EgoSchema [B], after improving long video understanding capabilities. It is worth noting whether the performance on general benchmarks is affected after enhancing long-context capabilities.\n* The proposed Multi-Modal Sequence Parallelism design is interesting and effective. However, it is not clear whether the source code will be released, which will be beneficial to the community.\n\n---\n\n[A]Patraucean, V., Smaira, L., Gupta, A., Recasens, A., Markeeva, L., Banarse, D., ... & Carreira, J. (2024). Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36.\n\n[B]Mangalam, K., Akshulakov, R., & Malik, J. (2024). Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ims2ZESiO", "reviewer": "Reviewer_MTsu", "summary": "This paper proposes LongVILA, a solution for long-context visual-language models. The authors incorporate two additional stages, i.e., long context extension and long video supervised fine-tuning. A long-context multi-modal sequence parallelism system is proposed to efficiently parallelize long video training and inference.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper implements a five-stage training curriculum: multi-modal alignment, large-scale pre-training, short supervised fine-tuning, context extension for LLMs, and long supervised fine-tuning.\n2. An efficient and user-friendly framework (multi-modal sequence parallelism) is proposed to support training and inferencing\nmemory-intensive long-context VLMs.\n3. The presentations are good.", "weaknesses": "1. I suggest the authors to involve more baselines in Table 3. For example, PLLaVA, Flash-VStream, Video-LLaMA-2.\n2. This paper has not reported results on some (long) video QA benchmarks, for example, MoVQA, ActivityNet-QA, Ego-Schema, MV-Bench. I suggest the authors to include them.\n3. Is there any model complexity analysis? For example, #Params, GFLOPs.\n\n\n[1] PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning\n\n [2] VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs \n\n[3] Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)