LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture

## Abstract

Abstract Expanding the long-context capabilities of Multi-modal Large Language Models (MLLMs) is crucial for video understanding, high-resolution image understanding, and multi-modal agents. This involves a series of systematic optimizations, including model architecture, data construction and training strategy, particularly addressing challenges such as degraded performance with more images and high computational costs . In this paper, we adapt the model architecture to a hybrid of Mamba and Transformer blocks, approach data construction with both temporal and spatial dependencies among multiple images and employ a progressive training strategy. The released model LongLLaVA ( Long -Context L arge L anguage a nd V ision A ssistant) is the first hybrid MLLM, which achieved a better balance between efficiency and effectiveness. LongLLaVA not only achieves competitive results across various benchmarks, but also maintains high throughput and low memory consumption. Especially, it could process nearly a thousand images on a single A100 80GB GPU, showing promising application prospects for a wide range of tasks.

## 1 Introduction

The rapid advancement of MLLMs (Liu et al., , ; Dong et al., ; Chen et al., ) has demonstrated their remarkable capabilities across various applications (Chu et al., ; Yang et al., ; Wu et al., ; Chen et al., ). However, multi-image scenario remain an important yet to-be-explored aspect. In particular, expanding the context of MLLMs to understand longer videos (Zhang et al., ; Cheng et al., ), higher-resolution images (Xu et al., ; Wu & Xie, ), and make decisions based on more historical messages (Wang et al., ; Liu et al., ) is crucial for enhancing user experience (Li et al., ) and further broadening MLLMs’ application scope (Apple, ).

However, extending the context length of MLLMs to improve their usability poses challenges related to degraded performance and high computational costs when processing more images.
To maintain the performance in longer context, some studies (Zhang et al., ; Zhao et al., ) have concentrated on curating long-context training data involving multiple images to enhance performance. Additionally, other research efforts have explored innovative training strategies (Liu et al., ; Zhang et al., ; Li et al., ; Zhang et al., ) to mitigate performance declines.
Regarding the issue of high computational costs, Xue et al. () have made strides in improving multi-node efficiency by reducing communication costs. However, there remains a gap in solutions for accelerating the computation itself when managing longer contexts.

To address the challenges mentioned above, we propose a systematic solution called LongLLaVA, especially using a hybrid architecture for acceleration.
This solution comprehensively optimizes across three dimensions: Multi-modal Architecture, Data Construction, and Training Strategy.

- •
For Multi-modal Architecture, we adopt a hybrid architecture combining Transformer with Mamba and propose an efficient image representation method that applies 2D pooling to compress image tokens, significantly reducing computational costs while maintaining performance.
- •
For Data Construction, we have designed unique formats for different tasks, enabling the model to distinguish between temporal and spatial dependencies among images.
- •
In terms of Training Strategy, we employ a three-stage method for multi-modal adaptation—Single-image Alignment, Single-image Instruction-tuning, and Multi-image Instruction-tuning—to incrementally enhance the model’s ability to handle multi-modal long contexts.

Experiemntal results show that LongLLaVA excels in understanding multi-modal long contexts with high efficiency. It leads in retrieval, counting, and ordering tasks in VNBench (Zhao et al., ) and achieves nearly 100% accuracy with 1,000 images on a single 80GB GPU for Needle-In-A-Haystack evaluation (Zhang et al., ).

Our summarized contributions are as follows:

- •
We introduce LongLLaVA, a solution optimized through data construction, training strategies, and multi-modal architecture, effectively balancing performance and efficiency. To the best of our knowledge, this is the first hybrid architecture for MLLMs.
- •
LongLLaVA demonstrates exceptional performance in multi-modal long-context understanding, excelling in retrieval, counting, and ordering tasks.
In our commitment to transparency and community research, we will open source all models, codes, and datasets associated with LongLLaVA.

## 2 Towards Scaling up the Image Number in MLLMs

### 2.1 The Curse of Image Numbers

The model’s input length increases rapidly with the number of images, leading to issues such as degraded model performance and high inference costs.
With the evolution of MLLM technology, many existing open-source MLLMs have demonstrated capabilities on par with closed-source models in single-image tasks (Bai et al., ; Li et al., ; Zhang et al., ; OpenAI, ; Google, ).
However, when the task expands to multiple images, the capabilities of these models degrade significantly. Whether it is a time-related or semantically-related multi-image task, the models struggle considerably, lagging far behind the performance of closed-source models (Song et al., ). Such capability degradation severely limits the application scenarios of open-source MLLMs, and the open-source community urgently needs a systematic solution to address this issue.

Excessive Input Length.
Aside from the degraded performance, another challenge is managing the excessive input length when processing a larger number of images.
The visual encoder components of MLLMs typically transform each image into a large number of tokens. For instance, the CLIP encoder(^1^11openai/clip-vit-base-patch32) (Radford et al., ) generates 576 image tokens for an image with a resolution of 336 pixels. Representing a three-minute video at 1 FPS would require $3\times 60\times 576=103,680$ tokens.
This results in a rapid increase in computational demand and memory usage, significantly degrading user experience and escalating service costs.
Although some works have proposed compressing image tokens in 2D or 3D formats (Chen et al., ; Zhang et al., ; Xu et al., ), this often comes at the expense of performance. Solely relying on compression methods to accommodate multiple images without substantial performance degradation is insufficient.

High Computational and Memory Complexity.
The excessive input length results in high computational and memory complexity.
The LLM part of existing MLLMs primarily relies on Transformer architectures, whose computational complexity grows quadratically with increases in sequence length. As the number of images (or sequence length) increases, memory usage becomes exceptionally high due to the need to store KV-Cache.
As shown in Figure 1, current models on a single 80GB GPU can theoretically handle up to 384 images using Int8 quantization.
Current efforts mitigate this through techniques like ring attention (Liu et al., ; Zhang et al., ) or sequence parallelism (Xue et al., ), but these solutions introduce greater time overhead, or by switching to the Mamba architecture (Gu & Dao, ; Zhao et al., ) to address these challenges. However, the capability for In-Context Learning (ICL) in multi-image scenarios is indispensable, posing a challenge for Mamba models (Lieber et al., ). Therefore, there is a need for a balanced approach that considers model architecture in multimodal contexts, addressing both the high number of image tokens and reducing computational complexity.

### 2.2 Motivation for Hybrid Architecture

**Table 1: Comparative Analysis of Architectures. A checkmark (✓) indicates that the architecture supports in-context learning (ICL) capabilities, while a cross (✗) denotes a relatively weaker ICL ability. For a more detailed experimental analysis, please refer to Sec. 5.1.**
| Architecture | Compute Complexity | ICL | Representative models |
| --- | --- | --- | --- |
| Transformer | Quadratic | ✓ | Gemma (Team et al., ), LLaMA (Touvron et al., ) |
| Mamba | Linear | ✗ | Mamba (Gu & Dao, ), Mamba-2 (Dao & Gu, ) |
| Hybrid | Quasi-Linear | ✓ | Jamba (Lieber et al., ), Zamba (Glorioso et al., ) |

To balance effectiveness and efficiency in multimodal long-context scenarios, we propose to use a Mamba-Transformer hybrid architecture.
As shown in Table 1, the Mamba-Transformer hybrid architecture (Lieber et al., ; Poli et al., ; De et al., ; Glorioso et al., ) is distinguished by its superior performance on ICL task and other benchmark tests, matching the performance of Mixtral-8x7B (Jiang et al., ).
In terms of efficiency, the Mamba-Transformer hybrid architecture is more efficient due to Mamba’s linear computational complexity. For example, a popular hybrid architecture Jamba’s throughput is typically three times that of Mixtral, which already have utilized sliding window techniques to reduce inference complexity. Jamba only requires 4GB of KV-Cache memory to process a 256K token sequence under 13B activation parameters with bf16 precision, substantially less than comparable models. Moreover, in scenarios using a single A100 80GB GPU with Int8 precision, Jamba can handle up to 140K tokens in context length, significantly surpassing the 80K of LLaMA2-7B (Touvron et al., ) and the 60K of Mixtral 8$\times$7B.

### 2.3 The Benefit of Scaling Up the Image Number

Adopting more images significantly broadens the application scenarios for current MLLMs. We will explore this from two dimensions: Temporal Expansion and Spatial Expansion.

Temporal Expansion.
Understanding the temporal dependencies between images is crucial for a variety of applications. In multi-modal assistants, it enhances real-time recall capabilities, which is particularly beneficial for the elderly (Li et al., ; Loveys et al., ). For mobile agents, it enables more personalized services and improves task planning (Deng et al., ; Li et al., ; Wu et al., ). In the healthcare sector, it assists in anomaly detection in 3D medical videos, thereby reducing diagnostic errors (Bai et al., ).

Spatial Expansion.
When dealing with high-resolution images (Xu et al., ; Dong et al., ) or when detailed understanding of images (Wu & Xie, ) is required, images are often decomposed into sub-images. This process highlights the importance of grasping spatial dependencies among these sub-images. In remote sensing, an increased number of images enhances both coverage and granularity (Guo et al., ; Liu et al., ). In pathology, it minimizes information loss and improves diagnostic accuracy (Sun et al., ; Xu et al., ). In the field of Molecular Learning, it facilitates the processing of complex reactions and the analysis of larger molecular graphs (Zhang et al., ; Le et al., ).

## 3 LongLLaVA: Scaling LLaVA to Longer Context

To address the aforementioned challenges and enhance the model’s adaptability to long-context, multi-image scenarios, we introduce improvements from three perspectives: multi-modal model architecture (Sec. 3.1), data processing protocol (Sec. 3.2), and training strategy (Sec. 3.3).

### 3.1 Multi-modal Architecture

Our multimodal architecture is constructed around three core components inspired by LLaVA (Li et al., ): the Vision Encoder, the Projector, and the LLM.
The primary strategies for adapting to multimodal long-context are predominantly derived from two aspects.

Figure: Figure 2: Architecture of LongLLaVA. The LongLLaVA model is capable of (1) accommodating a variety of multimodal inputs and efficiently processing image tokens via 2D token compression; (2) uniformly managing the preprocessed inputs within its hybrid LLM architecture, which comprises four stacks of hybrid layers, each blending Transformer and Mamba layers in a 7:1 ratio.
Refer to caption: x2.png

Vision Information Processing.
We employ CLIP(^2^22openai/clip-vit-base-patch32) as the vision encoder to encode visual information and a two-layer MLP as the projector to map vision features into the text embedding space suitable for the LLM. Prior to projection, bilinear pooling is applied, reducing the token representation of an image from 576 to 144 by aggregating $2\times 2$ patch units into a single token. This approach effectively conserves training and inference time while maintaining essential spatial relationships between patches. Further details on the effectiveness of this strategy are provided in Section 4.5.

Hybrid LLM Architecture.
Our model employs a hybrid LLM architecture comprising four stacks of hybrid layers, each integrates Transformer and Mamba layers in a 7:1 ratio, as depicted in Figure 2. It also features a Mixture of Experts (MoE) approach in every other layer, utilizing 16 experts and selecting the top-2 experts for each token. RMSNorm (Zhang & Sennrich, ) is used between layers to enhance normalization, although positional embeddings are omitted. The model incorporates Grouped Query Attention (GQA) (Ainslie et al., ) and SwiGLU activation functions (Shazeer, ), similar to other large language models.
The total parameter count of the model is 53B, with activation parameters during inference totaling 13B; we designate this model as LongLLaVA-A13B.
In an effort to make the model more efficient, we have retained only the Expert-0 in the Mamba MoE Layer, thereby constructing LongLLaVA-9B.

### 3.2 Data Processing Protocol

To ensure that the model effectively distinguishes between temporal and spatial dependencies among images in multi-image scenarios and performs well across various tasks, we meticulously differentiated special characters in different scenarios. As shown in Figure 3, these special characters comprehensively address the various relationships between images in different contexts, thereby enhancing the model’s adaptability to diverse tasks.

Figure: Figure 3: Data Processing Protocol for LongLLaVA.We utilized different tokens to distinguish various modal information, and to identify the spatial and temporal relationships within images.

Regular Single and Multiple Images. For this type of inputs, we use <img> and </img> to enclose image tokens, helping the model differentiate between image and text tokens.

Video. For video inputs, to enable the model to understand the temporal relationship between frames, we first use <vid> and </vid> to enclose image tokens. Additionally, we add the special symbol <t> between different frames to represent the temporal dependency between them.

High Resolution Image. For complex single-image understanding that require dividing an image into multiple sub-images, we use \n to separate the main image from its sub-images. For the arrangement of sub-images, we traverse from the top-left to the bottom-right, adding \n between split lines to preserve the relative spatial positions of the sub-images.

### 3.3 Training Strategy

In our training strategy, we implement single-modal and multi-modal adaptations to transform a pre-trained language model into a multimodal long-context model.

Figure: Figure 4: Dataset Taxonomy of LongLLaVA. Replay refers to data sampled from former phase to maintain single-image and dialogue understanding ability. SubImage denotes a constructed dataset for understanding complex single images divided into sub-images. Ins-T. and Align. refer to instruction-tuning and alignment, respectively.
Refer to caption: x3.png

Pure-text Instruction Tuning.
We initially enhance the pre-trained language model’s ability to follow instructions of varying lengths in pure-text contexts. This is achieved using a comprehensive dataset totaling 278k pure text entries from Evol-instruct-GPT4 (Xu et al., ), WildChat (Zhao et al., ), and LongAlign (Bai et al., ).

For multi-modal adaptation, following the Single-image Alignment and Single-image Instruction-tuning stages in LLaVA (Li et al., ), we introduce a Multi-image Instruction-tuning stage to progressively enhance the model’s long-context capabilities.
We adopt progressive training not only for better control of variables but also to increase model reusability (Fu et al., ). The specific dataset usage is detailed in Figure 4.

Stage I: Single-image Alignment.
This stage is to align visual modal features with textual modality. We utilize datasets such as ALLaVA-Caption (Chen et al., ) and ShareGPT4V (Chen et al., ), which comprise approximately 600K high-quality image-caption pairs. During this phase, only the projector is trained while freezing the parameters of the Visual Encoder and LLM.

Stage II: Single-image Instruction Tuning.
This stage aims to endow the model with multimodal instruction-following capabilities. We use datasets like LLaVA-1.5 (Liu et al., ) and Mantis-Single (Jiang et al., ), totaling around 932K high-quality question-answer pairs. Here, only the Visual Encoder is frozen, and the projector and LLM parts are trained. This process ultimately results in the development of LongLLaVA (single image).

Stage III: Multi-image Instruction Tuning.
In this stage, the model is trained to follow instructions in multimodal long-context scenarios. We sample 200K, 200K and 50K data items from Mantis (Jiang et al., ), VideoChat2 (Li et al., ) and ShareGPT4Video (Chen et al., ) respectively. To preserve the model’s single-image comprehension and pure-text dialogue capabilities, we include an additional 200K and 50K data items from the Single-image Instruction-tuning and Pure-text Instruction-tuning phases as the Replay component. Furthermore, to enhance the model’s ability to interpret complex single images segmented into multiple sub-images, we extract 50K data items from the Single-image Instruction-tuning phase, perform padding and segmentation, and divide the original images into sub-images of size $336\times 336$ as the Sub-Image component.
,For multi-modal adaptation, fo

## 4 Experiments

### 4.1 Training Details

For training, we utilize random sampling to concatenate data items into a token length of 176,000, separated by the <eos> token. This approach helps in managing extensive datasets and ensuring diverse coverage of different data segments. Training is executed across three compute nodes, each equipped with eight A800 GPUs, leveraging DeepSpeed Zero-3 as the distributed strategy to enhance scalability and efficiency. We employ a cosine learning rate scheduler with a warm-up rate of 0.03, set the training epoch to 1, and the learning rate to 1e-5. This configuration is designed to balance learning speed and model convergence effectively.

### 4.2 Evaluation Setup

We evaluate the model’s multimodal long-context understanding ability using the multi-image evaluation. For detailed exploration of the basic functionalities through single-image evaluation, please refer to Appendix B. Both LongLLaVA-9B and LongLLaVA-A13B are evaluated using Int8 quantization, a method designed to reduce computational costs while preserving performance. Additionally, the temperature is set to zero to guarantee consistent performance evaluation.
Unless specified otherwise, “LongLLaVA” in the subsequent text refers to LongLLaVA-A13B.

Benchmarks.
Our multi-image evaluation utilizes three multi-image benchmarks: MileBench (Song et al., ) for assessing multimodal long-context scenario performance, and Video-MME (Fu et al., ) along with MVBench (Li et al., ) for video analysis capabilities. Detailed descriptions of these benchmarks are available in Appendix A.

Baselines.
We compare our model against four commercial models: GPT-4V(^3^33gpt-4-vision-preview) (OpenAI, ), GPT-4o(^4^44https://openai.com/index/hello-gpt-4o/), Gemini-1.5-Pro(^5^55gemini-1.5-pro) (Google, ), and Claude3-Opus(^6^66claude-3-opus-20240229),
as well as five open-source models: Phi-3-Vision(^7^77https://huggingface.co/microsoft/Phi-3-vision-128k-instruct), OmChat (Zhao et al., ), LongVILA (Xue et al., ), Video-LLaMA-2 (Cheng et al., ) and VideoChat2 (Li et al., ).

### 4.3 Main Results

As shown in Table 2, LongLLaVA demonstrates superior performance among open-source models on MileBench, even surpassing Claude3-Opus, and particularly excels in retrieval tasks. This highlights LongLLaVA’s impressive capabilities in handling multi-image tasks. Notably, LongLLaVA’s effectiveness is further underscored by its performance on video benchmarks such as Video-MME and MVBench. It shows exceptional results, especially in tasks involving medium to long-length videos, outperforming traditional video models like Video-LLaMA2 and VideoChat2.

**Table 2: Results of Multi-image Evaluation. PFLOPs represents the number of floating-point operations required to infer 128 images. The highest scores for proprietary and open-source MLLMs are marked in bold. Video-MME is evaluated under the settings of without subtitles (w/o subs).**
| Model | PFLOPs | MileBench | VideoMME w/o subs | MVBench |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Temporal | Semantic | IR | Avg. | Short | Medium | Long | Avg. |  |  |  |
| Proprietary Models |  |  |  |  |  |  |  |  |  |  |
| GPT-4V | - | 45.6 | 58.9 | 86.7 | 63.7 | 70.5 | 55.8 | 53.5 | 59.9 | 43.5 |
| GPT-4o | - | 56.2 | 63.5 | 88.8 | 69.5 | 72.5 | 63.1 | 58.6 | 64.7 | - |
| Gemini-1.5-Pro | - | 50.2 | 58.3 | 88.0 | 65.5 | 78.8 | 68.8 | 61.1 | 69.6 | - |
| Claude3-Opus | - | 37.4 | 48.1 | 25.0 | 36.8 | 70.5 | 57.4 | 51.2 | 59.7 | - |
| Open-source MLLMs |  |  |  |  |  |  |  |  |  |  |
| Video-LLaMA2 | 3.71 | - | - | - | - | 55.9 | 45.4 | 42.1 | 47.8 | 34.1 |
| VideoChat2 | 0.24 | 25.5 | 25.5 | 9.2 | 20.1 | 48.3 | 37.0 | 33.2 | 39.5 | 51.9 |
| LongVILA | 3.90 | - | - | - | - | 61.8 | 49.7 | 39.7 | 50.5 | - |
| Phi-3-Vision | 2.68 | 46.9 | 50.0 | 18.7 | 38.5 | - | - | - | - | - |
| OmChat | 3.90 | 51.4 | 52.0 | 34.2 | 45.9 | - | - | - | - | 50.2 |
| LongLLaVA-9B | 0.15 | 47.3 | 46.8 | 45.4 | 46.5 | 52.4 | 42.2 | 36.4 | 43.7 | 49.1 |
| LongLLaVA-A13B | 0.22 | 52.7 | 52.1 | 67.5 | 57.4 | 60.9 | 49.7 | 44.1 | 51.6 | 54.6 |

Remarkably, despite achieving these impressive results, LongLLaVA operates with an order of magnitude fewer FLOPs compared to other models. This efficiency in computational resources not only underscores LongLLaVA’s advanced performance but also its optimization in resource management. These results reflect a significant advancement in the research community’s efforts to close the performance gap with commercial models.

### 4.4 Diagnostic Evaluation of Long-Context MLLMs

**Table 3: Long Context MLLMs’ Atomic Capabilities Analysis using VNBench (Zhao et al., ). PFLOPs refers to the number of floating-point operations required for inference on 54 images, which corresponds to the average number of frames extracted from the dataset videos at 1 FPS.**
| Video MLLM | PFLOPs | Retrieval | Ordering | Counting | Avg. |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| E | I-1 | I-2 | E | I-1 | I-2 | E-1 | E-2 | I |  |  |  |
| Proprietary Models |  |  |  |  |  |  |  |  |  |  |  |
| Gemini-1.5 | - | 100.0 | 96.0 | 76.0 | 90.7 | 95.3 | 32.7 | 60.7 | 7.3 | 42.0 | 66.7 |
| GPT-4o | - | 100.0 | 98.0 | 87.3 | 88.4 | 86.6 | 45.2 | 36.8 | 0.0 | 36.1 | 64.4 |
| GPT-4V | - | 100.0 | 99.3 | 82.0 | 42.6 | 22.8 | 23.0 | 37.6 | 0.0 | 32.4 | 48.9 |
| Open-source MLLMs |  |  |  |  |  |  |  |  |  |  |  |
| Video-LLama2 | 0.85 | 1.2 | 26.0 | 6.0 | 0.0 | 0.0 | 0.0 | 2.0 | 4.7 | 0.7 | 4.5 |
| VideoChat2 | 0.08 | 43.4 | 40.0 | 14.6 | 0.0 | 0.0 | 1.3 | 4.4 | 8.0 | 12.4 | 12.4 |
| LongLLaVA-9B | 0.07 | 98.3 | 57.2 | 96.3 | 24.2 | 57.2 | 24.3 | 24.5 | 21.0 | 26.0 | 44.4 |
| LongLLaVA-A13 | 0.09 | 100 | 73.3 | 100.0 | 37.5 | 35.3 | 34.8 | 36.0 | 23.7 | 28.0 | 52.1 |

Considering that former evaluations cannot adequately capture the abilities of MLLMs over long contexts, we employ a new diagnostic evaluation set, VNBench (Zhao et al., ), to further analyze the atomic capabilities of models in long contexts. VNBench is a benchmark construction framework based on synthetic video generation, encompassing tasks such as retrieval, ordering, and counting.

The results, as presented in Table 3, indicate that LongLLaVA exhibits performance that is on par with leading closed-source models in tasks such as cross-context retrieval, ordering, and technical capabilities, even outperforms GPT-4V. Among open-source models, LongLLaVA also shows its superior performance. This positions LongLLaVA as a prominent contender in the field, demonstrating its advanced capabilities in managing and interpreting long contexts.
To further assess the retrieval ability of LongLLaVA, we conducted multimodal needle-in-a-haystack experiment, the specifics of which are outlined in Appendix C.

### 4.5 Ablation Study

Figure: Table 4: Ablation Study of ◼ multi-modal architecture and ◼ dataset construction. Each strategy builds upon the previous row, except for 1D Pooling. Mile${}^{*}_{avg}$ represents the average score of MileBench. 1D and 2D denote different pooling strategies. #T refers to the token count for one image.
Refer to caption: x4.png

As shown in Table 4, significant improvements were observed across all evaluation sets when using the hybrid LLM architecture, Jamba, with identical data and model parameters, demonstrating its potential in multimodal scenarios.
For token compression, we choose the 2D pooling, which significantly reduces computational load while keeping performance degradation within acceptable limits (less than 2.2%). Compared to 1D pooling, the 2D pooling method, which pools along the height and width directions to obtain a 12x12 token arrangement, yields better results (0.1$\sim$1.5 improvement). As demonstrated in Figure 5, setting 144 tokens per image effectively maintains performance while significantly reducing inference costs, particularly noticeable in the case of SEEDBench.
Regarding data construction, after training on our single-image data, the model achieved a 1.5% accuracy improvement on SEEDBench and 12.3% on MileBench. Subsequent multi-image training led to a further 7.4% increase on MileBench, validating the dataset construction’s effectiveness.

## 5 More Analysis

In this section, we conduct further analysis to understand the inner workings and multimodal long-context capability of LongLLaVA.

### 5.1 On the Motivation for the Hybrid Architecture

We explore the strengths and weaknesses of different architectures in terms of ICL capabilities and inference efficiency, highlighting the balanced advantages of multimodal hybrid architectures that combine the strengths of both. For Mamba, we select Cobra (Zhao et al., ), which, to the best of our knowledge, is the first and only multimodal architecture to use Mamba as its LLM within the LLaVA framework. For Transformer, we choose the 13B parameter LLaVA-1.6, which has inference parameters consistent with LongLLaVA, to enable a more accurate efficiency comparison.

**Table 5: ICL Capability and Efficiency Analysis across Different Architectures. Cobra is the largest Mamba-based LLM to date. TP and Mem. refer to throughput and memory usage.**
| Model | Arch. | Active Param. | #Few-shot of VL-ICL | 100K Token (Efficiency) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 2 | 4 | 5 | Prefill (s) | TP (tokens/s) | Mem.(GB) |  |  |  |
| Cobra | Mamba | 3B | 48.7 | 50.3 | 51.0 | 51.5 | 10.2 | 42.7 | 29.9 |
| LLaVA-1.6 | Transformer | 13B | 50.0 | 52.3 | 54.6 | 58.9 | 34.0 | 14.7 | 79.4 |
| LongLLaVA-9B | Hybrid | 9B | 51.6 | 57.8 | 58.4 | 60.2 | 16.5 | 62.1 | 38.7 |
| LongLLaVA-A13B | Hybrid | 13B | 52.3 | 59.0 | 59.0 | 61.3 | 25.5 | 37.6 | 79.1 |

ICL Analysis. We evaluate the performance on the Matching Image task from VL-ICL benchmark (Zong et al., ) for multi-modal in-context learning. This task’s inputs contain an image pair $x=\{x1,x2\}$, and output $y$
indicates whether a specific relation $r$ holds between them. MLLMs are required to learn the relation
from examples. As shown in the Table 5, both Hybrid and Transformer architectures exhibit rapid performance improvements with the increase in examples, whereas the Mamba architecture shows a slower improvement, confirming its ICL shortcomings.

Efficiency Analysis. We focus on three aspects: Prefill Time (first inference latency), Throughput (next tokens per second), and Memory Usage. We control the input text length to 100K and measure time and maximum memory usage for generating outputs of 1 token and 1000 tokens. Throughput is calculated as $(1000-1)/(time_{1000}-time_{1})$. To better simulate real application scenario, Transformer and Hybrid architectures are evaluated using vLLM framework (Kwon et al., ) with Int8 quantization (Team et al., ; Frantar et al., ). As shown in the Table 5, Mamba architecture has the fastest Prefill Time, highest throughput. The Hybrid architecture achieves 2.5 times the Throughput, 75% of the Prefill Time, and reduced memory usage compared to the Transformer architecture with similar inference parameters.

### 5.2 Scaling Law of the Image Number

With more images processed, it could support more image patches for high-resolution image understanding and more video frames for video understanding. To explore the impact of increasing the number of sub-images and video frames, we evaluate LongLLaVA on the benchmarks V* Bench (Wu & Xie, ) and Video-MME (Fu et al., ) respectively.

Figure: Figure 6: Performance of LongLLaVA with increasing sub-image counts on V^∗Bench
Refer to caption: x5.png

Scale up Number of SubImages.
V* Bench evaluates a model’s ability to locate small objects within large images. As shown in Figure 7, increasing the number of sub-images initially improves model performance significantly, indicating better understanding of image details. However, we also find that further increasing the number of sub-images slightly degraded the performance, suggesting that an excessive number of sub-images may interfere with performance on this task.

Scale up Number of Frames.
Video-MME (Fu et al., ) is a benchmark that tests a model’s ability to extract information from videos. We can see from Figure 7 that as the number of sampled frames increases, the model’s performance on the benchmark improves significantly, reaching its peak when 256 frames are extracted. This indicates that the model can effectively understand and utilize the information contained in the additionally sampled frames to provide better responses.

## 6 More Applications for LongLLaVA

Apart from the long video understanding task introduced in Section 4.3, which demands a prolonged temporal image comprehension ability, we have investigated two additional domains, Healthcare and Science. These areas necessitate the utilization of LongLLaVA’s fine-grained image understanding capability, providing us a broader platform to further probe its potential applications.

### 6.1 Application in Healthcare

**Table 6: Performance of the models on the pathology image understanding tasks.**
| Model | Size | VQA-RAD | PathVQA |
| --- | --- | --- | --- |
| GPT-4V | - | 39.5 | - |
| LLaVA | 34B | 58.6 | 59.1 |
| LLaVA-Med | 7B | 55.5 | 35.9 |
| HuatuoGPT-V | 8B | 63.8 | 59.9 |
| LongLLaVA-Med | 9B | 68.5 | 55.0 |

To evaluate the potential of LongLLaVA in the Healthcare domain, we selected two tasks: Pathology Image Understanding and 3D CT Image Understanding. Initially, we trained LongLLaVA-9B for one epoch using PubMedVision to equip it with basic multimodal medical capabilities, a process which took five hours with eight A800 GPUs. As a result, we obtained LongLLaVA-Med.

#### Pathology Image Understanding.

In the healthcare field, we chose pathological image understanding task, which requires the model to possess fine-grained recognition capabilities and medical knowledge.
We evaluated LongLLaVA using two benchmarks, VQA-RAD (Lau et al., ) and PathVQA (He et al., ). As presented in Table 7, our model proves competitive in comparison to SOTA models (LLaVA-Med (Li et al., ) and HuatuoGPT-V (Chen et al., )), even with less training data.

#### 3D CT Image Understanding.

To evaluate LongLLaVA’s capability in 3D vision tasks, we selected the CT image understanding task. Since 3D CT images can be viewed as a combination of multiple slices of the human body, all slices were converted to RGB format and processed as a multi-image sequence for model interpretation.
We conducted a zero-shot evaluation on the CT-RATE (Hamamci et al., ) validation set, with random selection of varying resolutions from the same patients. The dataset contains 1304 samples, with slice resolutions ranging from 512$\times$512 to 1024$\times$1024, and an average of 690. The number of slices per sample ranges from a maximum of 984 to a minimum of 100, with an average of 300.
Our results in Table 7 indicate that LongLLaVA-Med outperforms the state-of-the-art model by 21.6 points in terms of accuracy, thereby establishing a new precedent in the task of understanding 3D CT images.

### 6.2 Application in Science

**Table 8: On the FIT-RSFG-VQA**
| Model | Size | Acc. |
| --- | --- | --- |
| Zero-shot Setting |  |  |
| LLaVA1.5-7B | 7B | 58.6 |
| GeoChat | 7B | 53.5 |
| LongLLaVA | 9B | 65.2 |
| Fine-tuned Setting |  |  |
| SkySenseGPT | 7B | 79.8 |
| LongLLaVA-RS* | 9B | 82.3 |

In science domain, we focus on geology and deal with understanding remote sensing images.
In the field of remote sensing, models are required to perform Visual Question Answering (VQA) based on high-resolution remote sensing images (Zhou et al., ). We followed SkySenseGPT (Luo et al., ), the latest MLLM in this domain. We selected the latest FIT-RSFG-VQA task from their work, which is designed to evaluate a model’s fine-grained perception capabilities and instruction-following ability in this domain.
As shown in Table 8, LongLLaVA maintains excellent performance among all models. Moreover, after fine-tuning the model on some remote sensing data, its performance improved further, surpassing existing SOTA models.

Since the image resolution in FIT-RSFG-VQA is only 512$\times$512, to showcase LongLLaVA’s capability in high-resolution remote sensing image understanding, we conducted a case study. We extracted two remote sensing images from STAR (Li et al., ), with resolutions of 1024$\times$768 and 3327$\times$4083 respectively.

Figure: Figure 8: A Comparative Study on Remote Sensing Image Understanding using LongLLaVA and GPT-4V.
Refer to caption: x7.png

The results in Figure 8 demonstrate that, after performing subimage segmentation, our model successfully answered VQA questions requiring high-resolution recognition capabilities, whereas GPT-4V failed.

## 7 Conclusion

In this study, we introduce LongLLaVA, an innovative hybrid architecture model that excels in long-context multi-modal understanding. The model integrates Mamba and Transformer blocks, leveraging temporal and spatial dependencies between multiple images to construct data, and employs a progressive training strategy. LongLLaVA demonstrates competitive performance across various benchmarks while ensuring efficiency, setting a new standard for long-context MLLMs.

## Appendix Table of Contents

[sections]
\printcontents[sections]l1

## Appendix A Details of Benchmarks

#### Single-image Benchmarks.

We select seven commonly used evaluations to assess the model’s single-image understanding capabilities. These include:

- •
GQA (Hudson & Manning, ): A benchmark for real-world visual reasoning and compositional question answering.
- •
MME (Fu et al., ): A comprehensive benchmark focused on perception and cognition, from which we use the perception component.
- •
MM-Vet (Yu et al., ): Examines six core visual-linguistic (VL) capabilities and sixteen integrations derived from these capabilities.
- •
ScienceQA (Lu et al., ): Consists of 4,210 questions across various science topics, with detailed annotations.
- •
SEED-Bench-v1 (Li et al., ): Evaluates comprehension across twelve dimensions in both image and video modalities; we use the image set.
- •
MMBench (Liu et al., ): A systematically-designed benchmark across twenty ability dimensions.
- •
MMMU (Yue et al., ): Tests multi-modal models on multidisciplinary tasks requiring university-level knowledge, covering 183 subfields and 30 types of images.

#### Multi-image Benchmarks.

To explore multi-image capabilities, we utilized:

- •
MileBench (Song et al., ): Assesses long-context scenario performance, focusing on Temporal, Semantic, and Information Retrieval (IR) components.
- •
Video-MME (Fu et al., ): Covers 30 sub-fields to evaluate video analysis capabilities. We analyze 128 frames extracted uniformly from each video, independent of subtitles.
- •
MVBench (Li et al., ): Addresses 20 challenging video tasks that are not effectively solved with a single frame.

## Appendix B Details of Single-Image Evaluation

The single-image evaluation aims to explore the model’s fundamental capabilities and the impact of extended long-context training on single-image understanding.

### B.1 Benchmarks

We utilized a series of benchmarks, including GQA (Hudson & Manning, ), MME (Fu et al., ), MM-Vet (Yu et al., ), ScienceQA (Lu et al., ), SEED-Bench-v1 (Li et al., ), MMBench (Liu et al., ), and MMMU (Yue et al., ). These benchmarks assess various aspects of visual understanding and cognitive processing within a single-image context.

### B.2 Comparison Models

For a comprehensive comparison, we selected three commercial models for the single-image evaluation: GPT-4V(^8^88gpt-4-vision-preview) (OpenAI, ), Gemini-1.5(^9^99gemini-1.5-flash) (Google, ), and Claude3-Opus(^10^1010claude-3-opus-20240229).
Additionally, we included five open-source models to broaden the scope of evaluation: LLaVA-1.5-13B (Liu et al., ), LLaVA-1.6-13B (Liu et al., ), Phi-3-Vision-4.2B(^11^1111https://huggingface.co/microsoft/Phi-3-vision-128k-instruct) and OmChat-8B (Zhao et al., ).

**Table 9: Single-image Evaluation. TFLOPs represents the number of floating-point operations required to infer 1 images. The highest scores for proprietary and open-source MLLMs are marked in bold. #Token refers to the token count for one image.**
| Model | TFLOPs | #Token | GQA | MM-Vet | MME^P | MMB | MMMU | SQA^I | SEED${}^{v1}_{img}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Proprietary Models |  |  |  |  |  |  |  |  |  |
| GPT-4V | - | - | - | 67.7 | 1926.5 | 81.3 | 56.8 | 82.1 | 69.1 |
| Gemini-1.5 | - | - | - | 65.8 | 2148.9 | 73.6 | 48.9 | 81.4 | 62.9 |
| Claude3-Opus | - | - | - | 74.2 | 1586.8 | 63.3 | 54.9 | - | 42.0 |
| Open-source MLLMs |  |  |  |  |  |  |  |  |  |
| OmChat | 7.61 | 576 | - | 39.6 | - | 78.8 | 45.9 | - | - |
| Phi-3-Vision | 3.56 | 576 | - | - | - | 80.5 | 40.4 | 90.8 | - |
| LLaVA-1.6 | 11.86 | 576 | 65.4 | 44.9 | 1445.0 | 70.0 | 36.2 | 73.6 | 71.4 |
| LLaVA-1.5 | 11.86 | 576 | 63.3 | 36.1 | 1531.1 | 67.7 | 34.4 | 71.6 | 68.2 |
| LongLLaVA-9B | 1.52 | 144 | 62.2 | 40.5 | 1630.1 | 72.6 | 42.1 | 75.9 | 68.9 |
| LongLLaVA-A13B | 1.52 | 144 | 59.9 | 35.2 | 1523.9 | 63.7 | 39.2 | 73.4 | 65.3 |

### B.3 Results Analysis

As shown in Table 9, LongLLaVA (single image) generally outperforms LLaVA-1.6-13B, despite both models having the same inference parameter size. This advantage is particularly notable in the MMMU benchmarks, highlighting LongLLaVA (single image)’s strengths in handling comprehensive knowledge-based questions. Although LongLLaVA (single image)’s performance is slightly lower compared to some recently emerged high-performing models, it still demonstrates the potential of hybrid architectures in multi-model scenarios. To ensure complete reproducibility of our results, we only focus on four representative public datasets. Additionally, we find that LongLLaVA tends to underperform relative to LongLLaVA (single image). Addressing this issue may require incorporating more single-image data during the Multi-image Instruction-tuning phase.

## Appendix C Multimodal Needle-in-the-aystack Evaluation

Figure: Figure 9: Video-NIAH (Zhang et al., ) evaluated on one A800 80GB GPU.
Refer to caption: x8.png

Using the V-NIAH evaluation framework proposed in LongVA (Zhang et al., ), we conduct a needle-in-the-haystack test to evaluate the model’s performance. As shown in Figure 9, LongLLaVA achieves nearly 100% retrieval accuracy on a set of 1200 images without requiring additional training.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Needle in a haystack benchmark missing? What is retrival rate for LongLLaVa

How the model (in Figure 2) is initialized for later VLM training?

L123-124: ring-attention and sequence parallel are actual same technique with different naming. 

L125: why sp or ring-atten introduce extra overhead? What is the comparison baseline here?

L127: Mama model … ICL capability … indispensable. Any evidence to support this is weakness from mamba arch itself rather than training data? Does this weakness exists in VLM or both LLM and VLM?

The paper titled LongLlava, seems to be like an extension of llava series work. But the architecture and training scheme have been both changed drastically.
soundness: 2
strengths: LongLLaVA can handle up to 1173 images on a single 80GB GPU, showing excellent processing power of handling more images, enabling more spatial and temporal information.

The proposed efficient hybrid architecture improving throughput and reducing memory usage while maintaining good performance in both ICL and VLM benchmarks. 

The enhanced data construction and progressive training strategy guide the model to distinguish temporal and spatial dependencies among images.
summary: The paper presents LongLLaVA, a novel solution to enhance the long-context capabilities. Architecture-wise, it combines Mamba (pseudo attention) and token compression.  It can process >1000 images on a single A100 80GB GPU. Experimental results show good performance in multi-modal long-context understanding tasks, surpassing many models in MileBench and VNBench. It also demonstrates strong application potential in healthcare and science domains
weaknesses: My main concern for the paper is that the motivation is not clearly justified. LongLLaVa aims to enable more frames (tokens) for Vision Language Models (VLM) and thus explores a Mamba-based architecture. However, the reason for choosing a hybrid architecture is confusing. Lines 127-128 mention the Mamba model's in-context learning (ICL) capability as indispensable. Is there any evidence or literature to support that this is a weakness of the Mamba architecture itself rather than a result of training data? Additionally, Cobra in Table 5 only has results with a 3B size model, while other models are 9B–13B. This is not a fair comparison and doesn't convincingly justify the choice of a hybrid architecture.

While supporting over 1,000 images on an A100 GPU looks promising, the performance is not satisfying. LongLLaVa-9B-MoE (with actual parameters of 53B) shows a significant performance degradation compared to LongVILA (VideoMME Average drops from 50.5 to 43.7). The 13B MoE model is also not comparable with LongVA (released in July and cited in the paper but not compared). Furthermore, what is the performance of LongLLaVa on regular VLM benchmarks like DocVQA, ChartQA, and MMMU? The results in Table 4 and Figure 5 are from different models than those in Tables 4 and 5, and the comparison is incomplete.

Furthermore, the design choices are not well-ablated. For example, why does the training pipeline (Figure 4) train the model sequentially? Many previous VLM works like LongVA, VILA, and InternVL mix most data in one stage instead of feeding it sequentially. Another point is the architecture design: why blend transformer and Mamba in a 7:1 ratio and only apply MoE in Mamba? Do other blending ratios hurt the performance or lead to less memory saving? Could you provide ablations and share the thoughts behind the design choices?

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: - What does 1D Pooling in table 4 specifically refer to?
- What does LLaVA-1.5-13B + Jamba in table 4 mean? Is it just replacing LLaVA-1.5-13B’s LLM with Jamba? Are those both trained with the LongLLaVA recipe?
soundness: 3
strengths: - The integration of Mamba and Transformer layers enables LongLLaVA to achieve quasi-linear computational complexity while supporting in-context learning. This design is well motivated to deal with long videos.
- Detailed experimental results, ablation studies, and diagnostic evaluations have been conducted on benchmarks like MileBench, VNBench, and Video-MME, showcasing that LongLLaVA has reasonable performances on multi-image and video tasks.
- LongLLaVA maintains a lower computational cost compared to full-attention counterparts, showing its potential in cost-effective deployment.
summary: This paper presents LongLLaVA, a MLLM designed for efficient long-video understanding. The proposed design applies a hybrid architecture inspired by Jamba that combines Mamba and Transformer layers, with 2D pooling applied for image token compression, and follows a progressive multi-stage training strategy. The paper conducts experiments in various benchmarks, including retrieval, counting, and video analysis tasks, while maintaining efficient resource use on an 80GB GPU for long video inference.
weaknesses: - The use of LongLLaVA-9B from Expert-0 in the Mamba (Jamba) MoE Layer appears unorthodox and lacks sufficient justification. The VLM's capabilities are heavily dependent on its underlying LLM, and it's unclear how well Jamba expert-0 performs as an LLM.
- The evaluation baselines are outdated and not sufficiently competitive. While LongVA-7B is mentioned, it's not directly compared against, and several newer 7B VLMs with superior performance are excluded. Although fair comparisons are challenging due to varying training data mixtures and model backbones, the authors should provide a more comprehensive overview of existing VLMs of similar size, including LongVA-7B at the least.
- More rigorous experimental design is needed to demonstrate that the hybrid architecture doesn't compromise VLM performance. While Tables 4 and 5 show LongLLaVA-9B outperforming LLaVA-1.5-13B and LLaVA-1.6-13B, this comparison is skewed since Jamba already outperforms LLaMA2-13B (and likely Vicuna-1.5) in LLM benchmarks [1]. Additionally, the VLM training data mixtures differ. To validate that the hybrid architecture matches traditional full-attention LLM performance as a VLM backbone, LLMs models in those two architectures with similar capabilities (e.g., Zamba-2-7B [2] and LLaMA3.1-8B) should be trained using the LongLLaVA training recipe.
- The training strategy lacks sufficient ablation studies, particularly regarding the impact of removing replay data.
- Tables require additional context. Tables 2 and 3 should include model sizes for open-source baselines. While PFLOPs indicates model efficiency, model sizes are crucial for understanding expected performance.

[1] Lieber et al. Jamba: A Hybrid Transformer-Mamba Language Model.  
[2] https://huggingface.co/Zyphra/Zamba2-7B

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: - Are there any qualitative examples where this method performs better or worse than the baselines? Are there certain subtasks that this method is particularly well-suited for?
- As a result of pooling are there any tasks that suffer as a result? It could be helpful to include OCR or fine-grained image recognition tasks
- Does this new architecture show improved many-shot ICL performance (https://arxiv.org/abs/2404.11018) as opposed to finetuning?
soundness: 2
strengths: - The motivations are outlined very clearly and the way the authors chose to address the challenges presented makes sense
- Hybrid architecture efficiency analysis
- Ablation of architecture choices
- Multiple model scales presents some scaling analysis
- Many experiments, including additional applications to healthcare and science
summary: The work introduces LongLLaVA, a multimodal large language model that aims to solve challenges related to scaling up the number of in-context images in similar models as well as study design choices in the architecture, data, and training recipe of multimodal LMs. The challenges and motivations are outlined clearly and the authors choose to explore a hybrid architecture inspired by recent work around state space models, token compression through 2D pooling, and various data augmentation strategies. The authors evaluate the work on three multi-image benchmarks and show strong performance against open-source baselines and comparable performance to some closed models, despite being much more efficient to serve. The authors run an ablation study on the architecture choices and showcase additional interesting applications for this new model.
weaknesses: - There is no ablation study showcasing the 3-stage finetuning vs a typical 1-stage finetuning step with all of the data across the three stages mixed in.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Could the authors provide a detailed comparison of system performance, specifically two graphs/tables maximum throughput and latency's relation on one/multiple GPUs, between the hybrid model, Transformer-based model, and Mamba-based model at a similar parameter size? This would give readers a clearer understanding of the efficiency gains and trade-offs associated with the proposed hybrid architecture.

2. I wonder if the authors could further emphasize the differences—such as model architecture, training methods, and other relevant aspects—between this work and Jamba, as well as other hybrid structured models, to better highlight the novelty of this paper.
soundness: 3
strengths: 1.The paper presents a comprehensive set of optimizations, including model architecture, data construction, and training strategy, showcasing a well-rounded and complete approach.

2.It features clear and well-structured illustrations alongside a compelling and coherent narrative.

3.The work addresses the urgent and impactful need for scaling up the number of images and handling long video generation, which is crucial for advancing the capabilities of MLLMs.

4.Extensive ablation studies are conducted, comparing the proposed model with both open-source and proprietary models across various benchmark datasets, including rigorous long-context tests like the Needle-In-A-Haystack evaluation.
summary: The paper focuses on enhancing the long-context capabilities of Multi-modal Large Language Models (MLLMs), essential for tasks like video understanding, high-resolution image analysis, and multi-modal agents.

 The authors address key challenges, such as performance degradation with increased image inputs and high computational demands. They introduce a hybrid model architecture combining Mamba and Transformer blocks, optimize data construction to account for temporal and spatial dependencies, and use a progressive training strategy. 

The resulting model, LongLLaVA (Long-Context Large Language and Vision Assistant), demonstrates competitive performance on multiple benchmarks, while also being memory-efficient and capable of processing nearly a thousand images on a single A100 80GB GPU, highlighting its potential for various applications.
weaknesses: 1.The paper does not provide a comprehensive comparison of maximum throughput and latency profiles between the hybrid-structured model, pure Transformer-based models, and Mamba-based models, leaving a gap in understanding the performance trade-offs.

2.The extension of an existing, widely-used hybrid architecture to a new use case may reduce the perceived novelty of the work, as the contribution could be seen as incremental rather than groundbreaking.(see more details at questions)

Meta-review: Summary: The paper introduces LongLLaVA, a novel multimodal large language model designed to enhance long-context capabilities. It employs a hybrid architecture combining Mamba and Transformer layers and uses token compression to efficiently process a large number of images. LongLLaVA demonstrates strong performance in multi-modal long-context understanding tasks and shows potential in healthcare and science applications.

Strengths:

LongLLaVA's hybrid architecture effectively balances computational efficiency with performance, allowing it to handle a large number of images on a single GPU.

The model shows promising results in long-context understanding tasks and has potential applications in various domains, indicating its versatility and practical value.

Drawbacks:

The paper lacks a clear justification for choosing the hybrid architecture over other potential solutions, which may affect the perceived novelty and necessity of the approach.

The performance comparison with other models is not entirely convincing, with some results showing significant degradation, raising questions about the model's robustness.

There is a need for a more comprehensive evaluation, including comparisons with more recent and competitive baselines, as well as a deeper analysis of the model's performance on various subtasks.

Given the above points, I must reject this work as it does not fully meet the acceptance criteria due to concerns about the clarity of its contributions, the completeness of performance evaluations, and the need for a more rigorous comparison with existing models.

Decision: Reject