Matryoshka Multimodal Models

## Abstract

Abstract Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in visual-linguistic reasoning. These models first embed images into a fixed large number of visual tokens and then feed them into a Large Language Model (LLM). However, this design causes an excessive number of tokens for dense visual scenarios such as high-resolution images and videos, leading to great inefficiency. While token pruning and merging methods exist, they produce a single-length output for each image and cannot afford flexibility in trading off information density v.s. efficiency. Inspired by the concept of Matryoshka Dolls, we propose M 3 : Matryoshka Multimodal Models , which learns to represent visual content as nested sets of visual tokens that capture information across multiple coarse-to-fine granularities. Our approach offers several unique benefits for LMMs: (1) One can explicitly control the visual granularity per test instance during inference, e.g. , adjusting the number of tokens used to represent an image based on the anticipated complexity or simplicity of the content; (2) M 3 provides a framework for analyzing the granularity needed for existing datasets, where we find that COCO-style benchmarks only need around 9 visual tokens to obtain an accuracy similar to that of using all 576 tokens; (3)
Our approach provides a foundation to explore the best trade-off between performance and visual token length at the sample level, where our investigation reveals that a large gap exists between the oracle upper bound and current fixed-scale representations.

## 1 Introduction

Large Multimodal models (LMMs) [, , , , , , ] have shown strong performance in visual-linguistic understanding and reasoning. Models such as LLaVA [, , ]
first embed the input image with a fixed number of visual tokens, and then feed them as prefix tokens to a Large Language Model (LLM) [, ] to reason about the input image. Similar model designs are borrowed in video LMMs [, ], where each frame contributes a fixed number of tokens to form the final video representation.

In reality, the number of visual tokens can be prohibitively large in the case of high-resolution images, and even more so for long videos. Existing works [, , , ] mainly tackle this issue by increasing the input context length and consequently, feeding a large number *e.g., *3-8k of visual tokens into the LLM. This approach has a couple of significant drawbacks: (1) the extremely long context makes both training and inference inefficient; (2) an excessive number of visual tokens can actually *harm* the LMM’s performance, distracting it from attending to the relevant information, as we show in Sec. 4.3.
Several recent works [, , ] use heuristics to prune and merge visual tokens to reduce the sequence length. However, they produce a single-length output and *do not afford control over the final sequence length*, which could be useful to trade information density versus efficiency while accounting for resource constraints in the deployment phase.

Figure: Figure 1: Matryoshka Multimodal Models. We enforce the coarser set of visual tokens $\mathbf{X}_{S_{i-1}}$ to be derived from the finer level of visual tokens $\mathbf{X}_{S_{i}}$. As a result, the granularity of Matryoshka visual tokens gradually changes in a controllable manner. The image is from MSCOCO [] validation set.
Refer to caption: x1.png

Images and videos naturally exhibit a hierarchical structure from coarse to fine details, and our human visual system has evolved to recognize visual information in this coarse to fine manner, as shown by biologists and psychologists decades ago [, ]. Can we create a similar structure for LMMs, where within one suite of model weights, the visual content tokens are organized into different scales of granularities? Conceptually, our goal is to learn the visual tokens to have a nested structure, similar to the Matryoshka Doll []. Matryoshka Representation Learning (MRL) [] builds the Matryoshka mechanism over a neural network’s representation vector, where each of the segments with various feature dimensions is capable of handling tasks like classification or retrieval. However, for LMMs, the inefficiency mainly comes from the number of tokens. Thus, inspired by, but different from MRL, our work is motivated to build Matryoshka Multimodal Models upon the *token length dimension*, so that we can flexibly adjust it.

Figure: Figure 2: MMBench evaluation results under M^3, oracle under LLaVA-1.5-M^3, LLaVA-1.5 with average pooling at inference time, LLaVA-1.5 separately trained for each specific scale, and other methods. M^3 shows as least as good performance as LLaVA trained for each specific scale. A large gap exists between the oracle upperbound and model’s actual performance on a specific scale.
Refer to caption: extracted/5762161/figures/scale_plot_submission.png

Specifically, we propose M^3: Matryoshka Multimodal Models, which enforces an LMM to learn a hierarchy of visual representation granularities at the token sequence level, instead of the feature dimension level as in MRL []. With this representation, at inference time, the visual granularity can be *flexibly controlled* based on specific requirements, e.g., to account for the input image’s information density and efficiency constraints. Our training process is simple and straightforward. During training, we encode the image into $M$ sets of visual tokens from coarse to fine, $\mathbf{X}_{S_{i}}$, $i=1,\cdots,M$, where the number of visual tokens gradually increases, *i.e., *$|\mathbf{X}_{S_{i-1}}|<|\mathbf{X}_{S_{i}}|$. And importantly, the visual tokens in a coarser level are derived from the visual tokens in a finer level, *i.e., *$\mathbf{X}_{S_{i-1}}\subset\mathbf{X}_{S_{i}}$, $\forall i$. In this way, the visual information in $[{\mathbf{X}}_{S_{1}},{\mathbf{X}}_{S_{2}},\cdots,{\mathbf{X}}_{S_{M}}]$ gradually includes more fine-grained details. For example, given a natural image as shown in Figure 1, $\mathbf{X}_{S_{1}}$ includes high-level semantics such as the restaurant and girl, while $\mathbf{X}_{S_{M}}$ includes more details such as the Pepsi cup and white paper bag. All other training settings, such as the loss function and model architecture, are kept the same as LLaVA [, , ].

Our approach, M^3, introduces several novel properties and benefits for LMMs. First, our approach can adaptively and efficiently represent visual content. Under one suite of weights, it generates multiple nested sets of visual tokens with different granualarities in information density. This enables flexibility in the number of visual tokens used for any image during inference, enabling control over the best tradeoff between cost and performance based on the image or video content. For example, one can use all visual tokens for images with dense details and use just a few tokens for simpler images. This flexibility can be particularly significant when handling very long visual sequences, such as videos. For instance, given a fixed budget of 2880 visual tokens, a user could represent a video of 2880 frames each with one token or represent the same video by sampling 5 frames each with 576 tokens.

Second, our approach can be used as a general framework to evaluate the visual complexity of vision-language datasets or benchmarks, *i.e., *which level of granularity is needed in order to perform the given task correctly. Surprisingly, we find that most benchmarks, especially those mainly crafted from natural scenes (such as COCO) [, , ], can be handled well with only $\sim 9$ tokens per image. In contrast, dense visual perception tasks such as document understanding or OCR [, ] require a greater amount of tokens ($144-576$ tokens) per image to handle the task well. The detailed findings are presented in Sec. 4.2.

Finally, our approach provides a foundation to tackle a critical task in LMMs: How to use the least amount of visual tokens while answering the visual questions correctly?. Based on the model’s predictions on the test set, we find that compared to full visual tokens, the oracle can use far fewer tokens while performing much better. For example, under six common LMM benchmarks used in LLaVA-NeXT [], the oracle with the trained M^3 model can use as few as 8.9 visual tokens on average to achieve performance that is 8% points better than LLaVA-NeXT which uses 576 tokens per image grid. This indicates that there is a large room for improvement compared to the oracle upperbound, as we show in Sec. 4.2.

To enable further research on adaptive LMMs that learn diverse information granularities, we publicly release our code and models.

## 2 Related Work

Large Multimodal Models.
Large Language Models (LLMs) like ChatGPT [], GPT-4 [], and LLaMA [] have demonstrated impressive reasoning and generalization capabilities for text. The landscape of LLMs has been significantly transformed by the recent introduction of models that also incorporate visual information, such as GPT-4V(ision)[]. Building upon open-source LLMs [, ], a plethora of multimodal models have made significant strides, spearheaded by models like LLaVA [, ] and MiniGPT-4 [], which combine LLaMA’s [] language capabilities with a CLIP [] based image encoder. Recently, LMMs on more tasks and modalities have emerged, such as region level LMMs [, , , , ], 3D LMMs [], and video LMMs [, , ]. However, existing LMMs typically represent the visual content with a large and fixed number of tokens, which makes it challenging to scale to very long visual sequences such as high-resolution images or long-form videos. In this work, we propose to adaptively and efficiently represent the visual content by learning multiple nested sets of visual tokens, providing flexibility in the number of visual tokens used for any image during inference.

Matryoshka Representation Learning. Matryoshka Representation Learning (MRL) [] addresses the need for flexible representations that can adapt to multiple downstream tasks with varying computational resources. This approach, inspired by the nested nature of Matryoshka dolls, encodes information at different granularities within the same high-dimensional feature vector produced by a neural network. The adaptability of MRL extends across different modalities, including vision (ResNet [], ViT []), vision + language (ALIGN []), and language (BERT []), demonstrating its versatility and efficiency. Recent work [] extends MRL to both the text embedding space and the Transformer layers space. Our approach is inspired by MRL, but instead of learning multiple nested embeddings for a high-dimensional feature vector, we learn * nested visual tokens along the token length dimension* for the visual input.
We are the first to show that the idea of Matryosha learning can enable explicit control over the visual granularity of the visual content that an LMM processes.

Token Reduction. One of the main causes of inefficiency in recent LMMs is their large number of prefix visual tokens that are fed into the LLM [, ]. The quadratic complexity in Transformers [] is the key issue in scaling the input sequence length for Transformers. Token reduction serves as an effective technique to reduce computational costs in Transformers. Sparse attention methods such as Linformer [] and ReFormer [] conduct attention operations within local windows rather than the full context, thereby reducing the quadratic complexity of the vanilla attention operation. Another notable method is Token Merging (ToMe) [], which utilizes full attention but gradually reduces the number of tokens in each transformer block by selecting the most representative tokens through bipartite matching for the Vision Transformer (ViT). A recent work [] further studies different families of token reduction methods for ViT. However, prior approaches produce a single length output per input image and do not offer multiple granularities over the reduced token sequence. Our M^3 approach instead learns a multi-granularity, coarse-to-fine token representation within the same model architecture and weights, enabling it to easily be adjusted to various computational or memory constraints.

A concurrent work [] shares a similar spirit with our approach, representing an image with varying numbers of visual tokens using a single set of model weights. While their method reformats the visual tokens into a sequential list via transformation layers, we use average pooling to preserve the spatial structure of the visual tokens, demonstrating effectiveness in our experiments.

## 3 M 3 : Matryoshka Multimodal Models

Figure: Figure 3: Architecture of our proposed Matryoshka Multimodal Models. The visual features from CLIP are represented as several groups of coarse-to-fine visual tokens. At test time, users can explicitly control the granularity of the visual features.
Refer to caption: x2.png

Our goal is to learn a Large Multimodal Model (LMM) that represents visual content as nested sets of visual tokens capturing information across multiple coarse-to-fine granularities, so that one can explicitly control the visual granularity per test instance during inference. Here we introduce how we learn a Matryoshka doll-like token sequence.

LMMs such as LLaVA [] typically input a sequence of visual tokens as prefix tokens to the LLM for visual-linguistic reasoning. The visual encoder from pretrained vision-language models, such as CLIP [] and SigLIP [], is typically utilized to project the images into the set of visual tokens.
In particular, the CLIP visual encoder represents an input image $\mathbf{I}$ as an $H\times W$ grid of visual tokens ${\mathbf{X}}_{H\times W}$, where each $\mathbf{X}_{i}\in\mathbb{R}^{C}$ is a $C$ dimensional feature vector. Our goal is to learn nested sets of visual tokens $[{\mathbf{X}}_{S_{1}},{\mathbf{X}}_{S_{2}},\cdots,{\mathbf{X}}_{S_{M}}]$ which encode the visual information in a coarse-to-fine manner. To this end, we enforce ${\mathbf{X}}_{S_{i}}\subset{\mathbf{X}}_{S_{i+1}},\forall i$. Importantly, we do not introduce any new learnable parameters to the LMM. We instead optimize the CLIP visual encoder to learn the nested visual representation directly, and train the ensuing LLM to adapt to the learned nested set of tokens.

For ease of exposition, we consider CLIP-ViT-L-336 [] as the visual encoder, where an image is encoded as $24\times 24$ visual tokens (576 total). We create $M$ sets of tokens e.g., $|S_{i}|\in\{1,9,36,144,576\}$, in which the visual tokens at the coarser level are derived directly from those at the finer level. Specifically, given the initial $24\times 24$ visual tokens, We sequentially apply $2\times 2$ pooling with a stride 2, resulting in $12\times 12,6\times 6$, and $3\times 3$ visual tokens. Finally, we apply $3\times 3$ pooling and get the most condensed single visual token. In this way, the sets of Matryoshka visual tokens can gradually preserve the spatial information in the original tokens while simultaneously forming a coarse-to-fine nested representation.

We train M^3 by averaging the autoregressive next token prediction loss for each scale $S_{i}$ for each image $\mathbf{I}_{i}$. Specifically, given a Matryoshka visual representation ${\mathbf{X}}_{S_{i}}$ for scale $S_{i}$, we maximize the likelihood of the predicted tokens matching the ground-truth answer $\mathbf{X}_{\mathrm{a}}$:

$$ $P(\mathbf{X}_{\mathrm{a}}\mid{\mathbf{X}}_{S_{i}},\mathbf{X}_{\text{q}})=\prod _{j=1}^{L}P_{\boldsymbol{\theta}}(x_{j}\mid{\mathbf{X}}_{S_{i}},\mathbf{X}_{ \text{q}},\mathbf{X}_{\mathrm{a},<j}),$ (3.1) $$

where $\boldsymbol{\theta}$ is the trainable parameters of the model, which includes both the CLIP visual encoder and the ensuing LLM. $\mathbf{X}_{\text{q}}$ denotes the question in text format, $L$ denotes the token length of the ground truth answer $\mathbf{X}_{\mathrm{a}}$, and $\mathbf{X}_{\mathrm{a},<j}$ denotes all the ground truth answer tokens before the current prediction token $x_{j}$, where $j$ denotes the token index during text token generation. We omit system messages for clarity, though they are part of the conditioning. Figure 3 shows our model architecture.

The final objective averages over all $M$ visual token scales:

$$ $\min_{\boldsymbol{\theta}}\frac{1}{M}\sum_{i=1}^{M}-\log P(\mathbf{X}_{\mathrm {a}}\mid{\mathbf{X}}_{S_{i}},\mathbf{X}_{\text{q}}).$ (3.2) $$

With this objective function, M^3 learns nested sets of visual tokens that gradually include more details with increasing scale. For example, in Figure 1, the smaller set of visual tokens describes the whole scene at a high level while the larger set of visual tokens includes more details such as the Pepsi cup. Our training objective affords our model to conduct visual question answering under any granularity during inference. This can be particularly useful in resource constrained applications; e.g., the visual granularity can be flexibly adjusted based on the anticipated simplicity or complexity of the visual content while taking into account compute and memory constraints.

## 4 Experiments

In this section, we first detail the experiment settings in Sec 4.1. Then we show the performance of M^3 on both image-level benchmarks 4.2 and video-level benchmarks 4.3. Finally, we analyze the behavior of Matryoshka Multimodal Models and provide ablations in Sec 4.4 and  4.5.

### 4.1 Experiment Settings

#### Model

We use LLaVA-1.5 [] and LLaVA-NeXT [] as the base LMMs, both with Vicuna 7B as the language model backbone. We finetune the whole model using the exact visual instruction data from LLaVA-1.5 and LLaVA-NeXT, respectively. The learning rate of LLM is $2\times 10^{-5}$ and $1\times 10^{-5}$, respectively for LLaVA-1.5 and LLaVA-NeXT. The learning rate for the visual encoder is $2\times 10^{-5}$ for both models. We train both models for 1 epoch using 8 NVIDIA H100 GPUs.

Instead of training the language model from scratch, we initialize the language model weights from pre-trained LLaVA-1.5 and LLaVA-NeXT, which we empirically works better. We name our Matryoshka Multimodal Models LLaVA-1.5-M^3 and LLaVA-NeXT-M^3.

#### Visual Token Scales

We design 5 scales for the visual tokens. LLaVA-1.5 [] and LLaVA-NeXT [] both leverage CLIP-ViT-L-336 [] as the visual encoder, where an image is embedded into $24\times 24$ visual tokens. We gradually apply $2\times 2$ pooling with stride 2, resulting in $12\times 12,6\times 6$, and $3\times 3$ visual tokens, where we finally apply a $3\times 3$ pooling to get the final single visual token. Therefore, the size of Matryoshka visual token sets are $S\in\{1,9,36,144,576\}$, following a nested manner. The efficiency anlaysis on the system level is shown in Appendix B, where M^3 boosts the speed of the LMM prefill process through diminished floating-point operations (FLOPs) and lessens computational memory requirements.

#### Evaluations.

For image understanding, we evaluate LLaVA-1.5 and LLaVA-NeXT on (a) diverse multimodal benchmarks: POPE [], GQA [], MMBench [], VizWiz [], SEEDBench [], ScienceQA [], MMMU [], and (b) document understanding/Optical character recognition (OCR) benchmarks: DocVQA [], ChartQA [], AI2D [] and TextVQA [].

For video understanding, we use both (a) open ended video question answering benchmarks evaluated by GPT-3.5: MSVD-QA [], MSRVTT-QA [] and ActivityNet-QA []; and (b) multi-choice video question answering benchmarks: NExT-QA [], IntentQA [], and EgoSchema [].

### 4.2 Image Understanding

#### LLaVA-1.5- M 3

We evaluate LLaVA-1.5-M^3 on the common multimodal understanding and reasoning benchmarks. Results are shown in Table 1. LLaVA-1.5-M^3 with full tokens maintains the performance of LLaVA-1.5 across diverse benchmarks. More importantly, our approach shows strong performance even with 1 or 9 tokens. Specifically, in MMBench, a comprehensive multimodal understanding benchmark, LLaVA-1.5-M^3 with 9 tokens surpasses Qwen-VL-Chat with 256 tokens, and achieves similar performance as Qwen-VL-Chat with even 1 token. Compared with InstructBLIP [], LLaVA-1.5M^3 with 9 tokens surpasses InstructBLIP-7B and InstructBLIP-13B across all benchmarks. This demonstrates that our model has both flexibility and strong empirical performance under diverse number of visual tokens.

Table: Table 1: Comparison between LLaVA-1.5-$M^{3}$ across various benchmarks under image understanding benchmarks. LLaVA-1.5-M^3 maintains the performance of LLaVA-1.5 while outperforming Qwen-VL and InstructBLIP with fewer tokens.

#### LLaVA-NeXT- M 3

We use the proposed Matryoshka Multimodal Models to finetune LLaVA-NeXT, and compare LLaVA-NeXT-M^3 with SS, which denotes the setting where the LLaVA-NeXT is trained under a Specific Scale of visual tokens also for 1 epoch. We also include the oracle upperbound performance. Specifically, ‘Oracle’ denotes the case where the best tradeoff between visual tokens and performance is picked for each test instance. Specifically, for each test instance, we select the the scale with the fewest amount of tokens but can answer the question correctly. Results are shown in Table 2. Our approach, M^3, is at least as good as SS, while performing better on tasks such as document understanding (TextVQA and ChartQA) and common benchmarks such as MMBench [].

Table: Table 2: Comparison of approaches with the SS baseline and $M^{3}$ across various benchmarks under LLaVA-NeXT []. Here # Tokens denotes the number of visual tokens per image grid in LLaVA-NeXT. SS denotes the baseline model trained with a Specific Scale of visual tokens. M^3 is at least as good as SS, while performing better on tasks such as TextVQA, ChartQA, and MMBench. Oracle denotes the case where the best tradeoff between visual tokens and performance is picked.

Our results also show that dataset level biases towards the visual token scales do exist. For example, ScienceQA maintains consistent performance across all visual token scales. AI2D and MMBench only encounter a small performance drop for even as few as 9 to 1 tokens. On the other hand, dense visual perception tasks such as TextVQA and DocVQA show a significant performance drop with fewer tokens. This analysis shows that M^3 could serve as a framework to analyze the granularity that a benchmark needs.

Furthermore, there is a large gap between the model’s actual performance under full tokens and the upper-bound oracle. This indicates that using full tokens cannot always result in the optimal performance for all samples; i.e., there is a large room of improvement towards the oracle point.

### 4.3 Video Understanding

Following IG-VLM [], we directly conduct zero-shot inference on diverse video benchmarks using LLaVA-NeXT-M^3. Specifically, 6 frames are uniformly sampled over the entire video, then arranged as a collage, which is fed into LLaVA-NeXT along with the question to get the response. Results under LLaVA-NeXT-M^3 and recent video LMMs are show in Table 3.

LLaVA-NeXT-M^3 with full visual tokens again shows comparable performance with LLaVA-NeXT. More interestingly, results indicate that full visual tokens usually *do not lead to the best performance* in video understanding tasks. Specifically, on 4 out of 6 benchmarks, full visual tokens show less desirable performance compared to 720 or 180 visual tokens. We suspect that very long visual context could bring distraction (e.g., too much focus on potentially irrelevant background) to the model’s prediction, where a compact representation of the video focusing on the more relevant information may be more advantageous.

Finally, for most video understanding tasks such as ActivityNet, IntentQA and EgoSchema, with 9 tokens per image grid (45 tokens in total), the accuracy difference compared to full tokens (2880 in total) is less than 1%. This demonstrates that the video questions in these benchmarks usually require very sparse visual information, as the source of such video understanding benchmarks mostly comes from natural scenes, which matches our observation in image understanding benchmarks.

Table: Table 3: Overall accuracy of LLaVA-NeXT-M^3 and recent video LMMs on various video understanding benchmarks. Here # Tokens denotes the overall number of visual tokens across all frames.

### 4.4 In-depth Analysis

#### M 3 shows much stronger performance compared to heuristics based sampling at test time.

A simple way to reduce the number of visual tokens via a training-free way is to conduct heuristic token merging or reduction. In Table 4, we compare M^3 with three training-free approaches: average pooling, spatial sampling, and sequential sampling. M^3 is much more resilient when the number of tokens decreases, while the heuristic based sampling approaches show dramatic performance drop. A visualization of the spatial and sequential sampling is shown in Figure 5.

Table: Table 4: Comparison between M^3, and heuristics based sampling baselines—average pooling, spatial sampling, and sequential sampling—at inference time on MMBench with the LLaVA-NeXT architecture.

#### M 3 serves as a good metric for image complexity.

We extract the response from LLaVA-NeXT-M^3 in the TextVQA benchmark, and show the samples where using visual tokens across different scales can answer the question correctly and incorrectly. Shown in Figure 4, the OCR performance aligns with the complexity of the images, which indicates that M^3 can be utilized as a metric towards sample level complexity.

Figure: Figure 4: TextVQA test samples with correct and incorrect predictions upon different scales. Answers vary with different number of visual tokens. In addition, M^3 can serve as a framework to evaluate the complexity of images.
Refer to caption: x3.png

#### Large gap between oracle and actual performance.

As shown in Table 2, the oracle upper-bound can use very few ($6\sim 64$) tokens yet achieve at least 10% better performance compared to full visual tokens. This suggests that a visual token scale predictor, where the model learns to automatically select the best visual token scale given the input images or both input images and questions, has potential to achieve a better tradeoff. This would be interesting future work.

#### Zero-shot generalization to longer visual sequences.

Here we extend the length of the visual tokens at inference time to study the model’s zero-shot generalization behavior. Results under LLaVA-NeXT are shown in Table 5. Here LLaVA-NeXT-M^3 is trained on $2\times 2$ image grids but evaluated on $3\times 3$ grids. We set the number of visual tokens to be 144 in each image during evaluation. The model obtains a significant improvement in document understanding by 2.12, 1.80, and 4.11 on TextVQA, ChartQA, and DocVQA, respectively, while maintaining the same performance on benchmarks mainly composed of natural scene images. $3\times 3$ image grids with 144 tokens per grid own 1440 tokens, yet achieve similar performance with the default LLaVA-NeXT $2\times 2$ image grids with 2880 total tokens (576 tokens per grid). This indicates it is promising to feed more subimages while making the number of visual tokens within each subimage much smaller.

Table: Table 5: Performance comparison of different image grid configurations with LLaVA-NeXT-M^3.

Figure: Figure 5: Visualization of sequential and spatial sampling. Given $24\times 24$ girds, the visualized cells denote the sampled tokens.
Refer to caption: x4.png

### 4.5 Ablation Studies

We ablate the key designs in M^3, including the sampling method of Matryoshka visual tokens, and training strategy.

#### Matryoshka visual token sampling.

Here we compare three different ways to select the visual tokens for Matryoshka Multimodal Models, including average pooling, spatial sampling, and sequential sampling, which is illustrated in Figure 5. Shown in Table 6, averaging pooling shows better performance than the two alternatives across diverse benchmarks. In general, sequential sampling performs the worst. We hypothesize that this is due to the visual tokens having spatial information, while sequential sampling does not naturally align with the spatial distribution of the visual tokens.

Table: Table 6: Ablation on Matryoshka visual token sampling including average pooling, sequential sampling, and spatial sampling.

Table: Table 7: Performance comparison of training LLaVA-NeXT-M^3 with and without training the LLM across diverse benchmarks. We see a clear drop when freezing the LLM.

Table: Table 8: Impact of (a) initializing the LLM weights from LLaVA, and (b) averaging the loss from all scales vs randomly selecting a scale for each sample during training.

#### Training the entire LMM vs only training CLIP.

Since the nested behavior of Matryoshka visual tokens is learned within the CLIP visual encoder, we next evaluate whether it is necessary to also finetune the LLM. Shown in Table 7, training the whole LLM achieves better performance. This demonstrates that by also training the LLM, the model can better adapt to the patterns of the visual tokens distributed in the Matryoshka manner.

As explained in Sec. 3 and 4.1, we (a) initialize the LLM weights from LLaVA and (b) minimize the loss averaged upon all visual token scales for each sample during training. An alternative choice is to randomly sample a visual token scale. Shown in Table 8, initializing the LLM weights from LLaVA and minimizing the losses over all scales shows consistent performance boost compared to using the vanilla text-only pre-trained LLM weights [] and randomly selecting a visual token scale. Initializing the LLM weights from LLaVA makes the training process of M^3 more stable. By learning all scales at once, the model is forced to learn the nested behavior for each sample, which leads to better performance.

## 5 Conclusion and Future Work

We introduced M^3: Matryoshka Multimodal Models, which learns to represent visual content as
nested sets of visual tokens, capturing information across multiple coarse-to-fine granularities. LMMs equipped with M^3 afford explicit control over the visual granularity per test instance during inference.
We also showed that M^3 can serve as an analysis framework to investigate the visual granularity needed for existing datasets, where we discovered that a large number of multimodal benchmarks only need as few as  9 visual tokens to obtain accuracy similar to that of using all visual tokens, especially for video understanding. Furthermore, we disclosed a large performance-efficiency gap between the oracle upper-bound and the model’s performance.

Our work can be naturally extended to other domains. For example, the long context in a text-only LLM or vision tokens in dense vision tasks can also be represented as nested sets of tokens in a Matryoshka manner. One limitation of our current approach is that we are lacking an effective visual token predictor that can bridge the gap between the oracle and LMM’s actual performance at a specific scale. We believe this would be an exciting next direction of research in this space.

## Acknowledgement

This work was supported in part by NSF CAREER IIS2150012, and Institute of Information & communications Technology Planning & Evaluation(IITP) grants funded by the Korea government(MSIT) (No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration) and (No. RS2022-00187238, Development of Large Korean Language Model Technology for Efficient Pre-training), and Microsoft Accelerate Foundation Models Research Program.

## Appendix A Broader Impact

The broader impact of M^3, a framework with nested visual representations, has potential benefits and risks associated with its deployment and release. Our model is trained using the exact same architecture and data of LLaVA-1.5 [] and LLaVA-NeXT []. All the concerns are same as LLaVA. Specifically, as one example, LLaVA conducts instruction tuning using GPT-4 and GPT-4V generated data. The bias from GPT-4 and GPT-4V would still exist in LLaVA.

## Appendix B Efficiency Analysis

To illuminate the computational benefits conferred by M^3, we employ the roofline-based LLM-Viewer analysis as detailed in []. Our analysis is set within a hypothetical context designed to emphasize the effects of M^3 on processing efficiency in LMMs. We study the LLaVA-1.5 case where a $336\times 336$ resolution image is processed using a CLIP-ViT image encoder, resulting in 576 visual tokens. Accompanied by a text prompt with an assumed number of 30 tokens, the nested visual tokens in M^3 substantially lowers the visual token count. The consequences of this reduction are substantial as outlined in Table 9, detailing the computational costs involved in the LMM prefill process. Notably, M^3 not only boosts the speed of the LMM prefill process through diminished floating-point operations (FLOPs) but also lessens computational memory requirements.

It is crucial to highlight that the advantages of M^3 are not limited to just efficiency improvements. The token reduction approach of M^3 can also enhance other LMM acceleration methods, such as quantization and factorization, as referenced in []. This complementary relationship accentuates the broad potential of M^3 to contribute to a wider array of efficiency-boosting strategies.

**Table 9: Computation Cost Analysis. The development device is Tesla V100 GPU, and time estimated by the roofline model represents the theoretical performance that the hardware can achieve.**
| # Tokens | FLOPs (TB) | Prefill Time (ms) | Total Memory (GB) | Storing Activation (GB) |
| --- | --- | --- | --- | --- |
| 576 | 8.0 | 58.1 | 21.6 | 3.8 |
| 144 | 2.2 | 19.5 | 15.0 | 0.7 |
| 36 | 0.9 | 18.0 | 13.8 | 0.3 |
| 9 | 0.5 | 17.7 | 13.6 | 0.2 |
| 1 | 0.4 | 17.6 | 13.5 | 0.1 |

## Appendix C More Visualizations on Nested Visual Representation

Shown in Figure 6, with more visual tokens, LMMs can discover more details, such as furniture and human attributes. Besides, LMMs can generate higher quality descriptions with more visual tokens, as demonstrated by the OCR capability in Figure 6 (b).

Figure: Figure 6: More visualization examples. With more visual tokens, LMMs can discover more details, and generate higher quality descriptions. The images are from MSCOCO [] validation set.
Refer to caption: x5.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: * **Symbol clarification**: At Line 203, it's stated that $X_{S_i} \subset X_{S_{i + 1}}$. However, if we consider $X_{\cdot}$ as a set of $C$-dimensional features, then this seems self-contradictory because the average of a subset of elements $\frac{1}{n} \sum_j X_{S_{i + 1}, j}$ is not necessarily one of the original elements. Could the authors provide a more rigorous definition about symbols $X_{S_i}$ and, in turn, the definition of Matryoshka property studied in this case?
soundness: 4
strengths: * The paper tackles a meaningful problem in practice. The motivation is coherent and easy-to-follow.

* The method description is mostly clear.

* Extensive experiments with strong results and insightful analysis.
summary: The paper proposes Matryoshka Multimodal Models (M^3), a MLLM framework supporting variable visual token length for flexible inference cost control. The multi-scale visual features are generated in a homogeneous manner by pooling visual encoder output with different kernel sizes, and the same set of weights are jointly trained on all scales, producing a single all-in-one model for easy deployment. Models are constructed in a continual finetuning setting on top of LLaVA and LLaVA-NeXT, and evaluated on a broad range of MLLM benchmarks with extensive ablation studies.
weaknesses: * **Comparison with Flamingo-style (i.e., cross-attention-based) methods**: Despite the popularity of LLaVa-style MLLMs which treat visual tokens as prompt, Flamingo-style MLLMs, which decode text-conditioned salient visual features with cross-attention modules, are also studied as an alternative paradigm in several previous works, e.g., [1, 2]. It's noteworthy that cross-attention alleviates most of the the performance penalty due to long visual sequences by nature, because the visual tokens do not go through the expensive MLP and quadratic-complexity self-attention in the language model part (it may still be quadratic in the visual encoder part though). Even if extensive experiments are infeasible within the rebuttal period, it might still make the argument stronger and benefit future readers if at least some discussions could be included (e.g., related works, theoretical analysis of both methods, FLOPS comparison, what if they are used together).

* **Training cost**: From the description at around Line 226, it seems that *all* scales of *all* images are used for training, which means the training cost could be a few times of single-scale training. A comparison with specific numbers and some discussions on scalability would be very helpful as training data of state-of-the-art MLLMs are approaching billion-scale.

* **Minor writing issues**: In Table 13, the unit of FLOPs should be in T instead of TB (TB=TeraBytes is for memory). Repeated reference item: Zhang et al 2023a / Zhang et al 2023b.

[1] Alayrac, Jean-Baptiste, et al. "Flamingo: a visual language model for few-shot learning." Advances in neural information processing systems 35 (2022): 23716-23736.

[2] Zhang, Renrui, et al. "LLaMA-adapter: Efficient fine-tuning of large language models with zero-initialized attention." The Twelfth International Conference on Learning Representations. 2024.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - How to extend M3 to any number of tokens. Currently, it can only support a set of token numbers (576->144->xxx). If we can enlarge the set, then switching between them will be smoother.
soundness: 3
strengths: - The motivation is clear. Current LMMs need more and more visual tokens to enhance their performance, the study of token reduction is important for efficient LMMs.
- The method is simple and easy to implement. Instead of tuning LLM for accepting varing number of tokens, M3 shows that tuning CLIP also works.
- The main evaluation and ablation analysis confirm M3's effectiveness.
summary: This paper proposed a multimodal LLM M3 supporting varing number of visual tokens, inspired by the Matryoshka Dolls. By changing the nbumber of visual tokens, M3 can understand the visual content at different granularity. The proposed method is simple and easy to follow, which has been verified by the experiments.
weaknesses: - Comparisions with dynamic sampling methods like Token Merging and Chat-Univi [1]. The performance drop is significant when reducing the number of tokens, while dynamic sampling methods like Chat-Univi can even surpass its full token baseline. Besides, M3 can be regarded as a special case of dynamic sampling. I suggest a fair comparison with these methods.
- High-resolution and long video evaluation and comparisons with other works (LLaVA-HD, SPHINX, LLaMA-VID etc.) . Since these tasks usually requires more tokens, M3 may lead to a better trade-off between performance and computation.
- Suggest to add speed and computation cost comparisions on the main paper, instead of the appendix.

[1] Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: - This paper is well-motivated, addressing the important capability of representing visual information at varying levels of granularity. This flexibility enables adjusting the number of visual tokens based on both computational budget and task complexity.
- The proposed method is effective, demonstrating comparable performance with the baseline LMMs (LLaVA-1.5 and LLaVA-Next) while using significantly fewer visual tokens, across benchmarks that do not demand dense visual perception.
- The empirical study offers several valuable insights:
  - A substantial gap exists between the naive use of all visual tokens and the upper-bound performance achieved by selecting the optimal number of tokens for each test instance.
  - Different vision-language tasks indeed require varying numbers of visual tokens to be addressed.
  - Reducing the number of visual tokens does not increase the level of hallucination for M3. 
  - The ablation study compressively compares different token reduction methods and suggests the advantage of average pooling.
summary: This paper addresses the problem of visual representation in large multimodal models (LMMs) with a method called Matryoshka Multimodal Models (M3). M3 sequentially applies average pooling to the initial visual tokens extracted by the CLIP-ViT to obtain visual representations at different granularity levels. During training, the LMM learns to autoregressively generate the next tokens based on visual representations at each granularity level individually. In inference, the level of visual granularity can be adjusted to balance performance and efficiency. Experimental results indicate that, on a number of benchmarks, M3 achieves performance on par with LLaVA-1.5 and LLaVA-Next, while requiring significantly fewer visual tokens. Additionally, M3 provides a tool for analyzing the visual complexity of vision-language benchmarks by assessing the granularity required to arrive at correct answers. The results reveal that dense visual perception benchmarks, such as TextVQA and DocVQA, indeed require a higher number of visual tokens compared to other benchmarks.
weaknesses: - Although M3 can produce visual representations at multiple granularity levels, the number of visual tokens used at inference must be predefined. In other words, the method cannot adaptively adjust the number of visual tokens for different instances.
- The baseline methods used for video understanding are relatively weak. For example, recent 7B-scale VLMs have achieved over 60% accuracy on EgoSchema, while the best baseline in this work only reaches 35.8%. M3 would likely benefit from integration with more advanced video LMMs. It would also be better to explore alternative video encoding methods other than "arranging video frames into a collage".
- The method for obtaining Oracle performance is not clearly explained. In line 323, the authors state, "for each test instance, we select the scale with the fewest tokens that can answer the question correctly." However, it is unclear what happens if the model cannot produce the correct answer at any scale.
- The capability of the LMMs is not taken into account when using M3 as a tool to analyze the visual complexity of vision-language tasks. If the LMMs cannot effectively consume additional visual information, increasing the number of visual tokens may negatively impact performance, even if the task itself requires more visual information.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: None
soundness: 3
strengths: 1. Presentation is clear and concise
2. The method is simple but effective
3. Experiments are well-designed and can support the value of the proposed method
4. The large number of visual tokens in multi-modal LLMs is a significant and urgent problem
summary: This paper focuses on the VLLM domain and proposes to use average pooling to downsample the visual tokens to a hierarchy of scales for efficient computation.
weaknesses: 1. Whereas the paper claims that the method can ''**adaptively** and efficiently represent visual content", I think the word **adaptively** is kind of misleading here as it makes readers feel that the method per se includes some dynamic inference strategy. I can understand that the authors hope to express that the users can tradeoff computation for accuracy, but it is important to make it more clear.


(**This is actually a weakness of mine instead of the paper**) I am not able to precisely evaluate the novelty of the paper because the VLLM domain is altering from day to day and I have not been closely tracing the frontier papers in the token reduction topic, and I hope other reviews can give better novelty judgments.

Meta-review: This paper introduces Matryoshka Multimodal Models, which make large multimodal models more efficient by representing visual content with adjustable token granularity. The strengths include clear motivation, simple but effective methods, and solid experiments showing flexibility and efficiency. Weaknesses are limited runtime adaptability and some gaps in using advanced video models or analyzing task complexity. Reviewers generally appreciated the idea and experiments. Overall, the work tackles an important problem in multimodal models and offers practical solutions. The AC recommends acceptance, as the approach is promising and impactful, though there's room to explore dynamic scaling further.

Decision: Accept (Poster)