MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model

Authors: chaoya  jiang
hongrui  jia
haiyang  xu
wei  ye
mengfan  dong
ming  yan
ji  zhang
fei  huang
shikun  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:29:37

Abstract: This paper presents MaVEn, an innovative Multi-granularity Visual Encoding framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. Current MLLMs primarily focus on single-image visual understanding, limiting their ability to interpret and integrate information across multiple images. MaVEn addresses this limitation by combining discrete visual symbol sequences, which abstract coarse-grained semantic concepts, with traditional continuous representation sequences that model fine-grained features. This dual approach bridges the semantic gap between visual and textual data, thereby improving the model's ability to process and interpret information from multiple images effectively. Additionally, we design a dynamic reduction mechanism by for long-sequence continuous features to enhance multi-image processing efficiency. Experimental results demonstrate that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts.

Main paper: 
 introduction: 
 Current multimodal large models (MLLMs) [37] concentrate on understanding single images [24, 43, 36, 22] , which significantly restricts their ability to interpret and integrate information across multiple images. As shown in Figure 1 , typical scenarios [17] involving multiple images include Knowledge Based VQA, Visual Relation Inference, Multi-image Reasoning and so on. These scenarios present a wide array of practical applications.Present strategies predominantly adopt a data-centric approach, where methods such as those proposed in [1, 15, 41, 2] aim to strengthen the multi-image capabilities of Multimodal Large Language Models (MLLMs) by introducing interleaved image-text data during the pre-training and fine-tuning phases. Although some efficacy has been achieved, training solely based on interleaved data still falls short in many multi-image scenarios. This is primarily because current MLLMs remain fundamentally designed for single-image scenarios. This raises the question of whether the visual feature encoding and bridging methods of MLLMs, originally designed for single-image input scenarios, are suitable for multi-image inputs.Current MLLMs encode visual inputs using either discrete symbol encoding [11, 13, 35, 4] or continuous sequence encoding [7, 27, 34] . For the continuous sequence feature encoding category, the The person in the first picture is Yao Ming. Which of the two people in the second picture is Yao Ming?Image 1: Image 2:Where is the man in the first image positioned in the second image?1. A plate of food, including a sandwich and onion rings, is on a table next to a glass. 2. A plate of food, including a sandwich and onion rings, is on a table next to a glass.In the first image, there is a cola, while in the second image, there are fried onions.LLaVA1.5Yao Ming is the player in the second picture.He is wearing a basketball uniform.The person in left of the second picture is Yao Ming, he is wearing a red jersey.LLaVA1.5The person in the first image is in the center of the second image.The older man in the first image in the left side of the second image.LLaVA1.5Figure 1 : We compared the performance of the classic single-image task trained MLLM LLaVA1.5 [22] and our model in three multi-image scenarios including Multi Image Reasoning, Knowledge Based VQA and Visual Relation Inference. LLaVA1.5 exhibits significant limitations in multi-image scenarios.following issues are present: (1) Excessively Long Visual Feature Sequences: Most current MLLMs utilize linear layers to bridge the visual sequence outputs from Vision Transformers (ViTs) [7] . Given the lengthy encoding sequences of images and the finite context input length of current MLLMs, the extended feature sequence inputs in multi-image contexts result in complex computational overhead and adversely affect model performance.(2) Imprecise Visual Information Encoding: Some MLLMs employ fixed-length latent queries to encode visual features through architectures like Q-Former [18] . While this approach somewhat reduces the length of visual sequences, recent studies [20] suggest that it still does not encode visual information from images with sufficient accuracy. There remains a misalignment with textual representations, leading to the model's confusion regarding visual information.Moreover, recent works [11, 13, 4] have started to explore encoding images as discrete symbol sequences. Compared to complex continuous visual representations, discrete visual representations offer simpler and clearer high-level semantic abstractions, closely aligning with the discrete nature of textual representations. Consequently, discrete visual encoding looks like more conducive to complex multi-image reasoning. However, given that discrete visual representations tend to be coarser in granularity, relying solely on them may overlook fine-grained details within images [40] .In this study, we introduce MaVEn: an effective and efficient Multi-granularity Hybrid Visual Encoding framework. MaVEn utilizes discrete visual symbol sequences to abstract coarse-grained semantic concepts, aiding in multi-image understanding, while traditional continuous representation sequences model fine-grained features to support detailed understanding. Accordingly, we investigate the synergy of multi-granularity visual features within the novel framework, design a dynamic reduction mechanism for long-sequence continuous features to enhance multi-image processing efficiency, and propose a multi-stage model training methodology aimed at improving multi-image comprehension. Experimental results demonstrate that the proposed method effectively enhances the understanding capabilities of MLLMs in complex multi-image scenarios, while also improving performance in single-image contexts. In summary, the contributions of this study include:2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: NA Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
multimodal large language models: 
 Existing Multimodal Large Language Models (MLLMs) [37] typically consist of a visual encoder, a visual interface, and a large language model (LLM). The visual encoder converts visual data into continuous sequence features. The visual interface then maps these features into the LLM's semantic space, allowing the LLM to process visual information. Current research focuses on developing effective visual interfaces. There are two main types: Latent-Query based Models: Used in MLLMs like BLIP-2 [18] and MiniGPT-4 [43] , this approach uses a fixed number of learnable latent vectors as query vectors in an attention mechanism. These vectors interact with visual sequence representations to summarize and integrate visual information, effectively reducing sequence length but potentially losing some visual details. Linear Mapping Models: Used in MLLMs like LLaVA [24] , this method directly maps visual feature sequences into the LLM's text embedding space via a linear layer. This approach retains complete visual information but results in longer output sequences.Efficient visual semantic encoding has become a key research area for MLLMs. Researchers have developed various methods to represent visual information, including: Continuous Visual Encoders: Examples include Visual Transformer (ViT) [7] and Swin-Transformer [27] . ViT segments images into patches and processes them sequentially, while Swin-Transformer uses a sliding window mechanism to capture local structures more efficiently. These methods excel in capturing image details but face challenges in aligning with textual encoding [20] . Discrete Visual Encoders: These methods encode images into discrete sequences similar to text tokens, aligning visual and textual information more closely. Examples include VQ-VAE [35] , VQ-GAN [13] , and SEED [11] . VQ-VAE uses self-supervised learning to create a visual vocabulary from image patches. VQ-GAN combines VQ-VAE with generative adversarial networks to capture semantic information and generate highquality images. SEED, the latest approach, encodes images into discrete visual sequences with one-dimensional causal dependencies, aiming to extract high-level semantics for visual understanding and generation tasks.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: NA Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 As illustrated in Figure 2 , we proposes an MLLM architecture that leverages multi-granularity visual features for enhanced multi-image understanding. Visual images are encoded as both discrete symbol sequences and continuous high-dimensional vector sequences. The discrete visual symbol sequences capture essential coarse-grained visual concepts from the images, while the continuous vector sequences encapsulate fine-grained details. Furthermore, to minimize redundant and irrelevant visual representations in the continuous visual sequences and thereby reduce the input context length in multi-image scenarios, we also introduces a dynamic reduction strategy for visual features, guided by textual semantics.As shown in Figure 2 (a), assume the input to the MLLM is {S, T }, where S = {I 1 , I 2 , . . . , I K } represents a collection of K images, and T denotes the corresponding textual content. For each image I k , k ∈ {1, 2, . . . , K}, we employ both the discrete visual encoder SEED [11] and the continuous visual encoder ViT [7] for encoding.Visual Continuous Encoding: we utilize the Vision Transformer (ViT) model, which is widely adopted by most modern Multimodal Large Language Models (MLLMs). For an RGB image I k with dimensions W × H × 3, the image is partitioned into patches of size p × p, resulting in W p × H p patches. These patches are then encoded by the ViT visual encoder into a continuous visual sequence:V k c = [⃗ v k 1 , ⃗ v k 2 , . . . , ⃗ v k nc ].Here, n c = W p × H p , and ⃗ v k i ∈ R z represents a continuous vector of z dimensions. Subsequently, we utilize the text-semantics-aware patch reduction module (details of which will be elaborated in Subsection 3.2) to select patch features relevant to the input textual content T , thereby reducing the sequence length of V k c , while preserving essential fine-grained information. The reduced feature sequence is denoted as Embeddling Layer an Multi-Layer Perceptron, akin to that used in LLaVA 1.5, as a bridging projector to project V c into the semantic space of the LLM embedding layer.V k c = [⃗ v k p1 , ⃗ v k p2 , . . . , ⃗ v k pm c ], m c ≪ n c .v! " , v" " , … , v# ! " v! ! , v" ! , … , v# ! ! v ! ! , v " ! , … , v #" ! ; v ! " , v " " , … , v #" " ; 𝑡!, 𝑡", … , 𝑡# # v$ $ " , v$ % " , … , v$ &! " v$ $ ! , v$ % ! , … , v$ &!Visual Discrete Encoding: image I k is tokenized by the image discrete tokenizer D v (for details on the design and training of the discrete tokenizer, please refer to the original work [11] ) into a visual discrete symbol sequenceV d = [d k 1 , d k 2 , . . . , d k n d ],whered k i ∈ [1, 2, . . . , N v ].Here, N v denotes the size of the visual discrete encoding vocabulary.Given that text modalities naturally possess a discrete vocabulary, merging the visual discrete vocabulary with the textual discrete vocabulary forms a unified multimodal vocabulary. The advantage of this approach lies in its ability to achieve a unified representation of both visual and textual modalities, effectively addressing the semantic gap between them. Assume that the vocabulary size of the LLM is N , and the vocabulary size of the visual discrete tokenizer is N v . The expanded multi-modal unified vocabulary size thus becomes N u = N + N v . Concurrently, we align each element in V d with the index of the unified vocabulary to obtain the final discrete encoding:V d = [v k 1 , vk 2 , . . . , vk n d ],where vk i = d k i + N . Finally, the weight matrix of the LLM's embedding layer, W , is also expanded from N × z to N u × z. Consequently, the weight matrix of the LLM's embedding layer, W , is expanded from N × z to N u × z. This adjustment enables the embedding layer of the LLM to concurrently encode features from both visual and textual discrete tokens. The embedded representation of V d is denoted as [ vk 1 , vk 2 , . . . , vk n d ] which is output by the expanded embedding layer. Finally, we sequentially insert the continuous visual tokens before the discrete visual token embeddings outputted by the embedding layer. The final visual representation inputted into the LLM is:[⃗ v k p1 , ⃗ v k p2 , . . . , ⃗ v k pm c , vk 1 , vk 2 , . . . , vk n d ].We aim for the discrete visual tokens to abstract high-level, coarse-grained semantics from the images, while the continuous visual tokens complement this with low-level, fine-grained details. However, we found that the continuous visual tokens output by the Vision Transformer (ViT) encompass a considerable amount of redundancy, with many tokens possessing repetitive or superfluous semantics. Consequently, as shown in Figure 2 (b), we propose a continuous visual token reduction mechanism guided by the coarse-grained semantics of discrete visual tokens, aimed at achieving semantic synergy between coarse-grained and fine-grained representations.Firstly, after obtaining the sequence of discrete visual tokensV d = [v k 1 , vk 2 , . . . , vk n d ],we append an <EOS> token to it. This sequenceV d = [v k 1 , vk 2 , . . . , vk n d , t eos ]is then passed through the LLM to obtain the final layer's output hidden state of the <EOS> token denoted as h eos , which represents the global information of the discrete visual tokens:h eos = LLM([v k 1 , vk 2 , . . . , vk n d , t eos ]). we then concatenate the EOS token with each image patch token as ⃗v k i = concat(⃗ v k i , h eos ), where ⃗ v k i ∈ R z , h eos ∈ R z , ⃗ v k i ∈ R 2z , i ∈ {1, 2, . . . , n c }. Then the concatenated patch features ⃗ v k i are EO S 5 6 9 3EO SText discrete Tokens fed to the patch selector. The patch selector is an Multilayer Perceptron (MLP) denoted as F that contains three linear layers and is used to predict the relevant score between patches and the discrete visual tokens. The output of the last linear layer has only one dimension and will be fed to a Sigmoid activation function to predict the relevant score a i with the discrete visual tokens as a i = F(⃗ v k i , h eos ) According to the prediction of patch selector F, the top-m key image patch tokens are kept and the unselected patch tokens which generally have lower relevant scores will be discarded. Finally, we reconstruct the reduced visual sequence asv k = ⃗ v k p1 , • • • , ⃗ v k pm, where m = n c × α, and α is a hyper-parameter and named Keeping Ratio which is used to control the proportion of selected patches to total patches.To train the patch selector, we constructed patch-level pseudo-label annotations based on Grounding SAM [32] (a recent state-of-the-art opentext vocabulary semantic segmentation model). We observed that the high-dimensional semantics encapsulated within the discrete visual tokens are largely consistent with the semantics of the image captions. Inspired by this observation, as shown in Figure 2 (b), we opted to use image captions as a proxy for the high-dimensional semantic abstraction of the image. We employed Grounding SAM to perform text-guided semantic segmentation of the images. After obtaining the semantic segmentation pixel masks, we computed the overlap between each patch and the mask labels. If there is an overlap, the corresponding patch label is set to 1; otherwise, the label is set to 0.The training process of MaVEn is divided into four stages. In the first stage, we utilized image-text datasets like COCO [21] and Visual Genome (VG) [14] to annotate 1 million semantic segmentation masks with textual annotations, based on Grounding SAM [32] . These masks were subsequently converted into patch-level pseudo-labels. Utilizing this dataset, we trained the Patch Selector while keeping other model parameters frozen.In the second stage, we exclusively trained the embedding layer of the LLM (Large Language Model) to adapt to our expanded vocabulary for the LLM. Consequently, we utilized the LLaVA 558k single-image pretraining dataset [24] and the MMC4 interleaved image-text dataset [45] for training. At this stage, we employed only the visual discrete encoding, eschewing the visual continuous encoding, with the aim of adapting the LLM embedding layer to the expanded unified vocabulary. We trained using a cross-modal autoregressive generation task; given an input that might contain images, we obtained tokenized discrete sequences through the text and image tokenizers. This enabled us to generate discrete image token sequences from text discrete token sequences and vice versa.In the third stage, our objective is to optimize the visual projector so that the continuous visual tokens, after being processed by the visual projector, align with the semantic space distribution of the unified multimodal vocabulary embeddings. Therefore, during this phase, we train solely the visual projector. Table 2 : Average results of zero-shot evaluation on each task category of SEED Benchmark [17] .We train using the LLaVA 558K image-text caption dataset, where images are encoded solely as sequences of continuous visual tokens:V k c = [⃗ v k p1 , ⃗ v k p2 , . . . , ⃗ v k pm c] without employing visual discrete coding. The model is required to generate captions for the images based on the visual input.In the fourth stage, we introduce instruction fine-tuning data with the aim of enhancing the MLLM's capability to follow human instructions. During this phase, the MLLM undergoes comprehensive fine-tuning with the LLaVA 665k instruction fine-tuning datasets, unfreezing all model parameters except for those of the visual encoder and patch selector for training and optimization.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment setting: 
 Dataset: We initially generated 1 million pseudo-labels for patch-level text semantic relevance by utilizing the COCO [21] , Visual Genome (VG) [14] , and RefCOCO datasets [38] , following the methodology delineated in Subsection 3.2 and leveraging Ground SAM. These pseudo-labels were subsequently employed to train the patch selector within MaVEn. During the second phase of model training, the embedding layer of MaVEn was refined using the MMC4-core dataset [45] and the LLaVA 558K single-image pre-training dataset [22] . In the third phase, the visual projector component of MaVEn was further trained using the LLaVA 558K single-image dataset. Finally, in the final phase, we fine-tuned the model with the LLaVA 665K instruction fine-tuning dataset.Training Settings: MaVEn utilizes the ViT-L model [31] with a patch size of 14 × 14 and is pre-trained at a resolution of 336 × 336, resulting in a continuous token length of 567 for the encoded image. For image discrete tokenization, SEED [11] is employed to tokenize the image into 32 discrete tokens. For the continuous visual tokens, during patch reduction, we set the Keeping Ratio to 0.25, meaning that only 25% of the continuous tokens are retained. Consequently, the length of the final continuous visual token sequence decreases from 576 to 144, while the length of the discrete token sequence is 32. Ultimately, the entire visual hybrid encoding sequence has a length of 176. The large language model Vicuna [42] , with 7 billion parameters, is used to handle multi-modal features. The AdamW optimizer [28] is used for optimization. During the instruction tuning stage, the entire model is trained for 1 epoch with a learning rate of 2e-5 and a batch size of 256. All experiments was performed using 8 NVIDIA A100 GPUs, each with 80GB of memory. To validate the effectiveness of MaVEn in multi-image scenarios, we evaluated MaVEn's performance multi-image visual understanding and reasoning. Within the multi-image visual understanding context, we assessed the model using DemonBench [17] and SEED-Bench [16] . DemonBench comprises seven scenarios involving multi-image reasoning and understanding, including tasks such as Multi-Modal Dialogue, Visual Relation Inference, Knowledge Grounded QA, and Multi-Image Reasoning. SEED-Bench, on the other hand, encompasses questions related to video comprehension. Additionally, we also tested the performance of MaVEn in single-image scenarios.Results on DemonBench: As shown in the Results on SEED-Bench: Furthermore, we assessed our model on SEED-Bench [16] , particularly focusing on video understanding tasks like action prediction, action Recognition and procedure understanding. As shown in Table 2 The experiments revealed that our approach significantly outperformed existing models like LLaVA 1.5, Otter and so on. For instance, MaVEn exhibited a 12-point improvement over Otter [15] in video understanding (30.4 -> 42.11 ). These findings underscore MaVEn 's effectiveness in multi-image understanding scenarios.We intend to explore the influence of MaVEn on the model's abilities of single image visual comprehension and generation. To achieve this objective, we carried out assessments on common benchmarks, such as Visual Question Answering (VQA) [12, 30, 33] and recently designed MLLMfocused Multi-modal Benchmarks including MME [9] , MMBench [26] , MM-Vet [39] .As summarized in Table 3 . We compared performance of MaVEn to other SOTA MLLMs such as BLIP2 [19] , InstructBLIP [6] , Shikra [5] , and Qwen-VL-Chat [3] .Our experimental results show that our approach can also successfully enhances the performance across a range of single image understanding task. Notably, MaVEn outperforms LLaVA-1.5 [25] in terms of consistency and accuracy across all VQA datasets.We also evaluate MaVEn on four recently popular single-image multi-modal benchmarks in a zero-shot manner. The results of our evaluation are listed in Table 3 . We discovered that after implementing MaVEn, all three models exhibited improvements To ascertain the efficacy of multi-granularity hybrid encoding, we conducted training using solely visual discrete encoding and visual continuous encoding, respectively. We then evaluated and compared the outcomes on both multi-image and single-image evaluation benchmarks. The results are detailed in the Table 4 below. We observed that, compared with utilizing only visual continuous encoding or employing a hybrid of visual discrete and continuous encoding, the model solely on visual discrete encoding exhibits subpar performance in both multi-image and single-image contexts. This underperformance is likely due to the nature of discrete visual feature encoding, which, while capturing the high-dimensional information of the image, forfeits a significant amount of low-dimensional, fine-grained details, resulting in a lossy encoding process. As a result, it fares poorly in tasks like image reasoning and understanding, which demand meticulously detailed information. Moreover, the model with only continuous encoding also do not deliver optimal performance, particularly in multi-image tasks. This further indicates that models based solely on visual continuous encoding are unsuitable for multi-image scenarios.To verify the effectiveness of patch reduction, we compared the length of visual tokens at different Keeping Ratios and analyzed the performance across various benchmarks. We experimented with the Keeping Ratios from 0.1 to 1.0. The experimental results are shown in the Figure 4 . We observed that when the Keeping Ratio was 0.1, the number of visual tokens decreased to 89, a significant reduction from the initial number of patches. However, the model's performance across multiple benchmarks also significantly declined. Therefore, despite the reduction in the number of visual tokens, the performance loss was too substantial, making it an unsuitable final choice. When the Keeping Ratio was 0.25, the number of visual tokens remained relatively low, but the model's performance was more stable. In this case, the model exhibited balanced performance across various benchmarks, effectively reducing the number of visual tokens while maintaining a high-performance level. Therefore, we ultimately chose a Keeping Ratio of 0.25.Selected Patches Discrete Index DistribuJon Relevant score To investigate the semantics of discrete tokens, we randomly selected an index from the visual discrete dictionary and searched the CC/SUB dataset for images containing this index in their encoding. As illustrated in Figure 5 , we randomly chose three images with index = 4568. We discovered that all three images featured the depiction of a snowman, suggesting that index = 4568 can represent high-level semantics such as a snowman or white snow. We provided the distribution of discrete tokens for these three images and observed that the position of index = 4568 in the discrete sequences was also notably consistent.Furthermore, as shown in Figure 5 , we also visualize the relevant score between the patches and the semantics of discrete visual tokens predicted by the patch selector, along with the patch tokens selected based on a Keeping ratio of 0.25. We found that the patch selector tends to choose patches related to the semantics of discrete visual tokens, thereby supplementing the missing low-level fine-grained information of the discrete tokens. The aforementioned findings further validate that multi-granularity hybrid encoding facilitates mutual assistance and synergy between discrete and continuous representations, thereby achieving efficient multi-granularity semantic encoding.To further validate the role of discrete visual tokens in the inference process of multi-image instructions, we visualize the attention weights of the last layer of the LLM. As illustrated in Figure 6 , we have MaVEn compare the commonalities between two images. For inputs using only continuous visual tokens, we observe that the model's attention during inference is primarily focused on text tokens, disregarding visual tokens. This may still be due to the lower semantic granularity of continuous visual tokens, making it difficult to establish direct semantic associations. However, with inputs encoded using multi-granularity visual hybrid encoding, we notice that the model establishes attention associations with discrete visual tokens when answering questions. This indicates that, in multi-image scenarios, discrete visual tokens guide the LLM to focus on visual information during decoding. 
conclusion: 
 In conclusion, we introduces MaVEn, a novel Multi-granularity Hybrid Visual Encoding framework designed to enhance multi-image reasoning in MLLMs. By combining discrete visual symbols for semantic abstraction with continuous sequences for detailed features, MaVEn improves both understanding and processing efficiency. Our dynamic reduction mechanism and multi-stage training strategy further enhance performance. Experimental results confirm that MaVEn significantly boosts MLLM capabilities in both multi-image and single-image contexts. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: NA Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: NA Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: NA Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: NA Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: The reviewers unanimously agree that the proposed approach is both novel and compelling, with some highlighting its significant technical contributions and broad applicability across various scenarios (HJUS, TdkD). The reviewers also acknowledge the thoroughness of the experiments and the clarity of the presentation. Initial concerns were raised regarding the computational cost (9o1r, Rxew), the effectiveness of continuous visual tokens (TdkD), and the adequacy of comparisons with other methods. However, the authors have addressed these concerns comprehensively in their rebuttal. After consideration of the reviews and the authors' responses, the Area Chair finds no remaining critical issues with the paper. The Area Chair agrees with the reviewers on the value of the proposed approach and recommends acceptance of the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes MaVEn, a novel multi-granularity hybrid visual encoding framework for multimodal large language models (MLLMs). MaVEn aims to improve MLLMs' capabilities in multi-image reasoning by combining discrete and continuous visual representations. The authors design a dynamic reduction mechanism to reduce the computational overhead of long continuous sequences. Experimental results demonstrate that MaVEn significantly improves performance on both multi-image and single-image benchmarks.
strength_weakness: strength_weakness: While the MaVEn proposed a novel approach to combine the advantage of discrete and continuous visual info, the system can be a bit over complex for serving / maintenance in real applications.

In addition, the paper didn't discuss the computation complexity of MaVEn and compare it with other existing models.
questions: questions: 1. how does MaVEn compare with other multi-granularity approaches? 
2. Could you elaborate more on the computation complexity and efficiency?
limitations: limitations: Need more discussion on the model's limitation and computational complexity
paper_summary: paper_summary: The paper introduces MaVEn, a framework designed to improve Multimodal Large Language Models (MLLMs) in understanding and reasoning across multiple images. Unlike current MLLMs, which are mainly focused on single-image interpretation, MaVEn integrates both coarse-grained semantic concepts and fine-grained details. This combination bridges the gap between visual and textual data, enhancing the model's ability to process multiple images. The framework also includes a mechanism for efficiently handling long sequences of features. Experiments show that MaVEn boosts performance in multi-image scenarios and also provides benefits for single-image tasks.
strength_weakness: strength_weakness: 1. In Tables 1, 2, and 3, the author does not compare some of the latest methods, such as mini-Gemini, MiniCPM, XComposer, and InternVL.
2. The paper primarily claims its main advantage is in multi-image tasks; however, the author only tests on DEMON and SEED benchmarks. Testing on additional multi-image benchmarks, such as MMBench-Video and MME-Video, would be more compelling.
3. Figure 6 is unclear. In the first figure, I see only two vertical lines on discrete tokens. Does this mean that only discrete tokens have attention weight?
4. The paper does not report the computational complexity compared to other methods.
5. Figure 5 shows that the selected tokens are mostly related to objects. Would it be beneficial to directly use Grounding Sam for token selection? A comparison might be interesting to see.
6. The training pipeline is complex and involves four stages. What is the training cost?
questions: questions: Please refer to the weakness section.
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces MaVEn, a Multi-granularity Visual Encoding framework that enhances Multimodal Large Language Models (MLLMs) in multi-image reasoning by combining discrete visual symbol sequences with traditional continuous representation sequences. Experimental results show that MaVEn significantly improves MLLMs' understanding in complex multi-image scenarios and boosts performance in single-image contexts.
strength_weakness: strength_weakness: see questions.
questions: questions: I am not an expert in this domain. But I do have two questions for authors:

1. Why not directly operate on continuous features to reduce feature redundancy since the approach doesn't learn from coarse to fine, like gating?

2. It's not quite clear how discrete tokens help non-single image understanding. The obvious advantage of using discrete tokens is to improve efficiency only?
limitations: limitations: n/a
paper_summary: paper_summary: This paper presents a Multi-granularity Visual Encoding framework (MaVEn) for better multi-image reasoning. MaVEn combines discrete visual symbols and continuous representation sequences, as well as designing a dynamic reduction mechanism to efficiently and effectively process and interpret information from multiple images. Experimental results demonstrate its effectiveness in various multi-image benchmarks.
strength_weakness: strength_weakness: 1. In Stage 3, would the adjustments to the Visual Projector affect the performance of the Patch Selector, since that component has been frozen?
2. The steps of training the patch selector using Grounded SAM annotated data seem a bit redundant. Directly selecting patches based on the similarity between the patch and the discrete token may be a simpler and more effective approach.
3. The role of the continuous tokens has not been well validated. In Figure 6, the attention seems to barely focus on the continuous tokens. Does this suggest that the continuous tokens have little impact on the performance, and they could potentially be discarded? Is it possible that the current evaluation design is unable to fully reflect the role of the continuous tokens? 
4. In line 202, '567' should be '576'.
questions: questions: Overall, I appreciate this paper, but some of the concerns raised in the 'Weaknesses' part, especially Q3, have prevented me from giving a higher score.
limitations: limitations: None.
paper_summary: paper_summary: The paper presents a novel approach, termed MaVEn, which aims to enhance the performance of MLLMs in multi-image scenario by integrating discrete visual symbol sequences with traditional continuous representation sequences. This dual strategy is designed to bridge the semantic discrepancies between visual and textual information. The approach also incorporates a dynamic reduction mechanism for long-sequence continuous features, aiming to boost processing efficiency in scenarios involving multiple images.
strength_weakness: strength_weakness: To solidify the claims regarding the efficacy of the discrete visual symbol sequences used in MaVEn, it would be recommended to conduct experiments comparing the performance of these different discrete representation techniques, such as VQGAN[1] or VQVAE[2].
[1]Taming Transformers for High-Resolution Image Synthesis.
[2] Neural Discrete Representation Learning.
questions: questions: The manuscript presents a well-structured and insightful study. I currently have no further questions.
limitations: limitations: yes.

Comments: title: Thank you very much to the authors for the detailed response.
Response to Reviewer HJUS
Gentle Reminder:   Review Our Response and Engage in Further Discussion
Thank you very much for your reply.
comment: Thank you for taking the time to read and consider our rebuttal. We greatly appreciate your positive feedback and are pleased to learn that you are now leaning towards a weak accept. We value your expertise and the insights you have provided throughout the review process. Your constructive comments have helped us identify areas for improvement and have contributed to enhancing the quality of our work.
We sincerely appreciate you taking the time to read our rebuttal and for your positive feedback. We are thrilled to learn that our clarifications and the additional comparison have addressed your concerns effectively. Thank you for your thorough evaluation and for your support. We deeply value your expertise and the insights you have provided throughout the review process.
thanks for the rebuttal! I am now leaning towards weak accept.
Thanks for the rebuttal! The comparison looks nice, I changed the rating to 7 (Accept)
We are truly grateful for the time and effort you invested in reviewing our paper and for your thoughtful feedback. Your insights have been invaluable, and we are glad that our clarifications effectively addressed your concerns.

Best regard.
Thanks for your response, which resolved most of my concerns. I would increase the score.
Thank you very much for your feedback and the time you dedicated to reviewing our paper. We greatly appreciate your insights and are pleased to hear that the clarifications we provided were helpful in addressing your concerns. If you have any further questions or suggestions, please do not hesitate to reach out. We would be happy to discuss them with you.
Dear Reviewer Rxew,

Thank you very much for taking the time and effort to review our paper. We sincerely appreciate your valuable feedback and insights.

We wanted to inform you that we have provided our responses in the comments section. We would greatly appreciate it if you could take a moment to review our replies.

Thank you once again for your time and consideration.
Dear Reviewer TdkD, 

First and foremost, please allow me to extend our deepest appreciation for the time and effort you have devoted to reviewing our paper.  

Moving forward, as the discussion phase is approaching its end, we are confident that we have comprehensively addressed the concerns you raised. We would greatly appreciate it if you could take a moment to review our responses. Your insights are important to us, and we are eager to hear your thoughts on the revisions we have made.

Thank you once again for your attention and assistance.
Dear Reviewer od3A,

Firstly, we would like to extend our sincere gratitude for the time and effort you have dedicated to reviewing our manuscript. Your insights and feedback are invaluable to us. As the discussion phase is nearing its conclusion, we believe we have addressed the concerns you raised in your review. We would be grateful if you could review our latest responses at your earliest convenience. Your further comments would be immensely helpful in refining our paper and moving forward in the review process.

Thank you once again for your attention and assistance.
Dear Reviewer 9o1r,

We would like to sincerely thank you for the time and effort you have dedicated to reviewing our paper. Your insights and feedback have been invaluable in helping us improve our work. 

As the discussion phase is nearing its conclusion, we believe we have addressed your concerns in our recent replies and would greatly appreciate it if you could take a moment to review our responses. Your insights are important to us, and we are eager to hear your thoughts on the revisions we have made. Thank you once again for your time and consideration.
## 1. The system can be a bit over complex for serving & compare computation complexity with other existing models.
Thank you for your insightful comments. We appreciate your concern and would like to address it comprehensively.
We acknowledge that the training process of our model might appear complex. However, we have conducted a thorough analysis comparing our model to recent SOTA models including QwenVL[1], InternVL[2], LLaVA1.5[3] in terms of training data size, training GPU count, inference FLOPs and latency . 
To evaluate the FLOPs and latency performance, we set the input image size of 336x336 and 24 text input tokens. We measured throughput and FLOPs over 10 inference steps using 4 input images, with an inference batch size of 1, on a single 80G A100 GPU. 

*Table 1. Comparision with different MLLMs on training data size, training GPU count, FLOPs, Latency and benchmark performance.*
| Model | Training Data Size | Training GPU Count | Inference FLOPs | Inference Latency | DemonBench ave Score (Multi-image) | VQAv2 dev (Single Image) |
|-------|--------------------|-----------|-----------------|-------------------|--------------------------|------------------------|
| QwenVL 7B |1.5B | 640×A100(80G) | 212 | 2.22 |29.9 | 78.2|
| InternVL 7B | 6B+ | Unkown| 340 | 1.53 | 30.3 |**79.3** |
| LLaVA 1.5 7B |**1.2M** | **8×A100(80G)** | 193 | 2.46 | 30.6|78.5 |
| MaVEN 7B | 7M | **8×A100(80G)**   | **163** | **2.64** | **39.0**|79.1 |

As shown in below table 2 and table 3. We also report trends in FLOPs and latency across different numbers (from 2 to 8) of image inputs. 

*Table 2. FLOPs across different numbers of image inputs for various models.*
| Image Nums | LLaVA | MaVEn | QwenVL | InternVL |
|------------|-------|-------|--------|----------|
| 2          | 104.44| 130.32| 120.33 | 210.33   |
| 4          | 193.6 | 163.21| 212.04 | 340.53   |
| 6          | 290.1 | 209.24| 324.23 | 480.2    |
| 8          | 402.4 | 268.42| 450.22 | 670.21   |

*Table 3. Latency across different numbers of image inputs for various models.*
| Image Nums | LLaVA | MaVEn | QwenVL | InternVL |
|------------|-------|-------|--------|----------|
| 2          | 3.24  | 2.94  | 3.05   | 2.34     |
| 4          | 2.46  | 2.64  | 2.22   | 1.53     |
| 6          | 1.75  | 2.48  | 1.45   | 0.89     |
| 8          | 1.21  | 2.22  | 0.88   | 0.33     |

We have following conclusion:
1. The results in Table 1 demonstrate that our approach actually incurs lower overhead in both training and inference stages compared to these models. **This suggests that, contrary to initial impressions, our method is indeed practical and efficient for deployment and maintenance in real-world scenarios.**
2. The results in Tables 2 & 3, we observed that as the number of images increases, MaVEn exhibits higher efficiency. This is primarily because **MaVEn encodes a lower number of continuous visual tokens, which reduces the computational burden.**

We hope this detailed analysis alleviates your concerns about the complexity of our model in practical applications. 
 ## 2. How does MaVEn compare with other multi-granularity approaches?
In response to your recommendation, we have conducted additional experiments to compare the performance of MaVEn using different multi-granularity  techniques, where we utilize VQGAN and VQVAE to replace the SEED tokenizer. Additionally, we explored the potential of combining these techniques.
Below are the results of our comparative experiments:

| Visual Discrete Representation | Code Book Size | DemonBench Ave score | SEED Bench acc-video | VQA  | MME  | MMbench test |
|-------------------------------|----------------|----------------------|----------------------|------|------|--------------|
| SEED                          | 8192           | 39.0                 | 42.1                 | 79.1 | 1530 | 65.2         |
| VQGAN                         | 1024           | 37.1                 | 39.2                 | 77.3 | 1441 | 61.3         |
| VQVAE                         | 1024           | 36.6                 | 38.4                 | 76.3 | 1380 | 60.2         |
| SEED+VQGAN                    | 9216           | 39.7                 | 42.8                 | 79.5 | 1521 | 65.8         |

Our conclusions are as follows:
1. Using SEED as the discrete visual token yields better performance compared to VQGAN and VQVAE.
2. Combining different discrete tokenizers can enhance the model's performance. We believe this improvement is due to the different visual semantic information encoded by the distinct codebooks. By integrating multiple codebooks, we achieve a richer and more comprehensive visual semantic representation, which in turn helps improve the model's overall performance.

[1] Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities.

[2] InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. 

[3] Improved Baselines with Visual Instruction Tuning.
You have addressed my concerns, and I will increase my score.
We sincerely appreciate your valuable suggestions and are honored by your recognition of our experiments. We will incorporate the aforementioned experiments and conclusions into the revised manuscript. Regarding the second issue you raised in your comments, we address your concerns from two perspectives:

1. **The semantic distribution of visual discrete tokens is closer to the text semantic distribution, aiding the alignment of image-text semantics by MLLM, which is the core reason for the model's performance improvement**: We encourage you to refer to Figure 6 in our paper and Figure 3 in the PDF provided in this rebuttal. We have conducted a visual analysis of the attention weight distribution of all features during the decoding phase of the LLM. Additionally, we clustered the discrete visual tokens. From the quantified experiments mentioned above, we observed that the semantics of visual discrete tokens are very close to the text semantics. Recent works [1][2][3] suggest that the most crucial step for multimodal large models is the alignment of image-text semantics, and visual representations based on discrete visual tokens are evidently easier to align with textual representations. Furthermore, the ablation study presented in Table 4 of our paper also supports this point where we found that the introduction of discrete tokens improved the model's performance on DemonBench from 30.66 to 39.51.

2. **We only used autoregressive image generation in the second phase of model training. The purpose of autoregressive image generation is solely to better aid the LLM in learning the embedding representation of visual discrete tokens, which is an essential step in introducing visual discrete representations**. During our training process, we only applied autoregressive image generation in the second phase of model training. Please refer to Figure 3 in our paper, where we show that only the embedding layer of the LLM was trained in this phase. Therefore, in our work, the use of autoregressive image generation is solely to better aid the LLM in learning the embedding representation of visual discrete tokens, helping the LLM understand visual discrete tokens. This is an essential step in introducing visual discrete representations. In fact, in previous experiments where we attempted to learn visual discrete tokens without autoregressive image generation, our results were as follows:

| w/ Visual Discrete Tokens | w/ Image Regressive Generation | DemonBench Performance |
|---------------------------|-------------------------------|------------------------|
| ✓                         | ✓                             | 39.0                   |
| ✓                         | ×                             | 32.14                  |
| ×                         | ×                             | 30.66                  |

We found that without autoregressive image generation, the embedding layer of the MLLM model struggled to effectively learn the embedding representation of visual discrete tokens, thereby limiting performance improvement on multiple images. Only by incorporating autoregressive image learning can the MLLM embedding layer better learn the embedding representation of discrete tokens and enhance performance.

We hope our response adequately addresses your concerns.

[1] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. NIPS 2022

[2] InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. CVPR2024

[3] Visual Instruction Tuning. NIPS 2023
Dear Reviewers,

We would like to kindly remind you to review our response to your comments and concerns. We have made our best efforts to address all the points raised by the reviewers and have provided detailed explanations and additional experiments to support our work.
The valuable insights and suggestions provided by the reviewers have helped us to strengthen our contributions and clarify the key aspects of our research. We are eager to engage in further discussion with the reviewers and welcome any additional feedback or questions you may have. Your expertise and comments are highly appreciated.

Thank you once again for your time and consideration.
My question has been resolved very well, and I am very satisfied with the experiments you added. Especially, the experiments you added have demonstrated that using multiple different discrete visual encodings simultaneously can further improve model performance, which is a very important discovery.  I'd like to try to follow this suit in my own work.
Additionally, I have read the comments from other reviewers as well as your reply, which has raised a new question for me. I noticed that in your reply, you mentioned that you believe the higher semantic granularity of discrete visual tokens helps MLLM better understand the semantic of images. But I also noticed that you used a self-regressive generation task for discrete image tokens in the second stage of training. Is it because you used this task that focused on discrete image tokens that it is the key to improving model performance? I would be very grateful if you could address this question and help me further improve my score.