MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model
Authors: Chaoya Jiang
Jia Hongrui
Haiyang Xu
Wei Ye
Mengfan Dong
Ming Yan
Ji Zhang
Fei Huang
Shikun Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Vision Language Model
Multi-image
Multi-granularity Visual Encoding
Primary area: machine_vision

Abstract: This paper presents MaVEn, an innovative Multi-granularity Visual Encoding framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. Current MLLMs primarily focus on single-image visual understanding, limiting their ability to interpret and integrate information across multiple images. MaVEn addresses this limitation by combining discrete visual symbol sequences, which abstract coarse-grained semantic concepts, with traditional continuous representation sequences that model fine-grained features. This dual approach bridges the semantic gap between visual and textual data, thereby improving the model's ability to process and interpret information from multiple images effectively. Additionally, we design a dynamic reduction mechanism by for long-sequence continuous features to enhance multi-image processing efficiency. Experimental results demonstrate that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts.

MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model
Chaoya Jiang' Hongrui Jia'! Haiyang Xu²; Wei Ye"; Mengfan Dong',
Ming Yan², Ji Zhang", Fei Huang", Shikun Zhang!
National Engineering Research Center for Software Engineering, Peking University
2 Alibaba Group
(jiancchooya, wye, zhangsk| @ (suuofeng.xhb .xyy, fei.huang @ alibaba-inc.com
Abstract
This paper presents MaVEn, an innovative Multi-grann gulti-granularity Visual Encoding framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) multi-image reasoning. Current MILMS primarily focus on single-image visual understanding, limiting their ability to interpret and integrate information across multiple images. MaVEn addresses this limitation by combining discrete visual symbol sequences, which abstract coarse -rarined semantic concepts, with traditional continuous representation sequences that model fine-grained fea- tures. This dual approach bridges the semantic gap between visual and textual data, thereby improving the model's ability to process and interpret information from multiple images effectively. Additionally, mechanism by for long-sequence continuous features to enhance multi-image processing efficiency. Experimental results demonstrate that MaVEn significantly enhances MI.LMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts.
design dynamic reduction
Introduction
Current multimodal large models (MLLMs) [37] concentrate on understanding single images [24, 43, 36, 22], which significantly restricts their ability interpret and integrate information across multiple images. As shown in Figure 1, typical scenarios [17] involving multiple images include Knowledge Based VQA. Visual Relation Inference, Multi-image Reasoning and on. These scenarios present a wide array of practical applications.
Present strategies predominantly adopt data-centric approach, where methods such those proposed in [1, 15, 41, 2] aim to strengthen the multi-image capabilities of Multimodal Large Language Models (MLLMs) by introducing interleaved image-text data during the pre-training and fine-tuning phases. Although some efficacy has been achieved, training solely based on interleaved data still falls short in many multi-image scenarios. This is primarily because current MLLMS remain fundamentally designed for single-image scenarios. This raises the question of whether the visual feature encoding and bridging methods of MLLMs, originally designed for single-image input scenarios, are suitable for mulli-image inputs.
Current MILMSs encode visual inputs using either discrete symbol encoding [11, 13, 35, 4] or continuous sequence encoding [7, 27, 34]. For the continuous sequence feature encoding category, the
corresponding authors.
'These authors contributed equally to this work.
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

Multi Inage Reasoning
Knowledge
Visual Relation Inference
Inaue
Image 3-
Imagee 2:
Image 3:
Image 2:
RENTTS
shat are the diffrenecce elenents in the first and second trages?
person potuue Ming. Which two people accund picture Ming?
the image positioned the sccund image?
1. A of feed, inchuing sandhich and onnn rags on ubil next to gloa. 2. A plate of food, inclulig sandwich and onion tigh, on ubile nou to play.
Ming the second basketaal uniforn.
inigg
L cea
Lavid.
LaVA1
while
older image
imago,
jarecy.
of the second image.
MaVEn
MaVEn
MaVEn
We compared the performance of the classic single-image task trained MLLM LLa [22] and our model in three multi-image scenarios including Multi Image Reasoning, Knowledge Based VQA and Visual Relation Inference. LLaVA1.5 exhibits significant limitations in multi-image scenarios.
Figure I:
following issues are present: (1) Excessively Long Visual Feature Sequences: Most current MLLMs utilize linear layers to bridge the visual sequence outputs from Vision Transformers (ViTs) [7]. Given the lengthy encoding sequences of images and the finite context input length of current MLLMs, the extended feature sequence inputs in multi-image contexts result in complex computational overhead and adversely affect model performance. (2) Imprecise Visual Information Encoding: Some MLLMs employ fixed-length latent queries to encode visual features through architectures like Q-Former [18). While this approach somewhat reduces the length of visual sequences, recent studies [20] suggest that still does not encode visual information from images with sufficient accuracy. There remains misalignment with textual representations, leading the model's confusion regarding visual information.
Moreover, recent works [11, 13, 4] have started to explore encoding images as discrete symbol sequences. Compared to complex continuous visual representations, discrete visual representations offer simpler and clearer high-level semantic abstractions, closely aligning with the discrete nature of textual representations. Consequently, discrete visual encoding looks like more conducive to complex multi-image reasoning. However, given that discrete visual representations tend to be coarser in granularity, relying solely on them may overlook fine-grained details within images [40].
In this study, we introduce MaVEn: effective and efficient Multi-granulare Hybrid Visual Encoding framework. MaVEn utilizes discrete visual symbol sequences to abstract coarse-grained semantic concepts, aiding in multi-image understanding, while traditional continuous representation sequences model fine-grained features to support detailed understanding. Accordingly, we investigate the synergy of multi- muuti-granularig visual features within the novel framework, design dynamic reduction mechanism for long-sequ oong-sequence continuous features to enhance multi-image processing efficiency, and propose A multi-stage model training methodology aimed improving mulli-image comprehension. Experimental results demonstrate that the proposed method effectively enhances the understanding capabilities of MILMMs in complex multi-image scenarios, while also improving performance in single-image contexts. In summary, the contributions of this study include:
We introduce a framework that combines discrete and continuous visual representations to enhance multi-image reasoning in MLLMs. This framework improves the model's ability to process and interpret information from multiple images effectively.
We design a dynamic reduction mechanism for long-sequence continuous visual features to increase the efficiency of multi-image processing in MLLMs.
Our approach demonstrates remarkable performance across various multi-image scenarios and also shows advantages in standard single-image benchmarks.

---

2 Related Work
2.1 Multimodal Large Language Models
Existing Multimodal Large Language Models (MLLMS) [37] typically consist of visual encoder, a visual interface, and large language model (LLM). The visual encoder converts visual data into continuous sequence features. The visual interface then maps these features into the LLM's semantic space, allowing the LLM to process visual information. Current research focuses developing effective visual interfaces. There are two main types: Latent-Query based Models: Used in MLLMS like BLIP-2 [18] and MiniGPT-4 [43], this approach uses fixed number of learnable latent vectors as query vectors in an attention mechanism. These vectors interact with visual sequence representations to summarize and integrate visual information, effectively reducing sequence length but potentially losing some visual details. Linear Mapping Models: Used in MLLMs like LLaVA [24], this method directly maps visual feature sequences into the LLM's text embedding space via a linear layer. This approach retains complete visual information but results in longer output sequences.
2.2 Visual Semantic Encoding Representations in MLLMs
Efficient visual semantic encoding has become key research area for MLLMs. Researchers have developed various methods to represent visual information, including: Continuous Visual Encoders: Examples include Visual Transformer (ViT) [7] and Trii--rranformer [27]. VIT segments images into patches and processes them sequentially, while Tren-Traanfoomer uses a sliding window mechanism to capture local structures face challenges aligning with textual encoding [20]. Discrete Visual Encoders: These methods encode images into discrete sequences similar text tokens, aligning visual and textual information more closely. Examples include VQ-VAE [35], VQ-GAN [13], and SEED [11]. VQ-VAE uses self-supervised learning create visual vocabulary from image patches. VQ-GAN combines VQ-VAE with generative adversarial networks to capture semantic information and generate high- quality images. SEED, the latest approach, encodes images into discrete visual sequences with one- one-dimensional causal dependencies, aiming to extract high-level semantics for visual understanding and generation tasks.
efficiently. These methods excel in capturing image details but
3 Method
As illustrated in Figure 2, we proposes an MLLM architecture that leverages multi-granularii visual features for enhanced multi-image understanding. Visual images are encoded as both discrete symbol sequences and continuous high-dimensional vector sequences. The discrete visual symbol sequences capture essential coarse coarse-grained visual concepts from the images, while the continuous vector sequences encapsulate fine-graine details. Furthermore, to minimize redundant and irrelevant visual representations in the continuous visual sequences and thereby reduce the input context length in multi-image scenarios, we also introduces dynamic reduction strategy for visual features, guided by textual semantics.
3.1 Multi-Granularity Hybrid Encoding
As shown in Figure 2 (a), assume the input to the MLLM (S,T), where S= {4, I2........ represents collection of K images, and T' denotes the corresponding textual content. For each image 4.. k E 21,2,.... K) J. we employ both the discrete visual encoder SEED [11] and the continuous visual encoder VIT [7] for encoding.
Visual Continuous Encoding:
we utilize the Vision Transformer (ViT) model, which widely adopted by most modern Multimodal Large Language Models (MLLMs). For an RGB image J, with dimensions W x x 3, the image is partitioned into patches size p x p. resulting
x # P x H and of F= represents continuous vector of patch reduction module (details of
patches. These patches are then encoded by the ViT visual encoder into a continuous visual sequence: V [ o.. ...... 1x J. Here, n2g p p. dimensions. Subsequently,
W
utilize the text-semantics-awall which will be elaborated in Subsection 3.2) to select patch features relevant to the input textual content T. thereby reducing the sequence length of V while preserving essential fine-graine fine-grained information.
The reduced feature sequence is denoted as V (ith ok de J, me & ne. Finally, we utilize

---

Targe Snggnge
Youd Takes
Vinaa Tolons
Lieee Prgezher
Grnzrdal SAM
Drnani Park Seloctoo
Irak SoaeesteSaaaa
Seperter
ViI
Veian Iranfferter
Iragee Diacce Tokestier
Dicore Vical Triee Sopercz
Vicual
Hyteri Flandlig
Figure 2: Subfigure (a) illustrates the structural schematic of our proposed Multi- Ilati-Grrnullrrty Hybrid Encoding, while subfigure (b) demonstrates the mechanism for the reduction of continuous visual tokens under the guidance of discrete visual information.
an Multi-Layer Perceptron, akin to that used in LLaVA 1.5, as a bridging projector to project Ve into the semantic space of the LLM embedding layer.
Visual Discrete Encoding: image I& tokenized by the image discrete tokenizer D, (for details on the design and training of the discrete tokenizer, please refer to the original work [11]) into a visual discrete symbol sequence Va dh .'..... d where dl E ...,..... N₂). Here, N. denotes the size of the visual discrete encoding vocabulary.
Given that text modalities naturally possess a discrete vocabulary, merging the visual discrete vocabulary with the textual discrete vocabulary forms a unified multimodal vocabulary. The advantage of this approach lies in its ability achieve a unified representation of both visual and textual modalities, effectively addressing the semantic gap between them. Assume that the vocabulary size of the LLM N, and the vocabulary size of the visual discrete tokenizer N.. The expanded multi-mont modai unified vocabulary size thus becomes N., = N + N₁. Concurrently, we align each element in Va with the index of the unified vocabulary to obtain the final discrete encoding: Va [ oh ..... 6%.]. where i d + N. Finally, the weight matrix of the LLM's s embedding layer. W, is also expanded from x 2 to N,, x 2. Consequently, the weight matrix of the LLM's embedding layer, W. is expanded from N x to N,, x This adjustment enables the embedding layer of the LLM to concurrently encode features from both visual and textual discrete tokens. The embedded representation of Va is denoted as [n] i .2...., ihh which is output by the expanded embedding layer. Finally, we sequentially insert the continuous visual tokens before the discrete visual token embeddings outputted by the embedding layer. The final visual representation inputted into the LLM is: [oth, Ti Tl
Unified Multimodal Vocabulary:
Pmm
3.2 Continuous Visual Tokens Reduction Mechanism
We aim for the discrete visual tokens to abstract high-level, coarse granind semantics from the images, while the continuous visual tokens complement this with low-level, fine-grained details. However, we found that the continuous visual tokens output by the Vision Transformer (ViT) encompass a considerable amount of redundancy, with many tokens possessing repetitive superfluous semantics. Consequently, as shown in Figure 2 (b), we propose a continuous visual token reduction mechanism guided by the parseegrained semantics of discrete visual tokens, aimed at achieving semantic synergy between coarse -raiined and fine-grained representations.
Firstly, after obtaining the sequence of discrete visual tokens Va [ef 05, oz..... ohgg. we append an <EOS> token to it. This sequence Va j o ok beos] then passed through the LLM to obtain the final layer's output hidden state of the <EOSS> token denoted as heas. which represents the global information of the discrete visual tokens: Mess LLM( o6 ....... ... 68 )
we then concatenate the EOS token with each image patch token as ot concat (T/) heos), where Jy E R Ieos
R 5 R²= 21,2.... ..... n.). Then the concatenated patch features of

---

Stage
Stage 2
Stage
Stage
Model
Model
@
Model
Embedding Layer
Embodding Layor
Embcdding
Embocding Layer
Patch Selecta
Paizs Se eetor
Paithh Selector
Parch Swlector
Visual Projector
Visual Projector
Visual Projector
Grounding
lmmaz
Tukoo
Image Conteccss
Image docnet
Tiken
Triles
Sinury
Irage
Tidies
Tokare
Prediction
Crnoomadial
Text auteragneshr georration
Tirst generation
The diagram illustrates the training schematic for MaVEn. We divide the training of MaVEn into four stages, where the snowflake icon indicates that the model parameters are frozen during training, and the flame icon indicates that the model parameters are updated during training.
Figure 3:
fed to the patch selector. The patch selector is Multilayer Perceptron (MLP) denoted as F that contains three linear layers and is used to predict the relevant score between patches and the discrete visual tokens. The output of the last linear layer has only dimension and will be fed to Sigmoid activation function to predict the relevant score a; with the discrete visual tokens as a; = F heos) According to the prediction of patch selector F, the top-m key image patch tokens are kept and the unselected patch tokens which generally have lower relevant scores will be discarded. Finally, we reconstruct the reduced visual sequence of [ hyper-parameter and named Keeping Ratio which is used to control the proportion of selected patches total patches.
p... where
n. a, and is
Construction of Patch-level pseudo-label annotation: train the patch selector, we constructed patch-level pseudo-label annotations based Grounding SAM [32] (a recent tfate-of-the-e art open- text vocabulary semantic segmentation model). We observed that the high-dimensional semantics encapsulated within the discrete visual tokens are largely consistent with the semantics of the image captions. Inspired by this observation, as shown in Figure 2 (b), we opted to use image captions as a proxy for the nigh-dimensional semantic abstraction of the image. We employed Grounding SAM to perform eext-guided semantic segmentation the images. After obtaining the semantic segmentation pixel masks, computed the overlap between each patch and the mask labels. If there is overlap, the corresponding patch label is set to I; otherwise, the label is set to O.
3.3 Training Paradigm of MaVEn
The training process of MaVEn is divided into four stages. In the first stage, we utilized image-text datasets like COCO [21] and Visual Genome (VG) [14] annotate million semantic segmentation masks with textual annotations, based on Grounding SAM [32]. These masks were subsequently converted into patch-level pseudo-labels. Utilizing this dataset, trained the Patch Selector while keeping other model parameters frozen.
In the second stage, we exclusively trained the embedding layer of the LLM (Large Language Model) to adapt to our expanded vocabulary for the LLM. Consequently, we utilized the LLaVA 558k single-image pretraining dataset [24] and the MMC4 interleaved image-text dataset [45] for training. At this stage, we employed only the visual discrete encoding, eschewing the visual continuous encoding, with the aim of adapting the LLM embedding layer to the expanded unified vocabulary. We trained using cross-modes modal autoregressive generation task; given an input that might contain images, obtained tokenized discrete sequences through the text and image tokenizers. This enabled to generate discrete image token sequences from text discrete token sequences and vice versa.
In the third stage, our objective is to optimize the visual projector so that the continuous visual tokens, after being processed by the visual projector, align with the semantic space distribution of the unified multimodal vocabulary embeddings. Therefore, during this phase, train solely the visual projector.
5

---

Multi Modal
Visual Story Telling Lisi
Visual Relation
Muini Modal
Knowledge
Text Rich
Multi Image
Method
Dialogue
Inference
Cloze
Grounded QA
Images QA
Reasoning
BLIP-2 [19]
11.96
20.10
3.67
18.25
39.73
30.53
39.53
mPLUG-OWl [36]
12.67
19.33
5,40
16.25
33.27
32.47
InstructBLIP [6]
33.58
24.41
11.49
21.20
47.40
44,40
48.55
LLaMA-Adapterv2 [10]
14.22
17.57
13.51
18.00
44.80
32.00
44.03
LLaVA [25]
7.79
10.70
8.27
15.85
36.20
28.33
41,53
MiniGPPT-4 [44]
13.70
17.07
7.95
16.60
30.27
26.40
43.50
LLaVA-1.5 [23]
27.17
14.32
11.62
31.65
46.4
38.87
Oiter [15]
13.37
15.37
17.39"
16.00
31.67
27.73
T3.83
OpenFliamingo
16.88
24.22
13.85
21.65
32,00
30.60
41.63
VPG-C [17]
37.50
25.20
25.90
22.15
48.60
44.93
50.28
MaVEn
34.63
21,33
30.24
33,35
51.33
47.33
54,38
Table Average results zero-shot evaluation on each task of DEMON Benchmark [17].
Method
Vision Encoder
Language Model
Avg.
Avg. Img
Avg. Video
BLIP-2 [19]
ViT-g (1.3B)
Vicuna (7B)
46.4
49.7
36.7
mPLUG-Owl [36]
VIT-L (0.3B)
LLaMA (7B)
34
37.9
23
InstruciBLIP
ViT-g (1.3B)
Vicuna (7B)
53.4
58.8
38.1
LLaMA-Adapter-22 [10]
VIT-L (0.3B)
LLaMA (7B)
32.7
35.2
25.8
Otter [15]
VIT-L (0.3B)
LLaMA (7B)
33.9
35.2
30.4
LLaVA [25]
VIT-L (0.3B)
Vicuna (7B)
33.5
37.0
23.8
MiniGPT-4 [44]
ViT-g (1.3B)
Vicuna (7B)
42.8
47.4
29.9
LLaVA-1.5
ViT-L
Vicuna (7B)
58.6
66.1
37.3
MaVEn
VIT-L + SEED (I.3)
Vicuna (7B)
60.89
65.85
42.11
Table Average results of zero-shot evaluation on each task category of SEED Benchmark [17].
We train using the LLaVA 558K image-text caption dataset, where images are encoded solely as 5k without employing visual discrete
sequences of continuous visual tokens: V [rol pp: p2:---3 Pm..
coding. The model is required generate captions for the images based the visual input.
the fourth stage, we introduce instruction fine-tuning data with the aim of enhancing the MLLM's capability to follow human instructions. During this phase, the MLLM undergoes comprehensive fine-tuning with the LLaVA 665k instruction fine-tuning datasets, unfreezing model parameters except for those of the visual encoder and patch selector for training and optimization.
4 Experiments
4.1 Experiment Setting
Dataset: We initially generated million pscudo-labels for patch-level text semantic relevance by utilizing the COCO[21], Visual Genome (VG)[14], and RefCOCO datasets [38], following the methodology delineated in Subsection 3.2 and leveraging Ground SAM. These pseudo-labels were subsequently employed to train the patch selector within MaVEn. During the second phase of model training, the embedding layer of Ma VE was refined using the MMC4-core dataset [45] and pre- pre-training dataset [22]. the third phase, the visual projector
the LLaVA 558K single-iman component of Ma Vn was further trained using the LLa VA 558K single-image dataset. Finally, in final phase, we fine-tuned the model with the LLaVA 665K instruction fine-tuning dataset.
Training Settings: MaVEn utilizes the VIT-L model [31] with a patch size of 14 x 14 and is pre-trained resolution of 336 336, resulting in continuous token length of 567 for the encoded image. For image discrete tokenization, SEED [1 1] is employed to tokenize the image into 32 discrete tokens. For the continuous visual tokens, during patch reduction, set the Keeping Ratio 0.25, meaning that only 25% of the continuous tokens are retained. Consequently. the length of the final continuous visual token sequence decreases from 576 to 144, while the length the discrete token sequence is 32. Ultimately. the entire visual hybrid encoding sequence has length of 176. The large language model Vicuna [42], with billion parameters, is used to handle multi-modal features. The AdamW optimizer [28] is used for optimization. During the instruction tuning stage, the entire model is trained for epoch with learning rate of 2e-5 and batch size of 256. All experiments was performed using 8 NVIDIA A100 GPUs, each with 80GB of memory.

---

General VQA
General VQA (Zero-shot)
Zero-shot Multi modal Benchmarks
MME
MMBench
MM-Vet
Method
#Params
VQAv2 GQA
VizWizQA TextVQA SciQA
BLIP-2 [19]
8.2B
63.0
41.0
19.6
42.5
61.0
1293.84
22.4
InstructBLII [6]
8.2B
49.2
34.5
50.17
60.5
1212.82
36.0
26.2
Unified-1Oxu [29]
2.9B
77.9
57.4
PaI.M-EE11B [8]
12B
76.2
Shikra [5]
7.2B
77.4
58.8
Qwen-VI.-Cha [3]
9.6B
78.2
57.5
38.9
61.5
68.2
1487.58
60.6
LLaVA [23]
7.2B
71.3
41.3
36.7
50.21
61.5
502.82
36.2
28.1
MiniGPT-4 [23]
7.2B
65.2
30.8
30.2
52.3!
58.4
581.67
23.0
22.1
LLaVA1.5 [23]
7.2B
78.5
62.0
50.0
58.21
66.8
1510.70
64.3
30.5
MaVEn
7.2B
79.1
62.5
50.5
59.8
67.3
1530.10
65.2
30.4
Table 3: Performance comparison on visual question answering and zero-shot multi-modal benchmarks. For VQA, accuracy is reported. Note that specialists are fine-tuned each individual dataset. f denotes OCR inputs are utilized. indicates the model has trained on the dataset.
4.2 Main Results
To validate the effectiveness of MaVEn in multi-image scenarios, we evaluated MaVEn's 's performance multi-image visual understanding and reasoning. Within the multi-image visual understanding context, we assessed the model using DemonBench[17] and SEED-Bench(16) DemonBench comprises seven scenarios involving multi-image reasoning and understanding. including tasks such Multi-Modal Dialogue, Visual Relation Inference, Knowledge Grounded QA, and Multi-Image Reasoning. SEED- Bench, on the other hand, encompasses questions related video comprehension. Additionally, we also tested the performance of MaVEn in single-image scenarios.
4.2.1 Effectiveness of Ma on Multi-image Visual Comprehension
Results on DemonBench: shown in the Table 1, we evaluated our model on DemonBench, comparing it with several multi-image data-trained MILM models such as Openflamingo, Otter, VPG- C, well as single-image scenario MLLM models. Our model attained the highest scores tasks such as Visual Relation Inference, Multi-Modal Cloze, Text-Rich Images QA, Knowledge-Grounder QA, and Multi Image Reasoning, underscoring MaVEn 's significant superiority in multi-image understanding and reasoning tasks. It also achieved comparable performance in Visual Storytelling and Multi-Modal Dialogue.
Results on SEED-Bench: focusing on video understanding tasks like action prediction, action Recognition and procedure understanding. As shown in Table 2 The experiments revealed that our approach significantly outperformed existing models like LLaVA 1.5, Otter and so on. For instance, MaVEn exhibited 12-point improvement over Otter [15] in video understanding (30.4 -> 42.11). These findings underscore MaVEn effectiveness in multi-image understanding scenarios.
Furthermore, we assessed our model on SEED-Bench [16], particularly
4.2.2 Effectiveness of MaVEn Single-image Visual Comprehension
We intend explore the influence of MaVEn the model's abilities of single image visual comprehension and generation. To achieve this objective, we carried out assessments on common benchmarks, such Visual Question Answering (VQA) [12, 30, 33] and recently designed MLLM- focused Multi-modal Benchmarks including MME [9], MMBench [26], MM-Vet [39].
Results on Benchmark Tasks As summarized Table 3. We compared performance of MaVEn to other SOTA MLLMs such as BLIP2[19], InstructBLIP [6]. Shikra [5], and Owen-VL-Chal [3]. Our experimental results show that our approach can also successfully enhances the performance across a range of single image understanding task. Notably, MaVEn outperforms LLaVA- 1.5 [25] in terms of consistency and accuracy across all VQA datasets.
Morinntee Multi-modal Benchmarks. We also evaluate MaVEn on four recently popular imngle-image multi-modal benchmarks a zero-shot manner. The results of our evaluation are listed in Table 3. We discovered that after implementing MaVEn, all three models exhibited improvements
7

---

Discrete
Continuous
SEED Bench
DEMONBench
VQA
MMBench
X
43.2
24.19
56.07
34.5
58.6
30.66
78.5
64.3
60.89
39.51
79.1
65.21
Table 4: Ablation evaluation on multi-modal benchmarks We evaluated the performance of various ablation targets on both multi-image (SEED-Bend)) DEMONBench and single-image (VQA, MMBench) benchmarks. MMBench [26].
Results Different Benchmarks with Varying Keep Ratios
Length
SIED Bench
Demo Bench
1161
31.2
Keazing ta00
Seeping 100
Kagging rado
Figure 4: Evaluation Results of Ma on different benchmarks with varying Keeping Ratios.
across multiple benchmarks. Notably, for LLaVA and MiniGPT-4, the enhancement was particularly evident on the MME [9] benchmark. For instance, after implementing MaVEn, LLaVA's MME score improved from 581.67 to 653.94. These results highlight Ma role in advancing the state-of-the-art in both single-image and multi-image visual comprehension tasks.
4.3 Ablation Study
4.3.1
Effectiveness of GraniiiGrrllartty Hybrid Encoding
To ascertain the efficacy of multi-granularity hybrid encoding, we conducted training using solely visual discrete encoding and visual continuous encoding, respectively. We then evaluated and compared the outcomes on both multi-image and single-image evaluation benchmarks. The results detailed the Table 4 below. We observed that, compared with utilizing only visual continuous encoding or employing a hybrid of visual discrete and continuous encoding, the model solely on visual discrete encoding exhibits subpar performance in both multi-image and single-image contexts. This underperformance is likely due to the nature of discrete visual feature encoding, which, while capturing the high-ue sigh-dimensional information of the image, forfeits a significant amount of dww-dimeniiona, fine-grained details, resulting in a lossy encoding process. As a result, it fares poorly in tasks like image reasoning and understanding, which demand meticulously detailed information. Moreover, the model with only continuous encoding also do not deliver optimal performance, particularly in multi-image tasks. This further indicates that models based solely on visual continuous encoding are unsuitable for multi-image scenarios.
4.3.2 Efficient of Continuous Visual Token Reduction Mechanism
To verify the effectiveness of patch reduction, we compared the length of visual tokens at different Keeping Ratios and analyzed the performance across various benchmarks. We experimented with the Keeping Ratios from 0.1 to 1.0. The experimental results are shown in the Figure 4. We observed that when the Keeping Ratio was 0.1, the number of visual tokens decreased 89, a significant reduction from the initial number of patches. However, the model's performance across multiple benchmarks also significantly declined. Therefore, despite the reduction in the number of visual tokens, the performance loss was too substantial, making it an unsuitable final choice. When the Keeping Ratio was 0.25, the number visual tokens remained relatively low, the model's performance was more stable. In this case, the model exhibited balanced performance across various benchmarks, effectively reducing the number of visual tokens while maintaining a nirh-performance level. Therefore, we ultimately chose a Keeping Ratio of 0.25.

---

Original Image
Selected Patches
Relevant score
Discrete Index Distribution
mr00
Figure 5: This figure visualizes the distribution of discrete tokens in an image containing index 4568 discrete tokens, along with the relevant score computed based on the Patch Selector and the patches chosen according the relevant score that are most semantically related to the discrete visual tokens.
Average Atteation Weights with Multi-ranndaartty Hybrid Visual Emcoding
Imagel
Image?
Text
attartion Weightt
0.00
-0.04
-6.02
Imagel
Discrete tokens
Continuous tokens
Discrete
Continuous
tokens
Average Attention Weights with Only Continuous Visual Tokens
Imagel
Image2
Text
ittentoon Weight ghtt
-0:5
-60s
Image2
0.00
Figure 6: Visualization of the attention maps with and without the visual discrete tokens demonstrate the attention maps for the 31-st layers, where the range of visual tokens is indicated by orange and the range of text tokens is indicated by blue.
We
4.4 Qualitative Analysis
Semantic Granularity of discrete and continuous visual tokens: To investigate the semantics of discrete tokens, we randomly selected index from the visual discrete dictionary and searched the CC/SUB dataset for images containing this index in their encoding. As illustrated in Figure 5, we randomly chose three images with index 4568. We discovered that all three images featured the depiction of snowman, suggesting that index 4568 can represent high-level semantics such as snowman or white snow. We provided the distribution discrete tokens for these three images and observed that the position of index 4568 in the discrete sequences was also notably consistent.
Semantic synergy between visual discrete and continuous representations: Furthermore, as shown in Figure 5, we also visualize the relevant score between the patches and the semantics of discrete visual tokens predicted by the patch selector, along with the patch tokens selected based on Keeping ratio of 0.25. We found that the patch selector tends to choose patches related to the semantics of discrete visual tokens, thereby supplementing the missing low-level fine-grained information discrete tokens. The aforementioned findings further validate that multi multi-grumularity hybrid encoding facilitates mutual assistance and synergy between discrete and continuous representations, thereby achieving efficient multi-granulen granularity semantic encoding.

---

The impact of discrete visual tokens on multi-image reasoning: To further validate the role of discrete visual tokens in the inference process of multi-image instructions, we visualize the attention weights of the last layer of the L.M.. As illustrated in Figure 6, we have MaVEn compare the commonalities between two images. For inputs using only continuous visual tokens, we observe that the model's attention during inference is primarily focused on text tokens, disregarding visual tokens. This may still be due to the lower semantic granularity of continuous visual tokens, making it difficult to establish direct semantic associations. However, with inputs encoded using hybrid encoding, we notice that the model establishes attention associations with discrete visual tokens when answering questions. This indicates that, in multi-imag scenarios, discrete visual tokens guide the LLM to focus on visual information during decoding.
visual
5 Conclusion
In conclusion, we introduces MaVEn, novel Multi-granularity Hybrid Visual Encoding framework designed to enhance multi-image reasoning in MILMs. By combining discrete visual symbols for semantic abstraction with continuous sequences for detailed features, MaVEn improves both understanding and processing efficiency. Our dynamic reduction mechanism and multi-stage training strategy further enhance performance. Experimental results confirm that MaVEn significantly boosts MLLM capabilities in both multi-image and single-image contexts.
Acknowledgement
This work is supported by the National Natural Science Foundation of China (623B2007) and CCF-Zhipu Large Model Innovation
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, volume 35, 2022.
[2] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.00 01 01090, 2023.
[3] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. ArXiv, 1is/2308.11966, 2023.
[4] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.002544 2021.
[5] Ke Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's 's referential dialogue magic. ArXiv, abs/2306. 01122306..119., 15195, 2023.
[6] Wenliang Dai, Jumnan Li. Dongxu Li, Anthony Meng Huat Tiong. Junqi Zhao, Weisheng Wang. Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiv, abs/2305.06500 2023.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[8] Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng. Igor Mordatch, and Peter R. Florence. Palm-e: An embodied multimodal language model. In International Conference on Machine Learning, 2023.
[9]
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. Sttii:2230.1399. 13394, 2023.
10

---

[10] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, W. Zhang, Pan Lu, Conghui He, Xiangyu Yue. Hongsheng Li, and Yu Jiao Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. ArXiv, abs/2304 2023.
[11] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint wwtii:223770080444 2023.
[12] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the in VQA matter: Flevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[13] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for -omaae synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696-10706, 2022.
[14] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[15] Bo Li, Yuanhan Zhang, Liangyu Chen, et al. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305003726, 2023.
[16] Bohao Li, Rui Wang, Guangzhi Wang. Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal lims with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.
[17] Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Fine-tuning multimodal Ilms to follow zero-shot demonstrative instructions. In The Twelfth International Conference on Learning Representations, 2023.
[18] Junnan Li, Dongxu Li, Silvio Savarese, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
[19] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv, abs/2301.12597, 2023.
[20] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in modai contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612-17622 2022.
[21] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer ViCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[22] Haotian Liu, Chunyuan Li, Yuheng Li, et al. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.
[23] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. ArXiy, abs/2310.03744. 2023.
[24] Haotian Liu, Chunyuan Li, Qingyang Wu, et al. Visual instruction tuning. In NeurIPS, 2023.
[25] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv, abs/2304.08485,, 2023.
[26] Yuan Liu, Haodong Duan, Yuanhan Zhang, et Mmbench: Is your multi-modal model an all-around player? arXiv preprint +xXiv:2307.0628.. 2023.

---

[27] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[28] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018.
[29] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: unified model for vision, language, and multi-modal tasks. ArXiv, abs/2206.08916 2022.
[30] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947-952. IEEE, 2019.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 9748-8763. PMLR, 2021.
[32] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401. 1411599900..4199 2024.
[33] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[34] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 2308.1588.,, 2023.
[35] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[36] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[37] Shukang Yin, Chaoyou Fu, Sirui Zhao, et al. A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
[38] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part II 14, pages 69-85. Springer, 2016
[39] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint urXiv:2308.02490 2023.
[40] Zheng Zhang, Yong Xu, Jian Yang, Xuelong Li, and David Dian Zhang. A survey of sparse representation: Algorithms and applications. IEEE Access, 3:490-530 03000053300000 2015.
[41] Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu. Sheng Wang, Wenjuan Han, and Baobao Chang. Mmiel: Empowering vision-language model with multi-modal in-context learning. arXiv preprint axXiv2200.009115 2023.
[42] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph F., Gonzalez, and lon Stoica. Judging alm-as-a-judge with mt-bench and chatbot arena, 2023.
[43] Deyao Zhu, Jun Chen, Xiaoqian Shen, al. Minigpt-4: Enhancing lision- understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
12

---

[44] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vanion-language understanding with advanced large language models. ArXiv, abs/2304.10592. 2023.
[45] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, sillion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 2024.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize to other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should diseuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification: NA
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification: NA
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.c
for more details. The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ys]]
Justification: NA
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer:
[Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the theorde package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
18

---

Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "VU2pvxaVIz", "reviewer": "Reviewer_Rxew", "summary": "This paper proposes MaVEn, a novel multi-granularity hybrid visual encoding framework for multimodal large language models (MLLMs). MaVEn aims to improve MLLMs' capabilities in multi-image reasoning by combining discrete and continuous visual representations. The authors design a dynamic reduction mechanism to reduce the computational overhead of long continuous sequences. Experimental results demonstrate that MaVEn significantly improves performance on both multi-image and single-image benchmarks.", "questions": "1. how does MaVEn compare with other multi-granularity approaches? \n2. Could you elaborate more on the computation complexity and efficiency?", "limitations": "Need more discussion on the model's limitation and computational complexity", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposed a novel approach to combine the strength of both discrete and continuous visual representation as well as dynamic reduction mechanism.  \nIn addition, the authors conduct comprehensive experiments on both multi-image and single-image benchmarks to demonstrate the effectiveness of MaVEn.", "weaknesses": "While the MaVEn proposed a novel approach to combine the advantage of discrete and continuous visual info, the system can be a bit over complex for serving / maintenance in real applications.\n\nIn addition, the paper didn't discuss the computation complexity of MaVEn and compare it with other existing models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iEV6jLogGm", "reviewer": "Reviewer_9o1r", "summary": "The paper introduces MaVEn, a framework designed to improve Multimodal Large Language Models (MLLMs) in understanding and reasoning across multiple images. Unlike current MLLMs, which are mainly focused on single-image interpretation, MaVEn integrates both coarse-grained semantic concepts and fine-grained details. This combination bridges the gap between visual and textual data, enhancing the model's ability to process multiple images. The framework also includes a mechanism for efficiently handling long sequences of features. Experiments show that MaVEn boosts performance in multi-image scenarios and also provides benefits for single-image tasks.", "questions": "Please refer to the weakness section.", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The concept is logical, and the paper is straightforward to read.\n2. The concept of using both discrete and continuous visual tokens is intriguing.\n3. I appreciate the author's use of figures 2 and 3, which help clarify the overall framework and training process.", "weaknesses": "1. In Tables 1, 2, and 3, the author does not compare some of the latest methods, such as mini-Gemini, MiniCPM, XComposer, and InternVL.\n2. The paper primarily claims its main advantage is in multi-image tasks; however, the author only tests on DEMON and SEED benchmarks. Testing on additional multi-image benchmarks, such as MMBench-Video and MME-Video, would be more compelling.\n3. Figure 6 is unclear. In the first figure, I see only two vertical lines on discrete tokens. Does this mean that only discrete tokens have attention weight?\n4. The paper does not report the computational complexity compared to other methods.\n5. Figure 5 shows that the selected tokens are mostly related to objects. Would it be beneficial to directly use Grounding Sam for token selection? A comparison might be interesting to see.\n6. The training pipeline is complex and involves four stages. What is the training cost?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Not compare some of the latest methods\nThank you for your valuable comments.  We have reproduced the results of mini-Gemini, MiniCPM-V, and InternVL and compared MaVEn with them. It is important to emphasize that our experiments were based on LLaVA 1.5. To validate our method's effectiveness on the most current MLLMs, we further retrained MaVEn on the latest SOTA single-image MLLM LLaVA Next, we named it MaVEn-NEXT. The results are as follows:\n\n|Model|LLM|DemonBench Ave score|SEED Bench acc-video|VQA|MME|MMbench test|\n|-|-|-|-|-|-|-|\n|miniGemini|Vicuna 7b| 31.4 | 38.6| 65.2| 1523 |69.3 |\n|InternVL-Chat-v1| Vicuna 7b | 30.3| 40.3| 79.3| 1525 |74.7 |\n|MiniCPM-V v2.5| Llama3 8b |33.9 | 40.5| 80.3 |**1916** | **77.2**|\n|LLaVA 1.5| Vicuna 7b |30.6 | 37.3 |78.5 |  1511| 64.3 |\n|LLaVA Next| Llama3 8B|31.3|39.1 | 79.3 | 1591| 72.6 |\n|MaVEn| Vicuna 7b| 39.0| 42.1|79.1 |1530 |65.2 |\n|MaVEn-Next| Llama3 8B| **41.2** | **44.3** | **80.7** | 1623 | 75.5|\n\nWe observed that MaVEn remains superior multi-image performance which demonstrating the effectiveness of our method. Moreover, MaVEn-NEXT shows improved single-image performance, reaching SOTA or comparable results.\n### 2. Testing on additional multi-image benchmarks\nThank you for your suggestion.\nIndeed, MMBench-Video and MME-Video were released after the NIPS submission deadline.  Moreover, based on your suggestion, we evaluated our model on MMBench-Video and MME-Video during the rebuttal period. During the evaluation, we extracted 8 frames from each video as input:\n\n|Model |CP | FP-S | FP-C| HL| LR| AR | RR| CSR| TR|\n|-|-|-|-|-|-|-|-|-|-|\n|InternVL-Chat-v1.5-[8f] |1.26 |**1.51** |1.22|**1.01**|**1.25**| **0.88** | 1.40 | **1.48**| 1.28 |\n|mPLUG-Owl2-[8f] |1.15| 1.34| 1.18 |0.99 |1.15| 0.63|1.33| 1.30| 1.03|\n|Qwen-VL-Chat-[8f] |0.52| 0.44| 0.62|0.33| 0.53| 0.45 |0.59| 0.50| 0.36|\n|Idefics2-8B-[8f]| 1.10| 1.23| 1.07| 0.89 |1.06 |0.77| 1.27| 1.41|1.11|\n|MaVEn-[8f] | **1.32** |1.32 |**1.24**| 0.96|1.18| 0.83 | **1.45**  |1.44| **1.31**|\n \nWe found that MaVEn performs better than MLLM models such as mPLUG-owl2, Qwen-VL, Idefics2-8B, and its performance is comparable to that of InternVL-Chat-1.5.\n\nFor the MME-Video results, please refer to Table 1 in the rebuttal pdf, where MaVEn also achieve competitive performance. \n### 3. Figure 6 is unclear.\n1. Clarification of Figure 6: We apologize for any confusion caused by Figure 6. In fact, the text question in the case is \"What is the similarity between Figure 1 and Figure 2?\" . **In this case, the American flag represents a coarse-grained semantic entity that continuous visual tokens struggle to effectively encode alone.** This results in the MLLMs paying less attention to continuous visual tokens during the decoding phase.\n2. Importance of continuous visual tokens: When the model faces scenarios that require understanding of fine-grained details in the image, MLLMs actually need continuous visual tokens. To better validate this point, as shown in Fugure 1 in the submitted pdf in this rubuttal, we have visualized the attention weights distribution of MaVEn when the model is asked about the details of the image. We found that MLLMs also pay attention to continuous visual tokens.\n3. Moreover, we conducted the following experiment: We randomly collected 500 images from the COCO dataset and tasked GPT4-o to generate questions about detailed object information in each image (e.g., asking about the shape, color, number, and size of an objects) . We then tested the accuracy on this dataset using MaVEN with only discrete visual tokens, only continuous visual tokens, and both, as shown in below table. We found the performance of the model using only discrete visual tokens was very poor.\n\n| Model | w/ Continuous token | w/ Discrete token | Accuracy |\n|-------|---------------------|-------------------|----------|\n| MaVEN | ✓       | ×                 |    35.2      |\n| MaVEN | ×       | ✓                 |      69.3    |\n| MaVEN | ✓       | ✓                 |      70.5    |\n\n### 4. Report the computational complexity compared to other methods.\nIn response, we have conducted a thorough evaluation:\n1. Experiment Setting: Specifically, we set the input image size to 336x336 and the number of text input tokens to 24. We then measured the  throughput and FLOPs of 10 inference step for different numbers of images (ranging from 2 to 8) across several models: LLaVA 1.5, MaVEn, QwenVL, InternVL. The inference batch size is 1 and was performed on single 80G A100 GPU. \n2. Experiment Results: As shown in Figure 2 of the submitted pdf file, we observed that as the number of images increases, MaVEn exhibits higher efficiency. This is primarily because MaVEn encodes a lower number of continuous visual tokens, which reduces the computational burden.\n### 5.Directly use Grounding Sam for token selection?\nVery insightful question!\n1. First, we want to highlight that Grounded SAM is engineered to segment image regions based on their corresponding textual information. However, if user instructions fail to provide explicit textual semantic cues for image segmentation (such as \"What are the differences between image one and image two?\"), it may lead to inaccurate region segmentation by Grounded SAM. \n2. To fully address your concern, we tested the direct use of Grounded SAM for selecting image patches in both multi-image and single-image benchmarks. The results are as follows:\n\n| Method                 | DemonBench Ave Score| VQA | MMbench |\n|-------------------------|-|--------|---------|\n| MaVEn-Patch Selector    | 39.0 | 79.1     | 65.2        |\n| MaVEn-Grounding SAM     | 27.3 |   71.2   | 52.5        |\n\nThe results indicate a significant decline when using Grounded SAM for selecting image patches, highlighting the effectiveness of patch selector. \n### 6. training cost?\nWe used 8*80G A100 GPU for training, which overall took about 122 hours. We give more detials in the table2 of the rebuttal pdf.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ufc2vjdwZa", "reviewer": "Reviewer_od3A", "summary": "This paper introduces MaVEn, a Multi-granularity Visual Encoding framework that enhances Multimodal Large Language Models (MLLMs) in multi-image reasoning by combining discrete visual symbol sequences with traditional continuous representation sequences. Experimental results show that MaVEn significantly improves MLLMs' understanding in complex multi-image scenarios and boosts performance in single-image contexts.", "questions": "I am not an expert in this domain. But I do have two questions for authors:\n\n1. Why not directly operate on continuous features to reduce feature redundancy since the approach doesn't learn from coarse to fine, like gating?\n\n2. It's not quite clear how discrete tokens help non-single image understanding. The obvious advantage of using discrete tokens is to improve efficiency only?", "limitations": "n/a", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-organized, from problem, motivation, approach and experimental validation. \n\n2. The proposed innovative multi-granularity approach includes 1) hybrid visual encoding and 2) dynamic reduction mechanism. The hybrid visual encoding captures both coarse-grained semantic concepts and fine-grained features, effectively bridging the semantic gap between visual and textual data.To enhance processing efficiency, a dynamic reduction mechanism is proposed, which selectively reduces long-sequence continuous features. This approach maintains essential information while reducing computational overhead.\n\n3. The paper validates MaVEn’s effectiveness using several benchmarks, including DEMONBench and SEED-Bench, which encompass multi-image reasoning and video understanding tasks. MaVEn achieves superior performance compared to state-of-the-art models like LLaVA1.5, Otter, and others.Besides multi-image tasks, MaVEn also performs well in single-image benchmarks such as Visual Question Answering (VQA) and MMBench, showcasing its versatility.", "weaknesses": "see questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our work and for your insightful questions. \n### 1.  Why not directly operate on continuous features to reduce feature redundancy ?\n\nThis is an excellent question. Directly reducing redundancy in continuous features can be approached in two main ways: \n\n1. **Token Selection with global semantic:** This could be done by using the attention weights from the image global semantic token (e.g., <EOS> token)  to select important continuous visual tokens [1], [2].\n2. **Merging Visual Tokens:** This can be done using latent queries (e.g., InstructBLIP[3]  used q-former to extract the fixed length visual continuous sequence) or convolutional network (e.g., QwenVL[4]) to merge long sequences of continuous visual tokens.\n\nWe have experimented with these methods as per your suggestion, and the results are as follows:\n\n| Model | Method | DemonBench Ave Score| VQA | MMbench |\n|-------|--------|----------|-------|-----|\n|MaVEn | Token Selection with global semantic|  33.8 | 74.5     | 62.1        |\n| MaVEn | Token Merging based on Q-former | 29.4 | 71.1  | 54.2 |\n| MaVEn | Our Method |  39.0 | 79.1  | 65.2  |\n\nWe found that these methods were less effective than our proposed approach, which combines discrete visual tokens with continuous tokens. From these results, we have the following inferences:\n\n1. Compared with the first type of methods, our method is similar to them but is more efficient and accurate because our method utilizes coarse-grained discrete visual features and fine-grained continuous features to encode complementary information. In our method, discrete visual features capture high-level, coarse-grained information (e.g., \"snowman,\" \"American flag\" as shown in Figure 6 in our paper), while continuous features capture fine-grained details of the image. In contrast, traditional methods based on global image semantic representations for token selection lack high-dimensional semantic guidance, resulting in lower accuracy in token selection.\n\n2.  The second type of method, which merges visual information, compresses the data and loses important information, thereby impairing the MLLM's understanding of the image. \n\nTo further illustrate the superiority of our method over traditional token selection methods, we conducted an experiment. We randomly selected 500 images from the COCO dataset and used Grounding SAM to segment relevant regions based on textual semantics of image caption as ground truth. We then measured the accuracy of selecting 20%, 40%, 60%, and 80% of tokens that hit the ground truth region for both approaches. As shown in below table, our method significantly outperformed the global visual semantic-based token selection method, demonstrating its effectiveness.\n\n| Token Selection Method |  20% | 40% | 60% | 80% |\n|------------------------|----------------|----------------|----------------|----------------|\n| Token Selection with global semantic | 93.4 | 90.5 | 84.4 | 79.4 |\n| Our Proposed Method | 74.8 | 66.3 | 61.1 | 52.3 |\n\n\n\n### 2. It's not quite clear how discrete tokens help non-single image understanding. The obvious advantage of using discrete tokens is to improve efficiency only?\n\nThank you for your insightful question. \n\n1. To better address your concern, we would like to draw your attention to Figure 6 of our paper, which visualizes the Average Attention Weights with Only Continuous Visual Tokens and the Average Attention Weights with Multi-granularity Hybrid Visual Encoding. The text question in the case presented in Figure 6 is, \"What is the similarity between Figure 1 and Figure 2?\" This case highlights the challenges faced by recent MLLMs, particularly in multi-image contexts, where **the semantic granularity of continuous visual representations and text tokens differ significantly, making it difficult for MLLMs to understand and capture high-dimensional semantic information from images.** In this specific case, the American flag represents a higher-dimensional, coarse-grained semantic entity that continuous visual tokens alone struggle to effectively encode. As shown in the visualization of Average Attention Weights with Only Continuous Visual Tokens in Figure 6, this results in the MLLMs paying less attention to continuous visual tokens during the decoding phase.\n\n2. **Upon introducing discrete visual tokens, which align more closely in semantic granularity with text tokens, the MLLM was able to better grasp the high-level semantics from the discrete tokens.** This alignment allowed the model to focus more attention on these tokens, thereby enhancing its ability to establish semantic associations across multiple images.\n\n3. Moreover, to further verify the semantics of the discrete visual tokens that the MLLM focuses on, as illustrated in the figure within the rebuttal PDF, we collected images that contain the discrete visual tokens targeted by the LLM. We discovered that these images consistently feature the American flag. Therefore, it can be inferred that the semantics of the corresponding discrete visual tokens are associated with the American flag.\n\n**In summary, while discrete tokens do improve efficiency, their primary advantage lies in their ability to bridge the semantic gap between multi-image visual representations and text representations.** This alignment enhances the model's ability to understand high-level semantics and establish meaningful associations across multiple images, ultimately improving multi-image understanding.\n\n### Reference\n\n[1] *An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models.* \n\n[2] *Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations.*  \n\n[3] *InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning* \n\n[4] *Q*wen-VL: A Frontier Large Vision-Language Model with Versatile Abilities*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S74MgwPAMN", "reviewer": "Reviewer_TdkD", "summary": "This paper presents a Multi-granularity Visual Encoding framework (MaVEn) for better multi-image reasoning. MaVEn combines discrete visual symbols and continuous representation sequences, as well as designing a dynamic reduction mechanism to efficiently and effectively process and interpret information from multiple images. Experimental results demonstrate its effectiveness in various multi-image benchmarks.", "questions": "Overall, I appreciate this paper, but some of the concerns raised in the 'Weaknesses' part, especially Q3, have prevented me from giving a higher score.", "limitations": "None.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-structured and clear in its presentation.\n2. I am highly impressed by the author's methodological design, particularly the Multi-Granularity Hybrid Encoding component, which I believe makes valuable and insightful contributions to the research community.\n3. The thorough experiments and visualizations presented in the paper effectively demonstrate the efficacy of the proposed method.", "weaknesses": "1. In Stage 3, would the adjustments to the Visual Projector affect the performance of the Patch Selector, since that component has been frozen?\n2. The steps of training the patch selector using Grounded SAM annotated data seem a bit redundant. Directly selecting patches based on the similarity between the patch and the discrete token may be a simpler and more effective approach.\n3. The role of the continuous tokens has not been well validated. In Figure 6, the attention seems to barely focus on the continuous tokens. Does this suggest that the continuous tokens have little impact on the performance, and they could potentially be discarded? Is it possible that the current evaluation design is unable to fully reflect the role of the continuous tokens? \n4. In line 202, '567' should be '576'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Would the adjustments to the Visual Projector affect the performance of the Patch Selector? \n\nThe adjustments made to the Visual Projector in Stage 3 will not affect the performance of the Patch Selector. As shown in Figure 2(b) in our paper, the process is designed such that **we first use the Patch Selector to select the important patch tokens. Only after this selection has been made, we then input the selected patch tokens into the Visual Projector.** This sequential process ensures that the modifications made to the Visual Projector do not influence the operation of the Patch Selector.\n\n### 2. Directly selecting patches based on the similarity between the patch and the discrete token may be a simpler and more effective approach.\n\nThank you for your insightful suggestion.\n\n1. We have indeed experimented with a similar approach where we directly select patches based on the attention weights between the <EOS> token in the discrete visual token sequence and all other continuous visual tokens. However, the results are shown below which were not satisfactory. This indicates that directly using attention weights may not be an effective method to measure the similarity between discrete tokens and continuous tokens.\n\n| Token Selection Method| DemonBench Ave Score| VQA | MMbench |\n| --------|----------|-------|-----|\n |  Selection based on attention weights |  33.8 | 74.5     | 62.1        |\n |  Selection with Patch Selector |  39.0 | 79.1  | 65.2  |\n\n\n\n2. On the other hand, Grounded SAM is designed to locate relevant visual regions based on textual semantics, and given the fact (as shwon in the figure 3 in the rebuttal pdf) that for the paired image-text sample, the semantics encoded by the text captions is similar to semantics encoded by the visual discrete tokens, we decided to use Grounded SAM to generate pseudo-labels. This allows the patch selector to learn the similarity between visual discrete tokens and patches more effectively.\n\nOverall, we appreciate your suggestion and believe that the use of Grounded SAM to train the patch selector, although seemingly redundant, is a necessary step to ensure the effectiveness of our model. \n\n\n### 3. The role of the continuous tokens has not been well validated. \n\nYour comments are greatly appreciated, and we will address your concerns in two parts:\n\n1. **Clarification of Figure 6 and the importance of discrete Tokens**: We apologize for any confusion caused by Figure 6. In fact, the text question in the case presented in Figure 6 is \"What is the similarity between Figure 1 and Figure 2?\"  This case was  to highlight the challenges faced by recent MLLMs, particularly in multi-image contexts, **where the semantic granularity of the continuous visual representations and text tokens differ significantly.** In this specific case, the American flag represents a higher-dimensional, coarse-grained semantic entity that continuous visual tokens struggle to effectively encode alone. This results in the MLLMs paying less attention to continuous visual tokens during the decoding phase.  However, upon introducing discrete visual tokens, which align more closely in semantic granularity with text tokens, the MLLM was able to better grasp the high-level semantic  from the discrete tokens, thereby focusing more attention on these tokens and enhancing its ability to establish semantic associations across multiple images.\n    \n   - To further verify the semantics of the discrete visual tokens that the MLLM focuses on, as illustrated in the figure 3 within the rebuttal PDF, we collected images that contain the discrete visual tokens targeted by the LLM. We discovered that these images consistently feature the American flag. \n\n2. **The Role of Continuous Tokens-Encoding Fine-Grained Visual Details**: We are not suggesting that continuous visual tokens are useless. In fact, continuous visual tokens and discrete tokens are complementary and indispensable. Continuous tokens encode a amount of fine-grained semantic information in the image, so when the model faces scenarios that require understanding of fine-grained details in the image (where discrete visual tokens often do not contain this information), we actually need continuous visual tokens. To better validate this point, we conducted the following experiment: \n    \n    - We randomly collected 500 images from the COCO dataset and tasked a GPT-4 model to generate questions about detailed object information in each image (e.g., asking about the shape, color, number, and size of certain objects) along with four different options and the correct answer. We then tested the performance on this dataset using MaVEN with only discrete visual tokens, only continuous visual tokens, and both, as shown below table, we found that the performance of the model using only discrete visual tokens was very poor, while the performance of the model using only continuous tokens was very close to that of the model using both, which also demonstrates the importance of continuous visual tokens for understanding detailed image information.\n\n    -  As shown in Fugure 1 in the submitted pdf in this rubuttal, we have also visualized the attention distribution of MaVEn for both discrete and continuous visual tokens when the model is asked about some details of the image (e.g.,\"what is the color the dog in this image?\"). We found that MLLMs also pay attention to continuous visual tokens. Which also suggests that the continuous visual tokens provide the fine-grained detial informations for our model.\n\n\n| Model | w/ Continuous token | w/ Discrete token | Accuracy |\n|-------|---------------------|-------------------|----------|\n| MaVEN | ✓       | ×                 |    35.2      |\n| MaVEN | ×       | ✓                 |      69.3    |\n| MaVEN | ✓       | ✓                 |      70.5    |\n\n\n### 4. In line 202, '567' should be '576'.\n\nWe appreciate your attention to detail and have corrected this error in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gMvmzoZQAQ", "reviewer": "Reviewer_HJUS", "summary": "The paper presents a novel approach, termed MaVEn, which aims to enhance the performance of MLLMs in multi-image scenario by integrating discrete visual symbol sequences with traditional continuous representation sequences. This dual strategy is designed to bridge the semantic discrepancies between visual and textual information. The approach also incorporates a dynamic reduction mechanism for long-sequence continuous features, aiming to boost processing efficiency in scenarios involving multiple images.", "questions": "The manuscript presents a well-structured and insightful study. I currently have no further questions.", "limitations": "yes.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The manuscript makes a significant contribution by proposing a hybrid model that combines both discrete and continuous data representations. This is a promising approach to mitigate the issues of semantic gaps in multimodal learning. The dynamic reduction mechanism for handling long visual sequences is also an innovative solution that could have broad applications in the field.\n2) The experiments conducted are robust and comprehensive,  as highlighted in section 4.4, is crucial in demonstrating the effectiveness of the visual hybrid encoding. This section effectively showcases how MaVEn performs under different scenarios, providing empirical evidence of its versatility and reliability.\n3) The paper is generally well-written and organized. The methodology section is well-articulated and provides a clear explanation of how MaVEn operates.", "weaknesses": "To solidify the claims regarding the efficacy of the discrete visual symbol sequences used in MaVEn, it would be recommended to conduct experiments comparing the performance of these different discrete representation techniques, such as VQGAN[1] or VQVAE[2].\n[1]Taming Transformers for High-Resolution Image Synthesis.\n[2] Neural Discrete Representation Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for your insightful comments and the recognition of our work. We have carefully considered your suggestion and provide a detailed response below.\n\n\n###  1.  it would be recommended to conduct experiments comparing the performance of these different discrete representation techniques, such as VQGAN or VQVAE.  \n\nThank you very much for your insightful suggestion. We fully acknowledge the importance of validating the efficacy of the discrete visual symbol sequences utilized in MaVEn by comparing them with other discrete representation techniques, such as VQGAN and VQVAE. These methods are indeed prominent in the field and have demonstrated significant success in various applications.\n\nIn response to your recommendation, we have conducted additional experiments to compare the performance of MaVEn using different discrete representation techniques, including VQGAN and VQVAE. Additionally, we explored the potential of combining these techniques to further expand the vocabulary of the large language model (LLM).\n\nBelow are the results of our comparative experiments:\n\n|   Visual Discrete Representation | Code Book Size | DemonBench Ave score | SEED Bench acc-video | VQA | MME | MMbench test |\n|-------|--------------------------------|----------------|----------------------|----------------------|-----| --------------|\n|  SEED                   |   8192     | 39.0           | 42.1                 | 79.1                 | 1530| 65.2| -            |\n|  VQGAN              |       1024   | 37.1           | 39.2                 | 77.3                 | 1441| 61.3| -            |\n|  VQVAE              |       1024  | 36.6           | 38.4                 | 76.3                 | 1380| 60.2| -            |\n|  SEED+VQGAN       |        9216      | 39.7           | 42.8                 | 79.5                 | 1521| 65.8| -            |\n\nOur conclusions are as follows:\n1. Using SEED as the discrete visual token yields better performance compared to VQGAN and VQVAE.\n2. Combining different discrete tokenizers can enhance the model's performance. We believe this improvement is due to the different visual semantic information encoded by the distinct codebooks. By integrating multiple codebooks, we achieve a richer and more comprehensive visual semantic representation, which in turn helps improve the model's overall performance.\n\nWe appreciate your valuable feedback, which has significantly contributed to enhancing the robustness of our findings. We look forward to any further comments or suggestions you may have.\n\nThank you for your thoughtful consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our sincere gratitude for your thorough and insightful reviews of our manuscript. We greatly appreciate the time and effort you have invested in providing valuable feedback and suggestions, which will undoubtedly help us improve the quality and clarity of our work.

We are pleased to note that the reviewers have acknowledged several strengths of our paper.  Reviewer **9o1r** found *"The concept of using both discrete and continuous visual tokens is intriguing."* Reviewer **od3A** commended our paper *"the innovative multi-granularity approach"*. Reviewer **TdkD** and Reviewer **HJUS** commended *"our work makes a valuable and insightful contribution to research community."* Reivewers **od3A, TdkD** and **HJUS**  think *our experiments effectively validates the effectiveness the proposed method.*

We are encouraged by these positive comments and will strive to address the concerns and suggestions raised by the reviewers to further enhance our manuscript.

Moreover, we would like to kindly remind you that we have included the visualized experimental data and certain tabular data within the PDF document submitted for this rebuttal phase. We encourage you to review these materials. Your attention to these details is greatly appreciated.

Meta-review: The reviewers unanimously agree that the proposed approach is both novel and compelling, with some highlighting its significant technical contributions and broad applicability across various scenarios (HJUS, TdkD). The reviewers also acknowledge the thoroughness of the experiments and the clarity of the presentation. Initial concerns were raised regarding the computational cost (9o1r, Rxew), the effectiveness of continuous visual tokens (TdkD), and the adequacy of comparisons with other methods. However, the authors have addressed these concerns comprehensively in their rebuttal. After consideration of the reviews and the authors' responses, the Area Chair finds no remaining critical issues with the paper. The Area Chair agrees with the reviewers on the value of the proposed approach and recommends acceptance of the paper.
Decision: Accept (poster)