Dense Connector for MLLMs
Authors: Huanjin Yao
Wenhao Wu
Taojiannan Yang
YuXin Song
Mengxi Zhang
Haocheng Feng
Yifan Sun
Zhiheng Li
Wanli Ouyang
Jingdong Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal Large Language Models
Vision-Language Model
Primary area: machine_vision
TL;DR: We introduce the Dense Connector - a simple, effective, and plug-and-play vision-language connecter that significantly enhances existing MLLMs by leveraging multi-layer visual features, with minimal additional computational overhead.

Abstract: *Do we fully leverage the potential of visual encoder in Multimodal Large Language Models (MLLMs)?* The recent outstanding performance of MLLMs in multimodal understanding has garnered broad attention from both academia and industry. In the current MLLM rat race, the focus seems to be predominantly on the linguistic side. We witness the rise of larger and higher-quality instruction datasets, as well as the involvement of larger-sized LLMs. Yet, scant attention has been directed towards the visual signals utilized by MLLMs, often assumed to be the final high-level features extracted by a frozen visual encoder. In this paper, we introduce the **Dense Connector** - a simple, effective, and plug-and-play vision-language connector that significantly enhances existing MLLMs by leveraging multi-layer visual features, with minimal additional computational overhead. Building on this, we also propose the Efficient Dense Connector, which achieves performance comparable to LLaVA-v1.5 with only 25% of the visual tokens. Furthermore, our model, trained solely on images, showcases remarkable zero-shot capabilities in video understanding as well. Experimental results across various vision encoders, image resolutions, training dataset scales, varying sizes of LLMs (2.7B→70B), and diverse architectures of MLLMs (e.g., LLaVA-v1.5, LLaVA-NeXT and Mini-Gemini) validate the versatility and scalability of our approach, achieving state-of-the-art performance across 19 image and video benchmarks. We hope that this work will provide valuable experience and serve as a basic module for future MLLM development. Code is available at https://github.com/HJYao00/DenseConnector.

Dense Connector for MLLMs
Huanjin Yao!.3; 1,3", Wenhao W Taojiannan Yang', Yuxin Song", Mengxi Zhang
Haocheng Feng", Yifan Sun Zhiheng Li Wanli Ouyang", Jingdong Wang"
Shenzhen International Graduate School, Tsinghua University
"The University of Sydney Amazon 5 Thee Chinese University of Hong Kong
Baidu Inc.
Equal Contribution
Corresponding Author
Abstract
Do we fully leverage the potential of visual encoder in Multimodal Large Language Models (MLLMs)? The recent outstanding performance of MLLMs in multimodal understanding has garnered broad attention from both academia and industry. In the current MLLM rat race, the focus seems to be predominantly We witness the rise of larger and higher-quality instruction datasets, as well as the involvement of larger-sized LLMs. Yet, scant attention has been directed towards the visual signals utilized by MLLMs, often assumed to be the final high-level features extracted by frozen visual encoder. In this paper, we introduce the Dense Connector a simple, effective, and plug-and-play vision-language connector that significantly enhances existing MLLMs by leveraging multi Multi-layer visual features, with minimal additional computational overhead. Building on this, we also propose the Efficient Dense Connector, which achieves performance comparable to LLaVA- v1.5 with only 25% of the visual tokens. Furthermore, our model, trained solely on images, showcases remarkable zero-shot capabilities in video understanding as well. Experimental results across various vision encoders, image resolutions, training dataset scales, varying sizes of LLMs (2.7B+700B) and diverse architectures of MILMs (e.g., LLaVA-v1.5, LLaVA-NeXT and Mini-Gemini) validate the versatility and scalability our approach, achieving state-of-the-art performance across 19 image and video benchmarks. We hope that this work will provide valuable experience and serve as a basic module for future MLLM development. wwwm///aaaoo//Deeeeonneeeeo
the linguistic side.
Code is available at httts:://tttaiiia
Introduction
In recent years, Large Language Models (LLMs) led by ChatGPT [ have made remarkable ad- vancements in text comprehension and generation. Furthermore, cutting-edge Multimodal Large have rapidly expanded the capabilities LLMs include visual understanding, evolving into models capable of integrating both vision and text modalities. This has elevated MLLMs to become new focal point for research and discussion
Language Models (MLLMs) 2 3
6 7
In broad terms, the architecture of existing MLLMs can be delineated into three components: the pre-trained vision encoder (e.g., CLIP's ViT-L g or EVA-CLIP's ViT-G ②) the pre-trained LLM (e.g., OPT
100 Llama [11. Vicuna 12) etc.), and the connector (e.g., Q-former [13] [4] or linear
projection 15
16 trained from scratch bridge the vision and language models. An intriguing
trend in current MLLM research is that the focus of model learning and performance improvement seems primarily center around the language aspect (e.g.. utilizing larger-scale and higher-quality visual instruction data [17] 18). larger-sized LLMs [19 20). with less exploration into the visual signals fed into the connector. Typically, the visual encoder is frozen extract high-level visual features, which are then fed into the connector. This leads us to rethink: Have we fully utilized the existing pre-trained visual encoder?
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

LIAN L/M
VOAW2
12
604
VNE
16.8 Laye
24 Layer
(Top Layen
24th Layer
MottWist
(a) Visualizing Attention Maps across ViT-I. Layers.
5Q4 rE.
Full Encoder
66.2
VIT Encoder
VIT
VIT
Blocks
Blocks
ToxtV
LLak VV
Final Visual Embeddings
Connected Visual Embeddings
(b) Visual Embeddings: Final
Dense Connectivity in MLLMs
(c) Plug-and-Play Dense Connector
Figure 1: Exploring Multi-layer Visual Features Empowering existing MLLMs.
In addition to the common practice of feeding the connector with final high-level visual features from visual encoder, an intuitive yet overlooked idea is to integrate visual features from various layers to complement the high-level features. In Fig. [ (a), we illustrate attention maps from different layers of 24-layer CLIP (8) pre-trained ViT-L [21], showing that different layers of the same visual encoder emphasize different regions of interest. Moreover, looking back at the history of computer vision, classic models (e.g., Densenet (22), FPN (23) utilize multi-layer features to enhance visual representations. In MILMs,, the vision encoder is typically frozen mitigate significant computational costs. In this context, our idea leverages the "free lunch" of utilizing offline features from different layers as an implicit enhancement of visual information without the need for additional computational overhead. Furthermore, this way also complements techniques
that directly increase visual signals, e.g., increasing image resolution | 25
26
27
28 or
introducing additional visual encoders [29 (0,]T8) This idea is both simple and efficient, while also
being sufficiently generic, logically allowing for seamless integration with any existing MLLMs.
In light of this, we propose the Dense Connector (DC), serving as a plug-and-pla vision-language connector that involves offline features from various layers of the frozen visual encoder provide the LLM with more visual cues. We explore three intuitive instantiations for the Dense Connector: 1) Sparse Token Integration (S77): We explicitly consider increasing the number of visual tokens by aggregating visual tokens from different specified layers and the final visual token. These tokens are then fed into learnable projector for mapping into the text space. 2) Sparse Channel Integration (SCI): To avoid increasing the number of tokens, we concatenate visual tokens from different specified layers in the feature dimension. They are then passed to the projector, which only maps visual tokens into the text space but also serves to reduce the feature dimensionality. 3) Dense Channel Integration (DCI): In addition to incorporating features from specified layers, we further attempt to utilize visual features from all layers. All of these instantiations yield significant improvements while utilizing just one simple learnable projector (comprising two linear layers) without introducing any extra parameters. Moreover, we conduct extensive empirical studies to demonstrate its scalability and compatibility. We summarize our contributions as follows:
We propose a simple, effective, and plug-and alug-and-pe and-play Dense Connector that enhances the visual representation of existing MILMs with minimal additional computational overhead. We hope it can serve as a basic module to continuously benefit future MI..Ms.
We demonstrate the versatility and scalability of our approach across various visual en- coders, image resolutions (336px-7768xx) training dataset scales, varying sizes of LLMs (2B (2B+70B), and diverse MLLMs architectures (e.g., LLaVA-v1.5 (16), LLaVA-NeXT 25 Mini-Gemini [88).
Our method exhibits exceptional performance across 11 image benchmarks and achieves state-of-the-an art results on 8 video benchmarks without the need for specific video tuning.

---

2 Related Work
2.1 Large Pre-trained Vision Models
The advent of pre-trained Vision Transformers (ViT) [21 has significantly propelled the advancement of computer vision. Furthermore, pre-training VIT models on web-scale image-texi pairs, e.g., CLIP g and its subsequent iterations [9] 32] 33, where vision and text encoders are simultaneously trained to bridge the gap between visual and textual modalities, has introduced zero-shot visual perception capabilities. Since then, CLIP-like models have served effective initializations and have been incorporated into various vision- -aiiuuee cross cross-modar modal models (e.g., video-text alignment [4]
35 36 371, large vision-language models [14 [5| 38 etc.). Recently, SigLIP B1] introduced pairwise sigmoid loss during training, enabling the visual encoder to demonstrate more advanced visual perception capabilities. To validate the compatibility of our Dense Connector, this paper conducted experiments on different visual encoders, including those of CLIP [8 and SigLIP (31).
2.2
Large Language Models
The exceptional text understanding and generation capabilities demonstrated by auto-regressive Large Language Models (LLMs)
40 41 have garnered significant attention. Subsequently. plethora II 4 have emerged, with notable open-source efforts like LLLMM 42 greatly
(39
of LLMS 42
propelling community contributions to LLMs research. Through instruction fine-tuning techniques, these models showcase human-like language interaction abilities, further further propelling advance- ments in natural language processing. Recent developments have seen LLMs scaled up or down to meet various application needs. Lightweight LLMs [44 45 20 46) have been developed to address computational constraints, facilitating edge deployment. Conversely, in the pursuit of exploring the upper limits of LLMs, works such as [47]19) [T] 20 have expanded LLM parameters, continuously pushing the boundaries of language capabilities. In this study, validated the scalability of our Dense Connector by employing multiple LLMs ranging from 2.7B to 70B parameters.
2.3 Multimodal Large Language Models
After witnessing the success of LLMs, researchers have shifted their focus towards enabling LLMs to understand visual signals. To achieve this, prior research has proposed compressing visual embeddings using Q-former [14 into query embeddings, followed by transforming them into text embeddings through linear projection, or directly employing MLP projection [15] to connect the visual encoder with LLM. Furthermore, following the instruction tuning paradigm 48 pioneering works 33 15]
50 significantly boost the development of MLLMs through visual instruction tuning. Subsequently, by introducing larger-scale and higher-quality datasets, efforts such as 18
have notably enhanced the visual understanding and reasoning capabilities of MLLMs. Additionally, there are
25
[7 24
works that introduce additional visual encoders 22 30 or utilize higher-resolution images [25 [8] 24
directly extend these above image-based methods to video conversational models by leveraging video instruction tuning datasets. In summary, these studies typically utilize high-level features from the frozen visual encoder as visual embeddings. However, we find that effectively leveraging offline features from different layers-the overlooked "free lunch"-car yield significant benefits. Then, we follow FreeVA (54) to directly extend the image model for video understanding without any additional video training.
to provide richer visual signal sources. Meanwhile, many studies [1
52 53
3 Method
3.1
Overview
In Fig. 2 a), we illustrate the overall architecture of our model, using the mainstream LLaVA [15] framework as an example. It includes the pre-trained visual encoder Vis(-) and the Large Language
Model LLM(()) (-), alongside with our proposed Dense Connector DC(-). Similarly, DC can be seamlessly extended to other high-resolution or dual-branch MLLMs, such as LLaVA-NeXT and Mini-Gemini (18). Formally, the introduction is follows:
Visual Encoder: We utilize a CLIP pre-trained Vision Transformer (VIT) 21 as the visual encoder for extracting visual features. Initially, ViT partitions image X; B XxxWxx into sequence of
3

---

(a) Overview
Tuned
Shallow
Middle
High Level
Question
Tokenizer
Level
Frezen
Large Language Model
Answer
Full Encoder
VIT
ViT
ViT Blocks
Dense Connector
Blocks
Blocks
(b) Three Instantiations of Dense Connector
Sparse Channel Integration
Projector
Prujector
21-2/00 NHO
N=30
D
N
NxD
NKD
NxD
N
Down Sample
Down Sample
Concatenate
N*D
NxD
NHD
N=D
N=D
safe W1T
11A
PM
safe 1it
Jafe 111
LIA
Jafe
1 Sparse Token Integration (STT)
2 Sparse Chunnel Inlegration (SCI)
3
Dense Channel Integration (DCI)
Figure Dense Connector MILM: Overview and Three Instantiations. the number of tokens,
D is the feature dimension, and a is the downsampling ratio.
non-overlapping patches. Each patch is then processed via convolution to produce visual tokens, which subsequently input into VIT. This procedure yields L layers of visual features V RIXNxD, where N denotes the number the visual tokens and D, denotes the feature dimension.
The Dense Connector comprises two components: the first integrates multi-layer visual features, elaborated upon detail in Sec. 3.2 while the second employs a learnable MLP to map the integrated visual features to the LLM's ' text space. The MLP consists of two linear layers with a GELU [55) activation function sandwiched between them. The first layer adjusts the visual hidden size D. to align with the LLM's hidden dimension Dr, while the second layer maintains the dimensionality at D.. Upon processing through the Dense Connector, we acquire visual embeddings C, Nx that encapsulate information from multiple layers.
Dense Connector:
Large Language Model:
The LLM processes textual data using a tokenizer and text embedding module to convert language into its input feature space. These text embeddings are concatenated with transformed visual embeddings before being fed into the LLM for subsequent predictions.
3.2
Dense Connector
Here we delve into three intuitive instantiations of the Dense Connector, each demonstrating superior performance compared to the baseline (e.g.. LLLV [66). Among them, Sparse Token Integration (STI) and Sparse Channel Integration (SCI) sparsely select visual features from K layers (indexed as I., where < I < L and < n < K) spanning shallow, middle, and high levels out of the total L layers of ViT, while Dense Channel Integration (DCI) utilizes features from all layers. These features are then fed into Dense Connector generate visual embedding that can be "understood" by LLM.
Sparse Token Integration (STI): While existing methods typically rely solely on features from the final layer as the visual representation input for the LLM, our STI approach diverges from this by integrating features from multiple layers enrich the visual input for the LLM. Recognizing that ligher-level features contain richer semantic information crucial for visual signal perception in VLMs, we maintain the final layer features unchanged while downsampling additional visual features from other layers by using average pooling avg(-) with stride a. This downsampling reduces the number of visual tokens to N' N/a, mitigating computational overhead and redundancy. These visual features from various layers are concatenated along the token dimension and processed through shared M LP(),, yielding more robust visual embedding e E NN'---Xn'''XXX

---

G. = (CCMI//CCoaatteeteeoa/a)))--)aa. (vg()))) Vi), dim = token)).
Sparse Channel Integration (SCI): We delve deeper into connecting multi-level features along the channel dimension. Subsequently, this feature is processed through an MLP projector to obtain visual embedding By Nx:
Cy = ML.P(Conadentte(VV Vige VE], dim channel)).
(2)
The MLP projector serves dual functions: integrating various features as fusion tool and facilitating the transformation of visual inputs into linguistic representations. This design ingeniously leverages dimensionality scaling effect of the MLP projector, enabling the transformation of connected multi- layer features into the feature space of the LLM without requiring additional modules. Moreover, this method does not increase the number of tokens fed into the LLM, thereby avoiding any increase in the computational overhead of the LLM.
Dense Channel Integration (DCT): While Sparse Channel Integration incorporates features from K layers, many visual feature from other layers remain unused. Concatenating all visual feature layers using STI or SCI leads excessively high dimensions, posing challenges during training. To address these issues, we propose DCI, which builds upon the SCI method by integrating adjacent layers to reduce redundancy and high dimensionality. This approach ensures dense connectivity across a wider range of visual layers. Specifically, we partition the features of L layers into G groups, where each group comprises M adjacent visual features, with M = L/G. Summing the features within each group, denoted as GVg. finally yields G fused visual representations:
gM
1 M
GV
Σ
V, 1595G.
Subsequently, we concatenate these features from the G groups with the final layer's features along the channel dimension before passing them through MIPP
e, MMLP((Cmeatennaeee(VV.. GVa, Ví), dim channel))-
3.3 Efficient Dense Connector for Visual Token Optimization
For MLLMs, each image is converted into hundreds or even thousands of visual tokens, and this large number tokens increases the computational burden on autoregressive models. However, reducing the number of visual tokens generally leads noticeable drop in performance. In this paper, we leverage multi-layer visual features to compensate for the information loss caused by reducing visual tokens, enabling
method to achieve performance par with LLaVA-v1.5 [16] while using several times fewer visual tokens, and surpassing other carefully designed efficient 56, 57] Specifically, after obtaining the visual embedding e, through the Dense Connector, we apply a parameter-fin free module, i.e. a 2D interpolation function, to downsample visual tokens. Then, these discrete visual tokens e,, concatenated with the text tokens and fed into the LLM, resulting in times improvement in inference speed.
connectors 24 50
3.4 Training-Free Extension from Image Video Conversational Models
we extend the iasee- models trained as described above to video domain for video understanding. Specifically, given video, we uniformly sample T frames, and each frame is processed through the visual encoder and dense connector to obtain visual embedding e,. Consequently, we obtain an embedding sequence {Eu, { Evr }. which is then fed into the LLM.
Following VreeVA 54
5

---

4 Experiments
4.1 Implementation Details
Architecture. 1) Visual Encoders: To explore the generalization of the Dense Connector across different visual encoders, we select two mainstream options, namely HIIP-ViT-L-3366 8 and SigLIP-ViT-SO (31). 2) LLMs: The Dense Connector is applied across various LLMs, spanning from
2.7B to 70B parameters. This includes Phi-2-2.7B 44, Vicuna- Cecuna-BB&133 13B (1). Hermes-2-Yi-34D [1]] and
3) Dense Connector: For the 24-layer CLIP-ViT-L-336p we specifically target visual features from the 8th, 16th, and final 24th layers for both STI and SCI. For STI, we apply a downsampling factor of a = 8 for the features from the 8th and 16th layers. For DCI, we divide all layer features into two groups, each containing 12 layers (i.e., 1-12, 13-24). Note that we use I to denote the stem layer of ViT, e.g., 2-25 correspond to the 24 layers of ViT-L.. Similarly, for the SigI.IP ViT-SO 31, which has 27 layers, we partition the first 26 layers into two groups (i.e., 1-13, 14-26)..
(8),
Training Datasets. Data quality plays crucial role in determining the performance of MILMS. In this study, we examine the impact of two high-quality training datasets on our model: LLaVA-1.5 [16] and Mini-Gemini [T8. The LLaVA-1.5 pre-training dataset comprises 558K image captions, while its instruction tuning dataset contains 665K conversations.
builds upon LLaVA-1.5, offering a larger dataset with 1.2M image-text caption pairs for aligmment and 1.5M conversations for instruction tuning. Unless otherwise specified, all experimental results are based on the LLaVA-1.5 dataset to reduce training costs.
Training Recipe. We train all models on 8 NVDIA A100 GPUs with 40GB VRAM, except for the Yi--3B- and LLLama---0B--nntruue
which utilize 32 NVDIA A100 GPUs with 80GB VRAM. Our training process comprises two stages: pre-training and instruction fine-tuning. In the pre-training phase, we initialize the visual encoder and LLM with pre-trained weights, while the Dense Connector is randomly initialized. Here, we freeze the visual encoder and the LLM, updating only the parameters of the Dense Connector. The model undergoes pre- pre-training for one epoch with a global batch size of 256 and learning rate of 1e-3. Subsequently, in the instruction fine-tuning stage, we maintain the visual encoder frozen while updating the Dense Connector and the LLM. Fine-tuning is performed for epoch with global batch size of 1 128 and a learning rate of 2e-5. For models using LoRA fine-tuning, we set the LoRA rank to 128 and LoRA alpha to 256. When scaling up the L.M to larger parameter sizes, such as L.aama---70B-Insttii
we apply LoRA fine-tuning due to memory constraints. this setup, we set the LoRA rank to 128 and LoRÁ alpha to 256.
Evaluation. We present comprehensive results across various image and video evaluation bench- marks. For image datasets, we include GQA [58], VQAV2 =2) [5], ScienceQA (SQA') (60] TextVQA (VQA7) ) 611, POPE 62), Math Vista (Math) 633, MMBench (MMB) 644, MM- Vet
(MMV) 65), MMMU 666, LLa VA- LLaVA--ench--n we evaluate zero-shot performance on open open-ended video question- -ansiirin benchmarks such as
(LBW)
1S and MME 67 Additionally.
MSVD-QA 688, ActivityNet QA (69) MSRVTT-QA 770. and the newly proposed generative per-
formance benchmark (51), include evaluation metrics such as Correctness of Information (CT), Detail Orientation (DO), Contextual Understanding (CU), Temporal Understanding (TU), and Consistency (CO).
4.2 Ablation Study
Study on Instantiations of Dense Connector. In Tab. [ we discuss three proposed methods of Dense Connector. 1) Sparse Token Integration (STT): This method uses visual features from different hierarchical levels independent visual prompts for the LLM, allowing to perceive a more diverse set of visual features. We select features from the 8th, 16th, and 24th layers, resulting in significant improvements across various datasets, particularly achieving a 2.9% increase on the MMB (64). Further expanding the selection to include the 8th, 16th, 20th, and 24th layers enhances performance but also increases token count, leading to higher training costs and inference time. 2) Sparse Channel Integration (SCI): This method efficiently uses the MLP projector for feature fusion and projection, integrating multiple layers of visual features into visual embeddings. SCI boosts performance with minimal additional computational cost, achieving peak performance with features from the 8th, 16th, and 24th layers, resulting in 1.7% improvement on GQA. SCI performs better than STI and mitigates the computational costs associated with higher token counts. However, expanding the range of visual

---

Table 1: Ablations on Visual Layer Selection in Dense Connector. Here, we explore three instantia- tions (STI, SCI, and DCT) of our Dense Connector integrated with the baseline (i.e., -LaVA-1.5 [T6]. which utilizes a 24-layer CLIP-ViT-L-333p3
Model
Layer Index
GQA
VQA
SQA
VQA
POPE
MMB
MMV
LBW
Baseline
24
62.0
78.5
66.8
58.2
85.9
64.3
31.1
65.4
STI
8.16.24
63.3
79.1
68.0
58.0
85.8
67.2
30.9
65.5
S77
8,16,20,24
63.0
79.1
68.0
58.8
85.9
67.6
30.8
65.7
+ SCI
8,16,24
63.7
79.2
68.9
58.2
86.1
66.2
32.2
66.0
+ SCI
16,24
63.0
79.0
67.6
58.2
86.0
65.6
31.7
65.6
+ SCI
8,16,20,24
63.6
79.2
67.0
58.1
86.0
65,8
31.9
66.0
+ DCI
(1-1))9-16)
63.6
79.3
67.8
58.6
86.3
66.5
32.6
66.0
DCT
(1-12)((13-24)
63.8
79.5
69.52-71
59.2
86.6
66,8
32,71.6
66,1% 9.71
Table 2: Exploring the Compatibility and Scalability of Dense Connector (DC). Scaling results on visual encoder (VE), resolution (Res.), pre-training (PT) / instruction tuning (IT) data, and LLM provided. "0.5M+0.6" 6M" denotes the training data from LLaVA-1.5 [[6]. while 1.22++1.5M denotes the data from Mini-Gemini [88. indicates results evaluated using official model.
Method
VE
Res.
PT+IT
L.M
GQA SQA' VQA MMB MMV MMMU" Math
Scaling to more powerfal visual encoder
LLaVA 16
CLIP-L
336
0.5M-0.6M Vicuna-7B 62.0
66.8
58.2
64.3
31.1
35.3"
24.9
LLaVA 16
CLIP-L
336
0.5M+0.6M Vicuna-13B 63.3
71.6
61.3
67.6
36.1
36.4
27.6
DC (w/ LLaVA)
CLIP-L
336
0.5M+0.6M
Vicuna-7B
63.8
69.5
59.2
66.8
32.7
34.8
26.9
DC (w/ LLaVA)
Sig
384
0.5M+0.6M Vicuna-7B
64.2
70.5
62.6
68.4
35.4
36.7
25.5
DC
(w/ LLaVA)
SigLIP-SO 384
0.5M+0.6M Vicuna- 13B 65.4
73.0
64.7
71.4
41.6
34.3
29.6
Scoling to larger training
DC (w/ LLaVA)
SO
1.2M+1.5M Vicuna-7B 63.8
72.9
64.6
71.7
45.0
35.8
33.1
DC (w/ LLaVA)
SigLIP-SO
1.2M+1.5M Vicuna-130 13B 64.6
77.1
65.0
74.4
37.2
Scaling high resolution with dual visual encoder
CLIP-L
336
MGM 18
1.2M+1.5M Vicuna-7B 62.6"
70.4"
65.2
69.3
40.8
36.1
31.4
+ConvX-L
+768
CLIP-L
336
MGM 13
1.2M+1.5M Vicuna- 63.4" 72.6*
65.9
68.5
46.0
38.1
37.0
+ConvX-L
+768
CLIP-L 336
DC (w/ MGM)
1.2M+1.5 Vicuna-7B 63.3
70.7
70.7
42.2
36.8
32.5
CConxX-L
+768
CLIP-L 336
DC (w/ MGM)
1.2M+1.5M Vicuna- 13B 64,2 74.9
66.7
70.7
49.8
39.3
38.1
X-L +768
Scaling to dynomic high resolution
NeXVA-NeXT 16 CLIP-L
AnyRes 0.5M+0.6M Vicuna-7B
64.0
69.5
64.5
66.5
33.1
35.4
25.7
DC (w/ LLaVA)
CLIP-L
AnyRes 0.5M+0.6M Vicuna-7B
64.6
70.5
65.6
67.4
33.7
37.6
26.2
DC (w/ LLaVA)
SigL.P-SO AnyRes 0.5M+0.6M Vicuna-7B
64.8
69.3
66.5
67.2
34.8
36.3
27.0
layers within SCI does not yield additional performance gains, suggesting that merely extending the range of visual feature layers ineffective. 3) Dense Channel Integration (DCI): Building on the performance enhancements of SCI, DCI integrates broader array of visual features using grouped additive fusion to produce robust visual representations. We divide the visual features into or groups, each with equal number of layers. For the CLIP-L model, each group combines features from 12 or 8 layers, respectively. Splitting them into groups demonstrates superior performance, achieving improvements of 2.7% on SQA [60] and 2.5% on MMB [64] compared to the baseline. The experimental results illustrate that utilizing multi-layer visual features enhances the visual perception capabilities of the MLLMs, leading to more accurate responses. Unless otherwise specified, employ DCI as the default instantiation of the Dense Connector for optimal performance.
Study on Visual Encoders and Training Dataset Impacts. Given our method's reliance multi- layer visual features, it is crucial to assess impact across various visual backbones. As shown in Tab. 2 we first replace the CLIP-ViT-L [ with the more advanced visual encoder SigLIP-ViT-
7

---

Table Comparison of Efficient Dense Connector with Other Efficient Methods. indicates results
evaluated using official model.
Method
Res.
#Token
PT+IT
LLM
GQA
VQA"
SQA'
VQA"
MMB
MMV
Math
LLaVA 16
576
0.5M+0.6M
Vicuna-7B 62.0
78.5
66.8
58.2
64.3
31.1
24.9*
Qwen-VL-Chal Chat 24
448
256
1.4
Qwen-7B
57.5
68.2
61.5
TokenPacker
56
336
144
0.5M+0.6M
Vicuna-7B
61.9
77.9
65.1
33.0
Dense Connector
336
144
0.5M+0.6M
Vicuna-7B
62.8
79.4
68.8
58.1
67.6
34.4
25.8
Leveraging the enhanced multi-layer visual features from SigLIP- VIT--O, our Dense Connector demonstrates further performance improvements. Additionally, we investigate the influ- ence of training datasets the effectiveness of the Dense Connector. By fine-tuning our model using the larger dataset [18), we observe notable performance gains across most benchmark evaluations. The results indicate that more training data significantly enhance model performance. Specifically, our model with Vicuna- 13B achieves accuracy of 36.5% on Math Vista [63] and 77.1% on SQA [60], underscoring the significant benefits of increased training data. Moreover, when using the same train- ing data and the LLM (i.e., Vicuna-7B), our Dense Connector surpasses the dual encoder structure of Mini- [8] across the majority of benchmarks. Specifically, it achieves performance gains of 2.4% on the MMB 664. 4.2% on MM- Vet [65]. and 1.7% on the Math Vista [63] benchmark.
SO B1.
Study High- ReshlResolttion Setting. The use of high-resolution images enhance detail repre- sentation in MI.LMs has garnered considerable attention 25 extend Dense Connector the Mini-Gemini (MGM) [88 and LLAVA-NeXTI [25]. showcasing its plug-and-play capability. For MGM, we keep the high-resolution features from ConvNeXT [71] intact, applying DCI exclusively the CLIP features. We observe significant improvements across various benchmarks, as detailed in Tab. 2 including Math Vista [63]. MMB [64], and MM-Vet [65], with enhancements of 1.1% %, 2.2%, and 3.8%, respectively. In addition to the high-resolution archi- tecture of the dual visual encoder, we also extend the Dense Connector to dynamic high-resolution. specifically using the AnyRes technology from LLaVA-NeXT (25). For a fair comparison, we provide a baseline for LLaVA-NeXT trained on the same dataset. As shown in Tab. [2] Dense Connector achieves overall improvements compared to the dynamic resolution method LLaVA-NeXT as well.
18 26 27
28). In this paper, we
Study on Efficient Dense Connector. To achieve faster inference speed, we investigate an efficient Dense Connector in this study, which can accelerate inference by 3 times. As described in Sec. 3.3] we use 2D bilinear interpolation function to downsample the visual tokens by a factor of 4, reducing the number of tokens from 576 144, which decreases the training time during in the second stage on 8 A100 GPUs from 9 hours to 6.5 hours. With the same configuration of using 144 visual tokens, Dense Connector outperforms the carefully designed efficient connector method Tokenpacker (56) by 0.9%, 1.5%, 2.5% and 1.4% on GQA [58] VQAv2 [59], MMB (64, and MM-Vet (65). respectively.
4.3 Main Results
Comparison with SoTAs in Image Understanding. In Tab. 4 we scale the LLMs from 2.7B to 70B parameters and compare them with state-of-ther -hr---rtttt--ttt-e MLLMs. When considering lightweight models, our Dense Connector surpasses the previous MLLM, Tiay (72), achieving a 1.7% enhancement
on the MM-Vet benchmark using the same fine-tuning data and foundation model. Furthermore, using same training data and LLM, our Dense Connector outperforms the VA-1.5 Vicuna 13B
[6] with substantial gains of 2. 2.1%, 3.7% and 5.5% on the GQA (88) MMB benchmarks, respectively. Notably, even with data solely from I.L our 13B model achieves performance comparable to MGM [18) which is trained on larger datasets, including 1.2M+1.5M data. Moreover, utilizing the advanced open-source LLM Llama3-8B-Insee
[4] and MM-Vet [65
our model significantly surpasses -L--VA- [76] with improvements of 5.5%, and 52 on MMB [64, and MME" [67], respectively, highlighting the contribution of our Dense Connector. By scaling up the LLM to 34B and 70B, Dense Connector achieves further improvements leveraging more powerful language models. The 70B model attains scores of 82.4% on SQA 600 and 79.4% on MMBench [64]. We then increase the resolution using AnyRes technology (25 and fully fine-tuned the LLM. Our 13B model outperforms MGM and L.LVV--NeTT on MMBench 644 and SQA 60) achieving scores of 72.3% and 72.6%. The 34B model achieves scores of 81 81.2%, 59.2%, and 97. 77. on MMBench (64 MM- Vet (65). and LLa VA-Benchh-B---e--. Bench-in-the- Wild [16), respectively.
8

---

Table 4: Comparisons with State-of-the-ANe indicates the dataset have been used for training. and indicates the dataset is not publicly accessible. "PT," "IT," and "Res." denote pre- vre-training data, instruction fine-tuning data, and image resolution, respectively.
Method
PT+IT
Res.
L.M
SQA' MMB MME! MM MMMU" Math LLaVA GQA
VL. V2 52
1.2M+3.6M
336
ML-2.7B
70.0
63.2
1441
61.1
LayyLLaVA 72
0.5M+0.6M 384
Phi2-2.7B
69.9
32.1
67.9
61.3
mPLUG-Ow2 73
348M+1.2M
448
Llama2-7B
68.7
64.5
1450
36.2
32.7
22.2
56.1
Qwen- Cha
1.4B450M
448
Qwen-7B
68.2
60.6
1488
57.5
LLaVA-vi.5
0.5M+0.6M
336
Vicuna-
71.6
67.7
36
27.6
72.5
63.3
ShareGPT4V
17
1.2M+0.7M
336
Vicuna133B
71.2
68.5
1619
43.1
79.9
64,8
MobileVI.M V2
57
1.2M+3.6M
336
Vicuna-73 7B
74.8
70.8
1559
64.6
LLaMA-VID
74
0.8M+0.7M 336
Vicuna-7B
70.0
66.6
1542
65.0'
SPHINX Plus
75
16M
448
Llama2-133 13B
74.2
71.0
1458
47.9
36.8
71.7
LLaVA-LLaMA3
76
0.5M+0.6M
336
Llama3-8B
73.3
1506
36.8
63.5
CuMo 77
0.5M+0.6M
336
Mistral-7B
71.7
1429
34.3
68.8
63.2
MMI
3B+14M
1344
MMI-7B
72.6
79.0
1529
42.1
37.0
35.9
81.5
50M+1M
Llamu-2-13B
73.7
73.0
63.3*
Mini Gemini 18
1.2M+1.5M
336+768
Vicuna 13B
72.6
68.5
1.65
46.0
38.1
37.0
87.7
63.4
LLaVA-NeXT
25
0.5M+0.7M
336 Amy
Vicuna- 13B
73.6
70.0
1575
48.4
36.2
35.3
87.3
65.4
Scoling wider runge of parumeter
708) LLMs
Dense Connector
0.5M+0.6M
384
Phi2-2.7B
70.3
70.5
1487
33.8
36.6
28.2
65.1
61.5
Dense Connector
0.5M+0.6M
384
Vicuna-7B
70.5
68.4
1523
35.4
36.7
25.5
67.4
64.4
Connector
0.5M+0.6M
Vicuna-
Dense Connector
0.5M+0.6M
384
Llama3-28 3-BB
75.2
74.4
1558
34.6
40.4
28.6
68.8
65.1
Dense Connector
0.5M+0.6M 6M 384
Yi-34B
80.5
77.7
1588
41.0
47.1
33.5
75.1
63.9
Connector
0.5M+0.6M
384
Llama3-70B LORA
82.4
79.4
1622
46.1
32.9
74.5
Dense Connector
1,2M+ 1,5M
384
Vicuna-
77.1
74.4
1579
47.8
37.2
36.5
88.9
64.6
Dense Connector
1.2M+
384.nnpes
Vicuna- 7B
72.0
69.2
1535
44.4
36.4
32.7
88.8
63.9
Dense Connector
1.2M+1.5M
384
3844Amgitee
Vicuna- 13B
75.2
72.3
47.0
36.8
35.5
93.2
64.3
Dense Connector
1.2M+
3384 384AnnRes
Yi-34B
78.0
81.2
1696
59.2
51.8
40.0
97.7
66.6
Table 5: Comparisons with Leading Methods Zero-shot Video QA Benchmarks. Following Free VA [54] we specify the GPT-3.5 versions used for evaluation to ensure fairness in performance
comparison across different versions. "MAR" denotes the GPT-3. --rrr--o--o000 0330, "JUN" denotes
the GPT-3. 5- -6633-0011, and "JAN" denotes the latest GPT 5 Tor--uurooo1110 0125.
L.MM GPT-3.5 Train MSVD-QA MSRVTT-QA ActivityNet-QA Video-ChatGPP henchmark Version
Method
Size
Free
Acc Score
Ace
Score
Ace
Score
CI
DO
CU
TU
CO
FrozenBiLM
33.8
16.7
25.9
Vidco-LLaMA
52
7B
MAR
X 51.6
2.5
29.6
1.8
12.4
1.96
2.18
2.16
1.22
1.79
LLaMA-Adapter
88
7B
MAR
X
2.03
2.32
2.30
1.98
2.15
VideoChat
[
7B
MAR
X
56.3
2.8
45.0
2.5
26.5
2.2
2.23
2.50
2.53
1.94
2.24
Video-ChatGPI
51
7B
MAR
X 64.9
3.3
49.3
2.8
35.2
2.7
2.50
2.57
2.69
2.16
2.20
VaQuirA
82
7B
MAR
X 74.6
3.7
68.6
3.3
48.8
3.3
LLaVA~Free
54
7B
MAR
81.5
4.0
72.9
3.5
58.3
3,5
2.88
2.52
3.25
2.32
3.07
B7-Adupter
[33
7B
JUN
X
67.3
3.7
57.0
3.2
"45.7
3,2
2.68
2.69
3.27
2.34
2,74
Video-LLaVA
[33]
7B
JUN
X
70.7
3.9
59.2
3.5
45.3
3.3
LLaMA-VID
F4
13B
JUN
X
70.0
3.7
58.9
3.3
47.5
3.3
3.07
3.05
3.60
2.58
2.63
LLaVA=Free
54
13B
JUN
71.8
3.8
59.2
3.3
54.5
3.5
2.90
2.52
3.26
2.32
3.07
LLiVAFfree
54
13B
JAN
74.4
4.1
61.1
3.6
551.6"
3.5
2.88
2.52
3225
2.34
3300
DC+FreeVA
7B
JAN
75.0
4.1
58.4
3.5
52.2
3.5
2,80
2.51
3.17
2.22
3.05
DC+FrecVA
13B
JAN
75.1
4.1
60.8
3.5
52.6
3.5
2.85
2.53
3.23
2.29
2.96
DC+FreeVA
34B
JAN
77.4
4.2
62.1
3.6
55.8
3.6
3.00
2.53
3.25
2.65
2.92
Comparison with SoTAs in Video Understanding. Building on the training-free paradigm of FreeVA [54] for image-to-vine to-ideoo adaptation, we directly apply our models, originally trained on image-text datasets, video dialogues. As indicated in Tab, [5] the visual enhancement capabilities of the Dense Connector significantly enhance the video comprehension of our 13B model. This model
surpasses the baseline VA-15 [6] with FreeVA (44 on MSVD 688 and ActivityNet [69] By scaling up the LLM to 34B, our model demonstrates marked improvement in video understanding capabilities, achieving oate-offtthearr performance on video evaluation benchmarks with accuracies of 77.4% MSVD (68), 62.1 % on MSR-VTT (70), and 55.8% ActivityNet [69].

---

Gonmecticut law if
OIL
If women fe witch- that Koth conoulteth with
familher spiris shny shall doath
User: What are all the seene text in the image?
displayed motorcycle rddden by the motorcyclist wearing yellow suif?
DC: The
the image
"Comnecticut Law of
DC: The matorcyclist wearing yellow suit is riding metorcycle
1642. any man woman be witch, that both or consulteth with a familiar spirit, they shall be put death."
with the number 16 on it.
(a) Detail Reasoning
(b) OCR
Ford
Toyata
Honds
figrr lengh of
Usert Answer the question the figure and provide options.
327 IFA4 wt,
Total fueling cost
$70.99
What the average fueling excluding the Ford F150?
(D)
18
DC: To calculate the uverage total lueling cost excluding the Ford
DC: To find the length of AR, we use the given ratio of the lengths ofAB to AC, The ratio 2:5, which means that every units of AB, there are can set up proportion to find the length of AB:
F150, we would add the costs Toyota RAV4 and the Honda Civic and then divide by twa. The Toyota RAV4's cost is $82.56 Honda Civic's
of Since
Adding two amounis
gives $155.11, 11, Dividing this sum gives average of 576.5.
AB/25
To solve for AB, we cross-multiply AB, 50 5 AB
User: What the highest fuellig cost?
2
Dividing sides by 5 gives
DC: The highest fucling cost, shown on the bar chart, for the Ford F-150, with a total fueling cost of S130.6..
AB 50/5, AB 10
Therefore, the length of AB 10 units.
The correct is
(c) Table Reading
(d) Math Solving
are they doing video?
User: What animal in video, and what doing?
DC: The people in the video ure playing with their dogs in grassy field. They are throwing a frishee for the dogs to catch, and the dogs running to catch
DC: The animal in the video is giraffe, and it is standing diving board, preparing to jump into the pool.
User: Is this video real or synthesized??
User: What breed is this dog?
DC: This compater-generatel video of giraffe diving into
The dog vidco German Shepherd.
pool, which real.
(e) Video Understanding
(f) Video Reasoning
Figure 3: Quantitative Results for Image and Video dialogues. Figures (a) through (d) pertain to image understanding, while figures (e) and (f) relate to video understanding.
Qualitative Results. In Fig. illustrate model's exceptional visual understanding and text generation capabilities across various scenarios, encompassing both image and video. More qualitative results are provided in Appendix A.5
Conclusion and Limitation
In this paper, we introduce the Dense Connector, novel plug-and-plan module that enhances visual perception capabilities of MILMS by densely integrating multi-layer visual features. We instantiated three types of Dense Connector and validate the efficacy of it across a diverse array of vision encoders, LLMs, and training datasets, demonstrating substantial improvements evaluation benchmarks. Dense Connector can be easily integrated into existing MLLMs. In this work, we incorporate the Dense Connector into mainstream model LLaVA and high-resolution method Mini- Mini-Gemini, demonstrating its versatility and generalization capabilities.
performance across multiple
Our three Dense Connector instantiations do not introduce additional parameters, leaving room for further exploration. We have not yet found an effective method for incorporating additional parameters. Future research will focus on discovering more efficient ways to connect visual and language models for better modality alignment.
Limitations:
10

---

References
[1] OpenAL. Chatgpt. https://openaii wtt.cooo//ba// www.///////////ooo.oooo..ooooeeo http://lloocchtgt////// 2023.
[2] OpenAL. 4viiion) system card. 2023.
[3] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- rean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family highly capable multimodal models. arXiv preprint arXiv:2312.11800 2023.
[4] Wenhao Wu, Huanjin Yao, Mengxi Zhang, Yuxin Song, Wanli Ouyang, and Jingdong Wang. Gpt4vis: What can gpt-4 do for zero-shot visual recognition? arXiv preprint arXiv:2311.157333 2023.
[5] Zhengyuan Yang. Linjic Li, Kevin Lin, Jianfeng Wang. Chung-Ching Lin, Zicheng Liu, and Lijuan Wang- The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint (xXi:23099.7422 9():1 2023.
Chaoyi Wu, Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou, Ziheng Zhao, Ya Zhang. Yanfeng Wang, et al. Can gpt-4v (ision) serve medical applications? case studies on gpt-4y for multimodal medical diagnosis. arXiv preprint arXiv:2310.09009 2023.
[7] Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi. Gpt-4v (ision) for robotics: Multimodal task planning from human demonstration. arXiv preprint arXiv:231..12015, 2023.
[8] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-878 17668 PMLR, 2021.
[9] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint 2rXiv:2303.15388 2023.
[10] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diah, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0m 01068, 2022.
[11] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.00 002888, 2023.
[12] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
[13] Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch. Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for 2022.
few-shot learning. Advances in neural information processing systems,
[14] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bontstrapping language-imen pre-training with frozen image encoders and large language models. In International conference machine learning, pages 19730-19742. PMLR, 2023.
[15] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information procesing systems, 36, 2024.
[16] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.
[17] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui Hc, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint urXiv:2311.12793. 2023.
[18] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Vixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 2024.
[19] Alex Young, Bei Chen, Chao Li, Chengen Huang. Ge Zhang. Guanwei Zhang, Heng Li. Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint 203.:2200..46652 2024.
11

---

[20] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2309.16600 2023.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2020.
[22] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4700 4708. 2017.
[23] Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie, Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2225, 2017.
[24) Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision- vision-language model with versatile abilitics. arXiv preprint arXiv:2308. 12966, 2023.
[25] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[26] Bo Li. Peiyuan Zhang, Jingkang Yang. Yuanhan Zhang. Fanyi Pu, and Ziwei Liu. Otterhd: high- resolution multi-modality model. arXiv preprint wrXiiv2311.042111 2023.
[27] Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.030F 2023.
[28] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng. Floris Weers, al. Mml: Methods, analysis insights from multimodal Ilm pre-training arXiv preprint arXiv:2403.090 2024.
[29] Dongsheng Jiang, Yuchen Liu, Songlin Liu, Xiaopeng Zhang, Jin Li, Hongkai Xiong, and Qi Tian. From clip to dino: Visual encoders shout multi-modal large language models. arXiv preprint arXiv:2310.0882. 2023.
[30] Shengbang Tong, Zhuang Liu, Yuexiang Zhai Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Ims.. arXiv preprint arXiv:2401.06209 2024.
[31] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975-11986, 2023.
[32] Mehdi Cherti, Romain Beaumont, Ross Wightman. Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-2829, 2023.
[33] Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu, Minzhe Niu. Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-iman -mmage pre-training. ICLR, 2021.
[34] Bo Fang, Wenhao Wa, Chang Liu, Yu Zhou, Yuxin Song, Weiping Wang, Xiangbo Shu, Xiangyang Ji, and Jingdong Wang. Uatvr: andertiinty-adaptivee vixt- retrieval. In ICCV, 2023.
[35] Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang. Cap4video: What can auxiliary captions do for text-video retrieval? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[36] Wenhao Wu, Zhun Sun, Yuxin Song, Jingdong Wang, and Wanli Ouyang. Transferring vision-language models for visual recognition: A classifier perspective. International Journal of Computer Vision, pages 1-18, 2023,
[37] Wenhao Wu, Xiaohan Wang. Haipeng Luo. Jingdong Wang. Yi Yang, and Wanli Ouyang. Bidirectional cross-modal knowledge exploration for video recognition with pre- pre-trained laniun-language models. In CVPR, pages 6620-6630, 2023.
[38] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arX4v:2304.10592 2023.
12

---

[39] Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learner. Advances in neural information processing systems, 33:1877-1901, 2020.
[40] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[41] Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Lc. XInet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[42] Hugo Touvron, Thibaut I Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.139) 2023.
[43] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
[44] Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorn, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023.
[45] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint Xii::2441.02.3850 2024.
[46] Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable Im 2 1.6 b technical report. arXiv preprint arXiv:2402. 178344 2024.
[47] Albert Q Jiang, Alexandre Sahlayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprini arXiv:2401.0m 04088, 2024.
[48] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[49] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quae V Le. Finetuned language models are zero-shot learners. urXiv preprint arXiv:2109.0163: 2021.
[50] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose lssoon-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[51] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05422 05424, 2023.
[52] Hang Zhang. Xin Li. and Lidong Bing. Video-llama: An instruction -ouneddeett-tededdddeeeee audio-visual language model for video understanding. arXiv preprint urXiv:2306.02858, 2023.
[53] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311. 10122, 2023.
[54] Wenhao Wu. Freeva: Offline milm as training-free video assistant. arXiv preprint arXiv.2405.07798 2024.
[55] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), arXiv preprint arXiv:1606.08415 2016.
[56] Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal Imm. arXiv preprint arXiv:2407.02399 2024.
[57] Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang. Xiaofei Sun, Yiming Hu, Xinyang Lin. Bo Zhang, et al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03770 037666 2024.
13

---

[58] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Pruceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[59] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904 6913, 2017.
[60] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan, Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances Neural Information Processing Systems, 35:2507-2521. 2022.
[61] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[62] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ro-R Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[63] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai-Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.0225: 2023.
[64] Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model all all-around player? arXiv preprint arXiv:2307.06288 2023.
[65] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02453 2023.
[66] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu. Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benehmark for expert agi. arXiv preprint arXiv:2311. 16502, 2023.
[67] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang. Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.133:
2023.
[68] David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL, 2011.
[69] Fabian Caba Heilbron, Victor Escorcia, Bemard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
[70] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
[71] Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976 1 11986, 2022.
[72] Baichuan Zhou, Ying Hu, Xi Weng. Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework mall--ccaee large multimodal models. arXiv preprint arXiv:2402. 14289, 2024.
[73] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint 2xXi:22311.04257 2023.
[74] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint 1xXx::2311117044 2023.
[75] Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijic Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x, x Scaling data and parameters for family of modai large language models. arXiv preprint arXiv:2402.05933. 2024.
[76] XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning 11m. http::://githuu..com/ninne xtuner 2023.
14

---

[77] Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-Wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin Wen. Cumo: Scaling multimodal Ilm with co-upeycled mixture-of-experts. arXiv preprint arXiv:2405.0594 2024.
[78] Ji Lin, Hongxu Yin. Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao. Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training visual language models, arXiv preprint arX1v:2312.07533, 2023.
[79] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. Advances in Neural Information Processing Systems, 35:124- 141, 2022.
[80] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama- Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 2023.
[81] KunChang Li. Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang. and Yu Qiao. Videnchat: Chat-centric video understanding. arXiv preprint 2.355:2305.0663 2023.
[82] Yizhou Wang, Ruiyi Zhang, Haoliang Wang, Uttaran Bhattacharya, Yun Fu, and Gang Wu. Vaquita: Enhancing alignment llm-assisted video understanding. arXiv preprint arXiv:2312.02310,, 2023.
[83] Ruyang Liu. Chen Li, Yixiao Ge. Ying Shan, Thomas H Li, and Ge Li. One for all: Video conversation is feasible without vidco instruction tuning. arXiv preprint arXiv: 27879:2309./5775, 15785, 2023.
[84] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[85] Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang. Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger ilms supercharge multimodal capabilities in the wild, May 2024.
[86] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint wxXv:2106.09685. 2021.
15

---

A Appendix
A.1 Further Exploration and Analysis of the Dense Connector
In Sec. 3.2 we provide detailed discussion of three instantiation methods of the Dense Connector:
Sparse Token Integration (STI), Sparse Channel Integration (SCT), and Dense Channel Integration (DCI). Applying our proposed Dense Connector to representative VLMs (e.g.. LLaVA (T6] results in notable performance improvements. In this section, building on the STI, SCI, and DCI methods, explore instantiations of Dense Connectors with additional learnable parameters.
Sparse Token Integration with ID Convolutional Downsampling. The STI instantiation method concatenates visual tokens from various layers, which significantly increases the total number of tokens. To reduce redundancy and computational overhead, we initially employ average pooling to reduce the number of tokens from shallow layer features. In computer vision, using convolution for downsampling a common practice X4 Here, we replace average pooling with a single 1D)
convolutional layer Conei (- (-), to downsample the shallow layer tokens. We set both the kernel size and stride to 8, maintaining consistency with the average pooling as detailed in Sec. 4.1 The formula is as follows:
Coi V2], dim token)).
(5)
Sparse Channel Integration with 2D Convolutional Modelling. The SCI method utilizes the projector as both a feature fusion tool and a modality mapper, transforming multi-layer visual features concatenated along the channel dimension into the input feature space of the LLMs. By straightforwarddy concatenating multi-layer visual features, the SCI method achieves significant performance enhancements for VLMs with minimal computational overhead. As a central element of the Dense Connector, we aim to enhance local perception abilities by processing visual features through 3 2D convolution ConUDD. with shared weights prior to feature concatenation:
e, LMLLLCCConteeeaeeeCCoCiiioi (Vi,), ConvaD (Vig) ConveD (on)),,,,CCCoa((( dim channel)).
Dense Channel Integration with Linear Layer. The DCI method initially groups visual features and then aggregates them, enabling the fusion of adjacent visual features without expanding the dimensionality. This technique mitigates the issues of feature redundancy and excessive channel dimensions that arise in the SCT method from incorporating too many layers. To enhance the integration of features from different visual layers across groups, each layer of visual features is processed through a linear layer with shared weights. The formula is as follows:
GV, = M
Σ
Linear(Ln(V.)), 1595.. G.
(7)
1)M41
e, = MMMMM(Conatonnte((VV..... GVG. V2), dim = channel)),
(8)
where Ln(-) and Linear(-) denotes Layer Normalization and Linear layer, respectively.
Experimental Results and Analysis. We present the detailed results of these described instantiations
in Tab. [ 1) For the STI method, using features from the 8th, 16th, and 24th layers, average pooling demonstrates superior performance compared ID convolutional downsampling, especially on the GQA [58] and MMBench (64] benchmarks. 2) In the SCI method, applying 2D convolution to enhance local feature modeling with offline VIT features from the 8th, 16th, and 24th layers achieves comparable performance to methods without 2D convolution. 3) Furthermore, incorporating additional aggregation information, such as a linear layer, into the DCI method does not lead to improved outcomes.
In summary, our attempts to introduce additional parameterized modules did not yield the anticipated improvements. We hypothesize that since the connector is randomly initialized, maintaining ease of convergence is crucial for effectively training the connector to align visual and language models under
16

---

Table 6: Additional studies on the Dense Connector,
Architecture
Layer Index
GQA
VQA
SQA!
MMB
MMVet
LLaVA 16
78.5
66.8
64.3
31.1
STI
8.16,20.24
63.0
79.1
68.0
67.6
30.8
STI
8,16,24
63.3
79.1
68.0
67.2
30.9
STI w/ ID Conv
8,16,24
62.6
78.9
67.4
65.0
30.6
SCI
8,16,24
63.7
79.2
68.9
66.2
32.2
SCY
4,8,16,24
62.9
78.9
68.3
65.2
30.2
SCI
8.12.16,24
63.5
79.0
67.4
65.6
31.7
SCI
8,16,20.24
63.6
79.2
67.0
65.8
31.9
SCI 2D Conv
8,16,24
63.6
79.0
68.1
66.0
32.2
DCI
(1-12)(13-24)
63.8
79.5
69.5
66.8
32.7
DCI w/ Linear
(1-12),(13-24)
63.7
79.4
68.5
66.4
32.8
SQA
GQA
VQAv2
64.0
IX11
DCI
63.5
SCT
79.5
69
Accuracy (%)
Accuracy (%)
Accuracy (%)
LLaWA-15
79.0
LLaVA-ES
LLaVA-1.5
78.5
67-
62.0
61.5
77.5
61.0
37.0
65
1L.VA-1.
SI1
SC1
DC1
Method
Method
Method
(a) Results comparison on GQA
(b) Results comparison VQAv2
(c) Results comparison SQA
POPE
MMBench
MM-Ves
DCI
DC
DC1
87.0
67-
SCL
32.5
Accuracy (%)
STI
Accuracy (%)
STI
Accuracy (%)
86.5
LLaVA-1.5
LLLVA-1.5
32.0
LLaVA-1.5
36.0
31.5
65
84.5
63
30.0
LLAVA-15
LLAVA-15
Method
Method
Method
(d) Results comparison POPE
(c) Results comparison MMB
(f) Results comparison MetVet
Figure 4: Comparison three instantiations of Dense Connector with LLaVA-1.5. STI stands for Sparse Token Integration, SCI for Sparse Channel Integration and DCI for Dense Channel Integration.
limited training data. In current MLLMs, the MLP structure of LLaVA [66 efficiently facilitates convergence between visual and language components. However, adding extra parameters may disrupt this inherent characteristic, leading diminished performance. Future research will explore more complex and effective implementations of the Dense Connector.
A.2 Visualized Analysis
Appendix A.I we further explored more complex Dense Connector structures. However, we found that con-parameterized methods yielded better average results. Therefore, we also visualized the comparison of three main instantiations the Dense Connector with LLaVA-115 in Fig. clearly demonstrating DCI's superior performance across different benchmarks.
17

---

Table 7: More comprehensive evaluation results of the Dense Connector. "PT", "IT", and "Res."
denote pre-trareeeee -ve-triinig data, instruction fine-tuning data, and image resolution, respectively. indicatess
pre-training data (0.5M) from LLa [66 and instruction data (1 (1.5M) from Mini-Gemini [18]
Method
FT+IT
Res.
LLM
GQA
VQA
SQA!
MMB MME"
MMML
Math
MMV
LLaVA
Dense Connector 0.5M+0.6M
384
Phi2-2.7B
61.5
55.8
70.3
70.5
1487
36.6
28.2
33.8
65.1
Dense Connector 0.5M+0.6M
384
Vicuna-7B
64.4
62.6
70.5
68.4
1523
25.5
35.4
67.4
Dense Connector 0.5M+0.6M
384
LJama3-8B
65.1
62.2
75.2
74.4
1558
40.4
28.6
34.6
68.8
Dense Connector 0.5M+0.6M
384
Vicuna-13B
65.4
64.7
73.0
71.4
1569
34.3
29.6
41.6
73.6
Dense Connector 1.2M+1.5M
384
Vicuna-133
64.6
65.0
77.1
74.4
1579
37.2
36.5
47.8
88.9
Dense Connector 0.5M+0.6M
384
Y.R3
63.9
66.7
80.5
77.7
1588
47.1
33.5
41.0
75.1
Dense Connector 0,5M+0.6M
384
Llama3-70B
64.0
66.0
82.4
79.4
1622
47,0
32.9
46.1
74.5
Dense Connector 0.5M+1.5M+
384
Llama3-708
2ORA
64.1
68.3
74.5
80.2
1649
43.6
40.7
53.3
97.8
Table 8: Ablation study fine-tuning Vision Transformer. In this table, all the results conducted
using LLaVA 1.5 data, comprising 558K pre-training data and 665K iestruction-tuning data.
Method
VE
FT-VIT
Res.
LLM
GQA
SQA'
VQA"
MMB
MMV
MMMU"
Math
DC (w/ LLaVA) CLIP-L
X
336
Vicuna-7B 63.8
69.5
59.2
66.8
32.7
34.8
26.9
DC (w/ LLaVA) CLIP-L
V
336
Vicuna-7B 63.7
67.4
60.2
68.6
34.4
35.4
26.0
DC (w/ LLaVA) SOO
X
384
Vicuna-7B 64.2
70.5
62.6
68.4
35.4
36,7
25.5
DC (w/ LLaVA) SO0
384
Vicuna-7B 65.0
71.6
63.4
69.3
35.8
35,2
27.0
A.3 Model Zoo
To probe the upper limits of MILMs, we attempt to utilize a larger dataset fine-tune our model based on Llama3-70B-Instee with the Dense Connector pre-trained on LLaVA's initial stage data [16] and subsequently refine it using the instructional dataset from Mini-Gemini [18]. This model achieves accuracy of 40.7% on the Mast (63] and 53.3% on MM- Vet (65] as illustrated in Tab. 7 Remarkably, our 70B model exhibits performance that is par with GPT-4V on the LLaVA-Bench-in-thhee Wild evaluation [6]. Specifically, our model achieves 97.8%, closely trailing GPT-4V's 98% (85). However, is worth noting that the full potential of our 70B model remains untapped due to the lack of initial stage pre- pre-training data and the inherent constraints of LoRA fine-tuning [86].
Given the limitations of computational resources, we commence
A.4 Further Exploration in Training Visual Encoder
In the experiments above, we used frozen multi-layer features from pre-trained VIT to enhance the model's visual perception capabilities. Recently, however, there has been growing trend of fine-tuning the entire model, including ViT, during the instruction-tuus tutiinnnnttingggg phase to achieve better results. Following this trend, we present additional results in Tab, 8 on the performance impact of fine- tuning the ViT within the multi-layer visual connector. Specifically, in the first stage, we maintain the original training strategy. In the second stage, we fine-tune the VIT with a smaller learning rate 2c-6, keeping other hyperparameters unchanged. Tab. shows that fine-tuning ViT yields performance gains on benchmarks with a higher reliance on visual information, such as MMBench [64] and TextVQA [64). For instance, fine-tuning CLIp-Large encoder resulted in improvements of 1.8% and 1.0% on MMBench and TextVQA, respectively. Conversely, on benchmarks that are less visually dependent, such SQA (60,, performance exhibited slight decline.
A.5 More Qualitative Results
From Fig. boo Fig. 13 we present additional multimodal dialogue illustrations with images and videos. Fine-tuned image-text datasets, Dense Connector achieves effective alignment between visual and linguistic modalities, enhancing the model's visual perception capabilities. Qualitative evaluations showcases that our model can adeptly handle a broad spectrum of challenges across various domains, including mathematics, flowcharts, and daily scenarios. Additionally, the incorporation of the Free VA method [54] equips model with video understanding capabilities, enabling our model to address dynamic temporal challenges. However, due to the lack of training on video data, our model may occasionally misinterpret images videos. For instance, shown in Fig. 12] the model mistakenly interprets videos as images, incorrectly responding with 'The image" instead of 'The video'.
18

---

Dense Connector
Lang paggedi
No
Play nioupp
Vo5
Hicer sero
No
Repariamp
What is the solution if the lamp is plugged in and the bulb has burned out?
The solution if the lamp is plugged in and the bulb has burned out, indicated by the flowchart, is to "REPLACE BULB."
Dense Connector
Walt for kettle to boll
Water bolled?
N
Wat
Pour water on teabag in CUD
Which step in the flowchart is drawn with diamond shape?
The step the flowehart that is drawn with a diamond shape is "Water boiled?" This is decision point in the flowehart, where the process continue in two different directions based on the outcome of the condition being tested.
Dense Connector
tire puncture for easy repair.
from the plastic.
4
5
What is the step after using probe to spread tire puncture?
After using the probe to spread the tire puncture, the next step is to remove the repair plug from the plastic. This is shown in step of the guide, where the hands are holding the repair plug and the plastic, preparing separate them.
Figure 5: Qualitative results of the flowchart understanding.
19

---

Dense Connector
4+
7+ ==
4+7
7+2-
=
9+3
3+52
9
3 + 8
What the answer to the bottom left equation?
The answer to the bottom left equation, which is "9 + 3 =", is 12.
What is the answer to the second equation on the right?
The answer to the second equation on the right, which =", is
Fix both ends (A and B) of naturally straight clastic band, measuring 16 em in length, horizontal tabletop. Then, vertically pull the midpoint C upward by em to point (as shown the diagram). How much has the elastic band been stretched?"
To determine how much the elastic band has been stretched, we need to calculate the length of the elastic band from point A to point D and then from point D to point B. This will give us the total length of the stretched elastic band.
Using the Pythagorean theorem, we can calculate the length of AD and DB as follows:
For triangle ACD:
AC em (half of the original length 16 cm)
CD em (the height which the midpoint is pulled)
AD V(AC=+ CD*) (88 6t) (66 36) V1000 10 cm
For triangle BCD:
BC 8 cm (half of the original length of 16 cm)
CD em (the height which the midpoint is pulled)
BD (BCC CD3) = v(82 6t) <66 + 36) v1000 10 cm
Since AD and BD equal, the total length of the stretched elastic band is AD - DB 10 cm + 10 cm = 20 cm.
Therefore, the clastic band has been stretched from its original length of 16 em to a total length of 20 cm, which means it has been stretched by 4 cm.
Figure 6: Qualitative results of the math solving skills.
20

---

Dense Connector
NICETRY
Can you help me understand or explain meme? Please describe the meme or provide details about its context.
The image is a humorous take on a McDonald's meal, with sandwich and fries made out of fruits and vegetables. The sandwich made from cut apple, and the fries made from carrot sticks. The arrangement is placed a dining table, and the words "Nice Try Mom" are written above the image.
The meme is poking fun at the idea of mother trying make healthy version of a fast-food meal, but the result is not quite convincing, The phrase "Nice Try Mom" is a lighthearted way of acknowledging the effort while also implying that the outcome not entirely successful.
MONDAY.
JONDA.
Please explain the meme in detail.
The meme features photograph of small, light-colored dog lying on its side wooden floor. The dog's posture is relaxed, with its head resting the floor and its body stretched out. The image is overlaid with text in bold, white font that reads "MONDAY, JUST... MONDAY." The text play on the common sentiment that Mondays are often dreaded as the start of the workweek, and the dog's posture humorously suggests reluctance to get up and face the day, which is feeling many people can relate The meme the dog's expression and body language to anthropomorphize the animal, attributing human emotions behaviors to it, which a common technique in meme creation to evoke humor and empathy.",
Figure 7: Qualitative results of the meme understanding.
21

---

Dense Connector
What the name of this dish, and how it made?
The dish in the image is commonly known as "Mapo Tofu," traditional
Sichuan dish. It is made by first marinating and then pan-frying tofu cubes until they are lightly browned. The sauce is prepared separately by stir-frying mixture of ground pork, garlic, ginger, and fermented soybeans in chili oil. The sauce is then seasoned with soy sauce, rice vinegar, and cornstarch to thicken it. The tofu then added to the sauce cooked until it is well coated. dish is garnished with green onions and served hot.
Connector
How to cook this dish?
To cook this dish, which appears to be type of sushi roll, you would typically follow these steps:
1. Prepare the sushi rice: Rinse the rice thoroughly and cook according to the package instructions. Once cooked, allow it to cool. Mix the rice with rice vinegar, sugar, and salt to give it the characteristic sushi flavor.
Prepare the filling: Slice the ingredients you want include the roll, such eucumber, avocado, and imitation crab, into thin strips.
3. Assemble the roll: Lay sheet of nori scaweed flat on sushi mat. Spread a thin layer of sushi rice onto seaweed, leaving small border at the top. Arrange the filling ingredients horizontally in the middle of the rice.
4. Roll the sushi: Using the sushi mat, roll the nori and rice tightly but gently, applying even pressure. Make sure the roll is tight and the ingredients are evenly distributed.
5. Slice the roll: Once the roll is formed, place in the refrigerator for about Then, using sharp knife, slice the roll into individual pieces. innn. Serve: Serve the sushi with soy sauce, wasabi, and pickled ginger, as shown in the image. The yellow condiment could be grated daikon similar garnish.
30 minutes
Remember to handle the sushi rice and ingredients with clean hands or utensils to maintain hygiene. Enjoy your homemade sushi!
Figure 8: Qualitative results of the delicacies understanding.
22

---

Dense Connector
Which movie is this image from?
The image appears be still from the movie "Pulp Fiction" (1994) directed by Quentin Tarantino. The character depicted is Mia Wallace, played by actress Uma Thurman. This particular scene is memorable for its iconic dialogue and the character's distinctive style, which has become cultural reference point. The image captures moment of Mia Wallace's character in a diner, which is a pivotal scene the film.
Dense Connector
What's the ending of this movie?
The ending of the movie '"Titanicl" is tragic The film is based the real-life story of the RMS Titanic, which sank its maiden voyage in 1912. The movie's plot follows the fictional characters Jack Dawson and Rose DeWitt Bukater, who fall in love aboard the ship during its ill-fated journcy. The ship's collision with an iceberg and subsequent sinking lead to the death of Jack, who sacrifices his
save Rose. The film ends with an elderly Rose, who has been telling the story of her time on the Titanic to her granddaughter and group of explorers, throwing the \"Heart of the Seat" diamond into the ocean, finally letting go of the past. The ending poignant, reflecting on the themes of love, loss, and the passage of time.
What is the significance of this scene in the context of the movic?
The significance of this scene in the context of the movic multifaccted. It represents pivotal moment of freedom and joy for the characters, who are from different social classes and are experiencing a moment of escape from their societal constraints. The of standing the bow of the ship with arms outstretched is a classic symbol of liberation and the pursuit of dreams. It also serves as a visual metaphor for the characters' aspirations and the limitless possibilities that lie ahead, both literally as they travel across the ocean and metaphorically in their personal lives. This scene is often associated with sense hope and the beginning of a new chapter, which a central theme in the narrative of the film.
Figure 9: Qualitative results of the movie analysis.
23

---

Dense Connector
Describe the video in detail.
The video depicts a person drawing a tree with pink flowers on white canvas. The artist is using a brush to create the intricate details of the tree, which is surrounded by a large circle. The tree appears to be a cherry blossom tree, as is covered in pink flowers. The artist's hand is visible, holding the brush and working on the painting. The scene captures the creative process of the artist as they bring the tree to life on the canvas.
Compose poem inspired by this artwork.
In the midst of tranquil scene, Its branches reaching out, As to embrace the world. A hand, holding brush, Paints the tree with care and grace, Creating a masterpicce, A reflection of nature's beauty. The artist's skillful strokes, Bring life to the canvas, A tree in bloom, A symbol of growth and renewal. The moon, gentle companion, Shines above the tree, Guiding the artist's hand, As the painting takes shape. moment captured time, tree bloom, hand at work, A testament to the power of To evoke emotions and inspire.
stands tall, adorned with pink leaves,
Dense Connector
Describe the video in detail.
The vidco depicts a group of young men gathered around a dining table, engaged in friendly arm wrestling competition. There least five people in the scene, with two of them actively participating in the arm wrestling match. The other individuals are watching the competition, likely cheering their friends waiting for their turn.
The table is cluttered with various items, including a bottle and a placed near the center of the table. The atmosphere appears to be lively and competitive, the young men enjoy their time together.
Figure 10: Qualitative results of the video understanding.
24

---

Dense Connector
Describe the video in detail.
The video features young child sitting on a bed, wearing glasses and holding book. The child appears to be reading or looking at the book, possibly enjoying story or learning something new. The covered with blanket, providing comfortable and cozy environment for the child to engage in their reading activity.
Why this video funny?
The video is funny because it features baby wearing glasses and holding book, pretending to read. The baby's actions and the fact that it is wearing glasses, which are typically associated with adults, create a humorous and adorable scene. The baby's innocent and curious expression book adds to the humor and charm of the moment. The combination of the baby's actions and the visual young child wearing glasses creates a lighthearted and entertaining scene that bring smile the viewer's face.
looks the
Dense Connector
What is strange about this video?
The strange aspect of this video is that a man is sitting in the middle of group of lions, which an unusual and potentially dangerous situation. Lions are wild animals and known for their strength and predatory instincts. It is highly unusual for person to be in such close proximity to group of lions without any protective measures or barriers. presence among the lions raises concerns about his safety and the potential risks involved in such an encounter.
man's
Figure 11: Qualitative results the video understanding.
25

---

Dense Connector
NET NOTEEEEEEETT
NOY INEREDIENTS
What is the video about?
The image features woman sitting table, holding tube of sunscreen in her hand. She is applying the sunscreen to her face, ensuring that she is protected from the sun's harmful rays. The sunscreen is a product from Skin Authority, and the woman is using it to moisturize her skin and maintain its health. The scene emphasizes the importance of using sunscreen as a daily skincare routine to prevent sun damage and maintain healthy complexion.
Could you recommend products similar to the one featured in the video?
Based on the image, the product featured is tube of sunsereen moisturizer from Skin Authority. Some similar products that you might consider are other sunsereen moisturizers from reputable brands, such as Neutrogena, Aveeno, CeraVe. These brands offer range of sunscreen moisturizers with varying SPF levels and formulations to cater to different skin types and preferences. is essential choose product that suits your skin type and provides adequate protection against harmful UV rays while moisturizing your skin.
Dense Connector
Can you identify any safety hazards in this video?
In the video, there is a baby playing with a toy truck near fireplace. This situation presents potential safety hazard, as the baby might accidentally fall into the fireplace or get too close the fire, which could lead burns or other injuries. It is important for parents or caregivers supervise the child closely and ensure that the baby is playing in a safe distance from the fireplace prevent any accidents.
Figure 12: Qualitative results of the video understanding.
26

---

Connector
Describe the video.
The image features a person standing in kitchen, preparing meal by cutting up a variety of bell peppers. The person is holding knife and is in the process of slicing the peppers into smaller pieces. The sliced peppers are placed tray, with some of them already cut and others still needing to be prepared. The scene captures the person's attention to detail and focus on preparing the meal.
How many slices are on the baking sheet.
There are four slices of red peppers on the baking sheet.
Dense Connector
Describe the video.
The image captures a snowy day at track and field event, where man is preparing to throw javelin. He is standing on a red track, wearing a blue shirt and black pants, and focused on his performance. There are several other people in the scene, some of whom are standing near the track, possibly watching the event or waiting for their turn.
In addition to the main javelin thrower, there two other people visible in the image, on the left side and another on the right side of the scene. The presence of multiple people suggests that this a competitive event or practice session with multiple participants.
Figure 13: Qualitative results of the video understanding.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide our contributions both in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Yes, we provide all necessary information to reproduce the experimental
results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
We will provide the anonymous code, and our code and data will be publicly available.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //nipscc//pie wwcppbblc/guuiicuccccccc
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are described in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
30

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these information the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research follows the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper for assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will provide anonymous URL.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "2XHzr0mY0K", "reviewer": "Reviewer_sj7M", "summary": "This paper introduces the Dense Connector, a simple idea that aligns visual and language modalities by utilizing multi-layer visual features. The authors explore three instantiation methods and demonstrates the effectiveness of the Dense Connector across various settings, including different backbones, modalities, datasets and architectures.", "questions": "Beyond the section on weaknesses, I have the following additional questions:\n1.\tThe decision to fine-tune ViT during MLLM training is still debated. Research findings are mixed; some [1] indicate that fine-tuning ViT may impair performance, while others [2] show it can lead to performance improvements. I wonder whether the Dense Connector, which utilizes multi-layer visual features, could benefit from fine-tuning ViT.\n2.\tThe authors could specify the additional training and testing time required by the Dense Connector.\n[1] Karamcheti, Siddharth, et al. \"Prismatic vlms: Investigating the design space of visually-conditioned language models.\" arXiv preprint arXiv:2402.07865 (2024).\n[2] Chen, Lin, et al. \"Sharegpt4v: Improving large multi-modal models with better captions.\" arXiv preprint arXiv:2311.12793 (2023).", "limitations": "The authors have addressed the limitations.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThis paper focuses on further exploring the utilization of visual signals in MLLMs. Compared to the attention given to language models, this perspective indeed lacks sufficient attention. It provides insights into utilizing MLPs for integrating multi-layer visual features at the channel level in MLLMs, reducing visual tokens and enhancing computational efficiency.\n2.\tDense Connector demonstrates effectiveness across a range of model architectures, including LLaVA and Mini-Gemini, as well as LLMs scaling from 2.7B to 70B parameters. \n3.\tThis paper is well-written and includes thorough experimental validation of the Dense Connector.\n4.\tI also tried running the code provided by the author, and it is indeed simple and effective. Personally, I feel dense connector has the potential to become a fundamental module in future MLLMs.", "weaknesses": "1.\tHigh-resolution approaches are widely utilized in vision-language models. This paper explores the application of the Dense Connector with dual visual encoders, specifically Mini-Gemini, in high-resolution settings. However, it lacks an exploration of dynamic high-resolution scenarios, such as those implemented by LLaVA-NEXT. I am curious about whether the Dense Connector could further enhance MLLM performance in this scenario.\n2.\tMy another concern is that although the Dense Connector performs well on the Vicuna 7B and 13B models, its effectiveness seems to diminish when scaling to larger sizes. I suspect this might be because training LoRA is not as effective as updating the LLM. If possible, the authors should provide results for updating the LLM to give readers a better reference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Combining Dense Connector with LLaVA-NEXT's AnyRes technology**\n\nThanks! This is a very worthwhile question to explore. We extended the Dense Connector (DC) to dynamic resolution scenarios. Using dynamic resolution, DC remains effective. Compared to the baseline, DC improved performance on TextVQA and MMMU by **1.1\\%** and **2.2%**, respectively. Additionally, **using only LLaVA-1.5 data, DC outperformed LLaVA-NeXT on several benchmarks**.\n\n| Model | Data | Vision Encoder | LLM | TextVQA | SQA | GQA | LLaVA-Bench | MM-Bench | MM-Vet | MMMU |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| LLaVA-NeXT (Baseline) | LLaVA-1.5 | CLIP-L/336px | Vicuna-7B | 64.5 | 69.5 | 64.0 | 68.2 | 66.5 | 33.1 | 35.4 |\n| DC  | LLaVA-1.5 | CLIP-L/336px | Vicuna-7B | 65.6 | 70.5 | 64.6 | 69.0 | 67.4 | 33.7 | 37.6 |\n| DC  | MGM | SigLIP-SO | Vicuna-7B | 70.0 | 72.0 | 63.9 | 88.8 | 69.2 | 44.4 | 35.8 |\n| DC  | MGM | SigLIP-SO | Vicuna-13B | 70.9 | 75.2 | 64.3 | 93.2 | 72.3 | 47.0 | 35.8 |\n\n**Q2. Explanation regarding the performance of larger language models**\n\nThank you for raising this concern. We will address it from two aspects and hope our response alleviates the reviewers' worries.\n\n- In previous experiments, constrained by limited computational resources, we **used LoRA to reduce memory consumption** for fine-tuning the 34B model. Here, we present a comparison of the 34B model w/ and w/o the DC under LoRA fine-tuning. The results indicate that the inclusion of DC consistently enhances the model's performance in this context.\n  \n\n|     | TextVQA | MM-Bench | MM-Vet |\n| --- | --- | --- | --- |\n| LLaVA-1.5-34B-LoRA w/o DC | 63.4 | 76.0 | 38.9 |\n| LLaVA-1.5-34B-LoRA w/ DC | 66.7 | 77.7 | 41.0 |\n\n- Now, with more computational resources, we **fully fine-tuned the 34B model**, addressing concerns about DC's performance on larger language models. The results are shown in the table below.\n  \n\n|     | Data | Vision Encoder | Res. | LLM | TextVQA | SQA | GQA | LLaVA-Bench | MM-Bench | MM-Vet | MMMU |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| DC  | MGM | SigLIP-SO | AnyRes | Yi-34B | 75.2 | 78.0 | 66.6 | 97.7 | 81.2 | 59.2 | 47.2 |\n\n**Q3.Will fine-tuning ViT improve performance?**\n\nThanks! That's a great question. We fine-tuned ViT using the settings from LLaVA-NeXT, specifically a learning rate of 2e-6 in the second stage. The results indicate that **fine-tuning ViT can further improve the performance of DC**.\n\n| Model | Data | Vision Encoder | Fine-tune ViT | Res. | TextVQA | VQAv2 | SQA | GQA | LLaVA-Bench | MM-Bench | MM-Vet |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| DC-7B | LLaVA-1.5 | CLIP-L | No  | 336 | 59.2 | 79.5 | 69.5 | 63.8 | 66.1 | 66.8 | 32.7 |\n| DC-7B | LLaVA-1.5 | CLIP-L | Yes | 336 | 60.2 | 80.5 | 68.4 | 63.7 | 66.2 | 68.6 | 34.4 |\n\n| Model | Data | Vision Encoder | Fine-tune ViT | Res. | TextVQA | VQAv2 | SQA | GQA | LLaVA-Bench | MM-Bench | MM-Vet |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| DC-7B | LLaVA-1.5 | SigLIP-SO | No  | AnyRes | 66.5 | 81.4 | 69.3 | 64.8 | 70.7 | 67.2 | 34.8 |\n| DC-7B | LLaVA-1.5 | SigLIP-SO | Yes | AnyRes | 67.6 | 82.5 | 69.2 | 64.8 | 71.1 | 68.9 | 35.0 |\n\n**Q4. About training time and inference time**\n\nFor the Sparse Token Integration method, the addition of visual tokens results in a 20% increase in training and inference time. However, for Sparse Channel Integration and Dense Channel Integration, the training and inference times **remain unchanged**. Specifically, using 8 A100 GPUs (40G) to train the Vicuna-7B model, DC takes approximately **4 hours** in the first stage and **12 hours** in the second stage, **similar to LLaVA-1.5**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xeq7hbdvhn", "reviewer": "Reviewer_2pJ5", "summary": "This paper endeavors to delve into the visual representations in MLLMs, introducing a module, plug-and-play component named as Dense Connector (DC). This  DC is designed to enhance visual representation. To this end, three  instantiations are presented, including Sparse Token Integration(STI), Sparse Channel Integration(SCI), and Dense Channel Integration(DCI).  Thorough experimental evaluations are meticulously carried out to validate the compatibility and scalability, thereby highlighting their potential in enhancing MLLM performance.", "questions": "It would be beneficial to include a line chart that incorporates a broader range of combinations, facilitating a clearer visual comparison and enabling the selection of groups more evidently.", "limitations": "Suggestions: The presented DC can be furthre to combine multiple vision encoder, like DINOv2.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. A plug-and-play module can be applyed into various MLLMs.\n\n2. The experiments, including different  training datasets, diverse MLLMs, are conduted to demonstrate the performance of DC. The video benchmarks are also adopted as an extension.", "weaknesses": "1. The insight of enhancing visual representations from multi-level features of vision encoder have been explored in previous works, like [1]. The work [1]  has provided the simliar evidences from multi-level features, and more comprehensive analysis based on a amout of experimetal results.\nWould authors provides any possible fair comparsions with  the work[1] (Only CLIP vision encoder)?\n\n   [1] Jiang et al.,  From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models, Arxiv 2023.\n\n2. While this work outlines three architectural configurations for the Dense Connector (DC),  it primarily employs a single variant – the Dense Channel Integration (DCI). It is important to note that the DC module serves more as an empirical trick rather than a theoretical methodology.\n\n3.  I think the performance gain is limited, especially on high-resolution MGM baseline in Table2. The  scalability of DC may be limited.\n To comprehensively assess its effectiveness, I encourage the authors to extend their comparisons utilizing a Vicuna-13B model based on the MGM framework, incorporating additional benchmark datasets such as VQAv2, MME, and POPE. It is speculated that the incremental performance gain might become even less pronounced when employing larger language models (LLMs),", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. More comparisons with COMM [1]**\n\n[1] combines visual features from all layers by simply adding them together, which can lead to **information loss**. Additionally, ViT's low, middle, and high layers contain different information. Therefore, COMM's approach of adding them all together **lacks prior knowledge**.\n\nIn contrast, DC groups multi-layer visual features, **providing prior knowledge during integration.** Additionally, DC concatenates multi-level features along the channel dimension, effectively utilizing the dimensional transformation characteristics of MLPs connectors. **It achieves feature fusion and transformation without additional modules.** The tables below show the performance comparison between [1] and DC.\n\n| | POPE | VQAv2 | MME |\n|-|-|-|-|\n|COMM|83.6|70.1|1235|\n|DC|86.6|79.5|1523|\n\n> Would authors provides any possible fair comparisons with the work [1] (Only CLIP vision encoder)?\n\nTo ensure a fair comparison, we fine-tuned the model using **LLaVA-1.5 data and a single ViT CLIP-L/336px**. For COMM, we used LLN-Layerscale(all) [1] for integrating visual features.\n\n||TextVQA|MME|SQA|POPE|LLaVA-Bench|AVG|\n|-|-|-|-|-|-|-|\n|LLaVA (baseline)|58.2|1511|66.8|85.9|65.4|70.4|\n|LLaVA+COMM|58.0|1493|67.4|86.0|60.5|69.3(**-1.1%**)|\n|LLaVA+DC |59.2|1511|69.5|86.6|66.1|71.4(**+1\\%**)|\n\nThe results above indicate that DC is a **more effective** method, achieving an average performance improvement of **2.1%** compared with [1].\n\n[1] Jiang et al., From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models, Arxiv 2023.\n\n**Q2.Statement about DC**\n\n> While this work outlines three architectural configurations for the DC, it primarily employs a single variant – the Dense Channel Integration (DCI).\n\nCurrent MLLMs research primarily focuses on high resolution and data, with less attention to multi-layer features. Previous work [1] has simply added visual features together, lacking further discussion. Our paper fills this gap by being the first to extensively discuss the fusion of multi-layer visual signals along the token (STI) and channel (SCI, DCI) dimensions.\n\nWe explored three instantiation methods, all enhancing MLLM performance across various settings. Among them, DCI achieved the best performance, so we primarily adopted it for scaling-up experiments.\n\n> It is important to note that the DC module serves more as an empirical trick rather than a theoretical methodology.\n\nDC was designed to enhance MLLM performance from a visual perspective. Inspired by DenseNet and FPN, we explored the use of multi-layer visual features in MLLM and sought effective model designs. Thus, we developed DC, which utilizes multi-layer features in a way **distinct from previous works**. This module is tailored to MLLM, utilizing the dimension transformation of connectors to fuse multi-layer visual features.\n\nImportantly, we believe this work **benefits the community**. First, DC outperforms previous connectors, demonstrating **applicability across architectures** (LLaVA-1.5, MGM, LLaVA-NeXT and Qformer). Second, we address the gap in integrating multi-layer visual features from the token and channel dimensions. Finally, we hope this paper will attract more attention to the visual modality in MLLM.\n\n**Q3.Performance of DC on MGM with Vicuna-13B**\n\n> I think the performance gain is limited, especially on high-resolution MGM.\n\nWe believe the improvement brought by DC is significant. DC improved performance on GQA, SQA and MMB by **1.8, 2.7 and 2.5**, respectively. In comparison, a more powerful ViT (CLIP-L->SigLIP) only improved GQA, SQA, MMB by **0.4, 1.0 and 1.6**, and a stronger LLM (7B->13B) only improved by **1.2, 2.5 and 3.0**.\n\nBased on reviewer sj7M's suggestion, we **extended DC to other high-resolution methods LLaVA-NeXT**, achieving performance improvements. Please refer to reviewer sj7M's Q1.\n\n> I encourage the authors to extend their comparisons utilizing a Vicuna-13B model based on the MGM framework.\n\nWe extended the MGM experiments to Vicuna-13B with additional benchmark results. The table shows that **DC can enhance MGM-13B's performance across various benchmarks.**\n\n||VQAv2|MMBench|POPE|MME|LLaVA-Bench|MMVet|\n|-|-|-|-|-|-|-|\n|MGM|80.8|68.5|85.1|1565/322|87.5|46.0|\n|MGM w/ DC|81.9|70.7|85.6|1573/355|92.0|49.8|\n\n**Q4. DC's performance on larger models**\n\n> It is speculated that the incremental performance gain might become even less pronounced when employing larger language models (LLMs)\n\nThank you for highlighting this concern. We will address it from two aspects and hope our response alleviates the reviewers' concerns.\n\n- Due to limited resources, we previously used LoRA for fine-tuning, which affected the performance. Here, we provide a comparison of **the 34B model w/ and w/o the DC under the LoRA fine-tuning.** The results show that DC consistently enhances the model's performance in this scenario.\n\n||TextVQA|MM-Bench|MMVet|\n|-|-|-|-|\n|LLaVA-1.5-34B-LoRA w/o DC|63.4|76.0|38.9|\n|LLaVA-1.5-34B-LoRA w/ DC|66.7|77.7|41.0|\n\n- We utilized more resources (32 A100 GPUs) to fully fine-tune 34B model, demonstrating that DC can achieve excellent results with larger models.\n\n||TextVQA|MM-Bench|GQA|MMVet|LLaVA-Bench|\n|-|-|-|-|-|-|\n|DC-34B-AnyRes|75.2|81.2|66.6|59.2|97.7|\n\n**Q5.Bar charts for clearer comparisons.**\n\nThanks. We provided bar charts in the PDF for clearer comparisons.\n\n**Q6. Combine multiple vision encoder**\n\n> DC can be further to combine multiple vision encoder, like DINOv2.\n\nWe extended DC to multiple vision encoder, structures. Based on the results in [2] and our findings, we discovered that DINOv2 may not perform well. Therefore, in addition to the combination of CLIP and DINOv2, we also combined CLIP with ConvNeXT. The results show that the CLIP and ConvNeXT performs better.\n\n||GQA|TextVQA|MMBench|\n|-|-|-|-|\n|CLIP+DINOv2|63.1|58.9|65.5|\n|CLIP+ConvNeXT|63.9|60.9|67.0| \n\n[2] Tong et al., Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs, CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k2U0QNz7DG", "reviewer": "Reviewer_qnn9", "summary": "The document presents a novel approach called the Dense Connector (DC), which is a simple and effective plug-and-play vision-language connector that enhances existing Multimodal Large Language Models (MLLMs) by leveraging multi-layer visual features from the frozen visual encoder.The authors propose three intuitive instantiations of the Dense Connector.The authors demonstrate the versatility and scalability of their approach across various visual encoders, image resolutions, training dataset scales, LLM sizes, and MLLM architectures, achieving state-of-the-art performance on 19 image and video benchmarks.", "questions": "I am curious whether the DC is compatible with BLIP2-like architectures which have cross-attention visual resamplers?\nCould authors give generalization of the compatible types of MLLM architecture?", "limitations": "authors adequately addressed the limitations", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the dense connector is easily plugged into mainstream MLLM architecture, and could enhance visual representation of existing MLLM with little additional computation.\n2. authors demonstrate the compatibility and scalability of proposed approach across visual encoders, image resolutions, scales of training datasets, sizes of LLMs and even for video understanding tasks.", "weaknesses": "1. authors do experiments on several MLLM models, however don't summarize the compatible scope of models architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.Can DC be compatible with the visual resampler architecture?**\n\nThis is a very worthwhile research question. Dense Connector (DC) is compatible with models similar to BLIP-2, which have visual resampler or Qformer structures.\n\nNotably, models like BLIP-2 use visual resampler to obtain learnable queries as final visual representations, **which still need MLPs for converting visual queries into the input space of the language model** in structures like BLIP-2. We can replace these MLPs with DC to enhance visual perception. Thus, the DC can be used wherever a vision encoder and a connector are present.\n\nSpecifically, the visual resampler compresses high-level features into learnable queries. We then use an interpolation function to downsample tokens of different layer features to align the number of queries. DC is then used to transform these features from different layers into the input space of the language model. The table below shows our experimental results: using the visual resampler architecture, DC improved the average performance by **2\\%**.\n\n| Model | GQA | VQAv2 | TextVQA | SQA | MMbench | MM-Vet | AVG |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| visual resampler w/o DC | 56.2 | 72.7 | 52.0 | 67.3 | 60.5 | 26.0 | 55.8 |\n| visual resampler w/ DC | 58.2 | 73.9 | 53.9 | 68.8 | 62.8 | 29.3 | 57.8 (**+2\\%**) |\n\nIt is important to note, as referenced in [1], that the larger number of parameters in the visual resampler leads to poorer convergence. As a result, when trained on the same data, the performance of the visual resampler is suboptimal. Nonetheless, the table above demonstrates that **DC is compatible with the visual resampler** and enhance its performance.\n\n[1] Liu, Haotian, et al. \"Improved baselines with visual instruction tuning.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n\n**Q2.Summarize the model architectures compatible with Dense Connector**\n\nThank you for pointing this out! We agree that summarizing the compatible scope of model architectures for DC would enhance the clarity and comprehensiveness of this paper.\n\nDC can be widely applied to current MLLM architectures. In current MLLMs, **any model utilizing a vision encoder** can integrate the DC. The reason is that, regardless of whether the models are based on MLPs (LLaVA) or Qformers (BLIP-2), they require linear layers to convert visual features into the language space. Specifically, LLaVA employs two linear layers for high-level feature transformation, while BLIP-2 initially uses Qformer to obtain learnable queries, followed by a single linear layer for feature conversion. **This linear layer is where the DC comes into play.** Therefore, any model using a vision encoder can be compatible with the DC.\n\nIn summary, in this paper, we applied DC to a wide range of different architectures, including LLaVA-1.5, Mini-Gemini, **LLaVA-NeXT (please refer to reviewer sj7M's Q1)**, and the Visual Resampler architecture. We will include these in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their time and insightful comments. We are glad the reviewers find that Dense Connector (DC) is a novel plug-and-play module, which is the first in the MLLM field to explore the fusion of multi-layer visual features from both token and channel dimensions. We introduced three fusion methods: STI, SCI, and DCI, all of which achieved significant performance improvements. We are encouraged that the reviewers acknowledge our extensive experiments demonstrating its effectiveness across different encoders, resolutions, model frameworks, LLMs, training data, and modalities (sj7M, 2pJ5, qnn9). We are also delighted that the reviewers tried using our module and found it effective (sj7M).

**[Paper improvements made in response to feedback]**

In response to the reviewers' valuable suggestions, we have made the following improvements to the paper:
- We have extended DC to more architectures, including dynamic resolution methods **LLaVA-NeXT** and cross-attention **visual resamplers**, demonstrating that DC can be widely applied in MLLMs (sj7M, qnn9).
- Our experiments show that **fine-tuning ViT can benefit DC**, which utilizes multi-layer features (sj7M).
- We have validated that our method is **superior to previous methods** that utilize multi-layer features (2pJ5).
- We extended DC to the high-resolution method MGM-13B, where **DC significantly improved MGM-13B's performance** (2pJ5).
- By **fully fine-tuning the 34B model**, we hope this result alleviates reviewers' concerns about DC's performance on larger models (2pJ5, sj7M).
- We have also validated that DC can be applied to **multiple vision encoder architectures** (2pJ5).
- We have included **charts** in the PDF, which provides a clearer visual comparison (2pJ5).

We did our best to address the questions within the given time. We appreciate the reviewers' comments and believe the revisions have strengthened the paper. We thank all the reviewers for their help. Please find individual responses to your questions below.

Meta-review: This work intoduced dense connector for multimodal LLMs to enhance to the visual percetion capability, akin to densenet or FPN in vision models. The method is simple yet effective across the board, spanning from different benchmarks with different resolutions and variou LLM sizes. The paper is also well-written and method is clearly motivated. During the discussion, the authors and reviewers were engaged in intensive discussions. The authors provided a detailed and constructive rebuttal, addressing all reviewer concerns effectively. As a result, it recevied all positive final ratings (5, 6, 7). 

Overall, this work presents a technically sound contribution to the field of MLLMs. According to the submission and discussions, the proposed Dense Connector is a versatile module that offers consistent performance improvements with minimal additional computation. While some concerns were raised about the incremental nature of the gains and the generalizability of the approach, the authors’ thorough rebuttal addressed these issues convincingly. As a result, the ACs recommend an acceptance, and suggest the authors integrate the additional experimental results and analysis during rebuttal to the final version, and release the implementation code to public for benefit to the community.
Decision: Accept (poster)