Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
Authors: Jiachen Jiang
Jinxin Zhou
Bo Peng
Xia Ning
Zhihui Zhu
Venue: NeurIPS 2025 poster ()
Keywords: Multimodality Alignment; Token-Level Alignment; Multimodal Large Language Models; Representation Similarity
Primary area: deep_learning
TL;DR: Enhancing the patch level alignment of Multimodal Large Language Models

Abstract: Achieving better alignment between vision embeddings and Large Language Models (LLMs) is crucial for enhancing the abilities of Multimodal LLMs (MLLMs), particularly for recent models that rely on powerful pretrained vision encoders and LLMs. A common approach to connect the pretrained vision encoder and LLM is through a projector applied after the vision encoder. However, the projector is often trained to enable the LLM to generate captions, and hence the mechanism by which LLMs understand each vision token remains unclear. In this work, we first investigate the role of the projector in compressing vision embeddings and aligning them with word embeddings. We show that the projector significantly compresses visual information, removing redundant details while preserving essential elements necessary for the LLM to understand visual content. We then examine patch-level alignment---the alignment between each vision patch and its corresponding semantic words---and propose a $\textit{multi-semantic alignment hypothesis}$. Our analysis indicates that the projector trained by caption loss improves patch-level alignment but only to a limited extent, resulting in weak and coarse alignment. To address this issue, we propose $\textit{patch-aligned training}$ to efficiently enhance patch-level alignment. Our experiments show that patch-aligned training (1) achieves stronger compression capability and improved patch-level alignment, enabling the MLLM to generate higher-quality captions, (2) improves the MLLM's performance by 16% on referring expression grounding tasks, 4% on question-answering tasks, and 3% on modern instruction-following benchmarks when using the same supervised fine-tuning (SFT) setting. The proposed method can be easily extended to other multimodal models.

Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
Jiachen Jiang", Jinxin Zhou', Bo Peng', Xia Ning Zhihui Zhu
Department of Computer Science and Engineering. The Ohio State University Trranslational Data Analytics Institute, The Ohio State University Department of Biomedical Informatics, The Ohio State University [jiang 2880, zhou. 3820, peng. 707, ning. 104, 2hu.34400 Cosu. edu
Abstract
Achieving better alignment between vision embeddings and Large Language Mod- els (LLMs) is crucial for enhancing the abilities of Multimodal LLMs (MLLMs), particularly for recent models that rely on powerful pretrained vision encoders and LLMs. A common approach to connect the pretrained vision encoder and LLM is through a projector applied after the vision encoder. However, the projector is often trained to enable the LLM to generate captions, and hence the mechanism by which LLMs understand each vision token remains unclear. In this work, we first investigate the role of the projector in compressing vision embeddings and aligning them with word embeddings. We show that the projector significantly compresses visual information, removing redundant details while preserving essential elements necessary for the L.M to understand visual content. We then examine patch-level alignment-thh alignment between each vision patch and its corresponding mantic words-and propose a multi-semantii alignment hypothesis. Our analysis indicates that the projector trained by caption loss improves patch-level alignment but only to a limited extent, resulting in weak and coarse alignment. To address this issue, we propose patch-aligned training to efficiently enhance patch-level alignment. Our experiments show that patch-aligned training (1) achieves stronger compression capability and improved patch-level alignment, enabling the MILM to generate higher-que quality captions, (2) improves the MLLM's performance by 16% on referring expression grounding tasks, 4% on question-anas -answerng tasks, and 3% modern instruction-following benchmarks when using the same supervised fine-tuning (SFT) setting. The proposed method can be easily extended to other multimodal models.
higher-quality
Introduction
Multimodal Large Language Models (MLLMs) 21 4, 4, 6, 7] have recently gained significant attention and made notable progress. These models possess the ability to process and understand both visual and textual information, enabling them to perform complex reasoning [8, 9], generate textual deseriptions from images [10], and answer image-related questions [11].
Consider an MLLM M (E,C,)) where E is the vision encoder, C is the LLM, and P is the lightweight projector that connects the two parts. The standardized training paradigm follows two key phases: pretraining and instruction-tuning [1, 2]. During the pretraining phase, only the lightweight projector P is trained leaving the vision encoder E and the LLM L frozen. In inorucctio-tuuinng stage, both projector P and the LLM L are trainable. Despite the remarkable progress achieved through
'The corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Patch-Level Localization
Multi-Semantic Alignment
1.00
McT
Cos
Lor
'Hand"
"Phone'
Matching Pursuit Iterations
Mpred
Randem Projector
RRandom
Mpred
LLLVA 1.5 Projector
PLLAVA
M pred
PutchAlighed
(Ours)
Patch Allgned
Figure The patch-level alignment is measured in two aspects: Left) patch-level localization. Using LLM word embeddings of labels LGI, predict the most relevant image regions by cal- culating cosine similarity with the vision embedding. Right) multi-semantic alignment. Using matching pursuit, decompose each vision embedding into several discrete words by treating LLM word embeddings as a basis. Since the vision embedding are obtained after the MILM projector, compare across three projectors: random projector PRamiom (top), LLaVA1.5 projector PLlaVA (middle), and our patch-aligned projector Pranh Aligned (bottom). Results show that PlLAVA exhibits weak patch-level alignment abilities, while our Paach Aligned significantly enhances these two aspects.
this training paradigm, recent works [12, 13, 14, 15, 16] reveal that these MILMS still struggle with region-specific understanding and tend to hallucinate with irrelevant incorrect information. The reason of these issues remains an active area. In addition to limitations in the vision encoder or the capabilities of the language model, a significant contributing factor lies in the projector [4, 17].
The platonic representation hypothesis [18] suggests that representations in deep networks are converging across data modalities?, implying a shared structural alignment. However, for vision encoder and an LLM L that are pretrained separately, there is no guarantee that the embedding space induced by E will share the same basis the embedding matrix W in L, even if they have the same dimensionality. Thus, as the only connection between the two modalities, the projector P plays crucial role. However, current work remains at superficial understanding that the projector performs alignment, lacking a thorough and systematic analysis of its function. Thus, in this paper, are motivated by the following questions:
How does the projector align multimodal information in MLLMs? How to quantify and improve alignment in current models?
Contributions. In this work, we provide detailed analysis of the alignment between each vision patch and its corresponding semantic words and develop methods to improve patch-level alignment, enabling the LLM to better understand visual content in the input space. Our contributions as follows,
Projector compresses visual information. Vision embeddings are naturally continuous and contain redundant information, whereas LLM input word embeddings are discrete. Thus, natural question arises: Is the information contained in the vision embeddings compressed through the projector? To address this question, we propose quantifying the amount of information contained the embeddings
In the sense that vision and language models measure the distance between data points in similar way.

---

using Von Neumann entropy [19]. Our experiments show that information is significantly compressed after projection, indicating that the projector plays a crucial role in eliminating redundant information while preserving the essential elements needed for the L.M to understand the visual content.
Analyzing patch-level alignment. We then examine each image patch in detail and study how its embedding aligns with the corresponding text embedding. However, challenges arise due to (I) a lack of text labels for each image patch and (2) the possibility that each image patch contains multiple semantic meanings. To address these challenges, we propose two complementary approaches to quantitatively and qualitatively study patch-level alignment.
We first focus on alignment with respect to the objects in the image. Specifically, for an input image X with embedding V = P {(X), which serves input to an LLM L, we propose patch-level alignment measure Align(V (V W) to quantify the alignment between V and the word embedding W. A higher Align(V (V, W) indicates a greater ability of the L.MM to identify objects, even in the word embedding space. Inspired by previous work on decomposing word embedding vectors [20, 211, we then propose multi-semantic alignment hypothesis. which states that the embedding for each vision patch can be decomposed as a linear combination of word embeddings corresponding to all semantic meanings within the patch. To verify this hypothesis, we apply the matching pursuit algorithm [22] to identify the most relevant tokens from the LLM dictionary for each vision patch.
As shown in Figure 1, our analysis indicates that the LLaVA projector improves patch-level alignment but only to a limited extent, resulting in weak and coarse alignment. This is due to (1) the caption loss only implicitly enforces token-level alignment, (2) captions tend be short and primarily focus on a few prominent regions of interest, often neglecting many other regions. Consequently, numerous visual tokens (e.g., floor and TV cabinet) are often aligned with meaningless or garbled words.
Patch-Aligned Training for Improving Patch-Level Alignment. To address this issue, we propose simple yet effective method called Putch-Aligned Training to enhance fine-grained alignment. In addition to the standard image caption loss in the pretraining state, we introduce additional patch loss, similar to Align( V, W), to capture the alignment between V and the word embedding W. Notably, the patch loss relics only on the LLM embedding matrix W and is therefore computationally negligible compared to the caption loss, which requires inference and backpropagation through the L.M. As demonstrated in Figure 1, experiments show that Patch-Aligned training achieves stronger compression capability and improved patch-level alignment, enabling the LLM generate higher-quality captions. Moreover, under the same SFT setting, the enhanced projector improves the MLLM's performance by 16% referring expression grounding tasks, 4% on question-answeriig benchmarks.
tasks, and 3% on modern
Patch-Alignel Dataset with Detailed Patch-Level Semantic Labels. To enable patch-aligned training. we address the lack of patch- patch-level annotated data by introducing an automated data annotation pipeline that sequentially leverages RAM [23], Grounding DINO [24], and SAM [25]. Applying this pipeline to the 558K LLaVA pretraining dataset, we construct the Patch-Alige Aligned Dataset (PAD), which provides extensive and diverse patch-level annotations. To support future research, we publicly release both the annotation pipeline and the resulting dataset.
2 Related Works
Multimodality Large Language Models. Many MLLMs, such as LLaVA-I.5/1.66 [2, 26). BLIP-2
[4], InstructBIIIP [27], MiniGPT-4 [5], Otter [28], and mPLUG-Owl [29], can be viewed as stitched models, formed by connecting a pretrained (and often frozen) vision encoder (such as ViT) to a pretrained LLM through projector or connector. The projector can be trained using either (0) I-stage approach, where is directly trained alongside the fine-tuning LLM during instruction training [30], or (ii) a 2-stage approach, where the projector is first pretrained on adapter data before unfreezing the LLM and connector during instruction tuning [2]. The 2-stage approach has been widely adopted since LLaVA and has been shown to be beneficial [31]. However, during the pretraining stage, these models primarily rely on caption loss to achieve coarse alignment between modalities, which tends to lack region- understanding abilities. Recent efforts, such as GPT4Rol [13], Kosmos-2 [14], and GLaMM [15], have attempted to improve region-specific, fine-grained understanding. However, these approaches often rely on grounding techniques or introduce additional tokens to enhance inference-ting time capabilities. They focus improving inference rather than representation analysis of
3

---

fine-grained alignment. In contrast, our approach seeks to enhance fine- tne-grained understanding by improving patch-level alignment without requiring additional training or tokens.
MultiModal Alignment Analysis. Existing works analyze cross-modal alignment from two perspec- tives: coarse alignment and fine-grained alignment. Coarse alignment is evaluated using metrics such as AC Score [32], which heavily depends on the CLIP [33] model, and Modality Integration Rate (MIR) [34], a statistic-basee measure akin FID. While these metrics provide insights into pretrain- ing performance, they fail to address fine-grained token-level alignment. For fine-grained alignment, methods like Logit Lens Analysis [35] show that
information is localized to token positions corresponding to image regions but lack proposals for improvement. Other works align coordinate, text, and image modalities through question-anse ansseer formats but overlook feature-level understanding [36]. Concurrently, SEA [37] enhances token-level alignment using predefined word lists and contrastive loss, but its reliance on the CLIP model and fixed vocabularies limits accuracy and flexibility. In contrast,
fine-grained alignment model, our approach employs annotations from the RAM [23] model, which avoids predefined word lists for more accurate tagging, and introduces a cosine similarity loss, offering simpler and more efficient alternative to contrastive loss.
3 Understanding Multimodal Projector by Compression and Alignment
In this section, we study the projector from both macro and micro perspectives: 1) information compression in Section 3.1 and 2) patch-level alignment in Section 3.2.
3.1 Macro-scale Analysis: Information Compression
Consider a MILMM M = (E, P). For each input images X, the vision embedding of the n-th image before and after the projector is sequence of embeddings
V before {X) Rdxs
V after
P PoE(X) Rd
where 5' is the number of vision tokens and d, d' the embedding dimensions of the vision encoder and LLM L. For N images, we compute the embeddings for each image and stack them together. We denote the resulting embeddings as before E R and Vater E R xNS
We hypothesize that the projector plays a crucial role in eliminating redundant information while preserving essential elements for the LLM to understand vision content. To quantify this, we measure the information using the basis- vasis-independent,
cunsformation-invariin Von Neumann entropy [19].
Definition 3.1 (Von Neumann Entropy of Feature Embeddings). Let V E Rdx= be set of n feature vectors, each of dimension d, and let v; E Rdd denote the i-th column of V. Define the normalized empirical covariance matrix by,
Ev Pv Tv
Σv
U U
pddd
where Ey is normalized by its trace to obtain the density matrix of trace 1. Then, compute the information contained in the embeddings V through the Von Neumann entropy [19] as follows,
H(V) - TTr(pv log Pv) =- -E log(A,).
where { are the eigenvalues of Pv-
The Von Neumann Entropy measures how evenly information spreads across the feature space of learned embeddings, indicating their effective dimensionality. Higher values show well-distributed, high-rank representation with diverse features, while lower values indicate compression cunpression-resulting in information loss and reduced effective rank of the covariance matrix.
To measure the compression abilities of different projectors, we compare pretrained (stage 1) and randomly initialized variants. We evaluate several projector types commonly used the MLLM field, including Linear, 2-layer MIP, and C-Abstractor [38]. We evaluated these across 100 selected images from the COCO2017 dataset [39]. The Von Neumann Entropy of vision embeddings before and after projection are shown in Table 1.

---

Table I: Comparison of Von Neumann Entropy of vision embedding before and after projection.
Projector
PLLAV Linear
Random Linear
PLLAVA MLP
PRandom MLF
PLLAVA C-Abs
Raanom C-Abs
H betors
4.8353
4.8353
4.8353
4.8353
4.8353
4.8353
H V after)
2,4829
4.8197
2.0362
4.8245
3.5850
7.4913
Based on Table 1, we make several key observations:
Pretrained v.s. Random. The vision feature after the pretrained projectors (PILAVA Linears PILAVA MLP and PILaVA C-Absractoo)) exhibit lower entropy compared to random initialized ones (PRandom Linear. PRandom MLP and Ramdom C-Abstractar); indicating that the pretrained project ac- tively compresses the vision features. By contrast, the random projector barely changes the entropy, suggesting no meaningful compression occurs.
MLP v.s. Linear. The vision feature after the MLP projector PILAVA MIP yields larger drop in entropy than a linear projector PILAVA Linear; suggesting that a deeper, non-linear transformation can better remove "redundant" information. A simple linear mapping merely rotates or shifts the embedding space; has limited capacity that discards irrelevant information. This provides explanation for the performance advantage of MLP over linear porjector [2].
The compression appears essential for alignment since text embeddings, unlike visual inputs, compact and discrete- structured around a finite vocabulary and ooken-based representation. The vision projector must therefore transform high-dimensional, continuous visual data into a format that aligns with text embeddings, naturally producing a more condensed output. However, entropy analysis alone cannot reveal how this alignment occurs at the patch level. Therefore, examine patch-level alignment from a micro-scale perspective in the next section.
3.2 Micro-scale Analysis: Patch-Level Alignment
Unlike the CLIP model, where an image and its corresponding caption are encoded as an embedding vector, alignment can be simply measured using the cosine similarity between the two embedding vectors. Here, however, since the image patch embeddings are given as input to the LLM L, we aim to measure their alignment with word embeddings W. This presents several challenges: (1) There lack of labels for each patch; (2) each word may decomposed into multiple tokens or subwords in the LLM; (3) each image patch may contain multiple semantic meanings. To address these challenges, we propose two complementary approaches study patch-level alignment.
3.2.1 Patch-Level Localization
Given input image X with S patches, embeddings. Due to the lack of labels for each patch, we adopt a simpler approach that relies only on labels for the objects within the image. In particular, suppose the image X contains P objects, each defined by its object tags and bounding box locations. We will develop a mask-label annotation pipeline in the next section. Let the ground-truth mask-label pairs be denoted as {(M(p) )
V = = E(X) RaxS denote the S vision
For each label (P) which may contain multiple words or single word that is decomposed into multiple subtokens in the LLM, we compute its text embedding {(p) by averaging the LLM word embeddings of all its subtokens:
(P)) ΣV
w (p)
JK = -W(((()))
where e is the tokenizer that converts [P] into K subtokens, and wo() W represents the k-th subtoken embedding. As each object may occupy multiple patches, we identify the relevant patches by computing the cosine similarity ())( vx) between vision patch v; E V and the text embedding g(r) We then select the patches whose similarity score exceeds an adaptive threshold > 0, i.e.
Idx = {i COS( m.) c, Vos V).
which further gives the predicted bounding box locations Mprend A visualization of the ground truth mask M and predicted mask Mpre is shown in Figure I (left).
5

---

We now quantify the patch alignment Align(V, (V. W) between the image embeddings V and the word embeddings W through the Intersection over Union (IoU) against the ground-truth mask M:
P Intersection
(M pred: M(()
Align(V V. W)
(4)
P
Union (M prod M
To measure the patch-level alignment of projector, we compare the above measure over three variants: the projec- tor after pretraining (PLLaVA Stagel), ). the projector after SFT (PLLaVA Suge2), and a random MLP projector (Raadomm MLP). The results on GranDf dataset [15] are shown in Table 2.
Table 2: Patch alignment of projectors.
Projector
Align(V (V, W)
PRamko MLP
0.065
PELAVA Sagel
0.142
PLLAV Suge2
0.152
Projector improves patch-level alignment. As shown
in Table 2, the pretrained projector achieves higher Align(V, W) than random one, with the measure further improving after SFT, indicating better alignment between the vision and word embedding spaces. However, we also observe that the LLLVV projector exhibits low mloU values in both Stage and Stage 2, suggesting that text labels derived from LLM embeddings cannot accurately identify their corresponding image patch positions. This underscores the limitation of the original LLaVA projector in patch-level alignment.
3.2.2 Multi-Semantic Alignment
Algorithm 1 Matching Pursuit for Vision Embedding
While the above approach provides quan- titative method to measure patch-level alignment, the label for each object is of- ten very short. For example, TV may be labeled simply as "TV," without additional attributes such as color. On the other hand, the continuous vision embedding v is ex- pected to carry multiple semantic meanings that are understandable by the LLM. We express this as the following hypothesis.
Input: Vision embedding v E Rd; LLM word em-
bedding matrix W = [W1, W2,..., warl E RdxM, Number of selected word embeddings K.
Output: Top-K matched word embeddings {w 1=1
Initialize (1) +-
Initialize an empty set S + for i 1 to K do
Find the most relevant word embedding w!() +- arg max (w, v()))
Hy Hypothesis 3.1. In an MLLM, the embed- ding v for each vision patch can be de- composed as sparse linear combination of word embeddings: v = Even wg where Z is the set of subtokens represent- ing all semantic meanings within the patch, and 0g is the coefficient for each subtoken.
weW
Store selected embedding
+ --S (w((3) I/ Remove projection
((+1) + x(()) - (w ((0))
end for Return S
This hypothesis is similar to previous ones on (contextualized) word embedding vec-
tors as a sparse linear combination of word/transformer factors [20, 21J, but extended to multimodal embeddings. To support this hypothesis, we utilize the matching pursuit algorithm [22] to identify the top- K most relevant subtokens. Specifically, at the i-th iteration, the algorithm selects the discrete word embedding (+) from the LLM embedding space W that has the highest similarity with the current vision embedding (1), ensuring is the most semantically aligned token. Once selected, its contribution is removed from the vision embedding. Through this iterative process, we identify the key semantic components within the vision embedding. We present the details in Algorithm 1.
As in [20, 21], we provide qualitative results due to the lack of ground truth multi semaiic labels. As shown in Figure 1, while LLaVA vision embeddings can encode basic object information (c.g.. "TV") and color attributes (e.g., "white"), many patches remain uninterpretable. This analysis reveals that the projector has limited multi-semantic alignment capabilities. More results in Appendix D.
4 Patch-Aligned Training and Analysis
The previous section demonstrates that training with image caption task leads to weak peteh- alignment. In this section, we propose patch-aligned training to enhance fine-grained alignment.

---

Vision Enooder &
Projector
"stndwichh" "cell phone"
Vision Embeddings
"cardboardd" "dining table"
Recognize (R))
Tags
Global Level Alignment
Patch-J avel Alignment
M
Grounding DINO
Bounding
LLM
8
Vision
VOD
Cos(y (lo))
Captiun
Text
e
ASSthing(SAMM
Segment
Masks
surfboord high
LIM Embelding
harnessing the wind
"Sumboard"
19
Figure 2: Annotation pipeline.
Figure 3: Overview of patch-level alignment method.
4.1 Patch-Aligned Training
Mask-Label Annotation Pipeline. We develop an automated pipeline to create the Patch-Aligna Dataset (PAD), which enriches the LLaVA -reetriiid dataset with fine-grained annotations including object tags, bounding boxes, and segmentation masks. Our pipeline combines state-of-the-an models (RAM[23), Grounding DINO[24), and SAM[25]) for object recognition and segmentation. As shown in Figure 2, RAM first generates object tags, which Grounding DINO uses to create bounding boxes. After filtering with non-maximum suppression, SAM generates segmentation masks for each object. More details about the format of PAD can be found in Appendix A.
Patch-Aligned Training. Given an input image X with P associated mask-label pairs {(M1() ( the vision embedding after projection is represented as V = P E R For each mask ((P), let Idx (P) represent the set of vision tokens that are covered by the mask for at least half the patch area. We use (p) to denote the embeddings of the selected vision tokens in Figure 3. Using the same approach as in eq. (2) to compute the text embedding {(P) for the label L(6), similarly represent the vision embedding of the object by taking the mean of the selected vision embeddings (P)) 7 Evela(()) E Rdd We then introduce the patch-alignment loss to maximize the cosine similarity between the mask-selected vision embedding (v)) and the corresponding text embedding {(p);
P Σ P
I patch = 1 cOS( {(P)
To achieve global-level alignment, retain the commonly used caption loss. Specifically, given tokenized caption tokens (x) (Z,,,2,.... IT) for the input image, the caption loss computes the ability to predict each subsequent word in the caption sequence
Leaption
Σ logpe (ax V, Veee),
where Pc (xx, V, .rc))) denotes the predicted probability for the 1-th token x, based on the previous tokens Jet and the vision embedding V.
Our patch-aligned training combines both the caption loss and the patch-alignment loss
I Tepption + 3 Tpatchs
where 3 > 0 is used to balance the global-level aligmment and patch-level alignment.
Efficiency of the patch-alignment loss. The patch-alignment loss Lpatch is computationally more efficient than the caption loss, does not rely on the L.M.. only requires calculating cosine similarity with the LLM word embedding matrix W, making it lightweight to compute and optimize.
7

---

4.2 Ablation Study on 33
In this section, we present our ablation studies on hyperparameter 3 in Equation (7) in two aspects:
Linear increasing vs. fixed schedule. The linear schedule gradually increases to impose patch- level alignment progressively. This design choice stabilizes early training and prevents premature over-compression. As shown in Table 3, when comparing fixed / = 5 versus linearly increasing from 0 to 5, the linear schedule showed better performance. Therefore, adopted the linear increasing schedule for all subsequent experiments.
Optimal final 3 value: Since 3 in the objective function balances global-tevel and patch-level alignment, a larger 3 emphasizes patch-alignment loss over caption loss. We examined the impact of [ {0, 10), where the baseline LLaVA-1.5 corresponds when 33 is too small (3 = 0 or 2), patch-level alignment remains insufficient, leading to suboptimal performance. Conversely, when 3 is too large (3 10), the model overly focuses local regions, compromising its ability to establish comprehensive correspondence between the entire image and sentences. This imbalance degrades global-level alignment and ultimately harms overall performance.
3 0. As shown in Table 3,
Table 3: Comparison of different 8 schedules on various benchmarks.
Setting
GQA
Science QA
VizWiz VQA
OKVQA
Avg
fixed (3 = 00)
61.93
66.80
50.00
53.42
58.04
fixed (3 5)
62.16
68.07
54.86
56.51
60.40
linear increasing (33 E [0, 2])
62.64
68.12
50.84
57.52
59.78
linear increasing (3 E [0, 10)
62.64
68.57
50.82
56.85
59.72
linear increasing (3 E 50)
62.99
68.67
52.29
58.29
60.56
4.3
Compresion-Infornatii Loss Tradeoff
There exists a tradeoff between redundancy removal and semantic information loss when the compres- sion level continues to increase. Within proper compression range, more compression improves per- formance by removing redundant information and enhancing alignment. However, over-compression may potentially cause useful semantic information loss and performance degradation. To empirically validate this tradeoff, we conducted controlled ablation where we varied the patch loss weight / in Equation (7). Larger 33 encourages stronger patch-level alignment and induces more compression. As shown in Figure 4, ÁH increases, the overall performance first improves then declines. Before the tipping point, redundant information removed, which improves performance compared to the original LLa VA with 3 O. However, after the tipping point, performance drops rapidly as over-come over-compression causes semantic information loss.
0.70
3.68
Accuracy
3.66
59.
0.64
59.0
0.62
38.5
0.60
0.58
Figure 4: Tradeoff between compression and information loss.
Here, the change of entropy measured normalized AH(V) (Hhefore (V) Haner(V)) / (V) E [0, Task performance is measured by taking the average of the per- formance over the QA datasets (GQA, Science QA, VizWiz VQA, and OKVQA).
5 Experiments
In this section, we first introduce the experiment setup and training details for Patch-Aligned Training, which is used only in the pretraining stage for training the projector P. We evaluate the effectiveness of our methods in two stages: pretraining stage and SFT stage. the pretraining stage, we verify that the patch-aligned training achieves better compression and patch-level alignment abilities, enabling

---

Table 4: Compression and alignment.
Table 5: LLaVA v.s.
Patch aligned method for caption generation qualities.
Projector
AH(V)
Align(V, W)
Cos Sim
Model
METEOR
ROUGE
SPICE
PRimdom
0.0108
0.065
0.06
PLLVV
2.7991
0.142
0.07
MILAVA
0.1220
0.1661 0.1759
0.1571
Paichh Aligned
3.8352
0,279
0.56
Mparch Aligned
0.1256
0.1710
Table 6: Comparison refer expression comprehension benchmarks.
Models
RefCOCO test-A
ReSCOCO+ test-A
RefCOCOg
val
test-B
val
test-B
val
test
LLaVA 1,5-7B
56.22
64.43
47.38
50.00
59.2
39.0
48.8
48.4
Patch Aligned (Ours)
65.97
72.26
55.82
58.49
66.87
48.09
55.78
56.24
the LLM to generate higher-quality captions. In the SFT stage, we verify that fine-tuning with the new projector yields better performance across three aspects: (1) refer expression comprehension, (2)
visual question answering and (3) instruction following benchmarks.
5.1 Experiment Setup
For a fair comparison, we follow LLaVA-1.5 [1]'s architecture, training setup, and datasets. Our approach differs in two key aspects: (1) introduce patch-aligned loss where increases linearly from 0 to 5 during stage 1, and (2) we use the PAD dataset with detailed annotations to pretrain the projector. See Appendix B for details.
5.2 Stagel: Pretrained Model Evaluation
5.2.1 Compression and Patch-Level A Alignment
We evaluate the patch-aligned projector Prach Aligned using: Von Neumann entropy reduction before) AH(V) Heéoré H(V anter) patch alignment Align(V, W) (Section 3.2.1), and vision- text embedding cosine similarity (Section 3.2.2). As shown in Table 4, compared to baselines PRandom and PILAAA 100 COCO 2017 images [39], our projector achieves higher entropy reduction and better performance on both mIoU and cosine similarity, demonstrating superior patch-level alignment.
5.2.2 Measuring Caption Quality
To examine the advantages of improved patch-level alignment, we first evaluate the stagel MLLM MPasch Aligned = (E, PPPac Aligned) directly on caption generation while keeping both the vision encoder and LLM frozen. To measure the caption quality, utilize three metrics: METEOR, ROUGE-L, and SPICE. As shown in Table 5, MPacc Aligned generate higher quality of captions that benefits from the explicit patch-level alignment process in the pretraining stage.
5.3 Stage2: SFT Model Evaluation
5.3.1 Refer Expression Comprehension
To better demonstrate the method's enhanced fine-grained image understanding and localization capabilities, we further evaluate our approach on the refer expression comprchension(REC) tasks, including the RefCOCO[40), Re/COCO+[41], and Re/COCOg[41]. Specifically, the REC task requires the model to localize the target object under the guidance of a description. Here we report Acc@0 (cc@0.5(hin) 0.5 is better). As shown in Table 6, our method achieves significant improvements across all test splits, with approximately a 16% improvement on average. Notably, our approach uses the same architecture and training data as LLaVA- 1.5, only adding patch alignment during pretraining. This minimal change yields substantial improvements in grounding and localization abilities.
5.3.2 Visual Question Answering
Visual understanding plays an important role in many real-world applications. We test how well our models perform on text-based visual question answering tasks using multiple benchmark datasets. As shown in Table 7, our method outperforms the -.. baseline under identical conditions, demonstrating that initializing the MILMM with improved patch-level aligned vision embeddings leads to better fine-grained understanding and enhanced overall performance.

---

Table 7: Comparison on visual question answering benchmarks.
Models
LM
Img Sz
GQA
SciQA
VizWiz
OKVQA
BLIP-2 [4]
Vicuna-133 13B
224
41.0
61.0
19.6
InstructBLIP [27]
Vicuna-7B
224
49.2
60.5
34.5
InstructBIIIP [27]
Vicuna-133
224
49.5
63.1
33.4
Shikra [42]
Vicuna-13BB
IDFFICS-9B [43]
LLaMA-7B
224
38.4
35.5
IDEFICS-80B [43]
LLMMA-65B
224
45.2
36.0
Qwen-VL [44]
Qwen-7B 7
448
59.3
67.1
35.2
Owen-VL-CChe [44]
Qwen-7B
57.5
68.2
38.9
LLaVA [1]
Vicuna-77 7B
224
LLaVA-1.5 [2]
Vicuna-7B
336
62.0
66.8
50.0
53.4
PatchAligned (Ours)
Vicuna-7B
336
63.0
68.7
52.3
58.3
Table 8: Comparison on instruction following benchmarks.
Models
MMMU
MMVet
CMMMU
MMB
MME
MME
LLaVA 1.5
35.30
30.70
21.80 22.70
64.00
316.10 339.64
1510.75
Patch Aligned (Ours)
36.56
31.61
63.14
153133
Table 9: Analysis of compatibility when switching between base LLMs and projector types.
Models
RefCOCO
RefCOCO+
RefCOCOg
val
test-A
test-B
val
test-A
test-B
val test
Applying to different L.Mss (Vicuna 7B [45] - Llama 3.1 8B(46])
LLaVA (LLaMA3.1 8B)
66.32
74.30
56.09
58.92
68.62
48.21
56.90
55.87
Patch Aligned (LLaMA3.1 8B)
68.27
74.99
58.49
61.25
68.88
50.93
58.41
57.58
Applying to different projectors (MLP
C-Abstractor (38J)
LLaVA (C-Abstractor) 58.08
65.83
49.44
50.28
59.10
41.01
49.38
49.81
Patch Aligned (C-Abstractor)
60.89
67.70
51.21
53.56
60.74
42.01
51.91
51.61
5.3.3 Instruction Following Benchmarks
In addition to conventional vision-language evaluations, we assess our method's real-world capabili- ties by conducting evaluations on modern ciiruction-folowiing benchmarks. As shown in Table our model demonstrates superior performance in understanding when following user instructions.
5.3.4 Compatibility Patch Aligned Training
We demonstrate our method's general effectiveness by replacing both the MLLM (from Vicuna 7B[45] to Llama 3.1 8B[46]) and the projector (from MLP to C-Abstractor(38)). Using C-Abstractor, set the number of output visual tokens to 256. All models follow the same training LLaVA 1.5. We focus our evaluation on referring expression comprehension capabilities. As shown in Table 9, patch-aligned methods achieve consistent improvements compared their variants.
Conclusion
In this paper, we examine the projector's role at and micro scales, showing it compresses visual information while improving patch-level alignment. Though alignment remains coarse, our proposed patch-aligned training enhances both compression and alignment capabilities. This leads better caption generation and grounding performance, providing insights into multimodal reasoning.
However, despite the improvements, certain limitations remain to be addressed. First, while introduce the selli--eemntic alignment hypothesis, finding an optimal representation for each visual token in the embedding space remains significant challenge. Simply aligning with the same averaged word embedding may limit the interpretability and expressive power of LLMs. Moreover, due to the inherent compactness of language, manually guiding the projector for semantic alignment raises concerns about potential information loss visual tokens. Addressing these challenges requires the development of more effective alignment strategies, which will be crucial for further enhancing the capabilities and robustness of multimodal LLMs.
10

---

Acknowledgements
We acknowledge support from NSF grants IIS-2312840 and IIS-2402952, as well as the ORAU Ralph E. Powe Junior Faculty Enhancement Award.
References
[1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[2] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[3] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12999 12900. PMLR, 2022.
[4] Jumnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[5] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing languala understanding with advanced large language models. arXiv preprint ar04v:2304. 10592, 2023.
[6] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35: 237 23716-23736, 2022.
[7] Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems, 36: 72096-72109, 2023.
[8] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Mul- timodal chain-of Mlain-of-thoue of-thhutttttt reasoning in language models. arXiv preprint arXiv:2302.00923, 2023.
[9] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang. Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang. Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang. Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuchai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang. Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with Ilms, 2025. URL httph:ttt////cctt /arxiv.ivv.. wwrrvi.og////eeee.. wwwgobs/////oeooooooooooooeeeeee abss/501... 12599.
[10] Jun Yu, Jing Li, Zhou Yu, and Qingming Huang. Multimodal transformer with multi-view visual representation for image captioning. IEEE transactions on circuits and systems for video technology, 40(12)4447744480 2019.
[11] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnikk, and Devi Parikh. Vqa: Visual question answering. Proceedings of the IEEE international conference computer vision, pages 2425-2433, 2015.
11

---

[12] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9568-9578 2024.
[13] Shilong Zhang. Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interee arXiv preprint arXiv:2307.003000 2023.
[14] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[15] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, AbdeIrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13009-13018, 2024.
[16] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404. 18930, 2024.
[17] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality- enhanced projector for multimodal 11m. 2024 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 13817-13827, 2023. URL https://api.
semanticscholar org/ :etg:/orppsI::::000 :6:6177117...
[18] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024.
[19] John von Neumann.
Mathematische Grundlagen der Quantermechanik. Springer, Berlin, Germany, 1932. English translation: Mathematical Foundations of Quantum Mechanics, Princeton University Press, 1955.
[20] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495, 2018.
[21] Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding linear superposition of transformer factors. arXiv preprint arXiv: 15949, 2021.
[22] Joel A Tropp and Anna C Gilbert. Signal recovery from random measurements via orthogonal matching pursuit. JEEE Transactions on information theory, 53(12):4655 2007.
[23] Youcai Zhang. Xinyu Huang, Jinyu Ma. Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: strong image tagging model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1724 1732, 2024.
[24] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference Computer Vision, pages 38-55. Springer, 2025.
[25] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. In Proceedings of the IEEF/CVF International Conference Computer Vision pages 4015-4026, 2023.
[26] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-mttelleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024.
12

---

[27] Wenliang Dai, Jumnan Li, D Li, AMH Tiong, Zhao. W Wang, B Li, P Fung, and S Hoi. Instructblip: Towards general vision- language models with instruction tuning, arxiv 2023. arXiv preprint wrXiv:2305.065000 2, 2023.
[28] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprini urXiv:2306.05425 2023.
[29] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan. Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
[30] Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vims: Investigating the design space of visually-conditioned language models. arXiv preprini ar urXiv:2402.07863. 2024.
[31] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-I: A fully open, vision-centric exploration of multimodal Ilms. arXiv preprint arXiv:2406. 16860, 2024.
[32] Shijia Yang. Bohan Zhai, Quanzeng You, Jianbo Yuan, Hongxia Yang, and Chenfeng Xu. Law of vision representation in milms. arXiv preprint arXiv:2408. 16357, 2024.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8766 8763. PMLR, 2021.
[34] Qidong Huang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Deciphering cross-modal alignment in large vision-language models with modality integration rate. arXiv preprint arXiv:2410.07166 2024.
[35] Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. To- wards interpreting visual information processing in vision-language models. arXiv preprint arXiv:2410.07149, 2024.
[36] Wei Wang, Zhaowei Li. Qi Xu, Linfeng Li, YiQing Cai, Botian Jiang, Hang Song, Xingean Hu, Pengyu Wang, and Li Xiao. Advancing fine- the-graiied visual understanding with multi-scale alignment in multi-modal models. arXiv preprint 2rXiv2241.00969, 2024.
[37] Yuanyang Yin, Yaqi Zhao, Yajie Zhang, Ke Lin, Jiahao Wang, Xin Tao, Pengfei Wan, Di Zhang, Baoqun Yin, and Wentao Zhang. Sea: Supervised embedding alignment for token-level visual- textual integration in milms. arXiv preprins arXiv:2408, 1813, 2024.
[38] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality- enhanced projector for multimodal Ilm. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 13817-13827, 2024.
[39] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[40] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[41] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11 11-20, 2016.
[42] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Im''s referential dialogue magic. arXiv preprint 230.22306.5195 2023.
13

---

[43] Hugo Laurençon, Daniel van Strien, Stas Bekman, Léo Tronchon, Lucile Saulnier, Thomas Wang. Siddharth Karamcheti, Amanpreet Singh, Giada Pistilli, Yacine Jernite, and Victor Sanh. Introducing idefics: An open reproduction of state-of-the-art visual language model. 2023. URL https://huggingfaaee wt/blo///cicccccccc idsfics. Accessed: 2025-01-30.
[44] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilitics. arXiv preprint arXiv:2308. 12966 2023.
[45] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprini arXivv2304.03277, 2023.
[46] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[47] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicua. Imsys. org (accessed April 2023), 2(3);6,23 2023.
[48] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8317-8326, 2019.
[49] Drew A Hudson and Christopher Manning. Gqa: new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[50] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947-952. IEFE, 2019.
[51] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 23:32-73, 2017.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction is accurate.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations or our work in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We provide the assumptions and proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper provide the information to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We would release the dataset and code to reproduce the main experimental
results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. ec/
Policy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips... cc/ htapllicggide///eeee
sssisssnn ssisssolicyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide experimental setting in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resources in experimental setting in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: h http///eerrp...ee wttppppssccppppcccpeeeeeeeee httpplc/Ethissuuleeeeee ctcuidell ines?
Question:
Answer: [Yes]
Justification:
This paper conducted in the paper conform.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: all datasets used in this paper are public available.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
This paper do research related to multimodal LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: / for what should or should not be described.
21

---

Appendix
The appendix is organized as follows. First, we show the details of mask-label annotation pipeline and the the patch-aligned dataset format in Appendix A. Then, we introduce the details about experiment setting in Appendix B. Finally, we present visualizations related to token-level Sppecficiiil patch-level localization Appendix C and multi-semantic alignment Appendix D.
A Patch Aligned Dataset
In this section, present the format of the Patch Aligned Dataset(PAD) in comparison the LLaVA pretraining dataset following the mask-label annotation pipeline.
Mask-Label Annotation Pipeline. To address the lack of patch-level annotated data, we develop an automated annotation pipeline for generating the Patch-Aligned Dataset (PAD), designed to refine dataset by incorporating fine-grained details. PAD enriches this dataset with detailed annotations, including object tags, bounding box locations, and segmentation masks for individual objects. By incorporating dense, pixel-level grounding information, PAD is designed to enhance fine-grained image-text alignment during the pretraining stage, thereby improving the model's s ability to understand localized regions within the image.
the
As illustrated Figure 2, our automated annotation pipeline consists of diverse state-of-the models, including Recognize Anything Model (RAM) [23], Grounding DINO [24], and Segment Anything Model (SAM) [25]-t0 perform grounded image segmentation and object recognition. First, RAM generates object tags from the input image. These tags are then passed to Grounding DINO, which generates bounding boxes for each identified object. Afterward, a Non-Maximum Suppression (NMS) process is applied filter overlapping bounding boxes based Intersection over Union (IoU) thresholds. The remaining bounding boxes are passed to SAM, which generates segmentation masks for each object. The pipeline outputs the segmented image with bounding boxes, along with metadata in ISON format, including object tags, bounding box coordinates, and RlE-eecodee masks for further analysis. As shown in Table 10. annotate the images of the LLaVA pretraining dataset with additional object tags, bounding box coordinates, and RL RLE-encoded masks stored JSON file. The RLE-encoded masks can be decoded back into binary masks that have the image size.
Table 10: Comparison of LLaVA Pretraining Dataset and Patch Aligned Dataset (Ours).
"inage_ id : "G0000/00000005" JPB"
LLaVA Pretraining Datasct
"size": [448, 336]
"caption"; "a canyon wall reflects the sunny day in utah."
"inage id : 0000000000000500 jPg"
"aize": [448, 336]
"caption": "a canyon wall reñlecta the water on a aunny day in utab."
"labele":
"tag": "vater",
"bbor": [-0. 000.02043447733285, 182. 072.5599499748093.
447.99961171877, 335. 27926662222266611
Patch Aligned Dataset
>ele_naek": )
(ours)
"tag": "cliff"
"bbox": [-0. 00..1117111140625,, 0.3400475463B671875,
447. 9346005859376, 182. (82.6725887666231
"rle_maak":
"]832. :0e80V 5004LXY2: LMN00H02220P22333000000000000000
}]
22

---

To find the optimal hyperparameters in our mask-label annotation pipeline, we conducted thorough ablation study using the coco-val 2017 [39] dataset, which provides ground truth bounding boxes. We focused on two key hyperparameters:
Score threshold: Only boxes with confidence scores above this threshold are selected.
NMS threshold: During non-maximum suppression (NMS), this determines the maximum allowed overlap between two boxes-if their IoU exceeds this threshold, the box with the lower confidence score is removed.
We evaluated performance using Fl score at IoU 0.5, which classifies predictions as true or false positives based on an IoU threshold of 0.5. First, we tested various score threshold values:
Table 11: Effect of varying score thresholds.
Score threshold
0.1
0.2
0.3
0.4
0.5
F1 @ [loU=0.5]
0.2762
0.4931
0.6326
0.6677
0.6207
Next, we fixed the score threshold at 0.4 and evaluated various NMS threshold values:
Table 12: Effect of varying NMS thresholds.
NMS threshold 0.3
0.5
0.7
0.8
0.9
F1 @ [loU=0.5]
0.6530
0.6677
0.6702
0.6722
0.6717
Based on this analysis, we selected the optimal hyperparameters (Score threshold 0.4 and NMS threshold = 0.8) for our final implementation.
To evaluate our pipeline with optimal syper-parameters, we compare it with the original Grounding DINO[24] on coco-val 2017 [39]. The results are as follows:
Table 13: Comparison of between our pipeline and original Grounding DINO.
AP0160U-0550:0.55 AP@[IoU=0.50)
AP@ [IoU=0.75]
Original Grounding DINO Our pipeline
0.485
0.644
0.529
0.531
0.676
0.572
where AP@ [I0==0.50:0.55] is the mean precision across IoU thresholds from 0.50 to 0.95 (stepped by 0.05). A prediction is considered correct if its overlap with ground-truth exceeds the IoU threshold. As the results demonstrate, our pipeline consistently outperforms the baseline.
B Experiment Setup
Architecture To evaluate the effectiveness of our method, we ensure a fair comparison by following the same architecture as LLaVA 1.5. Specifically, we use CLIP- VVT--L@33666 [33] as the vision encoder ε, Vicuna- ciatnaa-1.55777 1.577B[47] as the I.LM L, and a 2-layer MI.P as the projector P. The parameter 3 follows linear schedule, increasing from 0 to 5.
Training Details Following the standard training paradigm in LLaVA [1], our training pipeline consists two stages. In stage 1, keeping the vision encoder & and LLM L frozen, we train only the projector P using our proposed Patch Aligned Training method obtain the patch-aligned projector Prach Aligned- In stage 2, we perform supervised fine-tuning on both the L.M L and the patch-aligned projector PPruchh Aligned- Following I.LaVA's hyperparameters, we optimize all models for epoch using the Adam optimizer with a cosine learning schedule. The learning rates are set to 1e-3 for pretraining and 2e-5 for instruction tuning. Pretraining requires approximately 8 hours using 8xA5000 GPUs (24G), while visual instruction tuning takes about 10 hours for LLaVA-vI.5-7B on 8xH100 (80G).
Dataset For pretraining dataset, utilizing our automated annotation pipeline, we annotate the 558K subset of the LAION-CC-SBR dataset, which is used as the pretraining dataset of LLaVA. The
23

---

resulting dataset comprises 2.3M regions, each associated with segmentation mask, and includes 33.5K unique tags. For fair comparision, we use the same vision instruction tuning dataset as the one in the LLaVA-115, containing I.LaVA-Inttruct [1], TexiVQA [48], GQA [49], OCR-VQA [50], and Visual Genome[51].
C Patch-Level Localization: More Visualizations
Following the micro-scale analysis on patch-level localization Section 3.2.1, we provide more examples comparing the ground truth mask MGT and predicted mask Mpred generated by three pro- jectors: the random projector P Random- pretrained LLaVA 1.5 projector PaLaaA and our PatchAligned Projector Patch Aligned- As shown in Figure 5, the P Patch Aligned predicts more accurate masks.
"fence"
"giraffe"
"tree"
"grass"
GT
Raandon
PILVVA
Patch Aligned
Figure 5: Additional visualization for patch-level localization.
24

---

D Multi-Semantic Alignment: More Visualizations
We begin by showing the first iteration of matching pursuit, which finds the token in the LLM embedding space that has the highest similarity with the vision embedding. We show the full tokenmap in Figure 6, displaying the found token for each vision patch. We use font size to represent similarity. Tokens recognizable by NLTK are shown in color, while unrecognized tokens remain black. For LLaVA, only partial areas objects achieve alignment. In contrast, PatchAligned LLaVA achieves better alignment across most patches.
LLaVA
PatchAligned LLaVA
dky
day
slay
a
Cir GD
Figure 6: Additional visualization for tokenmap comparing LLaVA and PatchAligned LLaVA.
25

---

Next, following Section 3.2.2, we apply matching pursuit on PatchAligned LLaVA for 5 iterations. As shown in Figure the semantic meanings are decoded for each iteration, with cosine similarity decreasing across iterations.
(a) Matching Pursuit Iter (b) Cosine Similarity
(c) Matching Pursuit Iter
(d) Cosine Similarity Iter
(e) Matching Pursuit Iter
(t) Cosine Similarity Iter 2 (g) Matching Pursuit Iter 2 (h) Cosine Similarity Iter
(i) Matching Pursuit Iter 3 (j) Cosine Similarity Iter 3 (k) Matching Pursuit Iter 3 (l) Cosine Similarity Iter
(m) Matching Pursuit Iter
(n) Cosine Similarity Iter 4 (o) Matching Pursuit Iter 4 (p) Cosine Similarity Iter
(q) Matching Pursuit Iter (r) Cosine Similarity Iter 5 Matching Pursuit Iter 5 (t) Cosine Similarity Iter
Figure 7: Perform Matching Pursuit using PatchAligned LLaVA. Each row represents an iteration, with selected tokenmap (Column 1,3) and cosine similarity maps (Column 2,4).
26


Reviews:
[{"review_id": "QvCNfa50vI", "reviewer": "Reviewer_UhGf", "summary": "This paper tries to enhance the performance of multimodal language models through fine-grained alignment between text and visual cues. It first analyzes the role of projector in compressing vision embeddings and aligning with word embedding. It hence proposes the patch-aligned training, which boosts the performance.", "questions": "The authors should further clarify the technical novelty and significance of the proposed method and conclusions.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\nThe proposed method boosts the performance of MLLM as shown in experiments. \n\nWeakness:\n- The proposed method simply adds a patch loss, which is computed based on automatically generated patch-text pairs. The technical contribution is quite limited. \n- The findings in Sec 3 reveals that existing projectors show limited multi-semantic alignment and patch-level alignment. Those findings, however are not quite innovative and can be inferred from the performance of MLLMs. E.g., the poor localization capability indicates the poor patch-level alignment.\n- The presentation should be improved. Many symbols are used with different meaning, e.g., V_before, and V_after show different dimensionalities in Sec. 3.1.", "was_revised": true, "final_justification": "The rebuttal has addressed part of my concern on the novelty and presentation. I thus would like to raise my score.", "rebuttal": "We sincerely thank the reviewer for the feedback. Below, we respond to each of the main concerns individually and clarify the corresponding points.\n\n> Q1: The proposed method simply adds a patch loss, which is computed based on automatically generated patch-text pairs. The technical contribution is quite limited. \n\nA1: Thank you for your comments. We appreciate the opportunity to clarify our contributions. We want to first note that the patch loss only serves as one component of our contributions. In this work, we develop a comprehensive framework for both analyzing and improving fine-grained vision-text alignment in MLLMs. We highlight our key contributions as follows: (1) examining the projector's role in compressing vision embeddings and aligning them with word embeddings from an entropy perspective, (2) proposing metrics that quantitatively reveal how projectors trained only with caption loss (the standard approach) achieve limited patch-level alignment, resulting in weak and coarse-grained alignment,  (3) using matching pursuit to find the proper word embeddings to preserve multiple semantic meanings,  and (4) finally proposing a patch-alignment method to enhance patch-level alignment. \n\nWith respect to the proposed patch loss, as mentioned by Reviewer zak5, it offers several key advantages: (1) Efficient and simplicity. The cosine similarity between visual tokens and LLM’s embedding tokens is computationally efficient. Since the patch-level loss is independent only requires the LLM's embedding matrix and operates independently of the LLM's forward pass, the training time of PatchAligned LLaVA remains comparable to the original LLaVA. (2) Effectiveness. Despite being a straightforward modification, the proposed align loss significantly improves the projector's fine-grained visual-text token alignment, yielding substantial improvements: $16$% on referring expression grounding tasks, $4$% on question-answering tasks, and $3$% on modern instruction-following benchmarks. (3) Adaptable to any MLLM framework. Our approach requires no specialized architecture and can be easily integrated into various MLLMs to enhance fine-grained alignment between different LLM and vision encoders.\n\n> Q2: The findings in Sec 3 reveals that existing projectors show limited multi-semantic alignment and patch-level alignment. Those findings, however are not quite innovative and can be inferred from the performance of MLLMs. E.g., the poor localization capability indicates the poor patch-level alignment.\n\nA2: We respectfully disagree that these findings lack novelty. To the best of our knowledge, no prior work has formally studied patch-level alignment and multi-semantic alignment. If the reviewer is aware of prior work that directly quantifies these alignments, we would be happy to cite it. While poor localization performance of MLLMs may hint at weak patch-level alignment, it can also arise from several other factors, such as limitations of the vision backbone or the language model’s ability to interpret visual tokens. Without a formal metric and analysis, these factors cannot be disentangled. To bridge this gap, Section 3 introduces a quantitative metric ($\\mathrm{Align}(\\mathbf{V}, \\mathbf{W})$) for patch-level alignment and proposes an algorithm to analyze multi-semantic alignment. Using our $\\mathrm{Align}(\\mathbf{V}, \\mathbf{W})$ metric, we show that the original LLaVA projector exhibits weak alignment, improving $\\mathrm{Align}(\\mathbf{V}, \\mathbf{W})$ only from $0.065$ to $0.142$ compared to a random projector. Our proposed  patch-align training further improves $\\mathrm{Align}(\\mathbf{V}, \\mathbf{W})$ to $0.279$, leading to a $16$% gain on referring expression grounding tasks. We will incorporate this point at the beginning of section 3.2.2. in the revision.\n\n> Q3: The presentation should be improved. Many symbols are used with different meaning, e.g., V_before, and V_after show different dimensionalities in Sec. 3.1.\n\nA3: The $\\mathbf{V}\\_{\\mathrm{before}}$ and $\\mathbf{V}\\_{\\mathrm{after}}$ could have different dimensionality. As explained in Eq. (1),  $\\mathbf{V}\\_{\\mathrm{before}}$ and $\\mathbf{V}\\_{\\mathrm{after}}$ refer to features before and after the projector, in MLLMs such as LLaVa which have a structure Vision Encoder $\\rightarrow$ Projector $\\rightarrow$  LLM. For example, the vision encoder CLIP-ViT-Large-Patch14-336 has an embedding size of $1024$ (the dimension of $\\mathbf{V}\\_{\\mathrm{before}}$), whereas Vicuna 7B has an embedding size of $4096$ (the dimension of $\\mathbf{V}\\_{\\mathrm{after}}$). We will add a clarifying comment after Eq. (1) to make this point explicit. If the reviewer has noticed any other issues with the notation, we would be happy to address them."}, {"review_id": "vDswFCdqQ0", "reviewer": "Reviewer_ZSmz", "summary": "This work treats the projector in multimodal large language models (MLLMs) as having a dual role of “information compression + fine-grained alignment.” First, it uses Von Neumann entropy to quantify and demonstrate that the projector substantially compresses visual redundancy. Next, it introduces Patch-Aligned Training: leveraging PAD data automatically generated by RAM, Grounding DINO, and SAM to apply a cosine alignment loss to each visual patch. Without altering the network architecture, the new projector yields a marked performance boost for LLaVA-7B.", "questions": "- This paper relies on a combination of RAM, Grounding DINO, and SAM for automatic annotation. Could you provide evaluation results comparing with manual annotations in terms of IoU/mAP, so as to quantify the true impact of false positives and false negatives on training?\n\n- In the RefCOCO and instruction-following sections, is there a side-by-side comparison with models of the same parameter size?\n\n- Von Neumann Entropy may indicate better alignment, but is there a ΔH versus task performance curve that reveals the tipping point where over-compression causes semantic information loss?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- This paper introduces Patch-Aligned Training based on information compression and fine-grained alignment. By changing only the loss function—with no network modifications—it incurs very low computational overhead yet delivers significant gains on tasks such as localization and VQA.  This paper writing is well written and easy to follow, and performance well.\n\n\n## Weaknesses\n- When Grounding DINO produces erroneous boxes, non-maximum suppression (NMS) and confidence‐score thresholding are employed to remove overlapping or low-quality detections. However, the quality of this filtering has not been evaluated, and the effectiveness of the automatic annotations remains unquantified.\n\n- There is a similar work *SEA: Supervised Embedding Alignment*, which aligns each visual token to the LLM word embedding to solve the problem that caption-loss can only bring coarse-grained semantics.", "was_revised": true, "final_justification": "This paper identifies the drawback of projector between LLM and vision encoders. It proposed a multi-semantic alignment hypothesis and patch-aligned training to address the drawback, which achieve good improvement and contribute to the community. The rebuttal address all my concerns. Thus, I recommend to accept this paper.", "rebuttal": "We sincerely thank the reviewer for the feedback. Below, we respond to each of the main concerns individually and clarify the corresponding points.\n\n> Q1: Grounding DINO uses NMS and confidence-score thresholding to remove overlapping or low-quality boxes, but the quality of this filtering and the effectiveness of the automatic annotations have not been evaluated.\n\nA1: Thank you for your suggestion. The Mask-Label Annotation Pipeline serves primarily as an automated method to quickly obtain bounding box and segmentation information from the original LLaVA pretraining dataset to support our patch-aligned training. Since our main contributions are the analysis of multimodal projector and the patch-aligned training method, due to the limited space, we omitted the presentation of evaluation of the Mask-Label Annotation Pipeline in the main paper. We will now present the details of our evaluation.\n\nTo optimize our automatic annotation pipeline, we conducted a thorough ablation study using the coco_val_2017 dataset, which provides ground truth bounding boxes. We focused on two key hyperparameters:\n\n1. Score threshold: Only boxes with confidence scores above this threshold are selected.\n2. NMS threshold: During non-maximum suppression (NMS), this determines the maximum allowed overlap between two boxes—if their IoU exceeds this threshold, the box with the lower confidence score is removed.\n\nWe evaluated performance using F1 score at IoU of 0.5, which classifies predictions as true or false positives based on an IoU threshold of 0.5. First, we tested various score threshold values:\n\n| Score threshold | 0.1   | 0.2   | 0.3   | 0.4   | 0.5   |\n|------------------|-----|--------|--------|--------|--------|\n| F1 @ [IoU=0.5]     | 0.2762 | 0.4931 | 0.6326 | **0.6677** | 0.6207 |\n\nNext, we fixed the score threshold at 0.4 and evaluated various NMS threshold values:\n\n| NMS threshold   | 0.3    | 0.5    | 0.7    | 0.8    | 0.9    |\n|------------------|--------|--------|--------|--------|--------|\n| F1 @ [IoU=0.5]     | 0.6530 | 0.6677 | 0.6702 | **0.6722** | 0.6717 |\n\nBased on this analysis, we selected the optimal hyperparameters (Score threshold = 0.4 and NMS threshold = 0.8) for our final implementation. Following the reviewer's suggestion, in the revision, we will include this evaluation in the appendix. \n\n> Q2: There is a similar work SEA: Supervised Embedding Alignment, which aligns each visual token to the LLM word embedding to solve the problem that caption-loss can only bring coarse-grained semantics.\n\nA2: We thank the reviewer for pointing out the related work. While SEA shares the motivation of improving alignment between visual tokens and language embeddings beyond caption-level supervision, our work differs in several significant ways:\n\n1. Fine-grained alignment measurement: SEA focuses primarily on introducing a new training loss, whereas our work provides a systematic analysis of existing projectors using novel metrics, including *Von Neumann Entropy* (quantifying information compression) and $\\mathrm{Align}(\\mathbf{V}, \\mathbf{W})$ (measuring fine-grained alignment). These analyses reveal the structural limitations of current projectors and provide actionable insights.\n\n2. Multi-semantic alignment: SEA treats each patch as corresponding to a single label. In contrast, we introduce a Matching Pursuit-based decomposition to extract multiple aligned tokens per patch, capturing fine-grained multi-token semantics—crucial for complex understanding.\n\n3. Vocabulary coverage: SEA relies on a fixed, predefined vocabulary, which inherently limits semantic granularity. We use RAM to recognize a broad and dynamic vocabulary of visual concepts, allowing our patch supervision to adapt to diverse real-world entities and contexts.\n\n4. Loss formulation and performance: While SEA adopts a contrastive loss with handcrafted negative sampling, we use a simpler cosine loss that eliminates the need for sampling strategies. Since SEA doesn’t release their code, we implement the contrastive loss on our own. Our ablation studies show that our approach outperforms contrastive loss in downstream tasks:\n| Dataset     | Original LLaVA | SEA (contrastive loss) | Ours (cos loss) |\n|-------------|----------------|-------------------------|------------------|\n| GQA         | 61.93          | 60.95                   | **62.99**        |\n| Science QA  | 66.8           | 67.58                   | **68.67**        |\n| OKVQA       | 53.42          | 51.59                   | **58.29**        |\n\nWe will incorporate these key distinctions and comparisons with SEA in the revision.\n\n> Q3: Could you provide evaluation results comparing with manual annotations in terms of IoU/mAP, so as to quantify the true impact of false positives and false negatives on training?\n\nA3: Thank you for the question. Directly creating manual annotations for the entire LAION dataset would require an impractical amount of human effort and time, so we are unable to provide a direct comparison with fully manual annotations. However, we used the coco-val 2017 dataset, which has high-quality annotated bounding boxes, to evaluate our pipeline with optimal hyper-parameters and compare it with the original Grounding DINO. The results are as follows:\n\n|                      | AP@[IoU=0.50:0.95] | AP@[IoU=0.50] | AP@[IoU=0.75] |\n|----------------------|-------------------|---------------|---------------|\n| Original Grounding DINO | 0.485             | 0.644         | 0.529         |\n| **Our pipeline**         | **0.531**         | **0.676**     | **0.572**     |\n\nAP@[IoU=0.50:0.95] is the mean precision across IoU thresholds from 0.50 to 0.95 (stepped by 0.05). A prediction is considered correct if its overlap with ground-truth exceeds the IoU threshold. As the results demonstrate, our pipeline consistently outperforms the baseline. Following the reviewer's suggestion, we will include this evaluation in the appendix of the revision. \n\n> Q4: In the RefCOCO and instruction-following sections, is there a side-by-side comparison with models of the same parameter size?\n\nA4: Thank you for your question. In both Table 5 (referring expression comprehension) and Table 7 (instruction-following),  our comparisons between Patch-aligned LLaVA (7B) and LLaVA-1.5-7B are conducted using exactly the same models (the same architecture and parameter size (7B)), ensuring a fair evaluation. We additionally compare with LION (Dual-Level vIsual knOwledge eNhanced Multimodal Large Language Model) [1] because it reports zero-shot performance on RefCOCO—without training on RefCOCO—consistent with our evaluation setting. In contrast, many other models (e.g., QwenVL) only report results after fine-tuning on RefCOCO, which leads to higher scores. Although LION uses a larger 12B model, as shown in the table, our 7B model still achieves higher average performance on referring expression grounding tasks, highlighting the effectiveness of our method.\n\n| Model     | RefCOCO val | RefCOCO test A | RefCOCO test B | RefCOCO+ val | RefCOCO+ test A | RefCOCO+ test B | RefCOCOg val | RefCOCOg test  | Avg   |\n|-----------|-------------|--------|--------|---------------|--------|--------|---------------|--------|--------|\n| LION-12B  | 58.54       | 56.41  | **59.36** | 45.93       | 45.73  | 47.89  | **66.12**     | **64.69** | 55.58  |\n| Ours-7B   | **65.97**   | **72.26** | 55.82  | **58.49**   | **66.87** | **48.09** | 55.78       | 56.24 | **59.94** |\n\n[1] Chen, G., Shen, L., Shao, R., Deng, X., & Nie, L. (2024). Lion: Empowering multimodal large language model with dual-level visual knowledge.\n\n> Q5: Von Neumann Entropy may indicate better alignment, but is there a ΔH versus task performance curve that reveals the tipping point where over-compression causes semantic information loss?\n\nA5: Thanks for the question. Yes, there exists a tipping point in $\\Delta H$ versus task performance curve. As shown in the table, as $\\Delta H$ increases, the overall performance first improves then declines. Before the tipping point, redundant information is removed, which improves performance compared to the original LLaVA with $\\beta = 0$. However, after the tipping point, performance drops rapidly as over-compression causes semantic information loss. \n\n| $\\beta$ setting                                 | $\\beta = 0$ (original LLaVA) | $\\beta = 2$ | $\\beta = 5$| $\\beta = 10$ | $\\beta = \\infty$(Patch Loss Only) |\n|------------------------------------------|-------------------------|-----------|------------------|------------|--------------------------|\n| $\\Delta H$                                        | 0.5789                  | 0.6039    | 0.6646           | 0.6726     | 0.6972                   |\n| Performance (Avg)                        | 58.54                   | 59.78     | **60.56**        | 59.72      | 57.77                    |\n\nHere, the change of entropy is measured as normalized one: $\\Delta H(V) = \\frac{H_{before}(V) - H_{after}(V)}{H_{before}(V)} \\in [0,1]$. Task performance is measured by taking the average of the performance over the QA datasets (GQA, Science QA, VizWiz VQA and OKVQA). The compression level is controlled by the weight of the patch-aligned loss in the aligned training:  $L = L_{caption} + \\beta L_{patch}$, where larger $\\beta $ leads to larger $\\Delta H$, indicating more compression. \n\nDue to space limitations, we omitted this ablation study in the current version. We will include it in the revised version, either in the appendix or in the main paper, depending on the available space."}, {"review_id": "ZWJjDLw9Z3", "reviewer": "Reviewer_zak5", "summary": "This paper addresses the issue of fine-grained patch-level alignment of the projector in Multimodal Large Language Models (MLLMs). The paper proposes a novel lightweight alignment loss that improves the token-level semantic matching of visual to textual tokens, and through quantitative and qualitative analysis shows significant improvements to VQA, referring expression, and captioning performance, as well as token-level semantic matching. The paper argues that compression of visual information is critical to the performance of a projector in MLLMs; that while the standard caption loss enables coarse alignment with lower compression, the proposed align loss enables strong compression capability and better patch-level alignment.", "questions": "1. Averaging embeddings of tokenized words seems noisy, since the word parts likely carry very different semantic meaning compared to the whole. \n   - It would be interesting to understand the effect on the align loss and multi-semantic alignment when it comes to word parts vs whole words. \n2. The choice of measuring Von Neumann entropy is deliberate, and Line 141-142 does not clarify why this particular metric was chosen, as opposed to Shannon's entropy, to help quantify the degree of \"compression\" in visual tokens after projection. \n   - It would be helpful for the reader, for instance, to note that Von Neumman metric unlike Shannon's is basis independent, an important consideration as the projector is projecting from the visual to the textual space.", "limitations": "Yes. Addressed.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\n - **Contributions seem significant and original:**\n    - Patch-Level Localization: The proposed align loss is novel and seems to be light-weight, relatively efficient, and performative. Based on both qualitative and quantitative results, the proposed align loss significantly improves the projector's fine-grained visual-text token alignment, with significant improvements across captioning, VQA, and referring expression comprehension. Further, the patch level loss is independent of the LLM's forward pass (requires only the LLM's embedding matrix), and is efficient to compute. \n    - Multi-Semantic Alignment: The approach to semantically mapping visual tokens to sets of text embeddings, using the matching pursuit algorithm, is another interesting and novel contribution. This appears to provide, to my knowledge, a first in efficient and effective token-level visual-text semantic matching. The resulting qualitative analysis of mapping the visual tokens to interpretable text tokens is also intriguing, and likely opens up a number of future ideas towards semantic matching analysis in MLLMs. \n- **Quality** : The efficacy of the proposed method seems well supported by the qualitative and quantitative results. A number of ablations are provided applying these methods to varying LLMs and projectors, although there is scope for more detailed ablations supporting varying design decisions (see below on ablating beta hyperparameter in align loss). \n- **Clarity** : The paper is well written, easy to follow, and provides a good introduction and motivation to the problem and approach. There is some room to strengthen the arguments (see weaknesses below), but a strong write-up overall.\n\n\n**Weaknessess:**\n\n- **Weak Entropy argument, with potentially contradictory claims**: \n\n  - The observations in Table 1, justifying the claim of projector compression, by using the comparison of Neumann entropy of vision embeddings before/after projection seem unsupported and even contradictory. Specifically, the relatively larger drop in entropy using MLP projectors, compared to linear projectors, is used to justify that \"deeper, non-linear transformation can better remove 'redundant' information\".  Additionally, this is used to \"provide an explanation for the performance advantage of MLP over linear projector\". However, C-Abstractor has a higher entropy than both MLPs and linear projectors, and has been reported to be more performative than both, for general visual understanding [1]. This contradicts the reasoning and claim made here. \n  - Additionally, the drop in entropy cannot be claimed as being simply due to the removal of \"redundant information\"; potentially useful visual information may have been lost. Certainly, compression is occurring, and does not detract from the central claims of the paper. However, it is unclear if relating that to performance is well-founded. \n  - The same argument appears to be used in Lines 281-282 for Table 3.  It would be helpful to clarify this. \n\n\n[1] Cha, Junbum, et al. \"Honeybee: Locality-enhanced projector for multimodal llm.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n- **Need more detailed ablations on design choices**: \n    - The choice of linearly increasing beta (hypeparameter for the align loss weight) is not justified or ablated in the paper. In general, it would be helpful to provide more detailed ablations on the choice of the beta hyperparameter, linearly increasing it from 0 to 5 during stage 1. \n    - Additionally, it would be helpful to see ablations with other LLM families (beyond LLAMA), as well as other projector types (such as transformer projectors).", "was_revised": true, "final_justification": "Authors have clearly and strongly clarified all concerns; primarily along the lines of some conflicting results in the paper. \n\nThere are no major concerns, and the paper is a technically strong contribution in my opinion. \n\nMany congratulations to the authors on a great contribution.", "rebuttal": "Thank you for your detailed and positive feedback. We greatly appreciate your recognition of our contributions in patch-level localization and multi-semantic alignment. Below, we respond to the remaining concerns individually and clarify the corresponding points.\n\n> Q1: However, C-Abstractor has a higher entropy than both MLPs and linear projectors. This contradicts the reasoning and claim made here. \n\nA1: Thank you for this observation. In our paper, we primarily compare the Von Neumann Entropy $H(\\mathbf{V}\\_{after})$ of linear and MLP projectors since they yield similar values when randomly initialized. We don't directly compare with the C-Abstractor because a random initialized C-Abstractor has a much higher baseline entropy ($7.4913$) compared to Linear ($4.8197$) or MLP ($4.8245$). This demonstrates that absolute entropy is influenced by the inductive bias of the model architecture. The C-Abstractor uses a deep CNN-based architecture with residual connections (ResNet blocks), which—even when randomly initialized—produces more diverse outputs. This higher initial entropy reflects the architectural prior of the network, not its training quality. \n\nTo fairly assess the projector’s training effect (i.e., information compression caused by learning), we can measure the entropy drop ΔH, which removes the influence of architecture priors: $\\Delta H(V) = \\frac{H_{random}(V) - H_{trained}(V)}{H_{random}(V)} \\in [0,1]$. where we get: $\\Delta H_{C-Abs}  = 0.729> \\Delta H_{MLP} = 0.578 > \\Delta H_{Linear} = 0.485$. This demonstrates that the C-Abstractor achieves higher entropy reduction after training. Thank you for these insightful comments. We will include this discussion with a comprehensive comparison of the C-abstractor in our revision.\n\n> Q2: Additionally, the drop in entropy cannot be claimed as being simply due to the removal of \"redundant information\"; potentially useful visual information may have been lost. Certainly, compression is occurring, and does not detract from the central claims of the paper. \n\nA2: Thank you for the comments. As briefly discussed in the conclusion, we agree with the reviewer’s perspective that over compression may potentially lead to useful visual information loss. There exists a tradeoff between redundancy removal and semantic loss when the compression level continues to increase. Within a proper compression range, more compression improves performance by removing redundant information and enhancing alignment. However, over-compression causes semantic information loss and performance degradation. To empirically validate this tradeoff, we conducted a controlled ablation where we varied the patch loss weight $\\beta$ in the joint objective: $L = L_{caption} + \\beta L_{patch}$. Larger $\\beta$ encourages stronger patch-level alignment and induces more compression. As shown in the table, performance first improves then drops as $\\Delta H$ increases, which validates our conclusion.\n\n| $\\beta$ setting                                 | $\\beta = 0$ (original LLaVA) | $\\beta = 2$ | $\\beta = 5$| $\\beta = 10$ | $\\beta = \\infty$(Patch Loss Only) |\n|------------------------------------------|-------------------------|-----------|------------------|------------|--------------------------|\n| $\\Delta H$                                        | 0.5789                  | 0.6039    | 0.6646           | 0.6726     | 0.6972                   |\n| Performance (Avg)                        | 58.54                   | 59.78     | **60.56**        | 59.72      | 57.77                    |\n\nIn the revision, we will clarify this in section 3 and Lines 281-282 and include these experiments in the appendix. \n\n> Q3: The choice of linearly increasing beta (hypeparameter for the align loss weight) is not justified or ablated in the paper.\n\nA3: Thanks for the suggestions. Due to the limited space, we omit the ablation study in our main paper.  We will now present our ablation studies on the hyperparameter $\\beta $ in two aspects: \n\n1. Linear increasing vs. fixed schedule: The linear schedule gradually increases $\\beta $ to impose patch-level alignment progressively. This design choice stabilizes early training and prevents premature over-compression. As shown in the table, when comparing fixed $\\beta=5 $ versus linearly increasing $\\beta $ from $0$ to $5$, the linear schedule showed better performance. Therefore, we adopted the linear increasing schedule for all subsequent experiments.\n\n2. Optimal final $\\beta $ value: Since $\\beta $ in the objective function balances global-level and patch-level alignment, a larger $\\beta$ emphasizes patch-alignment loss over caption loss. We examined the impact of $\\beta \\in \\{0, 2, 5, 10\\}$, where the baseline LLaVA-1.5 corresponds to $\\beta = 0$. As shown in the table, when $\\beta $ is too small ($\\beta = 0$ or $2$), patch-level alignment remains insufficient, leading to suboptimal performance. Conversely, when $\\beta $ is too large ($\\beta = 10$), the model overly focuses on local regions, compromising its ability to establish comprehensive correspondence between the entire image and sentences. This imbalance degrades global-level alignment and ultimately harms overall performance.\n\n| Setting                                 | GQA   | Science QA | VizWiz VQA | OKVQA | Avg   |\n|----------------------------------------|-------|------------|------------|-------|-------|\n| fixed ($\\beta = 0$)                           | 61.93 | 66.80      | 50.00      | 53.42 | 58.04 |\n| fixed ($\\beta = 5$)                           | 62.16 | 68.07      | **54.86**  | 56.51 | 60.40 |\n| linear increasing ($\\beta \\in [0,2]$)           | 62.64 | 68.12      | 50.84      | 57.52 | 59.78 |\n| linear increasing ($\\beta \\in [0,10]$)          | 62.64 | 68.57      | 50.82      | 56.85 | 59.72 |\n| linear increasing ($\\beta \\in [0,5]$) (Optimal)    | **62.99** | **68.67** | 52.29      | **58.29** | **60.56** |\n\nWe will include the complete ablation study in the appendix.\n\n> Q4: It would be helpful to see ablations with other LLM families (beyond LLAMA), as well as other projector types (such as transformer projectors).\n\nA4: Thank you for the suggestion. We have ablations on different LLMs (Vicuna and LLAMA) and projector types (linear, MLP, C-Abstractor). Due to time constraints, we were not able to include additional ablations, which we plan to explore in future work. Specifically, we aim to incorporate LLMs with different parameter sizes or architectures such as Phi-3 (3.8B) and Mixtral-8x7B (MoE), and to investigate transformer-based projectors, including Q-Former[1] and D-Abstractor[2].\n\n[1] Li, J., Li, D., Savarese, S., & Hoi, S. (2023, July). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning (pp. 19730-19742). PMLR.\n\n[2] Cha, J., Kang, W., Mun, J., & Roh, B. (2024). Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13817-13827).\n\n> Q5: Averaging embeddings of tokenized words seems noisy, since the word parts likely carry very different semantic meaning compared to the whole. It would be interesting to understand the effect on the align loss and multi-semantic alignment when it comes to word parts vs whole words.\n\nA5: Thank you for pointing this out. Ideally, we would prefer that each word is tokenized into a single unit by the LLM tokenizer. However, in practice, many words—especially longer or less frequent ones—are split into multiple sub-tokens. Let a visual patch embedding be $\\mathbf{v}$ and a word *w* be decomposed by the LLM’s tokenizer into $n$ sub‑tokens with embeddings $ [ \\mathbf{w}\\_{1},  \\mathbf{w}\\_{2}, …  \\mathbf{w}\\_{n} ]$. In that case, we aim to maximize total similarity between the patch and **all** sub‑tokens: $\\max \\sum_{i = 1}^{n} \\left <\\mathbf{v}, \\mathbf{w}\\_i \\right> $, which is equivalent to $ \\max \\left <\\mathbf{v}, \\frac{1}{n}\\sum_{i} \\mathbf{w}\\_i \\right >$, i.e., the alignment between the vision token and the average embedding of sub-tokens.  \n\nEmpirically, Trager et al. [3] show that vision-language models exhibit approximately linear semantic structure, where meanings of complex concepts can be composed by summing component representations. This also supports our choice of averaging sub-token embeddings. That being said, as briefly discussed in the conclusion, we acknowledge that averaging embeddings over sub-tokens might not be the optimal choice and could potentially dilute semantic meaning. If the reviewer has alternative suggestions, we’d be happy to test and incorporate them. \n\n[3] Trager, M., Perera, P., Zancato, L., Achille, A., Bhatia, P., & Soatto, S. (2023). Linear spaces of meanings: compositional structures in vision-language models. In *Proceedings of the IEEE/CVF International Conference on Computer Vision* (pp. 15395-15404).\n\n> Q6: The choice of measuring Von Neumann entropy is deliberate, and Line 141-142 does not clarify why this particular metric was chosen, as opposed to Shannon's entropy, to help quantify the degree of \"compression\" in visual tokens after projection.\n\nA6: Thank you for the notes. As reviewer suggested, we will provide stronger motivation for our choice of Von Neumann entropy. Von Neumann entropy is calculated from the eigenvalues of the normalized covariance matrix and remains invariant under orthogonal or unitary transformations. This means it captures the intrinsic \"spread\" of information regardless of the coordinate system used. In contrast, Shannon entropy is defined for discrete distributions or requires discretization in continuous cases. It lacks basis independence—changing the coordinate axes or binning scheme can significantly alter the measured value, creating unwanted variability when comparing embeddings before and after projection. We will clarify this in the final revision."}, {"review_id": "3xxa4Q6qeO", "reviewer": "Reviewer_7Lxw", "summary": "This work first analyses the alignment between visual and LLM feature spaces by evaluating several custom metrics before and after projectors in MLLMs. The results reveal that existing MLLMs, such as LLaVA, achieve only weak and coarse alignment, lacking patch-level alignment after pretraining. To address this issue, we introduce a Patch-Aligned Dataset and design a patch-alignment loss to enhance fine-grained alignment. Experiments demonstrate that the proposed patch-alignment training not only improves fine-grained alignment but also boosts the MLLM’s performance after instruction tuning.", "questions": "See weakness section.", "limitations": "Limitations already discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1)Visual-LLM feature alignment is an underexplored topic in MLLMs. This work provides a quantitative analysis of both coarse and fine-grained alignment using custom metrics.\n2)The proposed patch-aligned training is simple yet effective, as validated by experiments.\n3)The method improves not only fine-grained alignment but also downstream MLLM performance after instruction tuning.\n\nWeaknesses:\n1)The idea of enforcing token-level alignment between visual features and text embeddings is quite similar to SEA, even thought the implantation is different.\n2)Since the patch-aligned training uses additional labeled data, the performance gains after SFT—especially on fine-grained recognition tasks—are somewhat expected. A deeper discussion of this limitation is needed.", "was_revised": true, "final_justification": "I have read author's rebuttal other reviewer's comments. Most of my concerns have been addressed and I decide to keep my original score.", "rebuttal": "We sincerely thank the reviewer for recognizing the novelty of our work, the effectiveness of our proposed patch-aligned training, and its impact on both alignment and downstream MLLM performance. Below, we respond to the remaining concerns individually and clarify the corresponding points.\n\n> Q1: The idea of enforcing token-level alignment between visual features and text embeddings is quite similar to SEA, even though the implantation is different. \n\nA1: We thank the reviewer for pointing out the related work. While SEA shares the motivation of improving alignment between visual tokens and language embeddings beyond caption-level supervision, our work differs in several significant ways:\n\n1. Fine-grained alignment measurement: SEA focuses primarily on introducing a new training loss, whereas our work provides a systematic analysis of existing projectors using novel metrics, including *Von Neumann Entropy* (quantifying information compression) and $\\mathrm{Align}(\\mathbf{V}, \\mathbf{W})$ (measuring fine-grained alignment). These analyses reveal the structural limitations of current projectors and provide actionable insights.\n\n2. Multi-semantic alignment: SEA treats each patch as corresponding to a single label. In contrast, we introduce a Matching Pursuit-based decomposition to extract multiple aligned tokens per patch, capturing fine-grained multi-token semantics—crucial for complex understanding.\n\n3. Vocabulary coverage: SEA relies on a fixed, predefined vocabulary, which inherently limits semantic granularity. We use RAM to recognize a broad and dynamic vocabulary of visual concepts, allowing our patch supervision to adapt to diverse real-world entities and contexts.\n\n4. Loss formulation and performance: While SEA adopts a contrastive loss with handcrafted negative sampling, we use a simpler cosine loss that eliminates the need for sampling strategies. Since SEA doesn’t release their code, we implement the contrastive loss on our own. Our ablation studies show that our approach outperforms contrastive loss in downstream tasks:\n| Dataset     | Original LLaVA | SEA (contrastive loss) | Ours (cos loss) |\n|-------------|----------------|-------------------------|------------------|\n| GQA         | 61.93          | 60.95                   | **62.99**        |\n| Science QA  | 66.8           | 67.58                   | **68.67**        |\n| OKVQA       | 53.42          | 51.59                   | **58.29**        |\n\nWe will incorporate these key distinctions and comparisons with SEA in the revision.\n\n> Q2: Since the patch-aligned training uses additional labeled data, the performance gains after SFT—especially on fine-grained recognition tasks—are somewhat expected. A deeper discussion of this limitation is needed.\n\nA2: Thank you for your thoughtful comment. We agree with the reviewer that our method introduces additional supervision through bounding boxes. However, the core contribution of our work isn't adding strong supervision, but rather designing a lightweight mechanism to improve fine-grained visual–language alignment in multimodal large language models (MLLMs).\n\nOur performance gains don't come from new module design or additional training data, but from addressing the misalignment in current MLLMs trained solely with caption-level objectives. Compared to the baseline LLaVA model, our method:\n\n- maintains exactly the same architecture,\n- uses the same pretraining dataset, with labels generated by our automatic detection pipeline (no additional datasets or human-annotated labels),\n- adds only a minimal patch-level cosine loss to guide alignment.\n\nThis demonstrates that our approach doesn't rely on rich supervisory information, but rather serves as a method that directly addresses a fundamental alignment gap in the existing training pipeline.\n\nImportantly, the supervision signal is automatic, scalable, and practical. It requires no human annotation. Our pipeline generates annotations directly from existing image data. Using standard hardware (A5000 GPUs), the pipeline annotates 558K samples in ~13 hours, demonstrating real-world scalability.\n\nAs suggested, we will incorporate the above discussion in the revision."}]

Author rebuttal:

Meta-review: The paper proposes a detailed analysis of visual-text alignment in MLLMs and introduces a lightweight patch-aligned training objective that improves fine-grained alignment and downstream performance. Across thorough reviews, the consensus is generally positive, with all reviewers ultimately recommending acceptance (1 Strong Accept, 1 Accept, 2 Borderline Accepts).

While some concerns were raised—especially regarding the novelty over existing methods like SEA and the use of automatically generated annotations—these were convincingly addressed in the rebuttal, with additional evaluations and ablations provided.

Given the significance of the fine-grained alignment problem in MLLMs and the empirical rigor demonstrated, I recommend acceptance. The paper provides meaningful insights and a practical solution likely to benefit the community.
Decision: Accept (poster)