Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment
Authors: Xin Xiao
Bohong Wu
Jiacong Wang
Chunyuan Li
zhou Xun
Haoyuan Guo
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision Language Model
Contrastive Alignment
Selective Token Re-weighting
Primary area: generative_models

Abstract: Existing image-text modality alignment in Vision Language Models (VLMs) treats each text token equally in an autoregressive manner. Despite being simple and effective, this method results in sub-optimal cross-modal alignment by over-emphasizing the text tokens that are less correlated with or even contradictory with the input images. In this paper, we advocate for distinct contributions for each text token based on its visual correlation. Specifically, we present by contrasting image inputs, the difference in prediction logits on each text token provides strong guidance of visual correlation. We therefore introduce Contrastive Alignment (CAL), a simple yet effective re-weighting strategy that prioritizes training visually correlated tokens. Our experimental results demonstrate that CAL consistently improves different types of VLMs across different resolutions and model sizes on various benchmark datasets. Importantly, our method incurs minimal additional computational overhead, rendering it highly efficient compared to alternative data scaling strategies.

Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment
Xin 1.20 Bohong Wu Jiacong Wang2.3, Chunyuan Li3, Xun Zhou', Haoyuan Guo?
School of Computer Science, Wuhan University "ByteDance Inc.
Schooo of Artificial Intelligence, University of Chinese Academy of Sciences
Abstract
Existing image-text modality aligmment in Vision Language Models (VLMs) treats each text token equally in an autoregressive manner. Despite being simple and effec- tive, this method results sub-optimal cross-modal alignment by over-emphasizing the text tokens that are less correlated with or even contradictory with the input images. In this paper, we advocate for assigning distinct contributions for each text token based its visual correlation. Specifically, inputs, the difference in prediction logits on each text token provides strong guid- ance of visual correlation. We therefore introduce Contrastive ALignment (CAL), a simple yet effective re- weighting strategy that prioritizes visually correlated tokens. Our experimental results demonstrate that CAL consistently improves different types of VLMs across different resolutions and model sizes on various bench- marks. Importantly, our method incurs minimal additional computational overhead, rendering it highly efficient compared to alternative data scaling strategies.
present by contrasting image
Introduction
Recent advancements in Large Language Models (LLMs) [1-4] have opened up new avenues in multimodal understanding, giving rise to novel model category known Vision Language Models (VLMs) [5-8]. Many recent studies on VLMs are centered around enhancing their capabilities, either through increasing the resolution of input images [9-12] or incorporating higher-quality training datasets [13-16]. Additionally, research efforts have been directed towards exploring variations of vision models, such as replacing or augmenting vision encoders beyond CLIP [17, 18], including approaches like SigLIP [19]. DINO [20, 21] or ConvNeXt [22, 23]. The integration of these techniques has spurred the development VLMs, continually enhancing their performance across various benchmarks including visual question answering [24-28], image captioning [29, 30], and
visual grounding [31,3]]. 32].
Despite these advancements, whether the current alignment strategy on existing image-text datasets performs satisfactorily is often less studied. Existing alignment strategies simply treat all text tokens equally in an auto-regressive manner. Although such a method has been proven to be simple and effective, many text tokens exhibit limited relevance to the visual inputs, which contribute little to image-text modality alignment. Figure la presents a sample drawn from the ShareGPT4V [16] dataset, where a large proportion of text tokens including unique, context presents little visual correlation. Treating these text tokens in equal weights results in ineffective training and can introduce negative effects by prioritizing more on fitting the distribution of these visually irrelevant tokens, rather than the image-text modality alignment.
* "qqual contribution
i Work done during internship in ByteDance
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ShareGPT4V
DeaVall
Q: Please describe the image in detail.
45
47 47.67%
image features unique The backdrop of the image provides context to the loca- -tion of the truffic lights tree. It situated on a busy street with a red truck and black car captured in motion.
$7,33%
Samples contain visually contradictory tokens (cleun samples)
visually correlated tokens
visually contradictory tokens
Samples contain visually contradictory tokens (noisy samples)
(a) Tokens correlate differently with the image.
(b) Human evaluations.
Figure la is sample drawn from the ShareGPT4V dataset, which contains text tokens that are even contradictory with the given image. Figure lb further presents our human evaluation results on the proportion of noisy samples that contain contradictory tokens.
Figure 1:
Moreover, there also exists a proportion of tokens that contradicts visual conditions, which is inevitable in model generated datasets [7, 16, 13, 15, 14), presented in Figure 1a. In particular, we conduct human evaluations on the broadly used GPT-assisted datasets including ShareGPT4V and the detail caption subset of LLLa VA-Inssttt in Figure 1b by sampling 100 samples from each dataset. We score each sample by 0 and based on whether there exist text tokens that contradict with visual input, and the score is averaged by three annotators. We found approximately half of the sampled datasets contain visually contradictory tokens in both datasets. Imitating the text distribution on these contradictory tokens further harms the image-text modality alignment. Consequently, recent evaluations on existing VLMs [33, 34, 5, 35, 6, 8] present the shortcomings of current alignment strategy from various aspects, including ballucination [36, 37, 33] and responding without depending on visual conditions 38].
Fortunately, inspired by recent training-free visual contrastive decoding researches [36, 39], we present that the visual correlation can directly indicated by contrasting input image conditions. In particular, we investigate the change in prediction logits of text tokens with or without the image input and observe strong relevance between the logit change of each text token and its visual correlation. We therefore propose Contrastive ALignment (CAL), which is surprisingly simple re- re-weighting strategy to prioritize the training of text tokens that are highly correlated with the input image to enhance image- text modality alignment. Experiments have shown that our proposed method can improve leading VLMs of different kinds including LLaVA- --..VA--.55L... 1.5/LL VA- [7, 6, 10]. MiniGemini(MGMMMMM HD [11], across different resolution and model size various types of benchmarks including visual question answering, captioning and grounding. Especially, CAL on L.a VA-VV--
[10] can bring an impressive performance of 1.7 ANLS on VQA Doe [26], 3.4 relaxed accuracy on VQA Chart [25],
2.2 CIDEr [40] on COCO [30] and 6.3 CIDEr on TextCaps [29]. 0.6/0.7 IoU on validation/test set of method introduces little computational overhead, with one auxiliary gradient-free forward operation in each training step. The lightweight feature while the impressive performance of CAL brings current VLMs a new stage, highlighting the importance of a delicate image-text modality alignment strategy design. We further conduct extensive qualitative analysis for CAL and present the improved ability of OCR recognition and maageccaptioning.
RefCOCOg [32]. Moreover,
In summary, our contributions are listed as follows.
We present that by contrasting image inputs, existing VLMs able distinguish visually correlated tokens both visually irrelevant and visually contradictory ones.
We propose CAL, contrastive image-text alignment method via token re-ieighting, which is lightweight and effective. CAL requires little additional training cost and no additional inference cost.
Experiments show that our CAL can consistently improve VLMMs of different kinds, across different resolutions and sizes in various types of benchmarks.

---

dhop
Image
T
Text
imrgee
Dic
The
grad
Contrusting logits
tratfic
lights
rree
grad
Trikste weights busved visual corrolation
CAL
MELE
E30t0on
Weight distribation
Loss for takens
(a) Ao heat map.
(b) Training procedures of CAL
Figure 2: Overview of CAL. Figure 2a presents sample drawn from the ShareGPT4V dataset. We calculate the logit difference w/ or w/o image inputs and plot the heat map on partial text tokens. Figure 2b presents the training procedure of CAL., which re-weights the importance of label tokens based on the contrasting logits.
Contrastive Alignment
In this section, we describe the detailed design of CAL. First of all, review the existing image-text modality alignment method and provide the notations in Section 2.1. Secondly, we show the token discrepancy in cross-modal datasets can be inferred via contrasting image inputs in Section 2.2. Finally, we present a detailed description of our proposed CAL in Section 2.3.
2.1 Preliminary and Notations
Preliminary Most existing VLMs adopt a two-stage strategy to align pre-trained image features with text embeddings Large Language Models, i.e., PreTraining (PT) stage that uses quantitative while noisy datasets for rough alignment, and an Instruction-Tunne Tuning (TT) stage that uses high-quality datasets enhance the alignment. Both stages treat all tokens equally an auto-regressive generation manner, i.e., the Maximum Likelihood Estimation (MLE) objective.
Notations In this paper, denote the alignment dataset D consisting paired image-text samples D {(1, ), (12, (?,1")... (1", }"),, and denote the logit computation function as f(0) where the weight of VLMs. For the 1th sample (I', T") in the training dataset, where T' consists of sequence of tokens 7 [1A.1 fa.2 (')), denote the prediction logit distribution without input
I as o, and prediction logit distribution with input I as ō, depicted in the following equation:
à (TT ت.ر fo(I", (T.<))
(1)
(.) or to represent the prediction logit in the tth sample at token Ij. As a result, the MLE loss objective for the given ith sample at token Lj is written the following equation by treating the weight e of each token equally, where is set to 1:
where T < represents all previous tokens before position j in the th sample. We further use (ty)
sijj
[t,]
Ch.. = c- -....oooooee Logsoftmax
(2)
2.2 Tokens Differ in Image-text Modality alignment
In this section, we present the necessity of token re-weighting in Section 2.2.1, and show that the re-weighting guidance could be naturally inferred by contrasting image inputs in Section 2.2.2.
2.2.1
Discrepancy exists text tokens
Our proposed method begins with the discrepancy in the training label tokens. For image-text modality alignment, the training labels are usually natural texts, where not all text tokens have

---

strong correlation with the image inputs. Moreover, due to the existence of model generated datasets, there also exist noisy text tokens that harm the alignment process, which depicted in Figure 1.
Based on the relevance between corresponding input images, text tokens can be naturally divided into three kinds. (1) Visually correlated tokens, which contain clear visual concepts and movements depicted in the image. (2) Visually irrelevant tokens, which contain either irrelevant to the image inputs or could be easily inferred by previous text tokens. (3) Visually contradictory tokens, which contain hallucinated objects, especially in the model generated datasets.
2.2.2 Visually correlation can be inferred by contrasting image inputs.
Inspired by VCD [36] and IBD [33], which both enhance the generation by contrasting image inputs, we further present that the contrastive method can also provide clear guidance for visually correlation
on each token.
We take the prediction logit of each label token under two cireumstances, i.e., with or without the
image inputs, which we denote as 6.. and 4.j [ts] [t5]" Then denote Ao(6 [6,] 16.1 [6,]
d.j as the difference
between the predictions logits across two circumstances, we plot Ao on each token in Figure 2a to [y]
visualize the effect of image conditions.
From Figure 2a, Ao [t] performs impressively in distinguishing text tokens three kinds. By Ao [ty]" the visually correlated tokens the traffic lights tree. busy street, red truck are specially high-lighted while other tokens, especially the visually contradictory tokens a black car are light-colored.
In summary, the discrepancy in the label tokens motivates us to apply the token-wise dynamics on loss enhance the image-text modality alignment. By contrasting the image inputs, the difference in is aids with clear guidance for visually correlation of each text token. [c]]
the prediction logits A
2.3 Contrastive Alignment (CAL)
In this section, present the details of proposed CAL. CAL proposed to re-assign the contribution of each token based on their visually correlation weights. The overview of our method is shown in Figure 2b and the detailed algorithm is depicted in Algorithm 1.
Following the previous section, CAL first dy- namically computes the visually correlation weight Ao of each token tj by contrast- ing the image conditions. To avoid the effects of extreme values, we additionally introduce post-proc post-processing methods including clamping and average pooling. We clamp Alogit by setting the upper bound to 3 and the lower bound to a(a >= 0). By setting a to the extreme value CAL neglects the visually irrelevant tokens and visually contradictory to- kens. By setting 3 the extreme value +ox, CAL tolerates the circumstances where some visually correlated tokens occupy most of the importance weights in all label tokens:
Algorithm Detail Procedure of Litl
Input: 1th Image I", tth Text T = t1, tus VLM fo
I: Compute contrastive logit.
ot fo(T".<)). .' fo (I Thiss)
2: Compute Alogit.
Ao 1,51
#5]
3: Compute weights by post- post-processing Ao
whth = pealinggwwddampp..oo s (Ao ))
4: Compute CAL loss by re-weighting tokens.
w wh log fo (o(o 53
Elmn whit
(t,)
Output: cul
wi's clampa,
(Aop))
ig (ty)
We further introduce average pooling with window size of W smooth the visually correlation weights of each token, where we denote as:
who, poolingw (wadinggwww"t)))
The final loss objective of CAL is the weighted average of the original MLE objective based on w and is defined as:
cily
1
wiis log fo (o)
(5)
Ek will

---

Table Visual Question Answering benchmarks of CAL on leading methods including LLa LLa VA VA-NeXT',, and MGM/MGM- Our results are marked with VQA Text is evaluated without OCR tokens, Abbreviations: OCRB. BeCcBBench,, MMS. (MMStar), MMT. (MMTTBench).
VQA
Method
LLM
OCRB
SQA'
Win/All
Doc
Chart
Text
Low resolation setting
MGM
Gemma-2B
335
39.8
23.4
48.1
60.6
25.5
43.4
6/7
MGM+CAL
Gemma-28
360
44.8
27.0
51.8
64.0
25.4
45.4
MGM
Vicuna-7B
431
57.7
43.2
61.1
69.9
32.8
50.3
6/7
MGM+CAL
Vicunu-7B
443
58.0
42.8
63.0
70.4
35.5
51.4
MGM
Vicuna 13B
48.8
62.6
5/7
MGM+CAL
Vicuna 13B
466
61.6
48.0
63.8
71.9
33.7
51.9
LLaVA-1.5
Vicuna-71 7B
315
28.5
17.5
47.6
68.2
32.4
48.6
5/7
LLaVA-I,5+CAL
Vicuna-7B
328
30.6
17.5
47.5
68.7
32,9
48.8
1LaVA-1.5
Vicuna-1 13B
341
31.1
18.3
49.0
72.1
33.5
51.1
7/7
LLaVA-1.5+CAL
Vicuna-133 13B
347
32.6
18.4
49.6
72.7
35.7
51.2
High resolution setting
MGM-HD
Vicuna-7B
477
72.0
49.3
65.5
68.4
31.0
47.9
6/7
MGM-HD+CAL
Vicuna-7B
503
73.4
49.6
67.1
69.2
30.1
50.5
MGM-HD
Vicuna-13B
502
77.7
55.8
67.2
73.5
34.2
50.9
6/7
MGM-HD+CAL
Vicuna 13B
535
78.0
57.2
68.8
73.1
38.5
51.4
LLaVA-NeXT
Vicuna-7B
542
75.1
62.2
64.2
68.5
33.7
49.5
7/7
LLaVA NeXT=CAL
Vicuna 7B
561
77.3
64.3
65.0
70.1
35.5
50.7
LLaVA-NeXT
Vicuna-130 13B
553
78.4
63.8
67.0
71.8
37.5
50.4
6/7
NeXT-NA
Vicuna 13B
574
80.1
67.2
67.1
71.5
38.1
52.4
3 Experiments
3.1 Experimental Setup
Implementation Details In this paper, we verify proposed CAL two leading model structures: LLaVA-I.SLLLVAA-NeTTI [6, 10] and
Miini-Gemmiii-DD [11]. LLaVA-1.5 uses CLIP- pretrained ViT-L as the visual encoder. For resolution scaling, LLaVe-XXXT employs a simple while adaptive image cropping strategy, encodes each image and concatenates them in one single sequence. Mini-Gemini (MGM) further introduces LAION-pretrained ConvNeXI-L [22, 41] for high-resolution refinement. For original paper as is public available. where the learning rate for the PT stage is set to 1 and the IT stage is set to 2e for both Vicuna-7B and Vicuna-13B For LLaVA-NeXT. where only the evaluation code made public, we reproduce L.L VAA.eT- with the same learning rate as MGM, and set the learning rate of VTT to 1/10 of the base learning rate (our reproduction presents on-par performance with the original paper/blog. We present comparison of our reproduction results with those of the original papers Appendix A.I). We also set the lower bound and upper bound / in Equation (3) and 5 respectively, and we set in Equation (4) to for all experiments. We use 16 A100 for experiments, except for 8 GPUs in JLavA-1.5/Gemmas 2B and 32 GPUs in MGM-HD-13B.
we follow the same setting as the
Datasets For experiments on LLaVA-NeXT [10], since the detailed composition of training datasets not publicly available,
slightly different training dataset combination, where we include the
mixture of LLaVAGe5t [6]. VQA Doc [26]. VQA Chart [25] and the ShareGPT4V [16]. For experiments
of LLaVA-115 [6] and MGM/MGM-HH
the same dataset combination with original paper. The training datasets include LLa CC3M [42], ALLaVA [14], ShareGPT4V [16],
[1]],
LAION-GPT-4V [43], LIMA [44], OpenAssistant2 [45], VQA Doc [26]. VQA Chart [25]. DVQA [46]
and A12D [47]. Finally, we report results on widely-adopted VLM benchmarks, including VQA [4thhwtth providing OCR tokens), VQA Doc [26], VQA Chart
[25]. OCR-Bench [49].
MMT [28], MMStar [27]. SQA [50], COCO Caption [30], TextCaps [29]. and RefCOCOg [32] in
our main experiments which observe significant improvement in majority settings, and additional
benchmarks MME [24], POPE [37], SEED-I [51], VQA Text [48h(with OCR tokens given), with comparable performance Appendix A.2.
'ee reproduce the results on LLaVA-Nec -eXT on slightly different training data combination, since the instruction- instruction-tuning data not made public.
5

---

Table 2: Image captioning and visual grounding benchmarks on LLaVA-1.5, LLaVA-NeXT, and MGM/MGM-HII HDD Our results are marked with
COCO Caption
TextCaps
Refcocogool
Refcoeigess
Low resolution serting
MGM
Gemma-2B
8.0
14.1
46.2
46.7
4/4
MGM+CAL
Gemmu-2B
29.7
25.1
50.9
51.1
MGM
Vicuna-7B
18.2
31.4
66.4
4/4
MGM+CAL
Vicuna-7B
21.8
33.6
MGM
Vicuna-13B
17.6
27.1
73.2
73.5
4/4
MGM+CAL
Vicuna- 13B
18.6
32.1
73.7
74.1
LLaVA-155
Vicuna 7B
111.1
100.7
72.2
72.2
4/4
LLaVA-LS5+CAL
Vicuna-7B
111.8
106.7
73.5
72.6
LLaVA-1.5
Vicuna- 13B
115.9
102.4
74.6
75.2
3/4
L.5VA-
Vicuna-1 13B
119.4
105.6
75.3
75.2
High resolution setting
MGM-HD
Vicuna-7B
33.4
42.4
70.2
70.6
3/4
MGM-HD+CG HD&CAL
Vicuna-7B
31.3
52.6
70.9
71.5
MGM-HD
Vicuna-
22.2
42.1
76.8
3/4
MGM-HD&CAL
Vicuna-133 13B
30.0
51.6
77.2
77.1
LLaVA-NeXT
Vicuna-7B
112.0
115.0
77.6
77.5
4/4
LLVV-NNXT+CAM
Vicuna 7B
114.7
124.7
78.4
78.1
HLaVA-NeX
118.5
4/4
LLaVA-NeXT+CAL
Vicuna- 13B
120.6
124.4
80.4
3.2 Main Results
CAL effectively improves various VLMs in Visual Question Answering scenarios Table presents the performance of VLMs on Visual Question Answering. Our CAL consistently improves the performance on most of the understanding benchmarks with impressive margin, on different resolution settings on both MGM/MGM-HH and LLaVA-1.5/LLLVA--N vision model architectures. Especially on the high resolution setting, our CAL presents impressive performance improvement on OCR centric benchmarks. CAL can bring improvement in 7 out of 8 benchmarks on LLaVA-NeXT-7B MGM-HD-7B/1333 and 6 out of 8 benchmarks on LLaVA- NeXT-13B, Especially, on LLaVA-NeXT-I32 CAL improves VQA Doc by 1.7 ANLS, VQA Chart by 3.4 relaxed accuracy, and OCR-Bench by 21 points.
which have different
Effectiveness of CAL consistent on other benchmarks including Image captioning and Ground- ing More promisingly, found the improvement of CAL is not limited to Visual Question An- swering benchmarks, but even more impressive on image-caption and visual grounding benchmarks. Table 2 presents the performance of VLMs COCO Caption, TextCaps and both validation and test set of RefCOCOg. Especially, CAL improves LLaVA-NeXTI
2.1 CIDEr score on COCO Caption benchmark, 6.2 CIDEr score TextCaps, 0.6/0.7 70.7 IoU on the validation/test set of RefCOCOg.
3.3 Ablation Studies
Stages of Conducting CAL within VLM Training CAL can be integrated into both the PreTraining (PT) stage and the Instruction Tuning (IT) stage in existing VLMs. In this section, investigate which stage benefits the most from CAL in Table 3. Integrating CAL into the IT stage contributes most of the improvement in all listed benchmarks, and CAL in the PT stage further enhances the performance with notable improvement on MMT-Bench and OCR-Bench.
CAL relieve the effect of noisy labels in the training data CAL impressively reduce the effect of the contradictory label tokens training data. To distinctly presents the denoising ability of CAL pollute the original training data with ratio of 10 %, 20 % and 30 % by exchanging the training images and their labels, i.e., we select (I', 7 and (13, T with probability of p and replace them with (1 T and (1), T*.. We plot the performance difference when the noise rate varied on four
Note that the caption score of MGM/MGM- HD is significantly lower than LLaVA-NeXT. The training split of MGM does not include the training set for both COCO Caption and TextCaps, where we provide the zero-shot performance for MGM in all settings these benchmarks.

---

Acturas
118
555
016
550
bascline CAL
baseline
beseline
baseline
CAL
Rate
Noise
(a) COCO Caption
VQA Dre
(c) VQA
(d) OCR-Bench
Figure 3: Accuracy difference when different noise ratios applied. The performance of the baseline is marked with red lines, and CAL is marked with green lines. The dashed line represents the asymptote.
Prpetheri
4.9
031
Paperke
0.0
F0.0
En15
g015
0.0
0.00
0.99
005
035
0.55
000
20
9.0
16
13
20
10
3.0
(a) LLaVA-1.5-7B
(b) LLa V3-1.5-33
(c) LALLVA-NEETTT
(d) LLAVA-NEXT--ER
Figure 4: Ao distribution for LLaVA models on 100 random sampled cases.
Table 3: Performance difference when CAL is
Table 4: Performance difference when applying
applied at different training stages.
different weights [a, 3| for clamping.
PT I
"QA
TextCaps
MMT.
OCRB.
a
"QA
VQu
OCRB.
Refcncog.al
78.4
118.1
50.4
553
+00
79.4
66.6
563
79.4
[0.5]
80.4
66.3
570
79.7
78.5
116.7
50.9
554
1,00
79.1
66.8
78.5
79.9
124.8
51.5
564
[1.5]
80.1
67.2
80.1
124.4
52.4
Buseline
78.4
63.8
553
79.8
different benchmarks, including COCO caption, VQA Doc VQA Text and OCR-Bench. From Figure the performance of baseline drops significantly when the noise rate is increased, while CAL performs more robust to noise.
flyper-parameed for Ja, 3] in elamping We further conduct ablation study and study the effect the hyperparameters in our clamping operation.
First, we plot the Ao distribution on LLaVA series in Figure 4. Tokens whose Ao is lower than 5 nearly occupy approximately 90% of the total label sequences. Therefore, we select 5 as the upper bound B. To prevent the context dependent tokens being totally neglected, we set the lower bound 1. We then extend both lower bound and upper bound to extreme values, i.e., and +oc. The results are shown in Table 4. (1) Both setting the lower bound to 0 and setting the upper bound to +oo causes performance degradation. (2) Even when setting the lower bound to CAL still achieves better performance compared with the original baseline, indicating that totally neglecting the context dependent/visually contradictory tokens while focusing on the visually dependent ones still brings steady improvement.
We further provide ablations on the image contrasting conditions in Appendix A.3, where compare CAL when different kinds of augmentation strategy is used.
Complementary Ablations
3.4 Analysis
CAL presents better capability in OCR recognition and capturing details We provide qualitative analysis in Table to analyze the improved caption ability CAL. the table, CAL presents much better ability in OCR recognition by accurately distinguishing bewildering OCR cases, e.g., CAL accurately tells the difference between hand-writter consciousness and construction, world and would. Such ability contributes to the superior performance of CAL on the TextCaps benchmark.
7

---

Meanwhile, CAL also presents better ability in capturing compared with baseline, CAL captures the material details cd cover, and the numerical details by telling two men on horses from a mun on a horse. The capability of CAL to capture intricate details leads to sustained enhancements the COCO caption benchmark. CAL. empowers the model to identify more accurate elements within images, including objects like a frolley and the number 8, which might otherwise be incorrectly recognized or overlooked.
details. For instance,
Complementary Analysis We first provide statistics for computational overhead in Appendix A.6. And we further provide more qualitative analysis on studying the quality of image-text segeettxtttee modality alignment. The attention map scores are visualized in Appendix B. and the aligned image features are visualized in Appendix B.2.
4 Related Work
Vision Language Models LLMs [1, 52, 2, 53,4, 4] have made significant strides in Natural Language Processing (NLP) tasks, including text generation and guestion-answering, paving the way for VLMs that integrate vision ability with LLMs. In the realm of visual language learning, CLIP [17, 54] has set milestone by employing extensive image-text pair contrastive learning to achieve multimodal alignment. Recently, numerous VLMs [5. 7, 8, 55, 34, 56-59, 35, 60, 61] have leveraged the robust capabilities of LLMs for cross modal understanding and generation tasks. Models like BLIP-2 [5] and MiniGPT-4 [8] have improved cross- cross-modal aligmment through comprehensive image-text pair pre-training. LLaVA [7] has further advanced its comprehension of complex prompts via refined instruction fine-tuning. Additionally, recent research [9-11] has incorporated higher resolution input images and longer sequences enhance VLMs' understanding capabilities. Mini-Gemini (MGM) [11] introduces a LAION-pretrained ConyNeXt-L [22] for high-resolution refinement.
Image-text Modality Alignment Image-text modality alignment has long been regarded as the core problem in modal understanding and generation tasks. Traditional image-text alignment strategies include both contrastive learning across different modalities and generative learning that train text tokens in an autoregressive manner [17, 62-64]. The combination of both techniques also proven to be effective in the early era of VLMs, where BLIP [5] proposes a multi-stage alignment strategy, with contrastive learning in early alignment and generative learning in the latter stage. However, recent researches [7, 10], contrastive learning is discarded for being redundant in image-text modality alignment of VLMs, and researchers propose to enhance the cross-modke alignment through dataset scaling and image resolution scaling [53, 9, 65, 10]. Despite being simple in application, the existing generative alignment method simply treats each text token with equal importance, resulting in sub-optimal alignment performance.
More recently, due to the great success in Reinforcement Learning (RL) in the alignment of LLMs, many recent works [66, 37, 67] have also integrated Reinforcement Learning methods align existing VLMs with human preference. However, these RL-based methods require high-quality human-labeled pair-wise data and focus more on aligning with human preference rather than modality alignment.
Training-free Contrastive Decoding Recently, many researchers have proposed to improve the
generation quality via contrastive decoding [68-70, 36, 33J. Especially, in the field of LLMs, CID [71]
utilizes contrastive decoding on paired text inputs for model de-biasing. Such method is also proven
to be effective in enhancing the reasoning ability of LLMs in various aspects [72-74]. Similar investigations have also been taking in VLMs. Recently, both VCD [36] and IBD [33] propose to
enhance the generation of VLMs by contrasting the prediction logits between the original visual input
and the perturbed ones. CRG [39] further proposes to improve the grounding ability without training via contrasting differently masked images. However, these training- free methods require additional computation during the decoding stage, making it highly ineffective for application.
5 Limitation
Despite the superiority of CAL in various model structures, resolution settings and model scales on various benchmarks, limitations still exist in our proposed method. First of all, there lacks clear and
8

---

Table 5: OCR and captions generation comparison based on LLaVA-NEXT-13B model.
OCR cases
ConSHuetuv
Would
Question:
What written in the image?
What is the number the image?
Baseline:
The word "consciousness" is written in
The word "world" is written in the image.
the image.
CAL:
The word "construction" is written in the image.
The word "would" is written in the image.
6O
Question:
What is the Mosman Manly exit going to?
How many items purchased from Amazon?
Baseline:
The mosman manly exit is going to mos- man.
There are 2.43 million items purchased from amazon. 902k.
CAL:: The mosman manly exit going to
chatswood epping.
Short caption cases
Question:
Provide one-sentence caption for the provided image.
Provide a one-sentence caption for the pro- vided image. A book is open to a page with a picture of a man horse.
Bascline:
A red background with sun and birds and the words Sibelius Symphonies No 2
& 5.
CAL:
A ed cover for Sibelius Symphonies Nos
A book is open to a page with picture of two men on horses.
&
TIA GeR
Question:
Provide one-sentence caption for the provided image.
Provide one-sentence caption for the pro- vided image.
Baseline:
A silver car parked in front of fence and a bus. A silver car is parked behind fence in front of trolley.
A person riding horse with cart attached to it.
CAL::
A horse pulling cart with the number 8 on

---

quantitative discrepancy between the three kinds of label tokens. More discussion of the importance weights guidance can be further investigated future works.
The selection of lower bounds and upper bounds in Equation (3) are empirically decided based on the frequency of the prediction logits, which could be extended more adaptive settings in further explorations. Nevertheless, the simple while broadly effective nature of CAL indicates the importance of delicate image-text modality alignment strategy for leading VLM structures.
Conclusion
In this paper, we investigate the in-completeness of current image-text alignment in leading VLMs by treating all text tokens with equal weights. We present by contrasting input images, the difference in the prediction logits for each token naturally reveals their visual correlation. We therefore propose a token re-weighting strategy that prioritize the training of highly visually correlated tokens. Our proposed strategy, CAL is simple while impressively effective, achieving consistent performance gain across various benchmarks including visual question answering, image-capiooiig and grounding.
Our work raises a question about the potential optimal learning strategy of image-text modality align- ment. Both the imperfectness of training data and over concentration on visually irrelvant/visually contradictory tokens hinder the performance of current VLMs. We hope the proposed CAL can inspire more investigation better alignment strategy to enhance the capabilities of existing VLMs.
10

---

References
[1] OpenAL. Chatgpt. http:///nna...oo/olll////catta/a 2023. 1,8
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar0iv 12922.13999 13971, 2023.
[3] Susan Zhang, Stephen Roller, Naman Goyal. Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint ar 2205 01066, 2022.
[4] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401. 044888 2024.
[5] Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-imin image pre-training with frozen image encoders and large language models, In International conference on machine learning, pages 19742. PMLR, 2023. 1, 2, 8
[6] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. In NewrlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. 2,5
[7] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2, 8
[8] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing vision- language understanding with advanced large language models. In The Twelfth International Conference on Learning Reprexentations, 2023. 1, 2,
[9] Xiaoyi Dong. Pan Zhang. Yuhang Zang. Yuhang Cao, Bin Wang. Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining et al. Internlm xcomposer2-4hie 4khd:: A pioneering large vision- language model handling resolutions from 336 pixels to 4k hd. arXiv preprint 2rXiv:2404.06512, 2024. 1,
[10] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https: v1. ww..iitttttt io/ blog/2024- 30--001100 1011-30-1laaa-- next/. 2,
[11] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 2024. 5,8
[12] Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. hoplug-dccow 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXvv:2403. 12895, 2024. 1
[13] Weiyun Wang, Min Shi, Qingyun Li, Wenhai Wang, Zhenhang Huang, Linjie Xing, Zhe Chen, Hao Li, Xizhou Zhu, Zhiguo Cao, et al. The all-sceing project: Towards panoptic visual recognition and understanding of the open world. In The Twelfth International Conference Learning Reprexentations, 2023. 1,2
[14] Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesizedd data for lite vision-language model. arXiv preprint arXiv:2402. 1684, 2024. 2,5
[15] Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang, Qingyun Li, Lewei Lu, Xizhou Zhu, et al. The all-seeing project v2: Towards general relation comprehension of the open world. arXiv preprint 40X: 19474, 2024. 2
[16] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui Hc, Jiaqi Wang. Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.1279
2023. 1, 2,5
[17] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021. 1,8

---

[18] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint 0rXiv:2303.15388 2023.
[19] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975 11986, 2023.
[20] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[21] Maxime Oquab, Timothéc Darcet, Théo Moutakanni, Huy V Vo, Mare Szafranicc, Vasil Khalidov, Pierre Fernandez. Daniel HAZIZA, Francisco Massa, Alaaekdin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions Machine Learning Research, 2023.
[22] Zhuang Liu, Hanzi Mao, Chao-Yuan Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1197--11988 2022. 1,5,8
[23] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu. Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autocncoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16133-16142, 2023.
[24] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. 1,5
[25] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqu: A henchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.1022 10244, 2022.
2,
[26] Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Document collection visual question answering. In Document Analysis and RecognätionICDARR 2021: 16th International Conference, Lausanne, Switzerland, September 5 5-10, 2021, Proceedings, Part II 16, pages 778 -922. Springer, 2021. 2, 5
[27] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et al. Are preprint arXiv:2403.2000 20330, 2024. 5,5
the right way for evaluating large vision-languad models? arXiv
[28] Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404. 16006, 2024. 1,5
[29] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part II 16, pages 742-758. Springer, 2020. 1,2,5
[30] Xinlei Chen, Hao Fang. Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.003325 2015. 2,
[31] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to- region-to-phrase correspondences for richer mmage-to-sentent models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[32] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan I. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1 1--00, 2016. 1, 2.
[33] Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision language models via miased decoding, arXiv preprint arXiv:2402. 18476, 2024. 2, 4,
[34] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint 24118704..44188 2023. 2,
12

---

[35] Anas Awadalla, Irena Gao, Josh Gardner. Jack Hessel, Yusuf Hanafy. Wanrong Zhu, Kalyani Marathe. Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive -anguage models. arXiv preprint N.XXv::3388.11999 2023. 2, 8
[36] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing- Mitigating object hallucinations in large vision-language models through visual contrastive decoding. arXiv preprint arXiv:2311.16922 2023. 2, 4,8 8
[37] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large lasoon-anggaaee models, arXiv preprint arXiv:2305. 10355, 2023. 2, 5,
[38] Renrui Zhang, Dongzii Jiang. Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403. 14624, 2024." 2
[39] David Wan, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal. Contrastive region guidance: Improving grounding in vision-language models without training. arXiv preprint arXiv:2403.02332 023325, 2024. 2, 8
[40] Ramakrishna Vedantam, C Lawrence Zitncck and Devi Parikh. Cider: Consensus-basse image description evaluation. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[41] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman. Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-Sb: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 0055278-252944 2022. 5
[42] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed. image alt- dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 2556 2565, 2018. 5
[43] LAION eV. Laion/gpt4v-dataset- datasets at hugging face. URL.tttps:/mugggagaacccomm cottaaaas /aion/ pattvvdattsst.....
[44] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024. 5
[45] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations- democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 2024. 5
[46] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In CVPR, 2018. 5
[47] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. diagram is worth a dozen images. In ECCV, 2016. 5
[48] Amanprect Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlci Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8322 2019. 5
[49] Yuliang Liu, Zhang Li, Hongliang Li. Wenwen Yu, Mingxin Huang, Dezhi Peng. Mingyu Liu, Mingrui Chen, Chunyuan Li. Lianwen Jin, et al. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.00 008999 2023. 5
[50] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507 -2211, 2022. 5
[51] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal lims with generative comprchension. arXiv preprint 2337 2023. 5
[52] Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 23030000 2023. 8
13

---

[53] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrae, Jiahui Yu, Radu Soricut, Johan Sehalkwyk, Andrew M Dai. Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint urXiv:2312.11800 2023.
[54] Yunhao Gou, Tom Ko, Hansi Yang, James Kwok, Yu Zhang, and Mingxuan Wang. Leveraging per image-token consisteney for liiounlanguuag pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19155-19164, 2023. 8
[55] Zhiliang Peng. Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023. 8
[56] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.088 054005555555.... 2023, 8
[57] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.094778 2023.
[58] Wenliang Dai, Junnan Li, Dongxu Li. Anthony Meng Huat Tiong, Jungi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung. and Steven Hoi. Instructblip: Towards reneral-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[59] Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199 2023. 8
[60] Pan Zhang, Xiaoyi Dong Bin Wang. Yuhang Cao. Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Internim -oommm-e A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309. 12, 2023. 8
[61] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internim-xcone xcoopoosr:: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint (iXi:2240111422) 2024.
[62] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXi::2205.0191 2022.
[63] Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Benjamin Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew M Dai, Zhifeng Chen, Claire Cui, et al. Mammut: A simple architecture for joint learning for multimodal tasks. Transactions on Machine Learning Research, 2023.
[64] Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang. Jian Jiao, Nan Duan, et al. Rh Not all tokens are what you need. arXiv preprint 27X1::2404.07665, 2024. 8
[65] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding. localization, text reading, and beyond. 2023.
[66] Tianyu Yu, Yuan Yao, Haoye Zhang. Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai- Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milmss via behavior alignment from fine-grained correctional human feedhack. arXiv preprint 2rXXi:23/22000444 2023.
[67] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang. et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 145.2 2023. 8
[68] Xiang Lisa Li. Ari Holtzman, Daniel Fried, Percy Liang. Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint 222: 1509,, 2022. 8
[69] Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based hias in nlp. Transactions of the Association for Computational Linguistics, 9: 1408-1424, 2021.
[70] Mian Zhang. Lifeng Jin Linfeng Song, Haitao Mi, Wenliang Chen, and Dong Yu. Safeconv: Explaining and correcting conversational unsafe behavior. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22 22-35,20 2023. 8
14

---

[71] Gal Yona, Or Honovich, Itay Laish, and Roee Aharoni. Surfacing biases in large language models using contrastive input decoding. arXiv preprini arXiv:3305.07375, 2023.
[72] Sean O'Brien and Mike Lewis. Contrastive decoding improves reasoning in large language models, arXiv preprint arXiv:2309.08 09117, 2023. 8
[73] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality large language models. arXiv preprint arXiv:2309.0888.
2023.
[74] Alisa Liu, Xiaochuang Han, Yizhong Wang. Yulia Tsvetkov, Yejin Choi, and Noah A Smith. Tuning language models by proxy. arXiv preprint arXiv:2401.0855 088565. 2024.
15

---

Outline
A. Complementary Experimental Analysis
A.1. Comparison of our reproduced results with the official results in Table 6.
A.2 Results of CAL on other Visual Question Answering benchmarks in Table
A.3, Ablations of CAL when different contrasting method is used in Table 8 and Table 9.
A.4. Ablation for w/ w/o average pooling in Table 10.
A.5, Ablation for pre-trained model in Table 11.
A.6. Training time comparision w/ or w/o CAL on LLaVA-NeXT-7B-13B in Table 12.
B. Complementary Qualitative Analysis
B.1. Visualization of attention map between image tokens and text tokens in Figure 5.
B.2. Visualization of image-text modality alignment quality in Figure 6.
16

---

A
Complementary Experimental Analysis
A.1 Ablations for Comparison with Official Results
Table 6: Comparisons between our reproduced results and official results. denotes providing OCR tokens for evaluation. Official results are marked with f-
VQA
COCO
Method
LLM
SQA!
MME
POPE
SEED-1
TextCaps
Text"
Caption
Low resolation setting
MGM
Gemma- 2B
39.8
23.4
48.1
56.2
60.6
1335
85.7
64.6
8.0
14.1
MGM
Gemma-2B
56.2
1341
MGM
Vicuna-7B
57.7
43.2
61.1
65.7
69.9
1551
85.7
68.2
18.2
31.4
MGM
Vicuna-P7
MGM
Vieuna-1 13B
61.7
48.8
66.1
69.1
1550
86.2
69.5
17.6
MGM
Vicuna- 13B
65.9
1565
LLaVA-155
Vicuna-22 7B
28.5
17.5
47.6
58.2
68.2
1468
86.2
66.6
111.1
100.7
LLaVA-1.5
Vieuna-7B 7B
28.1
18.2
46.1
69.5
1508
85.9
66.2
110.4
98.1
LLaVAL5
Vicuna-1 13B
31.1
60.6
72.1
1574
85.7
68.6
115.9
102.4
LLaVaL5
Vicuna-1
72.9
1523
85.9
68.2
115.5
Hligh resolation aetting
MGM-HD
Vicuna-PP 7B
72.0
49.3
65.5
68.2
68.4
1521
85.7
65.7
33.4
42,4
MGM-HD
Vicuna-7B
68.4
1546
MGM-HD
Vicuna-13B
55.8
67.2
73.5
1633
86.3
70.1
22.2
42.1
MGM-HD
1597
LLaVA-NeXT
Vicuna-7B
75.1
62.2
64.2
67.1
68.5
1514
86.8
69.6
112.0
115.0
LLaVA-NeXT
Vicuna-78 7B
74.4
54.8
64,9
70.2
1519
86.4
70.2
99.9
71.8
LEXVA-NeXT
Vicuna- 13B
78.4
63.8
67.0
70.0
71.8
1574
87.2
71.8
118.5
118.2
LexV
Vieuna- 13B
77.5
62.2
66.9
73.6
1575
86.3
71.9
102.0
67.4
We compare our reproduced results with those reported in the paper or from reliable sources in Table The results for Mini-Gemini taken from their paper, while the results for LLaVA are sourced from this sheet provided by the LLaVA authors. We can observe that our reproduced results are comparable the official ones.
A.2 Results on other Visual Question Answering benchmarks.
Table 7: Results additional Visual Question Answering benchmarks. denotes providing OCR
tokens for VQA Text Our results are marked with
Method
LLM
MME
POPE
SEED-1
VQA Text
Low resolation setting
MGM
Gemma-2D
1335
MGM+CAL
Gemma-2B
1333
85.8
65.0
58.1
MGM
Vicuna-7B
1551
85.7
68.2
65.7
MGM+CAL
Vicuna-7B
1509
69.0
66.7
MGM
Vicuna-1 3B
1550
86.2
69.5
66.
MGM+CAL
Vicuna-135 13B
1506
86.4
69.4
67.1
LLaVA-155
Vicuna-7P 7B
1468
86.2
66.6
58.2
LLAVA-1.5-CAL
Vicuna-7B 7
1500
85.5
66.5
58.6
LLAVALS
Vicuna- 13
1574
85.7
68.6
60.6
LLaVaLS+CAL
Vicuna- 13B
1567
85.8
69.3
60.1
High resolation setting
MGM-HD
Vicuna-7B
1521
85.7
65.7
68.2
MGM-HD+CAL.
Vicuna-7B
1531
68.6
69.5
MGM-HD
Vicuna-
1633
86.3
70.1
69.8
MGM-HD+CAL
Vieuna- 3B
1582
86.4
70.5
70.3
LLLVV-NNXT
Vicuna-78
1514
69.6
67.1
LLaVA-NeXTICAL
Vieuna-7B 7B
1490
86.7
70.7
67.1
LLLVA-NeXT
Vicuna-
1574
87.2
71.8
70.0
LLaVA-NeXTICAL
Vicuna-
1606
87.2
71.6
68.9
17

---

Table 8; Ablations for contrasting image conditions on Visual Question Answering benchmarks using
VQA
Method
Mask
MMS.
MMT.
SQA'
OCRB.
Text
Text"
Doc
Chart
MaVA-NeXT
37.5
50.4
71.8
78.4
63.8
553
LLaVA-NeXT+CAL
38.1
52.4
71.5
67.1
68.9
80.1
67.2
574
LLaVA-NeXT+CAL
0.5
37,9
51.9
70.5
67.1
69.5
79.0
66.0
555
HaVA-NeXT+CA mack
0.7
38.9
51.3
72.1
66.1
65.5
LLaVA-NeXT+CG CAL
0.9
38.3
52.0
71.8
67.0
69.4
78.9
65.9
580
LLVA-NNCT+CALL geasssinn
37,9
51.7
71.2
66.8
69.6
78.7
65.7
554
MaVA-NeXT+CAI geassian
38.5
52.0
70.2
67.3
564
Table 9: Ablations for image contrasting conditions on image captioning and visual grounding benchmarks using
Method
Mask
COCO Caption
TextCaps
Refcocog.al
Refcocogrest
LLaVA-NeXT
118.2
79.8
79.6
CLVA--NeX++CAL
120.6
124.4
80.4
80.3
LLaVA-NeXT+CC CAN
0.5
114.6
122.9
80.9
80.8
LLaVA-NeXT+ CALmaak
0.7
116.0
122.9
80.6
81.7
LLaVA-NeXT+ CA Allasuo
0.9
117.7
121.5
80.6
81.1
LLaVA-NeXT+ Caganniin
117.3
118.9
80.3
80.1
LLaVA-NeXT+CAl Colounrio
116.8
123.4
79.2
79.8
In the primary sections of this manuscript, CAL achieves better performance benchmarks of OCR centric VQA, image-captioning, visual grounding, and other image-dependent benchmarks. this section, extend our analysis by presenting additional results on MME, POPE and SEED-I and Text VQA Text (with OCR token given). CAL presents comparable performance on these benchmarks, without clear distinction performance in different training settings. With steady improvement on OCR centric or image dependent VQA benchmarks, CAL also presents satisfying quality on the remaining VQA tasks.
A.3 Ablations for Image Contrasting Conditions
In this section, we further conduct additional analysis the image contrasting conditions. Except from masking the whole image token sequence, which is the method we use our paper, we further study two kinds of contrasting conditions, including random patch masking and Gaussian blurring. For random patch masking, we cut images into 20 10 grids, with 10 pieces in the width and 20 pieces in the height. Therefore each images is cut into 200 pieces. We then randomly select 50 %, 70 % and 90 of the patches to mask, and contrast them with the original image. For Gaussian blurring, set the kernel to and 10 to simulate the extreme blurring circumstances. By using 1, we adopt light blurring which only brings significant effect on OCR centric images. By using 10, all images will be heavily affected.
Table 8 presents results on Visual Question Answering benchmarks of different image contrasting conditions on LLa LLaVV-NeeT-1133 and Table 9 further presents results image-caption benchmarks and visual grounding benchmarks. From these two tables, either random patch masking or adding Gaussian blurring performs sub-optimal in different benchmarks. By CAL mask. except for the signifi- cant improvement on the test split of RefCOCOg, performance especially on the OCR-centric benchmarks. By CAL Gaussians although the performance on OCR centric benchmarks (VQA Doc VQA Chart DCR-Bench) is less affected, it performs less effective in nearly all benchmarks than CAL
CAL Gaussian drops significantly,
A.4 Ablations for Average Pooling
We fix the window size (W) to 3 in Equation (4) to smooth the weight distribution. An additional experiment in Table 10, where we removed the average pooling step, shows slightly inferior perfor- mance, supporting our chosen.
18

---

Table 10: Comparison of benchmarks with and without Average Pooling.
ChartVQA
DocVQA
SQA'
COCO Caption
TextCaps
OCRB,
Refcocogual
w/ AvgPool
67.2
80.1
71.5
120.6
124.4
574
80.4
w/o AvgPool
66.3
79.5
72.4
116.7
123.8
581
79.5
A.5
Ablations for Pre-trained Model
Table 11: Comparison of pre-trained models for CAL on
Model
Pretrain ChartVQA DocVQA
SQA' COCO Caption TextCaps
OCRB.
Refcocog.ol
Baseline
Original 63.8
78.4
71.8
118.5
118.2
553
79.8
CAL
Original 67.2
80.1
71.5
120.6
124.4
574
80.4
CAL
Baseline 66.3
79.5
72.4
116.7
123.8
581
One assumption of CAL is that after training simple projector only, the VLM is capable of distinguishing visually correlated tokens. In the first phase VLM training, it is common practice to freeze the ViT and the LLM, and only train projector to align their features. In the second phase, we finetune the model using high-quality data to enable it to answer image-related questions. To validate this assumption, finetune CAL models using types of pre-trained versions: the original pre-trained model (only train the projector) and one with the fully trained baseline model (after the finetuning phase). As shown in Table 11, compare the performance of these two versions and found no significant differences in performance.
with
A.6 Computational Overhead.
Table 12: Training time comparison with and without CAL in the instruction-tuningg stage on LLaVA- NeXT.
Pretrain
LLaVA-NeXT-78
LLaVA-NeXT-13B
Baseline
15.6h
27.4h
Baseline CAL
19.14
33.7h
Each iteration of our proposed method requires forwarding text tokens twice effectively enhance image-text modality alignment. Table 12 presents the training time for the instruct-tuning stage on LLaVA-NeXT, with each experiment conducted on 16 100 GPUs. CAL introduces approximately 20% computational overhead on both -LaVA-NeeX-773 and LLavA-NeXT-I33 Our implementation currently uses an attention mask to neutralize the effect of the image tokens, meaning that the image tokens are still forwarded twice. This additional burden could be further reduced by completely removing image tokens.
19

---

B Complementary Qualitative Analysis
B.1 Attention Map Visualization
In this section, we visualize the attention maps generated by both the baseline model and our proposed model. These visualizations provide insight into how each model focuses on different parts of the input image. The attention weights are calculated by accumulating the attention score between image tokens and text tokens across all layers. As shown in Figure 5, model w/ CAL provides clearer attention maps with less noisy points in the background area, indicating more precise focus on relevant regions.
(a) a white van on highway from Monks & Crane.
(b) $8.49.
Figure 5: Comparison of attention maps with and without CAL on LLaVA-NeXT-I3B, The left side of each sub-figure shows LLaVA-NeXT-13B without CAL, while the right side shows LLaVA-NeXT- 13B with CAL
B.2 Image-text Modality Alignment Visualization
In this section, we visualize the image-text modality alignment by retrieving the nearest text words each image patch feature from the LLM vocabulary. We plot the results in Figure 6. CAL brings better image-text modality by accurately retrieving the OCR information from the language vocabulary. E.g., LLaVA1.5-7B. CAL correctly retrieves Prices, expert, O, shadow, which is exactly the OCR information the original image. We do select LLaVA-NeXT in this section due the presence of token overlaps from sub-crops.
SHADOWRUN IN WA GHOST CARTELS
Sales Price per Unit
Varlable Cost per Unit
Product
Junior Adult Expert
50
25 50
110
purch
Baseline
0
dark studie share dark
ew
wouldn
svg studio dark White
took took late
Price
purch
shadowshadow dark Moscom
won wy
unit
unit
shadswwhhddowpersee persoon
wy
xter
Figure 6: Visualization of image-text modality alignment for each image patch. We filtered out some nonsensical patches for better visualization.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims fully reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the specific parameters used in the experiment and the information needed to reproduce it.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
We will release our code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips.com wttpccccpppccccceeececceccccccct httpa//gggiieee/eeeuuuuiiiiiii
/CodeSubmie ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We show the training and test details in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We follow the same guideline in our experiment as those previous works.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://meet :/tppr/:///uuipp//// cc Cwwublic/Ethissuite deluiddeeiing
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of our work performed for our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
All the creators or original owners of assets used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected. All datasets we use are from internet open source datasets under CC-BY licenses and are cited properly.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwoee com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "NnCXLexxPM", "reviewer": "Reviewer_DwVB", "summary": "This study proposes a reweighting strategy, namely contrastive alignment, to enhance model learning on visually correlated tokens. Specifically, the authors divided the tokens into three sub-groups, virtually related, non-related, and contradictory, and assigned different weights to each group. The weight is calculated based on the predicted contrastive logit (w/o visuals) and further post-processed by clamping and average pooling. This study is evaluated on vision question answering, image caption, and grounding tasks using two foundation model structures, LLaVA and Mini-Gemini.", "questions": "See above weakness.", "limitations": "As I can see, this paper has no potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-- The paper is well-written. \n\n-- Deploying a reweighting strategy on different types of tokens seems feasible for performance improvement.\n\n-- The division of token groups is straightforward yet reasonable.", "weaknesses": "-- This study does not compare with any baseline models within the topic of resisting noisy tokens, such as [*] and [**].\n\n-- In [*], a study with similar aims was presented. In particular, they also studied the image-token consistency. Thus, detailed baseline comparisons, including overall performance and token consistency evaluation/comparison, are expected. \n\n-- The authors claim that the proposed strategy is simple yet efficient. However, it may rely upon several strong assumptions, such as high capability requirements of the pre-trained model. These assumptions are not presented, discussed, and validated in the paper. \n\n-- Weight distribution on different token sub-groups is not presented. This would contribute to the validation of weighting.\n\n[*] Gou, Yunhao, et al. \"Leveraging per image-token consistency for vision-language pre-training.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[**] Wu, Cheng-En, et al. \"Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your constructive feedback and highlighting the strengths of our work. We appreciate the detailed insights and suggestions provided. Below, we address the key points raised in your review.\n\n> **W1: Lack of Baseline Comparisons**\n\nWe are happy to include these papers in the related work, however, we would like to clarify that these works might not be suitable for comparison as baseline methods in this paper. First of all, the two methodologies mentioned are primarily **designed for visual-language representation tasks, focusing on classification and retrival tasks**. However, CAL is designed for generation task, so there are no common benchmarks for comparision. Secifically, Gou et al. (2023) use a method involving masked images and a **BERT-like** approach to identify salient tokens, which is not applicable to the **causal decoder-only large language models** used in our method. Wu et al. (2023) focus on enhancing the CLIP pretraining scheme, which is different from the multimodal generation tasks we address. Secondly, our method focuses on ensuring better alignment between visual and textual feature spaces by prioritizing visually correlated tokens. While our method does reduce the influence of noisy tokens, its primary aim is to diminish the impact of irrelevant tokens, which constitute a much larger proportion. Therefore, addressing noise tokens is not its central goal.\n> **W2: Strong Assumptions**\n\nThank you for pointing out this concern regarding the assumptions in our proposed strategy. \n\n1. The basic assumption of CAL is the **high image-text matching capability of Vision Transformer** and a **well-trained Large Language Model**, which are already facts in current structures of Generation VLMs (like LLaVA). The current standard practice in training VLMs primarily aims to map image features to the text embedding space. Before this alignment stage, the CLIP ViT is pre-trained on a billion-level dataset, and the LLM is trained on a trillion-level dataset. They are of great capability to encode images and deal with complex language tasks. In comparison, the data (about 1M samples in LLaVA) used during the VLM training stage mainly serves to align the feature spaces of the two models and to enable the model to answer image-related questions.\n\n2. Another assumption of CAL is that **after training a simple projector only, the VLM is capable of distinguishing visually correlated tokens**. In the first phase of VLM training, it is common practice to freeze the ViT and the LLM, and only train a projector to align their features. In the second phase, we finetune the model using high-quality data to enable it to answer image-related questions. To validate this assumption, **we finetune CAL models using two types of pre-trained versions: one with the original pre-trained model (only train the projector) and one with the fully trained baseline model (after the finetuning phase)**. We then compared the performance of these two versions and found no significant differences in performance. We will include a more detailed discussion of these assumptions and their validation in the revised version of the paper to address this concern explicitly.\n\n| Model | Pretrain | ChartQA | Docvqa | SQA-I | COCO Caption | TextCaps | OCRBench | Refcocog val |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Baseline(LLaVA-Next-13B) | Original Pretrain | 63.8 | 78.4 | 71.8 | 118.5 | 118.2 | 553 | 79.8 |\n| CAL | Original Pretrain | 67.2 | 80.1 | 71.5 | 120.6 | 124.4 | 574 | 80.4 |\n| CAL | Baseline Pretrain | 66.3 | 79.5 | 72.4 | 116.7 | 123.8 | 581 | 80.2 |\n\n\n> **W3: Weight Distribution Not Presented**\n\nThank you for your insightful comment on the weight distribution across different token sub-groups.\n\nWe appreciate your recognition of our token classification approach. However, it is important to note that there is no existing labeled dataset that directly identifies these three types of tokens. Given the impracticality of manually annotating a large volume of data at the token level within a short time frame, we utilized the RLHF-V[1] dataset, **which includes annotations for incorrect words in captions**.\n\nUsing this dataset, we conduct a statistical analysis and find that **the average logits difference (diff) for erroneous tokens is significantly lower than that for correct tokens**. This finding supports our hypothesis about the effectiveness of our weighting mechanism. We will include these details and present the weight distribution analysis in the revised version of the paper to provide a more comprehensive validation of our weighting approach. It is worthy to mention that since this dataset is specially collected, the number of false tokens is much larger than other tokens. (LLaVA-15)\n\n| Token Type | Token Num | avg_diff_value (7B model) | avg_diff_value (13B model) |\n| --- | --- | --- | --- |\n| False token | 344197 | 0.163 | 0.277 |\n| Correct token | 76783 | 2.271 | 2.10 |\n\n---\n\nWe hope that these clarifications and additional details adequately address your concerns. Thank you once again for your valuable feedback.\n\n[1] Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZoiiokIzfM", "reviewer": "Reviewer_tsLt", "summary": "This paper points out that parts of samples in the broadly used datasets contain visually contradictory text tokens. To mitigate the sub-optimal cross-modal alignment in VLMs, this proposed method is to assign distinct contributions for each text token based on its visual correlation.", "questions": "1. It’s unclear how to calculate the prediction logit distribution o without input I for the VLM like CLIP. Is this method not applicable to the pure image-text matching VLM like CLIP? \n2. There’s no l in Equation 4. Does it mean W in Sec. 3.1 implementation details “we set l in Equation 4 to 3 for all experiments”?\n3. For Figure 3, it is better to describe what the dashed line represents as well.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The proposed method is easy to implement and effective. The idea of assigning distinct weights for each token is intuitive. \n2. This paper is well organized and provides clear preliminaries, helping readers understand the method.\n3. CAL achieves consistent and solid performance across different benchmarks.", "weaknesses": "Please see the Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewers for their positive feedback and for highlighting the strengths of our work. We appreciate the detailed insights and suggestions provided. Below, we address the key points raised in your review.\n\n> **Q1: How to Use on CLIP**\n\nThank you for your question regarding the calculation of the prediction logit distribution (o) without input (I) for Vision-Language Models (VLMs) like CLIP.\n\nFor models like CLIP, we can assess the image-text relation by evaluating the change in CLIP score between the original and the noise-added images. This approach aligns with the essence of our method, which is focused on contrastive learning.\n\n> **Q2: There’s No 'l' in Equation 4. Does It Mean 'W' in Sec. 3.1?**\n\nThank you for pointing out this issue. We acknowledge the typo in our manuscript and will correct it in the revised version of the paper.\n\n> **Q3: Dashed Line in Figure 3**\n\nThe dashed line represents the asymptote. We will update the figure description in the revised version of the paper to clearly indicate this and prevent any further confusion.\n\n---\n\nWe hope that these clarifications and additional details adequately address your concerns. Thank you once again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0BEzQ9Qima", "reviewer": "Reviewer_ewhK", "summary": "This paper introduces Contrastive Alignment (CAL), a straightforward yet effective re-weighting strategy designed to improve multimodal alignment. Specifically, the authors propose contrasting image inputs and calculating the differences in prediction logits for each text token to determine its training weights. Extensive experiments are conducted to demonstrate the effectiveness of CAL across various tasks, providing a robust validation of the proposed approach.", "questions": "1. Why MGM-HD-13B perform worse that -7B model in caption tasks as shown in Tab-2 in the paper ? \n2. The visually correlation weights in CAL are smoothed using average pooling with a window size of W. How is W determined? Is this parameter set consistently across all datasets, or does it require adjustment for each training run to optimize performance?\n3. The paper mentions that CAL adds minimal computational overhead compared to other data scaling strategies. However, could the authors provide more quantitative examples or an intuitive comparison? It seems that processing very long texts could significantly increase the overhead.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The Contrastive Alignment (CAL) proposed in this paper is an efficient method for multimodal alignment, particularly beneficial when training data is heavily contaminated with noise. The dynamic adjustment of training weights represents a lightweight yet innovative solution to this problem.\n2. The experimental section of this paper is thorough. It includes extensive quantitative analysis and demonstrates the performance of CAL across a broad range of scenarios. These results not only allow for a deep understanding of the CAL method but also provide valuable insights into its practical applications.", "weaknesses": "1. While CAL is innovative for aligning multimodal data, its robustness and comprehensiveness in complex task settings, such as multimodal question answering (VQA), appear limited. The method ensures correct alignment but may inadvertently impair other model capabilities, such as reasoning ability or retention of original world knowledge. This issue is critical in VQA scenarios where data often include elements not directly related to image content, potentially conflicting with it and incorporating extensive external knowledge. The application of CAL in such contexts might suppress training on these critical aspects, introducing uncontrollable biases and worsening model performance. Although CAL shows promise in simpler tasks like captioning and grounding, its performance in VQA tasks is inconsistent, and its effectiveness in addressing hallucination issues, such as in the POPE dataset, is negligible.\n2. CAL’s effectiveness appears contingent upon the pre-existing performance of the underlying multimodal model. Since it calculates visually correlated weights based on the model itself, any hallucinations or errors present in the base model are not only perpetuated but potentially exacerbated by the CAL method. This limitation restricts CAL’s applicability and caps its performance, as there is no mechanism within CAL to correct these amplified errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewers for their constructive feedback and for recognizing the strengths of our paper. We appreciate the detailed insights and suggestions provided. Below, we address the key points raised in your review.\n\n> **W1: Robustness in Complex Task Settings Like VQA**\n\nActually, aligning with previous works, we have conducted extensive validation on various VQA tasks, as shown in Table 1(Doc, Chart, Text VQA, mmstar, mmtbench and etc). And our methods surpass the baseline in most cases.\n\nAs for the retention of original world knowledge, we present the performance on OKVQA[1], which is used to test the model's ability to answer questions requiring external knowledge beyond what is explicitly shown in the image. The CAL models exceed baseline models in most cases.\n\n| Model  | Baseline | +CAL |\n|-------|-------------|-----------------|\n| LLaVA-15 7B | 56.46       | 58.77           |\n| LLaVA-15 13B| 53.09       | 56.20           |\n| LLaVA-NEXT 7B | 56.08       | 55.26         |\n| LLaVA-NEXT 13B| 55.09       | 55.69         |\n\nWe also evaluate the pure text capability of the baseline and CAL models. Among the tests used, BBH is specifically employed to assess reasoning ability. There are no significant differences compared to the baseline.\n\n| Method         | AGI-eval | BBH(For reasoning)| MMLU | \n| -------------- | :------: | :----: | :---: |\n| LLaVA-15-7B    | 38.14    | 40.78  | 50.56 |\n| LLaVA-15-7B + CAL | 37.65 | 40.33  | 50.63 |\n| LLaVA-15-13B   | 39.55    | 49.27  | 55.11 | \n| LLaVA-15-13B + CAL | 39.52 | 49.67 | 55.14 |\n \n \nAs for hallucination issues, our method prioritizes visually correlated tokens to enhance cross-modal alignment. Nevertheless, since we maintain a minimal weight for all tokens, some contradictory tokens inevitably continue to impact the training process, which can potentially lead to hallucinations as baseline model does. Furthermore, it's important to recognize that LLMs inherently exhibit hallucination issues. Merely enhancing cross-modal alignment, although beneficial, is not a comprehensive solution. This challenge necessitates the adoption of additional techniques tailored to address this specific problem.\n\n> **W2: Dependency on Base Model Performance and Amplification of Existing Errors**\n\nThank you for your observation regarding CAL's dependence on the base model's performance. \n\nWhile it is true that CAL could potentially amplify errors, we would like to clarify that **this scenario is rare**. The amplification of errors only occur when there are both **hallucinations in the training dataset** and **such hallucinations are also captured by CLIP**, which is rare in practice. Moreover, as demonstrated in our **experiments on the POPE** (for hallucination evaluation) dataset, our method achieves comparable performance with the baseline.\n\n| POPE Performance     | Baseline | +CAL  |               | Baseline | +CAL  |\n|----------------------|----------|-------|---------------|----------|-------|\n| MGM 7B               | 85.7     | 87.5  | LLaVA-15 7B   | 86.2     | 85.5  |\n| MGM 13B              | 86.2     | 86.4  | LLaVA-15 13B  | 85.7     | 85.8  |\n| MGM-HD 7B            | 85.7     | 87.0  | LLaVA-NEXT 7B | 86.8     | 86.7  |\n| MGM-HD 13B           | 86.3     | 86.4  | LLaVA-NEXT 13B| 87.2     | 87.2  |\n\n\n> **Q1: Performance Discrepancy Between MGM-HD-13B and MGM-HD-7B**\n\nThe performance discrepancy arose from the initial zero-shot evaluation approach in MGM, where results were largely influenced by output style. We retrained the MGM-HD model using the LLAVA-15 dataset, which includes training data from both COCO and TextCaps. The retrained results showed that the performance trend in MGM aligns with the LLAVA models, resolving the discrepancy.\n\n| Model          | COCOCaps | TextCaps | Model          | COCOCaps | TextCaps |\n| -------------- | -------- | -------- | -------------- | -------- | -------- |\n| MGM 7B         | 107.54   | 108.17   | MGM 7B + CAL   | 110.33   | 113.33   |\n| MGM 13B        | 113.00   | 112.47   | MGM 13B + CAL  | 113.91   | 118.10   |\n\n> **Q2: Smoothing Window Size (W) Determination**\n\nWe fixed the window size (W) to 3 in all our experiments to smooth the weight distribution. An additional experiment where we removed the average pooling step showed slightly inferior performance, supporting our chosen.\n\nWe will include these findings and a discussion on the fixed window size in the revised paper.\n\n| Benchmark      | ChartQA | Docvqa|  SQA-I | COCO Caption| TextCaps| OCRBench | Refcocog val|\n| -------------- | --- | --- | --- | --- | --- | --- | --- |\n| w/ AvgPool     | 67.2 | 80.1 |  71.5 | 120.6 | 124.4 | 574 | 80.4 |\n| w/o AvgPool    | 66.3 | 79.5 |  72.4 | 116.7 | 123.8 | 581 | 79.5|\n\n> **Q3: Computational Overhead of CAL**\n\nAs detailed in Table 10, CAL generally introduces an additional computational cost of approximately 20%. This overhead is primarily due to **an extra forward pass without gradient computation**.\n\nIn exceptionally long texts scenarios, the original forward and backward process also need to deal with long sequences. Therefore, although the absolute overhead increases in such scenarios, the relative increase compared to shorter samples remains consistent.\n\n---\n\n[1] OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XjdoDjjLwt", "reviewer": "Reviewer_mF2S", "summary": "The paper proposes a new re-weighting strategy for training VLMs to improve visual-language alignment. The motivation is to assign higher weight to visually relevant tokens and lower weight to visually irrelevant and visually contradictory tokens. The re-weighting factor relies on the logit difference with and without image inputs for each token. Using the proposed CAL, the paper demonstrates general improvement in VQA and captioning tasks.", "questions": "1. The trend shown in Figure 2a is based on which model? Is this a common phenomenon across existing VLMs?\n\n2. How do you deal with cases where a semantic word has been split into two subparts, each with a different value, such as \"tr\" and \"uck\" in Figure 2a?\n\n3. The CAL seems to rely heavily on the model's logits with and without image input. However, can the model accurately predict changes in logits, as shown in Section 2, during the initial alignment stages?\n\n4. The training requires an additional forward pass of the LLM. Why is the training described as \"lightweight\" in the paper? Doesn't this double the cost?", "limitations": "not applied", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The motivation to dynamically assign weights to different types of tokens is intuitive and well-studied in the language domain. Several studies have attempted to apply this method to vision-language tasks in a zero-shot manner, which incurs high inference costs. This paper proposes training the model with this method instead. The approach is simple and effective, demonstrating general improvements over existing state-of-the-art models across benchmarks. The analysis further validates the effectiveness of the proposed method.", "weaknesses": "1. **Lack of Clarification in Presentation**: Some content is confusing. For example, equation 2 is averaged across the entire sentence with length $l$, yet the description in lines 88-89 refers to the loss objective for the $i^{th}$ sample at token $t_j$, which seems more like the loss at a single token. Additionally, equation 4 lacks clarification; it's unclear how to apply average pooling with a window size from this equation. Some experiments also lack necessary context. For instance, it's unclear which model is evaluated in Figure 2a, and the ablation study in section 3.3 doesn't specify which model is used for evaluation. The main result in section 3.3 doesn't disucuss how to apply CAL into model training, at which stage for example. The annotation is inconsistent; in equation 4, it's $\\tilde{w}^{i,t_j}$, while in equation 5, it's $\\tilde{w}(i.t_k)$\n\n2. **More Experiments Are Required**: The paper presents decent improvements over some baselines, mainly in vision-language tasks. However, since the training recipe has changed, it would be better to also evaluate other tasks, such as language-only tasks, to see how the CAL affects the model's ability in standard language understanding. Additionally, the method seems quite relevant to hallucination tasks, so it would be beneficial to report results on those as well.\n\n3. **Lack of Originality**: The motivation and methodology are already well-studied in language models [1,2], and prior studies have also applied changes to logits based on the presence or absence of images to measure the relevance of visual and language inputs. In this context, the paper simply applies the method during the training stage, instead of the inference stage as in [3]. Therefore, the novelty is somewhat limited.\n\n[1] Jiang S, Wolf T, Monz C, et al. TLDR: token loss dynamic reweighting for reducing repetitive utterance generation[J]. arXiv preprint arXiv:2003.11963, 2020.\n\n[2] Lin Z, Gou Z, Gong Y, et al. Rho-1: Not all tokens are what you need[J]. arXiv preprint arXiv:2404.07965, 2024. \n\n[3] Zhu L, Ji D, Chen T, et al. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding[J]. arXiv preprint arXiv:2402.18476, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewers for their constructive feedback and for recognizing the strengths of our work. We appreciate the detailed insights and suggestions provided. Below, we address the key points raised in your review.\n\n> **W1: Lack of Clarification in Presentation**\n\nWe acknowledge the need for better clarification and consistency in the presentation. We will fix these typos in the revision. We would like to clarify some of the points for your concern.\n\n> **W2: More Experiments Are Required**\n\nThank you for your constructive advice. Following most of the existing works on VLMs, we report the same VLM benchmarks to examine the cross-modal ability for better comparison.\n\nWe also understand and agree with your concern that CAL might affect performance on language-only tasks. To address your concern, **we experimentally verified that CAL did not interfere the performance on language-only tasks**. We further  conducted additional evaluations on language-only tasks including AGI-eval, BBH, and MMLU. These statistics show that CAL achieves comparable results, and even outperforms the baseline on BBH on LLaVA-15-13B. \n\n| Method         | AGI-eval | BBH    | MMLU  | \n| -------------- | :------: | :----: | :---: |\n| LLaVA-15-7B    | 38.14    | 40.78  | 50.56 |\n| LLaVA-15-7B + CAL | 37.65 | 40.33  | 50.63 |\n| LLaVA-15-13B   | 39.55    | 49.27  | 55.11 | \n| LLaVA-15-13B + CAL | 39.52 | 49.67 | 55.14 |\n\nAs for hallucination issues, our method prioritizes visually correlated tokens to enhance cross-modal alignment. Nevertheless, since we maintain a minimal weight for all tokens, some contradictory tokens inevitably continue to impact the training process, which can potentially lead to hallucinations as baseline model does. Furthermore, it's important to recognize that LLMs inherently exhibit hallucination issues. Merely enhancing cross-modal alignment, although beneficial, is not a comprehensive solution. This challenge necessitates the adoption of additional techniques tailored to address this specific problem. Besides, CAL shows comparable performance on POPE with the baseline models, as presented in the Appendix in Table 7.\n\n> **W3: Lack of Originality**\n\nThank you for your feedback. We would like to clarify the originality of both our motivation and methodology.\n\nFor motivation, we are the first to study token discrepancy in the cross-modality alignment stage of existing VLMs, while [1] and [2] focus on language-only scenarios. We are glad to include them in the related works, but we would like to clarify the significant difference from these works. Our work and [2] are concurrent but use different techniques: [2] trains a token-level selective model using high-quality data, while we select tokens based on contrastive image inputs. Unlike [1] and [2], our motivation is to enhance cross-modal alignment by prioritizing visually correlated tokens rather than handling noise/hard tokens.\n\nFor similarity with contrastive-decoding methods, we emphasize that contrastive learning is widely used but applied differently across scenarios. Our innovation lies in token-level training data selection by identifying visually correlated tokens in training labels using contrastive learning and utilizing logit differences for token loss reweighting.\n\n> **Q1: Trend in Figure 2a**\n\nThank you for your insightful question. The trend shown in Figure 2a is a common phenomenon across various VLMs, including LLaVA-15-7B-PT, LLaVA-15-7B-SFT, and MGM-7B-SFT. The high cosine similarity in weight assignment distribution across these models on ShareGPT4V dataset, as shown below, supports this:\n\n| Model              | LLaVA15-7B-SFT | MGM-7B-SFT | \n| ------------------ | :------------: | :--------: |\n| LLaVA15-7B-PT      | 0.9470         | 0.9352     |  \n| LLaVA15-7B-SFT     | -              | 0.9191     |\n\n> **Q2: Handling Split Semantic Words**\n\nThank you for your question. We do not strictly distinguish or design special methods for subwords since CAL is orthogonal to word tokenization. Currently, there is no theoretical basis to decompose sub-word level interactions within a semantic word. We consider the logit change at the token level.\n\n> **Q3: Model's Ability to Predict Logit Changes**\n\nThank you for your feedback. The current standard practice in training VLMs primarily aims to map image features to the text embedding space. Before this alignment stage, the CLIP ViT is pre-trained on a billion-level dataset, and the LLM is trained on a trillion-level dataset. These models are highly capable of encoding images and handling complex language tasks. In the first phase of VLM training, it is common practice to freeze the ViT and the LLM, and only train a projector to align their features. This step can be quite straightforward and quick. The figure in Section 2 comes from the pre-trained LLaVA-NeXT-13B, demonstrating that the ability to distinguish tokens emerges early in the alignment process.\n\n\n> **Q4: Training Cost and \"Lightweight\" Claim**\n\n\nFor the concern of \"lightweight\". **Experimentally**, as discussed in the Appendix (Table 10), CAL adds only a 20% increase in training time, far from \"doubling\". **Theoretically**, the additional forward pass of CAL neither needs gradient back-propagation nor needs storing the intermediate activation values, which is both computation time friendly and memory efficient. Besides, the second forward pass does not need image tokens as the original forward process does, resulting in reduced sequence length and increased speed.\n\n---\n\nWe hope these clarifications and additional details adequately address your concerns. Thank you once again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces an approach for multimodal alignment through contrastive alignment by assigning different weights for different tokens. All reviewers agree that the paper is easy to follow and well-written, and the idea is interesting. The authors' rebuttal addressed misunderstandings in terms of experiments and all reviewers recommends acceptance. The AC thus recommends acceptance and encourages the authors to revise the paper accordingly.
Decision: Accept (poster)