Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment

Authors: xin  xiao
bohong  wu
jiacong  wang
chunyuan  li
xun  zhou
haoyuan  guo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 07:39:47

Abstract: Existing image-text modality alignment in Vision Language Models (VLMs) treats each text token equally in an autoregressive manner. Despite being simple and effective, this method results in sub-optimal cross-modal alignment by over-emphasizing the text tokens that are less correlated with or even contradictory with the input images. In this paper, we advocate for distinct contributions for each text token based on its visual correlation. Specifically, we present by contrasting image inputs, the difference in prediction logits on each text token provides strong guidance of visual correlation. We therefore introduce Contrastive Alignment (CAL), a simple yet effective re-weighting strategy that prioritizes training visually correlated tokens. Our experimental results demonstrate that CAL consistently improves different types of VLMs across different resolutions and model sizes on various benchmark datasets. Importantly, our method incurs minimal additional computational overhead, rendering it highly efficient compared to alternative data scaling strategies.

Main paper: 
 introduction: 
 Recent advancements in Large Language Models (LLMs) [1] [2] [3] [4] have opened up new avenues in multimodal understanding, giving rise to a novel model category known as Vision Language Models (VLMs) [5] [6] [7] [8] . Many recent studies on VLMs are centered around enhancing their capabilities, either through increasing the resolution of input images [9] [10] [11] [12] or incorporating higher-quality training datasets [13] [14] [15] [16] . Additionally, research efforts have been directed towards exploring variations of vision models, such as replacing or augmenting vision encoders beyond CLIP [17, 18] , including approaches like SigLIP [19] , DINO [20, 21] or ConvNeXt [22, 23] . The integration of these techniques has spurred the development of VLMs, continually enhancing their performance across various benchmarks including visual question answering [24] [25] [26] [27] [28] , image captioning [29, 30] , and visual grounding [31, 32] . Despite these advancements, whether the current alignment strategy on existing image-text datasets performs satisfactorily is often less studied. Existing alignment strategies simply treat all text tokens equally in an auto-regressive manner. Although such a method has been proven to be simple and effective, many text tokens exhibit limited relevance to the visual inputs, which contribute little to image-text modality alignment. Figure 1a presents a sample drawn from the ShareGPT4V [16] dataset, where a large proportion of text tokens including unique, context presents little visual correlation.Treating these text tokens in equal weights results in ineffective training and can introduce negative effects by prioritizing more on fitting the distribution of these visually irrelevant tokens, rather than the image-text modality alignment. 2a presents a sample drawn from the ShareGPT4V dataset. We calculate the logit difference w/ or w/o image inputs and plot the heat map on partial text tokens. Figure 2b presents the training procedure of CAL , which re-weights the importance of label tokens based on the contrasting logits. 
contrastive alignment: 
 In this section, we describe the detailed design of CAL. First of all, we review the existing image-text modality alignment method and provide the notations in Section 2.1. Secondly, we show the token discrepancy in cross-modal datasets can be inferred via contrasting image inputs in Section 2.2. Finally, we present a detailed description of our proposed CAL in Section 2.3.Preliminary Most existing VLMs adopt a two-stage strategy to align pre-trained image features with text embeddings in Large Language Models, i.e., a PreTraining (PT) stage that uses quantitative while noisy datasets for rough alignment, and an Instruction-Tuning (IT) stage that uses high-quality datasets to enhance the alignment. Both stages treat all tokens equally in an auto-regressive generation manner, i.e., the Maximum Likelihood Estimation (MLE) objective.Notations In this paper, we denote the alignment dataset D consisting paired image-text samples D = {(I 1 , T 1 ), (I 2 , T 2 ), ..., (I n , T n )}, and denote the logit computation function as f (θ) where θ is the weight of VLMs. For the i th sample (I i , T i ) in the training dataset, where T i consists of a sequence of l tokens T i = [t i,1 , t i,2 , ..., t i,l ], we denote the prediction logit distribution without input I as o, and prediction logit distribution with input I as õ, depicted in the following equation:EQUATIONwhere T i,<j represents all previous tokens before position j in the i th sample. We further use o i,j[tj ] or õi,j[tj ] to represent the prediction logit in the i th sample at token t j . As a result, the MLE loss objective for the given i th sample at token t j is written in the following equation by treating the weight c of each token equally, where c is set to 1:EQUATIONIn this section, we present the necessity of token re-weighting in Section 2.2.1, and show that the re-weighting guidance could be naturally inferred by contrasting image inputs in Section 2.2.2.Our proposed method begins with the discrepancy in the training label tokens. For image-text modality alignment, the training labels are usually natural texts, where not all text tokens have a strong correlation with the image inputs. Moreover, due to the existence of model generated datasets, there also exist noisy text tokens that harm the alignment process, which is depicted in Figure 1 .Based on the relevance between corresponding input images, text tokens can be naturally divided into three kinds. (1) Visually correlated tokens, which contain clear visual concepts and movements depicted in the image. (2) Visually irrelevant tokens, which contain either irrelevant to the image inputs or could be easily inferred by previous text tokens. (3) Visually contradictory tokens, which contain hallucinated objects, especially in the model generated datasets.Visually correlation can be inferred by contrasting image inputs.Inspired by VCD [36] and IBD [33] , which both enhance the generation by contrasting image inputs, we further present that the contrastive method can also provide clear guidance for visually correlation on each token.We take the prediction logit of each label token under two circumstances, i.e., with or without the image inputs, which we denote as õi,j[tj ] and o i,j[tj ] . Then denote ∆o i,j [tj ] = õi,j [tj ] -o i,j[tj ] as the difference between the predictions logits across two circumstances, we plot ∆o i,j[tj ] on each token in Figure 2a to visualize the effect of image conditions. From Figure 2a , ∆o i,j[tj ] performs impressively in distinguishing text tokens of three kinds. By ∆o i,j[tj ] , the visually correlated tokens the traffic lights tree, busy street, red truck are specially high-lighted while other tokens, especially the visually contradictory tokens a black car are light-colored.In summary, the discrepancy in the label tokens motivates us to apply the token-wise dynamics on loss to enhance the image-text modality alignment. By contrasting the image inputs, the difference in the prediction logits ∆o i,j[tj ] aids us with clear guidance for visually correlation of each text token.In this section, we present the details of our proposed CAL. CAL proposed to re-assign the contribution of each token based on their visually correlation weights. The overview of our method is shown in Figure 2b and the detailed algorithm is depicted in Algorithm 1.Algorithm 1 Detail Procedure of L i,tj CAL Input: i th Image I i , i th Text T = t1, t2, ..., tn, VLM f θ1: Compute contrastive logit. o i,j = f θ (T i,<j ), õi,j = f θ (I i , T i,<j ) 2: Compute ∆logit. ∆o i,j [t j ] = õi,j [t j ] -o i,j [t j ]3: Compute weights by post-processing ∆o i,j[t j ] . wi,t j = poolingW (clamp α,β (∆o i,j [t j ])) 4: Compute CAL loss by re-weighting tokens.L i,t j CAL = - 1 l k=1 wi,t k wi,t j • log softmax f θ (õ i,j [t j ] ) Output: L i,t j CALFollowing the previous section, CAL first dynamically computes the visually correlation weight ∆o i,j[tj ] of each token t j by contrasting the image conditions. To avoid the effects of extreme values, we additionally introduce post-processing methods including clamping and average pooling. We clamp ∆logit by setting the upper bound to β and the lower bound to α(α >= 0). By setting α to the extreme value 0, CAL neglects the visually irrelevant tokens and visually contradictory tokens. By setting β to the extreme value +∞, CAL tolerates the circumstances where some visually correlated tokens occupy most of the importance weights in all label tokens:EQUATIONWe further introduce average pooling with a window size of W to smooth the visually correlation weights of each token, where we denote as: wi,tj = pooling W (w i,tj )The final loss objective of CAL is the weighted average of the original MLE objective based on w and is defined as: 3 ExperimentsEQUATION 
experimental setup: 
 Implementation Details In this paper, we verify our proposed CAL on two leading model structures: LLaVA-1.5/LLaVA-NeXT [6, 10] and Mini-Gemini/Mini-Gemini-HD [11] . LLaVA-1.5 uses CLIPpretrained ViT-L as the visual encoder. For resolution scaling, LLaVA-NeXT employs a simple while adaptive image cropping strategy, encodes each image and concatenates them in one single sequence. Mini-Gemini (MGM) further introduces a LAION-pretrained ConvNeXt-L [22, 41] for high-resolution refinement. For MGM/MGM-HD/LLaVA-1.5, we follow the same setting as the original paper as it is public available, where the learning rate for the PT stage is set to 1e -3 and the IT stage is set to 2e -5 for both Vicuna-7B and Vicuna-13B. For LLaVA-NeXT, where only the evaluation code is made public, we reproduce LLaVA-NeXT with the same learning rate as MGM, and set the learning rate of ViT to 1/10 of the base learning rate (our reproduction presents on-par performance with the original paper/blog. We present a comparison of our reproduction results with those of the original papers in Appendix A.1). We also set the lower bound α and upper bound β in Equation ( 3) to 1 and 5 respectively, and we set l in Equation ( 4) to 3 for all experiments. We use 16 A100 for experiments, except for 8 GPUs in LLavA-1.5/Gemma-2B and 32 GPUs in MGM-HD-13B.Datasets For experiments on LLaVA-NeXT [10] , since the detailed composition of training datasets is not publicly available, we use a slightly different training dataset combination, where we include the mixture of LLaVA 665k [6] , VQA Doc [26] , VQA Chart [25] and the ShareGPT4V [16] . For experiments of LLaVA-1.5 [6] and MGM/MGM-HD [11] , we use the same dataset combination with original paper. The training datasets include LLaVA-filtered CC3M [42] , ALLaVA [14] , ShareGPT4V [16] , LAION-GPT-4V [43] , LIMA [44] , OpenAssistant2 [45] , VQA Doc [26] , VQA Chart [25] , DVQA [46] and AI2D [47] . Finally, we report results on widely-adopted VLM benchmarks, including VQA Text [48] (without providing OCR tokens), VQA Doc [26] , VQA Chart [25] , OCR-Bench [49] , MMT [28] , MMStar [27] , SQA I [50], COCO Caption [30] , TextCaps [29] , and RefCOCOg [32] in our main experiments which observe significant improvement in majority settings, and additional benchmarks MME [24] , POPE [37] , SEED-I [51] , VQA Text* [48] (with OCR tokens given), with comparable performance in Appendix A.2. Effectiveness of CAL is consistent on other benchmarks including Image captioning and Grounding More promisingly, we found the improvement of CAL is not limited to Visual Question Answering benchmarks, but even more impressive on image-caption and visual grounding benchmarks. CAL relieve the effect of noisy labels in the training data CAL can impressively reduce the effect of the contradictory label tokens in training data. To distinctly presents the denoising ability of CAL , we pollute the original training data with a ratio of 10 %, 20 % and 30 % by exchanging the training images and their labels, i.e., we select (I i , T i ) and (I j , T j ) with a probability of p and replace them with (I i , T j ) and (I j , T i ). We plot the performance difference when the noise rate is varied on four different benchmarks, including COCO caption, VQA Doc , VQA Text and OCR-Bench. From Figure 3 , the performance of baseline drops significantly when the noise rate is increased, while CAL performs more robust to noise.We further conduct ablation study on α and β to study the effect of the hyperparameters in our clamping operation.First, we plot the ∆o distribution on LLaVA series in Figure 4 . Tokens whose ∆o is lower than 5 nearly occupy approximately 90% of the total label sequences. Therefore, we select 5 as the upper bound β. To prevent the context dependent tokens being totally neglected, we set the lower bound α to 1. We then extend both lower bound and upper bound to extreme values, i.e., 0 and +∞. The results are shown in Table 4 . (1) Both setting the lower bound to 0 and setting the upper bound to +∞ causes performance degradation. (2) Even when setting the lower bound to 0, CAL still achieves better performance compared with the original baseline, indicating that totally neglecting the context dependent/visually contradictory tokens while focusing on the visually dependent ones still brings steady improvement.We further provide ablations on the image contrasting conditions in Appendix A.3, where we compare CAL when different kinds of augmentation strategy is used.CAL presents better capability in OCR recognition and capturing details We provide qualitative analysis in and MiniGPT-4 [8] have improved cross-modal alignment through comprehensive image-text pair pre-training. LLaVA [7] has further advanced its comprehension of complex prompts via refined instruction fine-tuning. Additionally, recent research [9] [10] [11] has incorporated higher resolution input images and longer sequences to enhance VLMs' understanding capabilities. Mini-Gemini (MGM) [11] introduces a LAION-pretrained ConvNeXt-L [22] for high-resolution refinement.Image-text Modality Alignment Image-text modality alignment has long been regarded as the core problem in cross-modal understanding and generation tasks. Traditional image-text alignment strategies include both contrastive learning across different modalities and generative learning that train text tokens in an autoregressive manner [17, [62] [63] [64] . The combination of both techniques is also proven to be effective in the early era of VLMs, where BLIP [5] proposes a multi-stage alignment strategy, with contrastive learning in early alignment and generative learning in the latter stage. However, in recent researches [7, 10] , contrastive learning is discarded for being redundant in image-text modality alignment of VLMs, and researchers propose to enhance the cross-modal alignment through dataset scaling and image resolution scaling [53, 9, 65, 10] . Despite being simple in application, the existing generative alignment method simply treats each text token with equal importance, resulting in sub-optimal alignment performance.More recently, due to the great success in Reinforcement Learning (RL) in the alignment of LLMs, many recent works [66, 37, 67] have also integrated Reinforcement Learning methods to align existing VLMs with human preference. However, these RL-based methods require high-quality human-labeled pair-wise data and focus more on aligning with human preference rather than modality alignment.Training-free Contrastive Decoding Recently, many researchers have proposed to improve the generation quality via contrastive decoding [68-70, 36, 33] . Especially, in the field of LLMs, CID [71] utilizes contrastive decoding on paired text inputs for model de-biasing. Such method is also proven to be effective in enhancing the reasoning ability of LLMs in various aspects [72] [73] [74] . Similar investigations have also been taking in VLMs. Recently, both VCD [36] and IBD [33] propose to enhance the generation of VLMs by contrasting the prediction logits between the original visual input and the perturbed ones. CRG [39] further proposes to improve the grounding ability without training via contrasting differently masked images. However, these training-free methods require additional computation during the decoding stage, making it highly ineffective for application. 
limitation: 
 Despite the superiority of CAL in various model structures, resolution settings and model scales on various benchmarks, limitations still exist in our proposed method. First of all, there lacks a clear and What is the number in the image? Baseline:The word "consciousness" is written in the image.The word "world" is written in the image.The word "construction" is written in the image.The word "would" is written in the image.Question: What is the Mosman Manly exit going to?How many items purchased from Amazon?Baseline:The mosman manly exit is going to mosman.There are 2.43 million items purchased from amazon. CAL :The mosman manly exit is going to chatswood epping.902k.Question: Provide a one-sentence caption for the provided image.Provide a one-sentence caption for the provided image. Baseline:A red background with a sun and birds and the words Sibelius Symphonies No 2 & 5.A book is open to a page with a picture of a man on a horse.A cd cover for Sibelius Symphonies Nos 2 & 5.A book is open to a page with a picture of two men on horses.Question: Provide a one-sentence caption for the provided image.Provide a one-sentence caption for the provided image. Baseline:A silver car is parked in front of a fence and a bus.A person riding a horse with a cart attached to it. CAL :A silver car is parked behind a fence in front of a trolley.A horse pulling a cart with the number 8 on it.quantitative discrepancy between the three kinds of label tokens. More discussion of the importance weights guidance can be further investigated in future works.The selection of lower bounds and upper bounds in Equation ( 3) are empirically decided based on the frequency of the prediction logits, which could be extended to more adaptive settings in further explorations. Nevertheless, the simple while broadly effective nature of CAL indicates the importance of a delicate image-text modality alignment strategy for leading VLM structures. 
conclusion: 
 In this paper, we investigate the in-completeness of current image-text alignment in leading VLMs by treating all text tokens with equal weights. We present by contrasting input images, the difference in the prediction logits for each token naturally reveals their visual correlation. We therefore propose a token re-weighting strategy that prioritize the training of highly visually correlated tokens. Our proposed strategy, CAL is simple while impressively effective, achieving consistent performance gain across various benchmarks including visual question answering, image-captioning and grounding.Our work raises a question about the potential optimal learning strategy of image-text modality alignment. Both the imperfectness of training data and over concentration on visually irrelevant/visually contradictory tokens hinder the performance of current VLMs. We hope the proposed CAL can inspire more investigation on better alignment strategy to enhance the capabilities of existing VLMs. In the primary sections of this manuscript, CAL achieves better performance on benchmarks of OCR centric VQA, image-captioning, visual grounding, and other image-dependent benchmarks. In this section, we extend our analysis by presenting additional results on MME, POPE and SEED-I and VQA Text (with OCR token given). CAL presents comparable performance on these benchmarks, without a clear distinction on performance in different training settings. With steady improvement on OCR centric or image dependent VQA benchmarks, CAL also presents satisfying quality on the remaining VQA tasks. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: In this section, we further conduct additional analysis on the image contrasting conditions. Except from masking the whole image token sequence, which is the method we use in our paper, we further study two kinds of contrasting conditions, including random patch masking and Gaussian blurring.For random patch masking, we cut images into 20 × 10 grids, with 10 pieces in the width and 20 pieces in the height. Therefore each images is cut into 200 pieces. We then randomly select 50 %, 70 % and 90 % of the patches to mask, and contrast them with the original image. For Gaussian blurring, we set the kernel σ to 1 and 10 to simulate the extreme blurring circumstances. By using 1, we adopt light blurring which only brings significant effect on OCR centric images. By using 10, all images will be heavily affected.Table 8 presents the results on Visual Question Answering benchmarks of different image contrasting conditions on LLaVA-NeXT-13B, and Table 9 further presents results on image-caption benchmarks and visual grounding benchmarks. From these two tables, either random patch masking or adding Gaussian blurring performs sub-optimal in different benchmarks. By CAL mask , except for the significant improvement on the test split of RefCOCOg, performance of CAL Gaussian drops significantly, especially on the OCR-centric benchmarks. By CAL Gaussian , although the performance on OCR centric benchmarks (VQA Doc , VQA Chart , OCR-Bench) is less affected, it performs less effective in nearly all benchmarks than CAL .We fix the window size (W) to 3 in Equation ( 4) to smooth the weight distribution. An additional experiment in Table 10 , where we removed the average pooling step, shows slightly inferior performance, supporting our chosen.One assumption of CAL is that after training a simple projector only, the VLM is capable of distinguishing visually correlated tokens. In the first phase of VLM training, it is common practice to freeze the ViT and the LLM, and only train a projector to align their features. In the second phase, we finetune the model using high-quality data to enable it to answer image-related questions. To validate this assumption, we finetune CAL models using two types of pre-trained versions: one with the original pre-trained model (only train the projector) and one with the fully trained baseline model (after the finetuning phase). As shown in Table 11 , we compare the performance of these two versions and found no significant differences in performance.A.6 Computational Overhead. Each iteration of our proposed method requires forwarding text tokens twice to effectively enhance image-text modality alignment. Table 12 presents the training time for the instruct-tuning stage on LLaVA-NeXT, with each experiment conducted on 16 A100 GPUs. CAL introduces approximately 20% computational overhead on both LLaVA-NeXT-7B and LLaVA-NeXT-13B. Our implementation currently uses an attention mask to neutralize the effect of the image tokens, meaning that the image tokens are still forwarded twice. This additional burden could be further reduced by completely removing image tokens. In this section, we visualize the image-text modality alignment by retrieving the nearest text words to each image patch feature from the LLM vocabulary. We plot the results in Figure 6 . CAL brings better image-text modality by accurately retrieving the OCR information from the language vocabulary. E.g., LLaVA1.5-7B + CAL correctly retrieves Prices, expert, 0, shadow, which is exactly the OCR information in the original image. We do not select LLaVA-NeXT in this section due to the presence of token overlaps from sub-crops. Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of our work performed for our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).We reproduce the results on LLaVA-NeXT on a slightly different training data combination, since the instruction-tuning data is not made public.Note that the caption score of MGM/MGM-HD is significantly lower than LLaVA-NeXT. The training split of MGM does not include the training set for both COCO Caption and TextCaps, where we provide the zero-shot performance for MGM in all settings on these two benchmarks. 


Decision: This paper introduces an approach for multimodal alignment through contrastive alignment by assigning different weights for different tokens. All reviewers agree that the paper is easy to follow and well-written, and the idea is interesting. The authors' rebuttal addressed misunderstandings in terms of experiments and all reviewers recommends acceptance. The AC thus recommends acceptance and encourages the authors to revise the paper accordingly.

Reviews: review: paper_summary: paper_summary: This study proposes a reweighting strategy, namely contrastive alignment, to enhance model learning on visually correlated tokens. Specifically, the authors divided the tokens into three sub-groups, virtually related, non-related, and contradictory, and assigned different weights to each group. The weight is calculated based on the predicted contrastive logit (w/o visuals) and further post-processed by clamping and average pooling. This study is evaluated on vision question answering, image caption, and grounding tasks using two foundation model structures, LLaVA and Mini-Gemini.
strength_weakness: strength_weakness: -- This study does not compare with any baseline models within the topic of resisting noisy tokens, such as [*] and [**].

-- In [*], a study with similar aims was presented. In particular, they also studied the image-token consistency. Thus, detailed baseline comparisons, including overall performance and token consistency evaluation/comparison, are expected. 

-- The authors claim that the proposed strategy is simple yet efficient. However, it may rely upon several strong assumptions, such as high capability requirements of the pre-trained model. These assumptions are not presented, discussed, and validated in the paper. 

-- Weight distribution on different token sub-groups is not presented. This would contribute to the validation of weighting.

[*] Gou, Yunhao, et al. "Leveraging per image-token consistency for vision-language pre-training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[**] Wu, Cheng-En, et al. "Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
questions: questions: See above weakness.
limitations: limitations: As I can see, this paper has no potential negative societal impact.
paper_summary: paper_summary: This paper points out that parts of samples in the broadly used datasets contain visually contradictory text tokens. To mitigate the sub-optimal cross-modal alignment in VLMs, this proposed method is to assign distinct contributions for each text token based on its visual correlation.
strength_weakness: strength_weakness: Please see the Questions
questions: questions: 1. It’s unclear how to calculate the prediction logit distribution o without input I for the VLM like CLIP. Is this method not applicable to the pure image-text matching VLM like CLIP? 
2. There’s no l in Equation 4. Does it mean W in Sec. 3.1 implementation details “we set l in Equation 4 to 3 for all experiments”?
3. For Figure 3, it is better to describe what the dashed line represents as well.
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces Contrastive Alignment (CAL), a straightforward yet effective re-weighting strategy designed to improve multimodal alignment. Specifically, the authors propose contrasting image inputs and calculating the differences in prediction logits for each text token to determine its training weights. Extensive experiments are conducted to demonstrate the effectiveness of CAL across various tasks, providing a robust validation of the proposed approach.
strength_weakness: strength_weakness: 1. While CAL is innovative for aligning multimodal data, its robustness and comprehensiveness in complex task settings, such as multimodal question answering (VQA), appear limited. The method ensures correct alignment but may inadvertently impair other model capabilities, such as reasoning ability or retention of original world knowledge. This issue is critical in VQA scenarios where data often include elements not directly related to image content, potentially conflicting with it and incorporating extensive external knowledge. The application of CAL in such contexts might suppress training on these critical aspects, introducing uncontrollable biases and worsening model performance. Although CAL shows promise in simpler tasks like captioning and grounding, its performance in VQA tasks is inconsistent, and its effectiveness in addressing hallucination issues, such as in the POPE dataset, is negligible.
2. CAL’s effectiveness appears contingent upon the pre-existing performance of the underlying multimodal model. Since it calculates visually correlated weights based on the model itself, any hallucinations or errors present in the base model are not only perpetuated but potentially exacerbated by the CAL method. This limitation restricts CAL’s applicability and caps its performance, as there is no mechanism within CAL to correct these amplified errors.
questions: questions: 1. Why MGM-HD-13B perform worse that -7B model in caption tasks as shown in Tab-2 in the paper ? 
2. The visually correlation weights in CAL are smoothed using average pooling with a window size of W. How is W determined? Is this parameter set consistently across all datasets, or does it require adjustment for each training run to optimize performance?
3. The paper mentions that CAL adds minimal computational overhead compared to other data scaling strategies. However, could the authors provide more quantitative examples or an intuitive comparison? It seems that processing very long texts could significantly increase the overhead.
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes a new re-weighting strategy for training VLMs to improve visual-language alignment. The motivation is to assign higher weight to visually relevant tokens and lower weight to visually irrelevant and visually contradictory tokens. The re-weighting factor relies on the logit difference with and without image inputs for each token. Using the proposed CAL, the paper demonstrates general improvement in VQA and captioning tasks.
strength_weakness: strength_weakness: 1. **Lack of Clarification in Presentation**: Some content is confusing. For example, equation 2 is averaged across the entire sentence with length $l$, yet the description in lines 88-89 refers to the loss objective for the $i^{th}$ sample at token $t_j$, which seems more like the loss at a single token. Additionally, equation 4 lacks clarification; it's unclear how to apply average pooling with a window size from this equation. Some experiments also lack necessary context. For instance, it's unclear which model is evaluated in Figure 2a, and the ablation study in section 3.3 doesn't specify which model is used for evaluation. The main result in section 3.3 doesn't disucuss how to apply CAL into model training, at which stage for example. The annotation is inconsistent; in equation 4, it's $\tilde{w}^{i,t_j}$, while in equation 5, it's $\tilde{w}(i.t_k)$

2. **More Experiments Are Required**: The paper presents decent improvements over some baselines, mainly in vision-language tasks. However, since the training recipe has changed, it would be better to also evaluate other tasks, such as language-only tasks, to see how the CAL affects the model's ability in standard language understanding. Additionally, the method seems quite relevant to hallucination tasks, so it would be beneficial to report results on those as well.

3. **Lack of Originality**: The motivation and methodology are already well-studied in language models [1,2], and prior studies have also applied changes to logits based on the presence or absence of images to measure the relevance of visual and language inputs. In this context, the paper simply applies the method during the training stage, instead of the inference stage as in [3]. Therefore, the novelty is somewhat limited.

[1] Jiang S, Wolf T, Monz C, et al. TLDR: token loss dynamic reweighting for reducing repetitive utterance generation[J]. arXiv preprint arXiv:2003.11963, 2020.

[2] Lin Z, Gou Z, Gong Y, et al. Rho-1: Not all tokens are what you need[J]. arXiv preprint arXiv:2404.07965, 2024. 

[3] Zhu L, Ji D, Chen T, et al. Ibd: Alleviating hallucinations in large vision-language models via image-biased decoding[J]. arXiv preprint arXiv:2402.18476, 2024.
questions: questions: 1. The trend shown in Figure 2a is based on which model? Is this a common phenomenon across existing VLMs?

2. How do you deal with cases where a semantic word has been split into two subparts, each with a different value, such as "tr" and "uck" in Figure 2a?

3. The CAL seems to rely heavily on the model's logits with and without image input. However, can the model accurately predict changes in logits, as shown in Section 2, during the initial alignment stages?

4. The training requires an additional forward pass of the LLM. Why is the training described as "lightweight" in the paper? Doesn't this double the cost?
limitations: limitations: not applied

Comments: title: Response to rebuttal by authors
comment: Thank you for your detailed feedback and for raising your score. We appreciate your recognition of our work's potential and your valuable suggestions.

Regarding the computational cost, we can preprocess and store the CAL weights using the trained model offline, which will not increase the cost in subsequent training. This is one of our future improvement directions. 

In the pretraining stage, since the number of trainable parameters (only projector) is limited and there is a minimum weight for all text tokens, inaccuracies in predicting logits in the early stages do not significantly impact the overall performance. As shown in Table 3, applying CAL in both pretrainig and finetuning stages yields the best performance. 

Regarding the issue of positional embeddings, in the current version, we ensure that the positional embeddings for text tokens are consistent in both two scenarios.

Once again, thank you for your valuable insights and support. We are committed to addressing these issues in our future work.
Thank you for your detailed feedback and for raising your score. We appreciate your recognition of our work's potential and your valuable suggestions. 

Regarding the computational cost, we can preprocess and store the CAL weights using the trained model offline, which will not increase the cost in subsequent training. This is one of our future improvement directions.

In future versions of our paper, we will add more analyses, including exploring weight distribution and scenarios where CAL may not enhance performance. In our subsequent work, we will strive to address the uncertainties introduced by the base model.
Thank you for your constructive feedback and for maintaining your positive rating. We appreciate your suggestion and will explore implementing CAL on CLIP with noise-added images in future work.
Thank you for your feedback and for raising your score. We appreciate your recognition of our work's potential and your valuable suggestions.
The authors partially addressed my concerns. So, I increased my rate.
Thanks for the response. The Answer for Q1 is still a vague idea in its infancy. In future work, the authors may implement the CAL on CLIP by using "noise-added images" to decide the token scores and see if it works. I choose to keep my original positive rating.
Hi Reviewer  tsLt,

Could you take a look at the authors' rebuttal and finalize your rating?

Thanks,
AC
Hi Reviewer  DwVB,

Could you take a look at the authors' rebuttal and finalize your rating?

Thanks,
AC
I appreciate the authors' response in addressing my questions and concerns. The explanation effectively clarifies the concern regarding training costs, particularly noting that the second forward pass does not require storing intermediate activations or performing backward propagation. However, the claim of a 20% increase in time is not scientifically grounded, as it can vary depending on the dataset used for training.

My third question was not fully understood by the authors. The inquiry was not about the training recipe for MLLMs like LLaVA. In the early stages of training, one might assume the model lacks the ability to generate captions from images but the LLMs can perfectly generate text from a partial caption. The question is whether the gap in predicted logits with and without the image is accurate in the initial stage where the model lacks good captioning capacity. It might be more appropriate to apply this training during the instruction-tuning stage, where it is assumed the model can generate captions after pretraining. Another way to frame my question is: considering that language models typically use RoPE as a positional embedding, how does prepending image tokens affect logits given the altered positional embedding of text tokens in this scenario?

Overall, while some concerns remain, particularly regarding the limited originality of the methodology in light of prior works, I find the work interesting and believe it may inspire future research. As such, I have raised my score to 5.
Thanks for the response, most of my concerns have been addressed. However, for an algorithm that introduce additional computational costs, mere performance enhancements in "most" scenarios are insufficient, particularly as we observe that the CAL method results in performance degradation in the LLaVA-Next model. This phenomenon could be anticipated given the lack of interpretability of CAL in complex settings, or as discussed in W2, the introduction of uncontrollable biases stemming from inherent limitations of the base model. Despite its flaws, this work challenges the dominant paradigm of pure supervised learning in multimodal training, making it a valuable contribution to the field. In recognition of its potential, I will raise my score.  I urge the authors to provide more quantitative analyses in future version of the paper, such as exploring the weight distribution highlighted by Reviewer DwVB, or presenting scenarios where CAL fails to enhance performance, to deepen our understanding of these issues.