Self-Supervised Visual Preference Alignment

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 16:51:57

Abstract: This paper makes the first attempt towards unsupervised preference alignment in Vision-Language Models (VLMs). We generate chosen and rejected responses with regard to the original and augmented image pairs, and conduct preference alignment with direct preference optimization. It is based on a core idea: properly designed augmentation to the image input will induce VLM to generate false but hard negative responses, which helps the model to learn from and produce more robust and powerful answers. The whole pipeline no longer hinges on supervision from GPT-4 or human involvement during alignment, and is highly efficient with few lines of code. With only 8k randomly sampled unsupervised data, it achieves 90\% relative score to GPT-4 on complex reasoning in LLaVA-Bench, and improves LLaVA-7B/13B by 6.7\%/5.6\% score on complex multi-modal benchmark MM-Vet. Visualizations shows its improved ability to align with user-intentions. A series of ablations are firmly conducted to reveal the latent mechanism of the approach, which also indicates its potential towards further scaling.

Main paper: 
 introduction: 
 Large Vision-Language Models have recently emerged and greatly advanced current researches [2, 27, 36, 41] . Representative methods in this family, such as MiniGPT4 [47] , LLaVA [27] and Instruct-BLIP [8] , try to properly align vision signals to Large Language Models (LLMs) to better conduct multi-modal comprehension. These methods usually undergo a pretraining stage with abundant imagetext pairs for modality alignment before supervised finetuned (SFT) with academic [8] or GPT-4 generated [4, 43] SFT data. There are variants of them (e.g., LLaVA-based) that try to improve the VLMs' instruction following ability by leveraging high quality pretraining pairs [4, 24] or scaling up SFT database [23, 38, 43] .Despite their success [4] in boosting the comprehension skills of current VLMs, these models are not properly aligned with userintentions. As a result, they lack the ability to reject samples and could induce more unintended output. For instance, Instruct4V [38] construct 220k SFT data from GPT-4, but still struggle on hallucination benchmark POPE [21] . The same can be observed in Fig. 7 , where LLaVA failed to follow user instructions and provide meaningless information. Recently, there are few trials that incorporate alignment techniques into vision-language fields [20, 37, 44] . However, we found that they only emphasize on quite limited task domains [44] , and, more importantly, their data construction pipeline requires extra knowledge source either from GPT-4 or human feedbacks. This might hinder them from further data scaling, as preference data is not cheap (e.g., a 10k collected human-evaluated instances requires a cost of 3000$ [22] ).In this paper, we propose a self-supervised paradigm that can freely obtain arbitrary number of preference samples with minimal code implementation (cf. Alg. 1). Simply but, we have enjoyed the advantage of this pipeline and observed a significant improvement in the capabilities of current VLMs from various aspects: stronger chain-of-thought skills, better OCR [30] ability, proper alignment with user-intentions, less hallucinations, etc (cf. Fig. 7 ).Our motivation came from an interesting discovery in Fig. 2 , where we apply commonly adopted image augmentations in visual contrastive learning [13, 48, 49] during LLaVA inference. The effect of all these test-time augmentations are evaluated on three common multi-modal benchmarks (MMVet [40] , MMBench [28] , POPE [21] ). It is not surprising to observe in Fig. 2 that vision-language models are quite sensitive to data-augmentations and, slightly distortion will induce the model to output different semantic tokens. Then, an natural question arises: are the original and distorted responses valuable to construct preference data for DPO training? Since this will totally free the data construction from any extra source, and might relieve the difficulty of scaling up preference data as well [42] . idx = (C != R) C 𝑡 , R 𝑡 = C[idx], R[idx] Q 𝑡 , I 𝑡 = Q[idx], I[idx] # DPO training, omit reference model for simplicity DPOTrain(model, (Q 𝑡 , I 𝑡 ), (C 𝑡 , R 𝑡 ))Our motivation was then firmly verified by quantitative experiments in Table 1 , in which we randomly sampled 8k image-question pairs from the subset of LLaVA665k [26] (cf. Sec. 4.1 for details), and choose 5 augmentations to generate the preference data using LLaVA-7B. All these self-generated data are again fed into the same LLaVA-7B model for DPO training. As Table 1 shows, all the augmentations are helpful for model comprehension, especially on GPT-4 evaluated benchmark MM-Vet, which makes our motivation valid. In the meanwhile, we found that either too strong (e.g., Diffusion-S) or too weak data augmentation (e.g., RandFlip) are sub-optimal for the whole pipeline, and medium is the best (e.g., Diffusion-W, MOCO). Our final conjecture is: self-generated augmentation pairs are suitable preference sample to improve multimodal abilities, and hard negatives are most preferred.We name our methodology as: Self-supervised Visual preference alignment (SeVa), and summarize its whole pipeline in Alg. 1. Though being embarassingly simple with few lines of code to implement, we mathematically showcase its strong relation with visual contrastive learning in maxmizing a preference distribution, where SeVa could probably be viewed as a special form of contrastive learning with one negative sample. This makes SeVa easily extendable if more negatives are involved (cf. appendix).Experiments in common multi-modal benchmarks demonstrate the effectiveness of our method, where SeVa improves the VLM's comprehension ability by a large margin (e.g., SeVa-7B even surpasses LLaVA-1.5-13B by 1.8% in MMVet). In addition, we carefully design detailed ablations to reveal the latent mechanism of SeVa from different angles. We found that SeVa shows surprisingly results like detailed descriptions, less hallucinations, stronger OCR skills and chain-of-thought ability, etc. Our contributions are:• For the first time, we conduct visual preference alignment in an un-supervised manner. The whole pipeline do not require any GPT-4 or costly human annotated data. • We found such an alignment pipeline have numerous benefits such as enhanced multi-modal comprehension ability, better alignment with user-intentions, etc. Visualizations in Fig. 7 and ablations firmly verify our hypothesis as well. • Our method, SeVa, enjoys efficiency in pipeline and simplicity in implementation, which paves way for future preference alignment in visual-language domain. 
related works: 
 Large Vision-Language Models (VLMs). The great development of Large Language Models (LLMs) has facilitated the community in both academics [2, 8, 19, 26, 47] and industries [1, 7, 36, 46] . Recently, large vision-language models (VLMs) emerged, aiming to extend the reasoning brain of LLM to vision modality. The majority of VLMs undergo a two-stage training manner, with the pretraining period focusing on aligning the vision and text signals before finetuned with instruction following data in the second stage. LLaVA [27] , as one of its representatives, has attracted numerous researchers [4, 22, 23, 38, 39] since it provide a valuable opportunity for them to reproduce and built upon. Subsequent works based on LLaVA try to leverage more image-text data pairs of higher quality [4] or to construct more abundant instruction following database [38] .Despite their achievements, these model are not preference aligned with user intentions neither implicitly nor explicitly, which might limit their further deployment. Therefore, an suitable alignment technique is of great importance.Preference alignment in LLM/VLM. Training an LLM to align with human or user preference is called RLHF [31] , which usually happens after the instruction following stage. The core concept of RLHF is to reduce un-intented or toxic output produced by LLMs [10, 50] . Variants of RLHF include, but not limited to, DPO [33] , PPO [34] and RLAIF [3] . Alignment has been fruitfully researched in natural language processing (NLP) domains [3, 12] , but relatively less visited in vision-language fields. There are some latest attempts [20, 37, 44] that integrate alignment techniques in vision-language domains. However, they only emphasize on task specific ability (e.g. hallucinations in HA-DPO [44] ), and, more importantly, it requires tedious GPT-4 or human interventions to construct the preference data, which could diminish the pipeline efficiency. In this paper, we propose an unsupervised data construction pipeline that not only solve the data hungry issue in preference alignment, but also greatly boosts the comprehension ability of current VLMs, as well.Contrastive learning. Self-supervised learning (SSL) of visual representation are popularized in the past years [5, 11, 13, 48] . Traditional SSL are mostly contrastive based, where strongly augmented positive views of the same image will share similar deep semantic in the hidden space that the model could capture. Nevertheless, we found that a similar augmentation pipeline does not holds true in VLMs. Inspired by unsupervised property of SSL, we design an selfsupervised pipeline to construct the preference data, and empower the current VLM with improved capability. 
method: 
 We will first show the preliminaries of language modeling and direct preference optimization, then move onto the procedure of the proposed SeVa pipeline. Finally, we discuss its strong relation with visual contrastive learning.Language modeling. We start from the vison-language modeling perspective and take LLaVA [27] as an example for illustration. Given an image input 𝐼 , it is first passed through a vision encoder Table 1 : Six data augmentations selected to generate preference data for DPO training (w/ LLaVA-1.5). Each strategy was adopted to distort the image (together with the questions) to produce rejected response, which is paired with the chosen response from the original image input (cf. Alg. 1). The models after DPO training with these preference pairs are then evaluated on MMVet [40] , MMBench [28] (in english 'en' and chinese 'cn') and POPE [21] (e.g., ViT-L [32] ) to get:EQUATIONwhere 𝑔 denotes a combination of vision encoder and projection layer. 𝒉 represents a sequence of deep image embeddings. The embeddings before the last transformer layer are considered in LLaVA-1.5 [26] , which are then concatenated with the question token 𝑞 and fed into the auto-regressive LLM 𝜋 that sequentially generates the next target token 𝑦:EQUATIONwhere 𝑥 = (𝒉, 𝑞), and is considered as prompt sent to the LLM parameterized by 𝜃 . 𝑦 <𝑖 is the generated token before the current prediction 𝑦 𝑖 , and 𝐿 is the length of the produced token sequece. We denote the LLM trained with SFT data as 𝜋 SFT for clarity. Direct preference optimization. DPO is first proposed in [33] , and can be viewed a new parameterization of the reward model in RLHF [31] that can directly optimize the policy (the LLM's parameter 𝜃 ). Here we start with RLHF for better illustrations. In RLHF, a Bradley-Terry (BT) reward model is often adopted, which stipulates the human preference distribution as:𝑝 * (𝑦 𝑐 ≻ 𝑦 𝑟 |𝑥) = exp (𝑟 * (𝑥, 𝑦 𝑐 )) exp (𝑟 * (𝑥, 𝑦 𝑐 )) + exp (𝑟 * (𝑥, 𝑦 𝑟 )).(The 𝑦 𝑐 and 𝑦 𝑟 are the chosen and rejected response conditioned on the LLM's prompt 𝑥. The reward model 𝑟 𝜙 (•) is paramerized by 𝜙 and can be obtained with maximum likelihood using preference database D annotated by human [33] :EQUATIONThen we maximize the a preference policy as follows:maxEQUATIONThe reference model 𝜋 ref are initialized by 𝜋 SFT , which prevents the learned parameter 𝜃 ′ from much deviation. The finally learned policy 𝜋 𝜃 ′ (•) are better equipped with human or user intentions. DPO uses a closed form [33] derived from Eq. 5 to represent the optimal reward 𝑟 * model by the learned optimal preference model 𝜋 * as (with 𝐾 the constant factor): By substituting the 𝑟 * in Eq. 3 with Eq. 6, we get the final optimized loss function of DPO as follows (cf. [33] ):EQUATION)EQUATIONThe trainable parameter Eq. 7 is 𝜃 ′ , which is the same with RLHF. But it do not need any online optimization process like PPO [34] . Instead, DPO samples (𝑥, 𝑦 𝑐 , 𝑦 𝑟 ) from preference database D and optimize the VLM's parameters 𝜃 ′ with a simple classification loss.This section will introduce our pipeline: self-supervised visual preference alignment (SeVa). For a language model 𝜋 SFT after supervised finetuning stage, we feed it with the original and augmented image input as (for simplicity, we omit the generated token as conditional input shown in Eq. 2):EQUATIONEQUATIONThe 𝑦( 𝑗 )𝑐 and 𝑦( 𝑗 )𝑟 are regarded as a pair of raw chosen and rejected response. For each image 𝐼 𝑗 and its question 𝑞 𝑗 , we generate only one paired preference data (𝑦 ( 𝑗 ) 𝑐 , 𝑦 ( 𝑗 ) 𝑟 ). Suppose there are total 𝑀 image-question pairs and its corresponding generated preference pairs, we simply filter those equaled responses to get remained preference data pairs (supposed 𝑁 𝑑 ), which constitute the final preference database D SeVa for DPO training (cf. Alg. 1 and Eq. 7).EQUATIONEQUATIONWe choose 6 different data augmentation strategies to construct our database and conduct DPO training. As shown in Table 1 , all self-constructed preference data are helpful for multi-modal comprehension, showing the validity of this pipeline. What surprises us is that training samples built from 'RandFlip' are quite helpful (e.g., an increase of 3.2% score on MMVet). We conjecture that this might derive from the natural property of the text data we used, since flipped images can make OCR and recognition harder and induce potential negative responses (cf. appendix for more details). Besides, all models have seen a noticeable improvement in adversarial settings of POPE, indicating the models trained by our self-supervised pipeline might be less inferior to potential interruptions.In contrastive learning, augmented views of the same image will share similar semantic in its deep embeddings, which optimize the following InfoNCE loss:EQUATIONin which 𝑞, 𝑘 + are the positive embeddings from augmented views of the same image, while 𝑘 -could be negatives embeddings from other images [5] or memory bank [13] . If we consider only one negative, and denote 𝑓 (𝑞, 𝑘) = (𝑞 • 𝑘)/𝜏 as the scoring function, we can reformulate Eq. 12 as:EQUATION)It is not hard to find that the optimzed loss function derived from Eq. 3 are quite similar with Eq. 13. The main difference lies in the definition of negative samples. Unlike contrastive learning, the predicted tokens from the augmented views of the same image will be counted as negatives, but not positives. If we imitate traditional InfoNCE loss in Eq. 12, we could inject multiple negative reward terms in Eq. 3 (by augmenting multiple views of the same image to [27] , MMBench [28] , MMBench-Chinese [28] , POPE [21] , SEED-Image [18] , SHR [44] , SQA [15] , GQA [15] . Among them, SHR are newly proposed [44] to evaluate the hallucination ratio of VLMs.  
experiment: 
 In this section, we conduct experiments to verify the effectiveness of the proposed SeVa pipeline. Firstly, we provide the experimental settings (e.g., the data construction process). Then we show the results on multi-modal comprehension benchmarks. Finally, we provide fruitful ablations and visualizations to analyze SeVa with more details. For simplicity, we name LLaVA-1.5-7B/13B trained with SeVa pipeline as SeVa-7B, SeVa-13B, respectively.Data construction. The source data we obtained are from LLaVA-665k SFT dataset [26] , and we choose image-question pairs from TextVQA and OCRVQA [30, 35] (denoted as 'text+ocr') in 665k to generate the DPO preference data. This setting is kept by default throughout all the experiment unless otherwise noted. For each data instance in 'text+ocr' of LLaVA665k, we randomly choose 2 questions in its multi-turn dialogue, and pair each question with the image. We treat each image-question pair as one data instance, which leads to nearly doubled instance as the original 'text+ocr'. Then we randomly choose 8k such image-question pairs in 'text' and 'ocr' each, to form the original data source of 16k. Finally, these 16k image-question instances will go through SeVa pipeline to produce preference data with filtering (cf. Alg. 1), which are cut down by half to about 8k. For clarity in our later ablations, we denote data nums as those unfiltered preference pairs instead of the filtered ones (e.g., 16k but not 8k), since the num of raw unfiltered data is fixed and immune to filtering process (cf. appendix). Data augmentation choice. As verified in Table 1 , all the selected data augmentations are all helpful for multi-modal comprehension. In our later experiment, we choose diffusion noise as the default augmentation in SeVa pipeline, since it can easily control the intensity of distorted level, which is helpful for ablations (cf. Fig. 6 ). The noise steps for training SeVa-7B and SeVa-13B are set as 800 and 500, respectively. As MOCO augmentation strategy is also highly effective as shown in Table 1 , we are glad to verify more data augmentation strategies in our future work.Training details. Following previous works [4, 22, 23, 38] , we choose LLaVA-1.5-7B/13B as our base models in all our experiments. The LLaVA's weights are pretrained and SFT tuned before our DPO training, which is the by default pipeline as in many previous literatures [12, 33, 34, 44] . We adopt deepspeed ZeRO stage-3 [26] during DPO training and use Vicuna-7B/13B, CLIP-VIT-L-336px as our LLM and vision encoder, respectively. The total epoch, batch size, learning rate and weight decay are set as 1, 128, 2e-6 and 0, respectively, following previous work [44] . The hyper-parameters of lora 𝑟 are set as 1024, and the scale parameter 𝛽 in DPO is fixed as 0.1. DPO optmization are conducted with 8-A800 GPU, with LLaVA-7B/13B tuned for about 0.7/1.3 hours.Firstly, we compare the proposed SeVa with state-of-the art VLMs. A total of 9 benchmarks are included, with multi-modal benchmarks as: MMVet [40] , LLaVA-Bench-in-the-wild [26] (LLava W ), MMBench [28] (MMB), MMBench-Chinese (MMB CN ), POPE [21] , SEED-Bench-Image [18] (SEED 𝐼 ) and SHR [44] , and traditional Question-Answer benchmarks ScienceQA [29] (SQA) and GQA [15] . Among them, MMVet, LLaVA W are two GPT-4 evaluated benchmarks. MMB and MMB CN are multiple choice question answer dataset that requires minimal GPT-4 involvement (e.g., answer choice aligning [28] ). POPE and SHR are two hallucination benchmarks, and SQA & GQA are two traditional QA benchmarks.As shown in Table 2 , SeVa consistently improve the LLaVA-1.5 7B/13B models on 8 out of 9 multi-modal benchmarks. Specifically, it improve its 7B/13B models by 6.7%/5.6% on the most complex multimodal dataset MMVet, showing the great power of the proposed pipeline. The same can be observed in LLaVA W , where SeVa-13B achieves a 80% relatively score to GPT-4, boosting LLaVA-1.5-13B by a large margin of 9.4%. The great boost on GPT-4 evaluated datasets suggested a better alignment with user-intentions, as demonstrated in Fig. 5 and Fig. 7 . We also observe a consistent improvement on POPE and SHR, which indicates the effect of SeVa to relieve object hallucinations, and thus we may not resort to specialized hallucination methods [17, 44] . Please note that the improvement on hallucinations are not marginal, since previous work that constructed huge amount of SFT data can lead to decreased performance on POPE (e.g., LVIS-Instruct4V [38] ). There is also a higher multiple choices accuracy in MMB, MMB CN and SEED 𝐼 , showing the broad adaptability of SeVa. In Table 2 , a little bit drop was found on traditional benchmarks SQA/GQA. We conjecture that this phenomenon might arise from a trade-off between stronger instruction following ability in traditional QA and better comprehension ability in modern VLMs benchmarks, which is also pointed out in [12] that stronger instruction following do not guarantee a more helpful and better VLMs. In fact, how to better trade-off between benchmarks remains a discussing issue [4] and is out-of-scope of this paper.Now we ablate the factors in our pipeline, to help readers better understand the success behind SeVa and its generalized ability. We choose SeVa-7B as the default model unless otherwise noted.Compare with SFT. We start from comparing with continual SFT, as this is the most straight forward method to verify its effectiveness with supervised training (note SeVa do not need target answer labels). We conduct three different types of SFT: a randomQuestion textvqa: -Provide a one-sentence caption for the provided image. Reference OCR token: [task-specific prompt]-Who is the author of this book? -What is the title of this book?-Is the cheese to the left of the food on the plate? -Which kind of furniture are the cups hanging on?-Where is the cat positioned in the image? -Please provide a short description for this region: [task-specific prompt]-Please provide a short description for this region: [task-specific prompt] -Provide the bounding box coordinate of the region this sentence describes. [task-specific prompt] Figure 4 : Illustration of representative questions in five databases from LLaVA665k. In our main experiment, we adopt a combination of 'textvqa' and 'ocrvqa'. The results of applying the other 3 question types can be found in Table 4 . selection of 2%/10%/20% in LLaVA665k and a collection of TextVQA and OCRVQA in 665k (denoted as 'text+ocr'). We evaluate its downstream performance on MMVet, MMB and POPE, which can be found in Table 3 . For a fair comparison, we strictly follow LLaVA-1.5's SFT settings during experiments. It can be observed that all SFT data are helpful for multi-modal comprehension. Specifically, text related data are more helpful for dealing with model hallucinations (e.g., on POPE). In comparison, our SeVa achieves the overall best results on all benchmarks with both less training time cost and data usage. More importantly, SeVa enjoys its unsupervised nature, making it more generalizable in reality.Generalize to other datasets. Since we adopt the questions in TextVQA and OCRVQA as a default fashion, we now ablate on more dataset selection on top of these two dataset. For more Table 5 : GPT-4 evaluated consistency score from 1-10. Qconsistency measures how the model's answers are aligned with the asked question, while A-consistency measures how answers (𝑡 > 0) are aligned with its generated tokens without sampling (𝑡 = 0). We adopt 1.5 version of LLaVA. clarity, we choose other part of LLaVA665k including GQA [15] , COCO [25] and Visual Genome [16] to form image-question pairs. We randomly select 2k-16k these data and conduct SeVa-7B training pipeline. The results are then evaluated on MMVet. As shown in Table 4 , all the choosen dataset lead to consistent improvement over LLaVA-1.5-7B (30.5% score), showing the generalized ability of SeVa to different image and question domains. Besides, involving more unsupervised data leads to consistent better performance, which indicates the great power of SeVa towards further scaling. Why SeVa works? Attentive readers might have a natural question: why rejecting inferior answers lead to better model output than the original? Here, we conduct a detailed experiment to show that, SeVa can be viewed as a special form of model calibration and rejecting negatives does help. Specifically, we collect image-question pairs from LLaVA W to prompt LLaVA and SeVa with higher sampling temperature 𝑡. Note that 𝑡=0 means no sampling in token generation, which is the default and most stable evaluation settings. Then we utilize GPT-4 (cf. appendix for prompt) to evaluate the its output consistency with scoring from 1 to 10, namely Q-consistency (how the model's answer are aligned with the input questions) and Aconsistency (how the model's answer are aligned with its generated tokens w/o sampling). As shown in Table 5 , the consistency score of both models fluctuates as 𝑡 grows. However, SeVa has seen an consistent better score in all different 𝑡s, especially when temperature becomes higher. Since VLMs are easily affected by randomness in generating tokens [20] , SeVa has potentially made a calibrating effect by reducing the sampled probability of those bad tokens, thus being more tolerable to interruption. As a result, the model trained with SeVa can produce more robust and correct answers.Besides, we also observed a longer response in LLaVA W and MMVet (cf. Fig. 5 ). This indicates that models trained with SeVa pipeline could potentially produce more detailed and meaningful responses that greatly benefits multi-modal comprehension.Hard negatives matter. As we previously observed the phenomenon of hard negatives in Table 1 , we now design a controllable experiment to study how strength of data augmentation affect the final training performance. Specifically, diffusion gaussian noise with different steps are chosen to generate preference data for DPO training, which are then evaluated on MMVet benchmark. As shown in Fig. 6 (a), larger noise steps will increase the reward margin (cf. DPO [33] for more details) during training, indicating the increased preference divergence. In the meanwhile, a trade-off exists in the noise steps (possibly between 500 and 800), as shown in Fig. 6(b) . We thus attribute the preference samples produced within this regime as hard negatives that are most valuable for DPO training.Effect of Lora. For completeness, the hyper-parameters of LoRA are ablated on three multi-modal comprehension benchmarks, as shown in Table 6 . In PETL [9] , this parameter can be quite important, as it decides how much new knowledge the model can be absorbed during finetuning. As shown in the table, a relatively LLaVA-1.5-13B: Yes, you should add sugar when mixing egg and milk in the chocolate cake recipe. This is because sugar helps to balance the flavors and create a more harmonious taste in the final product. Q: Can you give a short introduction to this movie? LLaVA-1.5-13B: The image features two men in suits, standing close to each other and looking at each other intently. They appear to be having a conversation or sharing a moment of connection. The scene, suggests that it might be a social gathering or a formal event. The men's attire and the presence of other people in the room indicate that this could be a scene from a movie or a special occasion. Since we built-on LLaVA-1.5, we directly make comparison between them to show the effectiveness of our proposed method. For clarity, we highlight the incorrect response in LLaVA with red color, and emphasize ours with blue italic character. This figure is best viewed in color.higher 𝑟 is helpful to downstream tasks, showing the increased acquired knowledge of the VLM. However, too large 𝑟 will decrease performance, which can be attributed to the catastrophic forgetting in training a LLM (a similar point also pointed out in [4] ). In conclusion, we set 𝑟 as 1024 to seek a best trade-off, and always keep the delta importance as 2 (the 𝛼 is always twice as 𝑟 ).Aligning with user-intentions. Finally, we visualize the improved VLM with SeVa. We selected image-question pairs from MMVet and LLaVA W , which are sent to the original VLM (LLaVA-1.5-7B) and the improved model (SeVa-7B), respectively. As shown in Fig. 7 , Our SeVa demonstrates superiority over LLaVA in various aspects: stronger OCR ability, where SeVa could recognize the exact number of the gas price; less hallucinations can be observed in the second picture, where SeVa accurately comprehends the process of 'chocolate cake recipe' and produces correct answers. It is also surprising that SeVa could potentially recovers more world knowledge after DPO training, as it gives detailed and accurately response to the introduction of a movie (the third picture). Finally, we found that through our DPO training, the models provide more detailed and helpful answers, as suggested by the last picture of a meal ordering. Following Vicuna [6] , we also conduct a competition game between SeVa and LLaVA-1.5 under 7B and 13B settings on LLaVA W and MMVet, respectively. We introduce GPT-4 as judge to evaluate the score of each model's responses (similar to the evaluation process in these two benchmarks) and calculate the statistics of 'win' and 'lose'. This could serve as a result that indicates how the models are aligned with user-intentions [45] . Note we didn't include 'tie' [6] , since we observed a majority of meaningless 'tie', where both SeVa and LLaVA achieves a a score of zero. As shown in Fig. 5 , SeVa has occupied most of the winning rounds in comparison with LLaVA, across both model sizes and benchmark datasets. Together with the visualizations in Fig. 7 , we believe that our SeVa could probably serve as a strong pipeline to improve current VLMs' chat ability, and makes them more suitable towards practical usage. 
conclusions and future work: 
 In this paper, we introduce SeVa: Self-supervised visual preference alignment, for multi-modal comprehension. It first went through a data collection and filtering process with response from the original and distorted image, respectively. Then, the standard DPO training are applied to improve the model's capability. Experiments in various benchmarks clear verify and efficiency of the approach. We also conduct ablations and quantitative visualizations to reveal the latent mechanism of SeVa, which demonstrate the great merit of aligning large models in vision-language domains.In the future, we might consider generalize SeVa to various domains. We will study the effect of data scale: how the model will benefit from DPO training if more unlabeled data are involved, On top of that, we would investigate the potential to generalize our SeVa pipeline RLHF [31] domain. (e.g., whether the generated preference data could be helpful in building a reward model).  
 
 Appendix:  


Decision: This paper investigated unsupervised preference alignment in Vision-Language Models (VLMs). The idea is simple but intuitive: properly designed image augmentation will induce VLMs to generate hard negative responses. Motivated by this idea, this work further proposed self-supervised visual preference alignment that takes the original and augmented images for both supervised fine-tuning and direct preference optimization. Experiments on 9 benchmark datasets show the significant improvement of the proposed SeVa compared to LLaVA-1.5 on 7B and 13B models.

This paper received 1 Accept, 1 Weak Accept, and 1 Borderline Reject. After rebuttal, the reviewer who gave borderline reject increased the score to borderline accept. Considering the strengths on motivation, novelty, effectiveness, and writing, the final recommendation is a clear accept.

Reviews: review: paper_summary: paper_summary: The paper introduces a method that optimizes vision-language models by self-generated chosen or rejection pairs with DPO. It designs augmentation to the image to generate false but hard negative answers. The method does not rely on human annotation for alignment. Extensive experiments and analyses show the proposed method is effective.
strength_weakness: strength_weakness: - The paper is well-written and easy to follow.
- The proposed methods are novel and effective.
- The analyses of the connections to contrastive learning are insightful.
- The methods proposed do not rely on human or GPT-4 annotations, which is a more elegant and scalable approach.
- The paper conducts extensive analysis, such as comparisons with SFT and the importance of hard negatives, which helps to better understand why SeVa is effective.
limitations: limitations: - I've noticed a phenomenon where after DPO, the outputs of the model seem to have become longer. Could this affect the evaluation metrics, especially those that rely on LLM evaluation, as LLM might prefer longer answers?
- The augmentation for creating negative examples seems to be degradation operations from the CV field. Could adding more purposeful and interpretable operations, such as replacing objects or changing attributes,  potentially enhance the performance, as this would enhance the model's accurate understanding of visual content?
paper_summary: paper_summary: This paper presents a novel self-supervised approach to visual preference alignment, which aims to address the preference alignment problem in Visual-Language Models (VLMs) in an unsupervised context. Researchers generate chosen and rejected responses based on the original and augmented image pairs, respectively, and carry out preference alignment employing direct preference optimization (DPO). The underlying principle involves the use of carefully designed image augmentations that lead VLMs to generate false yet challenging negative responses. This, in turn, facilitates the model's learning process, enabling it to produce more robust and potent answers. The entire procedure is no longer dependent on supervision from GPT-4 or human involvement during the alignment phase, and it is highly efficient, requiring only a few lines of code to execute effectively.
strength_weakness: strength_weakness: 1. This work marks the first attempt at unsupervised preference alignment within VLMs, breaking free from reliance on either GPT-4 or expensive human-annotated data, charting a new path for preference alignment.

2. The proposed SeVa methodology is achievable with just a few lines of code, and introduces an efficient data construction pipeline.

3. Through abundant experimental data and illustrative visualization results, the paper extensively demonstrates SeVa's capability to enhance VLMs across various dimensions.
limitations: limitations: This paper innovatively presents a self-supervised method for visual preference alignment and validates its efficacy across several pivotal benchmarks. Nevertheless, there remains room for further refinement and deeper investigation, especially in terms of experimental details, selection of data augmentation strategies, choice of evaluation metrics, and theoretical elucidation.

1. The details of the experimental setup are not sufficiently clear, including the criteria for selecting specific image augmentation strategies and the precise implementation details of DPO. Moreover, the model diagram is too simple and should show the model network structure and the pipeline of data processing in the model.

2. The research does not offer a systematic approach to selecting the most appropriate augmentation strategies for different tasks or datasets. This may lead to increased practical difficulty in finding optimal augmentation when applied in new environments.

3. The evaluation of GPT-4 itself may also introduce some subjectivity, especially for complex reasoning tasks. Additionally, the experiments may overlook neutral responses where the model fails to distinctly differentiate between superior and inferior options, which may affect the comprehensiveness of model performance evaluations.

4. Although the authors propose SeVa as a special form of single-negative sample contrastive learning, the theoretical exploration remains superficial with no rigorous proof, thereby limiting a deep understanding of its working principles.
paper_summary: paper_summary: This paper introduces a data augmentation method specifically designed for self-supervised learning, which is particularly aimed at large language models (LLMs). They propose a method that employs the strategy of rejecting generated samples, which are the same answer with the chosen data, to enhance the quality of augmentation. This approach achieves significant improvement on MMVet and $LLaVA^{W}$. The ablation studies show that the generation, consistency score, hyper-parameter in LoRA, and hard-negatives.
strength_weakness: strength_weakness: 1) This method is easily understand and reproduce.
2) The self-supervised method can be adapted to other multi-modal areas.
3) The approach leverages a few samples to make a great improvement on same datasets. (e.g., 16k vs. 102k on POPE).
4) The authors conduct substantial compared experiments on different metric and with different setting, and the results are encouraging.
limitations: limitations: 1) This approach is only conducted on LLaVA, which cannot demonstrates its generation on different models.
2) It is appreciate for the comparison experiments about data scale (8k and 16k in Table 3, and generation in Table 4). Although the motivation is to maximize performance with minimal data usage, the upper bond performance of this method is unclear for readers on the data scale metric.

Typo errors:
The quotation mark in the caption of Figure 2.