Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach
Authors: Shijian Deng
Wentian Zhao
Yu-Jhe Li
Kun Wan
Daniel Miranda
Ajinkya Kale
Yapeng Tian
Venue: COLM 2025 (2025)
Keywords: Self-Improvement
Multimodal Large Language Models
TL;DR: A novel judge-free self-improvement framework for multimodal large language models (MLLMs) efficiently enhances reliability by controlling hallucinations without costly model-level verification loops.

Abstract: Self-improvement in multimodal large language models (MLLMs) is crucial for enhancing their reliability and robustness. However, current methods often rely heavily on MLLMs themselves as judges, leading to high computational costs and potential pitfalls like reward hacking and model collapse.
This paper introduces a novel, model-level judge-free self-improvement framework. Our approach employs a controlled feedback mechanism while eliminating the need for MLLMs in the verification loop. We generate preference learning pairs using a controllable hallucination mechanism and optimize data quality by leveraging lightweight, contrastive language-image encoders to evaluate and reverse pairs when necessary.
Evaluations across public benchmarks and our newly introduced IC dataset, designed to challenge hallucination control, demonstrate that our model outperforms conventional techniques. We achieve superior precision and recall with significantly lower computational demands. This method offers an efficient pathway to scalable self-improvement in MLLMs, balancing performance gains with reduced resource requirements.

Published as a conference paper at COLM 2025
Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach
Shijian Deng' Wentian Zhao²
Yu-Jhe Li2
Kun Wan
Daniel Miranda²
Ajinkya Kale²
Yapeng Tian'
'The University of Texas Dallas Adobe Inc.
{shi jian. deng, yapeng. tian (lann@utin @utdallas edu, (wezhao, jhel, kuwan, miranda, akale @@adobe.com
Abstract
Self-improvement in multimodal large language models (MLLMs) is crucial for enhancing their reliability and robustness. However, current methods often rely heavily on MLLMs themselves judges, leading to high compu- tational costs and potential pitfalls like reward hacking and model collapse. This paper introduces novel, model-level judge-free self-improvement framework. Our approach employs a controlled feedback mechanism while eliminating the need for MLLMs in the verification loop. We generate preference learning pairs using controllable hallucination mechanism and optimize data quality by leveraging lightweight, contrastive language- image encoders to evaluate and reverse pairs when necessary. Evaluations across public benchmarks and our newly introduced cataset, designed to challenge hallucination control, demonstrate that model outper- forms conventional techniques. We achieve superior precision and recall with significantly lower computational demands. This method offers an efficient pathway to scalable self-improvement in MLLMs, balancing per- formance gains with reduced resource requirements. The code is available at https:/ github.com/ ShijianDeng/ Ee////SSIMMMM...
1 Introduction
cier Saforpowaeer
Frethokk
Tredeck
(8) Conventional Improvement
(b)
Sell-improvement
(e) Efficient Self-improvement
(d)
Comparing Paradigms
Figure 1: Comparison of three different improvement paradigms. (a) The conventional improvement paradigm requires humans annotate feedback data and feed it into the model for improvement, making it the least efficient approach. (b) The self-improvement paradigm leverages the model itself to provide feedback; however, this approach is still inefficient due to the high cost and potential bias of using large models as verifiers. (c)
Our efficient rel--improvement paradigm improves the model without human feedback or model-level self- fef-ffee by using predefined data generation strategy combined with lightweight verifier, achieving both efficiency and performance improvement. (d) Among all three paradigms, our approach offers the best trade-off between performance and cost.
Self-improvement is natural way for humans to learn independently, enabling them to acquire knowledge and skills beyond what they learn from their teachers. This same paradigm is being gradually adapted for large language models (LLMs) and multimodal large language models (MLLMs) to achieve performance improvements beyond the seed model with minimal human supervision.

---

Published as a conference paper at COLM 2025
Recent studies have explored various approaches (Favero et al., 2024; Zhou et al., 2024; Deng et al., 2024; Yu et al., 2024b) for self-improvement in MLLMs. For instance, RLAIF- V (Yu et al., 2024b) uses MLLMs to evaluate and score responses generated by another MILM, creating preference learning pairs from responses to the same image and question. M3ID (Favero et al., 2024), POVID (Zhou et al., 2024), and STIC (Deng et al., 2024) employ techniques like bad prompts, image corruption, unconditioned generation, and response injection to generate hallucinated responses as negative samples for preference learning.
However, several issues limit this paradigm: 1) it relies heavily on the quality of the verifier (e.g,, reward model); 2) the process can be resource-intensive, generating numerous samples but only using a tiny subset; 3) the cost multiplies when another large model is needed for verification, especially when generating reasoning or comments for final evaluation. Past studies (Valmeekam et al., 2024a;b) have underscored the necessity of an external verifier.
To overcome these challenges, we propose an alternative approach, illustrated in Fig 1, enabling self-improvement without directly using an MLLM as a verifier for dataset filter- ing. Our method involves controlled hallucination to generate preference learning pairs, lightweight evaluation with a contrastive language-imaae encoder to optimize data quality, and direct preference optimization (DPO) (Rafailov et al., 2024) to train the seed model.
First, we employ an efficient, controllable approach to generate simple negative or hard- negative samples, thereby creating the initial preference learning pairs. We use controller that ranges from 0 to to regulate the level of hallucination in responses. After generating the initial dataset, we leverage lightweight, contrastive aanguage-image pretrained encoder to compute the average sentence-level CLIPScore (Hessel et al., 2021). This score is used to identify and update pairs in which the negative sample's 's score exceeds that of the positive sample by set threshold, thereby refining our preference learning dataset. Finally, we use the optimized datasst train the seed model via DPO, resulting in a self-improved model. Extensive evaluations on both in-house and public benchmarks demonstrate significant gains over the original seed model.
Our primary contributions are as follows: 1) We propose novel and efficient framework for self-improvement in MLLMs that: (a) combines a predefined, controllable mechanism for efficient negative sample generation, and (b) uses a lightweight verifier to effectively control positive and negative pairs, automatically reversing them when necessary. 2) We collected new IC dataset, which includes GPT-4o assisted evaluation of both precision and recall of MLLMs. 3) Experimental results demonstrate that we achieve significantly better performance over the seed model on both our IC dataset and other public datasets.
2 Related Work
2.1 Multimodal Large Language Models
To leverage the knowledge and reasoning capabilities of LLMs in multimodal settings and address broad multimodal comprehension challenges, MILMs have been developed. Significant work has been done in this field, such as LLaVA (Liu et al., 2024c), which connects CLIP with the LLaMA model through an adapter; Qwen-VL (Bai et al., 2023), which implements grounding and text-reading abilities by aligning image-caption-box tuples; CogVLM (Wang et al., 2023b), which uses trainable visual expert module in the attention and FFN layers to enable deep fusion of vision-language features without sacrificing NLP task performance; VL (Chen et al., 2024b), which employs both contrastive and generative tasks to better align the large-scale vision foundation model with MLLM; Pixtral (Agrawal et al., 2024), which processes images through the vision encoder at their native resolution and aspect ratio, converting them into image tokens for each patch in the image, allowing it to handle any number of images of arbitrary sizes in its large context window; and LLaMA3.2 Vision, which incorporates LLaMA (Dubey et al., 2024) via a compositional approach to ensure that text-only task performance is not affected by the addition of visual-recognitionn capabilities.
capabilities into
2

---

Published as a conference paper at COLM 2025
2.2 Self-Improvement
Even after large-scale pretraining, instruction tuning, and reinforcement learning from human feedback (RLHF) (Quyang et al., 2022), large models may still show vulnerabilities in various cases, Although new data can always be prepared to improve specific missing capability of the model, this is not a sustainable long-term solution to fix all issues. To enhance large model's helpfulness and trustworthiness without exhausting human effort, a new self-improvement paradigm has been adopted, as systematically discussed in the survey (Tao et al., 2024). For MLLMs, this often involves two key steps: sampling and verification.
Sampling. To improve the seed model's performance, the first step is to sample the necessary data. The simplest approach is to change seeds and randomly sample large number of outputs, though this may not be efficient. Instead, users can predefine the type of data to generate by employing improved prompts and chains of thought to produce high-quality data, or by using corrupted images, attention masks, and text generate negative data, as explored in POVID (Zhou et al., 2024), STIC (Deng et al., 2024), and BDHS (Amirloo et al., 2024). In M3ID (Favero et al., 2024), the authors also use mutual information from information theory to better control the quality of generated outputs and therefore achieve more effective sampling. In our work, we further simplify this sampling approach, making the process more practical for optimizing the margin (Deng et al., 2025) in preference learning.
The model would not significantly improve if it simply reuses any generated data for retraining. A more effective approach is to perform data selection before training. There are many ways to achieve this. The simplest method is majority voting, though this may fail when the correct output is not the most common. A verifier, while optional, is commonly used as an additional quality control layer for data. The most straightforward and widely used verification method is to use an MLLM as reward model, as seen in RLAIF-V (Yu et al., 2024b). However, this approach has limitations related cost and potential bias due to the reward models' own limitations. An external verifier can help address these issues. For example, CLIP-DPO (Ouali et al., 2024) utilizes CLIP to rank short descriptions generated by the MLLM. We adopted a similar approach and extended it to suit long captions, seamlessly integrating it into our self-improvemmnt framework along with our sampling methods to further enhance the robustness of our pipeline.
Verification.
3 Method
This section describes our approach to efficient self-improvement in MLLMs. We begin with description of our controllable method for generating positive and negative data pairs for training. Next, we highlight the importance of incorporating lightweight quality control mechanism to ensure that the generated data effectively guides the learning process. Finally, we explain how the generated data is used to train the seed model with DPO, culminating in self-improved model.
3.1 Motivation
To perform well, preference learning requires diverse data and accurate preference labels for each pair, making it critical to establish a fully controllable approach for generating the required dataset. While it is challenging to produce data that surpasses the quality of what the seed model can generate, it is relatively feasible to create data that worse than what the model can typically produce. It is also important to know how much worse the sample we need before we generate it since both too hard or too simple pairs may not work best. Based on these observations, propose simple yet efficient method for generating preference learning data pairs with any difference level between positive samples and negative samples.
The high computational cost of running models with a large number of parameters, com- bined with the inherent inductive biases of MLLMs, imposes significant limitations
3

---

Published as a conference paper at COLM 2025
Suptram
S
Seed
Pootiw Captoo
Token Dastbulcn Imagee
Instraciion:
Deecitte image-i wn debei
Per Seep When Necwzary
Final Tokan Dastitution
Sy
Bood
Bowk Inpul
bution
Image
(a)
Data Generation
(b) Verification
(c) DPO
Figure 2: Overview of our framework. Our efficient self-improvement framework combines two main strategies: (a) We use a simple yet effective predefined preference dataset genera- tion approach, employing two decoding paths during response generation. By adjusting the hallucination ratio fratio, we can control whether a negative or positive sample is generated for preference learning. (b) After the initial preferences are generated, we use lightweight contrastive language-image pretrained encoder to calculate the average sentence-leve CLIP-score difference between the initial positive and negative samples, swapping them when necessary to ensure the quality of the final preference dataset. (c) Finally, we apply DPO with the resulting dataset to improve the model.
relying on large models as verifiers. To address these challenges, we introduce an objective and lightweight alternative for verification purposes.
3.2 Controllable Dataset Generation
To train self-improving model with preference learning, we first need to prepare a suitable dataset. To generate preference pairs, we use the seed model mo to create a positive response Yw and a negative response yi from the same input image Ximg and instruction Xinstruct- To obtain the negative sample, we introduce interventions during the decoding process of the MILM. We use two decoding paths: a conditional path Pe that generates a response based on both the input image Xing and instruction Ximstruct, and an unconditional path Pu that uses only the instruction Ximstructr without the image Xing- The generation is controlled by the hallucination ratio, i.e., Nratio, which determines the level of hallucination to be injected into the generated caption, ranging from to 1. A higher Itatio denotes injecting more hallucinations into the response.
As shown in Fig. 2, for each output token, the distribution is determined by combining the token distribution l. from the conditional path and the token distribution tu from the unconditional path, weighted by the hallucination ratio (i.e., Iratio):
f = (1 - Nratio) to + Rratio fu-
(1)
The two paths, Pe and Pu, do not interact, ensuring that the unconditional path never accesses any information from the input image, thus serving as a "pure" hallucination source. Each pair is initially labeled, with the response generated under the lower hratio assigned as positive and the other as negative. The hratio follows a predefined distribution, such as uniform or Gaussian, and remains fixed for each decoding process once assigned.
3.3 Lightweight Preference Data Inversion
Although the generated pairs initially have assigned positive negative labels, these labels may not always be accurate, as the conditional generation process with the seed MLLM can sometimes introduce certain level of hallucination in the decoded text. To address this,
4

---

Published as a conference paper at COLM 2025
Algorithm 1 Efficient Self-Improving MLLM with Preference Learning
Require: Seed model Mhy dataset ((rigg Mnstmme))
Ensure: Improved model it)
for cach (Ximg. Tinstuct) in dataset do
Sample hallucination ratio Natio [0, from predefined distribution S: for each time step do
Compute conditional
distribution Po(.e Ximg- -inetrcc Mett
5: Compute unconditional token distribution Pu(h Xinstruch
Compute final distribution (1 Resio) f Pratio x t
7:
Sample token ~ end for
9: Obtain responses View (lower firatu) and Yhigh (higher fraauu)
10: Assign initial labels: positive response yi = Moow negative response yf
Mhigh
11: Compute average CLIP scores CLIP scorel, and CLIP scoref
12.
if CLIP ..caPl. -LiPPscone threskuid then
13:
Swap Y' and
14.
end if
15:
final pair (Xing Xmstract yar to preference dataset D 16. end for
subset Das
based Coooesccm difference
18. Initialize improved model my <-
19. each (x,y", in Duub [log ( 1og) mp(y/ [x]] [logmoly"" 1x) log
Compute A(x,y".,'500
21: Compute loss 7((9) logor (A(x,"",y;;) .y:0))
22. Update model parameters by minimizing ((0)
end for
implement an additional quality control step to manage cases where initial labeling may be incorrect.
Specifically, we use a lightweight CLIP model, which the vision-language contrastive pretrained encoder of the MLLM. For each initial pair (y we calculate the CLIP -score between the image and each decoded sentence. Since CLIP has 77-token limit and cannot accommodate overly long captions, we compute the average sentence-level CLIP scores for the initial positive caption,
and the initial negative caption, CLIP LcllPsscoree.. CLIP score - CLIP score/ < threshold, indicating that the initial positive is rated much lower than the initial negative, we swap the preference labels, designating yi, as the final negative y/ and yj as the final positive y Otherwise, we retain the original order the final pair.
This process prevents cases where an initial negative sample might outperform its counter- part, which could undermine subsequent preference learning. After this step, we obtain the final preference pairs (ylwr y/) which are used in preference alignment training to improve the seed model my.
3.4 Preference Learning Finetuning
After obtaining the final pairs of positive caption y and negative caption of generated from the same input image Ximg and instruction Xinstruct, we select a subset of the pref- erence dataset within a certain range of informativeness (Wu et al., 2024), defined by the CLIP score difference: CLIP score CLIP score,, forming Duub- We then DPO, commonly used, low-cost alternative to PPO, to train the seed model mo, further enhancing its performance. Through this finetuning process, we obtain an improved model my which is self-improved from the seed model my using its own generated dataset. The detailed process is illustrated in Fig. 2 and Algorithm
4 Experiments
To evaluate the effectiveness of our proposed self-improvement framework, we tested it both our IC dataset using GPT-4o series evaluation and other commonly used benchmarks. We introduce the experimental settings for dataset generation and verification, followed by

---

Published as a conference paper at COLM 2025
detailed analysis of results and ablation studies to demonstrate the effectiveness of our framework and each of its design modules.
4.1 Datasets
IC Dataset. Current hallucination benchmarks primarily evaluate the precision of captions while often ignoring recall. To comprehensively assess MLLMs' captioning abilities, we have collected a new dataset containing 150 challenging images prone to hallucination across wide range of domains and scenarios. These include abstract concepts, animals, animations, artistic content, common sense violations, documents, events, fashion, food, handwriting, illustrations, objects, people, posters, scenes, technology, and vehicles.
After generating captions, we use the GPT-4o series to evaluate them based on precision (elements in the caption that are present in the image) and recall (elements in the image that are captured in the caption) to calculate a final Fl score, which serves as a measure of caption quality. Some examples
shown in Fig. 6.
Public Benchmarks. Besides our newly collected IC dataset, we also selected Object Hal- Bench (Rohrbach al., 2018), classic benchmark that focuses on evaluating object-level hallucination in vision-language models. To further explore the generalizability of our framework, we incorporated Qwen2-VL (Wang et al., 2024), along with LLaVA, evalu- ate their performance on additional benchmarks, including AMBER (Wang et al., 2023a), MMHal-Bench (Sun al., 2023), MMStar (Chen al., 2024a), GQA (Hudson & Manning, 2019), OCRVQA (Mishra et al., 2019), Math (Lu et al., 2024), and ChartQA (Masry et al., 2022). These benchmarks cover various tasks such as VQA, OCR, math, and chart understanding.
4.2 Experiment Setup
For the seed model mo, we used LLAVA-1.5-13B (Liu et al., 2024a), popular and representa- tive MILMM. An 8xA100 node with 80GB VRAM per GPU was used for DPO training, while data generation and other processes were performed on single GPU. We trained the model for 672 steps with a learning rate of 5e -7
During data generation, we sampled 100k images from the LLaVA instruction tuning dataset, Ilava_v1.5 mix665,, and removed all question-answer pairs. Using the prompt "Describe image in detail," " the model generated responses with an Atatio ranging from 0 to 1. Initially, captions generated with lower hratio were assigned the initial positive samples, yo while captions generated from the same inputs Xinstruct and Ximg with higher Matio were assigned as the initial negative samples, y/. This process resulted in 100k initial preference pairs.
For the obtained caption pairs, each sentence was extracted, and the CLIP model was used to compute the CLIP_scon for each image-caption pair. For sentences longer than the CLIP model's context limit, we we split them into shorter sub-sentences, computed their respective CLIP scores, and calculated the average CLIP score for each caption by averaging the scores from all sentences and sub-sentences. If the average CLIP_score score higher than that of the positive caption, we swapped the positive and negative samples. The pairs were then sorted by CLIP_scOn sccore difference, from low to high, and organized into 10 splits, each containing 10k pairs. For each split, we trained a LLa VA model and conducted inference on the IC dataset and other benchmarks to gather results. For the IC dataset, GPT-4o was used as the evaluator to compute precision, recall, and F1 score.
negative caption was
4.3 Results
With the improved model mı derived from the original seed model mo, we evaluated performance across different benchmarks and presented the results in Tab and Tab 2. As shown, the self-improved model outperforms previous models on both benchmarks. In particular, compared to the original seed model LLaVA-115-13BB, performance has improved significantly, clearly demonstrating the effectiveness of our framework. Compared to

---

Published as a conference paper at COLM 2025
LLaVA 1.5 18b
ratlo: 0.1 Reconstruct
Ours Reconstruct
Original
Reconstrust
Figure 3: Image reconstruction examples. To further demonstrate the effectiveness of our training framework, we use a text-to-image diffusion model, DALL-E 3, to convert captions generated by the models back into images. The reconstructed image from the original model's caption contains significant hallucination, while increasing the Prttio during genera- tion produces negative caption that, when reconstructed, shows even more hallucination in attributes like style and emotion. However, after training with these generated caption pairs, the reconstructed image from the improved model's caption closely resembles the original, surpassing both the positive and negative samples.
Model
Size
Feedback
Object HallBench
Resp.
Ment.
VCD (Leng et al,, 2024)
7B
No
48.8
24.3
OPERA (Huang
2024)
7B
No
45.1
22.3
Less-is-more (Yue et 2024)
No
40.3
17.8
LURE (Zhou et al, 2023)
7B
No
27.7
17.3
OWEN-VL. (Bai et 2023)
MiniGemini (Ti et 2024)
2024)
34B
No
14.5
8.0
LaVA-NeXT NeXT (Liu
2024b)
34B
No
12.6
6,4
HA-DPO (Zhao et al,, 2023)
7B
Rule
39.9
19,9
POVID (Zhou al., 2024)
7B
Rule
48.1
24.4
Silkie eLi 2023)
108
GPT-VV
27.1
13.4
LLaVA-RLHF (Sun et al., 2023)
138
Human
38.1
18.9
RLHE-V al., 2024a)
Human
LLaVA-1.5 (Liu et al,, 2024a)
7B
No
53.6
25.2
[LaVA-1.5 (Liu al,, 2024a)
13B
No
51.6
24.6
+ Ours
138
Self-Ffficiency
9.4
5.1
Table Main results of our experiments on Object HallBench. Comparison of various models across different metrics. Resp. indicates the response-level metric, and Ment. represents the mention-level metric. The best results are highlighted.
previous methods, ours is the first to emphasize an efficient self-improvement approach that balances efficiency and effectiveness.
With different CLIP score difference pairs generated with hatio sampled from a Gaussian distribution, we also present the ablation study results in Fig. 4. We observe clear perfor- mance gain for each component added to our framework, compared to the seed model mo and the model without that component, shown in Fig. 5.
In Fig. 6, we show qualitative results to demonstrate the differences between our self- improved model and the original seed model. We also use the generated captions to perform image reconstruction with the DALL-E 3 model, as shown in Fig. 3. For other benchmarks, results are presented in Tab. 3.
4.4 Experimental Analysis
From the comprehensive evaluation results, we observe that our self-improved model shows significant performance gains across various benchmarks compared the initial seed model in all evaluation dimensions. Here are some detailed discussions and findings from our experiments:
Our model performs substantially better than the initial seed model, as demonstrated by both quantitative and qualitative results. In Tab. 1, our model achieves scores of 9.4 for object response level and 5.1 for mention level, ranking it among the best of all models. These results on popular public benchmark for evaluating hallucination demonstrate that our model outperforms all others at multiple evaluation levels, despite not using additional
7

---

Published as a conference paper at COLM 2025
Medel
Precision
Recall F1
Change
1LaVA-15-1156
6.6
0.00
sandom negative somples TPPO
Ours
7.74
7.76
Table 2: Main results on the IC dataset. For the random negative samples DPO method, the model generates random captions without an image to serve as negative samples; these are paired with the original captions (positive samples) to train an improved model using DPÓ. Precision measures how many elements in the caption are also in the image (higher scores indicate lower hallucination in the caption). Recall measures how many elements in the image are included in the caption, providing a complementary metric for hallucination evaluation. F1 is the harmonic mean of precision and recall. All scores are on scale from (worst) to 10 (best). The best scores are highlighted.
Performance Comparison DPO Training with Gaussian Distribution
Best F1: 7.76
Precision
Recall
7.5
Score (%)
7.0
6.5
Basaline
Spiit
Spilt 2
Spit
Split 4
Sprit
Spiit 6
Spilt 7
Spii
Spitt g
Spiit 10
Model
Figure 4: Performance comparison of DPO training using various _LIP_score generated with lryaio sampled from Gaussian distribution and ranked from low to high. For these experiments, we set μ = 0.5 and o = 0.15.
score differences
human feedback during finetuning or requiring feedback from an MLLM. Instead, it relies on lightweight CLIP encoder, highlighting the efficiency and effectiveness of our proposed framework.
Each of our designed modules contributes to performance improvement. As shown in Fig. 5, using Iraaio to generate preference learning pairs and fine-tuning the seed model with DPO result in substantial gain in both precision and recall compared to the original model. Adding CLIP-score _ccore difference filtering to exclude pairs with negative differences further enhances the model's performance. Instead of discarding these pairs, swapping the positive and negative samples when their CLIP score difference is negative leads to another notable improvement. This highlights the necessity of a lightweight, post-hoc guardrail with CCorescore score difference to further boost performance. Finally, rather than using a fixed Bratio, we experimented with more diverse approach by randomly sampling Iratio values from a Gaussian distribution. This method introduces greater diversity, likely because encompasses broader range of cases, making the dataset more generalizable. This approach further improves the model's performance by increasing the variety of negative samples.
The visual-language correspondence difference matters. To evaluate model performance across settings with different preference pair combinations, we conducted extensive ex- periments varying the average sentence-level CLIP sccre differences using Gaussian dis- tribution sampling (see Fig. 4, highlighting structured variability). The results indicate that selecting an optimal CLIP score difference is critical, with performance peaking when differences are moderate: neither too large nor too small. This aligns with human learning patterns, where understanding improves most when examples are distinct enough to differ- entiate yet similar enough to allow meaningful comparisons. This finding also matches the data selection theory in recent research (Deng et al., 2025).
8

---

Published as a conference paper at COLM 2025
AMBER
HallusionBench
MMStar
GQA
OCRVQA
MathVista
CharlQA
LLaVA-L3-138
52.09
26.70
Ours
82.35
50.79
0.36
54.08
60.25
28.00
13.84
Qwen2VI.228
62.79
Ours
83.41
62.46
0.47
99.87
68.98
48.00
73.88
QwenZVL-7B
59.50
+ Ours
85.81
68.03
0.60
62.47
71.91
60.10
81.24
Table 3: Performance comparison with more models across other benchmarks. Best perfor- mance for each model is highlighted.
Precision, Recall, and Fl. Scores with Gains Relative to Previous Model
Prectdion
7.75
M00 Cain
7.25
seed moitel
with rate
remove pala
reveree pañlo
Model
Figure 5: Comparison of model performance when adding certain mechanisms in our framework. The striped sections indicate the performance gain after adding each module. We use LLaVA-1.5-13B
the seed model. The modules gradually added to the seed model are: using Azatio, removing pairs with negative CLIP_score _ccre differences, swapping pairs instead of removing them, and adding Gaussian distribution sampling. Each module our design contributes to the final performance.
Hallucination reduction and better reconstructions. From the qualitative examples in Fig. 6 and the reconstruction results in Fig. 3, we observe that while the seed model tends to hallucinate significantly, our model generates far more accurate content when provided with the same image and text prompt. These results also demonstrate how hallucinations can impair the reconstruction of an original image given a caption from hallucination-prom model, and how our approach mitigates this issue. This could potentially contribute to building better reconstruction or generation models by using captions generated by our model.
Generalization to Other Tasks. As we observed in Tab. 3, the results confirm that our self-improved model generalizes well across wide range of tasks, including OCR, VQA, ChartQA, and even for visual reasoning tasks such as Math Vista. This demonstrates that the gains from our framework benefit broader domain.
Limitations and Future Work
Although our experiments demonstrate that our framework is highly effective for enhancing the initial model's performance, we acknowledge some limitations and highlight areas for exploration and improvement in future work.
Advanced Guardrail. Our framework's 's primary reward signal operates on the assumption that higher Aratio indicates worse response. CLIP -core serves secondary role guardrail to handle instances where this assumption fails. Consequently, a more robust guardrail than CLIP score would ultimately improve the model's performance.

---

Published as a conference paper at COLM 2025
image feat and dog playing logether grassy chasing The ball
mage woman
posilioned lowards the side the with the standing
the left side.
middle
playing
grassy
located
standing
There are a fow other people in the background, but they the main foous of the acene. The yard surrounded fenco, which adds the cusdoor satting The miverall aimosshhnne of the image fivily and fun, as the worsen and the dag enjoy their playimm logadite air.
weichiing the dogg rure towerns the ball. The scine oaplures fun and enargittic moment behween the worsen and her dog, they anjoy playing togethor outtooos.
ANGRY
0
The image features a woman with curly halt making surpnsed or shocked expression, She appears to be reacingg something uneopected eurpriing The main focus image, expression dramatic.
The imaga features a woman with surpr sed expression her toos, as the is reading to something unaxpected The womem's face is displayed computer screen, emolions baing messuree by a solhoarn program, program shows emations percentage, with
SAD
0
her
SURPRISE
49
FEAR
NEUTRAL
figuree, other background, These figures additional the
background,
surprice. woman's facial express surprise, makinng quite dramatic.
50
98%
Input Image
LLaVA 1.5 12b
LLaWK 13b Ours
Figure 6: Examples of qualitative results. With the same input image and instruction prompt "Describe image in detail", the caption generated by the original seed model, LLaVA-1.5- 13B, contains many hallucinations. In contrast, the model trained through our efficient self-improvement framework describes the image accurately, without hallucinated content. Hallucinated content is highlighted in red, and accurate content is highlighted in blue for easy identification.
Recursive Self-mmprovement. Due to limited resources, we were unable to investigate whether recursive self-improvement is feasible by iteratively applying our framework in multiple rounds, from data generation to preference learning finetuning, to go beyond Mi and potentially achieve 112,, MM3, and so on. This could reveal whether further improvements are possible or if an upper performance bound exists.
Scaling with Larger Models and Datasets. Because of training costs, we were unable to experiment with even larger models or larger datasets. Exploring the scaling laws of the framework with additional resources would be an interesting avenue for future research.
Extending to Other Modalities. Although our experiments focused solely on vision- language tasks, the framework should be extendable to other modalities, such as video and audio. These directions present promising topics for future exploration.
Conclusion
In this paper, we propose a novel and efficient self-improvement framework for MLLMs that does not require model-level self-feedback. We demonstrate that using our methods: 1) Significantly reduce hallucinations, enhance image-caption correspondence, and generalize to other domains, including visual reasoning, compared to the original seed model across different benchmarks. 2) Our approach enables precise control over the pair generation process, allowing us to efficiently generate preference pairs with any desired level of difference between samples. 3) We prevent cases where the positive sample is much worse than the negative one by using a lightweight CLIP model to flip samples when the score difference is too negative. Unlike traditional self-improvement methods, our approach dramatically reduces the number of parameters required during the verification process by eliminating the need for model-level judge. Extensive experiments demonstrate that our framework effectively balances superior performance and efficiency. We hope our work inspires new strategies for managing trade-offs in the self-improvement process for MLLMss.
10

---

Published as a conference paper at COLM 2025
References
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chud- novsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, et al. Pixtral 12b. Xiv preprint ar Xf1::2410007073, 2024.
Elmira A Jean-Philippe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, et al. Understanding alignment multimodal Ilms: comprehensive study, arXiv preprint arXivx2207.002 02247777 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arviv preprint arXiv:2308.11990 2023.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arviv preprint ar Stt:::20032233300 2024a.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24111 24198, 2024b.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507 006201,06666,. 2025.
Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangnan He. Less is more: Improving Ilm alignment via preference data selection. arXiv preprint ar X10:2550..14560, 2025.
Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. Enhancing large vision language models with self-training on image comprehen- sion. arXiv preprint ar 2rXio:2405.11777 19776,, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arxiv preprint or Xiv:2407 217883 2024.
Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13303-14312, 2024.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint ar arXiv:2104.08715, 2021.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allcaatie IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
In Proceedings of the
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 6700-6709, 2019.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. Xiv preprint ar Xii::441002222 2024.
11

---

Published as a conference paper at COLM 2025
Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du Faithscore: Fine-grained evalua- tions of hallucinations in large vision-language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 5042-5063, 2024.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large lision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13872-13882, 2024.
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. ar Xiv preprint ar Xit::3122.10666 2023.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. ar Xiu preprint ar Xrix:2403.188/ 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024c.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathe- matical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 findings-acl. 177.
aindings-acl.177. URL https: http:///aaaattooo.oorrrro
Anand Mishra, Shashank Shekhar, Aject Kumar Singh, and Anirban Chakraborty, Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019.
Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos, Clip-dpo: Vision- language models as a source of preference for fixing hallucinations in Ivims. ar Xiv preprint 1:Xi::2408.1033,, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-277441 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward p model. Advances in Neural Information Processing Systems, 36, 2024.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. Xix preprint ar arXit:1809.02156 2018.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yas Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. ar preprint ar Xiv::9330 1ee00. 2023.
Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou. A survey on self-evolution of large language models. arXiv preprint arXio:2404.1438) 2024.
12

---

Published as a conference paper at COLM 2025
Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Systems, 36, 2024a.
Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati, Lims still can't plan; can lrms? preliminary evaluation of openai's ol on planbench. arXio preprint #rXio:2409.13373, 2024b.
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An llm-free multi-dimensional benchmark for milms hallucina- fion evaluation. ar Xio preprint ar Xxi::331..03997 2023a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vi: Enhanc- ing vision-language model's perception of the world at any resolution. arviv preprint arXio:240.112991 2024.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogylm: Visual expert for pretrained language models. Xiv preprint ar axXiv:2311.0307, 2023b.
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. B-dpo: Direct preference optimization with dynamic B. Advances in Neural Information Processing Systems, 37:129944 1299666, 2024.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVE Conference on Computer Vision and Patlern Recognition, pp- 13807-13816, 2024a.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open- source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.172200 2024b.
Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigating multimodal hallucination from an eos decision perspective. arXiv preprint 1454:2240..14455, 2024.
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing Ivlms through hallucination-aware direct preference optimization. arv preprint ar xxXio:2311.16839, 2023.
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint ar XxX::22100.0054, 2023.
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. ar Xiv preprint vxXi=2242..14411 2024.
13

---

Published as a conference paper at COLM 2025
A Ablation Studies with Different rratio Strategies
For a fixed patio during dataset generation, we present the ablation study results in Fig. 7, which illustrate performance trends. When using uniform distribution, obtained the experimental results shown in Fig. 8, which illustrate variations.
Performance Comparison of DPO Training with h=0.0, h=0.1
8.0
Precision
Best Fl: 707
F1
7.5
Score (%6
7.0
Baseiie
Spii
Splr
Spilt
Split
Spiit
Spilt
Spiit
Spiit
Spiit
Split 10
Model
Figure 7: Performance comparison of DPO training using various CLIP score differences generated with hratio = 0.0 and ratio = 0.1, ranked from low to high. The best performance highlighted.
Performance Comparison of DPO Training with Uniform Distribution
Precision
F1
Score
Baseline
Spit
Split
Spift
Spit
Spiit
Spilt
Spit
Spiit
Spiit
Spli 10
Model
Figure 8: Performance comparison of DPO training using various CLIP score differences generated with /rtaii sampled from a uniform distribution and ranked from low to high. The best performance is highlighted.
We also conducted experiments with different combinations of the threshold and CLIP score, as shown in Fig. 9.
B CLIP Reliability for Long Sentences
known limitation of contrastive llsguugla models like CLIP their fixed context length (77 tokens), which can challenge the evaluation of long, descriptive captions. To address this, our framework employs robust atomic claim splitting strategy similar to recent work (Jing et al., 2024; Yu et al., 2024b). This process, detailed in Algorithm 2, first splits a caption into sentences. Then, any sentence that still exceeds the token limit is broken information is lost and avoiding common iteration bugs. The final score for the entire caption is the average of the CLIP scores computed for each individual chunk against the image.
down further into the largest possible chunks of words that fit, ensuring
14

---

Published as a conference paper at COLM 2025
fatvvll
Figure 9: Comparing the performance over different combinations of a threshold and CLIP score, with Aratio sampled from Gaussian distribution. We observe that hallucina- tion reduction depends on the CLIP score threshold and is sensitive to larger deviations. Furthermore, after swapping, pairs with mid-level CLIP score difference tend to perform best, finding that aligns with recent work (Wu et al., 2024; Deng et al., 2025).
To quantitatively validate the reliability of this approach, especially for longer captions, we conducted a rigorous analysis comparing our lightweight CLiP-based verifier against judgments from two state-of-the-art MLLMs: GPT-4o (Hurst al., 2024) and Gemini 2.5 Pro (Comanici et al., 2025). We randomly sampled 600 preference pairs from our training data, stratified into four groups based on caption length, and measured the agreement rate on which caption in pair was superior. The agreement between our CLIP verifier and the SOTA MLLMs was compared baseline agreement between the two SOTA MILMs themselves.
The results, presented in Tab. 4, demonstrate a consistently high agreement rate that does not degrade for longer captions. Our CLIP-based verifier achieves up to 93.2% agreement with GIT-40, and its agreement level is comparable to the baseline agreement between GPT-4o and Gemini. This confirms that our sentence-level averaging strategy is robust and faithful method for evaluating long captions.
Furthermore, audited our full 100k training dataset to assess the prevalence of individual sentences exceeding the 77-token limit. We found this an extreme edge case, occurring in only 0.003% of the data (3 captions). This indicates that while our sub-sentence splitting mechanism is an effective safeguard, the primary sentence-level splitting is sufficient for the vast majority of cases. These findings validate that lightweight verification process effectively handles long captions without sacrificing scoring fidelity.
Table 4: Agreement analysis of our CLIP-based verifier against SOTA MLLMs (GPT-4o, Gemini 2.5 Pro) for captions of varying lengths. We sampled 150 preference pairs for each length category. The 'CLIP Agreement' columns show the percentage of pairs where our verifier's preference matches the MLLM's. The 'Baseline' shows the agreement between the two SOTA MLLMs. The results show our lightweight verifier's 's judgment is stable and highly correlated with SOTA models, even for long captions.
Caption Length (okena)
Sampled
CLIP Agreement
Agreement
Agreement
<77
18-104
89.7%
>132
87.9%
3.
83.3%
15

---

Published as a conference paper at COLM 2025
Algorithm 2 CLIP Scoring with Greedy Chunking
Require: Image Caption
Ensure: Average CLIP Score 5
1: Let Lmox <- 77 (Maximum token length for the text encoder]
2: Preprocess image: I' + Preprocess)()
3: Split caption into sentences: +- SentenceSplit((C
4: Initialize list of chunks: Cohunks < []
5: for each sentence $ E S do
6: if token length of I Lmax then
7: Add to Cehunks
8: else
9: Split sentence into words: W <- WordSplit(s)
10:
Initialize current chunk: Ceurr f- []
11:
for each word W do Form a test chunk by appending the new word: Ctest + Join( w) if token length of Ctest Lmax then
12:
14:
Append w to Cearr
else
16:
Add joined Cearr to Cchunks Start new chunk with the current word: Ceurr f- [z0]
18:
end if
end for
20: Add the last remaining chunk to Cehunks
21: end
22: end for
23: Initialize list of scores: S
24: for each chunk Cohunks do
25: Compute similarity score: Se <- CLIPSimilarity (I,cc
26: Add s. S
27: end for
Compute the average score: 5 Average(S)
29: return
C Unconditional Path vs. Empty String Prompt
We also conducted experiments to compare two strategies for generating negative samples: the unconditional path, which uses an instruction (e.g., "Describe image in detail") without an image input; and the empty string prompt, which uses an image with an empty string (" ") as the prompt. We analyzed responses from LLaVA-1.5-13B to both unconditional path and empty prompt inputs (n = 10, 000 each), providing a coverage analysis in Fig. 10 and t-SNE visualization in Fig. 11.
D Experimental Results on Efficiency
We included additional comparisons in terms of computational resource consumption and runtime efficiency, as shown in Tab. 5. All experiments were conducted using -.L.VA-1.5-13B single A6000 Ada GPU. Due to limited resources, provide estimated runtimes when full execution was not feasible. As can be seen from the results, our framework significantly reduces the resources and time needed, demonstrating greater efficiency compared to traditional self-improvement methods.
Method
Active Parameters
Runtime 10 minutes
GPU Requirements 892ViB
Ours
T49M
RLHF/RLAIE- 22,000M 104,000M
100 hrs
40560MiB
Table 5: Efficiency comparison of different methods.
16

---

Published as a conference paper at COLM 2025
Vocabulary Comparison
Response Length
20200
Uncondilana Emolyy fromps
0.00
0300
0.00
6000
De-sily
Undogn
0.04
4000
2000
2,114
Jhrandh
Empty Prmpt
50
100
150
200
Reiporic Length
Lexical Diversity Metrics
Semantic Category Usage
Uncondi onal Path
Uncondilara Path
Erppyy Promp:
Emssy fromst
12500
12090
Uniue
2500
5000
17R
MSTTR EnPoay
perapiees
gatial
eljets
People
Calegorie
Figure 10: Coverage Analysis. (a) A vocabulary size comparison reveals 9.1x larger lexicon in empty prompt responses (10,119 vs. 1,114 unique words). (b) Response length distributions show comparable overall distributions and mean token counts (45.3 vs. 44.8 tokens, p < 0.05, Mann-Whitne, U test). (c) Lexical diversity metrics demonstrate similarity between the sets of responses. (d) Different distributional patterns across predefined linguistic categories.
E Details of the IC Dataset
As mentioned in the main paper Sec 4, to comprehensively evaluate the model's performance across different caption cases, including the most challenging types, it was necessary to build diverse dataset to address this issue.
To validate its reliability, we conducted user study comparing human ratings of generated text with GPT-4o evaluations on the IC benchmark. We found that, on scale from to 10, the mean absolute error (MAE) between human scores and GPT-4o scores was only 1.252, demonstrating strong alignment between GPT-4o judgments and human preferences.
We provide details of each category and the number of samples collected in our IC dataset in Tab. 6.
More example visualizations of our proposed dataset can found in Fig. 12.
Demo Examples
Qualitative Examples Across Different Categories. To better demonstrate the usefulness of our proposed framework, we have included additional qualitative example comparisons such as an animation image in Fig. 13, a document image in Fig. 14, and common sense violation image in Fig. 15.
17

---

Published as a conference paper at COLM 2025
Emply Frompt Uncendiional Centriis Conred
60
40
I-SNE Dimerzzion
20
20
40
-00
1-SNE Dimension
Figure 11: A t-SNE visualization of response embeddings comparing unconditional path generation (blue, = 2, 000) and empty prompt generation (red, и = 2, 000) from LL- 13B. Points represent individual responses, projected into 2D space from 1, 1,000-dimensional TF-IDF features using t-SNE (perplexity=30, 1,000 iterations). The centroids for each re- sponse type are marked with crosses.
Category
Count
abstract
animal
animation
artistic
common
13
documents
12
events
10
fashion
food
handwritten
illustration
object
12
people
10
poster
scenes
technology 10
vehicle
Total
150
Table 6: Category and image counts of our IC dataset.
GPT-4o evaluation. For the GPT-4o evaluation, each caption was processed by GPT-4o to separately generate precision analysis and recall analysis. The precision analysis was used to compute the precision score, and the recall analysis was used to compute the recall score. The detailed prompts are shown in Fig. 16.
Detailed examples of precision analysis and scores are provided in Fig. 17, and examples of recall analysis and scores are shown in Fig. 18.
18

---

Published as a conference paper at COLM 2025
(a)
Abstract Concepts
(b)
Animals
(c)
Animations
(d)
Artistic Content
(e)
Common Sense Violations
data empli OVEER
Documents
(g)
Events
(h)
Fashion
Jour too and done
Aandenagy aalay
The Cutet
i
Food
Handwriting
(k)
Illustrations
Wilson
NIGHT PARTY
10.3-R EORIN
Objects
(m)
People
(n)
Posters
(o)
Scenes
(p)
Technology
(q)
Vehicles
Figure 12: Examples from the IC dataset. The dataset covers a wide range of diverse categories important for image captioning tasks, including abstract concepts, animals, animations, artistic content, common sense violations, documents, events, fashion, food, handwriting, illustrations, objects, people, posters, scenes, technology, and vehicles.
19

---

Published as a conference paper at COLM 2025
The image features a colorful and vibrant scene with a woman in the center, surrounded by varlous characters. She wearing white dress and holding a sword, possibly a ninja sword. woman appears the main focus the scene, with her purple hair standing out.
The image features a colorful and vibrant anime-style artwork with various characters. The main focus a girl dressed in kimono, standing center of the scene, surrounded by characters. The girl is holding sword, and her outfit is adomed with belt.
There several other characters in the image, including dragon on the left side and monkey on the right side. The monkey is holding guitar, adding a unique element to the scene. combination woman, sword, and the diverse characters creates a lively engaging visual experience.
In addition to the girl, there are several other characlers in the image, including a blue monster and dragon, both of which are positioned on the left side of the scene. The artwork showcases the diverse and lively nature characters, creating captivating and engaging visual experience.
LLavA 1.5 13b
Ours
Figure 13: Qualitative comparison of animation image captions. The left panel shows the description generated by LLaVA-1.5-13B, while the right panel presents our method's output. Both descriptions capture the vibrant, anime-style scene featuring diverse characters, but our method demonstrates superior accuracy and specificity detailing the central figure (a girl in kimono) and surrounding elements (a blue monster and dragon on the left). This highlights our approach's improved understanding nuanced visual elements and context, enabling richer and more precise visual descriptions.
20

---

Published as a conference paper at COLM 2025
The image is bar graph displaying the results of the 2004 presidential vote by race and gender. The graph shows the percentage of votes received by each candidate, with the total votes being 100%. The bar graph divided into four sections: White men, White and Non-White women.
The image features a bar graph displaying the 2004 presidential vote by race and gender. The graph is divided into different groups, including white men, white women, non-white men, non-white women. bars represent the percentage votes each group cast for President Bush and President Kerry.
Non-White
The bar graph shows that the majority of the votes were received by the candidate for White men, with 44% of the total votes. White women followed closely behind with 41% of the total votes. Non-White men and Non-White women received the lowest percentage votes, with 11% 4% respectively. The graph also includes line indicating the total percentage of voters.
The graph shows that white men and women, as well as non-white men, predominantly voted for President Bush, while non-white women and the overall population leaned more towards President Kerry. The provides breakdown of voting pattemss by during the 2004 presidential election.
Vida
gender
LLavA 1.5 13b
Ours
Figure 14: Qualitative comparison of document image captions. The left panel contains the description generated by LLaVA-1.5-13B. while the right panel shows the output from our method. Both descriptions analyze a bar graph depicting the 2004 U.S. presidential vote by race and gender, but our method provides a more concise and focused interpretation. highlights key trends such voting patterns for Bush and Kerry across demographic groups, emphasizing accuracy and clarity in extracting meaningful insights from visualized data.
21

---

Published as a conference paper at COLM 2025
The image features a unique landscape with a large, white, frozen crater in the middle a desert-like terrain. The crater filled with snow and ice, creating a striking contrast against the surrounding red dirt. The crater's center large, frozen lake, which adds the overall beauty of the scene. The combination of the frozen lake and the desert-like environment makes for intriguing and unusual landscape.
The image features a large, round, white crater in the middle a desert landscape. The crater appears to filled with water, creating a striking contrast against the surrounding dirt sand. water crater clear and blue, giving serene and picturesque appearance. The crater's size is quite impressive, occupying significant portion of the image.
LLavA 13b
Ours
Figure 15: Qualitative comparison common sense violation image captions. The left panel shows the description generated by LLaVA-1.5-13B, while the right panel presents our method's 's interpretation. While both descriptions recognize the unique setting of white crater within desert-like terrain, method provides fying the crater as frozen and filled with snow and ice, rather than water. This enhanced understanding highlights our model's ability handle complex and counterintuitive visual elements, ensuring clarity and correctness in scenarios that defy common expectations.
accurate depiction by identi-
Caption: (caption)
Caption: (caption)
Please analyze how well the caption describes what is present in the image.
Please analyze how well the caption covers the content of the image. Focus on the recall (i.e., what is in the image that is mentioned in the caption).
Focus on the precision (i.e., what is the caption that is in the image).
(a) Precision Analysis
(b)
Recall Analysis
Based your precision analysis, rate how image on completely inaccurate 10 is highly accurate. Just output 1-10.
Based on your recall analysis, caption covers the content the image to 10, where completely inaccurate and highly accurate. Just output 1-10.
well
the caption describes what is the where
(c) Precision Rating
(d)
Recall Rating
Figure 16: Prompts used for GPT-4o evaluation. We first use (a) and (b) to obtain precision and recall analysis separately, and then, combining these analyses, we use (c) and (d) to calculate the final precision and recall scores accordingly.
22

---

Published as a conference paper at COLM 2025
The caption provices general ovensew of scone and capturos many elements presint in the image. but there insouuaacis and Here's anslyss of precision:
Betting:
describes setting brankel
indeed
depicted
pizza wooden correctly baing part the spread, caption more specific
aligne
contents the saled.
somewhat misleading while there snaces that fruit (the amall dish has fruit, there Isn't clear bowl of only fruit.
mantion
Beverages:
The image fostures pienic selting with a blankst
The oxption comeecly states the presence of mi mnulippe zupe. However, inacourately refers to a wine glass; the
The biankat creating deiigtful inviting almosphare.
Image shows glesses of what appears juice or iced tea rather than wine glasses.
adomned variety
Decorative Elements:
Thare are saveral plates of food, including pizza and salad, well bowl of fruit.
vase with flowers acourately referenced, adding decorative touch: idemtfled potted plant nacourate:
backaround, clearly
addittinn the food, glass plazed offering a selection beverages for the guantts vase is also prosent, adding docorative touch the scano.
multipio cups
capton utensils forks, knives, spoons, bppe than actually present.
Seating Description:
positooned doss the blnneet, which visible image: instwed, there are sushions, not chair.
capaion
forks, knives, spoons laid the blankat, ready use. potted
Utansils
plant can be sean nearby, further onhancing the ouidoor ambience. A chair is posilioned close to the blanket, providing comfortable seating
Ovarall, the eaption captures the general essence of the picric scane, but contai inaccuracies regarding the specilic food Hemss the types dirice the presence of chair, and dets sbout the utensllis More preczie descriptions and accuracy what astually visible would mpprvv the capson significantly
onjoying
(a) Image caption
Precision analysis and score
Figure 17: GPT-4o analyzing the precision of caption given food image and rating the score.
Bonadlyy fore Yo donied Kyych vorere
provides solid overview of the content the image, focusing on several key elements:
sammeroe en alleuae weee wilk padion dlpa
ugu unfirryaaton pattn ofgeei th
1. Notebook Paper and Handwritten Essay: The caption correctlyy noles image depicts piece essay, which matches the visual content.
paper handwritten
Topic of the Essay: accurately identifies that the essay concerns the ancient Greeks and their mathernatica contributions, which is central theme in the shown the image.
to bette whenl wheel aoneeleee of seontyy Bund cullen weneey leggle eed nome safe saladd.
usdad
Godd geosaly pnoiplan baok sar whera in cedtertto watering ond
Diagram in ink, which aligns with what is visible in the image. connects this diagram essay's content, although does not specify what diagram illustrates.
caplion
diagram
Cursive Writing: The caption highlights that the essay is written cursive, reflecting the personal touch and historical context, which corresponds style
image.
The image features a piece notebook paper with handwrilten essay it. The essay about the ancient Greeks and their mathematical contributions. The paper diagram drawn in ink, which likely related the essay's content. The essay is wrilten in cursive, giving it personal and historical feel. The combination handwritten text and the diagram suggests that the essay is a detailed exploration of the ancient Greeks' mathematical achievements.
Detailed Exploration: assessment essay delailed exploration of the ancient Greeks' mathernticaal achievements aligns with informative nature text present image.
Overall, the caption elleclively caplures the primary elements of image, including the handwriling, topic, and visual characteristics. However, could enhance its precision by briefly mentioning the content diagram deepen the connection between textual components.
(a) Image with caption
(b) Recall analysis and score
Figure 18: GPT-4o analyzing the recall caption given a handwritten image and rating the score.
23


Reviews:
[{"review_id": "6UspsVq05u", "reviewer": "Reviewer_Kgrp", "summary": "This paper presents a novel model level judge free framework for self improvement in multimodal large language models. The core idea is to generate preference learning pairs via a controllable hallucination mechanism and to use a light weight contrastive language image encoder to verify and swap pairs when necessary. Finally, direct preference optimization (DPO) is applied to fine-tune the seed model. In particular, each token distribution is computed as an convex combination of $t_c$ and $t_u$, where where $t_c$ and $t_u$ are the conditional and unconditional token distributions respectively. The paper introduces a new IC dataset of 150 challenging images evaluated by GPT-4o on precision and recall, and demonstrates consistent gains on both public benchmarks (such as Object HallBench) and the IC dataset with significantly lower computational cost.  The originality lies in removing any large model feedback loop during verification and replacing it with a single light weight encoder.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel framework.\nIntroduces the first model level judge free self improvement approach for MLLMs, eliminating the need for expensive verifier models.\n\n- Clear methodological presentation.\nProvides a detailed algorithm (Algorithm 1), an explicit formula for token distribution, and illustrative figures that make the approach easy to follow.\n\n- Strong experimental validation.\nShows consistent improvements on both the IC dataset and Object HallBenchwith far less compute.\n\n- New benchmark contribution.\nThe IC dataset fills a gap by evaluating both precision and recall of captions across diverse categories, backed by human alignment studies with GPT-4o.", "reasons_to_reject": "- Limited theoretical insight.\nThe paper lacks a deeper theoretical explanation for why moderate CLIP score differences yield maximal learning gains.\n\n- Generality beyond images. \nAlthough claimed, extension to other modalities (video, audio) is not experimentally verified.", "questions_to_authors": "- How was the CLIP score difference threshold chosen, and how sensitive is performance to its value?\n\n- Could you provide more details on the sampling distribution fo $h_\\mathrm{ratio}$? Why choosing Gaussian with $\\mu = 0.5$ and $\\sigma = 0.15$?\n\n- Have you tested recursive self improvement by applying your framework for multiple rounds?", "extra_scores": {}}, {"review_id": "jSpKJ5Nm50", "reviewer": "Reviewer_LhJg", "summary": "This study proposes a cost-effective self-improvement method for enhancing the performance of multimodal large language models (MLLMs). The approach involves linearly interpolating token probabilities obtained with and without the use of images, controlled by a parameter $h_{ratio}$. By adjusting this parameter, the model generates captions under varying hallucination conditions. Pairs of chosen and rejected captions are then constructed and used for preference learning (DPO). To ensure the quality of these pairs, the method employs CLIP scores to correct label assignments: if the quality of the positive caption is lower than that of the negative one by a predefined threshold, the pair is swapped. Experimental results show that this self-improvement strategy contributes to performance gains in the model.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The study presents a method for collecting preference data without relying on external judges or expensive feedback sources.\n\n- The proposed technique effectively mitigates hallucination while maintaining general performance, demonstrating its practical utility.", "reasons_to_reject": "- There are significant concerns regarding the proposed CLIP-based scoring mechanism. Since the target task is image captioning and CLIP tends to focus on single salient objects, it may not fully capture the diverse details present in longer captions. Although the method is lightweight, its reliability as a scoring function is questionable. Furthermore, the use of average CLIP scores over sub-sentences to handle long captions presents limitations in accurately measuring similarity between the generated caption and the image.\n\n- Baselines such as LLaVA-RLHF, RLHF-V, and Silkie were not specifically designed to address hallucination. Therefore, comparisons with hallucination-focused baseline models are essential for a fair evaluation. (This concern is resolved by the author response)\n\n- The threshold used for pair swapping may significantly impact the quality of the training data. A deeper analysis of how this threshold affects learning outcomes would strengthen the paper.", "questions_to_authors": "Suggestion: The proposed method uses the hallucination control parameter $h_{ratio}$ to modulate the distance between positive and negative captions. This implicitly relates to the concept of margin in preference learning. Given the recent studies [1,2] on learning with dynamic or adaptive margins, connecting the proposed method to this line of work and analyzing the effect of margin control could enrich the paper’s theoretical and empirical contributions.\n\n[1] Wu, Junkang, et al. \"$\\beta $-DPO: Direct Preference Optimization with Dynamic $\\beta$.\" Advances in Neural Information Processing Systems 37 (2024): 129944-129966.\n[2] Deng, Xun, et al. \"Less is More: Improving LLM Alignment via Preference Data Selection.\" arXiv preprint arXiv:2502.14560 (2025).", "extra_scores": {}}, {"review_id": "w8znOjFMPu", "reviewer": "Reviewer_WN6m", "summary": "This paper introduces a judge-free self-improvement pipeline that eliminates the need for a MLLMl evaluator. The proposed framework generates controllable hallucinated negatives during decoding with a tunable hallucination ratio, filters and auto-relabels paired outputs using sentence-level CLIPScore comparisons and fine-tunes the base model with DPO. This approach aims to reduce computational costs and mitigate risks such as reward hacking and model collapse associated with traditional judge-based self-training methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper effectively addresses the high computational costs associated with judge-based self-training methods and introduces a lightweight alternative that uses only a 149M-parameter adapter and CLIP for scoring, making it more efficient.\n2. The dual-path decoding, with continuous control over the hallucination ratio, automatically produces hard negative samples. Additionally, the CLIPScore flip rule enhances dataset quality by correcting mis-labeled pairs without requiring additional learned judges.", "reasons_to_reject": "1. The reliance on external models like CLIP and GPT-4o introduces potential biases, but these biases are neither quantified nor sufficiently addressed. This reliance limits the generalizability and objectivity of the results.\n2. While the method demonstrates significant improvements on hallucination benchmarks, the improvements on broader vision-language tasks are marginal. This suggests the framework may be overfitting to hallucination-specific benchmarks, and failure cases or diagnostic analyses are not explored in sufficient detail.\n3. The IC dataset, with only 150 samples and roughly 10 samples per category, sounds quite small. The representativeness and effectiveness of such a limited dataset might be questionable, particularly in terms of its generalizability.", "questions_to_authors": "1. The method relies heavily on GPT-4o for evaluation. Could the authors explore and discuss the potential limitations and biases introduced by GPT-4o in the evaluation process?\n2. Some hyperparameters, such as the CLIPScore flip threshold and hratio scheduling, seem to be chosen arbitrarily. Could the authors provide a more detailed analysis or justification for these choices and include sensitivity tests to support the robustness of the method?", "extra_scores": {}}, {"review_id": "tPk5bbuY4M", "reviewer": "Reviewer_17vT", "summary": "This paper introduces an uncritical self-improvement pipeline for multimodal logical language models (LLMs), which (i) synthesizes positive/negative caption pairs by mixing conditional and unconditional decoding and adjusting the hallucination rate; (ii) flips or discards caption pairs using a lightweight CLIP evaluator; and (iii) trains a seed model via direct preference optimization (DPO). Experiments on Object HallBench, a new 150-image IC set, and multiple VQA/Chart/OCR benchmarks show significant hallucination reduction and much lower GPU cost than RLHF-like systems.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method is easy to reproduce: only a seed MLLM, a CLIP scorer, and an off-the-shelf DPO implementation are needed to reproduce the entire pipeline.\n\nThe authors show significant reductions in memory and time consumption.", "reasons_to_reject": "The core of the method proposed by the authors is essentially \"DPO + CLIP scoring\", which lacks sufficient innovation. The controllable h_ratio and pair-flip logic only implement the most basic negative sample sampling; the lack of comparison with stronger baselines (such as random negative sample DPO, paired subtitle dropout DPO, etc.) makes it impossible to prove the core contribution.\n\nOnly systematic experiments were conducted on LLaVA-13B, and mainstream MLLMs such as Qwen2-VL, Qwen2.5-VL and VideoLLaMA were not covered, which limits the extrapolation of the conclusions.\n\nThe key settings such as pair-flip threshold, h_ratio distribution and DPO temperature are not given ablation or the reason of value choice.\n\nIn the author's results, their method outperforms all public methods on Object HallBench, but it does not report whether \"direct DPO + random negative samples\" on the original LLaVA can achieve a similar magnitude.", "questions_to_authors": "If negative samples are generated using random truncation or random replacement, and then DPO is performed directly, what is the performance gap?\n\nHow is the CLIP difference threshold determined? What is the trade-off between model performance and training sample size at different thresholds?\n\nHas the model's true hallucination rate been tested on a human question-answering benchmark unrelated to CLIP?\n\nWhat is the statistical difference between the titles generated by the unconditional path and a simple empty string prompt? Can you provide coverage analysis or t-SNE visualization?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces an efficient, judge-free self-improvement method for multimodal large language models (MLLMs), specifically targeting hallucination reduction through controlled negative sample generation and lightweight CLIP-based verification. The reviewers recognized its novelty, simplicity, and practical efficiency, alongside strong empirical results on relevant benchmarks. Although some concerns were raised regarding theoretical depth and hyperparameter sensitivity, the authors provided thorough clarifications and additional experiments, which largely addressed reviewers' concerns. Overall, this submission offers valuable insights and demonstrates clear benefits, making it suitable for acceptance.
Decision: Accept