Towards Self-Refinement of Vision-Language Models with Triangular Consistency
Authors: Yunlong Deng
Guangyi Chen
Tianpei Gu
Lingjing Kong
Yan Li
Zeyu Tang
Kun Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Vision-Language Models
Self-Refinement
Unsupervised Learning
Primary area: deep_learning

Abstract: Vision-Language Models (VLMs) integrate visual knowledge with the analytical capabilities of Large Language Models (LLMs) through supervised visual instruction tuning, using image-question-answer triplets. However, the potential of VLMs trained without supervised instruction remains largely unexplored. This study validates that VLMs possess inherent self-refinement capabilities, enabling them to generate high-quality supervised data without external inputs and thereby learn autonomously. Specifically, to stimulate the self-refinement ability of VLMs, we propose a self-refinement framework based on a Triangular Consistency principle: within the image-query-answer triangle, any masked elements should be consistently and accurately reconstructed. The framework involves three steps: (1) We enable the instruction generation ability of VLMs by adding multi-task instruction tuning like image$\rightarrow$question-answer or image-answer$\rightarrow$question. (2) We generate image-query-answer triplets from unlabeled images and use the Triangular Consistency principle for filtering. (3) The model is further updated using the filtered synthetic data. 
To investigate the underlying mechanisms behind this self-refinement capability, we conduct a theoretical analysis from a causal perspective. 
Using the widely recognized LLaVA-1.5 as our baseline, our experiments reveal that the model can autonomously achieve consistent, though deliberately modest, improvements across multiple benchmarks without any external supervision, such as human annotations or environmental feedback.
We expect that the insights of this study on the self-refinement ability of VLMs can inspire future research on the learning mechanism of VLMs. Code is available at https://github.com/dengyl20/SRF-LLaVA-1.5.

Towards Self-Refinement of Vision-Languaag Models with Triangular Consistency
Yunlong Deng"
Guangyi Chen 1.2" Tianpei Gu', Lingjing Kong', Yan Li Zeyu Tang', and Kun Zhang!?
Mohamed bin Zayed University of Artificial Intelligence Carnegie Mellon University
ByteDDnce US
Abstract
Vision- LanguaLe Models (VLMs) integrate visual knowledge with the analytical capabilities of Large Language Models (LLMs) through supervised visual instruc- tion tuming, using image-question-anse answer triplets. However, the potential VLMs trained without supervised instruction remains largely unexplored. This study validates that VLMs possess inherent self-refinement capabilities, enabling them generate high-quality supervised data without external inputs and thereby learn autonomously. Specifically, to stimulate the self-refinement ability of VLMs, we propose self-refinement framework based Triangular Consistency principle: within the image-query-answee triangle, any masked elements should be consis- tently and accurately reconstructed. The framework involves three steps: (1) We enable the instruction generation ability of VLMs by adding multi-task instruction tuning like caauge--questoon answer or image-answer -qqussiinn (2) We gener- ate saune--uurr-answeee triplets from unlabeled images and use the Triangular Consistency principle for filtering. (3) The model is further updated using the filtered synthetic data. To investigate the underlying mechanisms behind this self- refinement capability, we conduct a theoretical analysis from causal perspective. Using the widely recognized LLa VA-1.5 as our baseline, our experiments reveal that the model can autonomously achieve consistent, though deliberately modest, improvements across multiple benchmarks without any external supervision, such as human annotations or environmental feedback. We expect that the insights of this study the self-refinement ability of VLMs can inspire future research on the learning mechanism of VLMs. Code is available at SRF-LLLVA
Introduction
Recent advancements in Large Language Models (LLMs) have demonstrated remarkable capabilitiess natural language understanding and generation. By incorporating visual features into the linguistic modalities understandable by Large Language Models (LLMs), Vision-Language Models (VLMs) have developed the capability to interpret visual content effectively. A key factor in the success of
these VLMs [1 2 3 4 5 6 7 is visual instruction tuning, which aligns visual content with the representation of LLM by optimizing the model's responses to visual instructions.
Acquiring supervised data with high-quality annotations is crucial for the success of visual instruction tuning. However, compiling visual instructions, such as image-question-anse answeer triplets, is non-trivial Such data cannot be directly crawled from the web and requires human involvement, making the process both challenging and costly. Careless data collection may lead to copyright issues. These factors motivate researchers leverage synthetic data instead.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

MMB
33.4
50N
Ours
M
Task 1 / (Q.4)
S4:1
57.7
Task 2 (1,Q) ->
61.3
33.5
60.3
Task 3 (1,
HMB
50.2
50.3
LLavA>
(a) The illustration of the Triangular Consistency principle. This principle evaluates the consistency within image-question-answe
116.1
18.5
62.0
triplets. Specifically, we independently mask A, Q, both, and use VL.Ms to infer the masked elements. We then check consis- tency by assessing whether the generated outputs align with the original ones.
MME
VON
61.4
GOA
(b) Stimulating self-improvement boosts down- stream performance.
The key to generating synthetic data lies in the creation of question-answer instructions. To ensure high-quality supervision signals, several methods [8] IO utilize advanced VLMs, such as GPT-
4V [12] and Gemini [33 to generate captions for unlabeled images. For example, ShareGPT4V g
applies GPT-4V to generate a curated set of 100K high-quality captions and expands the dataset to
1.2M using captioner trained on these captions. Additionally, some methods [14, 15 16] leverage advanced LLMs, such as GPT-4 [[7]. as expert evaluators of generated data quality. However, these approaches depend on high-quality VLMs, which may face bottlenecks due to usage limits or the cost of proprictary models. As models improve, it becomes increasingly challenging to find superior teacher models for data annotation. To address this issue, some methods propose deriving annotation signals from environmental feedback, such as evaluation performance [18] 19, 20 21]. However, relying environmental feedback for supervision signals may be inefficient for model training. More discussion of the related work can be found in Appendix A.
Motivated by the challenges outlined above, this paper seeks to address the following question:
Can we refine VLMs without relying on external supervision, using only the model itself?
To stimulate the self-refinement capabilities of VLMs, we propose a framework using a Triangular Consistency principle generate high-quality instructions by the model itself. This framework consists of three steps. (1) We fine-tune the VLMS to enhance their ability to generate instructions through multi-task objective. This involves randomly masking the question, answer, or both, and training the VLMS to reconstruct the missing components. Consequently, the models can generate query-answer pairs from unlabeled images. (2) To ensure high-quality generated instructions, as shown in Figure Ta we filter them with Triangular Consistency by inferring one component (question or answer) based the other, and then compare the consistency between new-inferred ones and the original parts in the instructions. By filtering, we select the instructions with high consistency. (3) Finally, we leverage the selected instructions refine the VLMs. This structured framework enhances the model's 's capability through continuous improvement, using its outputs as feedback mechanism. Consequently, supports multiple iterations of enhancement using only unlabeled images.
To investigate the underlying mechanisms behind the self-refinement capability of VL.Ms, we propose both theoretical and experimental analyses for validation. Theoretically, formulate the refinement process as a semi-supervised learning task. Initially, we identify the causal relations between natural language and images. Subsequently, we offer guarantees that learning can be effectively conducted using only unlabeled images, grounded in the principles of causality. Then, experimentally, we employ 1L. [ as our baseline, and evaluate whether the self-generated instructions can refine the models. As demonstrated by the results across multiple benchmarks, which are shown in Figure [b] the proposed framework can efficiently stimulate the relf-refinemen ability of VLMs. Our primary contributions are summarized as follows:
Triangular Consistency Principle:
We introduced the Triangular Consistency principle as a measure for testing the reliability of generated instructions on unlabeled data.
Self-Refinement Framework: We developed a self-refinement framework to update the model with self-generated instructions without the usage of any external annotations by humans or other stronger VLMs.
Theoretical and Experimental Analysis: We present theoretical analysis from a causal perspective to investigate the mechanisms self-refinement, Furthermore, we experimen- tally validate method using LLaVA-115 the baseline. The dataset, comprising 2

---

Fmining
Specific Taoks
New Data
(+:2)) +
Re Re-Prediction Filtered JQA's
Augmented Training Set
T
- (2) Ag
Pe uniabeeed
P
Pritere
(FAQ)
M
salabcled Irages only
(A)
Yes
Instrctionn Tuming
(Q2.0)
Q-
Tuning
(3)
Consistency
Chock
No
Drop
Emhancing Instruction Generation
Triangular Consistency Filtering
Instruction Tuming
An overview of the Seff-Refinement framework. The Self-Refinement framework comprises three stages. First, we design multi-task fine-tuning to enhance the model's instruction generation capabilities. Next, we apply the Triangular Consistency principle to filter high-quality instructions based on their scores. Finally, the selected data is used to refine the model. By using the updated model and synthetic data as the starting point for the next iteration, the framework naturally supports an iterative process.
Figure 2:
million images accompanied by generated instructions, will be publicly released facilitate further research.
2 Method
In this section, we first define the task and present the overall framework, which consists of three stages. We then detail each stage to demonstrate how the framework facilitates self-refinement in VLMs. Finally, we show that the framework is flexible enough to support multi-round refinement.
2.1 Overall framework
Our primary objective is to enable VLMs to refine themselves by extracting supervisory signals from an unlabeled image dataset Dunlabeled {1 images. To achieve this, we utilize the VLM itself to generate image-Question-Answer (IQA) triplets for the unlabeled images, thereby constructing synthetic dataset Dsymhetic {[T, Qi A,)). Here, I, denotes the image, Q; represents question generated based on the content of the image, and A, is the corresponding answer the question.
1.2..... .N. where N denotes the number of
As illustrated in Figure 2. we propose a three-stage self-refinement framework. Our preliminary experiments (Table 6)) indicate that directly utilizing existing VLMs, such LLaVA, struggles to generate precise and diverse instructions (question-answer pairs). To address this, we introduce multitask learning objective in the first stage. incorporating an amage--oonditioal instruction- generation task. Next, a filtering process is applied to collect refined data set Dihered. ensuring high-quality supervisory signals. In the final stage, we retrain the VLM using this filtered dataset by tuning the visual instruction.
2.2 Enhancing instruction generation
Although current VLMS exhibit strong visual understanding and perform well in answering visual questions, they often lack the ability to generate rich and diverse instructions (e.g., question-answer pairs). To address this limitation, we propose retraining the VLM to generate a synthetic instruction tuning dataset D Dyymhetic from an unlabeled image dataset Dunlabeled- For clarity, we denote the original
VLM as ()) and the model after fine-tuning for instruction generation as M
Specifically, we modify the typical instruction tuning stage by introducing three specific tasks:
- QA: Given an image I,, generate corresponding question question-answer pair Az).
Below is an example:
Human: For this image, what can be the instruction and answer pair?
VLM: Instraction: What object is the man holding above his head in the image?
Answer: The man is holding skateboard above his head in the image.
3

---

IQ - A: Given an image I and a question Qú, predict the masked answer A, This task is the typical instruction tuning task []] The following an example:
Human: What other object is the person in the image holding or wearing?
VLM: The person is also holding or wearing a guitar around his neck.
IA + Q: Given an image I and an answer Ai, predict the masked question Qi Here is an
example:
Build the instruction based on the answer. Answer: Yes, the man in the image appears to be in a band or a performer, as he is holding a guitar around his neck and is in location that could be nightelub performance venue. VLM: Instruction: Does the man in the image appear to be in band or a performer?
Human:
Task I QA focuses on enhancing the model's ability generate instructions effecctvvlyy. Tasks IQ + A and IA + Q are aimed at improving data filtering in the subsequent stage by ensuring the consistency and relevance of the generated question-ansee -answer pairs.
We preprocess the dataset 2 used in the instruction tuning of the original model M to create training data suitable for these tasks. The templates to generate multi-task training data can be found in Figure A2 of the Supplementary Materials. By organizing the training data in a way similar to an cestruction-tunigg task, we can directly apply the training strategy used in visual instruction tuning. Formally, the final multi-task loss function for this stage is defined as:
Lal Lqp + La Lap
(1)
where each loss term L corresponds to of the tasks and is computed using the cross-entropy loss:
Σ log P (we) Vinstract, wet):
where W; represents the target token at time step t. Vinstruct is the task-specific instruction including the input information such as the image I, question Qi or answer A,, depending the task.
2.3 Triangular consistency filtering
In this stage, we introduce the principle of Triangular Consistency filter high-quality instructions from the generated dataset. This principle is based on a simple hypothesis: robust instructions should demonstrate consisteney when any component of mmage-qusstion-answee (IQA) triplet is masked and subsequently re-pi re-predicted. Specifically, for given instruction triplet (I,, Q, As), where Qi, A; generated by the VLMs:
If we mask the answer A, and let the same VLM predict it based image T, and question
Qi, the predicted answer A should closely match the original A,
Conversely, if we mask the question Q: and predict it using the image 7; and answer A,, the predicted question Q' should align with the original Q.
According to the principle of triangular consistency, the predicted elements A and Ø' should be consistent with the originals, A, and Q. respectively. To quantify this consistency, we define consistency score as:
S = v Sim(Q: Q3 x Sim AQ),
where Sim(Qi, Q and Sim A,, AQ) represent similarity measures between the original and predicted questions and answers, respectively.
In particular, in order to compute the similarity score across diverse types of -restion-answer pairs, have developed appropriate similarity metries tailored to effectively handle cach data type. For single-sentence texts, we utilize a Sentence Transformer [22], while for longer texts, we employ BERTScore 23 When similarity can be precisely determined, such as in multiple-choice questions, we apply fixed metrics by directly comparing the answers to assess the consistency score. In question- answer pairs involving region descriptions and localization, we measure similarity by calculating the Intersection over Union (loU).
For each data type, we identify the top 20% of data exhibiting the highest triangular consistency scores. These selected data points form a filtered synthetic instruction tuning dataset, Dittered which is then used to refine the VLMs.

---

VLM
X1
1
S
X2
Nx
Ny
(a) causal relations
(b) input-ouiput of VLMS
Causal modeling and understanding of involved data-genett Lata-generating processes. Panel (a) presents causal relations among the language X, the semantic concept S, and the image Y. Panel (b) summarizes the input-output relation of VLMs.
Figure 3:
2.4 Instruction tuning & iteration
After obtaining the synthetic instructions about the unlabeled images, we merge this newly generated data with the original seed dataset (e.g. 1lava_vi__ _11__mmix665k used in LL. (2] as the final training dataset. Then we finetune the VLM ( on this merged dataset to obtain the self-refined VLM M The training strategy is the same as typical instruction tuning shown in Equation 2
We demonstrate that this three-stage process can be iteratively applied to enable continuous self- refinement of the model. Starting with the initial VLM M and unlabeled image set D(0), execute the self-refinement procedure outlined, producing enhanced model A) In the next iteration, we gather additional unlabeled data ((1) and repeat the process, treating M1) as the new base model. Through these repeated steps, the model is refined iteratively, resulting in (2).
By iteratively repeating this process with the vast amount of unlabeled data available online, the model can continually integrate real-world visual information from diverse image distributions. This iterative approach drives progressive self-refinement, allowing the model to adapt to ever-expanding range of visual data. The self-refinement loop can continue until the VLM has learned from sufficiently comprehensive data distribution, encompassing most new images within its existing training domain.
Theoretical analysis
This section explores the self-refinement capability of VLMs from a causal perspective. Specifically, demonstrate that the language is selected by the semantic concept, and through which, eventually, it serves as a cause of the image. We also present the theoretical foundation for the effectiveness of self-refinement framework.
3.1 Causal relations among language, image, and semantic concept
In order to see the causal relations between the language and the image, let us consider the roles played by the semantic concept as an auxiliary variable. Such a semantic concept is, in general, regarded as (latent) common cause of the observations for both image and text modalities [2]] 25 Here, we illustrate the asymmetric relationship between concepts and observations image (with pixels (Y,, Y2)) and text X (with words X2)). In this context, the semantic concept serves as selection mechanism language. In particular, as illustrated in Figure 3 (a), the semantic concept serves as selection for
words rather than common cause, considering the characteristies of natural
language, i.c., (x) + S. and as a (latent) common cause for image, i.c., S + (Y1, Y2).
Selection mechanisms indicate that samples are chosen based on specific criteria (e.g., only when they meet certain principles) before being observed. Recent literature 26 language processing explores selection mechanisms, showing that language outputs are modulated by semantic concepts specified in advance. Specifically, the semantic concept that the language needs to convey a goal to achieve, instead of a common cause for language. The key statistical difference between selection and common cause lies in conditional independence. If concept acts as a common cause, observations (e.g., words) become more independent when conditioned on that concept. In contrast, selection leads to the opposite effect. Clearly, when conditioned on certain topics, the relations of words would be more dependent, indicating the selection instead of the common cause structure.
5

---

Table 1: Comparison with advanced methods and baselines on 8 vision-language benchmarks. Due to space limitations, the names of the benchmarks are abbreviated. LLaVA W: -LaVA-Bench (In-
the-Wild) M MM-Vel (28]; MME MME Cognition; MMB: MMBench (29]; MMB MMBench- Chinese; VQA VZ [0:; GQA 31: SQA": ScienceQA-IMG [32]. More evaluation results on additional benchmarks are provided in Appendix C.I. Data: Number of Imaee-Instruction Pairs used during
visual instruction tuning. Recap-LLaVA-IA. is obtained by using LLa itself to caption one million images and retraining on those new image-instruction pairs. For comparisons based on the VA-155 7B baseline, the best results are bolded and the second- results are underlined.
Method
L.M
Data
LLaVA
MM-Vet
MME
MMB
MMB
VQA
GQA
SQA
InstmetBLIP
Vicuna-7B
1.2M
60.9
26.2
23.7
49.2
60.5
IDEFICS-BB
LLaMA-7B
1M
25.2
50.9
38.4
Qwen-Vi VL
QWen-7B
50M
78.8
59.3
67.1
Qwen-VL--C VL--Cht
QWen-7B
50M
360.7
56.7
78.2
57.5
68.2
LLaVA
Vicuna- 1.5-B
158K
63.0
26.7
247.9
34.1
14.1
79.0
38.5
LLLVA-L.5
Vicuna 1.5 13B
665K
70.7
35.4
295.4
67.7
63.6
80.0
63.3
71.6
SRE-L.aVV-LI.5 13B
Vicuna 1.5. 13B
665K 200k
73.5
37.7
334.2
68.6
64.0
81.3
65.1
72.2
LLaVA-1.5 7B
Vicuna-I.5-- 1.5-7B
665K
63.4
30.5
316.1
64.3
58.3
78.5
62.0
66.8
Recap-LLaVA-1.5 7B
Vicuna- 1.5-7B
665K + 1M
63.3
29.9
321.3
66.1
58.5
79.2
62.5
69.01
SRF-LLaVA-LL 7B
Vicuna-1.55 1.5-7B
665K + 200k
66.9
33.1
331.8
66.3
59.2
79.6
63.35
67.72
3.2
Self-Refinement explanation
The input-output relations of a VLM can be summarized as generating the text X based on the visual information Y and conditioned on a question Q, represented as P(X Y: Q). To simplify the notation, we fix the question Q and use the shorthand P(X Y) in the following discussions without ambiguity. The initial VL.M models the mapping between Yola and Xata- To further refine the VLM, we incorporate additional unlabeled images, Yiew, to improve the estimation of P(X Y). We introduce the following mild assumptions, which are widely used in the causality theory (27
Sufficiency and Independence. As shown in part (b) of Figure 3 the conditional distribution P(Y | X), is defined by a deterministic function io and an independent noise variable Ny, such that Y (xx,Nyy) with Ny P(Nr). Similarly, X is governed by independent noise variable Nx with Nx ~ x.Nx). The mechanism yp is irrelevant to the input distribution P(X).
ANM and Decomposition. The Additive Noise Model (ANM) 3 assumes that the relationship between X and Y can be expressed in the format of cxX,N, Ny) = (X) + Ny. We further posit that the marginal distribution P(Y) can expressed as the convolution of two distributions:
P(Y) = = F(=)G(Y- -) 2),2,
where F and G are component distributions. This decomposition is valid under conditions such as Ny being Gaussian and (()())) being indecomposable. The latter assumption reasonable, as X represents diverse textual data, making (()((X) unlikely to be decomposable.
Both the independence--of-mechhni extensively used to identify causal directions and to separate mechanisms from input distributions. In our setting, they provide a principled reason why better estimating the image marginal P(Y)-viia additional unlabeled images Ynew can improve inference target P(X Y): the forward (causal) model P(Y |X) is stable across changes in P(X). whereas the backward direction lacks such invariance. Consequently, unlabeled images carry exploitable signal for refining P(Y) and, through Bayes' rule, for improving P(X Y). We emphasize that Eq. (4) specifies the generative process used for theoretical analysis and is not enforced during VLM training. While these assumptions cannot be directly verified for modern VLMs, they are useful theoretical tools to capture plausible structural properties and to derive interpretable insights.
and ANM are standard functional assumptions that have been
Our goal is to estimate P(X Y) (())))((X) By incorporating unlabeled images Y Yiews can P(Y)
refine the empirical estimate P(Y) with more observations. Since is observed, improving the estimate of P(Y X) consequently improves our goal P(X Y).
Formally, under ANM assumptions, P(Y) can be expressed as a convolution of the distributions of (x(X and Ny. From the paired data (Xold. Yota), can estimate the function Pood and corresponding

---

distribution P(fota(X)) Having estimates of both P(Y) and P(Pola(X)) we can identify the noise distribution P(Ny) via deconvolution..P(Nyy
P(Y) * (X)) Due the unique decomposition of P(Y), we can identify whether P(Ny) corresponds to F or G, and thus get the true distribution P(Ny). This correction of P(Ny) can help obtain a better estimation of P(Y X).
4 Experiments
4.1
Experimental settings
We adopted LLa 7B 2 as our baseline, which uses CI.IP-Large as visual encoder and Vicuna- 1.5 35 as LLM. To rule out the possibility that our gains stem merely from more data or longer training, we constructed an additional strong baseline, Recap-LLaVA-I which uses the same unlabeled images but replaces the Self-Refinemes procedure with a straightforward self-annotation pipeline: (i) starting from the same M unlabeled images as SRF-LLaVA-L., we use a frozen LLaVA-1.5-7B to generate one detailed caption per image; (ii) we retain all I M captions without filtering and merge them with the original LLaVA-1.5 training set; and (iii) we fine-tune the model using the identical training schedule as SRF-LLaVA-
+4
Beyond the original training dataset 1lava v1 v1_5_ _____mix6655 used in LLaVA-1.5, we randomly selected 2.8 million images from LAION 36 as the unlabeled image set to validate the self- refinement ability. During fine-tuning on the merged dataset, we follow the standard LLaVA-1.5 instruction-tuniin recipe. Specifically, we keep the vision encoder frozen and fine-tune both the image-text projection layer and the LLM parameters. In the evaluation, we followed the setup of LLaVA-1.5, which contains 8 benchmarks. For traditional VQA tasks, we used the VQAv2 (30), GQA 31I, and ScienceQA [2] datasets. For visual perception and reasoning tasks, we employed MMBench 22), MMBennhh-hinese [29]. MME (37), and MM- Vet (28) benchmarks. To assess visual dialogue ability, we utilized the LLaVA-Bench(In- Bench(In-thee Wild) I benchmark. Please note that the goal of this work is to explore the self-refinement capability of VLMs without distilling knowledge from other models like GPT-4V. Therefore, we did not incorporate such knowledge into our model and did not conduct many comparisons with the methods using such models.
4.2 Implementation details
In the Enhancing instruction generation stage, we constructed the dataset using Laav__v1___mix6661 as the seed. Specifically, we masked both the question (Q) and the answer (A) in 50% of the data, only Q in 20%, and only A in the remaining 30%. For each task, we randomly selected prompt templates combined with unmasked elements as inputs, with the masked parts serving as ground-truth targets. Then we generate synthetic data from one million unlabeled images and select the top 20% ones with higher Triangular Consistency scores. We design hybrid scoring approach suited to different instruction types. For shorter texts, we employed UAE-Large-Vi V [22). compact Sentence Transformer with only 335 million parameters, that effectively compares single-sentence text similarity without introducing external knowledge. For longer texts, since the performance of the Sentence Transformer diminishes, we opt for BertScore 23), based the cutt-bas-unnaseee model, text comparisons. To ensure fair comparisons with baseline, we adopted a head-to-heald setup by using the same base language model and prompts as LLa prompt templates and data preprocessing methods utilized by L.L. The specific calculation procedure for Triangular Consistency scores is provided in Section B of Supplementary Materials.
provides robust discriminative capabilities for lengthier
Also, we adhered strictly to the
4.3 Quantitative evaluation
We evaluated our framework alongside advanced VLMs, the baseline LLaVA-1.5 and the strong baseline SRF-LLaVA- 1.5 on benchmarks. Overall, our framework offers consistent improvements over V1. on most of the benchmarks. For instance, in traditional VQA tasks -ickusiinn VQAv2 and GQA, our model outperformed 1La by an average of 1. 1.22%. For visual perception and reasoning tasks such as MMBench, MM-VET, and LLaVA-Wild, we achieved improvements of 2.1 %, 1.7% and 3.5%, respectively. Although Recap-LLLa -A---1 shows only small gains on most benchmarks, RRF-LLaVA-1.. improves them much more. This suggests that SRF-LLaVA-1.57
7

---

Question What's the anding of thi movie?
LLaVA-1.TTte mnding of this movoe is not provided, as scent from a movie ara TV show However It likely that the couple will continge their idventure, as they are already bool and enjoying their tine togetiee:
Ourss the ending of novie Titasic knoww, on a ship oil led the Titani.. They fall lve and shom a memomble night tygether on the deck of the ship. Howneer as the ship eooomtor an loebery and begins to ank, Jack socifiices his life giving Rose a chance to survive jumping into the water, The morie ends with Ros bring rescued and the sidking of the Titanic The emotonal impart film and the ending have made ata beloved classic symbol
classic romantic crama. In the the main characters Jack and Roen,
saotifice
Question Describe
detail
LLAVA-15 The image fectues a peinking a dog weentng Reneissccoo--d dress, giving ita hunnnn like appestance. The dog is sitting in chair, looking dinecly at the viewer The painring is quiee demiled, eapturing the dog's ficial expresiio and the inticate ded of the dress in the background, there person partally vishle on the ieft of the painting: The avenal somee is quito cipticating and unique, as combones the artutte
the Renusstnnca with the subject mather of dog.
Ours The image uss peinting of dog dressed in Renaisaace style outfit resembiling a woman. The dog is weering a long dress and a lat, giving it unique and artistic appearanoe. The painting see lindecspe ambicace of
with its hands mountatoon backggound, adding the overall lo,,
The postiioned in the omber the painting captures the dog's unique crative portnyal, making it interesting and eyo ratching puece
human. The
Figure 4: Two comparison examples between our SRF-LLaVA-1.5 and LLaVA-1.5 in visual chat. Red highlights indicate factual errors or irrelevant content in the response, while green highlights emphasize image details critical for providing accurate answer.
Table 3: Evaluations on Moblie VLM Baseline
Method
LLM
GQA
SQA!
VQA
POPE
MME
MMB
Mo
MobileLLaMAA114B
56.1
57.3
41.5
84.5
1196.2
53.2
CRF-MobileVLLM MobileLLaMA-1.4B
58.1
59.8
43.3
85.3
1220.3
56.1
advantage comes mainly from the Triangular Consistency principle, which enhances self-refinement, rather than from training on extra data.
Training cost analysis. At inference time, SRF-LLaVA- 1.5 and LL. are identical in parameter count and therefore require the same FLOPs and GPU memory. Dur- processes larger
Table 2: Training time compari-
son between LLaVA-1.5 and SRF. LLaVA-1.5.
ing training, however, SRF-LLaVA number of images, leading to longer training duration. Ta- compaee the total wall-clock times SRF-LLaVA-1.5 and LLaVA-1.5 both trained on cluster equipped with 8
Model
Total wall-clock time GPU-hours
LLaVA-15
7h
SRF-LLAVA-LS
12h
96
NVIDIA H100-NVL GPUs (96 GB each). The refinement procedure costs approximately 40 ad- ditional GPU-hours (=00.7x), which report explicitly to inform practitioners of the trade-off between accuracy and computational expense.
4.4 Qualitative evaluation
In this section, we illustrate how our self-refinement framework enhances the real-world visual dialogue capabilities. Figure Aresents two representative examples.
Enhanced World Knowledge. For the first example in Figure 4 model correctly identifies the individuals in the image as the protagonists the movie Titanic and accurately describes the film's ending, whereas the original LLaVA-1.5 fails to do so. Though no external information is involved, our framework can use the consistency principle to filter possible errors and do the refinement.
Improved Generalization Ability. For the second example, model generates description that is significantly more detailed and accurate than that of LLaVA-1.55 without exhibiting hallucinations. We attribute this improvement to the exposure to wider variety of visual scenes and objects provided by the unlabeled images. By learning from these diverse visual features during training, the model enhanced its ability recognize and represent them.
4.5 Ablation studies
Does Self- Reff-Refinement Generalize Across Different Scales and Architectures of VLMs? Yes. To assess the generalization of our Self-Refinement framework across VLMs with different parameter scales and architectures, conducted experiments MobileVLLM-1.7 38), QWen2.5-VL 3B (3),

---

Table 4: Evaluations on QWen2.5-VL Baseline
Method
LLM
MMMU
MMMU Pro Math
Math
MMMtta
MMB
Qwen2.5-VL
Qwen2.5-3B
53.1
31.6 31.9
62.3
21.2
55.8
81.5
SRF-Qwen2.5-VI.
Qwen2.5-3B
55.0
64.7
23.3
56.5
80.8
and LLaVA-1.5 13B using an identical dataset and training pipeline. For MobileVLM and QWen2.5- VL 3B, we followed their original configurations and, for each model respectively, evaluated SRF- MobileVL.M and SRF-Q SEFF-Wen2..5-V 5L on six benchmarks selected from their official evaluation suites. For LLaVA-1.5 13B, we mirrored the experimental setup used for LLaVA-1.55 7B. As shown in Table and T SRF delivers consistent improvements in all baselines, indicating that self-refinement is generalized across various VLM architectures. In particular, as shown in Table SRF-LLaVA-I.. 13B exceeds LLaVA-115 13B in all 8 benchmarks, demonstrating effectiveness at larger parameter scales.
Can Multiple Iterations Lead to Better Per- formance? Yes. As shown in Table Round 2 performance was comparable to Round on GQA and SQA, with noticeable improve- ments on MM- Vet and LLaVA-Bench How- ever, the overall improvement in Round 2 was smaller. We hypothesize this satura- tion stems from the finite exploitable signal that unpaired images provide about the un- derlying image distribution; once largely dis- tilled, subsequent iterations yield diminishing returns. Nevertheless, a second round still produced measurable gains, implying that single pass does not fully harvest the avail- able signal. Round 2 exhausted all our collected images, suggesting that further performance gains would require additional data.
Table 5: Ablation study on Framework Itera- tion, Consistency Criterion, Selection Thresh- old. Complete results are provided in Table A3 of Supplementary Materials.
Ablations
Patterns
GQA SQA'
Me-Yet LLaVA W
SRF-LLaVA-LL5
Default
63.35
67.72
33.1
66.9
LLaVA-LS
Baseline
62.00
66.8
30,5
63.4
Multi-Round
Round-2
63.25
67.77
33,9
67.7
Consistency
Bottom 20%
62.85
66.70
30.4
64.4
Top 5%
62.90 67.48
32.4
63.6
Top 20%
63.35 67.72
33.1
66.9
Threshold
Top 50%
63.02 68.77
30.4
62,8
Top 80%
62.9 67.03
29.1
64.0
Top 100%
62.87 67.48
30.7
63.7
Does the Self-Refinement Framework Pro- duce Better Instructions? Yes. Table 6 compares QA pairs generated by our Self- Refinement framework against those from prompted original LLa VL-155 evaluating GPT- 4o accuracy and diversity metrics (TTR 44) and D 41) on 1000 samples. Our model demonstrates clear advantages over prompted LLaVA-1.55, achieving superior accuracy and di- versity. This indicates that explicitly training dedicated model for instruction generation ef- fectively improves QA pair quality, particularly diversity.
Table 6: Evaluations of GPT-4o accuracy, Type- Token Ratio (TTR), and Distinct-2 on the gen- erated QA pairs from SRF-LLaV. VA, baseline LLaVA, and LLaVA with in-context learning.
SRF-LLavA 7B
LLaVA-1.5 7B
Metrics Accl TTR Distinet-2
0-shot
0-shot
I-shot
5-shot
85.3
73.9
79.2
63.6
0.1144
0.0888
0.0775
0.0692
0.4790
0.3216
0.3075
0.2737
To evaluate the effectiveness of the Triangular Consistency Filtering stage, we conducted two comparisons: (1) between the retained and excluded QA pairs after filtering and (2) by altering the filter criterion from the top 20% to the bottom 20%. both quantitative and qualitative evaluations of the retained versus excluded synthetic data subsets. For the quantitative analysis, we used GPT-4o to evaluate 1,000 samples from each subset. We also conducted human study with 12 volunteers, each of the 100 samples reviewed by 3 volunteers. The results confirm that the retained subset significantly outperforms the excluded subset. Qualitative comparisons of the QA pair samples at the bottom of Table further emphasize the superior quality of data selected by the Triangular Consistency Principle. Additionally, as shown in the "Consistency" group of Table [5] training the model on the lower 20% (ranked with consistency score) of synthetic data resulted in performance decline compared to SRF-LLaVA-Is. (which using top 20%), further validating the effectiveness of Triangular Consistency. More experimental details on Triangular Consistency Filtering can be found in Section C of Supplementary Materials.
How does Triangular Consistency Principle work?
How do Different Instruction Types Influence Performance?

---

To further investigate which data types have the greatest impact on model performance, we replaced the filtered synthetic data with different subsets of data categories. Specifi cally, we retained only five types in- cluding VQA (Visual Question An- swering), Visual Chat, REG&REC (Region Expression and Recog- nition), Caption, and Multiple- Choice/Judgment, and retrained LLaVA-1.5 exclusively on these subsets to observe any changes in downstream performance. As shown in Figure different instruc- tion types yield corresponding im- provements, with captioning tasks contributing the most overall en- hancement. Please refer to Section C of Supplementary Materials for comprehensive comparisons, statis- tical details on the distribution of generated instructions, and further
Table 7: Quantitative and Qualitative Analysis of Retained and Excluded QA Pairs, We evaluate the quality of the gener- ated QA pairs through both GPT-4o assessments and a human user study. Additionally, we present an example to qualitatively compare the consistency between the Original and Recovered QA pairs.
Type
Evaluation Examples
Subset
GPT
Human
Quant.
Retained
85.3
89.0
Excluded
59.9
83.0
Subset
Original QA
Recovered QA
Q: What is the color of the painting displayed in the im- age?
Q': What color the picture displayed in the image? A:: The color the painting displayed in the image blue.
Retained
A: The painting displayed in the image is blue picture.
Qual.
Q': What is the setting of the image??
Q: What setting the image?
Excluded
A: setting image indoors, with the person using green case in a white room.
A": The setting outdoors, with the person hold- ing the small ease or container outside.
image
experiments on the impact of different instruction partitions.
How Much Data is Required? More data of- fers additional information but also introduces more noise. We investigated the impact of vary- ing the threshold of the triangular consistency score used to screen high-qualit, quality synthetic data, Specifically, we combined synthetic data with scores ranking the top 5%, 20%. 50%, 80%, and 100% (i.e., without filtering and detailed captions) with the 1L. 665k dataset to form the new training set, as illustrated in Ta- ble 5 The results indicate that model perfor- mance improves across all thresholds. However, the best overall performance is achieved when using the top 20% threshold.
2:00
1,0
1:22
O0A
5:04
MMVS
FallOOk
WallCapto
Cal
Figure 5: Ablation study specific synthetic data types. The numbers denote the percentage increase compared LLa
5 Conclusion
In this work, we explored the self-refinement capability of Vision-Language Models (VLMs) through a framework grounded in the Triangular Consistency principle. Our results demonstrate that this self-refinement ability can be effectively activated, enabling VLMs autonomously generate and leverage high-quality supervision from unlabeled data. We further provided a causal explanation for this phenomenon, showing how the model's ability to infer missing modalities supports its internal consistency and learning dynamics. Empirical evaluations confirm that our framework consistently improves baseline performance without external supervision, highlighting the potential of self-refinement as a pathway toward autonomous and continually improving multimodal intelli- gence. Limitations: Although our approach successfully demonstrates self-refinement, the model's improvement diminishes when the newly introduced images closely align with the distribution of the model's existing training data. Societal Impact: By enabling reploimmpoveeeett from unlabeled images, our framework can reduce reliance on expensive and labor-intensive human annotations and help democratize access to rerfopeeformance VLMs, particularly in low-resource settings. However, learning from model-generated synthetic data risks reinforcing or amplifying biases and hal tions present the pretrained model, and the absence of human-in human-in-the-loge in-the-loop oversight can make unintended behaviors harder to trace audit.
10

---

Acknowledgment
We would like to acknowledge the support from NSF Award No. 2229881, AT Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NIH) under Contract R01HL159805 and grants from Quris AI, Florin Court Capital, and MIBZUAI-WIS Joint Program, and the Al Deira Causal Education project.
References
[1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neurul information processing systems, 36, 2023.
[2] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22226-26306, 2024.
[3] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In The Twelfth International Conference Learning Representations, 2023.
[4] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-2377 2022.
[5] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Advances in neural information processing systems, 2023.
[6] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint a:X.v:2308.01390 2023.
[7] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vasion-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
[8] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311. 12793, 2023.
[9] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[10] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.000 03326, 2024.
[11] Zhiyuan Zhao, Linke Ouyang, Bin Wang. Siyuan Huang, Pan Zhang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Milm-dataengine: An iterative refinement approach for milm. arXiv preprint 2308v:2308. 13566, 2023.
[12] OpenAI. Gpt-4v(ision) system card. 2023.
[13] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Saan-Bapistte Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 /805, 2023.
[14] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self self-generated instruc- tions. arXiv preprint arXiv:2212.10566 2022.

---

[15] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principledrrvven self- ali-neiigment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2023.
[16] Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoye,, Jason We- ston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arXiv:2308.000 06259, 2023.
[17] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[18] Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. Iterative translation refinement with large language models. arXiv preprint WVXi:22306003856, 2023.
[19] Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon. RI4f: Generating natural language feedback with reinforcement learning for repairing model outputs. arXiv preprint wrXiv:235508844 2023.
[20] Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representations. arXiv preprint ar arXiv:2304.01904, 2023.
[21] Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302. /28 2813, 2023.
[22] Xianming Li and Jing Li. Angle-optimized text embeddings, 2024.
[23] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: /904.09690 2019.
[24] Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. In International conference on machine learning, pages 11455-11472. PMLR, 2022.
[25] Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf. Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451-16467, 2021.
[26] Jingling Li, Zeyu Tang. Xiaoyu Liu, Peter Spirtes, Kun Zhang, Liu Leqi, and Yang Liu. Steering Ilms towards unbiased responses: A causality-guideer debiasing framework. arXiv preprint arXiv:2403.088 08743, 2024.
[27] Bernhard Schölkopf. Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint arXiv: 64071, 2012.
[28] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023.
[29] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024.
[30] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Flevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904 6913, 2017.
[31] Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019.
12

---

[32] Pan Lu, Swaroop Mishra, Tony Xia. Liang Qiu, Kai- Wei Chang, Song-Chur Shng-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. 2022.
[33] Patrik Hoyer, Dominik Janzing, Joris M Mooij. Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[35] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging -undds-- with mt- bench and chatbot arena, 2023.
[36] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filterd 400 million image-text pairs, 2021.
[37] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: comprehensive evaluation benchmark for multimodal large language models, 2024.
[38] Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevim: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312 260.2666,,,,,, 2023.
[39] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprini 2r0i 13923, 2025.
[40] Mildred C Templin. Certain language skills in children; their development and interrelationships. 1957.
[41] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity- pronetttty-rmootiggg objective function for neural conversation models. arXiv preprint 1510000050 2015.
[42] Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive language-image image pre-training with synthetic caption. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2922-2931, 2023.
[43] Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang. Hiclip: Contrastive language-image pretraining with hierarchy-aware attention. arXiv preprint arXiv:2303.02995. 2023.
[44] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021.
[45] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23390-23400, 2023.
[46] Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Madappally Jose, Alexander Toshev, Jonathon Shlens, Ruoming Pang, and Yinfei Yang. Stair: learning sparse text and image representation in grounded tokens. arXiv preprint 23X.0:2301.13081, 2023.
13

---

[47] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint arXiv: 1ti:2//11.11332 2021.
[48] Gabriel Ilarrco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong. John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. If you use this software, please cite it as below.
[49] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. Neurocomputing, 555: 556:126658, 2023.
[50] Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annota- tions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11162-11173 2021.
[51] Michael Tschamnen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. Advances in Neural Information Processing Systems, 36, 2023.
[52] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai- Rei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint (00i::190000355577 2019.
[53] Yicong Hong. Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazz and Stephen Gould. VIn bert: A recurrent vision-and-lawe and-language bert for navigation. In CVPR, pages 1643-1653, 2021.
[54] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104-120. Springer, 2020.
[55] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020.
[56] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for iision-anddLanguage representation learning. Advances in Neural Information Processing Systems, 33:6616-6625 16-6628, 2020.
[57] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision- language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5579-5588, 2021.
[58] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[59] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904-4916. 2021.
[60] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing Dision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[61] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint (.t::2400000332555 2024.
[62] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713. 2024.
14

---

[63] Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen. Daniel Khashabi, and Yejin Choi. Generating sequences by learning self-correct. arXiv preprint arXiv:2211.00055. 2022.
[64]
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. Rarr: Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16477-16508,
2023.
[65] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305. 1738, 2023.
[66] I Chern, Steffi Cher,, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Gra- ham Neubig, Pengfei Liu, et al. FacTool:: Factuality detection in generative ai-a tool augmented framework for multi-task and multi-domain scenarios. arXiv preprint arXiv:2307. 13528, 2023.
[67] Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar- Lezama. Is self-repair silver bullet for code generation. In arXiv preprint arXiv:2306.09896 2023.
[68] Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint SeXXv:24050006755 2024.
[69] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXhv:2303.176517 2023.
[70] Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896, 2023.
[71] Jie Huang. Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023.
[72] Loka Li, Zhenhao Chen, Guangyi Chen, Yixuan Zhang, Yusheng Su, Eric Xing, and Kun Zhang Confidence matters: Revisiting intrinsic self-correction capabilities of large language models. arXiv preprint arXi: 2024.
[73] Qwen Team. Qwen: Open large language models by alibaba cloud. https: /ttps://github.. thub. com/ Treali//Dreen3////rreee/aaaaaaai qwen- vl finetune, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims in the abstract accurately reflect our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide complete experimental details in Appendix B. In addition, the full code for our main experiments has been open-sourced in an github repository (the link is given in Abstract.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide an github link to the code and describe how to reproduce the experimental results in the README file of the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/ccpppblligggddeeeeeeeeeeee Cotee//ddeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the complete experimental details and hyperparameter choices in Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report the error bars of main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the computational resource requirements of our proposed method in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: stttps://neurips.coo
Answer: [Yes]
Justification:
Our research aligns with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts of framework Section 5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The code and datasets used in the paper are publicly available and properly credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have provided detailed documentation for our open-source code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This paper does not involve LLMs as any important, original, non- non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy huttep//hearipssccc//offrrrreeee for what should or should not be described.
22

---

Appendix of "Towards Self-Refinement of Vision-Language Models with Triangular Consistency"
Yunlong Deng",
Guangyi Chen 1.2" Tianpei Gu', Lingjing Kong', Yan Li!, Zeyu Tang', and Kun Zhangl?
Mohamed bin Zayed University of Artificial Intelligence Carnegie Mellon University
ByteDance US
Appendix organization:
Related Work
24
B
More Implementation Details
24
B.1
Templates for Generating Multi- Training Data
24
B.2
Similarity Calculation across Diverse Data Types
25
B.3
Hyperparameters)
26
B.4
Training Details of Other Baseline VLMs
27
Pseudo Code
27
More Quantitative Results
27
Evaluation on More Challenging Benchmarks
27
Ablation Results on More Benchmarks
27
Effectiveness of Consistency Score Measurement
28
Data Types Investigation
29
Ablation Study on Alternative Masking Ratios for QA ................................
29
C.6
Error Bars in the Main Experimental Results
30
More Qualitative Results
30
Empirical justification of Equation (4) on synthetic data
31
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A Related Work
Vision Language Models. Vision-Language Models (VLMs) leverage large datasets to learn joint representations of visual and linguistic information. These models commonly employ training frameworks such as contrastive matching 42 43
49), cross-modal genera- 56 571, or a combination of both. Key examples include contrastive frameworks like CLIP 58 and ALIGN [59] which synchronize text and image representations through contrastive objectives. Conversely, the cross-modal generation approach, exemplified by works such as VirTex 50 and VisualBERT (2]. emphasizes generative tasks like captioning or masked content reconstruction for learning representations. Additionally, the emergence of large language models has significantly enhanced the capabilities of multimodal systems. These meth- ods 112 D 2
45
46
47
48
tion (50 51 52 53
54
55
4
6
[1, 60) integrate and expand upon this rich multimodal knowledge by aligning
the pre-trained LLMs with the visual modality.
Visual Instruction Tuning with Synthetic Data. To address the high costs and complexities of
acquiring copyrighted visual instruction data, recent approaches [8] 9 10 [1 62 have advocated for the use of synthetic data in model training. These methods typically utilize advanced VLMs such as GPT4V (12) to analyze requirements, generate detailed captions or instructions, and assess synthetic data quality. For instance, ShareGPT4V [8 and LLaVA-Next 99 employ GPT4V to produce captions and instructions, respectively. while HLLL--DaaEnnee
61
[1]] focuses on identifying model weaknesses and generating instructions with specific targeting. While these methods enhance performance, they encounter two major limitations tied to the reliance advanced VLMs: firstly, high-quality VLMs often come with restrictive usage limits and high costs; secondly, as models evolve, sourcing superior teacher models for effective data annotation becomes increasingly difficult.
Self-Refinement of LLMs. While discussions on VLMs' self-refinement are limited, debates about LLMs' self-refinement persist. These methods fall into two categories: extrinsic and intrinsic self- refinement. Extrinsic methods utilize feedback from external sources, such as evaluation models (20] or interactions with humans or tools 65] 66 67 68). More related to this work
19
63
21
64
the intrinsic self-refinement [69 18 external feedback. For instance, Self-Refine 69 employs the in-context learning of LLMs to generate and iteratively correct instructions using seed examples. Sun et al. [15] apply templates and text- based rules for instruction selection, while loE Prompt [72) uses carefully designed prompts for self-correction through re-inference. However, applying these methods to VLMS presents significant challenges. Multiple inferences [72] and multi-example in-context learning [69 are resource-intensive with images. In addition, the use of image modality means that sample templates or text-only rules, which only focus on textual quality without considering alignment, are less effective.
70
71 72,, which utilizes the model's capabilities without
B More Implementation Details
In this section, present some key implementation details of the Self Self-meffnement Framework to facilitate a better understanding of its mechanisms.
B.1 Templates for Generating Multi- Masti-TaskTie Training Data
We provide the prompts used to generate multi-task training data. During the training phase of Enhancing Instruction Generation, transform the original training data from the LLaVA-1.5 database,
665k, into data suitable for multi-task training by setting specific prompt templates. Specifically, for the maage-Question-Asse we established the following task types: - QA, IQ + A, and IA- +
(IQA) data triples in original dataset,
For the IQ - A task, we use the original data without modification. For the I + QA and IA + Q tasks, we prepend a randomly selected prompt to the inputs to assist the VLM in more accurately aligning the instructions with the respective downstream tasks. Additionally, we have also redesigned the system prompts used by the VLM due to changes in training tasks. The complete prompts used for training are detailed in Figure Al
24

---

System Prompt
You are visual- Lisual-language assistant. You will be given one of the three kinds of tasks with specified task type. Complete the task by generating a response.
Prompts for + QA Instructions
Generate the instruction and answer pair corresponding to the image.
Build the instruction and answer pair for this image.
What can the question and answer pair corresponding to the image?
For this image, what can be the instruction and answer pair?
Given the image, generate the question and answer pair.
Can you generate the question and answer pair for this image?
Prompts for IA + Q Instructions
Generate the instruction based the answer.
Build the instruction based on the answer.
What can the instruction for the answer?
What's the question for this answer?
For the response, what can be the instruction?
Figure Al: The list of prompts for Enhancing Instruction Generation. Given that IQ + A task the same as typical instruction tuming, We provide the system prompt and template prompts for I - QA, and IA -> Q tasks. Notably, we do not enforce any particular question-answe type. The prompts simply ask the model to generate question and its answer. The distribution of generated QA types therefore emergent
B.2 Similarity Calculation across Diverse Data Types
We demonstrate detailed information on the similarity indices designed for different types of data during the Triangular Consistency Filtering stage. We process various types of question-answer data generated during the previous stage and flexibly adjust the similarity calculations based on data characteristics, follows:
To assess text similarity, we use either Sentence Embedding or BertScore. As detailed in Section 2.3, we separately calculate the similarities of questions (Q,Q') and answers (A, A'), then compute their geometric mean. For Visual Question Answering (VQA) data, we employ Sentence Embedding for both questions answers. contrast, for Visual Chat data with longer answers, we apply Sentence Embedding to questions and use BertScore for answers, as performs better with extended texts.
When similarity values can be precisely calculated, we compute them directly using specific metries. For example, in Region Description (REG) and Recognition (REC) data involving a region description query (Q) and corresponding coordinates (A), or vice versa, we use Intersection over Union (IoU) measure the similarity of region coordinates. The similarity of region descriptions is calculated using Sentence Embedding, and we take the geometric mean both similarities. Similarly, for multiple-choice and true-false data consisting of only a question and an answer, we do not compare question similarity since the same answer might correspond to different questions. Instead, we check the answer remains the same before and after reconstruction; if identical, the similarity score is 1; otherwise, it is 0.
25

---

VQA
Question: What the color of the man's shirt? Answer the question using a single word
phrase.
Answer: Blue
Visual Chat
Question: Why might each of the three shots in this photograph be taken?
Answer: Each of the three shots in the photograph captures different high angle angle and side view of a man wearing a suit and tie, possibly for a wedding, as one of the angles suggests. These shots might be taken to showcase the suit, tie, and overall look from different angles and perspectives, providing a more comprehensive and visually appealing coverage the desired outfit. The variations in camera angles also provide sense of uniqueness and individuality. helping the viewer better understand the significance of the clothing and the overall attire being presented.
REG REC
Question: Please provide the bounding box coordinate of the region this sentence describes: the mans right arm.
Answer: [0.82, 0.83, 0.97, 0.98]
Question: Please provide a short description for this region: [0.67, 0.78, 0.8, 0.93].
Answer: Two buttoned up shirts.
Caption
Question: Provide one-sentence caption for the provided image. Reference OCR token:
Book, BEST
Answer: A poster of smiling cartoon character in white and blue shirt with the words best at the bottom of the photo.
Multiple-Choiee & True-False
Question: What kind of bag is shown? A. backpack B. tote C. purse D. briefcase Answer with the option's 's letter from the given choices directly.
Answer:
Is there balcony in the picture? Answer the question using a single word phrase.
Question:
Answer: Yes
Figure A2: The list of Q-A pairs of different types. The parts marked in orange are fixed prompt templates, which are not included in similarity calculations.
For data where the questions are in fixed format, we only compute the similarity of the answers. For example, for Caption data, which includes instruction (Q) and its corre- sponding image description (A), we focus solely on the similarity the image descriptions. This is because the instructions primarily contain hints and OCR tokens without significant meaning.
To fairly treat different types of data, we select the top 20% of data from each category. Additionally, to prevent the similarity calculations from being influenced by repeated instruction templates, we exclude all fixed-format instructions. Examples of each data type are shown in Figure A2
B.3 Hyperparameters
To ensure a fair comparison, we adopted head-to-head setup by using the same base language model and prompts as LLaVA-1.5. The training hyperparameters were consistent with those of LLLVA-1.5 an initial learning rate of 10-5 and a batch size of 128. Most of the training (including all ablation studies) was conducted on 8 NVIDIA A100 GPUs, with both the first and third stages
26

---

each taking approximately 20 hours. For evaluation, we compared our SRF-LLa VA-155 against the original LLaVA-1.5. We adhered strictly the prompt templates and data preprocessing methods utilized by L.L.V-1.55. Decoding was performed using greedy search to maintain consistency and reproducibility our results.
B.4 Training Details of Other Baseline VLMs
The merged dataset for MobileVLM consists of (i) samples generated by MobileVLM itself through self-reffi reffrrefinemen loop the 1M unlabeled images, and (ii) its original supervised training set. For QWen2.5- VL. since its original training set is not publicly available, we follow the official repository's recommendations by combining our self-refinement dataset with the officially suggested datasets [73] Inyu-visionx/Cambriin-- 10M,, Laaa-aabbl.LaV--eee..
Prredoomnnelligenne/AAA.. LaVAA44 and
and subsequently perform SFT the combined corpus.
B.5
Pseudo Code
Algorithm A1 lterative Self-Refinement
Input: Mo ... initial VLM
2:
Do human-labeled (T, Q, A) dataset
3:
U ... unlabeled images
4:
K number of refinement rounds
5:
for iterative self-refinmment
K do
Stage Multi-task fine-tune (caption, VQA, instruction)
6:
M f- TRAIN Do)
7: S + { (I,Q.A) produced by M for every I E U
8:
S' <- (1,
M (I) A A' M (1, (1
Stage 2 Generate & filter synthetic IQA
9:
for all (I, A) S and (I, Q' A S" do
10:
if F Q' and A then F+ A)
12:
end end for
Stage 3 Instruction tuning with filtered data
D1 Do F
15:
M1 + TRAIN(Mo, D1)
>> instruction-only objective
end for
17: return M
As shown in Algorithm A1 This pseudo code serves as a high-level description of the algorithm's 's workflow, outlining its core computational steps and logical structure.
provide the pseudo code for the algorithm of our SRF-LLaVA-1155
C More Quantitative Results
Table Al: Comparison on more chal- lenging benchmarks. For both LLaVA- 1.5-7B and SRF-LLaVA-I.5-7B. we ob- tained the results through local evalua- tion using VLMEvalKit.
C.1 Evaluation on More Challenging Benchmarks
To evaluate whether our method generalizes to more chal- lenging benchmarks. we further assess SRF-LLaVA-IA. 7B and LLaVA-1.5 7B on MMMU and MMMU-Pro, The results are shown in TT
Model
MMMU
MMMU
Baseline
34.7
17.6
C.2 Ablation Results on More Benchmarks
After refinement
36.6
18.3
This section presents the complete ablation study data conducted within the Self-RReinement Frame- work. We examined the impact of various factors on the framework's performance, including the number of framework iterations, the selection of consistency criteria, the setting filtering thresholds, and the adjustment of data distribution. As shown in Table A2 observed the following phenomena:
27

---

Table A2: Complete ablation study results Framework Iteration, Consistency Criterion,
Selection Threshold, and Data Partition. LLaVA W; LLaVA-Bench On-the-Wiidd) [i MM-Ver Vet
28
MME MME Cognition 371: MMB: MMBench
29
MMB HMMBench-Chiness VQA
30
GQA 31: SQA'; ScienCeQA-IMG
32
Ablations
Patterns
LLaVA
MM-Vet
MME
MMB
MMB
VQA
GQA
SQA
SRF-LLaVA-1.5
Default
66.9
33.1
331.8
66.3
59.2
79.6
63.35
67.72
LL5
Baseline
63.4
30.5
316.1
64.3
58.3
78.5
62.0
66.8
Multi-Round
Round2
67.7
33.9
335.0
66.3
60.3
79.52
63.25
67.77
Consistency
Lower Score
64.4
30.4
337.5
66.0
60.3
79.19
62.85
66.7
without long caption
64.8
30.1
321.4
66.3
60.2
79.41
63.27
67.03
Partition
LLaVA-1.5 partition
62.2
30.2
282.
66.8
60.0
79.42
63.17
68.12
top 5%
63.6
32.4
337.5
65.8
61.2
79.34
62.9
67.48
top 20%
66.9
33.1
331.8
66.3
59.2
79.6
63.35
67.72
top 50%
62.8
30.4
319.3
67.1
60.9
79.44
63.02
68.77
Threshold
top 80%
64.0
29.1
316.4
66.5
60.4
79.33
62,9
67.03
top 100%
63.7
30.7
312.5
65.4
59.4
79.35
62.87
67.48
Table A3: Consistency score between reference and candidate answers.
Type Ref
Answer
Similarity
The answer is an apple.
Cand1
The answer is an orange.
0.79
Cand2
The answer happens be apple.
0.98
Cand3
An apple is the correct answer.
0.94
Improved Performance with Additional Iterations With more iterations, the model's performance improves. The Round-2 model slightly surpasses ARF-LLaVe VA-1.5 on 5 benchmarks and is comparable on others, achieving an average improvement of 0.4% across all benchmarks (excluding MME-C due to unnormalized values, and keeping this setting for all following discussions). The performance gain more significant on woal-world visual chat benchmarks than on traditional VQA tasks, suggesting that supervisory information from unlabeled images in the new distribution benefits real-world dialogue scenarios.
Preserving Data Distribution and Detailed Captions Improves Performance Maintaining the original distribution of synthetic data, along with detailed captions, enhances self-refinement performance. Adjusting the data distribution to match that of the LLaVA-1.5 training set causes slight performance drop. Likewise, removing all long-caption data decreases performance.
Impact of Filtering Threshold The filtering threshold affects model performance. 20% threshold yields the best results among thresholds of 5%, 20%, 50%, and 80%, suggesting the importance of balancing data quality and quantity. Even without filtering (threshold of 100%), the model shows some performance improvement (about 0.73%). This indicates that the VLM can learn supervisory information from new images, even with low-quality synthetic annotations.
C.3 Effectiveness Consistency Score Measurement
We used to explore several consistency score measurements, including RoBERTa and Vicunal.5 which yielded lower accuracy and higher costs, so we adopted the current method. Specifically, we employ Sentence Transformer for shorter sentences, while for longer sentences or paragraphs, we utilize BertScore, ensuring an optimal balance between evaluation accuracy and computational efficiency. Table presents consistency scores for sentence meaning shifts due to word changes. The score is notably lower when the factual word "apple" changes to "orange."
28

---

Ours
llava v1 5 mix665k
9%
5%
5%
33%
18%
13%
2%
33%
1%
19%
17%
13%
32%
VQJ
Cantion REG REC
VQA
Caption
Visual Chat
Multiple- Choice
True-Falss
Visual Chat
REG & REC
Long Caption
Multiple-Choice
Fulsea
Figure A3: The comparison between the data distributions of synthetic data and LLaVA-1.5 training dataset.
Table A4: Complete ablation study results on specific synthetic data types. Each value in the Data Partition column corresponds to retaining only the indicated synthetic-data type.
Data Partition
LLaVA
MM-Vet
MME
MMB
MMB
VQA
GQA
SQA
Add VQA
68.7
31.1
322.5
65.8
59.5
79.28
63.12
69.66
Add Caption
63.5
33.0
326.4
67.2
61.4
79.20
62.90
70.10
Add Visual Chat
69.5
32.8
335.4
66.4
60.5
79.27
62.35
68.82
Add REG&REC
68.4
33.0
287.9
66.6
59.5
79.20
62.45
68.62
Add True- ThueFaals/MulilleCChole
66.5
32.0
309.3
66.8
59.0
79.09
62.49
67.92
C.4 Data Types Investigation
This section presents the complete results of ablation studies specific synthetic data types. This experiment aims to comprehensively assess the impact of integrating different data types on the Self-Refinement performance of VL.Ms.
Figure A3 illustrates the proportional differences in distribution between our synthetic data and the original VLa1--1.5 data. Compared to the original distribution, our synthetic data is more balanced, without categories that are excessively underrepresented.
As shown in Table A4 we retained data solely from five types: VQA (Visual Question An- swering). Visual Chat, REG&REC (Region Expression and Recognition). Caption, and Multiple- Choice/Judgement tasks, and tested the performance of VLMMs when re-instructed to fine-tune on these filtered data distributions. The results indicate that the VLM generally shows performance improvements in the training sets corresponding to the integrated data types. For instance, the VLM integrated with VQA data outperformed others on the VQAv2 and GQA datasets. However, this trend is not without exceptions. For example, after integrating multiple-choice data, the VLM not surpass the performance achieved with other data types on the ScienceQA and MM-Vet benchmarks, which are also multiple-choice in nature. Moreover, the integration of Visual Chat data resulted the highest average performance improvement of 2.3%, while the performance gain from Multiple-Choice/ Choice/ TraiceeTrre-aee -alse data was the least. This could be due Visual Chat data containing richer image features compared to other types, thus providing more supervisory information during the fine-tuning phase.
C.5 Ablation Study on Alternative Masking Ratios for QA pairs
Pilot results suggested limited sensitivity to the exact Q/A masking split, so we used a balanced 50/30/20 (Q-mask/A-mast/Noone) Lalittask-LLaAA-L. with two alternatives: Q-heavy (60/20/20) and A-heavy (40/40/20).
To verify this, we retrained

---

Table A6: Comparison with advanced methods and baselines 8 vision-language benchmarks with error bar. Note: SRF-LLaVA-I.3 results are reported as mean + standard deviation over three independent fine-tuning runs with identical hyperparameters and random seeds.
Method L. 7B
LEM
Data
MaaAA"
MM-Vet
MMEE
MMR
MMK
VQA"
GQA
SQA
Vieume
.578
665K
63.4
10.5
316.1
64.3
58.3
76.5
62.0
66.5
Recap
521.
79.2
SRP LL-.5- 7B paean 1 sid)
Vicuno 1.5 665% e 200K.
6.62
33.1 0.06
3
0.21
66.4
0L16
79.6 59.1
0/03
6.22
Then, in Stage generated QA pairs for 1,000 unlabeled images and evaluated their accuracy and diversity: Acct (GPT-40 judged image-QA consistency), TTR (type-token ratio), and Distinct2* (unique bigrams ratio). As shown in Table A5 all strategies yield comparable results; we therefore keep 30/30/20 for simplicity and reproducibility.
Table A5: Masking configurations and results (Accuracy, Hype-Token Ratio, and Distinct-2).
Mask Type
Mask Q&A
Mask Q
Mask
Acc(%)
TTR
Distinct-2
Original
50%
20%
30%
85.3
0.1144
0.4790
More Balanced
33%
33%
33%
87.7
0.0995
0.4381
C.6 Error Bars in the Main Experimental Results
We have repeated the entire cestruction-ine-turee identical hyperparameters, the same random seed, and the same training-datanne order. The resulting mean + standard deviation across the three runs has been reported in Table A6
phase of SRF-LLaVA three times, each time using
D More Qualitative Results
Baseis conditonnrr
Oees ohe
conditinne and by Those
dloce prodacts of the Tlicker
Fuller colloction.
Cas you gote
perten?
LaVA--15 The inape finteres ihssed the icreic dieeader the fiker, from the mucce The Deek Kright Hu holding playin cadd in las laxal possilly rofareaze the famono soase the usnto whace fir Joker dosls cad Buraan The mren makinu and costaaze deigard resetible tae Joker's appeatuee, anking bun look like coastooing represattion to chartctss
The uarviz The Dark Kangla" oetor Jooth Lodgts Tae Joker eay kro In the image, the loker bolding plaiing card. winch is eigante element of obateeer and syntte of expretieebbit and dispes the morce is leaona for inhenae action, compelling storlinns, and the aepenssive purfommance of teath I Ledge as Joker.
Oure:
the DC Couaice univone and fentures issat charaia, the loke, playal by the chopotic cumuio who woralo bavoo on Godian City wtale te pootaguast, Dotuaee nios to
Qucaiom Can
sdes: introsaationn to dii paining!
1L. The imsge is paratig of's night soms fietring bepa. dank like sly with a les stas vidile. The pairtng shso inclucks maocn. wieck locead lowe fhe te right conox Bhe soae:. The sky is Eled with willin dlud, cnatine a chamnic and drantic atmrspphii In the seqgorod teee is csuall Lswn with clurd, wisdaill, and fow bosss The lowe siturd belon the sight dky croding a scme of dayth and perpectree the pourtng The combonscce of uglr say, towa,
drack custes axprooing and sercce coece.
Ours: The pointiss is Gensse sishtin dasopoe Vincai Vaa Gool fastnins boutidd sight da Gibal wils can. The scene i viina villggo with dludhstegdo and de arhi skitffilly ceptured Dhe of the tigle de with trikking stas. The silage is cememaaded by mewstii, aldiing the petrooque qualiy for wome The parting copiivting remouretic bearty nuttere trangeliit sigls.
Figure A4: More qualitative examples of our SRF-LLaVA-1.5 and LLaVA-1.5. In all three examples, SRF-LLaVA-1.5 provides more detailed and instruction- aligned responses compared to LLaVA-1.5. Red highlights indicate factual errors or irrelevant content in the response, while green highlights emphasize image details critical for providing an accurate answer.
In this section, we present series of qualitative results intuitively compare the differences between SRF-LLaVA-115 and the original VL..5.
Improvements in Model Capabilities From these analyses, we observed several enhancements in our model. As mentioned in Section 4.4, SRF-LLaVV-1..5 exhibits enhanced detail recognition, capturing more subtle features images than I.L. VA- and possesses richer world knowledge, successfully identifying artists and interpreting underlying meanings in artworks, shown in
30

---

Studying online
Qwalion: Can
monc?
LLaVV-L.5
difiseal
Zoom
Teacher
festame with clwn fie,, poohlyy making farny enssaining foutth image sbowe with and lis pasibly thowal forell ovant work exptices undemesth cadh imags scited to stalling vellae, with the finst caption moaling SSudying Ouliaz? The soaud cqpiaas caah "Zonn, the this cug roads "Toocher" and the fruth cggton mads "Stdeers" "Th mene seame be pliyfil og eseraaiio of ditiverr srpert stadying online, with the images and captices prossingg a ligttooral tukz watso.
pouitly noboing neinng
dares
possilyy coktesting
The socond shows
dewn. The
company
Studonts
meme coninas Sour differett pisuuos of people, cepoertting diffeen sapodt of stadiin online. The fiost sloure picuue
money sver lin, symboliziag
suubing suite. Finally, picure planse mon repowworing inteppettdd
PAY
mon laytop, tigsitiing stichhologgg
entetsising atayiuu
Figure A5: Failure cases. Both SRF-LLaVA-1.5 and LLaVA-1.5 fail correctly understand the implications of the meme. Red highlights indicate factual errors or irrelevant content in the response, while green highlights emphasize image details critical for providing an accurate answer.
Figure A4 In addition to the aforementioned capabilities, we have also observed that SRF-LLaVA- 1.5 demonstrates superior instruction-following capabilities, allowing it to flexibly adjust its responses based on different types of prompts. For instance, in the second example of Figure A4] the user's instruction is introduce the character "Joker." While LLaVA-1.5 continues to focus describing the image content, our SRF-LLaVA-115 provides a comprehensive summary of the Joker as per the user's request, without being confined to the content displayed in the image.
Failure Case Analysis Despite these improvements, SRF- LAR-LLVVA-155 shares some shortcomings with the original LL. In meme presented in Figure AS which superficially consists of four unrelated images, a deeper analysis reveals concerns about students' situations in online learning. However, both models provide only surface-level analyses of the image. While this limitation likely due to constraints in the logical reasoning capabilities of the underlying language models used, we optimistic that future advancements through our Self-Refinement approach will help overcome this boundary, enabling SRF-LLaVA-1.5 to understand deeper meanings and provide more insightful analyses.
E Empirical justification of Equation (4) on synthetic data
To examine the conditions underlying Equation (4), we design a controlled synthetic experiment that follows the generative process in Section 3.3, where the output distribution P(Y) arises from the convolution of a deterministic component with independent noise. Concretely, we draw ~ Laplace(O, 1) and independent noise N Laplace(O, 0.6) dimension d 50, then set
Y X + N,
with random full-rank matrix & E 50 The dataset is split into Ni.b 1900 labeled pairs (X, Y) for training, Nunl = 4900 unlabeled observations Yuew for self-refinement, and Nes = 1000 labeled pairs for evaluation.
three-layer MLP with two hidden layers of 128 ReLU units takes a test vector Y 50 input and outputs two 50-dimensional vectors (n, b), where estimates the conditional median of X and b E TR50 the coordinate wise Laplace scales. The network trained on labeled data by minimizing the Laplace negative log- ligliiil
XX Mg by
L(X, = Σ log(2b;)
using Adam (learning rate 10 batch size 128. 50 epochs). During self-refinement, the model predicts pseudo-labels Xnew = (4(Yew) N, retains the top 40% most confident samples (smallest predicted scales 6), augments the training set with these pseudo-labels, and retrains; this process is repeated for several rounds. On the test set, we report (i) Laplace NLL as above; (ii) mean squared error
Nist
MSE-
Xx - 44 11.12
Nicu
31

---

Table A7: Synthetic-data self-refinement under the generative model in Section 3.3. Lower NLL/MSE is better, higher R is better.
Setting
NLL
MSE
R²
Baseline
1.1840
0.4691
0.7635
After refinement
0,9046
0.2872
0.8551
Improvement (A)
0.2794
0.1819
0.0916
and (iii) R² computed by scikit-learn's r2. score with miliootpuut"uniforee _aarrrge" (higher is better). Results are summarized in Table[A7] 'hee self-refinement procedure consistently improves NI.L, MSE, and R², supporting the premise that additional observations the effect variable Y help estimate P(X | Y) under the assumed decomposition. We note, however, that directly transferring this decomposition to real-world VLMs remains a theoretical abstraction; see Section 3.3 for discussion of this limitation.
32


Reviews:
[{"review_id": "XURVC12Oba", "reviewer": "Reviewer_6Atd", "summary": "In this paper, the authors leverage the inherent self-refinement capabilities of VLMs to generate high-quality supervised instruction data without external supervision. To achieve this, the authors propose a triangular consistency principle within the image-question-answer triplets: any masked elements should be consistently and accurately reconstructed. Based on this principle, the authors construct a synthetic instruction tuning dataset to further fine-tune the VLMs. Experimental results on standard multimodal benchmarks demonstrate the effectiveness of the proposed method.", "questions": "See the questions mentioned above. Given the current status of the paper, I am leaning towards borderline reject and hope the authors could address my concerns during the rebuttal.", "limitations": "The authors briefly discussed the limitations in Sec. 5. However, I cannot find a specific section to discuss the societal impacts of their work. Some broader impacts that come to my mind: positive impacts: reduce human supervision and costs, negative impacts: induce potential bias using the synthetic data from the model.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n-\tThe idea of using internal feedback from the model itself to enhance VLMs is interesting.\n-\tThe paper is generally well-written and easy to follow.\n-\tThe experiments are extensive and the results are promising, while some parts need to be improved (see weaknesses).\n\n**Weaknesses:**\n\n-\tThe authors mainly focus on LLaVA. What is the generalizability of the proposed method to other kinds of VLMs (e.g., Qwen-VL, DeepSeek-VL, Intern-VL)?\n-\tFor LLaVA, the authors mainly focus on 7B and 1.7B models. What is the scalability of the proposed method for larger models (e.g., 13B)?\n-\tAccording to L237-238, the authors use different masking ratios for Q and A. How to decide the ratios? Using different ratios can potentially influence the final performance. However, it seems that no relevant ablations are conducted.\n-\tSome details are missing exact descriptions. When the authors fine-tune the VLM on the merged dataset, it is unclear which specific modules are fine-tuned (the whole model or some sub-modules)? And what is the effect of fine-tuning different modules?\n-\tAlthough the authors mention that the self-refinement process can be conducted continuously, it seems that the performance has already saturated after one-round update. This makes the potential of the proposed method somewhat limited.", "was_revised": true, "final_justification": "The rebuttal well addressed my concerns. After carefully reading other reviewers' comments, I would like to increase my score to borderline accept. I suggest the authors incorporate all the results/comments discussed in the rebuttal into the final version.", "rebuttal": "Dear Reviewer 6Atd, \nWe sincerely thank you for recognizing the originality of our work, for your encouraging comments on our experimental design and results, and for the valuable suggestions you offered to further enrich our study. We are also deeply grateful for the time and effort you devoted to reviewing our manuscript.\n\nIn response to your recommendations, we have conducted additional extension and ablation experiments and have expanded the description of our implementation details. Our point-by-point replies are presented below:\n\n> **Q1.** The authors mainly focus on LLaVA. What is the generalizability of the proposed method to other kinds of VLMs (e.g., Qwen-VL, DeepSeek-VL, Intern-VL)?\n\n**A1.** Thank you for raising this question regarding the generalizability of our method across different VLMs. To demonstrate that our self-refinement framework (SRF) is not limited to LLaVA, we validated its effectiveness on MobileVLM, as discussed in Section 4.5. In light of your suggestion, we further evaluated the SRF framework using the Qwen 2.5-VL-3B model, one of the strongest open-source VLMs of similar scale. Specifically, we applied our entire pipeline using the same 1 million unlabeled images. The results, summarized in the table below, confirm that our method maintains its effectiveness across different VLM architectures.\n\n\n| Model[1]             | MMMU[2] | MMMU Pro[3]  | MathVision | MMStar | MMBench1.1[4] |\n| ----------------- | ---- | -------- | ---------- | ------ | ---------- |\n| Qwen2.5-VL-3B     | 53.1 | 31.6     | 21.2       | 55.8   | 81.5       |\n| SRF-Qwen2.5-VL-3B | 55.0 | 31.9     | 23.3       | 56.5   | 80.8       |\n\n[1]: SRF-Qwen 2.5-VL-3B was evaluated with VLMEvalKit; Qwen 2.5-VL-3B scores are taken from the official model documentation.  \n[2]: Results are obtained on the MMMU validation (VAL) set.  \n[3]: Results are obtained on the MMMU Pro Standard (10-choice) split.\n\n[4]: Results are obtained on the MMBench Dev English split.\n\nTo provide a clear, side-by-side comparison of model performance, we have added the evaluation results of LLaVA-1.5 7B and SRF-LLaVA-1.5 7B on MMMU and MMMU-Pro, as shown in the table below.\n\n| Model[5]            | MMMU | MMMU Pro \n| -----------------| ---- | -------- | \n| LLaVA-1.5-7B     | 34.7 | 17.6 |\n| SRF-LLaVA-1.5-7B | 36.6 | 18.3 |\n\n[5]: For both LLaVA-1.5-7B and SRF-LLaVA-1.5-7B, we obtained the results through local evaluation using VLMEvalKit.\n\n\n> **Q2.** For LLaVA, the authors mainly focus on 7B and 1.7B models. What is the scalability of the proposed method for larger models (e.g., 13B)?\n \n**A2.** We appreciate the suggestion to conduct experiments on larger backbones. Following your suggestion, we applied our self-refinement framework to LLaVA‑1.5‑13B. We evaluated SRF-LLaVA-1.5-13B on eight standard benchmarks. As illustrated in the following table, our approach yields consistent performance gains over the original LLaVA-1.5-13B across all tasks. These updated results have been incorporated into Table 1 of the revised manuscript.\n\n\n| Model | LLaVA-Wild | MM-Vet | MME-C | MMBench | MMBench-CN | VQAv2 | GQA | ScienceQA |\n|------|--------|--------|------|-----|-------|-------|-----|------|\n| LLaVA-1.5-13B | 70.7 | 35.4 | 295.4 | 67.7 | 63.6 | 80.0 | 63.3 | 71.6 |\n| SRF-LLaVA-1.5-13B | 73.5 | 37.7 | 334.2 | 68.6 | 64.0 | 81.3 | 65.1 | 72.2 |\n\n\n> **Q3.** According to L237-238, the authors use different masking ratios for Q and A. How to decide the ratios? Using different ratios can potentially influence the final performance. However, it seems that no relevant ablations are conducted.\n\n**A3.** Thank you for highlighting this point. In our pilot experiments, we found that model performance was largely insensitive to the exact masking ratios applied to questions and answers. Consequently, we adopted a balanced split of 50 %, 30 %, and 20 % chosen at random.\n\nTo address your concern, we have now added an ablation study on alternative masking ratios (i.e., different Stage-1 training-data partitions) in Appendix C of the revised manuscript. Specifically, we retrained Multitask-LLaVA-1.5 under two distinct masking configurations. In the first stage of the Self-Refinement framework, this model generates QA pairs from unlabeled images. We then used each retrained version to create QA pairs for 1,000 unlabeled images and evaluated their diversity and accuracy with GPT-4o. As the table below shows, both masking strategies delivered comparable performance. The Acc metric assesses how well the generated question-answer pairs align with the image content, whereas TTR and Distinct2 measure the lexical diversity present in those QA pairs. (all higher is better)\n\n| Mask Type     | Mask Q&A  | Mask Q    | Mask A    | Acc  | TTR    | Distint2 | \n| ------------- | ---- | ---- | ---- | ---- | ------ | -------- |\n| Original      | 50%  | 20%  | 30%  | 85.3 | 0.1144 | 0.4790   | \n| More Balanced | 33%  | 33%  | 33%  | 87.7 | 0.0995 | 0.4381   | \n\nIf you would like us to include results for additional partitions, please let us know—we would be more than happy to add them.\n\n \n> **Q4.** Some details are missing exact descriptions. When the authors fine-tune the VLM on the merged dataset, it is unclear which specific modules are fine-tuned (the whole model or some sub-modules)? And what is the effect of fine-tuning different modules?\n\n**A4.** Thank you for this question, which helped us to clarify the training details.  During fine‑tuning on the merged dataset, we follow the standard LLaVA‑1.5 instruction‑tuning recipe. Specifically, we keep the vision encoder frozen and fine-tune both the image‑text projection layer and the LLM parameters.\nEmpirically, updating only the projection layer yields a severe drop in multi‑turn reasoning and fails to correct linguistic hallucinations. Conversely, updating only the LLM impairs pure‑text tasks because the LLM over‑fits to visual cues. Jointly tuning both components achieves the best trade‑off. We have added these implementation details in Section 4.2 of the revised manuscript.\n\n\n> **Q5.** Although the authors mention that the self-refinement process can be conducted continuously, it seems that the performance has already saturated after one-round update. This makes the potential of the proposed method somewhat limited.\n\n\n**A5.** Thank you for this insightful observation. We acknowledge that in our experiments, the performance improvements tend to plateau after the first round of self-refinement. To provide clearer guidance to readers, we explicitly acknowledged the experimental findings in Section 4.5 and the limitations section. In light of your suggestions, we have added more discussions about this. We believe the diminishing returns may be due to the finite amount of useful information that unpaired images can provide about the underlying image distribution. Once this information is sufficiently leveraged, further gains naturally converge. At the same time, we observed that a second round of self-refinement still brings measurable improvements, suggesting that the full potential of this information cannot be fully exploited in just one iteration.\n\n\n\n> **Q6.** The authors briefly discussed the limitations in Sec. 5. However, I cannot find a specific section to discuss the societal impacts of their work. Some broader impacts that come to my mind: positive impacts: reduce human supervision and costs, negative impacts: induce potential bias using the synthetic data from the model.\n\n**A6.** We greatly appreciate your suggestion to include a dedicated discussion on the societal impacts of our work. We fully agree with the positive and negative aspects you mentioned. In light of your suggestions, we have added the following section about social impact:\n\nBy demonstrating that VLMs can autonomously improve themselves using only unlabeled images, our approach significantly reduces the reliance on expensive and labor-intensive human annotations. This has the potential to democratize access to high-performance VLMs, especially for low-resource domains and communities with limited labeled data. Despite the benefits, the use of synthetic data generated by the model itself introduces a risk of reinforcing or amplifying existing biases present in the pretrained VLM. Since no human-in-the-loop supervision is involved during the self-refinement process, biases or hallucinations in the model’s generation may propagate or compound. Furthermore, the autonomous nature of the learning pipeline could lead to unintended model behaviors that are harder to trace or audit."}, {"review_id": "BgMgb8UmO0", "reviewer": "Reviewer_rr1J", "summary": "This work introduced a metric based on Vision-Question-Answer data to measure the reliability of generated instructions on unlabeled data. This work presented Self-Refinement Framework for data generation and provided theoretical analysis for the framework.", "questions": "1. Was the dataset used in the MobileVLM model generated by itself or LLaVA-1.5?\n2. How do you tell the model to generate specific type of data as the prompts shown in Figure A1 do not provide the data type information?\n3. The link shown in line 100 of Appendix is expired.\n4. What is the meaning of \"k\" and \"k+1\" shown in Figure2? Please provide more explanations about the overview of the framework.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper introduced a self-refinement framework to utilize the capacity of the pre-trained VLM in data generation.\n2. This work conducted comprehensive experiments in ablation study for different types 他the generation data.\n\nWeaknesses:\n1. It is not fair for comparison with baseline as the SRF-LLaVA-1.5 is trained on a larger dataset than as shown in Table 1. And the paper does not provide sufficient details about used data and training procedures about Recap-LLaVA-1.5.\n2. There seems to be a conflict between the 2.8 million images, mention in line 227, and the external knowledge, mention in line 73", "was_revised": true, "final_justification": "After reading the answers, my questions about the methods and content have been resolved, so I will keep my original score. I hope the author can address these concerns in the final version of the paper.", "rebuttal": "Dear Reviewer rr1J,\n\nWe are deeply grateful for your positive assessment of our ablation-study design and for the valuable time you devoted to a thorough review of our manuscript. Your insightful remarks regarding the experimental workflow, baseline settings, and data-generation details have provided us with an excellent opportunity to enhance the paper’s clarity and readability. In accordance with your suggestions, we have added a more comprehensive discussion of the model-training pipeline and the data-generation process.\n\n\n> **Q1.** It is not fair for comparison with baseline as the SRF-LLaVA-1.5 is trained on a larger dataset than as shown in Table 1. And the paper does not provide sufficient details about used data and training procedures about Recap-LLaVA-1.5.\n\n\n**A1.** Thank you for raising this concern. SRF‑LLaVA‑1.5 is trained on an expanded dataset that includes an additional 200K unlabeled images compared to the original LLaVA. Thus, to ensure a fair comparison, we included evaluations against more baseline models that are also trained with the same augmented dataset.\n\nSpecifically, we compare SRF‑LLaVA‑1.5 with Recap‑LLaVA‑1.5 (Table 1), and the “Consistency” Bottom 20% model (4th row in Table 3), both of which use the additional unlabeled images. These comparisons demonstrate that the performance gains primarily stem from our self-refinement process, rather than simply the inclusion of more data.\n\nTo isolate the effect of self-refinement, we construct Recap‑LLaVA‑1.5 using the following two-stage procedure:\n1. Use LLaVA‑1.5 (7B) to caption 1M unlabeled images. (We call this process Recap.)\n2. Combine the resulting self-captioned image-text pairs with the original LLaVA‑1.5 training set, and finetune using the same training configurations as LLaVA‑1.5 and SRF‑LLaVA‑1.5.\nAlthough Recap‑LLaVA‑1.5 sees the same enlarged corpus, its performance gains are significantly smaller than those of SRF‑LLaVA‑1.5, as shown in Table 1.\n\nSimilarly, the “Consistency” Bottom 20% baseline follows the same self-refinement procedure but selects self-generated image-text pairs with lower consistency scores. While this model also leverages the unlabeled data, it does not apply our consistency-based filtering. The accuracy gap between our model and this baseline further supports that the improvement comes from our consistency-guided self-refinement, not just the volume of unlabeled data.\n\nIn light of your suggestion, we have added the training details of Recap‑LLaVA‑1.5 to Section 4.3 of the revised manuscript.\n\n\n> **Q2.** There seems to be a conflict between the 2.8 million images, mention in line 227, and the external knowledge, mention in line 73\n\n**A2.** We apologize for the unclear wording and thank you for pointing it out. In line 73, we meant “no external annotations were used.” The model is trained only on image–text pairs that are self-produced by the model itself; No additional human-labeled data or significantly stronger VLMs, such as GPT-4V, were employed. In light of your suggestions, we have replaced \"external knowledge\" with \"external annotations by human or other stronger VLMs\" in the revised version. \n\n\n> **Q3.** Was the dataset used in the MobileVLM model generated by itself or LLaVA-1.5?\n\n**A3.** Thank you for this question. The merged dataset for MobileVLM consists of (i) samples generated by MobileVLM itself through our self‑refinement loop on the 1 M unlabeled images, and (ii) its original supervised training set. We have added a detailed breakdown and training recipe in the appendix.\n\n> **Q4.** How do you tell the model to generate specific type of data as the prompts shown in Figure A1 do not provide the data type information?\n\n**A4.** We highly appreciate your insightful question. We do not enforce any particular question–answer type. The prompts simply ask the model to generate a question and its answer. The distribution of generated QA types is therefore emergent; Fig. A3 reports those proportions. We have clarified this in the caption of Fig. A1 in the revised version.\n\n\n> **Q5.** The link shown in line 100 of Appendix is expired.\n\n**A5.** Thank you for pointing it out. Because the NeurIPS 2025 rebuttal rules prohibit updating anonymous repositories during the discussion phase, the link must remain frozen for now. We will refresh the repository and verify permanent access immediately after the rebuttal period.\n\n\n> **Q6.** What is the meaning of \"k\" and \"k+1\" shown in Figure2? Please provide more explanations about the overview of the framework.\n\n**A6.** Thanks for your question. In Fig. 2, k denotes the current self‑refinement iteration and k + 1 the next. For example, the initial model LLaVA‑1.5 is M(0); after one self‑refinement round, it becomes SRF‑LLaVA‑1.5, labeled M(1). M(1) can seed another round, yielding M(2), and so on. We have added an explanatory sentence to the caption and extended the main‑text overview in the updated version for clarity. Below is our core algorithm (presented in pseudocode): \n```\nInput:\n    M0 ... initial VLM\n    D0 ... human‑labeled (I,Q,A) dataset\n    U  ... unlabeled images\n    K  ... number of refinement rounds\n\n\nfor k = 1 … K:\t# ----- iterative self‑refinement -----\n\n    # Stage 1 – Multi‑task fine‑tune (caption, VQA, instruction)\n    M0_gen ← train M0 on D0\n    S ← { (I,Q,A) produced by M0_gen for every I ∈ U }\n    S'← { (I,Q',A') | Q' ← M0_gen(I,A) ∧ A' ← M0_gen(I,Q)}\n    \n    # Stage 2 – Generate & filter synthetic IQA\n    For (I,Q,A) ∈ S, (I,Q',A') ∈ S':\n    \tif Q == Q' and A == A':\n    \t\tF ← (I,Q,A)\n\n    # Stage 3 – Instruction tuning with filtered data\n    D1 ← D0 ∪ F\n    M1 ← train M0 on D1  (instruction‑only objective)\n\nreturn M\n```"}, {"review_id": "dLEM21Uq2U", "reviewer": "Reviewer_zL5Z", "summary": "This work investigates the self-refinement capabilities of VLMs by introducing a training scheme where the model is trained to learn not only how to generate an answer (A) given an image (I) and a question (Q), but also how to generate A given (I, Q) and Q given (I, A). By forcing the model to learn these extra tasks, it is then used to generate IQA triplets for unlabeled images. The generated instructions are then filtered by the proposed triangular consistency check: for each synthetic triplet, the model tries to regenerate the question from the I+A and the answer from the I+Q, and the results are compared to the original Q and A. The model is then fine-tuned on the filtered synthetic data, improving its capabilities without any human annotations. Empirical results indicate that the method yields consistent yet modest improvements.", "questions": "- In line 201, what is the rationale for this relation $Y = \\phi (X, N_Y)$? I read [27], but I cannot see how you can explicitly use their assumption to VLMs.\n- Is there any empirical justification for (4) specifically for VLMs?\n- Given that SRF-LLaVA-1.5 has an unfair advantage over vanilla LLaVA-1.5 since the former has been expose to more images (even though unlabeled), have you tried to check the performance if you just use the same extra images with their true (Q, A) pairs (assuming there are available) to check how close you are to the fully supervised model. That could be considered as an upper bound for SRF.", "limitations": "Yes, but the discussion about limitations is too short.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The proposed three-step framework is logically sound and practical to implement.\n- The authors conduct extensive experiments on eight benchmarks, with results consistently favouring the self-refined model over the baseline, demonstrating the robustness of the improvement.\n\n\n# Weaknesses\n- Despite the consistent improvement of SRF-LLaVA over the vanilla version, most of the accuracy gaps are narrow, and there are no error bars to justify any statistical significance.\n- There is no total running time comparison between SRF-LLava and SRF-LLaVA. In general, SRF-LLava would require more computational resources, and it is not clear to a practitioner what is the computational cost is required to gain a few percentages of accuracy improvement.\n- The theoretical analysis is based on several arbitraty assumtions that is hard to verify and there is no discussion why the authors resort to these assumptions.", "was_revised": true, "final_justification": "My concerns have been addressed in the author's responses, so I will keep my initial score. Incorporating this discussion in the final version would strengthen the readability of the paper.", "rebuttal": "Dear Reviewer zL5Z,\n\nWe sincerely thank the reviewer for their thoughtful evaluation and recognition of our framework’s soundness, practicality, and consistent improvements across eight benchmarks. We appreciate the opportunity to clarify several points and address the concerns raised.\n\n> **Q1.** Despite the consistent improvement of SRF-LLaVA over the vanilla version, most of the accuracy gaps are narrow, and there are no error bars to justify any statistical significance.\n\n**A1.** Thank you for pointing this out. While the accuracy gaps are sometimes narrow, we respectfully emphasize that the consistent improvements of SRF‑LLaVA are achieved under the same training configurations as LLaVA, without relying on any additional human annotations or more powerful VLMs such as GPT‑4V.\n\nIn line with your advice, we have repeated the entire instruction-fine-tuning phase of SRF-LLaVA three times, each time using identical hyperparameters, the same random seed, and the same training-dataset order. The resulting mean ± standard deviation across the three runs has been reported in the updated Table 1 of the main paper, as shown in the table below.\n\n\n\n| Model                         | LLaVA-Wild  | MM-Vet      | MME-C        | MMBench     | MMBench-CN  | VQAv2       | GQA         | ScienceQA   |\n| ----------------------------- | ----------- | ----------- | ------------ | ----------- | ----------- | ----------- | ----------- | ----------- |\n| LLaVA-1.5-7B                  | 63.4        | 30.5        | 316.1        | 64.3        | 58.3        | 78.5        | 62.0        | 66.8        |\n| SRF-LLaVA-1.5-7B (mean ± std) | 67.2 ± 0.62 | 33.1 ± 0.06 | 332.0 ± 0.21 | 66.4 ± 0.16 | 59.1 ± 0.12 | 79.6 ± 0.03 | 62.4 ± 0.25 | 68.2 ± 0.35 |\n\n\n\n\n\n\n\n\n\n> **Q2.** There is no total running time comparison between SRF-LLaVA and LLaVA. In general, SRF-LLava would require more computational resources, and it is not clear to a practitioner what is the computational cost is required to gain a few percentages of accuracy improvement.\n\n**A2.** We regret the omission and have added full training‑time statistics to Section 4.4 in the revised version. At inference time, SRF‑LLaVA‑1.5 and LLaVA‑1.5 are identical in parameter count and therefore require exactly the same FLOPs and GPU memory.\n\nDuring the training of SRF-LLaVA-1.5, the larger number of images in the dataset naturally led to a longer training time. The table below compares the training durations of SRF-LLaVA-1.5 with those of the baseline LLaVA-1.5, both trained on a cluster equipped with 8 NVIDIA H100-NVL GPUs (96 GB each).\n\n\n| Model | Total wall‑clock time | Effective GPU‑hours |\n| -------- | -------- | -------- |\n| LLaVA‑1.5     | 7 h     | 56     |\n| SRF‑LLaVA‑1.5  | 12 h     | 96     |\n\nThus the refinement procedure costs an additional 40 GPU‑hours (≈0.7×). We now state this trade‑off explicitly in the paper so that practitioners can make an informed decision.\n\n\n>**Q3.** The theoretical analysis is based on several arbitraty assumtions that is hard to verify and there is no discussion why the authors resort to these assumptions.\n\n\n**A3.** Thanks for the questions. These assumptions we employed include (1) the independence of mechanism and input distribution, and (2) the Additive Noise Model (ANM).\nThese assumptions are well-established functional assumptions in causal inference literature, such as [1,2,3] rely on ANM and the independence of mechanism and input distribution to identify the causal directions.\n\n1. Why do we adopt these assumptions? \n\n(1) The independence of mechanism and input indicates that P(Y; image|X; text) contains no information about P(X; text). It introduces an asymmetry between cause and effect, since it will usually be violated in the backward direction. In Section 3.1, we discussed why this asymmetry holds true. Empirically, this asymmetry aims to provide implications about VLMs that we can use unlabeled images to produce the self-refinement of VLMs, but it is hard to use unpaired text sentences to produce the self-refinement.\n\n(2) The ANM, especially $Y = φ(X) + N_Y$, provides a functional assumption of the model in which the causal direction can be proved for identification. With such an assumption, the causal directions can be verified. \n\n2. Verifiability.\nWhile these assumptions may not be directly verifiable in the context of vision-language models (VLMs), they serve as necessary theoretical tools to justify plausible structural properties and to derive interpretable insights. These simplifying yet principled assumptions allow us to provide a theoretical explanation for why self-refinement is achievable.\n\nIn response to your comment, we have added a discussion in Section 3.3 of the revised manuscript to explicitly motivate the choice of these assumptions, clarify their connection to prior foundational work, and acknowledge their limitations.\n\nPlease let us know if any part is unclear. We are more than happy for a deeper discussion. \n\n\n[1] Zhang, Kun, and Aapo Hyvarinen. On the identifiability of the post-nonlinear causal model. 2012.\n[2] Shohei Shimizu, Patrik O Hoyer, Aapo Hyv¨arinen, and Antti Kerminen. A linear non-Gaussian acyclic model for causal discovery. 2006.\n[3] Janzing, D. and Scholkopf, B. Causal inference using the algorithmic Markov condition.  2010.\n\n> **Q4.** In line 201, what is the rationale for this relation $Y=\\phi(X, N_Y)$? I read [27], but I cannot see how you can explicitly use their assumption to VLMs.\n\n**A4.** Thank you for the question. The equation $Y = \\phi(X, N_Y)$ is intended to model the data generation process, not the behavior of the trained system. It follows the functional causal model framework, where text X is treated as the cause and images Y as the effect, which is consistent with the causal direction illustrated in Figure 3.\n\nWe do not impose this assumption during VLM training. Instead, we adopt this formulation to theoretically motivate our self-refinement approach. The key intuition is that unlabeled image data enables a more accurate estimation of the marginal distribution P(Y) (i.e., of images). Under the assumptions of additive noise and causal sufficiency, this marginal can reveal structure useful for learning a VLM that predicts text from images,i.e., modeling $\\hat{X} = f(Y)$.\n\n\n> **Q5.** Is there any empirical justification for (4) specifically for VLMs?\n\n**A5.** Thank you for the question! Regarding Equation (4), we currently provide empirical justification only in the context of synthetic data. Specifically, we construct data that follows the generative process outlined in Section 3.3, where the output distribution P(Y) is formed via a convolution of an underlying deterministic component and an independent noise variable.\n\n\nWe draw $X\\sim\\mathrm{Laplace}(0,1)$ and independent noise $N\\sim\\mathrm{Laplace}(0,0.6)$ in dimension $d=50$, then set $Y=X\\Phi^{\\top}+N$ with a random full-rank $\\Phi\\in\\mathbb{R}^{50\\times50}$. The data are split into $N_{\\text{lab}}=1900$ labeled pairs $(X,Y)$ for training, $N_{\\text{unl}}=4900$ unlabeled observations $Y_{\\text{new}}$ for self-refinement,  and $N_{\\text{test}}=1000$ labeled pairs for evaluation. A three-layer MLP with two hidden layers of 128 ReLU units takes a test vector $Y\\in\\mathbb{R}^{50}$ as input and outputs two 50-dimensional vectors $(\\mu,b)$: $\\mu$ estimates the conditional median of $X$ and $b>0$ its coordinate-wise Laplace scale. The network is trained on labeled data with the Laplace negative-log-likelihood loss $\\mathcal{L}(X,\\mu,b)=\\log(2b)+|X-\\mu|/b$ using Adam (learning rate $10^{-3}$, batch 128, 50 epochs). During self-refinement, the model predicts pseudo-labels $\\hat X_{\\text{new}}=\\mu(Y_{\\text{new}})$, keeps the 40 % most confident samples (smallest predicted scales $b$), retrains on the augmented set, and repeats for several rounds. On the test set we report **MSE**: $=\\frac{1}{N_{\\text{test}}\\!d}\\sum_{i=1}^{N_{\\text{test}}}\\lVert X_i-\\mu_i\\rVert_2^{2}$  and **$R^{2}$** computed by scikit-learn’s `r2_score` with `multioutput=\"uniform_average\"` (higher is better).\n| Setting            | NLL   | MSE    | $R^{2}$ |\n|--------------------|:-----:|:------:|:-------:|\n| Baseline           | 1.1840 | 0.4691 | 0.7635 |\n| After refinement   | 0.9046 | 0.2872 | 0.8551 |\n| Improvement        | $\\Delta=0.2794$ | $\\Delta=0.1819$ | $\\Delta=0.0916$ |\n\n\nThis controlled setup allows us to empirically verify the assumed conditions, and we can use more data of the effect variable Y to help the estimation of P(X|Y). However, we acknowledge that applying this decomposition directly to real-world VLMs remains a theoretical abstraction. We have clarified this limitation in Section 3.3 of the revised manuscript.\n\n> **Q6.** Given that SRF-LLaVA-1.5 has an unfair advantage over vanilla LLaVA-1.5 since the former has been expose to more images (even though unlabeled), have you tried to check the performance if you just use the same extra images with their true (Q, A) pairs (assuming there are available) to check how close you are to the fully supervised model. That could be considered as an upper bound for SRF.\n\n**A6.** We appreciate the constructive suggestion! We totally agree that it is an ideal manner to give an upper bound for SRF. Unfortunately, the extra images come from LAION and are unlabeled. Obtaining automated annotation with state‑of‑the‑art VLMs is beyond our budget: using GPT‑4V api to produce detailed captions for 2.8 M images would cost roughly $19,152 (estimated from the current OpenAI pricing). \nWe therefore apply an alternative manner to use open source sub-optimal model, Qwen 2.5-VL-32B to annotate the (Q, A) pairs.However, because of the model’s limited inference speed, the image annotations are still underway; we will update the experimental results during the discussion phase."}, {"review_id": "rhjoqLARut", "reviewer": "Reviewer_7zkJ", "summary": "The authors demonstrate that vision–language models (VLMs) can autonomously improve without any external supervision by leveraging a “triangular consistency” among images, questions, and answers: first, the base VLM is instruction-tuned to handle tasks like “image → question-answer” and “image+answer → question,” enabling it to generate candidate triplets from unlabeled images; next, each synthetic triplet is filtered by ensuring that any one element (image, question, or answer) can be accurately reconstructed from the other two, thereby retaining only high-quality pseudo-supervision.", "questions": "Refer to Weaknesses", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- Novel Self-Refinement Framework: Introduces a triangle-consistency principle to activate and guide the self-refinement capability of vision-language models without external supervision.\n\n- Causal-Theoretical Explanation: Provides a clear causal perspective that explains why and how triangle consistency leads to better model performance.\n\n\nWeaknesses:\n\nI feel it is great work, but there are some four concerns. First one is that the authors only use LLaVA-1.5 as baseline to apply your method, but this paper could be improved by using multiple recent released models such as Qwen2/2.5-VL and InternVL2.5/3 or the models you are using for evaluation baselines. The second is that there are none of challening benchmarks like MM-Vet-v2, MMMU, and MMMU-Pro. Third is that I am wondering if this paper could be improved by the causal theoretical analysis like backdoor adjustment (Treatment T, Outcome Y, Confounder X, and Counterfacture Z). The last one is that there are none of ablation studies regarding your dataset can be used for just SFT or different way training instead of your method. This aims to identify if the performance gain is not just using more dataset or more training time.", "was_revised": true, "final_justification": "The authors addressed the concerns that initially I mentioned, so I've updated the score. Especially, I recommend the authors should present many variants of models and challenging benchmarks' score as well as clarify ablation studies to be emphasized.", "rebuttal": "Dear Reviewer 7zkJ,\n\nWe sincerely thank the reviewer for their thoughtful and constructive comments, and for recognizing the novelty of our self-refinement framework and its causal-theoretical grounding. Your comments helped us to improve the completeness of the evaluations of our methods.  In light of your suggestions, we add the evaluation with recent released backbone and new challenging benchmarks and clarify the ablation studies.  We respond to the four main concerns raised below:\n\n> **Q1.** The authors only use LLaVA-1.5 as baseline to apply your method, but this paper could be improved by using multiple recent released models such as Qwen2/2.5-VL and InternVL2.5/3 or the models you are using for evaluation baselines. \n\n**A1.** Thank you for highlighting the importance of demonstrating the generality of our Self‑Refinement (SRF) framework across diverse model families.\nFollowing your suggestion, we have added Qwen‑2.5‑VL‑3 B as a new baseline. We reused the exact 1 M unlabeled images and three‑stage training pipeline described in the manuscript, and we kept all hyper‑parameters identical to those of the original Qwen 2.5‑VL‑3B release. The resulting model SRF‑Qwen 2.5‑VL‑3B is compared with the vanilla Qwen 2.5‑VL‑3B on 6 widely‑used benchmarks in table below. \n\n\n\n| Model[1]             | MMMU[2] | MMMU Pro[3] | MathVista | MathVision | MMStar | MMBench1.1[4] |\n| ----------------- | ---- | -------- | --------- | ---------- | ------ | ---------- |\n| Qwen2.5-VL-3B     | 53.1 | 31.6     | 62.3      | 21.2       | 55.8   | 81.5       |\n| SRF-Qwen2.5-VL-3B | 55.0 | 31.9     | 64.7      | 23.3       | 56.5   | 80.8       |\n\n[1]: SRF-Qwen 2.5-VL-3B was evaluated with VLMEvalKit; Qwen 2.5-VL-3B scores are taken from the official model documentation.  \n\n[2]: Results are obtained on the MMMU validation (VAL) set.  \n\n[3]: Results are obtained on the MMMU Pro Standard (10-choice) split.\n\n[4]: Results are obtained on the MMBench1.1 Dev English split.\n\n\n> **Q2.** There are none of challenging benchmarks like MM-Vet-v2, MMMU, and MMMU-Pro.\n\n\n**A2.** We appreciate the recommendations of these challenging benchmarks. In light of your suggestions, we have now evaluated SRF‑LLaVA‑1.5 and the original LLaVA‑1.5 on MMMU and MMMU‑Pro. The new results (see revised Table 1) are shown in the table below.\n\n| Model[5]            | MMMU | MMMU Pro \n| -----------------| ---- | -------- | \n| LLaVA-1.5-7B     | 34.7 | 17.6 |\n| SRF-LLaVA-1.5-7B | 36.6 | 18.3 |\n\n[5]: For both LLaVA-1.5-7B and SRF-LLaVA-1.5-7B, we obtained the results through local evaluation using VLMEvalKit.\n\n\n\n\n> **Q3.** I am wondering if this paper could be improved by the causal theoretical analysis like backdoor adjustment (Treatment T, Outcome Y, Confounder X, and Counterfacture Z). \n\n**A3**: Thank you for the insightful suggestion! We carefully considered the potential of applying causal theoretical frameworks, such as backdoor adjustment, to improve VLMs. Backdoor adjustment is primarily designed to enable causal inference by mitigating bias introduced by observed confounders. However, in our current setting to learn VLMs \\hat{X} = f(Y), we do not explicitly identify any clear confounders that would justify applying such an adjustment.\n\nAt this stage, it appears that backdoor adjustment may not directly benefit our approach. That said, we are very open to further exploring this direction and would greatly appreciate any additional insights you may have on how such causal tools could be effectively integrated into VLM training or evaluation.\n\n\n> **Q4.** There are none of ablation studies regarding your dataset can be used for just SFT or different way training instead of your method. This aims to identify if the performance gain is not just using more dataset or more training time.\n\n\n**A4.** Thank you for insisting on this clarification. We performed an ablation in which we replaced the Self‑Refinement procedure with a straightforward recapitulation baseline Recap‑LLaVA‑1.5, which was  constructed as follows (The detailed comparison can be found in Table 1:\n\n(i)Starting from the same 1 M unlabeled images used by SRF‑LLaVA‑1.5, we employ the frozen LLaVA‑1.5‑7 B to produce one detailed caption per image.\n\n(ii) We take all 1 M generated captions without any filtering and mix them with the original LLaVA‑1.5 training set.\n\n(iii) We fine‑tune the model under the same schedule as SRF‑LLaVA‑1.5.\n\nDespite having the same data volume and training steps as SRF‑LLaVA‑1.5, Recap‑LLaVA‑1.5 achieves only marginal improvements over the vanilla model (see Table 1). By contrast, SRF‑LLaVA‑1.5 attains substantially larger gains. These results indicate that the benefits stem chiefly from the Triangular Consistency constraint and iterative error‑correction built into Self‑Refinement, rather than from simply exposing the model to more unlabeled image data or longer optimization.\n\nAdditionally, the ablation studies presented in Table 3 (the “Consistency” group) further support that the performance gains are not solely due to the use of unlabeled data. In particular, the “Consistency” Bottom 20% baseline applies the same self-refinement procedure but selects self-generated image-text pairs with low consistency scores. Although this baseline also leverages the same unlabeled data, it lacks our consistency-based filtering mechanism. The notable accuracy gap between our model and this baseline highlights that the improvement is primarily driven by consistency-guided self-refinement, rather than merely the presence of additional data."}]

Author rebuttal:

Meta-review: This paper introduces a triangular consistency self-refinement framework for vision–language models (VLMs), demonstrating that models can autonomously improve using unlabeled images without external supervision. The idea is original and practically relevant, with clear causal-theoretical grounding and consistent empirical improvements across multiple benchmarks.

Pros:
* Novel principle (triangular consistency) to activate self-refinement in VLMs.
* Comprehensive experiments, including on LLaVA-1.5 (7B, 13B), MobileVLM, and Qwen-2.5-VL-3B.
* Ablation studies clarify that gains stem from consistency filtering rather than dataset size.
* Rebuttal and final revisions added error bars, training cost analysis, and societal impact discussion, addressing reviewer concerns.
* Reviewers agree that clarifications and additional experiments sufficiently resolved initial weaknesses.

Cons:
* Improvements, while consistent, are modest in scale.
* Training overhead (≈40 extra GPU hours) may limit practicality for some practitioners.
* Theoretical analysis relies on assumptions that remain somewhat abstract and hard to verify empirically.

Although the gains are incremental, the contribution is conceptually strong, methodologically sound, and well-supported by additional experiments. The reviewers’ concerns were adequately addressed, leading to raised or maintained positive scores. Overall, this is a solid and timely contribution to the study of self-refinement in VLMs and merits acceptance.
Decision: Accept (poster)