Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs
Authors: Zhiyu Pan
Yizheng Wu
Jiashen Hua
Junyi Feng
Shaotian Yan
Bing Deng
Zhiguo Cao
Jieping Ye
Venue: ICLR 2026 Oral (2026)
Keywords: Reasoning
Vision-Language Models
Contrasting
Primary area: foundation or frontier models, including LLMs

Abstract: Reasoning has emerged as a key capability of large language models. In linguistic tasks, this capability can be enhanced by self-improving techniques that refine reasoning paths for subsequent fine-tuning. However, extending these language-based self-improving approaches to vision language models (VLMs) presents a unique challenge: visual hallucinations in reasoning paths cannot be effectively verified or rectified. Our solution starts with a key observation about visual contrast: when presented with a contrastive VQA pair, i.e., two visually similar images with synonymous questions, VLMs identify relevant visual cues more precisely compared with when given a single VQA sample. Motivated by this observation, we propose Visual Contrastive Self-Taught Reasoner (VC-STaR), a novel self-improving framework that leverages visual contrast to mitigate hallucinations in model-generated rationales. We collect a diverse suite of VQA datasets, curate contrastive pairs according to multi-modal similarity, and generate rationales using VC-STaR. Consequently, we obtain a new visual reasoning dataset, VisCoR-$55$K, which is then used to boost the reasoning capability of various VLMs through supervised finetuning. Extensive experiments show that VC-STaR not only outperforms existing self-improving approaches but also surpasses models finetuned on the SoTA visual reasoning datasets, demonstrating that the inherent contrastive ability of VLMs can bootstrap their own visual reasoning. The code, dataset and trained models will be released upon acceptance.

Published as a conference paper at ICLR 2026
THROUGH THE LENS OF CONTRAST: SELF-IMPROVING VISUAL REASONING IN VLMS
Zhiyu Pan Yizheng Wur Jiashen Hua" Junyi Feng", Shaotian Yan', Bing Deng',
Zhiguo Cao', Jieping Ye²
Huazhong University of Science and Technology. PAlibaba Cloud
ABSTRACT
Reasoning has emerged key capability of large language models. In linguistic tasks, this capability can be enhanced by self-improving techniques that refine rea- soning paths for subsequent finetuning. However, extending these kunguage-based self-improve approaches to vision language models (VLMs) presents unique challenge: visual hallucinations in reasoning paths cannot be effectively verified rectified. Our solution starts with a key observation about visual contrast: when presented with contrastive VQA pair, i.e,, two visually similar images with syn- onymous questions, VLMs identify relevant visual cues more precisely. Moti- vated by this observation, we propose Visual Contrastive Self-Taught Reasoner (VC-STaR), novel self-improving framework that leverages visual contrast mitigate ballucinations in model-generated rationales. We collect a diverse suite of VQA datasets, curate contrastive pairs according to multi-modee similarity, and generate rationales using VC-STaR. Consequently, we obtain new visual reason- ing dataset, VisCoR-55K, which is then used boost the reasoning capability of various VLMs through supervised finetuning. Extensive experiments show that VC-STaR not only outperforms existing self-improving approaches but also sur- passes models finetuned on the So'T visual reasoning datasets, demonstrating that the inherent contrastive ability of VLMMs can bootstrap their own visual reasoning. com/ hhtoo/hhhyyaaa22// STaR
Project at:
INTRODUCTION
The scaling of large language models (LLM) has led to the emergence of reasoning capabilities (Wei (ct 2022a), making a transition from System 1 to System 2 (Kahneman (011]) and enabling lan- guage models tackle complex, multi-step problems (Wei al. 2022b Kojima al.. 2022) This emergent ability can be further enhanced by various techniques (Wang et al. 2023b Li et al. 2023b
Hao et al. 2023 Gao al., 2023 OpenAl 2024b Guo al. 2025). Among them, self-improving
approaches (Zelikman et al. 2022 Gulcehre et al. 2023 Madaan et al. 2023 Qu et al. 2024 Ma
al. 2025) form a prominent branch, mainly because they can be easily applied and extended with-
out external reward models (Lu et al. (024a),, predefined step decomposition (Liu et al. 2025), or
specially designed reasoning structures (Li al. 2025).
However, it is infcasible to directly adapt such language-based self-improving methods to vision language models (VLMs) (Liu al., [203]] Bai al. 2025). Previous self-improving approaches focus on textual coherence and the quality of the final answer (Zelikman et al., 2022] Zhang et al.) 2024a), while they are unable to verify or rectify the visual hallucinations that persist in current VLMs (Tong et al.] 2024 Li et al.) 2024). Even worse, they may get stuck in speculative reasoning that privileges textual priors over real visual evidence (Favero et al.] 2024, Wu et al., 2025). We claim that the key problem for self- infroving in VLMs is: how rectify visual hallucinations in VLMs' reasoning paths for high-quality visual rationale generation.
Our solution is built upon interesting observation: VLMs can see better during contrasting. As shown in Fig. Ta the VLM generates a wrong rationale with visual hallucinations given a single visual question answering (VQA) sample. Instead, when presented with contrastive VQA pair,
Same contribution.
Corresponning author and project leader.

---

Published as a conference paper at ICLR 2026
Naive Answering
Contrasting and Answering
Disotmictte between two images for two images respectvels, for the fist mage: What lind or trick is the skatcbeander pe perfoning? for second image: What' skotebborddi
Think step by step to an- twer this queetion what boarder porforming?
User
User
doing?
the skatebnarder appears eaccuting an olts, one most fundamental trikk they jame skatchpording ding their board into mida attached undermeeth them without uning any handrais objects...
The first image shows siotebcander babancing with one foot whill skiting along the edge of an elecated mmpp using their back wheels (bxll, Indiratin they might be evecuting attempting wher anuld revemble a
VLMss better during contrasting
both quections regarding dent fying specific tncks based purely sisualy from these descr pilons alone: 13 **Fist Image." The trick sppears more aligned towards "hill Siiee, where balance and control over near portion dominstte...
(a) Visual hallucinations within the reasoning paths can mislead the model. Contrasting within a contrastive VQA pair, the VLM may rectify its own hallucinations.
Results and Statistic
Dow Creck
Warbbe of Herd Sample Pairs
(b) Results and statisties of rectifying hallucinatory outputs by three settings. H: with hint; C: via contrasting.
Figure 1: Contrasting makes the VLM see better. (a) Contrastive VQA pairs compels a more accurate response. (b) Compared with a previous self-improving method STaR (Zelikman et al.] 2022) that enhances the quality of reasoning with hints (ground-truth answers), contrasting with hints can rectify more cases. The blocks along the x-axis mark initial VLM failures. The color of each block indicates the outcome of rectifying: green for success and gray for failure. Tested VLM is Qwen2.5VL-7B (Bai ct al. 2025).
Le., two similar images with synonymous questions (Setting C in sub-figure Tb). the model captures fine-grained visual evidence more accurately and rectifies the erroneous rationale. Statistics of this phenomenon on group of failure cases are shown in Fig. [b] Compared with the hints-only (Setting in sub-figure Tb) self-improving (provide the model with the ground-truth answers), the hints and contrasting (Setting C&H in sub-figure lb) setting not only prevents the model from making new errors but also leads to the rectification of its original hallucinations.
Motivated by this, we propose a new self- improvipg framework, Visual Contrastive Self-Taught Reasoner (VC-STaR). VC-STaR contains three steps: (1) think step by step and generate coarse rationale; (2) compare visual queries in contrastive VQA pair and provide contrastive analysis; (3) rethink and refine the coarse rationale via an LLM based on the contrastive analysis. In order to guarantee the scalability of VC-STaR, we also propose a task-agnostic contrastive VQA pair curation framework, which can be readily adapted to various VQA tasks, e.g., reasoning (Lu et al. 20216). math (Gao et al. 2025), chart Liu et al. (2024a), and OCR (Yuan et al.] (022). Specifically, curate the contrastive VQA pairs within individual datasets, based on the similarity of both images and questions. We utilize these contrastive VQA pairs to generate faithful rationales, resulting in
novel Visual Contrastive Reasoning dataset (VisCoR-55K3 as illustrated in Fig. 2 Finetuning with VisCoR-55K enhances VLMs' visual reasoning capability.
VC-STaR achieves prominent results on a wide range of challenging benchmarks, including MMVP (Tong al. (2024). HallusionBench (Guan al. 2024). MathVista (Lu et al. 2024b), Math Vision Wang et al. (2024), and MMStar (Chen et al. 2024c). On the one hand, VC-STaR outperforms existing self imfroviigigg baselines. On the other hand, it exhibits clear advantage over models trained on recently proposed reasoning datasets. The experimental results validate that visual reasoning capability of VLMS can be bootstrapped through the lens of contrast.
RELATED WORKS
Reasoning in Language. Dual-system theory (Kahneman 201T) illustrates two systems in human cognition: fast, intuitive System 1 and slow, deliberate System 2 which is akin to emergent
2

---

Published as a conference paper at ICLR 2026
General (42.7%) (%)7%)
Reasoning 25.0%)
Reasoning
VisCoR 55K
Chaph
General
Where are they located relative to each other? dog on
If you select marble without looking, which color are you less likely to pick?
positioned the right the left the dog.
21 VQA Datasets Categories
The answer blue.
The answer black.
Math
Graph/Chart (8.8%)
QCR (7.8%)
Orange candies
Stenl
Lest
Drange candiea bag
i
Siem
SSLS 8o-X
2.
25.
099
11:4866
1244459
345ES
USL3
B
6600
27
Assume that segments appear to tangant tangent.
What the smallest number of orange candies?
1.1.66
Please write out the expression formula image using
smallest number of orange candies is 11 orange candies.
The smallest number of orange candies is 31 orange candiess
Veas langle 5 Vrac (B) (frrcc (1)(2)=)(4xx)
leos Yangla 8
The answver 30.
The anower is
(4){)
Figure 2: Vi5CoR-55K. We introduce the Visual Contrastive Reasoning dataset (VisCoR-55K)) new collection of 55K high-quality visual reasoning samples. Spanning the domains of general VQA, reasoning. math, graph/chart, and OCR, each sample is created by leveraging contrastive counterpart to generate a faithful rationale. Rationales are shown in the Sec. A.4
reasoning capability of LLMs (Wei et al. 2022a). Consequently, reasoning enhancement (Wei et al, 2022b Kojima et al. 2022) is considered pathway to elevate LLMs' cognitive performance. One solution involves a reward model (Li et al. 2023b Lu et al, 2024a). often coupled with Monte Carlo tree search (Hao et al.) 2023 Zhang al. 2024a). to discover optimal reasoning paths. However, this solution is constrained by the need of an auxiliary model and the requirement for reasoning step dividing (Liu et al. 2025). Another way employs macro reasoning actions (Gao et al. 2023 Khot et al. [223] Yang et al.] 2025a) to inject human prior knowledge, however, hand-crafted macro diverse reasoning scenarios. While reinforcement learning (Rafailov et al., 2023 Trung et al. 2024, Guo et 2025) has also attracted the attention, its success relies on the data format and the design of reward functions. Self-improving methods (Zhang al. 2024a) offer more scalable alternative, enabling LLMs to refine its own reasoning by constructing high- quality reasoning data (Wang et al. 2023b), utilizing ground-truth answers as hints (Zelikman et al, 2022). leveraging internal feedback (Qu et al. 2024). With fewer external constraints, self- improving methods pave the way for more flexible and general language reasoners.
actions struggle
adapt
Reasoning in Vision. Human reasoning is stimulated not only by textual input but also by visually- related queries. Fostering the visual reasoning ability (Zhang et al.] (024c) for VLMs (Liu et al.] 2023 Li et al. 2023a) therefore critical frontier topic. Harly attempts often rely external scaffolding like scene graphs (Mitra et al.) 2024), macro actions (Xu et al. 2025) Dong et al. 2025), or bounding boxes highlighting key region in images (Shao al. 2024). However, such approaches suffer from fundamental limitations: they are constrained by the data structure or tend to generate stereotyped reasoning paths. Despite these advances, the self-improving paradigm which has shown its effectiveness in text-only domain is underexplored for visual reasoning. The primary obstacle the visual hallucinations embedded in reasoning paths cannot be easily rectified by existing text- centric self infi-iirriing frameworks /Zhang et al. 2024a Zelikman et al. 2022) Qu et al. 2024). The proposed VC-STaR attempts to bridge this gap through the lens of contrast.
Power of Contrasting. Contrasting has shown effectiveness in wide range of machine learning topics. By comparing different views (Tian et al.) 2020), e.g., data augmentations, of the same sample while distinguishing them from others (Wang & Isola 2020), contrastive self-supervised learning methods (He et al.) 2020; Grill et al., 2020 Radford et al. 2021 Liang et al. 2022 Pan et al. 2023) excel at learning potent feature representations. Explicitly cross image contrasting is also studied under uni-modal setting (Pan et al.
2023 Ding et al. 2024 Chen et al. 2024a) and
multi-modal setting (Park al.
2019 Kim al.
2021 Yao et al. 2022 Dunlap et al. 2024) Based
3

---

Published as a conference paper at ICLR 2026
on these advancements, VLMs endowed with robust capabilities for multi-image -mage comprehension
and comparison (Alayrac et al. 2022 Bai et al. 2025 Chameleon 2025 Lin et al. 2025). Some
prior works have leveraged contrasting create better perrucctiontuuingg data (Jiao et al. 2025 Ma
et al. 2024). However, how contrasting can help visual reasoning remains an open question. We observe that VLMSs' inherent comparative ability be repurposed to actively suppress its own visual hallucinations, bootstrapping their visual reasoning capability. This discovery offers a new perspective about the power of contrasting in reasoning.
VISUAL CONTRASTIVE SELF-TAUGH REASONER (VC-STAR)
Let f be VLM and D = {(us; qi, a:) be a set of visual question answering (VQA). The VQA set consists N triplets, where Vis qi and a, represent the i-th image, question, and corresponding ground-truth answer, respectively. Following previous self-taught reasoners (Zelikman et al. [2022] Madaan et al. 2023), the original VQA dataset D can be enriched by generating rationale r with 8 for each triplet. which transforms D into visual reasoning dataset R {ee, a,, r.) } However, as mentioned in Sec. D rationale r, may be contaminated by visual hallucinations. Mo- tivated by the observation illustrated in Fig. [ VC-STaR aims to refine rationale т; into more faithful one r by contrasting the (Ma, qi, a:) with a contrastive VQA counterpart sample (6, 4t as) where qe is synonymous with di and U, shares similar context with é. The contrastive VQA pairs P = {(v qi, az), (ti, q., d.)) }ー support the contrasting and rationale refining process. The con- trastive VQA pairs are curated by searching (V, G. di) for (u; qt. a,) within diverse data groups in D for different VQA tasks, ensuring the generalization of VC-STaR. The VC-STaR is designed to address two key challenges: (1) how to curate meaningful contrastive VQA pairs; (2) how to transfer the fine-graings discriminative ability from dual-image contrasting to refine the single-image reason- ing. Sec. 3.1 elaborates on the pipeline for curating contrastive VQA pairs. Building upon this foun- dation, Sec. 3.2 introduces our contrasting and rethinking procedure which embeds the dual--imgee comparison into a new reasoning path, guided by an LLM, to produce more faithful rationale. The refined rationales are then used to construct a new reasoning dataset R = {(11, 9s. ag F.) which we name the Visual Contrastive Reasoning dataset (VisCoR-55K), The VLM @ updated to new version with improved reasoning capability by finetuning on VisCoR-55K.
3.1 CONTRASTIVE VQA PAIR CURATION
To ensure the generalization of VC-STaR, the contrastive VQA pair curation pipeline should be flexible enough across wide spectrum of VQA tasks. For better contrasting, each contrastive VQA pair ((vs, 9s: az), (0i, q., d.)) should possess three key properties: (1) 9i and q, are synonymous. This shared question acts as a semantic anchor, grounding the two images e; and & at the same point in the semantic space. The images thus represent different manifestations of this anchor, providing solid basis for contrasting: (2) 2/ and iú are visually similar. U; and i; should not be trivially distinct but exhibit visual similarity, creating confusing contrasting. This visual proximity compels VLMs to engage in fine-grained contrasting to discriminate subtle differences; (3) qu is reasoning dependent. qi should be sessoning-provoking rather than one that can be solved by straightforward answer. To achieve these requirements, as illustrated in Fig. 3 we propose a three-stage curation pipeline:
Data Collection. We collect 21 VQA datasets spanning five categories: reasoning (Zhang et al. 2019 Kiela et al. 2020 Lu et al. 22221b), graph/chart (Kembhavi et al. 2016,] Mathew et al. 2022 Masry al. [022] lang et al. [2023] Lu et al. 2023 [iu el | (0224a), math (Lu et al.) 202 Cao
& Xiao 2022 Gao et al. (025), general (Zhu et al. [2016] 6 Johnson et al. 2017 Acharryy et al.) 2019
Schwenk et al.| 2022 Wang et al. 2023a Chen et al. al.] 2024b). and OCR [CDAR][009]] Yuan al.
2022 Zhang et al., 2024b). This broad collection enriches the diversity of our curated pairs, which ensures the generalization ability of the finetuned model.
Contrastive VQA Pair Hunting. In order to compute the similarity of VQA pairs, we first rep- resent the question gi and the image v, by high-dimensional embeddings, denoted as eğ and ej respectively. We use GTE (Li et al. 2023c) text embeddings to represent the questions. In terms of image embedding, existing models fall into two types. i.e., langun-language contrastive learning approaches (Radford et al. 2021: Tschannen et al. 2025) while vision-only self-supervised learn- ing methods (Zhang et al. 2023 Oquab et al. 2024). The former ones mainly capture at global semantic information, while the later ones are good at instance discrimination. Neither of them
4

---

Published as a conference paper at ICLR 2026
Data Collection
Contrastive VQA Pair Hunting
Difficulty Based Sampling
Right Anower
What person ssing"
Sample
Median
Vasterting
Cample
Tlasaresin
LangerCChatt
Wrong AreaNe
VUA dotasets for different tacks
Mash
Cerional
DCT
Imaps Erhecding Misching
Sample
Wronn Arower
Figure 3: Contrastive VQA pair curation pipeline. To facilitate effective contrastive analysis, we curate corresponding challenging counterparts for VQA samples from pool of diverse datasets. Each curated pair consists of two samples that share a synonymous question but feature distinct yet semantically similar images. Collected pairs are filtered by a difficulty-bssed sampling procedure.
Contrastive VQAs
Think
Rethink
Please think
this question.
Please refine rationale based analysis.
answer this question. I am focusing on the woman's behavior and interaction sitting actively engaging anything: simply the direct concusion that woman on the bench.
surroundings. notice
To the doing focusing bohavlor her interaction with surroundings. She sitting bench
Oo What is the person doing? She reading book.
Contrast
the detail she's holding: booklet. looking off distance holding something else. probobly enjoying the the content point interacting with hands. the likely that she's reading.
Please discrmminate clearly between provided images, answer question both images respectively.
looks like
The first image shows woman sitting bench park or garden. she's holding what looks like book or some paperrs so soems
all in
reading, contrast, second cloutyy day. She's holding red umbrolla and looking out at the horizonn. So, for questions reganding their actions: 1) most reading. 2) "Sacord ** simoly enjoying the
Qu sho doing? A: She enjoying the sconery.
**First Imaga**
Figure 4: Faithful rationale generation pipeline. A contrastive analysis can be obtained based on the curated contrastive VQA pair. Leveraging the property of VLMs illustrated in Fig. [T] the contrastive analysis then used to trigger rethinking procedure, which refines the naive rationale into a more faithful one. This pipeline is designed to generate rationales for supervised finetuning.
generic enough adapt to the diverse domains. To tackle the dilemma, build versatile visual embedding model based on ID-based visual metric learning (ypsilantis et al.) 22024 An et al. 2023
Hunting for counterpart (0,, q., d;) is then performed dataset-by-dataset A sample (vy, q a, is recalled as a valid counterpart if it satisfies: >(e,, e²)) < o, and <(e7,e)) ez) < Og. where ((s is the cosine distance, and o, and oy are pre-defined thresholds for visual and question similarity, respectively. Any sample that fails to meet both conditions is dropped.
Difficulty-Based Data Sampling. For the goal of developing visual reasoning capability, should be a difficult question requires reasoning rather than a straightforward one. We define the levels of difficulty based on the performance of VLM θ: (I) easy samples are with the simple qi which can be correctly answered by 0 without any auxiliary help; (2) median samples are with & which makes 0 initially fails but succeeds when contrasting with (6;, 4s, á;) based on provided hint (the C&H setting introduced in Fig. [] (3) hard samples are the ones with gi that cannot be correctly addressed by 0 even with the help of contrasting. We only keep median-di difffcuult contrastive VQA pairs for the rationale generating.
3.2 CONTRASTING AND RETHINKING
Rationales in the reasoning dataset R [(V,, di a; TM) generated by the VLM 0 itself include visual hallucinations. To achieve the goal:
R {1U, 9t a; M.) + R = {(v, qs 0; P.M
(1)
where f is the rectified rationale, we use the contrastive VQA counterpart (v,, q:, de) to provoke a rethinking action refine r; into F.. As illustrated in Fig. 4 this pipeline includes three steps:
Thinking step. Following the design of Zelikman et al. 2022 to provide the VLM 9 with ground- truth answer a; as hints, prompt the VIM / to generate the coarse rationale r; for the target VQA
5

---

Published as a conference paper at ICLR 2026
sample (t,, (s: 0z) as follows:
f(v,, Qs a,)0,8")
where f is inference process with "thinking prompt" 5'. Details of 8 are in the Sec.
A.3
Contrasting step. Asking the VLM to compare the target VQA sample (U; 9: a;) with its con- trastive counterpart (ve, gi, úz) results in a contrastive analysis c, which may provide more faithful visual information:
= (((r, 4i 0s), (6i, 4i, 0,,0)
where the 8 is the "contrasting prompt". When a, has the same meaning as d;, 8 requires summa- rizing the common patterns of v; and 8;; When a, is different from d., 8 expects the analysis about the fine-grained differences between and vi. Details of de are in the Sec.A...
Rethinking step. As demonstrated Fig. 1 G is more trustworthy than ri- Hence, we adopt a LLM vp to transfer the information from q to new reasoning path according to r;:
f = [r;; G.l.""")
(4)
where 8" is the "rethinking prompt" which asks the LLM & to rectify the visual hallucinations in r; according to the visual information from 5" requires LLM & to respond like directly answering the question q; details are in the Sec. A.3
To ensure the quality of the R. we finalize the visual reasoning dataset by employing a text-macching post-prov post-processing to filter out samples that contain incorrect reasoning patterns. The final visual rea- soning dataset contains 55K VQA samples with corresponding rationales, a.k.a., the VisCoR-55K.
EXPERIMENTS
Section details experimental setup, including the supervised finetuning process and the benchmarks used to evaluate the effectiveness of the VC-STaR. In Section[4.2] we present a compre- hensive performance comparison. As self-improving method for visual reasoning, benchmark VC-STaR against two primary groups: (1) other self-improving baselines adaptable to visual reason- ing, and (2) models trained on off-the-shelff visual reasoning datasets. Finally, Section 4 provides in-depth ablation studies on designs of our method, including the contrastive VQA pair construction, the generalization on other base models, the difficulty sampling strategy, and the effect of the types of contrastive VQA counterpart.
4.1 SETUP
Implementation Details. Using the LLaMA-factory framework (Zheng et al. 2024). finetune the model for 3 epochs via fall-parameter supervised finetuning (SFT), with the vision tower's s pa- rameters frozen. The SFT utilizes learning rate of 1e-5, batch size of 256. The inference process of the finetuned model does not require such a contrastive pipeline illustrated in Fig. 4 and it fol- lows the standard inference paradigm of VLMs. As for the curation of contrastive VQA pair, the question similarity threshold σ, is set to 0.15 and the visual similarity threshold é, is set as 0.5 for datasets of general images. For the datasets including icon, geometry, chart or graph images, the visual similarity threshold & is set as 0.3. The LLM v used in the rethinking step of our rationale generation pipeline is the oounesourced Qwen2.5-72B.
Evaluation Benchmarks. We employ 6 benchmarks designed to assess its robustness against hal- lucination, mathematical reasoning, and general abilities. The MMVP (Tong et al., (022) and Hal- lusion (Guan et al. 2024) benchmarks focus on visual hallucination, and the Math Vista (Lu et al.) 2024b) and Math Vision [Wang et al.) 2024) benchmarks are about the mathematical reasoning. The MMStar (Chen al.) 2024c) is highly curated benchmark, composed of purified samples from multiple benchmarks, e.g., MMMU (uue et al. 2024) and MMBench Liu et al. (2024b). The MME- RealWorld benchmark Zhang et al. (2025b) is large-scale,
benchmark for diffi- cult, real-world tasks. Therefore, MMStar and AME-RealWort are suitable to evaluate the general perceptual and cognitive abilities under varied scenarios.

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison with self-improving baselines and the models trained on off- the-shelf visual reasoning datasets on hallucination, math, and general benchmarks. We adopt the 2ven2.5VL--B as our base model, and report its reasoning performance as a baseline. MME- RW is short for MME-RealWorde Zhang et al. (2025b); RI-OV is short for R1-Onevision (Yang) et al. 2025b). Blue (red) numbers in parentheses represent performance gains (drops) relative to the baseline. The best performance is in boldface, and the second best is underlined.
Bench.
Hallucination
Math
General
Avg.
Method
MMVP
Hallusion
MathVista
Math Vision
MMStar
MME-RW
Base Model
70.0
53.1
68.4
24.0
61.8
55.9
55.5
VQA SFT
74.3643)
54.2(.1.))
65.4(..0)
19.4.4.6
59.7G2.1
56.8
55.0ka.))
Self-Improving Approcchss
STaR(2022)
73.0ca3.)
55.9(22)
66.9(1.5)
19.8:4.2)
58.9(-2.9
58. (42.
55.4
Verifier 2024a
73.7(437)
(3.2(40.))
67.0c.44)
20.3
58.2(336)
56.7(4a.)
54.9.006)
Feedback 2024
75.0x+5.0y
53,4
68.8
22.16L19
63.2(+1.4)
+6.0
56.4(+0.9)
Off-the-Shelf Visual Reasoning Datasets
Virgo 2025)
68.0(2.0)
(7.2(559)
(395
21.5(2.5)
59.7(2.))
29.4: 26.5)
48.2(-1.)
LLaVA- 2025
71.7(+1.7)
50.3
68.4(.0.0)
24.4c+0.4)
(3.1
59.3(+3.4)
56.2(+0.7)
R1-0V
68.0k2.0)
55.8(4277)
68.2(-02)
25.4(+1.4)
53.2(8.6)
(6.3
52.8
(025)
74.0c+4.0)
53,4<40.3)
69.2(+0.8)
24.2
64.3(+2.5)
(
56.9(+14)
VC-STaR(OUrs)
75.70557)
56.3()3.)
69.7(.1.))
25.3
62.4c10.)
59.3
58.102.6)
4.2
MAIN RESULTS
Comparison with the base model. To evaluate the effectiveness of our approach, we employ Qwen2.5VL-7B as the base model and adopt the "think step by step" prompt to enable chain-of- thought reasoning. We compare our method against this baseline, with results summarized in Ta- [ VC-STaR demonstrates consistent performance gains across diverse challenging benchmarks, achieving an average improvement 2.4%. Notably, it yields substantial improvements of 5.7% and 3.2% on MMVP and the Hallusion Benchmark, respectively, validating its efficacy in mitigating hallucinations within the reasoning process. Our approach also shows its enhanced reasoning capa- bilities mathematical benchmark, i.e., Math Vista and MathVision. Furthermore, the improvement on the MMStar and AME-Reall World underscore the generalizability of the VC-STaR under varied challenging peurroe-purpose scenarios.
For qualitative validation, F visual comparisons that offer deeper insights. The visu- alizations reveal that our model excels grounding its textual rationales in the corresponding visual evidence. This capability remains robust even when confronted with visually complex patterns, thereby effectively mitigating hallucinations.
Comparison with self-improving baselines. We reproduce three self-improving baselines and compare VC-STaR against them. Each baseline is applied to the Owen2.5VV-7B base model and generates rationales on VisCoR-555 -55 for finetuning, differing in their core improvement mechanism: (1) STaR (Zelikman et al. 2022): Leverages ground-truth answers to regenerate rationales for in- correct predictions. (2) Verifier (I.u et al. 2024a): Filters out visually hallucinated rationales via a self-verification step (Zhang et al. (025a) ensure visual grounding. (3) Feedback (Qu et al.) 2024): Refines rationales based
-eferaeed feedback in recursive manner. Table [ reveals critical trade-off: existing self-improving methods boost performance on hallucination benchmarks the expense of math and general capabilities. Our approach mitigates this pattern and achieves robust, consistent performance gains.
Comparison with off-the-sheil visual reasoning datasets. We also evaluate VC-STaR against base model finetuned on four off-the-shelf visual reasoning datasets. These datasets represent diverse strategies for rationale generation. For instance, Virgo at. 2025) makes the VLM think slowly
7

---

Published as a conference paper at ICLR 2026
Question
meaning white sign?
visb ee the image?
The in picture red with through 2. symtol indicating panting sign typicilly The duration parking
Owar2
archibtidd bo the lisalt
period
mage, paste
spccited add tonilllnal indicates parking allowed.
the
To desermiine how many lorks are visble the image. let's follow these sisps: Identily the objecte: Look droelly wi the image and kdwniily all the objests that could be torks. 2. Count the torks: There one lork placed the napkin nn next to the slice of cake on the left plate. There is another fork pariially visible on the ligtt side of the image, rosiing on the plate with the chooseate coiis. Verty count Double-check lonis hiddion cartally ocscured.
The white sign in the mmge hase a red cirde with tie through rt which is unhvorsal symbol coling "no" or "prohtotod"" Spociiialll, this sion is "No Fariina sign, moons that parking is not allowed the area where sign possed. This type of sign typicely used to present velicles from blocking bulfic causing congestion owtaain aress.
Owan2 SVL-78 thank step by step"
ensure notitional kellwingg these stops, we can conduue that are two vobble the image.
symbel pointing resogrized proh against right that location The atsence any corflising symools aeditiona hot corfirr this intepreiaton. Furthemmoe, Bhe consirieccy talfic signage conventions reinorces the condisson that the dignt purpose is prch bi right turrs.c/hiik whhiioo-aaaneeeeee The while sigh means "No Right Turn It protibes vehicles from making a right sum at that partoular locaion chereoowr
age image
sign bocauso
symbol painting unversslly reoograzod prohibilion
right, This
the
Owwr2.VVL.7
right that location The atsence comlisting symboli sccitione lest confime this intepretaaton. Furthermree the consinencc in taafie signage conventions rentorces the conduuson that the eights purpose is to prohili right tumn, <fhink-smmweee--T white sign means "No Rightt Turis" protiotss vehicles from makiing a. right sum at that partioular loowion-chereeeee
VO-STaR
Figure 5: Qualitative Comparison with base model. The second row shows the directly response from the base model, the third row shows the response when the base model is prompted to "think stey by step", the last row shows the model improved with our VC-STaR. We highlight the key visual evidences with red boxes for clarity of visualization. More results are in Sec. A.4
MMVP
Hallusion
62.4
Performance
76
Performance
50
55.9
56.3
Performance
62
61.2
74
72
69.7
60
59.8
70
59
HQ-Edit
DOCCI MathVista
VisCoR
HQ-Edit
DOCCI
VisCoR
w HQ-Edit
w DOCCI
VisCoR
Performance
70
69.7
Penformance
26
MathVision 25.1
25.3
Performance
MME-RealWorld
59.3
69
25
24.3
68
24
67
66.5
23
49.6
50.4
W HQ-Edit
DOCCI
VisCoR
HQ-Edit
DOCCI
VisCoR
HQ-Edit
w DOCCI
VisCoR
Figure 6: Performance comparison with other contrastive VQA pair construction strategies. Rationales in all settings are generated from the proposed VC-STaR, The red dashed line represents the base model (Qwen2.5VL-77) p performance.
with purely textual rationales. In contrast, LLaVA-COT (uu et al. 2025) leverages hand-crafted templates filled by the powerful GPT-4o (OpenAl 2024a). Other approaches first convert visual information into text; R1-Onevision Yang et al. 2025b) generates rationales from image captions using the DeepSeek-R1 model (Guo al. 2025), while Long Perceptual Thought (LPT) (Liao jet al., 2025) extends this by using dense captions (Onoe et al. 2024) and keywords like "wait" to elicit more detailed outputs from similar LLM. In our experiments, we directly finetune the base model on each of these datasets. Based on the results shown in Table [ we can draw the following conclusion: (a) Enhancing visual reasoning with purely textual rationales from Virgo is ineffective. This strongly indicates that visual modality matters. (b) The model trained on LLaVA- CoT suffers limited improvement, which demonstrates that the hand-crafted template struggle to generalize across diverse VQA tasks. (c) The models trained on datasets generated by DeepSeek-RR based on captions achieves notable improvements. However, the performance gap between them and ours highlights the clear advantage of our visually-native approach over relying on textual captions.
4.3 ANALYSIS
Can contrastive VQA pairs constructed in other ways? To answer this, we explore alternative strategies for curating contrastive VQA pairs. The first strategy is editing-based, utilizing the HQ-
8

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation of the effect of VC-STaR on other base models. Blue numbers in parentheses represent performance gains.
Table 3: Effect of the easy samples adding to VisCoR-55K, Red numbers in paren- theses represent performance drops.
Model
VC-STaR Hallusion Math Vision MMStar
Neasy Hallusion Math Vision MMStar
Qwen2.5VL
46,9
18.4
55.0
Ok
56.3
25.3
62.4
3B
V
53.2.+63)
21.9(43.5)
55.7(+0.)
+20k
52.2(4.1)
23.3
61.3c-.1)
VL.25
48.2
21.3
61.1
8B
55.44<7.2))
23.4(+2.1)
(2.5(+1.)
+40k 55.7c-0.)
21.9(34)
59.5
Table 4: Analysis about the effect of positive and negative contrastive VQA counterparts on GQA benchmark. We adopt the @wen2.5VVI-77 our base model, and report its reasoning performance as a baseline. QR: query for relationships; QA: query for attributes; QG: query for global informa- tion; QC: query for category; CA: comparing of attribute; CC: choosing the object of one certain category; CAt: choosing the object of one certain attribute. Blue (red) numbers in parentheses rep- resent performance gains (drops) relative to the baseline.
Setting
Pos. Neg.
QR
QA
QG
QC
CA
CC
CAt
Total
Base Model
43.8
51.4
31.5
44.4
30.2
60.6
44.4
45.4
VC-STaR
(44.5(+)))) 52,8(+1.4 46.8(+15.3) (7.1 42.9(+12.7) 60.1(-0.5
44.4(+0.0) 50.6(+3.2)
VC-STaR
x
(51.6(+7.8) 56.8(4 33.9(+2.4) (9.2() 46.0(+15.8) 70.8(+102) 66.7 53.7(+8.3)
VC-STaR
V (53.5(497)) 57.3(+5.9) 46.3(118)) 55.6(-112) 36.5(-6.3) 72.70122 55.66-12) 54.7(+9.3)
Edit dataset (Hui al. 2025). By prompting an LLM to create questions from editing instructions, we generate pairs where an original and an edited image yield different answers. The second strat- egy is caption-based, leveraging a dense caption dataset, DOCCI (Onoe al., 2024). For this, we instruct an LLM to parse dense captions of visually similar images and generate a question that hinges on their subtle differences. For both strategies, we generate rationales for these newly cre- ated contrastive pairs using our proposed VC-STaR and finetune the Owen2.5VL-7B. The results, presented in Fig. 6 lead several observations: (a) VC-STaR is broadly effective, but performance is data-dependent. This is attributable to the biased data distribution of HQ-Edit and DOCCI, high- lighting a key limitation of their curation scope. (b) VisCoR-550 includes contrastive pairs from a broader range of reasoning tasks, resulting in a more balanced performance.
Does VC-STaR generalize to other base models? We conduct experiments on Qwen2.5VL-3B and InternVL2.5--83 (Chen et al. 2025). Following the same self-improving procedure, we use VC-STaR to generate visual reasoning datasets from our VisCoR contrastive pairs, specifically for the two base models. We then finetune the Owen2.5VL-33 and Intern VL2.5-8B via the LLaMA- factory and SWIFT (Zhao et al., 2025), respectively. The results, presented in Table 2 demonstrate the model -agno-tg effectiveness of our approach. These consistent and significant gains confirm that VC-STaR is versatile and broadly applicable strategy for enhancing the visual reasoning ability.
What is the effect of easy samples on visual reasoning? Starting with our VisCoR-55K datasets, we incrementally add easy samples of two batches with 20K each. As illustrated in Table 3 we observe that the inclusion of easy samples is harmful. Specifically, when the number of easy samples increases, performance decreases. Therefore, we do not use the easy samples to avoid the potential "overthink" for straightforward problems.
How the contrastive VQA pairs of different types contribute? A contrastive VQA pair can be cat- egorized as "positive" if both samples yield the same answer, and "negative" if their answers differ. To investigate the respective contributions of these two types of counterparts to our method's 's per- formance, we conducted a controlled experiment on the GQA dataset (Hudson & Manning 2019). The structured nature of GQA allows for the reliable curation of both positive and negative pairs via simple text matching. We applied VC-STaR to three distinct training sets: one generated from only positive contrastive pairs, one from only negative pairs, and combined set including both. The results, detailed in Table g reveal a clear and significant trend. While both types of pairs are benefi- cial, negative counterparts are substantially more effective than positive ones, and their combination
9

---

Published as a conference paper at ICLR 2026
yields the optimal total gain, highlighting their complementary roles. We attribute the superior effi- cacy negative counterparts to their ability to induce stronger semantic contrast. Accordingly, our approach incorporates both positive and negative pairs without restriction to achieve optimal gain.
5 CONCLUSION
We demonstrate that visual hallucination can be effectively mitigated through the lens of contrast, thereby enhancing visual reasoning. Based the insight that VLMS better by contrast, propose the VC-STaR. The VC-STaR refines hallucinatory reasoning paths through analysis over curated contrastive VQA pairs, which yields our high-quality VisCoR-55K. Finetuning on VisCoR- 55K delivers a consistent performance gain across six benchmarks, significantly surpassing other self-improving baselines and models trained on state-of offe-of-thee-ar visual reasoning datasets. Looking forward, we hope our work will offer new perspective visual reasoning and inspire the explo- ration of novel contrast-driven training and inference paradigms.
REFERENCES
Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. AAAI pp. 8076-8084, 8084, 2019.
Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sehastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. NeurIPS, pp. 23716-23736, 2022.
Xiang An, Jiankang Deng, Kaicheng Yang, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Unicom: Universal and compact representation learning for image retrieval. In ICLR, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL httph::t///// //apxii//axxvvv.. o abs/ 13923
Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In COLING, pp. 1511-1520, 2022.
Chameleon. Chameleon: Mixed-modal early-fusion foundation models, 2025. URL https://
arxiv. abs 24005 09818
Hongruixuan Chen, Jian Song, Chengxi Han, Junshi Xia, and Naoto Yokoya. Changemamba: Re- mote sensing change detection with spatiotemporal state space model. TGRS, pp. 1-20, 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In ECCV, pp. 370- 387, 2024b.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are the right way for evaluating large vision- language models? In NeurlPS, pp. 27056-27087 2024c.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance bound- aries of open-source multimodal models with model, data, and test-time scaling, 2025, URL ittps://aate http:///arxvv.ooo....oo//// wwtp://ggg/....../ 05271
10

---

Published as a conference paper at ICLR 2026
Lei Ding, Jing Zhang, Haitao Guo, Kai Zhang, Bing Liu, and Lorenzo Bruzzone. Joint spatio- temporal modeling for semantic change detection in remote sensing images. TGRS, 62:1-14, 2024.
Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring cong-chain visual reasoning with multimodal large language models. In CVPR, pp. 9062-9072, 2025.
Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Virgo: A preliminary exploration on reproducing ol- like mllm, 2025. httptpss:///aar...rrrraaas//////
Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E Gonzalez, and Serena Yeung-Levy Describing differences image sets with natural language. In CVPR, pp. 24199-24208, 2024.
Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. In CVPR, pp. 14303-14312, 2024.
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. G-llava: Solving geometric problem with multi-modal large language model. ICLR, 2025.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In ICML, pp. 10764-10799 2023.
Dedre Gentner. Structure-mapping: A theoretical framework for analogy. Cognitive science, 7(2):
155-170, 1983.
Yean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In NeurIPS, pp. 21271-21284, 2020.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models. In CVPR, pp. 11375-14335,, 2024.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas Reinforced self- ooff-training (rest) for language modeling, 2023. URL hottb:ts:://// http://axvvvv.o www.:/rgg/aaaa..a 208.. 08998
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Peiyi Wang. Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma. Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhi- hong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li. Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang. Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue
11

---

Published as a conference paper at ICLR 2026
Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunx- ian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang. Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Zizheng Pan, Zhen Huang. Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(808 ():388)))333333 2025.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong. Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In EMNLP, pp.
2023.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729-9738, 2020.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real- world visual reasoning and compositional question answering. In CVPR, pp. 6700-6709, 2019.
Mude Hui. Siwei Yang. Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang. Cihang Xie, and Yuyin Zhou. HQ-edit: A high-quality dataset for instruction-based image editing. In ICLR, 2025.
ICDAR. Overview icdar 2019 robust reading challenge scanned receipts ocr and information extraction, 2019. URL https: Frc. cvc. uab. e
Qirui Jiao, Daoyuan Chen, Yilun Huang. Bolin Ding, Yaliang Li, and Ying Shen. Img-diff: Con- trastive data synthesis for multimodal large language models. In CVPR, pp. 9296-9307, 2025.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, pp. 2901-2910, 2017.
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. diagram is worth a dozen images. In ECCV, pp. 235-251, 2016.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. ICLR, 2023.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ring- shia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. NeurlPS, pp. 2611222122000 1-2224,, 2020.
Hoeseong Kim, Jongseok Kim, Hyungseok Lee, Hyun a Park, and Gunhee Kim. Viewpoint-agnostic change captioning with cycle consistency. 2021 ieee. In ICCV, pp. 2075-2084, 2021.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. NeurlPS, pp. 22199-229222 222133, 2022.
Baiqi Li. Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang. Zixian Ma. Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. Naturalbench: Evaluating vision- language models on natural adversarial samples. NeurlPS, pp. 17044-17068, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023a.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In ACL., pp. 5315-5333 2023b.
Zehan Li. Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023c. URL https: : //arxiv. org/ 2308 03281
12

---

Published as a conference paper at ICLR 2026
Zhuoqun Li, Haiyang Yu, Xuanang Chen, Hongyu Lin, Yaojie Lu, Fei Huang, Xianpei Han, Yongbin Li, and Le Sun. DeepSolution: Boosting complex engineering solution design via tree-based exploration and bi-point thinking. ACL, pp. 4380-4396, 2025.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS, pp. 17612-17625, 2022.
Yuan-Hong Liao, Sven Elflein, Liu He, Laura Leal-Taixé, Yejin Choi, Sanja Fidler, and David Acuna. Longereeptulttooughs: Distilling system-2 reasoning for system-1 perception. In COLM, 2025.
Wei Lin, Muhammad Jehanzeb Mirza, Sivan Doveh, Rogerio Feris, Raja Giryes, Sepp Hochreiter, and Leonid Karlinsky. Comparison visual instruction tuning. In CVPR, pp. 2973-2983 2025.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. ICLR, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, pp. 34892-34916, 2023.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? ECCV, pp. 216-233, 2024b.
Yuliang Liu, Junjie Lu, Zhaoling Chen, Chaofeng Qu, Jason Klein Liu, Chonghan Liu, Zefan Cai, Yunhui Xia, Li Zhao, Jiang Bian, Chuheng Zhang, Wei Shen, and Zhouhan Lin. Adaptivestep: Automatically dividing reasoning step through model confidence. ICML, 2025.
Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yunlong Feng, and Zhijiang Guo. Autopsv: Automated suppess-upee verifier. NeurIPS, pp. 79935-79962, 2024a.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song- Cang-Chunn Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. ACL, 2021a.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chur Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. NeurlPS DB, 2021b.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. ICLR, 2023.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024b.
Ji Ma, Wei Suo, Peng Wang, and Yanning Zhang. C3 I: content correlated vision- languaee instruc- tion tuning data generation via contrastive learning. In IJCAI, 2024.
Ruotian Ma, Peisong Wang, Cheng Liu, Xingyan Liu, Jiaqi Chen, Bang Zhang, Xin Zhou, Nan Du, and Jia Li. S²R: Teaching LLMs to self-verify and self-correct via reinforcement learning. In ACL, pp. 22632-22654, 2025.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder. Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. NeurIPS, pp. 46534-46594, 2023.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. ACL findings. pp- 2022.
13

---

Published as a conference paper at ICLR 2026
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographievqa. In WACV, pp. 1697-1706, 2022.
Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of-
thought prompting for large multimodal models. CVPR, pp. 1440111431, 2024.
Yasumasa Onoe, Sumayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-TTuset, Garrett Tanzer, Su Wang, and Jason Baldridge. DOCCI: Descriptions of Connected and Contrasting Images. In ECCV, 2024.
OpenAl. Gpt-4o system card, 2024a. URL httpss//axxxv..orr/////////1 200.02//2222..720000 16720
OpenAl. Openai ol system card, 2024b. URL
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve. Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without super- vision. TMLR, 2024. ISSN 2835-8856.
Zhiyu Pan, Yinpeng Chen, Jiale Zhang, Hao Lu, Zhiguo Cao, and Weicai Zhong. Find beauty in the rare: Contrastive composition feature clustering for nontrivial cropping box regression. AAAI, pp. 201220112000 1-2019, 2023.
Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. ICCV, pp. 4624-4633, 2019.
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how self-improve. NeurlPS, pp. 55249-555555 55225,, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In JCML, pp- 8748-8762. 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. NeurIPS, 2023.
pp.
Eleanor Rosch. Cognitive representations of semantic categories. Journal of experimental psychol- ogy: General, 194(3):192, 1975.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In ECCV, pp- 146-162, 2022.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hong- sheng Li. Visual cot: Advancing multi-modal language models with comprehensive dataset and benchmark for chain-of-thought reasoning. NeurIPS, pp. 8612-8642. 2024.
Benny Tang, Angie Boggust, and Arvind Satyanarayan. Vistext: A benchmark for semantically rich chart captioning. In ACL, pp. 7268-7298, 2023.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, pp- 776-794, 2020.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, pp. 9568-9578, 2024.
Luong Trung, Xinbo Zhang, Zhanming Jie. Peng Sun, Xiaoran Jin, and Hang Li. ReFT: Reasoning with reinforced fine-tuning. In ACI., pp. 7601-7614, 2024.
14

---

Published as a conference paper at ICLR 2026
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision- vision-language encoders with improved semantic understanding. localization, and dense features, 2025. URL.
htt.p:://arxiis org. abs 2502 14786
Junke Wang. Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning, 2023a. URL https://arxivv
arg//bs/2311..00 075744
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with MATH-vision dataset. In NeurlPS DB, 2024.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. ICML, pp. 9929-9939, 2020.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023b.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. TMLR, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le. and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurlPS, pp.
2022b.
Shengqiong Wu, Hao Fei, Liangming Pan, William Yang Wang, Shuicheng Yan, and Tat-Seng Chua. Combating multimodal Ilm hallucination via bottom-up holistic reasoning. AAAI, pp. 8460 8468, 2025.
Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. In ICCV, 2025.
Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E Gonzalez, Bin Cui, and Shuicheng Yan. Supercorrect: Advancing small I1m reasoning with thought template distillation and self- correction. ICLR, 2025a.
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, Bo Zhang, and Wei Chen. RI RI-onevision: Advancing general- ized multimodal reasoning through cross-modas formalization. In ICCV, pp. 2376 2385, 2025b.
Linli Yao, Weiying Wang, and Qin Jin, Image difference captioning with pre-training and contrastive learning. In AAAI, pp. 3108-3116, 2022.
Nikolaos-Attonios Ypsilantis. Kaifeng Chen, André Araujo, and Ondřej Chum. Udon: Universal dynamic online distillation for generic image representations. NeurIPS, pp. 86836-86859, 2024.
Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai. Syntax- aware network for handwritten mathematical expression recognition. In CVPR, pp. 4553-4562, 2022.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin. Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. CVPR, 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. In NeurlPS, pp. 15476-15488, 2022.
15

---

Published as a conference paper at ICLR 2026
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song- Jong-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. CVPR, pp. 5317-5327, 2019.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts": Llm self-training via process reward guided tree search. In NeurIPS. pp. 647722,,,, 2024a.
Fuxiang Zhang. Jiacheng Xu, Chaojie Wang, Ce Cui, Yang Liu, and Bo An. Incentivizing Ilms to self-verify their answers, 2025a. URLhttps::
wwwxiv.vvv/v.////aa//e..//aaaava http://wwwwvv.....aaa. wwt. 2506 01369
Hao Zhang. Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. ICLR, 2023.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding, 2024b. URL
https://arxa
17107
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, Liang Wang. Rong Jin, and Tieniu Tan. Mme- realworld: Could your multimodal IIm challenge high-resolution real-world scenarios that difficult for humans? In ICLR, 2025b.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. TMLR, 2024c.
Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Hong Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swift: scalable lightweight infrastructure for fine-tuning, In AAAI, pp. 29733-29735, 2025.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In ACL, pp. 400-410, 2024.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answer- ing in images. In CVPR, pp. 4995-5004, 2016.
A APPENDIX
A. RETHINKING VC-STAR FROM A COGNITIVE PERSPECTIVE
Learning and reasoning are inherently comparative and contrastive processes. Humans rarely learn concepts in isolation. Instead, humans refine our understanding by comparing examples, identifying distinguishing features, and reasoning through analogies and differences. The prototype theory concludes this cognitive behavior as that our human beings identify new identities by comparing them with the prototype concept (Rosch) 1975). Besides, the structure- structure-mapping theory says that analogical reasoning can recognize the relationships shared by two domains (Gentner) (983). This mapping can be treated as a fine-grained contrasting process. In our work, the contrasting process provides an opportunity to learn visual concept from the prototype, and our rethinking strategy reinforces the structure-mapping by generating new reasoning paths via contrasting. We hope to highlight the potential of porting such human-like cognitive behaviors to the domain of reasoning.
A.2 DETAILS ABOUT VISCOR-55K
The construction of our VisCoR-55K dataset is a multi-stage process involving efficient pair cu- ration, difficulty-based filtering, and quality-controlled rationale generation. The entire pipeline is designed to produce a high-quality, challenging visual reasoning dataset. Our curation process for contrastive VQA pairs begins with dytaset-by-dataset, divide-and-conquer strategy. To maintain computational tractability and avoid a costly O(n?) search complexity across the entire data pool, implement greedy, fist--tath-exxi search algorithm: for each sample within a given source dataset, the search for contrastive VQA counterpart terminates as soon as the first valid match is identified. This efficient approach allows to scale the curation process effectively. Following this
16

---

Published as a conference paper at ICLR 2026
Distribution
Easy
Reasening
Reasening
120%
Modian Sample Ratio (%)
Samples
OCR
General
Mats
OCR
GraphyChart
GrahhCChart
Graphchaan
20k
80%
15k
60%
10k
0%
wris
Tallya
Visue7WW
IconOA(Parti)
CLEVR
RAVEN
IconoA(Parta)
TabMwp
HME100K
SROIE
GeoQA4
SPVQA
Geol Geo170KqA
Vietest
Geomatry 3k
LLaVAR
HotefulMemas
AOKVOA
LRV
UReador
InfoGraphievaa
A220
Figure 7: Statistics of the contrastive VQA pair curation. The bar chart (left y-axis) displays the total number of contrastive VQA pairs in each dataset, with colors indicating the data category. The line graph (right y-axis) plots the ratio median samples identified within those pairs for each dataset. In the upper right, the pie charts provide categorical breakdown of the sample.
procedure, we initially curated large pool 240k contrastive VQA pairs. The distribution is visualized by the bar chart in Fig. 7 with the left y-axis indicating the number of samples.
This initial pool of 240k pairs then undergoes rigorous filtering and refinement pipeline. First, we apply the difficulty- faaseuulty-baee sampling strategy (as detailed in Sec. 3.I) to select only the median samples, which are most effective for enhancing the model's reasoning capabilities. The proportion of median samples varies significantly across datasets, and is illustrated by the line graph in Fig. 7 (plotted against the right y-axis). This critical filtering step narrows our collection down to 86k challenging contrastive pairs. Subsequently, we leverage the contrasting and rethinking pipeline to generate a high-quality rationale for each of these 86k samples. a final quality control measure, we employ a text-matching-oae containing unexpected or erroneous reasoning patterns. This process culminates in our final VisCoR- 55K dataset, collection of high-fidelity visual reasoning samples ready for finetuning. The pie charts in Fig. provide a categorical overview of the data composition throughout this pipeline.
post-processing step to automatically filter out any rationales
A.3 PROMPTS FOR THINKING, CONTRASTING, AND RETHINKING
As introduced in Sec. 3.2 steps lead to the final rationales. We design prompts for the thinking, contrasting, and rethinking steps. The thinking prompt is:
You are helpful assistant to answer the question by thinking step by step. ### INPUT ####
Image: The image that serves as the basis for answering the question.
Question: The question pertains the content of the image.
Answer: The correct answer for the question about the image.
### INSTRUCTION ###
You should analyze the question and decide to focus on which visual content.
You should parse the details of visual content based on the question.
You should conclude the visual evidence to answer the question.
### OUTPUT ###
The returned content MUST be in the natural flow.
The contrasting prompt is:
17

---

Published as a conference paper at ICLR 2026
You are a helpful assistant to think step by step for discriminating between two images to
answer two synonymous questions.
### INPUT ###
First Image: One image that serves as the basis for answering the question.
Second Image: The other image that serves as the basis for answering the question.
First Question: The question pertains to the content the First Image.
-Second Question: The question pertains the content of the Second Image.
Answwr: The correct answer for the question about the images.
### INSTRUCTION ###
-When the correct answers for the two images are the same, you should summarize the com-
mon patterns in the visual content of the two images.
When the correct answers for the two images are different, you should identify the differ- ences in visual content between two images.
Conclude the visual evidence to answer the questions respectively.
### OUTPUT ####
Return in the natural flow.
firt Imsge>><Seecodddaaaeaeeee
F TLL.ooeees.aoxxeeeeeeeeeeeeeaeae
The "Answer" here is the concatenation of both samples. The rethinking prompt is:
You are helpful assistant rewrite the coarse rationale into a more correct and more
logical one based on a contrastive analysis. ### INPUT ####
Question: The question to be answered based one given target image.
Answer: The correct answer to answer the question.
Coarse Rationale: The naive reasoning process answering the question.
Contrastive Analysis: The reasoning process when comparing the first image with the second image for synonymous questions.
### INSTRUCTION ###
The contrastive analysis is more reliable than the coarse rationale.
If the answers in the contrastive analysis are the same for the two images, the model should formulate summary reasoning schema. This schema must summarize the key visual features and confirm that the provided visual evidence aligns with this schema to derive the conclusion.
If the answers in the contrastive analysis are different for the two images, you can employ backward chaining hypothesizing the visual cues that would present if the alternative answer were correct, and then highlighting the critical distinctions between this hypothetical scenario and the actual visual evidence. ### OUTPUT ####
The output MUST be in the format of <thhink>the thinking con- answering content</answeer
The content of thinking content MUST be between the special token of '<think>>> and </think>'
<
The content of answering content MUST be between the special token of 'aanswe>'' and </answer>'.
<Questi <enestiioon<nnswr>>>>>>> oewp://wwaaeeeeoooooeeeeeoooooea
A.4 ADDITIONAL QUALITATIVE RESULTS
Examples of rationales generated by VC-STaR in VisCoR-55K are illustrated in Fig. X After ob- taining the model finetuned with VisCoR-55K on Qwen2.5VL-7B, we test it on some customized visual question answering cases and observe the interesting results shown in Fig. 9
18

---

Published as a conference paper at ICLR 2026
Question
Rationates
It ceternane the ligue that inginall companees the sequenos, let's coretully anatyee the target image: 1. "Shopp Analysis": shapes in sequenie are perrazone they remain consiston
acreso all positions "Blze Progression": Otsening the ligures, there a dear trand inreasing sae as the scquence progresees left to anghtand top
subusquert shape gass large
deservaioon
spparent charge in coor or shading pattems intuending the plapement primary gradual expension, the nest figure
spures, The consemency color suggests that the
than the preceding pallir must continue frend reles adley incremental growth sizee whoout intiodusing adiitonal complextios such as color changes now shapes. Thorafore, the figure that aligni porfectly with this prinbble option aiseaaaneeeeeessseecc...
sequerte
same
detats the contem that the
Choose fiuue that would
lagically complese the sequaiiio..
<hinko-Thh image shows red London United buses parked side by side, with their front ends visible. Therefore, based on this observation:
There
are
""wo" buses shown
in the
thene?
<thinks-The image an animal posttioned head partially obscured by archway. This indicates while there may another horn present, It not visible from this angle or due to the obstruction. The vieual evidence supports the condusiion that only one hom identiiable in the given tew.cithincc-anewee elanswerr
way only one clearly visible,
How many hores are visible??
< detorning the measuro of angle wo start by analyonng the glrnn information and loveraging goomettic procertes. Thierge gla is insolaed didcl D as the macpoiet of side BC Bioe D mispoiet perpencioula BC, and OD extended intersctss the dicck point E, Connecting EC. we are piven that angle CEC From the proparty iscacales orde, trangle OEG isoseeles, thus anglne perpendicutar Angll ICC) that Gereequantly the owniol angle BDC, which aubtends am BC, twios angle FOB: (( (Tedlange RGO) Vimes 50%aro subterding
mangies, DE
equal Theretore
givn clegramm Mangle insoriedd
ECB mippoint
bC
BE,
imsles
BC. intrrce cirde poin and EC is then
being incerieed { ('aollAnnglee A) 199) lames 100More Sprcirc.)) () This ressoring signa with the specile detall provided in the problem and confiin the measure of angle Authiink canswers 50 S0rolnssers
100"ats. Finally
the centrol BOC:
connected, what the measure angle CEC is 8517
A
Cholces: A: 50 B: 55" C: 60 65
Figure 8: Examples of rationales in VisCoR-55K.
Duestion
Unvend DVL-77
Owen2 DVL-7B "thikk step by step"
Qwerr BWL. VVV--TT
shocked,
denenane wtir makeo pooii roop
comtan deait
<heniio-hh image depicts soene at what apcears
The image shows group ol paoole what
The imape appean takan at bastnal gane
basobal gane, with spectstrrn showing rppression suppiss bey
agpears sports staiumm reacting
brokets tauitall
suppiss person could
baseball
"he could be play thet vauuedd conteverna or
in mid-ar, which suggeers an unexpected has someth cur the game causing the audience's reactoon
body tanguage of the spectation indicate
has just takion placo. Without additional
comteot it's not possible to determine esact cause whatever happened was surpestng and possibly dangerous.
foul ball hilking someone
audden and intense reaction to an event that
suggest samwhiig
extracndinry play. The sogressione and
occured
shack, is clesr that
making
with
wen ssce way that to sos spentioon, prisaae
wor flcar inpuii the wa Conaarid od Clusc The
ex 3. Wei duau This peopll anisn
posere inaily disrppton. diven thes cisamnation
white othere are lockng mupresiora abock uniogpacid This
monerny
While other sconarios might Involve difiorent
here points dreectl to the but as the source swaas-Th people are shoolee bece uon hat has been thron into the clowd, likey during a surprise
cassea for shock ool---n suach ass soodents or
sudden and
unrelated incidents- the visual evidence
This
aworishmmentt wot.emettttttt------oo
the
makes people shockedd?
conas, colevvnn slos desi sas of
the spealic draun starov the garw, bul the genea
gamn oal
<thinkoThe The suestion osto about the color of the in parking spot 33, The ana yois shows thet there is no car occupy parking spot 53 the image, appeors empry. Since thre no veriide presect
parked parking spot
The car the parking spot labeled
specite 1pet, bs impossible to dete any color associated with
labeled "33" is blue.
"33" is blue.
obsenvttio that absence of car directly with
What color is the oar on the perking spot
at Remeebe wowe Borgit
cetermine what single breid will willin on the board,
the girl
me rac toerdyyth with Sanga Ce
The girl with single braid is working on the problem "11 2", The answer this problem 9.
need salve the matiematical expressioon
the working on The expression on the
the boad
Adlan with chill diracl belon
Boliing pie
Lat's breas it cown stop by stop: 1, idonoty the openition: the operation here is
howed for the gir with a sinlaa breid is 7x2.
Vesil beteroe oe The up waer (iree 34 woush The drt
reuu lece in lime debumai an, ciee thr ged
muttpacation. 2. Miltply the numbers: 7
What will the boar??
multpleed by its 18. the gin with single
bosd weter the nanber
will who the number
the board
torn fus du bookaar clenueeso
Figure 9: Additional qualitative comparison.
19


Reviews:
[{"review_id": "SKafm22Cl2", "reviewer": "Reviewer_EV9r", "summary": "This paper addresses the challenge of improving visual reasoning in vision-language models, where conventional self-improving methods fail to correct visual hallucinations in reasoning paths. To overcome this limitation, the authors propose Visual Contrastive Self-Taught Reasoner (VC-STaR), a novel framework that leverages contrastive visual question-answering pairs to refine model-generated rationales. VC-STaR constructs a new dataset, VisCoR-55K, by curating contrastive VQA pairs across multiple domains and generating faithful visual rationales through a “think–contrast–rethink” pipeline. Fine-tuning VLMs on VisCoR-55K significantly enhances their reasoning performance across five benchmarks, surpassing existing self-improving baselines and models trained on state-of-the-art reasoning datasets. These results demonstrate that contrastive learning enables VLMs to self-correct visual hallucinations and advance visual reasoning.", "questions": "Please refer to the weak points.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)\tThis paper proposes a task-agnostic, three-stage pipeline to construct contrastive pairs, with principled text/vision similarity thresholds to ensure both semantic anchoring and visual proximity.\n2)\tThe paper takes an insightful contrastive perspective, showing that learning from visual contrasts effectively reduces hallucinations and strengthens reasoning accuracy in VLMs.\n3)\tAnalyses isolate the impact of curation strategies, sample difficulty, and pair types, producing actionable design guidance.", "weaknesses": "1)\tIn Table 1, the method fails to achieve top results on MMStar. While it improves over its backbone, it still lags behind several baselines, and the paper does not explain why its gains do not generalize to this benchmark.\n2)\tIn line 258, the authors state that only median-difficulty contrastive VQA pairs are retained for rationale generation. However, the paper does not discuss why hard samples are excluded or how they might be addressed. Exploring strategies for handling these challenging cases could further enhance the model’s robustness and generalization.\n3)\tIn Figure 1b, the meaning of Setting C is not clearly explained. Although readers can infer its general idea, the paper should explicitly define it to avoid ambiguity and ensure the experimental setup is fully transparent.\n4)\tThe paper’s main contribution lies in constructing the VisCoR-55K dataset, yet neither the dataset nor the accompanying code is released. Without open access, it is difficult to verify the results or reproduce the data-curation process, which weakens the paper’s practical impact and credibility.\n5)\tThe additional data generation and contrastive fine-tuning steps likely increase training cost substantially, but the paper does not report resource usage or efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0u667FVGKp", "reviewer": "Reviewer_RMKH", "summary": "Authors propose Visual Contrastive Self-Taught Reasoner (VC-STaR), a novel self-improving framework that leverages visual contrast to mitigate hallucinations in model-generated rationales.", "questions": "see the weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The ideas presented are interesting, and the proposed approach has potential to spark further research.\n\n2. The article is well-structured and easy to read, providing a smooth reading experience.", "weaknesses": "1. Unclear Motivation for Using Input Comparison to Solve the Hallucination Problem: While the paper proposes using input comparison to mitigate hallucination issues, the motivation or theoretical basis for this approach is not clearly explained. Could the authors elaborate on why this method is effective from a cognitive perspective? For instance, are there relevant studies from cognitive models or psychological theories that can support the effectiveness of this approach?\n\n2. The Problem Being Addressed is Not Clearly Defined: I understand that the goal is to use the reasoning dataset generated after rethinking to train the model. However, my question is: compared to previous reasoning datasets, what exactly is the advantage of your approach? Are there hallucinations in the reasoning data from existing datasets, or is it because you are using a contrastive dataset that allows you to fine-tune the data in a new way to improve the model’s generalization? If it’s the former, are you addressing the problem that existing reasoning datasets are inadequate? If it’s the latter, you need to explain this in more detail and position it as a new contribution.\n\n3. The Inference Method is Not Clearly Defined: How is the trained model used after training? Does it follow the same inference paradigm as before, or does it adopt the contrastive reasoning method shown in Figure 1b? If it’s the latter (option b), does the model need to search for relevant question-answer pairs during inference? Additionally, since you modify the user’s query during rethinking, how does the model generate an appropriate response from the modified query? If it’s not option b, then your inference method hasn’t changed and you are simply training a better model with higher-quality data, correct?\n\n4. Lack of Experimental Details and Limited Experiments: The experimental section lacks sufficient details, and the number of experiments is limited. The specific VLMs used are not mentioned, and the experiments could benefit from being conducted on a wider range of benchmark datasets. Providing more details on experimental settings and parameters would enhance the credibility and reproducibility of the results.\n\n5. Variability in Benchmark Experiment Results: In some benchmarks, the results show significant improvements, while in others, the improvements are small. Has there been any analysis of the reasons behind these discrepancies? It would be helpful if the authors could provide an explanation or discussion regarding the variation in results across different benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GBMzl1MptG", "reviewer": "Reviewer_EeLq", "summary": "Motivated by an observation that VLM performs better when presented with a contrastive pair of VQA, the authors propose a “Visual Contrastive Self-Taught Reasoner” to leverage visual contrasts to mitigate hallucinations in the model generated rationales. The authors construct contrastive pairs in different datasets for selected samples which do have a similar enough counterpart within each dataset, respectively. The authors further divide the samples into easy, medium and hard and only keep the medium and hard samples. These samples from different datasets together form a new dataset and the authors leverage this dataset for supervised fine-tuning of VLM. Experimental results show that fine-tuning models in this dataset could improve the performance in hallucination, math and general benchmarks.", "questions": "Is it important whether the second contrasted image’s question is correctly answered? How is the correctness of that image? Does the order of the image pairs matter? How many images are ideal for this contrast (e.g., how about contrasting 3 or more images)?\n\nAny statistics on how many images are selected as useful in each of the 21 datasets?\n\nAfter fine-tuning, will the tuned model exhibit a “contrastive rethinking pattern” even on simple tasks? Will the performance degrade on these simple tasks? Will the model hallucinate a “none-existing contrasted image” during the inference?\n\nDuring inference, does it require a contrast or not? If yes, what to do when no such pair can be found?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed framework works in different VQA tasks, including math reasoning, general tasks, and hallucinations.\n- The SFT dataset construction process is intuitive.\n- A multi-step prompting strategy for contrasting and rethinking is proposed to properly leverage the “reference information” offered by the contrasted image pairs.", "weaknesses": "-\tThe definition of “self-improving” is not clear to me. The collected public datasets for selecting samples to construct the proposed dataset is quite large (21 datasets). Does evaluating a model fine-tuned in 21 datasets in 5 downstream benchmark datasets really count as self-improving? It’s more a strategy to carefully select external knowledge instead of “self-improving.”\n\n-\tIf I understand correctly, after the model is tuned on the curated dataset consisting of contrastive pairs, the inference process of the tuned model does not include any contrasting any more. Will the model still perform better after being presented with a pair of images?\n\n-\t[line 279-line280, line 910-912] how does the model know which one is correct? Why is it expected to pick a correct one to make the framework work?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "abWqdcEg6u", "reviewer": "Reviewer_mDXQ", "summary": "This paper introduces VC-STaR, a contrastive self-improvement framework that reduces visual hallucinations in vision–language models by pairing similar images with synonymous questions and applying a three-stage process (coarse reasoning, contrastive analysis, rationale rewriting). Using this pipeline, the authors build VisCoR-55K from 21 VQA datasets and fine-tune models, yielding consistent gains on hallucination, math, and general benchmarks over STaR-like baselines and text-only rationale training.", "questions": "1. Visual Similarity Computation\nPlease provide details on how visual similarity is computed, including the architecture, training data, and training objectives of the “general visual embedding model.” Indicate whether the model reuses or fine-tunes existing models (e.g., CLIP, DINOv2, UNICOM, UDON), and include specific implementation details.\n\n2. Similarity Threshold Selection\nPlease explain how the text and visual similarity thresholds (φq and φv) are determined and provide corresponding sensitivity analyses.\n\n3. LLM (ψ) in the “Rethinking” Stage\nPlease clarify the relationship between ψ and the vision-language model θ: is ψ identical to θ, or is it a stronger external LLM? If the latter, please describe its role and contribution to performance improvements.\n\n4. Additional Benchmark Evaluations\nPlease include comprehensive evaluation results on more benchmarks (e.g., POPE (hallucination), MMBench, MME-RealWorld, TallyQA, LLaVA-Eval) to provide a more complete view of the model’s performance.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel and Elegant Methodology: The core insight—that visual contrast mitigates hallucinations—is intuitive yet effective. The proposed three-step VC-STaR framework is a simple, \"visually-native\" self-improvement pipeline that avoids complex external components like reward models.\n\n2. Systematic Data Contribution: The paper introduces VisCoR-55K, a high-quality visual reasoning dataset created through a scalable and principled curation pipeline. It also provides valuable design insights, such as the superior efficacy of \"negative\" (different-answer) contrastive pairs.\n\n3. Broad and Robust Empirical Validation: The approach achieves consistent and significant performance gains across a diverse set of challenging benchmarks (hallucination, math, general reasoning). Its effectiveness is demonstrated across multiple base models, showing strong generalization and outperforming key self-improvement baselines.", "weaknesses": "1. Some key details could be clarified. For example, how the similarity thresholds (φv/φq) were chosen and their sensitivity, as well as the configuration/source of the LLM (ψ) used in the rethinking stage. Adding these would help with reproducibility and understanding the method’s scope.\n\n2. The evaluation coverage feels a bit limited. Including results on benchmarks such as POPE, MMBench, MME-RealWorld, TallyQA, and LLaVA-Eval would provide a more complete picture of performance across settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rzZGj9pVzO", "reviewer": "Reviewer_4FzS", "summary": "This paper introduces Visual Contrastive Self-Taught Reasoner (VC-STaR), a self-improving visual reasoning framework. The key insight is that VLMs can identify more relevant visual cues when comparing two visually similar images paired with synonymous questions. Building on this observation, the authors collect visually similar image–question pairs and generate rationales for self-improvement training. Experiments across multiple benchmarks demonstrate that the contrastive capability of VLMs can effectively bootstrap their own visual reasoning ability.", "questions": "1. Can the authors provide a quantitative or qualitative comparison of VC-STaR’s computational cost relative to STaR?\n2. How sensitive is VC-STaR to noisy or low-quality contrastive samples?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is straightforward. By leveraging the intrinsic contrastive ability of VLMs, VC-STaR enables the model to generate improved rationales through self-prompting, making it readily adaptable to new architectures and settings.\n\n2. VC-STaR achieves substantial performance gains, surpassing prior self-improvement methods and even models fine-tuned on carefully curated datasets. Moreover, the method generalizes well across different backbone models, further validating its effectiveness and robustness.", "weaknesses": "1. Construct data in a contrastive manner is not a new idea, there are several previous works already explored training VLMs with collected contrastive samples, such as [1] and [2]. I suggest authors to include a discussion with these works for a better understanding with training VLMs using contrastive data samples.\n    \n    [1] Jiao et al. Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models\n    \n    [2] Ma et al. C3L: Content Correlated Vision-Language Instruction Tuning Data Generation\n    via Contrastive Learning\n    \n2. Despite the effectiveness, the proposed VC-STaR framework appears to be significantly more computationally expensive than existing methods such as STaR. Pairing visually similar samples from a large dataset may hinder the method’s scalability and efficiency when adapting to new domains. A discussion or analysis of VC-STaR’s efficiency relative to previous methods would strengthen the paper.\n\n3. Table 1 does not include a baseline that trains models solely with instruction–response pairs. Including such a baseline would help readers assess the specific contribution of rationales and better understand the overall performance gain.\n\n4. The paper does not analyze how the quality of the contrastively selected samples affects performance. Since these samples are central to the proposed pipeline, a discussion or empirical analysis of their quality’s influence on model outcomes would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)