Sherlock: Self-Correcting Reasoning in Vision-Language Models
Authors: Yi Ding
Ruqi Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Vision-Language Models
Self-Correction
Reasoning
Chain of Thought
Primary area: deep_learning

Abstract: Reasoning Vision-Language Models (VLMs) have shown promising performance on complex multimodal tasks. However, they still face significant challenges: they are highly sensitive to reasoning errors, require large volumes of annotated data or accurate verifiers, and struggle to generalize beyond specific domains. To address these limitations, we explore self-correction as a strategy to enhance reasoning VLMs. We first conduct an in-depth analysis of reasoning VLMs’ self-correction abilities and identify key gaps. Based on our findings, we introduce \emph{Sherlock}, a self-correction and self-improvement training framework. \emph{Sherlock} introduces a trajectory-level self-correction objective, a preference data construction method based on visual perturbation, and a dynamic $\beta$ for preference tuning. Once the model acquires self-correction capabilities using only 20k randomly sampled annotated data, it continues to self-improve without external supervision. Built on the Llama3.2-Vision-11B model, \emph{Sherlock} achieves remarkable results across eight benchmarks, reaching an average accuracy of 64.1 with direct generation and 65.4 after self-correction. It outperforms LLaVA-CoT (63.2), Mulberry (63.9), and LlamaV-o1 (63.4) while using less than 20\% of the annotated data.

O.O
Sherlock: Self-Correcting Reasoning in Vision-Language Models
Yi Ding, Ruqi Zhang
Department of Computer Science, Purdue University, USA [ding432, rugiz)Opurdue.edd
Project Page: https: http://drippnooi hwtppoongggghuu.....p.pp.p ho/Sherlock/
Abstract
Reasoning Vision-Language Models (VLMs) have shown promising performance complex multimodal tasks. However, they still face significant challenges: they are highly sensitive to reasoning errors, require large volumes of annotated data or accurate verifiers, and struggle generalize beyond specific domains. To address these limitations, we explore self-correction as a strategy to enhance reasoning VLMs. We first conduct in-depth analysis of reasoning VLMs' self-correction abilities and identify key gaps. Based on our findings, we introduce Sherlock, a self-correction and celf-improvement training framework. Sherlock introduces a trajectory-level self-correction objective, a preference data construction method based on visual perturbation, and dynamic for preference tuning. Once the model acquires self-correction capabilities using only 20k randomly sampled annotated data, it continues self-improve without external supervision. Built on the Llama3.22 2--....--iiiooo-. 1 model, Sherlock achieves remarkable results across eight benchmarks, reaching an average accuracy of 64.1 with direct generation and 65.4 after self-correction. It outperforms LLaVA-CoT (63.2), Mulberry (63.9), and L-o (63.4) while using less than 20% of the annotated data.
Introduction
Teaching Large Language Models (LLMs) to reason through long, step-by-step thinking processes during post-training has been shown to significantly improve their performance complex tasks, such as mathematical and code benchmarks [13, 33, 11, 34]. Similarly, as Lision-Language Models (VLMs) demonstrate strong capabilities [18, 2, 591, recent studies have incorporated reasoning into VLMs through Supervised Fine-Tuning (SFT) [44, 35, 7, 48, 12, 31] or Reinforcement Learning (RL) [37, 38, 47, 54, 27, 26), boosting performance in solving challenging multimodal tasks.
Although reasoning VLMs such as [24, 31, 3, 32, 19] have demonstrated impressive performance on multimodal mathematical reasoning tasks, such MathVista [21] and Math Verse [55]. they still face significant challenges. First, they are highly sensitive to reasoning steps-once an error occurs in multi-step reasoning process, it often propagates through subsequent steps, leading incorrect final answers. Second, they are extremely deta-demanding [12, 44], requiring large-scale, high-quality annotated data easily verifiable answers to achieve consistent improvements. Third, they struggle to generalize to broader domains where such precise supervision is scarce, limiting their applicability beyond mathematical or VQA problem settings.
In this paper, we study self-correction as promising strategy address these challenges in reasoning VLMs. While self-correction has been explored in LLMs [14, 16), it remains underutilized in reasoning models or VLMs, despite being particularly well-suited. Self-correction allows the model to revise and improve its own prior outputs. This is particularly useful for challenging samples where the model may produce partially correct reasoning. such cases, correcting specific mistakes through
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Case of Self- CeffCCoreectio
Problem: The image shows the Chubb illusion. the center area on the smaller
than the center area on the right?
Sherlock Direct Generation Response. <SUNMARY> I will analyze the image to compare the size of the ..... </ </SUNNARY>
<CAPTION>> The image divided into two halves: the left side is solid gny background with small circular pattern in the center, while the right side features black and white noise pattern with central circle that identical size the one the left. </CAPTIOK>
<BEASONIKG>> central circle both sides
image identical size. However, right side's central circle is embedded size. ... Despite this visual effect, the physical dimensions of the central cirele do not change, and it remains smaller than the entire right half the image. </REASDNING> <CONCLUSION> Yos </CONCLUSION>
in busy black and white pattern, which can create an illusion of
Sherlock Self-Correction Response. <SUMNARY> <CAPTIOR> The image is divided into two halves. The left side shows solid gray background with small, circular, speekled pattem in the center. The right side features a dense, random black and white pattern covering the entire area, with central circular region that less donse. </CAPTION>
analyze image determine the relative sizes of the </SUNMARY>
<REASORIKG> To assess the size of the central areas, I will compare the two: the circular pattern on the left side is isolated and clearly defined against density within sea of busy black and white patterns, suggesting a larger visual impact. However, the physical dimensions of the two areas are the same. </REASONING>
gray backgruund suggesting a smaller presence. 'The area the right delined by lesser
<COKCLUSION> No / </CONCLUSIOR>
Figure 1: Example of Sherlock's self-correction ability. The direct generation contains errors that
cause the trajectory to deviate from the correct path and result in an incorrect answer. Sherlock successfully corrects the previous response and obtains the correct answer.
self-correction is simpler than generating fully accurate answer from scratch [46]. Moreover, the responses before and after self-correction naturally form preference pair [5, 52], where the inherent ordering acts as implicit supervision. By treating self-corrected outputs as preferred alternatives, the model not only reduces its dependency on extensive annotated data but also enhances its ability to generalize to broader domains.
To enable self-correction in reasoning VLMs, we introduce Sherlock: self-correction and self- improvement training framework enhance reasoning. Unlike existing self-correction approaches. Sherlock introduces reasoning trajectory- self-correction objective that focuses on correcting only the erroneous steps rather than the entire answer. Additionally, it leverages visual noise perturbations to construct preference datasets with controllable quality gaps. To further stabilize preference training, we introduce dynamic |3 that adapts to the quality gap between each sample pair. Once the model learns self-correction, it is able to self-improve without external supervision. Our main contributions are summarized as follows:
We conduct in-depth analysis of the self-correction capabilities of reasoning VLMs trained with either SFT or RL. We find that these models generally camoot self-correct. Self-correction within reasoning trajectory was observed in fewer than 10% of samples, with only half leading to correct final answer. When prompted with external critiques or self-correction prompts, the models fail to improve and may even see accuracy drop.
To teach reasoning VLMMs self-correct, we propose trajectory-level self-correction objective, a preference data construction pipeline, and a dynamic / for preference tuning. This approach enables fin-grained correction at the reasoning trajectory level rather than overhauling the entire answer, providing a clearer learning signal.
We present Sherlock, a training framework designed to enhance self-correction and reasoning in VLMs. It includes three stages: SFT, offline, and online preference learning. Through the entire training, Sherlock requires only 20k randomly sampled annotated data. In the online stage, the model continues improve both reasoning and self-correction abilities without any external supervision, leveraging olff-constructed preference datasets.
Across eight benchmarks, the Sherlock model achieves the best reasoning performance while using the least amount of annotated data (20k), reaching an average accuracy of 64.1 (direct reasoning) and 65.4 (after selffcorrection). (260k), and LlomaV-ol (175k) achieve 63.2, 63.9, and 63.4, respectively. To efficiently scale inference-time compute, we combine Sherlock with a verifier as stopping criterion, reducing GPU usage by 40% while achieving even higher accuracy (54.0 - 55.9).
comparison, LLaVA-CoT (100k), Mulberry

---

2
Related Works
VLM Reasoning. LLMs with reasoning abilities (i.e., thinking before answering) have been shown to significantly improve performance across diverse tasks [40, 13, 11]. Building on this insight, researchers are now equipping VLMs with deeper reasoning capabilities enhance their performance on visual tasks. Reasoning abilities are primarily enabled by supervised fine-tuning (SFT) or reinforcement learning (RL) methods. SFT uses external supervision signals combined with (Co'T) [40] to teach models think step-by-step [56, 44, 12, 7, 6, 48, 35j. However, generating multimodal CoT annotations often involves multi-turn prompting or Monte Carlo Tree Search (MCTS) on powerful non-reasoning VLMs, resulting in substantial computational costs. RL methods elicit reasoning behaviors through reward maximization. Inspired by DeepSeek-RI [11], recent studies [54, 3, 24, 27, 26, 31, 38, 37, 31, 19, 58] use carefully designed, rule-based rewards to induce multimodal "aha moments". Despite notable improvements, particularly in mathematical reasoning, these approaches remain highly dependent on carefully curated multimodal datasets and accurate verifiers. Training samples are often filtered based on complexity and reasoning difficulty, making these methods resource-intensive and challenging generalize to diverse domains [24, 38], In contrast, our method improves VLM reasoning by requiring only 20k randomly sampled CoT-labeled data to start, and then self- celf-improves without relying ground truth labels.
Self-correction. Self-correction refers the model's ability to revise its previous responses to generate higher-quality answers. It has emerged as an intuitive and promising approach for enhancing model capabilities through inference-time scaling [22, 23, 57]. Studies [36, 28, 52] show that directly prompting LLMs to perform self-correction using generic instructions often leads to degraded performance. To address this, SFT and RL approaches [39, 49, 5, 16, 52] are developed to equip models with critique and self-correction capabilities. Step-wise correction methods [48, 37] use SFT to teach models to reflect and revise within a single reasoning process, but often fail to effectively trigger intrinsic correction behavior. The most relevant to our work is [43], which integrates response- wise correction into LLMs for mathematical reasoning tasks using correction prompts and trains with DPO and PPO. However, this approach is limited to domains with verifiabe answers and depends on such rules to construct self-correction datasets. In multimodal settings, some approaches train an additional critic model to generate sample-specific critiques [42, 53, However, deploying these extra models and collecting large-scale critique data are resource- -houureeintensivee and their effectiveness in judging long CoT responses remains uncertain. Unlike prior work, Sherlock is specifically designed for multimodal reasoning models. It introduces trajectory-level self-correction objective and a corresponding data construction pipeline that guides the model to revise only the erroneous suffix of the reasoning trajectory, removing the need for verifiers or large-scale critique data.
3 Can Reasoning VLMs Self-Correct?
Reasoning VLMs explicitly generate step-by-step thoughts along with the final answer during infer- ence [44, 35, 48, 12, 38, 47, 54, 37]. This process can be denoted as (.1.... Yn a) ~ x x-1(2182)) where y represents the i-th reasoning step, is the final answer, is the reasoning VLM, and T&&T denotes the input image and text. Given language models' ruto-regressive decoding mechanism, an error in any reasoning step yi may propagate and result in incorrect reasoning in the subsequent steps. Therefore, equipping reasoning models with self-correction capabilities holds great potential for improving overall reasoning quality. For reasoning models, self-correction behavior can be implemented in two ways: (i) Step-wise correction: The model reflects on its previous incorrect i-th step within a single thinking process and revises it to arrive the final answer:
(r, (r,yt+1,".. a) I (-12113r (irraer;;1.... V1. v. E ("wait", "however", "check",
(1)
Here, y" is the erroneous reasoning step, and is a reflection token indicating the model's intention to correct its previous reasoning. (ii) Response-wise correction: The model is prompted to revise and improve its previously generated response:
(v² ... ".....,,::: a²)) ~ T-H
at;t),
(2)
where (vL. .a) denotes the j-th attempt and is an additional instruction guiding the model to perform correction. In this section, we analyze both types of self-correction behaviors of reasoning VLMs, including the SFT-based LLaVA-CoTT [44] and the RL-based Ret-inkhn [37], on two widely-used benchmarks, MMStar [4] and Math [21].
3

---

Original Reasaning
VL-otlaber aie
Slep
Stop
Choice:
57.6
3744
573
575
$3.4
cabieed acalsterd dis
As Find RS
cORS
30
equilsternl
lerght shilet GE de. 95 be and be
friangls.
Intoral
Reesenig VIM Villeteeeer
Modiied Oue Sky
Qweet 5-Tl-mmmee
(4)
Your
Meditled One Step Trajectery
535
540
550
15.8
75.2
External Ontipp
anternal Corraction
30.7
CHIl VIM
Readdring Waiginn
15.
Figure 2: Left: Overview of experimental settings for self-correction analysis. Blue block illustrates the Modified One Step process using Bren2.5-7B-Instruue [45], while Green block represents two correction strategies applied to direct generations: external critique-based correction and self- correction prompt. Right: Reasoning performance of LLaVA-CoT [44] and VL-Rethinker [37] under different settings, evaluated on MMStar [4] and MathViste [21].
3.1 Step-wise Seff-correction
Setup. To test whether the model can perform step-wise self-correction, we conduct a controlled experiment using Qwen2.5-7B --Inttrcett [45]. In this experiment, we prompt the Qwen model autonomously select one influential step yt in the first half of the original reasoning trajectory and apply Modify One Step, result- ing in an erroneous step denoted as VI. Then, VLMs generate the subse- quent trajectory (W/+11 y a²)
Table I: Step-wise self-correction results of reasoning VLMs under the Modify One Step setting. The numbers following the benchmark names (1500 and 1000) indicate the total number of samples in each benchmark.
MMStar (1500)
Math Vista (1000)
w/ Aha Moment
w/ Aha Moment
Model
Num
Acc
Num
Acc
LLaVA-CoT [44]
121 (8. 1%)
47.1
104 (10.4%)
27.9
VL-Rethinker-77B [37] 136 (9.1%)
52.2
94 (9.4%)
63.8
T (-1æ114; "1.-." , 9:-1, vf conditioned on the previous reasoning steps with incorrect y This process illustrated in the blue block of Fig. 2. To examine whether models exhibit step-wise self-correction behavior, which is often accompanied by the emergence of an "aha moment", and whether such behavior contributes to successful correction and the correct final answer, we analyze whether the suffix (V'+1, .Wn)) contains indicative expressions such as "wait", "however". or "check", which we denote as w/ Aha Moment. We report both the overall accuracy under the Modify One Step setting and the accuracy within the subset of responses that exhibit an Aha Moment.
Findings. We report the performance of reasoning VLMS under Modify One Step in the right part of Fig. 2. By comparing direct generation with the Modify One Step setting, we observe that an in single step often leads to an incorrect final answer, causing the overall accuracy to drop to the level of a random guess (~25%) Additionally, Table shows the number of responses exhibiting "aha moments" (i.e., signs of self-reflection) after an error step. The results indicate that even the RL-based model VL-Rethinker exhibits "aha moments" in fewer than 10% of cases. Moreover, the presence of self-reflective behavior does not necessarily lead to correct final answer. More details about the settings are provided in Appendix C.3.
Takeaway 1: Step-wise Seff-Correction
Reasoning VLMs struggle with step-wise self-correction. Even when reflection signals are present, they often fail to correct their reasoning to reach the correct answer.
3.2 Response-wise Self-correction
Setup. To analyze the models' response-wise self-correction behavior, we evaluate two strategies: (i) applying an self-correction prompt to guide the model in refining its responses; (ii) leveraging external critiques generated by critic models, such as Critie-V [53] and Qwen2.5- -1--------- [2]. to prompt VLMs to improve their responses based critique feedback. The evaluation procedure

---

consists of: (1) generating the initial Direct Generation response; and (2) applying two correction strategies obtain revised responses, reported as Internal Correction and External Critique, with the latter involving two different critique models, as illustrated in the green block of Fig. 2. Complete prompts and more details are shown in Appendix C.3.
Findings. The results in the right part of Fig. 2 show that neither the correction prompt nor the external critiques effectively improve the reasoning trajectory. In some cases, model accuracy even declines after correction. Moreover, we observe that, regardless of which critic model is used to provide feedback, the accuracy of the corrected responses tends to converge to level similar to direct generation, showing little variation based on the quality of the critique.
Takeaway 2: Response-wise Self-Correction
Reasoning VLMS struggle with esponse-wise self-correction, regardless of whether using self-correction prompts external critiques.
Step-wise vs. Response-wise Self-Correction Step-wise self-correction depends on the internal reflection during intermediate reasoning, which hard to control reliably trigger. In contrast, response-wise self-correction is externally guided (e.g., by self- cofrectortt prompts), making it more controllable and learnable. Hence, this work focuses teaching VLMs response-wiis wise self ceff-correction
Teaching VLMs to Self-Improve Their Self-Correction Abilities
Takeaway I and 2 reveal a critical gap in current reasoning VLMs: models trained with either SFT or RL lack the ability to perform effective step-wise and response-wise self-correction. Once an error occurs, the models struggle to revise their reasoning, often failing to recover from mistakes. To address this limitation, we introduce Sherlock teach the model self-correction, thereby enhancing its reasoning capabilities. The framework consists of three stages, which will detail below.
4.1 Stage I: SFT Cold-start for Reasoning and Self-correction
Recent studies [44, 48, 12, 35] have shown that supervised fine-tuning VLMs on long CoT annotated data can significantly enhance their general reasoning capabilities. To teach VLMs to self-correct, we draw inspiration from [52, 48] and aim design an objective that simultaneously improves reasoning and self correction. To achieve this, we introduce pairwise training objective:
(FT(T = Exxierr """ DDDDttrncc log x. (Y (xT1eT) log x
Y y',t)) t)]-
(3)
Direct Generation
Self Self-Corection
We begin by randomly sampling 10k examples from the LLaVA-CoT dataset [44], denoted as DA. We then train a base VLM on DA using vanilla SFT, resulting in RO VLM, which is capable of generating CoT template responses. Next, we randomly sample an additional 10k examples, denoted as DB- Since DB is annotated with high-quality reasoning trajectories, we assume its responses are of higher quality than those sampled from the model fine-tuned on limited data, i.e., Y TRo VLM (- We then construct Sherlock-SFT dataset Dsbetookk (#f Y,": Y2)) )=)=, where Y is from Dg and y' is generated by TRO VLM- Finally, we apply the loss defined in Eq. 3 to cold-start the base VLM, jointly training the Sherlock SFT model to acquire both reasoning and self-correction capabilities.
4.2 Stage II: Offline Preference Training with Trajectory-Ievel Self-correction
An incorrect reasoning process may contain partially correct steps, but the loss in Eq. enforces correction over the entire response, potentially introducing noise by forcing the model to revise already correct steps. To avoid such undesirable behavior and inspired by Takeaway 1, which shows that early mistakes can derail the entire reasoning trajectory, we adopt a fine-grained trajectory-level offline training strategy. Instead of rewriting the full response, the model is guided to revise only the erroneous suffix, allowing to make targeted adjustments while preserving correct reasoning steps. Our goal is enable reasoning VLMs to self-correct faulty steps Y3, (w) ... whi a²) in the initial y (wi.... ... y at) and produce a higher-quality trajectory Y2 (V?... ... 22: a²), conditioned on input I&TT, correction instruction t, and the original response y Since the prefix
5

---

Ressoning Dataset LLaVA-CoT
D
Random sampling 10k 10k Dn
Unlobeled
Duta
1-U(L2,..
ST
910
Sherlock
Re-Inilidiee
Sherlock SFT
Sherlock SFT
Inference
Y
Coro
If of
(-M
Inrennnce
DIO
BAO
Select
Preference Datoset
Sa Foin Ka)
Dr
10k Sharlock SFT Dale
Sherlock Offiine
Dataset Dulling append ,Y,"")
Sherlock teer
Dicer A .appppdd (x. 1"
SFT Cold-Start
Offine Preference Training (Reuse 10K DA GT)
Iterative Online Preference Training
D2 D2 w/ GT)
(0 GT)
Figure 3: Training pipeline of Sherlock, including: (Left) SFT cold-start stage, (Middle) offline preference training, and (Right) online iterative self-improvement. In the SFT and offline stages, randomly sample 10k D and 10k DB with ground truth from the 100k LLaVA-CoT [44] dataset as supervision. During the online stage, each iteration samples only 5k unlabeled inputs, from which self-constructed and self-labeled dataset is built using the selection rule illustrated the (Right) part.
Yd is assumed be correct, we do enforce revisions on these steps. The objective instead focuses on generating higher-quality suffix Y2,
max Bygga - (MIr. Y (.)2))
[e(2 X I (T [ Y 2..)
+ Byg ( (" [XY2 I +2) (F Y t; D.] (4)
Here, i is randomly sampled from model to prefer the higher-quality suffix Y2, over Y. given the prefix Y2, while remaining close to the reference policy Tee-- The second expectation does the same but conditioned on Yt. This objective has two key benefits: it leverages both prefixes Y2, and Y which contain no explicit preference signals, and it explicitly models the preference between Y and Y promoting correct self-correction while discouraging incorrect updates. Following prior work [25, 5, 1, 52], we denote p(Y22' > 1 y as the human preference probability over suffix reasoning trajectories given the instruction and prefix, expressed as:
for each example. The first expectation encourages the
p I Y2,) | (1 prefers Y Y given æι& and 1. (1,2). (5)
Building upon the formulation in [5, 52], we extend it to the trajectory-level and solve the objective in Eq. 4, resulting in the following self-correction loss:
To (Y [x, Y Y2.))
To (Y Y t
u(2, Y t;
ma) = 9 log Y1 <in
B Bog
Teat Y +YE
Teet
x Y
Fngnurage positive self correction
Disenurage negative self correction
To (Y Y t; Y
To (Y [xx Y t; Y2.)
u(x,Y), tt; YY Y2 Y2, = =lg
Blog
Tret (Y3 [x, YI, xY
Neat (Y 1 +22))
Encourage positive self correction
Discourage negative correction
Lsc(TO) Trat) = E yt) v(ankr Y L; yl < Yu m6) u(Iiter Y ,4 Y Y me)
+ +B ((,,"", [1 o(FI&T, "oiusr,Y""Kr <0 y mo) + u(aner Y T", Y Y m0)
(6)
The detailed derivation and explanation provided in Appendix A. To further enhance the model's direct reasoning ability using the preference dataset, we additionally incorporate the DPO loss [29], which is jointly optimized with the self-correction loss:
LShertock (mT; meer) = [sc Teet) + aLLPo (TT;; Wrer)
Data Construction. We construct negative responses with controlled quality gaps by randomly truncating reasoning steps and adding visual noise. From the 10k annotated examples in DA.

---

sample a truncation point U[1, .N) and generate a suffix Y' by applying visual perturbation e and decoding from Y2.. This produces low-quality reasoning trajectories for preference training. The model is then trained on this dataset using Eq. 7 to obtain the Sherlock Offline model. The complete data construction process described in Appendix B.2.
Dynamic 3 Stabilize Preference Training. Prior work shows that the choice of 3 strongly affects preference optimization [5, 52, 41]. In framework, we build controllable preference dataset by randomly selecting a truncation step i and applying visual perturbation enabling a sample-specific dynamic adaptively scale learning signals:
1
( n, c) 4(0.5 + (1)
(8)
Larger values are assigned samples with earlier truncation stronger perturbation, where quality gaps are wider, yielding more conservative updates. Smaller gaps result in smaller 8, promoting stronger learning from subtle preferences. Further details on 3 design are provided in Appendix B.2.
Takeaway 3: Key of Sherlock Objective
The key insight of Sherlock is to fully leverage the constructed trajectory-level preference data and explicitly encourage models to revise only the incorrect suffix reasoning trajectory.
4.3 Stage III: Iterative Online Preference Training with Self-generated Data
The Sherlock model has acquired both self-correction and reasoning capabilities using only a small amount of data in the SFT and offline stages. Inspired by recent LLMs self- iefrovmmroeeeentt works [52, 43, 5]. we observe that the original and corrected responses naturally form preference pairs. Motivated by this, we develop a self-improvement framework to further enhance the model's capabilities.
The online iterative training differs from the offine stage (Sec. 4.2) only in the absence of ground-truth responses y". As previously discussed, reasoning quality improves through sequential self-correction. In each iteration, randomly sample 5k unlabeled questions. For each initial generation Y three rounds of self-correction produce Y y and Y We then apply self-consistency filtering: the final answers are semantically identical (a² a a²4), y is selected as the preferred response. To further reduce noise, the rejected response is derived from Y by keeping the prefix Y Y2, and generating a perturbed y via visual noise injection, following the offline stage. We perform two iterations of self- celf-improvement training using Eq. 7, resulting in the Sherlock Iterl and Iter2 models.
Takeaway 4: Self- Improvement Training Enabled by Self-Correction
When the model can self-correct, naturally generates a pair responses- the original and the corrected versions with clear quality preference, enabling iterative self-improvement.
Experiments
5.1 Setup
Training Details. Following prior SFT-based methods [44, 48, 35], Sherlock builds on Llama3.2V V- 11B- Instruct [9], focusing on integrating self-correction and reasoning capabilities. Additional training details, including data construction and hyperparameters, are provided in Appendix B.
Baselines. We evaluate Sherlock from two aspects: reasoning and ciff-correctinn. For reasoning, compare against LLaVA-CoT [44], Mulberry [48], and LlamaV-ol [35], all built on Llama3.2- Vision-1 I--Instruct [9]. For self-correction, we include inference-time scaling baselines such as Majority Vote and critique-base methods using LLaVA-Critic [42], Critic-V V [53], and Qwen2.5-VL. 7B-Instruct [2]. Details of these baselines are provided in Appendix C.2.
Evaluation Details. We evaluate on eight challenging multimodal benchmarks, including VQA (MMBench- VI.1 [20], MMVet [50], MME [17], MMStar [4)), math and science (Math Vista [21], A12D [15], MMMU [51 D), and hallucination (HallusionBench [10]). Sherlock is tested under two settings: direct generation and after three rounds of self-correction. All evaluations are conducted
using the VLMEval [8] for consistency, Additional details provided in Appendix C. and C.4.
7

---

Table 2: Performance comparison across 8 benchmarks. #Daata GT indicates the number of ground- truth annotated samples used during training. MMB refers to AMBench-V1.II Hallus denotes HallusionBench, and Math corresponds to Math Bold and underline indicate the best and second-best results, respectively. The red number represent the accuracy change after self-correction.
#Data w/ GT
Models
MMB MMVet Hallus MMMU MMStar A12D MathV MME Avg.
Llama3.2V-111 1V- [9]
65.8
57.6
42.7
47.8
53.0
88.2
49.7
1822
58.7
Reasoning Models
LLaVA-CoT [44]
75.0
61.7
47.7
49.1
57.6
82.9
53.7
2177
63.2
-Seff-Coreeee Correction
100k
74.4
62.3
46.4
49.2
57.8
82.9
53.0
2183
63.0
Mulberry [48]
75.2
58.3
47.8
46.7
57.8
86.2
61.9
2170
63.9
260k
Celf-Correction
59.0
46.6
46.9
57.4
86.3
62.3
2177
63.8
LlamaV-ol [35]
75.6
61.9
45.6
52.3
56.5
86.4
53.3
2125
63.4
Self- Leff-Coreection
175k
18.4
50.9
39.4
43.9
47.1
76.9
44.0
1823
48.2 18.2
Ours Sherlock Models
Sherlock SFT
72.2
61.4
45.5
47.1
54.9
86.6
52.0
2170
62.2
10k
Self-Correction
73.8
62.8
47.5
46.2
55.9
87.9
52.2
2172
63.0
Sherlock Offline
73.2
61.4
48.1
47.6
57.5
88.4
52.2
2162
63.2
Self-Correction
10k
74.7
48.9
49.0
57.7
89.5
53.9
2171
64.4
Sherlock Iterl
74.9
62.3
49.7
48.2
57.0
88.9
52.2
2177
63.9
Self-Correction
0
76.6
62.7
50.6
49.2
58.8
90.0
54.4
2195
65.1 1.21
Sherlock Iter2
74.6
48.7
49.7
57.7
89.6
52.0
2197
64.1
Self--orreetion
77.2
62.6
51.2
50.1
59.0
90.6
54.0
2204
65.4
5.2 Main Results
Advancing Reasoning Ability with Minimal Annotation. The results in Table 2 show that Sher- lock Iter2, trained on only 20k annotated examples randomly sampled from LLaVA-CoT, achieves the best overall reasoning performance. It outperforms LLa VA-CT [44] (100k annotations), Mul- berry [48] (260k), and Llama V-ol [35] (170k) across 8 benchmarks. Notably, even before online self-improvement, Sherlock Offline performs comparably to LLaCoT,,,, highlighting that incorpo- rating self-correctee correction significantly enhances direct generation quality. During online training, both reasoning accuracy (63.2+63.95 1..111 and celf-correction gains (1.2f-1.311) steadily improve, demonstrating the effectiveness of self-genere geferaeee preference data and self-improvement training.
Existing Models Fail Self-correct Their Reasoning. Table shows the performance of rea- soning VLMs after three rounds of self-correction. Consistent with Takeaway 2, existing reasoning models fail to improve through internal prompt- based self-cooreetion and even show performance drops across eight benchmarks. Notably. Llama degrades sharply because its step-by-step reasoning generated via multi-turn prompting: adding a correction prompt after the final turn breaks its original training format, leading to poorer responses.
Sherlock Further Enhances Performance via Ceff-Correction. Table also presents the results of three-round ceff-correction on Sherlock. During SFT, we use only response-wise self-correction objective without explicitly revising incorrect trajectories (Y3,). After both offline and online training, the model shows clear gains over Sherlock SF7, validating the effectiveness of our trajectory-level objective. Table compares different inference-time scaling strategies. Except for Majority Vote, all methods rely external critics for response critiques. Unlike in Sec. 3.2, find critique- yields the
based correction highly dependent on critic quality: the strong Qwen2.5-Ve best results, while weaker critics cause degradation, likely due to difficulty processing long CoT outputs. This sensitivity highlights Sherlock's ' learned responsiveness to contextual feedback. Notably, self-correction outperform Majority Vote while requiring only half the inference time.
5.3 Ablation Studies on Training Objective
We ablate our Sherlock pipeline, including the combination DPO loss and self-correction loss in Eq. 7, the selection strategy for the prefix step i in Eq. 6, and the design of the dynamic in Eq. 8.

---

Table 3: Performance of inference-tinne titeeneee-tieeeeeee methods. LLaVA-Critie, Critic-V, V, and Qwen2.5-VL-7B pro- vide critiques for correction, while Majority Vote @8 selects an answer from sampled generations.
Methods
MMB
MMVet
Hallus
MMMU
MMStar
A12D
Math
MME
Avg.
Sherlock Iter
74.6
62.4
48.7
49.7
57.7
89.6
52.0
2197
64.1
-LLaVA-Critic [42]
75.5
58.9
45.9
47.0
58.7
89.1
52.6
2122
62.91.2)
Critic-V [53]
73.9
61.8
47.0
47.7
58.1
88.9
50.2
2192
63.2
Qwen2.5-VL -L- [2]
76.5
64.4
48.6
47.9
59.3
89.1
55,5
2189
64.9
Majority Vote @8
78.5
62.2
49.3
49.7
58.0
91.1
54.0
2195
65.1 L
Self-Correction
77.2
62.6
51.2
50.1
59.0
90.6
54.0
2204
65.4
Offline
Tterl
Tter2
66
66
66
65-
65
65
Accurncy
64
64
64
63.9
63.9
64
63.7
M42
67.1
63
62.6
616
62
62
62.0
61
61
60
Sherlack
LoRD w/o Ese w
Static
60
Sherlock
we Copo
wlo Esc
w/ jm
w/ Static
60
Sherlock
w/o Copo
w/o Esc
w/ jm)
w/ Stalic
Direct Generation
CorteCoottttnnn
Average accuracy across benchmarks for ablation settings. w/ i=1 indicates that the objective in Eq. performs self-correctin on the entire response instead of trajectory-
Figure 4:
Finding I: Self-correction and Reasoning are Not Orthogonal Capabilities. As shown Fig. 4, we compare the full Sherlock model with two ablated variants trained using only the DPO loss or only the self-correction loss. We observe that using only the DPO loss brings minimal improvement over Sherlock- SFT. In contrast, the model trained solely with the self-correctic objective continues to improve across iterations. Although its overall performance is slightly lower than the full Sherlock model, it achieves direct reasoning accuracy comparable to other baselines, despite receiving explicit supervision for direct generation. This finding aligns with prior work showing that critique- based supervision
benefit direct reasoning [39]. Interestingly, even the DPO-only model shows gains in self-correction over iterations, despite having access to self-correction signals only during the initial SFT stage. These results collectively suggest that self-correction and reasoning are not independent abilities, but mutually reinforcing: learning one promotes the other.
Finding 2: Trajectory-level Objective Yields Superior Self-correction in Reasoning VLMs. In Fig. 4, we compare different strategies for selecting step in Eq, 6. Setting i = reduces the loss to full-response correction, forcing revision the entire response. In contrast, Sherlock adopts trajectory-Ievel correction, updating only the low-quality suffix Y while leaving prefixes Y2, and Ya untouched, they contain no clear preference signals. The offline model remains stable since both preferred and rejected responses share identical prefixes, contributing no gradients. However, in online iterations, full-response correction degrades self-correction performance, as strong preference signals mainly lie in y and updating non-informative prefixes hinders learning.
Finding 3: Dynamic 3 Leads to Stable Training. We present results in Fig. 4 comparing whether the dynamic B is applied to Eq. 6. Experiments show that incorporating dynamic / consistently improves both direct generation and self-correction performance across iterations. These findings validate that our sppple-specific dynamic contributes to more stable training and stronger models.
5.4 Why Does Sherlock Need Less Annotated Data?
Sherlock requires only 20k randomly sampled annotated examples because it fully leverages each example through its training objectives. To highlight Sherlock's ' data efficiency, we compare it against two alternative training strategies under the same 20k data budget: (i) training with standard SFT loss as adopted LLaVA-CoT,, without any self-correction objective, and (ii) training only using Sherlock- SFT loss defined in Eq. 3, without applying the offline preference training objective.

---

Table 4: Performance comparison of different methods using the same 20k annotated data.
Methods
MMB
MMVet
Hallus
MMMU
MMStar
AT2D
Math
MME
Avg.
LLaVA-CoT 20k
72.0
58.9
47.9
47.9
55.3
88.3
48.7
2158
62.0
+ -Self-Correction
71.6
58.6
46.8
48.7
56.0
87.7
49.8
2109
61.8
Sherlock SFT 20k
72.1
61.2
47.6
45.6
57.0
88.0
52.5
2186
62.8
+ Celf-Correctton
74.1
62.2
47.8
46.0
58.1
89.0
53.2
2159
63.4%6t
Sherlock Offline
73.2
61.4
48.1
47.6
57.5
88.4
52.2
2162
63.2
Self-Correction
74.7
63.8
48,9
49.0
57.7
89.5
53.9
2171
64.4
Integrating Self-correction into SFT: A Free Lunch for Enhancing Reasoning. Under the same 20k data setting, the only difference between Sherlock SFT and LLaVA-CoT is the self-correction term in Eq. 3. As shown in Table 4, Sherlock achieves 0.8 higher average accuracy in direct reasoning, consistent with Finding (Sec. 5.3) that learning self-correction enhances reasoning ability. After applying self-correction, Sherlock further improves, widening the gap over LLaVA-CoT.
Trajectory-level Objective Maximizes the Use of Preference Data. Table 4 shows that Sherlock- Offline consistently outperforms Sherlock-SFT,, both terms of direct reasoning performance (63.2 & 62.8) and improvements (1.2f & 0.61) brought by self-correction. While the Sherlock-SF7 loss (Eq. 3) focuses solely on guiding the model to revise low-quality responses into high-quality ones, the Sherlock- Sherlock-Offttnee loss (Eq. 6) incorporates three additional objectives: (I) discouraging the model from generating the same low-quality response again, (2) preventing quality degradation when the initial response is already high-quality, and (3) encouraging the model to reproduce the same output when the initial reasoning is correct. These complementary signals help the model fully utilize the preference data to learn more fnn-grained and robust self-correction behavior.
5.5 Efficiently Scaling Up Sequential Self-Correction with Verifiers
Recalling the results in Sec. 5.2, where we val- idated that the Sherlock model achieves stable inference-time scaling through self-correction. In this section, inspired by recent work MM- Verify [31], we conduct deeper exploration aimed at simultaneously improving the effi- ciency and capability of self-correction-bet based inference-time scaling.
Table 5: Inference-time scaling results with MM- Verify on Vasta.
Methods
Acc
GPU Hours
Sherlock Iter2
52.0
+ Self-Correction w/o Verify
54.0
13.2
Parallel Vote w/ Verify
55.1
40.2
Celf-Correction w/ Verify
55.9
8.7
Unlike MM-Verffy, which generates N responses in parallel and selects the top i verified ones for majority voting, we adopt Sherlock's s sequential self-correction during inference. After each response, verifier checks correctness; if incorrect, MM- MM-Verify's critique guides the next round. Sampling stops once a correct response is verified. We compare this approach with parallel majority voting on Math Vista [21], reporting accuracy and inference time (A100 GPU hours) in Table 5. Results show that a strong verifier improves both accuracy and efficiency. Sherlock's s self-correction achieves higher accuracy with lower cost, demonstrating superior scalability for complex reasoning tasks.
Conclusion and Discussion
In this paper, we introduce Sherlock, the first framework to achieve intrinsic self-correction in reasoning VLMs, with significant improvements across diverse benchmarks, Our analysis reveals that existing reasoning VLMs, whether trained with SFT RL, struggle to self-correct. Sherlock addresses this gap with novel trajectory-level objective, enabling self-correction and self eef--immrovement using only 20k annotated examples. Our findings show that self-correction is not only feasible but also powerful strategy to improve reasoning and inference-time scaling in VLMs. This makes VLMs more suitable for complex tasks, especially those where generating a correct answer in a single pass is difficult. Looking ahead, Sherlock provides a promising approach for extending self-correction to other types of reasoning models. It can also be further enhanced by incorporating step-wise self-correction for more efficient self-correction.
10

---

Acknowledgements
We thank Xingchao Liu and the anonymous reviewers for their thoughtful comments on the manuscript. This research is supported in part by NSF ISS-2508145, Amazon Research Award, and OpenAl Researcher Access Program.
References
[1] M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[2] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge. S. Song, K. Dang, Wang, S. Wang, Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 2025.
[3] H. Chen, H. Tu, F. Wang, H. Liu, X. Tang, X. Du Y. Zhou, and C. Xie. Sft or rl? an early investigation into training r rlike reasoning large vision-language models, 2025.
[4] L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang. Y. Qiao, D. Lin, et al. Are we the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.200 20330, 2024.
[5] E. Choi. A. Ahmadian. M. Geist, O. Pietquin, and M. G. Azar. Self-improving robust preference optimization. arXiv preprint arXiv:2406. 01660, 2024.
[6] Y. Ding, L. Li, B. Cao, and J. Shao. Rethinking bottlenecks in safety fine-tuning of vision language models. arXiv preprint arXiv:2501. 18533, 2025.
[7] Y. Dong, Z. Liu, H.-L. Sun, J. Yang, W. Hu, Y. Rao, and Z. Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. arXiv preprint 2rXiv:2411.1445 2024.
[8] H. Duan, Yang, Y. Qiao, X. Fang, L. Chen, Liu, X. Dong, Y. Zang. Zhang, J. Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, pages 11198-11201 2024.
[9] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama herd of models. arXiv preprint 2024.
[10] T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14375-14385, 2024.
[11] D. Guo, D. Yang, H. Zhang, Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 12948, 2025.
[12] J. Guo, T. Zheng, Y. Bai, B. Li, Y. Wang, K. Zhu, Y. Li, G. Neubig, W. Chen, and X. Yue. Mammoth-vi: Eliciting multimodal reasoning with instruction tuning scale. arXiv preprint arXiv:2412.05237, 2024.
[13] A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et Openai ol system card, arXiv preprint arXiv:24/2.16720, 2024.
[14] R. Kamoi, Y. Zhang, N. Zhang. J. Han, and R. Zhang. When can 1lms actually correct their own mistakes? critical survey of self-correction of 1lms. Transactions of the Association for Computational Linguistics, 12:1417-1440, 2024.
[15] A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. A diagram is worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
[16] A. Kumar, V. Zhuang, R. Agarwal, Y. Su, J. D. Co-Reyes, A. Singh, K. Baumli, S. Iqbal, C. Bishop, Roelofs, et al. Training language models to self-correct via reinforcement learning, Xiv preprint X1X:2240.112111 2024.

---

[17] Z. Liang, Y. Xu, Y. Hong. P. Shang. Q. Wang, Q. Fu, and K. Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pages 405-409, 2024.
[18] H. Liu, C. Li. Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916. 2023.
[19] X. Liu, Ni, Z. Wu. C. Du, L. Dou, H. Wang, T. Pang, and M. Q. Shich. Noisyrollout: Reinforeing visual reasoning with data augmentation. arXiv preprint arXiv:2504. 2025.
[20] Y. Liu, H. Duan, Y. Zhang. B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. Mmhench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216-233. Springer, 2024.
[21] P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. 2023.
arXiv preprint arXiv:23
[22] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang. et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534-46673 2023.
[23] N. McAleese, R. M. Pokorny, J. F. C. Uribe, E. Nitishinskaya, M. Trebacz, and J. Leike. Llm critics help catch Ilm bugs. arXiv preprint 2xxiv2240.0022133 2024.
[24] F. Meng. Lingxiao du, zongkai liu, zhixiang zhou, quanfeng lu, daocheng fu, botian shi, wenhai wang, junjun he, kaipeng zhang, et al. mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arXiv preprint arXivv2503.073665 1, 2025.
[25] R. Munos, M. Valko, D. Calandriello, M. G. Azar, M. Rowland, Z. D. Guo, Y. Tang, M. Geist, T. Mesnard, A. Michi, et al. Nash learning from human feedback. urXiv preprini
arXiv:2312.00886 18, 2023.
[26] Y. Peng, X. Wang, Y. Wei, J. Pei, W. Qiu, A. Jian, Y. Hao, J. Pan, T. Xie, L. Ge, et al. Skywork rlv: Pioneering multimodal reasoning with chain-of-thought. arXiv preprint arXiv:2504.05599 2025.
[27] Y. Peng, G. Zhang M. Zhang, 7. You, J. Liu, Q. Zhu, K. Yang, X. Xu, X. Geng, and X. Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based arXiv preprint arXiv:2503.07536, 2025.
[28] Y. Qu, T. Zhang, N. Garg, and A. Kumar. Recursive introspection: Teaching language model agents how to self-improve. Advances in Neural Information Processing Systems, 37:55249- 55285, 2024.
[29] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly reward model. ArXiv, abs/2305. 188290 2023.
[30] X. Rong, W. Huang, J. Liang, J. Bi, X. Xiao, Y. Li, B. Du, and M. Ye. Backdoor cleaning without external guidance in milm fine-tuning, arXiv preprint arXiv:2505. 16916, 2025.
[31] L. Sun, H. Liang, J. Wei, B. Yu, T. Li, F. Yang, Z. Zhou, and W. Zhang. Mm-verify: Enhancing multimodal reasoning with chain-of-thouge verification. arXiv preprint arXiv:2502.13381, 2025.
[32] H. Tan, Y. Ji, X. Hao, M. Lin, P. Wang. Z. Wang. and S. Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprini arXiv:2503. 20752, 2025.
[33] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai,
A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv 2023.
preprint 2rXiv:2312../888
[34] K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. Kimi k1. 5: Scaling reinforcement learning with 1lms. arXiv preprint arXiv:2501. 125999 2025.
[35] O. Thawakar, D. Dissanayake, K. More, R. Thawkar, A. Heakl, N. Ahsan, Y. Li, M. Zumri,
J. Lahoud, R. M. Anwer, et al. Llamav-ol: Rethinking step-by-step visual reasoning in llms. 2025.
arXiv preprint arXiv:2501.061
[36] G. Tyen, H. Mansoor, V. Cärbune, P. Chen, and T. Mak. Lims cannot find reasoning errors, but can correct them given the error location. arXiv preprint arXiv:2311.088 2023.
12

---

[37] H. Wang, C. Qu. Z. Huang, W. Chu, F. Lin, and W. Chen. Vl-rethinker: Incentiviz ing self-reflection of vision- language models with reinforcement learning. arXiv preprint arXiv:2504. 08837, 2025.
[38] X. Wang, Z. Yang. C. Feng, H. Lu, L. Li, C.-C. Lin, K. Lin, F. Huang, and L. Wang, Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement. arXiv preprint 25t.::2550009333344 2025.
[39] Y. Wang, X. Yue, and W. Chen. Critique fine-tuning: Learning to critique is more effective than learning to imitate. arXiv preprint arXiv:2501. 17703, 2025,
[40] J. Wei, X. Wang, D.. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. Le, D. Zhou, al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837 2022.
[41] Wu, Y. Xie, Z. Yang, J. Wu, J. Gao, B. Ding, X. Wang, and X. He. beta-dpo: Direct preference optimization with dynamic 3. Advances in Neural Information Processing Systems, 37:129944-129966, 2024.
[42] T. Xiong. X. Wang. D. Guo, Q. Ye, H. Fan, Q. Gu, H. Huang, and C. Li. Llava-critic: Learning to evaluate multimodal models. arXiv preprint arXiv:2410.02771 2024.
[43] W. Xiong, H. Zhang, C. Ye, L. Chen, N. Jiang, and T. Zhang. Self-rewarding correction for mathematical reasoning. arXiv preprint arXiv:2502 19613, 2025.
[44] G. Xu, P. Jin, L. Hao, Y. Song, L. Sun, and L. Yuan. Llava-ol: Let vision language models reason step-by-step. arXiv preprint 1xXiv:2411.14444 2024.
[45] A. Yang. B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F, Huang, H. Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 151111,,,. 2024.
[46] S. Yang, E. Gribovskaya, N. Kassner, M. Geva, and S. Riedel. Do large language models latently perform multi-hop reasoning? arXiv preprini arXiv:2402. 16837, 2024.
[47] Y. Yang. X. He, H. Pan, X. Jiang, Y. Deng, X. Yang. H. Lu, D. Yin, F. Rao, M. Zhu, et al. RI R1-onevision: Advancing generalized multimodal reasoning through moosl formalization. arXiv preprint arXiv:2503. 10615, 2025.
[48] H. Yao, J. Huang. W. Wu, J. Zhang. Y. Wang. S. Liu, Y. Wang. Y. Song, H. Feng, Shen, et al. Mulberry: Empowering mllm with ol-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.183:
2024.
[49] S. Ye, Y. Jo, D. Kim, S. Kim, H. Hwang, and M. Sco. Selfee: Iterative self-revising I1m empowered by self-feedback generation. Blog post, 2023.
[50] W. Yu, Z. Yang, I.. Li, J. Wang, K. Lin, 7. Liu, X. Wang, Ind Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv: 02490, 2023.
[51] X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567, 2024.
[52] Y. Zeng, X. Cui, X. Jin, G. Liu, Z. Sun, Q. He, D. Li, N. Yang. J. Hao, H. Zhang, et al. Aries: Stimulating self-refinement of large language models by iterative preference optimization. arXiv 2025.
preprint arXiv:2502.00660
[53] D. Zhang, J. Li, J. Lei, X. Wang, Liu, Z. Yang, J. Li, W. Wang, S. Yang, J. Wu, et al. Critic-v: Vlm critics help catch vlm errors in multimodal reasoning. arXiv preprint 2rX1v:2411.18203, 2024.
[54] Zhang, J. Huang, H. Yao, S. Liu, X. Zhang, S. Lu, and D. Tao. RI-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.. 12937, 2025.
[55] R. Zhang, D. Jiang, Y. Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K.-W. Chang, Y. Qiao, et al. Mathverse: Does your multi-modal IIm truly the diagrams in visual math problems? In European Conference on Computer Vision, pages 169-186. Springer, 2024.
[56) R. Zhang, B. Zhang, Y. Li, H. Zhang, Z. Sun, Z. Gan, Y. Yang, R. Pang, and Y. Yang. Improve vision language model sfain-of-thouglhe reasoning, arXiv preprint arXiv:2410.16198. 2024.
13

---

[57] Y. Zhang, M. Khalifa, L. Logeswaran, J. Kim, M. Lee, H. Lee, and L. Wang. Small language models need strong verifiers self-correct reasoning. arXiv preprint arXiv:2404.177 171440, 2024.
[58] K. Zhou, C. Liu, X. Zhao, S. Jangam, J. Srinivasa, G. Liu, D. Song, and X. E. Wang. The hidden risks of large reasoning models: A safety assessment of rl. arXiv preprint arXiv:2502. 12659, 2025.
[59] J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, Y. Duan, H. Tian, W. Su, J. Shao, et al. Internv3:: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv: 2025. 2025.
14

---

A Mathematical Derivations
In this section, provide detailed derivation of our Sherlock objective leading to Eq. 6. We extend the derivation structure of prior methods such as SRPO [5] and ARIES [52] from the response level more fine-grained trajectory level.
max E ( yt 2.)) p(2 - - IIar, Y t; xx
+ Eyg X (.23)) p(Y > 12 x2) - ( Y t; (i1)
(9)
* 1 Y X), xx)))
max Byz M p(Y + Y2 FI&T; x2) Blog
N 2 II&T Y Y2.)
π T IKT Y (2,
Byg, " p(Y > 1 Y2i)
Blog Tel
2 IKT Y
(10)
p(
Y 1 V2.)
T ( 17,4
logexp
max B (f
log Teal I Y1, Y2, (2D)
B
(IIT: Y21)
+ [I&T Y1, 1; (2,))))
logexp
+ (
log
Fral(
I&&T, Y y1,4; 221))
(11)
x x [ ,1, (
(TRRT, Y 11,4Y3)
max - 31 ((( r
log
Tee Tre [IITT YI, t1,+) exp
3
1 Y f2 ( Y 22
- - (olllll
log
Tel (Y3 [IIr. YI, 1t: exp
Blog (IIITT Y (2,)))) + Bog Z(INET Y Y (4;2))
(12)
plY plY 2.
Trel (Y [IITT Y r; 12,1) exp
max -3DLL xYSilEET Y 12,1
Z(I Y Yi,,; Y2,)
Teet Te3((1l(1 Y1,t exp
-3DKL
TY2illETTT y ft (2.))
9
Z Yl, (2,4;Y2)
+ log (XIITT Y ,4;Y22) + 3 log Z(ZINT, Y 122)),
(13)
where 2(X, XY 22; Y2,) and Z(X, Y 2; Y2,) denote the partition functions. Noting the non- negativity of the KL divergence, the optimal proxy is given by:
P(Y YY (
Tief (Y2 Y1, 12,4Y2)) exp
x" T"Y2, ,1,
(14)
Z(ZIET,YI ,1,,
p(Y (
Teel (Y2 Y1,422) exp
*" "YY2,lIEIET Y¹ t; Y2d))=
(15)
Z(Z Y1,t;
Therefore, we have:
TYYY3,
II&T. Y Y1,+:Y2,)
MY K v2,) =Blog
+ + log Z( Y .442)) (16)
Neel Y2
Fier, Y1,4;22]
π 3
II&T, Y Y1,()
P(Y > KallEeT; X21)) Bllog
+ ((xeeT Y :t: Y2) (17)
[IIT, Y1,))
15

---

Recalling the definition of p(Y > (2( in Eq. 5, we have p( X 2 Y2)) = 21
and p
Y x1&:;
Y 21) 1.2 Then, combining with Eq. 16, and 17, we obtain:
n- (Y I Y ,1,)
2 = 3 log
FINT ,1, + Bog Y
(18)
Tef (Y
n 2 [ Y ,1,;; 22,11%,)
3 log
Tee Z IINT, (1,4)))) + Bog ( ,4Y2))
(19)
2
By subtracting Eq. 18 from Eq. 16, and Eq. 19 from Eq. 17, we obtain:
p(rz - 12 - = log T T( Y ,2,422))
" TY (144
-log
Teat (T 2i [INT, ,7,4
Tee Tee |2 x1,t;Y2]
+ BBlog (IIKTT ,4; 22)) - log ( ,1,4;22))
T"" (Y [I&T Y ,,, Y Tar
II&T Y t; ((,l [IITTT Y ,
Z(TIST Y (1,1 Z(II&T, y1, x2,)
= Blog
+ +Boog
(Y 21 [I&T Y1, , Y |T
(11) (X
(20)
P(Y > F 22) 3 log T' T' [I&T Y X2, ,t;
" T Y 222l)
log
TTat| 2,1 IINT Y1 1t:
Teet (Y [IRT, Y t; Y2,1)
+ Blog ((IINT, Y t; x1) - log ( Y ,,, 1,,;Y2211 221)
Y Y, Tar [I&TT Y ,4:Y221)
Z(TIRTT Y 11,4 Z(TTKTT 11,4
= Bog
+ 3log
Tref 21 [I&T Y1, 4; Y T [IITT 11,4
(21)
Next, by adding Eq. 20 and Eq. 21, we obtain the following expression:
To [z, Y', x222))
To (Y
[z. St,YY,,;Y (2,))
v(2, y1 11 23: 2o) mo) ,3 log
Blog
2i
T
21 [z r1,,;22
21
[x,
( [z, Y', X2,))
Tol (Y
[z, Y',t;
u(r, y1 11 2 Y2 nO) =3llgg
BBlog
21
nell Y
21
[z, Y1,
Ter 21
[r, ,1,t;Y2
p(Y
Y
IiRe; + P(
Y SERTIT -1
=(IITT Y Y Y2) + u(TIeT. Y
Y2.)
(22)
Finally, we follow prior work [52, 25, to adopt the Mean Squared Error (MSE) as the loss function and parameterize the policy model as Ta. The training is then performed on the preference dataset D using the following loss:
To (Y Y f; Y
To Ta [x,
1; Y2,)
(zz Y ,Y,t =0) 3log
3 log Tha ( I
Teaf (Y I Y Y
Encourage posilive sell colf-corecction
Discourage no self-corrctiinn
To [x, Y Teet
Y
To (Y Y 4; Y Tael [z, Y +, (2,)
u(z, Y t; YL TT) = 3 log
3 log
YI,
Encourage positive self-correcton
Discourage self-correction
To TY Y 1;
To TY Y Y Y
u(x, y t; 2e y m0) )2)
Blog
3 log Teet
Teet (Y [x, Y t DY.
Discourage negative self-comection
Encourage no colf-coorection
To (Y [x. Y 1;
no (Y
w
[r, V Y!
w(x, 4 1,; <3>
y1 Y2 = Blog
3 log
Tee Ir Y t:
neel
W t;
Discourage nogative self-correction
Encourage self-correction
16

---

L Tret)
[1 v(aTT y 14,4 Y2 (2)))) ulanT, Y y 11 V mo)] + (ly [] o Y", 1; Y Y mo) + u(are,, Y Y", 44: Y 3 mT) (23)
Our self-correction loss is designed to primarily encourage the model to learn desirable serf-coore behavior. Specifically, promotes positive self-correction: revising a low-quality response Y' into a high-quality one y", and supports from Y to y". Conversely, the loss penalizes negative self-correction, such as degrading a good response y into poor one Y', well as failing revise incorrect response, i.e., y' to y
self-correction when the initial response is already correct, i.e.,
B Implementation Details
B.1 Self-Correction Prompt
Considering the four-stage thinking structure of the LLaVA-CoT [44] dataset, we design the following self-correction template by modifying the template proposed in ARIES [52].
Self-Correction Prompt
Below QUESTION from user and an EXAMPLE RESPONSE.
Please provide a more helpful RESPONSE, improving the EXAMPLE RESPONSE by making the content even clearer, more accurate, and with reasonable logic. Focus on addressing the human's 's QUESTION step by step based on the image without including irrelevant content.
QUESTION: {question)
EXAMPLE RESPONSE:
[cxample_response]
Now, refine and improve the RESPONSE further. You can consider two approaches:
1. REFINEMENT: If the SUMMARY section in the response is closely related to the question,
the CAPTION section accurately describes the image, the REASONÍNG section logically clear and correct without any contradictions, and the CONCLUSION provides an accurate answer based on the previous steps, enhance clarity, accuracy, or reasoning logic as needed.
2. NEW RESPONSE: If the SUMMARY section incorrectly summarizes the intent of the issue, the CAPTION contains content unrelated to or incorrect about the image, there are logical errors contradictions in the REASONING, or the CONCLUSION incorrectly states the findings, please enhance the accuracy and quality of each step, and craft a more effective RESPONSE that thoroughly resolves the QUESTION.
RESPONSE:
B.2 Data Construction and Training Details
In this section, we present the detailed data construction procedures for each stage of the Sherlock framework. Specifically, a total of 20k annotated examples and 10k unlabeled questions are randomly sampled from the LLaVA-COT [44] dataset for training.
of 10k annotated examples from the LLaVA-CoT dataset, denoted as D and DB- We first use D to train the Llama3.2 2- Vision- IB- Instruct [9] model for one epoch using vanilla SFT loss follows, enabling the model generate responses in a fixed reasoning template:
Stage I: SFT Cold-Starting. We randomly sample
(TF((T) = -E (X(]rr))
(24)
We denote the resulting model as RO VLM. Next, we sample responses on Dn using RO VLM, i.e., y ~ TRo vLM(( #I&T), which yields relatively low-quality outputs following the CoT template. Each sampled response is paired with its corresponding annotated answer and input question to
17

---

Table 6: Detailed training hyperparameters for each stage of the Sherlock model.
Sherlock model Learning Rate
Max Length
Batch Size
α
BORO
Warm-Up Ratio
Epoch
SFT
le-6
4096
128
0.03
3
Offline
5c-6
4096
32
0.25
0.1
0,00
herl
5c-7
4096
0.25
0.1
0.00
Iter2
5e-7
4096
32
0.25
0.1
0,00
0.325
0.275
0.250
0.200
0.175
0.0
0.2
0.4
0.6
0.8
1.0
Figure 5: Values of dynamic 3 under different truncation steps and visual perturbation levels
form the Sherlock SFT dataset, denoted as SPT {z, yi, y"). Finally, we apply the Sherlock- SFT loss defined in Eq. to cold-start the base VI,, equipping it with both reasoning and self-correction capabilities.
Stage II: Offline Training. In the offline preference training stage, we construct trajectory- level preference data using the 10k examples in DA. The LLaVA-CoT dataset comprises four reasoning stages: <SUMMARY>, <CAPTION>, <REASONING>, and <CONCLUSION>, where the fi- mal <CONCLUSION> stage only provides the answer a. We randomly sample truncation step 2(1,2,3 3,4) 4} to determine the prefix. Specifically, when 1, no prefix retained; when i = 2, 3, or 4, we retain the SUMMARY, SUMMARY+CAPTION and SAMMMAR*CCAPTIN of the REASONING stage, respectively. To generate lower-quality suffix reasoning trajectory Y a-1 Yei), we introduce random visual perturbation to the image input [30], which affects the model's 's visual perception during subsequent decoding.
the first half
Stage III: Online Self- Self-Improvement. With the reasoning and self-correction capabilities of the Sherlock Offline model already activated, we aim to leverage the natural preference relationship between responses before and after correction self-construct a preference dataset for iterative self-improvement training. Specifically, for each randomly sampled imput question, we perform direct reasoning followed by three rounds of self-correction, obtaining responses Y', Y y and Y To reduce noise in the preference pairs, we apply self-consistense filtering strategy: we only retain Y as the preferred response " if the final answers y²2 y and y are semantically consistent. To construct the rejected response y we use y as the base and follow the same perturbation-hatt based procedure in Stage the same prefix by construction, i.e., Y Yč:. However, in Stage 3, since Y' is generated via three turn self-correction based on Y their prefixes often differ, and thus the prefix portions of the preference pairs Y and Y are typically not equal.
worth noting that, Stage 2, the chosen and rejected responses share
Dynamic B. We build upon static baseline of 3 = 0.25 and introduce controlled dynamism through randomly sampled truncation steps and image perturbation levels e. Considering that as reasoning progresses, the model's reliance on visual input typically decreases with longer generations, we aim make 3 less sensitive changes more conservative policy updates when the quality gap between preference pairs is large, that is, when the truncation step small and the image perturbation is large, while encouraging more assertive learning when the quality gap is small. Therefore, 3 should be negatively correlated with i and positively correlated with e. Fig. 5, we visualize how 3 varies with for = 1,2,3,4, demonstrating that the dynamic 3 design in Eq. 8 aligns with our preferred properties.
e when is large. In addition, we aim adopt
18

---

Training Details. Our Sherlock is built on Llama3,2 Vision- IB Instruct [35] model. In both the SFT and offline training stages, we randomly sample 10k annotated examples from the LLaVA- CoT [44] dataset, and follow the procedures introduced above to train the Sherlock SFT and Offline models with Eq. 3 and Eq. respectively. We then perform two rounds of online self-improvement using 5k unlabeled data per round, containing only input questions and images. Following the self-consistency selection rule in Sec. 4.3, we construct self-generated preference dataset. We then train the models using the objective in Eq. 7, resulting in Sherlock Iterl and Iter2. The detailed training hyperparameter provided Table 6.
Sherlock Algorithm. To provide a more intuitive understanding of our Sherlock, we summarize the detailed algorithm in Algorithm including the training data, loss functions, and learning procedures.
Algorithm 1: Sherlock: Self-correction and Self-improvement training framework
Input:
D (Zi&rT, Y"),, DB (11&T, Y"), base VLM iteration T, correction prompt Sherlock model Thherookk
Output:
Stage 1: SFT Cold-Start
2
TVLMER < arg min (TF (Eq. 24) on DA
0
3 DSherlock < {(TTTTT Yi, "" TBn (IM( (IINT, Y²)= E Dn)
SST arg min SFhedockkSFT (mT)
(Eq. 3) on SFTT
Stage 2: Offline Preference Training 6 Domtine f- for (I1 Y in DA do 1(0,1)
i i 1,
Y f Y 12 T (H Y B(i, 4, (Eq. 8)
Dottine + Domin U {xIaT, Y',Y",""" y",
10 TSherkock-Offinee - arg min Sherlook (To; Tref) (Eq. 7) on Doffine
Stage 3: Online Iterative Self-Improvement
12
Sherllokk-terr f TSheelockOOfflin
for t
do Duer& t t /
14
15
while 'Duertl < 5000 do
16
Randomly sample question F&& without any annotation
17
Y1 +- TTeerlock her: ( for j = to 3 do
18
yi+1 < Therrluck tte 1-1 [1&r. Y),t)
19
2 a then i 2f1,2,1,,3 4), ((, Y + y4 y 8 3(i,4,c) (Eq. 8)
20
V < Sherrheck her -- (1IIT, (i)
21
22
Duert + Dieri fZier, Yi, y
23
else
24
Skip
TThherlock-terr +- arg min CShertock (To; Tret) (Eq. 7) on D
0
C Evaluation Detail
C.1 Evaluation Benchmarks
We conduct experiments on eight multimodal tasks, covering comprehensive VQA benchmarks (MMBench- V [20], MMVet [50], MME [17], MMStar [4]), math and science benchmarks (Math- Vista [21]. A12D [15], MMMU [)1]), and hallucination benchmark (HallusionBench [10]). We evaluate Sherlock under two settings: direct generation and after three rounds of self-correction. For
consistency and reproducibility, all models are evaluated using the VLMEvalKit [8] pipeline.
19

---

MMB.nch-V1.1. MMBench-vI.J contains a total of 3217 samples and is designed to evaluate the visual perception and visual reasoning capabilitties of VLMs. It adopts the CircularEval protocol, which presents each question to a VI.M multiple times with shuffled answer choices. A model considered successful only if it selects the correct answer in all attempts, thereby reducing the influence of random guessing in multiple-choiee questions.
MMVet. MM consists of 218 open-ended questions designed to comprehensively assess VLMs across six dimensions: Recognition, OCR, Knowledge, Language Generation, Spatial Awareness, and Math. The final evaluation results are obtained by scoring the model-generated open-ended responses using GPT4-TuTra Turbo as the grader.
MME. MME is a comprehensive multimodal benchmark designed to assess two core capabilities of VLMs: visual perception and reasoning. It consists of 10 perception tasks and 4 reasoming tasks. The overall perception and reasoning (cognition) scores are computed as the sum of their respective subtask scores, with the total maximum score across all tasks being 2800.
MMStar.
MMStar consists of 1500 test samples and aims to address key issues in evaluation, such low visual-textual alignment and potential data leakage during training. The benchmark is carefully curated and covers 6 core capabilities and 18 detailed evaluation axes.
Math Vista. Math Vista introduces unified benchmark with 6141 examples collected from 28 existing datasets and newly curated ones (IQTest, FunctionQA. and PaperQA). In our experiments, we evaluate models on the best-mine split, which contains 1000 samples.
MMMU. MMMU is an expert-level multimodal benchmark developed to assess the perception, knowledge, and reasoning abilities of VLMs. For our experiments, we use its validation set, which comprises 900 multimodal samples.
AI2D. We evaluate model performance on the Tee__TETT____LS setting from VLMEvalKit, which assesses VLMs' ability to interpret grade-school science diagrams.
Benc. HallusionBench is a challenging benchmark designed to evaluate image-context reasoning in VLMs. It consists of 346 images and 1129 expert-crafted questions, targeting nuanced visual understanding. We report the results as the average of the three evaluation metrics provided by VLMEvalKit.
C.2 Evaluation Baselines
Considering that our approach is primarily initialized using the reasoning data from LLaVA-CoT [44], our main experiments compare with other SFT-based methods built on the Llama3.22 2- Vision-1 1B- Instruct [9] model, including LLaVA-CoT, Mulberry [48], and Llama [35]. In addition, to further investigate the self-correction behavior of reasoning VL.M, we also include experiments and analysis on the RL-based method VL-Rethinker [37].
LLaVA-CoT. LLaVA-CoT constructs 100k seage-level Chain-of- Thaun-of-Thought (CoT) annotations using multi-turn prompts to GPT-4o, and then trains the model via standard supervised fine-tuning (SFT) acquire reasoning ability. Each response is structured into four stages: <SUMMARY>, <CAPTION>, <REASONING>, and <CONCLUSION>
Mulberry. Mulberry constructs 260k long CoT annotations by applying Monte Carlo Tree Search (MCTS) over strong models such Qwen2- VL and GPT-4o, with a particular focus the mathemat- ical domain. addition, incorporates supplementary self-reflection SFT data to encourage models to engage in step-wise reasoning. first reflecting prior errors and then performing corrections.
LlomaV-ol. LlomaV-ol extends the LLaVA-CoT dataset to 175k samples and adopts a multi-turn questioning and training strategy during inference to simulate the four reasoning stages: summary, caption, reasoning, and conclusion, thereby enabling step-by-step reasoning behavior.
20

---

VL-Rethinker: VL-Rethinker is built upon the Qwen2.5-VL series models and employs GRPO- based reinforcement learning to enhance reasoning capabilities. Additionally, introduces limited amount of self-reflection and correction data to enforce a self-reflection objective, aiming to train the model revise earlier responses within a single thinking process.
C.3 Evaluation Setup in Section 3
Prompt for Modify One Step Setting. Below is the prompt specific LLaVA-CoT models.
Modify One Step Prompt for Qwen2.5 Innnnutt-Isttuut-77 on LLa VA-CoT
You are professional sentence rewriting expert. Your task is to modify the given sentence by altering key numbers, key phrases so that the resulting sentence conveys different or in- correct meaning, finally leads to wrong answers. In essence, you should transform a logically correct factually accurate statement into one that contains logical or mathematical error. You should output the original format but with incorrect logic or meaning thay may leads wrong answers. Ensure that the intention or logic of your version clearly differs from the original input.
<EXAMPLE>
[INPUT]
### Question
Find RS if AQRS is an equilateral triangle.
### Answer
<SUMMARY> To solve the problem, will use the properties of an equilateral triangle to find the length of side RS. will up equations based on the given side lengths and solve for RS. </SUMMARY>
<CAPTION> The image shows an equilateral triangle AQRS with side lengths labeled as QR = 4z, QS = 6F 1, and RS = 2x + 1. </CAPTION> <REASONING> The triangle is equilateral,
all sides are equal. Therefore, we can up the equation QR QS RS. This gives us the equations 4r = 6r and 4r = 2r + 1.
[YOUR REWRITTEN]
<SUMMARY> To solve the problem. will use the properties of an equilateral triangle to find the length of side RSS. will set up equations based on the given side lengths and solve for RS. </SUMMARY>
<CAPTION> The image shows equilateral triangle AQRS with side lengths labeled as QR = 4z, QS = 9x 7, and RS = 2x </CAPTION> <REASONING> The triangle is equilateral,
all sides equal. Therefore, the equation QR = QS RS. This gives us the equations 4r = 9r and 4r 2r Make sure only output the rewritten answer with original format.
up
[INPUT]
[YOUR REWRITTEN]
Make sure only output your rewritten answer with original format, not output the question.
RESPONSE:
Below the prompt specific VL-Rethinker models.
Modify One Step Prompt for Qwen2.5- Instruccttt on VL--Rthinner
You are professional sentence rewriting expert. Your task is modify the given sentence by altering key numbers, or key phrases so that the resulting sentence conveys different or in- correct meaning, finally leads to wrong answers. In essence, you should transform a logically correct or factually accurate statement into one that contains a logical or mathematical error. You should output the original format but with incorrect logic or meaning thay may leads wrong answers. Ensure that the intention or logic of your version clearly differs from the original input.
21

---

<EXAMPLE>
[INPUT]
### Question
Find RS if AQRS is an equilateral triangle.
### Answer
Since triangle QRS is an equilateral triangle, all its sides are equal in length. Therefore, we the expressions for the sides equal to each other and solve for The expressions for the sides are: QR 4x RS 2x
[YOUR REWRITTEN]
Since triangle QRS an equilateral triangle, all its sides are equal in length. Therefore, we can set the expressions for the sides equal to each other and solve for x. The expressions for the sides are: QR RS 2x
Make sure to make the rewritten sentence with explicit wrong steps or logic.
[INPUT]
{input]
[YOUR REWRITTEN]
Prompt for External Critique. In the external critique setting, we primarily follow the setup of Critic-V [53], which includes both the critique generation prompt and the correction prompt.
External Critique Generation Prompt
#### Question
{question]
#### Answer
[result]
#### Task
Please provide a critique of the answer above.
Self-Correction Prompt via External Critique
Reflection on former answer:
#### Former Answer
(answer)
(critics) }
#### Question (original_questionn
C.4 Evaluation Setup in Section 5
We report Sherlock's 's performance under direct reasoning and three-round self-correction settings. We also evaluate MM-Verify verification mechanism accelerate inference-time scaling, with the corresponding prompt shown below.
Prompt for MM- Verify
MM-Verify
Solve the math problems and provide step-by-step solutions, ending with "The answer is [Insert Final Answer Here]".
When asked "Verification: Is the answer correct (Yes/No)?", respond with Yes" or No" based on the answer's ' correctness.
When asked "Verification: Let's verify step step.", verify every step of the solution and conclude with "Verification: Is the answer correct (Yes/No)?" followed by Yes" or No".
Q:
{question]
A: Let's think step by step.
22

---

{answer]
Verification: Let's verify step by step.
Self-Correction Prompt Based on Critique from MM- MerV-erify
Below QUESTION from user and an EXAMPLE RESPONSE.
Please provide a more helpful RESPONSE, improving the EXAMPLE RESPONSE by making the content even clearer, more accurate, and with reasonable logic. Focus on addressing the human's QUESTION step by step based the image without including irrelevant content.
QUESTION:
{question}
EXAMPLE RESPONSE:
exxample_repponss
CRITIQUE:
{critique)
Now, refine and improve the RESPONSE based on provided CRITIQUE. You can consider approaches:
REFINEMENT: If the SUMMARY section in the response is closely related to the question, the CAPTION section accurately describes the image, the REASONING section is logically clear and correct without any contradictions, and the CONCLUSION provides an accurate answer based on the previous steps, enhance clarity, accuracy, or reasoning logic as needed. 2. NEW RESPONSE: If the SUMMARY section incorrectly summarizes the intent of the issue, the CAPTION contains content unrelated or incorrect about the image, there are logical eπrors or contradictions in the REASONING, or the CONCLUSION incorrectly states the findings. please enhance the accuracy and quality of each step. and craft more effective RESPONSE that thoroughly resolves the QUESTION.
RESPONSE:
D Experimental Results
D.1 Training Efficiency of Sherlock
Table Training efficiency and performance compared to other baselines.
Methods
Training Time (L)
Accuracy (f)
LLaVA-CoT [44]
160
63.2
LlamaV-or V-ol [35]
288
63.4
Sherlock
128
64.1
Although Sherlock adopts multi-stage framework, its data efficiency and RL-free design make training highly efficient. As shown in Table 7, Sherlock outperforms LLaVA-CoT and LlamaVool while requiring the least training time.
D.2 More Results of Sherlock Performance
Sherlock Performance of Each Correction Turns. We report the average performance of the Sherlock Iter2 model across eight benchmarks under different numbers of self-correction rounds. As shown in Fig. the Sherlock model consistently improves its accuracy with more rounds of self-correction, demonstrating stable diference-time scaling.
Initialize Sherlock on Reasoning VL.Ms. The goal of Sherlock not merely to improve upon existing CoT models, but to introduce new framework that transforms base VLM into reasoning
23

---

Accuracy
65.7
65.4
65.3
65.1
64.9
64.5
64.1
64.1
Generation
Turn2
Figure 6: Average accuracy of the Sherlock Iter2 model across eight benchmarks under direct generation and varying numbers of self-correction turns.
Direct Inforence
Average Accuracy (%)
Self-Comeation
65.4%
66
64.4%
64.2%
84.1%
63.6%
63.2%
63.2%
63.2%
63.0%
63
62.8%
LLaVA-CoT
LLaVA-CoT Cold-Start
LLaVA-CoT Offine
Sherlock Offine
Sherlock Iter2
Figure 7: Performance comparison different initialization strategies. LLaVA-CoT Cold-Start and LLaVA-CoT Offline denote post-training the LLaVA-CoT model using the "Cold-Start" and "Offline" stage strategies of Sherlock, respectively.
VLM with strong reasoning and self-correction capabilities. However, if we use VAaCoOT the base VLM for Sherlock's multi-stage training, severe overfitting may occur because Sherlock's train- ing data randomly sampled from LLaVA-CoT 100k, meaning the model has already extensively seen these samples during its original training. To address this issue, we instead consider applying the offline stage's ' preference training objective to post fine-tune the LLaVA-CoT model directly. Results in Fig. 7 reveal two key observations. First, initializing LLaVA-CoT with the Cold-Start stage enhances the model's 's self-correction ability but slightly degrades its direct inference accuracy, which consistent with our previous analysis. Second, applying offline preference training on LLaVA-CoT successfully improves both direct generation and self-correction capabilities. However, compared to Sherlock Offline, the self-correction ability remains limited, as the training data have already been seen during LLaVA-CoT's SFT stage.
Table 8: Performance comparison between LLaVA- Reasoner-DDD and Sherlock across 8 benchmarks.
#Data
Models
MMB MMVet Hallus
MMMU MMStar
AI2D MathV
MME
Avg.
w/ GT
Llama3.2V-11BBmm.
65.8
57.6
42.7
47.8
53.0
88.2
49.7
1822
58.7
LLaVA-Reaoner-DOO
73.5
56,2
45.9
42.4
57.1
87.6
50.8
2001
60.6
Seff-Correction
258k
75.5
53.9
43.1
43.9
54.6
87.8
46.0
1875
59.0
Sherlock Iter2
74.6
62.4
49.7
57.7
89.6
52.0
2197
20k
Self- Seff-Correction
77.2
62.6
51.2
50.1
59.0
90.6
54.0
2204
65.4
Comparison with More Baseline.
To further validate the effectiveness of Sherlock, we compared DPO [56] across eight benchmarks.
its performance with trained via DPO [29] by distilling from stronger model (GPT-4o) and constructing preference pairs based on answer accuracy. For fair comparison, we trained Llama3.2 Vision- IB-Inttruct [9] as the base model. The results in Table show that, even with stronger model initialization, CoT training, revealing limitations. contrast, Sherlock significantly enhances both the model's direct reasoning and self-correction capabilities.
-Do using
fails to further improve performance through

---

E
Case Study
Case of Sherlock Self-Correction
The United Nationn
Problem: What color the text image? (A) White (B) Blue (C) Black (D) Grey
Sherlock Direct Generation Response.
<SUMHARY> will analyze the image determine the color of the text by examining its appearance and comparing to the given options. </SUMMARY>
<CAPTION> The image is a slide titled "The United Nations," containing bullet points about the organization's 's formation and functions. The text is set against blue background, and there is an image a UN General Assembly meeting. </CAPTION>
<REASONING> To determine the color of the text, will look hue brightmess in contrast to the background. The text appears to be lighter shade compared to the blue background, indicating it a neutral color such as grey white, Since it is not black, option C can be eliminated. Comparing it to the other options, white would be too bright, so it is likely grey. </REASONING> <CONCLUSION> D </CONCLUSIDN>
Sherlock Ceff-Corecction Turn Response.
<SUMMARY> will analyze the image to determine the color of the text by examining its appearance and comparing to the given options. </SUMMARY>
<CAPTION> The image a slide titled "The United Nations." contains bullet points about the United Nations' formation and functions, with an accompanying image of a UN General Assembly meeting, The text is prominently displayed against blue background. </CAPTION> <REASONING> To determine the text color, I will look at its contrast with the background. The text appears lighter than the blue background, suggesting is neutral color. Comparing it the options provided, the closest match is grey, as it typically appears lighter than black or blue and not as bright as white. </REASONING>
<CONCLUSION> D x </CONCLUSION>
Sherlock Self- Celf-Correction Turn Response.
<SUMMARY> To determine the color of the text in the image, I will carefully examine the image to identify and describe the text's color, then confirm it matches one of the given options. </SUMMARY> <CAPTION> The image a slide titled "The United Nations," containing information about the organi- zation's 's formation and functions. It includes a photograph of meeting at the United Nations General Assembly. The text is presented in clear, legible font against gradient blue background. </CAPTION> <REASONING> Upon examining the image, the text stands out against the blue background. Its color appears be neutral shade that contrasts with the background, which typically characteristic of color gray. This observation aligns with the options provided, helping to identify text color accurately. </REASONING>
<CONCLUSION> D X </CONCLUSION>
Sherlock Self-Correction Turn Response.
<SUMMARY> To determine the color of the text in the image, I will carefully analyze the visual elements and identify the color of the text by comparing it to the background and other elements in the image. </SUMMARY>
<CAPTION> The image a slide titled "The United Nations," containing bullet points about the history and functions of the United Nations. The text is presented against gradient blue background, with an image of a United Nations General Assembly session the right. </CAPTION> <REASONING> The text the slide clearly visible and stands against the blue background. By comparing the text's color to the surrounding blue hues, determine the text is different color. The text appears to neutral, light color that contrasts with darker shades of in hackground. </REASONING>>
<CONCLUSION> A / </CONCLUSION>
25

---

Case 2 of Sherlock Self- Correctontt
Problem:
What's the function of the demonstrated object?
(A) Cooking (B) Cook soup (C) Fry (D) steam
Sherlock Direct Generation Response.
<SUMMARY> I will analyze the image to identify the object and its function, focusing its features and context to determine the correct answer from the given options. </SUMMARY>
<CAPTION> The image shows a stainless steel pan on a stovetop. Inside the pan, there a piece of fish being cooked in water. There are also vegetables, such as corn and a tomato, visible nearby on the stove. </CAPTION>
<REASDNING> The object in the image is stainless steel pan placed on stove, containing a piece of fish. The presence heat from stove and the fish heing submerged water indicates that it is being used for cooking. Among the options provided, "Cooking" is the most general term that encompasses the activity taking place. While the could be used to fry, cook soup, steam, the most straightforward answer based the image alone that it is being used for cooking general. </REASONING> <CONCLUSION> (A) Cooking </CONCLUSION>
and
Sherlock Self-Correction Turn Response.
<SUMMARY> analyze the image to identify object function, focusing on features and context to determine the correct answer. </SUMMARY>
<CAPTION> The image shows a stainless steel frying pan of fish being cooked in oil melted fat, indicating that it being used for cooking. There are also some vegetables,
stove. Inside the pan, there is a piece
as corn and tomato, visible nearby. </CAPTION> <REASDNING> The object in the image a frying pan, which is typically used for frying foods. The presence of the fish being cooked in the pan suggests that it is being used for frying. The pan's shape and the cooking environment further support this conclusion, as frying usually involves cooking in pan over direct heat. </REASONING> <CONCLUSION> (C) Fry </COICLUSION>
Sherlock CeffCorrectiin Turn Response.
<SUMMARY> I will analyze the image identify the object and its function, then determine which option best describes its use based on its features and context. </SUMMARY>
<CAPTION> The image shows a stainless steel pan on stovetop, containing water with a piece of fish submerged. There are vegetables, such as corn and tomato, nearby the counter. </CAPTION> <REASDNING> The object contains water and a piece of fish, which suggests that is being used to cook the fish. The presence of the stovetop and the way the fish is submerged in water further implics that the pan is being used for a cooking method that involves boiling frying. Given the options, frying typically involves submerging food in hot oil or water and is consistent with the setup shown. </REASOKING> <CONCLUSION> C </CONCLUSION>
stainless steel pan placed
stove, indicating it designed for cooking.
Sherlock Self-Correction Turn Response.
<SUMMARY> I will analyze the image identify the object and its function, using visual cues determine the correct answer from given options. </SUMMARY>
<CAPTION> The image shows a stainless steel frying pan on gas stove. Inside the pan, there a piece of fish being cooked in oil or melted fat, indicating that it is being used for cooking. There are also some vegetables, such as and tomato, visible in the background. </CAPTION> <REASDNING> The object in the image frying pan, which is typically used for frying foods. The presence the fish cooking involves cooking food in hot oil or fat, which matches the scenario depicted. Other options like cooking soup or steaming require different types of vessels, such as pots with a lid for steaming or larger pots for soup. </REASONING> <CONCLUSION> C / </CONCLUSION>
pan with visible layer oil further supports this function. Frying
26

---

Case 3 of Sherlock Self-Correction
Problem: Hint: Please answer the question and provide the correct option letter, c.g., A, B, C, D at the end. Question: Is it split in half?
Choices: (A) Yes (B) No
Sherlock Direct Generation Response.
<SUMHARY> I will amalyze the image to determine if the pizza is split in half by examining its shape and visible cuts. will then compare my findings with the given options to provide the correct answer. </SUMMARY>
<CAPTION> The image shows pizza white plate. The pizza has thin crust and topped with cheese and basil. It appears to have been cut into several slices, with two main cuts extending from the center towards the edge, suggesting it is divided into halves. </CAPTION>
<REASONING> To determine if the pizza is split in half, I will look for clear cuts that divide the pizza into main portions. The pizza has prominent diagonal cuts starting from the center, meeting the edge and creating two large halves. Additional smaller cuts are also visible, but the primary division into two is clear. </REASONING> <CONCLUSION> X </CONCLUSION>
Sherlock Self-Correction Turn Response.
<SUMMARY> I will analyze the image to determine if the pizza is split in half by examining its cut pattern and deseribe my findings systematically reach the correct conclusion. </SUMMARY> <CAPTION> The image shows pizza white plate. The pizza has thin crust and topped with cheese and basil strips. is cut into several slices, with two prominent cuts that divide </CAPTION>
halves.
<REASONING> To determine if the pizza is split in half, I will look at the number and pattern of the cuts. The pizza has main cuts that from the center the edge, dividing into two equal halves. There are additional smaller cuts, but the primary division into two halves. </REASONING> <CONCLUSION> X </CONCLUSION>
Sherlock Self-Correction Turn Response.
<SUMMARY> analyze the image to determine the pizza is split half by examining the number of slices and their arrangement. will then compare my findings with the given options to provide the correct answer. </SUMMARY>
<CAPTION> The image shows pizza white plate. The pizza has thin crust and is topped with cheese and hasil. It appears to be cut into eight slices, arranged in symmetrical pattern. </CAPTION> <REASDNING> To determine the pizza split in half, will count the number slices. it two large slices separated by clear line down the middle, it would be considered split half. However, the pizza is cut into eight slices, indicating is not divided simply into two halves. The slices are evenly distributed across the pizza with no large halves. </REASONING> <CONCLUSION> B </CONCLUSION>
Sherlock Aff-Correction Turn Response.
<SUMMARY> I will analyze the image determine if the pizza is split half by examining the number of slices and their arrangement. will follow structured approach to reach the conclusion. </SUMMARY> <CAPTION> The image shows a pizza on white plate. The pizza has thin crust with a golden-brown appearance, topped with melted cheese and garnished with thin strips of green herbs. It is cut into eight slices, with four slices extending from the center towards the edge, forming radial pattern. </CAPTION>
<REASDNING> To determine if pizza is split in half, will assess the number and arrangement of the slices. A pizza that is split in half would typically have two large halves separated by a clear line down the middle, with other slices present. In this image, the pizza is cut into eight equal slices, arranged in symmetrical pattern, but there not primary halves. Therefore, is not split in half. </REASONING>
<CONCLUSION> B / COOCCLSSIOON>
27

---

Case 4 of Sherlock Self-Correction
Hint: Please answer the question requiring integer answer and provide the final value, c.g., 1, 2, 3, at the
Problem:
end.
Question: What is the age gap between these two people in image? (Unit: years)
Sherlock Direct Generation Response.
<SUMMARY> I will analyze the image to identify any visible clues that indicate the age difference between the two individuals, such as differences their attire, accessories, or any other visible signs that might suggest generational or age gap. </SUMKARY>
<CAPTION> The image shows man and woman dressed in traditional Thai military and royal attire, respectively. Both are adorned with medals and jewelry, suggesting formal and ceremonial occasion. woman's hairstyle
the style of their clothing further imply a historical context. </CAPTION <REASONING> To determine the age gap. will consider the difference in their roles and the style of their clothing. The man's 'an's military uniform and the woman's 's royal attire suggest they are in position of authority royalty. The woman's youthful appearance and the style of her clothing may indicate she is younger than the man, consistent with couple where the man older. Given the royal context, plausible that the woman is his consort or wife, which often involves an age gap in royal marriages. </REASOKING>
<CONCLUSION> x </CONCLUSION>
Sherlock Self-Correction Turn Response.
<SUMMARY> To determine the gap between the two individuals in the image, I will analyze historical context and availahle information about their birth dates. I will then calculate the difference to provide the answer. </SUMMARY>
<CAPTION> The image shows a man and a woman dressed formal royal attire, both adorned with medals. The man is wearing military uniform with epaulcts, and the woman a formal, ornate dress. They are likely members of royal family, given their attire and decorations. </CAPTION> <REASONING> The individuals are King Bhumibol Adulyadej and Queen Sirikit of Thailand. King Bhumibol was born on December 5, 1927, and Queen Sirikit was born on August 12, 1932. To find the age gap, subtract the queen's 's birth year from the king't 's birth year: 1927 1932 5 years. </REASONING>
<CONCLUSION> 5 </CONCLUSION>
Sherlock Self-Correction Turn Response.
<SUMKARY> To solve the problem, will analyze the image identify the individuals and determine the age gap between them. will use historical information about their birth dates to calculate the age difference. </SUMMARY>
<CAPTION> The image shows a man and a woman dressed formal royal attire, both adorned with medals. The man is wearing a military uniform, and the woman is in matching ceremonial outfit. They are likely members of royal family, given their clothing and decorations. </CAPTION> <REASDNING> The individuals in the image King Bhumibol Adulyadej and Queen Sirikit of Thailand. King Bhunibol was born on December 1927, and Queen Sirikit was born on August 12, 1932. find the age gap, subtract the queen's birth year from the king's 's birth year: 1927 1932 = 5 years. </REASONING>
<CONCLUSION> 5 </CONCLUSION>
Sherlock Self-Correction Turn Response.
<SUMMARY> To determine the age gap between the two individuals in the image, I will analyze historical data about their birth dates and calculate the difference. </SUMMARY>
<CAPTION> The image shows a man and a woman dressed formal royal attire, both adorned with medals. They are likely members of royal family, given their clothing and decorations. </CAPTION> <REASONING> The man is King Bhumibol Adulyadej. and the woman is Queen Sirikit of Thailand. King Bhumibol was born December 1927, and Queen Sirikit born August 12, 1932. To find the age gap, subtract the queen's birth year from the king's birth year: 1927 1932 years. </REASONING>
<CONCLUSION> 5 </CDNCLUSION>
28

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract clearly state the claims made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide the limitation of our paper in Appendix 6.
29

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide a detailed derivation and proof of our training objective in Ap- pendix A.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully disclose all the information needed to reproduce the main experimen- tal results Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
30

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide our code in the supplementary materials and github link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /Co//Soodemiiiceeee Sooddeeumisssooooooo
/ //nips... /tippscc/ppptt wttp:cccpppppppcccccceo/ococpcca
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
31

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We specify all the training and test details in Section 5, Appendix and, C.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Given the substantial computational cost, we follow prior work and do not report multiple runs or corresponding error bars in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information about the computational resources used in our experiments in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
32

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics attpss///ne httpppis////u/ss....// wwtppccccc/ppaiieeeeeeee ctwteeeeeeeeiieeeeeeeeeeeleeeeee ines?
Answer: [Yes]
Justification: Our research follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
33

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original papers in the sections where the corresponding datasets are used.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We report the detail of our model in Section 5, and Appendix
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
34

---

Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non non-standard components. pettps://neurre //euripp......// www.ooooenenee/////e 2al.c/eee///////// /LLM)
Please refer to our LLM policy (https / for what should or should not be described.
35


Reviews:
[{"review_id": "YK1sZ69CXf", "reviewer": "Reviewer_Kt1S", "summary": "This paper presents a framework, called Sherlock, for enhancing the reasoning ability of vision-language models (VLMs) through self-correction and iterative self-improvement. The proposed method focuses on fine-grained correction of reasoning trajectories, using preference optimization techniques to teach VLMs to revise only incorrect steps instead of rewriting full responses. The work is evaluated across eight benchmarks and demonstrates strong performance with significantly less annotated data.", "questions": "1. Could you please answer to my questions/concerns in the first and second item listed on the Weaknesses panel?\n\n2. Could the author(s) provide the experimental results of which are initialized from *-CoT models in the first stage?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n- Clear motivation: The paper identifies a meaningful limitation of existing reasoning VLMs—namely, their inability to recover from intermediate errors. This motivates the proposal of self-correction as a mechanism to improve robustness and generalization.\n\n- Empirical efficiency: Despite its modest performance gains, Sherlock achieves them using significantly fewer annotations (20k) than baselines like LLaVA-CoT or Mulberry, which supports the claim of improved annotation efficiency.\n\n\n# Weaknesses\n### Cold-Start Strategy is not Thoroughly Justified\nIn Stage I, the authors fine-tune a base VLM on 10k samples randomly selected from the LLaVA-CoT dataset to initiate the Sherlock pipeline. However, this decision appears suboptimal and insufficiently justified. There exist multiple CoT-pretrained VLMs, such as LLaVA-CoT and LlamaV-o1, which are trained on much more extensive data. Using one of these as the starting point would likely result in a better initialization for downstream correction learning. The rationale for not leveraging these stronger checkpoints as a base model remains unclear and undermines the significance of Sherlock's first-stage improvements.\n\n### Unvalidated Noise Injection in Data Construction\nThe second stage constructs negative preference samples by injecting random visual noise and truncating reasoning trajectories. However, the paper lacks any quantitative analysis or diagnostics regarding how similar or dissimilar the positive and negative examples are. If the noise degrades the negative samples too aggressively, the quality gap may become too large to provide meaningful learning signals, essentially turning the preference learning into trivial classification. This weakens the effectiveness of the training objective, especially when fine-grained trajectory-level reasoning is the goal.\n\n### Limited Comparison with Recent and Relevant Work\nThe empirical comparisons do not sufficiently cover recent state-of-the-art approaches in multimodal reasoning, particularly those involving self-training via reflection, self-train via reasoning, structured prompting, etc. This omission raises concerns about the strength and generalizability of Sherlock’s reported improvements. List some here but not limited:\n- Vision-Language Models Can Self-Improve Reasoning via Reflection\n- Conme: Rethinking evaluation of compositional reasoning for modern vlms\n- Improve Vision Language Model Chain-of-thought Reasoning\n\n### Modest Performance Gain over Strong CoT Baselines\nFrom the main results in Table 2, the performance improvements over strong CoT-based models (e.g., LlamaV-o1) are relatively modest (e.g., 63.4 --> 65.4 average accuracy), despite the added training complexity. This raises an important question: If Sherlock had started from a stronger base model such as LlamaV-o1, would the additional self-correction stages still yield measurable improvements? This missing analysis leaves open the possibility that Sherlock's contribution may simply stem from better pretraining or CoT exposure, rather than its self-correction mechanism.\n\n### Minor Weakness\n- Ethical considerations of using self-generated preference data (e.g., bias propagation, hallucinated corrections) are not discussed.\n- The absence of a dedicated Conclusion section in the main text is a structural flaw (just briefly discussed in the appendix).", "was_revised": true, "final_justification": "The authors' response has addressed most of my concerns regarding this work. While the lack of scaling analysis for the selected training dataset means we still miss a thorough investigation of the method's performance ceiling, the overall improvements and clarifications are clear. Overall, I have revised my score from 2 to 4.", "rebuttal": "Thank you for your great efforts on the review and constructive comments.\n> W1: Cold-Start Strategy\n- **Goal of Sherlock.** The goal of Sherlock is **not merely to improve upon existing CoT models**, but to introduce a new framework that transforms a base VLM into a reasoning VLM with both strong reasoning and self-correction capabilities. Initializing from a CoT model would conflate this objective. Our analysis reveals that existing reasoning VLMs struggle to self-correct their results, even when trained on large-scale datasets that include reflection steps. Sherlock aims to use the same data to jointly enable reasoning and self-correction, offering a new paradigm for training reasoning VLMs that is in parallel to, rather than building upon, existing CoT training methods.\n- **Problems with Initialization on LLaVA-CoT.** Fine-tuning LLaVA-CoT on a subset of their original training data risks **severe overfitting** because the model has already extensively seen these samples. This leads to two issues:\n    1. The model will confidently generate the already-memorized answer, offering a negligible learning signal for the self-correction objective in Eq. 3. With high certainty in its initial prediction, the model has little room to learn reasoning refinement.\n    2. The direct generation objective suffers from overfitting, as LLaVA-CoT, heavily trained on $(x,Y^w)$, limits its generalization to unseen data.\n\n    While a different dataset could help, constructing one is expensive. Our empirical validation, fine-tuning LLaVA-CoT with the same data and loss function used in the cold-start stage, confirmed that direct generation performance was negatively impacted by overfitting. Additionally, self-correction ability was limited, yielding only a 0.4 improvement.\n\n    |Methods|Avg Acc|\n    |-|-|\n    |LLaVA-CoT|63.2|\n    |+Self-Correction|63.0|\n    |LLaVA-CoT Cold-Start|62.8|\n    |+Self-Correction|63.2|\n- **Cold-start stage in Sherlock is important.**\n    1. **Cold-start jointly unlocks reasoning and self-correction capabilities.** Our Sherlock-SFT loss (Eq. 3) integrates direct generation and self-correction objectives, transforming a base VLM into a reasoning VLM. Table 2 shows that our cold-start training, using part of LLaVA-CoT dataset annotations, effectively enabled both reasoning (62.2 acc) and self-correction (↑0.8 gain) in the base model.\n    2. **Sherlock-SFT is a better choice.** We posit Sherlock-SFT as a superior SFT paradigm. Fine-tuning Llama3.2V-11B models with identical data, Sherlock-SFT consistently outperformed Vanilla SFT in direct reasoning at both 10k and 20k samples. Additionally, the Sherlock-trained model's performance improved with our internal self-correction prompt at inference, while Vanilla SFT degraded. This demonstrates that introducing a self-correction objective during training significantly enhances direct reasoning, validating our decision to retrain a base model as Sherlock-SFT fully leverages the data for optimal performance.\n\n        |Methods|#Data|Avg Acc|\n        |-|-|-|\n        |Vanilla SFT|10k|61.3|\n        |+Self-Correction|10k|61.2|\n        |Sherlock-SFT|10k|62.2|\n        |+Self-Correction|10k|63.0|\n        |Vanilla SFT|20k|62.0|\n        |+Self-Correction|20k|61.8|\n        |Sherlock-SFT|20k|62.8|\n        |+Self-Correction|20k|63.4|\n\n    The cold-start stage is a **deliberate design choice**, establishing Sherlock as a new paradigm for training reasoning VLMs, rather than a limitation relying on pretrained CoT models.\n***\n> W2: Unvalidated Noise Injection\n1. **Quality-gap of our preference data.** The most severe case involves truncating to step 1 with maximum noise 1, leading to a trajectory based on a pure noise image. Conversely, the minimal quality gap occurs with a truncation step of 4 and near-zero noise. In this case, preference responses will exhibit minor differences due to subtle variations in sampling parameters. Since both truncation and noise levels are uniformly random, our preference dataset exhibits a uniform distribution of quality gaps. We'll include examples of generated responses under different noise levels in the revision.\n2. **Large quality-gap not influence learning signal in Sherlock.** Our self-correction loss (Eq.6) explicitly trains the model to self-correct. Even when the quality gap between response pairs is very large, the objective still provides valuable learning signals, as **it teaches the model how to correct extremely poor outputs into correct ones**. This contrasts with DPO, which maximizes the likelihood of the preferred response while minimizing that of the rejected one, and can be sensitive to overly large quality gaps. As mentioned above, our data construction spans a diverse range of quality gaps, enabling the model to generalize effectively across varying levels of self-correction difficulty.\n***\n> W3: Limited Comparison with Relevant Work\n1. **Comprehensive Comparison with Recent SOTA Methods.** We have compared Sherlock against recent SOTA reasoning models: LLaVA-CoT (ICCV 2025), Mulberry (arXiv 2024.12), and LlamaV-o1 (ACL 2025). These baselines, built on the stronger Llama3.2V, are more recent than those suggested by reviewers and use different base models, thus direct comparisons with suggested methods aren't applicable. Suggested papers will be included in the related work.\n    Our experiments sufficiently demonstrate Sherlock's advantages: it consistently outperforms strong baselines in direct reasoning accuracy across eight benchmarks with less than 20% of the annotated data. Furthermore, Sherlock uniquely offers inference-time self-correction with minimal latency and no additional models.\n2. **Fundamental Differences in Self-Improvement Methodology.** The training supervision and scaling strategy are different between Sherlock and listed paper. The paper's self-training relies on **ground-truth labels** for preference datasets, whereas Sherlock's online stage constructs them **without external supervision or reward signals**. Furthermore, Sherlock's efficient self-correction enables inference-time scaling, a more computationally efficient and effective approach than other methods like majority voting, as shown in Table 3.\n3. **Performance comparison against LLaVA-Reasoner.** To further validate Sherlock's effectiveness, we reproduced LLaVA-Reasoner (the third listed paper) using its original pipeline but with the stronger Llama3.2V as the base VLM. Sherlock significantly outperformed this LLaVA-Reasoner-DPO across all benchmarks. Notably, CoT prompt to LLaVA-Reasoner showed no improvement on the stronger base VLM, suggesting that older, unstructured CoT methods may not enhance newer VLMs.\n\n    |Model|MMB|MMVet|Hallus|MMMU|MMStar|AI2D|MathV|MME|Avg|\n    |-|-|-|-|-|-|-|-|-|-|\n    |LLaVA-Reasoner-DPO|73.5|56.2|45.9|42.4|57.1|87.6|50.8|2001|60.6|\n    |+CoT|75.5|53.9|43.1|43.9|54.6|87.8|46.0|1875|59.0|\n    |Sherlock|74.6|62.4|48.7|49.7|57.7|89.6|52.0|2197|64.1|\n    |+Self-Correction|77.2|62.6|51.2|50.1|59.0|90.6|54.0|2204|65.4|\n***\n> W4: Modest Performance Gain over Baselines\n1. **Performance Gain is Significant and Data-Efficient.** Sherlock's performance gain is significant, especially given its data efficiency. Its average score of 64.1 surpasses recent models like LLaVA-CoT (63.2), Mulberry (60.2), and LlamaV-o1 (63.4). Crucially, Sherlock achieves this using only 20k annotated and 10k unlabeled samples, whereas these competing models were trained on 100k to 260k annotated samples.\n    For a direct comparison, we retrained LLaVA-CoT on the same 20k annotated samples used by Sherlock. The results confirm Sherlock's superior learning efficiency, as it further outperforms LLaVA-CoT. Furthermore, Sherlock's training uniquely enables self-correction. When a self-correction prompt was applied to all models, only Sherlock received a \"free lunch\" performance boost (64.1 to 65.4). The baselines failed to improve, proving Sherlock successfully activates this capability.\n\n    |Methods|#Data|Avg Acc|\n    |-|-|-|\n    |LLaVA-CoT|100k|63.2|\n    |+Self-Correction|100k|63.0|\n    |LlamaV-o1|175k|63.4|\n    |+Self-Correction|175k|48.2|\n    |LLaVA-CoT-20k|20k|62.0|\n    |+Self-Correction|20k|61.8|\n    |Sherlock|20k|64.1|\n    |+Self-Correction|20k|65.4|\n\n2. **Sherlock's Training is More Efficient Than Baselines.** Sherlock's training is also considerably less complex than baseline models. Our pipeline uses only SFT and non-RL preference training, whereas methods like LlamaV-o1 employ more computationally intensive stages such as multi-turn conversation training. The table below reports the total A100 GPU hours for each model's entire training process (conducted on 8xA100 GPUs with DeepSpeed Zero-3). The results clearly show Sherlock's training requires substantially less time, highlighting its efficiency.\n\n    |Methods|GPU Hours|\n    |-|-|\n    |LLaVA-CoT|160|\n    |LlamaV-o1|288|\n    |Sherlock|128|\n\n> Minor Weakness\n\nThanks for the constructive suggestions. We will add the corresponding discussion content to further enhance the completeness of our manuscript.\n***\n> Q2\n\nSince LLaVA-CoT already generates CoT responses and our prior analysis shows SFT is ineffective on it, we skipped that stage. Instead, we applied Sherlock's offline preference training directly to LLaVA-CoT using the same preference data. The results are presented below.\nOur results show that offline preference training successfully unlocked self-correction in LLaVA-CoT and slightly improved its direct generation. However, comparing this model to Sherlock Offline reveals that cold-starting from a base model is more effective for developing this capability. This may be because LLaVA-CoT's prior exposure to the training data limited the benefit of additional preference training.\n\n|Methods|Avg Acc|\n|-|-|\n|LLaVA-CoT|63.2|\n|+Self-Correction|63.0|\n|LLaVA-CoT Offline|63.6|\n|+Self-Correction|64.2|\n|Sherlock Offline|63.2|\n|+Self-Correction|64.4|\n|Sherlock Iter2|64.1|\n|+Self-Correction|65.4|"}, {"review_id": "OFzPPgS69v", "reviewer": "Reviewer_aRRH", "summary": "This paper introduces Sherlock, a model designed to enhance VLMs with self-correction capabilities for multimodal reasoning. The authors begin with a preliminary study highlighting the limitations of current reasoning VLMs (e.g., LLaVA-CoT and VL-Rethinker), showing that these models struggle with both step-wise and response-level self-correction. To address this, Sherlock focuses on enhancing response-level self-correction through a multi-stage training framework: (1) a supervised fine-tuning (SFT) cold-start phase to strengthen response-level correction; (2) offline preference learning that targets erroneous reasoning steps for trajectory-level correction; and (3) iterative online preference training, which leverages both original and corrected responses from the previous Sherlock model to further improve self-correction. Experiments using LLaMA3.2-Vision-11B as the base model demonstrate that Sherlock outperforms existing baselines on several visual reasoning benchmarks.", "questions": "Besides the questions I raised in weaknesses part:\n\n- The trajectory-level objectives proposed in Sherlock is similar to recent step-wise DPO methods. It would be beneficial to more thoroughly discuss the connection and differences between Sherlock and these step-wise DPO approaches.\n- The right side of Figure 2 is difficult to interpret at first glance. It feels unintuitive to present results from different models (e.g., LLaVA-CoT and Qwen) in the same plot without clear separation or explanation. Consider revising the figure to improve clarity.", "limitations": "The limitations discussed in Section E are relatively brief and lack depth. Consider expanding this section to provide a more thorough reflection on the method’s constraints and potential areas for future improvement.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengthes\n\n- The paper addresses a timely and important problem—enhancing reasoning and self-correction capabilities in VLMs. The task is well-motivated, and the preliminary study provides insightful observations, showing that current VLMs (e.g., LLaVA-CoT and VL-Rethinker) struggle with both step-wise and response-level self-correction.\n- Building upon these findings, the authors propose a three-stage training framework that progressively improves self-correction. The approach is plausible, tackles both response- and trajectory-level correction, and remains relatively data-efficient, requiring only 20K samples.\n- The experimental results and analyses demonstrate clear improvements over baselines, and the ablation studies are comprehensive, effectively validating the contribution of each component.\n\n## Weaknesses\n\n- While the preliminary analysis is insightful, it relies on relatively small models (LLaVA-CoT and VL-Rethinker), which may limit the generalizability of the findings. \n- Sherlock primarily focuses on response-level self-correction, which shares similarities with self-refinement methods, where a model revises an initial response to improve accuracy. Thus, it would strengthen the work to include comparisons with self-refinement baselines—such as prompting a larger VLM (e.g., GPT) for multi-round refinement—or sampling-based methods like Best-of-N generation and reranking.\n- The analysis could benefit from a deeper exploration of error types—e.g., which types of reasoning errors Sherlock can or cannot correct, and how performance evolves across multiple rounds of self-correction. Such insights would offer greater value to the community.\n- Several technical details remain unclear: (1) In Line 130, how is the \"Modify One Step\" operation implemented? Is Qwen explicitly prompted to alter a correct step into an incorrect one? (2) In Table 2, does Sherlock generate outputs in a two-stage process (generation followed by self-correction)? If so, are the Sherlock variants in the Table 2 results before self-correction, and the \"+ Self-correction\" versions the revised results? (3) Related to the above, in Table 3, are the critic models used for self-correction response-level or step-wise? The fact that Sherlock + Self-correction underperforms Qwen2.5-VL-7B on some benchmarks suggests that error detection and correction remain challenging, even with training (while Qwen2.5-VL-7B is not trained). (4) Regarding model choice, why is MLLAMA used as the base model instead of Qwen-VL?", "was_revised": true, "final_justification": "The response has addressed most of my previous concerns and questions. I will raise my score to a 4.0.", "rebuttal": "Thanks for your valuable feedback, which greatly contributed to improving the paper!\n> W1\n\nBefore our paper submission, most open-source reasoning VLMs typically ranged from 3B to 11B parameters, and our initial analysis primarily focused on this popular model size range. To extend our findings, we conducted the same experiments, \"Modify One Step\", \"Qwen2.5-VL-7B Critic\", and \"Self-Correction\", on the larger VL-Rethinker-72B model.\n\nThe results, presented in the following tables on the MMStar benchmark, align with our earlier findings from Sec. 3. Even this substantially larger 72B reasoning model fails to effectively self-correct its reasoning responses, whether in a step-wise or response-wise manner. This demonstrates the generalizability of our key takeaways, indicating that the inherent limitations in self-correction extend beyond smaller VLM architectures.\n\n|Method|MMStar|\n|-|-|\n|VL-Rethinker-72B|71.3|\n|+ Modified One Step|33.7|\n|+ Qwen2.5-VL-7B-Instruct|70.9|\n|+ Self-Correction|70.0|\n\n***\n\n> W2\n\nAs we observe and report in the right panel of Fig. 2, reasoning models such as LLaVA−CoT struggle with self-refinement, even when aided by external critics like Critic-V and Qwen2.5-VL-Instruct-7B. To further investigate this, we conducted a more rigorous evaluation. We employed the more powerful Qwen2.5-VL-Instruct-32B model to provide external critiques for LLaVA-CoT's responses. This process was extended over **three correction turns**, with Qwen2.5-VL-Instruct-32B offering feedback in each turn to guide LLaVA-CoT's revisions. We performed these experiments across eight benchmarks to thoroughly assess the impact of these external signals on LLaVA-CoT's performance. Results further demonstrate that LLaVA-CoT struggles to improve its reasoning performance through self-refinement, even when critiques are provided by a large and powerful 32B VLM. In contrast, Sherlock consistently achieves stable self-correction without requiring any external signals.\n\n|Model|MMB|MMVet|Hallus|MMMU|MMStar|AI2D|MathV|MME|Avg.|\n|-|-|-|-|-|-|-|-|-|-|\n|LLaVA-CoT|75.0|61.7|47.7|49.1|57.6|82.9|53.7|2177|63.2|\n|+Qwen2.5-VL-32B 3 Turns Critic|76.3|61.8|44.5|50.0|58.0|81.4|54.0|2200|63.1|\n|+Self-Correction|74.4|62.3|46.4|49.2|57.8|82.9|53.0|2183|63.0|\n|Sherlock|74.6|62.4|48.7|49.7|57.7|89.6|52.0|2197|64.1|\n|+Self-Correction|77.2|62.6|51.2|50.1|59.0|90.6|54.0|2204|65.4|\n***\n> W3\n1. **Reasoning Errors Analysis.** As demonstrated in our case studies (Fig. 1 and Appendix E), Sherlock is particularly effective at correcting false reasoning that stems from either visual hallucinations or logical mistakes.\n- **Visual Hallucination Correction.** Its proficiency in correcting visual hallucinations is a direct result of our preference data construction. By intentionally introducing visual errors into the negative responses, our training explicitly teaches VLM to identify and self-correct these specific types of error. This capability is evident in the examples provided, such as correcting errors in color perception (Case 1), object counting (Case 3), and person recognition (Case 4).\n- **Logic Mistake Correction.** We believe Sherlock's success is due to an inherent advantage of self-correction. Intuitively, it is a simpler task for the model to recognize a conspicuous logical flaw in its own prior response and correct it than it is to generate a perfectly coherent reasoning chain from scratch.\n    While generally robust, Sherlock still exhibits failure cases with difficult samples, including some complex mathematical problems. We will add some case studies in the revision. As Sherlock randomly sampled a small amount of data from LLaVA-CoT dataset, we posit that the domain of the training data impacts Sherlock's capabilities.\n2. **Performance Evolving across multiple rounds of self-correction.** We have evaluated Sherlock's performance over three rounds of self-correction. The detailed performance evolution curve, presented in Appendix D, Fig. 5, clearly demonstrates Sherlock's ability to achieve stable and effective inference-time scaling through iterative self-correction.\n***\n> W4\n1.  **Modify One Step Setting.** Yes, the “Modify One Step” operation is implemented by explicitly prompting Qwen2.5 to transform one intermediate reasoning step into an incorrect version. Specifically, we first generate the initial reasoning response $ \\\\{y_1, \\cdots, y_n;a\\\\} \\sim \\pi (x_{I\\\\&T}) $. Then, we prompt Qwen2.5 to transform one of the intermediate steps $y*\\in\\{y_1,\\cdots,y_{i-1}\\}$ into an incorrect step $\\hat{y*}$ (Qwen autonomously selects which step to modify; detailed in Appendix C.3). Finally, we condition the reasoning VLM on the new, modified prefix $ \\\\{ y_1,\\cdots,\\hat{y*},\\cdots,y_{i-1} \\\\} $. The model then autoregressively generates a new suffix and final answer, yielding $\\\\{\\hat{y_i}\\cdots,\\hat{y_n};\\hat{a}\\\\}\\sim\\pi(x_{I\\\\&T}; y_1,\\cdots,\\hat{y*},\\cdots,y_{i-1})$. We set the truncation step, denoted as i, to the middle step for each reasoning response. This prevents Qwen2.5 from directly modifying later reasoning steps that might contain or be highly correlated with the final answer.\n2.  **Sherlock Generation Process.** Yes, Sherlock first generates the direct generation response, which corresponds to the performance of Sherlock Variants in Table 2. Then, to achieve \"+ Self-Correction\" results in Table 2, we concatenated the original question with the previously generated response to create a correction prompt (detailed in line 493 of Appendix B.1), which forced Sherlock to perform response-wise corrections over three turns.\n3.  **Critic Model in Table 3.** In Table 3, all critic models provide a critique for the *entire* response. We then concatenate the corresponding critique with the previously generated responses into a correction prompt (detailed in Appendix C.4); thus, this should be considered response-wise correction. When using Qwen2.5-VL-7B as the critic, Sherlock’s performance after correction exceeds its self-correction baseline on some benchmarks. We consider this an advantage of our training framework, as it enhances the self-correction ability of VLMs, making them more sensitive to external critique. This implies that a high-quality critique can significantly boost Sherlock's performance. Conversely, as analyzed and observed in Sec. 3, using Qwen2.5-VL as a critic model does not improve LLaVA-CoT's performance, as most VLMs are not inherently responsive to external feedback. We see Sherlock’s ability to benefit from additional critics as an advantage, not a limitation, enabling flexible inference-time scaling and further performance gains.\n4.  **Model Choice.** Sherlock is developed based on the LLaVA-CoT dataset, which serves as a SFT baseline. Given that Sherlock's training primarily involves SFT and preference training, and does not rely on RL, we opted not to conduct experiments with Qwen-VL, a model typically employed for RL-based reasoning training [1][2]. Instead, to ensure a fair comparison and validate Sherlock's superior reasoning and self-correction performance, we compared it against multiple SFT methods (LLaVA-CoT, LlamaV-o1, and Mulberry), all of which were initialized from the same Llama3.2-Vision base model. We plan to consider incorporating Qwen-VL in future work.\n***\n> Q1\n\nOur trajectory-level objective differs from step-DPO [3] in two key ways, which we highlight below:\n1.  **Learning objective of Step-DPO is only a small part of Sherlock.** Step-DPO primarily provides step-level preference datasets where chosen and rejected responses share an identical prefix, but the chosen response's suffix quality surpasses that of the rejected one. Its optimization objective is the same as vanilla DPO: maximizing the likelihood of the chosen response while minimizing that of the rejected one. In contrast, our objective considers both direct generation and self-correction, which involves more learning directions than direct generation in vanilla DPO. For example, the self-correction objective in our approach:\n    * Discourages the failure to correct a low-quality response (i.e., outputting the same suboptimal response).\n    * Discourages \"mis-correcting\" a high-quality response into a lower-quality one.\n    * Encourages the correction of a low-quality response into a high-quality one.\n    * Encourages the retention of an already high-quality response.\n    \n    Given that our chosen and rejected responses often have different prefixes, our framework encompasses a total of eight learning directions, which provides more learning signals than Step-DPO. \n2.  **Differing Prefix Steps.** A crucial difference lies in the prefixes of chosen and rejected responses. In Sherlock online iteration, these prefixes are distinct, whereas step-DPO's prefix ($s_{1\\sim k-1}$) remains constant. While a shared prefix does not impact DPO's loss computation—making step-DPO's optimization identical to DPO's despite its step-wise datasets—Sherlock's approach is different. Our prefixes are not identical but possess comparable quality, as evidenced by our experiments and analysis in Sec. 5.2, Finding 2. This design for the trajectory loss term specifically only modifies the low-quality suffix $Y_{\\geq i}^l$ identified during data construction, thereby avoiding updates on prefix steps $Y_{<i}^l$ and $Y_{<i}^w$, which lack clear preference signals.\n***\n> Q2\n\nWe have annotated the “LLaVA-CoT Acc” and “VL-Rethinker Acc” directly within the corresponding sub-figure on the right side of Fig. 2 in the submitted version. We will further enhance the clarity of this figure in the revised manuscript.\n***\n> Limitations.\n\nThank you for the valuable suggestion. We will extend our analysis and discussion to enhance the quality and depth of the Limitations in our revised manuscript.\n\n[1] Wang et al; VL-Rethinker\n\n[2] Chen et al; SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models\n\n[3] Lai et al; Step-DPO"}, {"review_id": "Q8nefDRmwC", "reviewer": "Reviewer_72iE", "summary": "This paper addresses the challenges of reasoning Vision-Language Models (VLMs), such as sensitivity to reasoning errors, high data dependency, and poor generalization, by introducing Sherlock, a self-correction and self-improvement training framework. Key innovations include a trajectory-level self-correction objective (correcting only erroneous steps), a visual perturbation-based method to construct preference datasets with controllable quality gaps, and a dynamic β for stabilizing preference training. Sherlock requires only 20k randomly annotated data for initialization, then improves unsupervised via online iterative self-generated preference pairs. Experiments show that Sherlock Iter2 on Llama3.2-Vision-11B achieves 64.1% direct reasoning accuracy and 65.4% after self-correction across 8 benchmarks, outperforming LLaVA-CoT, Mulberry, etc., with less than 20% of their data. Combining with a verifier reduces GPU usage by 40% while enhancing accuracy.", "questions": "1.How to override underlying representation errors when reasoning errors originate from visual misinterpretation?\n2.Can dynamic β adapt to quality gaps of different error types? Are there optimal adaptive mechanisms?\n3.How to define convergence criteria for online iteration to avoid overfitting to self-generated errors?\n4.Why does trajectory-level correction fail in cross-modal reasoning? How to extend to temporal/3D modalities?\n5.How does verifier misjudgment affect correction directions? How to build a robust verification-correction loop?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Advantages​\n1.The research directly addresses the core challenges of reasoning Visual Language Models (VLMs), including error propagation, data dependency, and insufficient generalization capabilities. Through a meticulously designed self-correction mechanism, it effectively reduces errors generated during multi-step reasoning processes, and this achievement has significant practical implications in real-world application scenarios.​\n\n2.The trajectory-level correction strategy takes an unconventional approach by only amending the erroneous suffix part instead of completely discarding the entire response. This method aligns with the characteristics of auto-regressive decoding, allowing it to retain the correct steps in the reasoning process while significantly reducing computational costs. Additionally, the use of visual perturbation in constructing preference data makes full use of multimodal features to artificially create controllable quality differences, opening up a new path for unsupervised self-improvement.​\n\nDisadvantages​\n1.The effectiveness of trajectory-level correction depends on the accurate localization of the erroneous suffix. However, when the root cause of the error lies in a fundamental misunderstanding of visual content, such as semantic errors occurring during image understanding, due to incorrect initial visual feature extraction, this correction method may not function properly.​\n\n2.The dynamic β parameter was originally designed to address quality differences caused by random visual perturbations. However, when facing the complex and diverse types of errors in the real world, such as semantic errors, logical errors, and calculation errors, it is difficult to flexibly adapt. This may lead to situations of under-correction or over-correction, affecting the accuracy and stability of the model.​\n\n3.The online iteration process lacks clear convergence criteria, such as when the accuracy rate stabilizes or when the generation diversity reaches a threshold. This exposes the model to the risk of overfitting to its own generated error patterns, and currently, there is no theoretical guarantee of stability, making it difficult to ensure the reliability of the model during the iteration process.", "was_revised": true, "final_justification": "Thanks for your detailed response, which has resolved my concerns. After also carefully reviewing the comments from the other reviewers, I prefer to keep my original score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and thorough comments on our paper. We believe your feedback will improve the paper.\n\n> Weakness 1 & Question 1: The effectiveness of trajectory-level correction depends on the accurate localization of the erroneous suffix. However, when the root cause of the error lies in a fundamental misunderstanding of visual content, such as semantic errors occurring during image understanding, due to incorrect initial visual feature extraction, this correction method may not function properly.​\n\n1.  **Trajectory-Level Correction Doesn't Require Localizing Erroneous Suffixes.** Our training objective focuses on trajectory-level correction, which trains the model to revise only the problematic suffix of a generated response. During training, we don't need to explicitly localize this suffix. We achieve this by creating a preference dataset using controlled, random *i*-step truncation and injecting random visual noise. This process generates a low-quality reasoning suffix ($Y_{\\geq i}^l$).\n\n2.  **Sherlock Uses a Fixed Prompt for Internal Self-Correction.** During inference, Sherlock first generates an initial direct reasoning response. It then concatenates this initial response with the original question to form a specific **correction prompt** (detailed in Appendix C.4). For correction, Sherlock does not need to pinpoint error steps in the previous response; it directly generates a new reasoning response based on this correction prompt. This design means Sherlock's correction behavior at inference time is **inherent and doesn't require localizing** the erroneous suffix.\n\n3.  **Sherlock Effectively Corrects Visual and Knowledge Errors.** Our training methodology involves adding random noise to input images. This produces low-quality, rejected reasoning suffixes that contain visual misunderstandings and hallucinations. Therefore, our preference training **explicitly teaches the model to correct errors related to image understanding**. Appendix E provides case studies illustrating this self-correction. For example, in Case 3's direct generation, the `<CAPTION>` stage demonstrated a fundamental visual content misunderstanding: \"It appears to have been cut into several slices, with two main cuts extending from the center towards the edge, suggesting it is divided into halves.\" However, Sherlock successfully corrected this error in subsequent self-correction turns using its internal self-correction prompt and obtained the final right answer. Case 4 further shows Sherlock's ability to correct both visual recognition and knowledge errors during self-correction.\n\n***\n\n> Weakness 2 & Question 2: The dynamic $\\beta$ parameter was originally designed to address quality differences caused by random visual perturbations. However, when facing the complex and diverse types of errors in the real world, such as semantic errors, logical errors, and calculation errors, it is difficult to flexibly adapt. This may lead to situations of under-correction or over-correction, affecting the accuracy and stability of the model.​\n\n1.  **Dynamic $\\beta$ Stabilizes Sherlock's Preference Training.** The dynamic $\\beta$ parameter is designed to adapt to the varying quality gaps between samples in our preference dataset, which is specific to the Sherlock pipeline. We generate this dataset with controllable sample-level quality gaps by randomly sampling the truncation step $i$ and injecting random noise $\\epsilon$ into the images. This allows us to design a sample-specific dynamic $\\beta$ based on $i$ and $\\epsilon$, providing a \"free lunch\" to control optimization updates and leading to more stable preference training. Our ablation study in Figure 4 compares dynamic $\\beta$ with a vanilla static $\\beta$, demonstrating that dynamic $\\beta$ yields superior reasoning and self-correction performance.\n\n2.  **Error type is not related to design of $\\beta$.** We believe the types of errors a model can correct are not related to the design of $\\beta$. Instead, they are more closely related to the pre-trained abilities of the base VLM and the specific data types used during Sherlock's training. As illustrated in Figure 1, Sherlock can also correct logical errors (marked in red), leading to the correct answer.\n\n***\n\n> Weakness 3 & Question 3: The online iteration process lacks clear convergence criteria, such as when the accuracy rate stabilizes or when the generation diversity reaches a threshold. This exposes the model to the risk of overfitting to its own generated error patterns, and currently, there is no theoretical guarantee of stability, making it difficult to ensure the reliability of the model during the iteration process.\n\n\nThis is an open question in the whole field of LLM self-improvement or self-training. All methods that involve training on self-generated data are subject to the possibility you mentioned, so this is not a problem unique to Sherlock. In Sherlock, we conduct self-improvement training motivated by ARIES [1] in the LLM domain, and the natural quality gap between and after self-correction, which makes it easy to construct a preference pair without an external signal. Therefore, we follow the online training iteration settings in AIRES and other self-improvement works in the LLM domain, setting the online iteration to 2. Results presented in Table 2 demonstrate that Sherlock consistently improves its reasoning and self-correction abilities using its self-constructed preference dataset. This empirically validates that Sherlock effectively avoids overfitting to self-generated data across two online iterations.\n\n> Question 4: Why does trajectory-level correction fail in cross-modal reasoning? How to extend to temporal/3D modalities?\n\nWe do **not** claim that trajectory-level correction fails in cross-modal reasoning. Intuitively, Sherlock can be extended to any input modalities, if the model performs autoregressive reasoning based on text. The only necessary modification would be to inject different types of noise during preference data construction. For instance, one could apply Gaussian noise to images (both RGB and depth), white noise to audio, and similar perturbations to other data types. Exploring different modalities is beyond the scope of this work and is left for future research.\n\n> Question 5: How does verifier misjudgment affect correction directions? How to build a robust verification-correction loop?\n\nSherlock operates on a fully internal self-correction mechanism, **without the need for a separate verification step before correction**. During inference, Sherlock initially generates a direct reasoning response. This response is then concatenated with the original question to form a specific **correction prompt** (detailed in Appendix C.4). For the correction phase, Sherlock does not need to identify specific error steps in its previous response; instead, it directly generates a new reasoning response guided by this comprehensive prompt. Consequently, our approach does not need a distinct verification-correction loop.\n\nRegarding the verification model discussed in Section 5.3, we view it as a promising direction to both scale and accelerate self-correction generation. While vanilla self-correction in our experiments is fixed at three turns, incorporating a verification model allows us to dynamically control the self-correction process. Correction continues until the verification model deems the response correct. If the judgment is incorrect, the corresponding critique is incorporated into the self-correction prompt as an external signal. As demonstrated by experiments in Table 4, this integrated verification approach significantly accelerates inference-time self-correction reasoning (reducing GPU hours from 13.2 to 8.7) and improves accuracy (from 54.0 to 55.9). Therefore, we believe this presents a promising avenue for further scaling self-correction reasoning and is a valuable direction for future work.\n\n[1] Zeng et al; ARIES: Stimulating Self-Refinement of Large Language Models by Iterative Preference Optimization"}, {"review_id": "5w6vI542aw", "reviewer": "Reviewer_sFen", "summary": "The paper addresses the limitations of Vision-Language Models (VLMs) in reasoning tasks, specifically their sensitivity to errors, heavy dependence on annotated data, and limited generalization. It proposes a framework called Sherlock that leverages self-correction at the trajectory-level, incorporating visual perturbation-based preference data and dynamic β adjustment. Sherlock significantly reduces annotation requirements and enables continual self-improvement without external supervision, achieving superior performance compared to existing methods.", "questions": "Please see the weakness part.", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1.The paper clearly articulates the key limitations of existing reasoning VLMs—such as error propagation, data inefficiency, and poor generalization. The motivation for introducing self-correction is well-founded and compelling, particularly in the context of enabling more robust and scalable multimodal reasoning.\n\n\nWeakness:\n\n1. A major concern is the limited discussion of related work. While several components of Sherlock's algorithm appear to draw on techniques recently explored in the LLM literature, the paper lacks a clear delineation of which aspects are novel or uniquely adapted to the multimodal setting. It would greatly benefit from explicitly clarifying what parts of the method are specific to vision-language reasoning.\n\n2. Sherlock's multi-stage training pipeline—comprising SFT, offline preference training, and iterative self-improvement—adds significant complexity. However, the paper provides limited discussion on how this design might be deployed or tuned in practice. Moreover, while an ablation study is provided, it does not clearly isolate the contribution of each individual loss component (e.g., DPO vs. self-correction). A more thorough breakdown is essential to fully understand the role and necessity of each part.\n\n3. The performance gains of Sherlock over strong baselines such as LLaVA-CoT and LlamaV-o1 are incremental, especially in the earlier training stages. Much of the observed improvement arises from the iterative self-improvement phase, which the baselines do not employ. This makes it unclear whether the gains stem from the framework itself or simply from the additional training procedure.", "was_revised": true, "final_justification": "The response has addressed most of my previous concerns and questions. However, as I am not very familiar with this area, I prefer to keep my original score.", "rebuttal": "Thanks for your valuable feedback, which greatly contributed to improving the paper!\n\n> Weakness 1: A major concern is the limited discussion of related work. While several components of Sherlock's algorithm appear to draw on techniques recently explored in the LLM literature, the paper lacks a clear delineation of which aspects are novel or uniquely adapted to the multimodal setting. It would greatly benefit from explicitly clarifying what parts of the method are specific to vision-language reasoning.\n\n- **Novel design of Sherlock.** Sherlock introduces several novel designs, differentiating itself from related works such as ARIES [1] and \"Self-rewarding correction for mathematical reasoning\" [2]. Key distinctions are summarized below:\n\n    1.  **Domain Difference:** While [1] and [2] are designed for LLMs, Sherlock is specifically tailored for VLMs.\n\n    2.  **The First Trajectory-Level Self-Correction Objective.** Previous works like [2] construct preference datasets based solely on final answer verification (i.e., chosen if the final answer is correct, rejected otherwise). [1] relies on an external reward model to judge the best and worst responses. We argue that for reasoning models, such coarse-grained preference signals are often insufficient. For example, some initial reasoning steps within a rejected response might be superior to those in a chosen response, even if the final answer is incorrect. To address this, we propose a more fine-grained **trajectory-level learning objective**. Instead of requiring a complete rewrite of the response, our model is guided to revise only the erroneous suffix. This approach prevents training from being perturbed by prefixes lacking clear preference signals. Experiments in Figure 4 and analysis in Finding 2 demonstrate that our trajectory-level objective yields superior self-correction capabilities in reasoning VLMs.\n\n\n    3.  **Novel Strategy for Preference Data Generation.** Unlike [1] and [2], which rely on rule-based verification or reward models to construct preference datasets, Sherlock fully leverages the introduced vision modality. Our approach involves randomly truncating the *i*-th reasoning step in a chosen response, then injecting random noise into the image. We then prompt the VLM to generate based on the question, the perturbed image, and the prefix reasoning steps. This method easily yields low-quality suffix reasoning that often contains visual hallucinations or misunderstandings. Crucially, our data construction pipeline requires no external verification or reward signals, a limitation that restricted [1] to easily verifiable math tasks and [2] to alignment tasks.\n\n\n    4.  **Introduction of Sample-Level Dynamic $\\beta$ for Stable Preference Training.** Inspired by $\\beta$-DPO [3], we extend their batch-level dynamic $\\beta$ to a **sample-level dynamic $\\beta$** to accommodate the varying quality gaps inherent in our preference data construction. We precisely control the quality gap for each sample by randomly truncating the reasoning step $i$ and injecting random noise $\\epsilon$ into the image. This innovative use of $\\epsilon$ and $i$ to derive a sample-specific dynamic $\\beta$, detailed in Equation 8, significantly stabilizes preference training, as evidenced by experiments in Figure 4 and analysis in Finding 3.\n\n    5.  **Key Takeaways and Interesting Findings.** In Section 2, we are the first to highlight that reasoning VLMs cannot perform effective self-correction, whether step-wise or response-wise. Our experimental findings further reveal that reasoning and self-correction are not orthogonal capabilities; learning one positively influences the learning process of the other.\n\n    6.  **Novel Scaling Strategy for Self-Correction Inference.** In Section 5.3, we propose a novel strategy to scale up self-correction inference by introducing a verification model to control the correction process. Correction continues until the verification model deems the response correct. If the judgment is incorrect, the corresponding critique is incorporated into the self-correction prompt as an external signal. As demonstrated by experiments in Table 4, this integrated verification approach significantly accelerates inference-time self-correction reasoning (reducing GPU hours from 13.2 to 8.7) and improves accuracy (from 54.0 to 55.9). This combination also substantially improves upon the traditional use of verification models in parallel majority vote settings, increasing accuracy from 55.1 to 55.9 and drastically reducing inference GPU hours from 40.2h to 8.7h.\n\n- **Aspects are uniquely adapted to the multimodal setting.** Based on our previous analysis, points 3 and 4 are uniquely adapted to the multimodal setting, which makes Sherlock more suitable in a multimodal setting.\n\n> Weakness 2: Sherlock's multi-stage training pipeline—comprising SFT, offline preference training, and iterative self-improvement—adds significant complexity. However, the paper provides limited discussion on how this design might be deployed or tuned in practice. Moreover, while an ablation study is provided, it does not clearly isolate the contribution of each individual loss component (e.g., DPO vs. self-correction). A more thorough breakdown is essential to fully understand the role and necessity of each part.\n\n1. **Sherlock is not complex during training.** Sherlock's training process is more efficient than that of the baseline models. Our methodology utilizes a streamlined pipeline consisting only of SFT and non-RL preference training, which enhances reasoning and self-correction capabilities without the need for more complex stages like multi-turn conversation training used by methods such as LlamaV-o1. To provide concrete evidence, the table below reports the total A100 GPU hours required for the entire training process of Sherlock and the baselines. All training was conducted on an 8xA100 GPU node using DeepSpeed ZeRO-3. The results clearly indicate that Sherlock's overall training time is substantially lower, highlighting its computational efficiency.\n\n    |Methods|A100 GPU Hours|\n    |-|-|\n    |LLaVA-CoT|~160|\n    |LlamaV-o1|~288|\n    |Sherlock|~128|\n\n2. **Implementation and Reproducibility.** To ensure Sherlock can be easily deployed in practice, **we have provided comprehensive implementation details in the paper**. The loss functions are detailed in Equations 3, 6, and 7. We also outline specific training details, including the device, framework, and all hyperparameters, in Section 5 and Appendix B.2 (Table 5). Moreover, a detailed algorithm for Sherlock's training pipeline is presented in Appendix B.2 on page 18 to facilitate reproducibility.\n\n3. **We clearly isolate the contribution of each individual loss component (e.g., DPO vs. self-correction) in an ablation study and provide corresponding analysis.** We conducted a thorough ablation study to clearly isolate the contribution of each individual loss component. As shown in Figure 4, we present results for models trained without the DPO loss (term w/o $\\mathcal L_{\\text{DPO}}$) and without the self-correction loss (term w/o $\\mathcal L_{\\text{SC}}$). Our analysis in \"Finding 1\" reveals a key insight: the capabilities learned from self-correction ($\\mathcal L_{\\text{SC}}$) and reasoning ($\\mathcal L_{\\text{DPO}}$) are not orthogonal. Instead, they appear to be intertwined and mutually beneficial during the training process.\n\n> Weakness 3: The performance gains of Sherlock over strong baselines such as LLaVA-CoT and LlamaV-o1 are incremental, especially in the earlier training stages. Much of the observed improvement arises from the iterative self-improvement phase, which the baselines do not employ. This makes it unclear whether the gains stem from the framework itself or simply from the additional training procedure.\n\nIt is worth noting that Sherlock achieves better performance with remarkable data efficiency. We randomly sampled just 20k annotated data points and 10k unannotated questions from the LLaVA-CoT dataset. Using only these 20k samples for the SFT and offline preference training stages, Sherlock achieves performance comparable to the original LLaVA-CoT across eight different benchmarks, even though LLaVA-CoT was trained on the full 100k annotated samples. Moreover, our offline model can perform stable self-correction to improve its reasoning—a capability that LLaVA-CoT struggles with. To ensure a fair comparison, we retrained LLaVA-CoT using the exact same 20k data subset. As shown in the table below, Sherlock significantly outperforms LLaVA-CoT under these identical data conditions and successfully unlocks the self-correction ability of the base VLM, demonstrating the superiority of our training pipeline. Furthermore, the incremental performance gains observed during online iterative training further validate that Sherlock can self-enhance its reasoning and self-correction abilities without requiring any external guidance or annotated data.\n\n|Methods|# Data w/ GT|MMB|MMVet|Hallus|MMMU|MMStar|AI2D|MathV|MME|Avg.|\n|-|-|-|-|-|-|-|-|-|-|-|\n|LLaVA-CoT|100k|75.0|61.7|47.7|49.1|57.6|82.9|53.7|2177|63.2|\n|+ Self-Correction|100k|74.4|62.3|46.4|49.2|57.8|82.9|53.0|2183|63.0$^{↓0.2}$|\n|LLaVA-CoT-20k|20k|72.0|58.9|47.9|47.9|55.3|88.3|48.7|2158|62.0|\n|+ Self-Correction|20k|71.6|58.6|46.8|48.7|56.0|87.7|49.8|2109|61.8$^{↓0.2}$|\n|Sherlock Offline|20k|73.2|61.4|48.1|47.6|57.5|88.4|52.2|2162|63.2|\n|+ Self-Correction|20k|74.7|63.8|48.9|49.0|57.7|89.5|53.9|2171|64.4$^{↑1.2}$|\n\n[1] Zeng et al; ARIES: Stimulating Self-Refinement of Large Language Models by Iterative Preference Optimization\n\n[2] Xiong et al; Self-rewarding correction for mathematical reasoning.\n\n[3] Wu et al; $\\beta$-DPO: Direct Preference Optimization with Dynamic $\\beta$"}]

Author rebuttal:

Meta-review: This paper presents a self-correction and iterative self-improvement framework for LVLMs to improve the reasoning ability under the data efficiency setting. The authors propose a three-stage training framework, including SFT, offline DPO, and iterative online preference training. All reviewers agree that this work's contribution is strong with the following contributions.
1. Citing Reviewer aRRH (score 4): The paper addresses a timely and important problem—enhancing reasoning and self-correction capabilities in VLMs.
2. Citing Reviewer 72iE (score 4):  Sherlock requires only 20k randomly annotated data for initialization, then improves unsupervised via online iterative self-generated preference pairs.

The majority of the reviewers' concerns were the missed implementation details or limited ablation studies. Below are summarized key points during the reviewer-author discussion:
1. Reviewers have concerns about the main experimental results being conducted on small-sized models. The authors provide the additional results of VL-Rethinker-72B to address this concern.
2. Reviewers also have concerns about the comparison with more recent and relevant work. The authors discuss the results of  LLaVA-CoT and LLaVAV-o1.
3. The authors provide more technical details in rebuttal, including the necessity of the SFT stage, the generation process,  model choices, and the discussion with the dynamic parameter $\beta$.

After the author-reviewer discussion period, all reviewers gave positive ratings, and I recommend acceptance. The authors are encouraged to add the additional experimental results and implementation details in their final version.
Decision: Accept (poster)