Title: Sherlock: Self-Correcting Reasoning in Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision-Language Models

Self-Correction

Reasoning

Chain of Thought
Abstract: Reasoning Vision-Language Models (VLMs) have shown promising performance on complex multimodal tasks. However, they still face significant challenges: they are highly sensitive to reasoning errors, require large volumes of annotated data or accurate verifiers, and struggle to generalize beyond specific domains. To address these limitations, we explore self-correction as a strategy to enhance reasoning VLMs. We first conduct an in-depth analysis of reasoning VLMs’ self-correction abilities and identify key gaps. Based on our findings, we introduce \emph{Sherlock}, a self-correction and self-improvement training framework. \emph{Sherlock} introduces a trajectory-level self-correction objective, a preference data construction method based on visual perturbation, and a dynamic $\beta$ for preference tuning. Once the model acquires self-correction capabilities using only 20k randomly sampled annotated data, it continues to self-improve without external supervision. Built on the Llama3.2-Vision-11B model, \emph{Sherlock} achieves remarkable results across eight benchmarks, reaching an average accuracy of 64.1 with direct generation and 65.4 after self-correction. It outperforms LLaVA-CoT (63.2), Mulberry (63.9), and LlamaV-o1 (63.4) while using less than 20\% of the annotated data.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the limitations of Vision-Language Models (VLMs) in reasoning tasks, specifically their sensitivity to errors, heavy dependence on annotated data, and limited generalization. It proposes a framework called Sherlock that leverages self-correction at the trajectory-level, incorporating visual perturbation-based preference data and dynamic β adjustment. Sherlock significantly reduces annotation requirements and enables continual self-improvement without external supervision, achieving superior performance compared to existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1.The paper clearly articulates the key limitations of existing reasoning VLMs—such as error propagation, data inefficiency, and poor generalization. The motivation for introducing self-correction is well-founded and compelling, particularly in the context of enabling more robust and scalable multimodal reasoning.\n\n\nWeakness:\n\n1. A major concern is the limited discussion of related work. While several components of Sherlock's algorithm appear to draw on techniques recently explored in the LLM literature, the paper lacks a clear delineation of which aspects are novel or uniquely adapted to the multimodal setting. It would greatly benefit from explicitly clarifying what parts of the method are specific to vision-language reasoning.\n\n2. Sherlock's multi-stage training pipeline—comprising SFT, offline preference training, and iterative self-improvement—adds significant complexity. However, the paper provides limited discussion on how this design might be deployed or tuned in practice. Moreover, while an ablation study is provided, it does not clearly isolate the contribution of each individual loss component (e.g., DPO vs. self-correction). A more thorough breakdown is essential to fully understand the role and necessity of each part.\n\n3. The performance gains of Sherlock over strong baselines such as LLaVA-CoT and LlamaV-o1 are incremental, especially in the earlier training stages. Much of the observed improvement arises from the iterative self-improvement phase, which the baselines do not employ. This makes it unclear whether the gains stem from the framework itself or simply from the additional training procedure.\n\nQUESTIONS:\nPlease see the weakness part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenges of reasoning Vision-Language Models (VLMs), such as sensitivity to reasoning errors, high data dependency, and poor generalization, by introducing Sherlock, a self-correction and self-improvement training framework. Key innovations include a trajectory-level self-correction objective (correcting only erroneous steps), a visual perturbation-based method to construct preference datasets with controllable quality gaps, and a dynamic β for stabilizing preference training. Sherlock requires only 20k randomly annotated data for initialization, then improves unsupervised via online iterative self-generated preference pairs. Experiments show that Sherlock Iter2 on Llama3.2-Vision-11B achieves 64.1% direct reasoning accuracy and 65.4% after self-correction across 8 benchmarks, outperforming LLaVA-CoT, Mulberry, etc., with less than 20% of their data. Combining with a verifier reduces GPU usage by 40% while enhancing accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nAdvantages​\n1.The research directly addresses the core challenges of reasoning Visual Language Models (VLMs), including error propagation, data dependency, and insufficient generalization capabilities. Through a meticulously designed self-correction mechanism, it effectively reduces errors generated during multi-step reasoning processes, and this achievement has significant practical implications in real-world application scenarios.​\n\n2.The trajectory-level correction strategy takes an unconventional approach by only amending the erroneous suffix part instead of completely discarding the entire response. This method aligns with the characteristics of auto-regressive decoding, allowing it to retain the correct steps in the reasoning process while significantly reducing computational costs. Additionally, the use of visual perturbation in constructing preference data makes full use of multimodal features to artificially create controllable quality differences, opening up a new path for unsupervised self-improvement.​\n\nDisadvantages​\n1.The effectiveness of trajectory-level correction depends on the accurate localization of the erroneous suffix. However, when the root cause of the error lies in a fundamental misunderstanding of visual content, such as semantic errors occurring during image understanding, due to incorrect initial visual feature extraction, this correction method may not function properly.​\n\n2.The dynamic β parameter was originally designed to address quality differences caused by random visual perturbations. However, when facing the complex and diverse types of errors in the real world, such as semantic errors, logical errors, and calculation errors, it is difficult to flexibly adapt. This may lead to situations of under-correction or over-correction, affecting the accuracy and stability of the model.​\n\n3.The online iteration process lacks clear convergence criteria, such as when the accuracy rate stabilizes or when the generation diversity reaches a threshold. This exposes the model to the risk of overfitting to its own generated error patterns, and currently, there is no theoretical guarantee of stability, making it difficult to ensure the reliability of the model during the iteration process.\n\nQUESTIONS:\n1.How to override underlying representation errors when reasoning errors originate from visual misinterpretation?\n2.Can dynamic β adapt to quality gaps of different error types? Are there optimal adaptive mechanisms?\n3.How to define convergence criteria for online iteration to avoid overfitting to self-generated errors?\n4.Why does trajectory-level correction fail in cross-modal reasoning? How to extend to temporal/3D modalities?\n5.How does verifier misjudgment affect correction directions? How to build a robust verification-correction loop?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Sherlock, a model designed to enhance VLMs with self-correction capabilities for multimodal reasoning. The authors begin with a preliminary study highlighting the limitations of current reasoning VLMs (e.g., LLaVA-CoT and VL-Rethinker), showing that these models struggle with both step-wise and response-level self-correction. To address this, Sherlock focuses on enhancing response-level self-correction through a multi-stage training framework: (1) a supervised fine-tuning (SFT) cold-start phase to strengthen response-level correction; (2) offline preference learning that targets erroneous reasoning steps for trajectory-level correction; and (3) iterative online preference training, which leverages both original and corrected responses from the previous Sherlock model to further improve self-correction. Experiments using LLaMA3.2-Vision-11B as the base model demonstrate that Sherlock outperforms existing baselines on several visual reasoning benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengthes\n\n- The paper addresses a timely and important problem—enhancing reasoning and self-correction capabilities in VLMs. The task is well-motivated, and the preliminary study provides insightful observations, showing that current VLMs (e.g., LLaVA-CoT and VL-Rethinker) struggle with both step-wise and response-level self-correction.\n- Building upon these findings, the authors propose a three-stage training framework that progressively improves self-correction. The approach is plausible, tackles both response- and trajectory-level correction, and remains relatively data-efficient, requiring only 20K samples.\n- The experimental results and analyses demonstrate clear improvements over baselines, and the ablation studies are comprehensive, effectively validating the contribution of each component.\n\n## Weaknesses\n\n- While the preliminary analysis is insightful, it relies on relatively small models (LLaVA-CoT and VL-Rethinker), which may limit the generalizability of the findings. \n- Sherlock primarily focuses on response-level self-correction, which shares similarities with self-refinement methods, where a model revises an initial response to improve accuracy. Thus, it would strengthen the work to include comparisons with self-refinement baselines—such as prompting a larger VLM (e.g., GPT) for multi-round refinement—or sampling-based methods like Best-of-N generation and reranking.\n- The analysis could benefit from a deeper exploration of error types—e.g., which types of reasoning errors Sherlock can or cannot correct, and how performance evolves across multiple rounds of self-correction. Such insights would offer greater value to the community.\n- Several technical details remain unclear: (1) In Line 130, how is the \"Modify One Step\" operation implemented? Is Qwen explicitly prompted to alter a correct step into an incorrect one? (2) In Table 2, does Sherlock generate outputs in a two-stage process (generation followed by self-correction)? If so, are the Sherlock variants in the Table 2 results before self-correction, and the \"+ Self-correction\" versions the revised results? (3) Related to the above, in Table 3, are the critic models used for self-correction response-level or step-wise? The fact that Sherlock + Self-correction underperforms Qwen2.5-VL-7B on some benchmarks suggests that error detection and correction remain challenging, even with training (while Qwen2.5-VL-7B is not trained). (4) Regarding model choice, why is MLLAMA used as the base model instead of Qwen-VL?\n\nQUESTIONS:\nBesides the questions I raised in weaknesses part:\n\n- The trajectory-level objectives proposed in Sherlock is similar to recent step-wise DPO methods. It would be beneficial to more thoroughly discuss the connection and differences between Sherlock and these step-wise DPO approaches.\n- The right side of Figure 2 is difficult to interpret at first glance. It feels unintuitive to present results from different models (e.g., LLaVA-CoT and Qwen) in the same plot without clear separation or explanation. Consider revising the figure to improve clarity.\n\nLIMITATIONS:\nThe limitations discussed in Section E are relatively brief and lack depth. Consider expanding this section to provide a more thorough reflection on the method’s constraints and potential areas for future improvement.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework, called Sherlock, for enhancing the reasoning ability of vision-language models (VLMs) through self-correction and iterative self-improvement. The proposed method focuses on fine-grained correction of reasoning trajectories, using preference optimization techniques to teach VLMs to revise only incorrect steps instead of rewriting full responses. The work is evaluated across eight benchmarks and demonstrates strong performance with significantly less annotated data.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- Clear motivation: The paper identifies a meaningful limitation of existing reasoning VLMs—namely, their inability to recover from intermediate errors. This motivates the proposal of self-correction as a mechanism to improve robustness and generalization.\n\n- Empirical efficiency: Despite its modest performance gains, Sherlock achieves them using significantly fewer annotations (20k) than baselines like LLaVA-CoT or Mulberry, which supports the claim of improved annotation efficiency.\n\n\n# Weaknesses\n### Cold-Start Strategy is not Thoroughly Justified\nIn Stage I, the authors fine-tune a base VLM on 10k samples randomly selected from the LLaVA-CoT dataset to initiate the Sherlock pipeline. However, this decision appears suboptimal and insufficiently justified. There exist multiple CoT-pretrained VLMs, such as LLaVA-CoT and LlamaV-o1, which are trained on much more extensive data. Using one of these as the starting point would likely result in a better initialization for downstream correction learning. The rationale for not leveraging these stronger checkpoints as a base model remains unclear and undermines the significance of Sherlock's first-stage improvements.\n\n### Unvalidated Noise Injection in Data Construction\nThe second stage constructs negative preference samples by injecting random visual noise and truncating reasoning trajectories. However, the paper lacks any quantitative analysis or diagnostics regarding how similar or dissimilar the positive and negative examples are. If the noise degrades the negative samples too aggressively, the quality gap may become too large to provide meaningful learning signals, essentially turning the preference learning into trivial classification. This weakens the effectiveness of the training objective, especially when fine-grained trajectory-level reasoning is the goal.\n\n### Limited Comparison with Recent and Relevant Work\nThe empirical comparisons do not sufficiently cover recent state-of-the-art approaches in multimodal reasoning, particularly those involving self-training via reflection, self-train via reasoning, structured prompting, etc. This omission raises concerns about the strength and generalizability of Sherlock’s reported improvements. List some here but not limited:\n- Vision-Language Models Can Self-Improve Reasoning via Reflection\n- Conme: Rethinking evaluation of compositional reasoning for modern vlms\n- Improve Vision Language Model Chain-of-thought Reasoning\n\n### Modest Performance Gain over Strong CoT Baselines\nFrom the main results in Table 2, the performance improvements over strong CoT-based models (e.g., LlamaV-o1) are relatively modest (e.g., 63.4 --> 65.4 average accuracy), despite the added training complexity. This raises an important question: If Sherlock had started from a stronger base model such as LlamaV-o1, would the additional self-correction stages still yield measurable improvements? This missing analysis leaves open the possibility that Sherlock's contribution may simply stem from better pretraining or CoT exposure, rather than its self-correction mechanism.\n\n### Minor Weakness\n- Ethical considerations of using self-generated preference data (e.g., bias propagation, hallucinated corrections) are not discussed.\n- The absence of a dedicated Conclusion section in the main text is a structural flaw (just briefly discussed in the appendix).\n\nQUESTIONS:\n1. Could you please answer to my questions/concerns in the first and second item listed on the Weaknesses panel?\n\n2. Could the author(s) provide the experimental results of which are initialized from *-CoT models in the first stage?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}