Title: Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning
Venue: NeurIPS 2025
Accepted: no
Keywords: Large Language Models

Reasoning

Long Chain-of-Thought

Critique
Abstract: While slow-thinking large language models (LLMs) exhibit reflection-like reasoning, commonly referred to as the “aha moment”, their ability to generate informative critiques and refine prior solutions remains limited. In this paper, we introduce Double-Checker, a principled framework designed to enhance the reasoning capabilities of slow-thinking LLMs by fostering explicit self-criticism and iterative refinement of their previous solutions. By fine-tuning on a curated DC-1.7K dataset of 1,730 self-critical instances, Double-Checker empowers long-CoT LLMs to iteratively critique and refine their outputs during inference until they evaluate their solutions as correct under self-generated critiques. We validate the efficacy of Double-Checker across a comprehensive suite of reasoning benchmarks, demonstrating that iterative self-critique significantly enhances the reasoning capabilities of long-CoT LLMs. Notably, Double-Checker increases the pass@1 performance on challenging AIME benchmarks from 4.4\% to 18.2\% compared to the original long-CoT LLMs. These results highlight a promising direction for developing more trustworthy and effective LLMs capable of structured self-critique.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper steers the LLM to do the self-critique-and-refine process to improve the answer correctness. The authors first probe that the current deepseek-distilled models do not self-improve the answer during the long-chain reasoning. Then the authors synthesize the SFT dataset will delicate framework and prompt design in which the critique and refinement processes are included. The SFT training with self-critique makes the performance better in both Math and GPQA.\n\nOverall, I think the main point is clear and the writing is good. However, my biggest concern is whether the innovation part is enough. The self-reflection for LLM reasoning has been extensively studied. The process to synthesize the dataset and train the model with SFT is also not such novel. Forcing LLM to critique and refine the answer is revealed to be effective in case of the testings in the study. However, as authors discussed, whether it will deteriorates model's reasoning capability remains to be answered.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The idea is clear and the writing is easy to follow. The experimental results show the effectiveness of the methd.\n\n\nMy questions are as follows:\n\n1) Lacks the baseline to do both SFT and further RL optimization\n\n2) The authors use Qwen3-235B and DeepSeek-R1 for initial data construction and annotation. That may have the issue of knowledge distillation since the training model 7B and 32B are much smaller. Whether the improvement is not because of the critique-and-refine mechanism but the models are taught to be better with the extra knowledge.\n\n3) The authors only test on GPQA to test generalizability. Whether testing on other domains will be possible?\n\ntypo: Line 55, proposeDouble -> propose Double; Line 793, Sec. ??;\n\nQUESTIONS:\nI am curious whether 8*H800 GPUs can fine-tune 32B models with full parameters, as claimed in Line 836, since H800 only has 80GB memory each. The maximum 16384 tokens and 32 batch size will also consume much memory.\n\nIn Line 249, what is the reason that the conclusion of the other study is different from this study.\n\nLIMITATIONS:\nYes, but in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Double-Checker, a framework designed to enhance the reasoning capabilities of slow-thinking large language models (LLMs) through self-critical fine-tuning. The authors identify that while long-CoT LLMs exhibit \"aha moments\" in reflection-like reasoning, they lack the ability to generate informative critiques and iteratively refine their solutions. Double-Checker addresses this limitation by fine-tuning models on a curated dataset of 1,730 self-critical instances, enabling iterative self-critique and refinement during inference. The method demonstrates significant improvements across reasoning benchmarks, notably increasing pass@1 performance on AIME benchmarks from 4.4% to 18.2%.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. Clear Problem Identification: The paper provides a well-motivated investigation showing that \"aha moments\" in long-CoT LLMs do not translate to effective self-critique capabilities, establishing a clear research gap.\n\n2. Thorough Ablation Studies: The paper includes comprehensive ablation studies examining the contribution of different components (direct data, critique data, number of rounds).\n\n### Weaknesses\n\n1. Inconsistent Baseline Performance and Unclear Evaluation Framework: The reported baseline performance in Table 1 raises significant concerns. While the average performance improvement appears to stem from GPQA results, there are substantial discrepancies with official DeepSeek reports. Specifically, DeepSeek's official documentation reports GPQA accuracy of 49.1% for DeepSeek-R1-Distill-Qwen-7B and 62.1% for the 32B variant, whereas this paper reports only 20.7% and 50.5%, respectively (https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B). Similarly, the paper reports 88.8% accuracy for DeepSeek-R1-Distill-32B on MATH500, while the official report shows 94.3%. Critically, the paper fails to specify the evaluation codebase used, which is crucial for mathematical reasoning tasks, as different frameworks employ varying answer extraction methods (e.g., some extract only content within \\box{} tags, while others like LightEval are more robust). The results of other baselines reported in the paper do not seem to be as good as the original ones, such as LIMO and s1.1.\n\n2. Limited Domain Generalization: The evaluation is restricted exclusively to mathematical reasoning tasks. A critical question remains unanswered: how does the fine-tuned model perform in other domains? The authors should evaluate performance on coding tasks (e.g., LiveCodeBench) and other reasoning domains to demonstrate the broader applicability of the self-critique mechanism.\n\n3. Incomplete Computational Cost Analysis: Section 5.3's focus solely on token consumption provides a misleading picture of computational overhead. Compared to the DS-Qwen-7B baseline, Double-Checker requires multiple model forward passes for each critique round, which significantly increases inference time. Additionally, Figure 4 should include accuracy curves for each round; if substantial accuracy improvements only occur in later rounds, the claim that \"tokens spent on direct inference of Double-Checker are fewer than the naive SFT baseline\" becomes meaningless from a practical standpoint.\n\n4. Missing Critical Baseline: The paper lacks a fundamental baseline comparison with \"base model + Self-Critique & Refine\" without the specialized training, which would help isolate the contribution of the training versus the inference procedure itself.\n\nQUESTIONS:\nSee the weaknesses part.\n\nLIMITATIONS:\nSee the weaknesses part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces so-called Double-Checker, a framework aimed at improving the reasoning of \"slow-thinking\" or long-CoT. The authors first observe that existing long-CoT models, despite showing signs of reflection, struggle to effectively critique and refine their own incorrect solutions. The proposed solution involves fine-tuning these models on a curated dataset of 1,730 instances that explicitly teach a self-critique and refinement loop. During inference, the fine-tuned model iteratively generates a solution, critiques it, and refines it until the internal critique judges the answer to be correct. The authors demonstrate that this method significantly boosts performance on several reasoning benchmarks, most notably improving pass@1 accuracy on the challenging AIME benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n-\tThe paper's main strength is its impressive empirical results. The Double-Checker framework achieves substantial performance gains over strong long-CoT baselines across multiple challenging benchmarks, including AIME, MATH500, and GPQA. The 18.2% absolute improvement on AIME25 for the 32B model is particularly noteworthy.\n\n-\tThe authors demonstrate that significant improvements can be achieved by fine-tuning on a relatively small dataset of only 1,730 high-quality, curated instances. This highlights the value of data quality over quantity for teaching complex reasoning skills.\n\nWeakness:\n\n-\tThe core concept of iterative refinement with self-generated feedback is not new. The framework is conceptually similar to prior work like Self-Refine and Reflexion. The main contribution of Double-Checker appears to be a specific, well-engineered implementation and data formatting strategy for long-CoT models, rather than a significantly novel idea. The paper essentially boils down to finetuning on self-critique data and accuracy may be inflated by simply training on more domain-similar data (shown by improvements in the naïve-SFT baseline) \n\n-\tDomain of testing is narrow, mainly limited to math and logic datasets\n\n-\tThe data curation pipeline is a critical component, yet it relies on extremely powerful models like Qwen3-235B and DeepSeek-R1 as \"teachers\" to generate the initial solutions, critiques, and refinements. This raises a significant concern: the performance gains may stem from distilling the superior capabilities of these teacher models into the smaller student models, rather than from teaching a generalizable skill of self-critique. This dependency limits the method's accessibility and scalability, as it requires access to SOTA models for data creation.\n\n-\tThe comparison being drawn is between an iterative double-checking framework and other reasoning model baselines. I am not convinced this is a fair comparison and that other self-reflection baselines or at least multiple sampling strategies /test-time scaling strategies would be a more accurate comparison.\n\n-\tThe Double-Checker N=0 model (trained on critique data but performing only direct inference) consistently outperforms the naive SFT baseline. This implies that the data format itself, not just the iterative inference, is a major source of improvement. Therefore, framing the main results as a comparison between the iterative Double-Checker (N>0) and the naive SFT model conflates the benefits of the data format with the benefits of the iterative process, potentially overstating the impact of the latter. Also, additional baselines comparation between Double-Checker and other similar approaches would greatly enhance the paper.\n\nQUESTIONS:\n-\tThe data curation pipeline relies heavily on powerful teacher models (Qwen3-235B, DeepSeek-R1). To what degree are the observed performance gains attributable to knowledge distillation from these models versus the learning of a genuine, generalizable self-correction skill? What would happen if the teacher models used for data generation were the same size and capability as the student models being trained?\n\n-\tGiven that your framework is iterative a test time which requires sampling multiple rounds from the llm, How does your framework compare in performance to other test-time scaling strategies?\n\n-\tI am not sure I agree that “critique models or reward estimators” are a parallel line of research. It seems like your method is very similar except your original large LLM is being used as the critique model. In this vein, how does your method compare to other critique methods which use critique models? What is the performance gap and compute trade-off compared to these methods? Keep in mind that while the split architecture strategy requires two LLMs the second one can be much smaller than the first and that would more than offset the additional overhead. \n\n-\tDoes your improved performance carry over to other testing areas? I believe this is needed if you want to claim “strong generalizability”. \n\n-\tIn Section 3.2.1 where the authors generate fine-tuning data using a dedicated refinement model to improve the quality of the initial critique content, there is no explicit explanation whether this operation will introduce reasoning traces that are unnecessary to yield the final result.\n\n-\tWhen comparing the results of the Double-Checker framework with other baselines in Table 1 in Section 4.3, consider including comparison between the fine-tuning approach with other self-correction / self-critique framework(s) that is either prompt-based (i.e. with no fine-tuning involved) or fine-tuning based (e.g. Learning to Check: Unleashing Potentials for Self-Correction in Large Language Models URL: https://arxiv.org/abs/2402.13035). This could make the result more comprehensive and demonstrate more clearly the advantage of fine-tuning in LLM’s ability to conduct self-critique.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}