SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction

Authors: Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E. Gonzalez, Bin CUI, Shuicheng YAN
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, LLM Reasoning

Abstract:
Large language models (LLMs) like GPT-4, DeepSeek-R1, and ReasonFlux have shown significant improvements in various reasoning tasks. However, smaller LLMs still struggle with complex mathematical reasoning because they fail to effectively identify and correct reasoning errors. Recent reflection-based methods aim to address these issues by enabling self-reflection and self-correction, but they still face challenges in independently detecting errors in their reasoning steps. To overcome these limitations, we propose SuperCorrect, a novel two-stage framework that uses a large teacher model to supervise and correct both the reasoning and reflection processes of a smaller student model. In the first stage, we extract hierarchical high-level and detailed thought templates from the teacher model to guide the student model in eliciting more fine-grained reasoning thoughts. In the second stage, we introduce cross-model collaborative direct preference optimization (DPO) to enhance the self-correction abilities of the student model by following the teacher's correction traces during training. This cross-model DPO approach teaches the student model to effectively locate and resolve erroneous thoughts with error-driven insights from the teacher model, breaking the bottleneck of its thoughts and acquiring new skills and knowledge to tackle challenging problems. Extensive experiments consistently demonstrate our superiority over previous methods. Notably, our SuperCorrect-7B model significantly surpasses powerful DeepSeekMath-7B by 7.8\%/5.3\% and Qwen2.5-Math-7B by 15.1\%/6.3\% on MATH/GSM8K benchmarks, achieving new SOTA performance among all 7B models. Code is available at: https://github.com/YangLing0818/SuperCorrect-llm

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730175619666

thread_type:
official_review

<thread_object>
creator: reviewer_huwp

<content>
summary:
The paper introduces SuperCorrect, a two-stage framework to improve the mathematical reasoning of smaller models like Llama-3-8B and DeepSeekMath-Base. While large language models such as GPT-4 and PaLM excel, smaller models struggle with error detection and correction. SuperCorrect addresses this by using a large teacher model to guide the student model through reasoning and reflection. The first stage extracts hierarchical thought templates from the teacher model to refine the student's reasoning. The second stage uses cross-model direct preference optimization (DPO) to enhance the student's self-correction abilities. SuperCorrect-7B achieves state-of-the-art results, outperforming DeepSeekMath-7B and Qwen2.5-Math-7B on MATH and GSM8K benchmarks.

strengths:
1. The writing of the paper is smooth and clear, making it easy to understand.
2. Significant performance improvements were achieved in mathematical tasks.
3. The paper has open-sourced 10K high-quality SFT data and 1K preference alignment data.

weaknesses:
1. The paper introduces two strategies for data augmentation using the teacher model (corresponding to SFT and DPO training), but there is a lack of a clear logical connection between these two methods, as they appear to be separately designed.
2. The motivation behind the design of the "Hierarchical Thought Template" is insufficiently explained. In the Introduction, author mention that the capability deficits in smaller models stem from "failing to effectively identify and correct reasoning errors". However, this doesn't naturally lead to the idea that hierarchical reasoning, incorporating both generalization and details, would be effective.
3. Using a teacher model to correct the student's reasoning results is not a novel idea. Although this paper emphasizes the use of DPO training with these data, there is a possibility that the performance gains primarily come from the expansion of data after correction, and using the SFT approach on this data may achieve similar results.

questions:
1. Are the training data for Qwen2.5-Math-7B-Instruct, DeepSeekMath-7B-Instruct, and Meta-Llama3.1-8B-Instruct the same as the data used for training SUPERCORRECT-7B as mentioned in the paper? Including the types of datasets and the data volumes used.
2. What LLM was used to generate the high-quality dataset ? Considering that LLM can still generate incorrect data (both of Hierarchical Thought process and error correction data), were any measures taken to address this issue?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730383357108

thread_type:
official_review

<thread_object>
creator: reviewer_grhy

<content>
summary:
The paper introduces SUPERCORRECT, a two-stage framework aimed at boosting smaller LLMs' reasoning and self-correction abilities. It leverages guidance from larger teacher models, such as GPT-4 or LLaMA, in two core phases: hierarchical thought templates and cross-model collaborative Direct Preference Optimization. Experiments on MATH and GSM8K benchmarks reveal that SUPERCORRECT consistently outperforms traditional reflection and fine-tuning approaches. Notably, the SUPERCORRECT-7B model exceeded DeepSeekMath-7B by 7.8% on MATH and 5.3% on GSM8K, showcasing improvements in both accuracy and stability.

strengths:
1. By utilizing the teacher model to identify and correct errors in the student model's reasoning, SUPERCORRECT not only corrects mistakes but also teaches the student model to avoid and rectify specific errors. This approach breaks the bottleneck of the student model's thought process and equips it with new skills and knowledge to tackle challenging problems.

2. When leverage a large teacher model to supervise and correct a smaller student model, SUPERCORRECT combines hierarchical thought templates and cross-model collaborative direct preference optimization (DPO). I think this method is innovative.

3.  This advancement is particularly significant as it shows the potential for smaller models to compete with or even surpass the capabilities of much larger models in complex mathematical reasoning tasks.

weaknesses:
1. The paper primarily focuses on 7B models. It may not be immediately clear how well the SUPERCORRECT framework would scale to larger models or generalize across different types of reasoning tasks beyond mathematical problems.

2. The success of SUPERCORRECT relies heavily on the quality of the fine-tuning datasets and the paired correction traces. The paper mentions constructing high-quality datasets, but it may face challenges in scenarios where such curated datasets are not available or the domain of interest is very niche.

questions:
1.	Could you elaborate on the potential strategies for scaling the SUPERCORRECT framework to even larger language models, and how you might address the computational efficiency challenges that come with such an increase in model size?

2.	How does SUPERCORRECT handle systematic biases or errors that may be present in the teacher model's corrections? Additionally, could you discuss how the framework evaluates and ensures the robustness of the student model against such potential inaccuracies in the supervision process?

3.	In the context of the two-stage training process, have you observed any stability issues or challenges in long-term training? If so, what techniques or modifications are employed to ensure the stability and convergence of the models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660717892

thread_type:
official_review

<thread_object>
creator: reviewer_bwuv

<content>
summary:
The paper tries to improve the mathematical reasoning of small LLMs by (1) improving their thought template using their proposed Hierarchical Thought  template and (2) enhancing the self-correction of the model by making it to learn how to self-correct like a larger SOTA model using the proposed cross-model DPO. The experiments are done over three small LLMs on two math benchmarks.

strengths:
**(a)** I think the paper is very well-written and even someone who is not entirely familiar with self-correction and reflection concepts can follow. The formatting is also clean and makes the story again easier to follow. The preliminary concepts such as DPO and how it relates to other methods such as RLHF is again very well presented.

**(b)** The cross-model DPO, to me, seems quite close to a distillation framework of teaching the smaller model how to identify wrong steps from a larger model. I think it is very simple and quite applicable to many domains with different tasks (see also Question **(a)** below).
 
**(c)** The quantitative results seem quite promising, especially in Table 1 and 2. Both HTSF and cross-model DPO seem to give significant gains. (see also weakness **(c)** below)

weaknesses:
**(a)**  While I find the quantitative results quite promising, I think there are a few tiny tables that can be added to further prove the point and support the proposed intuition. Specifically, the way I see it, is that the self-correction ability of the smaller (student) LLM is improved by comparing it with the correction provided from a larger, more powerful teacher, as if the paper is ‘distilling’ the self-correction. While it seems to work well for final accuracy, it is not quantitatively supported that this distillation was successful. For example, I would like to see if the two models agree on each reasoning step after the distillation and can locate correct and wrong steps the same way. This should be quantifiable, as you already have discrete reasoning steps in XML format. Specifically, I suppose you can measure the ratio of steps that are predicted the same way (both larger LM and smaller LM) say `correct’ or `wrong’ over the entire number of steps. Then, based on what the paper claims, I expect this ratio to increase after the cross-model collaborative DPO stage. I let the authors decide what sort of metrics demonstrates this the best, but I think right now such an inspection is missing.


**(b)** Regarding the Thought Template, the main contribution seems to be the hierarchical aspect of the thought. However, I think this needs to be supported by measuring variance over different prompt styles, while I would maintain the hierarchical aspect. For example, would we get the same benefits if the <Generalized> statement is moved to be before all of the <Step>s? (I understand if one works better than the other, but overall I would expect this should still give similar gains?) The other way to test this would be to only remove the Generalized prompt (or negatively prompt the model to not provide this). Or you could even use the same exact prompt as the baseline, and only improve it with a statement to include the <Generalized> template as well. In short, I’m currently finding the gains from HSFT alone, compared to SFT, hard to associate **only** to the hierarchical aspect and not to the different prompt formatting used.

**(c)** I would like to ask the authors to clarify why Table 2 only shows numbers for the Qwen model and not for the DeepSeek and Llama, as this would not require any additional experiments!? In the current version the individual gains for HSFT(without cross-model DPO) is only demonstrated for Qwen and not the other two models.

questions:
**(a)** It is stated multiple times that self-correction of reasoning steps is additionally difficult for mathematical tasks. While I mostly agree with this, the current proposed method has no part in its design that limits itself to Math benchmarks; the proposed Hierarchical Thought Template, to some extent, can be used for any task and the cross-model correction with a teacher model can also be used in other tasks. Therefore, it currently seems strange to me that the paper only evaluates on the math benchmarks and only for smaller LM models tuned for math questions. I would like to ask the authors to clarify this.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686339599

thread_type:
official_review

<thread_object>
creator: reviewer_qt8r

<content>
summary:
The paper proposes SuperCorrect, mainly focuses on addressing the issue of unability of detecting erroneous steps in their reasoning steps of small language models. SuperCorrect firstly prompts powerful LLMs to generate correct CoT reasoning steps and their detailed thought templates, then employs cross-model collaborative DPO to enhance the inner reasoning and self-correct abilities of small language models. Extensive experiments have demonstrate the effectiveness of proposed SuperCorrect framework.

strengths:
1. The paper proposes hierarchical thought template that more effectively captures the underlying reasoning mechanisms of powerful LLMs compared to simply prompting them to generate reasoning steps.
2. Cross-model collaboration applies step-level correction and DPO, which can provide small language models more detailed supervise signals and than solution-level training, thus leading to more powerful reasoning capabilities.
3. Extensive experiments are convicing, demonstrating the effectiveness of SuperCorrect.
4. The paper is well-structured and well-written.

weaknesses:
Teacher LLMs generated contents lack evaluation. For example, the logic flaws and errors found by teacher LLMs in student LLMs generated reasoning steps (Line 331). In Line 333, are the analysis $a_i$ and correction $c_i$ annorated by humans or generated by teacher LLMs? If it’s the latter case, what are the quality of these generated contents?

It’s seems like the cross-model collaborative DPO is nearly identical to the Step-DPO [1] paper, with the main difference being that in Step-DPO, $a_i$ and $c_i$ are sampled directly from the policy model, whereas in this paper, they are annotated/generated by humans/teacher LLMs. Can you explain the core differences? Or are these simply parallel works?

[1] Lai, X., Tian, Z., Chen, Y., Yang, S., Peng, X. and Jia, J., 2024. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732090907754

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We sincerely thank all the reviewers for their thorough reviews and valuable feedback. We are glad to hear that  our proposed framework is novel and generalizable (reviewer Bwuv and gRHY), the paper is well-written and presented (reviewer qT8r,Bwuv and hUwP), and the performance improvements demonstrated in experiments are significant and promising (all reviewers).

Here, we want to highlight the main contributions and novelties of our proposed framework, "SuperCorrect" :

**Hierarchical Thought Based Reasoning:** SupperCorrect introduces a novel first-stage finetuning paradigm that leverages hierarchical thought based reasoning to improve reasoning ability of LLMs. This approach not only significantly enhancing accuracy, but also enable LLMs to consistently generate reasoning thought during the reasoning process.

**Cross-model Collaborative DPO:**  SuperCorrect proposes a novel second-stage Cross-model Collaborative DPO which utilizes teacher LLM to correct the erroneous thought exposed in the self-correction traces of student LLMs. Regarding cross-model DPO innovatively learns to correct self-correction errors instead of only correcting reasoning errors, the student LLMs can focus on both correct and erroneous correction traces which help to better improve the self-correction ability and further break the original thought bottleneck. 

**Empirical Validation:** Extensive experiments on MATH and GSM8K datasets demonstrates significant performance improvements over state-of-the-art models, including substantial accuracy improves and promising generalization ability to larger sized models and more various domains.

We summarize our responses to the reviewers' comments as follows:

* We additionally provide more examples and conduct more experiments to show the quality of our dataset and **updated our manuscript in Appendix.D**. (Reviewer qT8r and hUwp)

* We additionally conduct ablation experiments and provide more quantitative and qualitative comparisons for the difference between Step-DPO and our Cross-model DPO and **updated our manuscript in Appendix.C** (Reviewer qT8r)

* We additionally conduct ablation study on the impact of prompt style, and prsent quantitative analysis. (Reviewer Bwuv)

We reply to each reviewer's questions in detail below their reviews. Please kindly check out them. Thank you and please feel free to ask any further questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
We recommend the paper to be accepted for Poster.

Below a more detailed description of the paper.

The paper introduces SuperCorrect, a two-stage framework to improve the mathematical reasoning of smaller models like Llama-3-8B and DeepSeekMath-Base. The papers shows that new method achieves SOTA performance across all models considered.
The main strengths (S#) of the paper are

- (S1)	the proposed methodology that seems to more effectively capture the reasoning mechanisms of powerful LLMs.
- (S2)	the proposed methodology, especially the cross-model DPO, can be applied to other domains.
- (S3)	the promising quantitative results obtained across different models, that achieve SOTA performance.
- (S4)	the paper is well written and easy to follow, also for people not familiar with the specific topic.

Many of the weaknesses pointed out by the reviewers were addressed by the authors.
Some weaknesses (W#) that may remain are the following:

- (W1)	The paper primarily focuses on small models and it is not clear whether the methodology proposed can scale to larger models. This in turn means that the methodology may be only applicable to a niche of available models; aspect that may still be useful, but limit its interest to the wider community.
- (W2)	The success of the proposed methodology relies on the quality of the fine-tuning datasets and the paired correction traces. The paper mentions constructing high-quality datasets, but it may face challenges in scenarios where such curated datasets are not available or the domain of interest is very niche.

Some discussions on how to overcome these weaknesses have been mentioned by the authors, but are not conclusively addressed, and may limit the success and adoption of the methodology.