Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation

Authors: Chengwei Dai, Kun Li, Wei Zhou, Songlin Hu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: CoT distillation, key reasoning steps, dual CoTs

Abstract:
As Large Language Models (LLMs) scale up and gain powerful Chain-of-Thoughts (CoTs) reasoning abilities, practical resource constraints drive efforts to distill these capabilities into more compact Smaller Language Models (SLMs). We find that CoTs consist mainly of simple reasoning forms, with a small proportion (~4.7%) of key reasoning steps that truly impact conclusions. However, previous distillation methods typically involve supervised fine-tuning student SLMs only on correct CoTs data produced by teacher LLMs, resulting in students struggling to learn the key reasoning steps, instead imitating the teacher's reasoning forms and making errors or omissions on these steps. To address these issues, drawing an analogy to human learning, where analyzing mistakes according to correct solutions often reveals the crucial steps leading to successes or failures, we propose mistak\textbf{E}-\textbf{D}riven key reason\textbf{I}ng step distilla\textbf{T}ion (\textbf{EDIT}), a novel method that further aids SLMs learning key reasoning steps rather than mere simple fine-tuning. Firstly, to expose these crucial steps in CoTs, we design specific prompts to generate dual CoTs data with similar reasoning paths but divergent conclusions. Then, we apply the minimum edit distance algorithm on the dual CoTs data to locate these key steps and optimize the likelihood of these steps. Extensive experiments validate the effectiveness of EDIT across both in-domain and out-of-domain benchmark reasoning datasets. Further analysis shows that EDIT can generate high-quality CoTs with more correct key reasoning steps. Notably, we also explore how different mistake patterns affect performance and find that EDIT benefits more from logical errors than from knowledge or mathematical calculation errors in dual CoTs. Code can be found at https://anonymous.4open.science/r/eb77sh-F564

Decision: reject
Reviewer scores: 3, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730355570517

thread_type:
official_review

<thread_object>
creator: reviewer_7kvh

<content>
summary:
This paper propose “Mistake-driven key reasoning step distillation”(EDIT) which aims to improve the reasoning capability of SLMs, focusing specifically on key reasoning steps. Unlike previous standard SFT techniques, EDIT fine-tunes SLMs solely on tokens within key reasoning steps, disregarding training loss for tokens that are not in key reasoning steps. To obtain the data for SFT, this paper prompts powerful LLMs to generate dual CoTs data by correcting erroneous CoTs and introducing errors into correct ones. Empirical results demonstrate the effectiveness of EDIT in significantly improving SLM performance on reasoning tasks.

strengths:
1. EDIT innovatively fine-tunes SLMs solely on tokens that are different in dual CoT reasoning steps to prevent simply mimicking the reasoning form, which is resource-efficient and reasonable.
2. The paper is well-written and easy to follow.

weaknesses:
1. Teacher LLMs generated contents lack evaluation, e.g, the correctness of reasoning steps after applying AHP to rectify wrong CoTs, and the ratio of correctly corrupted key reasoning steps?
2. The experiments of this paper are relatively weak. First, the compared baselines are all works essentially accomplished in 2022. Are there any brand new methods in 2023-2024? Second, the paper conducts in-domain experiments on only one dataset (BBH) which is released in 2022. It’s better to provide at least one more in-domain experiment on some convicing benchmarks to demonstrate the effectiveness of EDIT.
3. The structure of the paper can be optimized. I think it’s better to move some the results from Figure.5(middle) and Table.9 in Appendix to Table.1, making your main results more convicing. Figure 5 (right) is referenced in Section 5, yet it is included in the same figure as the content from Section 4. I believe it would be better to format it separately under Section 5.

questions:
See Weaknesses.1.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730382788425

thread_type:
official_review

<thread_object>
creator: reviewer_9wvq

<content>
summary:
The paper introduces EDIT (mistakE-Driven key reasonIng step distillaTion), a method aimed at enhancing the reasoning ability of smaller language models (SLMs) by distilling key reasoning steps from large language models (LLMs). Unlike traditional approaches that focus solely on correct answers, EDIT leverages dual chains of thoughts—pairs of reasoning processes that share a similar path but reach different conclusions—to identify critical reasoning steps using a minimum edit distance algorithm. This technique improves the SLM's ability to capture essential reasoning steps, thereby increasing the accuracy of its conclusions, as demonstrated through experiments on both in-domain and out-of-domain reasoning tasks.

strengths:
1.	The introduction of dual chains of thoughts (CoTs) and the application of a minimum edit distance algorithm to identify crucial reasoning steps represent a unique approach to improving the reasoning capabilities of smaller language models (SLMs).
2.	The method used in this work can improve the interpretability and generalization of reasoning abilities in smaller language models. 
3.	The methodology is sound, incorporating the generation of corrected CoTs and corrupted originally correct CoTs to create a rich training set for the SLMs. The results indicate that EDIT outperforms simple fine-tuning techniques.

weaknesses:
1.	The paper’s presentation could be enhanced by offering a clearer explanation of the mistake-driven key reasoning step distillation process shown in Figure 2. Readers may find it challenging to understand the purpose of obtaining the corrupted CoT and rectified CoT from the figure alone, so providing a more explicit explanation at the beginning of the methodology section is recommended.
2.	In the process of "Rectifying Wrong CoT," does providing the correct answer to the large model guarantee that it will generate a completely correct output? The quality of the data generated through this part of the method lacks further discussion. Additionally, how is it ensured that the rectified CoT retains similar reasoning steps, and does this impact the calculation of the minimum edit distance in later stages?
3.	The cases lack a comparison between the reasoning process generated by Edit and that of the ground truth. It remains unclear whether the reasoning process generated by Edit can truly capture the key steps in the ground truth.

questions:
1.	How are the weight values of α and β set, and how do they impact the performance of KRSL?
2.	Are there metrics to assess whether the reasoning steps generated by EDIT are more accurate compared to the original SFT?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730447211033

thread_type:
official_review

<thread_object>
creator: reviewer_bt6k

<content>
summary:
This paper focuses on distillation methods for SLMs through learning key reasoning steps from dual CoTs data. Specifically, they first collect CoTs data annotated by teacher LLMs including both the correct and wrong ones, and then generate their dual CoTs data, which possess similar reasoning paths but divergent conclusions. On this basis, they identify the key reasoning step via the minimum edit distance and employ a fine-grained loss for finetuning, thereby allowing students to learn key reasoning steps instead of simply imitating teachers’ reasoning forms. Extensive experiments demonstrate their effectiveness and superiority.

strengths:
1.This work designs dual CoTs data by prompting teacher LLMs, which can be utilized to  identify key reasoning steps through the minimum edit distance algorithm.

2.This work proposes a fine-grained loss based on the dual CoTs data, which enables students to focus more on key reasoning steps and reduce errors, rather than simply imitating teachers’ reasoning forms, further improving reasoning.

3.The proposed EDIT outperforms baselines on both in-domain and out-of-domain benchmarks, demonstrating its effectiveness and versatility. Further analysis also shows that EDIT can generate high-quality CoTs with more correct key reasoning step, and logical errors in dual CoTs benefit more for student SLMs.

weaknesses:
1. In L312, why is greedy decoding used for text generation in inference? Is it used to ensure the similarity between dual CoTs data? If not, how can we guarantee the formal similarity of dual data, that is, only the key steps are different, while the reasoning steps, forms, and even words of other parts remain consistent? And will there be a situation where non-key reasoning steps are identified as key steps?

2. It seems that the rectified teacher’s wrong CoTs contributes more in EDIT, as in Table 1, EDIT w/o KRSL outperforms w/o RWC, and in Table 2, D_dual^- outperforms D_dual^+. Therefore, what if not using RWC in the first step and only using D_dual^ + in the second step? 

3. To better show the superiority of the method, can you please compare EDIT with some of the latest baseline methods?

4. Typo: Table 24 and Table 25 are reversed.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721474379

thread_type:
official_review

<thread_object>
creator: reviewer_6mqt

<content>
summary:
This paper introduces a novel distillation method based on dual Chain-of-Thought (CoT) data that share similar reasoning paths but reach different conclusions. The approach begins by retaining all CoT data generated by the teacher model (GPT-3.5-turbo), regardless of correctness. The authors then design two prompts to guide the teacher in producing dual CoTs, ensuring they follow similar intermediate reasoning steps but arrive at divergent conclusions. Using a minimum edit distance algorithm, they identify the critical reasoning steps in these dual CoTs and apply a fine-grained loss function to optimize the likelihood of these key steps.

strengths:
1. Distilling the strong reasoning capabilities of a large language model (LLM) into a smaller LLM is an intriguing and valuable research topic.

2. The analysis of dual CoTs with different error types provides a more comprehensive approach to the distillation process, offering insights that could guide further refinement of Chain-of-Thought distillation methods.

weaknesses:
1. The novelty of the approach is limited, as many existing works already utilize mistake retrieval to provide tailored guidance and improve model performance during Chain-of-Thought (CoT) inference.

2. The method relies on constructing dual CoT datasets composed of positive-negative CoT pairs with similar intermediate reasoning steps but divergent conclusions. In practice, however, such pairs are often scarce, and the annotation process can be costly and labor-intensive, typically requiring manual selection. This could be a significant limitation unless the generation prompt achieves a high level of accuracy without human oversight.

3. The performance improvement over the teacher model is minimal, suggesting that the distillation process achieves only marginal gains and has room for further enhancement.

4. The method's reliance on teacher model errors could introduce inconsistencies, particularly if certain error types are more prevalent or systematically biased in the teacher model. Also, by focusing intensively on key reasoning steps, there may be a risk of overfitting certain reasoning patterns, potentially impacting generalization in scenarios that require varied reasoning styles.

questions:
1. How about distilling the strong reasoning capabilities of a large language model (LLM) into a smaller LM? Let's say a 1.5B model.

2. Why is there a huge gap in performance between the teacher model and the student model?

3. What is the scalability of EDIT? What if there are not enough "high-quality" positive-negative CoT pairs that follow similar intermediate reasoning steps but lead to divergent conclusions?
</content>
</thread_object>

</thread>


Metareview: