Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
Authors: Xinghao Chen, Zhijing Sun, Guo Wenjin, Miaoran Zhang, Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, Xiaoyu Shen
Accepted at: acl25

Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning

Xinghao Chen1,2,3, Zhijing Sun3, Wenjin Guo3, Miaoran Zhang4, Yanjun Chen1,3*, Yirong Sun3, Hui Su5, Yijie Pan3, Dietrich Klakow1, Wenjie Li1*, Xiaoyu Shen2,3*
1 Department of Computing, The Hong Kong Polytechnic University
2 Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative
3 Institute of Digital Twin, EIT 4 Saarland University 5 Meituan Inc.
xing-hao.chen@connect.polyu.hk cswjli@comp.polyu.edu.hk xyshen@eitech.edu.cn

Abstract

Large Language Models (LLMs) excel in reasoning tasks through Chain-of-Thought (CoT) prompting. However, CoT prompting greatly increases computational demands, which has prompted growing interest in distilling CoT capabilities into Small Language Models (SLMs). This study systematically examines the factors influencing CoT distillation, including the choice of granularity, format and teacher model. Through experiments involving four teacher models and seven student models across seven mathematical and commonsense reasoning datasets, we uncover three key findings: (1) Unlike LLMs, SLMs exhibit a non-monotonic relationship with granularity, with stronger models benefiting from finer-grained reasoning and weaker models performing better with simpler CoT supervision; (2) CoT format significantly impacts LLMs but has minimal effect on SLMs, likely due to their reliance on supervised fine-tuning rather than pretraining preferences; (3) Stronger teacher models do NOT always produce better student models, as diversity and complexity in CoT supervision can outweigh accuracy alone. These findings emphasize the need to tailor CoT strategies to specific student model, offering actionable insights for optimizing CoT distillation in SLMs. The code and datasets are available at https://github.com/EIT-NLP/Distilling-CoT-Reasoning.

1 Introduction

Large Language Models (LLMs) have demonstrated exceptional capabilities through extensive pretraining on diverse human language data (Brown et al., 2020; Hoffmann et al., 2022; Team et al., 2024a; Meta, 2024a; OpenAI, 2024a,b). Chain-of-Thought (CoT) prompting has further enhanced their abilities by guiding LLMs to generate intermediate reasoning tokens, which emulate human cognitive processes and improve interpretability (Kojima et al., 2022; Wei et al., 2023). Advances in CoT prompting have explored techniques like extending reasoning steps (Jin et al., 2024; Merrill and Sabharwal, 2024) and refining reasoning formats (Deng et al., 2024; Xu et al., 2024a). However, CoT’s token-intensive nature significantly increases computational demands (Zhao et al., 2024), limiting its practicality in resource-constrained settings. This has spurred interest in distilling CoT capabilities into Small Language Models (SLMs) as a more efficient alternative (Team et al., 2024b; Meta, 2024b).

Since SLMs often struggle to independently generate effective CoT reasoning solutions (Kaplan et al., 2020; Stolfo et al., 2023), distilling CoT capabilities requires fine-tuning SLMs on teacher-annotated CoT datasets, where the teacher can be either human experts or more powerful LLMs. While previous research has demonstrated successful distillation of CoT capabilities into SLMs (Ho et al., 2023; Magister et al., 2023; Xu et al., 2024b; DeepSeek-AI et al., 2025), the choice of teacher annotators and CoT generation methods has often

![Overview of CoT Distillation. Different teacher models generate CoT supervision with varying levels of granularity and formats to fine-tune the student model.](page_1012_370_388_246.png)

Figure 1: Overview of CoT Distillation. Different teacher models generate CoT supervision with varying levels of granularity and formats to fine-tune the student model.
been arbitrary. A critical, yet unexplored, research question remains: What is the most effective CoT supervision for training a student model to develop robust reasoning capabilities? Analogous to how human teachers instruct students, there are three key factors that influence how effectively a student absorbs knowledge:

• Choice of teacher: This defines who teaches the student. Different teachers bring varying levels of knowledge, teaching styles, and problem-solving approaches. In reality a student’s performance can vary significantly depending on the teacher, and the most knowledgeable person is not always the best teacher.

• Granularity of teaching: This defines what level of detail is provided. Teachers may provide varying levels of explanation: some offer detailed, step-by-step reasoning, while others skip over simpler steps, assuming they are self-evident. The optimal level of granularity depends on the student’s perspective of what needs to be explained.

• Format of teaching: This defines how the reasoning is structured and presented. Even with the same teacher and granularity level, the way information is organized and expressed can significantly impact learning outcomes. Some students may prefer plain language explanations, while others may thrive with more technical, mathematical language.

Building on this analogy of how human teaching impacts student performance, we conducted extensive experiments on four mathematical reasoning datasets of varying difficulty and three commonsense reasoning datasets, using four teacher models to distill reasoning skills to seven student models. We adopted a 1-shot prompting approach for generating CoT annotations, which we found to be the most effective in maintaining consistency in teaching style while controlling granularity. Our key findings are: (1) While LLMs benefit monotonically from detailed steps, SLMs exhibit an non-monotonic relationship. Stronger student models benefit from finer granularity, while weaker ones can be overwhelmed by excessive explanations and prefer simpler CoT annotations; (2) CoT format changes influence LLMs, likely due to their pre-training preferences for certain structures, but this effect is less pronounced in SLMs, which adapt more readily to diverse formats during fine-tuning; (3) Contrary to prior research suggesting that better teacher models invariably lead to better student performance (Zong et al., 2023), in the task of distilling CoT capabilities, we find that better teacher models do not always produce better student models. Stronger student models benefit more from advanced teacher model. Human-annotated CoTs, despite their near-perfect accuracy, often underperform LLM-generated CoTs. Our work presents the first systematic framework for optimizing CoT distillation, laying the groundwork for enhancing the reasoning capabilities of SLMs.

2 Related Work

CoT prompting CoT prompting (Wei et al., 2023) has become a pivotal technique for enhancing reasoning capabilities in LLMs by introducing intermediate reasoning steps. Automated approaches like Auto-CoT (Zhang et al., 2023), Tree-of-Thoughts (Yao et al., 2023) and Self-play Mutual Reasoning (Qi et al., 2024) explore multiple reasoning paths to expand the search space and improve task accuracy. These methods focus on increasing the reasoning length or expanding the reasoning horizon to handle complex tasks. Recent studies have underscored the importance of reasoning granularity and formats in enhancing LLM performance. For instance, Jin et al. (2024) identified that longer reasoning steps improve task success for complex problems, while overly concise steps can reduce effectiveness. Tailored reasoning formats(Khot et al., 2023; Zhou et al., 2023; Deng et al., 2024; Xu et al., 2024a) have demonstrated substantial improvements across tasks. However, these reasoning optimization strategies often comes with significant computational costs (Nayab et al., 2024), raising concerns about the trade-off between accuracy and efficiency.

Knowledge distillation While direct prompting enables LLMs to perform complex reasoning through CoT, SLMs struggle due to limited capacity (Stolfo et al., 2023). Knowledge distillation (KD) provides an effective framework for transferring the reasoning capabilities of teachers to SLMs (Xu et al., 2024b). A simple yet effective approach is using a teacher-student paradigm, which employs teacher-generated CoT steps to guide SLMs, addressing their limitations and enhancing reasoning-intensive task performance (Magister et al., 2023; Ho et al., 2023; Shridhar et al., 2023).
Despite these advances, a systematic exploration of how to balance reasoning granularity, format, and teaching strategies remains lacking. Addressing these gaps is crucial for optimizing CoT distillation and enabling efficient reasoning in SLMs.

3 Problem Formulation

Let \( \mathcal{D} = \{(x_i, y_i)\}_{i=1}^N \) denote a reasoning dataset with \( N \) \((x_i, y_i)\) pairs. Chain-of-Thought distillation aims to train a student \( S \) to generate intermediate reasoning steps \( C_i \) for each input \( x_i \) in order to generate the right \( y_i \). The optimal \( C_i \) to train the student model is influenced by three key pedagogical factors: Choice of Teacher, Granularity of Teaching, and Format of Teaching:

Choice of Teacher  The teacher model \( T \) generates a reasoning chain \( C_T(x_i) \) for each input \( x_i \), which guides the student model in producing the correct answer. The teacher can either be an LLM or a human with varying styles and expertise.

Granularity of Teaching  Granularity refers to the level of detail in the CoT reasoning. A high-granularity annotation provides detailed, step-by-step reasoning, while a low-granularity annotation skips steps and provides a more abstract summary. We represent the CoT chain with granularity \( g \) as \( C_g(x_i) = (c_{g,1}, c_{g,2}, \ldots, c_{g,k_g}) \) where \( k_g \) is the number of reasoning steps. Higher \( k_g \) and more tokens in \( c_{g,i} \) indicates higher granularity.

Format of Teaching  Format refers to the structure in which the CoT reasoning is presented. It could be in natural language, formal logic, or symbolic representation. We denote the CoT chain in format \( f \) as \( C_f(x_i) \). The format impacts how the reasoning steps are conveyed.

Given these three factors, the distillation process involves supervised fine-tuning of the student model \( S \) on generated CoT annotations:

\[
\mathcal{L}_{\text{distill}} = \sum_{i=1}^N \mathcal{L}(S(x_i), C_{T,g,f}(x_i) \oplus y_i)
\]

where \( S(x_i) \) is the generation from \( S \), \( C_{T,g,f}(x_i) \) denotes the CoT annotation generated under teacher \( T \) with granularity \( g \) and format \( f \), \( \oplus \) denotes concatenation and \( \mathcal{L} \) measures the discrepancy between \( S(x_i) \) and the ground truth.

4 Experimental setup

4.1 Generation of CoT Annotation

Teacher Models  We use three teacher models: GPT-4o (OpenAI, 2024a), Gemini-1.5-Flash (Team et al., 2024a), and LLaMA 3 70B (Meta, 2024a), chosen for their diverse architectures and reasoning capacities. Additionally, we include human-annotated CoTs, typically considered the ground-truth reasoning steps (Kumar et al., 2024).

Generation Method  The CoT generation process begins with selecting a representative problem from the training split as a 1-shot example. This example is used to prompt teacher models for generating annotations under various configurations. For granularity, we prompt the model to generate CoTs with varying levels of detail simultaneously. For format, we prompt the model to generate CoTs for each format individually.\footnote{For granularity, we also investigated other data collection strategies, such as generating reasoning steps forward and backward simultaneously, but these methods did not produce better data, as shown in Appendix C.1.} Details regarding the workflow, prompt designs, and case studies are included respectively in Appendix C and D.

4.2 Tasks and Datasets

Mathematical Reasoning  To evaluate mathematical reasoning, we utilize four datasets with varying complexity levels: SVAMP (Patel et al., 2021), GSM8K (Cobbe et al., 2021), AQuA-RAT (Ling et al., 2017), MATH (Hendrycks et al., 2021). SVAMP, GSM8K, and MATH require numerical answers, while AQuA-RAT adopts a multiple-choice format. From the MATH dataset, we randomly sample problems from subcategories such as prealgebra, algebra, number theory, and counting and probability, ensuring a representative coverage of diverse mathematical domains.

Commonsense Reasoning  For commonsense reasoning, we use three datasets: CommonsenseQA (CSQA, Talmor et al. 2019; Aggarwal et al. 2021), OpenBookQA (OBQA, Mihaylov et al. 2018), and StrategyQA (STQA, Geva et al. 2021). These datasets test the models’ ability to handle everyday reasoning and general knowledge tasks. CSQA uses a 5-class multiple-choice format, OBQA has 4 classes, and STQA is binary.

For evaluation, answers are extracted from generated responses using predefined templates and
Figure 2: Performance of student models with different granularity. Most models achieve peak accuracy at intermediate granularity levels.

regular expressions. We use accuracy as our evaluation metric, which is calculated as the ratio of correctly predicted instances to the total number of instances: Accuracy = \( N_{\text{correct}} / N_{\text{total}} \). The complete details can be found in the Appendix A.

5 Effects of Granularity

While previous research has shown that increasing reasoning granularity improves LLM performance through detailed intermediate steps (Jin et al., 2024; Merrill and Sabharwal, 2024), SLMs differ fundamentally from LLMs in their ability to process complex reasoning chains. This raises a critical question: does increasing reasoning granularity still yield consistent benefits for SLMs in the task of CoT distillation? In this section, we investigate this question using GPT-4o as the teacher model.

Non-Monotonic Scaling in Student Models As shown in Figure 1, our experiments reveal a non-monotonic relationship between CoT granularity and student model accuracy. Most models exhibit peak performance at intermediate granularity levels. Further increasing granularity leads to diminishing returns and even performance declines. It suggests that intermediate granularity strikes a balance between informativeness and efficiency in CoT, whereas overly detailed reasoning chains may introduce redundant information which is overwhelming especially for weaker models.

Table 1 presents the performance of three representative student models across seven evaluation datasets. We include a baseline called Only Answer, where student models are fine-tuned to predict answers without CoT. Similar to System 1’s automatic thinking (Yu et al., 2024), higher baseline score suggests that the model may have implicitly learned the relevant knowledge during pretraining (Prabhakar et al., 2024).

Notably, stronger and more recent student models, such as those from the Gemma and LLaMA family, achieve significant performance gains from KD at higher granularity levels. In contrast, smaller and weaker models like BLOOM family improve on simpler tasks such at the intermediate granularity levels but struggle on more challenging datasets, sometimes performing no better than random guessing. This trend of BLOOM family aligns with parameter scaling laws (Kaplan et al., 2020) for simpler tasks but breaks down for more complex ones, where smaller models fail to acquire the reasoning abilities due to limited training data. Full results are provided in Appendix E.

These findings emphasize that CoT granularity plays a crucial role in CoT distillation. Customizing granularity levels to align with the student’s abilities is thus critical for maximizing the efficiency and effectiveness.

Distinguishing Granularity from Length Effect
Increasing reasoning granularity often leads to longer sequences as a byproduct. To isolate the impact of granularity from sequence length, we pad CoT training samples for a lower granularity level \( g_1 \) with non-informative filler content to match the sequence length of a higher granularity \( g_2 \), such that \( \text{avg\_len}(\mathcal{D}_{g_1}) \approx \text{avg\_len}(\mathcal{D}_{g_2}) \). This modification allows us to assess whether reasoning accuracy stems from granularity or simply sequence length. The specific padding procedure can be found in the Appendix F.

As shown in Table 2, padding Level 1 reasoning chains to level 5 consistently failed to replicate the gains observed with actual higher-granularity reasoning, which demonstrates that simply increasing
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Only Answer</th>
    <th colspan="6">Gemma 2B Performance</th>
  </tr>
  <tr>
    <th>Level 1</th>
    <th>Level 2</th>
    <th>Level 3</th>
    <th>Level 4</th>
    <th>Level 5</th>
    <th>Level 6</th>
  </tr>
  <tr>
    <td>SVAMP</td><td></td><td>47.70</td><td>59.48<sup>11%</sup></td><td>64.33<sub>±0.01</sub></td><td>65.22<sub>±0.01</sub></td><td>65.89<sub>±0.38</sub></td><td><b>67.11<sub>±0.38</sub></b><br><span style="color:red;">+13.74%</span></td>
  </tr>
  <tr>
    <td>GSM8K</td><td></td><td>8.20</td><td>46.09<sub>±0.27</sub></td><td>52.36<sub>±0.08</sub></td><td>53.37<sub>±0.01</sub></td><td>52.69<sub>±0.15</sub></td><td>53.42<sub>±0.15</sub><br><span style="color:red;">+5.31%</span></td>
  </tr>
  <tr>
    <td>AQuA-RAT</td><td></td><td>20.47</td><td>40.68<sub>±0.27</sub></td><td>42.91<sub>±1.01</sub></td><td>43.72<sub>±0.58</sub></td><td>39.91<sub>±0.15</sub></td><td><b>44.88<sub>±0.15</sub></b><br><span style="color:red;">+12.48%</span></td>
  </tr>
  <tr>
    <td>MATH</td><td></td><td>9.00</td><td>23.41<sub>±0.06</sub></td><td>21.53<sub>±0.16</sub></td><td><b>24.44<sub>±0.20</sub></b><br><span style="color:red;">+16.19%</span></td><td>21.93<sub>±0.42</sub></td><td>23.01<sub>±0.22</sub><br><span style="color:red;">+21.06%</span></td>
  </tr>
  <tr>
    <td>CSQA</td><td></td><td><b>69.86</b></td><td>67.38<sub>±0.82</sub></td><td>67.98<sub>±0.37</sub></td><td>68.74<sub>±1.30</sub></td><td>66.75<sub>±0.53</sub></td><td>67.54<sub>±0.47</sub><br><span style="color:red;">+6.61%</span></td>
  </tr>
  <tr>
    <td>OBQA</td><td></td><td>69.60</td><td>71.53<sub>±0.94</sub></td><td>69.93<sub>±0.90</sub></td><td>69.93<sub>±0.36</sub></td><td>68.33<sub>±1.27</sub></td><td><b>72.00<sub>±1.64</sub></b><br><span style="color:red;">+25.37%</span></td>
  </tr>
  <tr>
    <td>STQA</td><td></td><td>60.69</td><td><b>67.92<sub>±1.11</sub></b><br><span style="color:red;">+11.11%</span></td><td>63.17<sub>±1.79</sub></td><td>64.03<sub>±1.36</sub></td><td>65.87<sub>±1.24</sub></td><td>65.57<sub>±1.77</sub></td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Only Answer</th>
    <th colspan="6">LLaMA 3.2 3B Performance</th>
  </tr>
  <tr>
    <th>Level 1</th>
    <th>Level 2</th>
    <th>Level 3</th>
    <th>Level 4</th>
    <th>Level 5</th>
    <th>Level 6</th>
  </tr>
  <tr>
    <td>SVAMP</td><td></td><td>53.70</td><td>69.38<sub>±1.53</sub></td><td>65.89<sub>±3.53</sub></td><td>68.33<sub>±1.86</sub></td><td>68.11<sub>±1.27</sub></td><td><b>69.78<sub>±1.07</sub></b><br><span style="color:red;">+74.33%</span></td>
  </tr>
  <tr>
    <td>GSM8K</td><td></td><td>9.30</td><td>59.59<sub>±1.59</sub></td><td>62.29<sub>±1.41</sub></td><td>62.57<sub>±0.80</sub></td><td><b>63.48<sub>±0.16</sub></b><br><span style="color:red;">+6.53%</span></td><td>62.29<sub>±1.22</sub><br><span style="color:red;">-6.08%</span></td>
  </tr>
  <tr>
    <td>AQuA-RAT</td><td></td><td>19.60</td><td>44.36<sub>±3.31</sub></td><td>44.88<sub>±2.19</sub></td><td>45.01<sub>±2.17</sub></td><td>46.19<sub>±1.94</sub></td><td><b>47.24<sub>±1.47</sub></b><br><span style="color:red;">+7.64%</span></td>
  </tr>
  <tr>
    <td>MATH</td><td></td><td>9.40</td><td>19.07<sub>±0.00</sub></td><td>19.61<sub>±0.06</sub></td><td>19.79<sub>±0.12</sub></td><td><b>20.27<sub>±0.42</sub></b><br><span style="color:red;">+11.37%</span></td><td>19.93<sub>±0.20</sub><br><span style="color:red;">+18.21%</span></td>
  </tr>
  <tr>
    <td>CSQA</td><td></td><td>62.00</td><td>72.01<sub>±0.82</sub></td><td>70.71<sub>±1.70</sub></td><td><b>74.12<sub>±0.50</sub></b><br><span style="color:red;">+7.42%</span></td><td>71.71<sub>±0.62</sub></td><td>71.71<sub>±0.62</sub><br><span style="color:red;">+14.44%</span></td>
  </tr>
  <tr>
    <td>OBQA</td><td></td><td>74.40</td><td>79.33<sub>±2.34</sub></td><td>79.73<sub>±2.30</sub></td><td>78.84<sub>±2.06</sub></td><td>77.88<sub>±2.06</sub></td><td><b>79.27<sub>±2.06</sub></b><br><span style="color:red;">+8.02%</span></td>
  </tr>
  <tr>
    <td>STQA</td><td></td><td>55.52</td><td>66.44<sub>±1.55</sub></td><td>62.76<sub>±2.82</sub></td><td>67.47<sub>±2.44</sub></td><td>67.84<sub>±1.39</sub></td><td><b>68.62<sub>±1.29</sub></b><br><span style="color:red;">+7.94%</span></td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Only Answer</th>
    <th colspan="6">BLOOM 3B Performance</th>
  </tr>
  <tr>
    <th>Level 1</th>
    <th>Level 2</th>
    <th>Level 3</th>
    <th>Level 4</th>
    <th>Level 5</th>
    <th>Level 6</th>
  </tr>
  <tr>
    <td>SVAMP</td><td></td><td>5.00</td><td>15.44<sub>±0.51</sub></td><td>23.67<sub>±0.00</sub></td><td>23.11<sub>±1.26</sub></td><td><b>24.00<sub>±0.47</sub></b><br><span style="color:red;">+55.44%</span></td><td>22.22<sub>±0.69</sub></td>
  </tr>
  <tr>
    <td>GSM8K</td><td></td><td>4.60</td><td>18.24<sub>±0.57</sub></td><td>24.31<sub>±1.14</sub></td><td><b>25.81<sub>±0.88</sub></b><br><span style="color:red;">+70.82%</span></td><td>25.77<sub>±0.88</sub></td><td>25.85<sub>±0.86</sub><br><span style="color:red;">+20.85%</span></td>
  </tr>
  <tr>
    <td>AQuA-RAT</td><td></td><td>28.00</td><td>52.41<sub>±0.82</sub></td><td>54.41<sub>±0.26</sub></td><td>54.20<sub>±0.41</sub></td><td>53.85<sub>±0.41</sub></td><td>53.85<sub>±0.41</sub><br><span style="color:red;">+24.58%</span></td>
  </tr>
  <tr>
    <td>MATH</td><td></td><td>4.60</td><td>3.21<sub>±0.40</sub></td><td>2.80<sub>±0.40</sub></td><td>2.33<sub>±0.61</sub></td><td>2.73<sub>±0.41</sub></td><td>3.53<sub>±0.50</sub><br><span style="color:red;">+28.57%</span></td>
  </tr>
  <tr>
    <td>CSQA</td><td></td><td>20.56</td><td><b>57.44<sub>±1.12</sub></b><br><span style="color:red;">+11.38%</span></td><td>55.23<sub>±1.47</sub></td><td>55.42<sub>±0.64</sub></td><td>53.65<sub>±0.22</sub></td><td>52.96<sub>±0.29</sub><br><span style="color:red;">+51.57%</span></td>
  </tr>
  <tr>
    <td>OBQA</td><td></td><td>37.80</td><td>57.21<sub>±1.59</sub></td><td>52.33<sub>±0.61</sub></td><td>54.87<sub>±0.42</sub></td><td>54.64<sub>±0.46</sub></td><td><b>57.47<sub>±0.46</sub></b><br><span style="color:red;">+79.82%</span></td>
  </tr>
  <tr>
    <td>STQA</td><td></td><td>54.14</td><td>58.85<sub>±1.74</sub></td><td><b>61.04<sub>±1.73</sub></b><br><span style="color:red;">+7.32%</span></td><td>60.58<sub>±1.01</sub></td><td>59.89<sub>±1.11</sub></td><td>59.19<sub>±1.21</sub><br><span style="color:red;">+59.08%</span></td>
  </tr>
</table>

Table 1: Performance of Gemma 2B, LLaMA 3.2 3B and BLOOM 3B at six granularity levels. For each dataset, the best performance is boldfaced, and red text shows the relative improvement (%) for highest vs. lowest performance in six levels. Only Answer: Student models are fine-tuned to directly predict answers without CoT.

sequence length without introducing meaningful reasoning steps does not enhance model performance. Furthermore, adding filler content may introduce noise or distract the model, leading to degraded performance (Zhou et al., 2024; Li et al., 2024). This highlights the critical role of granularity, rather than sequence length alone, in driving reasoning efficacy.

<table>
  <tr>
    <th rowspan="2">Granularity</th>
    <th colspan="2">GSMB8K</th>
    <th colspan="2">AQuA-RAT</th>
  </tr>
  <tr>
    <th>Acc</th>
    <th>Seq. Length</th>
    <th>Acc</th>
    <th>Seq. Length</th>
  </tr>
  <tr>
    <td>Level 1</td><td>47.61</td><td>100.93</td><td>40.15</td><td>149.31</td>
  </tr>
  <tr>
    <td>Level 1 Padded</td><td>46.62</td><td>143.43</td><td>37.80</td><td>220.34</td>
  </tr>
  <tr>
    <td>Level 5</td><td><b>52.92</b></td><td>138.16</td><td><b>42.51</b></td><td>216.13</td>
  </tr>
</table>

Table 2: Performance and sequence length of Gemma 2B on GSM8k and AQuA-RAT with varying granularity levels and padding conditions.

Correlation between Granularity and Student Models Figure 3 illustrates the overall relationship between teacher and student performance across datasets at varying granularity levels. We further list the Pearson correlation coefficient (\( r \)) for each student model. The results reveal a clear trend: <i>as student model capacity increases, its performance aligns more closely with the teacher model’s preferences for reasoning granularity</i>

Stronger student models demonstrate significantly higher alignment with the teacher’s optimal granularity, indicating better transferability of reasoning structures. In contrast, weaker student models show a lower correlation, suggesting the limited ability to adapt to the teacher’s granularity preferences. This highlights the importance of tailoring granularity configurations to match the capabilities of student models, rather than relying solely on the teacher’s performance trends.

Conclusion

Increasing CoT granularity does not lead to monotonic improvements in student models. Stronger models benefit from higher granularity, whereas weaker models peak at intermediate levels and struggle with complex tasks. Optimizing granularity based on student capacity, rather than uniformly following the teacher model, is key to maximizing CoT distillation efficiency.

6 Effects of Format

Beyond granularity, the format of reasoning has been believed to influence model performance in prior research. However, SLMs often face limitations in processing complex reasoning structures. This raises a research question: <i>Do these alternative formats improve student model performance, or are their benefits task-specific and limited?</i>
<table>
  <tr>
    <th>Dataset</th>
    <th>CoT Format</th>
    <th>BLOOM 560M</th>
    <th>BLOOM 1.1B</th>
    <th>BLOOM 1.7B</th>
    <th>BLOOM 3B</th>
    <th>Gemma 2B</th>
    <th>LLaMA 3.2 1B</th>
    <th>LLaMA 3.2 3B</th>
  </tr>
  <tr>
    <td rowspan="4">SVAMP</td>
    <td>Original CoT</td>
    <td>5.56<sub>±0.41</sub></td>
    <td>10.67<sub>±1.00</sub></td>
    <td>16.56<sub>±0.51</sub></td>
    <td>22.22<sub>±0.69</sub></td>
    <td>67.11<sub>±1.35</sub></td>
    <td>52.44<sub>±1.71</sub></td>
    <td>69.78<sub>±1.07</sub></td>
  </tr>
  <tr>
    <td>Least-to-most</td>
    <td><b>6.11<sub>±1.07</sub></b>↑</td>
    <td>10.44<sub>±0.69</sub></td>
    <td>14.67<sub>±1.00</sub></td>
    <td>24.00<sub>±1.45</sub></td>
    <td>66.56<sub>±0.69</sub></td>
    <td>54.44<sub>±1.26</sub></td>
    <td><b>75.00<sub>±0.67</sub></b></td>
  </tr>
  <tr>
    <td>RaR</td>
    <td>4.89<sub>±0.19</sub></td>
    <td>9.00<sub>±0.58</sub></td>
    <td>14.11<sub>±0.69</sub></td>
    <td><b>24.22<sub>±1.58</sub></b></td>
    <td>65.67<sub>±1.73</sub></td>
    <td><b>54.56<sub>±0.69</sub></b></td>
    <td>73.89<sub>±1.26</sub></td>
  </tr>
  <tr>
    <td>Symbolic CoT</td>
    <td>5.89<sub>±0.51</sub></td>
    <td>9.00<sub>±0.67</sub></td>
    <td>19.22<sub>±1.07</sub></td>
    <td>64.78<sub>±1.77</sub></td>
    <td>51.78<sub>±1.07</sub></td>
    <td>72.89<sub>±1.50</sub></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">GSM8K</td>
    <td>Original CoT</td>
    <td>8.19<sub>±0.27</sub></td>
    <td>13.09<sub>±0.83</sub></td>
    <td>16.86<sub>±1.25</sub></td>
    <td>22.47<sub>±0.86</sub></td>
    <td>53.42<sub>±0.83</sub></td>
    <td>39.58<sub>±1.04</sub></td>
    <td>62.29<sub>±1.22</sub></td>
  </tr>
  <tr>
    <td>Least-to-most</td>
    <td>7.88<sub>±0.35</sub>↓</td>
    <td><b>13.52<sub>±0.87</sub></b></td>
    <td>15.54<sub>±1.22</sub></td>
    <td>21.86<sub>±0.56</sub></td>
    <td>51.93<sub>±0.71</sub></td>
    <td>39.25<sub>±1.04</sub></td>
    <td>62.07<sub>±0.74</sub></td>
  </tr>
  <tr>
    <td>RaR</td>
    <td>5.89<sub>±0.22</sub></td>
    <td>10.84<sub>±0.40</sub></td>
    <td>13.72<sub>±0.59</sub></td>
    <td>20.02<sub>±0.27</sub></td>
    <td>51.99<sub>±1.22</sub></td>
    <td>38.09<sub>±0.46</sub></td>
    <td><b>63.02<sub>±0.58</sub></b></td>
  </tr>
  <tr>
    <td>Symbolic CoT</td>
    <td>5.94<sub>±0.62</sub></td>
    <td>10.74<sub>±0.64</sub></td>
    <td>13.27<sub>±0.20</sub></td>
    <td>19.33<sub>±0.73</sub></td>
    <td>47.12<sub>±0.39</sub></td>
    <td>34.70<sub>±0.89</sub></td>
    <td>58.94<sub>±0.83</sub></td>
  </tr>
  <tr>
    <td rowspan="4">AQuA</td>
    <td>Original CoT</td>
    <td>18.64<sub>±1.98</sub></td>
    <td>21.92<sub>±1.66</sub></td>
    <td>22.31<sub>±1.38</sub></td>
    <td><b>25.85<sub>±0.45</sub></b></td>
    <td><b>44.88<sub>±0.79</sub></b></td>
    <td><b>33.20<sub>±2.17</sub></b></td>
    <td><b>47.24<sub>±4.77</sub></b></td>
  </tr>
  <tr>
    <td>Least-to-most</td>
    <td>19.69<sub>±1.22</sub></td>
    <td>20.73<sub>±0.91</sub></td>
    <td>23.10<sub>±1.17</sub></td>
    <td>24.41<sub>±1.42</sub></td>
    <td>38.32<sub>±1.86</sub></td>
    <td>28.48<sub>±2.17</sub></td>
    <td>41.60<sub>±3.77</sub></td>
  </tr>
  <tr>
    <td>RaR</td>
    <td><b>21.26<sub>±1.94</sub></b>↑</td>
    <td><b>22.57<sub>±1.79</sub></b></td>
    <td><b>24.38<sub>±1.50</sub></b></td>
    <td>25.07<sub>±1.94</sub></td>
    <td>41.86<sub>±1.16</sub></td>
    <td>31.10<sub>±2.99</sub></td>
    <td>45.93<sub>±4.60</sub></td>
  </tr>
  <tr>
    <td>Symbolic CoT</td>
    <td>16.14<sub>±1.97</sub></td>
    <td>19.16<sub>±2.11</sub></td>
    <td>21.00<sub>±1.38</sub></td>
    <td>20.87<sub>±2.36</sub></td>
    <td>40.94<sub>±0.81</sub></td>
    <td>28.08<sub>±3.66</sub></td>
    <td>42.13<sub>±3.81</sub></td>
  </tr>
  <tr>
    <td rowspan="4">OBQA</td>
    <td>Original CoT</td>
    <td>36.73<sub>±7.76</sub></td>
    <td>46.07<sub>±2.25</sub></td>
    <td>48.00<sub>±4.40</sub></td>
    <td>54.87<sub>±2.02</sub></td>
    <td>69.93<sub>±1.36</sub></td>
    <td>63.60<sub>±2.12</sub></td>
    <td>78.80<sub>±0.80</sub></td>
  </tr>
  <tr>
    <td>Least-to-most</td>
    <td>31.40<sub>±7.44</sub></td>
    <td>43.33<sub>±2.02</sub></td>
    <td>45.53<sub>±3.39</sub></td>
    <td>53.20<sub>±2.50</sub></td>
    <td>68.27<sub>±1.03</sub></td>
    <td>62.80<sub>±2.23</sub></td>
    <td>78.33<sub>±1.62</sub></td>
  </tr>
  <tr>
    <td>RaR</td>
    <td><b>40.47<sub>±1.68</sub></b>↑</td>
    <td><b>47.47<sub>±2.23</sub></b></td>
    <td><b>49.87<sub>±1.42</sub></b></td>
    <td><b>56.40<sub>±1.97</sub></b></td>
    <td><b>72.73<sub>±2.39</sub></b></td>
    <td><b>64.40<sub>±2.78</sub></b></td>
    <td><b>82.00<sub>±0.20</sub></b></td>
  </tr>
  <tr>
    <td>Symbolic CoT</td>
    <td>31.67<sub>±1.36</sub></td>
    <td>35.13<sub>±2.25</sub></td>
    <td>37.73<sub>±3.25</sub></td>
    <td>41.13<sub>±1.81</sub></td>
    <td>61.80<sub>±0.92</sub></td>
    <td>52.47<sub>±0.70</sub></td>
    <td>72.13<sub>±0.64</sub></td>
  </tr>
</table>

Table 3: Performance of student models with different CoT formats. For each dataset, the best performance is boldfaced, and arrows show that the performance is increased (↑) or decreased (↓) over original CoT.

![Scatter plots of teacher model (GPT-4o, x-axis) vs. student accuracy (y-axis) across datasets and granularity levels. Each point marker represents a specific dataset.](page_1012_1012_420_312.png)

Figure 3: Scatter plots of teacher model (GPT-4o, x-axis) vs. student accuracy (y-axis) across datasets and granularity levels. Each point marker represents a specific dataset.

Choice of Reasoning Formats In this section, we systematically evaluate the impact of alternative reasoning structures on student model performance. Since student models tend to perform stably at intermediate granularity levels, we let GPT-4o modify the format of the original CoT without changing the granularity (More details can be seen in Appendix D). We compare the original CoT format with three alternative structures:

• Least-to-most (Zhou et al. 2023): A reasoning approach that decomposes a complex problem into a sequence of sub problems. Least-to-most excels in systematically breaking down problems into manageable parts to facilitate understanding and solution synthesis.

• Rephrase and Respond (RaR) (Deng et al., 2024): A method where questions are rephrased to reduce ambiguity before answering, enabling iterative clarification and improving the LLM’s ability to respond accurately to nuanced queries.

• Symbolic CoT (Xu et al., 2024a): A reasoning structure that combines symbolic logic and CoT prompting, translating natural language into symbolic expressions for step-by-step logical deduction, enhancing faithfulness and flexibility in problem-solving.

Our results, summarized in Table 3, highlight a clear trend: the original CoT format often outperforms more complex or modified structures, primarily due to its simplicity and adaptability. This contrasts with previous findings in LLMs, where these alternative formats frequently yield improvements. For SLMs, however, the added complexity of alternative formats generally increases cognitive load and hardly improve the performance.

Task-Specific Gains While most tasks favor the original CoT format, certain alternative structures offer measurable benefits for specific scenarios. For example, RaR improves commonsense reasoning tasks by reducing ambiguity and enabling iterative clarification. Least-to-most sometimes excels in mathematical reasoning by breaking problems into logical steps or symbolic expressions. 2

2A possible reason for the suboptimal performance of Symbolic CoT is analyzed with examples in the Appendix G.
Model-Specific Trends  Stronger student models, such as LLaMA 3.2 3B, show improved performance under alternative formats, leveraging structural cues to refine problem-solving processes. However, these improvements are tied to specific tasks and do not generalize across all datasets.

Overall, our findings indicate that while CoT formats can occasionally enhance performance, their benefits are often task-dependent and come at the cost of introducing additional tokens. Moreover, SLMs have relatively limited pretraining corpora, which likely contain fewer instances of these reasoning formats. As a result, weaker models struggle to effectively learn and utilize them, making it even harder for CoT format variations to yield consistent improvements. Given these observations, we argue that adjusting CoT formats alone may not be the most effective approach for improving SLM performance. This contrasts with the consistent impact of granularity, as highlighted in Section 5, suggesting that focusing on granularity is a more effective strategy than altering CoT formats.

Conclusion
While alternative CoT formats sometimes offer some task-specific benefits, the original CoT format from teacher models often remains the most effective for general-purpose SLM training.

7 Effects of Teacher Model
In CoT distillation, teacher models serve as the source of CoT reasoning annotations for training student models. Prior research of KD assumes that teacher models with better performance naturally lead to better student models (Zong et al., 2023). This assumption stems from the belief that higher-performing teachers generate more accurate reasoning steps and answers, which, when distilled into student models, enhance their capabilities. However, this assumption may not hold universally as SLMs might have limited capacity to replicate the reasoning complexity of strong teachers.

In this section, we analyze the performance of student models under four teachers, GPT-4o, LLaMA 3 70B, Gemini-1.5-Flash and the human expert. We aim to determine whether the choice of teacher affects the ability of student models to effectively distill and replicate CoT reasoning.

Is Higher Teacher Accuracy Always Better?
We first investigate whether a higher teacher accuracy directly translates into improved student performance. As shown in Figure 4, we selected the best-performing student model for each dataset under different teacher models’ CoT supervision. It can be observed that points closer to the right side of the x-axis are not always positioned near the top of the y-axis. This indicates that, contrary to intuitive expectations, while a reasonably accurate teacher can effectively impart essential reasoning patterns, excessively high teacher accuracy does not always yield proportional improvements in student accuracy. Teacher accuracy alone is not the determining factor for student performance, which aligns with our findings in Section 5.

Moreover, we do not observe a significant preference pattern within the same model family. However, we find that stronger student models tend to benefit more significantly when trained under the guidance of stronger teacher models. This suggests that the trade-off between teacher model capability and computational cost should be carefully adjusted based on the target student model’s capacity. For simpler tasks, a less advanced teacher model is often sufficient, producing results comparable to those obtained from more powerful, computationally expensive teachers.

Human vs LLM: Task-Specific Effectiveness
As seen in Figure 5, in mathematical reasoning tasks, student models achieve higher accuracy when fine-tuned on LLM-generated CoTs compared to human annotations, although the accuracy of the teacher model itself performs poorly

![Scatter plots of teacher (x-axis) vs. student model accuracy (y-axis) across datasets. GPT refers to GPT-4o, LLaMA refers to LLaMA 3 70B, and Gemini refers to Gemini-1.5-Flash.](page_1016_180_393_312.png)

Figure 4: Scatter plots of teacher (x-axis) vs. student model accuracy (y-axis) across datasets. GPT refers to GPT-4o, LLaMA refers to LLaMA 3 70B, and Gemini refers to Gemini-1.5-Flash.
Figure 5: Student model performance across different teacher models. Each bar represents the average accuracy of a specific student model trained on CoT from different teacher models.

on difficult mathematical datasets compared with human-labeled data. Conversely, for commonsense reasoning tasks like StrategyQA, human-annotated CoTs dramatically improve student model performance. This phenomenon arises because LLMs generate structured and detailed reasoning chains that closely align with the symbolic and procedural nature of mathematical tasks. In contrast, human-annotated CoTs often lack the rigorous step-by-step structure required for effective mathematical reasoning. However, Human annotations excel at capturing nuanced contextual understanding, creative inferences, and interpretive reasoning, which are crucial for tasks involving ambiguous or open-ended questions. These findings underscore the importance of selecting CoT sources based on task characteristics rather than assuming a universal superiority of either LLMs or human annotations. Full results can be found in Appendix H.

The Matthew Effect in SLMs We explore the relationship between student model capacity and the benefits gained from CoT distillation, shedding light on the uneven distribution of performance improvements across models of varying capabilities. Figure 6 presents two heatmaps comparing student model performance before and after CoT distillation. The results reveal a Matthew Effect: stronger student models achieve greater performance gains from CoT distillation than weaker models, demonstrating their potential ability to leverage detailed reasoning steps. This phenomenon aligns with Vygotsky’s Zone of Proximal Development (ZPD) (Vygotsky, 1978), where weaker student models have a narrower ZPD, limiting their ability to absorb complex CoT reasoning. If reasoning complexity is too high relative to a model’s ZPD, it may fail to extract useful patterns, limiting the effectiveness of CoT distillation. In contrast, stronger models have a wider ZPD, enabling them to integrate and generalize from multi-step reasoning. CoT distillation provides gains on more challenging ones, where their capacity allows them to fully leverage structured reasoning. This highlights the need for adaptive CoT supervision, where reasoning depth is modulated based on the student’s ability to process and learn from it.

Figure 6: The Only Answer heatmap represents the baseline accuracy of student models without reasoning supervision, while the Average Performance heatmap shows the average accuracy of student models trained on CoT from ChatGPT-4o.

Conclusion

The assumption that a better teacher always produces a better student does not universally hold for SLMs. Stronger student models benefit more from advanced teacher models. Teacher choice should be task-specific: LLM-generated CoTs improve mathematical reasoning, while human annotations excel in commonsense reasoning.

8 Conclusion

This study systematically examined key factors influencing CoT distillation in SLMs, including teacher selection, granularity, and format. First, We found that finer-grained CoT benefits stronger SLMs, and weaker models perform better with simpler annotations. Then, while CoT format significantly impacts LLMs, its effect on SLMs is more subtle. Importantly, better teacher models do not always yield better students, as the effectiveness of CoT distillation depends on a model’s ability to absorb reasoning complexity within its ZPD. Notably, human-annotated CoTs underperform on mathematical tasks but can surpass LLM-generated CoTs
in certain commonsense reasoning tasks. Overall, CoT distillation proves more effective for stronger SLMs and complex tasks, emphasizing the need for tailored granularity and teacher selection strategies to optimize reasoning performance in resource-constrained settings.

Limitations

Despite the promising results of our study, several limitations must be acknowledged. First, during data generation and testing, some tasks triggered safety concerns in the models, causing them to refuse to generate CoTs. In these cases, we resorted to directly using the provided answers for fine-tuning, which may have constrained the diversity and quality of the reasoning chains, potentially affecting the distillation outcomes. Second, the ability of teacher models to generate CoTs is inherently tied to their reasoning capabilities. For certain tasks, teacher models were unable to reverse-engineer plausible CoTs from the given answers due to their limited capabilities, resulting in incomplete or suboptimal reasoning chains. Lastly, this study did not focus on exploring novel KD techniques but instead aimed to systematically analyze the effects of existing approaches on CoT granularity, format, and teacher selection. We recognize that alternative training strategies may influence distillation outcomes. These limitations highlight the need for further exploration of CoT generation methods and distillation techniques tailored to task complexity and student capabilities. Future work may benefit from integrating curriculum-based strategies or lightweight multi-stage distillation protocols to enhance the fidelity and generalizability of SLMs.

Ethics Statement

The datasets used in our experiment are publicly released and labeled through interaction with humans in English. In this process, user privacy is protected, and no personal information is contained in the dataset. The scientific artifacts that we used are available for research with permissive licenses. And the use of these artifacts in this paper is consistent with their intended use. Therefore, we believe that our research work meets the ethics of ACL.

Acknowledgement

We thank EIT and IDT High Performance Computing Center for providing computational resources for this project. This work was supported by the Research Grants Council of Hong Kong (PolyU/15209724), Innovation Consortium Program for Green and Efficient Intelligent Appliance of Ningbo City (No.2022H002), Young Tech Innovation Leading Talent Program of Ningbo City (No.2023QL008) and 2035 Key Research and Development Program of Ningbo City under Grant No.2024Z127.

References

Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021. Explanations for CommonsenseQA: New Dataset and Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3050–3065, Online. Association for Computational Linguistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucang Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuan Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuteng Pan, S. S. Li, Shuang Zhou, Shaoqing
Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yu-jia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhenhao Wen, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. 2025. Deepseek-r1: Incen-tivizing reasoning capability in llms via reinforce-ment learning. Preprint, arXiv:2501.12948.

Yihe Deng, Weitong Zhang, Zixiang Chen, and Quan-quan Gu. 2024. Rephrase and respond: Let large language models ask better questions for themselves. Preprint, arXiv:2311.04205.

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Preprint, arXiv:2101.02235.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS.

Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. Large language models are reasoning teachers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852–14882, Toronto, Canada. Association for Computational Linguistics.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.

Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. 2024. The impact of reasoning step length on large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1830–1842, Bangkok, Thailand. Association for Computational Linguistics.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361.

Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu-taka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.

Shanu Kumar, Saish Mendke, Karody Lubna Abdul Rahman, Santosh Kurasa, Parag Agrawal, and Sandipan Dandapat. 2024. Enhancing zero-shot chain of thought prompting via uncertainty-guided strategy selection. arXiv preprint arXiv:2412.00353.

Ming Li, Yanhong Li, and Tianyi Zhou. 2024. What happened in llms layers when trained for fast vs. slow thinking: A gradient perspective. Preprint, arXiv:2410.23743.

Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167, Vancouver, Canada. Association for Computational Linguistics.

Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2023. Teaching small language models to reason. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1773–1781, Toronto, Canada. Association for Computational Linguistics.

William Merrill and Ashish Sabharwal. 2024. The expressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations.

Meta. 2024a. Introducing meta Llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3/.

Meta. 2024b. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. Preprint, arXiv:1809.02789.
Sania Nayab, Giulio Rossolini, Giorgio Buttazzo, Nico-lamaria Manes, and Fabrizio Giacomelli. 2024. Concise thoughts: Impact of output length on llm reasoning and cost. Preprint, arXiv:2407.19825.

OpenAI. 2024a. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/.

OpenAI. 2024b. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/.

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? Preprint, arXiv:2103.07191.

Akshara Prabhakar, Thomas L. Griffiths, and R. Thomas McCoy. 2024. Deciphering the factors influencing the efficacy of chain-of-thought: Probability, memorization, and noisy reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3710–3724, Miami, Florida, USA. Association for Computational Linguistics.

Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. 2024. Mutual reasoning makes smaller llms stronger problem-solvers. Preprint, arXiv:2408.06195.

Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. Distilling reasoning capabilities into smaller language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7059–7073, Toronto, Canada. Association for Computational Linguistics.

Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Schoelkopf, and Mrinmaya Sachan. 2023. A causal framework to quantify the robustness of mathematical reasoning with language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 545–561, Toronto, Canada. Association for Computational Linguistics.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.

Gemma Team, Petko Georgiev, Ying Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024a. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Riviere, Mihir Sanjay Kale, Juliette Love, et al. 2024b. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295.

Lev S Vygotsky. 1978. Mind in society: The development of higher psychological processes, volume 86. Harvard university press.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichtier, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.

Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. 2024a. Faithful logical reasoning via symbolic chain-of-thought. Preprint, arXiv:2405.18357.

Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024b. A survey on knowledge distillation of large language models. Preprint, arXiv:2402.13116.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems.

Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. 2024. Distilling system 2 into system 1. Preprint, arXiv:2407.06023.

Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wen-hao Huang, Huan Sun, Yu Su, and Wenhui Chen. 2024. MAmmoTH: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations.

Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2024. A survey of large language models. Preprint, arXiv:2303.18223.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyuan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand. Association for Computational Linguistics.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations.

Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zeng-mao Wang, and Bo Han. 2024. Can language models perform robust reasoning in chain-of-thought prompting with noisy rationales? In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Martin Zong, Zengyu Qiu, Xinzhu Ma, Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. 2023. Better teacher better student: Dynamic prior knowledge for knowledge distillation. In The Eleventh International Conference on Learning Representations.
Appendix

A  Overview of Training and Test Datasets

For our experiments, we used three models (Llama3 70B, Gemini-1.5-Flash, GPT-4o) on multiple existing datasets, including mathematical reasoning datasets (SVAMP, GSM8K, AQuA-RAT, MATH) and commonsense reasoning datasets (OpenBookQA, CommonsenseQA, and StrategyQA) to generate CoT outputs. Table 4 shows the overview of the training and test datasets (Yue et al., 2024). Table 5 shows some examples of our datasets.

<table>
  <tr>
    <th>Training Dataset</th>
    <th colspan="2">Samples</th>
    <th>Fields</th>
    <th>Human Annotation</th>
  </tr>
  <tr>
    <th></th>
    <th>Training</th>
    <th>Testing</th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>SVAMP</td>
    <td>700</td>
    <td>300</td>
    <td>Arithmetic problems</td>
    <td>Yes</td>
  </tr>
  <tr>
    <td>GSM8K</td>
    <td>7.4k</td>
    <td>1.3k</td>
    <td>Grade-school math</td>
    <td>Yes</td>
  </tr>
  <tr>
    <td>AQuA-RAT</td>
    <td>6.1k</td>
    <td>254</td>
    <td>Algebraic reasoning, multi-step</td>
    <td>Yes</td>
  </tr>
  <tr>
    <td>Math</td>
    <td>1.3k</td>
    <td>500</td>
    <td>Pre-Algebra, Algebra, Counting & Probability, Number Theory</td>
    <td>Yes</td>
  </tr>
  <tr>
    <td>CommonsenseQA</td>
    <td>9.7k</td>
    <td>1.2k</td>
    <td>Commonsense knowledge</td>
    <td>Yes</td>
  </tr>
  <tr>
    <td>OpenBookQA</td>
    <td>4.9k</td>
    <td>500</td>
    <td>Domain-specific knowledge</td>
    <td>No</td>
  </tr>
  <tr>
    <td>StrategyQA</td>
    <td>2k</td>
    <td>290</td>
    <td>Multi-step reasoning</td>
    <td>Yes</td>
  </tr>
</table>

Table 4: Overview of Training and Test Datasets.

<table>
  <tr>
    <th>Dataset</th>
    <th>Problem</th>
    <th>Characteristics</th>
  </tr>
  <tr>
    <td>SVAMP</td>
    <td>There are 87 oranges and 290 bananas in Philip’s collection. If the bananas are organized into 2 groups and oranges are organized into 93 groups How big is each group of bananas?</td>
    <td>290.0 / 2.0 = 145.0. The answer is 145.0.</td>
  </tr>
  <tr>
    <td>GSM8K</td>
    <td>Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?</td>
    <td>Natalia sold 48/2 = «48/2=24»24 clips in May. Natalia sold 48+24 = «48+24=72»72 clips altogether in April and May. 72</td>
  </tr>
  <tr>
    <td>AQuA-RAT</td>
    <td>A man can swim in still water at 7.5 km/h, but takes twice as long to swim upstream than downstream. The speed of the stream is? Answer Choices: (A) 3 (B) 2.5 (C) 2.25 (D) 1.5 (E) 4</td>
    <td>M = 7.5 S = x DS = 7.5 + x US = 7.5 + x 7.5 + x = (7.5 - x)2 7.5 + x = 15-2x 3x = 7.5 x = 2.5 Answer: C</td>
  </tr>
  <tr>
    <td>Math</td>
    <td>Find the sum of all positive divisors of 50 that are also divisors of 15.</td>
    <td>The positive factors of 50 are 1, 2, 5, 10, 25, 50. Of these, only 1 and 5 divide 15. Their sum is 1+5 = 6.</td>
  </tr>
  <tr>
    <td>CommonsenseQA</td>
    <td>Bill did not abandon the fight, but did what to the enemy? Answer choices: A: arrogate, B: retain, C: embrace, D: smile, E: engage</td>
    <td>Bill engaged in a fight with enemy. Other options are not a type of fights one takes with enemy. The answer is E.</td>
  </tr>
  <tr>
    <td>StrategyQA</td>
    <td>Are more people today related to Genghis Khan than Julius Caesar?</td>
    <td>Julius Caesar had three children. Genghis Khan had sixteen children. Modern geneticists have determined that out of every 200 men today has DNA that can be traced to Genghis Khan. The answer is True.</td>
  </tr>
</table>

Table 5: Examples of Human Annotation for All Datasets.
B  Training setup

Our experiment uses the LLaMA-Factory framework (Zheng et al., 2024) to fine-tune models, and the training parameters are as follows:

<table>
  <tr>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>3e-5</td>
  </tr>
  <tr>
    <td>Num Train Epochs</td>
    <td>3</td>
  </tr>
  <tr>
    <td>LR Scheduler</td>
    <td>Cosine</td>
  </tr>
  <tr>
    <td>Max Grad Norm</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>Template</td>
    <td>gemma/alpaca/llama3</td>
  </tr>
</table>

Table 6: Configuration for training parameters.

C  Different CoT Granularity Dataset Collection

C.1  Workflow

The granularity dataset processing steps are detailed below:
  1. 0-Shot Example Generation: The same question is first provided to the three models using a 0-shot prompt. The models generate a single 1-shot example (including both a question and a corresponding output) as the output. This step ensures that the models first generate a baseline example. The generated example serves as a guide for subsequent responses.
  2. Input Construction: Each question is provided to the models, along with its corresponding ground-truth answer from the original dataset and the generated 1-shot example (from Step 1). Including the 1-shot example in the input establishes a reference point for the model, enhancing coherence and quality of generated outputs.
  3. Generation with Multi-Granularity Outputs: Using the constructed input (original question + ground truth + 1-shot example), all three teacher models are prompted to generate answers at multiple granularity levels (\( G = \{g_1, g_2, \ldots, g_6\} \)). These levels range from concise summaries to highly detailed, step-by-step reasoning. By solving each question across six levels of granularity, this step systematically evaluates the models’ ability to adapt their reasoning to different levels of abstraction.
  4. Ranking and Alignment: The generated outputs are sorted to align with the original dataset’s order, ensuring consistency and enabling a systematic evaluation of the results. Sorting the generated outputs ensures that the evaluation is systematic and comparable against the original dataset.

Why do we use 1-shot example in the prompt:
We have decided to incorporate a 1-shot example into the prompt instead of using a 0-shot prompt, based on our trial-and-error findings.

Our initial attempt used a forward-generation approach, where we prompted the model to produce the most succinct response and then enrich it level by level. However, we encountered significant challenges with this approach. The model struggled to demonstrate consistent incremental increases in granularity, as the initial requirement for conciseness often constrained its reasoning and led to inaccuracies or incomplete answers. The model’s inability to build upon a succinct base made this method unsuitable for achieving the desired level of granularity.

To address this, we reversed the approach by asking the model to provide the most elaborate response, intending to progressively reduce the level of detail in subsequent steps. While this method initially produced more detailed outputs, the responses often lacked sufficient depth and structure to support multiple rounds of granularity reduction. As a result, achieving consistent decreases in detail also proved to be a challenge.

These findings highlighted the need for a more structured and balanced approach. We identified that including a 1-shot example in the prompt could effectively guide the model to produce outputs with consistent and balanced granularity across levels. A well-designed 1-shot example helps the model
demonstrate high-quality reasoning even in concise answers, ensuring alignment with task requirements regardless of the level of detail. It also provides a clear reference for maintaining consistency when transitioning between levels of granularity.

In summary, 1-shot prompts strike an effective balance between flexibility and structure, enabling the model to generalize across tasks while maintaining coherence and consistency. This approach significantly enhances the model’s ability to generate high-quality training samples with varying levels of reasoning granularity.

As a result, we have decided to generate a 1-shot example to include in the prompt. The first prompt will be used to create the 1-shot example, and the second prompt will leverage it for data generation.

C.2 Prompts

<table>
  <tr>
    <th>CoT Prompt Template</th>
  </tr>
  <tr>
    <td>You are a math teacher. Please think step by step for the following question.<br>
Output the result strictly in the following format. DO NOT generate any other explanations.<br>
The generated answer must be consistent with the given answer.<br>
Question: "&lt;your question&gt;" Answer: "&lt;original answer&gt;"<br>
The output format should be as follows:<br>
"instruction": "&lt;your question&gt;", "output": "&lt;Solution Path&gt; The answer is &lt;answer&gt;"<br>
Here is the example:<br>
&lt;example&gt;</td>
  </tr>
</table>

Figure 7: Prompt for generating CoT dataset.
Synthetic 1-shot CoT Example Based on Granularity Prompt Template

You are a math teacher. Please think step by step for the following questions in six different Granularity levels.

Ensure that the explanations become progressively more detailed as the Granularity increases. The difference in the number of words between each Granularity should be as large as possible.

The generated answer must be consistent with the given answer. DO NOT generate any other explanations. Output the result strictly in the following format:

"instruction": "<your question>", "output": "<Solution Path>\n The answer is <answer>"

Granularity definitions:
- Level 1: Provide the most essential steps to reach the answer, minimizing explanations and focusing on the direct path to the solution.
- Level 2: Provide the essential steps required to reach the answer, including some intermediate calculations. It should be more detailed than level 1 but shorter than level 3.
- Level 3: Provide a detailed breakdown that includes all necessary calculations and explanations but shorter and less detailed than level 4. Ensure it is more detailed than level 2.
- Level 4: Provide a very detailed breakdown that includes all necessary calculations and explanations but avoids extra clarifications that would belong to level 5. It should be more detailed than level 3.
- Level 5: Provide an extremely detailed breakdown, covering almost all aspects and sub-steps, but shorter and less detailed than level 6. It should include additional clarifications and checks beyond level 4.
- Level 6: Provide the most detailed and comprehensive breakdown, covering all possible aspects and sub-steps. Ensure this explanation is significantly longer and more detailed than level 5.

The output format should be as follows:
{ "Granularity": "level 1", "instruction": "<your question>", "output": "Explanation: <Explanation>\n The answer is <answer>" }
{ "Granularity": "level 2", "instruction": "<your question>", "output": "Explanation: <Explanation>\n The answer is <answer>" }
{ "Granularity": "level 3", "instruction": "<your question>", "output": "Explanation: <Explanation>\n The answer is <answer>" }
{ "Granularity": "level 4", "instruction": "<your question>", "output": "Explanation: <Explanation>\n The answer is <answer>" }
{ "Granularity": "level 5", "instruction": "<your question>", "output": "Explanation: <Explanation>\n The answer is <answer>" }
{ "Granularity": "level 6", "instruction": "<your question>", "output": "Explanation: <Explanation>\n The answer is <answer>" }

Figure 8: Synthetic 1-shot CoT prompt example based on granularity levels, demonstrating progressively detailed reasoning steps from Level 1 to Level 6 explanations.
C.3  Granularity case study

Question: At the beginning of every period of British Literature, Mrs. Crabapple picks a random student to receive a crabapple as a gift. Given that there are 11 students in her class and her class meets four times a week, how many different sequences of crabapple recipients are possible in a week?

C.3.1  GPT-4o

<table>
  <tr>
    <th>Granularity</th>
    <th>Generated Output</th>
  </tr>
  <tr>
    <td>Level 1</td>
    <td>
      1. Number of students = 11<br>
      2. Class meets per week = 4 times<br>
      3. Each day, 1 student can receive a crabapple<br>
      4. Choices per day = 11<br>
      5. Total choices = \( 11^4 = 14641 \)<br>
      The answer is 14,641
    </td>
  </tr>
  <tr>
    <td>Level 2</td>
    <td>
      1. Each period is an independent random event where 1 of 11 students is chosen.<br>
      2. There are 4 periods per week.<br>
      3. Each of the 4 periods has 11 possible choices.<br>
      4. The total number of sequences is \( 11^4 \).<br>
      5. Calculate \( 11^4 \): \( 11 \times 11 = 121, 121 \times 11 = 1331, 1331 \times 11 = 14641 \)<br>
      The answer is 14,641
    </td>
  </tr>
  <tr>
    <td>Level 3</td>
    <td>
      1. Each time Mrs. Crabapple picks a student, she has 11 choices.<br>
      2. Since the class meets four times a week, there are four independent selections, each with 11 choices.<br>
      3. The number of different sequences of crabapple recipients is calculated by raising the number of choices to the power corresponding to the number of selections: \( 11^4 \).<br>
      4. Calculate this value: \( 11 \times 11 \times 11 \times 11 = 14,641 \).<br>
      The answer is 14,641
    </td>
  </tr>
  <tr>
    <td>Level 4</td>
    <td>
      1. Each class session, Mrs. Crabapple picks one of the 11 students.<br>
      2. Multiply the number of choices per session: 11 choices per session.<br>
      3. Since the class meets four times a week, we need to raise the number of choices to the power of the number of sessions: \( 11^4 \).<br>
      4. Calculate \( 11^4 \): \( 11 \times 11 \times 11 \times 11 \).<br>
      5. Perform the multiplication step-by-step: \( 11 \times 11 = 121, 121 \times 11 = 1331, 1331 \times 11 = 14,641 \).<br>
      6. Review all steps for consistency: 11 choices per session, 4 sessions total, \( 11^4 \) computation confirms 14,641 possibilities.<br>
      The answer is 14,641
    </td>
  </tr>
  <tr>
    <td>Level 5</td>
    <td>
      1. Identify key variables: 11 students, 4 class meetings.<br>
      2. Recognize the problem’s nature: finding the number of different sequences (order matters) of recipients over 4 periods.<br>
      3. Analyze each period’s choice: Each of the 4 periods can be filled by any one of the 11 students.<br>
      4. Calculate total sequences using the multiplication principle: Each of 4 periods has 11 possible choices.<br>
      5. Compute total different sequences: \( 11 \times 11 \times 11 \times 11 = 11^4 \).<br>
      6. Evaluate numerical result: \( 11^4 = 11 \times 11 \times 11 \times 11 = 14,641 \).<br>
      The answer is 14,641
    </td>
  </tr>
  <tr>
    <td>Level 6</td>
    <td>
      1. Determine possible choices per class: Mrs. Crabapple can choose one out of 11 students each time.<br>
      2. Confirm weekly meetings: The class has four periods a week.<br>
      3. Establish sequence combination formula: For each period, there are 11 choices, resulting in 11 choices for the first period, 11 for the second, 11 for the third, and 11 for the fourth.<br>
      4. Calculate total sequences: Multiply available choices for each period to find total combinations: \( 11 \times 11 \times 11 \times 11 = 11^4 \).<br>
      5. Compute final result: Raising 11 to the power of 4, we get \( 11^4 = 14,641 \) sequences.<br>
      The answer is 14,641
    </td>
  </tr>
</table>

Table 7: Comparison of Different CoT granularity for the same problem generated by GPT-4o.
<table>
  <tr>
    <th>Granularity</th>
    <th>Generated Output</th>
  </tr>
  <tr>
    <td>Level 1</td>
    <td>The number of possible recipients for each class is 11.<br>Since class meets 4 times a week, the total number of sequences is \(11*11*11*11 = 11^4 = 14641\).<br>The answer is 14,641</td>
  </tr>
  <tr>
    <td>Level 2</td>
    <td>First, we need to find the number of possible recipients for each class period, which is 11.<br>Then, we need to find the number of possible sequences for the week, which is the number of possible recipients for each class period raised to the power of the number of class periods in a week: \(11^4 = 14641\).<br>The answer is 14,641</td>
  </tr>
  <tr>
    <td>Level 3</td>
    <td>Step 1: Determine the number of students in the class. There are 11 students in Mrs. Crabapple's British Literature class.<br>Step 2: Determine the number of times the class meets in a week. The class meets 4 times a week.<br>Step 3: Calculate the number of different sequences of crabapple recipients possible in a week. Since there are 11 students and the class meets 4 times a week, we can calculate the number of different sequences by raising 11 to the power of 4, which represents the number of choices for each of the 4 days. \(11^4 = 14641\)<br>The answer is 14,641</td>
  </tr>
  <tr>
    <td>Level 4</td>
    <td>Step 1: Determine the number of students in the class. There are 11 students in Mrs. Crabapple's British Literature class.<br>Step 2: Determine the number of times the class meets in a week. The class meets 4 times a week.<br>Step 3: Calculate the number of different sequences of crabapple recipients possible in a week. Since Mrs. Crabapple picks a random student each time, the number of sequences is equal to the number of permutations of 11 students taken 4 at a time, allowing for repetition (since the same student can be picked more than once). This is calculated as \(11^4\).<br>Step 4: Calculate \(11^4\). \(11^4 = 11 * 11 * 11 * 11 = 14641\).<br>The answer is 14,641</td>
  </tr>
  <tr>
    <td>Level 5</td>
    <td>To solve this problem, we need to use the concept of permutations.<br>Since Mrs. Crabapple picks a random student each time, the order of selection matters. We have 11 students to choose from for each of the 4 periods in a week.<br>For the first period, there are 11 possible choices. For the second period, there are again 11 possible choices, and so on. Since the choices are independent of each other, we can multiply the number of choices for each period to get the total number of different sequences. This gives us (11 times 11 times 11 times 11 = \(11^4 = 14641\)).<br>Therefore, there are 14641 different sequences of crabapple recipients possible in a week.<br>The answer is 14,641</td>
  </tr>
  <tr>
    <td>Level 6</td>
    <td>To solve this problem, we need to use the concept of permutations.<br>Since Mrs. Crabapple picks a random student to receive a crabapple as a gift in each class, and there are 11 students in her class, there are 11 possible choices for the first class. For the second class, there are also 11 possible choices, and so on.<br>Since her class meets four times a week, we need to find the total number of different sequences of crabapple recipients for four classes. This can be calculated by multiplying the number of choices for each class together.<br>So, the total number of different sequences is (11 times 11 times 11 times 11 = \(11^4\)). To calculate (\(11^4\)), we can multiply 11 by itself four times.<br>This gives us 11 times 11 = 121, 121 times 11 = 1331, and 1331 times 11 = 14641. Therefore, the total number of different sequences of crabapple recipients possible in a week is 14641.<br>The answer is 14,641</td>
  </tr>
</table>

Table 8: Comparison of Different CoT granularity for the same problem generated by LLaMA 3 70B.
<table>
  <tr>
    <th>Granularity</th>
    <th>Generated Output</th>
  </tr>
  <tr>
    <td>Level 1</td>
    <td>There are 11 choices for the recipient of the crabapple in each of the four class periods.<br>The number of possible sequences is therefore \( 11 * 11 * 11 * 11 = 11^4 \). \( 11^4 = 14641 \).<br>The answer is 14,641</td>
  </tr>
  <tr>
    <td>Level 2</td>
    <td>There are 11 students in the class.<br>Mrs. Crabapple picks a student each of the four class periods. The number of choices for the first period is 11. The number of choices for the second period is 11. The number of choices for the third period is 11. The number of choices for the fourth period is 11.<br>To find the total number of different sequences, we multiply the number of choices for each period together: \( 11 * 11 * 11 * 11 = 11^4 = 14641 \).<br>The answer is 14,641</td>
  </tr>
  <tr>
    <td>Level 3</td>
    <td>There are 11 students in the class.<br>Mrs. Crabapple picks a student each of the four class periods. The number of choices for the first period is 11. The number of choices for the second period is also 11 (since the same student can receive a crabapple multiple times). Similarly, there are 11 choices for the third period and 11 choices for the fourth period.<br>To find the total number of different sequences, we multiply the number of choices for each period together. This is 11 times 11 times 11 times 11 = \( 11^4 \). Calculating \( 11^4 \) gives us 14641.<br>The answer is 14,641</td>
  </tr>
  <tr>
    <td>Level 4</td>
    <td>There are 11 students in the class, and the class meets 4 times a week.<br>Each time Mrs. Crabapple picks a student, there are 11 choices.<br>Since there are four periods, we multiply the number of choices for each period together to find the total number of different sequences. This is \( 11 * 11 * 11 * 11 = 11^4 \). Calculating \( 11^4 \): \( 11 * 11 = 121; 121 * 11 = 1331; 1331 * 11 = 14641 \).<br>The answer is 14,641</td>
  </tr>
  <tr>
    <td>Level 5</td>
    <td>There are 11 students in the class, and the class meets four times a week.<br>Each time the class meets, Mrs. Crabapple picks one student to receive a crabapple. The number of choices for the first period is 11. The number of choices for the second period is also 11 (since the same student can receive a crabapple again). Similarly, there are 11 choices for the third period and 11 choices for the fourth period.<br>To find the total number of different sequences of crabapple recipients, we multiply the number of choices for each period together. This is because each choice is independent of the others. Therefore, the total number of sequences is \( 11 * 11 * 11 * 11 = 11^4 \).<br>Calculating \( 11^4 \): \( 11 * 11 = 121; 121 * 11 = 1331; 1331 * 11 = 14641 \).<br>The answer is 14,641</td>
  </tr>
  <tr>
    <td>Level 6</td>
    <td>There are 11 students in the class, and the class meets four times a week.<br>Each time the class meets, Mrs. Crabapple picks one student to receive a crabapple. The number of ways to choose a student for the first period is 11. The number of ways to choose a student for the second period is also 11. Similarly, there are 11 ways to choose a student for the third period and 11 ways to choose a student for the fourth period.<br>Since these choices are independent, we multiply the number of choices for each period together to find the total number of different sequences of crabapple recipients in a week. This is \( 11 * 11 * 11 * 11 = 11^4 \).<br>Calculating \( 11^4 \), we get 14641. Therefore, there are 14,641 different sequences of crabapple recipients possible in a week.<br>The answer is 14,641</td>
  </tr>
</table>

Table 9: Comparison of Different CoT granularity for the same problem generated by Gemini-1.5-Flash.
D  Different CoT Format Dataset Collection

D.1 Workflow
The dataset with format processing steps are detailed below:
  1. 0-Shot Example Generation: The same question is first provided to ChatGPT using a 0-shot prompt in three formats. ChatGPT generates a single 1-shot example for each format (including both the question and a corresponding output in the specific format). This step ensures that ChatGPT establishes a baseline example for each format. These generated examples serve as templates for subsequent responses, ensuring consistency in style and logic.
  2. Input Construction: Each question is then re-input into ChatGPT, along with: - Its corresponding original ChatGPT-generated outputs. - The 1-shot examples generated in Step 1 for all three reasoning formats.
    Including the 1-shot examples in the input serves as explicit format demonstrations, guiding ChatGPT to generate outputs that align with the desired styles. This process improves the coherence and quality of the resulting outputs.
  3. Multi-Format Output Generation: Using the constructed input (original question + original outputs + 1-shot examples), ChatGPT generates reformatted outputs for each question across three reasoning formats while preserving the original logic: Least-to-most, RaR and SymbolicCoT.
  4. Ranking and Alignment: The reformatted outputs are then sorted to align with the original dataset’s order. This step ensures consistency and enables systematic evaluation. Sorting the outputs guarantees that the evaluation is both structured and comparable across different reasoning formats and the original dataset.

D.2 Prompt

Symbolic CoT Prompt Template

Please rewrite the output by following the Symbolic CoT (SymbCoT) reasoning to solve the given question step-by-step. You can ONLY change the format but not the original steps. In your rewrite, translate the question’s context into symbolic logic format, identifying key variables and relationships. Ensure to use logical symbols such as \( \exists \) (exists), \( \forall \) (for all), \( \wedge \) (and), \( \vee \) (or) and \( \implies \) (implies), etc., to represent relationships between variables.
You should use symbolic thinking steps in the output. The generated output must follow this specific structure and include logical symbols. Output the result strictly in the following format. DO NOT generate any other explanations. The generated answer must be consistent with the given answer. After modification, you must add The answer is <answer> at the end.
Here is the original output:

    {instruction: <question>,output: <solution_path>}

The output format should be as follows:

    {instruction: <question>,output: <SymbCoT Solution Path> The answer is <answer>}

Here is the example: <example>

Figure 9: Prompt for generating Symbolic Chain-of-Thought (SymbCoT) reasoning, requiring the transformation of problem contexts into symbolic logic representations using logical operators (\( \exists, \forall, \wedge, \vee \) and \( \implies \)).
Rephrase and Respond (RaR) CoT Prompt Template

Please rewrite the output and answer them individually. You can ONLY change the format but not the original steps.
The generated answer must be consistent with the given answer. Output the result strictly in the following format. DO NOT generate any other explanations.
After modification, you must add The answer is <answer> at the end.
Here is the original output:

    {instruction: <question>,output: <solution_path>}

Rephrase and expand the given question, and then respond carefully.
The output format should be as follows:

    {instruction: <question>,
        output: <Rephrase and expand the given question>
        <solution path> The answer is <answer>}

Here is the example: <example>

Figure 10: Prompt for Rephrase and Respond (RaR), which restructures the question while maintaining the original reasoning steps.

Least-to-Most CoT Prompt Template

Please rewrite the output by generating sub-questions and answer them individually. You can ONLY change the format but not the original steps.
The generated answer must be consistent with the given answer. Output the result strictly in the following format. DO NOT generate any other explanations.
After modification, you must add The answer is <answer> at the end.
Here is the original output:

    {instruction: <question>,output: <solution_path>}

The output format should be as follows:

    {instruction: <question>,
        output: <Subquestion_1><Explanation_1>
        <Subquestion_2><Explanation_2>... The answer is <answer>}

Here is the example: <example>

Figure 11: Prompt for Least-to-Most CoT, which restructures the solution by generating and answering sub-questions step-by-step while preserving the original reasoning process.
D.3  Format Case Study

Different CoT Format Examples

Original CoT:
Bert currently has 20 kangaroos. Kameron has 100 kangaroos. The difference in the number of kangaroos is 100 - 20 = 80 kangaroos. Bert buys kangaroos at a rate of 2 kangaroos per day. Therefore, the number of days required for Bert to have the same number of kangaroos as Kameron currently has is 80 / 2 = 40 days. The answer is 40.

Symbolic CoT:
1. Let \( K_0 = 100 \) \( \exists K_0 \). (Kameron has 100 kangaroos).
2. Let \( B_0 = 20 \) \( \exists B_0 \). (Bert has 20 kangaroos).
3. The difference in kangaroos, \( D = K_0 - B_0 \) \( \exists D \):
\[
D = 100 - 20 = 80
\]
4. Rate of buying kangaroos per day, \( r = 2 \forall r \):
\[
r = 2
\]
5. Number of days required, \( N = \frac{D}{r} \) \( \exists N \):
\[
N = \frac{80}{2} = 40
\]
The answer is 40.

Rephrase and Respond:
On Kameron’s large farm, there are currently 100 kangaroos, whereas Bert’s farm has only 20 kangaroos. Bert aims to increase his kangaroo population, purchasing 2 new kangaroos each day, until he reaches the current number of kangaroos on Kameron’s farm.
Calculate the number of days Bert will need to achieve this goal. Bert currently has 20 kangaroos. Kameron has 100 kangaroos. The difference in the number of kangaroos is 100 - 20 = 80 kangaroos. Bert buys kangaroos at a rate of 2 kangaroos per day. Therefore, the number of days required for Bert to have the same number of kangaroos as Kameron currently has is 80 / 2 = 40 days. The answer is 40.

LEAST-TO-MOST:
How many kangaroos does Bert currently have? Bert currently has 20 kangaroos. How many kangaroos does Kameron currently have? Kameron has 100 kangaroos. What is the difference in the number of kangaroos between Bert and Kameron? The difference in the number of kangaroos is 100 - 20 = 80 kangaroos. What is the rate at which Bert buys kangaroos per day? Bert buys kangaroos at a rate of 2 kangaroos per day. How many days will it take for Bert to have the same number of kangaroos as Kameron? The number of days required for Bert to have the same number of kangaroos as Kameron currently has is 80 / 2 = 40 days. The answer is 40.

Figure 12: Comparison of Different CoT Formats for the same problem: Kameron has 100 kangaroos on his large farm; Bert has 20 kangaroos on his farm. In how many more days will Bert have the same number of kangaroos as Kameron does now if he buys kangaroos at the same rate of 2 new kangaroos per day?
E Whole Results of Granularity Experiments

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Only Answer</th>
    <th colspan="6">Gemma 2B Performance</th>
    <th colspan="6">LLaMA 3.2 1B Performance</th>
    <th colspan="6">LLaMA 3.2 3B Performance</th>
    <th colspan="6">BLOOM 560M Performance</th>
    <th colspan="6">BLOOM 1.1B Performance</th>
    <th colspan="6">BLOOM 1.7B Performance</th>
    <th colspan="6">BLOOM 3B Performance</th>
  </tr>
  <tr>
    <th>Level 1</th><th>Level 2</th><th>Level 3</th><th>Level 4</th><th>Level 5</th><th>Level 6</th>
    <th>Level 1</th><th>Level 2</th><th>Level 3</th><th>Level 4</th><th>Level 5</th><th>Level 6</th>
    <th>Level 1</th><th>Level 2</th><th>Level 3</th><th>Level 4</th><th>Level 5</th><th>Level 6</th>
    <th>Level 1</th><th>Level 2</th><th>Level 3</th><th>Level 4</th><th>Level 5</th><th>Level 6</th>
    <th>Level 1</th><th>Level 2</th><th>Level 3</th><th>Level 4</th><th>Level 5</th><th>Level 6</th>
    <th>Level 1</th><th>Level 2</th><th>Level 3</th><th>Level 4</th><th>Level 5</th><th>Level 6</th>
  </tr>
  <tr>
    <td>SVAMP</td><td>47.70</td><td>59.48<sup>±1.10</sup></td><td>64.33<sub>±0.00</sub></td><td>65.22<sub>±0.01</sub></td><td>65.89<sub>±0.02</sub></td><td><b>67.11</b><sub>±1.78</sub><sup>†</sup></td><td>66.89<sup>±0.92</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td>
    <td>37.70</td><td>52.67<sub>±2.42</sub></td><td>52.11<sub>±2.07</sub></td><td>52.78<sub>±2.09</sub></td><td><b>53.44</b><sub>±1.07</sub><sup>†</sup></td><td>53.44<sub>±1.07</sub></td><td>52.84<sub>±1.71</sub></td><td>52.78<sub>±1.83</sub></td>
    <td>37.70</td><td>52.67<sub>±2.42</sub></td><td>52.11<sub>±2.07</sub></td><td>52.78<sub>±2.09</sub></td><td><b>53.44</b><sub>±1.07</sub><sup>†</sup></td><td>53.44<sub>±1.07</sub></td><td>52.84<sub>±1.71</sub></td><td>52.78<sub>±1.83</sub></td>
    <td>53.70</td><td>69.68<sub>±0.20</sub></td><td>65.89<sub>±0.00</sub></td><td>65.22<sub>±0.01</sub></td><td>65.89<sub>±0.02</sub></td><td><b>67.11</b><sub>±1.78</sub><sup>†</sup></td><td>66.89<sup>±0.92</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td>
    <td>37.70</td><td>52.67<sub>±2.42</sub></td><td>52.11<sub>±2.07</sub></td><td>52.78<sub>±2.09</sub></td><td><b>53.44</b><sub>±1.07</sub><sup>†</sup></td><td>53.44<sub>±1.07</sub></td><td>52.84<sub>±1.71</sub></td><td>52.78<sub>±1.83</sub></td>
    <td>53.70</td><td>69.68<sub>±0.20</sub></td><td>65.89<sub>±0.00</sub></td><td>65.22<sub>±0.01</sub></td><td>65.89<sub>±0.02</sub></td><td><b>67.11</b><sub>±1.78</sub><sup>†</sup></td><td>66.89<sup>±0.92</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td>
    <td>53.70</td><td>69.68<sub>±0.20</sub></td><td>65.89<sub>±0.00</sub></td><td>65.22<sub>±0.01</sub></td><td>65.89<sub>±0.02</sub></td><td><b>67.11</b><sub>±1.78</sub><sup>†</sup></td><td>66.89<sup>±0.92</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td>
    <td>53.70</td><td>69.68<sub>±0.20</sub></td><td>65.89<sub>±0.00</sub></td><td>65.22<sub>±0.01</sub></td><td>65.89<sub>±0.02</sub></td><td><b>67.11</b><sub>±1.78</sub><sup>†</sup></td><td>66.89<sup>±0.92</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td>
    <td>53.70</td><td>69.68<sub>±0.20</sub></td><td>65.89<sub>±0.00</sub></td><td>65.22<sub>±0.01</sub></td><td>65.89<sub>±0.02</sub></td><td><b>67.11</b><sub>±1.78</sub><sup>†</sup></td><td>66.89<sup>±0.92</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td>
    <td>53.70</td><td>69.68<sub>±0.20</sub></td><td>65.89<sub>±0.00</sub></td><td>65.22<sub>±0.01</sub></td><td>65.89<sub>±0.02</sub></td><td><b>67.11</b><sub>±1.78</sub><sup>†</sup></td><td>66.89<sup>±0.92</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td><td>53.45<sup>±1.08</sup></td>
  </tr>
</table>

Table 10: Performance of various models at six granularity levels, including standard deviation (±std). The best performance is boldfaced, and red text shows the relative improvement (%) for the highest vs. lowest performance in six levels. Only Answer: Student models are fine-tuned to directly predict answers without CoT.
F  Padding Procedure for Matched-Length CoT Variants

The following algorithm outlines the process of constructing matched-length CoT variants \( \mathcal{D}_g' \), ensuring that sequences from lower granularity levels are padded to match the length of higher granularity sequences. This process is designed to isolate the impact of granularity from sequence length during evaluation.

Algorithm 1: Dynamic Padding for Matched-Length CoT Variants

Input: \( \mathcal{D}_1, \mathcal{D}_5 \) (CoT outputs for levels 1, 5)
Output: \( \mathcal{D}_1' \): Padded \( \mathcal{D}_1 \) matching \( \mathcal{D}_5 \)
Load \( \mathcal{D}_1, \mathcal{D}_5 \), ensure lengths match;
foreach \( (x_1, x_5) \in (\mathcal{D}_1, \mathcal{D}_5) \) do
    Tokenize \( x_1, x_5 \) to get lengths len(\( T_1 \)) and len(\( T_5 \));
    if \( \text{len}(T_1) < \text{len}(T_5) \) then
        Compute repeat\_factor = \( \text{len}(T_5)/\text{len}(T_1) \);
        Repeat and truncate \( T_1 \) to match \( \text{len}(T_5) \), decode to \( x_1^{\text{tmp}} \);
        if "The answer is" in \( x_1 \) then
            Split \( x_1 \) as [prefix, suffix] at "The answer is";
            \( x_1' = \text{prefix} + x_1^{\text{tmp}} + \text{"The answer is"} + \text{suffix} \);
        else
            \( x_1' = x_1^{\text{tmp}} \);
        Append \( x_1' \) to \( \mathcal{D}_1' \);
    Save \( \mathcal{D}_1' \) and log statistics;

G  Analysis of Symbolic CoT Performance in CoT distillation

To investigate the suboptimal performance of Symbolic CoT, we conducted a case study.

Question: janet’s ducks lay 16 eggs per day. she eats three for breakfast every morning and bakes muffins for her friends every day with four. she sells the remainder at the farmers’ market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers’ market?

BLOOM 3B Output

Let:

\[
\begin{align*}
D &= 16 \quad \text{(eggs laid per day)} \\
E &= 3 \quad \text{(eggs eaten per day)} \\
B &= 3 \quad \text{(breakfasts per day)} \\
M &= 4 \quad \text{(muffins per day)} \\
S &= D - E = 16 - 3 = 13 \quad \text{(sold eggs per day)} \\
R &= \text{remaining eggs after breakfasts and muffins} \\
T &= \text{total daily earnings}
\end{align*}
\]

1. \( \forall E : E = 3 \) (Janet eats 3 eggs per day)
2. \( \forall B : B = 3 \) (Janet bakes muffins for her friends every day)
3. \( \forall M : M = 4 \) (Janet sells the remaining eggs at the farmers’ market daily for $2 per fresh duck egg)
4. \( \forall R : R = D - E - M = 16 - 3 - 4 = 9 \) (remaining eggs after breakfasts and muffins)
5. \( \forall T : T = R \times 2 \Rightarrow T = 13 \times 2 = 26 \) (total daily earnings)

The answer is **26**.
Gemma 2B Output

Let:

\[
E = 16 \quad \text{(total eggs laid per day by ducks)}
\]
\[
B = 3 \quad \text{(eggs eaten for breakfast per day)}
\]
\[
M = 4 \quad \text{(eggs baked into muffins per day)}
\]
\[
R = \text{remaining eggs after breakfast and muffins}
\]
\[
S = \text{eggs sold at farmers’ market per day}
\]
\[
P = 2 \quad \text{($ per egg at farmers’ market)}
\]
\[
T = \text{total earnings per day at farmers’ market}
\]

1. \( \forall E : E = 16 \) (total eggs laid per day)
2. \( \forall B : B = 3 \) (Janet eats 3 eggs for breakfast)
3. \( \forall M : M = 4 \) (Janet bakes 4 muffins daily)
4. Remaining eggs after breakfast and muffins:
\[
R = E - B - M \Rightarrow R = 16 - 3 - 4 = 9
\]
5. \( \forall S : S = R \Rightarrow S = 9 \) (Janet sells the remaining eggs)
6. Total earnings:
\[
T = S \times P \Rightarrow T = 9 \times 2 = 18
\]

The answer is 18.

The error in BLOOM 3B arises from unnecessary variable assignments and mismanagement of numerical values. While it correctly defines and calculates \( R = 9 \) (the remaining eggs), it later incorrectly substitutes \( R = 13 \) into the final computation, leading to an incorrect total earnings prediction of $26 instead of $18. The symbolic expressions themselves are structurally correct, but the model fails to maintain numerical consistency throughout the reasoning process. This suggests that Symbolic CoT lacks a robust mechanism for tracking variable dependencies and verifying intermediate values, especially for weaker SLMs. We identified and summarized several possible reasons for its poor performance:

1. Task Relevance and Reasoning Depth: According to the original paper, Symbolic CoT is primarily designed for logical reasoning tasks (Xu et al., 2024a). However, our datasets focus on mathematical and commonsense reasoning, where the advantages of symbolic reasoning—particularly its effectiveness in handling deeper reasoning—do not manifest as clearly.

2. Implementation Differences: The original study employed multiple stages and corresponding special tokens to enhance symbolic reasoning. In contrast, our implementation only adopted the symbolic reasoning format without these additional mechanisms, which might have impacted its effectiveness.

3. Pretraining Data Constraints: SLMs have relatively limited pretraining corpora, which likely contain fewer instances of symbolic reasoning formats. As a result, weaker models struggle to acquire symbolic reasoning capabilities with only a small number of training samples.
H  Student Performance across Different Teacher Models

(a) SVAMP
(b) GSM8K
(c) MATH
(d) AQuA-RAT
(e) CommonsenseQA
(f) StrategyQA

Figure 13: Student model performance across different teacher models. Each bar represents the average accuracy of a specific student model trained on CoT from different teacher models.

Reviewer paper summaries: This paper systematically investigates the factors influencing Chain-of-Thought (CoT) distillation in Small Language Models (SLMs), including the choice of granularity, format, and teacher model. Through extensive experiments on multiple datasets with various teacher and student models, it uncovers several key findings:

1.Non-monotonic relationship: SLM performance shows a non-monotonic dependence on CoT granularity.

2.Limited format impact: CoT formats significantly affect LLMs but have minimal impact on SLMs, likely due to SLMs' reliance on supervised fine-tuning rather than pretraining biases.

3.Teacher model trade-offs: Stronger teacher models do not always yield better students, as supervision diversity and complexity may outweigh pure accuracy.
Summary of strengths: 1.Comprehensive Exploration: The study comprehensively examines multiple factors affecting CoT distillation in SLMs, providing a systematic understanding of the problem.

2.Rigorous Experiments: Covers seven datasets, four teachers, and seven students, with controlled ablation studies.

3.Practical Value: Provides actionable strategies (such as granularity customization) and challenges the "stronger teacher is better" assumption.
Summary of weaknesses: 1.Lack of Innovation: Although the paper explores the factors influencing CoT distillation in SLMs, the methods and findings lack significant novelty. Many of the concepts and experiments are based on existing research, and there are no truly groundbreaking contributions in terms of methodology or theoretical understanding. This makes the paper less competitive compared to other works in the field that bring more innovative ideas and solutions.

2.Task Generalizability: Focus on math and commonsense tasks leaves other reasoning types (logic, code generation) unexplored.
Comments, suggestions and typos: Major Suggestions:

1.Clarify Innovation: The paper should further clarify its innovative aspects. Currently, it's not entirely clear how the proposed approach and findings significantly differs from existing research in the field. 

2.Analyze Underlying Reasons: There is a need to delve deeper into the underlying reasons behind the conclusions. Merely stating the findings is not enough. It could benifit provide in-depth analysis to explain why certain factors have the observed effects.

Minor Suggestions:

1.Visualize Methods: The specific methods used in the study would be more intuitive if presented with diagrams or flowcharts. This can greatly enhance the readability of the paper and make it easier for readers to understand the complex procedures involved.
Overall assessment scores: 2