Large Language Models Are Reasoning Teachers
Year: 2023

Abstract:
Recent works have shown that chain-of-thought (CoT) prompting can elicit language models to solve complex reasoning tasks, step-by-step. However, prompt-based CoT methods are dependent on very large models such as GPT-3 175B which are prohibitive to deploy at scale. In this paper, we use these large models as reasoning teachers to enable complex reasoning in smaller models and reduce model size requirements by several orders of magnitude. We propose Fine-tune-CoT, a method that generates reasoning samples from very large teacher models to fine-tune smaller models. We evaluate our method on a wide range of public models and complex tasks. We find that Fine-tuneCoT enables substantial reasoning capability in small models, far outperforming prompt-based baselines and even the teacher model in many tasks. Additionally, we extend our method by leveraging the teacher model’s ability to generate multiple distinct rationales for each original sample. Enriching the fine-tuning data with such diverse reasoning results in a substantial performance boost across datasets, even for very small models. We conduct ablations and sample studies to understand the emergence of reasoning capabilities of student models.1

heading: 1 Introduction
text: Language models (LMs) have demonstrated remarkable performance in a wide range of downstream tasks. Recently, large language models (LLMs) have demonstrated in-context generalization capabilities: performing downstream tasks simply by conditioning on few in-context exemplars or plain natural language task descriptions (Brown et al., 2020; Sun et al., 2021). Despite these advancements, even the largest LLMs have been found to struggle with complex tasks which require multiple reasoning steps (Rae et al., 2021).
1Our code implementation and data are available at https://github.com/itsnamgyu/reasoning-teacher.
To solve complex tasks, recent works show that it is possible to elicit reasoning abilities by prompting LLMs to perform chain-of-thought (CoT) reasoning, i.e., generate a series of intermediate reasoning steps. This can be achieved by providing CoT demonstrations as exemplars in prompting (Wei et al., 2022b). More recently, Kojima et al. (2022) found that LLMs can be prompted to perform CoT reasoning simply by providing a natural language instruction to think step-by-step.
A major drawback of prompt-based CoT reasoning methods, however, is their reliance on extremely large models that span hundreds of billions of parameters (Wei et al., 2022b; Kojima et al., 2022). These models are prohibitive to deploy at scale due to overwhelming computational requirements and inference costs (Wei et al., 2022b).
14852
Therefore, we strive to enable complex reasoning in small models which are more feasible for largescale deployment.
In this light, we propose an approach named Fine-tune-CoT, which utilizes the reasoning capabilities of very large LMs to teach small models how to solve complex tasks. We apply existing zero-shot CoT prompting (Kojima et al., 2022) to generate rationales from very large teacher models, and use them to fine-tune smaller student models2. We illustrate this in Figure 2. We note that standard fine-tuning without rationales has been shown to be inadequate for solving reasoning tasks with small models (Talmor et al., 2018). While there have been attempts to fine-tune small models with hand-annotated reasoning steps (Nye et al., 2021; Cobbe et al., 2021), they often require task-specific training setups and high-quality rationales which are costly to annotate (Wei et al., 2022b). In contrast, our approach can be readily applied to novel downstream tasks without hand-crafted reasoning or task engineering.
We also propose a novel extension to our method, termed diverse reasoning, to maximize the teaching effects of Fine-tune-CoT. Inspired by the intuition
2This can be interpreted as a variant of knowledge distillation (Hinton et al., 2015).
that complex tasks can have multiple solutions with distinct reasoning paths (Evans, 2010), we generate multiple reasoning solutions from teacher models using stochastic sampling to augment the training data for student models3. We find that this is a simple yet highly effective approach to maximizing student performance, which has not been explicitly recognized in concurrent works on fine-tuning with CoT reasoning (Huang et al., 2022; Li et al., 2022b; Magister et al., 2022; Fu et al., 2023).
We evaluate our method on 12 tasks using a wide range of publicly available models. We find that Fine-tune-CoT can elicit notable reasoning performance in small models while preserving much of the versatility of prompt-based CoT reasoning, which previously required >100B parameter models (Wei et al., 2022b). Diverse reasoning enables remarkable gains in performance at the minor cost of additional teacher inference at development time, by exploiting our unique learning setup. This enables models as small as 0.3B to outperform larger students, and even the 175B teacher model in some tasks. Our ablations show that performance is con-
3Diverse reasoning is orthogonal to existing data augmentation techniques (Yoo et al., 2021; Meng et al., 2022) which aim to augment new question-answer pairs rather than diverse reasoning solutions for complex questions.
sistently scalable across all axes considered: diverse reasoning, dataset size, teacher performance, and student model size. This shows the potential of our method to enable reliable performance in small models that are feasible for use in real-world applications. Lastly, we conduct thorough sample studies and analyses which shed light on crucial details previous overlooked in fine-tuning for CoT and provide intuition on the emergence of reasoning abilities in small models.

heading: 2 Related Work
text: Downstream transfer in language models Much previous work established a “pre-train and fine-tune” paradigm for enhancing LLM performance on downstream tasks (Radford et al., 2018; Dong et al., 2019; Vaswani et al., 2017; Devlin et al., 2018). However, fine-tuning is not always easily applicable (Hendrycks et al., 2020). More recent literature exhibits a paradigm shift towards “prompting” the model to predict the desired output (Liu et al., 2021; Raffel et al., 2020). Large LMs can exhibit strong performance in this setting (Brown et al., 2020). For smaller models to be able to perform similarly, additional engineering is usually required (Gao et al., 2021; Schick and Schütze, 2021b; Schick et al., 2020). For more complex tasks, the idea of using samples with explicit reasoning steps for fine-tuning a model (Nye et al., 2021; Cobbe et al., 2021) preceded the approach of chain-of-thought (CoT) prompting (Wei et al., 2022b), which enables very large LMs to perform well.
Chain-of-thought reasoning In few-shot CoT prompting, the model learns to generate intermediate reasoning steps that lead to a problem solution, after being fed examples of step-by-step reasoning. This enables very good performance on a wide range of tasks. (Wang et al., 2022). Additionally, LLMs can perform well in an unsupervised task-agnostic setting, using Zero-shot-CoT (Kojima et al., 2022). This requires no fine-tuning or task specific conditioning, and substantially outperforms standard zero-shot learning and sometimes even few-shot learning on a wide number of tasks.
Yet, prior work has shown that CoT requires extremely large models for optimal performance (Hoffmann et al., 2022; Chowdhery et al., 2022). In our work, we contrast this by showing how to utilize CoT reasoning methods for smaller models by fine-tuning them on rationales gener-
ated by a very large model. Using various LLMgenerated explanations for fine-tuning smaller models has been successfully used in prior work (Li et al., 2022a), with a focus on specific single tasks. Also, a similar approach to ours is mentioned in (Huang et al., 2022); however we note that this concurrent work focuses on using Few-shot-CoT to self-generate fine-tuning examples by and for very large proprietary models. There is a brief glimpse into fine-tuning on smaller distilled models, but the results are limited to one dataset and very large teacher models that are inaccessible to the general community. In contrast, we provide a rich set of results and qualitative/quantitative analysis on a wide range of datasets, using open-source models that are small and accessible to everyone.
Knowledge distillation Typically, knowledge distillation (KD) refers to training small models derived from large models in order to reduce model size and latency, while still preserving accuracy and capacity to generalize (Hinton et al., 2015; Sanh et al., 2019). Essentially, KD is a form of model compression, making efficient deployment to capacity-limited devices possible (Bucilua et al., 2006). We note that our work could also be considered a distant variant of KD (Gou et al., 2021), similar to works on improving prompt-based methods such as Yoo et al. (2021); Schick and Schütze (2021b,a); Zelikman et al. (2022), or works on datafree distillation (Micaelli and Storkey, 2019; Nayak et al., 2019; Shen et al., 2021), where the transfer data is synthetically generated from a large teacher model. Similarly, sequence-level distillation, i.e. training a student model on sequence distributions of a larger teacher, can make neural machine translation more efficient (Kim and Rush, 2016). Despite being similar in spirit, our method still distinguishes itself from such previous work. The role of the teacher model in our method is to teach the notion of intermediate reasoning. It is not the specific output that is the main supervising signal for reasoning, but rather the generation’s structure. Hence, we do not use a standard KD loss function that reflects trying to match the teacher output. Adding to this, we note that our diverse reasoning is also unusual in the context of KD, where it is e.g. sufficient in practice to only generate one teacher sequence for sequence level distillation.

heading: 3 Chain-of-Thought Fine-Tuning
text: We propose Fine-tune-CoT, a task-agnostic approach to enable chain-of-thought reasoning in small LMs. The core idea is to generate reasoning samples from very large teacher models using CoT prompting and subsequently fine-tune small student models using the generated samples. This approach preserves the versatility of prompt-based CoT methods while overcoming their reliance on prohibitively large models. To maximize versatility and minimize teacher inference costs, we use the task-agnostic Zero-shot-CoT prompting method (Kojima et al., 2022) on teacher models, as it does not require any reasoning examples or long inference context. We discuss our choice of teacher CoT prompting method in Section 7.3. In the following, we characterize Fine-tune-CoT in three distinct steps. We also provide a visual overview in Figure 2.
Step 1. Reasoning generation First, we utilize a large teacher model to generate CoT reasoning explanations for a given task. Consider a standard sample Si consisting of a question qi and its true answer ai. Using Zero-shot-CoT 4. we prompt the teacher model to generate a reasoning explanation, or rationale, r̂i to solve question qi and make a final answer prediction âi. The resulting text sequence, including the prompt and generations, takes the following form: “Q: <qi>. A: Let’s think step by step. <r̂i> Therefore, the answer is <âi>”.
Step 2. Curation Next, we filter the generated samples and reformat them into prompt-completion pairs. For filtering, we simply compare the final prediction of the teacher model âi with the groundtruth answer ai, following previous works (Zelikman et al., 2022; Huang et al., 2022). Note that this filtering incurs some loss of training samples. For all instances i where âi = ai, we repackage (Si, r̂i, âi) into a reasoning sample S′i = (pi, ci), a prompt-completion pair. To maximize inferencetime efficiency, we use special-character based delimiters to minimize token usage. Specifically, pi and ci each take the form of “<qi> ###” and “<r̂i> --> <ai> END”. We note that answer-based filtering does not ensure the correctness of the rationales, especially for multi-choice questions. We provide an analysis in Appendix E.1 regarding this important detail which has not been addressed in concurrent work.
Step 3. Fine-tune Finally, we fine-tune a small pre-trained student model on the assembled reasoning samples. We use the same training objective of that used during pre-training, i.e., autoregressive language modeling objective, or next-token prediction (Radford et al., 2018).
Diverse reasoning To maximize the teaching effects of Fine-tune-CoT, we can generate multiple reasoning explanations for each training sample. This approach is motivated by the intuition that multiple reasoning paths can be used to solve complex tasks, i.e., type-2 tasks (Evans, 2010). We posit that this unique feature of complex tasks, in tandem with the stochastic generation abilities of the teacher model, can enable diverse reasoning to significantly boost reasoning supervision simply through additional teacher inference. In detail, for a given sample Si, instead of applying Zero-shot-CoT using greedy decoding to obtain a single explanation-answer pair (êi, âi), we use a stochastic sampling strategy, i.e., temperature sampling with large T , to obtain D distinct generations {(r̂ij , âij)}Dj . Subsequent reasoning sample curation and fine-tuning then proceed as before. We refer to D as the degree of reasoning diversity. A similar approach is used in Wang et al. (2022); Huang et al. (2022), where multiple CoT outputs are generated and marginalized to find the optimal answer. However, the effects of such diverse reasoning on teaching student models has not been acknowledged or thoroughly investigated in concurrent work (Huang et al., 2022; Li et al., 2022a; Magister et al., 2022; Fu et al., 2023). We note that diverse reasoning imposes an important tradeoff between the development cost and inference cost/quality of student models which we discuss in Section 5.3.

heading: 4 Experiments
text: Tasks and datasets We evaluate our method on 12 datasets pertaining to four categories of complex reasoning, following Kojima et al. (2022). These include arithmetic (SingleEq, AddSub, MultiArith, GSM8K, SVAMP), other (Date Understanding, Tracking Shuffled Objects), symbolic (Last Letter Concatenation, Coin Flip), and common sense (CommonSenseQA, StrategyQA) reasoning. We provide details and references in Appendix B.
4Note that Zero-shot-CoT is itself a two-step prompting method. The reasoning (blue) is generated in the first step and answer prediction (red) is generated in the second step.
Models For teacher models, we use four variants of GPT-3 175B (Brown et al., 2020), provided by the OpenAI API. Unless otherwise stated, we use text-davinci-002 based on InstructGPT 175B (Ouyang et al., 2022) as the teacher for Finetune-CoT. For student models, we consider four popular model families. For our main experiments, we use GPT-3 {ada, babbage, curie} as they are readily available for fine-tuning via the OpenAI API. Due to the blackbox nature of the API, we also consider various open-source models under controlled settings. We use GPT-2 {Small, Medium, Large} (Radford et al., 2019) and T5{Small, Base, Large} (Raffel et al., 2020) as representative model families for decoder-only and encoder-decoder architectures, respectively. We also use the instruction-tuned version of T5, FlanT5-{Small, Base, Large} (Chung et al., 2022), to investigate the effects of instruction tuning on student models, prior to applying Fine-tune-CoT. These student models are 25–2500x smaller than the teacher model, thus considerably more feasible for realworld deployment. We provide details on models and API usage in Appendix C.
Baseline methods We provide a comparison of Fine-tune-CoT (ours) with four baseline methods: standard zero-shot prompting, vanilla fine-tuning, Zero-shot-CoT (Kojima et al., 2022), and Few-shotCoT (Wei et al., 2022b). Given a training sample {(qi, ai)}i, we use a simple format “Q: <qi>” for
zero-shot prompting. For vanilla fine-tuning, we format the prompt and completion as “<qi> ###” and “<ai> END”, respectively. We clarify the taxonomy of methods in Table 2. For text generation, we use greedy decoding following Wei et al. (2022b); Kojima et al. (2022) throughout our experiments, except for diverse reasoning. For diverse reasoning on the teacher, we use temperature sampling with T = 0.7, following Wang et al. (2022). We provide experimental details in Appendix A.

heading: 4.1 Results
text: In this section, we present the reasoning performance of models using Fine-tune-CoT and diverse reasoning. We compare with various baselines and demonstrate the scalability of our method across four axes: degree of diverse reasoning (Figure 3), dataset size (Figure 4), performance of the teacher (Figure 5), and size of the student model (Figure 6).
We present our findings on GPT-3 models in the main text and defer results on open-source models to Appendix G, with a brief summary at the end of this section.
Fine-tune-CoT elicits complex reasoning in small models Table 1 summarizes the accuracy of student models using the proposed Fine-tuneCoT, compared to prompt-based CoT baselines as well as standard fine-tuning. While Zero-shot-CoT exhibits remarkable performance on the very large 175B model (Kojima et al., 2022), it fails to enable complex reasoning in all three smaller models, showing near-negligible performance across all tasks. We also find that small models are unable to approach these tasks under standard zero-shot prompting. On the other hand, Fine-tune-CoT elicits notable reasoning performance, demonstrating significant gains over Zero-shot-CoT when using smaller models and outperforming both fine-tuning and Few-shot-CoT in more than half of the tasks. For complex arithmetic, Fine-tune-CoT achieves a notable 33% accuracy on MultiArith while Zeroshot-CoT only reaches 5%. Few-shot-CoT and fine-tuning only achieve 10% and 15%, respectively. For two commonsense reasoning tasks, our method outperforms the near-random performance of Zero-shot-CoT by 37% and 5%, respectively. Furthermore, it surpasses Few-shot-CoT on CommonSenseQA by 32% and performs similarly on StrategyQA. We observe that Fine-tune-CoT performance is most notable for tasks that are not overly complex, which include other reasoning tasks (Date Understanding, Shuffled Objects) and symbolic reasoning (Last Letter, Coin Flip), significantly outperforming other baselines. See Appendix Table 9 for performance of all students.
Small models can outperform very large teachers in reasoning Table 1 also shows that Finetune-CoT is highly effective on small models compared to the large 175B teacher model. For the tasks Shuffled Objects and Coin Flip, Fine-tuneCoT is shown to outperform the teacher model using either 1.3B or 6.7B parameters, i.e., reducing the number of required parameters by approx. 25–100x. We also find that Fine-tune-CoT with the very small 0.3B model consistently outperforms the 6.7B model under Zero-shot-CoT, demonstrating that our method is able to unlock a wider range of capabilities compared to the baseline, even when model size is vastly reduced.
Diverse reasoning substantially improves Finetune-CoT performance. To examine the learning effects of diverse reasoning and compare it with two baselines given by fine-tuning and Few-shotCoT, we apply Fine-tune-CoT using 1–64 reasoning explanations per sample across three model scales on MultiArith and SVAMP5. Figure 3 shows that diverse reasoning can significantly improve the performance of student models using Fine-tuneCoT. For the 6.7B student model, we find a boost of around 26% on MultiArith, and around 17% on SVAMP. We also note that using diverse reasoning always leads to outperforming the baseline within the respective model size, and can even boost performance of our method beyond that of a larger model that does not use diverse reasoning. This even includes the teacher in two cases (Date Understanding, Last Letter). Moreover, we find that diverse reasoning can boost the performance of Finetune-CoT to surpass that of both Few-shot-CoT and vanilla fine-tuning across all model sizes. We posit that due to our focus on complex tasks, the diversity of reasoning paths and linguistic templates can substantially aid in teaching student models to reason.
Fine-tune-CoT consistently benefits from more data. We perform an ablation on dataset size to study the performance scalability of our method with dataset size. We see that the performance of
5For diverse reasoning, we generate teacher rationales stochastically with T = 0.7 instead of greedy decoding, which accounts for small differences in absolute performance numbers between Table 1 and diverse reasoning with D = 1.
the 6.7B model clearly scales with the size of the dataset, independent of the task. In comparison, vanilla fine-tuning does not always exhibit this behavior. In fact, for Date Understanding, we find that an increase in dataset size harms the performance of fine-tuning. Furthermore, Fine-tune-CoT sees additional benefits from diverse reasoning, which is not applicable in standard fine-tuning.
Better reasoners are better teachers Next, we can ask the question of whether the performance of the teacher is correlated with that of their student when using Fine-tune-CoT. To test this, we use different versions of GPT-3 as teacher models, keeping the size of the student model constant at 6.7B parameters (Figure 5). We find that student performance indeed scales with teacher performance, particularly in the less complex tasks Date Understanding and Last Letter. There, the performance of the student matches the performance of the teacher very closely. This also fits with our observations in Appendix D, which show that the successes and failures of teachers are correlated with those of the students. We note that this scaling effect is in contrast not a given in knowledge distillation, where more accurate teachers do not always result in better students (Menon et al., 2021).
Fine-tune-CoT performance scales with model size for small LMs Finally, we explore the effect of scaling up student model size on our method, and compare it with the effects of increasingly larger student models in Few-shot-CoT as well as vanilla fine-tuning. We can observe that the performance of Fine-tune-CoT is consistently scalable with student size (Figure 6). In contrast, the two baselines do not always exhibit the same behavior: in Date Understanding, neither Few-shot-CoT nor vanilla fine-tuning results in scalable performance.
Results on open-source student models Overall, our findings on T5, Flan-T5, and GPT-2 show similar trends to those observed on GPT-3. Small models exhibit near-random performance under standard zero-shot or CoT prompting in nearly all cases. Notable, we find that encoder-decoder models, i.e., T5 and Flan-T5, show noteworthy performance under standard fine-tuning, suggesting that causal masking may be a bottleneck to reasoning in decoder-based language models in the absence of CoT output. Fine-tune-CoT consistently outperforms prompt-based baselines and is comparable or superior to vanilla fine-tuning. Diverse reasoning improves performance even further, often exhibiting significant gains. We report our full findings on open-source models in Appendix G.

heading: 4.2 Analysis
text: Sample study To identify the strengths and weaknesses of our method, we perform a thorough sample study across all datasets and methods. Across all arithmetic tasks, we find that a large portion of errors arises from calculations. MultiArith and SVAMP also show many semantic errors, but these are significantly reduced with diverse reasoning. For difficult tasks such as GSM8K and AQUA, we found that all methods tend to struggle. We found that our method is highly effective in text-based tasks, excluding commonsense reasoning, as well as tasks that contain common linguistic patterns. On the other hand, we find that students under Zero-shot-CoT often repeat questions or produce incoherent repetitive statements. While Few-shotCoT elicits step-by-step sentences, the student models rarely seem to understand the semantics of the question, and generations often contain logical or commonsense errors. For details on our sample study, see Appendix D.
Nuances of fine-tuning on CoT reasoning We shed light on nuances that have often been overlooked in previous or concurrent work (Wei et al., 2022b; Li et al., 2022a; Magister et al., 2022). First, we acknowledge the possibility that correct samples may contain incorrect reasoning. In fact, we find that 27.6% of correct teacher completions for Date Understanding contained reasoning errors.
However, ablations on rationale filtering suggest that these incorrect rationales can aid in student supervision (Appendix E.1). Secondly, we find that common maximum sequence lengths used for CoT generations often lead to incomplete answers. We observe that reasoning length differs among datasets, and longer generations typically improve accuracy, but may not be beneficial for fine-tuning (Appendix E.2). Lastly, we find that many datasets are comprised of samples that share common templates, potentially compromising the validity of our random train-test splits. To address this, we evaluate our method on manual template-wise data splits, and confirm that students retain meaningful reasoning capabilities (Appendix E.3).

heading: 5 Discussion

heading: 5.1 Accessibility of Fine-tune-CoT
text: Owing to the versatility of the teacher generation method, i.e., Zero-shot-CoT, our method can be readily applied to any complex task without taskspecific engineering. Rationales can be readily generated using publicly available APIs such as those provided by OpenAI or Anthropic. This makes it viable to obtain CoT training data in low-resource scenarios, which not only outperforms standard fine-tuning, but elicits the student to output interpretable explanations. Fine-tuning and inference on student models can also be performed on much more accessible hardware, in contrast to very large models. This can reduce long-term inference costs and minimize environmental impact while making our method fully accessible to a wide community.

heading: 5.2 Viability of Fine-tune-CoT
text: While Fine-tune-CoT elicits notable complex reasoning capabilities in small models, performance on some difficult datasets would not be considered viable for real-world use, such as 30.33% on SVAMP. However, our findings in Section 4.1 indicates significant potential for improvement, as our method is shown to be uniquely scalable with (1) diverse reasoning, (2) dataset size, (3) teacher model performance, and (4) student model size. The use of diverse reasoning and better teacher models is especially promising, as these can benefit from improved teacher LLM performance and inference costs in the future. In addition, it is possible to incorporate recent CoT methods, which lead to significant performance improvements, in student models, which we discuss in Section 7.3.

heading: 5.3 Tradeoffs of Fine-tune-CoT
text: The aforementioned opportunities to enhance Finetune-CoT also pose many important tradeoffs. We leave further analysis to future work.
Degree of diverse reasoning The performance benefits of diverse reasoning come at the cost of additional teacher inference. Therefore, diverse reasoning poses a tradeoff between development cost vs inference cost/quality. In other words, performance gains from diverse reasoning may be utilized to enhance student performance or alleviate the need for larger student models. This must also be taken into account for fair evaluation of similar distillation methods in the future.
Data acquisition Data annotation and diverse reasoning can both be used to enlarge fine-tuning data, but each have their associated costs. We note that the cost of diverse reasoning is linear to the number of generated rationale and the number of original samples. Despite this, it can still be a costeffective alternative to hand-annotating additional data. A preliminary cost analysis in Appendix F shows that the pareto front of data-acquisition-cost to performance always incorporates diverse reasoning. We expect that the cost benefits of diverse reasoning will continue to improve with improvements in teacher model performance and efficiency.

heading: 5.4 Emergence of CoT reasoning
text: The emergence of abilities such as CoT reasoning has become a point of interest in recent works (Wei et al., 2022b,a; Schaeffer et al., 2023). We note that the efficacy of Fine-tune-CoT on small models does not disprove this emergence, as our method is based on fine-tuning. However, we believe our results can provide some insight into this phenomena.
Why does Fine-tune-CoT work in small models? In a seminal work, Wei et al. (2022b) suggests that CoT reasoning is an emergent ability of scale—more specifically, a complicated phenomena involving a variety of emergent abilities, such as semantic understanding, symbol mapping, arithmetic ability. However, our sample studies suggest that Fine-tune-CoT elicits these emergent abilities even in relatively small models (see Appendix D). We explain this from two perspectives. First, Wei et al. (2022b) demonstrated the emergence of reasoning abilties by identifying a reduction in the frequency of reasoning errors with larger model scale. Similarly, we find that more potent forms
of supervision also lead to a gradual reduction in reasoning errors. For example, we found a clear distinction between Zero-, Few-shot-CoT and Finetune-CoT (with diverse reasoning) in the frequency and severity of semantic errors, i.e., understanding complex questions, and calculation errors. This suggests that explicit supervision on reasoning can also lead to the emergence of reasoning abilities. Second, we qualitatively find that students show capabilities that are reminiscent of the larger teacher model. We found that students can recognize common semantics and reasoning cues of the given task, and is able to imitate the process of splitting large tasks into subtasks. This suggests that it is possible to learn reasoning abilities pertaining to a particular domain. We posit that this is possible in small models due to the limited domain of reasoning, and may not be applicable in reasoning tasks that require large domains of knowledge.
Distillation of emergent abilities Chain-ofthought reasoning has been recognized as a prime example of emergent abilities in very large language models (Wei et al., 2022a). Our findings show that it is possible to distill this ability, under certain domains, to much smaller models simply through fine-tuning. The potential for distillation implies that future advancements in language models may lead to emergent abilities that are not only pertinent to those larger models, but could also have a broader impact, cascading benefits to smaller models.

heading: 6 Conclusion
text: We have proposed Fine-tune-CoT, a method that uses LLMs as reasoning teachers to transfer the broad reasoning capabilities previously found in >100B models to student models as small as 0.3B. We propose diverse reasoning as a novel approach to maximize these teaching effects, exploiting the unique characteristics of this new learning setup to vastly improve performance. Our extensive experiments show that Fine-tune-CoT elicits significant reasoning performance in small models, thus demonstrating the distillation of CoT reasoning which has been considered an emergent ability of scale. By leveraging publicly available models with zero-shot prompting, we demonstrate a taskagnostic approach to elicit reasoning performance in small models, making complex reasoning feasible for real-world deployment and accessible to the broader community.

heading: 7 Limitations

heading: 7.1 Towards concise answers
text: Sample studies show that rationales output from student models may occasionally be repetitive and digressive. This is undesirable in terms of inference-time efficiency as well as interpretability. As a minor optimization to inference computation, we construct our fine-tuning sample templates using special-character based delimiters instead of natural language used in concurrent work (Huang et al., 2022) to minimize sequence length. Preliminary findings showed this had no significant impact on reasoning performance. More importantly, it is desirable to train student models to generate concise answers in terms of substance. Appendix E.2 hints at the possibility for this, showing that finetuning on shorter reasoning samples causes the student model to also produce shorter rationales.

heading: 7.2 Exploring a wider array of models
text: We note that the performance of our method is currently not state-of-the-art. However, it can benefit from advances in teacher models as well as other prompting methods. For example, future work should include a wider array of teachers, such as the highly versatile ChatGPT, which typically generates detailed long responses that may be able to impart more knowledge to the student. More recent models such as GPT-4 have demonstrated significant advances in complex reasoning abilities, which may improve the efficacy of Fine-tuneCoT on very difficult datasets, such as GSM8K. Conversely, our method could prove even more advantageous when applied to recent models with improved efficiency, such as those based on the recent LLaMA model (Touvron et al., 2023), which has sparked a proliferation of work focused on compact language models. Both of these avenues are promising for future work.

heading: 7.3 Better CoT inference methods
text: The use of diverse reasoning and better teacher or student models is especially promising, as it is possible to leverage future improvements in model performance and decreased inference costs. However, we can also consider other ways to boost performance, such as using different prompting methods. For example, previous work shows that Few-shot-CoT (Wei et al., 2022b) can improve accuracy over Zero-shot-CoT by a wide margin, e.g., going from 78.7% to 93.0% on MultiArith (Kojima
et al., 2022). However, our choice to use Zeroshot-CoT to generate reasoning samples from the teacher model is motivated by the fact that Fewshot-CoT requires a significantly larger inference context. With the current pricing models based on token usage, the typical setup of 8-shot CoT would cost approximately 8 times more compared to Zero-shot-CoT. Therefore, we see a tradeoff between using the inference budget for Few-shot-CoT and using it for diverse reasoning with Zero-shotCoT. On the other hand, we also note that recent works introduce various ways to improve CoT reasoning performance substantially (often to nearperfect levels), which can be applied to our student models. These include refinement over repeated inferences (Wang et al., 2022; Li et al., 2022b) and self-improvement (Zelikman et al., 2022; Huang et al., 2022). In particular, self-consistency (Wang et al., 2022) can be utilizied on unlabeled samples to maximize the teaching signal. In contrast, we aim to achieve CoT reasoning without the inference time cost incurred by very large LMs. Future work is needed to incorporate these methods into Fine-tune-CoT while minimizing development and inference costs.

heading: 7.4 Connection with knowledge distillation
text: We assume that there is a lot of potential in strengthening the connections between knowledge distillation and our method. We have already seen in this work that our method shares some characteristics with KD, such as the fact that the knowledge of intermediate reasoning imparted by using also incorrect samples can have positive effects on student accuracy, akin to “dark knowledge” (Menon et al., 2021) that is transferred by training on teacher output logits and not one-hot labels. We have seen that this leads to a quantity-quality tradeoff when it comes to the ability of the student model to generalize: having fewer but perfectly curated reasoning samples is not necessarily as helpful as having a larger amount of reasoning samples that might not always be fully correct. On the other hand, we have also found that more accurate teachers do lead to more accurate students, which is not always the case in KD (Müller et al., 2019). It would therefore be of interest for future work to formalize the connection of Fine-tune-CoT with classic KD methods, and potentially test the use of a different distillation loss function that takes the teacher’s actual output into account.

heading: 8 Ethics Statement
text: Our work presents various challenges and opportunities in terms of bias and toxicity in language models. It is widely known that LLMs trained on large corpora have been shown to capture biases found in the training data (Brown et al., 2020; Chowdhery et al., 2022). Since our student models are trained on reasoning samples generated by these LLMs, it is possible that such characteristics of the teacher model can get passed along to the student. This is an important point to consider when selecting the teacher model for our method.
Our training setup, however, does offer a unique opportunity to minimize bias and toxicity in student models, by influencing the samples used for fine-tuning. One approach would be to augment the curating step of Fine-tune-CoT to filter out biased or toxic samples. It is possible to automate this via neural network-based verifiers, previously used to filter correct output (Cobbe et al., 2021; Li et al., 2022b). Alternatively, one may consider optimizing the CoT prompts to minimize bias and toxicity in teacher-generated rationales.
We note that bad actors can also potentially take advantage of our method to utilize complex reasoning for malicious purposes and deploy it at scale, using small models. This highlights the importance of safeguarding the potential capabilities of LLMs by major providers. To prevent the distillation of malicious reasoning abilities in small (or large) students, future work in identifying usage patterns involved in these distillation schemes may help providers apply more stringent precautions to these use cases.

heading: Acknowledgements
text: This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by Korea government (MSIT) [No. 2021-0-00907, Development of Adaptive and Lightweight Edge-Collaborative Analysis Technology for Enabling Proactively Immediate Response and Rapid Learning, 90%], [No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST), 5%], and the Stochastic Analysis and Application Research Center (SAARC) under the National Research Foundation of Korea grant (NRF-2019R1A5A1028324, 5%).

heading: A Experimental Details
text: A.1 Generation
Maximum sequence length For the maximum sequence length of teacher-generated rationales, r̂i, we use Lr = 128, following Kojima et al. (2022), unless stated otherwise. For the maximum sequence length of the student model predictions, we use Lp = 1024, unless stated otherwise. We retroactively applied Lp = 1024 as the default, after discovering that Lp = 128 is insufficient for many tasks, as discussed in Appendix E.2.
Sampling temperature We apply greedy decoding for all generations, except diverse reasoning, to obtain deterministic results following (Wei et al., 2022b; Kojima et al., 2022). For diverse reasoning, we use temperature sampling with T = 0.7 to obtain diverse samples, following a similar approach from Wang et al. (2022).
A.2 Answer cleansing
We follow the method used in Kojima et al. (2022) to cleanse answers generated by models to assess their correctness.
A.3 Few-shot-CoT exemplars
For Few-shot-CoT prompting, we use exemplars provided by Wei et al. (2022b), with some minor formatting adaptations for consistency with our other experiments. For Last Letter Concatenation and Coin Flip, for which Few-shot-CoT prompts are not provided, we use 8 training samples used in our 8-shot data experiments shown in Figure 4 and adapt them for Few-shot-CoT using the format of Wei et al. (2022b). This was not applicable to Tracking Shuffled Objects, therefore it was omitted from Few-shot-CoT experiments.
A.4 Fine-tuning OpenAI models
We use default hyperparameters set by the OpenAI API for both vanilla fine-tuning and Fine-tune-CoT. While the specifics of the fine-tuning API is not publicly known, some details on hyperparameters are documented in the API reference 6. According to the default settings, our models are trained for 4 epochs. The batch size and learning rate determined based on the number of examples used for training. The batch size is set to 0.2% of the number of training examples capped at 256. The
6https://platform.openai.com/docs/api-reference/finetunes/create
learning rate is set to 0.05, 0.1, or 0.2 times that of the learning rate used to pre-train the base model, depending on the batch size. Training loss is also applied to the prompt portion of the training examples, i.e., the question, with a small weight of 0.01. Based on API pricing , we posit that OpenAI employs a form of parameter efficient fine-tuning such as LoRA (Hu et al., 2021) for their fine-tuning API instead of updating all model parameters.
A.5 Fine-tuning open source models
For vanilla fine-tuning and Fine-tune-CoT on open source models, we strictly control for hyperparameters. Across all experiments, we fine-tune the entire model with a fixed learning rate of 3e-4 and batch size of 8. Upon inspection of model performance under various learning rates and batch sizes, we found that optimal parameters varies among datasets, even between those with similar number of reasoning samples. We train all models for a maximum of 20 epochs, which we found to be sufficient for test accuracy to plateau. We report the best test accuracy from 20 epochs, but found that performance varies significantly between epochs. Overall, we found that performance by epoch is stable for larger models, and that instruction-tuned Flan-T5 is more stable compared to T5. Similar to learning rate and batch size, the optimal number of epochs also varies between datasets, even those with similar number of reasoning samples. Based on the above, we note that our reported performances of fine-tuned open-source models may be significantly under-estimated compared to those with optimal hyperparameters, and recommend practitioners to optimize hyperparameters using a separate validation set, per each training setting.

heading: B Datasets
text: We provide a summary of datasets used in our experiments, including their original licenses, in Appendix Table 3. We consider the 12 datasets used in Kojima et al. (2022) to measure reasoning performance. For Last Letter Concatenation and Coin Flip, we use the publicly available data provided by Kojima et al. (2022).
Train-test split Contrary to previous works on prompt-based CoT such as Wei et al. (2022b); Kojima et al. (2022), our fine-tuning approach requires distinct sets of samples for training and testing. If
separate subsets for training and testing (or development) are provided, we use those. Otherwise, we perform a samplewise random split with a train-test ratio of 70:30. For AQUA, due to the disproportionately large size of the original training set, we randomly sample 10,000 instances for training in our experiments. Note that due to the highly templated nature of many datasets, this naive data split may not be appropriate for evaluating reasoning capabilities. This is an important nuance of fine-tuning on CoT reasoning, which we address in Appendix E.3.

heading: C Models and API Usage
text: Appendix Table 4 describes all teacher and student models used in our study. We use InstructGPT (Ouyang et al., 2022) as the default teacher model in our experiments, due to its superior zero-shot reasoning performance, compared to GPT-3 (Brown et al., 2020) of the same size (Kojima et al., 2022).
Specifically, we use text-davinci-002 at the default, as it was the best available model at the start of our experiments. We were unable to consider small InstructGPT models for fine-tuning, as it is not offered by the OpenAI API. We attach model size information based on (https://blog.eleuther.ai/gpt3-model-sizes/), following Kojima et al. (2022).
Our total expenditure for API usage, including all preliminary experiments, was $1,981 USD. The majority of this expenditure occurred after September 1st, 2022, from which the pricing for inference on davinci models was $0.02/1K tokens, among others. Between teacher model inference, student model {fine-tuning, inference}, the majority of API usage in terms of cost was focused on teacher model inference.

heading: D Sample Study
text: To understand where our method makes mistakes, where diverse reasoning can improve performance, and where our method always performs well, we observe randomly sampled instances and analyze the reasoning performance on them. To do so, we compare its generations for these samples with (1) the output of the large teacher model, (2) a student model using Zero-shot-CoT (3) a student model using Few-shot-CoT, and (4) a student model using fine-tuning without chain of thought reasoning. Our analysis reflects our overall findings, which we exemplify with representative examples in Tables 10–13.
D.1 Error analysis
For our analysis of the most common types of errors, we take a look at datasets where we find particularly bad performance of our vanilla method, also in comparison to other students. We also dis-
cuss the benefits of using diverse reasoning in D.2. We summarize our observations below.
Difficult datasets First, we observe that the sets GSM8K and AQUA appear to be too difficult for any small student model, in particular, given that the teacher model gets below 50% accuracy on both. In fact, even correct answers are usually correct only by chance, due to the high complexity of the tasks (Appendix Tables 10a,b). For AQUA in particular, we note that while we occasionally find meaningful reasoning in the 6.7B student model, students clearly cannot sufficiently learn to solve the tasks. We do note however that of all the student methods, Fine-tune-CoT still gets the best performance in these two datasets. A similar, if less salient, issue arises for StrategyQA. Here, the teacher also performs only 3% above the random guess accuracy of 50%. The smaller student models actually manage to improve on this performance as long as they do not use Zero-shot-CoT, in particular vanilla fine-tuning, but the errors arising in Fine-tune-CoT often look very similar to the ones in the large teacher model. We see that all models usually merely retrieve information related to the question, but cannot synthesize an answer from it (Appendix Tables 10c,11a).
Arithmetic mistakes Next, we note that small models overall exhibit weak arithmetic skills. This has already been discussed in previous literature, where calculation capability has been found to scale with model size (Wei et al., 2022a). Especially in SingleEq (Appendix Table 11b) and AddSub (Appendix Table 11c), a majority of errors in the output of student models using Finetune-CoT simply arise from wrong calculations, less so bad reasoning. This is also a major factor in the bad performance our method exhibits on SVAMP as well as GSM8K; even correct multistep reasoning cannot compensate for the fact that the model’s arithmetic tends to be wrong on intermediate steps (Appendix Tables 11d, 12a). Only the teacher model then does better on these tasks, given its much larger size, even though it does not get perfect accuracy either. However, we note here that very large language models, such as PaLM 540B, can be trained on arithmetic and scientific data to be able to reason correctly about a wide range of mathematical tasks in a step-by-step fashion (Lewkowycz et al., 2022).
Problematic benchmarks, impact of commonsense reasoning errors Meanwhile, when looking at our method’s performance in CommonsenseQA, we note that producing consistent multistep reasoning is not always the issue. We find that the student model utilizing Fine-tune-CoT can often generate logical reasoning paths for many of the samples that are marked as false (Appendix Table 13b). Rather, the exact answer is often subjective, making it difficult to guess the correct output from logical reasoning alone (Appendix Table 13c). CommonsenseQA thus is not always an ideal benchmark when judged on accuracy, but gives insight into how well the model can produce reasoning. We also note a difference compared to Few-shot-CoT in terms of the impact of reasoning errors: the latter only performs around 5% above random, lacks understanding of the question in many cases, and makes more severe logical and commonsense mistakes compared to our method. In fact, Fine-tune-CoT comes close to the teacher due to the relatively lower impact of errors that do arise (Appendix Table 13d). This suggests that Fine-tune-CoT enables stronger task-solving capabilities and avoids making serious commonsense mistakes that prevent it from arriving at a reasonable conclusion.
Aligned failures Importantly, we note that for each dataset, there seems to be a difference between “easy” and “hard” instances. When we consider the accuracy of the teacher and other student models (using fine-tuning, Zero-shot- or Few-shot-CoT) on tasks where our method fails, we find that it is always lower than on tasks where our method is successful. That is, successes and failures tend to be aligned across the different methods. We can hypothesize that factors such as content bias may play a role here; language models have been found to fail depending on context and content of the task, in a way similar to human reasoners (Dasgupta et al., 2022). We can identify samples that hint at this issue when we look at questions that include phrasing that seems contradictory or counterintuitive to the context that the model expects (see Appendix Table 13d, where the number of movies watched is larger than the number of available movies). Additionally, previous work shows that GPT-3 exhibits a performance gap between instances including terms that are frequent in the pretraining corpus, and instances including less frequent terms (Razeghi et al., 2022). This can
contribute to uneven performance on a multitude of (especially numerical) tasks across different methods and model sizes. We can then surmise the observed absolute differences in accuracy to stem from the various sources of errors for each method. For example, fine-tuning has much less room for error than Fine-tune-CoT, which can additionally make mistakes on intermediate reasonings such that errors compound.
D.2 Improvements from diverse reasoning
Semantic issues We find that models seem sensitive to how a question is formulated. This is noticeable in all datasets, in particular in SVAMP and to a certain degree in MultiArith. Besides arithmetic mistakes, we observe that such semantic issues are one of the main factors for uneven performance of vanilla Fine-tune-CoT on these two datasets.
In particular, we observe this issue when there is redundant information present in the question (Appendix Table 12b). Such cases elicit wrong reasoning, or lead the model to become stuck on the question, similarly to what usually happens with Zero-shot-CoT in the student model (i.e. repeating the question, or coming up with information that only vaguely pertains to the question). Other common sources of errors are when hidden variables make up the first part of the task (i.e. those tasks that force the model to calculate a previously unknown value that is described in the first sentence (Appendix Table 12c), or when the model encounters overloaded words (e.g., “landing” in Appendix Table 12d). We also observe samples where the model gets stuck on an intermediate result (Appendix Table 13a). This observation agrees with previous findings that language models have recency bias (Zhao et al., 2021).
However, this source of errors can be compensated for by using diverse reasoning. When comparing the generations from Few-shot-CoT, vanilla Fine-tune-CoT and Fine-tune-CoT with diverse reasoning on MultiArith, we find that diverse reasoning enables the model to understand the question better. While calculation errors are still relatively frequent, the generations show clear advantages in terms of semantic understanding and being able to reason logically as a consequence. This is especially clear when compared to Fewshot-CoT, which exhibits problems both in understanding the question and formulating coherent expressions, especially when three or more terms
are involved in the calculation, as mentioned in Kojima et al. (2022). By contrast, Fine-tune-CoT with diverse reasoning makes for a significantly smoother reasoning performance than using Fewshot-CoT or even vanilla Fine-tune-CoT. This results in vastly improved accuracy on both MultiArith and SVAMP.
D.3 Strengths Having analyzed the main sources of errors, we can now focus on the datasets that elicit good performance from our method, regardless of whether we use diverse reasoning.
Text-based datasets As arithmetic errors are one of the main reasons for the decrease in performance of small student models, it comes as little surprise that our vanilla method without diverse reasoning performs well on datasets that are mainly text-based and do not require actual calculation skills. This includes Date Understanding (60.4%) (Appendix Table 14a), Last Letter Concatenation (52.67%) (Appendix Table 14b), Coin Flip (98.7%) (Appendix Table 14c), and Shuffled Objects (64.4%) (Appendix Table 14d). Our methods performs significantly above random choice on these sets, and additionally beats the teacher on Shuffled Objects and Coin Flip. We find that accuracy metrics for these sets are mostly faithful: while the elicited reasoning is not always very detailed, and occasionally misses some reasoning steps (Appendix Table 14e) , the model draws correct conclusions from mostly correct steps. We also note that similar to MultiArith and SVAMP, performance on these four datasets can be even further boosted with diverse reasoning, outperforming the teacher model across all four.
Patterns These datasets also have very clear patterns in their tasks, which helps Fine-tune-CoT to perform well by providing cues on how to solve a specific task. We note that in contrast, classic fine-tuning does not have an advantage in these datasets, and it gets significantly lower accuracy than Fine-tune-CoT on all four. The same is also true for MultiArith, which we have used as a benchmark in the main text. While arithmetic errors cause the absolute accuracy of our method to be lower than the teacher, it significantly outperforms fine-tuning on MultiArith even without using diverse reasoning. Indeed, we find that also in the presence of arithmetic errors, our model reasons correctly in many cases. We can surmise that the
patterned nature of the tasks in MultiArith helps the student model to understand what is asked of it, eliciting the correct reasoning. Additionally, we note that the presence of such patterns in successful datasets does not mean that our method overfits to existing templates. In our template-split analysis (Appendix E.3), we in fact show that while tasks look similar to one another in certain datasets such as Date Understanding, the student model’s reasoning does not rely on simply matching templates or memorizing particular solutions. This implies that our method can generalize to previously unseen tasks; the patterns in the datasets do not produce overfitting, but can be surmised to act as cues for the model’s understanding of its current task. Thus, we observe that the reasoning skills of a student using Fine-tune-CoT can overcome the smaller model capacity (which proves to be completely prohibitive, e.g., for Zero-shot-CoT to have any success on the various tasks).

heading: E Nuances of Fine-tune-CoT
text: E.1 Rationale filtering
We investigate whether answer-based filtering is sufficient for selecting good teacher-generated reasoning samples. It is possible for the teacher model to answer correctly despite incorrect reasoning, especially in multi-choice questions where the random-guess probability is significant. To investigate the potential impact of a better filtering scheme (as opposed to our baseline answer-based filtering) we manually annotate the correctness of rationales from the teacher model and evaluate student performance when fine-tuning on correctly reasoned samples. We use the Date Understanding dataset for this ablation, as it is comprised of well-grounded multi-choice questions for which Fine-tune-CoT achieves adequate performance. Appendix Table 6 compares the Fine-tune-CoT performance of student models on Date Understanding when using correct samples filtered based on answer predictions vs golden samples, hand-picked based on the correctness of rationales. For golden samples, we exclude samples that contain incorrect reasoning steps or irrelevant steps which are misleading. We find that 28% of correct samples have incorrect rationales–significantly more than the randomguess performance of 17.12%, indicating the importance of filtering. Surprisingly, we however find that answer-based filtering outperforms the more stringent human filtering by 5-11%, given the same
initial samples. When we match the number of samples post-filtering (via undersampling), we do find that fine-tuning on golden samples outperforms that on correct samples by 5-8%. These results suggest that there is a tradeoff between the quality and quantity of reasoning samples which must be addressed when considering sample-filtering methods. We also note that this must be considered in tandem with diverse reasoning, which can drastically increase the quantity of reasoning samples.
E.2 Maximum sequence length Following the original setting for Zero-shot-CoT (Kojima et al., 2022), we limit the max sequence length, or max tokens, allowed for the teachergenerated rationale and student reasoning predictions, denoted Lr, Lp, to 128 initially. However, we find that this can be insufficient in many datasets. Allowing for longer inference, we observe that model performance improves significantly on AQUA and commonsense reasoning tasks (Appendix Table 5). Sample inspection shows that rationales with over ∼500 tokens are typically repetitive or too digressive. To investigate the effect of the max length Lr of the teacher rationale on fine-tuning, we compare student performance using Lr = {128, 512} (Appendix Table 7). The effect of Lr on student performance varies across datasets, and increased Lr does not necessarily improve student performance on tasks that require longer rationales, such as AQUA. Finally, we examine the length distribution of the generated rationales from the teacher model and student trained on short (Lr = 128) and long (Lr = 512) reasoning samples, respectively (Appendix Figure 7). We find that the distribution is different for each dataset. Notably, we find that while the distributions from the long students were similar to that of the teacher, the generated rationale from the short students were typically limited to less than ∼128 tokens. These findings are in line with the intuition that different tasks require different lengths of rationales, and suggest that careful consideration is needed in determining parameters related to sequence length.
E.3 Templated datasets Upon inspection, we found that many datasets contain groups of samples which share common templates. Therefore, naive samplewise data split has the potential to leak the same templates into the train and test sets, essentially demoting the learn-
ing problem into simple pattern matching, rather than complex reasoning. This brings into question the validity of naive samplewise data split, as it has the potential to leak the same templates into the train and test sets. To investigate whether the student models are truly learning to reason rather than matching simple patterns, we manually group samples by template and evaluate Fine-tune-CoT using a template-wise data split. We consider MultiArith and Date Understanding as they contain a moderate number of templates. Note that all datasets excluding GSM8K, CommonsenseQA, and StrategyQA contain templates to varying degrees. Appendix Table 8 shows the performance of Fine-tune-CoT when using sample-wise vs template-wise split, using the same train-test ratio of 70:30. While student performance is typically lower with a templatewise split, it still significantly outperforms random guess performance, as well as prompt-based baselines shown in Appendix Table 1. This reaffirms that Fine-tune-CoT is able to elicit complex reasoning capabilities in small language models.

heading: F Data Annotation vs Diverse Reasoning
text: In Appendix Figure 8, we analyze the cost of data annotation and diverse reasoning, based on current OpenAI API pricing and a low estimate of annotation cost at 30 annotations per hour at an hourly rate of $20, i.e., $0.67 per question-answer sample. When comparing the cost and student performance of models trained with D = 1 and D = 64, we can clearly see that using diverse reasoning can enhance the cost-effectiveness of data acquisition. However, as the cost of diverse reasoning correlates with the size of the dataset, it is important to consider the cost-performance tradeoffs.

heading: G Experiments on Open Source Models
text: To validate the generality of our method, we apply our method to a wide range of student models beyond variants of GPT-3. While the OpenAI API for GPT-3 inference and fine-tuning is accessible and does not require high-end GPUs, the model weights and implementation are not publicly available and may involve black-box processing. We therefore conduct experiments from Section 4 on open-source models under a standard setting with fixed hyperparameters, as explained in Appendix A and report our results in the following. Tables and figures include results from Section 4 on GPT-3 for reference.
Prompt-based baselines A comprehensive performance evaluation of student models across multiple tasks is encapsulated in Table 9, comparing Fine-tune-CoT against baseline methods. Performance of standard zero-shot prompting, predominantly insignificant, is omitted when negligible but does exhibit unexpected spikes on Flan-T5, such as 94.22% on Tracking Shuffled Objects on the smallest model. Few-shot-CoT likewise demonstrates inconsequential performance across most student models, yet the Flan-T5 models reveal significant performance on some tasks such as 7.51% on GSM8K and 83.87% on CommonSenseQA. This hints at the possibility that instruction tuning may empower models to comprehend and execute CoT prompts, unveiling a latent reasoning capacity within smaller language models.
Fine-tune-CoT vs vanilla fine-tuning Further examining Table 9, we note that vanilla fine-tuning achieves notable performance in encoder-decoder architectures, namely T5 and Flan-T5, achieving more than 80% on Date Understanding and 100% on Coin Flip, significantly outperforming vanilla fine-tuning on GPT-2 and GPT-3 student models. This leads us to believe that the causal attention masking present in decoder-only models could impede complex inter-token reasoning. CoT reasoning, in this regard, may serve to mitigate this limitation by repeating key information within the decoding context. Other the other hand, Fine-tune-CoT either surpasses or matches the performance of vanilla fine-tuning across a variety of tasks. Our method also displays consistent scalability with model size, in contrast to the fluctuating performance between model sizes for baseline methods. The incorporation of diverse reasoning enhances
this scalability. Particularly, we find that the FlanT5 models benefit more from Fine-tune-CoT compared to T5 models, implying a favorable role of instruction tuning. When enhanced with diverse reasoning, Fine-tune-CoT excels over vanilla finetuning across several complex reasoning tasks, notably observed in the performance of Flan-T5 on Tracking Shuffled Objects (44.00%→89.33%) and GPT-2 on MultiArith (11.67%→19.44%). Effects of diverse reasoning Figure 9 shows the performance of all student models on MultiArith and SVAMP under varying degrees of diverse reasoning. We observe that performance scales with diverse reasoning in all student models, with the exception of T5-Small. It is shown that diverse reasoning enables Fine-tune-CoT to outperform standard fine-tuning in all cases.
Effects of student model scale Figure 10 shows the performance of all student model families according to model size. While we observe performance scaling for Fine-tune-CoT on GPT-3 models, this is not apparent in other open-source models. We posit that this may be due to under-tuned hyperparameters, as we used fixed hyperparameters for all open-source models, in contrast to default suggested settings for GPT-3.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
3 A2. Did you discuss any potential risks of your work? Section 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
3 A4. Have you used AI writing assistants when working on this paper? We used ChatGPT sparingly for paraphrasing.
B 3 Did you use or create scientific artifacts? We create code for our experiments and provide a link to the anonymized code in a footnote within the abstract. We use 12 existing datasets which are listed in Section 4 and Appendix B.
3 B1. Did you cite the creators of artifacts you used? Appendix B
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We include an MIT license notice in our code repository.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix B
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3, Appendix B
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix C
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4. We use default hyperparameters.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4, Appendix A
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: To prevent the distillation of malicious reasoning abilities in small (or large) students, future work in identifying usage patterns involved in these distillation schemes may help providers apply more stringent precautions to these use cases