ELAD: Explanation-Guided Large Language Models Active Distillation (2024)

Abstract: The deployment and application of Large Language Models (LLMs) is hindered by their memory inefficiency, computational demands, and the high costs of API inferences. Traditional distillation methods, which transfer the capabilities of LLMs to smaller models, often fail to determine whether the knowledge has been sufficiently transferred, potentially resulting in high costs or incomplete distillation. In this paper, we propose an Explanation-Guided LLMs Active Distillation (ELAD) framework that employs an active learning strategy to optimize the balance between annotation costs and model performance. To improve the efficiency of sample selection, we introduce an explanation-guided sample selection method that identifies samples challenging its reasoning by exploiting uncertainties in reasoning explanation steps. Additionally, we present a customized LLM-annotated explanation revision technique where the teacher model detects and corrects flaws in the student model’s reasoning. Our experiments across various reasoning datasets demonstrate that our framework significantly enhances the efficiency of LLMs knowledge distillation.

heading: 1 Introduction
text: The advancement of LLMs (Brown et al., 2020; Hoffmann et al., 2022; Thoppilan et al., 2022; Touvron et al., 2023) has significantly impacted the natural language processing (NLP) domain, showcasing excellent in-context learning and complex reasoning capabilities. Yet, the deployment of these models is hindered by their extensive parameter count, leading to significant computational demands and financial burdens. For instance, deploying LLMs with 100-200 billion parameters would require a cluster of NVIDIA A100 GPUs, where each GPU costs $30, 000 in today’s market. While cloud computing offers a solution, the costs associated with such services can quickly accumulate.
Specifically, a cluster of A100 GPUs costs upwards of $25 per hour, which may lead to a staggering $18, 000 monthly if operated non-stop1. This financial barrier makes it impractical for many institutions and research labs to adopt LLMs widely, especially in resource-constrained environments (Bai et al., 2024), limiting their audience and application scope. Additionally, relying on API calls to access pre-trained LLMs, e.g., GPT-4, also presents its challenges, including high usage fees, inability to run models locally for customization or fine-tuning, potential data transmission issues, and privacy concerns (Yao et al., 2023c).
Recent research on LLMs knowledge distillation (Hinton et al., 2015) enables smaller models to achieve performance similar to LLMs by transferring reasoning capabilities to them, making them more computationally efficient. Tang et al. (2019), Wang et al. (2021), and Arora et al. (2022) demonstrate the training of smaller models using pseudo-labels generated by LLMs, wherein LLMs act as “teachers” to supervise the fine-tuning of these “student” models. Recent works (Magister et al., 2022; Ho et al., 2022; Chan et al., 2022; Zhang et al., 2023; Li et al., 2023; Hsieh et al., 2023; Zhao et al., 2024; Pan et al., 2024a) focus on multi-task fine-tuning of student models. They utilize chain-of-thought (CoT) reasoning (Wei et al., 2022) to generate both explanations and final answers generated by LLMs as pseudolabels to jointly supervise small model fine-tuning. However, a major issue with existing fully supervised learning methods is that they do not sense whether the knowledge has been sufficiently distilled into the small model. Insufficient distillation can lead to suboptimal performance of the small model, while excessive distillation may incur unnecessarily high costs.
To address this challenge, we take LLMs as an
1https://charshift.com/llm-true-cost/
4463
agent that guides small language models toward progressive improvement. Throughout this process, the LLMs (teacher) can sense the weaknesses of the small language model (student) and customize its teaching accordingly. Formally, we propose an Explanation-Guided LLMs Active Distillation (ELAD) framework that significantly enhances active learning through the explanations generated by language models. In each iteration, the framework encompasses a student reasoning task and a teacher reasoning task: the student model identifies samples it struggles to predict accurately and reasonably; subsequently, for these selected samples, the teacher model reviews the student’s explanations, correcting any erroneous reasoning.
However, first, it is nontrivial to tackle the student task. Current sample selection methods typically focus on finding samples with the wrong predictions (Lewis, 1995; Ren et al., 2021; Bansal and Sharma, 2023), but even if the prediction is correct, the reasoning process can be wrong or flawed. The selection of samples with bad reasoning goes beyond it and is yet to be well explored, which requires the student model to faithfully self-inspect its step-by-step explanation of its prediction and locate the flaw. To address this, we propose a novel explanation-guided sample selection method that identifies the samples that trouble its reasoning by exploiting explanation stepwise uncertainties. Second, how the teacher senses and corrects the flaws in student model reasoning is also a challenging problem. Merely prompting teacher and student to generate their respective explanations separately and compare their difference is problematic because a prediction could be led by different reasoning processes and different explanations (e.g., Rashomon Effect (Roth and Mehta, 2002)). We need the teacher model to check the student model’s explanation, locate the problem within its reasoning, and correct it, which is not well explored. To accomplish this, we propose a customized LLMannotated explanation revision technique. It entails sequentially prompting the LLM with the explanation from the small model and then asking it to assess whether the current step is reasonable or necessitates revision.
We evaluate our framework across six reasoning benchmarks, comparing it against existing sample selection methods for active learning and LLMs explanation and answer generation methods. Our findings indicate that the proposed framework notably enhances annotating efficiency.
We summarize our main contributions as follows: a) An Explanation-Guided LLMs Active Distillation framework that enhances active learning, guided by explanations from small models, during the distillation of LLMs to smaller models. b) An explanation-guided sample selection method that identifies the samples that trouble the reasoning of language model by exploiting explanation stepwise uncertainties. c) A customized LLM-annotated explanation revision technique that allows LLM to teach customized knowledge by guiding the LLM to pinpoint and correct inaccuracies in the reasoning steps of small models. d) Extensive experiments demonstrate that the proposed framework can improve annotating efficiency.

heading: 2 Related Work

heading: 2.1 LLMs Knowledge Distillation
text: DistilBert (Sanh et al., 2019) achieves efficient distillation of the BERT transformer into a student model with minimal performance loss. Tinybert (Jiao et al., 2019) introduces a loss term for matching hidden states between teacher and student. Works like Magister et al. (2022), Finetune-CoT (Ho et al., 2022), KNIFE (Chan et al., 2022), SCoTD (Li et al., 2023), and Distilling stepby-step (Hsieh et al., 2023) emphasize multi-task fine-tuning of student models using both CoT reasoning explanations and LLM-generated answers. Distilling step-by-step (Hsieh et al., 2023) specifically uses task-specific prefixes in prompts to tailor model responses. Li et al. (2022) investigate methods for generating explanations to aid student model learning. SOCRATIC CoT (Shridhar et al., 2023) decomposes problems into subproblems to guide reasoning, while SCOTT (Wang et al., 2023) uses teacher-generated explanations for training on a counterfactual reasoning objective, promoting self-consistency.

heading: 2.2 Efficient Annotating and Active Learning
text: Bansal and Sharma (2023) introduced a method using model uncertainty (Lewis, 1995), dataset density (Ren et al., 2021), and conditional informativeness (Bansal and Sharma, 2023) for one-shot informative sample selection to enhance annotation efficiency. However, these one-shot approaches fail to sense the sufficiency of annotation. Addressing this, active learning, particularly the pool-based paradigm, has been recognized as a crucial technique for reducing annotation costs (Krishnakumar,
2007; Ren et al., 2021). For NLP tasks, the most common strategy is based on the entropy of predicted tokens for sampling (Zhang et al., 2022). Additionally, Yao et al. (2023a) proposed a data diversity-based active learning sampling strategy, leveraging explanation annotations, with a humanin-the-loop setting. BSDETECTOR (Chen and Mueller, 2023) introduces an uncertainty quantification technique for black-box LLMs, focusing on consistency (Wang et al., 2022).

heading: 2.3 Explanation Generation for LLMs
text: Extractive explanations (Lei et al., 2016; Yu et al., 2019) focus on identifying key elements within the input that justify a prediction. However, they are limited in explaining complex reasoning tasks that require detailed natural language explanations (free-text explanation) (Camburu et al., 2018; Rajani et al., 2019). Narang et al. (2020) advanced this by training models to generate explanations post-prediction. Self-rationalization models, such as those discussed by Wiegreffe et al. (2020), aim to simultaneously predict labels and generate textbased explanations. STaR (Zelikman et al., 2022) generates explanations by augmenting ground truth answers as hints when predicted answers are incorrect. Wei et al. (2022) introduced CoT prompting, which uses demonstrations in LLM prompting to elicit intermediate reasoning steps for explanations. Kojima et al. (2022) demonstrated the zeroshot reasoning capabilities of LLMs by employing prompts like “Let’s think step by step” to generate an explanation. Tree of Thought (Yao et al., 2023b; Long, 2023) generates reasoning explanations by recursively decomposing complex questions into simpler sub-questions, solving them individually, and integrating their answers.

heading: 3 Preliminary Study

heading: 3.1 LLMs Reasoning
text: In the zero-shot CoT prompting scenario, prompting a question q to an LLM triggers the generation of the completion which consists of an answer a and reasoning path (explanation) r, modeled as (a, r) ∼ P (a, r | q). This process unfolds in an auto-regressive manner, generating r before a and formalizing the conditional probability of the answer as P (a | q) = P (a | q, r)× P (r | q), where P (a | q, r) represents the probability of a given both q and r, and P (r | q) denotes the probability of r given q. In few-shot scenarios, demonstration
triplets {(qpi , a p i , r p i )}mi=1 are included before q in the prompt, facilitating contextual guidance and reasoned answer generation, with m indicating the number of demonstrations in the prompt.
The CoT prompting approach facilitates sequential reasoning in LLMs, generating a series of reasoning steps r = {s1, s2, . . . , sn}, where n is the total number of steps. Each step si contributes cumulatively to the reasoning explanation, culminating in the final answer. Specifically, the probability of generating the explanation r given the question q, denoted P (r | q), is expressed as a product of conditional probabilities, representing the step-bystep reasoning:
P (r | q) = ∏n
i=1 P (si | q, s1, . . . , si−1) (1)
where each si is predicated on the question q and the preceding steps s1, . . . , si−1.

heading: 3.2 LLMs Knowledge Distillation
text: LLMs Knowledge distillation entails a process wherein a large teacher language model T transfers its knowledge to a small student language model S. In this framework, given an unlabeled dataset U , the teacher model generates a pseudo-answer and a pseudo-explanation for each question q in dataset U . These outputs are represented as the answer-explanation pair (completion) (â, r̂). This generation process is modeled as:
(â, r̂) ∼ T (a, r | q) (2)
The result is a collection of triplets {(q, â, r̂)}|U| as the training dataset D. Subsequently, the student model S is fine-tuned using D, employing the standard language modeling loss, formulated as:
maxE(q,â,r̂) ∼D [S(â, r̂ | q)] (3)

heading: 4 Problem Setup
text: In the context of knowledge distillation of LLM T , we address the problem of efficiently annotating an unlabeled question set U . This involves strategically selecting a subset Q from U , where Q ⊂ U and |Q| < |U|. Our study’s primary goal is to enhance the performance of a smaller model S through fine-tuning with a training dataset D = {(qi, âi, r̂i)}|Q|, where the LLM annotates an answer â and a corresponding explanation r̂ for each question q ∈ Q, forming D. The objective is to achieve a performance with S that is comparable
to that of T while minimizing the size of the annotated dataset Q. This approach aims to maximize the efficiency of the small model and minimize the amount of annotated data required from the LLM.

heading: 5 Methodology
text: In this section, we first present the overview of our proposed Explanation-Guided LLMs Active Distillation framework. We then proceed to present a novel explanation-guided sample selection method. Lastly, we present a customized LLM-annotated explanation revision method.

heading: 5.1 Explanation-Guided LLMs Active Distillation Framework
text: We propose a novel Explanation-Guided LLMs Active Distillation framework to optimize the tradeoff between sufficient distillation and annotation costs for LLM knowledge distillation tasks via an active learning strategy. Our overall framework is depicted in Figure 1. We first collect an unlabeled dataset U . At the n-th iteration of active learning, during the sample selection phase, as depicted in Figure 1 (a), we employ the standard pool-based setting. The small model S generates answers and explanations (completions) for all samples q ∈ U , resulting in a set {(qi, ai, ri)}|U|. Then, an explanation-guided sample selection method f (details to be provided in Section 5.2) is adopted to select m samples with high uncertainty in their generated answers and explanations from this set, forming the selected subset. This process can be
represented as
Qn = f({(qi, ai, ri)}|U|;m) (4)
We then create the batch Bn comprising triples {(qi, ai, ri)}m for each q ∈ Qn. Subsequently, we progress to the annotation phase of active learning, as illustrated in Figure 1 (b). The customized LLMannotated explanation revision function g (to be discussed in Section 5.3) annotates completions (â, r̂) for the selected samples in Qn using the LLM T , guided by the completion (a, r) generated by the small model, represented as
Dn = g({(qi, ai, ri)}m; T ) (5)
For each q ∈ Qn, this results in the dataset Dn = {(qi, âi, r̂i)}m for small model fine-tuning. The datasets are updated by removing Qn from U and adding Dn to the cumulative training set D. Finally, in the model update phase of active learning, depicted in Figure 1 (c), we fine-tune the small model S on the training set D. This process is repeated until the LLM annotating (labeling) budget Bu is depleted or other stopping criteria are met (e.g., marginal improvement of the small model falls below a certain threshold). The overall algorithm is summarized in Algorithm 1.

heading: 5.2 Explanation-Guided Sample Selection
text: This section presents the explanation-guided sample selection method to select samples with high uncertainty. This uncertainty stems from the complexity and instability inherent in the step-by-step
Algorithm 1: ELAD Require: U , D, T , S, annotating budget Bu, number of
samples to select each iteration m Ensure: Fine-tuned student model S 1: while Bu > 0 do 2: Generate answers and explanations for samples
from U using small model S 3: Select m most informative samples asQn using Equation 4 to form batch Bn 4: Annotate Bn using LLM T as per Equation 5, obtaining Dn 5: Update D ← D ∪Dn; U ← U \ Qn 6: Retrain S on updated D 7: Bu← Bu−m 8: end while 9: Perform final retraining of S on D
reasoning process. We estimate it across two dimensions: 1) Intra-explanation uncertainty, which explores the uncertainty within individual steps of an explanation, and 2) Inter-explanation uncertainty, which examines the uncertainty across the aggregated answers from different reasoning paths. Intra-explanation uncertainty As stated in Section 3, for an explanation r = {s1, s2, . . . , sn}, each reasoning step si builds upon the question and all preceding steps and influences subsequent steps and the final answer. To address the challenge of estimating the uncertainty in the explanation generation resulting from step-by-step reasoning, we introduce a novel method for estimating intra-explanation uncertainty. This method utilizes a step-wise technique to evaluate the consistency of final answers, whether they are conditioned on specific reasoning steps or not. By comparing outcomes in both scenarios, we effectively measure the uncertainty associated with each step in the explanation. To be more specific, for the i-th reasoning step, the reasoning and answer before the i-th step can be written as:
(a, s≥i) ∼ S(a, s≥i | q, s<n) (6)
Similarly, the reasoning process conditioned on the i-th reasoning step can be written as:
(â, s>i) ∼ S(â, s>i | q, s<i, si) (7)
where â is the sampled answer conditioned on ith reasoning step. The above two scenarios are illustrated in Figure 2. We prompt the small model with (q, s<i) and (q, s≤i) for each of the n steps to obtain the corresponding answers, resulting in a set {(âi, ai)}ni=1 that records the outcomes. The uncertainty of the explanation is then quantified by calculating the frequency of instances where
predictions remain unchanged despite the removal of a reasoning step in the prompt as
HReasoning := 1
n ∑n i=1 I (âi = ai) (8)
where I (âi = ai) is an indicator function that returns 1 (or 0) if the predicted answer is unchanged (or not). This intra-reasoning uncertainty score HReasoning measures the uncertainty of a single explanation.
Inter-explanation uncertainty The answers and explanations generated by language models can exhibit diversity due to the randomness introduced by sampling temperature. To assess the uncertainty arising from this randomness, we propose a consistency-based method for evaluating interexplanation uncertainty. For each question, we apply a multiple-path decoding strategy, prompting the model k times to generate k distinct reasoning paths. This process can produce different final answers, leading to N unique answer values, with each unique value occurring ci times. We assess the consistency among these multiple final answers by calculating the frequency of occurrence ci for every unique answer and subsequently computing the probability of each answer as pi = cik . To quantify the uncertainty in the probability distribution of the output answers derived from multiple promptings, we utilize Shannon entropy, calculated as follows:
HConsistency := − ∑N
i=1 pi log(pi) (9)
This inter-explanation uncertainty score serves as an indicator of the model’s reasoning uncertainty arising from different reasoning paths. Overall Uncertainty Estimation and Sample Selection Based on the two types of uncertainty illustrated in Equations 8 and 9, we define the overall reasoning uncertainty H as:
H = HConsistency + ∑k
i=1 H(i)Reasoning (10)
For all samples in dataset U , we select m samples with the highest uncertainty scores to form
the selected subset Q. Based on the above, the Equation 4 is formalized as:
Q = argmax-m q∈U H(q) (11)
where H(q) denotes the computed uncertainty score for a question q using Equation 10.

heading: 5.3 Customized LLM-Annotated Explanation Revision
text: After selecting the samples the small model troubles the reasoning, the next step is transferring knowledge from the LLM to the small model. This process involves using the LLM to generate pseudocompletion to fine-tune small model. To achieve this, we introduce a customized LLM-annotated explanation revision technique. This approach leverages the advanced capabilities of the LLM to provide customized guidance to the small model by allowing the LLM to detect and correct flaws in the small model’s reasoning.
As Figure 3 (a) and (b) illustrate, our method prompts the LLM to annotate customized completion for the selected questions, conditioned on the output completion provided by the small model to make detection and possible revision (correction), as (r̂, â) ∼ T (r̂, â | q, r, a). Specifically, we employ a Depth-First Search (DFS)-based strategy (Yao et al., 2023b), where for each reasoning step generated by the small model, the LLM is prompted to perform verification to ascertain the validity of the current step. This verification process is iterative, continuing along the current reasoning path until it becomes infeasible to proceed further. At this point, the LLM is prompted to complete the reasoning process by generating the remaining steps and providing the final answer. As depicted
in Figure 3 (c), the process is represented as: q → s1 → . . . → si−1 → × → ŝi → . . . → ŝn → a, denotes the si−1 reasoning step is infeasible. si are the steps generated by the small model, and ŝi are the steps generated (revised) by the LLM.
Based on the above statement, the Equation 5 can be formalized as the following process. We initiate the process with the question and prompt the LLM to determine if the current step si from the small model is valid for problem-solving. If the LLM’s response is “Yes”, we prompt it to generate its own reasoning step ŝi as ŝi ∼ P (ŝi | q, si, ŝ<i). This process is continued until the LLM responds “No” to the i-th step. At this point, we prompt the LLM to generate the remaining reasoning steps and the final answer as (a, ŝ≥i) ∼ P (a, ŝ≥i | q, ŝ<i). The specifics of this verification and prompting process take the following form. Initially, we combine the first reasoning step from the student model s1 with the question q to create the prompt: “For question <q>, can we start with this step: <s1>?” If the LLM’s answer is “Yes”, we adopt the first reasoning step from the LLM ŝ1 as the annotated step and proceed to the next reasoning step. For the i-th reasoning step from the student small model, we define the prompt as: “Can we continue with this step: <si>?” If the response is “No,” we then prompt the LLM with: “What are the rest of the reasoning procedures and the answer?” to generate the subsequent reasoning steps and the final answer. An example is shown below:
Customized LLM-Annotated Explanation Revision
Prompt: For question <q>. Let’s think step by step. Can we start with this step: <s1>? If yes, give me your step. If no, give me the rest steps and the final answer. Response: Yes, we can start with that <ŝ1> Prompt: Can we continue with this step <s2>? Response: Yes, the second step is <ŝ2>. Prompt: Can we continue with this step <s3>? Response: No, we should proceed as <ŝ≥3>, the final answer is <â>.

heading: 6 Experiments
text: Datasets The experiments are conducted on six well-known benchmark datasets across 3 different reasoning tasks: GSM8K (Cobbe et al., 2021) and AQuA (Geva et al., 2021) for arithmetic reasoning tasks, ANLI (Nie et al., 2019) and e-SNLI (Camburu et al., 2018) for natural language inference (NLI) tasks, and StrategyQA (Geva et al., 2021) and CommonSenseQA (Talmor et al., 2018) for
Method Annotating Arithmetic NLI Commonsense
GSM8K AQuA ANLI e-SNLI CommonSenseQA StrategyQA
Teacher: GPT-3.5-turbo
Zero-shot-CoT – 73.45 54.96 68.02 67.67 68.94 69.78
Student: LLaMA-2-7B
Zero-shot-CoT – 10.04 21.07 33.94 28.98 41.28 44.71
Fine-Tuned Student
Random CoT Prompting 28.42 26.86 54.22 48.60 45.66 48.76CLAER 30.31 27.05 57.12 48.56 48.54 50.89
Maximum Entropy CoT Prompting 27.58 27.67 52.56 47.98 46.35 49.03CLAER 29.04 27.42 53.75 51.76 48.86 51.05
Least Confidence CoT Prompting 28.42 25.8 52.26 48.21 45.93 47.53CLAER 28.68 27.19 53.63 48.65 48.52 51.23
Disagreement CoT Prompting 30.11 25.91 55.59 50.32 48.64 48.60CLAER 31.49 27.23 58.71 54.32 52.46 53.81
Self-Confidence CoT Prompting 26.41 26.04 52.69 46.01 48.53 49.69CLAER 27.95 25.57 54.32 49.21 49.03 52.44
common sense reasoning task. Further details are provided in the Appendix A.
Evaluation Metric To assess question-answering performance for the above reasoning tasks, we calculate the accuracy based on the final answers given by the student small model.
Setup We use GPT-3.5-turbo as a teacher via OpenAI API. We use LLaMA-2-7B (Touvron et al., 2023) as our student model. Further implementation details are provided in the Appendix A.
Baseline Methods We compare the proposed ELAD framework with two different categories of baselines: 1) sample selection methods, and 2) completion generation methods. To be more specific, we provide a comparison of ExplanationGuided Sample Selection (EGSS) with five different sample selection methods: Random, Maximum Entropy (Krishnakumar, 2007), Least Confidence (Culotta and McCallum, 2005), Disagreement (Vote Entropy) (Engelson and Dagan, 1996), and Self-Confidence (Kadavath et al., 2022). Further, we delineate the efficacy of our Customized LLM-Annotated Explanation Revision (CLAER) method in contrast to the conventional vanilla annotating with zero-shot-CoT prompting (Kojima et al., 2022) method applied post-sample selection. We also include comparative results from student and teacher models, assessed without fine-tuning
and using direct prompting to answer questions.

heading: 6.1 Results and Analysis
text: This section evaluates the reasoning performance of models using our proposed ELAD framework, comparing it with baseline methods. We highlight improvements in sample selection and completion generation. Performance trends from a 5% to 50% annotating budget are depicted in Figure 4, illustrating the effectiveness of our EGSS method against other selection baselines. Furthermore, we detail reasoning performance at a 50% annotating budget for our CLAER completion annotation method and the CoT prompting baseline in Table 1. Comparison with sample selection baselines From Table 1 we can observe that the EGSS method demonstrates significant performance improvements compared with traditional sample selection baselines for active learning. For arithmetic reasoning tasks (GSM8K and AQuA), EGSS with CLEAR exhibits a remarkable performance advantage. Specifically, it shows an increase of approximately 2.41% and 1.38% in accuracy over the next best-performing method for GSM8K and AQuA, respectively. In the context of natural language inference and commonsense reasoning tasks, such as ANLI, e-SNLI, CommonSenseQA, and StrategyQA, EGSS continues to set the benchmark. For
instance, in the ANLI dataset, EGSS achieves a performance boost of nearly 3.27% over the Least Confidence method with CLEAR. Similarly, for StrategyQA, EGSS demonstrates a substantial increase of 4.82% in accuracy compared to the Disagreement strategy. From Figure 4, It is evident that the proposed EGSS method effectively selects the most informative unlabeled questions, as evidenced by performance gains that align with increases in annotation budget. Initially, differences between EGSS and Disagreement strategies are minimal, likely due to the dominance of Inter-explanation uncertainty. However, as the annotation budget grows, EGSS significantly outperforms the Disagreement strategy, highlighting the crucial role of Intra-explanation uncertainty in identifying the most valuable samples for annotation. Evaluating customized LLM-annotated explanation revision method Table 1 showcases the CLAER technique’s effectiveness over the baseline CoT Prompting across several tasks. In arithmetic tasks like GSM8K and AQuA, CLAER outperforms vanilla CoT Prompting annotation method by up to 2.71% and 1.52%, respectively, under the EGSS framework, highlighting its superior capability in refining reasoning skills. In NLI and Commonsense Reasoning tasks, such as ANLI and StrategyQA, CLAER demonstrates notable accuracy improvements of 2.15% and 5.31%, respectively. These results underline the method’s strength in leveraging detailed explanations to enhance model understanding and performance significantly.

heading: 6.2 Ablation Studies
text: In this section, we conduct an ablation study to investigate the importance of each component in the ELAD framework we propose, and the results are reported in Table 2. The results reveal that the full proposed ELAD framework outperforms configurations lacking EGSS (w/o EGSS) and CLAER (w/o CLAER) across all tasks. Ours demonstrates a notable performance advantage, with improvements up to 2.41% in arithmetic tasks, 6.88% in NLI tasks, and 5.09% in commonsense reasoning tasks over the “w/o EGSS” setup. This highlights the critical contributions of EGSS and CLAER to the framework’s overall performance. The diminished performance in configurations without these components underscores their importance in enhancing model reasoning ability. Setting Arithmetic NLI Commonsense GSM8K AQuA ANLI e-SNLI CommonSenseQA StrategyQA ELAD (Ours) 32.72 28.43 58.02 54.44 53.53 55.63 w/o EGSS 30.31 27.05 57.12 48.56 48.54 50.89 w/o CLAER 30.01 26.91 55.87 51.16 49.64 50.32
Table 2: Ablation Study. We report the performance of our ELAD framework under different settings.

heading: 7 Conclusion
text: This paper introduced the Explanation-Guided LLMs Active Distillation framework to address the challenges of deploying LLMs due to the high memory and computational demands. Our proposed framework provides an efficient distillation
of LLMs, incorporating an active learning method. This includes an explanation-guided sample selection technique and a customized LLM-annotated explanation revision method tailored for active learning. Extensive experiments on various reasoning datasets demonstrate the effectiveness of our approach in enhancing the distillation efficiency.
Limitations
Our proposed ELAD framework, which utilizes LLMs as agents in active learning, is influenced by the design of prompts for the LLM, potentially affecting the quality of generated explanations and answers. Similarly, the prompt design for the small model can impact its reasoning abilities. Additionally, our approach requires submitting questions (data) to third-party services via APIs (e.g., OpenAI), posing a risk of data leakage. Additionally, due to budget constraints, we did not utilize the most recently released GPT-4.0 or other LLMs as the teacher model in our experiments. We plan to explore this in future research.
Ethical Considerations
All datasets and models used in this study are opensource, and references to previous work are properly cited. For fine-tuning the smaller student language model, we solely used triples generated by GPT-3.5-Turbo and LLaMA-2-7B, both of which are publicly accessible. This work complies with ethical guidelines, and no ethical concerns have been identified.

heading: Acknowledgments
text: This work was supported by the NSF Grant No. 2432418, No. 2414115, No. 2007716, No. 2007976, No. 1942594, No. 1907805, No. 2318831, Cisco Faculty Research Award, Amazon Research Award.

heading: A Appendix
text: Datasets Details
We provide more detailed descriptions of the datasets used in our experiments. We include a more detailed introduction and original sources released from the authors as follows: GSM8K (Grade School Math 8K) (Cobbe et al., 2021): A dataset containing approximately 8,000 math word problems designed for grade school students, testing a variety of mathematical skills in natural language. For more information, visit the GSM8K GitHub repository. AQuA (Algebra Question Answering) (Geva et al., 2021): Features algebraic word problems with multiple-choice answers, aimed at evaluating algebraic problem-solving in AI systems. Available on Kaggle. ANLI (Adversarial Natural Language Inference) (Nie et al., 2019): A dataset with natural language inference tasks, including adversarial examples, to test models’ understanding of human language beyond existing NLI datasets. For more details, refer to the ANLI GitHub repository. e-SNLI (Explainable Stanford Natural Language Inference) (Camburu et al., 2018): Extends the SNLI dataset by providing human-annotated explanations for NLI decisions, assessing models on inference and explanation generation. Visit the e-SNLI GitHub repository for more information. CommonSenseQA (Talmor et al., 2018): A question-answering dataset that focuses on commonsense reasoning, requiring an understanding of everyday concepts for correct answers. More details can be found on the CommonSenseQA website. StrategyQA (Geva et al., 2021): Tests models on strategic question answering, particularly on reasoning about implicit strategies for yes/no questions. Information is available on the AllenAI website.
For each dataset where a validation set is not originally provided, we randomly subsample 10% of the original training set to serve as a validation set. The dataset statistics are provided in Table 3.
Implementation Details
The main experiments were conducted on a single NVIDIA GTX 3090 GPU with 24GB of memory. We utilize QLoRA4 (Dettmers et al., 2023) by default to conduct parameter-efficient fine-tuning. We set the attention dimension as 64, the alpha parameter as 16, and the dropout rate as 0.1. For fine-tuning the Llama-2-7b-chat-hf model, we utilized the Huggingface libraries2 and set the batch size to 16 and the learning rate to 1e-4 using the AdamW optimizer.
Examples for Customized LLM-Annotated Explanation Revision
We present two examples of Customized LLMAnnotated Explanation Revision. Example 4 illustrates a case where the small model makes incorrect reasoning in step 3. The explanation from the small model is sequentially prompted to LLMs for verification. At step 3, the LLM responds with “No” and proceeds to revise step 3 and generate the remaining steps and the final answer. Example 5 provides an example where the small model errs in step 1 of its reasoning. The step 1 from the small model fails the LLM’s verification, and LLM directly generates the entire completion.
Discussion of API Costs
The proposed CLAER method leverages LLM APIs, such as GPT-3.5 and GPT-4, to generate pseudo answers and explanations. The costs associated with these API calls are based on the number of input and output tokens, with output tokens priced significantly higher than input tokens—approximately three times higher3. Compared to the zero-shot CoT prompting method, which generates both an answer and an explanation in a single prompt, our CLAER method does not substantially increase the output token counts. In fact, it can even result in fewer tokens if the
2https://huggingface.co/meta-llama/Llama-2-7b-chat-hf 3https://openai.com/api/pricing/
initial explanation steps input is accurate and succinct. Also, our designed prompts are not long, which does not substantially increase the input token counts (Pan et al., 2024b; Zhang et al., 2024). Therefore, the main additional cost arises from input tokens, which are considerably less expensive than output tokens. Our analysis, based on comparing the average input and output token counts between CLAER and zero-shot CoT prompting, suggests that the incremental costs are minor relative to the benefits of significantly reducing the need to annotate large numbers of unlabeled samples through an efficient sampling strategy.

heading: Future Work
text: We plan to explore this in future research. Ethical Considerations
All datasets and models used in this study are opensource, and references to previous work are properly cited. For fine-tuning the smaller student language model, we solely used triples generated by GPT-3. 5-Turbo and LLaMA-2-7B, both of which are publicly accessible. This work complies with ethical guidelines, and no ethical concerns have been identified