BayesKD: Bayesian Knowledge Distillation for Compact LLMs in Constrained Fine-tuning Scenarios
Authors: Wei Li, Lujun Li, Mark G. Lee, Shengjie Sun, Lei Zhang, Wei Xue, Yike Guo
Accepted at: acl25

BayesKD: Bayesian Knowledge Distillation for Compact LLMs in Constrained Fine-tuning Scenarios

Wei Li1*   Lujun Li2*   Mark Lee1   Shengjie Sun3   Lei Zhang4   Wei Xue2
Yike Guo2
1University of Birmingham   2HKUST   3AISpeech Co., Ltd.   4University of Exeter
WXL885@student.bham.ac.uk, lilujunai@gmail.com, M.G.Lee@bham.ac.uk
shengjie.sun@aispeech.com, l.zhang6@exeter.ac.uk, weixue@ust.hk, yikeguo@ust.hk

Abstract
Large language models (LLMs) have revolutionized various domains with their remarkable capabilities, but their massive parameter sizes pose significant challenges for fine-tuning and inference, especially in resource-constrained environments. Conventional compression methods often result in substantial performance degradation within LLMs and struggle to restore model quality during fine-tuning. To address this challenge, we present Bayesian Knowledge Distillation (BayesKD), a novel distillation framework meticulously designed for compact LLMs in resource-constrained fine-tuning scenarios. Departing from conventional LLM distillation methods that introduce time-consuming paradigms and fail to generalize in compressed LLM fine-tuning scenarios, our BayesKD develops the Logits Dual-Scaling, Knowledge Alignment Module, and Bayesian Distillation Optimization. In particular, our Logits Dual-Scaling strategy adaptively aligns the strength of the teacher’s knowledge transfer, while the Knowledge Alignment Module bridges the gap between the teacher and student models by projecting their knowledge representations into a shared interval. Additionally, we employ Logits-Aware Bayesian Optimization to swiftly identify optimal settings based on these strategies, thereby enhancing model performance. Extensive experiments across diverse tasks demonstrate that BayesKD consistently outperforms baseline methods on various state-of-the-art LLMs, including LLaMA, Qwen2, Bloom, and Vicuna. Notably, our BayesKD achieves average accuracy gains of 2.99% and 4.05% over standard KD for the 8B parameter LLaMA and Qwen2 model. Codes are available in the supplementary materials.

1 Introduction
The rapid advancements in Large Language Models (LLMs) have revolutionized natural language processing (Wei et al., 2022b,a), yet the substantial size and computational demands of these models pose significant challenges for practical deployment (Zhang et al., 2022). To address these challenges, various LLM compression techniques (Zhang et al., 2023b), such as pruning (Sun et al., 2024a), have emerged to develop streamlined LLMs, thereby boosting inference efficiency. Nonetheless, these compressed models often experience notable performance degradation and struggle to regain their original quality through additional fine-tuning (Ma et al., 2023).

Problem Statement: A promising approach to mitigating such performance gaps is to leverage the original dense model as a teacher for Knowledge Distillation (KD). However, current LLM distillation methods (Gu et al., 2023) encounter two key limitations: (1) They impose substantial computational overhead due to complex distillation paradigms, and (2) they are primarily designed for normal LLMs rather than compressed ones. For example, while MiniLLM (Gu et al., 2023) improves the performance of the original LLaMA 7B by 5%, its gain diminishes to only 1% on the sparse LLaMA 7B (Table 9). These failures are likely attributable to the immense gap between compressed LLMs and their teacher counterparts, which poses difficulties for general distillation methods.

While recent KD approaches have explored various angles. Teacher Assistant (TA)-based

<table>
  <tr>
    <th>Scaling Values</th>
    <th>BoolQ</th>
    <th>ARC_C</th>
    <th>OPQA</th>
    <th>PIQA</th>
    <th>SST-2</th>
  </tr>
  <tr>
    <td>8</td>
    <td>60.08</td>
    <td>26.56</td>
    <td>22.55</td>
    <td>69.81</td>
    <td>58.55</td>
  </tr>
  <tr>
    <td>16</td>
    <td>56.27</td>
    <td>26.61</td>
    <td>22.11</td>
    <td>70.82</td>
    <td>75.92</td>
  </tr>
  <tr>
    <td>20</td>
    <td>54.31</td>
    <td>26.54</td>
    <td>21.95</td>
    <td>70.95</td>
    <td>80.56</td>
  </tr>
</table>

Table 1: Hyperparameter sensitivity experiments: Performance generally improves with higher scaling value in TinyLLaMA1.1B, but tasks like BoolQ and OpenbookQA show a decline, indicating potential hindrances in knowledge transfer.
methods (Mirzadeh et al., 2020; Zhang et al., 2023a) improve distillation performance by inserting intermediate-capacity models between the teacher and student. However, identifying and deploying a TA model with an optimal capacity ratio (e.g., 1.5 times the student’s size) is computationally prohibitive at LLM scales, where even minor capacity adjustments lead to massive parameter counts and increased training costs. Similarly, logit standardization (Sun et al., 2024b), multi-level logit distillation (Jin et al., 2023), and intermediate-layer alignment methods (e.g., Universal-KD (Wu et al., 2021)) have delivered modest gains in smaller or less complex models but fail to yield substantial improvements for LLMs and compressed LLMs. For instance, applying these techniques (logit standardization, multi-level logit distillation, and Universal-KD) to Tiny-LLaMA1.1B and LLaMA-13B improves accuracy to only 52.33%, 51.89%, and 52.53%, respectively, across ten tasks (same tasks with Table 4 except Wiki). Such outcomes highlight that once models undergo compression or is LLM, established KD methods become less robust and struggle to restore substantial performance. Another issue arises from the sensitivity of KD to hyperparameters such as scaling values or temperature factors. Although adjusting logits can enhance certain tasks, it can also degrade others. Table 1 shows this challenge: increasing the scaling value from 8 to 20 boosts SST-2 accuracy from 58.55% to 80.56% but degrades results on BoolQ and OpenBookQA. These findings underscore that a one-size-fits-all scaling strategy is inadequate, and a more adaptive, task-aware tuning mechanism is required.

These observations raise two critical questions:
Why do compact LLMs struggle to learn knowledge edge from their teacher models, and can a more specialized distillation paradigm address these issues effectively for such compressed model fine-tuning scenarios.?

Our New Observations: We identify two factors that significantly influence KD performance in compressed LLMs: (1) Massive Logit Gap. The distributional discrepancy in logits between sparse and dense models is larger than that between dense models of different sizes. This gap intensifies with increasing model scale, contributing to the limited efficacy of current KD methods (Table 4). (2) Hyperparameter Sensitivity. As shown in Table 1, the optimal scaling value for knowledge distillation varies across different tasks. This issues significantly affect the performance of conventional distillation methods.

Our New Search Framework: Building upon these observations, we propose Bayesian Knowledge Distillation (BayesKD), a novel framework designed for LLMs and their sparse models under resource-constrained fine-tuning conditions. BayesKD integrates three core components: First, a logits dual-scaling technique dynamically adjusts teacher and student logits based on their standard deviations, thereby narrowing distributional gaps and enabling more effective, task-aware scaling. Second, a knowledge alignment module employs min-max normalization to better align the intermediate-layer representations of teacher and student models, enhancing the transferability of critical features. Third, to tackle the hyperparameter-sensitive issue, we first drew the paradigm of Bayes search in the LLM-KD field and propose Logits-Aware Bayesian optimization (SABO). Our SABO first builds a search space with Logits Dual-Scaling, Knowledge Alignment Module as the core and different key hyper-parameters distillation position, loss weight, temperature factor as options. To improve the search efficiency, we employ an advanced Bayesian search with faster convergence than random search. To optimize the search cost, we select only 5% of the sub-dataset for searching, which speeds up the search by 20 times compared to searching directly on the original dataset. Finally, we deeply analyze searched distillers and get some guidance: compressed LLM distillation always favors deep intermediate knowledge, logits dual-scaling, and smaller loss weights.

Evaluation and Results: We rigorously evaluate BayesKD on diverse tasks, including WikiText-2 (Merity et al., 2016), OpenBookQA (Mihaylov et al., 2018), HellaSwag (Zellers et al., 2019) and others, utilizing various teacher-student pairs such as LLaMA 13B for Tiny-LLaMA1.1B, LLaMA 7B, Vicuna 7B, and Bloom 7B; LLaMA-3 70B for LLaMA-3 8B; and Qwen-2 72B for Qwen-2 7B. Across these scenarios, BayesKD consistently outperforms baseline methods, delivering an average improvement of 4.4% over standard KD and a 2.6% advantage relative to vanilla LoRA fine-tuning baseline. Notably, the improvements are more pronounced for larger models, highlighting the scalability and robustness of the proposed framework.
The remainder of this paper is organized as follows: Section 2 reviews related work, Section 3 explains our methodology, Section 4 discusses results, and Section 5 concludes.

2 Related Work

Knowledge distillation (KD) (Hinton et al., 2015) methods can be broadly categorized into response-based methods focused on model output (Turc et al., 2020; Li et al., 2020; Jiao et al., 2020) and feature-based methods extracting intermediate layer features (Romero et al., 2015; Sun et al., 2019a; Tang et al., 2019). While these traditional approaches prove effective for BERT-scale models, they become inadequate for LLMs (Gu et al., 2023), where the vast capacity disparities present unprecedented challenges.

The capacity gap between teacher and student models remains a fundamental challenge, first systematically studied through Teacher Assistant Knowledge Distillation (TAKD) (Mirzadeh et al., 2020). Though TAKD effectively bridges smaller model gaps using intermediate networks, it becomes impractical for LLMs where the capacity gap between teacher and assistant often exceeds several orders of magnitude. This limitation significantly impacts the effectiveness of knowledge transfer in the LLM era.

Recent works have addressed these challenges with varying success. Pro-KD (Rezagholizadeh et al., 2022) introduces progressive distillation along the teacher’s training trajectory but focuses mainly on BERT-scale models, leaving larger capacity gaps unexplored. Their method also requires multiple checkpoints, causing significant storage overhead for LLMs. (Zhang et al., 2023a) establishes a linear law for capacity gaps, but tests only models under 3B parameters inheriting TAKD’s limitations in LLM distillation. Newer LLM-specific methods using reinforcement learning, which can better align the outputs of teacher and student LLMs, continue to face efficiency challenges (Ko et al.; Zhong et al., 2024).

Existing knowledge distillation methods face computational overhead and limited effectiveness for compressed LLMs. Capacity gaps challenge smaller models in replicating larger ones. Our BayesKD method bridges the logits distribution gap, enhancing student performance.

3 Methodology

Motivations and Overall Framework: Our BayesKD addresses a critical challenge in KD for LLMs and their sparse models: the pronounced logits distribution discrepancy between sparse and dense models. This discrepancy, which intensifies with model scale, significantly impairs traditional distillation methods. Our empirical analysis (Table. 4 and varying sparsity table in Appendix) reveals that single-temperature approaches, effective for dense model pairs, fail to adequately capture the complexity of compact LLMs. To overcome these limitations, we introduce a three-pronged approach. First, a Logits Dual-Scaling strategy bridges the teacher-student logits distribution gap while addressing the task-dependent scaling value sensitivity. Second, a Knowledge Alignment Module reconciles intermediate layer disparities. Inspired by prior works (Hou et al., 2020; Sun et al., 2019b), we focus on aligning the middle and final layers, which have been shown to capture task-specific knowledge effectively. Finally, a Logits-Aware Bayesian Distillation Optimization method ensures efficient hyperparameter tuning. This comprehensive framework specifically targets the unique challenges of compact LLMs distillation in resource-constrained environments.

3.1 Logits Dual-Scaling Strategy

The Logits Dual-Scaling strategy adjusts logits separately for the teacher and student models to address logits distribution discrepancies and task-dependent sensitivity in distillation. Scaling values are dynamically updated based on logits’ standard deviations during the process, allowing optimal knowledge transfer between models. The loss function \( \mathcal{L}_d \) is defined as:

\[
\mathcal{L}_d = \frac{1}{N^2} \sum_{i=1}^{N} \sum_{j=1}^{M} \gamma \cdot KL [P_{t,i,j}||Q_{s,i,j}]
\]
(1)

\[
P_{t,i,j} = \text{softmax} \left( \frac{T_{i,j}}{\sigma_{t,i}} \right),
\]
(2)

\[
Q_{s,i,j} = \text{softmax} \left( \frac{S_{i,j}}{\sigma_{s,i}} \right),
\]

Where \( P_{t,i,j} \) and \( Q_{s,i,j} \) are the teacher and student logits, \( N \) and \( M \) represent batch size and token length per sample, and \( \gamma \) is a scaling factor. \( T_{i,j} \) and \( S_{i,j} \) are the logits, while \( \sigma_{t,i} \) and \( \sigma_{s,i} \) are their respective dynamic standard deviations. This dynamic adjustment ensures optimal knowledge transfer between teacher and student models.
Figure 1: Schematic overview of our framework, which involves Logits Dual-Scaling, Knowledge Alignment, and Logits-Aware Bayesian optimization. Our Logits Dual-Scaling scaling and Knowledge Alignment enable the student to learn complex relationships and key features, while Bayesian optimization efficiently searches for observed optimal hyperparameters to maximize student performance.

Proof that Logits Dual-Scaling Bridges Teacher-Student Gap: Let the gap between the teacher model \( P \) and student model \( Q \) be:

\[
G(P, Q) = D_{KL}[P||Q],
= \mathbb{E}_u[\log P(v|u) - \log Q(v|u)],
\] (3)

Next, let us set the standard deviations of the teacher and student models, \( \sigma_t \) and \( \sigma_s \), as scaling value for teacher and student, respectively, such that:

\[
t_t = \sigma_t,
\]
\[
t_s = \sigma_s,
\] (4)
(5)

where \( T \) and \( S \) are the logits from the teacher and student models, and \( t_t \) and \( t_s \) are the corresponding scaling value coefficients. The standard deviations \( \sigma_t \) and \( \sigma_s \) of the teacher and student models, and substituting into the gap formula, we get:

\[
G(P, Q) = \mathbb{E}_u \left[ \left( \frac{1}{\sigma_t} - \frac{1}{\sigma_s} \right)^T \right.
\]
\[
\left. - \left( \log \sum_j \exp \left( \frac{T_j}{\sigma_t} \right) - \log \sum_k \exp \left( \frac{S_k}{\sigma_s} \right) \right) \right]
\] (6)

From this expression, we can observe two key scenarios: **Scenario 1** (\( \sigma_t > \sigma_s \)): The overall discrepancy \( G(P, Q) \) increases, primarily due to the dominance of the second term’s positive contribution. Despite this increase, the teacher model’s smoother output distribution facilitates the transfer of more complex information, enhancing the student model’s learning process. **Scenario 2** (\( \sigma_s > \sigma_t \)): The overall discrepancy \( G(P, Q) \) decreases, driven by the second term’s negative contribution. In this case, the teacher model provides concentrated, key decision information, while the student model’s smoother distribution effectively captures essential features, leading to improved performance. **Analyses**: The impact of the discrepancy variation on knowledge transfer depends on the relative standard deviations of the teacher and student models. When the teacher model’s standard deviation is greater, the second term in the discrepancy \( G(P, Q) \) contributes more positively, increasing the overall discrepancy. Despite this, the smoother distribution of the teacher model provides richer information, improving the student’s ability to learn complex relationships. Conversely, when the student model’s standard deviation is greater, the negative contribution of the second term becomes more significant, decreasing \( G(P, Q) \). In this case, the teacher’s output is more concentrated, providing key decision information, while the student’s smoother distribution better mimics the teacher’s. This dynamic, data-driven adjustment of scaling values softens the output distributions during the learning process, further enhancing knowledge transfer, as shown in Fig. 1.

3.2 Knowledge Alignment Strategy

During the distillation process, we align the varying intermediate distributions from the student and teacher models to a uniform size, employing a fixed-dimension method for consistency. The motivation behind this approach is to bridge the gap between the teacher and student models’ distributions, preventing gradient explosion and ensuring
<table>
  <tr>
    <th>Hyperparameter</th>
    <th>Values</th>
  </tr>
  <tr>
    <td>Hard Label Weight</td>
    <td>0.1, 1, 10, 20</td>
  </tr>
  <tr>
    <td>Soft Label Weight</td>
    <td>1e - 8, 1e - 7, 1e - 3, 1, 10, 20</td>
  </tr>
  <tr>
    <td>Starting scaling value</td>
    <td>1, 2, ..., 20</td>
  </tr>
  <tr>
    <td>Logits Normalization</td>
    <td>"none", "Max-Min", "standardize"</td>
  </tr>
  <tr>
    <td>Hidden States Normalization</td>
    <td>"none", "Max-Min", "standardize"</td>
  </tr>
  <tr>
    <td>Knowledge Alignment Strategy<sup>2</sup></td>
    <td>None, Last_Layer, Last_Middle Layer</td>
  </tr>
  <tr>
    <td>KD Type</td>
    <td>KL, Logits Dual-Scaling</td>
  </tr>
</table>

Table 2: Overview of hyperparameters search space. KLD is Kullback–Leibler divergence. Knowledge Alignment Strategy choices represent a range of layers instead of specific the last layer or the middle layer.

effective knowledge transfer. We define the loss function \( \mathcal{L}_k \) as follows:

\[
\mathcal{L}_k = \sum_{l=1}^L \frac{\beta_l}{N \times M} \sum_{i=1}^N \sum_{j=1}^M \| h_s[l, i, j] - h_t[l, i, j] \|_2^2
\] (7)

\[
\mathcal{L}_{total} = \mathcal{L}_k + \mathcal{L}_d
\] (8)

The total loss combines \( \mathcal{L}_k \) and \( \mathcal{L}_d \) as shown in Equation 8. In Equation 7, \( L, \beta_l, h_s, \) and \( h_t \) represent the selected layers, scaling factors, and hidden states of the student and teacher models. Since their intermediate layers often differ in distribution, we apply min-max normalization to align these before using Equation 7.

\[
L'_i = \frac{x_i - \min(L)}{\max(L) - \min(L) + \epsilon}
\] (9)

\( x_i \) denotes the distribution value, \( \min(L) \) and \( \max(L) \) are the distribution’s minimum and maximum, and \( \epsilon \) is a small constant for numerical stability. This normalization, applied to intermediate layers, narrows the distribution gap between teacher and student models and prevents gradient explosion.

3.3 Logits-Aware Bayesian Optimization

We utilize Bayesian optimization during the distillation phase to streamline the hyperparameter selection process and boost the model performance in Table. 2. The overview searching process is in Fig. 1 Bayes Optimization part.

Search Space: The hyperparameter search space encompasses a range of options, as summarized in Table. 2. It includes Logits-Aware parameters such as pruning thresholds and sparse regularization factors, alongside traditional distillation parameters like scaling value and layer configurations.

Search Method: We employ a modified Gaussian Process (GP) as a surrogate model, specifically tailored for logits discrepancy. The GP, denoted as \( f \sim GP(\mu, K) \), uses a mean function \( \mu \) initialized to reflect model characteristics and a covariance function \( K \) designed to capture sparse parameter interactions. This GP is iteratively updated based on hyperparameter evaluations (\( X, Y \)), refining its understanding of the model-specific hyperparameter space. To balance exploration and exploitation, we implement an adaptive sampling strategy, ensuring efficient optimization.

To guide the search for optimal hyperparameter configurations, we introduce a Logits-Aware Expected Improvement (LAEI) acquisition function:

\[
LAEI(x^*) = (\mu(x^*) - f(x^+) - \xi)\Phi(Z) \\
+ \sigma(x^*)\phi(Z) + \alpha S(x^*)
\] (10)

\[
Z = \frac{\mu(x^*) - f(x^+) - \xi}{\sigma(x)}
\] (11)

Here, \( \mu(x^*) \) and \( \sigma(x^*) \) represent the predicted mean and variance, \( f(x^+) \) is the best observed value, and \( \xi \) is a small parameter for balancing exploration and exploitation. \( \Phi \) and \( \phi \) represent the cumulative and probability density functions of the standard normal distribution. The novel term \( S(x^*) \) integrates information from the Logits Dual-Scaling strategy and Knowledge Alignment Module, with \( \alpha \) serving as an overall balancing factor for sparse models:

\[
S(x^*) = \lambda_1 D(\sigma_t, \sigma_s) + \lambda_2 A(h_t, h_s)
\] (12)

where \( D(\sigma_t, \sigma_s) = \left| \frac{1}{\sigma_t} - \frac{1}{\sigma_s} \right| \) measures logits discrepancy, and \( A(h_t, h_s) = \sum_{l=1}^L \| h_t[l] - h_s[l] \|_2^2 \) quantifies hidden state alignment across \( L \) layers. Coefficients \( \lambda_1 \) and \( \lambda_2 \) balance these terms.

Advantages: LAEI offers key advantages for sparse models by incorporating sparsity-related information into the hyperparameter search. It optimizes knowledge transfer through dynamic scaling value adjustment (\( D(\sigma_t, \sigma_s) \)) and hidden state alignment, while balancing exploration and exploitation. This approach enables LAEI to effectively navigate the complex landscape of compact LLMs, improving distillation performance in resource-constrained scenarios. A detailed theoretical analysis is provided in Appendix.

Compact LLM distillation guidelines: We summarize some deep insights based on search results in Fig. 2: Compressed LLM distillation always favors deep intermediate knowledge, greater scaling
<table>
  <tr>
    <th>Model</th>
    <th>Method</th>
    <th>OPQA</th>
    <th>Hella-SWAG</th>
    <th>Wino-grande</th>
    <th>PIQA</th>
    <th>ARC_E BoolQ</th>
    <th>ARC_C QNLI</th>
    <th>QQP</th>
    <th>SST-2</th>
    <th>Average</th>
    <th>WikiText2</th>
  </tr>
  <tr>
    <td rowspan="2">LLaMA3 70B</td>
    <td>Teacher Model</td>
    <td>48.60</td>
    <td>84.92</td>
    <td>81.14</td>
    <td>84.49</td>
    <td>85.94</td>
    <td>85.26</td>
    <td>64.33</td>
    <td>52.88</td>
    <td>65.92</td>
    <td>79.70</td>
    <td>73.32</td>
    <td>2.92</td>
  </tr>
  <tr>
    <td>Dense Model</td>
    <td>45.00</td>
    <td>79.11</td>
    <td>73.24</td>
    <td>80.74</td>
    <td>77.86</td>
    <td>81.16</td>
    <td>53.16</td>
    <td>50.49</td>
    <td>57.31</td>
    <td>68.23</td>
    <td>66.63</td>
    <td>10.18</td>
  </tr>
  <tr>
    <td rowspan="2">Sparse Model (wo LoRA)</td>
    <td></td>
    <td>39.00</td>
    <td>67.67</td>
    <td>69.22</td>
    <td>76.61</td>
    <td>60.86</td>
    <td>65.99</td>
    <td>37.88</td>
    <td>51.42</td>
    <td>62.67</td>
    <td>51.83</td>
    <td>58.32</td>
    <td>20.24</td>
  </tr>
  <tr>
    <td>Sparse Model (w LoRA)</td>
    <td>39.55</td>
    <td>70.85</td>
    <td>69.06</td>
    <td>76.62</td>
    <td>66.55</td>
    <td>69.87</td>
    <td>39.84</td>
    <td>48.68</td>
    <td>62.16</td>
    <td>64.51</td>
    <td>60.77</td>
    <td>17.00</td>
  </tr>
  <tr>
    <td rowspan="2">8B</td>
    <td>Stand KD</td>
    <td>38.2</td>
    <td>71.79</td>
    <td>71.36</td>
    <td>75.18</td>
    <td>68.76</td>
    <td>66.43</td>
    <td>40.86</td>
    <td>47.76</td>
    <td>63.59</td>
    <td>60.78</td>
    <td>60.47</td>
    <td>18.18</td>
  </tr>
  <tr>
    <td>BayesKD(Random Search)</td>
    <td>37.45</td>
    <td>72.37</td>
    <td>70.28</td>
    <td>76.69</td>
    <td>67.17</td>
    <td>68.48</td>
    <td>42.53</td>
    <td>49.29</td>
    <td>55.04</td>
    <td>62.31</td>
    <td>60.16</td>
    <td>17.21</td>
  </tr>
  <tr>
    <td rowspan="2">Qwen2 72B</td>
    <td>BayesKD (SABO)</td>
    <td><b>41.80</b></td>
    <td><b>74.32</b></td>
    <td><b>73.24</b></td>
    <td><b>78.29</b></td>
    <td><b>71.13</b></td>
    <td><b>76.24</b></td>
    <td><b>45.56</b></td>
    <td><b>49.86</b></td>
    <td><b>56.37</b></td>
    <td><b>67.78</b></td>
    <td><b>63.46</b></td>
    <td><b>15.77</b></td>
  </tr>
  <tr>
    <td>Teacher Model</td>
    <td>49</td>
    <td>85.59</td>
    <td>79.16</td>
    <td>83.24</td>
    <td>80.64</td>
    <td>89.54</td>
    <td>60.15</td>
    <td>73.7</td>
    <td>73.24</td>
    <td>93.58</td>
    <td>76.78</td>
    <td>6.65</td>
  </tr>
  <tr>
    <td rowspan="2">Qwen2 7B</td>
    <td>Dense Model</td>
    <td>44.40</td>
    <td>78.77</td>
    <td>71.9</td>
    <td>81.18</td>
    <td>74.58</td>
    <td>84.89</td>
    <td>50.00</td>
    <td>59.40</td>
    <td>73.28</td>
    <td>92.43</td>
    <td>71.08</td>
    <td>9.32</td>
  </tr>
  <tr>
    <td>Sparse Model (wo LoRA)</td>
    <td>38.52</td>
    <td>67.37</td>
    <td>68.11</td>
    <td>75.31</td>
    <td>58.17</td>
    <td>68.64</td>
    <td>35.64</td>
    <td>61.63</td>
    <td>63.68</td>
    <td>87.84</td>
    <td>62.49</td>
    <td>18.55</td>
  </tr>
  <tr>
    <td rowspan="2">Sparse Model (w LoRA)</td>
    <td></td>
    <td>39.06</td>
    <td>71.46</td>
    <td>69.93</td>
    <td>77.02</td>
    <td>63.62</td>
    <td>72.61</td>
    <td>38.52</td>
    <td>58.65</td>
    <td>70.41</td>
    <td>90.02</td>
    <td>65.13</td>
    <td>15.55</td>
  </tr>
  <tr>
    <td>Stand KD</td>
    <td>37.73</td>
    <td>70.53</td>
    <td>68.29</td>
    <td>75.57</td>
    <td>65.72</td>
    <td>69.26</td>
    <td>38.47</td>
    <td>57.31</td>
    <td>73.03</td>
    <td>88.19</td>
    <td>64.41</td>
    <td>16.62</td>
  </tr>
  <tr>
    <td rowspan="2">BayesKD(Random Search)</td>
    <td></td>
    <td>39.28</td>
    <td>69.66</td>
    <td>70.06</td>
    <td>76.78</td>
    <td>64.84</td>
    <td>76.99</td>
    <td>37.99</td>
    <td>56.66</td>
    <td>74.52</td>
    <td>90.18</td>
    <td>65.70</td>
    <td>15.74</td>
  </tr>
  <tr>
    <td>BayesKD (SABO)</td>
    <td><b>41.27</b></td>
    <td><b>73.98</b></td>
    <td><b>71.9</b></td>
    <td><b>78.7</b></td>
    <td><b>67.97</b></td>
    <td><b>79.53</b></td>
    <td><b>42.92</b></td>
    <td><b>58.5</b></td>
    <td><b>77.56</b></td>
    <td><b>92.31</b></td>
    <td><b>68.46</b></td>
    <td><b>14.42</b></td>
  </tr>
</table>

Table 3: The main results (Qwen and LLaMA3) from our multi-task testing, with the exception of WikiText-2, were derived from the Language Model Evaluation Harness1. For WikiText-2, the Perplexity (PPL) metric was employed, whereas accuracy served as the metric for all other tasks. Sparsity ratio is 25%. BayesKD (SABO) employs Logits-Aware Bayesian Optimization for hyperparameter tuning, while BayesKD (Random Search) uses random search. This methodological approach ensures a rigorous and comprehensive evaluation of our models’ effectiveness across a diverse array of tasks, adhering to the high standards of academic rigor and professionalism expected at scholarly conferences.

value sparsity, and smaller loss weights. This is evident from our exploration of different “KD Layers” options, including “Middle& Last” and “Max-Min Normalization”, which suggest leveraging intermediate representations from the teacher model’s deeper layers. Additionally, the wide range of scaling values, reaching up to 16, indicates a preference for higher scaling value sparsity. Furthermore, KD Loss Weight with options like potentially small values implies that compressed LLM distillation may benefit from using smaller loss weights, potentially to avoid overriding the student model’s original capabilities.

4 Experiments

Experimental Setup We extracted 13,000 training samples and 2,000 validation samples from the cleaned Alpaca dataset3 for LoRA fine-tuning and distillation. For evaluation, we selected 11 datasets across various NLP domains to assess model performance comprehensively on a zero-shot basis as introduced in the Introduction section. We established a comprehensive experimental setup using a diverse range of LLMs. Our student models include Bloom-7b1 (Workshop et al., 2022), LLaMA-7b-hf (Touvron et al., 2023), Vicuna-7b-v1.1 (Zheng et al., 2023), TinyLLaMA-1.1b (Zhang et al., 2024), LLaMA-3 8B (Dubey et al., 2024), and Qwen-2 7B (Yang et al., 2024). These were paired with appropriate teacher models: LLaMA-13b for the first four, LLaMA-3 70B for LLaMA-3 8B, and Qwen-2 72B for Qwen-2 7B. This setup allows for a comprehensive evaluation across various model architectures and sizes. All experiments were conducted on 8 NVIDIA A100 GPUs with a sparsity ratio of 25% for the student models.

4.1 Results and Analysis

In Table 4, we evaluate BayesKD on TinyLLaMA1.1B, LLaMA 7B, Vicuna 7B, and Bloom 7B models. Across these configurations, BayesKD consistently outperforms baseline methods. For instance, with the LLaMA 7B model, BayesKD achieves an average score of 60.08%, surpassing Standard KD (55.68%) and Sparse Model with LoRA (57.48%) by significant margins. Similar trends are observed for other models, with
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="1">OPQA</th>
    <th colspan="1">Hella-SWAG</th>
    <th colspan="1">Winogrande</th>
    <th colspan="1">PIQA</th>
    <th colspan="1">ARC_E BoolQ</th>
    <th colspan="1">ARC_C QNLI</th>
    <th colspan="1">QQP</th>
    <th colspan="1">SST-2</th>
    <th colspan="1">Average</th>
    <th colspan="1">WikiText2</th>
  </tr>
  <tr></tr>
  <tr>
    <td>LLaMA 13B</td>
    <td>Teacher Model</td>
    <td>44.80</td>
    <td>79.07</td>
    <td>72.77</td>
    <td>80.09</td>
    <td>74.71</td>
    <td>77.98</td>
    <td>47.61</td>
    <td>50.74</td>
    <td>46.37</td>
    <td>69.04</td>
    <td>64.32</td>
    <td>11.58</td>
  </tr>
  <tr>
    <td rowspan="4">Tiny-LLaMA 1.1B</td>
    <td>Dense Model</td>
    <td>36.00</td>
    <td>59.20</td>
    <td>59.12</td>
    <td>73.29</td>
    <td>55.35</td>
    <td>57.83</td>
    <td>30.12</td>
    <td>48.49</td>
    <td>54.28</td>
    <td>69.61</td>
    <td>54.33</td>
    <td>16.53</td>
  </tr>
  <tr>
    <td>Sparse Model (wo LoRA)</td>
    <td>32.00</td>
    <td>50.82</td>
    <td>55.72</td>
    <td>69.42</td>
    <td>46.84</td>
    <td>53.67</td>
    <td>27.47</td>
    <td>50.52</td>
    <td>50.33</td>
    <td>65.02</td>
    <td>50.18</td>
    <td>10.29</td>
  </tr>
  <tr>
    <td>Sparse Model (w LoRA)</td>
    <td>33.00</td>
    <td>52.13</td>
    <td>57.46</td>
    <td>71.22</td>
    <td>48.11</td>
    <td>53.70</td>
    <td>29.69</td>
    <td>50.49</td>
    <td>47.07</td>
    <td>73.05</td>
    <td>51.59</td>
    <td>27.59</td>
  </tr>
  <tr>
    <td>Stand KD</td>
    <td><b>34.40</b></td>
    <td>54.07</td>
    <td>57.46</td>
    <td>71.21</td>
    <td>49.24</td>
    <td>54.92</td>
    <td>29.18</td>
    <td>53.18</td>
    <td>55.73</td>
    <td>71.56</td>
    <td>53.10</td>
    <td>22.15</td>
  </tr>
  <tr>
    <td rowspan="2">BayesKD(Random Search)</td>
    <td>34.12</td>
    <td>53.29</td>
    <td>57.93</td>
    <td>69.15</td>
    <td>46.59</td>
    <td><b>63.27</b></td>
    <td>29.69</td>
    <td>49.46</td>
    <td>38.06</td>
    <td>50.92</td>
    <td>49.25</td>
    <td>25.43</td>
  </tr>
  <tr>
    <td><b>BayesKD (SABO)</b></td>
    <td><b>34.00</b></td>
    <td><b>54.34</b></td>
    <td><b>58.25</b></td>
    <td><b>70.78</b></td>
    <td><b>49.45</b></td>
    <td><b>54.19</b></td>
    <td><b>29.78</b></td>
    <td><b>53.22</b></td>
    <td><b>57.36</b></td>
    <td><b>80.96</b></td>
    <td><b>54.23</b></td>
    <td><b>22.00</b></td>
  </tr>
  <tr>
    <td rowspan="4">LLaMA 7b</td>
    <td>Dense Model</td>
    <td>44.40</td>
    <td>76.21</td>
    <td>69.85</td>
    <td>79.16</td>
    <td>72.81</td>
    <td>75.11</td>
    <td>44.71</td>
    <td>51.16</td>
    <td>48.00</td>
    <td>76.38</td>
    <td>63.78</td>
    <td>12.62</td>
  </tr>
  <tr>
    <td>Sparse Model (wo LoRA)</td>
    <td>36.20</td>
    <td>62.45</td>
    <td>59.43</td>
    <td>73.18</td>
    <td>51.14</td>
    <td>58.69</td>
    <td>32.17</td>
    <td><b>53.18</b></td>
    <td>46.14</td>
    <td>74.54</td>
    <td>54.71</td>
    <td>22.54</td>
  </tr>
  <tr>
    <td>Sparse Model (w LoRA)</td>
    <td>39.40</td>
    <td>67.18</td>
    <td>62.12</td>
    <td>74.65</td>
    <td>59.34</td>
    <td>57.37</td>
    <td>36.86</td>
    <td>51.53</td>
    <td>48.95</td>
    <td>77.40</td>
    <td>57.48</td>
    <td>19.58</td>
  </tr>
  <tr>
    <td>Stand KD</td>
    <td>38.60</td>
    <td>67.34</td>
    <td>62.67</td>
    <td>73.88</td>
    <td>59.81</td>
    <td>62.35</td>
    <td>37.97</td>
    <td>51.60</td>
    <td>46.50</td>
    <td>56.08</td>
    <td>55.68</td>
    <td>20.56</td>
  </tr>
  <tr>
    <td rowspan="2">BayesKD(Random Search)</td>
    <td>39.20</td>
    <td>64.17</td>
    <td>63.47</td>
    <td>73.23</td>
    <td>52.50</td>
    <td>67.74</td>
    <td>33.45</td>
    <td>51.39</td>
    <td><b>57.40</b></td>
    <td>49.29</td>
    <td>55.18</td>
    <td>27.29</td>
  </tr>
  <tr>
    <td><b>BayesKD (SABO)</b></td>
    <td><b>44.60</b></td>
    <td><b>72.01</b></td>
    <td><b>65.36</b></td>
    <td><b>78.10</b></td>
    <td><b>65.81</b></td>
    <td><b>67.80</b></td>
    <td><b>39.59</b></td>
    <td>52.08</td>
    <td>55.36</td>
    <td>60.08</td>
    <td><b>71.05</b></td>
    <td><b>19.05</b></td>
  </tr>
  <tr>
    <td rowspan="4">Vincuna 7b</td>
    <td>Dense Model</td>
    <td>43.40</td>
    <td>74.64</td>
    <td>70.09</td>
    <td>78.56</td>
    <td>72.01</td>
    <td>78.32</td>
    <td>43.77</td>
    <td>50.60</td>
    <td>60.70</td>
    <td>54.24</td>
    <td>62.63</td>
    <td>16.10</td>
  </tr>
  <tr>
    <td>Sparse Model (wo LoRA)</td>
    <td>34.20</td>
    <td>60.18</td>
    <td>59.04</td>
    <td>72.04</td>
    <td>54.46</td>
    <td>49.82</td>
    <td>33.02</td>
    <td>51.38</td>
    <td>58.85</td>
    <td>72.02</td>
    <td>54.50</td>
    <td>28.83</td>
  </tr>
  <tr>
    <td>Sparse Model (w LoRA)</td>
    <td>39.00</td>
    <td>65.72</td>
    <td>63.77</td>
    <td>73.40</td>
    <td>59.98</td>
    <td>53.00</td>
    <td>36.26</td>
    <td>50.54</td>
    <td>56.44</td>
    <td>76.95</td>
    <td>57.51</td>
    <td>20.95</td>
  </tr>
  <tr>
    <td>Stand KD</td>
    <td>33.20</td>
    <td>54.66</td>
    <td>58.64</td>
    <td>71.05</td>
    <td>49.83</td>
    <td>59.41</td>
    <td>30.29</td>
    <td>58.28</td>
    <td>55.11</td>
    <td>59.40</td>
    <td>52.99</td>
    <td>22.73</td>
  </tr>
  <tr>
    <td rowspan="2">BayesKD(Random Search)</td>
    <td>40.20</td>
    <td>68.33</td>
    <td>65.51</td>
    <td>75.30</td>
    <td>61.75</td>
    <td><b>62.22</b></td>
    <td>36.44</td>
    <td>54.70</td>
    <td>56.57</td>
    <td>70.30</td>
    <td>59.13</td>
    <td>23.35</td>
  </tr>
  <tr>
    <td><b>BayesKD (SABO)</b></td>
    <td><b>41.20</b></td>
    <td><b>69.59</b></td>
    <td><b>65.98</b></td>
    <td><b>76.39</b></td>
    <td><b>61.79</b></td>
    <td><b>56.15</b></td>
    <td><b>37.96</b></td>
    <td><b>56.40</b></td>
    <td><b>60.22</b></td>
    <td><b>77.18</b></td>
    <td><b>60.29</b></td>
    <td><b>20.93</b></td>
  </tr>
  <tr>
    <td rowspan="4">Bloom 7b</td>
    <td>Dense Model</td>
    <td>35.80</td>
    <td>62.26</td>
    <td>64.40</td>
    <td>73.56</td>
    <td>57.28</td>
    <td>62.91</td>
    <td>33.45</td>
    <td>51.18</td>
    <td>41.87</td>
    <td>49.08</td>
    <td>53.18</td>
    <td>26.58</td>
  </tr>
  <tr>
    <td>Sparse Model (wo LoRA)</td>
    <td>31.60</td>
    <td>38.13</td>
    <td>56.35</td>
    <td>67.79</td>
    <td>46.84</td>
    <td>61.99</td>
    <td>26.71</td>
    <td>49.33</td>
    <td>38.13</td>
    <td>61.01</td>
    <td>47.79</td>
    <td>19.07</td>
  </tr>
  <tr>
    <td>Sparse Model (w LoRA)</td>
    <td>31.20</td>
    <td>33.95</td>
    <td><b>57.22</b></td>
    <td>65.78</td>
    <td>44.49</td>
    <td>46.30</td>
    <td>25.85</td>
    <td>46.29</td>
    <td><b>49.78</b></td>
    <td>51.95</td>
    <td>45.28</td>
    <td>15.26</td>
  </tr>
  <tr>
    <td>Stand KD</td>
    <td>29.00</td>
    <td>35.01</td>
    <td>55.95</td>
    <td>65.28</td>
    <td>45.41</td>
    <td>60.86</td>
    <td>25.51</td>
    <td>49.50</td>
    <td>36.80</td>
    <td>50.80</td>
    <td>45.41</td>
    <td>149.58</td>
  </tr>
  <tr>
    <td rowspan="2">BayesKD(Random Search)</td>
    <td>31.60</td>
    <td>38.12</td>
    <td>56.35</td>
    <td>67.79</td>
    <td>46.84</td>
    <td>61.96</td>
    <td>26.71</td>
    <td>49.33</td>
    <td>38.13</td>
    <td>61.01</td>
    <td>47.99</td>
    <td>135.66</td>
  </tr>
  <tr>
    <td><b>BayesKD (SABO)</b></td>
    <td><b>32.20</b></td>
    <td><b>39.36</b></td>
    <td>55.09</td>
    <td><b>68.55</b></td>
    <td><b>46.89</b></td>
    <td><b>61.98</b></td>
    <td><b>27.65</b></td>
    <td><b>50.25</b></td>
    <td>39.39</td>
    <td><b>61.35</b></td>
    <td><b>48.25</b></td>
    <td><b>124.49</b></td>
  </tr>
</table>

Table 4: The main results (LLaMA1, TinyLLaMA, Vicuna) from our multi-task testing derived from the Language Model Evaluation Harness. Sparsity ratio is 25%. BayesKD (SABO) employs Logits-Aware Bayesian Optimization for hyperparameter tuning, while BayesKD (Random Search) uses random sampling.

<table>
  <tr>
    <th>Type</th>
    <th>Model</th>
    <th>Base Parameters (B)</th>
    <th>Pruned Parameters (B)</th>
    <th>Final Pruned Ratio</th>
  </tr>
  <tr>
    <td rowspan="6">Student</td>
    <td>LLaMA13B 13B</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Tiny-LLaMA 1.1B</td>
    <td>0.961B</td>
    <td>0.8735</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaMA1.1B</td>
    <td>7.069B</td>
    <td>6.282B</td>
    <td>0.8887</td>
  </tr>
  <tr>
    <td>Bloom7b</td>
    <td>6.738B</td>
    <td>5.423B</td>
    <td>0.8048</td>
  </tr>
  <tr>
    <td>Vicuna7b</td>
    <td>6.738B</td>
    <td>5.423B</td>
    <td>0.8048</td>
  </tr>
  <tr>
    <td>LLaMA7b</td>
    <td>6.738B</td>
    <td>5.423B</td>
    <td>0.8048</td>
  </tr>
</table>

Table 5: Overview of model parameter adjustments. All parameter values are expressed in billions (B).

BayesKD showing particular strength in challenging tasks such as ARC-challenge and HellaSwag. Table 3 extends our analysis to the latest models, LLaMA-3 and Qwen-2. For the LLaMA-3 8B model, BayesKD achieves an average accuracy of 63.46% across tasks, outperforming Standard KD (60.47%) and Sparse Model with LoRA (60.77%). Notably, on complex reasoning tasks like ARC-Challenge and HellaSwag, BayesKD shows substantial improvements of 4.7% and 2.53% respectively over Standard KD. The Qwen-2 7B results further corroborate BayesKD’s effectiveness. Our method achieves an average accuracy of 68.48%, significantly surpassing Standard KD (64.41%) and Sparse Model with LoRA (65.13%). Particularly impressive are the improvements in tasks like BoolQ and QQP, where BayesKD outperforms Standard KD by 10.27% and 4.53% respectively.

Across all model sizes and architectures, we observe that the performance gap between BayesKD and baseline methods widens as model size increases in both general NLU benchmarks and instruct-follow benchmarks 8. The consistent performance improvements validate the generalizability and robustness of BayesKD.

4.2 Ablation Study

We conducted an ablation study to evaluate the individual contributions of Logits Dual-Scaling (DS), the Knowledge Alignment Strategy (KAS), and Kullback–Leibler (KL) divergence to the distillation performance of the TinyLLaMA1.1B model.
<table>
  <tr>
    <th>Method</th>
    <th>OPQA</th>
    <th>Hella-SWAG</th>
    <th>Wino-grande</th>
    <th>PIQA</th>
    <th>ARC_E</th>
    <th>BoolQ</th>
    <th>ARC_C</th>
    <th>QNLI</th>
    <th>QQP</th>
    <th>SST-2</th>
    <th>Average</th>
    <th>Wiki-text2</th>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>32.00</td>
    <td>50.82</td>
    <td>55.72</td>
    <td>69.42</td>
    <td>46.84</td>
    <td>53.67</td>
    <td>27.47</td>
    <td>50.52</td>
    <td>50.33</td>
    <td>65.02</td>
    <td>50.18</td>
    <td>30.29</td>
  </tr>
  <tr>
    <td>+KL</td>
    <td>33.20</td>
    <td>53.52</td>
    <td>58.09</td>
    <td>69.53</td>
    <td>46.55</td>
    <td>54.92</td>
    <td>29.03</td>
    <td>49.28</td>
    <td>45.07</td>
    <td>69.27</td>
    <td>51.59</td>
    <td>25.85</td>
  </tr>
  <tr>
    <td>+ KL + KAS</td>
    <td>34.40</td>
    <td>54.07</td>
    <td>57.46</td>
    <td>71.22</td>
    <td>49.24</td>
    <td>53.88</td>
    <td>29.18</td>
    <td>53.18</td>
    <td>55.73</td>
    <td>71.56</td>
    <td>53.10</td>
    <td>22.15</td>
  </tr>
  <tr>
    <td>+ KL + DS</td>
    <td>33.00</td>
    <td>54.11</td>
    <td>57.3</td>
    <td>70.89</td>
    <td>50.00</td>
    <td>56.97</td>
    <td>30.2</td>
    <td>54.55</td>
    <td>55.96</td>
    <td>74.2</td>
    <td>53.72</td>
    <td>23.93</td>
  </tr>
  <tr>
    <td>+ KL + KAS + DS</td>
    <td>34.00</td>
    <td>54.34</td>
    <td>58.25</td>
    <td>70.78</td>
    <td>49.45</td>
    <td>54.19</td>
    <td>29.78</td>
    <td>53.21</td>
    <td>57.36</td>
    <td>80.96</td>
    <td>54.23</td>
    <td>22.00</td>
  </tr>
</table>

Table 6: Ablation study results in 0.25 sparsity ratio in TinyLLaMA1.1b on various NLP tasks. KL is KL divergence, KAS is Knowledge Alignment Strategy and DS is Logits Dual-Scaling. All parameters are the same and searched from Logits-Aware Bayesian Optimization

![Visualization of search results during Bayesian distillation optimization in LLaMA13B-TinyLLaMA1.1B. The yellow line illustrates the Logits Dual-Scaling Strategy approach combined with max-min normalization (Knowledge Alignment Strategy), leading to lower perplexity on WikiText-2 and higher accuracy across tasks.](page_370_613_1002_312.png)

Figure 2: Visualization of search results during Bayesian distillation optimization in LLaMA13B-TinyLLaMA1.1B. The yellow line illustrates the Logits Dual-Scaling Strategy approach combined with max-min normalization (Knowledge Alignment Strategy), leading to lower perplexity on WikiText-2 and higher accuracy across tasks.

<table>
  <tr>
    <th>Method</th>
    <th>Average</th>
    <th>Wiki-text2</th>
  </tr>
  <tr>
    <td>Naïve Bayesian Search</td>
    <td>53.37</td>
    <td>24.25</td>
  </tr>
  <tr>
    <td>SABO</td>
    <td>54.23</td>
    <td>22.00</td>
  </tr>
</table>

Table 7: Comparison of performance between SABO and Naïve Bayesian Search in 0.25 sparsity ratio in TinyLLaMA1.1b on Wiki-text2.

Starting from the full configuration, each strategy was removed in turn. As shown in Table 6, excluding DS reduced average accuracy by 1.13% across 10 tasks and increased perplexity on WikiText-2 by 2.29 points. Removing KAS produced similar effects, with a 1.13% drop in accuracy and a 2.85-point perplexity increase. Eliminating KL divergence resulted in a 1.41% accuracy decrease and a 4.44-point perplexity increase, demonstrating the effectiveness of incorporating KD regularization beyond the baseline sparse model. Furthermore, the main results in Table 4 already demonstrate the superiority of Bayesian search over random search, making a separate ablation study for Bayesian search unnecessary.

These findings confirm the contributions of Logits Dual-Scaling, Knowledge Alignment Strategy, and Bayesian optimization to the model’s overall performance, revealing their relative importance during the model training process.

<table>
  <tr>
    <th>Model</th>
    <th>Method</th>
    <th>gpt3_translation_tasks</th>
  </tr>
  <tr>
    <td rowspan="4">LLaMA3 70B</td>
    <td>Teacher Model</td>
    <td>54.31</td>
  </tr>
  <tr>
    <td>Dense Model</td>
    <td>31.94</td>
  </tr>
  <tr>
    <td>Sparse Model (wo LoRA)</td>
    <td>19.26</td>
  </tr>
  <tr>
    <td>Sparse Model (w. LoRA)</td>
    <td>22.14</td>
  </tr>
  <tr>
    <td rowspan="4">LLaMA3 8B</td>
    <td>Stand KD</td>
    <td>21.88</td>
  </tr>
  <tr>
    <td><b>BayesKD (SABO)</b></td>
    <td><b>24.59</b></td>
  </tr>
  <tr>
    <td rowspan="4">Qwen2 72B</td>
    <td>Teacher Model</td>
    <td>31.76</td>
  </tr>
  <tr>
    <td>Dense Model</td>
    <td>12.62</td>
  </tr>
  <tr>
    <td>Sparse Model (wo LoRA)</td>
    <td>7.62</td>
  </tr>
  <tr>
    <td>Sparse Model (w. LoRA)</td>
    <td>8.75</td>
  </tr>
  <tr>
    <td rowspan="2">Qwen2 7B</td>
    <td>Stand KD</td>
    <td>8.65</td>
  </tr>
  <tr>
    <td><b>BayesKD (SABO)</b></td>
    <td><b>9.72</b></td>
  </tr>
</table>

Table 8: Instruction-following datasets in sparse model. GPT-3 Translation Tasks are constructed using datasets such as WMT14 (French-English), WMT16 (Romanian-English, German-English), and IWSLT2017 (English-Arabic). Each task is augmented with instruction prompts to adapt the dataset for instruction-following evaluation.

5 Conclusion

In this paper, we propose BayesKD, a novel framework for distilling LLMs onto compact student models. Our approach introduces three key strategies: 1) a logits dual-scaling mechanism to bridge the logit distribution gap between teacher and stu-
dent models across tasks, 2) a knowledge alignment module using min-max normalization to align intermediate layer distributions, and 3) a logit-aware Bayesian optimization search to efficiently identify optimal hyperparameters tailored for compact model distillation.

6 Acknowledgements

The research was supported by Theme-based Research Scheme (T45-205/21-N) from Hong Kong RGC, and Generative AI Research and Development Centre from InnoHK.

7 Limitations

While this work enhances the generalized capabilities of pruned models, it does not specifically improve capabilities in categories such as inference and logical analysis, language generation, natural language understanding, knowledge retrieval, and integration. These areas present opportunities for detailed exploration in future research.

8 Ethics Statement

Our approach solely concentrates on the technical aspects of efficiently deploying LLMs. It does not involve any ethical or social implications.

References

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023. Knowledge distillation of large language models. arXiv preprint arXiv:2306.08543.

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531.

Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33:9782–9793.

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tiny{bert}: Distilling {bert} for natural language understanding.

Ying Jin, Jiaqi Wang, and Dahua Lin. 2023. Multi-level logit distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24276–24285.

Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models. In Forty-first International Conference on Machine Learning.

Jianquan Li, Xiaokang Liu, Honghong Zhao, Ruifeng Xu, Min Yang, and Yaohong Jin. 2020. BERT-EMD: Many-to-many layer mapping for BERT compression with earth mover’s distance. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3009–3018, Online. Association for Computational Linguistics.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305.11627.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. ArXiv, abs/1609.07843.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics.

Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2020. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 5191–5198.

Mehdi Rezagholizadeh, Aref Jafari, Puneeth S.M. Saladi, Pranav Sharma, Ali Saheb Pasand, and Ali Ghodsi. 2022. Pro-KD: Progressive distillation by following the footsteps of the teacher. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4714–4727, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

Julio Ariel Romero, Roberto Sanchis, and Elena Arrebola. 2015. Experimental study of event based pid controllers with different sampling strategies. application to brushless dc motor networked control system. In 2015 XXV international conference on information, communication and automation technologies (ICAT), pages 1–6. IEEE.

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2024a. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations.

Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. 2024b. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15731–15740.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019a. Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4323–4332, Hong Kong, China. Association for Computational Linguistics.

Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019b. Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4323–4332, Hong Kong, China. Association for Computational Linguistics.

Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971.

Julia Ture, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2020. Well-read students learn better: On the importance of pre-training compact models.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. Emergent abilities of large language models. Transactions on Machine Learning Research.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPs), 35:24824–24837.

BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Lucioni, François Yvon, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.

Yimeng Wu, Mehdi Rezagholizadeh, Abbas Ghaddar, Md Akmal Haidar, and Ali Ghodsi. 2021. UniversalKD: Attention-based output-grounded intermediate layer knowledge distillation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7649–7661, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.

Chen Zhang, Dawei Song, Zheyu Ye, and Yan Gao. 2023a. Towards the law of capacity gap in distilling language models. arXiv preprint arXiv:2311.07052.

Mingyang Zhang, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, Bohan Zhuang, et al. 2023b. Pruning meets low-rank parameter-efficient fine-tuning. arXiv preprint arXiv:2305.18403.

Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhui Chen, Christopher DeWan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685.

Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du, and Dacheng Tao. 2024. Revisiting knowledge distillation for autoregressive language models. arXiv preprint arXiv:2402.11890.

A Detailed Proof of Double Scaling Values Coefficients

The discrepancy between the teacher model \( P \) and the student model \( Q \) is defined as:

\[
G(P, Q) = D_{KL}[P||Q]
= \mathbb{E}_u[\log P(v|u) - \log Q(v|u)]
\]

In this context, \( D_{KL} \) denotes the Kullback-Leibler divergence, which measures how one probability distribution diverges from a second, expected probability distribution. Here, \( u \) represents the input data, while \( v \) denotes the corresponding output.

Given that \( T \) and \( S \) represent the logits of the teacher and student models, respectively, and \( t_s \) and \( t_t \) are the corresponding scaling value coefficients, we can express the softmax functions for the teacher and student models. The softmax function is defined as:
softmax\((z_i) = \frac{\exp(z_i)}{\sum_j \exp(z_j)}\)

By substituting the logits and scaling value coefficients into the definition of the discrepancy, we have:

\[
G(P, Q) = \mathbb{E}_u \left[ \log \operatorname{softmax} \left( \frac{T}{t_t} \right) - \log \operatorname{softmax} \left( \frac{S}{t_s} \right) \right]
= \mathbb{E}_u \left[ \frac{T}{t_t} - \log \sum_j \exp \left( \frac{T_j}{t_t} \right) \right.
- \left. \left( \frac{S}{t_s} - \log \sum_k \exp \left( \frac{S_k}{t_s} \right) \right) \right]
\]

(14)

Next, let us introduce the standard deviations of the teacher and student models, \( \sigma_t \) and \( \sigma_s \), respectively. These standard deviations are related to the scaling value coefficients as follows:

\[
t_t = \sigma_t t \\
t_s = \sigma_s t
\]

where \( t \) is a baseline scaling value. Substituting these expressions into the discrepancy formula, we get:

\[
G(P, Q) = \mathbb{E}_u \left[ \left( \frac{1}{\sigma_t} - \frac{1}{\sigma_s} \right) T \right.
- \left. \left( \log \sum_j \exp \left( \frac{T_j}{\sigma_t} \right) - \log \sum_k \exp \left( \frac{S_k}{\sigma_s} \right) \right) \right]
\]

(16)

To further analyze the impact of the standard deviations on the discrepancy, we consider two scenarios:
Scenario 1: \( \sigma_t > \sigma_s \)
In this scenario, the standard deviation of the teacher model is greater than that of the student model. This implies:

\[
\frac{1}{\sigma_t} < \frac{1}{\sigma_s}
\]

For the first term in the discrepancy formula:

\[
\left( \frac{1}{\sigma_t} - \frac{1}{\sigma_s} \right) < 0
\]

Since \( T \) is generally positive, it follows that:

\[
\left( \frac{1}{\sigma_t} - \frac{1}{\sigma_s} \right) T < 0
\]

This means that the first term contributes negatively to \( G(P, Q) \), thereby reducing the discrepancy.

For the second term:

\[
\log \sum_j \exp \left( \frac{T_j}{\sigma_t} \right) - \log \sum_k \exp \left( \frac{S_k}{\sigma_s} \right)
\]

Since \( \sigma_t > \sigma_s \), the logits of the teacher model are less magnified compared to those of the student model. Therefore, we have:

\[
\log \sum_j \exp \left( \frac{T_j}{\sigma_t} \right) < \log \sum_k \exp \left( \frac{S_k}{\sigma_s} \right)
\]

This indicates that the second term contributes positively to \( G(P, Q) \). To prove that the second term’s contribution is larger, we consider the properties of the log-sum-exp function. The log-sum-exp function is more sensitive to changes in variance compared to a linear function. Specifically, the log-sum-exp function can be approximated by the maximum logit value when the scaling value is low, making it highly sensitive to the largest logits. Using this approximation:

\[
\log \sum_j \exp \left( \frac{T_j}{\sigma_t} \right) \approx \frac{1}{\sigma_t} \max_j T_j \\
\log \sum_k \exp \left( \frac{S_k}{\sigma_s} \right) \approx \frac{1}{\sigma_s} \max_k S_k
\]

Thus, the second term becomes:

\[
\frac{1}{\sigma_t} \max_j T_j - \frac{1}{\sigma_s} \max_k S_k
\]

Given \( \frac{1}{\sigma_t} < \frac{1}{\sigma_s} \), we have:

\[
\left| \frac{1}{\sigma_t} \max_j T_j - \frac{1}{\sigma_s} \max_k S_k \right| = \frac{1}{\sigma_s} \max_k S_k - \frac{1}{\sigma_t} \max_j T_j
\]

For the first term:

\[
\left| \left( \frac{1}{\sigma_t} - \frac{1}{\sigma_s} \right) T \right| = \left( \frac{1}{\sigma_s} - \frac{1}{\sigma_t} \right) T
\]

Comparing the two terms:

\[
\left( \frac{1}{\sigma_s} - \frac{1}{\sigma_t} \right) T \\
\left( \frac{1}{\sigma_s} \max_k S_k - \frac{1}{\sigma_t} \max_j T_j \right)
\]
Since \( \max_j T_j \) and \( \max_k S_k \) are typically much larger than \( T \), the second term’s contribution is usually significantly larger than the first term’s contribution.

Summary:
In this scenario, the negative contribution of the first term is smaller in magnitude, while the positive contribution of the second term is larger. This leads to an overall increase in the discrepancy \( G(P, Q) \).

Scenario 2: \( \sigma_t < \sigma_s \)
In this scenario, the standard deviation of the student model is greater than that of the teacher model. This implies:

\[
\frac{1}{\sigma_t} > \frac{1}{\sigma_s}
\]

For the first term in the discrepancy formula:

\[
\left( \frac{1}{\sigma_t} - \frac{1}{\sigma_s} \right) > 0
\]

Since \( T \) is generally positive, it follows that:

\[
\left( \frac{1}{\sigma_t} - \frac{1}{\sigma_s} \right) T > 0
\]

This means that the first term contributes positively to \( G(P, Q) \), thereby increasing the discrepancy.

For the second term:

\[
\log \sum_j \exp \left( \frac{T_j}{\sigma_t} \right) - \log \sum_k \exp \left( \frac{S_k}{\sigma_s} \right)
\]

Since \( \sigma_s > \sigma_t \), the logits of the teacher model are more magnified compared to those of the student model. Therefore, we have:

\[
\log \sum_j \exp \left( \frac{T_j}{\sigma_t} \right) > \log \sum_k \exp \left( \frac{S_k}{\sigma_s} \right)
\]

This indicates that the second term contributes negatively to \( G(P, Q) \).

Why the Second Term’s Contribution is Larger:
Again, considering the properties of the log-sum-exp function, we use the same approximation:

\[
\log \sum_j \exp \left( \frac{T_j}{\sigma_t} \right) \approx \frac{1}{\sigma_t} \max_j T_j
\]
\[
\log \sum_k \exp \left( \frac{S_k}{\sigma_s} \right) \approx \frac{1}{\sigma_s} \max_k S_k
\]

Thus, the second term becomes:

\[
\frac{1}{\sigma_t} \max_j T_j - \frac{1}{\sigma_s} \max_k S_k
\]

Given \( \frac{1}{\sigma_t} > \frac{1}{\sigma_s} \), we have:

\[
\left| \frac{1}{\sigma_t} \max_j T_j - \frac{1}{\sigma_s} \max_k S_k \right| = \frac{1}{\sigma_t} \max_j T_j \\
- \frac{1}{\sigma_s} \max_k S_k
\]

For the first term:

\[
\left| \left( \frac{1}{\sigma_t} - \frac{1}{\sigma_s} \right) T \right| = \left( \frac{1}{\sigma_t} - \frac{1}{\sigma_s} \right) T
\]

Comparing the two terms:

\[
\left( \frac{1}{\sigma_t} - \frac{1}{\sigma_s} \right) T \\
\left( \frac{1}{\sigma_t} \max_j T_j - \frac{1}{\sigma_s} \max_k S_k \right)
\]

Since \( \max_j T_j \) and \( \max_k S_k \) are typically much larger than \( T \), the second term’s contribution is usually significantly larger than the first term’s contribution.

Summary:
In this scenario, the positive contribution of the first term is smaller in magnitude, while the negative contribution of the second term is larger. This leads to an overall decrease in the discrepancy \( G(P, Q) \).

Conclusion
The analysis reveals that the impact of the discrepancy variation on knowledge transfer differs depending on the relative standard deviations of the teacher and student models:
- When the standard deviation of the teacher model (\( \sigma_t \)) is greater than that of the student model (\( \sigma_s \)), the positive contribution of the second term is more significant. This results in an overall increase in the discrepancy \( G(P, Q) \). Although the discrepancy increases, the teacher model’s output distribution is smoother and contains more information. This helps the student model to learn more details and complex relationships, thereby improving its performance.
- Conversely, when the standard deviation of the student model (\( \sigma_s \)) is greater than that of the teacher model (\( \sigma_t \)), the negative contribution of the second term is more significant. This results in an overall decrease in the discrepancy \( G(P, Q) \). Although the
<table>
  <tr>
    <th>Model</th>
    <th>Method</th>
    <th>Average</th>
    <th>WikiText2</th>
  </tr>
  <tr>
    <td rowspan="3">LLaMA 13B</td>
    <td>Teacher Model</td>
    <td>64.32</td>
    <td>11.58</td>
  </tr>
  <tr>
    <td>Sparse Model</td>
    <td>54.71</td>
    <td>22.54</td>
  </tr>
  <tr>
    <td>Dense Model</td>
    <td>63.78</td>
    <td>12.62</td>
  </tr>
  <tr>
    <td rowspan="2">LLaMA 7B</td>
    <td>MiniLLM</td>
    <td>55.10</td>
    <td>22.15</td>
  </tr>
  <tr>
    <td>BayesKD (SABO)</td>
    <td><b>60.08</b></td>
    <td><b>19.05</b></td>
  </tr>
</table>

Table 9: MiniLLM in sparse model

discrepancy decreases, the teacher model’s output is more concentrated, providing key decision information. The student model’s output distribution is smoother and more accurately mimics the teacher model’s output distribution, capturing the main decision features and thus improving its performance.

Therefore, the impact of the discrepancy variation on knowledge transfer requires a detailed analysis of the specific probability distributions of the teacher and student models. Understanding these dynamics is crucial for optimizing the performance of the student model in various training scenarios.

B Experimental Details

All the experiments are run on 8 A100 GPUs. The experiments running time is 55 minutes in 7B models and 30 minutes in 1.1B model. The memory requires 75GB if we load 13B and 5.4B(compressed) model in 4 bytes/parameter to train in Batch size 4.

Ablation study with error bar:

![Ablation Study Results with Error Bar](page_1012_1042_377_246.png)

Figure 3: The ablation study results with error bar

P-values: We calculated the variance and mean based on 6 extra experiments in LLaMA 7B group. Then, we used independent samples t-test to statistically determine whether BayesKD (SABO) has a significant advantage in Table 10. The table shows the P-values of BayesKD (SABO) compared to other methods and whether it has a significant advantage.

Based on the P-values table (Table 10), we can summarize the performance of BayesKD (SABO) compared to Stand KD, Sparse Model (w LoRA), and BayesKD (Random Search) as follows:

BayesKD (SABO) vs. Stand KD: BayesKD (SABO) demonstrates significant advantages over Stand KD on 8 out of 10 datasets, with the exceptions being SST-2 and ARC_C. This indicates that BayesKD (SABO) generally outperforms Stand KD in most cases, suggesting its effectiveness as an improved knowledge distillation method.

BayesKD (SABO) vs. Sparse Model (w LoRA): BayesKD (SABO) shows significant advantages over Sparse Model (w LoRA) on all datasets except SST-2, where Sparse Model (w LoRA) significantly outperforms BayesKD (SABO). This suggests that while BayesKD (SABO) is generally more effective than Sparse Model (w LoRA), there may be specific cases where Sparse Model (w LoRA) is more suitable.

BayesKD (SABO) vs. BayesKD (Random Search): BayesKD (SABO) significantly outperforms BayesKD (Random Search) on 7 out of 10 datasets, with the exceptions being Winogrande, BoolQ, and QQP, where the differences are not statistically significant. This demonstrates the superiority of the Bayesian search approach over the random search method in most cases, highlighting the importance of an efficient search strategy in sparse knowledge distillation.

Overall, these comparisons provide strong evidence for the effectiveness and advantages of BayesKD (SABO) as a knowledge distillation method. However, it is important to note that there are specific cases where other methods may be more suitable, such as Stand KD for ARC_C and Sparse Model (w LoRA) for SST-2. Therefore, the choice of knowledge distillation method should take into account the characteristics of the target dataset and the specific requirements of the application.

Table 11 shows the mean and variance of different methods on different datasets corresponding to the P-values table.

BayesKD (SABO) vs. Stand KD in Translation tasks (Table 12 and Table 13): BayesKD (SABO) consistently outperforms Stand KD across all translation tasks for both LLaMA3 and Qwen2 models. For example, in Table 12, BayesKD (SABO) achieves an average score of 24.25 compared to Stand KD’s 21.63, while in Table 13, it improves the average score to 13.37, outperforming
<table>
  <tr>
    <th>Dataset</th>
    <th>vs. BayesKD (Random Search)</th>
    <th>vs. Sparse Model (w LoRA)</th>
    <th>vs. Sparse Model (wo LoRA)</th>
    <th>vs. Stand KD</th>
  </tr>
  <tr>
    <td>OPQA</td>
    <td><b>5.55e-16 (Significant)</b></td>
    <td><b>0.0018 (Significant)</b></td>
    <td><b>1.11e-16 (Significant)</b></td>
    <td><b>0.0002 (Significant)</b></td>
  </tr>
  <tr>
    <td>Hella-SWAG</td>
    <td><b>1.07e-08 (Significant)</b></td>
    <td><b>0.0008 (Significant)</b></td>
    <td><b>2.14e-10 (Significant)</b></td>
    <td><b>0.0002 (Significant)</b></td>
  </tr>
  <tr>
    <td>Winogrande</td>
    <td>0.0584 (Not Significant)</td>
    <td><b>0.0064 (Significant)</b></td>
    <td><b>1.23e-06 (Significant)</b></td>
    <td><b>0.0005 (Significant)</b></td>
  </tr>
  <tr>
    <td>PIQA</td>
    <td><b>3.58e-07 (Significant)</b></td>
    <td><b>0.0005 (Significant)</b></td>
    <td><b>4.41e-05 (Significant)</b></td>
    <td><b>4.61e-05 (Significant)</b></td>
  </tr>
  <tr>
    <td>ARC_E</td>
    <td><b>1.04e-12 (Significant)</b></td>
    <td><b>0.0042 (Significant)</b></td>
    <td><b>3.79e-15 (Significant)</b></td>
    <td><b>0.0002 (Significant)</b></td>
  </tr>
  <tr>
    <td>BoolQ</td>
    <td>0.6056 (Not Significant)</td>
    <td><b>1.03e-10 (Significant)</b></td>
    <td><b>1.39e-08 (Significant)</b></td>
    <td><b>0.0034 (Significant)</b></td>
  </tr>
  <tr>
    <td>ARC_C</td>
    <td><b>5.50e-07 (Significant)</b></td>
    <td><b>0.0179 (Significant)</b></td>
    <td><b>1.15e-07 (Significant)</b></td>
    <td>0.1008 (Not Significant)</td>
  </tr>
  <tr>
    <td>QQP</td>
    <td>0.5941 (Not Significant)</td>
    <td><b>6.57e-07 (Significant)</b></td>
    <td><b>6.02e-11 (Significant)</b></td>
    <td><b>2.90e-09 (Significant)</b></td>
  </tr>
  <tr>
    <td>QNLI</td>
    <td><b>1.36e-06 (Significant)</b></td>
    <td><b>2.06e-06 (Significant)</b></td>
    <td><b>0.0207 (Significant)</b></td>
    <td><b>0.0034 (Significant)</b></td>
  </tr>
  <tr>
    <td>SST-2</td>
    <td><b>6.51e-11 (Significant)</b></td>
    <td>5.17e-12 (Significantly Lower)</td>
    <td>8.78e-10 (Significantly Lower)</td>
    <td>0.1765 (Not Significant)</td>
  </tr>
</table>

Table 10: P-values of independent samples t-test comparing the performance of BayesKD (SABO) with other methods on various datasets in the LLaMA7B model. Values in parentheses indicate whether the difference is statistically significant at \( \alpha = 0.05 \). Significant results are shown in bold.

<table>
  <tr>
    <th>Dataset</th>
    <th>Metric</th>
    <th>BayesKD (SABO)</th>
    <th>Dense Model</th>
    <th>Sparse Model (w LoRA)</th>
    <th>BayesKD (Random Search)</th>
    <th>Stand KD</th>
    <th>Sparse Model (wo LoRA)</th>
  </tr>
  <tr>
    <td>OPQA</td>
    <td>Variance</td>
    <td>2.089256</td>
    <td>11.51863</td>
    <td>1.666222</td>
    <td>0.096833</td>
    <td>1.394181</td>
    <td>0.318456</td>
  </tr>
  <tr>
    <td></td>
    <td>Mean</td>
    <td>44.72667</td>
    <td>44.48</td>
    <td>39.51667</td>
    <td>39.31</td>
    <td>38.53167</td>
    <td>36.02667</td>
  </tr>
  <tr>
    <td>Hella-SWAG</td>
    <td>Variance</td>
    <td>9.417667</td>
    <td>9.729522</td>
    <td>2.284522</td>
    <td>2.719881</td>
    <td>1.034756</td>
    <td>0.539322</td>
  </tr>
  <tr>
    <td></td>
    <td>Mean</td>
    <td>72.62</td>
    <td>76.80667</td>
    <td>67.07333</td>
    <td>64.48167</td>
    <td>67.24333</td>
    <td>62.51667</td>
  </tr>
  <tr>
    <td>Winogrande</td>
    <td>Variance</td>
    <td>2.964481</td>
    <td>5.893789</td>
    <td>2.681358</td>
    <td>1.211381</td>
    <td>0.361258</td>
    <td>1.394889</td>
  </tr>
  <tr>
    <td></td>
    <td>Mean</td>
    <td>65.69833</td>
    <td>70.01333</td>
    <td>62.185</td>
    <td>63.75167</td>
    <td>62.805</td>
    <td>59.56667</td>
  </tr>
  <tr>
    <td>PIQA</td>
    <td>Variance</td>
    <td>5.300647</td>
    <td>1.431247</td>
    <td>0.897581</td>
    <td>0.2142</td>
    <td>0.401414</td>
    <td>1.021881</td>
  </tr>
  <tr>
    <td></td>
    <td>Mean</td>
    <td>78.46167</td>
    <td>79.40833</td>
    <td>74.83833</td>
    <td>73.24</td>
    <td>74.00167</td>
    <td>73.39833</td>
  </tr>
  <tr>
    <td>ARC_E</td>
    <td>Variance</td>
    <td>7.640856</td>
    <td>13.64682</td>
    <td>15.51068</td>
    <td>2.113767</td>
    <td>1.390933</td>
    <td>0.471181</td>
  </tr>
  <tr>
    <td></td>
    <td>Mean</td>
    <td>66.22333</td>
    <td>73.01333</td>
    <td>59.16833</td>
    <td>52.72</td>
    <td>59.72</td>
    <td>51.28167</td>
  </tr>
  <tr>
    <td>BoolQ</td>
    <td>Variance</td>
    <td>1.347033</td>
    <td>30.72663</td>
    <td>0.929522</td>
    <td>6.743989</td>
    <td>6.166856</td>
    <td>2.347567</td>
  </tr>
  <tr>
    <td></td>
    <td>Mean</td>
    <td>68.04</td>
    <td>75.935</td>
    <td>57.55667</td>
    <td>67.33667</td>
    <td>62.61667</td>
    <td>58.83</td>
  </tr>
  <tr>
    <td>ARC_C</td>
    <td>Variance</td>
    <td>2.630947</td>
    <td>10.13075</td>
    <td>3.144925</td>
    <td>1.903247</td>
    <td>1.539881</td>
    <td>0.712847</td>
  </tr>
  <tr>
    <td></td>
    <td>Mean</td>
    <td>39.97167</td>
    <td>45.45833</td>
    <td>37.435</td>
    <td>33.62833</td>
    <td>38.22833</td>
    <td>32.92167</td>
  </tr>
  <tr>
    <td>QNLI</td>
    <td>Variance</td>
    <td>0.779525</td>
    <td>0.698489</td>
    <td>0.009356</td>
    <td>0.031822</td>
    <td>0.067422</td>
    <td>1.713947</td>
  </tr>
  <tr>
    <td></td>
    <td>Mean</td>
    <td>52.235</td>
    <td>51.33333</td>
    <td>51.54667</td>
    <td>51.43667</td>
    <td>51.65667</td>
    <td>53.45167</td>
  </tr>
  <tr>
    <td>QQP</td>
    <td>Variance</td>
    <td>11.05059</td>
    <td>1.177981</td>
    <td>2.143789</td>
    <td>3.280289</td>
    <td>0.970514</td>
    <td>0.223514</td>
  </tr>
  <tr>
    <td></td>
    <td>Mean</td>
    <td>57.01667</td>
    <td>48.18833</td>
    <td>48.96667</td>
    <td>57.72667</td>
    <td>46.66167</td>
    <td>46.13833</td>
  </tr>
  <tr>
    <td>SST-2</td>
    <td>Variance</td>
    <td>5.483689</td>
    <td>3.024456</td>
    <td>17.58463</td>
    <td>1.010225</td>
    <td>9.477692</td>
    <td>12.31193</td>
  </tr>
  <tr>
    <td></td>
    <td>Mean</td>
    <td>60.31333</td>
    <td>76.67667</td>
    <td>85.685</td>
    <td>49.285</td>
    <td>57.355</td>
    <td>74.49</td>
  </tr>
</table>

Table 11: Variance and mean for each dataset across different methods in LLaMA7B group.

<table>
  <tr>
    <th></th>
    <th>gpt3_trans</th>
    <th>en-fr</th>
    <th>fr-en</th>
    <th>de-en</th>
    <th>en-de</th>
    <th>en-ro</th>
    <th>ro-en</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>Dense Model</td>
    <td>31.94</td>
    <td>29.93</td>
    <td>38.54</td>
    <td>41.40</td>
    <td>21.76</td>
    <td>19.99</td>
    <td>38.04</td>
    <td>31.66</td>
  </tr>
  <tr>
    <td>Sparse Model (wo LoRA)</td>
    <td>19.26</td>
    <td>7.83</td>
    <td>31.72</td>
    <td>32.05</td>
    <td>7.10</td>
    <td>7.07</td>
    <td>28.96</td>
    <td>19.14</td>
  </tr>
  <tr>
    <td>Sparse Model (w LoRA)</td>
    <td>22.14</td>
    <td>15.26</td>
    <td>32.58</td>
    <td>32.31</td>
    <td>11.16</td>
    <td>8.03</td>
    <td>29.64</td>
    <td>21.59</td>
  </tr>
  <tr>
    <td>Stand KD</td>
    <td>21.88</td>
    <td>17.48</td>
    <td>32.23</td>
    <td>32.40</td>
    <td>11.98</td>
    <td>7.00</td>
    <td>28.43</td>
    <td>21.63</td>
  </tr>
  <tr>
    <td><b>BayesKD (SABO)</b></td>
    <td><b>24.59</b></td>
    <td><b>19.54</b></td>
    <td><b>34.16</b></td>
    <td><b>35.45</b></td>
    <td><b>13.92</b></td>
    <td><b>10.21</b></td>
    <td><b>31.84</b></td>
    <td><b>24.25</b></td>
  </tr>
</table>

Table 12: Translation tasks on LLaMA3 8B and LLaMA3 70B models.
<table>
  <tr>
    <th></th>
    <th>gpt3_translation</th>
    <th>en-fr</th>
    <th>fr-en</th>
    <th>de-en</th>
    <th>en-de</th>
    <th>en-ro</th>
    <th>ro-en</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>Dense Model</td>
    <td>12.62</td>
    <td>8.55</td>
    <td>18.44</td>
    <td>20.92</td>
    <td>11.40</td>
    <td>8.35</td>
    <td>27.86</td>
    <td>15.45</td>
  </tr>
  <tr>
    <td>Sparse Model (wo LoRA)</td>
    <td>7.62</td>
    <td>2.24</td>
    <td>16.51</td>
    <td>16.82</td>
    <td>6.68</td>
    <td>4.17</td>
    <td>20.34</td>
    <td>10.24</td>
  </tr>
  <tr>
    <td>Sparse Model (w LoRA)</td>
    <td>8.75</td>
    <td>4.37</td>
    <td>17.09</td>
    <td>18.24</td>
    <td>5.85</td>
    <td>4.21</td>
    <td>21.54</td>
    <td>11.44</td>
  </tr>
  <tr>
    <td>Stand KD</td>
    <td>8.65</td>
    <td>5.00</td>
    <td>16.90</td>
    <td>17.00</td>
    <td>6.28</td>
    <td>3.67</td>
    <td>20.63</td>
    <td>11.16</td>
  </tr>
  <tr>
    <td><b>BayesKD (SABO)</b></td>
    <td><b>9.72</b></td>
    <td><b>9.59</b></td>
    <td><b>17.91</b></td>
    <td><b>19.59</b></td>
    <td><b>8.30</b></td>
    <td><b>5.36</b></td>
    <td><b>23.12</b></td>
    <td><b>13.37</b></td>
  </tr>
</table>

Table 13: Translation tasks on Qwen2 7B and Qwen2 72B models.

<table>
  <tr>
    <th></th>
    <th>arc_challenge</th>
    <th>arc_easy</th>
    <th>boolq</th>
    <th>hellaswag</th>
    <th>openbookqa</th>
    <th>piqa</th>
    <th>Avg</th>
    <th>wikitext</th>
  </tr>
  <tr>
    <td>25%-Logits Dual-Scaling</td>
    <td>45.56</td>
    <td>71.13</td>
    <td>76.24</td>
    <td>74.32</td>
    <td>41.80</td>
    <td>78.29</td>
    <td><b>64.56</b></td>
    <td><b>15.77</b></td>
  </tr>
  <tr>
    <td>25%-Single-Temperature</td>
    <td>40.86</td>
    <td>68.76</td>
    <td>66.43</td>
    <td>71.79</td>
    <td>38.20</td>
    <td>75.18</td>
    <td>60.20</td>
    <td>18.18</td>
  </tr>
  <tr>
    <td>Dense-Logits Dual-Scaling</td>
    <td>56.81</td>
    <td>80.39</td>
    <td>84.00</td>
    <td>82.72</td>
    <td>46.18</td>
    <td>84.19</td>
    <td><b>72.38</b></td>
    <td><b>8.84</b></td>
  </tr>
  <tr>
    <td>Dense-Single-Temperature</td>
    <td>54.35</td>
    <td>78.59</td>
    <td>82.00</td>
    <td>80.67</td>
    <td>45.70</td>
    <td>82.37</td>
    <td>70.61</td>
    <td>10.38</td>
  </tr>
  <tr>
    <td>Original Dense Model</td>
    <td>53.16</td>
    <td>77.86</td>
    <td>81.16</td>
    <td>79.11</td>
    <td>45.00</td>
    <td>80.74</td>
    <td>69.50</td>
    <td>10.18</td>
  </tr>
</table>

Table 14: Comparison of different sparsity rates and temperature/scaling value strategies on LLaMA3 models.

Stand KD’s 11.16.

BayesKD (SABO) vs. Sparse Model (w LoRA) in Translation tasks (Table 12 and Table 13):
Similarly, BayesKD (SABO) demonstrates clear advantages over Sparse Model (w LoRA) on both LLaMA3 and Qwen2 across all translation tasks. Specifically, in Table 12, BayesKD (SABO) achieves 24.25 compared to 21.59 for Sparse Model (w LoRA), and in Table 13, it achieves 13.37, surpassing the 11.44 scored by Sparse Model (w LoRA).

Comparison of different sparsity rates and temperature/scaling value strategies (Table 14):
In general tasks, BayesKD (SABO) with logits dual-scaling strategies, as shown in Table 14, outperforms single-temperature configurations. For example, the logits dual-scaling strategy achieves an average score of 64.56 for 25% sparsity, compared to 60.20 for the single-temperature approach. This demonstrates the robustness of the logits dual-scaling configuration in resource-constrained settings.

Reviewer paper summaries: This paper proposes BayesKD, a Bayesian Knowledge Distillation framework for distilling LLMs into sparse student models, which involves three components: 1) A Logit Dual-Scaling strategy to bridge the logit distribution gap between teacher and student models across tasks; 2) A Knowledge Alignment Module using min-max normalization to align intermediate layer distributions; 3) A Logit-Aware Bayesian Optimization search to efficiently identify optimal hyperparameters. Experiments on diverse tasks across various teacher-student pairs demonstrate the effectiveness of the proposed approach.
Summary of strengths: 1.	It is imperative to address the capacity gap between the teacher model and the student model, especially when encountering LLMs. 
2.	Designing adaptive logits scaling strategies and using Bayesian optimization for hyperparameter searching is conductive to the mitigation of the inherent limitations of knowledge distillation.
Summary of weaknesses: 1.	This paper has always emphasized that its purpose of mitigating the capacity gap when distilling LLMs to sparse student models, yet none of the related work such as TAKD, MiniMoE, and MiniMA have been introduced as baselines. Worse, as a paper to be published in 2025, I am surprised that the baselines used for comparison in the current version are not competitive or even outdated, lacking comparison with the latest white-box KD methods for LLMs to demonstrate its superiority.
2.	The proposed approach involves several components, especially Bayesian optimization, which perhaps increases the complexity of knowledge distillation, and the authors are strongly recommended to discuss the computational overhead of the proposed approach.
3.	A complete description of the algorithmic procedures is missing.
4.	Section 3.3 is unclear, e.g., how the sub-dataset used for Bayesian optimization search is obtained and used is never mentioned. 
5.	Lack of implementation details.
Comments, suggestions and typos: This paper still contains numerous errors that need to be carefully corrected, including but not limited to: 
1.	A typo in Figure 1: Our Logits Dual-Scaling scaling -> Our Logits Dual-Scaling strategy.
2.	A typo in Line 261: Where -> where.
Overall assessment scores: 2.5
Metareview score: 3
Metareview: This work proposes a novel knowledge distillation framework, BayesKD, designed to improve the efficiency and effectiveness of fine-tuning large language models (LLMs) in resource-constrained environments. The framework introduces three key components: Logits Dual-Scaling, Knowledge Alignment Module, and Bayesian Distillation Optimization. The authors claim that BayesKD consistently outperforms baseline methods across various tasks and state-of-the-art LLMs, including LLaMA, Qwen2, Bloom, and Vicuna.
Summary of reasons to publish: - The introduction of BayesKD, which integrates Logits Dual-Scaling, Knowledge Alignment Module, and Bayesian Distillation Optimization, is a good contribution. These components address the challenges of large model size discrepancies in distillation scenarios, leading to improved performance. 
- The paper presents extensive experiments across diverse tasks and multiple model combinations, demonstrating the robustness and generalizability of BayesKD. The results show consistent performance improvements over baseline methods. 
- All reviewers agree that this paper is well-written and organized, with clear explanations of the proposed framework, including mathematical formulations and algorithmic details.