Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
Authors: Nicolas Boizard
Kevin El Haddad
CELINE HUDELOT
Pierre Colombo
Venue: Accepted by TMLR ()

Abstract: Deploying large language models (LLMs) with billions of parameters is often impractical in industrial settings due to constraints like cost, latency, and hardware limitations. Knowledge distillation (KD) provides a solution by compressing the knowledge from large, resource-intensive models into task-specific smaller ones. Various strategies exist, some relying on the text generated by the teacher model, optionally, leveraging its output logits to improve learning. However, these logit-based methods usually require the teacher and student models to share the same tokenizer, which limits their applicability across different model families.
In this paper, we propose the Universal Logit Distillation (ULD) loss, which uses optimal transport theory to enable distillation across different architectures and tokenizers. Our results demonstrate that ULD loss effectively facilitates the distillation process, paving the way for a more widespread use of distillation.

Published in Transactions on Machine Learning Research (01/2025)
Towards Tokenize Distillation: the Universal Logit Distillation Loss for LLMs
Nicolas Boizard
nicolos. hinbaa..btarrdSeennooolsss...e.
Diabolocom, Paris, France
MICS, CentruleSupétee, Poris-Saclay University, Fronce
Kevin Haddad
kerin.tthaadad@@@iibbllc....m.
Diabolocom, Paris, France
ISIA Lab University of Mons, Belgium
Céline Hudelot
MICS, CentraleSupéles, Poris-Saclay University, Fronce
Pierre Colombo
pierre@equall.ee
Equall.ai
MICS, CentroleSupéles, Poris-Saclay University, France
Reviewed OpenReview: https: httpss//opeerevviau openreview. net/ pet/formm? id= bwRmxi
Abstract
Deploying large language models (LLMs) with billions of parameters is often impractical in industrial settings due to constraints like cost, latency, and hardware limitations. Knowledge distillation (KD) provides a solution by compressing the knowledge from large, resource- intensive models into task-specific smaller ones. Various strategies exist, some relying on the text generated by the teacher model, optionally, leveraging its output logits to improve learning. However, these logit-based methods usually require the teacher and student models to share the same tokenizer, which limits their applicability across different model families. In this paper, we propose the Universal Logit Distillation (ULD) loss, which uses optimal transport theory to enable distillation across different architectures and tokenizers. Our results demonstrate that ULD loss effectively facilitates the distillation process, paving the way for a more widespread use of distillation.
1 Introduction
A significant. irend in NLP involves the utilization of large language models (LLMs) such as LLaaa (Touvron)
et al. 2023a), Mistral (Jiang et al. 2023), Falcon (Almazrouei et al. 2023). GPT-NeoX (Black et al. 2022
or Mixtral (Jiang et al. 2024). While LLMs offer impressive performance [Bubeck et al. 2023 their deployment is often hampered by hardware availability, cost, and latency bottlenecks. Several strategies,
such as efficient decoding [Leviathan et al.
2023 Ye et al.) 2023). model recycling (Lester et al. 2022
and model size reduction (Dettmerss et al.
2023
Ma et al.J 2023), have been developed to streamline their
use. Among these, knowledge distillation
(KD)
Bueiluundffned et al. 2006:; Hinton al. 2015) a widely
adopted technique (Sanh et al. 2020 Jiao et al.)
2020: Mohammadshahi et al. 2022 He et. al. 2023 Raman
al. 2023 Dasgupta et al. 2023), transferring the capabilities of large, complex teacher models into
manageable and smaller student models, tailored for specific tasks. This approach is particularly valuable in contexts where the comprehensive knowledge contained within LLMs is not wholly necessary. This process aims to maintain the peak performance of general models on specific tasks while minimizing latency and memory usage.

---

Published in Transactions on Machine Learning Research (01/2025)
Two approaches can be considered. The "white box" approach, where researchers propose loss functions that require access to the model architecture to compute similarities across lay- ets, forcing adjustment for each situation. In contrast, the "black box" approach, indifferent models latent spaces, relies solely on the output logit vectors from teacher and student. The black-bor approach, due its flexibility and generality, is easily implemented by practitioners through libraries or APIs, facilitating its adoption.
Overlap Ratio Tokenizers Vocabularies
Uaaaa
28.10%
22.71%
28.80%
64. 66%
Teacher Mistral
31.23%
25.07%
30.03%
64.06%
Pythia
Opt
Student
Over the past years, NLP researchers have extensively explored these distillation methods with teacher and student models sharing similar architecture, notably the BERT encoder (Sanh ct al.) 2020 Jiao et al. 2020 Sun et al.) 2020). This desire to maintain similar architecture between teacher and student.
Figure 1: Vocabulary overlap between teacher and student models {Sec. 4.2)
illustrating the challenge of cross-tokenizer distillation. (e.g.. Bloomz tokenizer has 30.03% of Mistral's vocabulary).
models by mirroring some of the teacher blocks, hidden sizes,
or relying on the same tokenizer, came from the need to find similar information supports to apply distillation losses.
However, KD for generative models, those relying on encoder-decoder or decoder architectures, has received less attention due to the lack of common support between models. Although models of different sizes exist, they often diverge in architecture and tokenizer (ig.. 1), making logit distillation loss inapplicable. In fact, recent research predominantly focuses synthetic data fine-tuning (He et al., 2022 Kramchaninova &
Defauw 2022 Ouyang et al., 2022), rather than refining logits loss in the black box approach. Thus far, the primary method for KD with decoder models is to use the text generated by the teacher model (He et al., 2023 Hsieh et al., 2023), and if possible, when students and teachers models belong to the same family, improving KD by employing output-logit distillation with Kullback-Lcibler divergence (Mohammadshahi et al.
2022 Raman el. al. 2023 Wang al. (2020a). This raises the following research question:
How can build general distillation loss that leverages logits capacity whill staying within a black box framework that is easy to implement?
Contributions: In this paper, we make the following contributions:
1. A universal logit distillation loss. We introduce new loss, Universal Logit Distillation Loss (ULD loss), versatile to tokenizers and with minimal assumptions about the architectures of teacher and student models.
2. Experimental Results. We demonstrate the robust effectiveness of our loss function in transferring the capabilities of many teacher models to different smaller student models for variety of specific tasks, including extractive question answering, generative question answering, and summarization.
3. Contributing to future research. We make our code model weights, and generated dataset openly available to facilitate future research, minimizing computational overhead and lowering entry barriers.
2 Problem Formulation & Related Work
2.1 Notations
We define a model vocabulary set, Ω his size, and Ω signifies its Kleene closure Let P(2) denote the of probability distributions defined over the sample space Ω. is defined as P(2) = p E [0, 1]
3
with E/=1, = 1. Consider D as a non-empty set with independent and identically distributed samples
https://github www.e//.///oeeeeaeoeeoeeooooeeea recci
hittps://huggingfaet co/Sicolas-B2Rt
The Kleene closure corresponds sequences arbitrary size written with words in Formally: Q U Ω
2

---

Published in Transactions on Machine Learning Research (01/2025)
(x*)==1 E St. Each x" is a sequence of tokens, where wt E Ω represents the jth token of the ith sequence. The notation x²= = (xi-1
E S2" denotes the prefix of length t. In this paper, we will denote by 2t the student vocabulary and by 22 the teacher's vocabulary.
Remark. In general S2 + 22 and [S2*| + | but Ω2 n2 +0.
Conditional textual generation: Conditional textual generation aims to model the probability distri- bution p. (x) over variable-length text sequence x by approximating (x(x) parameterized by 0 E θ to p. for any x. In this work, we assume the presence of a teacher (fe,) and student (fes), two pre-trained condi- tional language models with a general form applicable to both fo S2 - RISi, where the output is provided in unnormalized logit score. The function fe parameterizes po (fe, and fes respectively parameterize pes and )0,), i.e., for any sentence x, Po(x) = softmax (1o(x) where T TR+ denotes the temperature set as default to 1. Given an input sequence x, the pre-trained language model fe can iteratively generate an output sequence during inference by sampling 6t41 ~ po(-(ee)) with &, generate token.
2.2 Knowledge Distillation Framework
In knowledge distillation (KD), the objective is guide the learning of student model on a specific task using complex teacher model (Buciluundefined al.) 2006 Hinton et al., 2015 that possesses generic knowledge. This paradigm includes two main components: the student model accurately predicts the gold tokens, and a secondary loss (LKD) tasked to align the probability distributions of the student model with those of the teacher model. This alignment transfers broader knowledge not encapsulated in gold vectors (similar to standard basis vectors) used in the cross- entropy loss. Formally, the goal of the student is to minimize L:
cross-entropy loss (CCE), which ensures
L= LCE À LKD
where À E R can be used to control the trade-off between learning exclusively from text and knowledge coming from the teacher.
2.3 Knowledge Distillation Related Work
Building on Eqq. various cases have been examined.
Distillation from leacher-generated text occurs when À - This strategy is particularly advantageous when dealing with syuthetic data (Kramchaninova & Defauw Ushio et al. 2023), fact highlighted by the effectiveness of instructing large language models such as -355/4 (Wu et al. 2023 Bubeck et al.] (023). Distillation from teacher-generater text He el al. 2023 Hsieh et al. 2023: Zhou & Chiam 2023) will be considered as a baseline throughout the paper. The primary drawback of these methods lies in their failure to fully leverage all the information that can be provided by the teacher.
Distillation from teacher-generated text:
2022
Du et al. 2023
[Kim & Rush 2016
White-box approach: A further refinement of Eq. occurs when LKD relies on the internal features of the teacher to transfer knowledge et al., 2020 Sun et al. 2020). Popular fcatures include transformer attention and internal layers within both encoder-only and encoder-decoder models (Raman et al. 2023 Wang et al. 2020a 2021). However, these methods require access to models' internal mechanisms, which are not available through API access, and assume similarities in architectural patterns between models, forcing adjustments for each situation.
Black-box approach: In the black-box approach, practitioners only used the output logits of the model. They use these logits to align the student's output probabilities with those of the teacher through Kull- back Leibler divergence (KL) (Sanh et al. ). This method has emerged as one of the most widely adopted approaches, successfully distilling encoder, decoder, or encoder-decoder models (Timiryasov & Tastet 2023 Mohammadshahi al.) [2022 Zhao et al. 2023a). However, employing KL divergence necessitates that both student and teacher share the same vocabulary, a requirement impractical with current large language models (L.MMs) as reported in Fig. 1. We dig into the limitations of this method in the next section.
3

---

Published in Transactions on Machine Learning Research (01/2025)
Pair of Teacher/Student model with different vocabulary size.
Distillation process with logit distillation loss (L LeE + 2 LKD
1
2
X [x] ... Mxe1l x== INf.
Teacher Model:
Text with synthetic labels generated by the Teacher Model.
Vocabulary: Q
40, ((xE))
Pay (a,(-1x35 (xxx
Student Model:
10
Student soft labels. |02
Vocabulary: Q
Q
LxD
Zoom in LnD term, tasked to align the probability distributions of the student model with those of the teacher at the step.
Rop xxxx)
With padding
|
|0=|= [R]
ULD Loss with Wasserstein distance
00.70
gay
(xxx)
w;
Liz1
Pas
X
40y
(z x
Sorting and Padding
Pe.( (x22
[0.62]
Kullback-Leibler eivergence
Po.(-1X(x)
0.1
UNDEFINED
Student soft labels.
|(Pl
Figure Distillation using ULD loss. In block 4, the KL divergence cannot be defined as the two distributions do not have the same support, breaking the absolute continuity of the quotient in the KL logarithmic term. To alleviate this we rely the ULD loss which leverages closed form of the Wasserstein distance.
2.4 KL Distillation loss
When distilling using the KT. [i et al. 2021 the goal to force the student to learn the teacher's output probability distribution at each generation step. The formal definition the objective function is provided
Eq.
CE(() + AKL [q0x (-1x:x), Pas (. (+Ix)()
(2)
where |x| the length the tokenized sentence x, qer (-Ix.qe) and pes (-Ix²() are the probability distribution for the Teacher and Student models at the 8th steps. CE (f) denotes the cross-entropy loss at each generation time step i, expressed CE (1) log (Pog (x)(x²()) with x, the gold token for the 9th steps and KL the
Kullback-Leibler divergence defined as: KL [go (-1x2)) Pos (xxx+)] Σ qot (x|x<() log
[or (x|xet) Peg (a(x)))
where À E R+ controls the trade-off between the two terms.
Remark. Eq. 2 relies on equality across the vocabulary of the student and the teacher, i.e. 2= Ω2 S2" ensuring similar support of probability distributions.
Remark. Eq. also suppose absolute continuity for the distributions (=x(-)Xe)) < (-xxx))) at each
generation step t, making the use of padding impractical.
Our examination Eq. iillustrating in Fig. highlights that both vocabulary (similar support probability distributions) and absolute continuity constraints pose challenges to distilling two distinct LLM families with logits using KL loss. In the following section, we introduce our ULD loss, providing flexible framework for knowledge distillation across wide range of L.Mss.
4

---

Published in Transactions on Machine Learning Research (01/2025)
3 Universal Logit Distillation
3.1 Background on Optimal Transport
Optimal transport provides a mathematical framework for transferring mass between distributions while minimizing cost (Villani et al. 2009 Peyré et al. 2019). Within this framework, the Wasserstein distance, also known as the Earth Mover Distance, a robust metric for quantifying dissimilarities between distri- butions. This distance metric has gained popularity in NLP applications such hallucination detection
(Guerreiro et al.) 2022: Shuster et al., 2021 clustering ZZuang al. 2022 Ye et al. 2017 Xu et al. 2018
sentence similarity (Colombo et al.
2021 Xu et al.
2018 Bahuleyan et al.
2018
Wasserstein distance: The Wasserstein distance minimizes transport costs between sampled points from all possible couplings. Let us consider two sets probability distributions, P (S2s) and P (Ωr), respectively containing discrete distributions over spaces Qg and Ωr. We denote p E P (Sts and q P (ΩT) as two discrete probability distributions with E!anl p=dzz 1 and 1Brl q:6.r =1, where dzg and 627 represent probability mass points at xf, xf with x; ag and of E fr for distributions p and q. The values p; and qi are weight factors ensuring that the sum of weights is equal to 1. Under this discrete setting, computing the Wasserstein distance defined as:
A&sl 10rl Σ
(p(PP min TEl((--))
Σ
Tú CP
(3)
where [[(p q) is the set of joint distributions with marginals p and q. Cjjy represents the cost matrix, and T is the transport plan. For the rest of the work, focus the Wasserstein distance related to p 1. Consequently, the Wasserstein distance seeks the optimal approach to transfer probability mass from p to q. minimizing the transportation cost defined by the absolute norm.
Remark. Note that the Wasserstein distance (see Eq. 3) makes no assumptions about the support of p or q, unlike the KL divergence, making a natural choice for distillation.
3.2 Universal Logit Distillation loss
The Universal Logit Distillation loss (ULD loss) is a novel distillation technique designed to virtually distill any generative model teacher into any student. It aims to overcome the limitations of KL divergence, as discussed in Soc. 2.4 and Fig.
Intuition. The ULD loss retains the CF loss term to guide the model in generating the target token and introduces a Wasserstein Distance term to transfer knowledge from the teacher to the student. By minimizing the distance between the soft probabilities of the teacher and the student, our goal is to reproduce not only the predictions for the gold token but also the near-zero labels, which are crucial for performance and gencralization.
ULD loss: Formally, the ULD loss function formulated as:
x
LULD CE(() + x Wi [pos (1x22)) qor (+xx21)
(4)
where |x| = min (xx*1|1x²)) the minimum length between the sentences tokenized with the teacher or student tokenizers. Respectively 9er (.xx2, and Peş (-Ix..) are the probability distribution for the Teacher and Student models at the tth steps, CE(t) the cross-cntropy loss defined in Sec. 2.4 and W1 represents the discrete Wasserstein distance defined in Eq. 3 where E R+ controls the trade-off' between the two terms and set to 1.5 the rest of this paper as discussed in Appendix: Sec. B.1
Explanation. Similar to the KL loss, the discrete Wasserstein distance ensures that the confidence of the student each time step is close to the from the teacher.

---

Published in Transactions on Machine Learning Research (01/2025)
3.3 Fast Computation & Approximations
To the best of our knowledge, we are the first to motivate and propose the Wasserstein distance learning loss for distillation in the scope of the LLM decoder. Prior efforts focus on Sinkhorn regularization (Cuturi 2013 with oncoder-dccoder for classification (Bhardwaj et al.] 2022), while our focus diverges as we concentrate on the generative setting. This shift presents inherent challenges, as the naive computation of the Wasserstein loss in Fq. 3 exhibits a complexity of O(n log n), where n signifies the size of the larger support. While manageable in small classification scenarios with encoders, the magnitude of the vocabulary, which can extend to 100K tokens in generative tasks, renders this approach intractable, particularly for long sequences.
Closed form solution for ULD loss: To achiove efficient computation of the Wasserstein distance in Eq. 4. we introduce two additional refinements:
Uniform Support Length: We augment either the student or teacher vocabulary size through distribution padding, ensuring equal support size for both (i.e., [Pc] [Ωa]] |Q]).
Uniform Cost: As teacher and student supports differ, and no vocabulary relationship is established, we assert that each transport cost is equal to 1. While this may seem strong assumption, we will demonstrate that the approximation we draw still achieves better results in our case.
Under this assumption the Wasserstein distance used in the LULD loss becomes the one introduced by Peyré ct al. (019):
[x|
103
Wi =
p (x - q(0 x
(5)
with a complexity of O (n log(n)) Sec. A.1 where の¹ and oJ the permutations that sort in decreasing order the probability of student and teacher probability vectors. An extensive demonstration of this closed form is provided in App. A. written clearly and accessible.
Intuition.
The final version of Eq. is decomposed into two sums:
The outer sum iterates over the sequence length (denoted as |x|) to compute the Wasserstein distance at each time step I.
The inner sum Σ, (a)((s) lo (ta) represents the Wasscrstein (W;) distance between the logits of the student and leacher models at that time step.
To compute the Wi distance efficiently, we use closed-form solution derived under certain assumptions: uniform support length and uniform transport cost, as detailed in App. A] This solution computes the absolute difference between the sorted probability masses of the teacher and student vectors at time step I, ensuring that the overall structure of the teacher's probability distribution is preserved.
This process allows the student model to learn not only from the golden tokens (as used in the Gross-Entropy Loss) but also from the surrounding probability distribution, capturing distributional information from the teacher's predictions.
4 Experimental Setting
4.1 Evaluation Scenarios
We avoid fine-tuning teacher models to ensure aligmment with the black-box approach as training teacher models may be unavailable. In this way, we enable ULD Loss to operate in an unsupervised environment by generating all answers text with teacher models. For repeatability and fair comparison between experiments, we opted to retain original answers for the test set split. We investigated various scenarios to evaluate the ULD loss performance across different datascts and tasks. These comprised Extractive QA (Ext.), 2 Generative QA (Gen.), and 1 Summary (Sum.) tasks:

---

Published in Transactions on Machine Learning Research (01/2025)
SQuAD (Ext.): The Stanford Question Answering Dataset (SQuAD) Rajjuurkrr et al. 2016 is reading comprehension dataset with 87,600 questions generated by crowdworkers from Wikipedia articles. Answers are text portions from the relevant sections of the articles.
QED (Ext.): The QED Laamm et al., 2020) dataset, expertly annotated, extends from a subset of the Google Natural Questions datasct, comprising 7,640 question-nsswerig pairs with explanations. Our focus is exclusively on extracted answers (spans).
FairytaleQA (Gen.):
The FairytaleQA Dataset (Xu et al. consists of 10,580 questions from 278 children-friendl) stories. Questions may be explicit or implicit.
2022), created by educational experts,
PubMcdQA (Gen.): The PubMedQA (Jin et al. 2019) dataset contains question-answer pairs extracted from medical papers. Questions are based on litles, context on abstracts, and responses on conclusions. Due the dataset size and context length of our student. models, we subset the dataset by selecting the first 50,000 smaller items.
DIALOGSum (Sum.): DialogSum (Chen et al. 2021 is large-scale dialogue summarization dataset, consisting of 13,460 spoken dialogues with corresponding summaries and topics.
4.2 Experimental Choices
Baseline: As far as we know, the only method currently capable of distilling any pair of teacher and student LLM models in a black-box approach is distillation from teacher-generated text scen in Sec.. 2.3 Throughout the remainder of this paper, distillation from teacher-generated lexts will serve as the baseline for evaluating the distillation process using the ULD loss across different teacher-student pairs, tasks, and datasets. Additional experimentation with KL Div methods between models from similar families can be found in Appendix Sec. B.2
We employed two teacher decoder models, cach with 7 billion parameters: LLama 2 7b Chat (LLama) by Touvron el al. (023b) and Mistral 7b Instruct (Mistral) by Jiang el al. (2023). These instruct models were chosen for their ability to generate few-shot answers (Brown et al. 2020 Wang et al. 2020b) across diverse tasks and their distinct vocabulary set as shown in Figure
Teacher Models:
Student Models: We chose student models from various LLM families and architectures with parameters ranging between 160 million to billion: OPT 350m (Zhang et al. 2022). Pythia 160m, Pythia 410m, Pythia 1b (Biderman et al. 2023); Bloomz 560m (Muennigholf el al., 2023) all decoder models and MTO 580m (Muennighoff et al. 2023) an encoder-decoder. It's important note that models can have been already pre-trained on some datasets such SQuAD for Bloomz and MTO.
Training process: ULD loss distillation and teacher-generated text distillation were processed uniformly. The two teacher models generate answers in inference mode for the five datasets. These answers then utilized to train student models. During training, student models are trained exclusively to predict answers, in teacher forcing configuration. Logits used for the ULD loss are calculated by applying teacher models to the same data points they generated. Teacher's weights were frozen, ensuring consistency in teacher- generated sentences during inference and training. Additional parameter details (learning rate, batch size, etc.) can be found in the Appendix App. G.
4.3 Teacher Performances
Distilling using synthetic teacher-generated answers might restrict student performance on teacher's ones. To measure distillation efficiency accurately, we report. the average native performances across tasks for both teachers Tab. 1 (details in Appendix Appp D.. We chose a primary metric for each task reflecting associate performances: F1 score for Extractive QA (Sokolova et al. 20066, BERTScore for Generative QA (Zhang* et al., (220)), and Rougo-Lsum for summary task (Lin) (2004). Comprehensive evaluation methods and outcomes, encompassing prompts and few-shot. examples, are provided in the Appendix App.
7

---

Published in Transactions on Machine Learning Research (01/2025)
Table I: Average performance teacher models across tasks with their main metrics. is important note relative difference of 30% in performance between teacher models on the summary task.
Extractive
Generative (BERTScore)
Summary (Rouge-Lsum)
Model
(F1)
LLama
69.51
36.11
23.90
Mistral
64.66
33.47
34.71
5
Empirical Results
Table 2: Overall performance of Teacher/Student pair models trained with ULD Loss and teacher-
generated text (Raw Text) across tasks with their main metrics. Evaluations performed over respective test splits.
Teacher
Model
Method
SQUAD
QED
FairytaleQA
PubMedQA (BERTScore)
DIALOGSum
(F1)
(F1)
(BERTScore)
(Ronge-Tsum)
LLama
81.30
57.72
41.59
30.62
23.90
Teacher
Mistral
76.31
53.01
36.01
30.93
34.71
Raw Text
70.78
48.64
33.78
27.99
20.58
OPT-350m
ULD Loss
72.97
49.06
33.03
30.01
20.11
Raw Text
71.39
47.04
33.02
29.86
20.94
LLama
Pythia-410m
ULD Loss
74.14
49.15
34.83
29.89
22.19
Raw Text
73.54
50.99
36.70
29.14
20.01
Bloomz-560m
ULD Loss
75.90
55.33
37.86
30.01
22.67
Raw Text
71.64
50.13
30.09
27.91
31.44
OPT-350m
ULD Loss
73.35
50.88
30.44
30.30
32.17
Raw Text
71.50
47.07
31.44
28.25
31.64
Mistral
Pythia-410m
ULD Loss
73.64
50.38
31.79
29.55
33.10
Raw Text
73.34
52.15
32.64
28.87
31.95
Bloomz-560m
ULD Loss
76.00
55.79
33.93
30.60
32.58
Raw Text
72.03
49.34
32.94
28.67
26.09
Average
ULD Loss
74.33
51.77
33.65
30.06
27.14
5.1 General Results
We empirically validate the effectiveness of the ULD loss step-by-step. First, report in Tab. 2 the aggre- gated key metrics performance over the different datasets and teacher/student pairs. ULD loss achieves the best overall results, which indicates that the proposed ULD loss effectively improves the performances of every student model on a varicty of downstream tasks using any Teacher. Notably, ULD loss exhibits an average improvement of 2.30 points over models trained on teacher-generated text for extractive QA lasks and Bloomz outperforms his teacher Mistral on the QED datasets. Furthermore, concerning summarization tasks, the 30% performance disparity between LLama LLama/Mistral Mistral TTb.. persists in their distilled counterparts Tab. 2), underscoring the critical role of teacher performances.
5.2 Student Size Ablation Study
General results in Tab. 2) show a consistent pattern regarding the model size and the gain achieved with the ULD loss, especially for challenging tasks such as generative QA. To understand the impact of student size on distillation capability, we performed an ablation study over the Pythia family. We hold the training dataset size fixed at 100% and compare the performance of models from 160m, 410m to 1b parameters and

---

Published in Transactions on Machine Learning Research (01/2025)
report results in Fig. 3
We observe that incorporating ULD loss consistently enhances student models across
SQUAD
QED
FairytaleQA
DIALOGSum
24
50
BERTScore
Rouge- -Lsum
22
35
70
40
30
20
60
ULD loss
25
18
Raw Text
Teacher
160M
410M
160M
410M
18
160M
410M
18
160M
410M
Figure 3: Student model size ablation with the Pythia family trained by LLama teacher. Trainings are conducted with ULD loss and teacher-generate text (raw text). Evaluation scores test sets are depicted the Y-axis, while Pythia model sizes are the X-axis.
various tasks. The enhancements are particularly noticeable for smaller models on simpler tasks, while ULD loss requires larger models for effectively distilling teacher logits harder tasks. This is espocially evident in lasks requiring reasoning, such as FairylaleQA. While using logits teacher improves training, deep reasoning tasks still require appropriate model sizes to process complex relationships taught by teachers. Generally, we observe a significant increase in capacity transfer from teacher to student models through the use of ULD, enabling student models to match models twice bigger trained with the teacher-generated text method. For example, Pythia 410m with ULD loss matches the performance of the Pythia 1b distilled with pacherrgenerated text on QED and DIALOGSum.
5.3 Dataset Size Ablation Study
In this section, we investigate and report in Fig. 4 the influence of the datasct size for models trained with ULD Loss or wracher-generater text. We perform ablations with respectively 25%. 50%. 75%, and 100% of dataset size while keeping training parameters constants. Training for each ratio was conducted five times to establish a range performance. For every ablation ratio, models trained with ULD loss achieved better performance than models trained on teacher-generated text. Specifically, with 50% of a dataset, ULD loss models overpass the performance of teacher-generated text models trained with full dataset.
SQuAD
FairytaleQA
35.0
74
Raw lixt
34.0
72
BERTScore
33.5
33.0
32.5
68
32.0
31.5
66
25%
50%
100%
25%
75%
100%
Figure 4: Dataset size ablation with a (LLama/Pythia-441 pair trained with ULD loss or teacher- generated lext. The X-axis indicates the % of data used during training while the y-axis represents the test set score. Minimum and maximum values are represented by the error bars in the plot while the mean is represented by points.

---

Published in Transactions on Machine Learning Research (01/2025)
5.4 Training Regularization
To understand the impact of the ULD loss during training we decide to compute the validation ULD and Cross-entropy loss values for two pairs of teacher/student on the SQuAD dataset every 200 steps during epochs. We report the curves formed by this point in Fig. It appears that using the ULD loss contributes to stabilizing the distillation process over training and mitigates overfitting issues, enabling the model to train more effectively across multiple epochs. It's worth noting that incorporating the ULD loss during training stabilizes both ULD and Cross-entropy loss.
LLama Pythia 410m
LLama Bloomz 560m
LLama Pythia 410m
LLama Bloomz 560m
1.3
0.95
1.2
1.2
Cross-entropy Loss
Cross ee-e-ettrroyy Loss
1.0
Haw Hext
ULD Loss
1.1
0.85
0.9
0.80
ULD
1.0
0.75
0.70
0.7
0.9
0.9
0.65
5000
10000
5000
10000
5000
10000
5000
10000
Step
Step
Step
Step
Figure 5: Evolution of validation ULD and Cross-entropy loss curves during training on SQuAD dataset for a LLama/PPaee Py0mhia4410m and LLama/Bloomz-55h Bloomz-560m Teacher/Stee /Student pair model. For teacher- generated text models (raw text), the ULD loss was only computed during validation and did not impact the training.
Remark. Euen though the entropy is lower with raw tert data than with ULD Loss, the data does not lead to beller results. In fact, during braining, models distilled with ULD Loss are trained to reproduce the predictions of the teacher model. Consequently, they are not trained to minimize cross-entropy and have mazimum confidence in each prediction.
For this reason, sludent model trained to reproduce the confidence of a teacher model of 0.92, compared to a model trained only on row tezt and trained predict will have a lower probability on the golden token and consequently a higher Cross Entropy Loss. However, this model will be better calibrated its true confidence for the sentence.
6 Distillation of Decoder Teacher to Encoder-Decodee Student
As shown in Sec. ULD loss effectively transfers knowledge from any pair of teacher/student decoders. Moreover, by leveraging solely on logit information and adopting black-box approach, ULD loss should be able to extend its versatility and improve cross-architecture distillation. To validate this, we distill a pair LLama/ Mlama/MTO-500m and focus our experimentation on PubMedQA, DIALOGSum, and QED to avoid any data seen during the pre-training of MTO with the xP3 dataset (Muennighoff et al. 2023 Sanh et al. 2022
teacher/studie
Table 3: Distillation of LLama teacher (decoder) to an MTO-580m (encoder-decoder) with ULD Loss and teacher-generatel text three data sets.
QED
PubMedQA (BERTScore)
DIALOGSum (Rouge-Lsum)
ULD loss
(F1)
Raw Labels
55.63
27.56
23.22
ULD Loss
56.01
30.19
23.92
10

---

Published in Transactions on Machine Learning Research (01/2025)
The results presented in Tab. demonstrate that incorporating logit information from decoder teacher using ULD loss can enhance the performance of an encoder-decoder student model. Notably, the inherent ability of the encoder-decoder in the summary task seems to be limited by the synthetic answers as teacher- generated text distillation matches the teacher's performance. However, by using the logit information with the ULD loss, the student model still leads improved results, suggesting a successful knowledge transfer through logits. With this additional knowledge, the student model slightly outperforms the teacher onc. Furthermore, in generative tasks where decoder architectures perform, the encoder-decoder student model gained 2.63 points over distillation with teacher-generated text.
7
Conclusions
In this work, introduce the Universal Logit Distillation (ULD) loss, a novel approach for distilling any decoder teacher model into any student model for LLM generative tasks, utilizing new closed form of the Wasserstein distance never seen before for distillation. ULD achieves superior overall results and matches the performance of traditional teacher-generated text distillation with only half the training dataset or student model size, while effectively preventing overfitting. Our comprchensive experiments demonstrate the efficacy of the ULD loss across a variety of tasks, datasets, and model architectures, showcasing its advantages over standard teacher-generated text distillation methods.
Broader Impact Statement
Knowledge distillation aims to reduce the size, cost, and energy consumed by a model at inference time. Our work opens new perspectives in this area, aligned with the desire for sobriety, notably for environmental reasons. Although KD allows partial transfer of larger model performance, smaller models remain limited their reasoning capacity and are more susceptible hallucinatory behavior (Rawte al. 2023), especially in open-ended generation tasks. This phenomenon has not been extensively studied in this work. Furthermore, by distilling knowledge from existing models, if bias is already present in the teacher model, it may be transferred to the student model. This is not unique to our method, but it's a common risk for all knowledge distillation
11

---

Published in Transactions on Machine Learning Research (01/2025)
References
Ebtesam Almazrouci, Hamza Alobcidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Maz- zotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models. Number arXiv:2311.16867 arXiv, November 2023. URL http::/arxixe http::/aaxwv.oog///o// www.ooggg///../ooooeo...oo...eat htrg/abs/2311... 2311 16867 arXiv:2311.16667 [cs].
Haroesh Bahuleyan, Lili Mou, Hao Zhou, and Olga Vechtomova. Stochastic wassorstein autoencoder for probabilistic sentence generation. 2018.
Rishabh Bhardwaj, Tushar Vaidya, and Soujanya Poria. Knot: Knowledge distillation using optimal trans- port for solving nlp tasks, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brion, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: suite for analyzing large language models across training and arXiv, May 2023. doi: (185550arXiv.23000 arXiv:2304.01373 [es].
scaling. Number arXiv:2304.0133 01..3044000033330:::300000000000 URL
23004 003373
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b. An open-source autoregressive language model, 2022.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information, 2017. URL btttp::/aaxii..oggabbb////
04606
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Langnage models are few-shot learners. volume 33, pp. 1877 1901, 2020.
Sébastion Bubeck, Varun Chandrasckaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Tee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang, Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
Cristian Buciluundefined, Rich Caruana, and Alexandru Niculescu- Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Dala Mining, KDD '06, pp. 133-5411 New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933395. doi: (0.1145/1150402.155444
URL htttpp::/aoo..rgg//// 1ee111111///1....1 115040 1150464
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. Dialogsum: A real-life scenario dialogue summarization dataset, 2021.
Pierre Colombo, Guillaume Staerman, Chloé Clavel, and Pablo Piantanida. Automatic text evaluation through the lens of Wasserstein barycenters. In Marie-Francins Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10450-10466, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 1036653/v12222.emm//// eeenl-emnaaeee mat.p-mmai.......1. URL https://aclanthoolgg/
org/2021. emm2pp1..mmmmmmmm..mmmmmmm2m1lm1 mww...11.1
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. volume 26, 2013.
Sayantan Dasgupta, Trevor Cohn, and Timothy Baldwin. Cost-offective distillation large language models.
Findings of the Association for Compulational Linguisties: ACL 2023, pp. 7346 7354, 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms, 2023.
12

---

Published in Transactions on Machine Learning Research (01/2025)
Zilin Du, Haoxin Li, Xu Guo, and Boyang Li. Training synthetic data beats real data in multimodal relation extraction, 2023.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Cham- bon, Laetitia Chapel, Adrien Corenñoos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python opti- mal transport. Journal of Machine Learning Research, 22(78):1-8, 2021. URL http://jmlr.... wwtph/rr..gppppppr/////oo 202/20-441.hhe
Nuno M Guerreiro, Pierre Colombo, Pablo Piantanida, and André FT Martins. Optimal transport for unsupervised hallucination detection in neural machine translation. 2022.
Nan He, Hanyu Lai, Chenyang Zhao, Zirui Cheng, Junting Pan, Ruoyu Qin, Ruofan Lu, Rui Lu, Yunchen Zhang, Gangming Zhao, Zhaohui Hou, Zhiyuan Huang, Shaoqing Lu, Ding Liang, and Mingjie Zhan. Teacherlm: Teaching to fish rather than giving the fish, language modeling likewise. Number mxXiv:2310.90011 arXiv, October 2023. doi: 10. 10.48550/arXiv.23100 23w...........000... URL http://arxiv. http://axxiv..r//bbbbb 2310.19011 arXiv:2310.19019 [cs].
Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Halfari, and Mohammad Norouzi. Generate, Annotate, and Learn: NLP with Synthetic Text. volume 10, pp. 826-844 842, 08 2022. doi: 10. (.1112/taa)) a 000992. URL
https://doi. org/ 100 1ttl///ecllaa00000000 004992
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. Number NMXXI=1160...233311xxx1:1111..11 025331. arXiv, March 2015. doi: 00488500/aXX1v.158...55. URL http: 02531 urXiv:1503.0253
[cs, stat].
Cheng- Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujäi, Alexander Ratner, Ran- jay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaune Lample, Lucile Saulnier, Lélio Renard Lavaud, Manie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. Number arXivv2310.06825 arXiv, October 2023. doi: 10.48550/arrii.222 06825. URL http://arxiv.rrrre httpg//bb/233100 06822
arXiv:2310.06825 [cs].
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Amn Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding, 2020.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical rescarch question answering, 2019.
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jamnalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A study of bfloat16 for deep learning training, 2019.
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317-1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. (.L1866//11/D161110 URL http://cclannooloogooo////////
13

---

Published in Transactions on Machine Learning Research (01/2025)
Alina Kramchaninova and Arne Defauw. Synthetic data generation for multilingual domain-adaptable ques- tion answering systems. In Helena Moniz, Lieve Macken, Andrew Rufener, Loic Barrault, Marta R. Costa-jussà, Christophe Declercq, Maarit Koponen, Ellie Kemp, Spyridon Pilos, Mikel L. Forcada, Car- olina Scarton, Joachim Van den Bogacrt, Joke Dacms, Arda Tezcan, Bram Vanroy, and Margot Fonteyne (eds.); Proceedings of the 23rd Annual Conference of the Europeon Association for Machine Transla- tion, pp. 151-160, Ghent, Belgium, June 2022. European Association for Machine Translation. URL org/ eamt-1.18
Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins, Qed: A framework and dataset for explanations in question answering, 2020.
VI Levenshtcin. Binary coors capable or 'correcting deletions, insertions, and reversals. In Soviet Physics- Doklady, volume 10, 1966.
Brian Lester, Joshua Yurtsever, Siamak Shakeri, and Noah Constant. Reducing retraining by recycling parammeter-fficient prompts, 2022.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decod- ing. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Tianda Li, Yassir El Mesbahi, Ivan Kobyzev, Ahmad Rashid, Atif Mahmud, Nithin Anchuri, Habib Hajimo- lahoseini, Yang Liu, and Mehdi Rezagholizadeh. dis, 2021.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74 81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https: //aclanthology..o http:/hoooooooooo/oo00--0-- org/w04-1011
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Lim-pruner: On the structural pruning of large language models, 2023.
Alireza Mohammadshahi, Vassilina Nikoulina, Alexandre Berard, Caroline Brun, James Henderson, and Laurent Besacier. Small-100: Introducing shallow multilingual machine translation model for low-resource languages. Number 22XIv=2200011222aaxv:20000...
arXiv, October 2022. doi: 10.48550 /ar// (0.48550//rXiv:22211
11621. URL
1162 11621 arXiv:2210.11621 [cs].
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Al- ham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and arXiv,
Colin Raffel. Crosslingual generalization through multitask finetuning, Number arXiv:2211.01723 May 2023. doi: 10.48550/ar 20..2222.111111 URL http:: htt///aaaixirrra/////aaaaaa wtwp/org////////ooo/e 22181111111111 01786 arXiv:2211.0178 [cs).
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. volume 11, pp. 355 607. Now Publishers, Inc., 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016.
Mrigank Raman, Pranav Mani, Davis Liang, and Zachary Lipton. For distillation, tokens are not all you need. In NewrIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. URL https:
Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models. 2023.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter, 2020.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiogler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Suezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization, 2022.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation, 2021.
Marina Sokolova, Nathalie Japkowicz, and Stan Szpakowiez. Beyond accuracy, f-score and roc: A family of discriminant measures for performance evaluation. volume Vol. 4304, pp. 1015- 1021,1 01 2006. ISBN 978-3-540-49987755 doi: 10.1007/11941439 114.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: compact lask-agnostic bert for resource-limited devices, 2020.
Inar Timiryasov and Jeun-Loup Tastet. Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. Num- ber
13971. arXiv, February 2023a. doi: 10.48550/arXiv 130771 URL http:://axxiv..rge wtwp///g//a///////a 13971 1eXiv:2302.13977 [cs].
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswarni, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Fric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyaan Zarov, Yuchen Zhang, Angela Fan, Melanic Kambadur, Sharan Narang, Aurelion Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama Open foundation and fine-tuned chat models, 2023b.
Asahi Ushio, Fernando Alva-Manchego, and Jose Camacho-Collados An empirical comparison of Im- question and answer generation methods. 2023.
Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020a.
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multi head self atlf-attention relation distillation for compressing pretrained transformers, 2021.
Yaqing Wang, Quamming Yao, James T Kwok, and Lionel M Ni. Generalizing from few examples: A survey on few-shot learning. volume 53, pp. 1-34. ACM New York, NY, USA, 2020b.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji. Lamini-In Im: A diverse herd of distilled models from large-scale instructions, 2023.
Hongteng Xu, Wenlin Wang, Wei Liu, and Lawrence Carin. Distilled wasserstein learning for word embedding and topic modeling, 2018
Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia- Jun Li, Nora Bradford, Branda Sun, Tran Bao Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer. Fantastic questions and where to find them: Fairytaloqa an authentic dataset for narrative comprehension, 2022.
Jianbo Ye, Panruo Wu, James Z. Wang, and Jia Li. Fast discrete distribution clustering using wasserstein barycenter with sparse support, 2017.
Qinyuan Ye, Iz Beltagy, Matthew Peters, Xiang Ren, and Hannaneh Hajishirxi. FiD-ICL: A fusion-in-decoderr approach for efficient in-context learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 1158-8183, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/v /2023 454. URL
acl- .54
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. Number (xXiv:2205.01066 arXiv, June 2022. URL. htttp://axxiv.org abs/2205. 01068 arXiv:2205.01065 [cs].
Tianyi Zhang*, Varsha Kishore*, Felix Wu", Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text gencration with bert. In International Conference on Learning Representations, 2020. URL https:
//openreview http:eeeeeeeoooommmmdeeeetttiiee
Jiachen Zhao, Wenlong Zhao, Andrew Drozdov, Benjamin Rozonoyer, Md Arafat Sultan, Jay- Lec, Mohit lyyer, and Andrew McCallum. Multistage collaborative knowledge distillation from large language models. Number arXiv:2311. 1.06640. arXiv, November 2023a. URL hhtt:://arx.a http:///xxxvvvorr//a/// htts/ab//231111 2311. 08640 arXiv:2311.08640 [cs].
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023b.
Tianxun Zhou and Hwee Chiam. Synthetic data generation method for data-free knowledge distillation in regression neural networks. volume 227, pp. 120327. Elsevicr, 2023.
Yubo Zhuang, Xiaohui Chen, and Yun Yang. Wasserstein k-means for clustering probability distributions, 2022.
16

---

Published in Transactions on Machine Learning Research (01/2025)
A Appendix Proof of the closed Form
In this section, we gather an extensive proof of the close form used. Although the proof is hard, we could not find it properly written in an easy and readable manner.
Step 1: Wasserstein Distance Definition
Given two discrete distributions and over a vocabulary Ω of size the Wasserstein distance W1 is given by:
Wi q) =min
F (1,))c((,
where Ti.,j) is the transport plan, and c(t,j) is the cost of transporting mass from to j.
Step 2: Uniform Cost Assumption
Assume that the cost d(i.j) = 1 for all i,j. This reduces the Wasserstein distance to:
W(p.q) = min
((.,)
min
T (i.
Step 3: Transport Plan Constraints
The transport plan xf,jj) must satisfy:
Σ (i.)) (() for all é
Σ t(i,)) q()) for all j
Step 4: Closed-Fore Solution Using Permutations
To derive a closed-form solution, we can utilize the assumptions and inherent structure of the problem:
Uniform Cost Implication: Given the uniform cost assumption, the cost of transporting mass from to j is constant. Consequently, the optimal transport plan minimizes costs by directly matching the masses without any additional expenditures.
Sorting and Matching: Assume the distributions p and q are sorted in decreasing order. If they are not already sorted, we can introduce permutations as and or such that p(as(i)) and (()r(i)) in sorted order.
Simplified Transport Plan: With the distributions sorted, the optimal transport plan *i.,) becomes straightforward. We can directly match p(as(i)) with q(ār(i)) for corresponding indices
With this optimal transport plan, the Wasserstein distance simplifies to:
w(p. Σ (i)o(()) q(or(1))
17

---

Published in Transactions on Machine Learning Research (01/2025)
Let's prove that the optimal transport plan is the that directly matching the masses.
We assume that the cost c(t,j) = 1 for all and j. This means that the cost of transporting any unit of mass from any to any j is identical. To prove that the optimal transport plan under the uniform cost is one that directly matches the masses, we need to show that any deviation from direct matching does not reduce the cost.
Step 1: Feasibility of the Direct Matching Transport Plan
Consider the direct matching transport plan π(i,j) = p(t) dig, where dig is the Kronecker delta function. This means that (V, is non-zero only when i = j, and directly assigns p(i) to q()).
p())
ii= otherwise
(i,
0
This plan satisfies the constraints because:
8 for all j
Σ x(.,)) Σ p(1) Jzg PP()) for all
(l.,) Σ
p()) 5, =9())
Step 2: Cost comparison
According to the direct matching plan, the cost is:
n Wi Σ (1, Σ p())
Similarly, for any alternative transport plan that satisfies the constraints, the total cost remains unchanged because the total amount of mass transported is conserved and each unit of mass incurs a uniform cost of 1:
x (i,j)
Σ p(i)
Conclusion
Given that the cost under any valid transport plan is equivalent to the total mass transported, which is per unit of mass due to uniform cost, the direct matching plan is optimal. Consequently, under the assumption of uniform cost, the optimal transport plan directly matches the masses without incurring additional costs, Thus, the Wassorstein distance can be approximated by summing the absolute differences between the aligned distributions, yielding the exact closed-form proposed under these assumptions:
Wi(p. = Σ (p()) q())
i=1
A.1 Complexity
[x]
x
19
W
p q(05r (t) x
CE(()
log (pag (c; (x)()))))
Cross Entropy Sec.. 2.4
Closed form Wasserstein Distance
Eq. 5
18

---

Published in Transactions on Machine Learning Research (01/2025)
gor xet)) x log q0p (x) X(t) Pag (rrg x(te
KL Σ
Kullback Leibler Divergence Seec. 2.4
As discussed in (Sec. 3.3), we emphasize the significance of newly closed-form Wasserstein distance for knowledge distillation, which from the base of our knowledge has not been previously seen. Indeed, the conventional approach poses challenges due to the Wasserstein high computational complexity of O(n--log n), where n signifies the size of the larger support, hindering scalability and practical adoption. However, our discovery of closed-form solution with the ULD Loss markedly reduces this complexity to O (n log(n)). Precisely, in the case of sentence of size [X] and output vector of support size [St] (vocabulary length) the complexity cost O (IX) (192) log (2))) with log (S2 the complexity induced by sorting algorithms in general
GPU frameworks !
chapter-46 40 improved gpw.--eddd--------- spu-sorting
19

---

Published in Transactions on Machine Learning Research (01/2025)
B Appendix Complementary experiments
B.1
Distillation Factor Ablation
Model
Dataset
À =
À = 0.5
X =
= 1.5
X = 2
À = 3
pythia-410m
FairytaleQA
33.02
34.83
35.52
34.83
34.93
33.99
pythia-410m
SQuAD
71.39
74.33
74.53
74.14
74.75
74.91
pythia-410m
Dialogsum
20.94
22.68
23.26
22.19
21.74
21.24
bloomz-560m
PairytaleQA 36.70
36.98
36.95
37.86
37.96
37.21
bloomz-560m
SQuAD
73.54
75.96
75.95
75.90
77.23
76.92
bloomz-560m
Dialogsum
20.01
22.26
22.34
22.67
21.90
20.02
opt-350m
FairytaleQA
33.78
34.60
33.95
33.03
33.45
32.72
opt-350m
SQuAD 70.78
73.43
73.78
72.97
74.03
73.26
opt-350m
Dialogsum 20.58
20.30
20.02
20.11
19.63
18.94
Table 4: Ablation Study on the Lambda Distillation Factor with the LLama teacher. The values compared for each dataset are as follows: FairytaleQA (BERTScorc), SQuAD (F1), Dialogsum (Ronge-Lsum)
Due to the prohibitive costs associated with training models across various datasets using multiple lambda values, restricted our analysis to specific values. These values were identified through ablation study involving three students across three distinct tasks, assessing six lambda values. A lambda value higher than implics greater emphasis on the ULD Loss than the cross-cntropy term, as detailed in Eq. I Conversely, lambda value of zero indicates that distillation proceeds without incorporating logit information. Results presented in Tab. 4 demonstrate that the ULD loss consistently enhances performance compared to baseline 0) in nearly experimental scenarios. Based on these findings, we have chosen to focus on À for baseline purposes and 1.5 for the ULD Loss provide the most general results in the remainder of this paper. Importantly, the ablation suggests that fine-tuning À could enhance performance by much 1.5 percentage points over the general results presented in this paper.
B.2 Comparison between KL Divergence and ULD Loss
Model
Dataset
Method
X = 0
À
0.5
À =
X = 1.5
À =
À = 3
TinyLlama-Ia
FairytaleQA
KL Div
38.45
39.21
39.20
39.67
39.61
39.88
TinyLJama-1.11
FairytaleQA
ULD Loss
38.45
38.53
39.37
39.39
39.75
39.36
TinyLlama-1.1. 1.1B
SQuAD
KL Div
81.06
81.87
81.95
82.12
82.06
82.02
TinyLlama-1.11
SQuAD
ULD Loss
81.06
81.52
81.90
82.04
81.72
81.47
TinyLlama-1.1B
Dialogsum
KL Div
24.77
24.36
24.17
24.17
24.03
24.07
TimyLJama-1.1B
Dialogsum
ULD Loss
24.77
24.97
24.74
24.08
24.48
24.85
Table 5: Comparison between a LLama-7b teacher distilled with h-l.laam--..---ttttetttttttttt 3T, model based on the same architecture, using the Kullback-Leible divergence the ULD Loss. The values compared for each dataset are as follows: FairytaleQA (BERTScore), SQuAD (F1), Dialogsum (Rouge- Lsum). The distillation performer with a lambda value = 0 is equivalent to training with raw data.
Although ULD Loss is designed to function across different model families, we opted to evaluate it alongside the Dallback-Leibler divergence (KL Div) method within the specific environment, where both the student and teacher models share the same architecture. In our case the student model is TinyLlama and the teacher is Llama-7b. As illustrated Tab. 5] ULD Loss demonstrates performance comparable existing black-box distillation methods. Notably, we observed that the most significant improvements between text versus
20

---

Published in Transactions on Machine Learning Research (01/2025)
logit distillation occur at identical lambda values for ULD Loss and KL divergence. This result raises the hypothesis of parallel behavior between the two methods when applied to models of the same architecture. However, ULD Loss offers the unique advantage of applicability across diverse model pairs, making it a superior alternative to KL divergence.
21

---

Published in Transactions on Machine Learning Research (01/2025)
C Appendix General Results
C.1 Summary
Toacher
Model
Meikod
Datase
Rouge=1
Rouge-2
Rouge-L
Roage-Lon
blama
Blooma- 500m
Raw Toxs
DIALOGSeM
24.71
10 06
19.90
20.01
Llama
Blaoms-5o0m
ULD Loua
DEALOCSum
28.08
11.68 68
22.04
22.67
Mastral
DIALOGSum
Mistral
Mloom 5-000
ULD Loos
DIALDGSem
40.37
84
32.6
32. 58
Llame
OPT-350m
Raw Toxi
DIALOCSum
25.4
10.48
20.57
20.68
Liama
OPT-310m
DIALOGSum
Mistrll
QPT-3500
Row Taxs
DEAL.GGSom
39.33
87
31.48
31 .44
Mistral
OPT-3500u
ULD Loss
DIALOGSum
39.8
16. 76
82.19
92:17
Llama
Pythia-410m
Raw Text
DIALOGSum
Pylhis-410u
ULD Lous
DIALOGSum
27.29
22.17
22.19
Mistral
Pyrthts-410m
Raw Taxt
DIALOGSum
39.69
15.0
31.42
$1.64
Mistral
Pythla-110m
ULD Loos
DIALOGSom
03
33.1
Pythis-160u
DIALOGSum.
20.34
7.40
Llama
Pythits160m
ULD Lous
DIALOGSum
22.94
8.30
19.58
19.55
Ilama
Pythta-1h
Now Taxt
DIALOGSum
08.08
21.88
Pythia-Ib
ULD
DIALOGSum
28.48
12.16
Table Details performance of Teacher /aaeeerrttuttttt pair models trained with ULD Loss and teacher-generated text (Raw Text) for the Summary task. Evaluations are performed over respective test splits.
C.2 Extractive QA
Llama
Bloome-560m
ULD Loos
SQuAD
75.9
77 2.37
77.85
Llama
Blooms-580m
Raw Text
QED
60.99
58.00 ,
62.38
Mistral
Mooms-550m
Raw Text
SQaAD
73.34
73 33:31
78.52
Mistral
Blooms-360m
ULD Loos
SQuAD
76.0
76.1 1
81.1
Mastral
Raw
Mittral
Blaoms-300m
ULD
QRD
55.78
61. 88
58.8
Llamm
GPT-350m:
Raw Teat
3QuAD
70.79
72 02
72.78
Llama
OPT-3550
ULD
SQuAD
74
74.99
OPT-350m.
Raw Text
QED
61.84
Llama
OPT-350m
ULD Loss
QED
49.06
55. 31
51.74
Mistral
OPT-350m
Raw Text
SQaAD
71.64
77.25
Mistral
OPT-350m.
SQUAD
78.
75.91
Mistral
OPT-3t0m
Raw Text
QED
60.13
55. 36
54.55
Mistral
GPT-350e
ULD Loss
QRD
56
54.53
Pylkis-410u
Raw Text
SQUAD
78.
72.85
Llama
Pyshin4400m
ULD Loas
3QsAD
74.14
75. .88
76.51
Llama
Pythoa-f1mm
Raw Text
QRD
47.00
59.
48.87
Pyshis-410m
QED
Mistral
Pyshin-410m
Raw Text
SQUAD
71.5
71. 38
77.64
Mustral
Pytho--11mm
ULD Lose
3QnAD
73.64
73
79.71
Mustra
Pyshin-410m
Mutrall
Pyshbs-410m
ULD
QRD
50.38
51.19
56.62
Llamn
Pystia-160m
Raw Text
9QuAD
02.53
53 68
06.67
Llama
Pyshiis-66mm
3Q+AD
68.19
Ldams
Pyshts-180m
Raw Text
QRD
21:11
21.
34.45
Llama
Pyshia-160m
ULD Loss
QED
27.48
30 3
33.5
Llama
Pyskin-1b
Raw Text
3QuAD
77.36
Pyshis-1b
ULD
SQaAD
77.1
78.37
79.55
Llama
Pyshia-1b
Raw Text
QBD
45.58
51 00
60:41
Llama
Pyshin-lb
ULD
QED
Table 7: Details performance of Teacher /Studee pair models trained with ULD Loss and Seacher-generaterr text (Raw Text) for Extractive QA task. Evaluations are performed over respec- tive test splits.
22

---

Published in Transactions on Machine Learning Research (01/2025)
C.3 Generative QA
Dataset
Bloome-5t0m
Raw
PairytaleQA
36.7
Llama
Blooms-58om
ULD Loss
EatrytaleQA
37.88
46 913
29.36
Llamn
Blooms-560m
Raw Text
PubMedQA
28.86
Bloome-5t0m
PubMesqA
Mastral
Blooms-580m
Raw Texe
PatrytaleQA
32.64
59. 46
26.09
Mistral
Blooms-560m
ULD
PatrytaleQA
Blooms-300m
Raw
PubMedQA
Mistral
Bloome-500m
ULD Loos
PubMedqA
30.6
32
29.13
Llama
OPT-5h0m
Raw Text
FairytaleQA
35.78
41.40
26.57
Llama
OPT-350m
Raw Text
PubMedQA
27.90
28 .66
27.56
Llama
OPT-350m
ULD Lows
PubMedqA
30.01
36
24.14
Mastral
Mistral
GPT-350m
ULD
PalrytaleQA
30.44
37
Mistral
OPT-350m
Raw Teat
PulMeeiQ
27.91
27. 06
26.75
Mastral
PubMedQA
Pyshis-410m
Raw Text
PatrytaleQA
33.02
Llama
Pysliio410m
ULD Loos
PuirytaleQA
34.53
42. .1
27.49
Llama
PubMedQA
Pyshis-f10m
PubMedQA
29.89
Mistral
Pychm-410m
Raw Text
PatrytaleQA
31.44
57. 97
25.18
Mistral
Pyshia-410m
ULD Loss
PalrytaleQA
Pyskio-410m
Raw
PubMedqA
Mistral
Pychm-410mm
ULD Loss
PubMedQA
29.55
50 00
29.01
Ilama
Pyshta-160m
Raw Text
PatrytaleQA
14.5
Pyshia-100m
PairytaleQA
Llama
Pyshin-160m
Raw Text
PubMedQA
26.54
28 26
26.85
Llama
Pyshos-180m
ULD Lowe
PuhMedQA
36.
23.38
Pyshia-1b
PairytaleQA
Llama
Pyshin-1b
ULD Loss
PuirytaleQA
37.34
48 93
28.33
Llama
Pyshta-1b
Raw Text
PubMedQA
30.12
31.
26.64
Pyshta-1b
PubMedQQA
Table 8: Details performance of Teacher /Student pair models trained with ULD Loss and teacher-generated text (Raw Text) for generative tasks. Evaluations performed over respective test splits.
23

---

Published in Transactions on Machine Learning Research (01/2025)
D Appendix Native Performances
Base models used as teacher and student can be respectively download on HuggingFace: LLama 2 7b Chat Mistral 7b Instruc Pythia 160m, Pythia 410m, Pythia 11 Bloonz 560m, MTO 580m
D.1 Summary
Model
Dataset
Nomber Shot
Few-
Pew-Sbot sled
Th
Rouge=1
Rouge-22
Rouge-L
Rouge- Lsam
Bloomz-660m
DIALOGS um
Faloc
15.35
1.47
11:92
11.9
OPT-350m
DIALOG3um
Falen
22.05
3.31
17.88
17.84
P10
Pythio-18 600m
DIALOGSSum
Falsc
16.0
4.72
13.58
18.88
Pythi--ib
DIALOGÉum
Palen
35.95
11.80
29.08
29.1
Mistral
DIALOGSum
Palse
0.43
0.18
0.35
0.36
Table 9: Native performance details of Teacher/Studeen pair models benchmark in few-shot setting for the Summary task. Evaluations are performed over respective test splits.
D.2
Extractive QA
Modal
Dataset
Number Fow-Shot
Pow-Shot Ticled
F1
Prectatoo
Recall
Bloomz-660m
SQuAD
Falae
66.06
68.0
06.09
Bloome-5600m
QRD
Palse
41.01
51.
38.83
OPT-330m
SQaAD
DPT-3350
QRD
Palen
30.21
32
37.6
Pythin-410m
SQuAD
Faloc
37.4
36 08
47.65
Palse
Pyrhea- 80m
SQUAD
Palze
16.
18.83
100m
QED
False
15:48
20:15
17.31
SQuAD
Palue
65.55
Pythoa-1b
QED
Palze
11:72
47
45.75
Llama
SQUAD
Falee
0.81
0.83
0.84
Llama
QED
False
Musral
SQaAD
True
0.78
Mistral
QRD
True
0.53
0.35
0.
Table 10: Native performance details of Caccher/Student pair models benchmark in few-shot setting for extractive QA tasks. Evaluations are performed over respective test splits.
D.3 Generative QA
Modell
Datueet
Number Fow-Shot
Pow-Shot Tidee
PBERT
RBERT
Blooune-660a
Palse
23.67
Bleomx-800m
PubMedQA
-20.3
-9.43
-30.97
DPTT-350m
ExtrytaleQA
Palze
3.82
-4.33
13.1
OPT-350m
PubMediQA
Palse
16.94
4ythia-410m
FuirytaleAA
Falue
6.70
Pythca-410m
PubMadQA
Falze
25.65
50 13
21.42
Pythia-1 60m
PairytaleQA
Polee
+6.57
Pythin- 100m
PubMedQA
Falue
Pythin-1b
FairytaleQA
Palac
20.59
22 44
19.23
Pythin-1b
PubMedAA
Falac
26.13
29.25 29
23.24
Liams
ExtrytaleQA
Palze
Mamma
PubMedQA
Palee
0.31
0.3
Motral
FuirytaleQA
True
0.41
0.38
0.45
Mustral
PubMadQA
Palee
Table 11: Native performance details of Teacher/Studen pair models benchmark in few-shot
setting for generative QA tasks. Evaluations performed over respective test splits.
httpap/s//agaaaaa...oooootttttt2 http:://agginglaeecco//tuut https//hhgging/ccccoo//iigggii
24

---

Published in Transactions on Machine Learning Research (01/2025)
E
Appendix Few-Shot examples and Prompt Systems
The few-shot technique was used to generate synthetic data with the teacher. The number of few-shots reported for evaluating teacher models in App- D are the same numbers used to generate the synthetic answers. It's also important to note that the few-shot method was only used to determine the native performance of the teacher and student, not the distilled versions.
E.1 Prompt Systems
List of prompt system used with teacher templates. The default templates for chat models provided with the huggingface tokenizer have been retained:
Extractive QA:
You are an agent answering questions part of reading comprehension activ- ity. You must read and understand the context text step by step, Answers are brief and consist exclusively of continuous words taken from the context text provided.
Generative QA: You must read and understand the contextual text step by step, then answer the question. The answer must be brief.
expert agent in reading comprehension (question answering). You
Summary: You're an expert at summarizing dialogues. You have read the dialogue between two people and summarize it in more than sentence. The summary should be as short possible, not re-explaining the dialogue in detail and using the person's name when implicitly mentioned.
E.2 Few-Shot examples
Context
Question
Answer
Chriatine's boyfrienn
Patrick Harris (Tim DeKay), Christine's new boyfriend, who meeta video store and stazts dating.
Who played patrick new odventures christiue?
DeKay
June 14, 2018: Dwach Row In- mutes
As of June 14, 2018, there were 2,718 death row inmates in the United States.
Total number death row tamates in the us?
2,718
muntem
Com-
communiiam Marxiem, ideology concelved noted sociologiet Karl Marx during the mid ninetccatà cextery.
modero
Who with Idea commontimm?
Karl Marx
French armyy armies Seventh Coalition
commaud Napolcon Bonapacic
Nepoleon's 's
commanded
Gebbard Wellington, Blücher,
battle waterloo?
Wellinton
Wablstatt.
Cauine charao- ter
Axtro canine character on Harna-Barbera exrtoon, The Jatsone.
What wes the dog's
name on the jossone??
Table 12: Few-shot examples for extractive QA used to benchmark models and generate synthetic answers from teachers.
25

---

Published in Transactions on Machine Learning Research (01/2025)
Context
Summory
PPrrrn Igg John, shall we go to Sun Store" have decided to bay that Murrberry bandbag, Anyway, I'm not carrying this one Mary's a weiding. #Pemon2dh: But, Jane, wky not rent one with Handbax Hire? lnstead 990, pay 50, and you have it for whole week. # Persall: Sounds great, but never knew I cun reat handbax, # Parsse2 Handbax ago.
Jane wants to buy Murrberry handbas carry Mary's ding,
busenase. Murrerry, 1500 Marrbery buy bag
many designer handhage @rarzenn::: Absolutaly. by rons7 online orders. @ Persson I'll Estill have one more question. Mary's as wedding is next Saturday. There are only five daya left. Do
twenty
suggeets Hire tells her about the service in de- tuil. Jane is pleased have
conditionn For example, telephone? Or In person? #Peroon2d: Bisher. And it online have euough time? PPProo22f: Don't worry. It promiac, that cas coutomero receive their orders by within two daya. Thiee more days to #Pesson di Ol, I'd bester order one right now,
8750. &Prrsoal Great, but
convenlently,
Perroon The cummere are great here! holday for Shall cooling breeses,
air, the greenery: really. had Bremen instead. Tomorrow
PPresnn @ Per- pond. Personl #rorrnn2 had plansed to Hamburg tonight. but they decide to stay in Bremen since they are not in a ruch.
Pe while? PPesonndd Let's do baskd! Are
the pond
thought we'd stay Hamburg tonight. bet can't unioe ruch Lec's can have lanch in Hamburg, then theck into bostel in Copenhapen and have dinner there. @Perseni de Sounds fine me. Whatever, let's enjoy this pend first. PPersen2d:: Sare. We can walk around to that path that leads into the woods there, Hey, look! There are some wild ducks over there in the reeds, A PPersonl d I see them! Wow! How do you know they're wild?? Per I used to go hunting with my unele, that's how. a #Person 1#: They're 're neat. Now lat's take that path into the woods and son what we can sen...
@Pemonig:: Well, Rehecca, there anything have covered card with mobile number. if other questions spring
you
for now? #Pezzon2s:: don't think va. for Personl
Parsons giver business after she interviow and tells Rebreca the doci- sion will be mnne by early wodt week and Mise Childe will contact Rebaecs.
Parcons, think
well listen, here Of course,
busiuess
don't healtate to contact expect to heat from you? Parsonii:; Well,
also call Mis Childs too. #Prroo2dd: Great. Rmm, when will certainly have decision made by carly next week. Miss Monday or Tuesday, How does that sound? #Persongl That sounds perfect. T Thank you very much for taking the time to spaak to me Mr. Parsone, #Pazson #: The pleaeuee' is all mine, Rebaccs. # Perrsonng:: hope to bear from you very soon. #Personi #i Absolutely. Thanke for coming Robsces, Goodbye.
are Euishing the shertiiut interviews lomorrow. Childs will call you to discus
Table 13: Few-shot examples for summary used to benchmark models and generate synthetic summary from leachers.
Title
Context
Queetion
A Anwwer
So, she jumped up with her lint and her lins carée, and the tatlor jumped up with his great shears, and one apprentice grasped the line messure, while snsther took up the suser dorlged
Dodged them round and
pins; and they all tried eatck got apprentions for
bannock.
bannack eecape the tailor's and the thaee lore?
Wes
rafely tried suip stopped
nock
safely of the door road.
him, Lowever, and
bock the house, whiile by the roadside. trundled in the door, and there weaver aitsing his loom, with his wife beside him, winding cluc
is
Then Look prince by the houd, him deep down n
carih altogether forged the earsh beside his ailver to the ground: The wild ride and your luckl the meantime will tend your oxenn The prince did not walt to be told sccond time; but put his belane and armor in all haste, secuzely buckled on his spurs, bunng his sword at bis side, and fols as light in his silver armor as a bird in the air. Then he leaped iato the saddle so that every claup and bucklle rans, laid his reim on the neek of his steed, and rode hastily toward the glass mountoin.
cave, sliver, saddled
hong
bright bridled,
Primcesz Moustain
Glare
hoofs, and champing his bit sill the foom
man said: 'now quickly into
What was the suit of given the wild forged from?
The clearest all-
well that had enough inside him buy all the other toys, and this gave him
He
thought of this fact also, although they did not express many other shings to talk about. A large doll, still handhome, shough racher ald, for her neck had been mended, bay maxdde one of the drawers which wae partly open. She called out to the others, 'et ue have a game at being men and woulnn, that is something worth playing at.'
good opinion of his
value. The for there
Why didn's the other toye salk about how valushie the pig woo?
There so many other things so talk about.
Money Bax
When oon fonfuciu the prince of all four-footed animalz, and only man on carth, sought the child and spat out jude whereon was writiem: 'son of the walercryatal you are destined to become and confuciun grew up, studied diligenly, lnaened windom and came to be a anint. He did much good on sarth, and ever since his death has been reverenced na the greatent of tenchers and masters. He had forcksewledde of many things and even after he had died, he gave evidienoo of this.
to the earth, the killin that strange besst which
when there great
Who was confuciuz's 's douth zeverenced as the grentest of teachere and mastere?
A Legond of Confucius
uncrowued king!"
He did much
good on earsh.
"Ob, let inl Let me in! I'm cold, and I'm wet!! Exciaimed suddenty the door and knocking for admistance, while the rain pouzed down, and the wind made all the windows ratsle. 'Poor thing!" said the old post, as he went to open the door. there stand little bay, quite naked, and the water ran down from his long golden hatr. He teombled with cold, and had he not counc inso warm room be would most certainly have perished in the brightfll lempest.
child that stood crying
Nanghty Bay
Why did the bay ask to come inside?
He wao cold and wet.
Table 14: Few-shot examples for generative QA used to benchmark models and generate synthetic answers from teachers.
26

---

Published in Transactions on Machine Learning Research (01/2025)
Question
Answer
Injury severity score (ISS), Glangow coma soore (GCS), and revised trauma soore (RTS) ars the most frequently used mathode to evaluate mnte the severtry of injury in blunt craums patients. 185 is tea complicated to nesess maily and CCS and RTS are easy to wses but somrwhal sukjeetiiee While bloud cell count (WBC) is an easy, quick nod objective less. This study was perfo performee to trauma
The leslosyle count at pre- sensation can be used as au adjunct the acverity of injury blunt trauma patients.
Docs the leukocyte couns correlate with severity injury
count preseatation in the trauma patients, who admitied versity Medical Center Emergency Depariment between 04.2000-31 12.220000 vel positively 188,
evalunte significance of the pasients, 713
evaluation
The sim of this study was to naee the dingnostic value of arsicular sounda, standardized alinical examination, and standardised arsicalan ultrasored the detection insernal derangements ticats eyyaptomatic volunteers undewent standardised interviow, plysical exomination, ity, speatficity, predietive were caleulated using magnetic resonance Based often hneffliieet detect dies displacement. highoreenlution ultrasound demonetrater considerably lower sensitivity when compared with magnetic resenance. Same the technical difficulties resulted from a limited access because the presence of sucrounding bose struciures.
camparomandihull joint. Forty pa-
dynamic articulor ulleasound. Sousitiv-
temporo-
derange-
present study sup- pors recommendation diag-
Joins:
articular
still
of temporomaandibs
static and dynamic
lar Joint.
personsel
Army
pereietently
had undertalea Service, collecting dental coclodemographii from ruceutt to each Service and examine the null hypotheis that dif- farences dental health exiseed. Dinguostic criteria were developed, snmple sise calonlated and data collacted at the mitial training setahlishments of each Service. Dats for 432 participants were entered into the analysis. ftecruite in the Army zample had a significanlly greater prevalence of dental dacay and greater treatment resouree need thon cither of the other two Services. Army recroits had mean number
Navy
Force.
aimed rellects
healtt
signtfeann difference den- sal health between recraite in the dental bealth of new recratts so the British Armed Farcas?
Is shere differential
each Service does exiat and is likely to be roflection of the sociedemosrrahit back- ground from which they are drawn.
59 08, 00) decayed teeth per recruit, com- (0.98, 1.58 from Index Multiple Depeivation compured
pazed 1.53 (1 49, 2.59 pcc..01) Royal Navy recruita pcc0,001) Royal Furee recruita. Among Army recruits most deprived quiatiles
Table 15: Few-shot examples for generative QA used to benchmark models and generate synthetic answers from teachers for medical topic.
27

---

Published in Transactions on Machine Learning Research (01/2025)
F
Appendix Non--niform transport costs Systems
We study how the uniform transport cost affects ULD Loss by comparing distillation results with and without it, as shown in Tab. 16] To emphasize the computational cost of the Wasserstein distance with non-uniform transport costs, which scales as O(n log with n being the size of the larger support, we use teacher and student models with both small and large vocabulary sizes. For small vocabularios, we select L.aMA (32, 000 tokens) as the teacher and Pythia-410 (50, 304 tokens) as the student. For large vocabularies, we pair LLaMA with Bloomz-560m (250, 880 tokens). We use 5k samples of the SQuAD dataset due to its short answers, averaging 5.52 tokens for LLaMA, 3.48 for Pythia, and 3.37 for Bloomz. For non-uniform transport costs, the Levenshtein distance and the L2 norm between embodding tokens are used to define the cost of transporting mass between points in the transport plan.
The Levenshtein distance (Levenshtein) 1966) measures how different two strings or tokens are. This distance is the smallest number of operations needed to turn one string into another. These operations are: inserting a character, deleting a character, or substituting character, each increasing the cost of transformation by 1. For example, converting "cat" to "cats" needs one insertion, giving distance of 1.
Before distillation training, we calculated the distance between all teacher and student tokens, resulting in matrix C' of size (20*) with [S2|] the teacher vocabulary size and [(2*]) the student vocabulary size. Each element Cig = C5a represents the transport cost between points i and j in the transport plan. This matrix was used as a cost dictionary during training in the Wasserstein distance computation Eq. 3
Embedding token (Bojanowski al.) 2017) a process that maps tokens (such words or subwords) into continuous vector representations in high-dimensional space. In this space, tokens with similar meanings contexts are positioned closer together, reflecting their semantic similarity. We use the L2 norm between the teacher and student token embeddings to compute the cost matrix. However, since the teacher and student models do not share a common embedding space, we employ the widely used Fest subword embeddings as a shared space. To achieve this, each token from the teacher and student models is mapped to its nearest token in the Fast vocabulary based on the Levenshtein distance. The corresponding Fast embedding is then assigned to the respective teacher or student token. This approach allows us to compute a matrix of L2 norm distances between all teacher and student token embedding representations. This distance matrix is then used a cost dictionary during training in the Wasserstein distance computation.
To compute the Wasserstein distance during training, we used the POT library (Flamary ct al. 2021 Ex- tensive GPU memory management including, Offloading Aplimizer/Graddine state states to CPU, activation offloading, and gradient accumulation, were employed to address the high memory demands calculating transport plans and cost matrices. We highlight, that these experiments were conducted on A100 GPUs with 80GB of memory, using SQuAD dataset with an average label length of 4.12 tokens. Longer sentences were infeasible due to severe memory constraints.
Distillation using ULD Loss for LLama teacher to a Pythia-410m and Bloomz-560m student with and without uniform transport cost.
Table 16:
Student
Transport Cost
F1 score
Training time
Pythia
Uniform
69.56
64 minutes (x1)
Pythia
Levenshtein distance
69.32
172 minutes (x2.68)
Pythia
Embedding L2 norm
69.61
203 minutes (x3.17)
Bloomz
Uniform
72.11
59 minutes (x1)
Bloomz
Levenshtein distance
72.08
221 minutes (x3.74)
Bloomz
Embedding L2 norm
72.37
265 minutes (x4.49)
For all experiments, in our limited setting, we do not observe consistent improvement between models trained with ULD Loss and uniform transport costs defined by the Levenshtein distance or Embedding L2
28

---

Published in Transactions on Machine Learning Research (01/2025)
Norm. These methods were however slower, requiring up to 4.49x the time of uniform cost training, par- ticularly as the model vocabulary size increased, such as when distilling Llama into Bloomz. Furthermore, training with non-uniform transport costs introduced additional challenges, including pre-training steps and intensive memory management to avoid GPU "out of memory" errors. These findings emphasize the practi- cality of using uniform costs for maintaining training efficiency while achieving comparable performance. We encourage future work explore novel cost matrices that balance computational complexity with potential gains in model performance.
29

---

Published in Transactions on Machine Learning Research (01/2025)
G Appendix Training Information
During training, all distillation processes were performed over epochs with batch size of 8 for the SQuAD dataset and 4 for the others. A one-cycle learning rate scheduler was used with the following configuration for decoder models: max Ir = 1e - 6, initial = max 1r/2, min Ir = initial 1r/5. For mt0 (encoder- decoder), the max learning rate parameter varied according to datasets: DIALOGSum: 1e-4, PubMedQQ: 3c-4, and QED: 7c-6. Finally, distillation was performed BFLOAT16 mode introduced by Kalammarr al.
(2019h, 4*NVIDIA A A100-SXM4-SOGB with the Fully Sharded Data Parallel (FSDP) technique Zhao
ct al. 2023b
In total 4.923 GPU hours were used (i.e. consumption for the entire project in tonnes of CO2:
0.268).
30


Reviews:
[{"review_id": "2dg3y6kdxL", "reviewer": "Reviewer_BZ5v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** \n\nULD loss  can bridge the gap between teacher and student models with inconsistent architectures and tokenizers.\n\n**Weakness:**\n\nHow would the ULD loss compare to logit distillation in eq (1) if the tokenizers of the teacher and student models were consistent? Can we assume the student model is trained from scratch?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an effective distillation mechanism that can bridge the gap between teacher and student models with inconsistent architectures and tokenizers. The core of this mechanism is the ULD loss, derived from the optimal transport closed-form solution. The paper's experiments are comprehensive, and the implementation results support the motivation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Perhaps some experiments comparing it with logit distillation could be added? As I mentioned above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L4TlF85rq4", "reviewer": "Reviewer_gWp9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper handles an important problem, which is knowledge distillation for LLMs to which only the output logits are available.\n- The proposed loss function is theoretically inspired, because it is based on a notion from optimal transport.\n- The performances in the experiments look promising, and indeed verify the consistent advantage of the proposed method over distillation using teacher-generated text.\n- The paper is well written. The studied problem if nicely motivated, and the contributions of the paper are clearly discussed.\n\nWeaknesses/Questions:\n- I think it would be helpful to add some high-level intuitions about how equation (5) is derived. No need for the technical derivations, some intuitions would suffice.\n- Section 3.3, the paragraph \"Intuitions\": If I understand correctly, the proposed distance in equation (5) does not account for the meaning of the tokens. For example, even if two tokens from the teacher LLM and student LLM represent the same word, this information will not be used by the proposed method. I think this approach may miss important information which is useful for measuring the similarity between the predictive distributions of two LLMs.\n- This is related to the point above, at the top of page 6, it is mentioned that \"no vocabulary relationship is established\", so each transport cost is set to 1. I wonder is it possible to establish relationship between a subset of common vocabularies for the teacher and student? Since I guess there are usually some overlaps between the token spaces of a pair of LLMs, as shown in Fig. 1. This may further improve the performance.\n- A limitation of the proposed method is that it requires access to the output logit of the LLM, which makes it not applicable to the currently most powerful LLMs such as ChatGPT and Gemini.\n- Figure 4, right plot: why aren't the curves monotonically increasing with the dataset size?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of knowledge distillation for LLMs. The paper proposes a novel loss function based on optimal transport, to measure the similarity between the predictive distributions of the teacher and student models. The proposed method is able to handle scenarios where the token spaces of the teacher and student LLMs are different. Extensive ablation studies and experiments demonstrate the strong performance of the proposed distillation method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the points listed under \"Weaknesses/Questions\" above.", "broader_impact_concerns": "The broader impact is discussed at the end of the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "goAwqexqwl", "reviewer": "Reviewer_cjLP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The paper tackles an important problem without an obvious solution: logit-based distillation has more signal but requires the same tokenizer, while text-based distillation is more general but has less signal.\n2. The method is exceedingly simple, enabling easy adoption within existing frameworks with near zero extra overhead\n3. The paper rigorously validates the method against reasonable baselines, establishing that the method works\n\nWeaknesses\n\n1. Though I can see that the idea was motivated by optimal transport, the assumptions that make the problem tractable remove the need for standard considerations in optimal transport. The final method simply makes the sorted logprobs as close to each other as possible. As the paper acknowledges, this is not necessarily principled, but is redeemed by its strong empirical performance. \n2. I am slightly confused by Figure 5: If we use early stopping on say the rightmost figure, can't we achieve better validation loss using raw text? Is the remark following this figure clarifying that the validation loss has the regularization term only for ULD loss?\n\nOverall, this is a good paper that demonstrates the empirical success of a method designed to solve a previously unconsidered problem.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Existing distillation methods either train on teacher-generated text or teacher-generated logits (only if the teacher and student have the same token space). This work addresses this shortcoming by proposing a generic loss that aligns students with teachers even when the tokenizers are different by aligning the sorted logprobs of both model's next token distribution. This works better than training on teacher-generated data for fine-tuning based distillation setups.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please use \\citet and \\citep appropriately when citing papers reflecting whether citations should go in parentheses. \n- Isn't the complexity of your distance n log n (not n) because of the sorting?\n- [The following experiment suggestion is not required] To evaluate how good the assumptions are in practice, one ablation could be to do distillation between a teacher and student models with the same tokenizer. Then, you could evaluate the \"oracle\" best performance of standard distillation and ULD which is oblivious of the token relationships. The gap between oracle, ULD, raw text, and no training could indicate how much best case performance is recovered by ULD.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This papers studies how to perform distillation with teacher/student models that may be unrelated, e.g., have entirely different model architectures or even tokenizers. This amounts to replacing a standard component in the distillation loss with a version that can accommodate these differences. Specifically, rather than minimizing the KL divergence between the output distributions of the teacher/student models, the authors use a Wasserstein distance term. This is natural but comes with additional complexity, which the authors seek to address through a simple assumption that leads to a clean and efficiently-computed form for the term. 

The reviewers are generally positive—this is a reasonable idea with good execution. It is on track for acceptance.

There are a couple of places that the authors could improve the draft:

- Additional experimental evidence. One of the authors’ core ideas is to require that the transport cost is equal to 1, which leads to much higher computational efficiency. The authors note that despite the assumption, this still produces strong results (and verify this experimentally). However, it would be useful to know how much is sacrificed here—how strong would the results have been without this assumption? That would produce a (computationally infeasible) upper bound. If there is indeed a large gap, it would imply that we should continue searching for more efficient ways to do OT without constraining it. 

I would suggest that the authors do a simple experiment where they use two synthetic token sets, varying their size (to keep these sufficiently small), run distillation with full OT and their own ULD approach and compare the results. 

- As spotted by the reviewers, there are a number of typos, so before camera ready submission, I encourage the authors to do a careful pass.
Decision: Accept (poster)