Knowledge Distillation of Uncertainty using Deep Latent Factor Model
Authors: Sehyun Park
Jongjin Lee
Yunseop Shin
Ilsang Ohn
Yongdai Kim
Venue: NeurIPS 2025 poster ()
Keywords: Knowledge Distillation
Deep Ensemble
Deep Latent Factor
Gaussian Process
Primary area: deep_learning

Abstract: Deep ensembles deliver state-of-the-art, reliable uncertainty quantification, but their heavy computational and memory requirements hinder their practical deployments to real applications such as on-device AI. Knowledge distillation compresses an ensemble into small student models, but existing techniques struggle to preserve uncertainty partly because reducing the size of DNNs typically results in variation reduction. To resolve this limitation, we introduce a new method of distribution distillation (i.e. compressing a teacher ensemble into a student distribution instead of a student ensemble) called Gaussian distillation, which estimates the distribution of a teacher ensemble through a special Gaussian process called the deep latent factor model (DLF) by treating each member of the teacher ensemble as a realization of a certain stochastic process. The mean and covariance functions in the DLF model are estimated stably by using the expectation-maximization (EM) algorithm. By using multiple benchmark datasets, we demonstrate that the proposed Gaussian distillation outperforms existing baselines. In addition, we illustrate that Gaussian distillation works well for fine-tuning of language models and distribution shift problems.

Knowledge Distillation of Uncertainty using Deep Latent Factor Model
Sehyun Park Department of Statistics Seoul National University ws__hen@smu.cc ac.kr kr
Jongjin Lee Samsung Research ga0408@snu, ac ac. .r
Yunseop Shin Department of Statistics Seoul National University dbst jo8ssmu ac kr
Ilsang Ohn
Yongdai Kim Department Statistics Seoul National University ydkim09038gmai11 com
Department of Statistics
Inha University ilsang. ohn@inha ac. kr
Abstract
Deep ensembles deliver state-of-the-ant reliable uncertainty quantification, but their heavy computational and memory requirements hinder their practical deployments to real applications such as on-device AL Knowledge distillation compresses an ensemble into small student models, but existing techniques struggle to preserve uncertainty partly because reducing the size of DNNs typically results variation reduction. To resolve this limitation, distillation (i.e. compressing a teacher ensemble into a student distribution instead of a student ensemble) called Gaussian distillation, which estimates the distribution teacher ensemble through special Gaussian process called the deep latent factor model (DLF) by treating each member of the teacher ensemble a realization of a certain stochastic process. The mean and covariance functions in the DLF model estimated stably by using the expectation maximization (EM) algorithm. By using multiple benchmark datasets, we demonstrate that the proposed Gaussian distillation outperforms existing baselines. In addition, we illustrate that Gaussian distillation works well for fine-tuning of language models and distribution shift problems.
introduce
method of distribution
Introduction
While DNNs have succeeded tremendously in various Al tasks, the rapid increase in their model sizes has raised concern about high computational resource demands, which limits their applications real world applications such as on-device AI [ and thus developers have increasingly compressed large-scale language models into much smaller models 2 3 4 representative tool for compres- sion is knowledge distillation (KD), which constructs smaller DNN that mimics given large-scale DNN [5].
Another concern that the inherent over parameterizaaiinn a single DNN makes them susceptible to overfitting, leading overconfident predictions. When training predictive models, it is essential to learn models not only accurate but also reliable. For reliable prediction, proper quantification of
uncertainty has become important topic in AI research 66] 7 8) Deep ensemble (an ensemble of multiple DNNs) has received much attention not only for its strong predictive performance but
Corresponding author.
TThe source code of DLF publicly available wttttpp://gtthhb..cm////uuu//uup
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

because of instability in also for its ability to quantify prediction uncertainty 10 An ensemble of DNNs can mitigate overconfident predictions by reflecting the uncertainty (i.e., the variation of multiple predictions made by members of an ensemble) when making final decision.
Since deep ensemble requires even more computational resources than DNNs, KD of deep ensemble is necessary for improving its applicability. Several works have focused on distilling a given teacher ensemble to student ensemble instead of distilling a teacher ensemble to single student DNN to keep the uncertainty as much as possible [8] 12 13 14) [3]. Algorithms for KD of deep ensemble
can be roughly categorized into two approaches: one-to- one distillation and distribution distillation. One-to-on one distillation compresses each member in a teacher ensemble to smaller DNN, which becomes member of student ensemble. Various weight-sharing architectures for student DNNs, along with their corresponding learning algorithms, have been proposed [T] [3] [4] [6]. On the other hand, distribution distillation treats each ensemble member in a teacher ensemble as an independent realization of a certain distribution whose parameters are modeled by student DNN. 17 and [18] assume that the conditional class probability vector of each member in a teacher ensemble follows Dirichlet distribution and devise method to estimate the parameters in the Dirichlet distribution using a student DNN.
There are still limitations in existing KD methods for deep ensemble. One-to-one distillation methods tend to lose sigmificant amount of uncertainty in teacher ensemble when they compress large DNNs into smaller ones, while performance of Dirichlet distillation (the distribution distillation with a Dirichlet distribution) is inferior one-to-one distillation partly learning the parameters in the Dirichlet distribution.
The aim of this paper is propose a new distribution distillation method that is numerically stable in learning and superior to other baselines in uncertainty quantification. In our proposed method, we treat each member in a teacher ensemble as an independent realization of Gaussian process and estimate the mean and covariance functions of the Gaussian process based on observed predictions of members in teacher ensemble. For this purpose, we propose the deep latent factor (DLF) model where the mean and covariance functions are modeled by student DNN and implement an EM algorithm estimate the maximum likelihood estimator (MLE) of the student DNN. We call our method Gaussian distillation.
Our contributions are summarized follows.
We propose new distribution distillation method based specially designed Gaussian process called the DLF model that achieves superior performance in uncertainty quantifica- tion to other baselines.
We develop an EM algorithm to estimate the student DNN in the DLF model. In particular, we propose way of finding a good initial solution by maximizing the penalized complete log-likelihood.
We do numerical experiments show that Gaussian distillation outperforms other baselines for both regression and classification. We also illustrate that Gaussian distribution is a useful tool for fine-tuning language models.
We apply the pre-trained DLF to distribution shift problems and show numerically that it outperforms baselines.
2 Preliminaries
2.1 Prediction uncertainty
In nutshell, quantifying prediction uncertainty in supervised tasks involves efficiently estimating the predictive distribution of the output y given a new input denoted as pey) The variation in the predictive distribution be used measure of uncertainty in prediction.
typical way of estimating the predictive distribution begins with a parametric generative model for the input and output pair. Let ply) 2z, 0)) be the conditional distribution of the output y E Y' given an input x E X' C Rd, where 0 E θ is an unknown parameter. Then, we try to estimate @ based on training data (æ1, F1) 01),..., ., (Xm, Fm) such that p(y (//,,,, 0) is as close as possible to p" (y) æ), where 0

---

is an estimate of f and p" (y/z) is the true conditional distribution. For example, the MLE minimizes the empirical KL divergence between ply æ 8) and p" (3/lz)
well known, however, that the variation ply|æ, f) is smaller than that in p" p"(y/æ: because p(y| xx 8) does not take into account the uncertainty in estimating 9. Thus, making a decision solely with p(y) az, 0) would lead in overconfident results. A proper uncertainty quantification in prediction should consider not only uncertainty in p" (3)lz) (alcatory) 66 7
but also uncertainty in & (epistemic).
popular way of considering both aleatory and epistemic uncertainties in prediction to use an ensemble. We construct multiple estimates ......, 6, of and then estimate the predictive distribution as p(y|=) = Σi= ply x, 0.)/n, which we call the averaged prediction model. For deep learning, the two most representative methods of constructing multiple estimates are deep ensemble 9 19 20
and Bayesian DNNs 21 22 23 24 25 26 27]. Deep ensemble generates multiple estimates by
learning a DNN with different imitial parameter, while Bayesian DNNs generate As from the posterior distribution. In this paper, we focus deep ensemble, but our proposed method can be applied to Bayesian DNNs without modification.
2.2 Review of ensemble distillation
As mentioned in Introduction, deep ensemble has an intrinsic limitation in its practical applications due to high computational costs and times along with demands for substantial memory to store and process multiple prediction models. To resolve this problem, KD of deep ensemble has received much attention [5] 8 12 3 4 [7] A basic idea of KD of deep ensemble is to approximate large DNNs in teacher ensemble by smaller student DNNs. A naive approach of KD of deep ensemble is to approximate the averaged prediction model p(y [z) of a teacher ensemble by a small single
DNN [1 28 29] This naive approach, however, does not perform well since it is hard to distill the uncertainty in p(yl (=) into a single student DNN.
A remedy is to distill teacher ensemble into student ensemble. Several methods have been proposed for this purpose, which can roughly divided into two categories that are explained in the subsequent subsections.
2.2.1 One-to-one distillation
The main idea of one-to-one distillation to construct multiple student models, each of which corresponds to each teacher model. That is for given many teacher models p (ylz), 1,..., n many student models Pi (yl az) are constructed. To save computation time and memory further, various special neural network architectures for n student models p (ulz) 1...., have been proposed. Examples are Hydra [II), Batch Ensembles (BE) and Latent Batch Ensemble (LBE) See Appendix A.I for details.
12 [4]
2.2.2 Distribution distillation
Distribution distillation assumes that teacher models are independent realizations of stochastic model with unknown parameters modeled by student DNN and estimates the student DNN based on the prediction values of the teacher models 17 18]. To be more specific, for classification problems,
we assume that (p." (v)x)) 1...., .), 1,..., n for given æ are independently generated from the Dirichlet distribution with parameters on (z())...... and model these parameters by student DNN. Once the student DNN is learned, ensemble members are generated from the learned
Dirichlet distribution and aggregated in the prediction phase. We call this method Dirichlet distillation. See Appendix A.2 for details.
3 The Proposed Method
We propose new method of distribution distillation. The main idea of the proposed method is that we treat members in teacher ensemble as independent realizations of a Gaussian process and estimate the mean and covariance functions of the Gaussian process by student DNN, Then, in the inference

---

phase, we generate ensemble members from the estimated Gaussian process. We call our proposed method Gaussian distillation. See Figure foor the overall process of Gaussian distillation.
CTeacher Ensemble Training
<Knowledge Distillation to DLF architecture)
(Inference on new
Prediction of
p((2x)
Teachers
Teachers
DLF
Sample 7--N
f
f
shared body
:
:
:
&++
DLF
shared body
Dataset
Dataset
New
x1
nEW
22
new
Figure 1: Overall process of Gaussian distillation
technical difficulty of this idea is model and estimate the covariance function. To resolve this problem, we use the DLF, which an extension of the standard linear factor model (30, where the mean and factor loading are modeled by a student DNN.
For the probabilistic model of data, we consider f(x) + where e ~ NO o²) for regression problems and p(y æ²) exp(ff E:= (x)(f((x:) for classification problems. Thus, teacher ensemble for regression problems consists multiple teacher models for f as well as multiple estimates of of, while teacher ensemble for classification problems consists of multiple teacher models for multivariate functions f(-) = (f). fc(-)).
3.1 Deep Latent Factor model
In this subsection, we introduce special Gaussian processes for f(-) and f(.), respectively,
Univariate case The DLF model for univariate random function f X E is defined as f(-) = (-)()) (-) + . T Z.
where (.) X + R is the mean function, lb(-) X Re is the factor loading function and Z N, T,) the latent factor. Here, N₁ is the q-dimensional Gaussian distribution and T, is the g-dimensional identity matrix. In the DLF model, we set (14(-), ( T) by a student DNN parameterized by l which has + output nodes.
It is easy see that the DLF a Gaussian process with mean function pa(+) and covariance function (: (.(-) (-)).. Once have many teacher models fi(.)...., fx(-), assume them to be independent realizations of the DLF model and estimate the mean and factor loading functions.
Multivariate case
The DLF model for multivariate function f(-) = (())))..,...))) is defined
as
f()) (.(()) LZ
(2)
where Ha (.) X' - R is the mean function, Tg(.) X - R2 is the factor loading function, Here, MA (0, T.. T,) is
T E Rexe lower-triangular matrix and matrix-variate Gaussian distribution. It can be shown that the DI.F model is a multivariate Gaussian process MGP.(pp Σ, A) with the mean function (.((-), covariance function )(.-) = (. (.)
MM (0,
and parameter matrix A LL T. For the definition of multivariate Gaussian process, see
3.2 Estimation of the mean and factor loading
The main idea of Gaussian distillation estimate the mean and factor loading functions by maxi- mizing the corresponding log- ligliikelhoodd assuming that teacher models are independent realizations

---

of the DLF. For optimization, we use the EM algorithm 332. In this section, we explain the EM algorithm for Gaussian distillation. For ease of notation, we only consider the univariate DLF model, and refer to Appendix B.2.2 for the multivariate DLF.
Suppose that many teacher models fi())...,, are given. Gaussian distillation consists of three steps. The first step is to choose m- design points Deesiga = { 3-... 2(d1) J. We will discuss how to choose the design points in Section 3.4 The second step is to calculate the vectors of
prediction values of each teacher model at the design points to have S; (f((zr) (d)) ).jj = -.....,,) m)
for i = 1,....,2 n. The final step is to estimate the parameter 0 in the DLF model assuming that fi())..... ...... (+(+) are independent realizations of random function following the DLF model. Since f,8 are independent Gaussian random vectors, the MLE can be obtained by use of the EM algorithm as follows.
To make the EM algorithm numerically stable, we consider the noisy DLF model which assumes that f. = f. +,, where v, ~ (0, (mmm and f. = (fi(z))...... )) with f(())s following the DLF model. Specifically, each f: is expressed [(()) = (-(() + Φb(-) TT,,, denotes the latent factor corresponding to the i-th function realization. Then, we obtain the MLE of the parameter 0 in the mean and factor loading functions as well as oz. We abuse the notation to write (6, o7:) unless there is any confusion.
u,
24, where 2; ( II)
The complete log-likelihood is given as
geom (flf 11n) = nm
log(2
nq
log (2T) log((2)
Ei=1 2, 2;
2
2
2
(3)
Σ (f+ 14 -Bozi) (f:
14 - fz0%))
20²
where S1n = (S1..... " 21m {z1,..., ( and b
= (
(1)
(2)( (... (2(11) )) is an m X q matrix.
For given parameter ((--1) at time t 1, the E-step is to calculate the conditional expectation of the complete ligllikel 1(010(t-1)) Ex @it. [ [ecom
1:n. f1n =1m)], whose formula is given in Appendix B In the M-step, we update g(t) by a stochastic gradient descent algorithm on mini-batches. The EM algorithm is summarized in Algorithm I Appendix B
Choice of the initial parameter Note that the EM algorithm may converge to a local optimum, and the choice of initial solution significantly impacts the final estimate. For the DLF model, where the factor loading involves a complex DNN structure, this issue becomes even worse. Moreover, the identifiability issue of the factor loading makes initializing the EM algorithm from a well-chosen starting point become even more crucial.
For searching good initial solution, pretrain the DLF model by maximizing the following penalized complete log- likeelhoo with respect to f and 2,8 where
geen (0, 21: If 1:n)) = goom (f] fin 1mm) - ADMMD <1..)
for À > 0, where z'.,. are samples generated from the standard Gaussian distribution and PXMD is the Maximum Mean Discrepancy (MMD) with the RBF kernel. The term MMD is introduced to make the distribution of the estimated Zi: similar to the standard Gaussian distribution.
KD for o² For regression problems, we need KD method for o². Let o2 22,1..., be estimators of of provided by each teacher model. We assume that o. i 1,..., n are independently generated from the inverse gamma distribution and estimate the parameter in the distribution accordingly. In the inference phase, we generate ensemble members of of from the estimated inverse gamma distribution.
3.3 Comparison with other baselines
Comparison with Hydra Recall that model (Hu()), TT by DNN with q many heads. Note that the DLF model assumes fe(-) (e(-) + ()(+) T for i 1, n. and thus
5

---

can interpret (po()), TT as the body and 2,8 as the model-specitfs weights at the head. In view of sharing the body, the DLF model is quite similar the conventional multi-head structure used in Hydra [T]]. The main difference is that the DLFF model treats 2,8 as random quantities and thus integrates out before estimating the MIE while Hydra treats 2,5 as fixed effects and estimates 0 and 2,8 simultaneously by minimizing given loss. It is well-known that treating random effects as fixed
effects is highly susceptible to bias [33] 34, 35 36). Our experimental results in Section amply demonstrate that treating 2,S random is better than treating 2,8 as fixed effects.
Comparison with Dirichlet distillation At least, there are two advantages of Gaussian distilla- tion compared to Dirichlet distillation. Gaussian distillation can be applied to both regression and classification models while Dirichlet distillation is only applicable to classification problems. The second advantage is that estimation of the mean and factor loading in the DLF model ís easier than estimation of the parameter in the Dirichlet distribution owing to the nice EM algorithm. This stability makes Gaussian distillation perform better than Dirichlet distillation. The inferior performance of Dirichlet distillation compared to one-to-one distillation, as reported in [T is confirmed by our experiments in Section 5
3.4 Choice of design points
and Let ja()) and ((.-) be the estimate of (-) and (.,-) by the DLF model with design points Delssiga
For given æ E X, let a and p be the distributions of (a) under the assumption that f-) is Gaussian process with the parameters (a, Σ) and (M+: Σ.), respectively. In Theorem G.3 in Appendix Ducsiga di (fm: p=,=)) converges to 0 as n. oc if we choose the architecture G.2. we prove that suPaeDaooie
of a student DNN for (-o(-), oo(.) )) appropriately, where d2 is the & metric.
For of Ddesign if fi and fc, well E and E. are (coordinate-wise) Lipschitz, it can be shown that di Pp.)). S d1 (Paray Pr.arr)) ) + C x1(1) for a positive constant C, where ((1) is the nearest point in Ddesign to æ. See Theorem G.3 in Appendix G.2 Note that the term || a is affected by the choice of design points. Suppose that x is realization of a random vector X IP. Then, the expected nearest- points are located in a higher-densit region of P. This observation suggests that design points similar to test data would be better. Validation data (dataset whose distribution is the same as the training data used for learning a teacher ensemble) would be a promising candidate for the design points. See Appendix D I.1.I for numerical experiments.
distance xx || X (1)l becomes smaller when the design
4 Application to distribution shift problems
The pre-trained DLF can be applied to distribution shift problems. We say that given new data is shifted in distribution if the distribution of new data is different from that of training data. Distribution shift problems, whose aim is to efficiently learn a prediction model new data when the size of new data is small, have been studied extensively (37 38 39 40
41 42 A popular method is to learn
DNN on training data first and retrain the head of the DNN on new data while the body is fixed
43 44
Note that the DLF model is given as
50) = P0)
+
(x(0) k=11=1
(5)
for j = 1, l,.....,, c, where ZykiS independent standard Gaussian random variables. For distribution shift problems, we can treat (f(-), (b-)T f.) as learned body and 2,s are the weights in the prediction head. Then we learn only the weights of the head on new data while fixing the body. In Section [5.] we show empirically that this method outperforms its competitors. The superior performance of the DLF model for distribution shift problems indicates that Gaussian distillation is good at not only uncertainty quantification but estimating the feature vector (f(-), (

---

5 Experiments
In this section, we investigate Gaussian distillation by analyzing multiple benchmark datasets, We compare Gaussian distillation with existing baselines including the naive distillation (one-to-one distillation without sharing weights between student DNNs, small-Ens), Hydra [ and BE 12 for regression and classification problems as well fine-tuning of language models in view of uncertainty quantification. For classification, we also evaluate Proxy-Diee Dirichlet Distillation (Proxy-End') [18] and Ensemble Distillation via Flow Matching (EDFM) [5]. In addition, we show that a traitre DLF outperforms its competitors for distribution shift problems.
5.1 Uncertainty quantification for regression and classification problems
5.1.1 Regression case
Datasets We analyze six benchmark datasets from the UCI repository [46] including Boston housing, Concrete, Energy, Wine, Power Plant, and Kin8nm. Each dataset is randomly split into 90% training and 10% testing, and teacher models are trained following the experimental protocol of [47]. We repeat this procedure 10 times to obtain 10 measures of the evaluation metrics of each methods and report the averages (with the standard errors). See Appendix C.T for details of implementation.
Results Table ! presents the results of the four evaluation metrics (see Appendix C.I.2 for the definitions) for performance and uncertainty quantification. DLF outperforms Hydra and BE in most cases. Even when not the best, DLF is at least the second best. The coverage probabilities of Hydra and BE are sometimes much lower than those of DLF (Boston housing and Concrete for Hydra, and Boston housing and kim8nm for BF), which suggests that deterministic distillation methods fail to fully preserve the uncertainty in a teacher ensemble. This observation is not surprising since variation of smaller models is in general smaller than that of larger models and weight sharing would reduce the variation further. Thus, way of adding additional uncertainty to student ensemble is needed, and distribution distillation is such a solution.
Table I: Results on UCI benchmark datasets. (+ closer to the coverage probability of teacher ensemble is better)
Datasets
Method
Boston housing
Concrete
Fnergy
Wine
Power
Kin8nm
Teachers
2.5786
5.6191
0.5692
0.5497
4,2197
0.0794
small-Ens
2.7280 (0.0184)
5.6952 (0.0494)
0.6367 (0.0182)
0,6002 (0.0083)
4,2430 (0.0864)
0.0865 (0.0007)
RMSE I
Hydra
2,8346 (0.0835)
6.0558 (0. 1366)
0.6549 (0.0366)
0.5689 (0.01 14)
4,2284 (0.0074)
0.0932 (0.0023)
BE
2,8375 (0.0729)
5.9777 (0.1475)
0.6661 (0.0354)
0.556 (0.0187)
4.2367 (0.00411)
0,0962 (0.0059)
DLF
2.6687 (0.1700)
5.6047 (0.1771)
0,5659 (0.0239)
0,5506 (0.0104)
4.2211 (0.0010)
0.0825 (0.0010)
Teachers
2,3850
3.1134
0,8533
0.7980
2,8586
-1.1109
small-Ens
2.4150 (0.0085)
3.1672 (0.0167)
0.9829 (0.0263)
0,8861 (0.0160)
2.8622 (0.0191)
-1.032 (0.0108)
NLL L
Hydra
2,4843 (0.0478)
3,2586 (0.0293)
0.9914 (0.0660)
0.8322 (0.0166)
2.8604 (0.0017)
-0,9434 (0.0307)
BE
2.4892 (0.0408)
3,2241 (0.0332)
0.9879 (0.0511)
0.8065 (0.0138)
2.8628 (0.0010)
-0.9115 (0.08008)
DLF
2.4346 (1.71
3.1584 (0.0463)
0.8525 (0.0471)
0.8230 (0.0199)
2,8591 (0.0010)
-1.0754 (0.0126)
Teachers
1.4425
2,9926
0,3137
0.2962
2,3360
0.0443
small-Fns
1.5233 (0.0087)
2.9953 (0.0304)
(0.0077)
0.3307 (0:0067)
2.3392 (0.0321)
0.0475 (0.0005)
CRPS
Hydra
1.6041 (0,0994)
3.3084 (0.0639)
0.3622 (0.0216)
0.3075 (0.0040)
2.3405 (0.0048)
(0.0012)
(0,0490)
(0.0894)
0.3617 (0.0168)
0.3020 (0.0059)
2.3483 (0.0024)
0.0532 (0.0035)
DLF
1.4317 (0.1029)
3.0622 (0.0954)
0,3163 (0.0119)
0.2980 (0.0043)
2.3364 (0.0010)
0.0458 (0.0005)
Teachers
0.9608
0,9515
1.0000
0.9750
0.9697
0,9610
95%
small-Ens
0,9408 (0.0016)
0.9431 (0.0041)
0.9921 (0.0011)
0.9569 (0.0042)
0.9669 (0.0012)
0.9649 (0.0016)
Covenge
Hydra
0.8995 (0.0109)
0.9097 (5.01
0.9948 (0.0091)
0,9594 (0.0053)
0.9782 (0.0003)
0.9407 (0.0046)
Probability
BE
0.9093 (0.0090)
0.9282 (0.0099)
0,9922 (0.0110)
0,9612 (0.0092)
0,9778 (0.0014)
0.9080 (0.0213)
DLF
0.9240 (0.0154)
0.9291 (0.0125)
1,0000 (0.0000)
0.9681 (0.0055)
0,9711 (0.0053)
0.9761 (0.0036)
5.1.2
Classification case
Datasets CIFAR-10 and CIFAR-100 consist of 50,000 training and 10,000 test images. In this experiment, the training data are further split into 80% training and 20% validation, and teacher models are trained on the training data and the number of epochs is determined by the validation
7

---

data. Implementation details of the distillation methods are given in Appendix C.2 Experiments are repeated with different random initializations for each method.
Results As shown in Table 2
DLF outperforms the other baselines consistently in terms of not only uncertainty quantification but also accuracy. In particular, improvements of DLF with respect to ECE noticeable. The definitions of the evaluation metrics are given in Appendix C.2.2
Table 2: Results on CIFAR-10 and CIFAR-100.
datasct
method
1
NLL
BCE(S)
Teachers
94.24
0.1539
0.9
small-Ens
92.87 (0.35)
0.2377 (0.0019)
(0.14)
Hydra
93.16 (0.06)
0.2660 (0.0063)
4.15
CIFAR-10
LBE
93.25 (0.26)
0.2480 (0.0053)
4.11 (0.10)
Proxy-EnD*
90.92 (0.28)
0.2861 (0.0027)
2.08 (0.19)
EDFM
90.62 (0.23)
0.2858 (0.0025)
2.78 (0.17)
DLF
93.40 (0.14)
0.2246 (0.0023)
2.79 (0.20)
Teachers
81.36
0.7167
1.41
small-Ens
79.29 (0.26)
1.0113 (0.0145)
12.90 (0.24)
Hydra
77.42 (0.15)
1,2912 (0.0272)
12.70 (0.37)
CIFAR-100
LBE
79.58 (0.40)
1.0110 (0.0087)
13.42 (0.42)
Proxy-EnD*
67.62 (0.22)
1,2355 (0.0151)
7,35 (0.25)
EDFM
64.17 (0.32)
1.6741 (0.0242)
11.35 0.47)
DLF
79.68 (0.23)
0.8974 (0.0042)
9.45 (0.31)
5.2 Application to fine-tuning of language models
In this section, we apply the proposed distillation framework to downstream binary classification tasks using pretrained language models. Given pre-trained teacher and student language models, fine-tuned teacher and student models are obtained using Low-Rank Adaptation (LoRA) 48 For
the teacher and student pre-trained language models, "RoBERTa" [49] and "DistilRoBERTa' [ are used. As a teacher ensemble, we obtain four fine-tuned models by combining LoRA and RoBERTa with randomly selected initializations for each task. Then, an ensemble of fine-tuned student language models constructed by applying LoRA to DistilRoBERTa for each distillation method.
We analyze three GLUE (50) and SuperGLUE [51] sub-tasks: RTE, MRPC, and WiC. All three datasets are binary classification tasks. Implementation details of the distillation methods given in Appendix
Datasets
As shown in Table Gaussian distillation outperforms Hydra and LBE with large margins. We conjecture that the performance gap of Gaussian distillation Hydra and LBE would become larger when the complexity gap between teacher and student models becomes larger. This is a reasonable conjecture since smaller models could preserve less variations in teacher models. Gaussian distillation would add additional variations to the student models through variations of the latent vector Z.
Results
5.3 Application to distribution shift problems
We compare DLF with two baselines which fine-tune only the head on new data while the body learned by either (1) a standard DNN or (2) applying Hydra on training data of CIFAR-10. For distribution-shiitedd new data, we swap the labels of CIFAR-10 as is done by [44). See implementation details in Appendix C.4 The results are given in Figure which amply show that DLF is superior. is interesting to see that DLF outperforms even when the sample size of the new data is large. which implics that the learned body by DLF is qualitatively different from those by DNN and Hydra. We do not know the reason but an implication is that Gaussian distillation is good at learning not only quantifying uncertainty but also learning the feature vector (i.e. the body).

---

Table 3; Results on GLUE and SuperGLUE benchmark datasets
dataset
method
Acc (%) f
NLL 4
ECE (%)
Teachers
75.09
0.8401
17.70
small-Ens
67.15 (0.0057)
0.6739 (0.0124)
13.09 (0.0148)
RTE
Hydra
62.82 (0.1650)
0,9034 1733)
23.31 (0.4907)
LBE
65.97 (0.1406)
0,9235 (0.0664)
25.63 (0. (9.1
DLF
67.06 (0.1040)
0.6658 (0.0762)
9.74 (0.5050)
Teachers
87.25
0.3435
4.77
small-Ens
83.092 (0.0172)
0.4596 (0.0396)
9.84 (0.0164)
MRPC
Hydra
82.19 (0.0357)
0.6429 (0.1274)
11.79 (0.0136)
LBE
82.23 (0.0153)
0.6527 (0,0544)
13.6 (0.0114)
DLF
83.094 (0.0035)
0.4526 (0.0113)
10.72 (0.0392)
Teachers
68.03
0.6395
9.14
small-Fns
65.02 (0.0062)
0.7674 (0.0206)
16.69 (0.01 16)
WiC
Hydra
65.05 (0.0210)
1.1628 (0.0355)
26.26 (0.0191)
LBE
65.36 (0.1209)
0.8809 (0.0950)
20.28 (0.0300)
DLF
66.18 (0.0146)
0.7706 (0.0543)
15.99 (0.0198)
Nogative Lischhood
Expcoted Collbration
030
088
2010
Number tuning
Figure 2: Comparison of performances of the three learning algorithms (DNN, Hydra and DLF) pretrained on CIFAR-10 and fine-tuned on CIFAR10-PH 10-Flip as the sample size of CIFAR 10-Flip data varies. The solid curves the means and the shaded bands the min-max spreads obtained from training models on 5 randomly selected new data of CIFARIO-Flip.
5.4 Ablation studies
In Appendix D we present the results of ablation studies including the sensitivity of Gaussian distillation to the choice of design points, the dimension of latent factor, the architecture size of student DNNs and the number of ensemble members used in teacher and student ensemble. In addition, we compare the results when the initial solution is randomly selected Gaussian distillation.
Conclusion
We proposed a novel method for distilling deep ensembles, specifically addressing the challenges associated with computational costs, inference time, and storage capacities inherent traditional deep ensemble approaches. The key innovation lies in modeling the covariance structure of deep ensembles through the DLF model, enabling efficient preservation uncertainty a teacher ensemble with significantly reduced inference costs.
There are several future research topies. First, in this paper, we only focused on deep ensembles. It would be valuable to consider Bayesian DNNs, as they provide framework for uncertainty quantification (25 26 27 and can potentially serve as a prior for on- on-device posterior updates. Second, for distillation of fine-tuned language model, we used DistilRoBERTa [I]. pretrained distilled language model. It would promising to distill the pretrained language model and the model for LoRA simultaneously. Third, the DI_F could be used for online Bayesian learning by approximating the posterior with respect to old data by the DLF and using it for the prior of new data, We will pursue this idea in a near future.

---

Acknowledgements
This work was partly supported by the National Research Foundation of Korea(NRF) grant funded by the Korea (MST) (No. Korea(NRF) grant funded by the Korea (MVT)) (RS-2025-00556779), and by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea governmen((MSIT) [NO.RS-202 1I22 1343, Artificial Intelligence Graduate School Program (Seoul National University)].
the National Research Foundation of
References
[1] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910001108,, 2019.
[2] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. arXiv preprint arXiv:2306. 08543, 2023.
[3] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean--appneetee Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katic Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1180. 2023.
[4] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
[5] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1.Xiv:5503.0255: 2015.
[6] Yarin Gal et al. Uncertainty in deep learning. 2016.
[7] Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018.
[8] Zelda E Mariet, Rodolphe Jenatton, Florian Wenzel, and Dustin Tran. Distilling ensembles improves uncertainty estimates. In Third symposium on advances in approximate bayesian inference, 2020.
[9] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[10] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903. /2261, 2019.
[11] Linh Tran, Bastiaan S Veeling, Kevin Roth, Jakub Swiatkowski, Joshua V Dillon, Jasper Snoek, Stephan Mandt, Tim Salimans, Sebastian Nowozin, and Rodolphe Jenatton. Hydra: Preserving ensemble diversity for model distillation. arXiv preprint arXiv:2001.04660 04694, 2020.
[12] Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715. 2020.
[13] Giung Nam, Jongmin Yoon, Yoonho Lee, and Juho Lee. Diversity matters when learning from ensembles. Advances in neural information processing systems, 34:8367-8377, 2021.
[14] Giung Nam, Hyungi Lee, Byeongho Heo, and Juho Lee. Improving ensemble distillation with weight averaging and diversifying perturbation. arXiv preprint 2ttp:::2226.154444 2022.
[15] Hailin Zhang, Defang Chen, and Can Wang. Adaptive multi-teacher knowledge distillation with meta-learning. In 2023 IEEE International Conference Multimedia and Expo (ICME), pages 1943-1948. IEEE, 2023.
[16] Martin Ferianc and Miguel Rodrigues. Simple regularisation for uncertainty-aware knowledge distillation. arXiv preprint arXiv:2205.099 2022.
10

---

[17] Andrey Malinin, Bruno Mlodozeniec, and Mark Gales. Ensemble distribution distillation. arXiv preprint arXiv:1905.00000 00076, 2019.
[18] Max Ryabinin, Andrey Malinin, and Mark Gales. Scaling ensemble distribution distillation many classes with proxy targets. Advances Neural Information Processing Systems, 34:6023-6035, 2021.
[19] Thomas G Dietterich. Ensemble methods in machine learning. In International workshop multiple classifier systems, pages 1-15. Springer, 2000.
[20] Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel, Jean-Mare Martinez, Andrei Bursuc, and Gianni Franchi. Packed-ensend Packed-ensembles for efficient uncertainty estimation. arXiv preprini urXiv:2210.09188 2022.
[21] Radford M Neal and Radford M Neal. Monte carlo implementation. Bayesian learning for neural networks, pages 55-98, 1996.
[22] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-II), pages 681-688. Citeseer, 2011.
[23] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference machine learning, pages 1050-1059. PMLR, 2016.
[24] Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization. Advances neural information processing systems, 0774497--778,, 2020.
[25] Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are bayesian neural network posteriors really like? In International conference on machine learning, pages 4629-4640. PMLR, 2021.
[26] Mrinank Sharma, Sebastian Farquhar, Eric Nalisnick, and Tom Rainforth. Do bayesian neural networks need to be fully stochastic? In International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2023.
[27] Insung Kong, Dongyoon Yang, Jongjin Lee, Ilsang Ohn, Gyuseung Baek, and Yongdai Kim. Masked bayesian neural networks: Theoretical guarantee and its posterior inference. In Interna- tional conference on machine learning, pages 17462-17491. PMLR, 2023.
[28] Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, and Bhuvana Ramabhadran. Efficient knowledge distillation from an ensemble of teachers. In Interspeech, pages 3697-3701, 2017.
[29] Kisoo Kwon, Hwidong Na, Hoshik Lee, and Nam Soo Kim. Adaptive knowledge distillation based on entropy. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7409-7413, IEEE, 2020.
[30] David Bartholomew, Martin Knott, and Irini Moustaki. Latent variable models and factor analysis: A unified approach, volume 904. John Wiley & Sons, 2011.
[31] Zexun Chen, Jun Fan, and Kuo Wang. Multivariate gaussian processes: definitions, examples and applications. Metron, 81(2):181-111, 2023.
[32] Donald B Rubin and Dorothy T Thayer. Em algorithms for ml factor analysis. Psychometrika. 47:69-76, 1982.
[33] Norman E Breslow and Xihong Lin. Bias correction in generalised linear mixed models with single component of dispersion. Biometrika, 82(1 (11))81-91, 1995.
[34] Xihong Lin and Norman E Breslow. Bias correction in generalized linear mixed models with multiple components of dispersion. Journal of the American Statistical Association, 1004 1996.
11

---

[35] Bas Engel. A simple illustration of the failure of pql, irreml and aphl as approximate ml methods for mixed models for binary data. Biometrical Journal: Journal of Mathematical Methods in Biosciences, 40(2):141-154, 1998.
[36] Anders Skrondal and Sophia Rabe-Hesketh. Generalized latent variable modeling; Multilevel, longitudinal, and structural equation models. Cre Press, 2004.
[37] Qi Lei, Wei Hu, and Jason Lee. Near-optimal linear regression under distribution shift. In International Conference on Machine Learning, pages 6164-6174, PMLR, 2021.
[38] Ruihan Wu, Chuan Guo, Yi Su, and Kilian Q Weinberger. Online adaptation to label distribution shift. Advances in Neural Information Processing Systems, 34:1 11111340-13511 2021.
[39] Yong Bai, Yu-Jie Zhang, Peng Zhao, Masashi Sugiyama, and Zhi-Hua Zhou. Adapting to online label shift with provable guarantees. Advances in Neural Information Processing Systems, 01229900229774 2022.
[40] Dheeraj Baby, Saurabh Garg. Tzu-Ching Yen, Sivaraman Balakrishnan, Zachary Lipton, and Yu- Xiang Wang. Online label shift: Optimal dynamic regret meets practical algorithms. Advances in Neural Information Processing Systems, (6:65703-657422 2023.
[41] Saurabh Garg, Nick Brickson, James Sharpnack, Alex Smola, Sivaraman Balakrishnan, and Zachary Chase Lipton. Rlsbench: Domain adaptation under relaxed label shift. In International Conference on Machine Learning, pages 10879-10928. PMLR, 2023.
[42] Elan Rosenfeld and Saurabh Garg. (almost) provable error bounds under distribution shift via disagreement discrepancy. Advances in Neural Information Processing Systems, 36:28761- 28784, 2023.
[43] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202. 10054, 2022.
[44] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation distribution shifts. arXiv preprint arXiv:2210.11466, 2022.
[45] Jonggeon Park, Giung Nam, Hyunsu Kim, Jongmin Yoon, and Juho Lee. Ensemble distribution distillation via flow matching. In Forty-second International Conference on Machine Learning.
[46] Arthur Asuncion, David Newman, et al. Uci machine learning repository, 2007.
[47] Thang Bui, Daniel Hermnndez-Lobato Jose Hernandez-Lobato, Yingzhen Li, and Richard Turner. Deep gaussian processes for regression using approximate expectation propagation. In International conference on machine learning, pages 1472-1481. PMLR, 2016.
[48] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[49] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint 1907. 2019.
[50] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1804 074611 2018.
[51] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances neural information processing systems, 32, 2019.
[52] Kaare Brandt Petersen, Michael Syskind Pedersen, et al. The matrix cookbook. Technical University of Denmark, 7(15):110, 2008.
12

---

[53] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[54] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009.
[55] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 00..001145 146, 2016.
[56] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinherger. On calibration modern neural networks. International conference on machine learning, pages 1321-1330. PMLR, 2017.
[57] Adam X Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. arXiv preprint arXiv:2.308.33 2023.
[58] Jose G Moreno-Torres, Troy Raeder, Rocío Aldiz-Rod Nitesh V Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern recognition, 45(1):521-530, 2012.
[59] Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[60] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[61] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-61 14. PMLR, 2019.
[62] Minwoo Chae, Dongha Kim, Yongdai Kim, and Lizhen Lin. A likelihood approach to nonpara- metric estimation of a singular distribution using deep generative models. Journal of Machine Learning Research, 24(77):1-42, 2023.
[63] Johannes Schmidt--Hieber. Nonparametric regression using deep neural networks with relu activation function. 2020.
[64] Wing Hung Wong and Xiaotong Shen. Probability inequalities for likelihood ratios and convergence rates of sieve mles. The Annals of Statistics, pages 339-362, 1995.
[65] Yongdai Kim, Issng Ohn, and Dongha Kim. Fast convergence rates of deep neural networks for classification. Neural Networks, 138:179-197, 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Introduction, provide the method of this paper and its key contributions. Abstract also briefly mentions these points.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss a potential limitation of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification: In Appendix G we provide the full set of assumption and a complete proof of the main theorem in this paper. Also, for the theoretical claims in Section 3.4 we provide complete mathematical proofs in Appendix G.2
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Appendix C we provide details of our experimental setups.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the source codes our proposed algorithm the supplementary material. Furthermore, we will publicly upload them after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guides/ wttaeees//ooeeSmmmiaiiiinooiioi
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppplcccgccceeeeeeco Chte://doeSSubmmcmic ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Appendix C we provide details of our experimental setups.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: In Section we report the mean and standard deviation from multiple runs.
And in Section 5.3 and Appendix D also plot the max-min area in figures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: In Appendix C we report the number of parameters for each regression and classification case. And we provide the used computing resources in Appendix C.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We thoroughly and carefully checked the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We give several societal broader impacts of our work in Section 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We give several societal broader impacts of our work in Section 6
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We mention all the owners/referencce//uus of methods/codeesdlataeet used in Section and Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, latasets,papersstt pwpppppppssiia..
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the source codes of our proposed algorithm in the supplementary material. Furthermore, we will publicly upload them after acceptance.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This study does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This study does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20

---

APPENDIX
A Review of Ensemble Distillation
A.1 One-to-one Distillation
(t) For given teacher members pi , 1..... n. student p, , 1, 1....., n are trained minimize
Σ Ex
KL (p. (p. (1/2))lp (v/z))
where Ez is the expectation operator of a certain probability distribution on the input space X. The following two methods use specially designed DNNs for student models which share the weights between student models.
A.1.1 Hydra
employs a multi-head architecture in which single shared body network extracts common features and a set of n distinct linear heads generates predictions for each ensemble member. Formally, the student DNN is parameterized by
Hydra 11
Ohydra (Obody: 0head,1, -.. )
where body used across all heads. At inference, the body is evaluated once, and each head heread (-)
produces member-specife output ( x:) her (benody (2)) This design captures ensemble diversity while reducing both computation and memory compared to maintaining n independent networks.
A.1.2 Batch Ensemble and Latent Batch Ensemble
introduces Batch Ensemble (BE) to reduce the memory and computational burden of deep ensembles. In this architecture, all student networks share a core weight matrix W' each layer, while individual ensemble members are differentiated by rank-one perturbations. Specifically, for the i-th student at layer
w = w' of
where ri E Red and sf E A modulate the rows and columns of W respectively. This construc- tion enables each sub-network maintain member-specific behaviors while reusing the majority of parameters, yielding significant savings in both storage and inference costs compared to training independent models.
Building on BE, [14 proposes Latent Batch Ensemble (LBE), which further compresses the ensemble inference time. Instead of maintaining n distinct perturbations, LBE computes the average rank-one mask across all students:
n
W!
W
n
The resulting single student network requires only one forward pass per input while capturing the ensemble's mean perturbation in weight space. Empirical results demonstrate that LBE matches or exceeds the calibration performance of standard BE, with inference cost reduced by factor of n. However, the proposed Latent Batch Ensemble is a specialized method designed for ensemble distillation for classification problems.
A.2 Distribution Distillation
Distribution distillation frames the output of an ensemble at the input æ samples from input- dependent Dirichlet distribution [7 18). Let f i I,...,, be given teacher models for a classi- fication task. For a given æ, let π (x) softmax (f (x)) and assume that π (1(x),... (æ),..., (x)
21

---

independently follow the Dirichlet distribution with parameter axxx) = (.. Then, we model (xx) by student DNN (z and estimate by maximizing
C(O, C(0,Ddesion Dd
[mp(n, (4)
v ((x) [0)]]
where p(T](a) is the density of the Dirichlet distribution with parameter a. See [7 for details.
A.2.1 Proxy-Dirichlet Distillation
18] proposed Proxy-Dirichlet Distillation (Proxy-EnD*) to mitigate the convergence difficulties of standard Dirichlet distillation [17] when the number of classes is very high. The method constructs
proxy Dirichlet distribution from the teacher models and trains the student DNN (x by minimizing the reverse KL divergence between p( π y (z) and the proxy distribution.
The proxy distribution is obtained from teacher models
n
(((x) n
Ta(x))
(xx (c (c-1)
1= 1,2,..
2 ii B((x)
(xx) - +/2 Eim Ia lm)a
Then, estimate 8 by minimizing
Σ KL. (pl (p(T (a)) (0)) || |-(a(((])"))
a(x,")) + +c))
where a(-) =
A.2.2 Ensemble Distribution Distillation via Flow Matching
Ensemble Distribution Distillation via Flow Matching (EDFM) 45 models conditional distribution of logits h1 given æ over RR. The method constructs the target distribution p1 (h) a:) as an empirical distribution using f; (ar).
They model using Flow matching such that
(x) = (l, x))1,, 1 E [0, ho( ~ NN(0,, and k/w _ hi (x)
where a is student DNN. Then, the network is trained by minimizing the conditional flow matching loss
Σ E ((el))) )
A(t) L (zx (00) (0)) - (h, (z0 - ho)l?3
where hg Ne(O, I2I,), he (1x pi( (- (d) ) U0. 1). (ax the (xx (1 t)ho
and X denotes a time dependent weighting function.
Sampling proceeds by numerically integrating the learned flow L (x)) x) with standard ordinary differential equation (ODE) solver. For more detailed information, see (45).
B
Details of Gaussian distillation
B.1 EM algorithm for the univariate DL.F model
The main idea of Gaussian distillation to estimate the mean and factor loading functions based on given teacher models assuming that teacher models are independent realizations of the DLF and estimate the parameter in the student DNNs by maximizing the corresponding log-likelihood. For optimization, use the EM algorithm (32..
22

---

f-(-) are given. Gaussian distillation consists of three steps. The first step is to choose m-many design points Ddosign
Suppose that n many teacher models
{-|| (d) =m)) }. The second step is to calculate the vectors of prediction values of each teacher model at the design points to have f: = (Jr ).jj I..==,..... m)
for l..... n. The final step to estimate the parameter A the DLF by the MLE assuming that fi(.).... (-) are independent realizations random function following the DLF model. Since f& are independent Gaussian random vectors, the MLE can be
obtained by use of the EM algorithm as follows.
To make the EM algorithm numerically stable, we consider the noisy DLF model which assumes that (zm)) with f((-)s following the DLF model. Specifically, each f; expressed as f.(.) = (-) + (e(-) T 24, where 2; ~ (O, 1,) denotes the latent factor corresponding the i-th function realization. Then, we obtain the MLE of the parameter 8 in the mean and factor loading functions as well as 07. We abuse the notation to write
f. - -ftt where v; Nr.. (0, (f and f. =
(0, of) unless there is any confusion.
The complete log- log-likelihood is given as
Σ 2 x;
geom (0lf ( l:a 11:n)
1877
log (og(2To7) )-
14
(og(2x)
2
22
2
Ei= (f: M4
-b
Tf; :40 (10%))
20²
where Jin (f1.... f kni Zim (. 14g = (uo(a)))),
)..... ..... uo(a)1))) and dbp
(2) (11) )..... Lo (aml )) an m X q matrix.
Thus, for given parameter ((t-1) at time t the E-step is to calculate the conditional expectation of the complete bog-likelihood which is given as
((010(1-11) -E in tm -ir-1) (com (c0mm(0) [0]f 1n #1:n]]
D B|,22 (0(t-2), s1n1]
nm
log Log(2moo
(TTO7)
12
log(2T)
2 + 2(f, m0) ) (0b((2z10((-1)) ff.m] + tr ( 2,12 (100,,,0000011 f1mul) }.
2
1
(f, (Ho) T(f: -HO)
20
where
([2z|0(1-)) . f1:n v(210(t-1)) 1) 1 (f, 1) 1/07 ///
2 10(1-1), ff.m = ([2]0(e-1)] E[zz (00-1) of f1.1111122 (( f1.nJ
with V[z(0(r-1)]
(I, 1)
+ go-n 000-n/0
In the M-step, usually update g( by maximizing Q ((c-1)), Instead of maximizing (t0))( we update 0 by using a stochastic gradient descent algorithm (i.e., gradient descent algorithm on given mini-batches). The EM algorithm is summarized in Algorithm
23

---

Algorithm 1: EM algorithm for the univariate DLF model
Input:
Design points Ddesign { (d)
(d) b.
Teacher ensemble members f. fa(-), learning rate >> 0
Output: Estimated parameter
Initialize parameter: g(0) for t = 1,2...7 do
Shuffle the dataset Deesign and divide into mini-batches (Bi...., BBM)
for 1 ........ .M do
Calculate prediction values of teacher model f 1, where
f 1,,,8, (f (2x ( ), design E Bi).
Calculate [[z]0[t-]]] and E[zz [10t-11] using f I.n,B,
Q(0)0(t-1))= EzinI s, oct (ycom (ccom (0; 1.m,B #1m)
Calculate gradient ((t) Vo (- (-Q(0))(t-1)))
Update g(t) g(t-1) 7) g())
end end
B.2 Multivariate DLF model
In this section, we provide details of the multivariate DLF model introduced in Section 3.1 With given the design points {-) (d) ....,...,yy xm }, let f = (f(zz) ).j 1, m.. and P4g (mo ),j =
m.) be matrices, and let 4b (oo(æ) (1) l,. m) be g matrix. Then, we assume that f follows the matrix-varie Gaussian distribution
MN (M4, P. LLT).
Using the properties of the matrix-variate Gaussian distribution, vectorize f, thereby allowing the multivariate DLF model to be handled the same as the univariate case.
B.2.1 Vectorization
According to Definition
31, the vectorization of f can be expressed as follows:
vec(f) -Nmo(ve LL 4b
where & denotes the Kronecker product. This association arises from factorization of the covariance matrix of the multivariate Gaussian distribution into the Kronecker product of matrices Φ 4 and LLT.
B.2.2 EM algorithm for the multivariate DLF model
We explain the EM algorithm for training the multivariate DLF model in the same way as in Appendix B.I. In the multivariate case, matrix vectorization and its associated properties serve the central tools, as detailed in Section 10.2.2 of [22]
Suppose that n teacher ensemble members f((.).... (-) are given, where each f.(+): X' - R a multivariate function. As in the univariate case, KD via the multivariate DLF involves three steps. First, we choose m design points Deesign = {x] (d)
( The second step is to calculate the prediction values of many ensemble members at the design points to have f. = (f (z)
..... (
m....,m) for i = 1, 1,..., n. Here, unlike in the univariate case, each f: should be regarded as an m e matrix. Similarly to the univariate case, the final step employs the EM algorithm obtain the MLE.
Similarly to the univariate case, assume that f. = f. v, where 8, MN (0, Ie) and f. - (f,(),,,... (zx))) with f.(:)s following the multivariate DLF model. From the property of the vatrix-variate Gaussian distribution, can rewrite the following vec(f.) vec(f.)
24

---

vec(v;), where vec(u;) ~ (0, 00, And in the case of multivariate case, we abuse the notation to write g(t) == (0(m), _ _ (*) like the univariate case.
From this, the complete log-likelihood is given as
goom (0lf 1:m 21:n) = cam
log log(2ro7) enq
log(2 Eiz voo(2i) voc(z;)
2
2 2 (voo(f)) voc( (f))) (vvo(f.)) voo(fi))
2
20
enq log(2T) Eis1 vec(z,) TT vec(zi)
Crum
log(2To77)
2
2
> 2i=1
TM
B; B:
20²
where B; = veo(f,) - vec(110) (oo 8 L) vec(zi), Sin = fi....:fn::: 2i:1 (no(2n) (1) ), .....,,....................... u
(mm)) )) is an m matrix and th (Pa(a) (11) ).... Do (2e'm) ))T is an m x matrix. Thus, for given parameter ((t-1) at time & -- 1, the E-step is to calculate the conditional expectation of the complete log-likelihood which is given as 1(010(t-1)
(21,..., 2n.. 14g
Be Im 0(+-1) [e {0f
f 21:m)]
enq log log(2T) Ex tr [(ve((z)) vec(z:) 10((-1)) (e((2)) f1m1
cam
1og(2To7}
2
(vec(f,) - vec(H))) T (vec(f.) vec(Hu))
20 1=1
+ 2(veo(f.) vec(Ma)) ( & [=v0vv((2:)) f fim]
+ tr (L T 2 2 2e @ L)e[(vc(2z) vec 10(t-1), ,f .1mm)) }
where
V (e S1mm (L @ $ 201111)) (S: 19(t-1))
E vecc(z) 22) (((-1),
f 11:
of (1-1-
vec(2;) voe(zi) T g((-1) f f1mm]
V [0o(2))0((-)) 1((-1), .1
E vec(2) (t(1-1) f 1:m
E [vc(2)]0((-))
i 1mm
where V [voc(=)]((t)] (Iyz (Z
क p (I & B 102(0))
In the M-step, instead of maximizing 1(0) update 0 by of stochastic gradient descent algorithm.
C Experimental details
In this section, we describe the overall setup of our experiments in detail, focusing on the datasets, model architectures, training procedures, and evaluation metrics used in regression and classification problems. Four baselines are considered: (i) small ensemble of lightweight networks ("small-Ens"), (ii) Hydra, and (iii) BE for regression and LBE for classification.
C.1 Regression case
We consider the standard regression problem
y y= fx) +E, (
25

---

Suppose that there are many teacher models l (-) and o,s- Then, the predictive distribution of given æ: is constructed as pl (y) [ = Ei= (z)) oe ) where N (ylu,, o²) is the density of the Gaussian distribution with mean ja and variance o² The predictive distribution p(*) (yl2z) based on student ensemble members is defined similarly.
(0)
(1)2
We obtain 50 teacher models of DNNs with two hidden layers and 100 nodes at cach layer which learned by minimizing the sum of squared residuals of the training data with 50 randomly selected initial solutions. For the design points used in the distillation, The architecture of student models comprises of an ame-iddeen-laye MLP with 50 units. The number of parameters in the student ensemble of each method is summarized in Table 4 Note that the number of parameters of DLF is smaller than those of the other baselines because the dimension the latent factor is 10 instead of 50.
the training data themselves.
Table 4: The number of parameters in the student ensemble.
Datasets Energy
Method
Boston housing
Concrete
Wine
Power
Kin8nm
Teachers
90,000
65,000
65,000
80,000
45,000
65,000
small-Ens
45,000
32,500
32,500
40,000
22,500
32,500
# of parameters
Hydra
6,650
6,150
6,150
6,450
5,750
6,150
BR
7,351
6,601
6,601
7,051
6,001
6,001
DLF(factor dim 10)
1,862
1,612
1,612
1,762
1,412
1,612
The Adam
53
is used for the optimization.
C.1.1
Dataset
We consider the following UCI datasets (Boston housing, Concrete, Energy, Wine, Power Plant, Kin8nm) [46]. We divide each dataset into a 9:1 ratio randomly for the training and test data. The experiment is repeated with 10 random split to have 10 evaluation metrics.
Table Description of UCT benchmark datasets used in the experiment
Dataset
size
# of features
Boston housing
506
13
Concrete
1030
Energy
1030
8
Wine
9568
Power
768
Kin8nm
8192
C.1.2
Evaluation metric
Let ((11.81). 81),.... (Emrntt Vmemn } be given test data.
Root Mean Square Error
Root Mean Square Error (RMSE) is defined
Micse
RMSE
Σ 1 B(ylæ,)))
Meest
where (Byl=) yp(y) by and p(y) x)) is estimated predictive distribution.
Negative Log Likelihood
Negative Log Likelihood (NLL) is defined as
Mect
NLL
log p(y; xz).
26

---

Continuous Ranked Probability Score
Continuous Ranked Probability Score (CRPS) is defined
CRPS
1
Meset
Σ
/
(B(()) 1( u/))]] du
Meest
where = fly x,
C.2 Classification case
We consider c-class classification problem where
exp (x))))
p(ul x. f) Ei=1 exp (fp(f(())) y E fI,....cc}
for given (vector-valued) function f()) = ( (.()). For given teacher ensemble p!
1...., the teacher predictive distribution is estimated by (g(=) = Σi= p(y|2, f(()) ) student predictive distribution for a given student ensemble defined similarly.
)/n. The
C.2.1 Dataset
In classification settings, we analyze two CIFAR datasets [54]. Each dataset contains 50,000 training and 10,000 test images of natural scenes, sized 32 32 pixels.
Table 6: Description of CIFAR-10 10 and CIFAR-100
Dataset
Train size
Test size
# of labels
CIFAR-10
50000
10000
10
CIFAR-100
50000
10000
100
We follow the set-up of experiments in 14 As teacher, we ensemble of four neural networks, where each model is a Wide-ResNet (WRN) [55]. Specifically, WRN-28-1 is used for CIFAR- 10 and WRN-28-4 is used for CIFAR- 100. And, the student model uses the WRN-16-1 network for CIFAR-10 and the WRN-28-1 network for CIFAR-100.
Training lasts 200 epochs on single GPU using SGD with Nesterov momentum of 0.9, weight decay of 10-1, and batch size of 128. A one-cycle cosine annealing schedule with a five-epoch linear warm-up (from 0.001 to 0.1) is employed. The number of parameters in each ensemble is summarized Table 7
Table 7: Number of model parameters in each ensemble.
Datasets
Method
CIFAR-I0
CIFAR-100
Teachers
1.48M
23,488M
small-Ens
0.70M
1.50M
# of parameters
Hydra
0.18M
0.39M
LBE
0.18M
0.38M
DLF(factor dim 8)
0.18M
0.38M
C.2.2 Evaluation metric
Accuracy Accuracy (ACC) is defined as
Maset
ACC
Σ
1 = 0,),
Miest
j=1
where ú arg max, p(y(2z,).
27

---

Negative Log-Likelihood
Negative log- rogelikelihodd (NLL) is defined as
Mhest
NLL
1yy k) log ply k xz))
Mtest
k=1
Expected Calibration Error
Expected Calibration Error (ECE)
[66 is defined as
|Brl
1
BCE
Σ
Σ
I 05)
p(y; I az))
Meest
B
B
(my-Dy)E
(E
where [BI..... BM) is partition of the test data Prest such that Bi (xx v) E Drest p( 2) (1-11)/M1//M) }. In this work, M is set to be 15.
C.3 Fine-tuning of language models
The experiments are conducted on three datasets from the GLUE [50] and SuperGLUE [5] mark: RTE, MRPC, and WiC. Table 8 summarizes the details of each dataset.
bench-
Table 8: Description of GLUE and SuperGLUE benchmark datasets used in the experiments
Dataset
Task Type
Description
RTE
Recognizing Textual Entailment
Determines whether given hypothesis can be inferred from given premise sentence.
MRPC
Paraphrase Detection
Identifies whether two sentences are semantically equivalent. The dataset consists of sentence pairs from news sources.
WiC
Word Sense Disam- biguation
Determines whether specific word used in two differ- contexts has the same meaning. Contextual under- standing of word senses is essential.
fine-tuned with the Low-Rank Adapta- tion (LoRA) method (48) on each task. For each dataset, four fine-tuned models are trained using different random initializations to construct teacher ensemble.
Model Construction For teacher models, RoBERTa 149
Student models are based on DistilRoBERTa [1] which is also fine-tuned with LoRA. Features are extracted through the shared backbone, and task-specific prediction heads are constructed depending on the design of each distillation method.
Training and Evaluation Settings The experimental settings, including learning rate, batch size, number of epochs, and the rank LoRA, follow the configuration used in [57]. Model performances evaluated using the three metrics described in Appendix C.2 Acc, NLL and ECE.
C.4 Application to distribution shift
To apply the framework introduced in Section 5.3 to classification tasks, consider the following model:
y W Multinommal(ssftmaan (fx;; W))),
[a; W) = no [ (
where W the weight matrix. For new data Dnww we estimate W by minimizing the cross-entropy on new data while and are fixed at and estimated on training data.
For numerical study, we consider distribution shift scenario where the conditional distribution P(X Y) changes [58). generate synthetic data, we use CIFAR- 10 and flip the labels by y ++ 9-y to construct CIFAR 10-Flip dataset (44). We first learn the body by DNN, Hydra, and DLF on the training data of 50,000 CIFAR-10 images under the WRN- architecture for the teacher and student ensembles. Then, train the weights of the linear head new data while the body is fixed.
28

---

C.5
Hardwares
All our experiments are done through Python 3.9.16 with Intel(R) Xeon(R) Silver 4310 CPU @ 2. 10GHz, NVIDIA TITAN Xp GPU and 128GB RAM.
D Ablation Study
We do the following ablation studies on Boston housing data.
We investigate how the choice of design points affects distillation performance.
The effect of the choice of latent factor dimension affects distillation performance.
The effect of the MMD-based initialization is investigated assess its role in stabilizing the EM algorithm during training.
The capacity of the student models varied by adjusting the network width, and the effect of this change is analyzed for each baseline.
We investigate the effect of ensemble size by varying the number of ensemble members used in the teacher and student ensembles.
Quantitative results are presented using RMSE, NLL, and CRPS, aggregated over ten independent
D.1 Choice of design points
D.1.1 Comparison of different design points selection methods
We investigate how different strategies of selection design points influence the performance of the Gaussian distillation. The entire dataset is partitioned into three disjoint subsets: 'D Dirain ( new + Deest with fixed ratio of 4.5 4.5 1. The teacher ensemble is trained on D teacher and the student model is distilled using various forms of design points Deesign To analyze the impact of the choice of design points, the four distinct strategies for selecting Deesigg are considered:
Design 1: Directly using the teacher training data, Ddesign = Dirain teacher
Design 2: Using mixup samples from Dirain teacher" DDdesign C mixup Diraan
teacher
Design 3: Using training data not used in training teacher ensemble, Delesign D Dirain
Design 4: Using mixup samples from Dtrain Deesignn C mixup(DDtrain)
Here, mixuppDirast DDtrain Dtrainer) denotes the set samples generated by linearly combining two randomly selected samples from Dieacher. For each index j, we randomly draw two data pairs (æz, y;) and (arf, v5) from Dtrain and form the mixed sample
v""""""))= (a v/) -1) (a)( X, E [0,
The set mixup Dirain Peacher) consists of all mixed pairs (z 1, with the number of generated design points in Designs and matched to the size of Dirainer- These strategies are designed to cover both scenarios where design points reused from the teacher training data and where additional or perturbed data are incorporated. All methods are evaluated on the reserved test data Diest
Table 9: Performances of Gaussian distillation for different strategies of the design point selection
Design Strategy
RMSE
NLL
CRPS
Design
3.2316 (0.0587)
2.6317 (0.0267)
1.7896 (0.0271)
Design 2
3,5856 (263)
2,8003 (0.0638)
1.9648 (0.0681)
Design 3
2.8786 (0.0322)
2.4816 (0.0129)
1.6427 (0.0165)
Design 4
3.1787 (0.2498)
2.6122 (0.1144)
1.792 (0.
The results summarized in Table visualized Figure 3 illustrate the effect of different choices of design points the performance of the Gaussian distillation.. Among the four strategies, the use

---

34
Design Design Design Design
29
2.5
RMSE
CAPS
28
2.5
Design
Design
Design
Design
Design Dasign
Design 3 Desigr
Design Design 2 Design Design
Design Type
Figure 3: Box plot of evaluation metrics (RMSE, NLL, CRPS) for different strategies of the design point selection
of new training data (Design 3) for design points consistently yields the best performance across all metrics. In contrast, strategies involving mixup consistently yield inferior performances regardless of whether teacher training data (Design 2) new data (Design 4) are used.
To sum up, the results suggest that using new training data is the best for Gaussian distillation. However, the size of the teacher training data becomes smaller, which might lead to performance degradation. In practice, we could find the optimal partition of teacher training data and new training data based additional validation data.
D.1.2 Comparison for different selection of the number design points
We also investigate the effect of the number of design points on the student model. As in the previous experimental setup, the teacher ensemble is trained on Diraim and the student model is distilled using various sizes of design sets Deesign We only consider the design type Dirain (Design 1) and Dirain (Design 3). The number of design points is controlled by the design ratio
(0.2, 0.4, 0.6, 0.8, 1.0)
meaning that Gaussian distillation uses x D dessign| samples for distillation.
Table 10: Performance for Gaussian distillation for the number design points
Design Type
Design
Design 3
design_ratin gariio
RMSE
NLL
CRPS
RMSE
NLL
CRPS
0.2
3,9829 (0.4823)
3,0244 (0.2738)
2.1773 (0.1704)
3.8643 (0.3277)
2.951 (0. (721)
2.145 (0.1349)
3,576 (0.3231)
(0.1612)
(0.1349)
3.0703 (0.0735)
2.5612 (0.0316)
(0.0396)
0.6
3.3996 (0.1750)
2.711 (0.0834)
1.9122 (0.0573)
3.0055 (0.0634)
2.5337 (0.0267)
1.7161 (0.0243)
0.8
3,2548 (0.1407)
2.6433 (0.0642)
1.8444 (0.0663)
3.0229 (0.0945)
2.5413 (0.0395)
1.7197 (0.0444)
3.2316 (0.0587)
2.6317 (0.0267)
1.7896 (0.0271)
2.8786 (0.0322)
2.4816 (0.0129)
1,6427 (0.0165)
The results summarized in Table and and Figure show that increasing the design ratio consistently improves the performance of the Gaussian distillation. That is, the larger the number of design points, the better the performance of Gaussian distillation is.
D.2 Dimension of the latent factor
We investigate the influence of the dimension of the latent factor on the performance of Gaussian distillation. The result visualized in Figure indicates that the performance Gaussian distillation not sensitive the dimension of the latent factor unless the dimension is too small.
30

---

Design Type Design
4.5
Design
2.4
425
32
2.2
400
RASE
CRPS
2.0
330
325
2.75
1.5
0.8
0.8
0.8
1.0
Design Ratio
Figure Evaluation metries (RMSE, NLL, CRPS) versus the design ratio.
30
37
1.7
CRPS
2.7
2.5
2.4
Latent Factor Dim.
Figure 5: Evaluation metrics (RMSE, NLL, CRPS) versus latent factor dimension.
D.3 MMD vs random initial
We compare the proposed MMD initialization in Section 3.2 with the random initialization. As shown Figure 6 the MMD initialization strategy outperforms the random initialization unless the dimension of the latent factor is too small. In addition, the variations of the evaluation metrics for the MMD initialization are much smaller than those the random initialization. That is, the MMD initialization indispensable for the superior performance of Gaussian distillation.
D.4 Capacity of student models
The effect of the capacity of student models examined by varying the number of nodes in one-layer DNN architecture. We increase the number of nodes gradually from 50 to 60, 70, 80, 90, and 100, and obtain the evaluation metrics the distillation methods. The results in Figure 7 show that the performances of DLF and small-Ens keep improving the number of nodes increases, while
31

---

MMD Initel selizininn Rendom intialication
30
CRPS
277
2.0
2.5
2.6
2.5
2.4
Latent Factor Dim.
Figure Evaluation metrics (RMSE, NLL., CRPS) versus latent factor size with without the MMD initialization
the performances of Hydra and BE are saturated. Apparently, DLF behaves similarly to small-Ens, which is interesting since small-Ens demands heavier computation and much more storage.
2.85
2.48
140
2,80
2.75
Nethod
RMSE
CAPS
small
1.50
ITLF
2.42
2.65
1.45
2.40
2.80
Number of Hidden Nodes
Figure 7: Evaluation metries (RMSE, NLL, CRPS) versus the number of hidden nodes.
D.5 Number of ensemble members
To evaluate the effect of ensemble size, we evaluate the performance of the KD methods by varying the number of ensemble members from 10 to 50. As we can see from Figure for all methods, the performances keep improving as the ensemble size increases. Note that the number weights DLFF model is not proportional to the ensemble sizes (instead, it is proportional to the dimension of the latent factor), while proportional for the other baselines. This is additional advantage of Gaussian distillation.
32

---

2.50
145
2.50
2.54
140
2.85
Method
RASE
CRPS
small
htdra
1.35
DL
2.40
2.75
130
2.70
1415
2,4
20
20
Number of Ensembles
Figure Evaluation metries (RMSE, NLL, CRPS) versus the number of ensemble members.
D.6 Application large dataset
To further examine the scalability and robustness of our proposed model, we conducted an additional experiment on larger and more complex dataset, Tiny ImageNet [59]. This dataset extends the original ImageNet hierarchy but contains reduced subset of classes and image resolutions, making challenging yet computationally manageable benehmark for evaluating model generalization on large-scale visual domains.
Tiny ImageNet consists of 200 classes with 100,000 training images and 10,000 test images, where each image is resized to 64x64 pixels. To ensure fair evaluation and avoid overfitting, we randomly split the training into 80% for training and 20% for validation.
We compared our method against Small Ensemble, Hydra, LBE, and Proxy-EnD*, The teacher network was implemented using WRN-28-4, while the student network employed WRN-16-4, Both teacher and student models were trained under an ensemble size of four, following the same distillation pipeline described in Appendix C.2
Table 11: Results on Tiny ImageNet.
dataset
method
Acc(56) 1
NLL
ECE(S) 1.
Teacher
68.74
1,2806
3.22
Small-Ens
57.11 (0.0027)
1.854 (0.0032)
13.51 (0.0032)
Hydra
(0.0037)
1.7503 (0.0092)
2.75 (0.092)
Tiny ImageNet
(0.0119)
2.24 (0.0407)
(0.0053)
Proxy-EnD2
51.64 (0.0022)
2.0195 (0.0069)
2.77 (0.0038)
DI.F
58.34 (0.2121)
1.6737 (0,0043)
1,92 (0.1098)
Table 11 shows that our approach outperforms all other methods across all measures, even when the number of classes and samples is large, suggesting that the proposed framework generalizes well beyond small-scale datasets. In contrast, both LBE and Proxy-EnD* exhibit significant drop in performance when applied to complex data, confirming that our latent factor modeling remains effective even in such challenging settings.
E
Uncertainty quantification
We conduct experiments to evaluate whether each model appropriately quantifies uncertainty by detecting out-of-distribution (OOD) data in classification problem. We adopt predictive mutual
33

---

information, which estimates epistemic uncertainty (6, as the OOD detection score:
ply x, x,) log Σ ply x, t..).)
==
y=1
i=1
n
plu az 0.) logply æ, 0.).
y
Specifically, we denote the in-distribution (ID) dataset by {æf wm J. the OOD dataset by {a]" Fm and y is the output of the model with corresponding predictive mutual informa- tion [ly 0 xx." for i = 1,.., 1 and I y, x: for = I.... m2..
For evaluation, assign label 0 ID data and label to OOD data, meaning that lower predictive
mutual information indicates ID whereas higher values indicate OOD. We then compute the AUROC between these labels and the predictive mutual information scores.
We use CIFAR-10 10 as the ID and SVHN, CIFAR- 100, and Tiny ImageNet as OOD. All models are trained on the CIFAR-1 10 training set with 50,000 images, and OOD detection is evaluated on the CIFAR-10 10 test set with 10,000 images versus the test sets of the OOD datasets, where SVHN has 26,032 images, CIFAR- 100 has 10,000 images, and Tiny ImageNet has 10,000 images. We repeat the entire procedure five times with different random seeds and report the mean of AUROC and its standard error. Table 12 show that DLF performs best on SVHN and remains competitive across CIFAR-100 and Tiny ImageNet.
Table 12: AUROC Results on in-distribution and out- diitrii-iitiiiito detection.
Out-Of-Diste Di-Distribttion Datasets
Method
SVHN
CIFAR-100
Tiny ImageNct
Teachers
0,9403
0.8604
0,9400
small-Ens
0.9093 (0.0037)
0.8000 (0.0036)
0.7956 (0. 0023)
Hydra
0.7178 (0.0107)
0.6644 (0.0083)
0.6709 (0.0059)
LBE
0,8329 (0.0249)
0.8107 (0.0130)
0,8152 (0.0141)
Proxy-EnD2
0.8427 (0.0344)
0.8427 (0.0183)
0.8456 (0.0175)
DLF
0.9359 (0.0212)
0.8357 (0.0062)
0.8291 (0.0067)
F
Computational Cost
In this section, we report comparison of the computational costs between our proposed method and baseline method during training.
F1 Training time
First, we evaluate the training time on the CIFAR-10 and CIFAR-100 datasets. For a fair comparison, all experiments are conducted under the same hardware environment. We follow the experimental setup described in Appendix C.2 Each experiment repeated four times, and the average training times are reported in hours.
Table 13: Comparison of training time on CIFAR-10 and CIFAR- datasets.
CIFAR-10 Training Time
CIFAR-100
Method
Training Time
Teachers
10.2
17.4
small-Ens
4.6
12.5
Hydra
3.1
10.28
IBE
6.5
21.5
PrDXy-EnD2
4.6
9.2
DL.F (Ours)
4.2
11.28
As shown Table 13, our proposed DLF model also exhibits relatively short training time compared to small ensemble, Hydra, and Proxy-EnD². In the DLF model, μ4 and dg are local networks whose
34

---

supremum norm output sizes depend only on the target and latent dimensions; hence, they do not scale with the student model size. Moreover, a key advantage of DLF that it can be effectively trained using EM algorithm. As detailed in Appendix B the E-step has closed-form solution, making the overall training process computationally efficient. In contrast, LBE requires more computational resources and longer training time as the model becomes larger, which will be discussed further in the following subsection.
F.2 Floating-poing operations
For a further analyze computational complexity, we compute Eloating-poin operations (FLOPs) per training for Hydra, Batch Ensemble and DLF. FLOOPs provide hardware-independerr measure of computational cost. For example, ResNet [60] and EfficientNet [6] use FLOPss to evaluate and compare model efficiency across architectures.
Let с denote the number of classes, the batch size, and n the number of ensemble members. The latent dimension in DLF is denoted by q. Faady and Phead represent the FLOPs for one forward pass through the share body (e.g.. Wide-ResNer (55]) and one Hydra head, respectively. In the DLF model, F corresponds to the FLOPs for the small fully connected layers that produce μs and Dg. Finally, a indicates the multiplier accounting for both forward and backward passes. Then, the FLOPs per training step for Hydra, Batch Ensemble, and DLF are formulated follows:
Hydra:
FLOP = a:B( (Foudyy + n. n.eeau
Batch Ensemble:
FLOPsgE 2 aB(1+ + Fbody
where (extra cost from rank-1 matrix) usually S 0.05.
DDLF:
FLOPP (Frody + F2,0)) By² + q² + (Bog + 9²)
Note that when the shared backbone is Wide-ResNet, the corresponding Fbody is typically on the order of 10 FLOPs. In Batch Ensemble, the additional cost comes from the overhead factor (1 + e) applied to the full backbone. Although is usually small (typically less than 0.05), its effect is not negligible due to the large scale of l'oody-
In contrast, Hydra shares the backbone and only adds small cost from the lightweight head modules. Since Fieead < Fbody: the total complexity remains dominated by the shared body, even with large ensemble sizes
Similarly, although DL.F requires additional computations such as F,. and matrix operations such as Bq², q², and n(Beq + q²), their contributions are negligible since q is typically small (often less than 20 in practice) and F2 Bq², x², n( Beg + g²) << Foody. Consequently, the dominant cost in DLF also comes from the shared backbone.
G Theoretical results
We intend to similarly investigate the convergence of the sieve maximum likelihood estimation (MLE), as discussed by [62], using the results of estimating smooth functions within the sparse deep neural network (DNN) function class proposed by [63] and sieve MLE's convergence rates by [64] We will investigate the convergence rate in terms of decaying rates of the eigenvalues.
Notation For a natural number m, we define [m]] = {11, 2,.,,...nm m... For a m x q-dimensional matrix A, we denote the spectral norm of the matrix A by || [A 12 and the Frobenius norm by || Fr
that is, || = suP Azlla and || = / /TT(AT A). For a square matrix A, let Amin ( (A) denote the smallest eigenvalue of A. For two positive sequences (a,, meN and (bw)neN. we write on S bn or b, Z an there exists a positive constant C' > 0 such that o, Cb, for any n E N. We write on x by if both a, 2 bn and a, S bnn hold. For vector-valued function f -J fmm defined on a domain X, we denote the dllementiise-maxinee
by |I foo maxisism |lo maxigiSm supxex [x)]. The Hölder space of smoothness
35

---

3 >> with domain [0, 1] and radius K > 0 is defined as, letting be the smallest integer larger than or equal to 3 1,
= ca: || - K
where Cj denotes the set of s-times differentiable functions on [0, 1] and [lx denotes the Hölder norm defined by
|l (ll())()))=
max (ALmaL))NN Σ E1=10 <,
1000 ... yod fllo0
800h ... yod f() g..... 8ou (1x22)
max (0l,. E
sup
|| 2221 8-
#02
where No = 10,1,2..... Let F be given class of functions defined on X. A collection {fi : i [N]} is called d-covering set of JF with respect to certain norm || I| defined on X, if, for all E F. there exists fi in the collection such that || fill S 6. The cardinality of the minimal d-covering set is called the 5-covering number F with respect to the norm || -1I and is denoted by (6, || II). A collection ((4, u²) : [N]) of pairs of functions with 4 S is called 5-bracketing set of J. with respect to a norm ||- I||| if, for all f E F, there exists (1, u.) in the collection such that 4 S f u, and |14, u4 8. The cardinality of the minimal ò-bracketing set is called the b-bracketing number of J with respect to the norm || |l. and is denoted by (6, Fo, || II). For two probability density functions p1 and p2, let denote the Hellinger distance between them by P2 1/2
h2)1.P2) [ (z) (x)
?
G.1 Problem formulation
Let fI,..., In be n independent realizations of the Gaussian process with mean function μ (-) and covariance kernel (-.)). Suppose that we have m many d-dimensional design points D {- ..... æm } (where we omit the superscript "design" unlike the main body of the manuscript for simplicity). We assume that without loss of generality, (d) E [0, 1]d for every j E [m]] by appropriate normalization. Then we observe fIpD = (f (z) (dd) + UsI...., f((=))))
(df) + vim)" TT, where VimS are
independent Gaussian random variables with mean 0 and variance o?? Note that f fap follows the multivariate normal distribution p. = N( E,p)). where
1 (1. )uE(m) E RR. Σ p = (E, (x. ( (uE()))E(() + I E mm
Let Xæ, and (-,s(-) be the j-th eigenvalues and eigenfunctions of the kemel Σ., ordered by their magnitude and (..((-) = (:) be the sealed eigenfunctions. Then, the covariance matrix can be decomposed into the q-leading part and the low-rank part
Σ. = P.IDD + + E.>g/D + of
where 2.1D = (0. (2) )..., Φ (a (d)
TT
E mRaxq with &- (xx) = (d))())))=(a) and Σ.>/P
(Ejon (e)(x!)) ( (2))(
ae(m],uE[m) For the sake of notational simplicity, we consider the case where of is known. The proof can be extended easily for the case of unknown of.
Our aim is to estimate p" based on observations f fimm...,,ff fajm by modeling μ., and Φ. by a specially design DNN. For given fa E 1T and Σ E _ let Pp. be the density of the Gaussian distribution with mean μ and covariance matrix Σ. For give mean function (a(+) and a vector of q many scaled eigenfunctions (.(.) parameterized by 0, we consider PH p. where
Help = (2o( (d) [u[[m] E R.
Enpp Polp Pop + If E gmmx
with 0 = (0)( ... 2 ( T. We model (1u()), (c(.) by a DNN with + many outputs and estimate by (sieve) maximum likelihood estimator (MLE) that is defined as 0
36

---

argmaX en (0), where
la (0) = log
71 2
Σ
pDl
E 1=1
(p - (aip) " [o i/D - pa)
Here, the sieve ,n depends on the architecture of DNNs. We will prove that the estimated Gaussian distribution converges to the true Gaussian distribution p. in probability as n + oo under regularity conditions while D is fixed, provided that the sieve θ, is selected appropriately.
G.2 Results
For the sieve Øx we consider set of parameters, whose elements are in [-1, (following (63 ).
of sparse DNNs with Ln many layers, In many nodes each hidden layers, S,, many nonzero elements and qn + 1 output nodes. When we would like to clarify such architectural choices, we will sometimes use the notation , ((L.: Sn, qr). For a DNN parameter 0 E On. we let go be the corresponding realized DNN function, but for technical reason, the outputs of this function are truncated [- B, B), so that Jn is a function from IR to [-B, B] We denote by {gn 8 E O, J. Such sparse DNNs have been considered in many previous studies [e.g.. investigate the asymptotic properties of DNNs.
g n
62
Given the design points = {0) (..
amm ), for each DNN parameter O, with the realized
DNN go (90,1..... 90.0g. +1)+ define the m-dimensional vector Mayp (90, (x.( ( Due(m)) and
Ldd PojD I where p (go.++1 (d)) )) wE Jel]
m symmetric matrix Egp T
For notational simplicity, write Payp PHoID: Eapes the density of the Gaussian distribution with
mean MelD and covariance matrix EgiD- We let the class of such Gaussian distributions P(O.; D) (PoaD A E O.))
Lemma G.1. Let D be an arbitrary set of m design points. There exists an absolute constant C₁ > 0
such that for any 8 (0, C1/qn), the following holds
o28
log-N (6, p(); D), S log. 26 max (2mq B, Vma.}" G(0,), 1l1o).
Theorem G.2. Suppose that μaa and H-.j.j = 1,... belong to H& (K). Consider the sieve MLE ÿ = P3 over O, = (Ln, Tn, Sn, qn) with In x log n and rn, Sn, In S n. Define
Sn
-23/d
(logn) n)²
(ct))² dn log n
13
Sn
n
j>an
Assume that In + 0o, Cn + 0 and n(c))) 00 + 00. Then, we have
P. (h (p.p.) > C2e +
as for some absolute constant C2 0.
We can make <t converge to by letting 9n and S, diverge with a appropriate speed provided the eigenvalues } 4u. converge 0 sufficiently fast (e.g. Xy exp(-j)).
The upper bound of Theorem G.2 is about the estimated Gaussian distribution at the design points Ddesiga, For prediction, we need an upper bound of the estimated Gaussian distribution at new point æ. The following theorem, whose proof is given Appendix G.4| provides an upper bound.
Theorem G.3 (Upper bound at a new input). If ja and & 1,... In are Lipschitz, the probability of
di (pe pe, x S di (P Pr,),))) + C2llx æ(1ill +1]]|| for a certain positive constant C3 and
sup di (p pe. S C2e =EDoneggn
converges to as -> so, where di h) lg(z) - Mz))ldz for given two probability densities
on R.
37

---

G.3 Auxiliary Lemmas
Before proving Lemma G.I we introduce the following two lemmas.
Lemma G.4. En - Σ1 is positive definite, then
Py, zi (x)
1 (1112-1))² - 1012-11) 112 Ei) -11 m1)
V
exp
P1z Eg (x)
Proof. Define n. = (Σj - En 1 (2)
[ M1 En M2). Note that by assumption En - E1 is
invertible, and thus we have
(x 14) Ej (x 11) - (x 142 T ΣΦ ( 12)
= -xx- (l,) -
14) - p. [Σ) - Σこ + 14 ET'M 11 112 ΣΦ 122
The sum of the second and third terms is further simplified as
m. (2) - -, ), + 14
M1 14 |! Σng 14
-r, (2) Σい ur +
14 - 12 2 142
= -Zi 11 Σし 4.2
[Σ] - - Σこ [E] f1 En M2) + + Ei
- M1 1 Eng P2
-- (I- Σ و.
11 - 11 ΣΦ
[Σ2Σ]' -1) 12
+ +2 Eg (ET - Σ
Σد 14 + u Σj #1 - U2 En H1
= -u M E -,- 11 - 12 Σこ Σ (Σ2KI 11) -1
+ 2 Σg (ΣT ΣØ- ΣТ 1/1
=1M (E2- Σn) 11 U2 (E2- -E1) f12 + 22p ( (E51-IA 1 k4
= (w4 - 122)
(Ee - E1) -1 (m1 -M2)
Therefore, we have
PH: 2, (z)
Σ
- (c 14 1 Σ (x- 141) - -(x - 112) T Σの (x - 122)
exp
Ppg. E2 (x)
Σ
2
Σ2
(z m.) (ET Σσ (x= - -..)
exp
Σ1
2
+ 1 (11 m₂) (22 - E (H, - (22)
Σ
2 (Mm
1
12) - E1)-t(1 - -H2))
exp
Σ
which completes the proof.
Lemma G.5. Let a2 be the lower bound of the minimum eigenvalues En and Ez. S
co for given >> O, the following inequalities hold:
[(1+ - -))))>)>>>>
2 o² (c (1 + 111)
and
x (1+ 2 (5 - 0)02
1 1+ C ||>||
where Ç = 30
Proof. The first inequality holds because x
((1 + <) - Ei)x 2 (1+ + ()x T (En- 10--), E1)x + Cx T E1Φ
2 -1 + ()] Eg - -Ellell<112 + co
> (6o² - (1 + ()coo))
The second inequality follows similarly.
38

---

G.4 Proofs
Proof Lemma G.1
Proof. Fix > O. Let (91...., .N with N = (e,, ((On), (1x)) be a e-covering of ((O,). For each i E [N], let 0g the parameter of g, and let μ4 Ma,ID and Σ; Eg.ID. Then for any 0 E On letting μ Mayz and Σ Egy for simplicity, we have
| Mill2 S Vmc.
and
|Σ Eill S (1 -,, )l2
< 21 + | || DFIlF
S 2mqn Bc,
where the third line follows from that || 81ll2
< Vmq, B. Now, let ô
mmax(2mq, /ma, h/o. so that 1 I o. and |Σ
2: I o28. Let 35. Then
will show that [4, uz] is a Hellinger bracket of the density (x) when we define
u, (1 (1+22) 2() PM, (1+(EE
1, = (1 2() (1t))))))
Then by Lemma G.5 (1 ()Σ,
Σ and E (1 ()-1))) E; are both positive definite. So by Lemma
G.4 we have for any æ E I
Pu,x(2)
2C) m m
(1 + C)
1 ( (1)=
Σ) - - (1)
exp
(x(x))
Σ
By Lemma again, we have [[(1+))) - (026) for any sufficiently small Moreover, by Weyl's inequality,
S [(( -2)(2-()0)--
1(1 + E
o.0) 0z
(1+
1+
$ (1+ 2()
Σ
Thus have
p,)x(x)
11 14-4:13 11l1
(1+ (1+ 2m/ exp
1x(x))
2028
Using the incquality log(1 + 2) 2 z/2 for E [0,2], we have
log
(()) (x)
m1
log(1 + 2C) +
11 14:11
2
20.5
<
m
+ m
(a_8)2
2
2028
3m
+ m
< 0,
2
which implies (x.) ux))). Similarly, we also have
(xx) (1 + 22)-2()/-- m/2 exp
|114 14,11
(x))
2028
and 1xxx) S Pu.x(x) for any R. We now bound the size of the bracket. Note that
12, u:) = (1 + 24) + 2() - ((2 122 12 (1t).. 1x.)).
39

---

Due to the inequality 22/22 2 =logg1 2) for any 2 2 0,
- KI.
ml 1ev(-logg log(1 + ()² (1 + 6)² -1)
((1 + C)² 1)
m (C + (2/2)²
9
mç²
4
Moreover, by taking e sufficiently small so that < 3/ m, we have
(1 (1+ + (1 2() - 2(1 + 2()
4mp(1 m
s 4m<(1 + 2/ 2/(3m))mm- S 4/2
Thus, we have (1, u:) S (9/4mC + 4,2/3) < (3/4 / + 42233 S 268. Hence, redefining constant as 265 - 5, we complete the proof.
L
Proof of Theorem G.2
Proof. The proof follows a similar reasoning in the proof of Theorem 3 in [62]. which is based on Theorem 4 in [64] with 0+. We divide the proof into the following four steps.
Bounding the estimation error: Check Eq. (3.1) of 64 For the class of DNN parameters O, O Fwy Sn, 4m), by Lemma 5 in 63), we can get the following covering number bound
log ((,)))
S (S,, + 1) log (25-1 (L,, + Iyd + 1)?
2 Sn log(nd-1)
for any 8 > 0. Applying Lemma G.I for 0 < 8 < C1/9n, we have
0.6
26 mamqq vmo.) } g(O.), l1x0 log,N( P(O.; D), h) s log
< Sn log(no-
Moreover, for a positive constant such that v2e S C1/qas we have
22/2 V (5, P(O.: D),
& cv C Ln log(ne-1).
Then the above display is bounded by 1222 up an absolute constant when we take En VS, (log ()ogn)2/ n)2/n i Ln x log n. Thus, Eq. (3.1) of [64] is satisfied.
Bounding the Kullback-Liete approximation error We first note that for any 0 e On, we have
KT. (PalIPpoip:
log
EgeD
E-1 ./D
Tr EgIDE.T - TTm) Tmm))
1
T
+
(P - M.ID
Σ
Loo0 1k/D))
2
log
Im + E
1/2
[Ep
1/2
2
+
1
Tr
1/2
(Eep
EnpD)
112
+
1
2
20
1
<
Eg
E 1-111
d
20
1
1
<
Enpp
E.Ipl
1|2 +
Mapo
M.Ip
(6)
4
20
40

---

To bound the two terms in Eq. (6) we use the well-known results about the approximation ability of sparse DNNs Hölder smooth functions [e.g.. Theorem of 63]. Namely, there exists @t 2 such that
max [loot,1 mells max jE(an]
1001,+11 - o x (5m 8/a
For the first term in Eq. (6) we define
1/2
Kn = 12
Σ X3
12
and establish the upper bound
[ Euppll <
DD 4 PD P.plpll + Kn
|1
+ / P.ID))
1| Kn
$ Pdip 2.p |/ + 2 1|ا P0IID P.ppl Knn
But by Eq. 7
0 D 2 p.pll
S v 4n Sn Ln) B/d which converges to 0 as n + 0o. This 1 01
implies that, as o pll2 is bounded, eventually. Therefore, have
1 $ 12 is smaller than 2/| $ P.lplllll Boup P.ppll
Egup - E.ppll? S 9n (S L.) 23/d + Kn
Moreover, by Eq. 7 it is immediate that | - 1= S (Sn Lm) - 23/d Adopting the notation of [64]. we have on 4n L., -28/d + Kn x 4n (Sw/ log n2) 23//d + Kn-
Bounding the Kullback-Lieble variation The last ingredient of the proof is to bound the so-called Kullback-Lieble variation defined as
KLV (pollPreID: Eopo) > R-RRun log
(=(x)
p. (w))x.
(
We will find a suitable network parameter to get manageable upper bound of the above, which is denoted by Tn in [64]. We use Lemma G.5 for this purpose. As in the argument used in the previous step. we can find network parameter H* such that
10 12 E.pplE S &",,, for some absolute 1
constant C" > 0. As In - D0, we have ş = Amia 1
$ 01 p
O. We then construct the network 4 PoipD. Then, by Weyl's inequality,
gt satisfying p = PlttID and & DD/pDD = (1 + (1
/
1/2
Amin Ea Eupp)
= Amin EEr E,i + (1+ C* o 0t $ @
2 Ami ((1 + C')6,, 60 p)
Eg - E.
2 (1 + C²) |E E.pol 2 5n.
which implies that Epr D E,jo is positive definite. Using Lemma
G.5 have
(x(x)
Σ Eg D Emppl
ク (Mo-)D - M.Ip) (E - - -k.lp))
exp
Protip: (x)
Σ y= D
-
exp
Σ E.ppl
25,
< (1 + Eggip - E.I
E.1pll//o2)m1
< (1 + [[(1 C²)6,// / ( o pll 1 Emppl2//22mm?
(1 (1+ + (Bmq.. ( //22 (Eo + (C.)//66.//0"""
where we use Weyl's inequality for the third inequality. Therefore, we have
KLV ppllPpottp S log(1 + 2/2 v/))..)
41

---

Adopting the notation of 64 we set = log(1 + (4x) 1/2
Combining the pieces together Let e = en V Vōn- Then by Theorem 4 of [64], there exists an absolute constant C" > 0 such that
P- (p.p..) > C2c,i & e- C"n(e
Tn
(TE,) }2
which tends to zero as n - oo by the assumptions cn - 0 and n(e,)) + 20.
Proof of Theorem
Proof. Since the total variation norm is upper bounded the Hellinger distance, the total variation norm between ÿ and p. is also upper bounded by C2e. with probability converging to 1. In turn, by the definition of the total variation norm, Du d1 (f P=, is upper bounded by C2t, with probability converging to 1. Due to the Lipschitz condition of μ, as well as px, E,, there exists a constant L > 0 such that d1 (Pa. Par) I Lix x' and d1 (Pr. p.,w) L|| æ'll for any and x' in Rd. Finally, have
di (Pw P=,
d1 (Par Pany) ) + d1 (Pmay) (., + d1 (py d1 (Pany) pr, + 2L/x 2(nill
C2a + 2L/x 2(1lll
with probability converging to The proof is complete by letting C3 = C2 and C4 2T.
42


Reviews:
[{"review_id": "d5Ti73vVpR", "reviewer": "Reviewer_vWb7", "summary": "The paper considers a problem of knowledge distillation of deep ensembles (though it is mentioned it could be applied to Bayesian NNs as well) and proposes a novel method for knowledge distillation based on a deep latent factor model where the latent factors follows the normal distribution. Parameters of the model are estimated with the EM-algorithm. The estimate of the parameters is based on predictions of the teacher ensemble models at so-called design points. The paper proposes a theoretical analysis for the choice of the design points.", "questions": "1. Could you please elaborate about why Dirichlet distillation has been left out of the comparison? \n2. Could you please comment anything about the computational cost?\n3. How are predictions are made and uncertainty is estimated with the proposed model?", "limitations": "The authors have discussed the areas of future work in the conclusion which can be considered as discussion of limitations to some extent, but it seems it could be more explicit discussion of limitations. No discussion about potential negative societal impact of their work is discussed, though it is a theoretical work which doesn't seem to have a lot of potential negative societal impact.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n*Quality*: The paper claims are supported with rather extensive empirical evidence, covering regression tasks, classification of image data, fine-tuning of LLMs. There is a theoretical base for the choice of design points. However, please see below.\n\n*Clarity*: The paper is generally easy to read, but please see below.\n\n*Significance*: The paper considers an important problem the experiments show promising results of the proposed methods over baselines.\n\n*Originality*: The idea appears to be novel. \n\n**Weaknesses:**\n\n*Quality*: \n\n* I am not convinced why Dirichlet distillation was left out of empirical comparison. This line of work seems like a very interesting baseline as the only other distribution distillation. Please see below for the details, why I am not convinced. \n* The main motivation of the paper is to provide a cost effective alternative to a deep ensemble, yet there is no computational cost discussion in the paper. \n* Image experiments are done on CIFAR-10/100 only, which is not too extensive. Regression datasets are UCI only, which is again is not very large and hard.\n\n*Clarity*: The paper is seemingly easy to read and follow, but it seems to be missing important details, or maybe I am missing them. After reading both the paper and the appendix, I realise I do not know how the model is making predictions and actually estimate uncertainty, which is the main selling point of the paper. I.e., the paper describes in details (well done) how to estimate the parameters with the EM-algorithm, but I am not clear how to use those parameters further for predictions.  \n\n*Originality*: The paper puts the proposed work rather well in the literature on knowledge distillation, but nothing at all is discussed about deep latent factor models existing in the literature. I believe it helps for the context and also to clarify what exactly is the novelty proposed in the paper. \n\nOther comments/suggestions:\n1. Lines 136-140. q is not defined\n2. Lines 169-171. $\\mathbf{z}_1, …, \\mathbf{z}_n$ are not defined\n3. Line 205-206. I am confused about the claim “It is known that Dirichlet distillation is inferior to one-to-one distillation [11, 13]”. I am not too familiar with this literature, but quick look at the provided references do not fully support this claim. Namely, [11] does show some inferiority of Dirichlet distillation it is not overwhelming and shows some trade-off in terms of computational costs. Whereas, I may be missing something, but I can’t see the direct comparison with Dirichlet distillation at all in [13]. The comparison seems to be with just Knowledge distillation [Hinton et al.], whereas [Malinin et al.] only mentioned in the related works but not in the direct comparison. Again, I am sorry if I am missing something as I only briefly looked at these mentioned references, but they do not seem to support the claim that Dirichlet distribution can be ignored for comparison with the proposed work. \n4. Line 271. The authors of DistilRoBERTa ask to cite this work Sahn et al. “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”, which is missing. (I do appreciate the link to hugging face as the paper covers a different distillation)\n5. Table 2. It seems the proposed method is referred to as KD here, which is very confusing considering it has been referred to as DLF before that.\n6. Figure 2. X-axis is not labelled.\n7. Line 303. “significantly reduced inference costs” – the claim is not confirmed by any theoretical or empirical findings.\n8. Lines 305-306. “Bayesian DNNs are known to be superior to deep ensembles in uncertainty quantification [25, 26, 27]”. This claim again is too strong and some argue the opposite and the provided references do not fully support this claim. [25] doesn’t show undouble inferiority of deep ensembles, [26] doesn’t compare to deep ensembles at all, [27] does show more undouble inferiority of deep ensembles, but it seems to be for the specifics of node-sparse models (sorry, I am not familiar with this), as it contradicts to findings on the same datasets by Ovadia et al. \"Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift.\" Advances in neural information processing systems 32 (2019). And Ashukha et al. \"Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning.\" International Conference on Learning Representations. (2020).\n9. Formulas are not numbered\n10. Line 579. Missing reference for the Adam optimiser\n11. Section 3.4. It is not reading well. Firstly, the theoretical start is quite heavy and, personally, I would prefer having some kind of high-level of motivation before diving into maths. Secondly, the leap from the design points being similar to test data to validation data being a good candidate should be elaborated. As validation data as described follows the training data distribution which is not necessarily the same as the test data distribution. \n12. Lines 612-614. L is overdefined. It has been already defined as a lower-triangular matrix for a multivariate case definition. \n13. Appendix C.5. Could it be really true that all the experiments were executed using only CPU?\n14. Lines 660-663. What are mixup samples?\n15. Figure 8. What data is used in this experiment? How would the authors explain why DLF obtain better results than small ensembles? \n16. Table 3. MRPC acc results. Bold results for KD is misleading as small-Ens has the same average\n17. The provided code does not cover all the experiments from the paper.", "was_revised": true, "final_justification": "I have read the other reviews and authors' responses. The authors have addressed most of my concerns therefore I have increased my score.", "rebuttal": "Thank you for your detailed and thoughtful review. We appreciate the time and care you took in carefully examining various aspects of our work. Your insightful comments helped us identify points for clarification and further improvement.\nThank you for your detailed feedback on items 2, 4, 5, 6, 9, 10, 12, 13 in the “Other comments/suggestions” part. I will address these points in the revision.\n\n# Weakness\n\n## Quality 1.\nIn the original manuscript, we did not include Dirichlet distillation (Proxy-EnD²) in the experiments since the papers for Hydra and BE reported that their algorithms are superior to Dirichlet distillation. We focused mainly on comparing these two deterministic distillation methods with our DLF. \n\nIn the Table below, we add Dirichlet distillation. In addition, we add a new distribution distillation method called Ensemble Distillation via Flow Matching (EDFM; Park et al., 2025), which we found after we submitted our paper. Note that DLF outperforms Proxy-EnD² and EDFM with large margins in terms of the accuracy and NLL while DLF is not much worse than Proxy-EnD² in terms of ECE.\n\n|dataset|method|Acc(%) ↑|NLL ↓|ECE(%) ↓|\n|--|--|--|--|--|\n|CIFAR-10|Teachers|94.24|0.1539|0.9|\n||small-Ens|92.87|0.2377|3.93|\n||Hydra|93.16|0.2660|4.15|\n||LBE|93.25|0.2480|4.11|\n||Proxy-EnD²|90.92|0.2861|2.08|\n||EDFM|90.62|0.2858|2.78|\n||DLF (Ours)|93.40|0.2246|2.79|\n|CIFAR-100|Teachers|81.36|0.7167|1.41|\n||small-Ens|79.29|1.0413|12.90|\n||Hydra|77.42|1.2912|12.70|\n||LBE|79.58|1.0110|13.42|\n||Proxy-EnD²|67.62|1.2355|7.35|\n||EDFM|64.17|1.6741|11.35|\n||DLF (Ours)|79.68|0.8974|9.45|\n\nWe want to emphasize that superior performance of DLF compared to other distributional distillations is not achieved by chance. Instead, there are various advantages of DLF compared to Proxy-EnD² and EDFM. We think that there are three properties that any desirable models for distribution distillation are expected to have: (1) large support, (2) easy and stable learning algorithm and (3) flexibility to be applied to downstream tasks. Below, we explain why DLF is desirable in view of these three properties.\n\n- DLF is a Gaussian process, which is well known to cover a large class of distributions (Garriga-Alonso et al., 2019; Neal et al., 1996; Calandra et al., 2016; Huang et al., 2015). Even if teacher ensembles are not Gaussian, a Gaussian process can approximate them closely. In addition, DLF treats teacher ensembles as random functions rather than random vectors as Dirichlet and Flow matching distillations do. See our reply to Question 1 of Referee 4 for detailed discussions for this point.\n\n- Existence of easy and stable learning algorithms is a critical fact when choosing an algorithm for distributional distillation. Learning the DLF model is numerically easy and stable since the efficient EM algorithm is available. It is well known that estimating the parameters in the Dirichlet distribution is difficult (Minka et al., 2000; Chai et al., 2013) and thus it would not be easy to learn the Dirichlet distillation model which would be a partial reason for inferior performance. For Flow matching distillation, pretrained features should be fed when learning. That is, end-to-end learning is not possible. In contrast, DLF is learned in an end-to-end fashion (i.e. feature vectors and distribution are learned simultaneously).\n\n- Flexibility is also an important aspect. As we have shown in Section **5.3**,\nthe learned features by DLF is superior to those by a single DNN or Hydra for downstream tasks.\n\n- To sum up, the idea of estimating the distribution for teacher ensembles is not new but an efficient implementation is not an easy task and DLF is such a method.\n\n\n## Quality 2.\nFor inference overhead, we reported the number of parameters in each model in the manuscript in Section C as inference cost is roughly proportional to the number of parameters. As the referee pointed out, in this rebuttal, we measure the computational costs not only for the   inference phase but also the training phase in the table below: (1) Multiply–Accumulate Operations (MAC, measured in MMac) for inference, (2) inference latency (ms), and (3) training time (hours). We did not include EDFM since it uses pretrained features (instead of learning features).\n\n|Method|CIFAR-10 MAC|CIFAR-10 Latency|CIFAR-10 Training Time|CIFAR-100 MAC|CIFAR-100 Latency|CIFAR-100 Training Time|\n|-|-|-|-|-|-|-|\n|Teacher|224|7.1|10.2|3400|7.6|17.4|\n|small-Ens|108|4.1|4.6|224|7.3|12.5|\n|Hydra|27|1.1|3.1|56|1.8|10.28|\n|LBE|108|2.4|6.5|224|4.3|21.5|\n|Proxy-EnD²|27|1.2|4.6|56|2.1|9.2|\n|DLF (Ours)|27|1.5|4.2|56|2.6|11.28|\n\n## Quality 3.\nFor a large and hard problem, in Section 5.2, we have conducted experiments using RoBERTa (approximately 355 million parameters) to evaluate our method. As the referee suggested, we analyzed Tiny-ImageNet, a 200-class subset of ImageNet whose results are provided below. DLF still outperforms the other baselines even including small ensembles.\n\n|dataset|method|Acc(%) ↑|NLL ↓|ECE(%) ↓|MAC|Latency|Training Time\n|--|--|--|--|--|--|--|--|\n|Tiny-ImageNet|Teachers|68.74|1.2806|3.22|166571|51.6|32.9|\n||small-Ens|57.67|1.7887|9.36|77382|26.0|53.6|\n||Hydra|55.94|1.7765|6.49|19353|6.3|24.2|\n||LBE|54.44|2.2804|6.04|77377|29.0|60.4|\n||DLF (Ours)|61.92|1.5450|3.06|19353|7.2|34.6|\n\nWe will conduct further experiments on larger-scale datasets (such as ImageNet) during the revision phase and include those results in the updated manuscript.\n\n## Clarity.\nAs explained in the Introduction, the predictive distribution is obtained by $\\hat{p}(y|\\mathbf{x})=\\sum_{i=1}^n p(y|\\mathbf{x},\\theta_i)/n,$ where $\\theta_1,\\ldots,\\theta_n$ are ensemble members. For DLF, as shown in Figure 1 (“Inference on new data”), we can generate $\\theta_1,\\ldots,\\theta_n$ by  sampling $n$ independent latent vectors  $\\mathbf{z}_1,\\ldots,\\mathbf{z}_n$ from $\\mathcal N_q(\\mathbf 0,\\mathbf I_q)$, forwarding each sample through the network to have ensemble members.  All of the uncertainty measures are calculated based on $\\hat{p}(y|\\mathbf{x}).$ We will state explicitly this procedure in the revision.\n\n## Originality.\nDLF is a kind of the conditional factor model (Chen et al., 2022; Gagliardini et al., 2019) where $\\mathbf{Z}$ is the factors and $\\Phi_{\\theta}(\\cdot)$ is the factor loadings depending on covariate $\\mathbf{x}.$ Even though there are much literature for the conditional factor models, as far as we know, there is no previous attempt to use deep neural networks for the conditional factor loadings (most models use linear factor loadings) presumably because the model is too complex to fit noisy data. Our key contribution is to suggest DLF to ensemble distillation. As we explained in the reply of Quality 1, our suggestion of using DLF is scientifically well motivated.\n\n# Other comments/suggestions:\n## 1.\nIn lines 137-138, we wrote $\\boldsymbol{Z} \\sim \\mathcal{N}_q(\\mathbf{0}, \\mathbb{I}_q)$ which indicates that $q$ is the dimension of $\\boldsymbol{Z}.$\n\n## 3 .\nPlease refer to our response to Quality 1.\n\n## 7 .\nPlease refer to our response to Quality 2.\n\n## 8.\nWe agree that the wording in Lines 305–306 was too strong. We mentioned Bayesian DNN since we tried to write that the learned DLF can be used as a prior for on-device update of the posterior. Due to the page limitation, we deleted this sentence. We will clarify this point in the revision. \n\n## 11.\nThe main motivation of the theoretical result in Section 3.4 is to answer the question about the choice of good design points for distribution distillation. Even if there are many strong assumptions (Gaussianity of teacher ensembles and so on), the theoretical result suggests that the design points whose distribution is close to that of test data are promising.\n\nRegarding the second point, we assume that the distributions of the training and test data are the same, and suggest using validation data randomly sampled from the entire data. To support this choice, we have empirically confirmed that using validation data for the design points works well in Appendix D.1.1. We will clarify this point in the revision.\n\n## 14.\nThe entire training dataset is denoted as $\\mathcal{D}^{\\text{train}} = \\{(\\boldsymbol{x}_1,y_1), \\ldots, (\\boldsymbol{x}_n,y_n)\\}$. For each $i$-th sample in the training set, another sample $(\\boldsymbol{x}_i^c, y_i^c)$ is randomly selected. A new mixed sample $(\\boldsymbol{x}_i^m, y_i^m)$ is generated by linearly combining the two samples. $(\\boldsymbol{x}_i^m, y_i^m) := \\lambda_i(\\boldsymbol{x}_i, y_i) + (1-\\lambda_i)(\\boldsymbol{x}_i^c, y_i^c), \\quad \\lambda_i \\in [0,1]$. We will add more explanation of mixup samples in the revision.\n\n## 15.\nThe experiment in Figure 8 uses the Boston Housing dataset, as stated in line 642.\n\n## 16.\nWhile the values are displayed as identical to two decimal places, DLF slightly outperforms small-Ens (83.094 vs. 83.0915). For clarity, we will report results to three decimal places in the revised version to avoid any potential misunderstanding regarding the bold formatting.\n\n## 17.\nWe plan to release the full code and scripts for reproducing all experiments in a follow-up update.\n\n# Question\n## 1. \nPlease refer to our response to Quality 1.\n## 2.\nPlease refer to our response to Quality 2.\n## 3.\nPlease refer to our response to Clarity.\n\n# Reference\n- Chen, Q. (2022). A unified framework for estimation of high-dimensional conditional factor models. arXiv preprint arXiv:2209.00391.\n- Gagliardini, P., \\& Ma, H. (2019). Extracting statistical factors when betas are time-varying. Swiss Finance Institute Research Paper, (19-65).\n- Please see the reference list of Reviewer e6dg."}, {"review_id": "TnCg9XmDND", "reviewer": "Reviewer_Lsh6", "summary": "This paper proposes Gaussian distillation, a distribution-distillation technique that treats every teacher in a deep ensemble as a realization of a Gaussian process whose mean and covariance are parameterized by a student network called the Deep Latent Factor (DLF) model. The parameters are learned with an EM procedure that integrates out latent factors, and ensemble members for inference are sampled from the estimated process. The empirical results on regression and classification tasks show the advantage of Gaussian distillation over several benchmarks, however, this paper failed to compare with current state-of-art methods, and are limited on small datasets such as small-to-medium vision and tabular benchmarks. Moreover, the computational burden and the stability of EM optimization are also concerns.", "questions": "•\tWhy “closer to the coverage probability of a teacher ensemble is better” in table 1? normally closer to nominal value should be better.\n\n•\tIn real-world applications, teachers models are pre-trained and likely with varying structures and trained over not necessarily the same data. I am interested in such settings, in the performance of the method proposed.\n\n•\tThe authors should attempt and evaluate the methods that have been suggested with larger dataset, such as ImageNet, and larger models.\n\n•\tThe comparison of state-of-art baselines are necessary.\n\n•\tHow does DLF scale to high-dimensional outputs (e.g., pixel-wise regression)? \n\n•\tOn line 187, any specific reason to choose to set \\sigma generated from the inverse gamma distribution?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Quality\n\nStrengths:\n\n•\tThe paper is technically sound, with support through empirical explorations and theoretical confirmation.\nThe suggested scheme is complete, aided by particulars such as design-point selection, choice of initial point etc.\n\nWeaknesses:\n\n•\tScalability of EM: The EM is computationally slow. A discussion of complexity would explain how the technique scales to, e.g. ImageNet. Computational time comparison with other baselines would also be required.\n\n•\tThe experimental part only compare techniques like Hydra, which are somewhat obsolete, and do not compare with a few recent distribution distillation baselines like packed-ensembles https://doi.org/10.48550/arXiv.2210.09184 or Bayesian neural-network distillation.\n\n•\tThe authors proposed the methods in the field of distribution distillation methods, and distribution distillation methods are introduced in, e.g., Dirichlet distillation, is introduced in section 2.2, but not compared in the experiment section.\n\n•\tNo Large-Scale Tests: Small models are used in experiments. Not known whether DLF scales to billion-scale LLMs.\n\nClarity\n\nStrengths:\n\n•\tThe paper is well written and well structured.\n\n•\tCore ideas, such as viewing teacher outputs as realizations of a DLF process, are illustrated clearly.\n\nWeaknesses:\n\n•\tClarity suffers from heavy notation and frequent cross-references to appendices for critical arguments.\n\n•\tThe theoretical findings are approximately discussed in the main part. Although some details are given in Appendix, it is not very well-organized. Also, the authors give many mathematical result in appendix, but the insights or conclusions gained are not clearly discussed in the main text. This issue also exists in the part of ablation study.\n\n•\tIt is not clear how authors define LLM in section 5.2. In today’s standard, models like RoBERTa are usually not called LLM.\n\nSignificance\n\nStrengths:\n\n•\tCompressing deep ensembles and retaining calibrated uncertainty is a timely practical problem, and the Gaussian distillation is a principled solution to it.\n\n•\tThe technique developed is applicable both to regression and classification methods, while original KD is designed for classifications.\nWeaknesses:\n\n•\tMost benchmarks are small to medium, the compared methods are naive, and the gains over small-Ens are modest, especially the coverage rate proposed in Table 1.\n\n•\tThe LLM example relies on DistilRoBERTa rather than a modern billion-parameter model, so the reader is left to extrapolate the method’s benefit in truly large-scale settings.\n\n•\tThe assumptions of the theorem are strong. The authors should be able to clearly illuminate in what way theoretical assumptions (e.g., strong convexity, good initialization) might constrain practical applications.\n\nOriginality\n\nStrengths:\n\n•\tThe paper is novel in framing distribution distillation as a latent-factor Gaussian process parametrizable through a student network.\nWeaknesses:\n\n•\tThe method integrates many components, Gaussian latent factors, EM optimization, and factor-analytic covariance parameterization, from established literature. Its move from Dirichlet to Gaussian distribution distillation is a careful extension, but could be perceived as incremental rather than a complete paradigm shift.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thorough review and the thoughtful questions you raised. Your comments not only helped us clarify key components of our work but also encouraged us to consider valuable directions for future extensions. Thank you for engaging deeply with our paper.\n# Weakness\n## Quality 1.\nA novelty of DLF is that it can be learned easily using the EM algorithm. As detailed in Appendix B, the E‑step has the closed‑form solution, so training time is not computationally demanding. To confirm this argument, we measure the computing times for the training as well as inference in the table below: (1) Multiply–Accumulate Operations (MAC, measured in MMac) for inference, (2) inference latency (ms), and (3) training time (hours). We will add this result in the revised manuscript.\n|Method|CIFAR-10 MAC|CIFAR-10 Latency|CIFAR-10 Training Time|CIFAR-100 MAC|CIFAR-100 Latency|CIFAR-100 Training Time|\n|-|-|-|-|-|-|-|\n|Teacher|224|7.1|10.2|3400|7.6|17.4|\n|small-Ens|108|4.1|4.6|224|7.3|12.5|\n|Hydra|27|1.1|3.1|56|1.8|10.28|\n|LBE|108|2.4|6.5|224|4.3|21.5|\n|Proxy-EnD²|27|1.2|4.6|56|2.1|9.2|\n|DLF (Ours)|27|1.5|4.2|56|2.6|11.28|\n## Quality 2.\nWe would like to clarify that packed-ensembles is not a knowledge-distillation method, but rather an efficient ensemble construction technique. We agree that including a recent distribution-distillation baseline would strengthen the evaluation. We have therefore added experiments with “Ensemble Distribution Distillation via Flow Matching” (EDFM; Park et al., 2025) which we found after the submission. \n## Quality 3.\nIn the original manuscript, we omitted Dirichlet distillation (Proxy-EnD²) from our experiments because the Hydra and BE studies demonstrated superior performance over it; instead, we focused on comparing those two deterministic distillation methods with our DLF.\n\nIn Table below, we add Dirichlet distillation and EDFM. Note that DLF outperforms Proxy-EnD² and EDFM with large margins in terms of the accuracy and NLL while DLF is not much worse than Proxy-EnD² in terms of ECE. \n|dataset|method|Acc(%) ↑|NLL ↓|ECE(%) ↓|\n|-|-|-|-|-|\n|CIFAR-10|Proxy-EnD²|90.92|0.2861|2.08|\n||EDFM|90.62|0.2858|2.78|\n||DLF (Ours)|93.40|0.2246|2.79|\n|CIFAR-100|Proxy-EnD²|67.62|1.2355|7.35|\n||EDFM|64.17|1.6741|11.35|\n||DLF (Ours)|79.68|0.8974|9.45|\n## Quality 4.\nWe agreed that RoBERTa is not of billion-scale LLMs. However, RoBERTa is used popularly for studying finetuning LLM (Yang et al., 2023). We could try to distill larger LLM but it would take significant amount of times. In our experiments, we only distilled the LoRA part of RoBERTa while we used the pretrained distilled RoBERTa for distillation of RoBERTa. We are trying to apply DLF to large LLMs such as (Liu et al., 2024) but we do not think we can get the results by the due date of rebuttal. If possible, we will report the results in the discussion period.\n## Clarity 1.\nWe are sorry for notational complexity. Due to page limitation, we moved most of notations into Appendix. In revision, we will try our best to improve the readabilty.\n## Clarity 2.\nThe main motivation of the theoretical result in Section 3.4 is to answer the question about the choice of good design points for distribution distillation. The theoretical result suggests that the design points whose distribution is close to that of test data is promising. We will polish Section 3.4 more in the revision.\n## Clarity 3.\nWe used the term LLM for RoBERTa since the previous literature (Yang et al., 2023) use this terms. It would be better to name it just SLM (smalle laguage model) or a high-capacity pretrained transformer. We will clarify this terminology in the revision.\n## Significance 1.\nFor a large and hard problem, in Section 5.2, we have conducted experiments using RoBERTa (approximately 355 million parameters) to evaluate our method. As the referee suggested, we analyzed Tiny-ImageNet, a 200-class subset of ImageNet. DLF still outperforms the other baselines even including small ensembles. \n|dataset|method|Acc(%) ↑|NLL ↓|ECE(%) ↓|MAC|Latency|Training Time\n|-|-|-|-|-|-|-|-|\n|Tiny-ImageNet|Teachers|68.74|1.2806|3.22|166571|51.6|32.9|\n||small-Ens|57.67|1.7887|9.36|77382|26.0|53.6|\n||Hydra|55.94|1.7765|6.49|19353|6.3|24.2|\n||LBE|54.44|2.2804|6.04|77377|29.0|60.4|\n||DLF (Ours)|61.92|1.5450|3.06|19353|7.2|34.6|\n\nBecause “small-Ens” consists of multiple models, it uses far more parameters. Although its accuracy gains are modest, DLF matches or exceeds its predictive performance, delivers higher coverage, and does so with only a fraction of the parameters and computation. We believe this efficiency–performance balance is exactly the strength of DLF. Please note that baselines such as Hydra and BE also share parameters in small-Ens.\n## Significance 2.\nWe are trying to apply DLF to large LLMs but we do not think we can get the results by the due date of rebuttal. If possible, we will report the results in the discussion period.\n## Significance 3.\nAssumptions like Lipschitz continuity and sparsity are common in theoretical analyses of deep generative models and nonparametric neural network regression (e.g., Schmidt-Hieber 2020; Chae 2023; Wing Hung Wong et al. 1995), and we agree they may not always hold in practice.  However, the main motivation of the theoretical result in Section 3.4 is to address how to choose design points for distribution distillation. Even if there are many strong assumptions (Gaussianity of teacher ensembles and so on), the theoretical result suggests that the design points whose distribution is close to that of test data is promising. We will polish Section 3.4 more in the revision.\n# Originality 1.\nWe want to emphasize that superior performance of DLF compared to other baselines\nis not achieved by chance. Instead, there are various advantages of DLF compared to Proxy-EnD² and EDFM. \nWe think that there are three properties any desirable models for distribution distillation are expected to have:\n(1) large support, (2) easy and stable learning algorithm and (3) flexibility to be applied to downstream tasks.\nBelow, we explain why DLF is desirable in view of these three properties.\n- DLF is a Gaussian process, which is well known to cover large class of distributions (Garriga-Alonso et al., 2019; Neal et al., 1996; Calandra et al., 2016; Huang et al., 2015). Even if teacher ensembles are not Gaussian, a Gaussian process can approximate it closely. In addition, DLF treats teacher ensembles as random functions rather than random vectors as Dirichlet and Flow matching distillations do. See our reply to Question 1 of Referee 4 for detailed discussions for this point.\n- Existence of easy and stable learning algorithms is a critical fact when choosing an algorithm for distributional distillation. Learning the DLF model is numerical easy and stable since the efficient EM algorithm is available. It is well known that estimating the parameters in the Dirichlet distribution is difficult (Minka et al., 2000; Chai et al., 2013) and thus it would not be easy to learn the Dirichlet distillation model which would a partial reason for inferior performance. For Flow matching distillation, pretrained features should be fed when learning. That is, end-to-end learning is not possible. In contrast, DLF is learned in a end-to-end fashion (i.e. feature vectors and distribution are learned simultaneously).\n- Flexibility is also an important aspect. As we have shown in Section **5.3**, the learned features by DLF is superior than those by a single DNN or Hydra for downstream tasks.\n- To sum up, the idea of estimating the distribution for teacher ensembles is not new but efficient implementation is not an easy task and DLF is such a method.\n# Question\n## 1. \nBecause our student model is distilled directly from the teacher ensemble, we chose the ensemble’s empirical coverage as the reference: matching it indicates that the student has inherited the teacher’s calibration. We agree, however, that proximity to the nominal target (e.g., 95 %) is also informative. In the revision we will report both values so that readers can assess calibration against the ensemble and against the nominal level.\n## 2. \nThank you for raising this interesting point. Due to the brevity of the rebuttal period, we were unfortunately unable to conduct experiments with teacher models pretrained on different datasets. We plan to use the upcoming discussion phase to perform these experiments and incorporate the results into the final manuscript. We appreciate your valuable suggestion.\n## 3. \nPlease refer to our response to Significance 1.\n## 4. \nPlease refer to our response to Quality 3.\n## 5. \nAs noted in line 146, DLF Framework depends on the output dimension—so as outputs grow, both the parameter count and $\\mathbf{Z}$’s dimensionality increase, which can slow training. However, we believe that selecting an appropriate latent dimension $q$ ensures efficient learning. \n\nMoreover, because DLF framework can estimate class‑specific covariance structures, it captures complex relationships more accurately and delivers superior performance on high-dimensional datasets like Tiny-ImageNet. For these reasons, we believe that DLF not only scales to tasks like pixel‑wise regression but may actually gain relative advantage as output dimensionality increases.\n## 6. \nWe use an inverse-gamma prior for the variance parameter $\\sigma^{2}_{\\epsilon}$ because it is conjugate to the Gaussian likelihood in our model. This conjugacy leads to closed-form updates within the EM algorithm and simplifies marginal-likelihood computations.\n\n# Reference\n- Yang, A. X., Robeyns, M., Wang, X., & Aitchison, L. (2023). Bayesian low-rank adaptation for large language models.\n- Liu, Z., Zhao, C., Iandola, F., Lai, C., Tian, Y., Fedorov, I., ... & Chandra, V. (2024, February). Mobilellm: Optimizing sub-billion parameter language models for on-device use cases.\n- Please see the reference list of Reviewer e6dg."}, {"review_id": "h9m7HNEReI", "reviewer": "Reviewer_e6dg", "summary": "This paper propose a novel distillation method that preserves the uncertainty estimation capabilities of the original model while achieving performance gains across multiple downstream tasks. The core innovation lies in the adoption of distributional distillation, wherein the teacher model is treated as samples from a Gaussian process via a deep latent factor model, and the optimization is performed using the Expectation-Maximization (EM) algorithm.", "questions": "The paper repeatedly claims that DLF reduces computational and inference overhead during distillation, yet lacks concrete analysis of actual costs and complexity. What is the practical computational cost of the model during distillation? According to Eq. in line169, both $\\mu$  and  $\\Phi$ are local networks dependent on design points—does the computational overhead increase with larger $m$?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n\n1.This paper conducts experiments across a wide range of tasks, including classification, regression, and distribution shift, demonstrating a well-rounded and comprehensive evaluation.\n\n2. The proposed DLF framework explicitly models both the mean and covariance structure, effectively preserving the uncertainty estimation capability of the teacher ensemble.\n\n Weaknesses：\n\n1.Table 1 does not compare the proposed method against other distributional distillation approaches, and instead focuses primarily on one-to-one distillation baselines. Although line 206 explains the exclusion of Dirichlet distillation, including such methods—especially those in Section 2.2.2 ([17, 18])—would strengthen the experimental completeness.\n\n2.While Section 3.3 briefly discusses the benefits of modeling with Gaussian distributions, the theoretical advantages over alternative distributional approaches are not sufficiently elaborated. Given that this is a core contribution, further justification is needed. For example, Dirichlet distributions naturally model class probabilities due to non-negativity and normalization, whereas Gaussians require additional normalization—does this lead to better uncertainty estimation?\n\n3. Most of the equations in the paper are not properly numbered, and the baseline models used in the experimental setup are not cited.", "was_revised": true, "final_justification": "My hesitation at the beginning of this article mainly came from the comparison of the missing baseline model and the further calrify of motivation, which were resolved by the author in the discussion. What's more, the author has further proved my doubts about computing expenses through theory and practical experiments. In summary, I decided to improve my score.", "rebuttal": "Thank you for your concise review. We appreciate your ability to highlight key aspects of our work, and your feedback helped us refine our focus in the revision process.\n\n# Weakness\n\n## 1.\nIn the original manuscript, we did not include Dirichlet distillation (Proxy-EnD²) in the experiments since the papers for Hyrda and BE reported that their algorithms are superior to Dirichlet distillation. We focused mainly on comparing these two deterministic distillation methods with our DLF. \n\nIn Table below, we add Dirichlet distillation. In addition, we add a new distribution distillation method called Ensemble Distillation via Flow Matching (EDFM; Park et al., 2025), which we found after we submitted our paper. Note that DLF outperforms Proxy-EnD² and EDFM with large margins in terms of the accuracy and NLL while DLF is not much worse than Proxy-EnD² in terms of ECE.\n\n|dataset|method|Acc(%) ↑|NLL ↓|ECE(%) ↓|\n|--|--|--|--|--|\n|CIFAR-10|Teachers|94.24|0.1539|0.9|\n||small-Ens|92.87|0.2377|3.93|\n||Hydra|93.16|0.2660|4.15|\n||LBE|93.25|0.2480|4.11|\n||Proxy-EnD²|90.92|0.2861|2.08|\n||EDFM|90.62|0.2858|2.78|\n||DLF (Ours)|93.40|0.2246|2.79|\n|--|--|--|--|--|\n|CIFAR-100|Teachers|81.36|0.7167|1.41|\n||small-Ens|79.29|1.0413|12.90|\n||Hydra|77.42|1.2912|12.70|\n||LBE|79.58|1.0110|13.42|\n||Proxy-EnD²|67.62|1.2355|7.35|\n||EDFM|64.17|1.6741|11.35|\n||DLF (Ours)|79.68|0.8974|9.45|\n\n\n## 2.\nSeveral prior works have shown that deep neural networks converge to Gaussian processes, including (Lee et al., 2018), (Khan et al., 2019), and (Gal et al., 2016). Based on these findings, we consider it natural to assume that a teacher models trained with a deep neural networks induce a distribution that can be approximated by a Gaussian process.\n\nWe want to emphasize that superior performance of DLF compared to other distributional distillations is not achieved by chance. Instead, there are various advantages of DLF compared to Proxy-EnD² and Ensemble Distillation via Flow Matching (EDFM; Park et al., 2025), a recent distribution distillation method. We think that there are three properties any desirable models for distribution distillation are expected to have: (1) large support, (2) easy and stable learning algorithm and (3) flexibility to be applied to downstream tasks. Below, we explain why DLF is desirable in view of these three properties.\n\n-  DLF is a Gaussian process, which is well known to cover large class of distributions (Garriga-Alonso et al., 2019; Neal et al., 1996; Calandra et al., 2016; Huang et al., 2015). Even if teacher ensembles are not Gaussian, a Gaussian process can approximate it closely. While Dirichlet distributions are often used to represent class probabilities due to their inherent non-negativity and normalization constraints, they are inherently limited to classification tasks as mentioned in answer of Weakness 1. In addition, DLF treats teacher ensembles as random functions rather than random vectors as Dirichlet and Flow matching distillations do. See our reply to Question 1 of Reviewer jk47 for detailed discussions for this point. \n\n- Existence of easy and stable learning algorithms is a critical fact when choosing an algorithm for distributional distillation. Learning the DLF model is numerical easy and stable since the efficient EM algorithm is available. It is well known that estimating the parameters in the Dirichlet distribution is difficult (Minka et al., 2000; Chai et al., 2013) and thus it would not be easy to learn the Dirichlet distillation model which would a partial reason for inferior performance. For Flow matching distillation, pretrained features should be fed when learning. That is, end-to-end learning is not possible. In contrast, DLF is learned in a end-to-end fashion (i.e. feature vectors and distribution are learned simultaneously).\n\n- Flexibility is also an important aspect. As we have shown in Section **5.3**,\nthe learned features by DLF is superior than those by a single DNN or Hydra for downstream tasks.\n\n- To sum up, the idea of estimating the distribution for teacher ensembles is not new but\nefficient implementation is not an easy task and DLF is such a method.\n\n\nAs explained in Introduction, the predictive distribution is obtained by $\\hat{p}(y|\\mathbf{x})=\\sum_{i=1}^n p(y|\\mathbf{x},\\theta_i)/n,$ where $\\theta_1,\\ldots,\\theta_n$ are ensemble members. For DLF, As shown in Figure 1 (“Inference on new data”), we can generate $\\theta_1,\\ldots,\\theta_n$ by  sampling $n$ many independent latent vectors  $\\mathbf{z}_1,\\ldots,\\mathbf{z}_n$ from $\\mathcal N_q(\\mathbf 0,\\mathbf I_q)$, forwarding each sample through the network to have ensemble members.  All of the uncertainty measures are calculated based on $\\hat{p}(y|\\mathbf{x}).$ We will state explicitly this procedure in the revision.\n\n\n## 3.\nIn the revised manuscript, we will number all displayed equations. Citations for the baseline methods (e.g., Hydra, Batch-Ensemble) are already provided in Section 2.2, but were omitted later due to the page limitation. For clarity, we will also include the same citations in the description of our experimental setup.\n\n\n# Question\n## 1.\nAs stated in line 137, $\\mu_{\\theta}$ and $\\Phi_{\\theta}$ are local networks whose output sizes depend only on the target dimension and latent dimension; they do not scale with the number of design points. Thus $m$ increases the amount of data processed during training but does not change the per sample inference cost. Moreover, DLF is that it can be learned easily using the EM algorithm. As detailed in Appendix B, the E‑step has the closed‑form solution, so training time is not computationally demanding. To confirm this argument, we measure the computing times for the training as well as inference in the table below:(1) Multiply–Accumulate Operations (MAC, measured in MMac) for inference, (2) inference latency (ms), and (3) training time (hours).  \nWe will add this result in the revised manuscript.\n\n|Method|CIFAR-10 MAC|CIFAR-10 Latency|CIFAR-10 Training Time|CIFAR-100 MAC|CIFAR-100 Latency|CIFAR-100 Training Time|\n|-|-|-|-|-|-|-|\n|Teacher|224|7.1|10.2|3400|7.6|17.4|\n|small-Ens|108|4.1|4.6|224|7.4|12.5|\n|Hydra|27|1.1|3.1|56|1.9|10.28|\n|LBE|108|2.4|6.5|224|4.6|21.5|\n|Proxy-EnD²|27|1.2|4.6|56|2.1|9.2|\n|DLF (Ours)|27|1.5|4.2|850|2.7|11.28|\n\n\n# Refference\n- Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Pennington, J., \\& Sohl-Dickstein, J. (2017). Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165.\n- Khan, M. E., Immer, A., Abedi, E., \\& Korzepa, M. (2019). Approximate inference turns deep networks into Gaussian processes. Advances in neural information processing systems, 32.\n- Gal, Y., \\& Ghahramani, Z. (2016, June). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning (pp. 1050-1059). PMLR.\n- Park, J., Nam, G., Kim, H., Yoon, J., \\& Lee, J. Ensemble Distribution Distillation via Flow Matching. In Forty-second International Conference on Machine Learning.\n- Minka, T. (2000, November). Estimating a Dirichlet distribution.\n- Chai, D., Forstner, W., \\& Lafarge, F. (2013). Recovering line-networks in images by junction-point processes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1894-1901).\n- Garriga-Alonso, A., Rasmussen, C. E., \\& Aitchison, L. (2018). Deep convolutional networks as shallow gaussian processes. arXiv preprint arXiv:1808.05587.\n- Neal, R. M. (1996). Priors for infinite networks. In Bayesian learning for neural networks (pp. 29-53). New York, NY: Springer New York.\n- Calandra, R., Peters, J., Rasmussen, C. E., \\& Deisenroth, M. P. (2016, July). Manifold Gaussian processes for regression. In 2016 International joint conference on neural networks (IJCNN) (pp. 3338-3345). IEEE.\n- Huang, W. B., Zhao, D., Sun, F., Liu, H., \\& Chang, E. Y. (2015, June). Scalable Gaussian Process Regression Using Deep Neural Networks. In IJCAI (pp. 3576-3582)."}, {"review_id": "3EGsvImviT", "reviewer": "Reviewer_jk47", "summary": "In this work, the authors introduce a new distribution distillation framework to estimate the distribution of a teacher ensemble through a special Gaussian process. In this distillation process, each member of the teacher ensemble is considered as a realization of a certain stochastic process. Multiple tasks, including regression tasks, classification, as well as fine-tuning of LLMs, among others, are conducted for experimental validation.", "questions": "**Q1**\n\nCould the authors provide some theoretical justifications or reasons for assuming that the many teacher models are independent realizations of the DLF model, and to estimate the mean and factor loading functions? In the settings of deep ensemble, the teachers (ensemble members) are trained using the same data but with different random initialization. How could we prove these modes are independent?\n\n**Q2**\n\nWould the DLF model perform better than the method in [1], which achieves comparable or even better uncertainty quantification performance of deep ensembles but requires a single model? \n \n[1] Mukhoti, Jishnu, et al. \"Deep deterministic uncertainty: A new simple baseline.\" CVPR. 2023.", "limitations": "Yes, the authors discussed them as future work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper has a relatively clear structure and is easy to follow.\n\n2. The experimental validations are clearly described, and the implementation details are given.\n\n3. The main results shown in the main body demonstrated the outperformance of the proposed method on the applied evaluation metrics. \n\n4. Some ablation studies are given to enhance the evaluation further.\n\n**Weaknesses**\n\n1. *Reliable or improved uncertainty quantification (UQ)* is claimed as one of the highlights of this work. So it would be better to briefly introduce how the prediction uncertainties are measured in the framework (especially, the authors mentioned about aleatory and epistemic uncertainty in the main body). Most importantly, the evaluation of UQ can hardly reflect the state-of-the-art benchmarks. For instance, out-of-distribution detection and active learning can be used for proxy tasks, as shown in recent literature, such as [1] [2] [3].\n\n[1] Mukhoti, Jishnu, et al. \"Deep deterministic uncertainty: A new simple baseline.\" CVPR. 2023.;\n[2] Wang, Kaizheng, et al. \"Credal deep ensembles for uncertainty quantification.\"NeurIPS 2024;\n[3] Mucsányi, Bálint, et al. \"Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks.\" NeurIPS 2024.\n\n2. I fully agree that Dirichlet distillation is limited to classification tasks. As shown in their original paper, they achieve comparable performance to the ensemble on ECE metrics and other UQ evaluations. Clearly, they also have a simpler architecture, the same as a single ensemble member of deep ensembles. So, would it be possible to also report the results of Dirichlet distillation in the classification tasks?\n\n3. It is true that the proposed methods significantly reduce the complexity of deep ensembles. I think it would be beneficial to include the training complexity and the inference complexity comparisons of the proposed method and the other baselines so that the community will have a better sense of applying the method in practice.\n\n4. I appreciate the ablation study of the latent factor dimension using the Boston housing data in regression. Would the conclusion hold also for more complex classification tasks? It might also be beneficial to report the applied latent factor dimension in the main result tables, if applicable.", "was_revised": true, "final_justification": "The responses from the authors and the new results conducted have basically addressed the concerns and enhanced my confidence in my assessments. Therefore, I increased my scores and confidence to 4. The main reasons for not further increasing my score are: There is no theoretical analysis/certificate for the main assumption in the proposed method, namely, each teacher model is a Gaussian process, and thus the prediction values from the same teacher model are dependent. The empirical evaluations do show the improved performance; I believe further experiments on larger-scale datasets (such as ImageNet)  would enhance the strengths of the proposed method.", "rebuttal": "Thank you for your constructive review and for expressing strong interest in our work. We are encouraged by your willingness to engage further in discussion and truly appreciate your thoughtful comments on the core contributions of our paper.\n\n# Weakness\n\n## 1.\nAs explained in the Introduction, the predictive distribution is obtained by $\\hat{p}(y|\\mathbf{x})=\\sum_{i=1}^n p(y|\\mathbf{x},\\theta_i)/n,$ where $\\theta_1,\\ldots,\\theta_n$ are ensemble members. For DLF, as shown in Figure 1 (“Inference on new data”), we can generate $\\theta_1,\\ldots,\\theta_n$ by  sampling $n$ independent latent vectors  $\\mathbf{z}_1,\\ldots,\\mathbf{z}_n$ from $\\mathcal N_q(\\mathbf 0,\\mathbf I_q)$, forwarding each sample through the network to have ensemble members.  All of the uncertainty measures are calculated based on $\\hat{p}(y|\\mathbf{x}).$ \n\nAs the referee suggested, we evaluated DLF and Deep Deterministic Uncertainty (DDU) under the same OOD-detection protocol. CIFAR-10 serves as the in-distribution dataset, while SVHN, CIFAR-100, and Tiny-ImageNet are treated as OOD. AUROC is the evaluation metric, and all tests use a Wide-ResNet-16-1 backbone (≈ 0.17 M parameters).\n\nIn (Mukhoti et al,. 2023), the authors report results with a much larger backbone (Wide-ResNet-28-10, $\\geq$ 214 $\\times$ Wide-ResNet-16-1). After downsizing DDU to match our architecture, DLF achieves higher AUROC on all three OOD datasets. **These findings demonstrate that DLF achieves well uncertainty estimation performance relative to its compact model size, which aligns well with the objective of knowledge distillation**. \n\n|method|# of parameters|SVHN|CIFAR-100|Tiny-ImageNet|\n|-|-|-|-|-|\n|small-Ens|0.70M|0.9040|0.7991|0.7952|\n|Hydra|0.18M|0.7331|0.6767|0.6794|\n|LBE|0.18M|0.8570|0.8033|0.8068|\n|DDU|0.18M|0.8772|0.6901|0.6660|\n|DLF|0.18M|0.8984|0.7897|0.7853|\n\nWe evaluated OOD detection performance using AUROC as our primary metric. Although LBE achieves slightly higher AUROC scores on CIFAR-100 and Tiny-ImageNet, its predictive probability densities assign nearly many in-distribution samples a probability very close to 1.0. This behavior indicates that the model is essentially overconfident rather than genuinely better at quantifying uncertainty.\n\n## 2.\nIn the original manuscript, we did not include Dirichlet distillation (Proxy-EnD²) in the experiments since the papers for Hydra and BE reported that their algorithms are superior to Dirichlet distillation. We focused mainly on comparing these two deterministic distillation methods with our DLF. \n\nIn Table below, we add Dirichlet distillation. In addition, we add a new distribution distillation method called Ensemble Distillation via Flow Matching (EDFM; Park et al., 2025), which we found after we submitted our paper. Note that DLF outperforms Proxy-EnD² and EDFM with large margins in terms of the accuracy and NLL while DLF is not much worse than Proxy-EnD² in terms of ECE.\n\n|dataset|method|Acc(%) ↑|NLL ↓|ECE(%) ↓|\n|--|--|--|--|--|\n|CIFAR-10|Teachers|94.24|0.1539|0.9|\n||small-Ens|92.87|0.2377|3.93|\n||Hydra|93.16|0.2660|4.15|\n||LBE|93.25|0.2480|4.11|\n||Proxy-EnD²|90.92|0.2861|2.08|\n||EDFM|90.62|0.2858|2.78|\n||DLF (Ours)|93.40|0.2246|2.79|\n|--|--|--|--|--|\n|CIFAR-100|Teachers|81.36|0.7167|1.41|\n||small-Ens|79.29|1.0413|12.90|\n||Hydra|77.42|1.2912|12.70|\n||LBE|79.58|1.0110|13.42|\n||Proxy-EnD²|67.62|1.2355|7.35|\n||EDFM|64.17|1.6741|11.35|\n||DLF (Ours)|79.68|0.8974|9.45|\n\nWe want to emphasize that superior performance of DLF compared to other distributional distillations  is not achieved by chance. Instead, there are various advantages of DLF compared to Proxy-EnD² and EDFM. We think that there are three properties thay any desirable models for distribution distillation are expected to have: (1) large support, (2) easy and stable learning algorithm and (3) flexibility to be applied to downstream tasks. Below, we explain why DLF is desirable in view of these three properties.\n\n- DLF is a Gaussian process, which is well known to cover large class of distributions (Garriga-Alonso et al., 2019; Neal et al., 1996; Calandra et al., 2016; Huang et al., 2015). Even if teacher ensembles are not Gaussian, a Gaussian process can approximate them closely.\n- Existence of easy and stable learning algorithms is a critical fact when choosing an algorithm for distributional distillation. Learning the DLF model is numerically easy and stable since the efficient EM algorithm is available. It is well known that estimating the parameters in the Dirichlet distribution is difficult (Minka et al., 2000; Chai et al., 2013) and thus it would not be easy to learn the Dirichlet distillation model which would be a partial reason for inferior performance. For Flow matching distillation, pretrained features should be fed when learning. That is, end-to-end learning is not possible. In contrast, DLF is learned in an end-to-end fashion (i.e. feature vectors and distribution are learned simultaneously).\n-  Flexibility is also an important aspect. As we have shown in Section **5.3**, the learned features by DLF are superior than those by a single DNN or Hydra for downstream tasks.\n- To sum up, the idea of estimating the distribution for teacher ensembles is not new but efficient implementation is not an easy task and DLF is such a method.\n\n## 3.\nThank you for this valuable suggestion. We categorize computational cost into two aspects: the cost incurred during training steps and that during inference. Since inference cost is roughly proportional to the number of parameters, we have reported each model's parameter count. Moreover, during training, the E-step expectation can be obtained in closed form; the formula is given in Appendix B of the paper, so training time does not increase significantly. In the revised manuscript, we will include a discussion of these computational cost. Specifically, we will report (1) Multiply–Accumulate Operations (MAC, measured in MMac), (2) inference latency (ms), and (3) training time (hours).\n        \nPreliminary measurements on CIFAR-10/100 are provided below.\n\n|Method|CIFAR-10 MAC|CIFAR-10 Latency|CIFAR-10 Training Time|CIFAR-100 MAC|CIFAR-100 Latency|CIFAR-100 Training Time|\n|-|-|-|-|-|-|-|\n|Teacher|224|7.1|10.2|3400|7.6|17.4|\n|small-Ens|108|4.1|4.6|224|7.3|12.5|\n|Hydra|27|1.1|3.1|56|1.8|10.28|\n|LBE|108|2.4|6.5|224|4.3|21.5|\n|Proxy-EnD²|27|1.2|4.6|56|2.1|9.2|\n|DLF (Ours)|27|1.5|4.2|56|2.6|11.28|\n\n## 4.\nThank you for your insightful suggestion. To verify that our observations generalize to more complex classification tasks, we have conducted an ablation study of the latent factor dimension on the CIFAR-10 and CIFAR-100 datasets. In the revised manuscript, we will include the corresponding figures, and we have also updated the main result tables to report the applied latent factor dimensions where applicable.\n\n# Question\n## 1.\nWe also think that ensemble members trained on the same data with different initializations are not strictly independent. The goal of distribution distillation is to find a proxy distribution that approximates\nthe distribution of teacher ensembles, and the proxy distribution assumes that teacher ensembles are independent.\nThe validity of this assumption in the proxy distribution is justified by the performance of the corresponding\npredictive distribution.\n\nNote that most methods for distribution distillation assume that teacher ensembles are independent.\nMoreover, Dirichlet distillation as well as Ensemble Distillation via Flow Matching (EDFM; Park et al. 2025)\nassume further that prediction values from a same teacher model at different inputs are independent.\nIn contrast, we assume that each teacher model is a Gaussian process and thus the prediction values from a same\nteacher model are dependent.\n\nThat is, DLF estimates the joint distribution of teacher ensembles while Dirichlet distillation and EDFM estimate the marginal distribution only. Estimation of the joint distribution would be useful when we have to predict multiple inputs simultaneously. \n\nFor example, consider a problem to decide whether two randomly given images belong to a same class or not. Let $\\mathbf{x}_1$ and $\\mathbf{x}_2$ be two given images and let $Y_1$ and $Y_2$ be the corresponding class labels. \nThen, the goal is to estimate $p(Y_1=Y_2|\\mathbf{x}_1,\\mathbf{x}_2)$. \n\nDLF estimates this predictive probability by $\\int_f \\sum_{c=1}^C p(y_1=c|\\mathbf{x}_1,f)p(y_2=c|\\mathbf{x}_2,f) \\pi(f)df$ while Dirichlet distillation and EDFM estimate it by $\\sum^C_c \\int_f p(y_1=c|\\mathbf{x}_1,f) \\pi(f)df \\int_f p(y_2=c|\\mathbf{x}_2,f) \\pi(f)df$. These two estimates are quite different and the former is expected to perform better. \n\nTo justify this conjecture, we investigate the performance of these two estimates on randomly selected pairs of CIFAR-10 test images. The results are as follows: \n\n|Method|AUROC|AUPRC|\n|-------------------------|--------|-------|\n|Dirichlet distribution|0.967|0.816|\n|DLF with dependency|0.985|0.900|\n|DLF without dependency|0.938|0.781|\n\n(AUROC and AUPRC) values are (0.967, 0.816) for Dirichlet distribution and (0.985, 0.90) for DLF while the ECE values are similar (0.11). Moreover, AUROC and AUPRC values of DLF without considering the dependency between $Y_1$ and $Y_2$ (i.e. using $\\sum_{c=1}^C \\int_f p(y_1=c|\\mathbf{x}_1,f)\\pi(f)df \\int_f p(y_2=c|\\mathbf{x}_2,f) \\pi(f)df$ where $\\pi(f)$ is estimated by DLF) are (0.938, 0.781), which are much lower than those of DLF when considering the dependency. These results amply support that considering dependency of prediction values from the same teacher model would be beneficial and DLF is such a model.\n## 2.\nPlease refer to our response to Weakness 1.\n\n# Reference\n- Mukhoti, J., Kirsch, A., Van Amersfoort, J., Torr, P. H., \\& Gal, Y. (2023). Deep deterministic uncertainty: A new simple baseline.\n- Please see the reference list of Reviewer e6dg."}]

Author rebuttal:

Meta-review: The paper is interested in the task of distilling an ensemble of deep networks into a single model for computational efficiency while keeping the desirable properties of an ensemble, particularly its predictive uncertainty quality. It uses student DNNs to model the mean and covariance of the teacher ensemble treated as a Gaussian process.  It tests the approach on various small datasets from UCI as well as CIFAR10, CIFAR100, and GLUE. It favorably evaluates the proposed model in terms of accuracy, negative log likelihood, and calibration error on their test data.

There was extensive rebuttal provided and the reviewers discussed it with the authors. After the rebuttal, all reviewers lean towards acceptance. During AC-reviewers discussion, all reviewers acknowledge the relevance and novelty of the work, clarity of the presentation, and significance of the obtained results with respect to prior work. Their only remaining concern/question is the transfer of the improved performance to larger benchmarks and/or more complex tasks. The authors provided additional results on Tiny ImageNet during the review process and have promised to add results on ImageNet.

The AC agrees with the reviewers that the strengths of the paper outweigh the criticism on missing large benchmarks. The fields of uncertainty estimation and distillation are of wide interest in academia and relevance in industry, and this paper makes a step forward. Therefore, the AC suggests acceptance.

The AC strongly recommends the authors to include the promised ImageNet results and possibly more large benchmarks and refer to the following currently-missing but directly-related works in the final version (and ideally compare with some of them as baselines):

[1] "Bayesian Dark Knowledge", NIPS 2015

[2] "Efficient Evaluation-Time Uncertainty Estimation by Improved Distillation", ICML 2019

[3] "A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness", JMLR 2022

[4] "Ensemble Distribution Distillation via Flow Matching", ICML 2025

[5] "Contextual similarity distillation: Ensemble uncertainties with a single model", arXiv 2025
Decision: Accept (poster)