Towards Undistillable Models by Minimizing Conditional Mutual Information
Authors: Linfeng Ye
Shayan Mohajer Hamidi
EN-HUI YANG
Venue: Accepted by TMLR ()

Abstract: A deep neural network (DNN) is said to be undistillable if, when used as a black-box input-
output teacher, it cannot be distilled through knowledge distillation (KD). In this case, the
distilled student (referred to as the knockoff student) does not outperform a student trained
independently with label smoothing (LS student) in terms of prediction accuracy. To protect
intellectual property of DNNs, it is desirable to build undistillable DNNs. To this end, it is
first observed that an undistillable DNN may have the trait that each cluster of its output
probability distributions in response to all sample instances with the same label should be
highly concentrated to the extent that each cluster corresponding to each label should ideally
collapse into one probability distribution. Based on this observation and by measuring the
concentration of each cluster in terms of conditional mutual information (CMI), a new
training method called CMI minimized (CMIM) method is proposed, which trains a DNN
by jointly minimizing the conventional cross entropy (CE) loss and the CMI values of all
temperature scaled clusters across the entire temperature spectrum. The resulting CMIM
model is shown, by extensive experiments, to be undistillable by all tested KD methods
existing in the literature. That is, the knockoff students distilled by these KD methods
from the CMIM model underperform the respective LS students. In addition, the CMIM
model is also shown to performs better than the model trained with the CE loss alone in
terms of their own prediction accuracy.

Published in Transactions on Machine Learning Research (06/2025)
Towards Undistillable Models by Minimizing Conditional Mutual Information
Linfeng Ye*
4.yye@uuaterooo.ca
Department of Electrical and Computer Engineering University of Waterloo
Shayan Mohajer Hamidi"
mmhhajenuuwatertto..om
Department Electrical and Computer Engineering University of Waterloo
En-Hui Yang
elyang@uwaterloo.cm
Department of Electrical and Computer Engineering University of Waterloo
Reviewed on OpenReview: /b net/ forum? id- JVABssDVV /
Abstract
A deep neural network (DNN) is said to be undistillable if, when used black-box input- output teacher, it cannot be distilled through knowledge distillation (KD). In this case, the distilled student (referred to as the knockoff student) does not oulperform student trained independently with label smoothing (LS student) in terms of prediction accuracy. To protect intellectual property of DNNs, it desirable to build undistillable DNNs. To this end, it is first observed that an undistillable DNN may have the trait that each cluster of its output probability distrributions in response to all sample instances with the same label should be highly concentrated to the extent that each cluster corresponding each label should ideally collapse into probability distribution. Based this observation and by measuring the concentration of each cluster in terms of conditional mutual information (CMT), new training method called CMI minimized (CMIM) method is proposed, which trains a DNN by jointly minimizing the conventional cross entropy (CE) loss and the CMI values of all temperature scaled clusters across the entire temperature spectrum. The resulting CMIM model shown, by extensive experiments, existing in the literature. That is, the knockoff students distilled by these KD methods from the CMIM model underperform the respective LS students. In addition, the CMIM model is also shown to performs better than the model trained with the CE loss alone in terms of their own prediction accuracy. The code for the paper is publicly available at htttps://anonnoon http://sww..o..teeeeeeeeoeeeoooa
be undistillable by all tested KD methods
1 Introduction
Originally siming for model compression, knowledge distillation (Buciluá et al., 2006 Hinton et al. 2015 (KD) has received siguificant altention from both academia and industry in recent years due to its remarkable effectiveness. The essence of KD transfer the knowledge of pre-trained large model (teacher)
a smaller model (student). Building the work of Hinton et al. (2015). numerous follow-up works have
endeavored to enhance the performance of KD (Romero et al. 2014 Anil al. 2018
Park et al.
2019) and
to gain deeper insights into why distillation is effective [Phuong & Lampert 2019
Mobahi et al.
2020 Ye
et al. 2024 Allen-Zhu & Li 2020 Menon et al. 2021 Borup & Andersen 2021
Authors contributed equally.

---

Published in Transactions on Machine Learning Research (06/2025)
In the scenario where the teacher does not want its knowlodge to be transferred, however, KD is undesirable and indeed poses threat to intellectual property (TP) of the teacher (Shokri & Shmatikov 2015). Developing and training high-quality large DNN requires significant investments of time, effort, finances, and resources, inchuding extensive data annotation and computational infrastructure. Developers of the large DNN may want to prevent the knowledge of the large DNN from being transferred by their competitors. However, once the large DNN is released as a "black box", anyone can apply a logit-basod KD method (or equivalently a distribution-based KD method (Zheng & YANG 2024)) to distill the DNN as a teacher. The goal is to train student, referred to knockoff student in the context of DNN IP protection, that mimics the teacher's behavior to gain competitive ad advantages. As such, in this case, it would be desirable for the developers to build the large DNN so that it is undistillable. The question, of course, is how.
Before delving deeper into the above question, let us first clarify what we mean by saying that a DNN is undistillable. At this point, we invoke the concept of distillable DNN introduced recently in Yang & Yo (0044):
Definition 1. [Distillability of a DNN YYnng& & Ye 2024) W When used as a black-box input-oulput teacher, a DNN is said to be distillableJ with respect to a student if there exists a KD method which, when applied to the teacher and student, yields a knockoff student outperforming the student trained alone with label
smoothing (LS student) in terms of prediction accuracy.
Therefore, a DNN is undistillable if no knockoff student can outperform the respective LS student regardless of which logit-based KD method is used. Since there are so many logit-based KD methods and so many students, what type of DNN is undistillable? In comparison with the training of LS student, Definition [] provides some insight into a trait that an undistillable DNN may possess. For each label, consider the cluster of the output probability distributions of the DNN in response to all input sample instances with that label. If each cluster corresponding to each label is highly concentrated to the extent that all probability distributions within the cluster more or less collapse into one probability distribution, then the student training within KD is similar to that of the respective LS student regardless of which logit-based KD method and which student are applied. In this case, one would expect that no knockoff student would perform significantly better than the respective LS student. Therefore, a DNN possessing this trait will likely be undistillable.
From theoretical perspective, KD relies the diversity within the teacher's predicted distributions to convey richer contextual information to the student. The greater the diversity in the cluster of predictions for a given class, the more nuanced guidance the student can receive (see Ye et al. (2024)). Conversely, when the teacher's outputs become highly concentrated meaning predictions for inputs with the same label converge to a single probability vector- contextual richness is lost. The student, in this case, receives largely redundant information, rendering the distillation process as ineffective as direct training with label smoothing.
To quantify this notion of concentration, we turn to conditional mutual information (CMI) YYngg et al., 2025 h. Specifically, let X denote the random input sample to the DNN, and Y be the ground truth label of X. Lot y denote the random label predicted by the DNN in response to input X. It was shown in Yang et al. (2025) that for each label y, the label specific CMI I(X; 81 = y) measures the concentration of the cluster corresponding to label y. and the CMI I(X; Y) measures the average concentration across all clusters. To build an undistillable DNN, one then is motivated to minimize jointly the conventional cross entropy (CE) loss and the CMI (X;;))))
In this paper, we will go one step further. In KD Hinton et al. 2015 temperature scaling of logits is often applied. It was shown in Zheng & YANG (2024) that logit temperature scaling with temperature T can be equivalently achieved by power transform of the output probability distribution with power = 1/T. Further, it was demonstrated in Ye et al. (2024) that the purpose of temperature scaling or power transform is to enlarge the CMI values of temperature scaled (or power transformed) clusters, and enlarging CMI values in turn improves the performance of distilled students. Since here we want to achieve the opposite,
TThere two reasons for us to adopt this definition of distillability. First, as shown theoretically in Zheng & YANG 2024). knowlodge distillation reduces to label smoothing in the limit as the temperature approaches infinity. Therefore, if the teacher is distillable, the distilled student should perform no worse than the L.S student. Second, knockoff student cannot outperform the L. student, then there no incentive for the teacher to be leveraged since the L. student can be trained on its own.
2

---

Published in Transactions on Machine Learning Research (06/2025)
want to make sure that all CMI values of all power transformed clusters can be made small. To this end, we further extend the label specific CMI T(X; 81 Y == y) and the CMT T(X; Y) to [(xp0g) Y = y) and 1(X;Yal)) Y), respectively, so that 1(X; 8a| Y = y) measures the concentration of the power transformed cluster corresponding to label y with power a, and I(X; I(X; YaMM) Y) mcasures the average concentration across all power transformed clusters with power a[Y], where different clusters may be power transformed with different power a. Notably, allowing separate temperature scaling (i.e., power transformation) for each class is essential, as an adaptive adversary may apply class-specific scaling to selectively alter cluster concentration and improve the knockoff student's performance.
Based on the above discussion and towards building undistillable DNNs, we then propose a new training method called CMI minimized method, which trains a DNN by jointly minimizing the CE loss and all CMI values of all power transformed clusters, ie., jointly minimizing the CE loss and I(x; (x(]) Y), Va(Y] > 0.
The resulting trained DNN is referred to as the CMI minimized (CMIM) DNN. The contributions of the paper are sunmarized follows:
An insight is provided that in order for DNN to be undistillable, it is desirable for the DNN to possess the trait that cach cluster of the DNN's output probability distributions corresponding to cach label is highly concentrated to the extent that all probability distributions within the cluster more or less collapse into one probability distribution close to the one-hot probability vector of that label.
We extend the label specific CMI I(X; 81 = y) and the CMI 8 Y) to I(X;8 Y = y) and [xa]] Y), respectively, so that I(X; Yo Y = y) measures the concentration of the power transformed [(x;yrl()] Y) measures the average concentration across all power transformed clusters with power a{Y], where different clusters may be power transformed with different power a.
cluster corresponding to label y with power a, and I(X; yrol]]
We develop a novel training method dubbed CMI minimized method to train a DNN by jointly minimizing the CE loss and all CMI values of all power transformed clusters with the resulting trained DNN referred to as the CMTM DNN.
To the best of our knowledge, our method is the first in the literature capable of training undistillable DNNs that remain robust against wide range of KD methods. Furthermore, for the notion of undistillability, we are the first to employ the formal definition introduced in Yang & Ye (2024).
We show, by extensive experiments over three popular image classification datasets, namely CIFAR-100 Krizhevsky et al., 2012). TinylmageNeet (Le & Yang 2015) and ImageNet (Deng et al. 2009 that CMIM DNNs have very small CMT values and are indeed undistillable by all tested KD methods existing in the literature. That is, the knockoff students distilled by these KD methods from the CMIM models underperform the respective LS students. On the other hand, models trained by defense training methods proposed in the literature are all distillable.
In addition, we show that the CMIM models achieve a higher classification accuracy compared to those trained with the conventional CE loss.
2 Related Works
In this section, we mention some defense methods against the threat posed by knockoff students attempting to steal the IP of pre-trained DNNs via logit-based KD methods. For thorough review of related works, including detailed discussions about recent logit-based KD methods, please refer to Appendix B These defense methods can be mainly categorized into two groups: (i) model stealing resistant training methods that specifically train DNNs to reduce the accuracy of knockoff students while maintaining the original classification accuracy of the model (Ma et al. 2021 Wang et al., 2022]; and (ii) post-training defense mothods that perform minimal perturbations to the pro-trained model's predictions to mislcad the knockoff
student Lee et al. 2019 Orekondy al. [220 Cheng & Cheng 2023). Nonetheless, in Section we will show that models trained by all these defense methods are indeed distillable.
3

---

Published in Transactions on Machine Learning Research (06/2025)
3 Notation and Preliminaries
3.1 Notation
The set of real numbers is denoted by R. Vectors are denoted by bold-face letters (e.g,, w). The i-th element of vector t is denoted by w[i]. For two vectors u, v E IRC, the inequality u S v implies that u(i] x o[i). Vi E [C]. For a positive integer K, let [K] & {.. Assume that there are class labels with [C] as the set of class labels. Let P([C]) denote the of all C dimensional probability distributions. For any two probability distributions Pı, P2 PP(C)), the CE and Kullback-Leibler (KT.) divergence between Pi and Pz are denoted by P2) and KL(PI, P2), respectively. For any y E [C] and P E P([C)), write the CE of the one-hot probability distribution corresponding to and P as H P).
For any differentiable function J(-), V (-) denotes its gradient vector w.r.t. vector wr
For any pair of random variables (x, Y), denote its joint probability distribution by Px,y y) or simply P(x,1) v) whenever there is no ambiguity, the marginal distribution Y by Pr(y), and the conditional distribution of Y given X x by Prix (-1z). The mutual information between two random variables X and Y is denoted by I(X, Y), and the CMI of X and Y given third random variable Z is I(X,YIZ).
We regard classification DNN
mapping from raw data Rd to probability distribution ea E P(CC). Given a DNN: E R + let 0 denote its weight vector consisting of all its connection weights; whenever there is no ambiguity, we also write fs as 4s.0-
3.2 Label Smoothing
Label smoothing (LS) Pereyra et al. 2017) a regularization technique that prevents peaked output probability distributions, leading better generalization, by minimizing the objective function:
CLS = () q=) eH(u, 9z).
(1)
where u is the uniform distribution over C classes, and e controls the strength of the regularization.
3.3 Power Transform of Probability Distribution
In "black-box" teacher setting, where only the output probability vectors (and not the logits) of the teacher are accessible to the public, applying temperature scaling directly over the logits of the teacher is not feasible in training knockoff students. In this case, KD training can resort to applying "power transformation of probability distribution" directly to the output probability vectors (Zheng & YANG 2024). Specifically, given P E P([C)), and a non-negative real number a, the power transform of P is another probability distribution define as
(Pli)) Ejeloy (PD))) Vi [C).
p*[]
(2)
It is not hard to verify that the power transformed probability distribution po is equal to the softmax of the logits scaled by temperature T =/=1/a Therefore, temperature scaling can be equivalently operated directly on the output probability distribution through power transform.
3.4 CMI value of a DNN
As discussed in Yang et al. (025). for classifier f x IRd + qr, let Ý be the random label predicted by the with probability qx[ in respond to the input X. For each cluster corresponding to label y E [C], we
4

---

Published in Transactions on Machine Learning Research (06/2025)
have
c
Prixy (8 i12,,1)
I(x; = Σ (xpr(
Σ
Pp (P 11) In
(3)
Prrr (8 = iiY = 5)
i=1
Exir [ qx ux In
qx Pr (Y i|Y v) 1 1 Exix [KL (9x, 8yy) A y)
(4)
where Prxxr (8 (|., v) 9=11] follows from the Markov chain Y + x V, and By = Prrr(1lu)) =
Exiy 19x = y]. 1(X;111 y) measures the concentration of the cluster corresponding to label y [C]. Averaging over all clusters corresponding to all labels y, we get
((;
Σ ( y) Exy (1x,
(5)
VE|C]
T(X; PIY) measures the average concentration across all clusters.
When the distribution Px, is unknown, we can approximate the CMI of f by its empirical value from a data sample (a training dataset or mini-batch thereof) D = {(z,, w)] To this end, let D, (1 j S m My = y). Denote the size of Dy by 12,l- The empirical values of each label specific CMI and the CMI can be calculatod follows
Iemp (X; NK = y) [Dyl
Σ ED,
KL (1x s)mp)
(6)
cmp (X; = 1 KL
(7)
1=1
where
cmp y
Σ
Urg Vy E [C].
(8)
Dyl
IED
4 CMI Minimized Method
In this section, we present our CMI minimized method. We begin with extending I(x; = y) and T(X; VIV) to the case of power transformed clusters.
4.1 Information Quantities for Power Transformed Clusters
Consider a classification DNN: f + qa which maps input sample instances æ with different labels into clusters of probability distributions 9u in the space P([C)), with one cluster per label. For each label y E [C], apply the power transform with power to each probability distribution da within the cluster corresponding to the label y. Then, obtain power transformed cluster. To measure the concentration of the power transformed cluster, we extend I(X; = y) to the following information quantity
1(X; = Ex [KL (9X, sy.a) |Y = y)
(9)
where 8y.a Exjy Y = y). Note that if we regard Pa as the random label predicted by f with probability xx (fra) in response to the input sample X, i.e., given X, aa is equal to label with probability y (c), Ve E [C], then T(X; (axy = y) is exactly the CMI between X and ya given == Thus, T(x; v) measures the concentration of the power transformed cluster corresponding to y.
Now, we go one step further and allow different clusters to be power transformed with different powers. Suppose that the cluster corresponding to label y is power transformed with power afy)- Let poll be the random label predicted by with probability 4x (fal()))) in response to the input sample X given Y. That
5

---

Published in Transactions on Machine Learning Research (06/2025)
is, given Y = y and X v, pa(M is equal to e with probability qu eivl (c) (c) for any c [C]. We can then extend I(X;8IY) to I(x;; (a(M))))
T(X; (xx:aMMlyy = Exx
KT. (1x)1
(x
a|l)]
(10)
Σ
Pr( [xxI KL @ofol 15x,80m s..aall)) 1 = w
(11)
vE[C
Σ Pr(y)[(X; galull
Py(y))(xxpabulx> = 9).
(12)
vE(C]
where for each y E [C],
Σ (x Exy 4x 1 -
syalll] (romi)p)()))
(13)
Note that T(X; (Y) is exactly the CMT between X and ÿa(M] given Y and measures the average con- centration across all power transformed clusters with power function a[Y]. However, Y, X, and yalM] do not form a Markov chain anymore.
When the distribution Px,y is unknown, we can approximate T(X; {X y) and I(X; (Xx;a(()| by their respective empirical values from data sample (a training dataset mini-batch thereof) D {(xx) )m))
doda &ooolli y)
1
KL )
(14)
[Dyl
ED
[mm xagaall(Y) = KL (1 llvl smr
demp (X;
(15)
where yyaifp] [Dyl
2 cbl
Vy E [C].
(16)
IED
As discussed in Section [ an undistillable DNN should exhibit the trait that each of these clusters is highly concentrated and ideally collapses into single probability distribution that closely resembles the one-hot probability vector for that label.
0.08
orchids motorcycle
0.06
Remark 1. While we leverage the concept of CMI from Yang et al. (2025), the way it calculated in our work significantly differs from how it is calculated in Yang et al. (2025). In Yang et al. (2025), I(x;Y])) is calculated under the assumption of Markov chain Y - X Y. In contrast, we quantify cluster compactness using T(X; )Y)))) where yo[r] explicitly depends on Y, violating the Markov assumption. Moreover, minimizing I(X; pa(xily) over all possible values a introduces additional challenges, which we ad- dress in the next subsection.
CMI
0.04
0.02
0.00
0.25
0.50
0.75
1.00
Figure
1:
The CMI value
I(X; yallly randomly selected (beaver, orchids, motorcycle) the power transform factor the model is ResNet-50 pre-trained on CIFAR-100. As observed, grows, the CMI value becomes larger, peaks, and then gradually becomes smaller.
= y) for three
4.2 Framework
classes
for Minimizing CMI Values of Power Transformed Clusters
Towards building an undistillable DNN, we now train DNN f ad + qu by jointly minimizing the CE loss and all CMI values of all power transformed clusters. Let
a = a2),),, a[C]],
and write each dr as gr our CMI minimized method, the objec- tive function want to minimize is
Exy [x())
+ Amax I(X; pa(M)y))
(17)

---

Published in Transactions on Machine Learning Research (06/2025)
where À > 0 is a hyper-parameter trading the CE loss with the maximum CMI, and the maximization over Or is taken over the region 0 S a<ii] - B, s sisc C. The optimization problem then becomes
min Exy
H(Y, [x
+ Amax 1(X; {XX])))))))
min Exx H(Y, xx,
+mmax
Py Pr[w]T(X; Polliy pol =1)
(18)
y
min
Exy qx,))
+ Σ Py lvl max I [[x pralll - w)
(19)
avy
In order to get a better understanding about the behavior of the second term in the objective function of equation 18 w.r.t. a, we depict in Figure [(x; y y) vs aly] for three randomly-selected classes y using a pre-trained ResNet-50 on CIFAR- 100. In Figure max, I(X; Pm Y y) is achieved value of which is between 0.25 and 0.75. In Theorem 2 of Appendix D we further show that for each label y, I(X; ([X:YaY y) function of is continuously differentiable.
However, finding algorithmic solution to the min-max problem in equation 18 to iquation 19 challenging. To overcome this difficulty, we next develop a more tractable expression for maxa 1 = v). At this point, we invoke the following theorem, which will be proved in Appendix
Theorem 1. For any label y.
max T(X;Pay = = lim 1 In / exp (wI(X; = w+20 w
(20)
Therefore, when w is large, max I(X; fo y = y) can be approximated by
max 1(X;80ll v) In 1
3 /' exp P 5)}da
(21)
Σ exp (wI y)}
In
(22)
where N is relatively large, and a; i3/N.
Now plugging equation
22 into equation
19
we have
N Σ
min Exy H[X
+ Σ y
Pylj
N exp ( = 8)} }
(23)
Note that the second term in the objective function of equation 23 is not amenable to parallel computation via GPU due to the dependency of KT. divergence on 8y,0,, the centroid of the power transformed cluster corresponding to Y = with power ag, To get around this difficulty, we follow the approach in Yang et al. (2025) and introduce dummy distributions Qy E P((C)) for cach (y. i) to rewrite I(X; Y(:YYa, Y y) as follows
T(X; = y) = EXIx [KL (xx sp,0s) $ 1 =
= min Exiy [KL (x: Qu.)) = w).
(24)
Que
where the minimum in the above is achieved when
[o @x Qy.t Sy.a, Exir
(25)

---

Published in Transactions on Machine Learning Research (06/2025)
Combining equation 24 with equation
23
are led to solve the double minimization problem
N
min Exy H(Y 9x.00 + A Σ
tw min Expr [KL (1x Qui) I - w]} }
Pr In
exp
(26)
0
N
Qu.t
1=1
{Exy (((Y22x.0)
-mnn
min (Qy,)) (9y:)0e10).ee111
N
exp (wlxx [KL (xx Qn.)) 1 w11
+
Pylyl hn
(27)
N
When the distribution Px, is unknown, it can be approximated by its empirical distribution from a data sample (a training dataset or mini-batch thereof) D (I,,,))) The objective function in the double minimization equation 27 then becomes
JD(0,(9q.
Σ
g2.0)+
[D]
(a.N)ED
A
D
yl
Σ KL (1x (8 jED,
In
exp
(28)
N
Dayl
i=1
4.3 Algorithm for Solving the Optimization in equation 27
The double minimization optimization problem in equation 27 naturally lends an alternating algorithm that optimizes f and (e():en alternatively to minimize the objective function in equation 27 Equation 28 given the other is fixed.
Given {Q 1Q 0 can be updated using the same first-order optimization method as in conventional deop learning, such as stochastic gradient descent applicd over mini-batches.
Following Yang et al. (2025), given 0, for each class y, HEtN] can be updated according to equation 25
in the following manner: (I) we randomly sample mini-batch of samples [Byl instances from the training set with ground truth label y: (2) (Q,,) can be updated as
Erer, Loo
Q
Vi [N].
(29)
B
The proposed alternating algorithm for optimization problem equation 27 is summarized in Algorithm [ To simplify notation, use (-) to indicate parameters at the b-th batch updation during the t-th alternating iteration of the algorithm. We further write (-) as (-) whenever needed, set (-)6 = (-)--1
The compactness of output clusters alone is insufficient to ensure an undistillable DNN. Undis- tillability is a significantly stronger property. For instance, LS can improve the compactness of the fcature space of a DNN, which may make the output probability of the DNN more compact (Müller et al. 2019 However, this compactness not enough to make the DNN undistillable (see Section
Remark 2.
5 Experiments
this section, we demonstrate the effectiveness of CMIM by comparing with several state-of-the-at alternatives. Specifically, we first report the accuracy that knockoff student can achieve by deploying
2In practice solving the double minimization problem introduces only minor runtime overhead, the inner optimization problem has analytic solution and can be parallelized efficiently modern GPUs. To demonstrate the efficiency of CMIC, report the wall-clock training time and compare the computational overhead of CMIM and CE in Appendix 3if the impact of the random mini-batch sampling and stochastic gradient descent is ignored, the alternating algorithm guaranteed converge theory since given A, the optimal (Qy.)vve CCCCC can be found analytically via equation 29 although it may not converge global minimum.
8

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm 1: Conditional Mutual Information Minimized (CMIM) Method.
Input: Training set T, mini-batches (o6))) number of epochs T, X, 8, w, N Initialization: Initialize 0 and Qu ve(c),,e[N]
for = 1 to T do
[Sampling aul Randomly select N samples Haste from interval [0,
for to B do
[Updating Qu.i)
For each class y, construct mini-batch {By)eejo Update 2z,,, Vy E [C]; Vi E [N],
according to Equation (29
[Updating 0] Fix Q VE[CTAE[N]] Update 0/_, to 0 by stochastic gradient descent over the objective
function 28
end end
Output: Global model A².
different logit-based KD (attack) methods in Section 5.1 In all the experiments, when testing the distillibality of the trained DNNs using the benchmark defense methods and CMIM, we compare the knockoff student's 's accuracy (i) when it attempts to steal the IP of protected DNN using logit-based (attack) methods with (ii) when it trains its model using the L.S. If the former outperforms the latter, conclude that the knockoff makes the underlying DNN distillable. Next, in Section 5.2 report the classification accuracy of the protected models trained by the different defense methods. Lastly, in Section 5.3 we visualize the oulput cluster of models trained by CMIM, CE and NT.
5.1 Knockoff Student Accuracy
Datasets: We conduct extensive experiments on three image classification dataset, namely CIFAR-100 (Krizhevsky et al. 2012) TinylmageNNtt (Le Yang 2015) and ImageNet (Deng et al. 2009). For description of each datasct, please refer to Appendix
different model architectural families for teacher and knockoff student models. To this end, we pick models from VGG family (Simonyan & Zisserman 2015 ResNet family (He et al. 2016) (shortened as RN), ShuffteNetV2 (Ma et al. 2018), shortened as SNV2, and Mobilenetv2 (Sandler et al. 2018) shortened as MNV2. Particularly, we have conducted experiments on the following (teacher-student) pairs for each dataset: (i) for CIFAR-100, we use four pairs (VYGG16-VGGID
Models: To show the effectiveness of CMIM,
(VGG16-SNV2), (GG50) (RN50-RN18)); (ii) for TinyImageNe we use two pairs ((RN34-RN18), (RN50-SNV2)): and for ImageNet we use two pairs (RN3)
Defense benchmark methods: For comprehensive comparisons, benchmark CMIM with seven recently published defense melhods: MAD (Orekondy et al. 2020), APGP (Cheng & Cheng
2023 RSP
[ee et al. 2019), ST (Ma et al. 2022), NT (Ma et al. 2021 SNT (Wang et al. 2022). and L Müüllr
ct al. 2019).
Logit-based KD (attack) methods: We use three logit-based KD methods that are primarily designed for when the teacher-student models are in cooperating mode, namely KD (Hinton et al.) (015), DKD (Zhao ctt al. 2022), DIST (Huang et al. 2022a);; and four logit-based KD attacks methods that a knockoff student can deploy to make the protected DNNs possibly distillable, namely MKD (Yang & Yel 2024). HTC (Jandial let al. 2022), AVG (Keser & Toreyin 2023), Knockoff (Orekondy et al.) (211)). We report all the training setups, including all the hyper-parameters used for both defense and attack methods in Appendix G.]
The accuracy that a knockoff student can achieve using the various (defense-attack) combinations is summarized Table [ For accuracy variances, please refer to Appendix[] In the table, we use the notation "K-student" to denote knockoff student. The numbers in the column titled "Best" represent the highest
Results:
4 'Although LS is not a defense method per se, is observed that the models trained by LS reduce the knockoff student's accuracy. We discuss the rationale behind this Appendix C
9

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: Top-1 accuracy (%) of the knockoff student on CIFAR-100, TinylmageNet and ImageNet dataset
(the results for CIFAR-100 100 and TinyImageNet are averaged over runs). Green upward arrows (t) and
red downward arrows (U) indicate whether the knockoff student was able render the underlying DNN distillable.
CIFAR-100
Defense
Model
K-student
LS
KD
MKD
DKD
DIST
НТC
AVG
Knockoff
Best
VGG11
71.94
68.55
72.08
53.32
69.21
71.19
70.03
61.44
72,08
VGG16
SNV2
72.65
72.50
72.46
7.64
4
69.91
71.37
72.86
70.87
72.86
MAD
VGGI
71.94
72.00
72.04
54.29
71.57
70.76
70.73
61.73
72.04
RN50
RN18
78.76
77.76
78.79
43.73
73.76
77.89
78.61
73.92
78.79
VGG11
71.94
71.92
72.27
27.24 4
69.25
70.08 4
72.01 4
45.98
72.27
VGG16
SNV2
72.65
73.10
73.75
12.52
71.04
71.66 1
73.20
9.48
73.75
APGP
VGG11
71.94
71.91
72.11
9.74
69.48 4
71.36
71.92
34.71
72.11
RN50
RNIS
78.76
78.04
79.06
62.71 4
77.32 4
77.82 4
77.90 4
2.57
79.06
VGG11
71.94
71.42
72.04
70.22 4
70.80
70.40
71.56
31.04
72.04
VGG16
SNV2
72.65
73.55
72.95
67.45
72.19
71.46
72.27
73.55
RSP
VGG11
71.94
71.97
72.01
69.53 4
72.18
70.87
70.85
46.68
72.18
RN50
RN18
78.76
77.78
77.79
77.01
78.88
78.00
78.13
55.86
78.88
VGGL1
71.94
71.40
73.44
71.47 4
71.33
70.77 4
71.58
63.56
73.44
VGG16
SNV2
72.65
72.44
72.70
6.24
72.04
70.75
72.83
6.32
72.83
NT
VGGII
71.94
72.01
72.03
71.55
71.88
70.16
71.94
62.94
72.03
RN50
RN18
78.76
78.41
78.92
79.26 1
78.99 f
77.94 1
78.33 L
68.96
79.26
VGG11
71.94
72.06
72.28
4.92
71.98
70.60
71.63
64.08
72.06
VGG16
SNV2
72.65
72.94
73.17
72.78
72.22
71.22 4
72.74
6.22
73.17
SNT
VGG11
71.94
72.02
72.12
72.32
71.70
70.66
71.65
62.94
72.32
RN50
RN18
78.76
78.25
78.48
78.82
78.14
78.45
78.38
67.71
78.82
VGGIT
71.94
72.09
72.01
71.63
71.93
71.16 4
71.63 4
63.32
72.09
VGG16
SNV2
72.65
72.64
72.67
70.53
72.24
71.32
72.42
69.46
72.67
ST
VGGI1
71.94
72.00
72.13
71.62
71.76
70.54
71.73
65.43
72.13
RN50
RN18
78.76
78.96
79.02
78.35
78.31 4
78.36 4
78.81 f
72.87
79.02
VGGLI
71.94
71.90
72.00
71.57
70.89
70.66
71.76
63.49
72.00
VGG16
SNV2
72.65
72.87
73.52
70.01 4
71.49 4
71.70 4
73.01
65.20 4
73.52
LS
VGG11
71.94
71.82
71.99
71.95 4
70.77 4
70.86 4
71.88
62.29
71.99
RN50
RNI8
78.76
77.72
77.82
79.37
78.33
78.31
77.91
63.36
79.37
VGGI
71.94
71.87
71.64
71.56 |
70.34 4
71.71 1
71.42
66.89
71.87
VGG16
SNV2
72.65
72.53
71.44
72.46
71.45 J.
71.59
71.94
64.45
72.53
CMIM
VGGII
71.94
71.54
71.34
71.77
4
71.86
69.32
71.70
60.58
71.86
RN50
RN18
78.76
78.21
78.16
78.13
77.56 4
77.23
78.64
65.88
78.64
T Tiny ImageNet
RN34
RNIS
63.56
63.54
64.32
64.01 1
63.27
63.54 1
62.15
55.43
6-1.32
RSP
RN50
SNV2
60.61
60.18
60.76
56.26
4
56.43
60.96
60.15
54.01
60.96
RN34
RNIS
63.56
63.96
64.12
63.25
4
63.51
63.49
1
63.84
57.42
6.1.12
ST
RN50
SNV2
60.61
61.23
61.36
60.43
60.32
60.22
61.13
55.84
61.36
RN34
RNI8
63.56
63.27
64.49
64.67
1 63.43
63.50
64.43
53.11
64.67
NT
RN50
SNV2
60.61
59.57
61.55
31.55
1 60.03
60.98
60.31
50.94
61.55
RN34
RNIS
63.56
63.74
64.01
64.23
1 63.51
64.20
63.04.
57.43
64.23
LS
RN50
SNV2
60.61
60.32
60.93
60.74
60.11
60.46
60.14
52.96
60.93
RN34
RNIS
63.53
62.89
63.15
62.94
63.28
61.57
62.96
56.13
63.28
CMIM
RN50
SNV2
60.61
57.57
59.32
60.58
59.41
59.33
60.42
56.91
60.58
ImageNet
RN18
70.89
70.74 74
71.02
70.02 4
69.94 4
70.91 1
71.00
63.24
71.02
ST
RN34
MNV2
70.93
71.03
71.25
69.32
70.53 |
70.69 |
71.06 1
54.53
71.25
RN18
70.89
70.44
70.69
69.97
70.59
70.63
70.53
59.34
70.69
CMIM
RN34
MNV2
70.93
70.21
70.72
69.97 4
70.44 4
70.86 1
70.20 J.
55.24
70.86
10

---

Published in Transactions on Machine Learning Research (06/2025)
accuracy obtained for each respective row, indicating the best possible performance knockoff student achieve using any of the listed distillation methods.
As observed in Table [ regardless of whether the teacher-student architectures are the same or different, DNNs trained with CMIM remain undistillable across all distillation methods. This is in stark contrast to DNNs trained using other defense techniques, which can still be successfully distilled a certain degree. The results indicate that prior defense strategies do not offer complete resistance against knockoff students, whereas CMIM effectively prevents distillation, making it significantly more robust in protecting model knowledge.
We also perform an ablation study on the hyperparameters of CMIM namely B, N, and w as detailed in Appendix K The key findings are as follows:
Effect of 3: The accuracy of the knockoff student drops sharply when 2 1, suggesting that large values may destabilize optimization excessively penalize the CMI term. This highlights the importance of carefully tuning 3 to effectively balance the cross-entropy and CMI objectives.
Effect of the number of samples N: Increasing the number of power samples N leads to a monotonic decrease the knockoff student's accuracy. This indicates that moderate values of are sufficient to capture the necessary diversity for robust CMI estimation.
Effect of the power coefficient w: Setting > 30 can lead to numerical instability, resulting in NaNs due to excessive exponentiation. Interostingly, the worst knockoff student performance is observed at w 20 and = 30, suggesting that these settings best approximate the extreme concentration behavior of w + oc, thereby enhancing undistillability.
Additionally, we investigate CMIC's impact on model calibration in Appendix M
5.2 Accuracy of Protected Models
In this section, we report the top-1 accuracy of the protected models in Table D trained using the benchmark defense methods with those trained by CMIM. The results are summarized in Tables 2 and 3 As observed, the models trained by CMIM have the highest classification accuracy compared to the benchmark methods. This is because for the models trained by CMIM, the clusters corresponding the output probability of the DNNs very concentrated, facilitating easier classification of samples from different classes.
Table 2:
Top-1 accuracy (%) of models trained by defense methods on CIFAR-100 and TinyImageNet. The best and second best results are bolded and underlined, respectively.
CIFAR100
TinyImageNet
Model
CE
MAD
APGP
RSP
ST
NT
SNT
LS
CMIM
Model
CE
RSP
ST
NT
LS
CMIM
VGG16
73.75
73.75
73.84
73.71
73.75
73.75
72.59
73.90
73.84
RN34
65.39
65.21
65.39
65.23
65.45
65.99
RN50
77.81
77.81
77.56
77.63
77.81
77.31
77.77
78.45
78.72
RN50
66.14
65.91
66.13
66.06
66.09
66.93
Table 3: Top-1 accuracy (%) of models trained by defense methods on ImageNet.
ImageNet
Model
CE
ST
CMIM
RN34
73.31
73.30
73.69
The results in Table motivate us to test the top-1 accuracy of additional models trained by CMIM and compare them with those trained by CE loss (see Appendix
It worth noting that the primary focus of this paper not on incrcasing the accuracy of DNNs but on developing method to train undistillable DNNs. While many existing methods in the literature can enhance DNN's accuracy, they do not address the critical challenge of making DNNs undistillable.
11

---

Published in Transactions on Machine Learning Research (06/2025)
(a) LS
(b) Nasty teacher
(c) CMIM
Figure 2: Visualization of three projected probability clusters for ResNet-50 trained on CIFAR-100 using (a)
LS, (b) NT, and (c) CMIM.
Our approach is the first in the literature that effectively trains undistillable models robust against wide range of existing KD methods. The improvement in accuracy observed in our results is by-product of our method and not its primary goal. This improvement arises from the unique properties of our approach rather than replicating the effects of label smoothing or similar techniques.
5.3 Visualizing the Output Clusters
In this subsection, we aim to visualize the output clusters for models trained using CE. NT, and CMIM. To achieve this, we follow the visualization approach introduced by Yang et al. 22025) Specifically, we randomly select three labels from the CIFAR-100 dataset. For each probability distribution corresponding to these three labels, we extract only the probabilities associated with these selected labels and normalize them to form threo-dimensional probability vectors. These vectors are then projected onto a two-dimensional simplex, allowing us to visualize the clustering behavior of each model. By applying this transformation, obtain clear representation of how the models distribute their probability mass across different output categories.
The resulting simplexes for ResNet-50 models trained with LS, Nasty Teacher, and the CMIM framework are shown in Figure 6 To ensure a consistent comparison, we applied the same power transform a = 4 for all visualizations. As observed, the clusters for the model trained with CMIM are highly concentrated near the corners of the simplex, closely resembling one-hot vectors. This indicates that the output distributions are highly compact, making it difficult for knockoff student to surpass LS regularization when attempting to distill knowledge from CMIM-trained model. This increased compactness plays a crucial role in enhancing the model's resistance to knowledge distillation.
Lastly, we elarify the distinction between "highly concentrated output clusters" and "overly confident pre- dictions". A highly concentrated output cluster does not necessarily imply that the model produces overly confident predictions. This is because the clusters can be concentrated around points that are not close to one-hot labels (the corners of the probability simplex). As result, the model can have concentrated outputs without being overly confident. These two separate concepts.
To illustrate this, we train an RN50 model on the CIFAR-100 dataset using three different methods: CMTM, CE, and LS. After training, we evaluate the model's confidence by measuring the average entropy of its output probability vectors on the test datasct. The results, presented Table 4 indicate that the entropy for CMIM is higher than that for CE, demonstrating that CMIM produces less confident output probabilities.
Additionally, our experiments in Table g further support the above-mentioned claim: the CMIM method generates more concentrated output clusters while also improving the model's accuracy on the held-out test dataset compared to models trained with the conventional CE loss. This observation is consistent with the
iIn Appendix C-FAR-100.
we present the visualization of three different projected prohability clusters for 5e trained
12

---

Published in Transactions on Machine Learning Research (06/2025)
Table Entropy Value of the Models Trained by CMIM, CE and
Methods
CMIM
CE
LS
Eutropy
0.102
0.064
0.152
CE
0.150
MAD
0.05
APGP
0.125
RSP
CM
SNT
0.075
CMIM (aurs)
0.02
0.000
0,000
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
(a) ResNet-50
(b) VGG16
Figure 3: The CMI values for the models trained by different KD-resistance defence methods Vs. power transform value a for (a) ResNet-50, and (b) VGG16 trained on CIFAR-100 dataset. Compared to all other defense methods in the literature, CMTM effectively reduces peak CMT values under temperature scaling, which prevents the teacher model from being distilled by the knockoff student.
findings of Yang et al. (2025 which demonstrate that training DNNs to produce highly concentrated output clusters can lead to improved test accuracy.
5.4 Why Prior Defense Methods Can be Made Distillable?
In this section, we address the question of why DNNs trained using all prior KD-resistance defense meth- ods can still made distillable, as demonstrated in Section 5.1 The key reason behind this lies in the fundamental characteristics these defense methods and their susceptibility to distillation under certain conditions. Specifically, by appropriately tuning the power transform parameter a, the models trained using these defense methods can attain relatively high CMI value in comparison to proposed method, CMTM (as illustrated in Figure 3 As though, compared to the CMIM model, all other defense methods provide more information for the student to leverage in improving their own performance.
This suggests that, despite their initial resistance, the defense methods fail to enforce undistillability rigor- ously across distillation settings. When the correct a value is selected during the distillation process, a logit-based KD approach can leverage this property effectively distill knowledge from these supposedly resistant DNNs. Consequently, these models can still be exploited produce distilled students that outper- form the baseline LS student, demonstrating that prior defense strategies do not provide robust protection against all KD methods.
6 Conclusion and Future Directions
In this paper, from information-theorrtii posed by knockoff students attempting to steal the IP of pre-trained DNNs via based KD methods. In particular, we proposed to minimize the CMI of the protected DNN across different power transform hyper- parameter values a, while minimizing the conventional CE loss simultaneously. We referred to model trained by these framework as CMIM models. By conducting a series of experiments, we showed that, unlike the prior defense methods proposed in the literature, a knockoff student cannot render CMIM models distillable. In addition, we showed that the models trained by CMIM achieve higher classification accuracy compared to those trained by CE loss.
perspective, proposed defence method against the threat
13

---

Published in Transactions on Machine Learning Research (06/2025)
Despite these promising results, our work has certain limitations. First, the evaluation of CMIM models is primarily empirical, as providing a formal theoretical proof of undistillability remains open challenge. Second, our approach introduces additional computational overhead compared to the conventional training using CE loss.
For future work; we aim to extend the CMTM method beyond standard classification tasks. Potential direc- tions include adapting it for multi-label classification, regression problems, and safeguarding the intellectual property of cutting-edge models such LLMs, CLIP, and diffusion models. Another promising direction is to enhance the model's undistilabiility by adapting CMIM for integration with contrastive learning frame- work, which regularizes the latent space rather than the output probability space. By broadening the scope of CMIM, we hope to further enhance its applicability and effectiveness across a wider range of machine learning paradigms.
References
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self- distillation in deep learning, arXiv preprint orXiv:2012.0955 2020.
Jure An, Doetsch Peter, and Pylvänäinen Thomas. Relation knowledge distillation. In International Con-
ference on Learning Represenlations, 2021.
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E Dahl, and Geoffrey E Hinton. Large scale distributed neural network training through online distillation. arXiv preprint (XXi:1804.03295 2018.
Kenneth Borup and Lars N Andersen. Even your teacher needs guidance: Ground-truth targets dampen regularization imposed by self-distillation. Advances in Newral Information Processing Systems, 34:5316- 5327, 2021.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535 541, 2006.
Anda Cheng and Jian Cheng. Apgp: Accuracy-preeerving generative perturbation for defending against model cloning attacks. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023. doi: 10.1109 D0.1099/CAASSP49337. 002.:/333372221009999990
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei- Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Ieee, 2009.
Kangning Guo, Hu Shengyuan, Yan Junjie, Liu Xin, Xu Dongbao, and Wang Ningbo. Logit-like knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10186 10193, 2021.
Shayan Mohajer Hamidi, Xizhen Deng, Renhao Tan, Linfeng Ye, and Ahmed Hussein Salamah. How to train the teacher model for effective knowledge distillation. In European Conference Computer Vision, pp. 18. Springer, 2024.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pallern Recognition (CVPR), pp. 770 778, 2016. doi: N0.1109/CVPR.200
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. URL https://arxiv. www.aaawww..gg.r/////aaaaa.. orw/aba/ 1503/ 125tt/253300001111111111
Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from a stronger teacher, In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022a. URL httppp//pppaaaaoaea.eeeee/ooooas
net./fooumnmddd117ll kbi
14

---

Published in Transactions on Machine Learning Research (06/2025)
Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from a stronger teacher. Aduances in Newral Information Processing Systems, 0-33376--3772) 2022b.
Surgan Jandial, Yash Khasbage, Arghya Pal, Vineeth N. Balasubramanian, and Balaji Krishnamurthyy Distilling the undistillable: Learning from nasty teacher. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022, pp. 587 603, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-19778-9,
Reyhan Kevser Keser and Behcet Ugur Toreyin. Averager student: Distillation from undistillable teacher, 2023. URL htttps://openrevier: wtt.://oooommmmiiii1/1111111111
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.
aiv preprinl
arXiv:1412.6900 6980, 2014.
Alex Krizhevsky, Vinod Nair, and GeolTrey Hinton. Cifar-10 (canadian institute for advanced research). University of Toronto, 2012. URL http::/www.cc cs. www.ttorttoo..oooooooooooooo...t serrnto..edu/-kriiccifaee html
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015. URL https://api. semanticscholas Trg//oorpsID: wtgg/oorpusD::16647999 16664790
Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Newral networks: Tricks of the trade, pp. 9 50. Springer, 2002.
Taesung Lee, Benjamin Edwards, Ian Molloy, and Dong Su. Defending against neural network model stealing attacks using deceptive perturbations. In 2019 IEEE Security and Privacy Workshops (SPW), pp. 43 49, 2019. doi: 10.1109/SPPI 2019.00020.
Haoyu Ma, Tianlong Chen, Ting- Kuei Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Undistillable: Making nasty teacher that cannot teach students. In International Conference on Learning Representa- tions, 2021. URL htttps:://opnreeviee: net/forummiddoovveeeeree
Haoyu Ma, Yifan Huang, Tianlong Chen, Hao Tang, Chenyu You, Zhangyang Wang, and Xiaohui Xie. Stingy teacher: Sparse logits suffice fail knowledge distillation, 2022. URL https::
forum?id=ae7BJJ00x
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenct v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In International Conference on Machine Learning, pp. 2021.
PMLR,
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351 3361, 2020.
Daniel Moldovan, lonescu Bogdan, Drimbä Alexandru, and Marius Popescu. Path-kg; Knowledge distillation with lath-level guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
pp.
Rafael Müller, Simon Korbllith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 4954-4963, 2019.
Tribhuvancsh Orekondy, Bernt Schicle, and Mario Fritz. Prediction poisoning: Towards defenses against dnn model stcaling attacks. In International Conference on Learning Representations, 2020. URL https:
15

---

Published in Transactions on Machine Learning Research (06/2025)
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE Conference on Computer Vision and Paltern Recognilion, pp. 3967-3976, 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions, 2017. URL httpss://pperrevie: net/forumiid- http:/pparrrrreeee.ee/fffrrrtttt
HkCjNI5ex
Mary Phuong and Christoph Lampert. Towards understanding knowlodge distillation. In International conference on machine learning, pp. 5142 5151. PMLR, 2019.
Radim Rehurck and Petr Sojka. Kensim-python framework for voctor space modelling. NLP Centre, Faculty of Informalics, Masaryk Universily, Brno, Czech Republic, 3(2), 2011.
Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Mathematical Slatistics, 22(3):400 - 407, 1951. doi: 10.1214 (0.121//aoms//177779555550 URL http::::///oii.r////.... hatp://.0..1///aa00 /omm// 1177729586
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatla, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint orXiv:1412.65500 2014.
Ahmed H Salamah, Shayan Mohajer Hamidi, and En-Hui Yang. A coded knowledge distillation framework for image classification based on adaptive jpeg encoding, Pattern Recognition, 158:110966, 2025.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. In 2015 53rd Annual Aller- ton Conference on Communication, Control, and Computing (Allerton), pp. 909 910, 2015. doi: 1A.100/AALERRTTO2200.......
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Zi Wang, Chengeheng Li, and Husheng Li. Adversarial training of anti-distilled neural network with semantic regulation of class confidence. In 2022 IEEE International Conference on Image Processing (ICIP), pp. 3576 3580, 2022. doi: 10. N0.1109/ICIP44570 20t.220022..9....
En-hui Yang and Linfeng Ye. Markov knowledge distillation: Make nasty teachers trained by self-undermining knowledge distillation fully distillable. In European Conference on Compuler Vision. Springer, 2024.
En-hui Yang, Shayan Mohajer Hamidi, Linfeng Ye, Renhao Tan, and Beverly Yang. Conditional mutual information constrained deep learning: Framework and preliminary results. In 2024 IEEF Internalional Symposium on Information Theory (ISIT), pp. 569 574, 2024. doi:
En-Hui Yang, Shayan Mohajer Hamidi, Linfeng Ye, Renhao Tan, and Beverly Yang. Conditional mutual information constrained deep learning for classification. IEEE Transactions Neural Networks and Learning Systems, pp. 1-13, 2025. doi:
Linfeng Ye, Shayan Mohajer Hamidi, Renhao Tan, and En-hui Yang, Bayes conditional distribution esti- mation for knowledge distillation based on conditional mutual information. In The Twelfth International Conference Learning Representations, 2024. URL
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the LEEE/CVF Conference on computer vision and pattern recognition, pp. 11953 11962, 2022.
Kaixiang Zheng and ENHUT YANG. Knowledge distillation based on tranformed teacher matching. In The Twelfth Internalional Conference on Learning Representations, 2024. URL https://openreviess 2orum22a=
16

---

Published in Transactions on Machine Learning Research (06/2025)
A Appendix
B A Thorough Study of Related Works
B.1 Logit-based KD methods
Knowledge Distillation (KD) has become cornerstone technique for compressing large teacher models into smaller student modcls. This section reviews key logit-basod KD methods and explores their advancements.
Hinton et al. (2015) introduced the foundational concept of KD, using KI, divergence to match the student's softmax outputs to the teacher's. This work laid the groundwork for logit-based methods, as the softmax output directly relates to the logits. Later, Moldovan et al. (2019) proposed Path-KD, a mothod that utilizes the paths leading to the correct class in both teacher and student models for distillation. While not directly logit-based, demonstrates the effectiveness of aligning decision-making processes. Guo al. (2021) proposed Logit-Like Distillation, addressing the capacity gap by matching the ranking of logits instead of their exact values [4]. This approach allows the student to learn the essential ordering of classes even with limited capacity. An et al. (2021) proposed relation knowledge distillation (RKD), focusing aligning relationships between class logits rather than individual values. This approach improves the student's ' ability to generalize to unseen data. Zhao et. al.] (202)) rodueed decoupled knowledge distillation (DKD), where il decouples the classical KD loss into two parts: target class knowledge distillation and non-target class knowledge distillation. Huang et al. (2022b) proposed DIST where they designed a KD method to distill better from stronger teacher; indeed they claim that preserving the relations between the predictions of teacher and student would suffice for an effective KD. Borup & Andersen (2021) provided theoretical arguments for the importance of weighting the teacher outputs with the ground- truth targets when performing self-distillation with kernel ridge regressions along with a closed form solution for the optimal weighting paramcter. Hamidi et al.
2024h propose to train the teacher model using mean square loss instead of the conventional cross
entropy loss. Salamah et al. (2025) proposed a KD framework that leverages adaptive JPEG encoding to make the teacher's output responses less peaky for image classification tasks.
B.2 Defense methods against Logit-based KD
As also discussed in Section 2 the defense methods against the threat posed by knockoff students attempting to steal the IP of pre-trained DNNs via logit-basod KD methods can be categorized into two approaches. Here, we elaborate on these two approaches.
In this approach, DNNs are trained to reduce the accuracy of knockoff students while maintaining the original classification accuracy of the model. In particular, Ma et al. (202)) proposed a training algorithm named sell-undermining KD to create nasty teachers (NT) that prevent knowledge leakage and unauthorized model stealing through KD, without compromising model accuracy. The nasty teacher trained by minimizing the following objective function:
(I) Model stealing resistant training:
NT H(y. 9z) KL( qu),
(30)
where Tz is a output of a pre-trained standard model.
Subsequently, Wang et al. [022)] proposed semantic nasty teachers (SNT) which improve the model stealing resistance of NT by disentangling semantic relationships in the output logits during teacher model training, which is crucial for successful KD.
(II) post-training defence methods: The aim of these approaches is to deceive the knockoff by imposing minimal perturbations to the model's predictions. Lee et al. (2019) tested variety of possible perturbation forms, and found that the reverse sigmoid perturbation (RSP) to be the most effective one. Orekondy et al. (2020) introduced maximizing angular deviation (MAD), a technique that perturbs the output probabilities, lcading to an adversarial gradient signal that deviates significantly from the original gradient of the knockoff. To this end, they applied randomly initialized model as the surrogate for the potential knockoff. More recently; Cheng & Cheng (0023) proposed plug-and-play generative perturbation model, dubbed accuracy
17

---

Published in Transactions on Machine Learning Research (06/2025)
preserving generative perturbation (APGP), which effectively defend KD-based model cloning, while preserve the model utility.
B.3 Attack Methods Using Logit-based KD
Jandial al. (2022) sought to circumvent the defense of nasty teachers and steal (or extract) its information. Specifically, they analyzed nasty teacher from two different angles and subsequently leverage them carefully to develop simple yet elficient methodologies, named as HTC and SCM, which enhance learning from nasty teacher.
In AVG Keser & Toreyin 2023), the authors noted that undistillable teachers exhibit multiple peaks in their softmax response, which are transferred the student models. These peaks are considered be the primary factor that mislcads the student models. To mitigate the influence of the multiple peaks in the softmax response of teachers, they proposed transferring the mean of features with the same labels the soft labels.
Orekondy et al. 2019) introduced technique called "Knockoff Nets" that allows an attacker to steal the lunctionality of black-box models. Remarkably, the atlacker only needs to interact with the model by feeding it input data and observing the resulting predictions. By training a new model ('knockoff") on these apputpprediction pairs, the attacker can create a copycat model that performs similarly to the original black box.
C Why LS reduce the knockoff student's accuracy?
Original aiming to prevent overfitting and improve generalization, label smoothing was observed by Müller et al. 2019) to reduce the accuracy of the knockoff student. The researchers found that "label smoothing encourages examples to lie in tight, equally separated clusters'. Consequently, label smoothing reduces the contextual information in the teacher model's output (Yang al. 2024).
D Power Transformation of Mutual Information
The following theorem implies that for each label (Xx = y) as a function of is continuously differentiable.
Theorem 2.
Let (X, 2) be a pair random variables, where Z is discrete, and X can be either discrete or continuous. Let P2ix|Ha denote the conditional probability distribution
given = x. Additionally, let [+1x] denote the power transformed version of PzixI-aa with power a, Z denote the random variable the conditional distribution of which given X = x is Pa x [ac, and 9t denote the probability distribution of za Then, the following holds
i lm)
1
Σ
Px|a] (m2 - m((P2 H Cov(P
(31)
a
where for probability vectors P and Q
(P) A
Pbl( 1011--nPP (P(()) H(P), (Shannon entropy)
(32a)
(P(P) A Σ Pb](- IIn(P)))) (Second moment)
(32b)
Cov(P,Q) e Σ
plb] (- (I((P)) - m1 In(()) - E p(]((-n(Q(i))
(32c)
3
18

---

Published in Transactions on Machine Learning Research (06/2025)
Proof. To simplify our notation, denote the conditional distributions Pz x||aa]] and P2 x [ā|aa] by Pa and Ps, respectively. Decompose [[X; 2 as follows
1(x;2") - -1(Z)- 1 = =H(g9) - Σ Ppt]H(p))
(33)
where for any random variables U and V, H(V and UVV are the entropy of V and the conditional entropy of V given U, respectively, and (p) denotes the entropy of the probability distribution p. Then the partial derivative of 2") w.r.t. a is equal to
01(X
(H(
2H(
Σ
P [
(34)
da
dx
da
To continue, we first compute the partial derivative in the second term of the RHS of equation 34
aH(
-
pi [i] In(p(()
Σ (mbb)(()+ +1)
Op%
Oa
da
aa
Σ (m(e2b61)+ +1)
(px 10) In(p
E, (p(li))))) (b=bi))
( in
Σ, (eallt)))
Σ (n(p + 1) (1l()(
Σ ps In(x(())))))
-1 Σ [n(p2:] (7)) p. 1sl (n(px()))
Σ
ps [il (1))
(35)
a
1
Σ [I((Cb)) + p. Lil Im(eاD)))
Σ
of 11
Σ ps [i] (In(P(bi))
Σ
pf Pfb bi))
(2())) - m (v
(36)
Note that
J = Σ Plxp-
19

---

Published in Transactions on Machine Learning Research (06/2025)
Then we have
(H()))
-0E, , (0] --E (mn(P()) +1)
da
da
da
3p
Σ [n(( + +1) Σ
Pla]
da
-1
Σ (In()"())) + +1) Σ PixlpSlil
(In(ll (In(p[[))
mi (p?))
(37)
-1
Σ (I((9((1))
Σ
Pi (n(p5(D) + (m)))))))++)))))))))
-1
Σ
P(x]
(n(0 p (n(pp()) + m1 mP()
Σ P(x] Σ ph In(p In((((7))
x
m1 (p
Σ D Pilil (
Pax] Cov (p
(38)
where equation 37 is due to equation 35
From Equations
36 and 38
Theorem follows.
E Proof of Theorem 1
Theorem 1 follows from Theorem and the following lemma.
Lemma 1. Let g(t) be continuously differentiable function over [0, 3]. Then the following holds:
1 In w-00 w 3
f exp (wg(t))dt.
max gt) = lim
(39)
Proof. Let t" be optimal point at which
g((") max g())
For any > let ((t,,e) denote closed interval containing with length It easy to verify that which implies that
1 In 3
1
8 exp (u
lim sup 1 In
1o exp (wg((g( s g((").
w+x w 3 [
(40)
On the other hand,
1 ld exp (wg(1))dt
1 1 /
In 3
2 - 3 3 [((
exp (d
1 In exp (w min g(()}
w
1EN(t".))
min g(t) + In e
1
(41)
(EN
g
20

---

Published in Transactions on Machine Learning Research (06/2025)
Letting w so in equation 41 yields
lim inf In 1 3
B /
exp (wg()))tt
min g()..
(42)
(EN()).))
Note that equation 42 is valid for any c <> Letting - 0 in equation we have
1
/ exp (wg(t))d 2 g(t)).
lim inf 400
In
(43)
w B
Then equation
39 follows from equation
40 and equation
43 This completes the proof of Lemma !
F Datasets description
CIFAR-100 (Krizhevsky et al. 2012) dataset contains 50K training and 10K test color images, each with size x 32, categorized into 100 classes.
TinyImageNet (Le & Yang 2015) contains 120K color images across 200 classes, each with a reso- lution of 64 x 64 pixels. For each class, there are 500 training images, 50 validation images and test images.
ImageNet (Deng et al. 2009 is a large-scale dataset used in visual recognition tasks, containing around 1.2 million training and 50K validation images.
G Experiments setup
All experiments detailed in this paper were conducted using publicly available national high-performance computer. For cach experiment, utilized 16 CPU cores, 64 GB of memory, and one NVIDIA V100 GPU. The software environment comprised Python 3.10, PyTorch 1.13, and CUDA 11.
For all experiments, including defenses and attacks, the SGD optimizer (Robbins & Monro ct al. 2002) with a learning rate of 0.1 used unless otherwise specified.
1951
LeCun
For the CIFAR-100 and TinyImageNet datasets, we train the model for 200 epochs, decaying the learning rate by 0.1 epochs 60, 120, 160.
For ImageNet, we follow the standard PyTorch practice G
The batch size is 128 for both CIFAR-100 and TinyImageNet, and 256 for ImageNet.
To get the accuracy that a knockoff student can achieve using label smoothing, we have tested a wide spectrum of label smoothing factor = {0.01, 0.05, 0.1, 0.15,,111..0 0.2, 0.25, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9}, and selected the value that yielded a classification accuracy excceding that of all knockoff students.
In the CMIM method, set T 20 and tested À 0.25, 0.5, 1, selecting the value that minimized the CMI value while maintaining improving classification accuracy.
G.1 Defense setup
We used the following parameters and settings for the defense models used in Section 5
G.1.1 Defense setup on CIFAR-100 and Tinylmagenet
MAD: We employ a randomly initialized VGG-8 as adversary's architecture, and following the implementation of MAD-argmax.
2. APGP: We apply a 3 layer MLP with residual connection as the generative model and set À = 0.1 for all experiments.
21

---

Published in Transactions on Machine Learning Research (06/2025)
3. RSP:
We
and B 20 for all the experiments.
NT: To ensure a acceptable accuracy sacrifice, we test three different values and select the largest one that results in accuracy drop of less than 0.5%. Specifically, we 0.01 for ResNet-50 and 0.005 for VGG-16 the CIFAR-100 dataset, while for TinyImageNet, we use 0.001 for both ResNet-34 and ResNet-50.
SNT: We use the pretrained word2vect model namely "fasttext-wiki-nnee www---i-eees-----oddd33000000000 provided by Gensim (Rchurck & Sojka 2011), and set X = 0.2 for all experiments.
ST: We use the teacher model trained by CE as the underlying model, and use the sparse ratio of 10% as suggested in their paper for all experiments.
7. LS: We apply label smoothing factor Of 0.05 for all experiments.
8. ISTM: We set the binary search parameters to Т₁ 20 and Qmax 2000. We use X = 0.2 for ResNet-50 and À = 0.5 for VGG-16 on the CIFAR-100 datasct, while for TinylmageNet, we use
À = 0.1 for ResNet-34 and = 0.5 ResNet-50.
G.1.2 Defense setup on Imagenet
1. ST: We use the teacher model trained by CE the underlying model, and use the sparse ratio of 10% as suggested in their paper for all experiments.
ISTM: We set the binary search parameters To 20 and ammx 2000. We use X 0.2 for all the experiments.
G.2 Attack setup
G.2.1 Attack setup on CIFAR-100 and Tinylmagenet
We use power transform parameter = 0.25 (or equivalently T 4) for all experiments unless otherwise specified.
1. KD: We set the CE-KL trade-off coefficient to À 0.9.
MKD: We use the intrinsic dimension of for CIFAR-100, and for TinylmageNet. We em- ployed the Adam optimizer (Kingma & Ba 2014 with learning rate 10-3 for the trainable Markov transform.
DKD: We test alpha, beta pairs of (1, 4) and (2, 8), and report the one with best accuracy.
4. DIST: We useß = 1.0, 1.0, r = 1.0 for all experiments,
5. HTC: We use 0.05(T 20), À 0.01 for all experiments.
AVG: 0.9.
7. Knockoff: We follow the implementation of the original paper.
G.2.2 Attack setup on Imagenet
We use a (T 1) for all experiments unless otherwise specified.
KD: À = 0.9.
2. MKD: We use the intrinsic dimension of 16. We employed the Adam optimizer Kingma & Ba
2014) with learning rate 1 for the trainable Markov transform.
3. DKD: We test alpha, beta pairs of (1,4) and (2, 8), and report the with best accuracy.
22

---

Published in Transactions on Machine Learning Research (06/2025)
H Accuracy of Protected models
In this section, we report the top-1 accuracy of some additional models that are trained using CMIM and compare them with those trained by CE method. To this end, we use 10 well-known models for CIFAR- 100 dataset namely ResNet (RN)-(18, 34, 50, 101, 152), SqueezeNet (SQN), ResNext (RNXT) 50, MobileNet (MN), Xception (XCP), DenseNet (DN) 121; and 2 models namely RN-(34, 50} for TinylmageNet and ImageNet. We follow the same training recipe as the one in Section 5.1 The results for CIFAR-100 and (Tanee)laggeNet are listed in Table 5 and Table
G respectively. As seen, the top-1 accuracy for all models trained by CMIM is consistently higher than those trained by CE counterpart, with the gain up to 1.15%.
Table 5: Top-1 accuracy (%) of models trained by CE and CMIM methods on CIFAR-100.
CIFAR-100
Model
CE
CMIM
Model
CE
CMIM
RN18
76.05
77.20
SQN
69.32
70.64
RN34
77.20
77.54
RNXT50
78.71
79.12
RN50
77.81
77.93
MN
67.26
67.51
RN101
79.07
79.12
XCP
77.37
77.64
RN152
79.21
79.43
DN121
79.16
79.33
Table 6: Top-1 accuracy (%) of models trained by CE and CMIM methods on TinylmageNett and ImageNet.
Tiny Tet
ImageNet
Model
CE
CMIM
Model
CE
CMIM
RN34
65.39
65.99
RN34
73.31
73.69
RN50
66.14
66.93
RN50
76.15
76.40
23

---

Published in Transactions on Machine Learning Research (06/2025)
Variance of Table 1
Table 7: Top-1 accuracy (%) and variance of the knockoff student on CIFAR-100 and TinyImageNett datasct (averaged over 3 runs)
CIFAR-I00 100
Defense
Model
K-studeut
LS
KD
MKD
DKD
DIST
ITC
AVG
Knochoff
VGGII
71:9110:00
68.5520.221
72.0820.29
55.3210.33
09:21220009
71.991008
70.0320.007
61.110006
VGG16
MAD
SNV2
72.66+0.16
72.00+0.11
72.40+0.113
7.64 + 0.70
09.91 + 0.18
T1.37+0.24
72 72.86+00.18
70.87=0.26
VGGII
71.94+0.09
72.00+00.21
72.0440..13
54.29400.52
71.57
70.764-0.20
70.73:40.22
61.73=0.23
RN80
RNIS
78.76+0:06
77.76+0.29 23
78.70++0.2
43.73+0.55
73.76=0010
17.80+0.25
78.61+0.155
13.92=0.19
VGG16
VGGII
71.91=0.9
T1.92+0.1 19
72.27:0.21
27.21+0.51
09. 09.25=0014
70.08+0.225
72.01+00.00
45,98=0.45
APGP
SNV2
72.65=00188
73.1020.27
78.75:10.23
12.5220.30
71.0120:31
71. 71.66200115
73.20:10.27
9.48
+ 0.73
RN50
VGGII
0109
71.91
0.17
72.11
+ 0.23
9.74 +
09.48
0.11
T1.38
0.25
71.92
34.71
0.30
BN18
78.76 = 0.08
78.01 +: 0.21
79.06
+ 0.14
62.71
+ 0.29
77.32 + 0.13
77.82
+ 0.13
77.90
+ 0.15
2.57
+ 0.95
VGG16
VGGIL
71.94 2: 0.09
71.42
0.24
72.04
1: 0.13
70,22
1: 0.19
70.80
0.17
70.40
0.17
71.56
1: 0.06
31.04
2: 0.62
RSP
SNV2
72,65
0.18
73.55
0.34
72.95
+ 0.96
67.45
1: 0.20
72.19
0.44
71.46
0.41
72.27
+ 0.27
26.09
0.40
VGGH
71.91
= 0.09
71.97
1 0.17
72.01
1 0.20
69.53
+ 0.12
72.18 = 0.21
70.87
2: 0.14
70.85
1 0.17
06.68
= 0.60
RN50
RNIS
78.76
0.08
77.78
0.09
77.79
77.01
78.88
78.00
0.28
78.13
55.86
VGG16
VGGIE
71,94
0.00
71.40
0.34
73.44 + 0.16
71.47 +
71.33
0.18
70.77
0.23
71.58 + 0.00
63.56
0.16
SNV2
72.65 + 0.18
72.44 + 0.43
72.70 + 0.35
6.24
+ 0.61
72.04 + 0.19
70.75
+ 0.13
72.83 + 0.20
6:32 + 0.26
NT
RN50
VGGIT
71.94
0.09
12.01
0.225
72.03
V1.55
0.38
71.88
0.31
70.16 0.22
71.94
0299
RNIS
78.76 + 0.08
78:41
0,25
78.92 + 0.14
79.26
+ 0.29
78 .9
+ 0.14
77.94
0.22
78.33
+ 0.05
68.06
0.18
VGG16
VGGIL
71.91 = 0.09
72.00 = 0.22
72.26 = 0.12
1.922
+ 0.222
71.98 = 0.18
70.60
= 0.13
71.63
+ 0.10
64.08 = 0.19
SNT
SNV2
72.65
72.94
0.41
73.17
72.78
72.22
71:22
0.18
72.74
RN50
VGGII
71,94 = 0.09
72.02
0.19
72.12
+ 0.39
72.32
+ 0.33
71.70 = 0.39
70.08
0.17
71.05
+ 0.20
62.94
0.29
RNIS
78.78 + 0.08
78.25
0.05
78.48
+ 0.24
78.82 + 0.30
78.14 + 0.28
78.45
0.15
78.38
+ 0.13
67.71
0.20
VGG16
VGGII
71.94
72.09
0,23
72.01
71.63 1:
71.93
0.12
0.27
71.65
63.32
SNV2
72.65 1: 0.18
72.64
+: 0.15
72.87
4 0.21
70.53
1: 0.48
72.24
4: 0.39
71.32
0.38
72.42
4: 0.12
60.46
0.28
RN50
VGGI1
71.94 = 0.09
72.00
1: 0.19
72.13 1 0.13
71.62
1 0.24
71.76 - 0.29
70.54
0.33
71.73 1: 0.11
65.43 = 0.24
RNIS
78.76
0.08
78.96
0.20
79.02 + 0.0G
78.35
78.31
78.36
0.25
78.81
72.87 =
VGC16
VGGUI
71.94 + 0.09
71.90
0.18
72.00 + 0.06
71.57
+ 0.26
70.89
0.12
70.06
0.17
71.76
+ 0.10
63.49 + 0.21
LS
SNV2
72.65 1: 0.18
72.87
0.28
73.52 1 0.25
70.01
1: 0.32
71.49
1: 0.38
71.70
0.42
73.01
1 0.27
65,20
1: 0.14
RN50
VGGIT
71.34
0.00
71.82
0.28
71.90
4 0.16
71.35 1 0.33
70.77
0.30
70.85
0.24
71.58 4 0.16
02:21
0.10
HN18
78.76 : 0.08
77.72
+: 0.301
77.82
1 0.12
79.37
1: 0.19
78.33
:: 0.06
78.31
:: 0.21
77.91
4 0.07
63,38
= 0.17
VGG16
VGGII
71.91 = 0.09
71.87
0.21
71.61
= 0.07
71.56
= 0.03
70.31 = 0.09
71.71
0.14
71.42
: 0.05
06.89
= 0.11
CMTM
SNV2
72.65
0.18
72.63
0.21
71.44
0.16
72.46
71.45
0.31
71.59
0.24
71.94
0.20
64.45
0.24
VGGII
71.94
= 0.09
71.54
= 0.30
71.31
= 0.1G
71.77
+ 0.06
71.86
= 0.28
09.32
: 0.09
71.70
+ 0.22
60.58
= 0.17
RN50
RNIS
78.78
0.08
78.21
0.13
78.16
0.00
78.13
0.06
77.56
77.23
0.09
78.64
0.06
65.88
TinylmageNet
RSP
RN34
RN18
63.58
0.06
63.54
0.00
64.32
0.07
64.01
1:
63.27
63.54
0.07
62.15
0.14
55.43
RN50
SNV2
60.61
0.15
60.18
0.28
60.76
4 0.16
56.26
1: 0.36
36.43
43
2 0.11
60.96
0.22
60:15
1 0.20
$4.01
0.22
ST
RN31
RNI8
63.56
= 0.06
63.96
+ 0.13
61.12
+ 0.07
63.25
+ 0.10
63.51
= 0.14
63:49
+ 0.19
63.84
+ 0.08
57.42
= 0.08
RN50
SNV2
0.15
61.23
0.24
61.36
0.14
60.43
+ 0.14
00.02
60,22
0.17
61.13
0.13
55.84
RN34
HN18
63.56
0.06
63:27
0.14
64.49
1 0.17
64.67
1 0.36
63.43
1 0.20
63.50
0.30
64.43
1 0.11
53.11
= 0.06
RN50
SNV2
60.61
# 0.15
50.57
1: 0.22
61.55
4 0.12
31.55
1: 0.28
00.03
#: 0.23
60.98
0.27
60:31
4 0.17
50.44
= 0.15
L.S
RN34
RN18
63.50
0.06
63.74
0.08
64.01
0.14
64.23
0.11
0.07
64.20
0.16
63.04 +
57.43
RNG0
SNV2
60.61
0.15
60.32
0.24
60:93
= 0.15
60.74
+ 0.20
60.
11 0.28
60:46
0.14
60:14
+ 0.21
52,96
= 0.23
CMIM
RN31
BN18
63.53 = 0.06
62.89
2 0.08
63.15
4 0.08
62.91
+ 0.03
63.25 08/28
= 0.05
61.57
2 0.06
62
02.96-
4 0.06
56.13
= 0.01
RN50
SNV2
57.57
0.21
39.92
60.58
0.10
60.42
0.04
56.91
Upon reviewing the variance estimates, we see that certain cases, such the (RN50, VGG11) pair on CIFAR-100, might suggest that the CMIM-trained model could be rendered distillable under naive statistical interpretations. For example, when the DIST method is applied to the CMIM model, the accuracy is reported as, which could potentially exceed the LS student accuracy of if variance is simply added to the mean.
However, this approach of directly comparing mean values with added variances does not provide an accurate or fair assessment of undistillability. To address this, we conducted more rigorous analysis where, across five different seeds, we compared the accuracy of the knock-off Student (VGG11 in this case) trained via label smoothing and the DIST method applied to RN50 trained with CMIM.
The results of this comprehensive comparison are summarized in the following table, which demonstrates that the CMIM model achieves undistillability across all different seeds.
J Computational Overhead
In this section we compare the computational overhead of our method with the CE counterpart on CIFAR-100 dataset.
24

---

Published in Transactions on Machine Learning Research (06/2025)
Table 8:
Variance analysis of Table
Seed
Seed 2
Seed
Seed 4
Seed
LS
72.28
71.83
72.06
72.25
71.53
CMIM
72.01
71.74
71.93
72.12
71.27
Table Training times of CMIM and CE for ResNet-10 and VGG-16 on the CIFAR-100 dataset.
CE
CMIM
RN50
hours 43 minutes
hours 13 minutes
VGG16
hours 57 minutes
hours 25 minutes
Note that the training time for CMIM is slightly higher than that of conventional CE method. This is primarily due to the additional inference samples required to estimate the CMI per class. We believe this is a reasonable trade-off given the significant benefits of the method. In addition, note that the number of samples N does not have any effects on the training time CMIM; this is because the power transform applied to the teacher's output probabilities, and when calculating the gradients during the backpropagation, different values of does not change the gradients.
Additionally, we note that among the existing KD protection methods in the literature, only CMIM (our method) and ST are scalable to larger datasets like ImageNet. This scalability due to the significant computational complexity of other benchmark methods, which limits their applicability to smaller datasets.
K Ablation on Hyper-parameterr
In this section, we perform an ablation study analyze the impact of three key hyper- pyper-parameters of CMTM: B, the number of samples N, and w.
For all experiments in this study, we employ the VGG-16 and SNV2 models as the teacher-student pair, using the CIFAR-100 dataset the evaluation benchmark.
K.1 Range of /
In this section, examine the impact of 3 on the performance of CMIM. For this analysis, we fix N 50 and w 25, while varying the value of B. The results are shown in Figure g show that the accuracy of the knockoff student decreases significantly when 3 2 1, highlighting the importance of choosing an appropriate B to effectively balance cross-cntropy and CMI objectives.
72.675
72,650
Student Accuracy
72.625
72,600
72.575
Figure The student accuracy when distilled from the teacher model trained by different values.
25

---

Published in Transactions on Machine Learning Research (06/2025)
K.2 Number of power samples N
In this section, analyze the influence of the number of samples N on the performance of the knockoff student. For this study, we set / = 2 and w = 25, varying N to observe its impact. The results, presented in Figure 5 reveal that the accuracy of the knockoff student declines monotonically as N increases, suggesting that moderate sampling size enable to capture sufficintt diversity for robust estimation.
72.700
Student Accuracy
72.625
72.600
72,575
72.550
20
40
60
80
100
Number of samples
Figure 5: The student accuracy when distilled from the teacher model trained by different number of sample
N.
K.3 Power Coefficient
In this section, we investigate the effect of the power coefficient w on the knockoff student's performance. For this analysis, we fix 3 2 and N = 25, while varying w. The results are summarized in Table [0] Notably, when w > 30, the simulation frequently results in NaN values due to excessively large exponent values. Furthermore, the table shows that w minimum, indicating that this value elfectively approximates the behavior of w oc.
w 30, the knockoff student's accuracy reaches its
Table 10:
The student accuracy when distilled from the teacher model trained by different values of w.
Value of w
5
10
15
20
25
30
40
50
100
200
Knock-olf off Student Accuracy
72.65
72.67
72.55
72.56
72.55
72.52
72.53
72.52
NaN
NaN
NaN
NaN
26

---

Published in Transactions on Machine Learning Research (06/2025)
L Visualization of Different Three Projected Probability Clusters
DAFC FNESSOO
(a) LS
(b) Nasty teacher
(c) CMIM
Figure 6: Visualization of three projected probability clusters for ResNet-50 trained on CIFAR-100 100 using (a)
LS, (b) NT, and (c) CMIM.
this section, we randomly selected three additional output clusters from the CIFAR-100 100 dataset, oak, raccoon, and ray, and visualized them using the method introducod in Yang ct (2025). The same conclusion continues to hold.
M Effect of CMIM on Model Calibration
In this section, we investigate the impact of the CMIM method on model calibration. To do so, we report the Expected Calibration Error (ECE) for ResNet-50 trained with CE, LS, NT and CMIM the CIFAR-110 dataset. As shown in Table 11 the CMIM-trained model achieves calibration par with its CE-trained counterpart and consistently perperforms both the LS and NT counterparts.
Table 11: Expected calibration error of three projected probability clusters for ResNet-50 trained CIFAR- 100 using CE, LS, NT and CMIM.
Defense
CE
T.
NT
CMTM
ECE
9.42%
21.48%
16.47%
11.24%
Accuracy
77.81%
78.45%
77.31%
78.72%
27


Reviews:
[{"review_id": "QUv0mzOMEd", "reviewer": "Reviewer_uaKc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n- CMIM is an **undistillable** deep neural network based on information information-theoretic approach.\n\n- The authors address a less-explored topic in knowledge distillation research: how to actively train models that are undistillable, which has significant implications for model intellectual property protection.\n\n- They provide a solid theoretical proof for approximating the maximum CMI across transformed output distributions using a log-sum-exp surrogate, which enables tractable optimization.\n\n- The proposed CMIM is tested on 3 image datasets and various model architectures and student-teacher pairs. They also benchmark it against seven defense baselines and multiple KD attacks.\n\nWeaknesses\n\n-  The proposed training objective involves non-trivial computation, particularly due to the dual optimization over cluster centroids and model parameters, potentially limiting scalability in large-scale settings.\n\n- Limited experimental results: It is unclear how well the proposed method can be generalized to other domains and tasks. \n\n-  While the method minimizes CMI to collapse output clusters, it may not fully eliminate all informative structure in the outputs (e.g., inter-class rankings), leaving open the possibility of distillation under alternative assumptions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The current manuscript introduces Conditional Mutual Information Minimization (CMIM) for an undistillable deep neural network. The authors propose a practical and effective defensive training strategy using information theory to protect deep learning frameworks from such attacks (KD, MKD, DKD, DIST, HT). \nSpecifically, they suggest that if for every class, the outputs of the neiral networks are highly concentrated (i.e., nearly identical for all inputs of the same label), then no meaningful information remains for a student to \"distill.\" This engagement is estimated using conditional mutual information (CMI). \nThe model is trained by a combination of 1)cross-entropy loss for classification and 2) maximum CMI across all transformations using numerical integration and surrogate KL loss with auxiliary distributions. \nThe authors provide theoretical proof for making the training tractable by approximating the maximum CMI value over power-transformed outputs by expressing it as a log-sum-exp integral (Theorem 1) .\nThe method has been tested on CIFAR 100, Tiny Imagenet and Imagenet against various attacks and taking into account different defence baselines such as MAD, ST, NT.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "i have several requests to change my decision from borderline to [weak] accept, \n\nMajor comments:\n\n- 1) Since one of the important contributions of this paper is the empirical results, can you please upload the code because I am getting a \"The requested file is not found\" error from the current GitHub repository?\n\n- 2)  Can you please consider an additional experiment on multi-label settings? Or can you please perform new experiments on a new task (regression )or new domain (time-series or NLP) rather than just image classification ?\n\n- 3) is it possible to evaluate the model via **adaptive attacker** (e.g. a few-shot tuning )?\n\n- 4) The experimental results indicate that the defence is empirically valid, however, deeper analysis of what knowledge remains in the compact output is needed to provide a robust justification for **undistillability**. Therefore, I would suggest : \n- 4-a) Can you theoretically justify what types of residual information (e.g., relative logit structure, rank correlation) might persist under low CMI but still be useful for distillation?\n- 4-b) Can you design targeted attacks or diagnostic metrics that measure whether such residual knowledge can be leveraged despite tight output clusters?\n- 4-c) empirically test adaptive KD attacks (e.g., rank-preserving distillation or adversarial students) to validate whether CMIM-trained models resist distillation beyond the tested methods.\n \nMinor comments: \n\n- Can you please provide more details for the caption of Figure 3 on what the curves imply for distillability?\n\n- Can you please add more details on why label smoothing (LS) is treated as the gold standard for undistillability comparisons? This makes the evaluation criteria more rigorous.\n\n- Can you please discuss/summarize the key findings of ablation analysis K1, K2,K3 on hyperparameters in the main paper?\n\n- Please consider the same format for all tables \n\n- While the connection between temperature scaling and power transforms is cited, a short visual or intuitive example (e.g., how α affects output softening or sharpening) could help readers unfamiliar with this concept.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fx8OG1mG7H", "reviewer": "Reviewer_AA75", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strenghths \n\nThe work is highly relevant to the TMLR community. It tackles an important and timely problem—protecting models from being distilled—by proposing a simple yet effective method to increase model undistillableness. The paper provides a practical solution supported by empirical evidence, which could inspire future research on model protection and robustness against distillation.\n\n# Weaknesses\n\n* While the authors propose some motivation for why the increased concentration of predicted probability vectors within each classification class would make the distillation process less effective, it does not entirely explain the observed phenomena. As an example, one can imagine that such an increase in prediction concentration could eventually hurt other aspects of the model beyond the predicted label (such as the model's calibration), which in turn could decrease overall performance. Are there any ways to formalize the described behavior, or at least explain it in more detail?\n\n* While the proposed method is defined for classification tasks, it’s not entirely clear how it could be extended to tasks such as regression.\n\n* It is not entirely clear why the authors introduce separate temperature scaling parameters for each of the classes, and why they were introduced in the first place. From the perspective of CMI optimization, these parameters are not necessary—so why are they used?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a novel method to make neural networks undistillable by introducing additional loss terms that optimize the concentration of predicted probability vectors within each classification class. The goal of making a model undistillable can be roughly formulated as preventing a student network from achieving better accuracy through teacher-student training compared to standard label smoothing. The proposed method incorporates extra loss terms to enhance the model’s resistance to distillation and demonstrates its effectiveness across several popular baselines, showing improvements over existing approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. [strengthen] Could the authors provide a more detailed explanation—or ideally some formalization—of why increased concentration of predicted probability vectors reduces distillation effectiveness? Additionally, how does this concentration impact other properties of the model, such as calibration or generalization?\n2. [strengthen] The motivation for introducing separate temperature scaling parameters for each class is unclear. From the perspective of CMI optimization, these parameters seem unnecessary—what role do they play in the method?\n3. [strengthen] Could the authors discuss whether and how the proposed method could be extended to regression tasks, given that it is currently defined only for classification?\n4. [strengthen] A comparison of the training time or computational overhead with existing approaches would help clarify the method’s practicality and efficiency in real-world scenarios.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "B97QbPYMgs", "reviewer": "Reviewer_9MCz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The storyline is very clear and complete. I like how the authors motivate the solution with a key observation, explain the method and support it with extensive empirical validations.\n2. The figures preemptively answer my questions that arise while I am reading the paper. For example, Figure 2 shows CMIM is isolating prediction probabilities of different classes more than cross entropy (baseline) and nasty teacher (a renowned anti-distillation method). Figure 3 demonstrates that CMI is indeed minimized by CMIM.\n3. The authors show evidence through extensive experiments that the proposed CMIM is making the model resistant to knowledge distillation in Table 1.\n4. The proposed CMIM method does not harm the teacher’s performance, which is not common to anti-distillation methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel training methodology called Conditional Mutual Information Minimization (CMIM), to protect deep neural networks from knowledge distillation, turning them into undistillable models. The main objective for building undistillable models is for protecting intellectual properties. The identifying behavior of such undistillable models is that when using them as black-box teachers, the student model cannot outperform a model trained from scratch using label smoothing. These two cases are respectively termed “knockoff student” and “LS student”.\n\nThe authors start with a key observation that some undistillable models have highly concentrated output probability distributions for instances sharing the same label, ideally collapsing into nearly identical distributions. The authors then propose to quantify this probability concentration using Conditional Mutual Information (CMI), and attempt to train undistillable models by minimizing CMI together with the main cross-entropy objective.\n\nThen the authors empirically verified the effectiveness of CMIM. Extensive experiments demonstrate that CMIM-trained models resist a wide variety of knowledge distillation methods across CIFAR-100, TinyImageNet, and ImageNet datasets. Notably, these models trained with CMIM are not only resistant to distillation but also shows improved classification performance compared to vanilla versions trained on cross-entropy only.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Weaknesses:\n1. In addition to Figure 2, it might be beneficial to add another figure that shows the distribution of predicted logits for the three training methods. Only showing three classes might risk cherry picking.\n2. [Not critical] In the Introduction section, the authors discussed how per-class probability concentration may help models become undistillable and motivates explicitly training for such a behavior in the prediction logits. This reminds me of contrastive learning, or more specifically, supervised contrastive learning, where the cosine similarities between latent embeddings for instances within the same class are minimized. Does that mean models trained using supervised contrastive learning are more undistillable than trained with standard supervised learning? It would be interesting to see that.", "broader_impact_concerns": "1. For the title of Algorithm 1, I would recommend putting the full name followed by abbreviation, instead of abbreviation only.\n2. Following Weakness #1, the authors might consider adding such a logit distribution visualization if they find it appropriate.\n3. Following Weakness #2, perhaps briefly mention the potential connection between per-class probability concentration and contrastive learning in the future directions or another section as appropriate.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)