Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information
Authors: Linfeng Ye
Shayan Mohajer Hamidi
Renhao Tan
EN-HUI YANG
Venue: ICLR 2024 poster (2024)
Keywords: Knowledge distillation
information theroy
bayes conditional distribution estimation
conditional mutual information
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: It is believed that in knowledge distillation (KD), the role of the teacher is to provide an estimate for the unknown Bayes conditional probability distribution (BCPD) to be used in the student training process. Conventionally, this estimate is obtained by training the teacher using maximum log-likelihood (MLL) method. To improve this estimate for KD, in this paper we introduce the concept of conditional mutual information (CMI) into the estimation of BCPD and propose a novel estimator called the maximum CMI (MCMI) method. Specifically, in MCMI estimation, both the log-likelihood and CMI of the teacher are simultaneously maximized when the teacher is trained. In fact, maximizing the teacher's CMI value ensures that the teacher can effectively capture the contextual information within the images, and for visualizing this information, we deploy Eigen-CAM. Via conducting a thorough set of experiments, we show that by employing a teacher trained via MCMI estimation rather than one trained via MLL estimation in various state-of-the-art KD frameworks, the student's classification accuracy consistently increases, with the gain of up to 3.32\%. This suggests that the teacher's BCPD estimate provided by MCMI method is more accurate than that provided by MLL method. In addition, we show that such improvements in the student's accuracy are more drastic in zero-shot and few-shot settings. Notably, the student's accuracy increases with the gain of up to 5.72\% when 5\% of the training samples are available to student (few-shot), and increases from 0\% to as high as 84\% for an omitted class (zero-shot).

Published as a conference paper at ICLR 2024
BAYES CONDITIONAL DISTRIBUTION ESTIMATION FOR KNOWLEDGE DISTILLATION BASED ON CONDITIONAL MUTUAL INFORMATION
Linfeng Ye; Shayan Mohajer Hamidi", Renhao Tan' & En-Hui Yang Department of Electrical and Computer Engineering, University of Waterloo (144ye, smohajer, cameron. ran, ehyang) @uwaterloo.co ca
ABSTRACT
is believed that in knowledge distillation (KD), the role of the teacher to provide an estimate for the unknown Bayes conditional probability distribution (BCPD) to used the student training process. Conventionally, this estimate obtained by training the teacher using maximum log- log-likelihood (MLL) method. To improve this estimate for KD, in this paper we introduce the concept of conditional mutual information (CMI) into the estimation of BCPD and propose novel estimator called the maximum CMI (MCMI) method. Specifically, in MCMI estimation, both the log-likelihood and CMI of the teacher are simultaneously maximized when the teacher is trained. Through Eigen- CAM, it is further shown that maximizing the teacher's CMI value allows the teacher to capture more contextual information in an image cluster. Via conducting a thorough set of experiments, we show that by employing a teacher trained via MCMI estimation rather than one trained via MLL estimation in various state-of of-the-art art KD frameworks, the student's 's classification accuracy consistently increases, with the gain of up to 3.32%. This suggests that the teacher's s BCPD estimate provided by MCMI method is more accurate than that provided by MILL method. In addition, we show that such improvements the student's accuracy are more drastic in zero-shot and few-shot settings. Notably, the student's accuracy increases with the gain of up to 5.72% when 5% the training samples are available to the student (few-shot), and increases from 0% to as high as 84% for an omitted class (zero-shot). The code is available at https: https:///gituus com http//cc120000 wtw.cl1222m4mmmmmmm 4wemmmmmmm//i.. /TM.emm//CCMMMMIIII
INTRODUCTION
Knowledge distillation (Buciluà et al. 2006 Hinton et al., 2015) (KD) has received tremendous attention from both academia and industry in recent years as highly effective model compression technique, and has been deployed in different settings (Radosavovic et al. 2018 Furlanello et al. 2018: Xie et al., 2020). The crux of KD is to distill the knowledge of a cumbersome model (teacher) into lightweight model (student). Followed by |Hinton al. (2015), many researchers tried to improve the performance of KD (Romero et al. 2014 Anil et al. 2018 Park et al. 2019h), and to understand why distillation works (Phuong Lampert 2019 Mobahi et al.] 2020) Allen-Zhu & Li.
2020 Menon et al., 2021 Dao et al. 2020).
One critical component of KD that has received relatively little attention the training of the teacher model. In fact, in most of the existing KD methods, the teacher is trained to maximize its own performance, even though this does not necessarily lead to an improvement the student's s performance (Cho & Hariharan 2019 Mirzadeh al.) (2200). Therefore, effectively train student to achieve high performance, it i8 essential to train the teacher accordingly.
Recently, Menon et al. the true Bayes conditional probability distribution (BCPD) of label y given an input æ. and that the closer the teacher's 's prediction BCPD, the lower the variance of the student objective function. On the other hand, (Dao et al. 2020) showed that the accuracy of the student is directly bounded
12021) observed that the teacher's soft prediction can serve proxy for
Authors contrihuted equally.

---

Published as a conference paper at ICLR 2024
by the distance between teacher's prediction and the true BCPD through the Rademacher analysis. In addition, Yang et al. (2023) showed that the error rate of a deep neural network (DNN) is upper bounded by the average of the cross entropy of the BCPD and the DNN's output. This together with the observations in Menon et al. (202T): Dao et al. (020)) implies that the better the estimation of the BCPD by the teacher, the better the student's performance would be.
In conventional KD, the BCPD estimate is obtained by training the teacher using maximum log- likelihood (MLL) method. In this paper, we instead explore a different way to train the teacher for the purpose of providing a better BCPD estimate so as to improve the performance of KD. To this end, we introduce novel BCPD estimator, the rationale behind which will be briefly explained in the sequel.
In a multi-class classification task, each of the "ground truth labels" can be regarded as a distinct prototype. In this sense, the images with the same label are different manifestations of a common protorype, each with its own context. For instance, consider the following two images whose prototypes are "dog": (i) dog in a jungle, and (ii) dog in a desert. The first and second images have some contexts related to jungle and desert, respectively, although both have the same prototype. As such, each manifestation (for instance, training/testing image) contains two types of information: (i) its prototype; and (ii) some contextual information on top its protorype. Therefore, in KD, it would be desirable for the teacher, providing an estimate of BCPD, to be capable of providing good amount of information about both of these types of information for the input images.
A teacher trained using MLL solely aims estimating the prototypes In order for the teacher to provide a good amount of the contextual information, first we have to find out how this information could be quantified. To this aim, we introduce an information quantity from information theory (Cover] 1999) to measure such information. Specifically, by treating the teacher model as a mapping function, we argue that this contextual information resides in conditional mutual information (CMI) T(X;Y) Y). where X, Y and Y are three random variables representing the input image, the ground truth label, and the label predicted by the teacher, respectively. As such, I(X Y Y) quantifies how much information y can tell about X, given the prototype Y: this aligns with the concept of contextual information. To provide good amount of contextual information, a logical step would be to maximize the CMI value under certain constraints during the teacher's ' training process.
Henceforth, to balance the prototype and the contextual information, we train the teacher simulta- neously maximize (i) the log- likeelili (LL) of the prototype (which is equivalent to minimize its cross entropy loss), and (ii) its CMI value. We refer such an estimator as maximum CMI (MCMI) method. As seen in section 6 although the classification accuracy of the teachers trained via MCMI lower than that of the teachers trained via MILL, the former can consistently increase the accuracy of the student models. In addition, we show that MCMI is general in that it could be deployed in different knowledge transfer methods. In summary, the contributions of the paper are as follows:
We argue that the so-called dark knowledge passed by the teacher to the student is indeed the contextual information of the images which could be quantified via teacher's CMI value.
Equipped with the concept of CMI, we attribute the reason of using temperature in KD to blindly increase the teacher's CMI value.
We develop novel BCPD estimator dubbed MCMI that simultaneously maximizes the LL and CMI of the teacher. This enables the teacher to provide more accurate estimate of the true BCPD to the student, thereby enhancing the student's 's performance.
We show that since larger models generally have lower CMI values, they are not often good teachers for KD. However, this issue is mitigated when teachers are trained using MCMI. We further explain why the models trained via early stopping are better teachers than those that are fully-trained.
To demonstrate the effectiveness of MCMI, conduct thorough set of experiments over CIFAR- 100 (Krizhevsky et al.) 2012) and ImageNet (Deng et al. 2009) datasets, and show that by replacing the teacher trained by MLL with that trained by MCMI in the existing state-of-the-art KD methods, the student's accuracy consistently increases. Moreover, these improvements are even more significant in both few-shot and zero-shot settings.
In this paper, we use the terms prototype and ground truth label interchangeahly.
2

---

Published as a conference paper at ICLR 2024
2
RELATED WORKS
Distillation losses in KD: The concept of knowledge transfer, as a means of compression, was first introduced by Buciluu et al. (2006). Then, |Hinton et al. (2013) popularized this concept by softening the teacher's 's and student's 's logits using temperature technique where the student mimics the soft probabilities of the teacher, and referred to it as KDD. To improve the effectiveness of distillation, various forms of knowledge transfer methods have been introduced which could be mainly categorized
into three types: (i) logit-based (Zhu et al. 2018: Stanton al. 202T Chen al. 2020a Li et al.
2020 Beyer et al. 2022 Zhao et al., (022), (i) ippresentationnbaseee based (Romero et al. 2014] Zagoruyko
& Komodakis| [016 Yim et al. 2017 Chen et al. |202lb] Yang et al., 2021 p. and (iii) relationship-
based (Park et al.
2019b; Liu et al.) 2019: Peng et al. 2019; Yang et al. (022) methods. Closely
related to the scope of this paper, some methods in the literature deploy the concept of information theory in KD to make the distillation process more effective. We defer discussing these works to
appendix A.I and note that our method is different from these works in that we use information- theoretic concepts for training the teacher only, rather than for building strong relationship between the teacher and student.
Training customized teacher: In the literature, only few works trained teachers specifically tailored for KD. Yang et al. (2019) attempted to train a tolerant teacher which provides more secondary information the student. They realized this via adding an extra term facilitating few secondary
classes to emerge to complement the primary class. Cho & Hariharan (2019); Wang et al.] (2022) regularized the teacher utilizing early-stopping during the training. Tan & Liu (2022) trained the teachers to have more dispersed soft probabilities. Additionally, Dong et al. (2023) stated that a model exhibiting local Lipschitz continuity around training inputs can serves as student-oriented teacher. Yet, our work distinguishes itself from these prior studies as it aims to estimate the true BCPD.
NOTATION AND PRELIMINARIES
3.1 NOTATION
For a positive integer C, let [C] A (1...., C}. Denote by P[:] the i-th element of vector P. For two vectors U and V, denote by U their inner product. We |C] to denote the cardinality of a set C. Also, C dimensional probability simplex is denoted by A The cross entropy of two probability distributions PI, P2 E A is defined as H(P, P2)
Pi [c] log Pa[c). and their
Bullback-Leible (KI. divergence is defined as (I. = > Pi [c] log 75lel
Relel
We use P)( to denote the probability of event E. For a random variable X, denote by Px its probability distribution, and by Ex the expected value w.r.t. X. For two random variables X and Y, denote by P(x,Y) their joint distribution. The mutual information between two random variables X and Y is given by I(X,Y) = ((X) - H(X)Y), and the conditional mutual information of X and Y given a third random variable Z is I I(X, Y Z) (X ()
3.2 THE NECESSITY OF ESTIMATING BCPD
In classification task with C' classes, a DNN could be regarded as mapping f i + P,. where E Rd is an input image, and Pz E A Denote by y the correct label of 2; then the classifier predicts y, as ÿ = arg max,ejo] Pa [c]. As such, the error rate of ʃ is defined as < Pr(ū + y), and its accuracy is equal to e. Assuming that X and are the random variables representing the input sample and the ground truth label, then one may learn such classifier by minimizing the risk
R(S.) A B(XY) (Y, Px)] = Ex [Eyix [e (Y, Px)]] = Ex [(P))" e(Px))
(1)
where E(-) is the loss function and e(.) A [((1.-).... ((C,-) is the vector of loss function, PX A [Pr(y] X is Bayes class probability distribution over the labels.
When using the cross entropy loss, Ely, Px) = e(y) F. log (Px) = - log (Px (B)). where e(v) {0, 13 denotes the one-hol vector of y. Therefore, R(S, e H) using cross entropy loss becomes
R(J H) -xx [(Px)T log (Px)]
(2)
3

---

Published as a conference paper at ICLR 2024
As discussed in Yang et al. 2202). the error rate of classifier f is upper-bounded
C R(J,H).
(3)
As such, by minimizing R(f. H) one can decrease (increase) the classifier's error rate (aceuracy). However, in a typical deep learning algorithm, both the probability density function of z, namely I'x and also Px are unknown. Hence, one may learn such a classifier by minimizing the empirical risk on a training sample _ {(I b== defined as:
Remp (f, (JH
Σ e(vn) log (Pa.). DE[N]
By comparing eq. and eq. 4 it is seen that in eq. (4) (i) Pxx is approximated by N. and (ii) Px is approximated by e(y) which is an unbiased estimation of Px. The former approximation is reasonable, however, the latter results in significant loss in granularity. Specifically, images typically contain a wealth information, and assigning a one-hot vector to Px results in a substantial loss of information. As will be discussed in the next subsection, KD alleviates this issue some extent.
3.3 ESTIMATING BCPD BY THE TEACHER KD
In KD, the role of the teacher is to provide the student with a better estimate of Px compared to one-hot vectors. Denote by Pt and Pa the teacher's and student's outputs sample z, respectively. First, the teacher is trained to minimize the empirical loss eq. (4). which is equivalent to train it to maximize the empirical L.L. of the ground-truth probability defined as
LI emp (
log (Ps, (wl))
N
nE[N]
Then, the teacher's ' output P could be regarded as an estimate of Px obtained using MLL method. Afterward, the student uses the teacher's 's estimate of Px, and minimizes
Ria Raa(S, H)
-1 N
Σ (Pt..) .log (Pa,.) WE[N]
(6)
where the one-hot vector e(y,.) in eq. is replaced by the teacher's output probability Pl.
In the next section, we aim to train the teacher such that it can provide the student with better estimate of P compared to the one that it can provide using MLL training method.
ESTIMATING BCPD VIA MCMI
In this section, we improve the teacher's estimate of the true BCPD by training to learn more contextual information of the imput images.
4.1 CAPTURING CONTEXTUAL INFORMATION VIA CMI
As discussed in section M each manifestation (a training/testing image) comprises two types of information: (i) its protolype (ground truth label), and (ii) some contextual information top its prototype. In the context of KD, if the teacher aims to estimate the true Px, these two types of information should be captured in its predictions. However, when teacher is trained using MLL, solely aims at the prototype information. Now the question is that how we can help the teacher to also learn the contextual information better.
Let y be the label predicted by the DNN with probability Px Px[ in response to the input X; meaning that for any input sample E IR and any ÿ [C], Pr(Ý = x x) Pe[i]. The input X and the probability of wrong classes have intricate non-linear relationships. For a classifier f. these complex relationships can be captured via CMI(f) [(X; Y Y). This value is the quantity of information shared between and Y when Y is known.
The CMI could be also written [(X; ÝJY) H (X] Y), i.e. the difference between the average remaining uncertainty of X when Y is known and that of X when both Y and Ý are
4

---

Published as a conference paper at ICLR 2024
known. As such, maximizing IXX; is equivalent minimizing H Y) (since H (XlY)
constant as it is intrinsic characteristic of the dataset), which, in turn, forces Y to have much information as possible about X, given its prototype.
4.2
CMI VALUE FOR A MODEL
As shown in Yang et al. 2023). for a classifier f we have
C
P( = 1/2))
1(X:1l y) = Σ (xlx
Σ
P( = i(x) x log
(7)
Prip (Y = 81Y = y)
Px
=xx
Px In
Y 18=11
(8)
Pri (Y = i|Y 3)
= ExIx XL(PP|IP&1))1X
(9)
On the other hand,
CMI(F) = I(X: Σ (x1 DEIC)
(10)
Therefore, from eqs. 9 and 10
CMI(F) could be calculated as
CMI()) = (X.))
KL (Px || @*)], with Q A EXIY) [Px Y].
(11)
For a given dataset D with unknown IP (x,Y) and IP ((]])) we can approximate CMI(f) by its empirical value. To this end, first we define D, {I, D Mj = y) the set of all training samples whose ground truth labels are y. Then, the empirical CMI(f) could calculated as
CMlemp (f)
1
Σ
Σ KL(P2, |l Qemp),
(12)
vE[C] ZyED,
where
Qy tim
Pzy, for y E [C].
(13)
iD 2yED,
Deploying eq. 12 in the next subsection we explain that the role of temperature T in KD is to naively increase the teacher's CMI value.
4.3 THE ROLE OF TEMPERATURE IN KD
In KD, using higher temperature T encourages the student to focus also on the wrong labels' logits, inside which the dark knowledge of the teacher resides (Hinton et al., 2015). Although some prior works tried to optimize as a hyper-parameter of the training algorithm (L et al. 2023 Liu et al.)
2022 Jafari al. 2021 Stanton al. 2021), the real physical meaning of T has yet be elucidated.
We argue that 7' is simple means to gauge the teacher's CMI value to some extent. To demonstrate this, fig. depicts the teacher's CMI value (right vertical axis) along with the student's classification accuracy (left vertical axis) Vs. the teacher's temperature (horizontal axis) for three different teacher- student pairs trained on CIFAR- dataset, where we use T = {1. 2, ....... 8} (here, following Hinton et al. (2015), both teacher and student models use the same T value). Let us, for instance, consider fig. Ta As seen, by increasing the value, the teacher's CMI initially increases reaching maximum value at T = 2, and then it continually drops. Interestingly, for T* 2, the student's accuracy reaches its highest value. The similar conclusions can be made from figs. lb and Ic suggesting that T used in KD as means of increasing the teacher's CMI value.
5
METHODOLOGY
Based on the discussions provided thus far, in order for the teacher to predict the true BCPD well, shall maximize the (i) log-likellng likelihoddo of the ground truth label, and (ii) its CMI at the same time.
To understand how CMI(f) affects the output probability space of f. please refer appendix A.4
5

---

Published as a conference paper at ICLR 2024
19--
429
704
450
704
Student Are
64.5
1.00
Saudert
065
Tomporature
Temperabure
(a) SesNett-6--ReNeel
(b)
(c) MoNNet-50-MobileNeer
Figure 1: The teacher's 's CMI value (red curve, right axis) along with the student's 's accuracy (blue bars,
left axis) Vs. the teacher's temperature in conventional KD for three different teacher-student pairs.
Hence, we train the teacher to maximize the following objective function
(fCMI()) = LLemm(f) + À CMIemp(f):
(14)
where >> 0 is a hyper-parameter. We refer to such a training framework as MCMI estimation. Yet, maximizing CMIsmp (f) faces some challenges as the term Qumpp in CMIsmp (f) is a function of Pz, (see eq. (12). Hence, the optimization problem in eq. (14) is not amenable to numeri- cal solutions. To tackle this issue, instead of training the teacher from scratch, fine-tune a pre-trained teacher in the manner explained in the following.
First pick a pre-trained teacher model that is trained via cross entropy loss. Then, we find {Qemp Ivecc for this model using eq. (13). Then, we fine-tune the pre-trained model by maximizing the objective function in eq. (14) while keeping the {Qump i ee(c values constant over the course of fine-tuning.
200
Esech
(a) WRN-40-2
Remark. One may wonder why training a teacher using tradi- tional MLL is not suitable for KD, and why teachers trained via early stopping are better teachers (Cho & Hariharan) 2019). To answer to these questions, we plot the curves for CMI and LL values for two models trained on CIFAR-100 using MLL loss in fig. [2] As seen, the teacher's 's CMI starts to sharply decrease after epoch ~ 150 Therefore, using an early-stopped teacher is beneficial since it has a higher CMI value compared to the fully converged one. Yet, the problem with an carly-stopped teacher is that it has a low LL value. This issue does not exist for model trained by MCMI as both CMI and LL are maximized Proposition 1. If fa is an intermediate feature map of a DNN corresponding to the input x, then, I(X; Y Y) S 1x;;xxl
64
Esech
(b) ResNet-110
Figure 2: The evolution of CMI and L.I values for teacher trained by MI.. during the training.
Based on proposition maximizing [(X; Ý Y). would also have the effect of increasing T(X; fx Y). Therefore, no matter what kind of KD method is used, logit-based or representation- based, maximizing the loss (MCM(()) would enable the teacher to provide more contextual informa- tion to the student.
EXPERIMENTS
Terminologies: For the sake of brevity, hereafter, we refer to the teachers trained by MLL and MCMI methods as the "MLL teacher" and "MCMI teacher", respectively.
In this section, we conclude the paper with several experiments to demonstrate the superior effec- tiveness of MCMI estimator compared to its MLL counterpart. Specifically, in section 6.1 we show that replacing the MLL teacher with the MCMI one in the sate-off-he-at KD methods leads to
FFor a comparison between the MCMI and carly-stopped teacher, please refer to appendix A.3
"ee appendix A.5 for the proof.

---

Published as a conference paper at ICLR 2024
consistent increase in the student's accuracy. In sections 6.2 and 6.3
demonstrate that the MCMI teacher can drastically increase the student's 's accuracy in the zero-shot and few-shot settings. In addition, in appendix A.2 we justify why larger teachers often harm the student's accuracy, and show that this problem is resolved by using MCMI teachers.
Plug-and-play nature of MCMI teacher: In all the experiments conducted in this section, when testing the performance of the MCMI teacher, we do not tune any hyper-parametert in the underlying knowledge transfer methods, all of which the same in the corresponding benchmark methods.
6.1 MCMI TEACHER IN KD METHODS
In order to demonstrate that Pf. s.MCMI serves as a better estimate of Px compared to Pa Z.MLL: in appendix A.6 we perform experiments on synthetic dataset with known Px Additionally, litmus test to see whether P McMI outperforms Pt, MI, as an estimate of Px the student's accuracy; where higher student's s accuracy indicates a better estimate provided by the teacher. In the following, we conduct extensive experiments over CIFAR- 100 (Krizhevsky et al. 2009) and ImageNet (Russakovsky et al. 2015) datasets, showing the MCMI teacher's s effectiveness.
This dataset contains 50K training and 10K test colour images of size 32 32, which are labeled for 100 classes. Following the settings of CRD (Tian et al.) (019),, employ 6 teacher-student pairs with identical network architectures and another 6 pairs with differing network architectures for our experiments (see table I. We perform each experiment across 5 independent runs, and report the average accuracy (for the accuracy variances, refer to appendix 12.
CIFAR-100.
compare using MCMI teacher Vs. MLL teacher in the existing state-of-the of--th--rtttttttt distillation methods, including KD (Hinton et al. 2015). , AT (Zagoruyko & Komodakis 2016), PKT (Passalis & Tefas 2018), SP (Tung & Mori 2 >. CC (Peng et al. 2019). RKD (Park et al. (019a),, VID (Ahn et al.] 2019), CRD (Tian et al. (2019). DKD (Zhao et al. 2022) REVIEWKD Chen et al. (2021a), and HSAKD (Yang et al., 2021). All the training setups, including
For comprehensive comparisons,
fine-tuning MCMI teachers, training students, and À values, along with a study on the effect of on MCMI teachers are provided in appendix A.8
As observed in table [ whether the teacher-student architectures are the same or different, the student's 's accuracy is improved across the board by replacing the MLL teacher with the MCMI one. In addition, such improvements are consistent in all the tested KD methods, and could be up to 3.32%; nevertheless, the gain more notable when the teacher and student have a different architecture, For both MIL and MCMI teachers, we reported their accuracies over the testing set, and both L.I and CMI values over the training set in table I As seen, the test accuracy of the MCMI teacher is always lower than that of the MLL teacher, confirming the fact that training the teacher in KD for the benefit of the student is a different task from training the teacher for its own prediction accuracy.
ImageNet. ImageNet is large-scale dataset used in visual recognition tasks, containing around 1.2 million training and 50K validation images. Following the settings of (Tian et al. 2 popular teacher-student pairs for our experiments (see ta- ble 2 We note that across all the knowledge transfer methods reported in table [ replac- ing the MLL teacher by MCMI teacher consistently leads an increase in the student accu- racy. For example, when considering teacher-student pairs ResNet34-ResNetl 8 and ResNet50-- MobileNetV2, the increase in the student's Top-1 accuracy in KD 0.51% % and 0.60%, respectively.
2019. Yang al.) 2020),
Visualization of contextual information ex- tracted by MCMI teachers. In this subsec- tion, we use Eigen-Cu CAM (Bany Muhammad & Yeasin 2021) to visualize and compare the fea- ture maps extracted by MCMI and MLL teach- ers. To this end, we randomly pick four images from the same class in ImageNet dataset, namely "Toy Terrier", and depict the respective MCMI
Original
MIL
MCMI
Original
MLL
MCMI
Eigen-CAM for MIL and MCMI teach- for 4 samples from class "Toy Terrier". and MLL teacher's ' Eigen-CAM CAM for them (see fig. B.. As seen, the activation maps provided by the MIL teacher always highlight the dog's 's head. However, for the MCMI teacher, the activation maps
Figure 3:
7

---

Published as a conference paper at ICLR 2024
Table 1: The test accuracy (%) of student networks on CIFAR- 100 (averaged over 5 runs), with teacher-student pairs of the same/different architectures. The subscript denotes the improvement achieved by replacing MLL teacher with MCMI teacher. We use bold numbers and asterisk (*) to denote the best results and identify the results reproduced on our local machines, respectively.
Teachers and students with the same architectures.
ResNet 56
ResNet I10
ResNet 110
WRN
WRN
40-2
VGG 13
Teacher
MLL
MLLL
MCMI
MCMI
MIL
MCMI
MLL
Accuracy
72.34
72.09
74.31
73.60
74.31
73.60
75.61
75.21
75.61
75.21
74.64
73.96
11.
-0.101
-0.344
-0.033
-0.246
-0003
-0.246
-0.052
-0.132
-0.052
-0.132
-0.007
-0.076
CMI
0.1583
0.4428
0.0605
0.3474
0.0605
0,3474
0.0255
0.1951
0.0255
0.1951
0.0152
0.1298
Student
ResNet-20
ResNes-20
ResNat-32
WRN-16-2
WRN-40-1
VGG-8
Accuracy
69.06
69.00
71.14
73.26
71.98
KD
70.66
70.84
70.67
70.85
73.08
73.48
74.92
75.42
73.54
74.53
72.98
73.83
-0.15
+4.15
-0.40
440.00
44,00
-0.45
AT
70.55
70.68
72.31
73.96
74.08
72.77
73.25
71.43
71.76
PKT
70.34
70.96
70,25
71.03
72.61
72.92
74.54
75.01
73.45
74.15
72.88
73,35
SP 69.67
70.98
70.04
70.83
72.69
73.34
73.83
74,60
72.43
73,60
72.68
73.29
+1.11
+4.79
+0.77
+1.33
-0.61
CC 69.63
69.98
69.48
70.02
71.48
71.71
73.56
74,00
72.21
72,50
70.71
71.02
+0.35
+4.54
-0.25
40.43
+4.20
-0.31
RKD 69.61
70.68
69.25
70.24
71.82
72.65
73.35
73.97
72.22
72,66
71.48
72.03
46.30
-0.85
44.82
44.44
-0.95
VID
70.38
70.64
70.16
70.69
72.61
73.10
74.11
74.44
73.30
73.58
71.23
71.93
CRD
71.16
71.40
71,46
71,93
73.48
74.03
75.48
75.82
74.14
74,86
73.94
74.23
DKD
71.97
72.31
71.51"
71,83
74.11
74.36
76.24
76.66
74.81
75.63
74.68
74.87
44.32
+41.22
-0.15
REVIEW
71.89
72.31
71.65*
72,11
73.89
74.01
76.12
76.29
75.09
75.47
74.84
74.96
60.40
44.49
40..7
10.30
-0.12
HSAKD
72.58
72,70
72.64
73,15 +0.51
74.97*
75.71
77.20
77.36
77.00
77.55
75.42"
75,86
-0.1
-0.24
+0.06
+0.05
-0.11
Teachers and students with different architectures.
Teacher
ResNet 50
ResNet-50 50
ResNet 32x4
ResNet 32x4
WRN 40-2
VGG-13 13
MLL
MCMI
ML.1
MCMI
Accuracy
79.34
78.45
79.34
78.45
79.41
78,70
79.41
78.70
75.61
75.21
74.64
73.96
11.
0.004
.0.083
0.004
.0.83
4.004
0.035
.0.044
40.35
.0.522
.0.132
.0.007
-0.076
CMI
0.0085
0.1065
0.0085
0,1065
0.0059
0.0586
0.0059
0.0586
0.0255
0,1951
0.0152
0.1298
Student
MobileNatV2
VGG-8 8
ShufleNerV1
Shuf
ShuffleNetVi
MobileNerV2
Accuracy
70.50
71.82
KD
67.35
70.23
73.81
74,59
74.07
75,90
74.45
76.32
74.83
76.45
67.37
69.14
-2.66
+6.75
-1.45
+1.57
+1.00
17
AT
58.58
60.03
71.84
72.19
71.73
75.05
72.73
75.21
73.32
75.61
59.40
62.07
+4.35
-3.32
+2.48
PKT
66.52
67.42
75.21
75.39
67.13
68.37
SP
68.08
69.07
73.34
74.14
73.48
76.56
74.56
76.70
74,52
76.82
66.30
67.83
+0.00
44.00
-3.05
+2.24
+2.30
-1.95
CC 65.43
66.76
70.25
70,90
71.14
71.77
71.29
73.02
71.38
71,80
64.86
65.45
+1.35
44.05
-0.65
61.75
40.42
-0.0
RKD 64.43
65.11
71.50
72,10
72.28
73.59
73.21
74.67
72.21
74,26
64.52
65.37
+0.6
+0.00 60
-1.11
61.40
+9.05
-0.45
VID
67.57
67.61
70.30
70.69
73.38
74.58
73.40
74.67
73.61
75.03
65.56
65.77
-1.20
+1.37
CRD
69.11
69.70
74.30
75.11
76.82
75.65
76.05
76.62
69.70
69.98
DKD
70.35
71.70
73.94
75.35
76.45
77.21
77.07
77.66
76.70
77.42
69.71
70.35
+1.41
440.3
44..72
-0.64
REVIEW
69.89
70.63
73.43"
74,20
77.45
77.78
77.78
78.23
77.14
77.56
70.37
71.70
.0.74
+0.17
-0.35
44.445
40.00
-1.5
HSAKD
71.83*
72.98
75.87"
76.45
79.51"
79.77
79.93
80.01
78.51
78.87
71.09*
72,80
+1.15
+0.08
-0.36
+0.00
+4.30
-1.11
Table 2: Top-1 and Top-5 student's is test accuracy (%) on ImageNet validation set for 4 different KD methods using MI.I. and MCMI teachers (RN and MN stand for ResNet and MobileNet, respectively).
Teacher-Studnnt
Teacher Performance
KD
DKD
Review KD
CRD
Topl
Top5
CMI
Top1
Top5
Topl
Top3
Top1
Top3
Topl
Top3
RN34-R18
MLL
73.31
91.42
0.7203
-0.5600
71.03
90.05
71.70
90.41
71.61
90.51
71.17
90.13
MCMI
71.62
90.67
0.8650
-0.6262
71.54
90.63
72.06
91.12
71.98
90.86
71.50
90.20
MLT.
76.13
92.86
0.6002
0.439
70.50
89.80
72.065
91.03
72.56
91.00
71.37
90.41
RN50-MNV2
MCMI
74.94
91.03
0.7150
-0.4943
71.10
90.16
72.61
91.26
73.00
92.19
71.63
90.53
as a whole capture more contextual information. including the dog's legs and some surrounding information (for comprehensive illustration of the Figen-CAM figures, refer to appendix A.10
8

---

Published as a conference paper at ICLR 2024
6.2 HERO-SHOT CLASSIFICATION IN KD
In the concept of zero-shot classification in KD, the teacher is trained on the entire dataset, but the samples for some of the classes are completely omitted for the student during the distillation. Then, the student is tested against the samples for the whole dataset (both seen and unseen classes). Some real-world scenarios of this setup is elaborated in appendix II.1 To show the effectiveness of using MCMI teacher in KD for zero-shot classification, here we report an example of such scenario on CIFAR-10 dataset, and further results including those for CIFAR- 100 are reported in appendix AA. We use MeasNe55-RReseelee methods on the whole dataset. Then, we entirely omit five classes from the student's training set, and train it once using MLL teacher and once using MCMI teacher. The student's 's confusion matrice for both scenarios are depicted in fig. 4 where the dropped classes are highlighted in red. As seen, the student's 's accuracies for the dropped classes are always zero when using the MLL teacher. On the other hand, these accuracies are substantially increased, by as much as 84%, when using the MCMI teacher. We conducted additional tests in which we dropped varying numbers of classes for the student, as elaborated in A1.4
as the teacher-student pair, and train the teacher via MLL and MCMI
6.3 FEW-SHOT CLASSIFICATION
MILL Teacher
MCH Thacher
In few-shot classification, the models are provided with an a percent of the instances in each
0.00 0.3 eice 1:53 D.00 0.00 0:00 0/12 OLEE
CIM 0.9 S
0:00 DU EXC 031 E/00 0.53 9
3.
0.00 0.2 6:6 3.00 9.00 G00 6:62 CLU 0.c0
0:00
6.6 3.00 606 5.88 9.67
52 0.0 (. 546 4.8 9.00 bol 0:00 GLS1 0.50
9.82 5:6 5:00 0.00
et al. (2018). Translat- ing this concept into the context of KD, the teacher is trained on the full dataset, yet an a percent of the samples in each class is used to train the student during the distillation process. Here, we show the effectiveness of MCMI teacher over MIL teacher us- ing the following experiments on
0.1 6:00 0.2 e:00
535 0.00 0.00 8:03 0.05 0.00
0.02 0.00
0:33
0.99 0:43 0.03
-a 0:00 uas ese 0.03
0.00
0.00 6.6 0.33 020
0.00 0.3 eiee 0.51 9.00 0.00 0:09 0.37 0.00
.EOT 0.02 9:89 0.00 0.00 0.0 033 0.00 0.55
Figure 4: The student's confusion matrices when it is trained through the MLL teacher (left), and by MCMI teacher (right). CIFAR- 100. We use KD and CRD with ResNet-56 and ResNet-20 as the teacher and student models. We conducted experiments for different values of a, namely {5, 10, 15, 25, 35, 50, 75). The results are summarized in table 3 where for a fair comparison, we employ an identical partition of the training set for each few-shot setting. As seen, the improvement in the student's 's accuracy is notable by using an MCMI teacher, which is particularly more pronounced for smaller a values (for instance, when 5, the improvement in the student's accuracy is 5.72% and 3.8% when KD and CRD are used, respectively).
Table 3: Top-1 I accuracy (%) of the student under few-shot setting (averaged over 5 runs).
15
75
Teacher
MLL
MCMI
MLL
MCMI
MLL
MCMI
MLL
MCM1
MLL
MCMI
MLL
MCMI
MLL
MCMI
KD
52.30
58.02
60.13
63.75
63.52
66.20
66.78
68.10
68.28
69.34
69.52
70.28
70.44
70,59
CRD
47.60
51.40
54.60
56,80
58.90
60.02
63.82
64.7
66.70
67.22
68.84
69.15
70.35
70.40
CONCLUSION
In conclusion, this paper has introduced a novel approach, the maximum conditional mutual infor- mation (MCMI) method, to enhance knowledge distillation (KD) by improving the estimation of the Bayes conditional probability distribution (BCPD) used in student training. Unlike conventional methods that rely on maximum log-likelihood (MLL) estimation, MCMI simultaneously maximizes both log-likelihood and conditional mutual information (CMJ) during teacher training. This approach effectively captures contextual image information, visualized through Figen-CAM. Extensive experiments across various state-of-the-an KD frameworks have demonstrated that utilizing a teacher trained with MCMI leads to a consistent increase in student's accuracy.
Please refer to appendix A.11.. for detailed explanation of how to interpret the confusion matrix.

---

Published as a conference paper at ICLR 2024
8 ACKNOWLEDGMENTS
We would like to thank the three anonymous reviewers for spending time and efforts and bringing in constructive questions and suggestions, which help us greatly to improve the quality of the paper. We would like to also thank the Program Chairs and Area Chairs for handling this paper and providing the valuable and comprehensive comments.
This work was supported in part by the Natural Sciences and Engineering Research Council of Canada under Grant RGPIN203035-22, and in part by the Canada Research Chairs Program.
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9163-9171, 2019.
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classification. IEEE transactions paltern analysis and machine 2015.
(8(())1145-14333)
Zeyuan Allen Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816, 2020.
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George Dahl, and Geoffrey E Hinton. Large scale distributed neural network training through online distillation. arXiv preprint arXiv:1804.032.. 03235, 2018.
Mohammed Bany Muhammad and Mohammed Yeasin, Eigen-cam: Visual explanations for deep convolutional neural networks. SN Computer Science, 2:1-14, 2021.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher patient and consistent. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10925-10934, 2022.
Maxime Bucher, Stéphane Herbin, and Frédéric Jurie. Generating visual representations for zero- shot classification. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 2666-2673. 2017.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizze Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Defang Chen, fian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 3430-3437,
2020a.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5008-5017, 2021a.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5008-5017, 2021b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33:22243-22255 2020b.
Wei-Yo Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu- Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2018.
Jang Hyun Cho and Bharath Hariharan. On the efficacy knowledge distillation. In Proceedings of the IEFE/CVF international conference on computer vision, pp. 3794-4807 2019.
10

---

Published as a conference paper at ICLR 2024
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Tri Dao, Govinda M Kamath, Vasilis Syrgkanis, and Lester Mackey. Knowledge distillation as
semiparametric inference. In International Conference on Learning Representations, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. pp. 248-2448880 255. leee, 2009.
Chengyu Dong, Liyuan Liu, and Jingho Shang. Soteacher: Toward student-oriented teacher network training for knowledge distillation, 2023. URL http:psss/pppeerreeee..eeess http://ppeneeeeeeeeeeeeeooooooee
aR3hRo 06cn
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pp. 1607-1616.
PMLR, 2018.
Wei Guo, Rong Su, Renhao Tan, Huifeng Guo, Yingxue Zhang, Zhirong Liu, Ruiming Tang, and Xiuqiang He. Dual graph enhanced embedding neural network for ctr prediction. KDD '21, pp. 504, New York, NY. USA, 2021. Association for Computing Machinery. ISBN
9781450383325. doi: 10. 10.1145/3445548.466717 34673388 3447548 3467384
URL. httpps:/ddo http::////oooo/////////// wrg/10..... 1145/
Shayan Mohajer Hamidi and Oussama Damen. Fair wireless federated learning through the identifi- cation of common descent direction. IEEE Communications Letters, 2024.
Shayan Mohajer Hamidi and En-Hui Yang. Adafed: Fair federated learning via adaptive common descent direction. arXiv preprint arXiv:2401 04493,, 2024.
Shayan Mohajer Hamidi, Mohammad Mehrabi, Amir K Khandani, and Deniz Gündüz. Over-the-air federated learning exploiting channel perturbation. 2022 IEEE 23rd International Workshop on Signal Processing Advances in Wireless Communication (SPAWC), pp. 1-5. IEEE, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 02531,2231,200 2015.
Dat Huynh and Ehsan Elhamifar. Fine-grained generalized zero-shot learning via dense attribute- based attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4483-4493, 2020.
Fotis Iliopoulos, Vasilis Kontonis, Cenk Baykal, Gaurav Menghani, Khoa Trinh, and Erik Vee. Weighted distillation with unlabeled examples. Advances Neural Information Processing Systems, 35:7024-7037, 2022.
Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma, and Ali Ghodsi. Annealing knowledge distillation. arXiv preprint w0xivv2214..0///0 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. University of Toronto, 2009.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). University of Toronto, 2012. URL http:://ww.w.c../ cs torooto.
eifar. html
Zheng Li, Ying Huang, Defang Chen, Tianren Luo, Ning Cai, and Zhigeng Pan. Online knowledge distillation via multi-branch diversity enhancement. In Proceedings of the Asian Conference on Computer Vision, 2020.
Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 1504-1512, 2023.
Jihao Liu, Boxiao Liu, Hongsheng Li, and Yu Liu. Meta knowledge distillation. arXiv preprint arXiv:2202.07) 07940, 2022.
11

---

Published as a conference paper at ICLR 2024
Li Liu, Qingle Huang, Sihao Lin, Hongwei Xie, Bing Wang, Xiaojun Chang, and Xiaodan Liang. Exploring inter-channel correlation for siversity-preserre knowledge distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8271 128200, 2021.
Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7096-7104, 2019.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-effccient learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In International Conference on Machine Learning, pp. 7632-7642. PMLR, 2021.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5191-5198, 2020.
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. arXiv:2002.03996, 2020.
Subclass distillation. arXiv preprint
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3967-3976,
2019a.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3967-3976. 2019b.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 268-284, 2018.
Nikolaos Passalis, Maria Tzelepi, and Anastasios Tefas. Heterogeneous knowledge distillation using information flow modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2339-2348, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5007-5016, 2019.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In International conference on machine learning, pp. 51 5142-5151. PMLR, 2019.
Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards omni-supervised lcarning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4128- 2018.
Yi Ren, Shangmin Guo, and Danica J Sutherland. Better supervisory signals by observing learning paths. In International Conference on Learning Representations, 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint 44Xiv:1412.65500. 2014.
12

---

Published as a conference paper at ICLR 2024
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-2555 2015.
Ahmad Sajedi and Konstantinos N Plataniotis. On the efficiency of subclass knowledge distillation in classification tasks. arXiv preprint arXiv:2109.05587 2021.
Aman Shrivastava, Yanjun Qi, and Vicente Ordonez. Estimating and maximizing mutual information for knowledge distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 48-57, 2023.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34: 6906-6919, 2021.
Chao Tan and Jie Liu. Improving knowledge distillation with a customized teacher. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Understanding and improving knowledge distillation. arXiv preprint arXiv:2002.0353. 2020.
Xudong Tian, Zhizhong Zhang, Shaohui Lin, Yanyun Qu, Yuan Xie, and Lizhuang Ma. Farewell to mutual information: Variational distillation for cross-modal person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1522- 1531, 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Interna- fional Conference on Learning Representations, 2019.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (inw), pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando Percira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Kari Torkkola. Feature extraction by non- non-parametric mutual information maximization. Journal of machine learning research, 3MMar): 2003.
Frederick Tung and Greg Mori. Similarity- knowledge distillation. 2019 IEEE/CVF
International Conference Computer Vision (ICCV), pp. 1374-1374 2019. URL https
Maria Tzelepi, Nikolaos Passalis, and Anastasios Tefas. Efficient online subclass knowledge dis- tillation for image classification. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 1007-1014, 2021a. doi: 10.1 0LL1009/CCC44888822211.91991 949999999.
Maria Tzelepi, Nikolaos Passalis, and Anastasios Tefas. Online subclass knowledge distillation. Expert Systems with Applications, 181:115132, 2021b.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Chaofei Wang, Qisen Yang, Rui Huang, Shiji Song, and Gao Huang. Efficient knowledge distillation from model checkpoints. Advances in Neural Information Processing Systems, 35:607-619, 2022.
Yongqin Xian, Bernt Schicle. and Zeynep Akata. Zero-shot learning- the good, the bad and the ugly. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4582-4591, 2017.
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10687-10698. 2020.
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan L Yuille. Training deep neural networks in generations: A more tolerant teacher educates better students. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 5628-5635, 2019.
13

---

Published as a conference paper at ICLR 2024
Chuanguang Yang, Zhulin An, Linhang Cai, and Yongjun Xu. Hierarchical self-supervised augmented knowledge distillation. arXiv preprint arXiv:2107.13715, 2021.
Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang, Cross-image relational knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12319-12328, 2022.
En-Hui Yang, Shayan Mohajer Hamidi, Linfeng Ye, Renhao Tan, and Beverly Yang. Conditional mutual information constrained deep learning for classification. arXiv preprint arXiv:2309.09//3 2023.
Jing Yang. Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos. Knowledge distillation via softmax regression representation learning. In International Conference on Learning Representa- fions, 2020.
Linfeng Ye, En-hui Yang. and Ahmed H. Salamah. Modeling and energy analysis of adversarial per- turbations in deep image classification security. In 2022 17th Canadian Workshop on Information Theory (CWIT), pp. 62-67, 2022. doi: 10. 1091C0TT5533300222222222 200.53980020222877767777
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4133-4141, 2017.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference on Learning Representations, 2016.
Borui Zhao, Quan Cui, Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVE Conference on computer vision and pattern recognition, pp- 11953-11962, 2022.
Jinguo Zhu, Shixiang Tang, Dapeng Chen, Shijie Yu, Yakun Liu, Mingzhe Rong, Aijun Yang, and Xiaohua Wang. Complementary relation contrastive distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9260-9269. 2021.
Xiatian Zhu, Shaogang Gong, al. Knowledge distillation by on-the-fly native ensemble, Advances in neural information processing systems, 31, 2018
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 RELATED WORKS ON MUTUAL INFORMATION BASED KD
Viewed through the lens information theory, knowledge distillation can be regarded as the process of maximizing the mutual information between teacher and student models. Considering that the true distribution between the teacher and the student representations are unknown, Ahn et al. (2019) maximizes variational lower-bound for the mutual information between the teacher and the student representations by approximating an intractable conditional distribution using a pre-defined variational distribution. To better retain sufficient and task-relevant knowledge, (Tin al. (2021) utilized the information bottleneck (Tiisby et al. 2000) to produce highly-represented encodings. In order encompass structural knowledge, specifically, the high-order relationships among sample representations, Zhu et al.] (2021); (Tian et al. (2019) have introduced an approach that involves maximizing the lower bound of mutual information between the anchor-teacher relation and the anchor-student relation. This approach offers a practical solution within the realm of contrastive learning. [Shrivastava
al. (2023) employs a contrastive objective, allowing to simultaneously estimate and maximize lower bound on the mutual information pertaining to local and global feature representations shared between a teacher and a student network. Furthermore, Passalis et al. (2020);
Passalis & Tefas (2018) employed the Quadratic Mutual Information (Torkkola (2033) (that replaces the KL. divergence with a quadratic divergence) to measure the information contained within teacher and student networks.
We note that our approach differs from these previous works in that we employ information- -theoreti principles primarily to optimize teacher training. opposed to utilizing them solely for the distillation process. Indeed, our approach aids the teacher in achieving more accurate estimation of the true BCPD.
A.2
WHY LARGER TEACHERS HARM KD
0.9
3.0
0.9
0.0
0.8
MLL
0.8
MLL
0.7
CHI
53.03
50.57
0.7
-8
CM
1:1.
60.98
60.29
-0.2
0.5
0.5
-0.5
-03
0.3
0.3
60.76
02
0.1
0.1
0.0
0.0
Reshiet52
Reshett6
ResMeti10
Reshett52
Bediet11
(a) KD
(b) CRD
Figure 5: The effect of teacher's size the (i) student's accuracy, (ii) teacher's CMI, and (iii) teacher's LL for both MLL and MCI teachers. The student model ResNet8, and the dataset CIFAR-
is known that using large model as the teacher would considerably degrade the student performance (Liu et al., 202T). Many papers attribute the cause of this issue to the capacity gap between the teachers and the students, and to that the small-scale model struggles to grasp the intricate, high-level semantics extracted by the larger model (Cho & Hariharan 2019 Mirzadeh al. 2020). Nevertheless. these explanations lack persuasiveness. In fact, we can answer to this phenomenon by observing the I.I. and CMI values for teachers of different sizes.
In particular, consider fig. 5a where KD method is used to train ResNet8 using MLL and MCMI teachers of different sizes. As observed, for both MLL and MCMI teachers, as the model size increases, the one hand, the teacher's LL value increases (beneficial), and on the other hand, the teacher's ' CMI decreases (detrimental). However, for MLL teachers with larger size, the CMI values rather small, although their LL values are quite big. This is the reason that why larger MLL teachers are not often hurt the student's 's accuracy (as seen the student's accuracy starts decreasing as the MIL teacher model size goes beyond ResNet56).
15

---

Published as a conference paper at ICLR 2024
However, this issue is mitigated when using MCMI teachers, in that the student's accuracy always rises as the teacher's 's size increases. This is due to the fact that compared to the MLL teacher, the CMI value for the MCMI teacher does not drop significantly the model size increases. Therefore, an MCMI teacher with a large size can establish a better trade-off between its LL and CMI values yielding better BCPD estimate.
A.3 COMPARISON BETWEEN MCMI AND EARLY-STOPPED TEACHER IN KD
In this subsection, we compare the effectiveness of using MCMI teacher Vs. early-stopped (ES) one in KD. In particular, we conduct experiments on ImageNet dataset, where the teacher-student pairs are ResNet34- 8 and ResNet50- Mooeee00--Milleeee222 The setup for training MCMI teacher the same as that used for generating the results in table 2 For the ES teacher, following Cho & Hariharan (2019), we train ResNet34 for epochs, and ResNet50 for 35 epochs. The results are summarized in table
As seen, the student's accuracy is higher when is trained by MCMI teacher compared to when is trained by the other counterpart teachers. In fact, the problem with ES and MLL teachers are that (i) the ES teacher has a low LL value and a relatively high CMI value (to see why an ES teacher has a higher CMI value compared to MLL teacher, please refer to Yang et al. (2023)), and (ii) the MLL teacher has a high LL value, yet a low CMI value. Therefore, none-of these methods can establish a fair trade-off between MLL and LL, and consequently they cannot properly estimate the true Px- This issue is alleviated for the MCMI teachers.
Table 4: Student's accuracy (%) on ImageNet when is trained by MLL, ES, and CMI teachers.
Teucher-Student
R34-R18
R50-MNV2
Teacher's 's Acc.
CMI
Student's s Acc.
Teacher's Ace.
CMI
LL.
Student's Acc.
MLL
73.31
0.7203
-0.3600
71.03
76.13
0.6002
40,4492
70,50
ES
68.68
0.9958
-0,9217
71.25
70.14
0,9598
-0.8840
70.76
MCMI
71.62
0.8650
-062662
71.54
74.94
0.7150
-0.4943
71.10
A.4 HOW CMI AFFECTS THE OUTPUT SPACE OF A DNN
A.4.1 VISUALIZE THE OUTPUT PROBABILITY SPACE OF DNNS WITH DIFFERENT CMI VALUE
Gf
100
50
-50
-50
-100
-100
50 (a) DNN with low CMI value.
-100
(b) DNN with high CMI value
Figure 6: Visualization of the output probability space of the DNN with (a) low CMI value, and (b)
high CMI value.
The DNN's outputs to the instances from specific class Y y. for [C]. form cluster in the DNN's output space. The centroid of this cluster is the average of Px w.r.. the conditional distribution P (-I = y), which indeed the definition of Qe in eq. 11
16

---

Published as a conference paper at ICLR 2024
Referring to eq. (1I). fixing the label Y y, the CMI(f.Y = measures the average distance
between the centroid Q" and the output probability vectors Px for class y. Therefore, CMI(f.Y = y) tells how all output probability distributions Px given y concentrated around its centroid Q". As such, the smaller CMI(S, J.YY Y = y) is, the more concentrated all output probability distributions Px given y are around its centroid.
To visualize this, we train two DNNs on CIFAR- 10 dataset, one with high and one with low CMI values. Then, we use t-SNE (Van der Maaten & Hinton 2008) to map the 10-dimensional output probability space of the DNN into 2-dimensional space. The result is depicted in fig. 6 As seen, the clusters for the DNN with lower CMI value are more concentrated (less dispersed).
A.4.2 COMPARING TEACHERS TRAINED BY THREE REGULARIZATION TERMS: CMI, ENTROPY, LABEL SMOOTHINNG
70.84
-0.5
5-
69.14
0.12
Entropy
0.10
Entropy
0.3
3-
Entropy CMI
Entropy
70.01
2-
67.37
0.04
57.01
66.97
-0.1
0.02
69,56
0.00
0.0
MLL
MCMI
LS
Entropy
MLL
MCMI
LS
Entrapy
(a) T: VGG-13, S: MobileNet-V2
(b) T: ResNet-56, S: S:ResNet-20
Figure 7: The effect of various regularization methods and MCMI method on (i) student's 's accuracy, (ii) teacher's entropy, (iii) teacher's 's CMI.
In this subsection, in order to unveil the reason why label smoothing hurts student's accuracy in knowledge distillation (Tang et al., 2020), we conduct a comparative study to see how training teachers with different regularization terms, namely {CMI, entropy, label smoothing). affect KD.
To this end, we train teachers using these regularization methods, and report the teacher's CMI value and the average entropy of its output probability vectors. Then, student via KD framework, and also report the accuracy of the resulting students. The results for two teacher-student pairs depicted in fig. where also depicted the results for an MLL teacher (the teacher with no regularization term).
these three teachers to train
As observed, the CMI regularization does not increase the entropy of the teacher's output predictions. On the other hand, the teachers trained by entropy and LS regularization have low CMI value, and high output's entropy. This shows that the CMI value probability vectors. In addition, we see that to train students with high accuracy, the teacher's CMI value should be high, and the entropy of its output is not important.
related to the entropy of the output
A.5 PROOF OF PROPOSITION
Proof. For given DNN, use fx to represent the intermediate feature sample X. As discussed in Tishby Zaslavsky 2015): Ye al. (2022), the layered structure of DNN forms a Markov chain
X X fx -
(15)
where Y is the output DNN the input X. By writing the CMI in terms of entropy we obtain
T(X; Y) = () H
(16)
T(X;fx Y) H(XY) HXXixx
(17)
17

---

Published as a conference paper at ICLR 2024
On the other hand,
H(X]fx,Y) = H(XIfx,
(18)
,H(XIY,Y),
(19)
where eq. 18) is due to the properties of Markov chain, and eq. 19 holds since removing random variable from condition increases the entropy. This together with eqs. (16) and (17) complete the proof of the proposition.
A.6 SYNTHETIC GAUSSIAN DATASET
First, let us denote the predictions of the MLL and MCMI teachers by P MLL and P MMMM respectively. In this subsection, we aim to examine whether Ps 2.MCMI better estimate of Px than Pt 2,MLL
Since the true BCPD is unknown for the popular datasets, we generate a synthetic dataset whose BCPD known in the manner explained in the sequel (following Ren al. (202I)).
We generate a 10-class toy Gaussian dataset with 10s data points. The dataset is divided into training, validation, and test with split ratio [0.9, 0.05, 0.05). The underlying model (for both teacher and student) in this set of experiments is 3-layer MLP with ReLU activation, and the hidden size 128 for each layer. We set the learning rate training epochs is 100.
10 the batch size 32, and the number of
The sampling process is implemented as follows: we first choose the label y using a uniform distribution across all the 10 classes. Next, we sample zly=k ~ N O2)) as the input signal. Here, μą is 30-dim vector with entries randomly selected from {-8 0, 5,
Then, we train the student using
Baa (J
A
Σ
(Par)²
-log (Pa.
(20)
N
DE[N]
for 4 different values of plr; (i) Ground Truth, Plar Px: (ii) MCMI KD, pur Pd
MCMII (iii)
MLL KD, ptar Pt. LL and (iv) On-Hot, ptar Zn = e(y)-
In experiments, we a 4 and different 8, values as 6, (0.5, 1,2,,1 4]. and report the student's accuracy and the L2 norm of the distance between Px and piar denoted by Pdist in table (the reason that we used L2 norm, and not other distance metries, is that the student's loss variance IS upper-bounded by function of || - par (Menon et al. 2021)).
As seen in table 5 compared to pt I.I. pt MMMC consistently provides better estimate of Px leading to higher test accuracy for the student.
Table 5: Results on synthetic Gaussian dataset.
Dataset parameters
Metrics
Ground Truth
MCMI KD
MIL KD
One-Hot
Accuracy
37.21 + 0.07
57.02 + 0.18
56.32 + 0.27
55.48 +:2 0.55 41.23
6n- = 0.5
Pdist
13.25 + 0.04
14.41 = 0.05
Accuracy Pdist
69.68
0.08
69.32 + 0.09
68.88 0.11
67.76 0.24
6n
9.02 0.07
9.65 = 0.06
34.52
Accuracy
76.98 + 0.04
76.54 4: 0.06
76.15 + 0.10
75.71 4: 0.15 7.11
6n =
Pdist
4.42 + 0.02
4.77 = 0.03
Accuracy Pdist
81.68 0.01
81.60 + 0.02
81.57 1 0.05
81.44 0.08
6g
2.44 0.02
2.54 = 0.02
3.18
18

---

Published as a conference paper at ICLR 2024
A.7 POST-HOC SMOOTHING
known that by tuning the teacher's temperature 7 and the student's temperature 7 one can improve the student's accuracy (Liu et al.) (022). Here, we show that the improvements in the student's i accuracy we obtained by using MCMI teachers cannot be obtained by such post-hoc smoothing method, i.e., changing the temperatures of the student and teacher's model.
To this end, we record student's accuracy when trained by MLL and MCMI teachers for three different teacher-studede pairs, where the teacher's temperature ranges in T" f1. 8), and the student's s temperature in T {1, 2, 4}. Note that the À value for the MCMI teacher is the same for different combinations of 7 and T and it not tuned for each combination.
The results are illustrated in fig. where each row corresponds to teacher-student pair. The observations follows:
As seen, no matter what combination of T and T used for the student and teacher model, the student's accuracy higher when using an MCMI teacher. Therefore, we conclude that, by using post-hoc smoothing method (temperature), the student's accuracy cannot improve as much as it can when using MCMI teacher.
MLL suclo
MLL hacles MC14 Stuttre lome
71.0
VEL
MCMI Time
VCV tucke,
71.--
20.5
St.tem Tere
13.3
Student Ace.
Stdeet
21.0-
12.
Temperature
Temperature
Teacher Temperxturn (c) Resseet0 T
(a)
7 =
(b) RessNN00 T 2
Soudet Tera
18.0
11.04
Statei Twee
2040
Soudert Tera
33.00
Student Acc.
2010
lucker
p'll tucher
03:2
Teache Temoerature
Teacher Tempecature
Teacher Tempreature
(d) ResNet1 Re-2ReNe120, T
(e) 1 T
2
(f)
10 T
12
67.
Student Poc. tome
08.5
MiLL
HiL
VILL
Ststent
036
612
Statiet cep
Staden Tamp 2
Soudert Terr --4
2
Teacher Temperature
Teacher Temcersture
Teacher Temperature
(g) ResNNe550=MMNVV T
(h) ResNet50-MMNVV T
(i) ResNet50-+MN-V V2,
Figure 8: The student's accuracy in KD with various combinations of T and T values, when both the MCMI and MIL teachers are deployed. The teacher-student pairs are ResNet56-ResNet20 (top), ResNetl (e-ResNe120 (middle), ResNet5U-MobileNet22 (bottom).
19

---

Published as a conference paper at ICLR 2024
A.8 IMPLEMENTATION DETAILS OF THE EXPERIMENTS IN SECTION 6
A.8.1 FINE-TUNING SETUP FOR TRAINING MCMI TEACHER
As discussed, to train MCMI teacher, we fine-tune pre-trained teacher. For all the datasets, obtain the pre-trained teacher from PyTorch official repository (Paszke et al. 2019) 6 Then, for fine-tuning,
cosine annealing learning rate scheduler with its initial value equals one-fifth of the initial learning rate that was used to train the pre- pre-trained teacher. We utilized the same set of teacher weights across all experiments, wherein various KD variants were employed to train different student models. All the other training setups are the same as those used to train the pre-trained teacher, except the number of epochs for fine-tuning which explained as follows:
For CIFAR-(10, (100 } we fine-tune the teachers with 20 epochs and test different À values, namely {0.1, 0.15, 0.2, 0.25 yn] then we pick the best À value, denoted by X² We list the X values for the teacher models on CIFAR-100 100 dataset in table 6
Table 6: X values used for fine-tuning different teacher models on CIEAR-(10,100) datasets.
Re
ResNet-110
WRN-40-2
VGG-13
ResNet-50
R2-Net-32-4
CIFAR- 100
0.13
0.10
0.15
0.20
0.20
0.15
CIFAR-10
0.25
For ImageNet, we use 10 epochs to fine-tune the pre-trained teachers, and test different À values, namely = {0.05, 0.1, 0.15, 0.2, 0.3). Specifically, we pick X 0.1 and X 0.15 for ResNet-34 and ResNet-50, respectively.
In fact, we opted to fine-tune the model for and 10 epochs for CIFAR and ImageNet, respectively, due to two primary reasons.
1. Tuning the trade-off: We can effectively establish the desired trade-off between LL and CMI values by adjusting the value of which serves as a weight parameter. Going beyond 20 (10) epochs on CIAFR (ImageNet) datasets does not offer significant benefits in achieving this trade-off, making a practical and efficient choice for model tuning.
2. Time complexity consideration: We aim to keep the number of fine-tuning epochs relatively low to avoid introducing excessive time complexity into the knowledge distillation (KD) process. This ensures that the overall training process remains efficient.
A.8.2 KNOWLEDGE DISTILLATION ALGORITHM SETUP
For all variants of knowledge distillation and settings in this paper, SGD is applied as the optimizer.
For CIFAR-100 dataset, train 240 epochs for all experiments with initial learning rate of 0.05 by default, which will be decayed by factor of 0.1 at epoch 150, 180, 210. For MobileNet ShuffleNetV1, and ShuffleNetV2. smaller initial learning rate of 0.01 is used. We adopt batch size of 64.
For ImageNet dataset, we used the same training recipe PyTorch official implementation, except train for 10 more epochs. Batch size of 256 is adopted.
In the paper, incorporate several state-of-the-art KD variants. Specifically, employ linear combination of cross-entropy loss and knowledge distillation loss, as outlined below:
i = tCE + 7
(21)
For different KD variants, we directly adopted the y values as those reported in their original papers. The y values and additional details for each KD variant used in the paper are outlined below:
1. AT Zagoruyko & Komodakis 2016 7 1000;
hhttps:/ppyorcch httpps///ptorcch.../viiioooo/// awg/vision/sablemmoods...m html.
In table 7 we study how different À values affect the teacher's CMI and L.I., and the student's accuracy.
20

---

Published as a conference paper at ICLR 2024
2. PKT Passalis & Tefas
2018): 30000;
3. SP (Tung & Mori
2019): 33000;
4. CC PPng et al.
2019)
0.02;
5. RKD (Park et al.
2019a): 71 25 for distance, and 7/2 = 50 for angle;
6. VID (Ahn et al.
2019
= 1;
7. CRD Tian al.
2019):
0.8;
8. DKD Zhao et al.
2022):
Consistent with their official implementation, select y from
the set [0.2, 1, 2, 4,
8):
9. REVIEWKD Chen from the set (0.6, 1,5,8) 8
2021a): Consistent with their official implementation, we select ?
10. HSAKD Yang et al. 2021
For KD, we follow the original implementation in (Hinton et al. 2013), set = 0.9, and T = 4.
For both zero-shot and few-shot, we follow the same training recipe that discussed in this subsec- tion.
A.8.3 EFFECT OF IN MCMI
In this subsection, our objective is to investigate the impact of the parameter À in MCMI estimation when maximizing the objective function in eq. (14). To this end, fine-tune pre-trained teacher employing À {0, 0.005, 0.01, 0.1, 0.15, 0.2, 0.25J. As expected, as À increases, the teacher's 's CMI value also increases while its LL value decreases. In addition, the student's accuracy forms a quasiconcave-like function w.r.t A, and it reaches its maximum value at A 0.15. In fact, at this point, a good trade-off between the teacher's CMI and LL values is established yielding good BCPD estimate by the teacher.
Table 7: The effect of on student's accuracy and Teacher's CMI and LL values. The teacher-student pair ResNet-56-RReNett-2 20, and the values are averaged over 5 different runs.
0.005
0.01
0.05
0.15
0.2
0.25
Student's Acc.
70.62
70.67
70.73
70.78
70.81
70.84
70.73
70.66
Teacher's CMI
0.1602
0.3045
0.3830
0.4053
0.4326
0.4428
0.4430
0.4424
Teacher's L.L
-0.114
-0.212
-0.317
-0.325
-0.336
-0.344
-0.352
-0.363
21

---

Published as a conference paper at ICLR 2024
A.9 EFFECTIVENESS OF MCMI TEACHER IN SEMI-SUPERVISED DISTILLATION
Semi-supervised learning (Chen et al.
2020b; Iliopoulos et al. 2022) is a popular technique due to its ability generate pseudo-labeis for larger unlabeled dataset. In the context of KD, the teacher is indeed responsible for generating pseudo-labels for new, unlabeled examples.
In order to evaluate the MCMI teacher's performance under semi-supervised learning scenario, we conducted experiments on CIFAR-I0 10 dataset, follow the setting of et al. (2020b). with ResNet18 the student model. The results present below are averaged over 3 runs. In this setup, although (500),
all the 50000 training images are available, only a small percentage of them are labeled- 2% (1000), 4% (2000), and 8% (4000) The results are depicted appendix where the student accuracy is plotted as function of number of labeled samples. As seen, not only can the MCMI teacher effectively perform in the semi-supervised distillation scenario, but it also outperforms the MLL teacher.
MLL
91
90
06
500
1000
2000
4000
Labeled
Figure 9: Effectiveness of MCMI teacher in semi-supervised distillation.
22

---

Published as a conference paper at ICLR 2024
A.10 EIGEN-CAL FIGURES
In this section, we visualize the activation maps of MCMI and MLL teacher for 4 different classes of ImageNet datasets, namely (goldfish, mountain tent, toy terrier, bee eater). To achieve this, we have randomly chosen 24 samples from each of these four classes. In addition, we compare the CMI value of MCMI and MLL teacher over these four classes in table
Focusing on Eigen-CAM figures for a specific class, let's say "mountain tent", we observe that the MCMI teacher activation maps have more randomness compared to the MLL teacher. To shed more light, the MLL teacher always highlights the tent itself and this is why it has higher accuracy compare to the MCMI teacher. On the other hand, the MCMI teacher more capable of capturing the contextual information as it sometimes highlight the tent as a whole, sometimes the details of the tent, and sometimes some other objects in the background of image.
Table Comparing CMI values between the MLL and MCMI teachers across four distinet classes from the ImageNet dataset.
C aasess
goldfish
mountain TenE
Toy terrier
bcc cater
0.2896
1.1694
0.4132
0.2954 0.3471
MCMI
0.4173
1,2312
0,5414
A.10.1
CLASS "MOUNTAIN TENT"
Original
MLL
MCMI
Original
Original
MCMI
Figure 10: Eigen-CAM for 24 randomly selected samples from class "tent" in ImageNet, for MLL and MCMI teachers.
23

---

Published as a conference paper at ICLR 2024
A.10.2 CLASS "GOLDFISH"
Original
MLL
MCMI
Original
MLL
MCMI
Original
MLL
MCMI
Figure 11: Eigen-CAN for 24 randomly selected samples from class "goldfish" in ImageNet, for MLL and MCMI teachers.
24

---

Published as a conference paper at ICLR 2024
A.10.3 CLASS "BEE EATER"
Original
MLL
MCMI
Original
MLL
MCMI
Original
MLL
MCMI
Figure 12: Eigen-CAM for 24 randomly selected samples from class "bee eater" in ImageNet, for MLL and MCMI teachers.
25

---

Published as a conference paper at ICLR 2024
A.10.4 CLASS "ToY TERRIER"
8
Original
MLL
MCMI
Original
MLL
MCMI
Original
MLL
MCMI
Figure 13: Eigen- CAM for 24 randomly selected samples from class "Toy Terrier" in ImageNet, for MLL and MCMI teachers.
26

---

Published as a conference paper at ICLR 2024
A.11 SHRO-SHOT KD
A.11.1 REAL-WORLD EXAMPLES FOR ZERO-SHOOT KD
Zero-shot learning in the context of knowledge distillation (KD) can be used when you want a student model to learn from a teacher model while omitting certain classes. In this subsection, we will provide detailed explanations of two real-world examples that show the application of zero-shot learning within the framework of KD.
Consider the scenario that you are tasked with developing disease classification system using medical images as input data. The central repository of these medical images is managed by the government, which possesses the teacher model designed for this purpose. However, local hospitals express interest in utilizing these images for disease diagnosis. Due to privacy concerns, sharing all the images with these hospitals is not feasible McMahan et al. (2017): Hamidi & Yang (2024); Guo eet al. 2021); Hamidi & Damen (2024); Hamidi et al. (2022).
Here's 's how shot learning in KD with omitted classes could be applied:
1. Teacher Model: You have a teacher model that has been trained on a large medical images including those sensitive ones.
2. Omitted Classes: Your objective is to create a student model for disease diagnosis. However, you want to exclude certain sensitive images from the training dataset.
You create student model for disease classification and initiate knowledge distillation from the teacher model.
Knowledge Distillation:
4. Zero-Shot Learning: The student model, can still perform diagnosing effectively, even for sensitive images that were omitted during training. It achieves this through zero-shot learning, it learns to predict diseases that have never seen directly in the training data by leveraging the knowledge from the teacher model.
In this scenario, zero-shot learning in KD allows you to build a classification system that respects privacy and confidentiality by omitting certain classes (sensitive images) while ensuring that the diagnosing quality remains high for other deceases.
A.11.2 COMPARING TO CONVENTIONAL ZERO-SHIOT LEARNING
As discussed in the main body of the paper, the MCMI teacher can help the student to perform generalized zero-shot learning, where the task is classify samples from both seen and unseen classes.
The methods in the literature proposed for generulized zero-shot learning are based on the assumption that we have access to the class semantic vectors that provide descriptions of classes (Akata et al.) 2015; Huynh & Elhamifar 2020: Xian et al., 2017: Bucher et al.) 2017). A common approach to use such semantics for classifying samples from both seen and unseen classes is to employ positive/negative scoring, which assigns positive scores related semantics and negative scores to unrelated semantics.
This method has major downfall: the semantic vectors for both seen and unseen classes should be available at the training time. This will, in turn, hinder applicability of these methods. However, our method for generalized zero-shot learning does not depend on such information. We further note that our method is generic in that it does not matter which class(es) is (are) missing in the student's s training dataset.
A.11.3 WIYY OUR METHOD WORKS FOR ZERO-SHOT LEARNING?
Here, we clarify the reason why our method can classify the missing classes. This is due to the fact that the information regarding the labels for the missing classes is still contained within P of the samples presented in the dataset. To elucidate, consider the following example in which the objective is to train a DNN to classify three classes: dog, cat, and car. Assume that the training samples for dog is entirely missing from the training dataset. Suppose that for particular training sample from the cat class, p = [0.30, 0.68, 0.02], with the probability values corresponding to the dog, cat, and car classes, respectively. The value of 0.30 suggests that this sample exhibits some features resembling
27

---

Published as a conference paper at ICLR 2024
the dog class, such as a similarity in the shape of legs. Such information provided from the BCPD vectors for the samples of existing classes can collectively help the DNN to classify the samples for the missing class. We note that the one-hot vector for this sample is [o. 1, 1, and therefore, such information does not exist when one-hot vectors are used as unbiased estimate of Ps
A.11.4 MORE ZERO-SHOT KD RESULTS
CIFAR-100: We consider the zero-shot in KD over CIFAR- 100 where the teacher is trained over the whole dataset while some number of classes are dropped during the distillation. Specifically, consider dropping f5, 10, 20, 30, 40, 50) classes The teacher and student models are WRN-40-2 and WRN-16-2, respectively. We compare using the MLL and MCMI teacher in the above scenario. Then, we report the accuracy of the student for both dropped and preserved classes, separately.
The results summarized in table As the results suggest, the accuracy for the dropped classes is significantly increases by using the MCMI teacher. In addition, the accuracy for the preserved classes almost the same when using either teacher. This suggests that the accuracy for the preserved classes is not sacrificed for a higher accuracy for the dropped classes.
Table 9: Student accuracy of zero-shot learning in CIFAR-100, where teacher is WRN-40-2, and student is WRN-16-2. Numbers red indicate the accuracy of omitted classes when training students, those in blue represent that of preserved classes. The reported results are averaged over different random seeds.
dropped classes
50
MILI
0.16/ 75.02
2.70 75.82
1.63 / 76.28
1.11 / 80.15
1.74 / 80.42
0.87 82.38
MCMI
25.64/75.08
32.78 75.78 30.61 76.22
21.72/ 80.27
23.74/ 80.79
22.26 82.46
10 10 confusion matrix C to better observe the model performance. Each row in represents the instances in an actual class, while each column represents the instances predicted by the model. In partieular, the entry C.,, [10], represents the probability that the instances in class are correctly classified. On the other hand, Ct i,j E [10]. i * j. represents the probability of the cases where instances from class are incorrectly classified as belonging to Class j.
CIFAR-10: For this dataset,
Then, we consider dropping {1,2,3, 4, 5, 6) classes from the student's training set. The dropped classes are denoted by red color in the confusion matrices.
28

---

Published as a conference paper at ICLR 2024
MLL Teacher
MCMI Teacher
0.99
9.01
0:00 0.00 9.00 0.02 0.00
9.
0.02 0.01 0.00 0.00 0:00 0.00 5.02 0.00
3.0
00.00 0:93
0.00
2.00
0.00 0.00 0.00 0.00 0.00 2:00
-0.00 5.69 0.00
0.00
0.00 0.00 0.00 0.00 2.00 0.01
0.
0:00
0.
3.02 0.02 0.02
5.02 0.00 0.00 5.00
-0.02
0.00
0:8
0.02 0.03
0.02
0.02 0.01 5 03 0.20
0.01 0:00 0.02
3.85
000 0.08 9.02 0.03 0.00 0.00
01
9.00
0.01
0:86
0.02
0.08 0.02 0.00 0.00 0.00
0.00
0..00
0.00
-04
0.00
0:00
0.00
0.00
0.90
0.00
5.00
00
0.00
0.02 0.00
0.00
0.00
2.00
0:00
0.00 0:00 0.00 3.01 0.01 0.02
9.00
0.94
0.00
0.00
.0.00 9.00
00 0.00 9.03 0.02
0:00
0.96
0:00
0.02 0.01 0.00 3.00 0.00 0.00 3.00
0.00
0.95
2.00
0.00 0.01 0.00 0.00 0.00 0.00
0.00
0.00
0.95
0.20
-02
0.00 0.00 0.01
2.00
8202 0:00 0.01 0.00 0.00 0.00
Archann
Autommbbii
Ong
Stip
Mork
Akplame Astommbil
Teachher
Teacher
0..00 0:04 0.57 8.09 0.05 0.01 9:03 0.03 0:27 0.13
0.8 9.02 0.08 0.02 9:03 0.00 0.01 0.01 9.00 0.04
0.00 0.00 3.00 0.00 0.00 5.02
0.97
0.00
0.00 5.00 0.00 0.00 0.00 0.02
0.00 0.00
0.02 0.87 0.02
0.00 0.01 5.00
0.00
0.00
0.02
0:83 0.62
0.09
0.01
0.01
0.6
0.00 0:00
0.01
9.02 0:96 0.91 8:00
0.01
0.00
0.00
00.00
9.00
0.01
0.01
0.55
0.01
0:01
0.01
9.00 0.00
0.00
0:00
0.01
3.07
0.02
0:30
2.00 0.00 0.00
5.00
-0.00
2.00
0.01
0.04
5.02
0.9
0.00 0.01
5.00
0.20
02
0.01
0.01
2.94
0.01
3.00
0.00
0.00
0.02
0.01
9.02
0.02
0.02 03 0:00 0.00 0.00 0.00
0.00 0.02 0.00 0.00 0.00 0:00 0.00
Abpllne Astommible
dey
Doge Cnyy 'rag
Horse
Alrplare Astomobly
-0.0
Figure 14: Zero-shot KD when one of the classes is entirely removed for the student.
MLL Teacher
MCMI Teacher
0. 0.51 2.01 0.00 0.00 0.00 0.00 0.01 5.00
0.02 0.00 0.00 0.00 0:00 0.00 5.02 0.00
1.0
0.00 0:95 0.00
9.00
0.00 0.00 8.00 0.00 0.01 0.00
.0.00 3.59 0.00 0.00
0.00 0.00 0.00 0.00 9.01 0.00
0.01 0:00
0.51
2.02 0.02
0.01 5.02 0.00 0.01 3.00
0.02
2.00
8.9
0.02
0.02
0.02 0.02 5.00 3.01 0.20
0.0000
0.02 0.67 9.01 0.00 0.81
0.02
0.84
0.02 0.07 0.02 0.00
0.6
0.01
0.95
0.02
8.00
0.01
0.01
0:56
3.00
0.00
0:00
0.00
0.05
0:02
0.00
0.01
0.00
0.01 2.02
0.00
0.00
2.09
0.00 0.00
2.00
-0.00 0.00
0.02
0.01
0.00
0.01
0:25
2.00
2.00
0.00
-0-
0.07
0:01
1.05
0.1 0.4 0.35 5.00 0.00 0.00 3.00
0.02 3.00 0.02 0.84 0.07
0.1
0:01
0:73
0.00
0.02 010100 1000 3.00 0.00 0.00 9.00 0.00
0.96 0.00
0.00 9:01 0.00 0.00 3.00 0.00
0.00
0.00
0.95
022
0:00 0.00 3.00
2.00
0.00 0.00 0.00 0.00 0.00
Desy Oug Ary Azpiann Aufommabiee
Sho Plck
Astomitii
Place
Teacher
Arguoee
Teacher
0.91 0G 82 0000 0:00 0.00 9.00 0.00 0.82 0.01
0.02 0.00 3.00 0.00 0.00 0.00 02 0.01
0.00 0.00 0.00 0.00 3:00
0.07
0.3
.0.01 0.00 0.00
0.00 0.00 0.00 3.02 0.36
0004 00
9.00 0.06 0.58
0.02 0.09
0.03
0:68 0.63
0.06
0.02 9.01 0.02
0.6
0.03
0:00
0.01
3.00
0.96 0.02 5.01
0.01
0.00 3.00
0.00
3.00
0.02
0.01
3:94
0.01
0:01
0.01
3 03 0.00
0..00
0:00
0.01
3.00 0.02 094
2.00
0.02
0.00
0.00
-0.00
9.00
0.01
0.02
3:03
0.5
0.00 0.01 3.00 0.00
0:00
001
0.02
0.95
0.00 5.00
0.00
0.00
0.00
0.03
0.00
0:00
0.00 0.00
a. 0. 0.00 00 0:00 0.00 0.00 0.00
01 0.03 0.00 0.00 0.00 0.00 0:00 0.00
Airsano Antommabio
Ony Dny Frg
Alpstare
Astomibble
Bag
0.0
Figure 15: shro-shot KD when two classes are entirely removed for the student.
29

---

Published as a conference paper at ICLR 2024
MLL Teacher
MCMI Teacher
BM
00 9.01 0:00 0.00 9.00 0.02 0.00
0.99
0.02 0.01 0.00 0.00 0:00 0.00 5.03 0.00
3.0
0.00 20 0.00 2.00 0.00 0.00 0.00 0.00 0.00 2:00
-0.00 5.69 0.00
0.00
0.00 0.00 0.00 0.00 2.00 0.01
0.02 0:00
0.91 3.02 0.02 0.00 5.02 0.00 0.00 5.00
-0.02
2.00
0.92
0.02 0.02
0.01
0.02 0.00 03 0.20
0.01 0.00 0.02
0.0 0.00 9.01 0.00 0.03 0.00
01
9.00
0.02
0.9
0.02
0.02 0.92 0.00 9.01 0.00
0.05
0.00
01.01
0.00
-04
0.01
0.00 2.02 0:03 0.00 0.93
0.00 0.00
2.00
00 0.00
0.02 0.00
0.00
0:90
0.00
2.00
0:00
0.07 0:03 0.09 0.24 0:55 0.00 9:00
0.00
0.00
0.00
.0.02 9.00 0.03 0.09 0:11 0.03
0:31
0:71
0.00
0.02 0.01 0.00 5.00 0.00 0.00 9.00 0.00
0.95
0.00
0.002 9.01 0:00 0.00 0.00 0.00
0.00
0.00
0.97
0.20
-02
0.15
0.00 0.00
0.00
0:31 0:00 0.01 0.00 0.00 0.00
Ouy Frog Autommibii
Stip
Mork
Akplame Astommbble
Teacher
Teacher
0.97 0.01 0.00 9.01 0.00 0.00 9:00 0.00 0.00 0.01
0.56 0:01 0.01 0.01 9:00 0.00 0.00 0.00 3.00 0
0.00 0.00 3.00 0.00 0.00 5.02
0.08
0.00
0.00 0.00 0.00 0.00 0.00 0.02
0.00
0.00
0.02 0.06
0.00
0.00 0.00 5.00
0.00
0.00
0.01
0.85 3
0.09
0.00
0.01
0.01
0.00
0.00 0.00 9.01
0:96 0.91 8.00
0.01 0.00 0.00
00.01
8.00
0.00
0.02
0:55
0.01
0.00
0.02
0.00 0.00
0.6
0.01
0:00
0.00
3.08
0.02
0.9 2.00 0.00 0.00 5.00
-0.00
2.00
0.00
0.06
0.02
0.9
0.00
0.01
5.00
0.20
0.05
0.02 0.
024 0.00
9.00 0.01
0.00
0.02 0.22
0.06
0.03
0.51
0.00
0.01
0.01
8.00
0.01
9.02
0.02
0.80
33
n.t 03 1.00 D 0.00 0.00 0.00
0.01 0.00 0.00 0:00
0.00
Airllnn Astormable
dey Doye Ong Frog
Horse
Truce
Airplare
Astomobll
Bey
-.00
Figure 16: shro KD when three classes are entirely removed for the student.
MLL Teecher
MCMI Teacher
BC 000000
0.00 0.00 0.00 0.00 0.02 5.00
95 2.03 0.02
0.00 0.00 0.00 0:00 0.00 5.02 0.00
1.0
0.00 0:99 0.00
9.00 0.00 0.00 8.00 0.00 0.00 0.00
.0.00 3.59 0.00
0.00
0.00 0.00 0.00 0.00 9.00 0.00
0.02 0:00
0.52
2.00 0.03
0.00 9.02 0.00 0.00 3.00
.002 2.00
0.2
0.00
0.03
0.01
0.02 5.00 3.00 0.20
0:02
0.25 5.01 0:33 0.00 0.00
0.01
0.11
0.19
0.11 0.12 0.01
0.6
0.00
0:3
0.00
0.00
0.56
0.00
0.00
0.80
0.05
0.34
0.00
0.07
0:09
0.02
0.00
0.00
0.02
0.02 2.00 0.01 0.00
2.57
0.00 0.00
2.00
-0.00 0.00
0.02 0.01 0.00
0.00
0:36
2.00
2.00
0.00
-04
0.1
0:01
0.1
5.0 00 0:72 0.00 5:03
0.00 0.03 5.00
0.03 2.01 05 0.02 5.18 0.04
0:01
0.63
3:01
0.00
02 010000 00 3.00 0.00 0.00 8.00 0.00 0.96
0.00
0.00 9:01 0.00 0.00 9.00 0.00
0:00
0.00
0.97
022
0.00 0.00 3.00 0.00
2.08
0.00 0.30 0.00 0.00 0.00
Oog Aryg Aufommabiee
Sho Plak
Akpotano Automibile
Astomiti
Place
Arpioo
Teacher
Teacher
0.92 01 00 0:00 0.00 9.00 0.00 0.00 0.01
0:01 0.01 0.01 8.00 0.00 0.00 0.00 3 0.00 0.01
0.00 0.00 0.00 3.00
0.00 0.02
.0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02
00 0000000
0.00 0.09
0.00 0.01
8.00
0.00
0:88 0.00 0.08
0:00 0.01
0.03
0.6
0.00
0:00
0.00
0.4 0.00 0.2
5.00
0.2%
0.00 3.00
0.00
0.00
0.61
0.34
6.62
0.11
0:01
0.07
3 03 0.00
0.00
0:00
0.00
3.07
0.00 0.1
3.00
0.01
0.00 0.00
0..01
3.00
0.00
0.06
3.00
0.9
0.00
0.01
3.00 0.00
0.00
9.00
0.03
5.02
0.00 0.01
2.02
0.00
0:35
0.00
0:00
0.15
0.01 0.00
0.0 02 0.00 00 0.00 0.00 0.00 0.00
01 0.02 0.00 0.00 0.00 0:00
0.00
Airsano Antommiio
Ony Dny Fry
Alpstare
Bag
3.0
Figure 17: Zero-shot KD when four classes are entirely removed for the student.
30

---

Published as a conference paper at ICLR 2024
Aryiice
Teacher
0.96
0.00 0:00 0.00 3.00 0.02 2.00
0.04
0.02
0.00
0.00
0.00
0:00
0.00 2.02 0:50
0.02 0:00 12
0.03
0.00
0.00
0.92
0.00
9.02
0.00
3.8
0.1
0.00
0.13 0.00 0:35
0.00 8.26
0.00 0.05
2.00
00.09
5.00
0.1 0:49 0:13
0.12
0.00
0.01
2.03 0:01
06
0.00
0:00
0.09 3.00 0.97
0.00 9.00
0.00
0.00 5.00
0.00
9.00
0.02
0.00
0.97
0.00
0:01
0.00
3 03 0.00
0.05
0.00
45
0.01 0.34
0.00 9:34
0.00
0.03 9.00
0..02
9.00
0.11
0.07
0.1 0.66 0.92
0.01
9.01 0.00
0.02
0.01
0.00
0.97
0.00
0.00
0.00
0.02
0.01
0.00 0.00
0:36
0.02 0.30
02 02 0:00 0.00 0.00 0.00
0:97
2.00
0202 0.00 00 0.00 0.00 0.00 0:00 0.00
0.59 0000 0.01 000 0.01 0.00 8.02 0.00
0:37
0.00
.0.07 9.02 0.00 0.00 0.00 0.00 0.00 0.00 3:05
Arrieno Automabble
Big
Dog Frag
Horte
Stuck
Alrpleno Automokie
Blog
Dog frog
0.0
MLL Teacher
Auplow
MCMI Teacher
1.0
00.02 0:12 0.00
0.4 0.00 0.06 0.00 0.00
0.33
0.13
0.01 0.00 0.00 0.00 0:00 0.03 2.03 0:07
0.00 94 1.00 03 0.00 0.00 3.00 0.00 0.00 5.00
...0
3.64
0.00
0.00
0.00 0.00 0:00 0.00 3 02 0.02
0..00 0:01 0.00
2.62 0.00 0.25 9.00 0.05 0.00 0.04
00.08
9:01
0:72
0.01 0.09
0.02
0.02 0.05 5.01 0.01
0.00
0.9 0.00 0.00
9.00
0.03 0.00 5.00
0.00
0.00
0:87
0.00 0.1 0.00 0.01 0.01
0.00
5.00
00.00
0.00
-04
-0.02
0.02
0.00 0.32
0.00
0.02 0.00 0.00
-0. 0.00
0:34 0.03 0.05
0.01
2.00
0.00
-0.00 0:00 0.00 3.02 0.00 0.02 8.00 0.96 0.00 0.00
-0.00 9.00 0.00 0.01 9.00 0.02 0:00
0.57
2.00
0.00
0.00 00 9.21 6.00 0.02 9.00 0.02
3.05 0.00 0.01 8.00 0.00
0.00
8.00 0.00 0.00
0.00
0.50
0:03 0.00 0.00 9.00 0.00
0.00
0:36
Degr Dng Froy Araaling Aatomubbee
Muck
Pro
Procy
Figure 18: shot KD when five classes are entirely removed for the student.
Aurtion
MLL Teacher
MCMI Teacher
0.05
0:15 0.01
5.13
0.02 0.00
0.03 0.00 0.09 0.00
0:80 5.00
0.01
000 0.00 0.00 0.00 0.02 0.00 5.12 0.26
0.8
0.00 0.00
1.00 0.03 0.00 9.02 0.00 0.01 0.00
00.01 9.00
0.53 0.00 0.02
0.01 0.02 0.00 0.00 0.00
0.00 0:00
0.32 5.00 0:35 0.00 9.25 0.00 0.05 5.00
00.01
2.00
0.12
0:49 0.12 0.12 0.1 0.01 3.02 0.01
0.00 0.00
0.42
0.00 9.01 0.00
0.00
0.02
0.00
0:97 0.00 0.03 0.00
0.00
0:35
0.00
0.07
9:08
8.01 0.80
0.00 0:00 18
0.76 0.00 9.04
5.00
01 0.00 06 0.01 0.17 0.05
0.00
0.00 0.00 0.01 3.03 0.00 0.00 5.00 0.00
0.49 5.00
0.01 2.00 0.00 0.00 0.00 0.00 0.00
0.00
0.00
-22
0.00 0.00 0.05 3.00 0.04 0.00 9.04 0.00
0:87
02 9:03 0.03 0.00 0.00 0.00 0.02 0.00
0.1
0:33
Deg. Dog Frag Aigiono Autommbbee
Bid
Morse
Thock
Aligtona Automobble
Bigg
Car
Dog
Png
Harso
Ship
Prrck
Alpllm
MUL Teacher
MCMI Teacher
Alsemobll
1.0
1000000
0.08 0.00
0.5 0.00 0.00 03 0.29 0.00 0:52
2.7
0.04 0.51 0.3 0:00 0.00 0.00 0.03 0.04 5.08
1.00 0.50 0.00 0.00 8.50 0.00 02 0. 3.00 0.02
0.00 0:50 3.00
0.00 0:00 00 0.50 0.00 0.00 0.02
0.8
0.000 0.01 0.03
0:81
0.00 0.02 0.35 0.00 0.04
0.05 0.00 2.51
0.03 0.02 0.05 0.00 8.02
0.
0.00 0.00 0.00
0.64
0.00 0.03 0:35
0.03 0.00 9.01
0.01 0.01 0.15 0.00
.0.00 0.00 0.02
0:86
0.00 0.00
0.05 0.13 0.00 0:00
.0.00 0.00 3.50 0.25 0.00
0.57 0.00 0.05
0.00 0.00
.0.00 0.01 0.00
0.94 9.00 0.00
02 0.20 0.00 0.02
...00 0.01 3.00 0.14 0:00
0.50
0:77
0.03
0.00
3.00
0.4
Ship
0.00 0.90 0.00 0.02 0.00 0.00
0.09 0.56
0.00
6.00 0.00 9.00 0.01 0.00
3.00
0.50
0:57
0.00 0.00
0.02 0.00
0,00 0.02
0.00
Airglon
000
Figure 19: Zero-shot KD when six classes are entirely removed for the student.
31

---

Published as a conference paper at ICLR 2024
A.12 ACCURACY VARIANCES
Table 10: The variance in accuracy for the results Table same architectures.
where the teacher and student have the
Teacher
ResNet-36
ResNet- 110
ResNet-110
WRN-40-2
WRN-40-2
VGG-13 13
Student
ResNet-20
ResNet-20
ResNet-32
WRN-16-2
WRN-40-1
VGG-8
KD
70.84 = 0.10
70.85 + 0.20
73.48
0.13
75.42
74.53 = 0.34
73.83 0.22
AT
70.89 0.28
70.68
0.32
73.96 0.18
0.15
73.25 0.28
73.35 + +.16
PKT
70.96 0.38
71.03
0.14 0.12
75.01
74.15
70.98 = 0.07
70.83 1.23:.229
0.114
7.160 0.22
73.29 - 0.14
69.98 0.47
70.02 0.18
0.23
72.50 0.20
71.02 = 0.29
RKD
70.68 0.13
0.17
0.22
7.26 0.157
72.03 0.28
VID
70.64 0.25
70.69 - 0.09
0.
0.25
73.58 + 0.28
71.93 0.10
CRD
=1.40=0.1
71.93 0.17
0.15
74.86 + 0.11
74.23 = 0.05
DKD
72.31 = 0.12
71.83 + 0.18
71.36 0014
76.66
+
75.63 + 0.09
74.87 + 0.11
REVIEWKD
72.31 = 0.26
72.11 + 0.30
74.01 - 0.43
76.29 0.16
75.47
+: 0.14
74.96 = 0.18
HSAKD
72.70 = 0.28
73.15 4 0.16
75.71 0.13
77.36 + 0.17
77.55 1 0.22
75.86 0.20
Table 11: The variance in accuracy for the results in Table different architectures.
where the teacher and student have
Teacher Student
ResNet-50
ResNet-50
ResNet-32xc
ResNet- 32x4
WRN-40-2
VGG-13
MobileNerV2
VGG-8
ShuffleNerVi
ShuffleNerV2 76.32 E 0.47
ShuffleNetVi
MobileNerV2
KD
70.23 1 0.34
74.59
1.166
75.90 0.111
76.45 0.27
69.14 0.22
AT
60.03 0.21
72.19
0.94
75.05 0.17
75.21 + 0.13
75.61 0.25
62.07 1: 0.26
PKT
67.42 0.18
73.43
0.17
75.21
0.27
76.34 + 0.18
75.39
68.37 1: 0.35
SP
69.07 0.17
0.114
76.56 0.29
76.70 0.18
76.82
67.83 0.28
CC
66.76 = 0.42
70.90 0.20
0.17
73.02
+ 0.17
7.19
65.45 0.39
RKD
0.49
72.10 0.27
0.118
74.67 + 0.29
74.26
03.37
VID
07.61
70.69 0.17
74.58
0.23
74.67 1 0.25
75.03
65.77 0.36
CRD
0.69
74.86
0.177
76.82
77.54 0.10
76.62 0.27
69.98 0.13
DKD
71.70=0.33
75.35
0.13
77.66 + 0.19
7.44
70.35 + 0.08
REVIEWKD
70.63 = 0.26
74.20 0.85
77.78 0.31
78.23 + 0.14
77.56 1: 0.32
71.70 + 0.46
HSAKD
72.98 1 0.26
76.45 0.19
79.77
1.09
80.01 1 0.16
78.87 2.47
72.80 1 0.11
Table 12: The variance in accuracy of few shot experiments in Table 3
Percentage
5%
10%
1.5
25%
35%
50%
75%
Teacher
MLL
MCKI
MCMI
MCMI
MCMI
KD
$2.30
58.02
60.13
63.15
63.32
66.20
66.78
68.10
68.25
69.34
69.52
70.28
70.44
70.39
10.26
10.21
1.0.9
10.17
10.32
20.25
20.29
=0.68
=0.37
0.32
0.22
40.23
10.21
4.046
CRD
47.60
51.40
$4.60
56.80
58,90
60.02
63.82
61.70
66.70
67.22
68.84
69.15
70.35
70.10
10.18
10.19
10.41
20.11
1..22
20.29
=0.19
=0.77
=0.23
00.20
=0.24
20.76
=.0.26
2.0.88
32

---

Published as a conference paper at ICLR 2024
A.13 BINARY CLASSIFICATION ON CUSTOMIZED CIFAR (10, 100)
is known that the improvement in student's accuracy in binary classification is typically lower than that observed in multi-class classification scenarios since the amount of information transferred from the teacher to the student network restricted (Sajedi & Plataniotis 2021 Müller et 2020).
Tzelepi et al. 202
In this section, we want to empirically verify the effectiveness of MCMI teacher in binary classification tasks. To this end, we create three binary classification datasets from CIFAR- (10, 100} datasets as explained in the sequel.
Dataset 1: Similarly (uüller al. 2020), we create CIFAR 2 dataset, where the input distribution has "sub-classes" structure, I.c., we merge the first classes of CIFAR-10 as class one, and the remaining classes as class two.
Dataset 2: We keep the training/testing samples from only two classes in the CIFAR-100 dataset: those belonging class 26 (crab) and class 45 (lobster), creating a dataset referred as CIFAR-26-45.
Dataset 3: Similar to dataset 2, but we keep class 50 (mouse) and 74 (shrew) from CIFAR- 100, which we refer as SIFAR-50-72
Now, we use VGG-13 and MobileNetv2-0.. as the teacher and student models, respectively; and use conventional KD, AT, CC and VID distill knowledge from MLL and MCMI teachers the student.
The results the three tasks are listed Table 13 with the values averaged over five different runs. As seen, the MCMI teacher yields better student's accuracy in most of the cases. Also, it is worth noting that in some cases, for instance, AT over CIFAR - 5 dataset, the distillation method hurts the accuracy of the student.
Table 13: Results of binary classification knowledge distillation variants of CIFAR dataset.
Dataset
CIFAR
CIFAR-26-45
CIFAR-50-1
Student Acc.
76.94
65.50
66.30
Teacher
MI.I
MCMI
MI..
MCMI
ML.I
MCMI
KD
77.00
77.41 (+0.41)
69.70
70.10 (+0.40)
71.40
71.20 (-0.20)
AT
67.39
68.19 (+0.80)
64.70
65.43 (+0.73)
67.10
69.50 (+2.40)
CC
77.19
77.85 (+0.66)
70.30
70.75 (+0.45)
69.70
71.40 (+1.70)
VID
77.16
77.34 (+0.18)
69.25
70.00 (+0.75)
69.80
69.80 (+0.00)
33


Reviews:
[{"review_id": "QeT4oMilyH", "reviewer": "Reviewer_oUDj", "summary": "This work  builds upon the insights from the previous study on knowledge distillation [1], which implies that producing a good teacher model \nsimilar to the optimal Bayes class probability $P^{*}_{X}$, is crucial for enhancing the performance of the student model. To convey this message, the authors propose a new training objective for the \"teacher model\" by introducing the empirical estimate of conditional mutual information as a regularizing term (MCMI). \n\nThe authors provide empirical evidence between MCMI and the accuracy of the student model; as the MCMI attains higher values, the the corresponding teach model obtains the highest accuracy. Furthermore, when using the teacher model trained with the MCMI regularizer, the corresponding teacher exhibits improved accuracy in most existing knowledge distillation algorithms. The proposed regularizer leads to improved performance of the student model in zero-shot and few-shot classification tasks  as well.\n\n[1] A Statistical Perspective on Distillation - ICML 21", "questions": "* Q1.  Could you elaborately explain why minimizing $I(X , \\hat{Y} | Y)$ can make the teacher model $f$ to be more similar to the optimal bayes classifier ? \n\n\n\n* Q2. It seems that the proposed regularizer requires the pre-trained model as the teacher model and apply the further training to the teacher model with the proposed objective of Eq. (14). How do we set the number of iterations further training? Based on my understanding, since we expect this regularizer to make the teacher model contain additional information as well as to be properly certain (not overconfident), setting the number of iterations is important hyperparameters and might significantly affect the performance of student model.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### Simple idea:\n\n> In implementation sense, the idea looks simple and easy to implement this idea; introducing a estimate of the MCMI in Eq (2) is additionally necessary.\n\n### Empirical improvement:\n> It seems that the proposed objective for the teacher model can be integrated with existing knowledge distillation algorithms which mainly focus on the distillation objective in view of \"student\" model. The proposed regularizer for the 'teacher' model seems to be effective in enhancing the performance of the 'student' model trained with existing knowledge distillation algorithms.", "weaknesses": "### Less elaboration on relationship between conditional mutual information $I(X , \\hat{Y} | Y)$ and optimal bayes classifier $P^{*}_{X}$\n\n> While it is intuitively clear that using the conditional mutual information as the regularizer term can capture the contextual information of $X$ (Image) and provide additional information to a student model, the direct connection between conditional mutual information and the optimal Bayes classifier is less explained. I believe explaining this connection is important because this approach is motivated from the importance of optimal classifier $P^{*}_{X}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J4o2wR26FP", "reviewer": "Reviewer_Nadg", "summary": "The paper proposes a method which aims to train the teacher to optimise the student. This is achieved through maximising the conditional mutual information between input and predicted label, conditioned on the true label. The approach demonstrates improved knowledge distillation on CIFAR100 and Imagenet using varies CNN architectures.", "questions": "* What is the value of $T$? \n* Does the CMI loss just reduce the entropy?\n* If so, is it possible that the same effect can be achieved by simply running this method with temperature scaling? I.e. drop the CMI term?\n* With regards to 6.2. my understanding is that this is using the negative scores during training, so is this really zero-shot classification? Why do you expect this?\n* Did you try varying different classes to drop? \n* In Figure 3, why is the heat map on the terrier not on the body of the animal? Bottom, third from left.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is very simple to understand and implement, which only a simple regulariser added to the training of the teacher model, which minimises the KL between the predicted probability and the average probability. \n* The results are conclusive and well presented on ImageNet using plenty of architectures. \n* The extension to few and single-shot experiments are nice.", "weaknesses": "In terms of weaknesses:\n* I'm interested to read more about what the role of the CMI regulariser actually does, is it just decreasing the variance of the predictions? Or leading to a distribution with higher entropy? Does this method work just as well if you add an entropy regulariser?\n* As far as I can tell, the value $T$ is not defined, is this for the softmax?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fLM5kciq3f", "reviewer": "Reviewer_AG5J", "summary": "This paper proposed a new distillation technique which is based on training teacher models so that they are well-suited for conveying information to the student models. Towards that end, the authors introduce a\"conditional mutual information\"(CMI) objective into the training process of the teacher model, whose goal is to improve the teacher's Bayes conditional probability estimates (via its soft-labels) — according to recent knowledge-distillation literature, more accurate Bayes conditional probability estimates result in better student's performance.\n\nOverall:\n\n(i) The authors argue that the so-called dark knowledge passed by the teacher to the student is the contextual information of the images which can be quantified via the conditional mutual information.\n(ii) They provide evidence that temperature-scaling in KD increases the teacher's CMI value\n(iii) They provide evidence that show that models with lower CMI values are not good teacher's, even if they're more accurate.\n(iv) They provide experiments on CIFAR-100 and Imagenet datasets showing evidence that their method helps in improving the student's performance, compared to other standard distillation techniques.\n(v) They show that their technique is especially effective in few-shot and zero-shot settings.", "questions": "— Does the proposed method and theory works well/applies in NLP datasets/binary classification contexts? \n— Is there a way to apply this technique in the context of semi-supervised distillation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is a well-written paper that presents a novel approach to knowledge distillation. They authors have provided extensive experimental evidence.", "weaknesses": "— The role of the teacher as a \"provider of estimates for the unknown Bayes conditional probability distribution\" is a theory for why distillation works that applies well mainly in the context of multi-class classification, and especially in the case where the input is images. (Indeed, there are other explanations for why knowledge distillation works, as it can be seen as a curriculum learning mechanism, a regularization mechanism etc see e.g. [1])\n\nIn that sense, I feel that the author should either make the above more explicit in the text, i.e., explicitly restrict the scope of their claims to multi-classifcation and images, or provide evidence that their technique gives substantial improvements on binary classification tasks in NLP datasets (but even in vision datasets).\n\n— One of the main reasons why knowledge distillation is such a popular technique, is because the teacher can generate pseudo-labels for new, unlabeled examples, increasing the size of the student's dataset. (This is known as semi-supervised distillation, or distillation with unlabeled examples, see e.g. [2, 3]. )  It seems that, in order to apply the current approach, one requires the ground-truth labels and, thus,  one has to give up a big part of the power of knowledge distillation as a technique.)\n\nTo be clear, I still like the paper and I am leaning towards acceptance even if the scope of the paper is more limited, but I think it would be beneficial to the research community if the above comments were addressed.\n\n[1] Understanding and Improving Knowledge Distillation [Tang∗, Shivanna, Zhao, Lin, Singh, H.Chi, Jain]\n[2] Big self-supervised models are strong semi-supervised learners [Chen, Kornblith, Swersky, Norouzi, Hinton]\n[3] Weighted Distillation with Unlabeled Examples [Iliopoulos, Kontonis, Baykal, Trinh, Menghani, Vee]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)