Enriching Online Knowledge Distillation with Specialist Ensemble
Authors: Mincheol Park
Woojeong Kim
Junsik Bang
Won Woo Ro
Suhyun Kim
Venue: Submitted to ICLR 2023 (2023)
Keywords: Online knowledge distillation
Label prior shift
Ensemble learning

Abstract: Online Knowledge Distillation (KD) has an advantage over traditional KD works in that it removes the necessity for a pre-trained teacher. Indeed, an ensemble of small teachers has become typical guidance for a student's learning trajectory. Previous works emphasized diversity to create helpful ensemble knowledge and further argued that the size of diversity should be significant to prevent homogenization. This paper proposes a well-founded online KD framework with naturally derived specialists. In supervised learning, the parameters of a classifier are optimized by stochastic gradient descent based on a training dataset distribution. If the training dataset is shifted, the optimal point and corresponding parameters change accordingly, which is natural and explicit.
We first introduce a label prior shift to induce evident diversity among the same teachers, which assigns a skewed label distribution to each teacher and simultaneously specializes them through importance sampling. Compared to previous works, our specialization achieves the highest level of diversity and maintains it throughout training. Second, we propose a new aggregation that uses post-compensation in specialist outputs and conventional model averaging. The aggregation empirically exhibits the advantage of ensemble calibration even if applied to previous diversity-eliciting methods. Finally, through extensive experiments, we demonstrate the efficacy of our framework on top-1 error rate, negative log-likelihood, and notably expected calibration error.

Under review as a conference paper at ICLR 2023
ENRICHING ONLINE KNOWLEDGE DISTILLATION WITH SPECIALIST ENSEMBLE
Anonymous authors Paper under double-blind review
ABSTRACT
Online Knowledge Distillation (KD) has an advantage over traditional KD works in that it removes the necessity for pre-trained teacher. Indeed, an ensemble of small teachers has become typical guidance for student's learning trajectory. Previous works emphasized diversity to create helpful ensemble knowledge and further argued that the size of diversity should be significant to prevent homogenization. This paper proposes well-founded online KD framework with naturally derived specialists. In supervised learning, the parameters of classifier optimized by stochastic gradient descent based on a training dataset distribution. If the training dataset is shifted, the optimal point and corresponding parameters change accordingly, which is natural and explicit. We first introduce a label prior shift induce evident diversity among the same teachers, which assigns skewed label distribution to each teacher and simultaneously specializes them through importance sampling. Compared previous works, our specialization achieves the highest level of diversity and maintains it throughout training. Second, we propose a new aggregation that uses cost-compensttion in specialist outputs and conventional model averaging. The aggregation empirically exhibits the advantage of ensemble calibration even if applied to previous diversity-eliciting methods. Finally, through extensive experiments, we demonstrate the efficacy of our framework top-1 error rate, negative log-likelihood, and notably expected calibration error.
INTRODUCTION
Knowledge Distillation (KD) has achieved remarkable success in model compression literature (Heo) et al. 2019; Park et al.) 2019 Tung & Mori
2019). KD traditionally employs a two-stage learning paradigm: training a large static model as a "teacher" and training a compact "student" model with
the teacher's 's guidance. Online KD (He et al. 2016 Song & Chai 2018 lan et al. 2018) emerged as a variant of KD, which simplifies the conventional two-stage pipeline by training all teachers and a student simultaneously. Previous works used limited number of small teachers and treated them as auxiliary peers that help a student learn. Especially, ensembling these teachers has become a typical direction to make knowledge guidance for the student.
core question in online KD is how to make teachers diverse for the ensemble, |Breiman (996) argues that traditional Bagging-style ensembles usually benefit from diverse and dissimilar models. Recent online KD studies (Chen et al. 2020 Li et al. [2000] Wu & Gong (021) support this claim and emphasize the importance of large diversity prevent homogenization. In supervised learning, the parameters of a classifier are optimized by stochastic gradient descent based on training data distribution. If the training dataset is shifted, the optimal point and corresponding parameters change accordingly, which is natural and explicit. That is, diversifying training data distribution sheds light on effectively generating diverse classifiers resorting to different features.
In this paper, we use label prior shift, where each teacher is assigned unique and non-uniform label distribution. This approach partially aligns with the specialization process in Mixture of Experts (MoE) literature, in which multiple experts with different problem spaces learn only the local landscape (Baldacchino et al., 2016). The most straightforward and prevalent approach to dealing with label imbalance to operate the shifted dataset itself (Japkowicz & Stephen 2002 Buda et al., 2018). However, online KD may have an inconvenient design that could necessitate sampling as much as the number of teachers because typical framework has shared
Chawla Buda

---

Under review as a conference paper at ICLR 2023
layers in multi-head architecture. As an alternative way, consider adjusting a cross-entropy loss of each teacher rather than recursive sampling. Therefore, we efficiently estimate the loss functions using importance sampling drawn from the usual uniform label distribution instead of directly multi-sampling from the truly shifted distributions. Our specialization exhibits the highest level of diversity and maintains it throughout the training compared prior works.
Furthermore, we propose a new ensemble strategy for aggregating specialist teacher outputs. From a perspective of Bayesian inference, can be interpreted that the conditional distributions of specialists become likewise distorted when a classifier learns the label-imbalance training dataset. Therefore, we need correct the distortion of conditional distributions before the aggregating process. We first use PC-Softmax (Hong et al. (021] to cospensaee Softmax outputs. Post-compensation adapts the shifted label priors according to the true label prior by manually adjusting teacher logits. It relaxes the disparity in negative log-likelihoods (Ren et al. (2200) for the same label. As result, PC-Softmax matches the uniform label distribution by modifying the teacher prediction trained by unique cross-entropy loss. Second, we apply standard model averaging method (Li et al. 2021) to all the PC-Softmax outputs. We empirically show that our aggregation policy, denoted "specialist ensemble," improves ensemble calibration even when applied to previous diversity-ellciting methods.
Our main contributions are summarized as follows:
(1) The proposed online knowledge distillation promotes diversifying teachers to be specialists through the label prior shift and importance sampling. As result, our diversity is at the highest level over previous works and maintained throughout training
(2) Our specialist ensemble, based on PC-Sofimax and averaging those probabilities, is beneficial in ensemble calibration. Moreover, this advantage is valid even when applied to previous diversity-eliciting methods.
(3) Through extensive experiments, we deseribe that a student distilled by our specialist ensemble outperforms previous works in top-1 error rate, negative log- likelihood, and notably expected calibration error.
2 RELATED WORK
Label prior shift. The label prior shift has been extensively discussed due to various degrees of imbalance in training (source) label prior (v( and test (target) label prior (v)).. Especially in most works closely related to ours, Post-Compensating (PC) strategy is typically chosen as the proper adjustment to estimate new conditional probability p(y) (r) approximated by pv(y) for given pely). When estimating a Softmax regression, [Ren et al. (2020) corrects the model outputs by the amounts of each class, assuming the uniform target distribution during training time. Many strategies for matching two priors at test time were investigated by rebalancing a different form of multiplying
(/)pp/p(//) to the output probability from a Bayesian perspective (Buda et al.) 2018]] Hong et al. 2021]] Margineantul 2000 TTin et al.] 2020). Here, Hong et al. (222)) carefully reconstruct each conditional probability that should satisfy a condition >. (y = c = 1. It is known as PC-Softmax. We use PC-Softmax of each teacher network to adapt entirely different label priors according to the student label prior.
Ensemble learning. Promoting diversity in traditional ensemble learning has been emphasized because the number of models, which acts as a factor of ensemble impact, becomes more crucial when they are gradually uncorrelated. (Breiman 1996; Ghojogh & Crowley| 2019). Lakshminarayan et al. (2017) used only different random initialization and weighted averaging on the same models. The models, as a result, can have similar error rates but converge to different local minima (Wen et al. 2020). Bringing in multiple models, however, requires prohibitively large computational resources, which frequently limits the ensemble's 's applicability. Thus, recent studies have found efficiencies in two approaches: sampling multiple learning trajectories with only a single model (Huang et al.) 2017a] Laine & Aila. 2017 Tarvainen & Valpola 2017) and building new structure architecturally efficient (Wen et al.) 2020 Li et al. 2021). Our ensemble overview can be aligned with the latter by modeling shared parameters and purposive heads to be diversified.
Online knowledge distillation. Online knowledge distillation works belong to two categories: network-based and peer-based. Network-based methods (Guo al. 2020 Zhang al. 2018) train
2

---

Under review as a conference paper at ICLR 2023
Teachers
Train Dataset
K:09
CR:
K200
CR2
(x, (xxx)))-(x.))
L2
Kg00
CR3
4
CR4
L4
"Samoyed" Class: 37
Shared Part
Student
Specialist Ensemble
LD
Lx
Figure I: Overview our online knowledge distillation framework with four teachers. Each teacher assigned different label prior by Class Reweighting (CR) function described in Section 3.3 Each teacher loss and a student loss is defined in Section 3.4 and Section [3.6] respectively. For knowledge distillation, specialist ensemble is obtained as described in Section 3.5
separate networks with identical architecture, employing mutual learning paradigm; every network interacts and provides knowledge guidance its cohorts. Sometimes, these allow independent data pre-processing per network. However, in peer-based methods (Song & Chai 2018 lan et al. 2018 Wu & Gong 2021 Li al. 2020), some parameters are shared among peer heads, which concurently benefit from computational efficiencies and generalization during training. Earlier works exploit surrogate modules to enlarge varying features or introduce each head unfairly for peer diversity. However, following trained parameters, the modules may ignore some peers useless, which prevents achieving the preferred diversity (Mullapudi et al. 2018). Chen et (2020); Kim et al.) (2021) can be applied to both network- and peer-based approaches, but they require extra modules as well. Our method is peer-based and shares two similarities with some previous works: first, designating "student" and "teachers" in advance of the training to equip one way guidance (Chen jtt al. 2020 Li et al., 2020), and second, creating dedicated dataset to train each peer (Feng et al. 2021) However, our specialty dataset differs from an arbitrary sampled subset (Feng et al. 2021) in that it purposely class-skewed.
3 METHODOLOGY
As shown in Figure I our model consists of three parts: shared part, multiple teacher heads and the student head. That we parameterize θ (0+) [0, N, < T) (0.) where T denotes the number of teachers. {0₂) represents the shared parameters; {0.} and {0.) represent the teacher and student head parameters, respectively. For simplicity, we will use the same notation for each teacher and student model including shared parameters, i.c., {0.} = {0+} U {0,} and
(0.) {0+} (0.). Note that each { and (0.) have the same dimension.
3.1 LABEL. PRIOR SHIFT
Prior works (Buda al. 2018 Hong et al.) 2021 Margincantu, 2000 Tian al. 2020) addressed the discrepancy between train and test class distributions due to the inherent difficulty in obtaining samples from certain classes and dealt with such class imbalance with post-scaling prior distributions. In our work, we adopt a similar method to manually shift the label prior distribution of teachers for the student learn from diverse teachers.
We want the model output p(y x; 0) to approximate true posterior distribution p(y|æ). which is conditional distribution of the labels y given input samples From the perspective of Bayesian
3

---

Under review as a conference paper at ICLR 2023
inference, true posterior distribution is defined follows:
p(ul=) p(xlu)p(v)
(1)
p(a)
where ply) represents a label prior distribution. We assume p(y) is a discrete uniform distribution y U(1/K) where K' is the number of classes since the datasets
using contain the uniform number of training samples for each class. In our setting, label prior shift refers to the label distribution distinction between the teachers and the student, i.c., Pr(y) * (y)). While px(y) p(y), we purposely make glaas-imbalancee settings by shifting label distributions of teachers. Also, ps Px(
differs from each teacher, i.e., (y(y) o Py (y), motivating each teacher be a diverse discriminative
classifier. We will discuss how to manipulate teacher label distributions in Section 3.3
3.2 IMPORTANCE SAMPLING
Under our class-imbalance setting, naive Monte-Carlo sampling unlikely to effectively approximate the target distribution. Therefore, we exploit importance sampling, which allows us to effectively approximate the target distribution only with the samples generated from a distribution we have.
We will denote the shifted label prior pt (v) as q(y) only in this section to avoid confusion with p(y). Theorem 1. Let q(x, y) and p(x.y) y) be joint probability distributions, and h(x,y) y) be a dif- ferentiable function with respect to x and y- Assuming q(zlv) p(rly), we can estimate (x( J = ((2,))~<(x..)) [h(x,, y)l as follows:
(1))((1,)) 3)
N
Aliihhaz wi) p(ya)
(d.,.)) ) ~ i..d p(a,
(2)
p(y)
N
During training, sample (ze, y:) is drawn from a joint distribution p(r, y) = p(zly)p(y)) However, what we actually aim to sample from the joint distribution with shifted prior q(V), q(x, y) = q(x ((xll))(()) Theorem 1 shows we can effectively estimate the expectation of the function 1( v) where (I. ~ q(2 y) with the samples drawn from p(r,y)
Often the target label distribution ( is intractable and we only know unnormalized ((y) (g(v) with unknown normalization constant, Z. We make use of an unnormalized distribution, 7(y), by our design choice.
Corollary 1.1. Let q(x. y) 7q(x, v) be an unnormalized distribution with unknown constant Z > O. Then, the unnormalized importance sampling estimator of μ. is follows:
N
k(y: k(y:)h(ra.vi) (I1, v.) i.i.d. p(z, y),
where k(v) g(v)/p(v) is the unnormalized reweighting function of y.
As shown in Corollary I.1, we can obtain the estimator of the data loss h(x y) under label prior shift, once k(y) is given. In the following section, we will discuss how to formulate the function k(v).
3.3 CLASS REWEIGHTING FUNCTION
Our goal is specialize each teacher in a specific subset of labels. To achieve this, each teacher is assigned "specialty" labels. We denote this "specialty" label set for i-th teacher y" y's can overlap, but all labels are assigned at least once and then for the same number of times, i.e., ) y .... ........ y."" where ) is the total label set. Although we sequentially allocate the label to teachers (see Appendix(B). clustering (Mullapudi et al. (0188) or human-crafted grouping (Krizhevsky 2009) can be also considered.
Now we introduce a Class Reweighting (CR) function, (x))) to assign high weights to "specialty" labels ) This function indicates how much the f-th teacher considers the label y against p(y). As mentioned before, p(y) is a uniform probability distribution. However, as Pr(y) is an unnormalized (distribution), we consider it as a function of y.
p.(v) [1,
if E yt [e, otherwise,
Ke (v)
05e51.
(4)
p(y)
4

---

Under review as a conference paper at ICLR 2023
We name exposure. If exposure is I, there is no label prior shift, pe(y) is also uniform distribution equal to the student label prior.
3.4 TEACHER Loss
Neural networks typically produce class probabilities by using "Softmax" output layer that converts the logit computed for each class into a probability by comparing it with the other logits. Let = denote the logit of class & given an i-th input sample (z,, ys) produced by the t-th teacher model. Then, the output conditional probability of Softmax layer is as follows:
exp (=)(vs))
Pe 0.)
EST.
(5)
Σ exp (=]():))
Using the unnormalized importance sampling estimator in Eq. 3 teacher loss for random samples {(rs, (:))::)) where is the number samples is defined as follows.
Fir (y:) log(pe 0+))
(e.)
t T.
(6)
Σ (e(e))
3.5 SPECIALIST ENSEMBLE FOR KNOWLEDGE DISTILLATION
Following the model averaging paradigm (Li et al. 2021 aggregate teachers' predictions to define
a guide signal of knowledge distillation loss. The original predictions, however, have shortcomings. 1-th teacher conditional distribution (y]:;; 0.) is closely related to each prior (y));; since supervision signals for minority classes are unlikely to occur, teachers may fail to introduce correct predictions on uniform p(y). In order to adapt according p(y), thus relax the minority classes' likelihood by manually adjusting logit values (Ren et al. 2020). We introduce further studies in Appendix
Adapting label prior. We adjust teacher output logits adapt shifted teacher priors Pe(y) to
uniform p(y). Following discussions of Appendix we employ PC-Softmax (Hong et al. 2021 1) to post-compensate teacher logits. Given the original teacher logits 2 and CR function Ki ( in Eq. post-compensated logits (PC-Logits) and conditional probabilities of the i-th sample produced by the t-th teacher are defined as follows:
2 [vl 2:(u:] log ( ( Pe bx; 0.)
exp (2)
ts
(7)
ΣK exp (2)(k))
where (3.) is pa (py:)) well defined by Section 3.3 Thus, if (<( is 1, the corresponding
class's logit is the same as the student's, but if ke (1)) ) c, its logit is compensated as much as log(1/c). Note that each head is trained with Eq. 5 and Eq. 7is used only to compensate for the label distribution shift before making ensemble prediction.
Model averaging. We now aggregate teacher predictions to form an ensemble prediction. Our fusion based on an averaged classifier manner (French al. 2018 Garipov et al./ 2018), com- monly used in statistical learning paradigm. Given conditional probabilities for an i-th sample, Pr (1 lxy f....... (r lx; Or). obtained from Eq
7
the aggregation is defined follows:
T
(e 01 O1.... 0r)
Σ 01)))l
p(0)) U(ITT).
Each 0, is uniformly chosen. We also employ logarithm for the aggregated probabilities (Stanton et al. 2021) to derive a class ensemble logit = [3. = log(p= (1nl( 0,,.., 0r)). In Section 4.3 we will compare this method to a convention of aggregating naive logits and show that our method has advantages in calibration.
3.6 STUDENT Loss AND DISTILLATION STEPS
Given an ensemble logit and a student logit, here we define the cross-entropy and a knowledge distillation (Hinton et al. 2015) loss update the student parameters (0.). A temperature is used
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Student Distilling Steps
Input: Training set D; (T+l)-head model parameterized O; mini-batch size N; learning rate
Output: A student model parameterized 0, oonneryee
1: Randomly initialize è
Set CR functions (y) with a choice of define each (.)))
3: while 0, not converged do
4: e. e-6- e
b Update epoch to adjust ramp-up X(e) of Eq]
13
sample mini batch ((x,, y:)3
6:
Compute the entire (())) and L. concurrently
> Use Eq 6 and Eq 13
7:
0, - 0: (..)
8:
+ 0 Vo.
end for
10: end while
soften probability distribution over classes, Same as Section 3.4 student loss is also defined by the same random samples.
exp (V)(vi
(u(iil, ...... 07) exp Wil/T
pa( 0.)
(9)
Σx= exp [2]
Σ exp (z=(k)/T)
For normal cross entropy loss, LCE, temperature is set 1. Knowledge distillation loss, LKD, is KL-divergence between the student and teacher ensemble posterior distributions.
LcF(O.) =
loglp (.) R0
log(p. b. 0.)).
(10)
LKD(O,) = (a (Ppeplpp)))
(11)
Pe Pe ( O 01...., OT)
(kila; ,..... 07) log
(12)
Pa (k; rz; 0.)
The final student loss is a weighted sum of Loe and LKD- We adjust À using a Gaussian ramp-up curve, which is X(e) exp( (5(1 e/a)²), where is an epoch and is the ramp-up period (Laine Aila 2017).
(.( = Lce + (())
(13)
Alg. T introdduess student distilling steps. All parameters are updated during training, and only the student, {0, ), is used at test time. Thus, our framework does not induce additional test-time costs.
EXPERIMENTS
In this section, conduct three experiments to assess the efficacy of the proposed method. First, we evaluate how well our student model generalized an image classification compared to previous methods with three measurements (Stanton et al. 202I): Top-1 error rate (ERR), expected calibration error (ECE), and negative log-likelihood (NLL). Second, we perform an ablation study on the exposure of the proposed CR function k(y) and the number of teachers T. For the analysis, include two metrics (Stanton et al. 2222]) to measure a student's fidelity on an ensemble's outputs: averaged top-1 agreement and averaged KL KL-divergence. We further show diversity change following the variation of ramp-up period in Appendix F.2 Finally, we empirically analyze why our student model has become calibrated. The evaluation settings are thoroughly summarized in Appendix E.
4.1 IMAGE CLASSIFICATION PERFORMANCE
We compare method to extensive online KD methods: CLILR (Sog & Chail 2018). ONE (lan et al. 2018), FFL-S (im et al. 202T) and OKDDip (Chen et al. (020) for CIFAR, as well as DML (Zhang et al. 2018), KDCL (Guo et al. 2020) and PCL (Wu & Gong 2021 | for ImageNet. Denoted "Vanilla" 1s to train target model from scratch without knowledge distillation loss. While CLILR, ONE, and FFL-S select the first network student after the whole training procedure,

---

Under review as a conference paper at ICLR 2023
Table 1: The generalization comparison with previous peer-based methods on the student model. ERR and ECE use a percentage (%), and NLL is loss value. Thus, the lower it is, the better. The numbers are the test results of three random experiments and filled in the mtdn(+std). The best result within each type is indicated in bold.
Donsett
Mothod
F30
ResNer-32 RCE:
NL.1
FRL
ResNe: BCE
NLLL
FIOR
DesseNer-40113
BCF
NL.1
FIO
EfBoceneeB0
BCE
NL.1
FRE
MobilcNeeV2
BCE
NIL1
Vasila
6.90
CLIIR
(20.18)
5.65
(50005)
3:38
(20.00)
0.33
(20.10)
4.50
(50.5)
2:55
(=0.00)
0.18
(20.05)
7,05
(50.22)
2:90
(=0.0)
0.34
(20.12)
7,45
(50.12
(20.01
0.75
(:0.27)
11:75
(50.34)
2.56
(=0000)
0.33
ONE
(50.13)
5.75
(20.13)
1:11
(20000
0.23
(20.13)
4.7%
(50.5))
2.11
(20.00
0.19
(1)
6.64
(20.24
1166
(2000)3
0.75
(24029)
7.47
(50.59)
5.70
(=0.02)
0.78
(31.33)
11:94
(40.17)
2:19
(=000))
0.36
C3BAR-10
5.04
6.90
0.34
OK/Mmp
(10.06)
5.76
(50.18) (8)
(30.0
0.23
(10.05)
4.58
(10.07)
(50.00
0.18
(10.15)
6.57
(50.12)
(5000
(20.17)
7.55
(10.23)
(5001)
E20.053
11.49
(10.35)
2.40
(50000)
0.35
PCL
(40.14)
5.12
(10.4))
3.76
(2010)
0.25
(40.25)
4.77
(10.11
3.42
(-0000
0.23
(40
5.54
(10.00
3.1
0.25
7.32
610 139
381
(2000))
0.5
11:35
(40.39)
3.06
0.35
561
0.23
4,49
0.17
6.78
0.24
7.08
0.24
11.77
0.34
Vasilla
(40-21)
28.30
(400220
13.04
1:21
(4022)
24.34
(40.37)
11.18
11011
1.27
(4
25.55
0106
( 18
1.25
23.41
(40.28)
13:22
(2000))
L.17
S3H
(50.15)
(=(UC)
1.34
20:45
21.49
28.51
1:04
27.70
3321
ONB
(40.20)
20.19
(50.32)
220.00
0.54
(50225)
21.58
(50.22)
799
(=0.00)
0.36
(50.45)
24 20.0
(5041)
(=0002
105
(50.35)
27-22
(50.23)
(20022
1.15
(:0.15)
3303
(10.55)
4.21
(5000)
1.17
CIFAR-100
FFL-8
(10.0))
26.19
(140780)
10:00
2-0000
1.85
(10.1)
22.1
(1080)
9:5
(-0.04)
0.02
[40
28.05
(1030
10.15
1.13
0.00
(10.11)
27
(10.4)
1041
(-0000
1.15
(10.57
53-97
(10.5%
5.14
(-010)
1.21
OKDDP
25:08
21.34
0.92
29
1.04
28,17
13:33
116
32.55
1,15
PC1
(40025)
26.78
(10.55)
10.56
(:0.00
1,0
(1022)
2100
(10.1)
10.81
(50.01
0.92
(10.4)
29.00
(10.7)
(:000)
L15
(30.25)
27.59
(10.3)
(2004
115
(20.4)
3484
(10,71)
13.46
(2002)
1.31
Ouss
(.4039)
25.45
650.00
5.30
[:000)
0.33
(.4.11)
20.44
(:433)
6.6
[:00
0.50
(34,31)
28.33
(1800))
521
(-000)
192
27.56
(14
0.75
(:000)
L15
32.11
(14,88)
301
(=000
LB3
Table 2: Top-1 ERR (%) comparison with previous methods on ImageNet validation set. The results
of R8 and ResNet-34 are each reported from Wu & Gong (202T) and Chen et al. (2000)
:
Note also ours has T = 2 and T = 3 on each model for fair comparison. We filled in mean( std)
through three random experiments on our validation results.
Medel
Vanilla
DMF
CLILR
ONE
FFL-8
ORDPip
KDCI.
PCL.
Ours
ResSet-18
30.4%:00114)
30.18/=0.08) 26.03
29.961=0.055
09 25.02
31.15120000
30.07
30.401:0.05)
29.50(20.13)
0..53
Re-Nei-34
26.76
26.06
25.53
25.60
25.45( -1.09)
OKDDip and ours designate a student and other peers (teachers) in advance. Thus, OKDDip and our approach use one less peer than the others. All methods have total four heads. We evaluate these methods on various deep neural networks (DNNs), i.e. ResNet- -2 and ResNet-1100 (He et al. 2016 DenseNet-40-11 12 (Huang et al., 20176), EfficientNetBO (Tan & Le 2019). MobileNet (Sandler jet al. 2018) on CIFAR as well as ResNet-18 and ResNet-34 (He et al. 2016) for ImageNet. For
building strategies of the peer-based architectures, we describe details in Appendix E.2
Results on CIFAR datasets. Table [ demonstrates that our method consistently outperforms previous methods to generalize a student. For all DNN models, our ERR and NLL are marginally better. However, our proposed method produces significantly calibrated student in ECE than in previous works and Vanilla; the gains improve as the class size increases. Section 4.3 will discuss ECE further. We provide an additional comparison with network-based methods in Appendix F.I]
Results on ImageNet datasets. In comparison to all the previous methods, Table 2 shows Top-1 1 ERR. Our proposed method improves 0.96% and 1 1.29% ERR against Vanilla for each ResNet-18 and ResNet-34 34 and still achieves competitive superior among previous methods.
4.2 ABLATION STUDY
We examine the effectiveness of exposure variation (e) and the number of teacher heads (T') As shown in Figure 2 we analyze results for teacher diversity, student generalization, and student fidelity on the ensemble posterior distribution. We utilize averaged pairwise
divergence (Appendix()) to measure diversity between given two distributions. We use ResNet- 1 10 trained on CIFAR- 100 when T = [2, 3, 4] and = [0.1, 0.3, 0.5, 0.5,0.8, 1.0]; Models with more than five peers are not practical due to computational efficiency and saturation performance (Stanton et al. (021). The range of is for our grid search. PC-Softmax is equally processed to evaluate the ensemble outcomes since p(y) is the same in the training and testing data distributions.
'The model architecture of CIFAR shallower than the plain version of ImageNet (He al. 2016
7

---

Under review as a conference paper at ICLR 2023
75
73
76
74
77
77
70
Tra Erann3
Euurdd
Tel Carrts
8.5
Dcponumecc
[ptarri)
Seposraad
Esposume/d
Exposimeei
Exporsrnnd
Top-1 accuracy (%)
(b) Expected calibration error (ECE) (%)
Suuert
3.40
0.3
3.55
0.35
025
2.50
0.00
0.50
02
2.75
0.75
0.75
T-2 Orerti
3.7
Cepoeumec)
Dpoountel
Desosumec))
Eepasuntel
Expasimie)
Eppaareti
(c) Negative Log- Likelihoodd (NLL)
(d) KL(p- (p.))
(e) Agreement
(1) Diversity
Figure 2: Ensemble and student generalization: top-1 accuracy, expected calibration error (ECE).
and negative log-likelihood (NLL) loss. Fidelity between ensemble and student conditional distribution: averaged KL-divergence and averaged ensemble-studen top-1 agreement. Diversity: averaged bensen-Shannon divergence between the posterior distributions each pair of teachers. The shaded region represents the meen(==tt) for three experiments with varying and T in test time.
0.50
1e-31
BesNeb-12
033
Beslee-110
145 2e33 1ensellet-40-12
2o13
Eflieentieen
2042
MohilekeV2
244
0.2
0.0
164
2.0
L27
143
0.2
132
3.50
0.35
6.00
8:16
3:64
197
F1.
DED0W Dam
FILSS
(ETTi Dam
C.E.
FILAA 2007
002
PLS
0LE1
030
Mathel
5e-11
36
Figure 3: Diversity comparison in various deep neural networks CIFAR-IO (up) and CIFAR-100 (down) with previous methods. For fair comparisons, we use Softmax to normalize teacher logits of the previous methods and PC-Softmax and the student is the best performer in validation time.
teacher logits. Each measure is obtained when T
Exposure variation. As discussed in Section 3 determines teacher diversity. The diversity is exceptionally higher when 0. as shown in Figure 2(1) At this point, the ensemble performs worse than a student, implying that the ensemble usually fails to discover the hidden knowledge in data. As result, the student has high disagreements against the ensemble; this implies that the student may experience significant confusion during distillation. Diversity is lower than 0.1 in the other ranges, decreasing in small amounts from 0.3. The value 0.3, in particular, quite encouraging. An outstanding generalized ensemble presents the potential to merge diverse teachers. As shown figure2(d)an and [(e) the fidelity also superior. The disparity in generalization is thus noticeably small in Figure 2(a) to 2(c) Furthermore, as shown in Figure 3 our diversity size by chosen in
various DNNs presents consistently high compared to earlier methods. In Appendix F. we further visualize how teachers' confidence varies while predicting specific k-class samples.
The number of teacher heads. The fidelity typically improves T increases shown in Fig-
ure 2[d] and 2(e) One possible explanation is that increasing the number ensemble components smooths the logits of unlikely classes, making the distribution easier for the student to match. This phenomenon may provide insight into how to improve overall fidelity. The student thus benefits from top-1 accuracy and NLL. loss, which improves ECE marginally, shown in Figure 2(a to 2(c) However, student generalization becomes increasingly saturated (Stanton et al.) 2021) as increases. Meanwhile, diversity falls slightly because label repetition can render class coverage redundant.
8

---

Under review as a conference paper at ICLR 2023
782
(a) Aggregation
(b) Generalizability in benchmark ensembles
(c) Choice of PC in our aggregation
Figure 4: Performance comparisons in ensembles using logits or probabilities (probs). All ensembles over the benchmarks are obtained when each student performs the best on accuracy at validation time. In particular, (a) probs are based on PC-Softmax for ours and Softmax for others. (c) The shaded region represents the mean( calculated from three trials with varying 6
DIIA
ONE
TTL-S
0100 p
Dun
59%
9.93%
1B
6.67%
kikel
kilel
2.6 0.6
05
Conftence
Confcence
Conficnce
Confience
Figure 5: Reliability diagrams of student model for ResNet-110 on CIFAR-100. The confidence intervals are divided into 20-bins to visualize outcomes. Each output bars and assessments, such as accuracy and ECE, are a mean of three test experiments.
4.3 ON CALIBRATION OF STUDENT MODEL
This section empirically explains why our ensemble usually leads to better student calibration than previous methods. The calibration considers the problem of predicting probability estimates representative of the true correctness likelihood (Guo et al. 2017). KD can regard type of learned label smoothing regularization (Yuan et al. 2020). The label smoothing can also calibrate a network, minimizing the miscalibration rate, i.e. ECE (Müller et al. 2019). Accepting such a KD effect, we conjecture two factors that our ensemble holds to transfer crucial scaling constraints for the student confidence: combining probabilities and diversity.
Combining probabilities. As shown in Figure 4(b) the posterior ensemble distribution with teacher probabilities rather than teacher logits significantly improves ECE and marginal gains with top-1 accuracy and NLL; we further provide ensembled confidence among them in Appendix F.4 Even after replacing the existing ensemble in previous methods with the probability-ban -based and altering our ensemble to the logit-based, using probability still outperforms in ECE, as shown in 4(a) Moreover, as shown in Figure [(c) PC-Softmax outperforms PC-Logit ECE, exhibiting comparable accuracy in varying e. Through three case studies, we hypothesize that probability- basedd ensemble effectively
regularizes student confidence by KD guidance.
Diversity. As shown in Figure 3 our diversity shows higher and more model-agnostic than previous works; previous works have trouble deriving diversity on DenseNet-40-12 12 and EfficientNetBO. It implies that when the number of teachers is constrained, using extra losses may fail to induce a helpful diversity. Apart from the size and robustness of our diversity, acceptable fidelity demonstrates that the diversity is implicitly fine for student to accommodate different signals, as shown in Figure 2(d) and [[e] Therefore, we know that a merged knowledge is made of our useful diverse teachers to a direction suitable to a student, and it exhibits generalized ensemble performance shown The student, as a result, can learn generalized potential knowledge well.
in Figure 2(a) to 2(c)
5 CONCLUSION
We propose enriching online knowledge distillation with the specialist ensemble. Proposed CR functions are equipped to model label prior shifts for large diversity among teachers throughout training. Averaging diverse teacher probabilities provides a significant advantage in ensemble calibration. This paper confirms KD with our ensemble enlarges student generalization: marginal improved ERR and NLL with notable ECE. Figure shows student becomes more predictable classifier than previous methods through reliability diagrams. For further discussions, the limitations and societal impacts are described in Appendices Gnadd H

---

Under review as a conference paper at ICLR 2023
REFERENCES
Tara Baldacchino, Elizabeth Cross, Keith Worden, and Jennifer Rowson. Variational bayesian mixture of experts models and sensitivity analysis for nonlinear dynamical systems. Mechanical Systems and Signal Processing, 06:178-200 2016.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140 1996.
Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks: the official journal of the International Neural Network Society, 106:249-259, 2018.
Nitesh V Chawla. Data mining for imbalanced datasets: An overview. Data mining and knowledge discovery handbook, 2009.
Defang Chen, fian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Shaoxiong Feng, Hongshen Chen, Xuancheng Ren, Zhuoye Ding, Kan Li, and Xu Sun. Collaborative group learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Geoff French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. In International Conference on Learning Representations, ICLR, 2018
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in Neural Information Processing Systems (NeurIPS), 2018.
Benyamin Ghojogh and Mark Crowley. The theory behind overfitting, cross validation, regularization, bagging, and boosting: tutorial. arXiv preprint arXiv:1905.12777 2019.
2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration modern neural networks. In International Conference Machine Learning, 2017.
Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. IEEE International Conference on Computer Vision (ICCV), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim. and Buru Chang. Disentangling label distribution for long-tailed visual recognition. Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. Snap- shot ensembles: Train 1, get m for free. In International Conference on Learning Representations, ICI.R, 2017a.
10

---

Under review as a conference paper at ICLR 2023
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition (CVPR), 2017b.
Nathalie Japkowicz and Shaju Stephen. The class imbalance problem: A systematic study. Intelligent data analysis, 2002.
Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. In Advances in Neural Information Processing Systems (NeurIPS), 2008.
Jangho Kim, Minsung Hyun, Inseop Chung, and Nojun Kwak. Feature fusion for online mutual knowledge distillation. In 2020 25th International Conference on Pattern Recognition (ICPR), 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations, ICLR, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Xu lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble, In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Xingjian Li, Haoyi Xiong, Zeyu Chen, Jun Huan, Cheng-Zhong Xu, and Dejing Dou. "in-network ensemble": Deep ensemble learning with diversified knowledge distillation. ACM Transactions on Intelligent Systems and Technology (TIST), 2021.
Zheng Li, Ying Huang, Defang Chen, Tianren Luo, Ning Cai, and Zhigeng Pan. Online knowledge distillation via multi-branch diversity enhancement. In Proceedings of the Asian Conference on Computer Vision, 2020.
Dragos Margineantu. When does imbalanced data require more than cost-sensitive learning. In Proceedings of the AAAI'2000 Workshop on Learning from Imbalanced Data Sets, 2000.
Ravi Teja Mullapudi, William R. Mark, Noam Shazeer, and Kayvon Fatahalian. Hydranets: Special- ized dynamic architectures for efficient inference. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Rafael Müller, Simon Korbliihh, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Pro- ceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Jiawei Ren, Cunjun Yu, shunan sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018
Guocong Song and Wei Chai. Collaborative learning for deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? In Advances in Neural Information Processing Systems (NeurIPS), 2021.
11

---

Under review as a conference paper at ICLR 2023
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning. 2019.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Junjiao Tian, Yen-Cheng Liu, Nathaniel Glaser, Yen-Chang Hsu, and Zsolt Kira. Posterior re- calibration for imbalanced datasets. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Frederick Tung and Greg Mori. Similarity-preeevinng knowledge distillation. In Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, ICLR, 2020.
Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2020)
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
12

---

Under review as a conference paper at ICLR 2023
A PROOFS
A.1 PROOF OF THEOREM 1
Proof. For discrete random variables and y, each joint probability mass function p(æ: y) and q( y) can be expressed as p(æ, y) = p(x ((p)))p() and (z. y) = (<<l))(()) by product rule. We assume that p(zly) g(x) ly) and only label priors are different ply) + gly). Also, let h(x, y) be differentiable function with respect to and y,.
E q( [b(,,))]
ΣΣ (e, (x,
Σg (y) Σ p(zlu)h(z,n)
Σ P(y) ((y)
Σ p(xly)h(x.1)
p(u)
(14)
((y)
Σ p(v)
Σp(y)
(E (Ez~p(slp) [1 v)))
p(u)
9(v) p(y)
yy-r(y)
E plapla [h(z, y)]l
By the associative law of multiplication and is constant with respect x. then have
(lv)
pxwp(e ( v\ (1(,,,)) v)ll pxpppl)) (1) v)
E
(15)
p(u)
p(y)
(1) v)
(16)
ply)
One can wonder about the case ply y/4 0 for some i, so the denominator becomes zero. In our setting, p(y) is a uniform distribution, thus, all the probabilities are strictly positive.
A.2 PROOF OF COROLLARY 1.1
Proof. Let Zq(x, y) be unnormalized distribution where Z > 0 is an unknown constant, then
ΣΣ g((, = ΣΣ ((x,))=
9
3
ΣΣ Σ (1 3) Z.
For every sample (I;, vs) i.i.d p(a,, y). then = 1 Ei ha,, 9:) is basic Monte-Carlo estimator of μ ((za))p(z [z. y) Σn Ey (z,, y)((1, y). From Eq. 16
N
(1))h(z, y) B(x ((x, (x..))
((v)/(x, v)
/1 = (pt.)))((a.)))
p(u)
Zp(y)
7v (u) y) ΣΣ
p(r. v)
Zp(v)
q(y)k(a, v)
p(x, v)
ply)
(17)
7y Σylp(y)
[>= p( h vill
plu)
Enlg(
g [y)l
Ky E Le(())
p (a
Mly)
(10 E,(p(1)
[(<ly)]]
p())
13

---

Under review as a conference paper at ICLR 2023
Let x(y) we have
d(v) ply) as the unnormalized class reweighting (CR) function of y and ((>ly) p(a (), then
En(p(u) flv p(v)
>> p(xly)h((, will
Ey (P()
70 ply]
((
Eylp(y) Ez p(aly)]]
Bymplc) (M((V) B Bempla ((.,v)))
Bympp(y). ()) [s()]
(18)
B py-ppt))a---aalll (1)
Eywpll) a p( [s(y)]
E ()x.)))
v)
[ () p [K()
Using Monte-Carlo estimation, we can estimate the above expectations:
(( (1).))ootes,///(y)((( (n(y))((,, v)] (x.)))((x.)) [s(y]] [x(y]
EEE H(U:)h(ss,, vi).
N
(Ir. v:) ~ iid. p(x, v)
(19)
N i==1 (()))
Σ 1= K((y (2i,,
(21.91) ~ iid. p(x, v).
(20)
E r((ss)
B FORMULATION OF SPECIALITY LABELS
This section describes how we create "specialty" labels y of each 1-th teacher (we already know t E N, < T where T' denotes the number of teachers). We allow the overlap between Y's as mentioned in the main paper. We introduce a parameter ? E [0, 1] to control the ratio of the classes that a teacher focuses on. Let us denote the first class of y as follows:
K c = T (1 - 1|,,
(21)
where is the total number of classes. Then, we can define the specialty set as:
y {k: ch
- 1}, if c + + - 15
(22)
Sometimes &s can go beyond the last index of the whole class set. In that case, we define the specialty labels yrr as follows:
y" = (k 4 < ks s KJU I k +
1)K- 1}.
otherwise.
(23)
In this paper, 7 = 0.5 is used to let specialty labels overlap at least once. There is no class overlap y is O. It is also worth noting that if K is not multiple of T, some classes may be exposed once less than the others. However, our experiments show that this does not significantly affect ensemble performance.
C WHY ADAPTING TEACHER LABEL PRIOR BEFORE AGGREGATION
C. TEACHER PREDICTION ON LABEL PRIOR SHIFT
In supervised learning, classifier parameterized by 0 tries to sample a correct label y the input æ by directly estimating conditional distribution p(y|ar; 0). In our online multi-head learning, each teacher parameterized 0, learns corresponding true distribution (r( y) = p(xll))p(()) whose label prior is differently class-skewed (y) # Pt (y). In Bayes rule, the empirical inference over parameters 0, given specialty dataset D: = {Z; y:) 1 on each class-imbalanced distribution is as follows:
M
p(0,ID,) x (())) II
p bm; 0,)
(24)
14

---

Under review as a conference paper at ICLR 2023
For unknown data (EM+1.3M+1) the predictive distribution is marginalized posterior distribution (t)::
MUMM+I<MM+1 = 00
(25)
Therefore, each teacher's prediction of given data is likely related to its class-skewed distribution.
C.2 BIASED ACCURACY ON LABEL. PRIOR SHIFT
In this section, we further discuss the class-imbalance causes the label prior shift can result in incorrect accuracy (Tian et al. 2020) when especially teacher's ' label prior has varying degrees of imbalance on uniform (student) label prior pe(y) # p(y). For the simplicity, we assume teacher classifier as on the K-way one-hot classification on D-dimensional inputs.
f(;;; 0.) D + (0, 13 K
N
Nu
Acc(I,
I(f(z;0) k. k)
I(f(x;;)) k. y
N
K
Σ ply k) Agremmentyy = = Eymp(ss) (Ageementt))))))
(26)
As shown in Eq. 26 the accuracy is equal to the expectation of agreement underlying the given label prior. When Pr(y) * p(y), training with imbalanced data maximizes accuracy on Pr(y) where majority classes are likely to observe. On the other hand, the accuracy of uniform data calculates the expectation of agreement on p(y). Therefore, training in (e is prone to bias towards large classes to maximize Eq. and and thus may result in inaccurate evaluation uniform p(y).
C.3 LIKELIHOOD RELAXATION
As shown in Eq. 25 and Eq. 26 training imbalanced implies that a given teacher cannot be accurate in the minority data and allows teacher prediction be closely related to its corresponding labels. Ren et al. (2220) introduce each negative-log likelihood (NLL) error of minority classes that should be adjusted more. They propose a manual relaxation method over the class-wise NLL by posing a discriminative "margin" denoted as where k is a class index. By carefully revisiting Theorem 2 in both Ren et al. (200); Kakade al. 2008), we discuss how we can quantitatively set the margin 7/k-
Suppose ę 2 0 is any threshold and L. (0.) is the standard NLL in Softmax regression of our teachers on the class-imbalanced dataset. Denoting Ω4 a subset of k-class, let errx (C) zero-one loss from empirical samples in k-class subset: errx(E) (r (0,) > g) In addition, we define err,, (EE) is the zero-one y--magiin loss from empirical samples in k-class subset: ert,,*(E)
(r (0.) + 7 > E
Theorem 2. (Ren et al. 2020) Assume that C, Lipschitz continuous and (x.().)) (c,(,))
< where 12 is an entire dataset. For any 8 with probability at least 8 over the samples, Vyk 0 and vʃ F in Theorem 2 Kakade et al. 20088, neglecting empirical noise, we have
4R
log(logz 40)
log(1/8)
erTi(E) S (E)
(27)
7kk
7ik
2n1
where RFF the Rademacher complexity of function family IF (Kakade et al. 2008) and my is the sample size of k-class subset. By discussion in Ren et al. (2020), we can have the relaxed generalization error bound err uniform (E) for the loss of uniformly class-distributed dataset.
1
K
((F) n2
log(log2
4C 7
(1g(1)))
erTuniform (E)
Σ
err (E)
(28)
K
7k
n4k
2n
where I can be measured as complexity of F, following Thereom 3 of Kakade al. (2008). To minimize the uniform error bound in Eq. [8 according to ny. we should minimize the second term because the first term is natural data loss and the other terms are negligible low-order losses.
15

---

Under review as a conference paper at ICLR 2023
With an equality constraint of Ek=1 7 p. solve the minimization problem of the second
term by applying Cauchy-Setie Schwarz inequality to get each optimal k-class margin %/2-
T(F)
K Σ
min
subject
7h p.
(29)
7ke
n
Proof. Given minimization problem can be written
I(F)
min
Tk
(30)
7k
1kk
By Cauchy-Schwarz inequality,
2
2
T(F) T1k
4
((F)
K
I(F)
7
7
(31)
T
Both sides are equal if and only if Mi and multiplier C² for ease of calculation. Then, we have
4
T(F)
are linearly dependent. Thus, we choose
4
I(F).
I(F).
T(F)
1/4
7h 5²
2h = 45
Tke
2
(32)
Th
71g
71k
11k
Substitute 7/ of Eq. 32 with those of the equality constraint in Eq. 29
((F)
1/1
((F)
1/4
p
Σ
2
p
2
(33)
12k
k=
k=
k=1
p
7kk
P
(34)
1/4
1/4
1/4
2 Σ K
[(F)
[(F)
2 K
[(F)
k=1
T1A
1/4
1/1
2p
[(F)
(F)(F)
1
7
1/4
1/4
(35)
Σ
T(F)
2T Σk=1
K
k=1
n
Finally, the optimal margin of the kc-laas subset, Nh is follows:
pm
7k
2k=1 12 11/1
(36)
As result of Eq. 36 " implies that independent margins are necessary according to ng. Thus, mi- nority classes sometimes require larger margins to be generalized. To make a uniform generalization error against each teacher prediction, denoted as Eq. 25 each teacher necessitates manually relaxing k-class NLL loss by adjusting Softmax outputs.
Corollary 2.1 of Ren of wisss-wise NLL loss and the given optimal margin. The straightforward derivation results in compensating method for the k-class logit value.
(2020) introduces that we can get the desired NLL loss from a sum
conditional distribution by adapted logit values on each t-th teacher be P. (2) 0,). Then, we can define
( = k1,,0,)) 0.) = exp(z log Y²,x)
n,, exp(2,1k)
(37)
- exp(2, [M] log y..))
Σ n E exp(2xlA(])
where V2,x and ni,k denote each optimal k-class margin and size of t-th teacher and each k-class logit value [k] is compensated as much log v However, Ren et al. (2020) suggests that since
16

---

Under review as a conference paper at ICLR 2023
Eq. 28 is not tight, a power of 1/4 for ny becomes not powerful condition than using 1. Therefore,
we can redefine
De.k exp(=; (A]])
De. exp(2((k))
Pr k/ 0,)
== ne, exp(== (k.)))
k
Σ kk=1 (xp(=
na.
(38)
pcly exp (2) [R])
Σ Pr exp((2=(k))))
where Pi = k) denotes glass-imbalancee probability of t-th teacher and is the total number of dataset Q2 Using ply = k), which is a uniform probability of k-class, then, we can have
po(y = [x](
(.(ym))
p(y=k) exp(ze(M))
p. k|2;; 0.)
Σ K k= (v = k²) exp(ze(k<])
ΣE =1=1
K
(e(x=)))
exp (=p(=1l1))) (kT])
p(y
exp(2:[<] log( (e(y=k))
exp( [k| log( kk pe ((()=))
p(y=k)
=1 exp( 26 | + log( Pe ()))))))))
K Σ k== exp(2 [K] - loog( )(
ii
exp(2 log( (y=k)
M-1 exp(=:[k<] log( (((
K
(39)
where ky k) our proposed CR function over t-th teacher. Eq. 39 is, as result, the same as Eq. 7 PC-Softmax (Hong et al. 2021). Thus, we little adjust each k-class logit value z [k] as much [og(L/e) in this paper.
D DIVERSITY: AVERAGED PAIRWISE FENSEN-SHANNEL DIVERGENCE
We measure the diversity of teacher outputs based on Jensen-Shannon Divergence (JSD), which assesses how similar two distributions are. The two distributions are mutually informative if the diversity zero. Given the i-th sample, we formulate the diversity among T teachers as follows:
N
Diversity =
Σ
Divg
(40)
N
Div: T(T - (E[L.T] PELLTT
Σ E 2
Σ
1
[KLL + DKL(Phlla))
(41)
where P the output probability distribution of 1-th teacher and = (pt + p/2/2. For our proposed method, probability distributions after post-compensation are used.
E EXPERIMENTAL SETTINGS
E. EXPERIMENTAL CONFIGURATIONS
Datasets. We compare our proposed method to previous online KD works using three datasets.
CIFAR-10 10 and CIFAR-100 100 datasets (Krizhevsky 2009) each have 50K training images and 10K test images, with each image belonging to one of 10 or 100 classes. ImageNet (Deng et al.) 2009) contains 1.2M training and 50K validation images in IK classes.
Training settings. For CIFAR datasets, we train all models for 300 epochs. We use SGD with momentum of 0.9. The learning rate begins at 0.1 and decreases by one-tenth every 150 and 225 epochs. We employ standard data augmentation strategy from He et al. (2016) and normalize all images by each channel mean and standard deviation. The batch size is set to 128, and the weight decay to x 10-1 The ramp-up up period of balancing factor X(e) for student knowledge distillation loss is 80, where e is an epoch. During the first 80 epochs, A(e) varies from 0 to 1. We perform a grid search to find each model's optimal exposure We find best among (0.1, 0.3. 0.5, 0.8, 1.0]
17

---

Under review as a conference paper at ICLR 2023
by measuring ERR, and choose 0.5 for ResNet-32. 0.3 for ResNett 110, 0.5 for DenseNet-40- 12, 0.3 for EfficientNetBO,, and 0.8 for MobileNetV2. We choose different exposure for models as they differ in deep network architecture and the ratio of peer/shared parameters, as shown in Section E.2 E.2 All parameters are initialized with MSRA initialization (He et al. 2013). To compare our method with previous works, use the officially released implementation code for the works and three evaluation metrics on each method is fairly measured with the training settings above. While we use for knowledge distillation temperature, DML uses T 1, and CLILR uses 2: those values are reported in the original paper.
We train all ImageNet models for 90 epochs. The learning rate begins at 0.1 and decreases by one-tenth at the 30 and 60 epochs. The mini-batche size is set to 256, and the weight decay to x 10 10-1. balancing factor X(e) has a ramp-up period a of 20 where distillation, we set т 3. For all models, the exposure of 0.7 is used. We also used MSRA initialization for ImageNet.
an epoch. For our knowledge
E.2 ARCHITECTURAL CONFIGURATION OF THE PEER-BASED METHOD
We separate the shared and teacher-specific parts from the start of the last block to build peer- based architecture for various deep models on both CIFAR datasets. For this purpose, we adhere the strategy in Chen et al. (2020). We divide the shared part from the teacher-specific part the beginning of the last two building blocks for all methods on MobileNetV2 and EfficientNe:BO built for more comparisons
models have far more parameters than peer-based models, as shown in T] [ We also follow the separating strategy in Chen et al. (2020) for ResNet on the ImageNet dataset; we split the last two residual blocks to build peer-based architecture.
this paper. As a result, network-
Table 3: The pure parameter ratio of DNNs. Network-bate based Peer-based denotes the parameter ratio of network-based models peer-based models. Peer / Shared denotes the parameter ratio of single peer head to the shared part. The models are in order of Table This ratio excludes additional parameters by extra modules generated in our benchmark works.
Dataset
Params. Ratio
ResNet-32
ResNet- 110
DenseNet-40-12
EfficientNetBO
MobileNetV2
Network-bused / Peer-based
1,22
1.22
3.94
1.22
1.22
CIFAR-10
Peer Shared
3.15
3.17
3.22
3.12
detwork-based / Peer-based
1.22
1.21
3.54
1.21
1.21
CIFAR-100
Peer Shared
3,20
3,25
0.05
3.25
3.33
Diversity disparity among various model architectures. Accepting that DNNs are architecturally distinct, we can empirically analyze through Figure 3 and Table that deriving diversity can be significantly difficull if the amounts of peer-head parameters part. DenseNet-40-12 12, for example, has almost all shared parameters because this architecture uses the teacher-specific part only fully-connected layer (Chen al. 2020). Thus, that only minor individual parameters are included for specialization, implying that diversity is possible (outperforms the previous methods), but specialization remains difficult. Therefore, The diversity of our proposed method on DenseNer-40-122 is lower than that of other models. Furthermore, we investigate why MobileNetV2 and EfficientNetBO have less diversity than ResNet-32. Despite having similar Peer/Shared parameter ratio, the aforementioned structural differences can be caused by an intermediate layer type, e.g., spatial or septtwwise-separable convolution; however, concrete analysis is our future topic.
considerably smaller than the shared
speculate
F SUPPLEMENTARY RESULTS
F.1 COMPARISON WITH NETWORK -AASDD METHODS
To make a fair comparison with DML and a network-based variant of OKDDip. we rebuilt our framework as a network-based one. From results of ECE in Table and Table [ network-
18

---

Under review as a conference paper at ICLR 2023
based online KD shows more effectiveness than peer-based in producing more calibrated student. Regardless of class size, our method outperforms previous works in ResNet-1 10, EfficientNetBO, and MobileNetV2. On CIFAR-10, 10, our method outperforms in ResNet-32 and DenseNet-40-12, 12, but falls short of OKD in ERR on CIFAR-100. However, ours is still better than OKDDip, about 2x in ECE. As result, our student is more accurately confident than OKDDip.
Table 4:
The generalization comparison with previous network-based methods on the student model. ERR and ECE use a percentage (%), and NLL is loss value. Thus, the lower it is, the better. The numbers are the test results of three random experiments and filled in the mean(=std). The best result within each type is indicated in bold.
Dutact
Method
ERK
RaaNa-12
ECE
NLL
ERK
RusNe-TT
ECE
NLL
ERK
BCE
NLL
ERK
ECE
NLL
ERK
MoNiEN
BCE
NLL
(50.15)
(20.100
(30020)
CIFAR-10
OXDDipp
(2:0.2))
5.13
(50.34)
5.71
(20.00)
0.34
(20.14)
4.45
(50.03)
2:43
(=0.00)
0.57
(50,05)
5.04
(50.16)
2.00
(=0.00
0.23
(20,07)
7.64
(50.15)
2.05
(=0.000
0.75
(20007)
5.87
(50,25)
(=0000)
0.90
KDCL
(10.16)
5.04
(10.70)
3.21
2:00001
0.22
(10.13)
5.07
(10.64)
2.0
(:0.01)
0.15
(40.15)
5.12
(1030
1.9
0.19
(10.13)
7,55
(10.37)
1.50
(:0000
0.24
(10007
10.75
(10.16)
1.51
(1.0.00)
0.23
5.68
7.64
DML
(4015)
25:22
(10.15)
4.72
2-0000
0.93
(10.5)
22:33
(1035)
7.85
(-0.0)
0.55
(48.5)
27.05
(10.18)
2.22
(-000)
0.93
(40.00)
27.75
(10.1%)
3.40
(-000)
0.97
(10.00
31.75
(10:95)
2:73
(-000)
1,10
CIFAR-100
OKDD
(2:9.44)
(50.33)
(50.15)
KDCL
(800.77)
25.35
(:062)
138
(:0.01
0.90
(21.14)
22.25
(:9.30)
3.11
(=802)
0.50
(20.12)
26.67
(50.30
(=0000)
0.91
(91119)
26.81
(51.34)
3.40
(20.0)
0.95
(54125)
31.37
(50,25)
1.05
25.32
3.1
0.30
21.44
7.2
0.82
26.29
1,66
0.91
26.66
2.99
0.94
31,14
1.43
1
(10.10)
(20.39)
(10.00)
(30.81)
(5035)
(:0.01)
(10.11)
(18.15)
( 0.00
(.0024)
(18.21)
(:0.00
(10331
(18.12)
(:0.00)
F.2 DIVERSITY CHANGE ON THE VARIATION OF RAMP-UP PERIOD
This section shows how our diversity is large and maintained well throughout training according to the variation of ramp-up up period a. In the online KD works, a has been used to modulate the power of KD strength to control homogenization. For example, when a is 80, X(e) in Eq 13 varies from O to during the first 80 epochs. As shown in Figure 6 CLIIR, FFI.-S, and ONE are sensitive to the
variation of a. In particular, when is small, the previous works have suffered from homogenization since early epochs. However, OKDDip and ours have not been affected by the variation of a. In addition, our method consistently exhibits the highest diversity over previous works.
PLA
Ond
MR.5
OM
0.15
0.10
3:32
5:10
655
3.52
6.05
50
2333
2.50
15.6
35.4
200
4.0
32
222
110
2.0
122
15.
150
Figure 6: Diversity comparisons during entire training time for ResNet-32 on CIFAR-100. The shaded region represents the mean(+std), calculated from three trials. We plot each diversity based on PC-Softmax (ours) and Softmax (others) while using training set.
F.3 VISUALIZATION OF TEACHER POSTERIOR DISTRIBUTION
This section depicts how the teacher's output varies when predicting specific class samples for each f-th teacher. When T 4, we apply two deep neural networks, ResNet-32 and ResNet- 10. on CIFAR-100 and test on exposure e E [0.1, 0.3, 1.0]. We create arbitrarily partial test set D including only the specific labeled data ) [1, 24]; we directly generate the skewed label distribution with the number of samples equal within " and zero in the reset of classes.
For each sample (x,, 31) ~ D. we can get averaged predictions for each teacher. We first define the conditional distribution Pr (yy z,; 0.) in K-class Softmax, which can be represented as a multinomial distribution:
K
(y = k kb Nu=kk; pr = k|a;; 0.) = exp(={(Al) KK exp(=i(R11) t 8 T, (42)
(y x.; 0.) = II
k=1
19

---

Under review as a conference paper at ICLR 2023
where } denotes the indicator function. The logit of class is given i-th input sample (I,, Us) produced by the t-th teacher model is denoted by [/] Second, we take the conditional distributions for each 1-th teacher and average them across all samples on D.
(v 0.)
Σ (,)l 1ST,
(43)
where N denotes the total number of samples in D. To demonstrate the post-compensation (PC) effect, adapt the original label prior introduced in Section 3.5 and thus replace pt [ 0,) with Pr (vla;; 0e).
we plot Eq 43 and variant of Eq replaced by Pe (y|l; lz; 0.). Smaller leads different conditional distributions when a model is slimmer, inducing specialization and dramatic diversity. After applying the PC strategy, we can see that teachers still maintain diversity in the uniform distribution.
In Figure
GUM
000
0.00
Clacs vodie
3002
042
042
50
24
Class
Class
(a) ResNet-32
(b) ResNet-1
Figure 7: Visualization of averaged teachers' posterior distribution the specific labeled dataset D. The front number of teacher is a teacher index. We plot each averaged conditional distribution based on Softmax (up) and PC-Softmax (down).
F,4 ENSEMBLE CONFIDENCE VISUALIZATION
As shown in Figure 8 we visualize ensemble confidence compared to previous methods on both positive and negative samples. As shown in Figure our ensemble less over-confident for positive samples than previous methods. Especially, our ensemble more confidently mispredicts for negative samples. It implies that our ensemble may have a lower chance of miscalibrated failure (being completely incorrect) than the others. That is, ensemble experiences failure uncertainly.
HLLS
Reg sampha
Repothve
Regethve sarplee
Reputhe Saralo
Figure 8: Confidence of an ensemble posterior distribution corresponds to each class. For each k class, positive samples denote the correct samples corresponding class k and negative samples denote the incorrect samples on the class k. The shaded area corresponds to the mean(=tstd)
G LIMITATION
Our ensemble method produces a better confidence calibration by leveraging two key factors: com- bining probabilities and teacher diversity. To resolve label prior shift among teachers and match the
20

---

Under review as a conference paper at ICLR 2023
student label distribution, the class probabilities are individually post-compensated. PC is necessary, but naively employing the PC strategy can result in sometimes overbalanced posterior probabilities on the rest of the specialty classes (Ren et al. 2020). We studied that generalization error bound for minority classes with fewer samples should be carefully considered in Appendiex C.3] We theo- retically discussed that tightness derive the post-compensation ratio is sometimes well-assumed from Eq. [28] We empirically discovered that our framework suffers from the overbalanced problem when re-scaling teacher outputs on the rest of the specialty labels before forming an ensemble. We conjecture that an estimator derived from importance sampling can have inherent difference from an estimator derived from basic Monte-Cart sampling of the actual imbalanced joint distribution; thus, we speculate that degree of experience with the out of the specialty classes will be little different from the actual situation. future work, we will investigate the posterior overbalanced problem after the PC strategy more thoroughly and fundamentally to obtain better predictions.
H POTENTIAL SOCIETAL IMPACT
This work has the same potential impact any neural network compression study. The positive effect first comes from reducing the resource overhead of deep learning models during inference time. Second, a compressed model with only essential knowledge has more potential because it achieves comparable performance with less power and can even exhibit better generalization than the larger capacity model. Therefore, can deploy neural network models to mobile phones edge devices, expecting acceptable performance. We thus take a step closer to energy-friendly deep learning, facilitating wider of Artificial Intelligence in industrial IoT smart home technology.
At the same time, research on neural network compression may have some negative consequences. For example, if neural network models are more widely used for wearable devices surveillance cameras, privacy invasion or cybercrime is possible. In addition, the malfunction of industrial IoT devices could cause severe problem for the whole production process.
21


Reviews:
[{"review_id": "wZJ3AJ8ks9", "reviewer": "Reviewer_HD42", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Towards the ensemble-based KD, this paper first presents a label prior shift to induce evident diversity among the same teachers. Then the authors propose an aggregation strategy that uses post-compensation in specialist outputs and conventional model averaging. Experiments on several datasets validate its effectiveness.", "summary_of_the_review": "The presentation and writing are good. But the novelty could be improved. And the experimental results are not convincing.", "strength_and_weaknesses": "Strength:\n\n(1) The motivation is clear, where this paper mainly aims to solve the issue of diverse teachers' construction.\n\n(2) The organization is good, which makes the paper easy to follow.\n\n(3) The authors also provide detailed proof for theorems and corollaries.\n\n(4) Sufficient experimental analysis is given for better understanding.\n\nWeaknesses:\n\n(1) My main concern lies in the experimental results. Several methods achieve much better results than the proposed method on all these datasets. The authors compared with PCL on ImageNet, and it achieves a 0.05% improvement over PCL. I wonder why the authors do not compare with it on CIFAR10/100. According to the results on PCL, the results on CIFAR under various backbones are better than the proposed method. Moreover, one important reference[1] is missing, which achieves much better results than the proposed method, including ImageNet.\n\n(2) As introduced by the authors, label prior shift has been extensively discussed by existing methods. Please analyze the difference between the adopted strategy and existing methods. Is this a simple combination with KD?\n\n(3) Sampling has also been well studied. Please also compare the importance sampling with existing methods.\n\n(4) Based on the above analysis, I think the novelty could be improved.\n\n[1] MULTI-VIEW CONTRASTIVE LEARNING FOR ONLINE KNOWLEDGE DISTILLATION, ICASSP, 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pZInYIjUAh", "reviewer": "Reviewer_Xcq2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a online knowledge distillation framework with specialized ensembles. It first uses a label prior shift to generate different teachers for student to learn from. Then it uses PC-Softmax to post-compensate teacher logits and an averaged classifier manner to aggregate teacher predictions. And the experiments part show the improvement over previous method on CIFAR and ImageNet.", "summary_of_the_review": "Overall, the paper proposes a new approach that uses specialized ensembles to do online KD. I would be happy to raise my score if the authors conducted the additional evaluations discussed in my review above.", "strength_and_weaknesses": "The paper is fairly well-written and the high-level problem is interesting. The writing is clear and easy to follow. And the experiment part especially ablation study is well designed to show the improvement over previous methods.\nHowever, there seems lack of comparison on CIFAR with more current works like CGL [1] or PCL [2]. Also there is a most recent one L-MCL [3] that has better performance. And the performance over those method seems not very significant. (It seems more common to report acc than error rate. Using error rate makes it hard to efficiently compare with the unlisted models.)\n\n[1] Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online\nknowledge distillation via collaborative learning. In Proceedings of the IEEE/CVF Conference on\nComputer Vision and Pattern Recognition (CVPR), 2020\n[2] Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. In\nProceedings of the AAAI Conference on Artificial Intelligence, 2021.\n[3] Yang C, An Z, Zhou H, et al. Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition[J]. arXiv preprint arXiv:2207.11518, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ndM2QEj-Yr", "reviewer": "Reviewer_uYCq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to diversify the models in an ensemble forming the teachers in distillation. The specific setup the authors consider is  on-line distillation (where the teachers are trained in parallel with the student) and peer-based (where the models share parts of their weights). \n\nThe proposed approach to achieve the aforementioned diversification of the teachers is based on varying the prior distributions of the labels in the train data used for training each teacher. Then, as an efficient approach to this, importance sampling is employed to avoid sampling from the data separately for each teacher. Then, before averaging the teacher outputs for the student supervision, the \"Post-Compensated Softmax\" loss correction is employed from a previous work.\n\nThe comparison against selected baselines shows improvements. Ablations are provided where the authors measure the effect of the teacher's diversity and the number of teachers within the proposed framework. ", "summary_of_the_review": "The use of the methods described by the authors looks interesting and promising. I suggest conducting more ablations and adding explanations for the decisions to strengthen the paper before resubmission.", "strength_and_weaknesses": "Strenghs:\n- The paper is mostly clear. \n- The Figure 1 is a very nice summary of the work.\n\nWeaknesses:\n- One limitation is that the setup seems very specialized (on-line distillation, peer networks). I don’t see why the technique is specific for that setup - diversifying the teachers (which is the main idea behind the authors’ method) could be well applied to off-line distillation or non-peer networks.\n- The baselines are not explained. It is not clear why these are chosen over other options (there are a lot of distillation methods one could compare to).\n- The ablation section is very useful to have, but I would like to see ablations for why using each of the steps suggested by the authors’ is useful. For example, does PC-Softmax  help? Is importance sampling actually good compared to sampling the data instead? (checking this even on a small dataset would be useful) What about the effect of the peer network choice in the light of your method, as opposed to using completely separate networks?\n- When averaging the teachers, why not weigh them differently depending on what labels they were trained on? Since each teacher focuses on a different set of labels, they should be experts with respect to such labels. It seems that averaging the teacher's predictions is naive and ignores useful information about the teachers.\n- As an application of label prior shift for diversifying the teachers, Why not try it in non-on-line distillation? Is there anything special about on-line distillation that diversifying the teachers is particularly interesting compared to off-line distillation? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "inLGsDcO1aL", "reviewer": "Reviewer_D2AU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a framework for online knowledge distillation from an intentionally diversified ensemble of teachers. A common hypothesis for the success of distillation from an ensemble is that diverse teacher models increase the efficacy of the ensemble. The paper provides a formal framework to enhance teacher diversity by introducing a prior shift - effectively turning each teacher head into a “specialist”. The paper includes a robust empirical analysis and demonstrates that the proposed method significantly improves student performance with respect to both accuracy and calibration.", "summary_of_the_review": "The paper proposes a new framework for online distillation with the intent to increase diversity among the teacher heads and demonstrates significant improvements over prior approaches in benchmark datasets.\n", "strength_and_weaknesses": "Strengths:\n+ The paper is written and structured well.\n+ While many of the component ideas have been introduced before, incorporating such specialists in a complete framework of online distillation is novel and interesting.\n+ The empirical analysis is thorough.\n\nWeaknesses:\n- The paper has a number of typos that are sometimes distracting. e.g. \n  - On page 3: “... both categories, using the extra modules as ever.”\n  - On page 4: “... target distribution while only with samples …”\n  - On page 18: “... optimal exposure differs from models…”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject