Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
Authors: Zeyuan Allen-Zhu
Yuanzhi Li
Venue: ICLR 2023 notable top 5% (2023)

Abstract: We formally study how \emph{ensemble} of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model using \emph{knowledge distillation}. We consider the challenging case where the ensemble is simply an average of the outputs of a few independently trained neural networks with the \emph{same} architecture, trained using the \emph{same} algorithm on the \emph{same} data set, and they only differ by the random seeds used in the initialization.

We show that ensemble/knowledge distillation in \emph{deep learning} works very differently from traditional learning theory (such as boosting or NTKs). We develop a theory showing that when data has a structure we refer to as ``multi-view'', then ensemble of independently trained neural networks can provably improve test accuracy, and such superior test accuracy can also be provably distilled into a single model. Our result sheds light on how ensemble works in deep learning in a way that is completely different from traditional theorems, and how the ``dark knowledge'' is hidden in the outputs of the ensemble and can be used in distillation.

TOWARDS UNDERSTANDING ENSEMBLE, KNOWL- EDGE DISTILLATION AND SELF- SELF-DISTILLATION IN DEEP LEARNING
Zeyuan Allen-Zhu Meta FAIR Labs
Yuanzhi Li
Mohamed bin Zayed University of Al Yuanzhi.. ..i Liimbbzua ac ae
zeyuaralleenzuueetttaacc com
ABSTRACT
We formally study how ensemble of deep learning models can improve test accu- racy, and how the superior performance of ensemble can be distilled into single model using knowledge distillation. We consider the challenging case where the ensemble is simply an average of the outputs of a few independently trained neu- ral networks with the same architecture, trained using the same algorithm on the same data set, and they only differ by the random seeds used in the initialization.
We show that ensemble/knowledge distillation in deep learning works very dif- ferently from traditional learning theory (such boosting or NTKs). We develop a theory showing that when data has structure we refer to as "multi-view" then ensemble of independently trained neural networks can provably improve test ac- curacy, and such superior test accuracy can also be provably distilled into single model. Our result sheds light on how ensemble works in deep learning in a way that is completely different from traditional theorems, and how the "dark knowl- edge" hidden in the outputs of the ensemble and can be used in distillation!
INTRODUCTION
Ensemble (Dietterich) 2000; Hansen & Salamon 1990, Polikar, 2006) one of the most powerful techniques in practice to improve the performance of deep learning. By simply averaging the out- puts of merely a few (like or 10) independently-trrin trainned neural networks of the same architecture, using the same training method over the same training data, it can significantly boost the prediction accuracy over the test set comparing to individual models. The only difference is the randomness used initialize these networks and/or the randomness during training. Moreover, it is discovered by Hinton et al. (2015) that such superior performance of the ensemble can be transferred into a single model (of the same size as the individual models) using technique called knowledge distil- lation: that is, simply train single model to match the output of the ensemble (such as "90% cat + 10% car". also known as soft labels) as opposite the true data labels, over the same training data,
On the theory side, there are lots of works studying the superior performance of ensemble from principled perspectives (see full version for citations). However, most of these works only apply to: (1). Boosting: where the coefficients associated with the combinations of the single models are actually trained, instead of simply taking average; (2). BootstrappingBBaggnng the training data are different for each single model; (3). Ensemble of models of different types and architectures: or (4). Ensemble random features or decision trees. To the best of our knowledge, none of these cited works apply to the particular type of ensemble that is widely used in deep learning: simply take a uniform average of the output the learners, which are neural networks with the same architecture and are trained by stochastic gradient descent (SGD) over the same training set. In fact, very critically, for deep learning models:
TRAINING AVERAGE DOES NOT WORK:
if one directly trains to learn average of individual neural networks initialized by different seeds, the performance much worse than ensemble.
KNOWLEDGE DISTILLATION WORKS: the superior performance of ensemble in deep learning can be distilled into a single model (Hinton et al. 2015).
Full version of this paper be found http:///arrv..oo//////////////

---

Neural Tangent Kernel: (random feature mappings)
train average models
ensemble 10
individual madel accuracies
knowledge distillation
self-distilattion accuracios
WRB 10 NYA on OFARID
72.88%
70.54%
66 68%
66.01% 61.92% 31.38% 27.54%
WRN 10 CIRARL0O
41.47%
38.32%
31.90%
Deep Learning:
ensemble over 10 models
self-distllattin accuracies
knowladge distillation
>>
train
10 models
Individual model accuracies
IN11--2-10 AN1D
97.20%
97.22% / 97.13%
96.46%
96. 70.70.0.21 81.51=0.16%%
WRN 28 10 CIRAR100
84.69%
83.81% / 83.56%
81.83%
Figure 1: Ensemble in deep learning very different from ensemble in random feature mappings. Details in Figure 6
SELF-DISTILLATION WORKS:
even distilling single model into another of the same size, there is performance boost. (Furlanello et al. 2018: Mobahi et al. 2020 Zhang et al. 2019)
We unaware of any satisfactory theoretical explanation for the phenomena above. For instance, as we shall argue, some traditional view for why ensemble works, such as 'ensemble can enlarge the feature space in random feature mappings'. even give contradictory explanations the above phenomena, thus cannor explain knowledge distillation or ensemble in deep learning. Motivated by this gap between theory and practice we study the following question for multi-class classification:
Our theoretical questions:
How does ensemble improve the test-time performance in deep learning when we simply (un- weightedly) average over few independently trained neural networks? - Especially when all the neural networks have the same architecture, are trained over the same data set using the same standard training algorithm and only differ by the random seeds, and even when all single models already have 100% training accuracy? How can such superior test-time performance of ensemble be later "distilled" into a single neural network of the same architecture, simply by training the single model to match the output of the ensemble over the same training data set?
Our results. We prove for certain multi-class classification tasks with special structure refer as multi-view, with a training set 2 consisting of N i.i.d. samples from some unknown distribution D, for certain two-layer convolutional network with (smoothed- Memootthdd)RRLL.U activation learner:
(Single model has bad test accuracy): there is a value μ > 0 such that when a single model f is trained over Z using the cross-entropy loss, via gradient descent (GD) starting from random Gaussian initialization, the model can reach zero training error efficiently. However, w.h.p. the prediction (classification) error of f over is between 0.49p and 0.51p.
(Ensemble provably improves test accuracy): let fI. f2, fL be L πi(1) independently trained single models as above, then w.h.p. G G= + Ee ft has prediction error < 0.01 over DD
(Ensemble can be distilled into single model): if we further train (using GD from random initialization) another single model fo (same architecture each fe) to match the output of = E. fe merely over the same training data set 2, then fo can be trained efficiently and w.h.p. fo will have prediction error < 0.01μ over D as well.
(Self-distillation also improves test accuracy): we further train (using GD from random ini- tialization) another single model f' (same architecture as fi) to match the output of the single model fa merely over the same training data set Z, then f can be trained efficiently and w.h.p. has prediction error at most < 0.26p over D. The main idea is that self-distillation is performing "implicit ensemble + knowledge distillation", as we shall argue in Section 4.2
We defer discussions of our empirical results to Section 5. empirical findings, as they shall confirm and justify our theoretical approach studying ensemble and knowledge distillation in deep learning. Specifically, we give empirical evidences showing that:
However,
highlight some of the
Knowledge distillation does not work for random feature mappings; and ensemble in deep learn- ing is very different from ensemble in random feature mappings (see Figure 1).
Special structures in data (such as the "multi-view" structure we shall introduce) needed for ensemble of neural networks to work.
The variance due to label noise or the non-convex landscape training, in the independently- trained models, may not be connected the superior performance of ensemble in deep learning.
2

---

2 OUR METHODOLOGY AND INTUITION
2.1 A FAILURE ATTEMPT USING RANDOM FEATURE MAPPINGS
The recent advance in deep learning theory shows that under certain cireumstances, neural networks can be treated as a linear function over random feature mappings - see (Allen-Zhu et al. 2019b; Arora et al. 2019b; Daniely et al. 2016, Du et al., 2018b; Jacot et al.) 2018; Zou et al., 2018) and the references therein. In particular, the theory shows when f R2 - IR is neural network with inputs x E R and weights E RD, in some cases, f(W, x) can be approximated by:
S(W, x) se J(Wo, .) + (W- - W. Vw](Wo,,
where Wo is the random initialization of the neural network, and &w, == Vw FWo.... x) is the neural tangent kernel (NTK) feature mapping. This known as the NTK approach. If this approxi- mation holds, then training neural network can be approximated by learning linear function over random features & w. (I), which is very theory-frieet -ecory-friendly..
Ensemble works for random features / NTK. Traditional theorems (Alhamdoosh & Wang. 2014, Brown et al., 2005a; Bryll et al. 2003; Tsymbal et al. 2005) suggest that the ensemble of indepen- dently trained random feature models indeed significantly improve test-time performance, as it enlarges the feature space from @w, (2) to { w- (1) [E(L) for many independently sampled
w((). This can be viewed a feature selection process (Alvarez et al. 2012; Cai al., 2018: Oliveira et al., 2003 Opitz, 1999, Rokach,, 2010), and we have confirmed it for NTK in practice, see Figure 1. However, can we understand ensemble and knowledge distillation in DL as feature selections using NTK? Unfortunately, our empirical results provide many counter examples towards those arguments, see discussions below and Figure
Contradiction I: training average works even better. Although ensemble of linear functions over NTK features with different random seeds: fi(r) = (W(),, d wo! (4)) does improve test accuracy, however, such improvement is mainly due to the use of larger of random features, whose combi- nations contain functions that generalize better. To see this, we observe that an even superior perfor- mance (than the ensemble) can simply be obtained by directly training F(x) = 1. (f1 f21+ fr.)) from random initialization. In contrast, recall if f((x)'s are multi-layer neural networks with dif- ferent random seeds, then training their average barely gives any better performance comparing to individual networks f.. as now all the fi are capable of learning the same set of features.
Contradiction 2: knowledge distillation does not work. For NTK feature mappings, we observe that the result obtained by ensemble cannot be distilled at into individual models, indicating the features selected by ensemble is not contained in the feature tb w (x) of any individual model. In contrast, in actual deep learning, ensemble does not enlarge feature spuce: so an individual neural network is capable of learning the features of the ensemble model.
In sum, ensemble in deep learning may be very different from ensemble in random features. may be more accurate to study ensemble knowledge distillation in deep learning as a feature learning process, instead of a feature selection process. But still, we point out a fundamental difficulty: Key challenge:
a single deep learning model is capable of - through knowledge distillation - learning the features of the ensemble model and achieving better test accuracy comparing to training the single model directly (and the same training accuracy, typically at global optimal of 100%), then why the single model cannot learn these features directly when we train the model to match the true data labels? What the dark knowledge hidden in the output of ensemble (a.k.a. soft label,? comparing to the original hard label?
2.2 ENSEMBLE IN DEEP LEARNING: A FEATURE LEARNING PROCESS
Before addressing the key challenge, we point out that prior works are very limited with respect to studying neural network training as a feature learning process. Most of the existing works prov- ing that neural networks can learn features only focus on the case when the input is Gaussian or
For a k-class classification problem, the output of model g(x) is usually k-dimensional, and represents soft-max probability distribution over the & target classes. This is known the soft label.
3

---

window
B
cal eye
wheel
missing feadiigh
missing wheel feature
headlight
Figure 2: Illustration of images with multiple views (features) in the ImageNet dataset.
Gaussian-like - see for instance (Kawaguchi 2016; Soudry & Carmon 2016 Xie et al. 2016) and many others. However, as we demonstrate in Figure 7 in the full version,
Ensemble in DL might not improve test accuracy when inputs are Gaussian-like:
Empirically, ensemble does not improve test accuracy in deep learning. in certain scenarios when the distribution of the input data is Gaussian or even mixture of Gaussians. This is true over various learner network structures (fully-connected, residual, convolution neural networks) and various labeling functions (when the labels are generated by linear functions, fully-comm conleccon residual, convolutional networks, with/without label noise, with/without classification margin).
Bias variance view of ensemble: Some prior works also try to attribute the benefit of ensemble as reducing the variance of individual solutions due to label noise or non-convex landscape of the training objective. However, reducing such variance reduce convex test loss (typically cross- entropy). but not necessarily the test classification error. Concretely, the synthetic experiments in Figure 7 show that, after applying ensemble over Gaussian-like inputs, the variance the model outputs is reduced but the test accuracy is not improved. We give many more empirical evidences to show that the variance (either from label noise or from the non-convex landscape) usually not the cause for why ensemble works in deep learning, see Section 5
Hence, understand the benefit of ensemble in deep learning theory, we would like study a setting that can approximate practical deep learning, where:
The input distribution is more structured than standard Gaussian and there no label noise. (From above discussions, ensemble cannot work for deep learning distribution-freely
The individual neural networks all are well-trained, in the sense that the training accuracy in the end is 100%, and there is nearly no variance in the test accuracy for individual models. (So training never fails.)
In this work, propose to study setting of data that we refer to multi-view, where the above two conditions both hold when we train a two-layer neural networks with (smoothed-)Rees Reelu activa- tions. We also argue that the multi-view structure consider is fairly common in the data sets used in practice, in particular for vision tasks. We give more details below.
2.3 OUR APPROACH: LEARNING MULTI-VIEW DATA
Let us first give thought experiment to illustrate our approach, and we present the precise mathe- matical definition of the "multi-view" structure in Section Consider a binary classification prob- lem and four "features" U1, U2 03 U4. The first two features correspond the first class label, and the next two features correspond to the second class label. In the data distribution:
When the label is class then
both U1, Ue appears with weight 1, one of v3, vs appears with weight 0.1
w.p. 80%:
only U1 appears with weight one of U3, 0n appears with weight 0.1
w.p. 10%:
only U2 appears with weight 1, one of Us, d4a appears with weight 0.1
w.p. 10%.
When the label is class then
both V3, 84 appears with weight 1, one of 01; 92 appears with weight 0.1
w.p. 80%;
only V3 appears with weight 1, one of U1, d2 appears with weight 0.1
w.p. 10%;
only 24 appears with weight one of U1, U2 appears with weight 0.1
w.p. 10%.
One can for simplicity think of "v appears with weight and appears with weight 3" as data oz
Iu noise.
4

---

ResNet-34 34 learns three features (views) a car: (1) front wheel (2) front window (3) side window
ResNet-34 learns three features (views) of horse: (1) tail (2) legs (3) head
L
Figure 3: Visualization of the channels layer- of ResNet-34 trained on CIFAR-10.
We call the 80% of the data multi-view data: these are the data where multiple features exist and can be used to classify them correctly. We call the rest 20% of the data single-view data: some features for the correct labels are missing. 4
How individual neural networks learn. Under the multi-view data defined above, if we train a neural network using the cross-entropy loss via gradient descent (GD) from random initialization, during the training process of the individual networks, we show that:
The network will quickly pick up one of the feature {vi, v2) for the first label, and of the features v' E {vg, J for the second label. So, 90% of the training examples, consisting of all the multi-view data and half of the single-view data (those with feature v or v'), are clas- sified correctly. Once classified correctly (with large margin), these data begin to contribute negligible to gradient by the nature of the cross-entropy loss.
Next, the network will memorize (using e.g. the noise in the data) the remaining 10% of the training examples without learning any new features, due to insufficient amount of left-over samples after the first phase, thus achieving training accuracy 100% but test accuracy 90%.
How ensemble improves test accuracy. It is simple why ensemble works. Depending on the randomness of initialization, each individual network will pick up 21 or 02 each w.p. 50%. Hence, as long as ensemble (1) many independently trained models, w.h.p. their ensemble will pick up both features {v1, U2 and both features (V3, ). Thus, all the data will be classified correctly.
How knowledge distillation works. Perhaps less obvious is how knowledge distillation works. Since ensemble learns all the features vi, 02, 13, u4, given a multi-view data with label 1, the ensem- ble will actually output x (2, 0.1), where the 2 comes from features 01, U2 and 0.1 comes from one of 03, Ug. On the other hand, an individual model learning only one of U3, will actually output x (2, 0) when the feature dy or 14 in the data does not match the one learned by the model. Hence, by training the individual model to match the output of the ensemble, the individual model is forced to learn both features 03, 24. even though it has already perfectly classified the training data. This is the "dark knowledge" hidden in the output of the ensemble model. (This theoretical finding is consistent with practice: Figure in the full paper suggests that models trained from knowledge distillation should have learned most of the features, and further computing their ensemble does not give much performance boost.)
U4
Significance of our technique. Our work belongs to the generic framework of feature learning in DL where one proves that certain aspects of the algorithm (e.g. the randomness) affects the order
"Meaningfulness of our multi-view hypothesis. Such "multi-views structure is very common in many of the datasets where deep learning excels. In vision datasets in particular, as illustrated in Figure 2, a car image can be classified as a car by looking at the headlights, the wheels, or the windows. For a typical placement of a car in images, we observe all these features and use any of these features to classify it a However, there are car images taken from a particular angle, where one or more features can be missing. For example, an image of a car facing forward might be missing the wheel feature. Moreover, some car might also have a small fraction of "cat features": for example, the headlight might appear similar to cat eyes the ear of a cat. This can be used as the "dark knowledge" by the single model to learn from the ensemble. In Figure 3, we visualize the learned features from actual neural network show that they can indeed capture different views. In Figure 5) we plot the "heatmap" for some car images to illustrate that single models (trained from different random seeds) indeed pick up different parts of the input image to classify it as a car. In Figure 9, we manually delete for instance 7/8 of the channels in some intermediate layer of a ResNet, and show that the test accuracy may not he affected by much after ensemble- - thus supporting that the multi-view hypothesis can indeed exist. even in the intermediate layers of neural network and ensemble indeed collecting all these views.
5

---

data patches
P data patches
App
Nigh MirP
Ypu wepar
Eattimnn Nedann Tome
sen man Tastars sirileege
0.00
0.Arya
wwwghtt u1
0.3eg
0
woight 11,
0.4
0. G.
0.01m1a
0.00yy
0.29g
0.15rg
un
0.39g
0.00igg
wought 22 20.10.55
example multi-view data
(b) example single-view
Figure 4: Illustration of multi-view and single-view data point; the feature vectors can also combined with feature noise and random noise, see Def. 3.1
where features are learned. This is fundamentally different from convex optimization, such as kernel method, where (with &g regularization) there is unique global minimum the choice of the ran- dom seed does not matter (thus, ensemble does not help). There are other works that consider other aspects, such as the choice of learning rate, that can affect the order where the features are learned (Li et al. 2019). Our work is fundamentally different: they only focus on the NTK setting where the features not learned: we study a feature learning process. Recall, the NTK setting cannot be used to explain ensemble and distillation in DL. Our work extends the reach of traditional machine learn- ing theory, where typically the "generalization" is separated from "optimization." Such "separate" treatment might not be enough to understand how deep learning works.
PROBLEM SETUP
The "multi-view" data distribution straight-forward generalization of the intuitive setting in Section 2.3, For simplicity, in the main body, we use example choices of the parameters mainly function of (such P k² 7 पो., 4 N. P k-001 ou 1/v/ as we shall see), and we consider the case when & is sufficienlly large. In our full version, we shall give a much larger range of parameters for the theorems to hold.
1122
3.1 DATA DISTRIBUTION AND NOTATIONS
We consider learning a k-class classification problem over P-patch inputs, where each patch has di- mension d. In symbols, each labelled data represented by (X, y) where X = (I1, -2. xPp) (R is the data vector and y E [k] is the data label. For simplicity, we focus on the case when P 122 and = pkly(k) for large polynomial.
We consider the setting when ic is sufficiently large We use "w.h.p." to denote with probability at +2log" k), and use Ô, Ô, notions to hide polylogarithmic factors in k.
least 1
We first assume that each label class j E [A] has multiple associated features, say two features for the simplicity of math, represented by unit feature vectors 1y U322 E Rd. For notation simplicity, we assume that all the features are orthogonal, namely,
Vj. [k], VE, l' [2], ||,
and vyee when (j. (),
although our work also extends to the "incoherent" case trivially. We denote by
: 2523,e( the set of all features.
We consider the following data and label distribution. Let Cp be global constant, E [1, k0.21 be a sparsity parameter. To be concise, we define the multi-view distribution D,, and single-view distribution D, together. Due to space limitation, here we hide the specification of the random "noise" and defer to the full version.
Definition 3.1 (data distributions Dm and D.). Given D E (Dms Da). we define (X,y) D as follows. First choose the label y [k] uniformly at random. Then, the data vector X is generated
'If we to work with fixed kt say k = 2, our theorem can also be modified that setting by increasing the number features per class. We keep our current setting with two features to simplify notations.
At a high level, we shall allow such "noise" be any feature noise plus Gaussian noise, such as noise Evey Smort + Ep Rd where each o,, E 7) can he arbitrary, and & (T) I).

---

as follows (also illustrated in Figure 4).
1. Denote V(X) = ( U V as the set of feature vectors used in this data vector X, where V is set features uniformly sampled from (. 21 each with probability i.
2. For each v E V(X), pick Op many disjoint patches in [P] and denote as (x(X) C [P] (the distribution of these patches can be arbitrary). We denote ((X) P.
3. IfD = Da is the single-view distribution, pick a value ? = x(x)= E [2] uniformly at random.
4. For each V and p E Po(X), we set Ip 2pU + "noise" Rdd, where, the random
coefficients Zp 2 O satisfy that:
In the case of multi-view distribution D -Dm
EpERL(X) 2p E [L1) when u E {Uy,11 2y EpEP.(x) 2p E (944).0.4) when v E V(X) \ (Ug,1. Fy.2),
7
In the case of single-view distribution D = DD.
EXER,(X) 3p E [1. O()) when u- yy
EXEP.(X) Zp [p, (p)] when e uy,3- EpER.(X) 2p E [T] when U E V(X) (Fy,1, 0.22
5. For each p E (PX), we set Zp to consist only of "noise".
Remark 3.2. The distribution of how to pick P(X) and assign ExEp,(x) Ep to each patch in p E (X) can be arbitrary (and can depend on other randomness in the data as well). In particular, we have allowed differrnt features UJ V2 to show up with different weights in the data (for example, for multi-view data, some view fy can consistently have larger 2p comparing to 6y Yet, we shall prove that the order to learn these features by the learner network can still be flipped depending on the randomness of network initialization.
Interpretation of our data distribution. As we argue more in the full paper, our setting can be tied to down-sized version of convolutional networks applied to image classification data. With small kernel size, good features in an image typically appear only at a few patches, and most other patches random noise ensure that, the concept class is not learnable by linear classifiers or constant degree polynomials. We believe (convolutional) neural network with ReI.U-like activation is somewhat necessary.
magnitude feature noises. More importantly, our noise parameters shall
Our final data distribution D, and the training data set 2 are formally given as follows.
Definition 3.3 (D and Z). The distribution D consists of data from Dm w.p. 1 1- and from D. w.p.
fL. We are given N training samples from D, and denote the training data set as Z = Zm, U 2, where
Zm and 2, respectively represent multi-view and single-view view training data. We write (X. y) Z
as (X.y) y) sampled uniformly random from the empirical data set, and denote N, = [zg- We again for simplicity focus the setting when μ poly(k] and we are given samples N = 1.22 so
each label appears at least (1)) in Zg. Our result trivially applies to many other choices of N.
3.2 LEARNER NETWORK
We consider a learner network using the following smoothed ReLU activation function ReLU:
funntiint Definition 3.4. For integer q 2 2 and threshold kojylog(kj the smoothed function ReLÚ(z) == 0
for z < 0: ReLU(z) 2² q for z e): and ReLU(z) = - (1 e for = 2 e.
Since ReLÚ is smooth we denote its gradient as ReLÚ (z). We focus on q 4 while our result applics to other constants 2 (see full version) or most other forms of smoothing.
For instance, the marginal distribution of Z Σ (x) &p can be uniform over [1, 2). For instance, the marginal distrihution of 7 (X) Ep can be uniform over (0.2, 0.4).
7

---

The learner network F (Fi(X), ....,........................... (x,,) R is two-layer convolutional network parameterized by War E R for E [k).r E [ml, satisfying
Vi [ (((X) -re(m)) Epe(ry ReLU((w,,
Although there exists network with = 2 that can classify the data correctly (e.g. Wirr Us.r for r E [2]). in this paper, for efficient optimization purpose is convenient to work on a moderate level of paraeetrttttii: upper bounds hold even for small over-paramettrriztion m polylog(k)..
[polylog(k)), kt]. Our lower bounds hold for any in this range and
Training a single model. We learn the concept class (namely, the labeled data distribution) using gradient descent with learning rate 1) > O, over the cross-entropy loss function L using N training data points {(Xi, 3:) vENNE We denote the empirical loss as:
L(F) # EiEIN) L(F;X,, vi) Exx.,)=
FF((x) where I(F;X,y) = - log Ezem es00 We randomly initialize the network F by letting each (0) W. NN off) for oā 1/k, which is the most standard initialization people use in practice.
To train a single model, at each iteration we update using gradient descent (GD).
(+++)
&- wir E(x,))> V my. L(F(;XX,)
(3.1)
1,
We run the algorithm for T plly(k))) iterations. We use F(*) to denote the model F with hidden weights ( at iteration
We denote by logit, (F,X) = ex2( Σjemy (x)) Using this, we can write down
Notations.
E [k], [m]: Ww,,L(FF: (1,4u (F, w., F(()
4 MAIN THEOREMS AND EXPLANATIONS
We now state the main theorems (and the one for self-distillation in the full paper). 10
Theorem (single model). For every sufficiently large > O, every m E [polylog (k), k). every n S k suppose we train single model using the gradient descent update (3.1) 1) starting from the random initialization defined in Section 3.2) then after poly(k) many iterations, with probability
- -0log" k), the model p(T) satisfies:
(training is perfect): meaning for all (X, v) E Z, all [6] (u): F (x) (x).
(test accuracy is consistently bad); meaning that:
Pr(X.))~D(ii e [K] (v):
1 (X) (X E [0.49p, 0.51,]
We shall give technical intuitions about why Theorem holls in the full version. But, at high-level. shall construct "lottery winning" set M C [k] x [2] of cardinality [M] [k<1 ()])), It only depends on the random initialization of F. Then, with some effort we can prove that, for every (i, f) E M, at the end of the training FTT will learn feature Ug,E but not learn feature U2,3-e- This means for those single-view data (X, y) with y and {xx) 3 l, the final network F(T) will predict its label wrong. This is why the final test accuracy is around 0.5yz.
Note the property that test accuracy consistently belongs to the range [0.49pu, 0.51 should be rem- iniscent of message @ in Figure 6, where multiple single models, although starting from different random initialization, in practice does have relatively small variance in test accuracies.
Our result also extends to the case when there is a weight decay, discussed in the full version. We shall restate these theorems in the full version with more details and wider range of parameters.
8

---

G(X) ((1) K Ee (X) Ensemble. Suppose {Flt) lej K S2(1) independently trained models of F with m polylog(k) for T = O p) iterations (i.e., the same setting as Theorem I except we only need small parrreeaemttiiziioonn polylog(k)). (k)). Let define their ensemble
(4.1)
Theorem 2 (ensemble). In the same setting as Theorem 1 except now we only need small m polylog(k),, we have for the ensemble model G in (4.1) with probability at least 1 e k);
(training is perfect): meaning for all (X,9) E 2, for all i E [b) {u}: G₂(X) > Gi(X).
(test accuracy is almost perfect): meaning that:
Pr(xw)~DBF Ei E [R] \ {v): G2(X) G,(X) 0.001
As we discussed in Section 2.3, the reason Theorem 2 holds attributes to the fact that those lottery winning sets M depend on the random initialization of the networks; and therefore, when multiple models are put together, their "union" of . shall cover all possible features (1t,e)
Moreover, our theorem only requires individual K (2(1) models for ensemble, which is indeed "averaging the output of few independently trained models".
4.1 KNOWLEDGE DISTILLATION FOR ENSEMBLE
We consider a knowledge distillation algorithm given the existing ensemble model G (see (4.1)) as follows. For every label E [k)]. let us define the truncated scaled logit as (for log k
unin(t " F,000.11
logit (F;
(4.2)
ESE(N] 00
(This should be reminiscent of the logit function with temperature used by the original knowledge distillation work (Hinton et al. 2015); we use truncation instead which is easier to analyze.)
Now, we train new network F from random initialization (where the randomness independent of all of those used in flf1). At every iteration we update each weight wir by:
(e+1) Wir
Vi,, w we, L(F(0) 1/' B(xaa== (logit) (F(), x) - logit (G,
(og (G, V Wir F! (x))
(4.3)
Notation. Throughout the paper we denote by [a] max a} and [a] min
This knowledge distillation method (4.3) is almost identical to the one used in the original work (Hinton al., 2015), except we use a truncation during the training to make more (the- oretically) stable. Moreover, we update the distillation objective using larger learning rate n/ comparing to 7) of the cross-entropy objective. This also consistent with the training schedule used in (Hinton et al., 2015).
Let (()) the resulting network obtained by (4.3) at iteration We have the following theorem:
Theorem 3 (ensemble distillation). Consider the distillation algorithm (4.3) in which is the ensemble model defined in (4.1). For every k 0. for polylog(k), for every 17 poly(R)' setting n' (k)) after T poly(k) many iterations with probability least 22log k). for at least 90% of the iterations 7:
(training is perfect): meaning for all (X.y) E 2, all i i [K]] fu): F (X) > (X)
(test accuracy is almost perfect): meaning that:
Pr Bi E [M] (u): (X) (X)) 0.001
(X.))~D
Remark. Theorem necessarily means that the distilled model F has learned all the features (ug,e (x2 from the ensemble model G. This is consistent with empirical findings in
Figure 8: if one trains multiple individual models using knowledge distillation with different ran- dom seeds, then their ensemble gives further performance boost.

---

REFERENCES
Monther Alhamdoosh and Dianhui Wang. Fast decorrelated neural network ensembles with random weights. Information Sciences, 264:104-117 2014.
Zeyuan Allen-Zhu and Yuanzhi Li. What Can ResNet Learn Efficiently, Going Beyond Kernels? In
NewrIPS, 2019a. Full version available at httttp://aaiii> wwt.oor///aao... 1905 10337
Zeyuan Allen-Zhu and Yuanzhi Li. Can SGD Learn Recurrent Neural Networks with Provable Generalization? In NeurlPS, 2019b. Full version available httt:://axiv..orrt
1992.010288
Zeyuan Allen-Zhu and Yuanzhi Li. Backward feature correction: How deep learning performs deep learning. arXiv preprint arXiv:2001.044/ 2020.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. On the convergence rate of training recurrent neural networks. In NeurIPS, 2019a. Full version available at
12065
Zeyuan Allen-Zhu Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In ICML, 2019b. Full version available htttp://axii. org/ abs/
039662
Jose M Alvarez, Yann LeCun, Theo Gevers, and Antonio M Lopez. Semantic road segmentation via multi-scale ensembles of learned features. In European Conference Computer Vision, pp- 586-595. Springer, 2012.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. arXiv preprint arXiv:19044 1955, 2019a.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterizee two-layer neural networks. CoRR, abs/1901. 088584, 2019b. URL http::ttt::/stttt wwgp:iiwivooraoooooooooo abs 1901.08584
Verónica Bolón-Canedo Canedo and Amparo Mlonso-Betanzor Ensembles for feature selection: A review and future trends. Information Fusion, 52:1-12, 2019.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
Gavin Brown, Jeremy Wyatt, Rachel Harris, and Xin Yao. Diversity creation methods: a survey and categorisation. Information Fusion, 6(1):5-20, 2005a.
Gavin Brown, Jeremy L Wyatt, and Peter Tiño. Managing diversity in regression ensembles. Journal of machine learning research, 1621 2005b.
Robert Bryll, Ricardo Gutierrez-Osuna, and Francis Quek. Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets. Pattern recognition, 3616):121-1100,, 2003.
Jie Cai, Jiawei Luo, Shulin Wang, and Sheng Yang. Feature selection in machine learning: A new perspective. Neurocomputing, 300:70-79, 2018.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. In Advances in Neural Information Processing Systems, pp. 10835-10845, 2019.
Victor Chermozhukov, Denis Chetverikov, and Kengo Kato. Comparison and anti-concentration bounds for maxima of gaussian random vectors. Probability Theory and Related Fields, 162(1): 47-70, 2015.
Amit Daniely. Sgd learns the conjugate kernel class of the network. In Advances in Neural Infor- mation Processing Systems, pp. 2422-2430, 2017.
Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks:
The power of initialization and a dual view on expressivity. In Advances in Neural Information Processing Systems (NIPS), pp. 2253-2261, 2016.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multi- ple classifier systems, pp. 1-15. Springer, 2000.
Simon S Du, Jason D Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. arXiv preprint 1811 03804, November 2018a.
10

---

Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint
2018b. David A Freedman et al. Bootstrapping regression models. The Annals of Statistics, 966):2118
1228, 1981.
Yoav Freund and Robert E Schapire. A decision-theoretie generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-139, 1997.
Yoav Freund, Robert Schapire, and Naoki Abe. short introduction boosting. Journal-Japanee Society For Artificial Intelligence, 14(771-780):1612, 1999.
Jerome Friedman, Trevor Hastie, Robert Tibshirani, al. Additive logistic regression: statistical view of boosting (with discussion and a rejoinder by the authors). The annals of statistics, 28(2):
337-407, 2000.
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of
statistics, pp. 1189-1232, 2001.
Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar.
Born again neural networks. arXiv preprint arXiv: 1805. 04770, 2018.
Mikel Galar, Alberto Fernandez, Edurne Barrenechea, Humberto Bustince, and Francisco Herrera. A review on ensembles for the class imbalance problem: bagging-. boosting-, and hybrid-based approaches. IEEE Transactions Systems, Man, and Cybernetics, Parl C (Applications and Reviews), 42(4))463-444, 2011.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension. arXiv preprint arXiv:1904. 12191, 2019.
Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. arXiv 2019.
preprint 1909:/900005500
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on pattern
analysis and machine intelligence, 12(10):9331001, 1990.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1rxiv:1503.0253: 2015.
Tin Kam Ho. The random subspace method for constructing decision forests. JEEE transactions on pattem analysis and machine intelligence, 20(8):832-8 834484,,, 1998.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. In Advances in neural information processing systems, pp. 8571- - 8580, 2018
Kenji Kawaguchi. Deep learning without poor local minima. In Advances in Neural Information Processing Systems, pp. 586-594, 2016.
Josef Kittler, Mohamad Hatef, Robert PW Duin, and Jiri Matas. On combining classifiers. IEEE transactions on pattern analysis and machine intelligence, 20(3):226-239, 1998.
Ron Kohavi, George H John, et al. Wrappers for feature subset selection. Artificial intelligence, 97
(1-2):273-324, 1997.
J Zico Kolter and Marcus A Maloof. Dynamic weighted majority: An ensemble method for drifting concepts. Journal of Machine Learning Research, 8(Dcc):2755-2799 2007.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Ludmila Kuncheva. Combining pattern classifiers: methods and algorithms. John Wiley & Sons, 2014.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient
descent structured data. In Advances in Neural Information Processing Systems, 2018.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. arXiv preprint arXiv: /99900499999 2019.
Pankaj Mehta, Marin Bukov, Ching-Hao Wang, Alexandre GR Day, Clint Richardson, Charles K Fisher, and David J Schwab. A high-bias, low-variance introduction machine learning for physicists. Physics reports, 810:1-124, 2019.
Hossein Mobahi, Mehrdad Farajtabar, and Peter L Bartlett. Self-distillation amplifies regularization in hilbert space. arXiv preprint arXiv:2002.05715, 2020.

---

M Arthur Munson and Rich Caruana. On feature selection, bias-variance, and bagging. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 144 159. Springer, 2009.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):e7,
2017. doi: 10.23915/disttll.00007, https://distill..uubbb// http://i......ooo/////aaarraaaaa
Luiz Oliveira, Robert Sabourin, Flávio Bortolozzi, and Ching Y Suen. Feature selection for ensembles: A hierarchical multi-objective genetic algorithm approach. In Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings, pp. 676-680. Citeseer, 2003.
David W Opitz. Feature selection for ensembles. In AAAI, pp. 379-384, 1999.
Robi Polikar. Ensemble based systems decision making. IEEE Circuits and systems magazine, (3):21-45, 2006.
Juan José Rodriguez, Ludmila Kuncheva, and Carlos J Alonso. Rotation forest: new classifier ensemble method. IEEE transactions on pattern analysis and machine intelligence, 28(10):1619 1630, 2006.
Lior Rokach. Puttern classification using ensemble methods, volume 75. World Scientific, 2010.
Lior Rokach and Oded Maimon. Data mining with decision trees: theory and applications, volume 69. World scientific, 2008.
Robert E Schapire, Yoav Freund, Peter Bartlett, Wee Sun Lee, et al. Boosting the margin: A new explanation for the effectiveness of voting methods. The annals of statistics, 26(5):6651-168) 1998.
Vaishaal Shankar, Alex Fang, Wenshuo Guo, Sara Fridovich-Keil, Ludwig Schmidt, Jonathan Kelley and Benjamin Recht. Neural kernels without tangents. arXiv preprint arXiv:2003.022.77 2020.
Daniel Soudry and Yair Carmon. No bad local minima: Data independent training error guarantees for multilayer neural networks. arXiv preprint arXiv:1605.0883: 2016.
Alexey Tsymbal, Mykola Pechenizkiy, and Pádraig Cunningham. Diversity in search strategies for ensemble feature selection. Information fusion, 6(1):83-98, 2005.
Giorgio Valentini. An experimental vias-variance analysis of svm ensembles based on resampling techniques. IEEE Transactions on Systems, Man, and Cybernetics, Part (Cybernetics). 35(6);
1252-1271,
Giorgio Valentini and Thomas G Dietterich. Bias-variance analysis of support vector machines for the development of svm-based ensemble methods. Journal of Machine Learning Research, 5(Jul): 725-775, 2004.
Bo Xie, Yingyu Liang, and Le Song. Diversity leads to generalization in neural networks. arXiv preprint 1xXIv:661..03111 2016.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint 2019.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
arXiv:1605.07116, 2016.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV, pp. 3713-3722, 2019.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes over-parameterized deep relu networks. arXiv preprint arXiv:1811.08888, 2018
12


Reviews:
[{"review_id": "uA3o3TQt7hc", "reviewer": "Reviewer_GAwQ", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the theories of ensemble and knowledge distillation. Insightful studies regarding NTKs and deep neural networks are performed and novel theoretical results based on the multi-view structure of data are established. I get many new insights after reading this paper.", "summary_of_the_review": "This is an outstanding paper in multiple aspects, so I suggest acceptance.", "strength_and_weaknesses": "# Strength\n- The writing and presentation are good. It is a pleasure to read this paper. \n- The empirical results on neural tangent features and deep neural networks are novel and interesting. The authors then provide thorough explanations for them, which is appreciated.\n- The theoretical results are rigorous.\n\n# Weaknesses\n- We know new theories usually motivate new algorithms. I want to know if there are some potential future works to better exploit these theories. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "nEOoJqx-z7W", "reviewer": "Reviewer_Y3qu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper takes a step toward understanding ensemble and knowledge distillation. The authors consider the challenging setting where the teacher model is an average of several models of the same structure, or even the teacher has an identical structure as the student model. The authors developed a theory that the distillation from several independently trained neural networks can improve the performance when the data has a \"multi-view\" structure. ", "summary_of_the_review": "This paper presents an interesting and intuitive explanation for distillation. However, one major concern is that I'm not able to open the supplementary material due to some file format error. I'm not able to fully assess this paper.", "strength_and_weaknesses": "Strength:\n\n1. The authors present theoretical results showing that a single model is guaranteed to have 0 training error while having a high testing error with high probability; the ensemble can provably improve the testing accuracy.\n\n2. The author shows that the ensemble can be efficiently distilled into a single model. This understanding is fundamentally different from the standard NTK settings.\n\n3. The idea of \"multi-view data\" is intuitive and provides a natural and convincing explanation for various empirical observations for distillation.\n\nQuestions and weaknesses:\n\n1. One major concern is that the supplementary material cannot be opened due to a file format error, and I'm not able to see the results of self-distillation and empirical evaluation, which seem to be important aspects of this work.\n\n2. In the example of Section 2.3 and for a single model, why the network can only pick up one of the features in $v_1, v_2$ with the 90% of the data, and only memorize the remaining 10% of the data, instead of learning from the other relevant feature in $v_1, v_2$?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FsEoteuji3y", "reviewer": "Reviewer_qyvd", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the theoretical explanation of why the ensemble and knowledge distillation works for the deep learning model. It shows the different behavior of classic parameterization of neural networks and the random feature model (e.g. NTK). It then explains via the idea of multi-view data that the network infers via generalizable learning of a subset of features and memorizing another subset of data. Such analysis differs from the classic analysis that decomposes optimization and generalization.\n\nOverall, I find the insight of the paper quite interesting, and such intuition is well supported by the empirical evidence using the practical network. The construction of the multi-view data for analysis is closely related to the practical settings.\n", "summary_of_the_review": "Questions:\n\n1. Almost all the result suggests a 100% training accuracy but this is different from almost 0 cross-entropy loss. I feel all the results will not hold if the model is large and trained with almost zero cross-entropy loss? At least, in this case, the distillation via logits seems to be no more useful. Pls correct me if I am wrong. \n\n2. Following 1, how do the results look like if we consider regression rather than classification?\n\n3. Following 1 and 2, I was curious if we do distillation on the latent feature space rather than logit space, will the result become very different? From the feature learning perspective, distill on the feature space seems more reasonable?", "strength_and_weaknesses": "Strength:\n\n1. The novelty of the paper is quite strong. I believe the such analysis is new and sound.\n\n2. There is quite a good match between theory and practice, especially the construction of the multiview dataset is an exciting and meaningful proxy for the practical case while being theory-friendly.\n\n3. The writing of the main context is well organized. It is easy for the reader to get to know the most critical intuition of the theory.\n\n4. I like the experiment used in this paper. It is concise but well supports the theory.\n\nWeakness:\n\n1. I am not able to open the supplementary of the paper but reviewing this paper, reading the appendix is very necessary and thus my reading of the appendix is via the arxiv. The structure of the appendix is good but I feel a more simplified overall technical/intuition is desirable.\n\n2. To obtain such a theory, there are still a lot of 'artificial' in the problem setup such as the use of smoothed ReLU, and specific restrictions of the data distribution. Those slightly increase the gap between theory and practice. But this point does not downgrade my point, understanding deep learning in theory is very hard and this paper has done excellent work toward that.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2UCozqHYTmk", "reviewer": "Reviewer_ZnbC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigated the superior performance of ensemble, knowledge distillation and self-distillation in the field of deep learning. As previous principles of ensemble on traditional machine learning algorithms do not apply to deep learning methods, a novel perspective of \"multi-view\" data was proposed to explain the phenomenon. Both empirical and theoretical analysis support multi-view theory for ensemble in deep learning.", "summary_of_the_review": "The paper made an attempt to explain the superior performance of ensemble, knowledge distillation and self-distillation with the theory of \"multi-view\" data structure and provided insights on understanding those techniques in deep learning. The analysis is rigorous and thus I tend to accept it.", "strength_and_weaknesses": "Strengths:\n- The paper analyzed an underexplored problem of how ensemble works in deep learning methods. It pointed out contradictions when applying previous principles of traditional algorithms and shed light on understanding deep learning.\n- For the theory side, the analysis was rigorous with clear definitions and concrete examples to help better digest the theorems.\n\nWeaknesses:\n- I am just curious whether the hypothesis of multi-view data structure can be extended to different data structures such as texts and graphs. Those data are very different from image data discussed in this paper and 'multi-view' might not be applicable to them.\n- It is not a good practice to include \"self-distillation\" in the title but defer all details about it to the appendix, which was somewhat misleading even though it was due to page limit. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%