Neighbourhood Distillation: On the benefits of non end-to-end distillation
Authors: Laëtitia Shao
Elad Eban
Yair Movshovitz-Attias
Venue:  (2021)
Keywords: distillation
deep learning

Abstract: End-to-end training with back propagation is the standard method for training deep neural networks. However, as networks become deeper and bigger, end-to-end training becomes more challenging: highly non-convex models gets stuck easily in local optima, gradients signals are prone to vanish or explode during backpropagation, training requires computational resources and time. 

In this work, we propose to break away from the end-to-end paradigm in the context of Knowledge Distillation. Instead of distilling a model end-to-end, we propose to split it into smaller sub-networks - also called neighbourhoods - that are then trained independently. We empirically show that distilling networks in a non end-to-end fashion can be beneficial in a diverse range of use cases. First, we show that it speeds up Knowledge Distillation by exploiting parallelism and training on smaller networks. Second, we show that independently distilled neighbourhoods may be efficiently re-used for Neural Architecture Search. Finally, because smaller networks model simpler functions, we show that they are easier to train with synthetic data than their deeper counterparts.

Under review as a conference paper at ICLR 2021
NEIGHBOURHOOD DISTILLATION: ON THE BENEFITS OF NON END-TO-ENE DISTILLATION
Anonymous authors Paper under double-blind review
ABSTRACT
End-to-end training with back propagation is the standard method for training deep neural networks. However, as networks become deeper and bigger, end-to- end training becomes more challenging: highly non-convex models gets stuck eas- ily in local optima, gradients signals are prone to vanish or explode during back- propagation, training requires computational resources and time. In this work, we propose to break away from the end-to-end paradigm in the context of Knowledge Distillation. Instead of distilling a model end-to-end, we propose to split it into smaller sub-networks also called neighbourhoods that are then trained indepen- dently. We empirically show that distilling networks in a non end-to-end fashion can be beneficial in diverse range of use cases. First, show that it speeds up Knowledge Distillation by exploiting parallelism and training on smaller net- works. Second, show that independently distilled neighbourhoods may be effi- ciently re-used for Neural Architecture Search. Finally, because smaller networks model simpler functions, we show that they are easier train with synthetic data than their deeper counterparts.
INTRODUCTION
As Deep Neural Networks improve on challenging tasks, they also become deeper and bigger. Image classification convolutional neural networks grew from 5 layers in LeNet (LeCun et al., 1998) to more than 100 in the latest ResNet models (He et al., (0166). However, as models grow in size, training by back propagating gradients through the entire network becomes more challenging and computationally expensive. Convergence a highly non-convex space can be slow and requires the development of sophisticated optimizers escape local optima (Kingma & Ba 2014). Gradients vanish or explode as they get passed through an increasing number of layers. Very deep neural networks that trained end-to-end also require accelerators, and time to train to completion.
Our work seeks to overcome the limitations of training very deep networks by breaking away from the end-to-end training paradigm. We address the procedure of distilling knowledge from teacher model and propose to break a deep architecture into smaller components which are distilled indepen- dently. There are multiple benefits working on small neighbourhoods as compared to full models: training a neighbourhood takes significantly less compute than a larger model; during training, gra- dients in a neighbourhood only pack-propagate through a small number of layers making it unlikely that they will suffer from vanishing or exploding gradients. By breaking a model into smaller neigh- bourhoods, training can done parallel, significantly reducing wall-time for training as well as enabling training on CPUs which are cheaper than custom accelerators but are seldom used in Deep Learning as they are too slow for larger models.
Supervision to train the components is provided by pre-trained teacher architecture, as is com- monly used in Knowledge Distillation (Hinton et al. (015)), a popular model compression technique that encourages student architecture to reproduce the outputs of the teacher. For this reason, call our method Neighbourhood Distillation. In this paper, we explore the idea of Neighbourhood Distillation number of different applications, demonstrate its benefits, and advocate for more research into non end-to-end training.
Contributions

---

Under review as a conference paper at ICLR 2021
We provide empirical evidence of the thresholding effect, a phenomenon that highlights deep neural networks' resilience to local perturbations of their weights. This observation motivates the idea of Neighbourhood Distillation.
We show that Neighbourhood Distillation is up 4x faster than Knowledge Distillation while producing models of the same quality. We demonstrate this on model compression and sparsification.
Then, we show that neighbourhoods trained independently can be used in a search algo- rithm that efficiently explores an exponential number of possibilities to find an optimal student architecture.
Finally, we show applications of Neighbourhood Distillation to zero-data settings. Shallow neighbourhoods model less complex functions which we can distill using only Gaussian noise as a training input.
RELATED WORK
Non end-to-end training Before the democratization of deep learning, machine learning methods relied on multi-stage pipelines. For example, the face detection algorithm designed by Vyola & Jones (200T) is a multi-stage pipeline relying first on handcrafted feature extraction and then on classifier trained to detect faces from the features. Then came the idea of directly learning classification from the input image, leaving the model to learn all parts of the pipeline through a series of hidden layers (LeCun et al. [1998 Fukushima & Miyake 1982) that could be trained with end-to-end with gradient prop-ppoonnnttion (Rumelhart et al. 1986) or layerwise training (Vincent et al. 2008). End- to -end deep learning gained traction with the success of the AlexNet model (Krizhevsky et al.) 2012) in image classification. It is now the main component of various state-of-the art approaches object detection (Redmon et al., 2016 Ren al. 2015), image segmentation (He et al.
2017).
speech processing (Senior et al. (2012), machine translation (Seo et al. 2016 Vaswani et al.
2017
However, gradient-based end-to-end learning comes with cost. Highly non-convex losses harder to optimize; models of bigger sizes also require more data to fully train; they suffer from vanishing and exploding gradients (Hochreiter 1998 Pascanu al. 2012).
Approaches to overcome these issues can be broken down into three categories. First, several meth- ods have been introduced to ease the training of deep models, such as residual connections (He et al. (216)), gated recurrent units (Cho et al. 2014), normalization layers (Toffe & Szegedy 2015
Ba et al. 20166] Salimans & Kingmal 2016). and more powerful optimizers (Kingma Ba 2014 Hinton et al.; Duchi et al. 2011). Second, engineering best practices have adapted to rise to the challenges raised by deep learning: pre-trained models trained on large-datasets can be reused for transfer learning, only requiring the fine-tuning of a portion of the model for specific tasks (Devlin)
ct al. 2018 Dahl et al. 201I). Distributed training (Krizhessyy et al. 2012 Dean et al. 2012) and
custom hardware accelerators (Jouppi et al. 2017) were also crucial in accelerating training.
The last category, which our work falls into, investigates non end-to mnd-to-end training methods for deep neural networks. One class of non end-to-end learning method relies on splitting a deep network into gradient-isolated modules trained with local objectives (Löwe et al. 2019 Nøkland & Eidnes 2019). Layerwise training (Belilovsky al., 2018 (uung et al.) 2017) also divides the target network into modules that are sequentially trained in a bottom-up approach. Difference Target Propagation (Lec) al., 2015)) seeks to optimize each layer to output activations close to a given target value. These values are computed by propagating inverses from downstream layers while ours are provided by pre-trained teacher model. All of these approaches also differ from ours as modules still depend on each other, while our neighbourhoods are distilled independently.
Knowledge Distillation Our work specifically draws from Knowledge Distillation (Hinton et al.) 2013), a general- purpose model compression method that has been successfully applied to vi- sion (Crowley et al. 2018) and language problems (Hahn & Choi 2019). Knowledge Distillation transfers knowledge from a teacher in the form of its predicted soft logits. Various variations have been developed to improve distillation. One direction is to transfer additional knowledge in the form of intermediate activations (Romero et al. 2014 Aguilar et al., 2019 Zhang et al., 2017). at- tention maps (Zagoruyko & Komodakis 2016), weight projections (Hyun Lee al. 2018) or layer
2

---

Under review as a conference paper at ICLR 2021
interactions (Yim al. 2017). Other methods also seek to directly address the capacity gap be- tween a teacher and student (Cho & Hariharan, 2019) by distilling from a series of intermediate
teachers (Mirzadeh et al.) 2019: Jin et al. 2019). These methods all distill the student end-to-end.
Neural Architecture Search
Recent papers study how to combine Knowledge Distillation with Neural Architecture Search methods, which automate the design process by exploring a given search space (Liu et al.; 2018: Pham al.] 2018: Liu et al. [2017] Tan & Le 2019 Zoph et al., (017). These methods have successfully been applied to find better suited students for a given teacher (Kag et al.)
2019 Liu al. 2020p. Closely related to work, [Li et al. (2020) divide supernet into blocks and use Knowledge Distillation to train it. However, they only focus on extracting the architectural
knowledge from the teacher, ignoring the parameters learned during the search process.
3 THRESHOLDING EFFECT
Due to their size, modern neural networks are usually overparameterized. Their learned represen- tations are redundant and recent empirical studies conducted on ResNets (Zhang et al.) 2019) Veit et al. 2016) showed that it is possible to drop or reset layers in a trained network without hurting their performance. We hypothesize further that less drastic modifications placing part or all of their cub-components by imperfect approximations, will not result in dramatic error accumulation. In the following section, we provide empirical evidence of an interesting prop- erty that supports this: sub-components of trained model may be perturbed without damaging the model's s accuracy, as long as individual local errors remain under a certain threshold. We call this phenomenon the thresholding effect.
network, such re-
First, we introduce the notion of neighbourhood that will be used throughout the rest of the paper. Deep neural networks are built by stacking a succession blocks of operations such as convolutions and non-linear layers. We express this by defining network T as composition of sub- neb-networks:
vi [1. [1...n), nn. T R. R.+
(1) We call neighbourhood any portion of the network that is delimited by one sub-networt T,. This neighbourhood represents an arbitrary logical construction block in the network and may be replaced by variants of its architecture that have the same input and output shapes. For any given netwrok, one can define multiple ways to break it up into neighbourhoods.
T= Tn.19
The question we set out to answer is the following. Imagine want to replace part all the neighbourhoods by imperfect approximations, how good do these approximations need to be to prevent drastic loss of performance in the modified model?
We consider different pre-trained models and estimate how their accuracy is impacted by perturba- tions of the network's 's intermediate features. To do so, we perturb each neighbourhood by artificially introducing some gaussian noise of amplitude e to each activation output.
S = +
8 ~~ x)
The students are then composed into a full network S S,
S which we evaluate.
On CIFAR- 10 (Kizhevsky et al. 2009), we train a ResNetV1-20 (He al.) 2016) model and define neighbourhood as one bottleneck block which consists of a two-layer convolutional network and a skip connection. We reiterate a similar experiment on large-scale dataset. On ImageNet (Deng al. 2009), train several EfficientNet (Tan & Le 2019) models and define neighbourhood as one mobile inverted bottleneck block.
Figure F shows how errors of different amplitudes accumulate across networks when replacing some or all neighbourhoods by approximations. In particular, we consistently witness a thresholding effect in all networks. When the amplitude of the noise is small enough, the final accuracy of the network is not impacted by accumulated perturbations. This threshold appears to depend on the number of neighbourhoods.
3

---

Under review as a conference paper at ICLR 2021
0
Accuracy change
Accuracy change
-5
-5
B0
-10
-10
blocks
blocks
B1
-15
blocks
blocks
-15
blocks
block
B3
-20
blocks
-20
0.0001
0.001
0.01
0.1
1.0
0.0001
0.001
0.01
0.1
1.0
g
E
(a) ResNetV1120 with 91.6% accuracy on CIFAR- 10.
(b) EfficientNet models trained on ImageNet. Each line represents different EfficientNet model for which all neighbourhoods have been perturbed.
Each line represents a different number of perturbed blocks.
Figure 1: We perturb the intermediate outputs regular locations in the network using gaussian noise of amplitude c and measure the effect of these perturbations on the accuracy of the model. We show that accuracy remains stable (accuracy change close 0) long remains under certain threshold. The threshold differs between network architectures and number of perturbed neighbourhoods. Note that for all models, even when perturbing all neighborhoods, there is still range for which there is virtually no loss in accuracy.
Student
MSE
Root
Teacher
Head
Neighbourhood Distillation
1 x 4D
1x 1x 4D
Student
Head
3 x
3 3 kD
MSE
1 Dx
1x 1x KD
Root
Teacher
Head
Teacher Network
Teacher Block
Student Block
Look-ahead loss
(b) Neighbourhoods parameter reduc- tion.
(a) Computational Graphs
Figure 2: (a) Computation graphs for Neighbourhood Distillation. Top: the teacher and the student neighbourhoods receive activations from the root of the teacher network. The student neighbour- hood is trained to reproduce the output of the teacher. Bottom: the teacher and student outputs propagated to the head of the teacher network and additional activations between teacher and student networks are compared. The look-ahead loss gives additional training signal for the student to reproduce the teacher. (b) Example of teacher and student neighbourhoods.
Our experiments on the thresholding effect show that is possible to locally replace sub-components of a model without hurting the performance of the reconstructed model. This observation is what motivates Neighbourhood Distillation: neighbourhoods trained to approximate their teacher outputs can be used reconstruct student networks with no or limited accuracy drop.
In the appendix, we also present preliminary results on understanding the thresholding effect and show how regularizing the teacher network can impact threshold.
4 NEIGHBOURHOOD DISTILLATION
In order to train new model in non end-to-end fashion, we leverage the representation of net- work as a stack of neighbourhoods. Supervision to train each component independently is provided by pre-trained teacher architecture which is also divided into small neighbourhoods. Each student neighbourhood S; is trained to match the outputs of teacher by minimizing the mean-square error between their produced features. The training inputs obtained by propaatpro of the
4

---

Under review as a conference paper at ICLR 2021
Table I: Neighbourhood Distillation results on CIFAR-10 for different values of multiplier k. Our Neighbourhood Distillation method reaches the same accuracy as Knowledge Distillation (KD) and performs better than retraining from scratch (CE). As decreases, fine-tuning becomes critical in order to recover good accuracy. Our method is 2.3 faster than Knowledge Distillation on GPU.
MULTIPLIER K
# PARAMS
CE
KD
ND
FT
ND
269K
91.6
91.9
91.6
91.6
0.9
239K
91.3
91.7
91.7
91.3
0.8
213K
90.8
91.5
91.7
90.7
0.75
202K
90.9
91.6
91.7
87.6
0.6
160K
90.6
91.1
90.8
0.5
136K
91.0
88.2
0.4
105K
89.7
90.0
83.5
GPU TIME(H) SPEED-UP
3.2
1.4
0.32
1x
2.3x
10x
original training images through the first part of the teacher network T .... Ti. The compu- tational graph for Neighbourhood Distillation shown in Figure 2a These activation maps can be pre-computed and stored before training. This makes each student neighbourhood extremely fast to train, as one does not need compute activations through big model in each training step.
Different student neighbourhoods can be distilled independently from each other and composed into final student model S Sn S1. The final student model is then fine-tuned by optimizing the Knowledge Distillation loss from Equation 3
LKD(x,y) CC . ALCEly,
T(x)
87 softmax
where LCE is the standard cross-entropy loss.
While the idea of minimizing the mean-square error between output features is similar to Hint- Training (Romero al. 2014), our method is more general as we do not limit ourselves to distill one prefix sub-network. Neighbourhood Distillation also provides more flexibility by enabling distilla- tion many sub-networks in parallel. Note also that although student neighbourhoods independently, we ultimately use them to build a bigger student model. As we want to be able to compose them properly, constrain the student Neighbourhood S, have the same input and output dimensions as its teacher.
distilled
We demonstrate the applicability of Neighbourhood Distillation ResNet models trained on two benchmark classification datasets: CIFAR-10 (Krizhevsky et al. 2009), and ImageNet (Deng et al.) 2009). Two different settings are considered for Neighbourhood Distillation: parameter reduction and sparsification.
Parameter Reduction We first explore the architecture space of ResNetV1 models by shrinking the number of parameters with multiplier k. shown Figure 2b On CIFAR-10, use ResNetV1 model as the teacher model where a neighbourhood is two-layer residual block. For ImageNet, we use a ResNetV1-50 model as the teacher where the neighbourhood is three-layer residual block. For both models, weights that are not affected by the multiplier (e.g: the first and last layer of the model) are kept as-is.
After the initial distillation step, we recompose the student neighbourhoods that have the same mul- tiplier into a final student model. The student model is then fine-tuned on the training set using the usual Knowledge Distillation loss until convergence. Table and Table 2 compare the final accuracies obtained for different student models trained with Cross- Entos-y loss, Knowledge Dis- tillation or Neighbourhood Distillation. We observe that Neighbourhood Distillation method yields similar result as Knowledge Distillation, and both distillation methods improve on training with Cross Cross-Entropy loss.
5

---

Under review as a conference paper at ICLR 2021
Table Distillation results on ResNet-50 for neighbourhoods of 2 and layers with different mul- tipliers. Neighbourhood Distillation (ND) performs similarly to Knowledge Distillation (KD) and better than retraining from scratch (CE). As the gap between the student and the teacher increases, fine-tuning becomes necessary to recover good model accuracy. Our method is 3.6x faster.
MODEL(E-NUM LAYERS)
PARAMS
CE
KD
ND
ND
RN50-1.0
23
76.2
RN50-0.75-2
M
73.4
76.0
76.3
72.8
RN50-0075-3
75.1
76.1
64.7
RN50-0.50-2
73.6
74.9
74.9
68.8
GPU TIME (H)
200
56
22.75
SPEED-UP
3.6x
8.8x
Sparsification We apply Neighhourhood Distillation with the goal of distilling a sparse student from fully-trained teacher network. During distillation, the student weights sparsified by mag- nitude pruning (Zhu & Guptal 2017) without changing the input or output shape. This allows us to consider one-layer neighbourhoods and initialize the student weights using the teacher's learned weights. We show in Table 3 that Neighbourhood Distillation is able to reach the same performace as Knowledge Distillation for low sparsity rates, without fine-tuning. For high sparsity rates, the sudden drop in accuracy for models distilled with Neighbourhood Distillation is symptom of the thresholding effect. When the target sparsity too high, the student doesn't have enough capacity to approximate its teacher well enough.
Timing The main advantage of Neighbourhood Distillation is the speed-up that be gained from training small blocks in parallel. We timed Neighbourhood Distillation and compare its run- time that of Knowledge Distillation and report the total GPU time for each, i.c. the total time that would have been needed for one GPU P100 to run. Results are reported in Table 2 and Neigh- bourhood Distillation is 3.6x faster on ImageNet and 2.3x on CIFAR-10 for parameter reduction, and 1.7x faster for sparsification. For parameter reduction, the end-to-end tne-tuning step is the main computational bottleneck but is faster than using Knowledge Distillation from scratch. Stu- dent neighbourhoods learn a meaningful approximation of their respective teachers, which allows fine-tuning step to converge faster than Knowledge Distillation. On the sparsification task, while don't fine-tune and each layer is fast to sparsify, the total number of layers (50) is what consti- tutes to main bottleneck when computing the total GPU time the procedure. further break-out of the time each phase of Neighbourhood Distillation needs is given in the appendix.
Table 3: Classification accuracies and GPU Time comparison be- tween Neighbourhood Distillation and Knowledge Distillation.
Table 4: Classification accuracies for different data-free dis- tillation methods. Different values of & are used for the stu- dent architecture including A: (self-distillation). We compare accuracies obtained after different training meth- ods: fully-supervised cross-entropy loss (CE), Gaussian Noise Neighbourhood Distillation (G-ND), Gaussian Noise Neighbourhood Distillation with Gaussian Noise Finetuning (G-ND+FT) Gaussian Noise Knowledge Distillation (GN- KD) and Zero-Shot Knowledge Distillation (Nayak et al,
Different values of target sparsities are used for the student architec-
ture.
SPARSITY
ND
KD
2019) (ZSKD). Gaussian Noise Neighbourhood Distillation
0.1
76.00
75,63
surpasses other data-free distillation methods we com-
0.2
76.03
75.94
pared to.
0.3
75.82
75.69
0.4
75.63
75,45
0.5
75.26
75.20
k
CE
G-ND
G-ND+FT
GN-KD
ZSKD
0.6
73.78
74.60
91.60
91.51
91.51
16.00
21.50
0.7
69.52
69.61
0.9
91.31
88.78
88.78
12.97
22.77
0.8
49.83
70.33
0.75
90.89
80.64
83.74
13.00
21.54
0.9
0.60
0.10
0.5
90.17
53.40
54.83
12.74
22.17
GPU TIME (H)
144
250
SPEED-UP
1.7x

---

Under review as a conference paper at ICLR 2021
5 STUDENT SEARCH
Usually, when performing Knowledge Distillation, the Student network is variant of the Teacher network with a smaller number of parameters (Hinton et al.) 2015 Romero et al.) 2014] /Zhang al.) (2017). Other candidate architectures with the same number of parameters could better capture the teacher's knowledge but searching for the ideal student architecture would require retraining different models from scratch and be computationally expensive. Here, we demonstrate how inde- pendently distilled neighbourhoods may be re- purposed for architecture search. Student Search, our architecture search method, the distilled neighbourhoods to make local decisions model's architectural design.
Formally, we consider C, the set of possible candidates for given neighbourhood S,. This could contain variants the same architecture with different parameters like the size of the inter- mediate layers, the level of sparsity, the non-linearity used. The number possible student model architectures to distill and evaluate would be I[ [C,]- Student Search leverages Neighbourhood Distillation by first distilling all Σ |Csl neighbourhoods completely independently. It then se- lects a candidate for each neighbourhood by solving constrained optimization problem that seeks to minimize the size of the student while maximizing the quality of the selected candidates. The selected candidate neighbourhoods are then combined together form the Student Network, which may then be fine-tuned. In practice, we find that solving the above optimization problem approxi- mately with greedy approach is enough yield good student architecture.
We demonstrate our method on the space of ResNet variants built with different multipliers. For each neighbourhood, we independently distill 10 candidates, obtained by varying the bottleneck multiplier l: in (0.1, 0.2, 1.0]. The quality of candidate (;(k) is measured by the accuracy of the partial model T, o T o (k) 7:-1 o Ti. We solve the optimization problem greedily by selecting for each neighbourhood the smallest candidate that leads an accuracy drop of less than x 2%.
76.5
76.0
Accuracy
90
Accuracy
75.5
88
Student Search (NDistill + FT)
Student Search (NDistill FT)
Width Multipliers (KD)
75.0
Width Multipliers (KD)
Width Multipliers (CE)
74.5
Width Multipliers (CE)
86
Bottleneck Multipliers (KD)
Bottleneck Multipliers (KD)
74.0
100k
150k
200k
250k
17M
20M
22M
25M
#Params
# Params
(a) ResNetVV-1 Student Search on CIFAR-10
(b) ResNetV1-50 Student Search on ImageNet
Figure 3: Trade-off curve between number of parameters and final accuracy for ResNetV1-20 and ResNetV1-50 models. Models found with Student Search are obtained by recombining units of different sizes that were distilled independently with Neighbourhood Distillation. Width multipliers refers to applying a uniform multiplier on all blocks. Bottleneck multipliers refers to applying a uniform multiplier the inner layers of all blocks. Student Search efficiently finds better student architectures than naively applying uniform transformations to the teacher.
Figure 3a and 3h show how our Student Search compares to using uniform multiplier on all lay- ers when searching for student architecture for Knowledge Distillation. Width multiplier refers to applying the same multiplier to all the bottleneck layers of the model. This setting cannot be explored with Neighbourhood Distillation bourhoods. Bottleneck multiplier refers to applying the same multiplier to the inner layers of each residual block: this setting is equivalent to applying Neighbourhood Distillation with the same mul- tiplier i for all neighbours. Student Search efficiently finds a better student model for distillation than uniformly applying the same transformations the teacher model. In particular, the found stu-
it modifies the input/output behavior of the neigh-

---

Under review as a conference paper at ICLR 2021
dent architecture can be directly fine-tuned using the student candidates trained by Neighbourhood Distillation.
6 DATA-FREE KNOWLEDGE DISTILLATION
Distillation in a context where the original dataset is not accessible is of critical interest for privacy- sensitive applications. Methods for distillation in this setting essentially rely on generating synthetic image datasets, which are usually obtained by directly optimizing some imput noise with regards to a pre-determined loss (Lopes al. 2017] Nayak et al. 2019 Bhardwaj et al. 2019 Yin et al. 2020)_ Generating a substitute dataset using these methods is costly and have shown limited success on very deep architectures trained complex and large-scale datasets. Previous work (Haroush al.) 2019) show that it is possible to successfully distill from Gaussian Noise in limited settings (fine- tuning or calibrating quantized models) where the gap between the student and the teacher weights is small. By switching the focus on small and shallow sub- uub-networss, we show that Neighbourhood Distillation can successfully adapted use gaussian noise inputs to distill models from scratch.
To that end, we distill ResNet-20 with bottleneck multiplier A = (1.0, 0.9, 0.75, 0.5) using only gaussian noise as the neighbourhood inputs. We compare our method with Zero-Shot Knowl- edge Distillation (Nayak et al., [2019), data-free method that generates a synthetic training set by optimizing noise inputs with gradient descent. To highlight the benefits of training on shallow sub-networks, we also attempt to use gaussian noise inputs to distill the entire student network end- to-end. Table shows that Neighbourhood Distillation outperforms both end-to-end baselines but shows degrading performance the compression rate decreases.
These results show that distillation in a data-free regime can benefit from looking at shallow sub- networks in isolation. As deep neural networks model complex and high-dimensional functions, it is not possible to distill them successfully from unstructured gaussian noise and generally requires generating complex inputs that simulate images. On the other hand, shallow networks represent less complex functions and we have demonstrated with Neighbourhood Distillation that it is easier to approximate them using gaussian noise. Notably, when neighbourhoods are only two-layer deep like in the ResNet-20 model, the function is simple enough that it can be distilled almost perfectly using gaussian noise when k
7 DISCUSSION
In this paper, we presented an approach to distillation that breaks away from end-to-end training. We demonstrated that distilling small neighbourhoods yields many advantages compared to traditional end-to-end distillation: we free ourselves from the computational limitations that come with train- ing very deep models and speed-up distillation by exploiting parallelism. Additionally, the trained candidates can be reused to efficiently explore an exponential number of local architecture changes with Student Search. Finally, we showed that distilling on small neighbourhoods allows us to easily distill in no-data context by generating synthetic inputs from gaussian inputs.
We also discovered interesting phenomenon in deep neural networks that empirically justifies the idea of Neighbourhood Distillation. The thresholding effect explains why replacing all neigh- bourhoods by distilled students can yield a reconstructed student with reasonable performance. also enables efficient diagnosis for cases where the student is too small to properly approximate the teacher and is not suited for Neighbourhood Distillation.
Moving forward, understanding the thresholding effect is one key to broadening the scope of Neigh- bourhood Distillation. Studying what impacts the model tolerance threshold could better inform how to limit failure cases and we present in the appendix preliminary results towards that direction. More generally, our paper highlights the benefits of distilling networks in a non end-to-end man- ner, reaping the benefits of training shallow networks, with the potential to revive methods such as second-order optimization methods which have been inapplicable to deep networks due to their complexity.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Chenlei Guo. Knowledge distillation from internal representations, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv: 1607.06077050 2016.
arXiv preprint
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise can scale to imagenet. CoRR, abs/1812. 1446, 2018. URL http:::/aaax.v.ot///////////////
Kartikeya Bhardwaj, Naveen Suda, and Radu Marculescu. Dream distillation: A data-independent model compression framework, 2019.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE International Conference Computer Vision, pp. 4794-480) 2019.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre. Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
Elliot J Crowley, Gavin Gray, and Amos J Storkey. Moonshine: Distilling with cheap convolutions. In Advances in Neural Information Processing Systems, pp. 2888-2898, 2018.
George E Dahl. Dong Yu, Li Deng, and Alex Acero. Context-dependent pre-traine pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on audio, speech, and language processing, 20(1):30-42, 2011.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. In Advances in neural information processing systems, pp. 1223-1231, 2012.
J. Deng. W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. CVPR09, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL
httt:://arxi: httg:/bb//1110 04805
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal machine learning research, 12(7), 2011.
Kunihiko Fukushima and Sei Miyake. Neocognitron: self-organizing neural network model for mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pp- 267-285. Springer, 1982.
Sangchul Hahn and Heeyoul Choi. Self-knowledge distillation in natural language processing, 2019.
Matan Haroush, Itay Hubara, Elad Hoffer, and Daniel Soudry. The knowledge within: Methods for data-free model compression, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv: 0oa.:t5033302225 2015.

---

Under review as a conference paper at ICLR 2021
Sepp Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Basse Systems, 6: 107-116, 04 1998. doi: 10.1 10.114225021884858980000099
Furong Huang, Jordan T. Ash, John Langford, and Robert E. Schapire. Learning deep resnet blocks sequentially using boosting theory. CoRR, abs/1706.04900 2017. URLattt:://zzix abs/ 0bs//706... 04964
Seung Hyun Lee, Dae Ha Kim, and Byung Cheol Song. Self-supervised knowledge distillation using singular value decomposition. In The European Conference on Computer Vision (ECCV), September 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:/502.083 1502.03167, 2015.
Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu. Knowledge distillation via route constrained optimization. CoRR, abs/1904.09149,, 2019. URL
owg. aaaaaasss199 09149
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analy- sis of tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 1-12, 2017.
Minsoo Kang, Jonghwan Mun, and Bohyung Han. Towards oracle knowledge distillation with neural architecture search, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6900 2014.
Alex Krizhevsky, Ila Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-23 23222224, 1998.
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation.
In Joint european conference on machine learning and knowledge discovery in databases, pp. 498-515. Springer, 2015.
Changlin Li, Jiefeng Peng, Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xi- aojun Chang. Block-wisely supervised neural architecture search with knowledge distillation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp-
2020.
Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan L. Yuille, Jonathan Huang, and Kevin Murphy, Progressive neural architecture search. CoRR, abs//111..00533 2017. URL http://aaxxv..rrg//////22...... 00559
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint 1806
2018.
Yu Liu, Xuhui Jia, Mingxing Tan, Raviteja Vemulapalli, Yukun Zhu, Bradley Green, and Xiaogang Wang. Search to distill: Pearls are everywhere but not the eyes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7539-7548, 2020.
Raphael Gontijo Lopes, Stefamo Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv: 2017.
10

---

Under review as a conference paper at ICLR 2021
Sindy Löwe, Peter O' O'Connor, and Bastiaan S. Veeling. Greedy infomax for biologically plausible self-supervised representation learning. CoRR, abs/1905.11786, 2019. URL http: http:://tp/aa:t arxiv. org/abs 1a0pa///995..111888
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, and Hassan Ghasemzadeh. Improved knowl- edge distillation via teacher assistant: Bridging the gap between student and teacher. arXiv preprint arXiv:1902.03393, 2019.
Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj. R Venkatesh Babu, and Anir- ban Chakraborty. Zero-shot knowledge distillation in deep networks. arXiv preprint arXiv:1905.081 14, 2019.
Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. arXiv preprint arXiv: I 111.0001.000 (001:0069000066 2019.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient prob- lem. CoRR, abs/ 1,5063, 2012. URL URL.http://arxxii //arxiva.ivvv//aa//aa wtwpiivovrrrraaaeoe absp///b///// 1211, 50
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. CoRR, abs/1802.03266. 2018. WRLL:
1802. 03269
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. CoRR, abs/1506.01497, 2015. URL http:
aaaarivv //axxi..rr/aabs. /abs hta.r//gab//55555 /150....//111115333000 01497
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, 323(6688):533-53, 1986.
Tim Salimans and Diederik Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. CoRR, abs/1602.07868, 2016. URL http://
arxiv. org/ abs / 602.0022.8870 07868
A Senior, Vanhoucke, P Nguyen, T Sainath, et al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal processing magazine, 2012.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv: 2016.
Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint 19X.vv:905..1990 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998- 6008, 2017.
Andreas Veit, Michael J. Wilber, and Serge J. Belongie. Residual networks are exponential en- sembles of relatively shallow networks. CoRR, abs/1605.06431, 2016. URL pRtp::/araa org/ Drg/abs/Iee 06431
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
Paul Viola and Michael Jones. Rapid object detection using boosted cascade of simple features. In Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001, volume pp. I-I. IEFE, 2001.
11

---

Under review as a conference paper at ICLR 2021
Junho Yim, Donggyu Joo, Jihoon Bae. and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp.
2017.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Ni- raj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 8715-8724, 2020.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint 1.XX::1612.03928, 2016.
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal?, 2019.
Zhi Zhang, Guanghan Ning, and Zhihai He. Knowledge projection for deep neural networks. arXiv preprint arXiv:1710.09500 1750000500 2017.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01877. 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. CoRR, abs/1707.0700 2017. URL http://arxiv..rrg
abs/1700.07011
A EXPERIMENTAL DETAILS
Data preprocessing The CIFAR-10 is standardized with per-channel train statistics and aug- mented at train time with random translations of 4 pixels.
ResNetV1-20 The teacher model is trained with standard cross-entropy loss with batch size 128 for 96k steps. We used Momentum Optimizer with a momentum of 0.9. The learning rate was increased linearly from 0.01 to 0.1 for 400 steps then decayed exponentially by a factor of 10 every 32k steps. The model regularized with L2 weight decay of 1e-4. The same settings were used for distillation. For Knowledge Distillation, we set À 1.0, and used random search on the parameter [1., 20) and learning rate in [1e-6 1.J. For Neighbourhood Distillation, we distill for 200k iterations with learning rate of le 1e-3. Finetuning is done with learning rate le 2, 2.5.
ResNetV1-50 The teacher is trained for 200 epochs with batch size 32, 12 weight decay of le - and Momentum optimizer. The learning rate is initialized 0.02 and decayed exponentially with factor 0.2 every 30 epochs.
Sparsification We use the Tensorflow Model Optimization Toolkit to sparsify our neighbour- hoods. For each neighbourhood, we train for 40k steps. The sparsification rate is ramped-up to the target sparsity rate with a polynomial decay schedule for 20k steps and then maintained for another 20k steps.
B ADDITIONAL RESULTS
B. THRESHOLDING EFFECT
We conducted an additional experiment to show how accuracy can be impacted by direct perturba- tion network's 's weights. For each comvolution layer kernel, we choose to add noisy perturbation with varying standard deviations. We then verify whether the perturbations have additive effect on the final accuracy by replacing weights from the original network by their perturbed version. Figure 4 shows how errors the CIFAR-10 10 test set accumulate when progressively replacing the layers of pre-trained ResNetV1 model, starting from the first layer. The figure compares that to the hypothetical additive regime. We see that when constrain individual layers cause small
12

---

Under review as a conference paper at ICLR 2021
Table 5: Impact of regularizing the teacher model on Neighbourhood Distillation. Accuracies are obtained before fine-tuning. Using regularized teacher improves the accuracy of the student before fine-tuning.
STUDENT
TEACHER
NDISTILL ACCURACY
CHANGE
RN50-0.75-3
RN-50
57.63
+0.%
RN-50+N = 0.1)
63.21
+5.6%
RN50-0.5-3
RN-50
25.7
+0.%
RN-50+NV(o= = 0.1)
32.9
+7.2 %
drops in accuracy (< 0.1%). the errors accumulate sub-linearly. On the other hand, when individual layers are perturbed much that they cause a drop of 2% in accuracy each, the errors accumulate super-linearly.
1.00
0.15
Acquracy Drop
Accurocy Drop
0.75
0.50
0.25
0.05
0.00
0.00
Perturbed weight
Perturbed weight
(a) Individual Drop < 0.1%
(b) Individual Drop 1%
80.00
Drog
60.00
Accumulated Emar
40.00
Cumulative sum of errors Individual Error
20.00
Perturbed weight
(c) Individual Drop 2%
Figure 4: Error accumulation caused by perturbing network layers. The bar plot shows the individual accuracy drop for network with perturbed weights for one layer. The blue line plot shows the cumulative sum of errors caused by perturbing a given number of layers from left to right. The black line plot shows the empirical error accumulation evaluated on the test set after perturbing given number of layers. (a) Individual accuracy drop of each layer is < 0.1% and the eπrоrs accumulate sub-linearly. (b) Individual accuracy drop of each layer 1% and errors accumulate linearly. (c) Individual accuracy drop of each layer is high (~~ 2%) and the errors accumulate super-linearly.
We show that a simple regularization technique applied during training can increase a network's resilience to local perturbations. Considering ResNetV1-50, add gaussian noise drawn from (0, a) at train time after every residual unit. Once these models are trained, we replace each units with perturbed versions again and report the drop in accuracy. Figure 5 shows that for high noise levels introduced between residual units at train time, the perturbation threshold is higher, although regularizing too much also degrades the overall performance of the original network.
More importantly, we observe that increasing the teacher's tolerance threshold directly impacts the accuracy of models trained with Neighbourhood Distillation. We take R 1-50 trained with regularization a = 0.1 as the teacher and consider bottleneck multipliers k = 0.5 and & = 0.75 for the student neighbourhoods.
B.2 NEIGHBOURHOOD DISTILLATION
Look-ahead losses using look-ahead losses. In addition to forcing the output of S, to be close to the output of T, feed each to the next teacher Neighbourhood T and minimize the Mean Square Error of its
In order to provide additional training signals to the student, also consider
13

---

Under review as a conference paper at ICLR 2021
Thresholding effect for different perturbation amplitudes E.
Accuracy change
0=0.00, acc=75.70
0=0.01, acc=76.03
-20
a00.10,
-40
a=0.20, acc=75.43
0=0.30, acc=74.57
-60
0=0.40, acc=73.85
-80
0=0.50, acc=72.95
0.00
0.25
0.50 0.75
1.00
0=1.00, acC=69.26
E
Figure 5: Drop in accuracy after perturbing diffferent ResNetV 1-50 models trained with Gaussian Noise (0, or) introduced after each unit. is the amplitude of perturbations introduced at test time. Original accuracies of the models before perturbation given in the caption. Models trained with high gaussian noise amplitudes a have higher resistance threshold to test time perturbations.
Table 6: Ablation Study. Impact of fine-tuning (FT) and look-ahead (2-LA) on final test accuracy for ResNet-20. While fine-tuned model always perform better than pre-finetuned models, the use of look-ahead tightens the gap between pre and post finetuned accuracy.
MULTIPLIER
NDISTILL
NDISTILL FT
2-LA NDISTILL
2-LA NDISTILL +
0.90
91.1
92.0
91.4
92.1
0.75
89.7
91.9
90.6
91.8
0.50
85.7
91.3
87.1
91.5
output:
Li, F+2 Ea-p(a) ((T()) - (T+1(T((a)) [
(4)
a a= R2(x) (z), x Darain
The look-ahead loss is shown in Figure 2a These losses can be combined to form total loss of:
C. = C.o +
CfCis
(5)
where a, are hyyer-parametern While lookahead losses may improve the accuracy of the distilled model by providing additional training supervision, it also leads to increased computational cost due to needing to propagate inputs and gradients through a higher number of layers.
Ablation Study We perform ablation study the CIFAR-10 model to show how look-ahead and fine-tuning impact our Neighbourhood Distillation results. We experiment with combining two look-ahead losses, i. Vj > 3, a, 0 and train each neighbourhood either without look-ahead or with two-neighbourhood lookahead. Models are evaluted before and after fine-tuning and test accuracies are reported in Table We observe that training neighbourhoods with look-ahead leads to better accuracy before fine-tuning. Fine-tuned models always perform better than their non-fing finefie- tuned counterparts. While there is no significant difference between the no look-ahead and the 2- look-ahead fine-tuned models, we also noticed that neighbourhoods distilled with look-ahead losses converge faster during the fine-tuning step 30k iterations instead 50k. Look-ahead losses can therefore be used to balance the cost of Neighbourhood distillation and the cost of fine-tuning,
B.3 STUDENT SEARCH
Motivation We observe empirically on the ResNet-20 and ResNet-50 models that there no rea- to think that applying the same multiplier all layers would yield the best compressed student
14

---

Under review as a conference paper at ICLR 2021
Table Comparison of training time in GPU hours for ResNet models.
(a) Timing for ResNet20
(b) Timing for ResNet50
Method
Training Time (h)
Method
Training Time (h)
NDistill
lunit
0.03
NDistill
l-unit
1.42
All units sequentially
0.32
All unit sequentially
22.75
Finetuning
1.07
Finetuming
33
All units Finctuning
1.39
All unit Finetuning
56
KD
3.2
KD
200
92.00
77.00
90.67
75.11
Accuracy aracy
89.33
73.22
88.00
71.33
86.67
Accur
69.44
1-1
85.33
1-1
67.56
3-1
84.00
2-2
65.67
3-2
82.67
63.78
81.33
3-2
61.89
4-2
80.00
0.2
0.4
0.6
0.8
1.0
60.00
0.2
0.4
0.6
0.8
Bottleneck
Bottleneck
(a) ResNet-20 on CIFAR 10
(b) ResNet-50 on ImageNet
Figure 6: Impact of applying a multiplier on the distilled accuracy of different Residual Units. x-y refers to unit of block x of the ResNet. For each unit, accuracy remains unchanged as long as the multiplier remains above certain threshold. This thresholding effect can be observed for different residual units, but each unit has different threshold.
model. For a number of chosen residual units, we reduce the number of parameters by applying multiplier < on the number of filters of the first layer and report the best accuracy we can get after training this variant with Neighbourhood Distillation. Figure 6 shows the final accuracy against for different units of ResNet-20 and ResNet-50. We observe another form of thresholding effect: the accuracy drops sharply below a given value of the multiplier, showing that residual units can be compressed up certain point before hurting the network's s accuracy. Most importantly, we observe that the threshold is different for each residual unit, which suggests that better strategy for a student architecture to choose a different multiplier per unit.
70
Accuracy
NDistill
60
KD
Student Search
50
0.2
0.4
0.6
0.8
Sparsity
Figure 7: Student Search for Sparsification ResNetV 1500. Models found with Student Search obtained by recombining layers with different sparsity rates that were distilled independently with Neighbourhood Distillation. Student Search efficiently finds a better student architectures for distillation than naively applying uniform sparsity rates.
15

---

Under review as a conference paper at ICLR 2021
Teacher Neighborhood
Neighborhood
Student 1d
Trainer
Teacher Neighborhaod2
Student 2a
Stucient
Teacher Neighborhood
Student 3b
Trainer
Neighborhood
Studiett 3c
Teacher Network
Neighborhood Distillation
Student Network
Figure 8: Student Search with Neighbourhood Distillation. A Teacher Network is broken down into several neighbourhoods. Each Neighbourhood can be used to train several student neighbourhoods independently from the rest. Selected student neighbourhoods are then merged back into a single Student Network. This allows to explore a large search space without having retrain all models.
16


Reviews:
[{"review_id": "rw7ktYvPKYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies knowledge distillation in the context of parallelly training sub-networks (called neighbourhoods) instead of commonly used end-to-end training paradigm. The authors explore the applications of the proposed neighbourhoods distillation in improving sparse networks,  searching a good student structure given the teacher and knowledge distillation merely using synthetic data. Both CIFAR and ImageNet datasets are considered in the experiments.\n\nThe proposed method is interesting. My main concerns are as follows.\n\n--- Thresholding effect.\n\nThe experimental findings regarding thresholding effect is interesting. In common understanding of CNNs, non-linear activations play a critical role in training. When perturbing a CNN or its  neighbourhoods by artificially introducing some gaussian noise of amplitude epsilon to each activation output, it will break non-negative property. What is the underlying reason why it does not affect final predication once the gaussian noise of amplitude epsilon is small enough?  Why the behavior for injecting noise to network weights is different from noise injection to activations? They are not clear enough. Besides, more details on how to inject noises to a network or its neighbourhoods would be useful.\n\n--- Neighbourhood distillation (ND).\n\nIn the formulation, the authors assume the teacher is pre-trained and available. And the authors claim that ND can be performed in a much faster manner, compared to conventional KD. However, to my understanding, it is not clear enough.\n \na) From Figure 2, I have not seen obvious differences against conventional KD or intermediate feature/attention maps guided KD, if we consider a subnetwork as the student.\n\nb) The authors claim \"These activation maps can be pre-computed and stored before training. This makes each student neighbourhood extremely fast to train, as one does not need to compute activations through a big model in each training step.\" This can also be used to conventional KD or intermediate feature/attention maps guided KD, if we consider a sub-network as the student.\n\nc) The authors claim \"Different student neighbourhoods can be distilled independently from each other and composed into\na final student model\" and then the final model is fine-tuned. How about computational resource cost when training them parallelly? Does the speed-up is also benefited from parallel training?\n\nd) In the equations such (1) and (3), many notations or terms are not clarified.\n\nI am confused by those points, which makes me think the presentation and applications of ND are not convincing.\n\n--- Experiments\n\nIn the experiments, the authors assume the student and teacher networks have strong relations, multiplier k. However, convention KD and its variants do not have any structural constraints between the student and teacher networks. That is, they usually have different structures, such as DenseNets vs. ResNets.\n\nKD methods are not limited to two-stage training, as many KD variants enable on the fly training of teacher/peer together with the student. Is it possible to apply ND to such kind of training scenarios?\n\nRegarding speed-up results over KD, it lacks details how they are implemented/measured. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "e65ePnScK6N", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces Neighbourhood Distillation (ND), a new training pipeline for knowledge distillation (KD), which splits the student network into smaller neighbourhoods and trains them independently. The authors breaks away from the end-to-end paradigm in previous KD methods and provides empirical evidence to reveal feasibility and effectiveness of ND. Specially, ND can: 1) speed up convergence, 2) reuse in neural architecture search and 3) adapt to the synthetic data.\n\nStrengths\n1) The paper is well written and easy to follow. An empirical evidence of the thresholding effect is provided to explain the motivation.\n2) The idea is simple and intuitive. The ND seems more like an initialization method for DNN’s local components and a finetuning procedure is sometimes needed for recovering the accuracies. Benefit from parallelism and small training components, such training schema can speed up the convergence of standard KD.\n3) Several different applications are conducted to demonstrate the flexibility of ND.\n\nWeaknesses\n1) Missing a relevant paper. [1] proposes a similar blockwise knowledge distillation method. The authors should cite and explain the differences between ND and [1].\n2) What is sparsification in Sec. 4? Is it the sparseness of the convolutional kernel or the channel? \n3) The authors mention that the work seeks to overcome the limitations of training very deep networks. However, the ResNet50 (the deepest model in experiments) is not deep enough. Usually, it is easy to converge.  \n4) In Sec. 5, only the width search experiments are conducted, which is more like layer-wise or block-wise pruning. However, architecture search is a general method that can not only search the widths but also the operations. Why only mentions “This set could contain variants of the same architecture...”? Is there any limitation when the searched candidates contain different architectures/operations? \n5) All the experiments are done on ResNet series. Different teacher and/or student architectures, such as VGG, ShuffleNet etc., should be considered.\n6) Does the observation of thresholding effect benefit from the shortcut in Resblok？Is it suitable for plain CNN, such as VGG? Which blocks are chosen in Fig. 1(a)? Does the shallow and deep blocks have the same phenomenon when perturb small number of blocks?\n7) How to record the GPU time of ND? Is it the time of paralleling on multi-GPUS or on single GPU?\n\nI am currently leaning towards a slightly negative score but would like to see the authors' responses and other reviewer's comments.\n\n\n[1] Hui Wang, Hanbin Zhao, Xi Li, Xu Tan. Progressive Blockwise Knowledge Distillation for Neural Network Acceleration. IJCAI, 2018.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LxZ5GQDHa0m", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I vote for rejection at this moment. This paper proposes a new knowledge distillation method called Neighbourhood Distillation to speed up knowledge distillation process. Compare to conventional KD approaches, this paper can achieve 3-4 times speed and obtain similar performance. Experiment on several benchmarks shows the proposed method can be beneficial in several application.\n Overall, I think the idea to speed up the process of knowledge distillation is a direction that could be discussed in the future,  while the training cost of current KD in experiments in this paper is almost affordable and the speedup is not very impressive. Besides, several concerns need to be addressed.\n\n####Pros:\n1) The observation of  Thresholding effect is somewhat novel to me.\n2) The methodology part is clear and easy to understand.\n3) This paper has investigated the ND in several tasks, including NAS, data-free KD,  and this is worth to mention. \n\n#### Cons:\n1)  For me, the connection in Thresholding Effect and NEIGHBOURHOOD DISTILLATION is a little weak. I noticed the author claim this motivated to use NEIGHBOURHOOD DISTILLATION: each student neighbourhood is trained to match its output of teacher. Such an idea is already used similarly way in Hint-Training and the following work.\n2)  The goal of this paper is to speed up the KD process, however existing cost of KD experiments in the paper is affordable. And the speed up is not very impressive.\n3)   The paper should add discussion some recently KD methods, especially matching intermediate features between student and teachers.\n4)  The results of Table 1 should be run several times and report the average mean.\n5)  The GPU time (H) is not a very objective index, could you provide the flops of each method?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a layer-wise or block-wise distillation scheme, Neighbourhood Distillation, that aims to reduce the training time and to improve parallelism when distilling large teacher networks. By breaking down the end-to-end distillation objective into blocks, the proposed method enables faster distillation when applied to model compression and block-wise architecture search. Several concerns of reviewers were addressed during the rebuttal period by the authors.

However, there are still some concerns among the reviewers after the discussion:

1) Computational cost-benefit of the proposed KD method seems marginal in comparison to the baseline, given that we still have to train all the neighborhoods in parallel and potentially to fine-tune in the end. 

2) It was brought up by a couple of the reviewers that the experiments lack diversity. It would be great to a clearly defined metric applied to a wide variety of the model architectures and datasets. It will also strengthen the paper by providing more details on the experiments.

The basic idea is interesting, but the paper needs further development and modification for publishing. 


Decision: Reject