Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies knowledge distillation in the context of parallelly training sub-networks (called neighbourhoods) instead of commonly used end-to-end training paradigm. The authors explore the applications of the proposed neighbourhoods distillation in improving sparse networks,  searching a good student structure given the teacher and knowledge distillation merely using synthetic data. Both CIFAR and ImageNet datasets are considered in the experiments.

The proposed method is interesting. My main concerns are as follows.

--- Thresholding effect.

The experimental findings regarding thresholding effect is interesting. In common understanding of CNNs, non-linear activations play a critical role in training. When perturbing a CNN or its  neighbourhoods by artificially introducing some gaussian noise of amplitude epsilon to each activation output, it will break non-negative property. What is the underlying reason why it does not affect final predication once the gaussian noise of amplitude epsilon is small enough?  Why the behavior for injecting noise to network weights is different from noise injection to activations? They are not clear enough. Besides, more details on how to inject noises to a network or its neighbourhoods would be useful.

--- Neighbourhood distillation (ND).

In the formulation, the authors assume the teacher is pre-trained and available. And the authors claim that ND can be performed in a much faster manner, compared to conventional KD. However, to my understanding, it is not clear enough.
 
a) From Figure 2, I have not seen obvious differences against conventional KD or intermediate feature/attention maps guided KD, if we consider a subnetwork as the student.

b) The authors claim "These activation maps can be pre-computed and stored before training. This makes each student neighbourhood extremely fast to train, as one does not need to compute activations through a big model in each training step." This can also be used to conventional KD or intermediate feature/attention maps guided KD, if we consider a sub-network as the student.

c) The authors claim "Different student neighbourhoods can be distilled independently from each other and composed into
a final student model" and then the final model is fine-tuned. How about computational resource cost when training them parallelly? Does the speed-up is also benefited from parallel training?

d) In the equations such (1) and (3), many notations or terms are not clarified.

I am confused by those points, which makes me think the presentation and applications of ND are not convincing.

--- Experiments

In the experiments, the authors assume the student and teacher networks have strong relations, multiplier k. However, convention KD and its variants do not have any structural constraints between the student and teacher networks. That is, they usually have different structures, such as DenseNets vs. ResNets.

KD methods are not limited to two-stage training, as many KD variants enable on the fly training of teacher/peer together with the student. Is it possible to apply ND to such kind of training scenarios?

Regarding speed-up results over KD, it lacks details how they are implemented/measured. 
review_title: An interesting work, but lacking details 

final_score: rating: 4: Ok but not good enough - rejection
review_content: I vote for rejection at this moment. This paper proposes a new knowledge distillation method called Neighbourhood Distillation to speed up knowledge distillation process. Compare to conventional KD approaches, this paper can achieve 3-4 times speed and obtain similar performance. Experiment on several benchmarks shows the proposed method can be beneficial in several application.
 Overall, I think the idea to speed up the process of knowledge distillation is a direction that could be discussed in the future,  while the training cost of current KD in experiments in this paper is almost affordable and the speedup is not very impressive. Besides, several concerns need to be addressed.

####Pros:
1) The observation of  Thresholding effect is somewhat novel to me.
2) The methodology part is clear and easy to understand.
3) This paper has investigated the ND in several tasks, including NAS, data-free KD,  and this is worth to mention. 

#### Cons:
1)  For me, the connection in Thresholding Effect and NEIGHBOURHOOD DISTILLATION is a little weak. I noticed the author claim this motivated to use NEIGHBOURHOOD DISTILLATION: each student neighbourhood is trained to match its output of teacher. Such an idea is already used similarly way in Hint-Training and the following work.
2)  The goal of this paper is to speed up the KD process, however existing cost of KD experiments in the paper is affordable. And the speed up is not very impressive.
3)   The paper should add discussion some recently KD methods, especially matching intermediate features between student and teachers.
4)  The results of Table 1 should be run several times and report the average mean.
5)  The GPU time (H) is not a very objective index, could you provide the flops of each method?

review_title: Official Blind Review #4

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces Neighbourhood Distillation (ND), a new training pipeline for knowledge distillation (KD), which splits the student network into smaller neighbourhoods and trains them independently. The authors breaks away from the end-to-end paradigm in previous KD methods and provides empirical evidence to reveal feasibility and effectiveness of ND. Specially, ND can: 1) speed up convergence, 2) reuse in neural architecture search and 3) adapt to the synthetic data.

Strengths
1) The paper is well written and easy to follow. An empirical evidence of the thresholding effect is provided to explain the motivation.
2) The idea is simple and intuitive. The ND seems more like an initialization method for DNN’s local components and a finetuning procedure is sometimes needed for recovering the accuracies. Benefit from parallelism and small training components, such training schema can speed up the convergence of standard KD.
3) Several different applications are conducted to demonstrate the flexibility of ND.

Weaknesses
1) Missing a relevant paper. [1] proposes a similar blockwise knowledge distillation method. The authors should cite and explain the differences between ND and [1].
2) What is sparsification in Sec. 4? Is it the sparseness of the convolutional kernel or the channel? 
3) The authors mention that the work seeks to overcome the limitations of training very deep networks. However, the ResNet50 (the deepest model in experiments) is not deep enough. Usually, it is easy to converge.  
4) In Sec. 5, only the width search experiments are conducted, which is more like layer-wise or block-wise pruning. However, architecture search is a general method that can not only search the widths but also the operations. Why only mentions “This set could contain variants of the same architecture...”? Is there any limitation when the searched candidates contain different architectures/operations? 
5) All the experiments are done on ResNet series. Different teacher and/or student architectures, such as VGG, ShuffleNet etc., should be considered.
6) Does the observation of thresholding effect benefit from the shortcut in Resblok？Is it suitable for plain CNN, such as VGG? Which blocks are chosen in Fig. 1(a)? Does the shallow and deep blocks have the same phenomenon when perturb small number of blocks?
7) How to record the GPU time of ND? Is it the time of paralleling on multi-GPUS or on single GPU?

I am currently leaning towards a slightly negative score but would like to see the authors' responses and other reviewer's comments.


[1] Hui Wang, Hanbin Zhao, Xi Li, Xu Tan. Progressive Blockwise Knowledge Distillation for Neural Network Acceleration. IJCAI, 2018.

review_title: A new training schema for KD with various applications, but inadequate experiments

Meta-review: The paper proposes a layer-wise or block-wise distillation scheme, Neighbourhood Distillation, that aims to reduce the training time and to improve parallelism when distilling large teacher networks. By breaking down the end-to-end distillation objective into blocks, the proposed method enables faster distillation when applied to model compression and block-wise architecture search. Several concerns of reviewers were addressed during the rebuttal period by the authors.

However, there are still some concerns among the reviewers after the discussion:

1) Computational cost-benefit of the proposed KD method seems marginal in comparison to the baseline, given that we still have to train all the neighborhoods in parallel and potentially to fine-tune in the end. 

2) It was brought up by a couple of the reviewers that the experiments lack diversity. It would be great to a clearly defined metric applied to a wide variety of the model architectures and datasets. It will also strengthen the paper by providing more details on the experiments.

The basic idea is interesting, but the paper needs further development and modification for publishing. 


Decision: Reject