Discovering and Overcoming Limitations of Noise-engineered Data-free Knowledge Distillation
Authors: Piyush Raikwar
Deepak Mishra
Venue: NeurIPS 2022 Accept (2022)
Keywords: Knowledge distillation
Gaussian noise
Batch normalization

Abstract: Distillation in neural networks using only the samples randomly drawn from a Gaussian distribution is possibly the most straightforward solution one can think of for the complex problem of knowledge transfer from one network (teacher) to the other (student). If successfully done, it can eliminate the requirement of teacher's training data for knowledge distillation and avoid often arising privacy concerns in sensitive applications such as healthcare. There have been some recent attempts at Gaussian noise-based data-free knowledge distillation, however, none of them offer a consistent or reliable solution. We identify the shift in the distribution of hidden layer activation as the key limiting factor, which occurs when Gaussian noise is fed to the teacher network instead of the accustomed training data. We propose a simple solution to mitigate this shift and show that for vision tasks, such as classification, it is possible to achieve a performance close to the teacher by just using the samples randomly drawn from a Gaussian distribution. We validate our approach on CIFAR10, CIFAR100, SVHN, and Food101 datasets. We further show that in situations of sparsely available original data for distillation, the proposed Gaussian noise-based knowledge distillation method can outperform the distillation using the available data with a large margin. Our work lays the foundation for further research in the direction of noise-engineered knowledge distillation using random samples.

think of Discovering and Overcoming Limitations of Noise-engineered Data-free Knowledge Distillation
Piyush Raikwar" ABV-IITM,, Gwalior, India hmt_20117020111t..ee in
Deepak Mishra IIT Jodhpur, India dmishra@iitj ac.
Abstract
Distillation in neural networks using only the samples randomly drawn from a Gaussian distribution is possibly the most straightforward solution for the complex problem of knowledge transfer from one network (teacher) to the other (student). If successfully done, it can eliminate the requirement of teacher's training data for knowledge distillation and avoid often arising privacy concerns in sensitive applications such as healthcare. There have been at Gaussian noise-based data-free knowledge distillation, however, none of them offer a consistent or reliable solution. We identify the shift in the distribution of hidden layer activation as the key limiting factor, which occurs when Gaussian noise is fed the teacher network instead of the accustomed training data. We propose simple solution to mitigate this shift and show that for vision tasks, such as classification, is possible achieve performance close the teacher by just using the samples randomly drawn from Gaussian distribution. We validate our approach on CIFAR 10, CIFAR 100, SVHN, and Food101 datasets. We further show that in situations of sparsely available original data for distillation, the proposed Gaussian noise-based knowledge distillation method can outperform the distillation using the available data with a large margin. Our work lays the foundation for further rescarch the direction of noise-engineered knowledge distillation using random samples.
recent attempts
Introduction
Deep neural networks are excellent choice for various real-world computer vision tasks. However, their high computational and space complexity becomes a bottleneck when it comes to deployment resource-constraiied devices. To address this issue, the idea of Knowledge Distillation (KD), proposed by I1. has recently gained significant attention. It enables the transfer of knowledge from large neural network referred to as student, gains the information learned by the over-parameterized large model, referred as teacher, and delivers a similar sometimes even better performance (2).
comparatively smaller one. The lightweight small neural network. often
Traditionally, KD, original data is passed through the accurate teacher model to collect soft labels, and subsequently, these soft labels used supervise the student
3. However, in certain applications such as medical, often becomes impossible to release the original data due to security concerns, sensitive information, storage restrictions. In contrast, trained model (teacher) usually does not hold a specific record or specific piece of information thus, no certainty or validity of the information extracted through neural network be guaranteed, thereby making them safer for public release. However, these models certainly contain information in the parameterized form for performing the downstream tasks, such as classification or object detection.
Work done during an internship at ITT Jadhpur. Presently at CERN, Geneva, CH ppiyush.rikkwar@ccmmm
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Gasser wále Ruenig Orggial Ravin Matikko
noise
Original Running stalisiis
Gaussian noise Running stalislics
-1
(a)
(b)
Figure 1: (a) Illustration of the effect of BatchNorm state a teacher network. Each curve represents the distribution of average activations of neurons belonging to some intermediate layer in a pretrained teacher network. These are obtained when a specific type of input (original data or Gaussian noise) is fed to the teacher network under different states of BatchNorm layers. The red curve is the oracle representing activation distribution when original data is fed to the teacher. The other two curves are obtained when Gaussian noise is fed as the input, and when BatchNorm layers use the statistics of the original data (blue), and when it uses the current mini-batch statistics (black). (b) Distribution of average activation values of all neurons in 'avgpool' layer of ResNet-34 teacher observed in case of CIFAR 10 experiments.
Due to these reasons, recently there has been interest in data-free distillation - that is, distilling knowledge from one neural network to the other without using any data. Majority of methods in this direction aim to create synthetic images either by optimizing the randomly generated samples 4 5 6,78 or by training a generative network 19 10), which often is computationally expensive. This motivates the research community to ask the question Can distillation be done by using the samples randomly drawn from a known distribution such as standard normal distribution. To best of our knowledge, for the first time provide satisfactory answer to the question and show that yes, it is possible to do data-free KD by only using images that are randomly sampled from a Gaussian distribution. Our experiments show that considerably better performance is possible using only the Gaussian noise as the input, in contrast to what prior works (19), (IO) have shown for this particular setting. We identify that the key limitation, which the prior works failed to identify is the covariate shift in the distribution of hidden layer activations of the teacher network caused by feeding Gaussian noise instead of the expected original data. As shown in Figure Ta the activation distribution when input is Gaussian noise (blue) is shifted, thus no longer aligns with the activation distribution when input is original data (red). We propose a simple yet surprisingly effective solution to reduce this shift by using the current statistics instead of using statistics of the original data (running statistics) in the teacher's batch normalization (BatchNorm) layers and thus distill the knowledge from teacher into student network using just the Gaussian noise. Our main contributions are as follows:
We explain how covariate shift in the teacher network interferes with Gaussian noise-based KD and propose a simple solution to eliminate it, thus enable knowledge transfer using Gaussian noise.
We show how this phenomena of covariate shift occurs in the student network as well and mitigate it in a similar way.
We obtain student performance close to the teacher without using any original or synthetic data. We further show that in the presence of limited data, the proposed approach outperforms the traditional distillation method with large margin.

---

Our work counters the commonly accepted notion of requiring realistic training data for data-free KD. Thus, we lay the foundation for noise-based data-free KD.
2
Related work
Knowledge Distillation: KD is the transfer of knowledge from one neural network to the other. First successful implementation of knowledge transfer was shown by [[1] for the purpose of model compression. Later, the term knowledge distillation was popularized by [. where the authors explored so called "dark knowledge" of a teacher model which is transferred to a student model with the help of soft labels. Since then, the domain of KD has witnessed many interested ideas related to improved distillation strategies such as teacher assistant [12] and self-distillation [13]. KD approaches for performing various downstream tasks such as object detection [14), semantic segmentation [1], and also the applications in different modalities, including speech 16 and graph
17 have been explored. detailed survey on KD is recently reported by 2].
The problem of data-free KD, i..., the constraint that the original dataset, which was used to train and supervise the teacher, is unavailable for distillation makes the tasks naturally challenging. This often is encountered in privacy-sensitive settings, where the original dataset cannot be shared. There are various prior works with similar context in mind. Most of these methods first construct a set of synthetic images that resembles original data, then distill the knowledge of the teacher network into the student network using that synthetic data 5 6 I0] Some approaches synthesize images one-by-one or batch-by-batch, whereas some train a generator model to gain access to limitless synthetic data. For example, 4 stored activation records of the teacher model when original data is crre-foowwdded through it, and then later used it to create synthetic images by optimizing random noise using gradient descent. [5] introduced DeepInversion intending to create realistic images, which makes use of feature map and BatchNorm statistics to optimize random images into high-fidelity images. [18] takes a different approach to simultaneously generate images, as well as train the student. They introduced Zero-shot knowledge transfer, which utilized adversarial learning between the generator model and the student model. Their intuition was to search for images on which the student poorly matches the teacher and then train the student with those.
Data-free Knowledge Distillation:
Gaussian Noise- based Knowledge Distillation: KD using Gaussian noise has been presented as a baseline numerous times in prior works, which resulted in a student accuracy slightly better than random chance. Although a decent accuracy was reported on MNIST (88% (9. 75%
19 ), probably
due to smaller networks and less complex dataset, on more challenging datasets like CIFARIO, a
poor performance (14% (5). 15% D. 11% [0]. 10% [9]] was observed. Apart from these, (6] presented an interesting, although a highly constrained scenario where the student had an identical architecture to the teacher and student's 's weights were initialized as quantized values of teacher's weights. Further when the BatchNorm layer parameters from teacher are copied to student and kept frozen, Gaussian noise based distillation resulted in good performance. Moreover, instead of training the student network end-to-end as in traditional KD, [6] partitioned the student and teacher networks across BatchNorm layers resulting in multiple teacher-student pairs of blocks and distilled each pair of blocks separately. Although this approach works, it cannot be used for a pair of architectures not having similar block structures and is unnecessarily complex.
In contrast to all above, our aim is to design simple approach for Gaussian noise-based data free distillation. This would not only relax the constraints of approaches such as [6] but also save the efforts required in creating synthetic dataset.
3 Methodology
3.1 Preliminaries
Knowledge Distillation: KD is achieved by training the student network using the output of the teacher network as soft labels instead of using the ground truth labels [3]. This results in matching of the output probability distributions of both the networks for a given input data. Formally, for a k-class classification, given a dataset D that consists of data samples xr, a teacher network T trained on D, and student network S with randomly initialized weights, the following cost function is optimized:
LKD Lxe (S(x) (x)),
(I)
3

---

where Lx.(..)) .) is the standard cross- entropy term.
Batch Normalization: BatchNorm was introduced by 20 that made the training of deep neural networks more stable and allowed to use larger learning rates, hence faster convergence. Further, it makes neural networks more robust to initialization schemes and changes in the learning rate. BatchNorm first normalizes the output of the hidden layer (input to BatchNorm layer) using the first (mean) and the second (variance) statistical moments of the current batch. Then, these normalized outputs are shifted and scaled according to the two trainable parameters and 8, respectively. These parameters are tuned as any other neural network parameters, and they help in choosing the optimum distribution as the imput for the next layer, thus regaining expressiveness lost in the normalization step.
Formally, let be the activations of a neuron in the previous layer (input to the BatchNorm layer), m be the batch-size, and be small constant used for numerical stability. Then,
h h; PB
(2)
VOB +
h = 7 x h4 +
(3)
where ua = = E, h; and - shifted values of h2 Above mentioned activation normalization using mini mini-batch or current statistics results in improved training. During evaluation inference, the batch statistics μu and Ø replaced with running or population statistics, H4 = EB[uB] and o, = -- Eslon] where Es[.] denotes expectation over training mini-batches. Accordingly, the normalization steps are modified as follows:
The h; and h2 show normalized and subsequently
h; H4
h4
(4)
Vo?- +
where aur, + (1 (1- p) and a, = aa, + (1 aaaon. Here a is a smoothing factor, which is there to prioritize statistics of the current batch.
3.2 Distillation using Gaussian Noise
We consider the most straightforward approach for KD in the absence of original data, just using random images sampled from Gaussian distribution This precondition to use Gaussian noise for KD might seem to make the task of distillation almost impossible since the input Gaussian distribution is notably different from the original data distribution, and does not contain any structural properties or representation as the original data. We hypothesize that, for successful KD, across two input data: first using which the teacher is trained, and second using which student is trained, having similar output distributions throughout the teacher network's hidden layers is more important than having the same structural representation properties at the input; thus, leading to the extraction of meaningful information from the teacher network. To understand this better let us consider trivial case where samples drawn from a standard Gaussian distribution (0, 1)) are passed through a BatchNorm layer optimized for some real data D. As during distillation only student network is trained, BatchNorm performs normalization of input according to (4). Consider the following statistics of the normalized signal.
input to teacher and student network.
En [h,] H4
0 0-
(0,
(5)
oo +
Vo?+e
Var[h.] 1
Var
(6)
o7 +
o7 +E
Equation 5 and (6) clearly show the deviation covariate shift in the output of BatchNorm when Gaussian noise is passed instead of the original data. This suggest that output of BatchNorm should be shifted by Vo?+e and scaled by of + before feeding to the subsequent layers. However,
this is not trivial for intermediate hidden layers of the teacher network as the activation statistics are unknown. Alternatively, as explained below, we may allow BatchNorm to simply use current statistics instead of running statistics as the corrections required are nothing but the factors of running statistics.

---

Oriine
Nose samenidaadis
Nerrallze
Figure 2: Inside the BatchNorm layer. The figure depicts expanded view of a BatchNorm layer when each of original data and Gaussian noise is fed to it. Initially, the activation distribution of the hidden layer (input to BatchNorm) is very different in the case of Gaussian noise (black) compared to original data (red). We use the current statistics derived from Gaussian noise instead of data derived running statistics in the normalization step mitigate the covariate shift. The rest of the shifting and scaling follows as usua Our method ensures that keeping y and B constant will yield similar P(h) irrespective of the input distribution P(h).
3.2.1 Mitigating the Shift
The solution to mitigate the observed shift and successfully perform distillation using Gaussian noise to effectively make use of the BatchNorm layers. Our goal is align the activation distribution of hidden layers throughout the teacher network when Gaussian noise is used as an input to that when original data is used as an input. We consider the case of BatchNorm layer placed after hidden layer H within a teacher network. Let B represent a batch of images sampled from original dataset D, G be batch of random samples sampled from Gaussian distribution, h be the vector of activations for some neuron n belonging to the hidden layer H. when inputs are fed to the teacher with a batch size m, and P(h) the activation distribution of that n neuron over the batch. Due to the covariate shift, we observe that: P(h G) & B). Generally, during KD, the BatchNorm layers the teacher network are setup for inference, hence use running statistics (H4,,, Ty); i.e., Equation to calculate the normalized activations h. This leads to:
P(h G, My σr) 4 P.h
Mr, σr))
Thus, P(N|G M,, o. v. B) # P(h o., 7.933
Here, h is the scaled and shifted activation vector, which is the output of the BatchNorm layer for the th neuron, as defined in Equation 4 This drifted h propagates to the next hidden layer and eventually to the output layer without any meaningful information being passed from the hidden layer H. Therefore, for KD using Gaussian noise, we propose use current batch statisties (MB, an) for the normalization step, i.e., Equation (2) in all BatchNorm layers of the teacher network. The intuition is to align the two drifted distributions the normalization step. This makes sense only because 7 and B have the inherent information about the distribution that the next layer expects and are designed to shift standard normal distribution. Therefore, even though P(h[G) # P(h(B) holds true:
PB. JB) P(h|B, pB B
NN 1) (B.Y,)) (3,,,,,)
Hence, P(f|G, Mg. TB, 7, P(hiBB
This reduces the covariate shift due to different input distribution from the expected one at every stage (BatchNorm layer) in the teacher network. It obtains P(h) similar to that of using original data irrespective of Ph)), and hence leads to the transfer of meaningful information. Our approach is visually explained in Figure 2 where we show an expanded view of BatchNorm layer for three neurons. shows how our proposed method effectively uses the normalization step to align the two activation distributions: (i) when Gaussian noise is fed inputs (ii) when original data is fed as
To understand this further, consider toy example presented in the Appendix A.I.
5

---

inputs. Note that the covariate shift discussed throughout this paper is not the same as the internal covariate shift hypothesized by [20]. even though ours might be happening in the hidden layers of the neural network as well. The internal covariate shift occurs due to constantly changing network parameters the preceding layer, thus affecting the input distribution for the following layer. On the other hand, we consider the shift due to change in the input that is being fed to the network, which in turn affects the input distribution of hidden layers as well.
3.3 Inference
Similar to how a teacher network trained using original data distribution cannot directly handle the Gaussian noise, the student network trained using Gaussian noise will also not be able provide correct output during test time, as the running statistics of BatchNorm layers in the student expect Gaussian noise activation patterns. The inference is, therefore, done using current statistics. Moreover, each mini- mini-batch needs to be independent and identically distributed, and large enough to approximate the test data distribution. Usually, it should not matter how the input is arranged during the inference, therefore, simple workaround to this problem is to first adjust the rumning statistics of the student network catering to the test data by forwadiigg some amount of shuffled test data before evaluating the student network. Note that no back- back-propagate propaaattion or optimization steps are required as only the running statistics need be adjusted. We summarize the training and evaluation phases of our approach in Algorithm I and Algorithm respectively.
Algorithm Training- KD
Algorithm 2 Evaluation
Requires: pretrained teacher T(-)
Initialize: student ((:: 0) with parameters
Requires: pretrained student S(.; 0) for B in 1.2.... ......,,, do X ~ D ys + S(X PB, σB)
for B in 1, 2, ..... B, do
G ~ (N,01) yr < T (B; RB)
ys - SGG HB, 0B)
Ytabel + argmar (ys) end for
0- 0 1) aLæD
30
end for
Experiments
In this section, we discuss the experiments we performed on various datasets in combination with different standard neural network architectures to demonstrate the working of our proposed method. We use CIFARIO (1 as proof of concept to establish the validity of our method, and for certain ablations. We further test our proposed approach on SVHN (2). CIFAR 100, and Food101 [23]. Note that, all the experiments, for Gaussian noise, we generate the image pixels from Gaussian distribution having mean, μ 0 and variance, o² 1.
4.1 State of BatchNorm
In this experiment, we discuss the effect of using the BatchNorm layer in two different states, i.e.. compare between using running statistics and current statistics. Note that we do this change only in the case of the teacher network. In contrast, the student network always uses current statistics, even at the time of evaluation on the validation set. We do this experiment on CIFAR1O by considering two independent parameters: (i) the state of BatchNorm layers in the teacher network, and (ii) the input that is being fed to both the networks.
We distill ResNet-34 teacher network pretrained on CIFAR10 into student networks of varying architectures, which are: ResNet-34, ResNet-18, and MobileNetV'2. In case of original data, we use standard data augmentations like random crop after padding, random horizontal flip, and normaliza- tion. Whereas, while using Gaussian noise, we independently sample pixels for creating a 32 x 32 x 3 image from a standard Gaussian distribution. In both cases, the batch size is 256, and an Adam optimizer with a learning rate of 10- for tuning the parameters of the student network is used
Code is available at: http://///ttttcoooooo////aaaaa/a

---

Table CIFARIO distillation in different cases (input fed to the networks and state of BatchNorm in teacher network) across various Student network architectures. The numbers are accuracies obtained on the test data (mean standard deviation from three runs). The teacher network here a ResNet-34, which has accuracy of 93.29%. The NatchNorm layers in the student model use current statistics during evaluation. Note that RS is running statistics and CS is current statisties.
Student
ResNet34
ResNet18
MobileNetV2
Supervised
93.29
93.22
91.61
Original data RS (Oracle)
92.74 0.21
92.44 1 0.05
90.57
0.22
Original data + CS
92.77 + 0.22
92.20 + 0.1
91.44 0.13
Gaussian noise + RS S 10
13.18 1 0.21
13.49 1 0.08
12.43 + 0.3
Gaussian noise + CS (Ours)
87.11 + 0.23
85.98 + 0.12
82.47 + 0.26
Table 2: Results SVHN, CIFARI00, and Food101 datasets. We see similar performance of the proposed method as the previous experiment. Although the gain for CIFAR 100 and Food101 is slightly lower, which may be due the complexity of datasets the supervised teacher also shows lower accuracy.
Dataset
SVHN
CIFARI00
Food101
Teacher
ResNet18
WideResNet-28-10
ResNet101
Student
MobileNetV2
WideResNet-16-8
ResNet18
Teacher supervised
94,48
80.6
73.4
Original data + RS (Oracle)
95.75
74.1
67.6
Gaussian noise + RS
45.03
1.2
0,9
Gaussian noise + CS (Ours)
92.93
65.7
54.16
The results shown in Table I As expected, the conventional setting of using the original data as input and the teacher with the running statistics computed from the original data is the ideal case, which results in the best performance. Using current statistics does not hinder the learning of the student network either, that is, in the case of original data. However, while using Gaussian noise with rumning statistics, the activation distribution gets mismatched. On the other hand, using current statistics results in the mitigation of this mismatch. Hence, in the former, the student network is unable to learn at all, whereas in the latter, it achieves accuracy close to the ideal case, considering that we rely on the random noise F
The distribution shifts observed in this experiment shown in Figure lb We use activations of 'avgpool' layer in the teacher network (ResNet-34) to obtain these plots. The figure shows the distribution of average activations across all neurons with respect to different settings discussed above. We observe that while using Gaussian noise, current statistics result in a distribution similar to that of using original data. Also, in the case of Gaussian noise with BatchNorm layers using the running statistics, most neurons have close to zero average activations, implying that they are rarely fired. Their information is, therefore, never used in deciding the outcome, hence that particular information is never learned by the student network. These plots are consistent with the results in Table [
4.2 Results on Other Datasets
We perform experiments similar that of CIFAR10 SVHN, CIFAR100 and Food101 to show the applicability of our method on various datasets and neural network architectures. Since we did not find a considerable variance in the case of CIFAR10. here report the results on single runs.
The teacher-student pairs are mentioned in Table 2 setup of hyprraarameters, i.e., batches of 256 samples, and optimization by Adam optimizer with learning rate of -3
We use standard data augmentation and a similar
An ablation study where consider percentage of BatchNorm layers is presented in Appendix A.2.
7

---

Table 2 shows the obtained results. We see similar trends as before, except that the Gaussian noise is random chance even with running statistics in the case of SVHN. We believe that this due to relatively lower covariate shift of activation distribution compared to other more sophisticate datasets. For CTFAR 100 and Food101 datsets we try out teacher-student architectures which are different from the previous experiments to introduce more variability. Table 2 that large amount of gap between real data-based distillation and data-free distillation is reduced by Gaussian noise and as we see in the next section, the remaining gap can be filled by other mechanisms.
4.3 Limited Data
In this experiment, we try to find the amount of additional data needed for extra efforts after Gaussian noise-based KD. It would be interesting to use the synthetic data for this, however, we rely on samples from the training dataset as proxy due large variation the nature of synthetic data generation approaches. We first pretrain our student model using Gaussian distillation and then finetune using original data. First, we show how the performance varies with respect to the amount of data used for finetuning, then we briefly show the same for different configurations. We compare our case of pretraining using Gaussian noise against traditional KD using the same fractional amount of data (baseline).
For finetuning, subset of training data is sampled randomly and a reduced learning rate of 10 is used. Figure 3 (Left) compares the accuracy of the ResNet-18 (student) on CIFARIO when it finetuned using the given fraction of training data its counterpart that trained from scratch. Further, Table 3 shows the student accuracy for other architectures and datasets with 1% of training data.
We observe considerable improvement in the student network after finetuning, specifically for MobileNetV2 on CIFAR10 (about 4%). In addition, pretraining using Gaussian noise outperforms the baseline by a large margin.
Orgai Burii cancis waygh
Can
200
70, 11.5 2002
Figure 3: (Left) Varying fractions of data used for finetuning or training from scratch. x-axis denotes the fraction of the training data used and y-axis denotes the corresponding student accuracy. (Middle) Effect of batch size during inference. Note that x-axis represents logg (batch, size). (Right) Scatter plot (clipped maximum value of 1.0) for average activation for 'avgpool' layer in student network trained using our approach for CIFAR10.
Table 3: Accuracy of student in various configurations after finetuning using 1 % original (training) data. Accuracies are averaged after runs.
Dataset
CIFARIO ResNet34
CIFAR10 MobileNetV2
SVHN MobileNetV2
CIFAR WideResNet-Iee 16-8
Fo0
Student
ResNet18
Limited data
35.46
43.17
29.54
9.28
9.20
Gaussian noise + Limited data
89.84
86.11
94.29
69.8
61.01

---

4.4 BatchNorm State in Student
Here, we explore how the batch size or inference, in general, affects the student network trained using our Gaussian noise-based based approach. We use the ResNet- 18 student network distilled using our approach from ResNet-34 teacher trained on CIFARIO. We use a sufficient amount of evaluation data to adjust the running statistics of the student's &s BatchNorm layers, which is 20 batches of size 16 eact
After the student is trained using Gaussian noise, its running statistics are not adjusted for test data distribution. Therefore, the accuracy using running statistics almost random chance (blue), shown in Figure (Middle) When we use the current statistics and vary the batch size, we see that the accuracy increases with an increase in batch size due better estimates of Hg and aB (black). Further, we adjust the running statistics using some amount of original data, while keeping the rest of the parameters fixed, considering the case in which the student model needs to infer on a single sample or batch of non-i..d. samples. We observe that the adjusted running statisties yield performance similar that of large batch sizes (red).
Discussion
5.1 Limitations
Absence of BatchNorm: If there are no BatchNorm layers present in the teacher-student architecture, then the proposed method will not work. In that case, one would need to learn the input data distribution to obtain activation distribution similar to that of original data.
Training: As opposed to traditional KD, the teacher in the proposed method is required to use the current statistics while distilling the knowledge to student. In order to correctly calculate these statistics and output correct soft labels, large enough batch size is required during training period. Results of study regarding batch size are included Appendix A.4.
Inference: As explained in Section 3.3 the student model might be limited by how it can be used during inference. The student network trained using our proposed approach should either be used with sufficiently large batch of test samples or its running statistics should be adjusted beforehand using a small amount of test data. Without doing so, the student model trained using Gaussian noise does not get adapted to real data distribution, as shown in Figure 3 (Right). The plot shows average activation values for neurons the 'avgpool' layer of a student network trained using our approach for CIFAR10. The distribution of points for Gaussian noise (blue) and original data with current statistics (orange) is similar. Whereas the points for original data with running statistics (red) either lie close to zero or explode to a very high value (clipped to 1.0).
5.2 Rethinking Knowledge Transfer
Recently, 24 explored different types of structured noise (which proves to far better than Gaussian noise) as way to learn representations using a specific contrastive loss and questioned the necessity of huge data-driven vision systems. [18] also observed that low-level features are sufficient for performing KD. This paper shows that using just the Gaussian noise (not even low-level features) we can successfully transfer knowledge from one neural network to the other to a large extent. We have not yet explored the possibility of using structured noise, which could prove to be more effective This challenges the generally accepted notion requiring synthetic data with representations similar to original data and provides motivation explore methods similar ours for tasks such domain adaptation, transfer learning, etc.
6 Conclusions
In this paper, we explore KD solely based on Gaussian noise. We do this by mitigating the covariate shift in the hidden layers of the teacher network, which is caused due to the change in the input data distribution. Further, we show how similar phenomena can occur in the student network, and how
5 Additional results in Appendix A.3.
Preliminary experiments with Dead leaves be found in Appendix A.5.

---

we can use similar approach there as well. The experiments show that student network trained using our approach can achieve a greater performance. contrast to what prior works have shown for Gaussian noise setting, and more so in the presence of limited data. Future work could be to try Gaussian noise-based distillation with lower input dimensions such that it red-ffowwrrss through both teacher and student network since the Gaussian noise does not contain structural information anyway. Further future work could involve comparing how our approach fares to using unrelated data for KD, trying approach structured noise, and extending our findings to other domains such as domain adaptation.
References
[1] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 5030031500.0253,,,200 2015.
[2] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):189--1819 2021.
[3] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? Advances neural information processing systems, 27, 2014.
[4] Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07533 2017.
[5] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715-8724, 2020.
[6] Matan Haroush, Itay Hubara, Elad Hoffer, and Daniel Soudry. The knowledge within: Methods for data-free model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8494-8502, 2020.
[7] Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj. Venkatesh Babu Radhakrishnan. and Anirban Chakraborty. Zero-shot knowledge distillation in deep networks. In International Conference on Machine Learning, pages
PMLR, 2019.
[8] Zi Wang. Data-free knowledge distillation with soft targeted transfer set synthesis. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10245-10253, 2021.
[9] Hanting Chen, Yunhe Wang. Chang Xu, Zhaohui Yang. Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 151-3522, 2019.
[10] Liangchen Luo, Mark Sandler, Zi Lin, Andrey Zhmoginov, and Andrew Howard. Large-scale generative data-free distillation. arXiv preprint arXiv:2012.05557. 2020.
[11] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541, 2006.
[12] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5191-5198. 2020.
[13] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3713-3722, 2019.
[14] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. Advances in neural information processing systems, 30, 2017.
10

---

[15] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2604 2613-2213, 2019.
[16] Yuchen Liu, Hao Xiong, Zhongjun He, Jiajun Zhang, Hua Wu, Haifeng Wang, and Chengqing Zong. End-to-end speech translation with knowledge distillation. arXiv preprint 1rXiv:1904.08075. 2019.
[17] Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7096-7104, 2019.
[18] Paul Micaelli and Amos Storkey. Zero-shot knowledge transfer via adversarial belief matching. arXiv preprint
2019.
[19] Gongfan Fang, Jie Song, Chengehao Shen, Xinchao Wang, Da Chen, and Mingli Song. Data-free adversarial distillation. arXiv preprint 191iv: 1006, 2019.
[20] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448-456. PMLR, 2015.
[21] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[22] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[23] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- - mining discriminative components with random forests. In European Conference Computer Vision, 2014.
[24] Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking at noise. arXiv preprint arXiv:2106.0540 05963,, 2021.


Reviews:
[{"review_id": "lhwn-WUQuX", "reviewer": "Reviewer_Ci53", "summary": "This paper proposes a super straightforward approach for data-free knowledge distillation (KD). Instead of generating pseudo samples to simulate the original training images, it uses Gaussian noise samples as the input and proposes to use current statistics rather than running statistics in the BatchNorm layers for knowledge transfer, which greatly improves the performance. The proposed approach is evaluated with several image classification datasets and the results indicates that it outperforms other noise input-based data-free KD approaches.", "questions": "Overall, I think this is an interesting paper that proposes a super simple but effective way that uses noise inputs for data-free KD. Compared to traditional KD which puts a lot of effort into sample generation, the proposed approach saves tremendous computation resources. I'd like to see how the authors respond to the cons that I'm concerned about and will consider raising my score accordingly.", "limitations": "The authors have adequately discussed the limitations in the manuscript.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. Using noise inputs from a Gaussian distribution is a more efficient way for data-free KD since it saves a lot of computation resources during the pseudo data generation process. \n2. The proposed approach is surprisingly easy.\n3. Results look good compared to other noise input-based data-free KD approaches. \n4. The paper is well written and easy to follow.\n\n\nCons:\n1. Some of the recent related works of data-free KD are missing:\n[1] Nayak, Gaurav Kumar, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. \"Zero-shot knowledge distillation in deep networks.\" In International Conference on Machine Learning, pp. 4743-4751. PMLR, 2019.\n[2] Wang, Zi. \"Data-free knowledge distillation with soft targeted transfer set synthesis.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 11, pp. 10245-10253. 2021.\n\n2. Although the proposed approach is surprisingly simple and I agree with the authors that using current statistics is much better than running statistics. However, it is not quite intuitive to imagine why the outputs with current statistics are very close to those from the original training samples because the inputs are quite different. It could be better if the authors could provide more evidence in the ablation study.\n\n3. I see that the performance of the proposed approach performs better than other works that use noise inputs as the baseline, but there is still room for improvement compared to pseudo image synthesized methods [1,2,3].\n[3] Yin, Hongxu, Pavlo Molchanov, Jose M. Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. \"Dreaming to distill: Data-free knowledge transfer via deepinversion.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8715-8724. 2020.\n\n4. As also discussed in the paper, there are some limitations of the proposed approach: (1) it only works when there are BatchNorm layers in the model, and (2) a large batch needs to be fed into the networks in order to get the statistics. These limitations slightly weaken the contribution of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "arB8A_LNlru", "reviewer": "Reviewer_fgeM", "summary": "This paper studies data-free knowledge transfer, specifically a new frontier using only Gaussian Noise as inputs. As known to literature, noise-based data-free KD imposes critical shift in intermediate distributions, and results in very limited efficacy in distillation. The paper proposes a very simple trick to overcome the issue using current batch mean and variances to normalize, as opposed to running stats, and the method seems to work surprisingly well. Method has been explored on CIFAR10, CIFAR100, SVHN, and Food101 datasets over changing network architectures.", "questions": "- How will this method scale to deeper networks, say ResNet-50, that has bottleneck blocks instead of basic blocks. Will this cause a difference?\n- How will the method scale to ImageNet level analysis?\n- How are the related work baselines in Table 1 implemented? Providing more details here can be helpful.", "limitations": "In my opinion the paper addresses these fronts well.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n+ The problem of using Gaussian noise only to data-free distillation is very interesting, as it bypasses the need to synthesize lots of data or train auxiliary networks.\n+ The approach is rather simple though, as the main trick is to replace the running stats with batch-wise stats, and has to slightly change the inference flow, that results in small degradation in accuracy. But it's pretty impressive observation that simply doing this trick allows for significant boost in distillation accuracy.\n+ I think this paper can inspire deeper analysis of the potential of noise-based KD. The analysis from various fronts are informative.\n+ Codes are provided.\n\nWeekness\n- It would be very beneficial to observe whether this holds for ImageNet1K-level KD, as the batch-wise stats and running stats discrepancy does impose performance drop, and it would be beneficial to see if the method works for large-scale high-resolution datasets. This can significantly improve the quality of paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3t4gephp913", "reviewer": "Reviewer_Ao4y", "summary": "The article proposes a method for distillation using Gaussian noise by modifying the distribution of BN layers. The article describes why the accuracy of the previous method is severely degraded and how the distribution can be adjusted to improve the distillation performance.", "questions": "See the weaknesses.", "limitations": "See the weaknesses.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "$\\textbf{Strengths}$\n\n1.It proposes a method for distillation using Gaussian noise by modifying the distribution of BN layers；\n2. Significantly better than the experimental results of other methods.\n\n$\\textbf{Weaknesses}$\n1. Fewer other methods of comparison in the experiments;\n2. It should try other network structures;\n3. Similar to the evaluation phase, if some Gaussian noise is first input to the teacher model before training, is it not necessary to modify the mean and standard deviation of the BN layer?\n4. Some experimental details are missing and there are some minor errors in the article that need to be fixed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept