Title: NOISY MACHINES: UNDERSTANDING NOISY NEURAL NETWORKS AND ENHANCING ROBUSTNESS TO ANALOG HARDWARE ERRORS USING DISTILLATION

Abstract: The success of deep learning has brought forth a wave of interest in computer hardware design to better meet the high demands of neural network inference. In particular, analog computing hardware has been heavily motivated specifically for accelerating neural networks, based on either electronic, optical or photonic devices, which may well achieve lower power consumption than conventional digital electronics. However, these proposed analog accelerators suffer from the intrinsic noise generated by their physical components, which makes it challenging to achieve high accuracy on deep neural networks. Hence, for successful deployment on analog accelerators, it is essential to be able to train deep neural networks to be robust to random continuous noise in the network weights, which is a somewhat new challenge in machine learning. In this paper, we advance the understanding of noisy neural networks. We outline how a noisy neural network has reduced learning capacity as a result of loss of mutual information between its input and output. To combat this, we propose using knowledge distillation combined with noise injection during training to achieve more noise robust networks, which is demonstrated experimentally across different networks and datasets, including ImageNet. Our method achieves models with as much as ∼ 2× greater noise tolerance compared with the previous best attempts, which is a significant step towards making analog hardware practical for deep learning.

1. INTRODUCTION:

Deep neural networks (DNNs) have achieved unprecedented performance over a wide variety of tasks such as computer vision, speech recognition, and natural language processing. However, DNN inference is typically very demanding in terms of compute and memory resources. Consequently, larger models are often not well suited for large-scale deployment on edge devices, which typically have meagre performance and power budgets, especially battery powered mobile and IoT devices. To address these issues, the design of specialized hardware for DNN inference has drawn great interest, and is an extremely active area of research. To date, a plethora of techniques have been proposed for designing efficient neural network hardware (Sze et al., 2017) .

In contrast to the current status quo of predominantly digital hardware, there is significant research interest in analog hardware for DNN inference. In this approach, digital values are represented by analog quantities such as electrical voltages or light pulses, and the computation itself (e.g., multiplication and addition) proceeds in the analog domain, before eventually being converted back to digital. Analog accelerators take advantage of particular efficiencies of analog computation in exchange for losing the bit-exact precision of digital. In other words, analog compute is cheap but somewhat imprecise. Analog computation has been demonstrated in the context of DNN inference in both electronic (Binas et al., 2016) , photonic (Shen et al., 2017) and optical (Lin et al., 2018) systems. Analog accelerators promise to deliver at least two orders of magnitude better performance over a conventional digital processor for deep learning workloads in both speed (Shen et al., 2017) and energy efficiency (Ni et al., 2017) . Electronic analog DNN accelerators are arguably the most mature technology and hence will be our focus in this work.

The most common approach to electronic analog DNN accelerator is in-memory computing, which typically uses non-volatile memory (NVM) crossbar arrays to encode the network weights as analog values. The NVM itself can be implemented with memristive devices, such as metal-oxide resistive random-access memory (ReRAM) (Hu et al., 2018) or phase-change memory (PCM) (Le Gallo et al., 2018; Boybat et al., 2018; Ambrogio et al., 2018) . The matrix-vector operations computed during inference are then performed in parallel inside the crossbar array, operating on analog quantities for weights and activations. For example, addition of two quantities encoded as electrical currents can be achieved by simply connecting the two wires together, whereby the currents will add linearly according to Kirchhoff's current law. In this case, there is almost zero latency or energy dissipation for this operation.

Similarly, multiplication with a weight can be achieved by programming the NVM cell conductance to the weight value, which is then used to convert an input activation encoded as a voltage into a scaled current, following Ohm's law. Therefore, the analog approach promises significantly improved throughput and energy efficiency. However, the analog nature of the weights makes the compute noisy, which can limit inference accuracy. For example, a simple two-layer fully-connected network with a baseline accuracy of 91.7% on digital hardware, achieves only 76.7% when implemented on an analog photonic array (Shen et al., 2017) . This kind of accuracy degradation is not acceptable for most deep learning applications. Therefore, the challenge of imprecise analog hardware motivates us to study and understand noisy neural networks, in order to maintain inference accuracy under noisy analog computation.

The question of how to effectively learn and compute with a noisy machine is a long-standing problem of interest in machine learning and computer science (Stevenson et al., 1990; Von Neumann, 1956) . In this paper, we study noisy neural networks to understand their inference performance. We also demonstrate how to train a neural network with distillation and noise injection to make it more resilient to computation noise, enabling higher inference accuracy for models deployed on analog hardware. We present empirical results that demonstrate state-of-the-art noise tolerance on multiple datasets, including ImageNet.

The remainder of the paper is organized as follows. Section 2 gives an overview of related work. Section 3 outlines the problem statement. Section 4 presents a more formal analysis of noisy neural networks. Section 5 gives a distillation methodology for training noisy neural networks, with experimental results. Finally, Section 6 provides a brief discussion and Section 7 closes with concluding remarks.

2. RELATED WORK:

Previous work broadly falls under the following categories: studying the effect of analog computation noise, analysis of noise-injection for DNNs, and use of distillation in model training.

Analog Computation Noise Models:

In Rekhi et al. (2019) , the noise due to analog computation is modeled as additive parameter noise with zero-mean Gaussian distribution. The variance of this Gaussian is a function of the effective number of bits of the output of an analog computation. Similarly, the authors in Joshi et al. (2019) also model analog computation noise as additive Gaussian noise on the parameters, where the variance is proportional to the range of values that their PCM device can represent. Some noise models presented have included a more detailed account of device-level interactions, such as voltage drop across the analog array (Jain et al., 2018; Feinberg et al., 2018) , but are beyond the scope of this paper. In this work, we consider an additive Gaussian noise model on the weights, similar to Rekhi et al. (2019) ; Joshi et al. (2019) and present a novel training method that outperforms the previous work in model noise resilience.

Noise Injection for Neural Networks Several stochastic regularization techniques based on noise-injection and dropout (Srivastava et al., 2014; Noh et al., 2017; Li & Liu, 2016) have been demonstrated to be highly effective at reducing overfitting. For generalized linear models, dropout and additive noise have been shown to be equivalent to adaptive L 2 regularization to a first order (Wager et al., 2013) . Training networks with Gaussian noise added to the weights or activations can also increase robustness to variety of adversarial attacks (Rakin et al., 2018) . Bayesian neural networks replace deterministic weights with distributions in order to optimize over the posterior distribution of the weights (Kingma & Welling, 2013) . Many of these methods use noise injection at inference time to approximate weight distribution; in Gal & Ghahramani (2016) a link between Gaussian processes and dropout is established in an effort to model the uncertainty of the output of a network. A theoretical analysis by Stevenson et al. (1990) has shown that for neural networks with adaptive linear neurons, the probability of error of a noisy neural network classifier with weight noise increases with the number of layers, but largely independent of the number of weights per neuron or neurons per layer.

Distillation in Training Knowledge distillation (Hinton et al., 2015) is a well known technique in which the soft labels produced by a teacher model are used to train a student model which typically has reduced capacity. Distillation has shown merit for improving model performance across a range of scenarios, including student models lacking access to portions of training data (Micaelli & Storkey, 2019) , quantized low-precision networks (Polino et al., 2018; Mishra & Marr, 2017) , protection against adversarial attacks (Papernot et al., 2016; Goldblum et al., 2019) , and in avoiding catastrophic forgetting for multi-task learning (Schwarz et al., 2018) . To the best of our knowledge, our work is the first to combine distillation with noise injection in training to enhance model noise robustness. Figure 1 : Deploying a neural network layer, l, on an analog in-memory crossbar involves first flattening the filters for a given layer into weight matrix W l , which is then programmed into an array of NVM devices which provide differential conductances G l for analog multiplication. A random Gaussian ∆W l is used to model the inherent imprecision in analog computation.

3. PROBLEM STATEMENT:

Without loss of generality, we model a general noisy machine after a simple memristive crossbar array, similar to Shafiee et al. (2016) . Figure 1 illustrates how an arbitrary neural network layer, l, such as a typical 3 × 3 convolution, can be mapped to this hardware substrate by first flattening the weights into a single large 2D matrix, W l , and then programming each element of this matrix into a memristive cell in the crossbar array, which provides the required conductances G l (the reciprocal of resistance) to perform analog multiplication following Ohm's law, i out = v in G. Note that a pair of differential pair of NVM devices are typically used to represent a signed quantity in G l . Subsequently, input activations, x l converted into continuous voltages, v(x l ), are streamed into the array rows from the left-hand side. The memristive devices connect row with columns, where the row voltages are converted into currents scaled by the programmed conductance, G, to generate the currents i(y l ), which are differential in order to represent both positive and negative quantites with unipolar signals. The currents from each memristive device essentially add up for free where they are connected in the columns, according to Kirchhoff's current law. Finally, the differential currents are converted to bipolar voltages, v(y l ), which are they digitized before adding bias, and performing batch normalization and ReLU operations, which are not shown in Figure 1 .

However, the analog inference hardware of Figure 1 is subject to real-world non-idealities, typically attributed to variations in: 1) manufacturing process, 2) supply voltage and 3) temperature, PVT variation collectively, all of which result in noise in the system. Below we discuss the two key components in terms of analog noise modeling.

Data Converters. Digital-to-analog converter (DAC) and analog-to-digital converter (ADC) circuits are designed to be robust to PVT variation, but in practice these effects do degrade the resolution (i.e. number of bits). Therefore, we consider effective number of bits (ENOB), which is a lower bound on resolution in the presence of non-idealities. Hence, we use activation and weight quantization with ENOB data converters and no additional converter noise modeling.

NVM cells. Due to their analog nature, memristive NVM cells have limited precision, due to the read and write circuitry (Joshi et al., 2019) . In between write and read operations, their stored value is prone to drift over time. Long-term drift can be corrected with periodic refresh operations. At shorter timescales, time-varying noise may be encountered. For most of the experiments in this paper, we model generic NVM cell noise as an additive zero-mean i.i.d. Gaussian error term on the weights of the model in each particular layer ∆W l ∼ N (∆W l ; 0, σ 2 N,l I). This simple model, described more concretely in Section 5, is similar to that used by Joshi et al. (2019) which was verified on real hardware. In addition, we also investigate spatially-varying and time-varying noise models in Section 5.2 (Table 1 ).

4. ANALYSIS OF NOISY NEURAL NETWORKS:



4.1. BIAS VARIANCE DECOMPOSITION FOR NOISY WEIGHTS:

Naively deploying an off-the-shelf pretrained model on a noisy accelerator will yield poor accuracy for a fundamental reason. Consider a neural network f (W; x) with weights W that maps an input x ∈ R n to an output y ∈ R. In the framework of statistical learning, x and y are considered to be randomly distributed following a joint probability distribution p(x, y). In a noisy neural network, the weights W are also randomly distributed, with distribution p(W). The expected Mean Squared Error (MSE) of this noisy neural network can be decomposed as

E (x,y)∼p(x,y),W∼p(W) [(f (W; x) -y) 2 ] = E (x,y)∼p(x,y),W∼p(W) [(f (W; x) -E W∼p(W) [f (W; x)] + E W∼p(W) [f (W; x)] -y) 2 ] = E x∼p(x) [E W∼p(W) [(f (W; x) -E W∼p(W) [f (W; x)]) 2 ]] + E (x,y)∼p(x,y) [(E W∼p(W) [f (W; x)] -y) 2 ].

(1)

The first term on the right hand side of Equation 1 is a variance loss term due to randomness in the weights and is denoted as l var . The second term is a squared bias loss term which we call l bias . However, typically a model is trained to minimize the empirical version of expected loss

l pretrained = E (x,y)∼p(x,y) [(f (E[W]; x) -y) 2 ].

We assume that the noise is centered such that pretrained weights are equal to E[W]. A pretrained model is therefore optimized for the wrong loss function when deployed on a noisy accelerator. To show this in a more concrete way, a baseline LeNet model (32 filters in the first convolutional layer, 64 filters in the second convolutional layer and 1024 neurons in the fully-connected layer) (LeCun et al., 1998) is trained on MNIST dataset to 99.19% accuracy and then exposed to Gaussian noise in its weights, numerical values of these loss terms can be estimated. The expected value of the network output E W [f (W; x)] is estimated by averaging over outputs of different instances of the network for the same input x. We perform inference on n = 100 different instances of the network and estimate the loss terms as

EQUATION

EQUATION

)

The above formulas are for a network with a scalar output. They can be easily extended to the vector output case by averaging over all outputs. In the LeNet example, we take the output of softmax layer to calculate squared losses. The noise is assumed i.i.d. Gaussian centered around zero with a fixed SNR σ 2 W,l /σ 2 N,l in each layer l. The numerical values of the above losses are estimated using the entire test dataset for different noise levels. Results are shown in Figure 2(a) . lbias is initially equal to lpretrained and lvar = 0 when there is no noise. However, as noise level rises, they increase in magnitude and become much more important than lpretrained . lvar overtakes lbias to become the predominant loss term in a noisy LeNet at σ N /σ W 0.6. It is useful to note that l bias increases with noise entirely due to nonlinearity in the network, which is ReLU in the case of LeNet. In a linear model, l bias should be equal to l pretrained as we would have f

(E[W]; x) = E[f (W; x)].

A model trained in a conventional manner is thus not optimized for the real loss it is going to encounter on a noisy accelerator. Special retraining is required to improve its noise tolerance. In Figure 2 (a), we show how the model accuracy degrades with a rising noise level for the baseline LeNet and its deeper and wider variants. The deeper network is obtained by stacking two more convolutional layers of width 16 in front of the baseline network and the wider network is obtained by increasing the widths of each layer in the baseline to 128, 256, 2048 respectively. Performance degradation due to noise is worse for the deeper variant and less severe for the wider one. A more detailed discussion of the network architecture effect on its performance under noise is offered in Section 4.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 

4.2. LOSS OF INFORMATION IN A NOISY NEURAL NETWORK:

Information theory offers useful tools to study noise in neural networks. Mutual information I(X; Y ) characterizes the amount of information obtained on random variable X by observing another random variable Y . The mutual information between X and Y can be related to Shannon entropy by

I(X; Y ) = H(Y ) -H(Y |X).

(6) Mutual information has been used to understand DNNs (Tishby & Zaslavsky, 2015; Saxe et al., 2018) . Treating a noisy neural network as a noisy information channel, we can show how information about the input to the neural network diminishes as it propagates through the noisy computation. In this subsection, X is the input to the neural network and Y is the output. Mutual information is estimated for the baseline LeNet model and its variants using Equation 6. When there is no noise, the term H(Y |X) is zero as Y is deterministic once the input to the network X is known, therefore I(X; Y ) is just H(Y ) in this case. Shannon entropy H(Y ) can be estimated using a standard discrete binning approach (Saxe et al., 2018) . In our experiment, Y is the output of the softmax layer which is a vector of length 10. Entropy H(Y ) is estimated using four bins per coordinate of Y by

EQUATION

where p i is the probability that an output falls in the bin i. When noise is introduced to the weights, the conditional entropy H(Y |X) is estimated by fixing the input X = x and performing multiple noisy inferences to calculate Ĥ(Y |X = x) with the above binning approach. Ĥ(Y |X = x) is then averaged over different input x to obtain Ĥ(Y |X). This estimate is performed for LeNet and its variants with different noise levels. Results are shown in Figure 2 (b). The values are normalized to the estimate of I(X; Y ) at zero noise. Mutual information between the input and the output decays towards zero with increasing noise in network weights. Furthermore, mutual information in a deeper and narrower network decays faster than in a shallower and wider network. Intuitively, information from the input undergoes more noisy compute when more layers are added to the network, while a wider network has more redundant paths for the information to flow, thus better preserving it. An information theoretic bound of mutual information decay as a function of network depth and width in a noisy neural network will be treated in our follow-up work. Overall, noise is damaging the learning capacity of the network. When the output of the model contains no information from its input, the network loses all ability to learn. For a noise level that is not so extreme, a significant amount of mutual information remains, which indicates that useful learning is possible even with a noisy model.

5. COMBINING NOISE INJECTION AND KNOWLEDGE DISTILLATION:



5.1. METHODOLOGY:

Noise injection during training is one way of exposing network training to a more realistic loss as randomly perturbing weights simulates what happens in a real noisy analog device, and forces the network to adapt to noise during training. Noise injection only happens in training during forward propagation, which can be considered as an approximation for calculating weight gradients with a straight-through-estimator (STE) (Bengio et al., 2013) . At each forward pass, the weight W l of layer l is drawn from an i.i.d. Gaussian distribution N (W l ; W l 0 , σ 2 N,l I). The noise is referenced to the range of representable weights W l max -W l min in that particular layer

EQUATION

where η is a coefficient characterizing the noise level. During back propagation, gradients are calculated with clean weights W l 0 , and only W l 0 gets updated by applying the gradient. W l max and W l min are hyperparameters which can be chosen with information on the weight distributions.

Knowledge distillation was introduced by Hinton et al. (2015) as a way for training a smaller student model using a larger model as the teacher. For an input to the neural network x, the teacher model generates logits z T i , which are then turned into a probability vector by the softmax layer

EQUATION

)

The temperature, T , controls the softness of the probabilities. The teacher network can generate softer labels for the student network by raising the temperature T . We propose to use a noise free clean model as the teacher to train a noisy student network. The student network is trained with noise injection to match a mix of hard targets and soft targets generated by the teacher. Logits generated by the student network are denoted as z S i . A loss function with distillation for the student model can be written as

EQUATION

)

Here H is cross-entropy loss, y true is the one-hot encoding of the ground truth, and R is the L 2regularization term. Parameter α balances relative strength between hard and soft targets. We follow the original implementation in Hinton et al. (2015) , which includes a T 2 factor in front of the soft target loss to balance gradients generated from different targets. The student model is then trained with Gaussian noise injection using this distillation loss function. The vanilla noise injection training corresponds to the case where α = 0. If the range of weights is not constrained and the noise reference is fixed, the network soon learns that the most effective way to decrease the loss is to increase the amplitude of the weights, which increases the effective SNR. There are two possible ways to deal with this problem. Firstly, the noise reference could be re-calculated after each weight update, thus updating the noise power. Secondly, we can constrain the range of weights by clipping them to the range [W l min , W l max ], and use a fixed noise model during training. We found that in general the second method of fixing the range of weights and training for a specific noise yields more stable training and better results. Therefore, this is the training method that we adopt in this paper. A schematic of our proposed method is shown in Figure 5 of the Appendix.

During training, a clean model is first trained to its full accuracy and then weight clipping is applied to clip weights in the range [W l min , W l max ]. The specific range is chosen based on statistics of the weights. Fine-tuning is then applied to bring the weight-clipped clean model back to full accuracy. This model is then used as the teacher to generate soft targets. The noisy student network is initialized with the same weights as the teacher. This can be considered as a warm start to accelerate retraining. As we discussed earlier, the range of weights is fixed during training, and the noise injected into the student model is referenced to this range.

Our method also supports training for low precision noisy models. Quantization reflects finite precision conversion between analog and digital domains in an analog accelerator. Weights are uniformly quantized in the range [W l min , W l max ] before being exposed to noise. In a given layer, the input activations are quantized before being multiplied by noisy weights. The output results of the matrix multiplication are also quantized before adding biases and performing batch normalization, which are considered to happen in digital domain. When training with quantization, the straight-throughestimator is assumed when calculating gradients with back propagation.

5.2. EXPERIMENTAL RESULTS:

In order to establish the effectiveness of our proposed method, experiments are performed for different networks and datasets. In this section we mainly focus on bigger datasets and models, while results on LeNet and its variants with some discussion of network architecture effect can be found in Figure 6 of the Appendix. ResNets are a family of convolutional neural networks proposed by He et al. (2016) , which have gained great popularity in computer vision applications. In fact, many other deep neural networks also use ResNet-like cells as their building blocks. ResNets are often used as industry standard benchmark models to test hardware performance. The first set of experiments we present consist of a ResNet-32 model trained on the CIFAR10 dataset. In order to compare fairly with the previous work, we follow the implementation in Joshi et al. (2019) , and consider a ResNet-32(v1) model on CIFAR10 with weight clipping in the range [-2σ W,l , 2σ W,l ]. The teacher model is trained to an accuracy of 93.845% using stochastic gradient descent with cosine learning rate decay (Loshchilov & Hutter, 2016) , and an initial learning rate of 0.1 (batch size is 128). The network is then retrained with noise injection to make it robust against noise. Retraining takes place for 150 epochs, the initial learning rate is 0.01 and decays with the same cosine profile. We performed two sets of retraining, one without distillation in the loss (α = 0), and another with distillation loss (α = 1). Everything else was kept equal in these retraining runs. Five different noise levels are tested with five different values of η: {0.02, 0.04, 0.057, 0.073, 0.11}.

Results are shown in Figure 3 (a). Every retraining run was performed twice and inference was performed 50 times on the test dataset for one model, to generate statistically significant results. Temperature was set to T = 6 for the runs with distillation. We found that an intermediate temperature between 2 and 10 produces better results. The pretrained model without any retraining performs very poorly at inference time when noise is present. Retraining with Gaussian noise injection can effectively recover some accuracy, which we confirm as reported in Joshi et al. (2019) . Our method of combining noise injection with knowledge distillation from the clean model further improves noise resilience by about 40% in terms of η, which is an improvement of almost 2× in terms of noise power σ 2 N . The actual noise level in a given device can only be estimated, and will vary from one device to another and even fluctuate depending on the physical environment in which it operates (Section 3). Therefore, it is important that any method to enhance noise robustness can tolerate a range of noise In the previous experiments, we assume a fixed noise level parameterized by η. On real analog hardware, there could be additional non-idealities such as variation in noise level due to temperature fluctuation and nonuniform noise profile on different NVM cells due to statistical variation in the manufacturing process. We have conducted additional experiments to account for these effects.

Results from the experiments are shown in Table 1 . Temporal fluctuation represents noise level variation over time. Noise η is randomly sampled from N (η; η 0 , σ 2 η ) for each inference batch. A noise temporal fluctuation level of 10% means that σ η = 0.1η 0 . Spatial noise level fluctuation introduces nonuniform diagonal terms in the noise covariance matrix. More concretely, each weight noise in our previous model is multiplied by a scale factor λ w with λ w drawn from a Gaussian distribution N (λ w ; 1, σ 2 λ ). A noise spatial fluctuation level of 10% means that σ λ = 0.1. The scale factors are generated and then fixed when the network is instantiated, therefore the noise during network inference is non i.i.d. in this case. Results from our experiments show that there is no significant deviation when a combination of these non-ideal noise effects are taken into account. The performance of our training method is also validated with quantization. A ResNet-18(v2) model is trained with quantization to 4-bit precision (ENOB) for both weights and activations. This corresponds to 4-bit precision conversions between digital and analog domains. A subset of training data is passed through the full precision model to calibrate the range for quantization -we choose the 0.1% and 99.9% percentiles as q min and q max for the quantizer. This range of quantization is fixed throughout training. The quantized model achieves an accuracy of 92.91% on the test dataset when no noise is present. The model is then re-trained for noise robustness. The noise level is referenced to the range of quantization of weights in one particular layer, such that W l min = q min,l and W l max = q max,l . Results are shown for the same set of η values in Figure 4 (a). In the distillation retraining runs, the full-precision clean model with an accuracy of 93.87% is used as the teacher and temperature is set to T = 6. Due to extra loss in precision imposed by aggressive quantization, accuracy of the pretrained quantized model drops sharply with noise. At η = 0.057, the model accuracy drops to 87.5% without retraining and further down to 80.9% at η = 0.073. Even retraining with noise injection struggles, and the model retrained with only noise injection achieves an accuracy of 90.34% at η = 0.073. Our method of combining noise injection and distillation stands out by keeping the accuracy loss within 1% from the baseline up to a noise level of η 0.07. One interesting aspect of using distillation loss during retraining with noise can be seen in Figure 4 (b). The evolution of model accuracy on the test dataset is shown. When no distillation loss is used, the model suffers an accuracy drop (difference between blue and orange curves) around 2.08% when tested with noise. The drop (difference between green and red curves) is significantly reduced to around 0.6% when distillation loss is used. This observation indicates that training with distillation favors solutions that are less sensitive to noise. The final model obtained with distillation is actually slightly worse when there is no noise at inference time but becomes superior when noise is present.

Results on the ImageNet dataset for a ResNet-50(v1) network are shown in Table 2 to demonstrate that our proposed approach scales to a large-scale dataset and a deep model. A ResNet-50 model is first trained to an accuracy of 74.942% with weight clipping in the range [-2σ W,l , 2σ W,l ]. This range is fixed as the reference for added noise. For ResNet-50 on ImageNet, only three different noise levels are explored, and the accuracy degrades very quickly beyond the noise level η = 0.06, as the model and the task are considerably more complex. Retraining runs for 30 epochs with an initial learning rate of 0.001 and cosine learning rate decay with a batch size of 32. For distillation, we used α = 1 and T = 6 as in previous experiments. Results are collected for two independent training runs in each setting and 50 inference runs over the entire test dataset. The findings confirm that training with distillation and noise injection consistently delivers more noise robust models. The accuracy uplift benefit also markedly increases with noise.

6. DISCUSSION:

Effects of distillation Knowledge distillation is a proven technique to transfer knowledge from a larger teacher model to a smaller, lower capacity student model. This paper shows, for the first time, that distillation is also an effective way to transfer knowledge between a clean model and its noisy counterpart, with the novel approach of combining distillation with noise injection during training. We give some intuition for understanding this effect with the help of Section 4.2: a noisy neural network can be viewed as a model with reduced learning capacity by the loss of mutual information argument. Distillation is therefore acting to help reduce this capacity gap.

In our experiments, distillation shows great benefit in helping the network to converge to a good solution, even with a high level of noise injected in the forward propagation step. Here, we attempt to explain this effect by the reduced sensitivity of distillation loss. An influential work by Papernot et al. (2016) shows that distillation can be used to reduce the model sensitivity with respect to its input perturbations thus defending against some adversarial attacks. We argue that distillation can achieve a similar effect for the weights of the network. Taking the derivative of the i-th output of the student network q S i at temperature T with respect to a weight w yields

EQUATION

The 1/T scaling makes the output less sensitive to weight perturbation at higher temperature, thus potentially stabilizing the training when noise is injected into weights during forward propagation. We plan to work on a more formal analysis of this argument in our future work.

Hardware Performance Benefits The improvements in noise tolerance of neural networks demonstrated in this work have a potential impact on the design of practical analog hardware accelerators for neural network inference. Increased robustness to noisy computation at the model training level potentially means that the specification of the analog hardware can be relaxed. In turn, this can make it easier to achieve the hardware specification, or even allow optimizations to further reduce the energy consumption. An in-depth discussion of the trade-off between compute noise performance and hardware energy dissipation is beyond the scope of this paper, but we refer the interested reader to Rekhi et al. ( 2019) for more details. In summary, we believe that machine learning research will be a key enabler for practical analog hardware accelerators.

7. CONCLUSION:

Analog hardware holds the potential to significantly reduce the latency and energy consumption of neural network inference. However, analog hardware is imprecise and introduces noise during computation that limits accuracy in practice. This paper explored the training of noisy neural networks, which suffer from reduced capacity leading to accuracy loss. We propose a training methodology that trains neural networks via distillation and noise injection to increase the accuracy of models under noisy computation. Experimental results across a range of models and datasets, including ImageNet, demonstrate that this approach can almost double the network noise tolerance compared with the previous best reported values, without any changes to the model itself beyond the training method. With these improvements in the accuracy of noisy neural networks, we hope to enable the implementation of analog inference hardware in the near future. 

O + k = " > A A A C J n i c b V D L S s N A F J 3 U V 6 2 v q E s 3 g 0 V o Q U o i g o I U i m 6 6 r N g X t D V M p t N 2 6 E w S Z i Z C D f k a N / 6 K G x c V E X d + i p M 2 i L Y e u H A 4 5 1 7 u v c c N G J X K s j 6 N z M r q 2 v p G d j O 3 t b 2 z u 2 f u H z S l H w p M G t h n v m i 7 S B J G P d J Q V D H S D g R B 3 G W k 5 Y 5 v E r / 1 Q I S k v l d X k 4 D 0 O B p 6 d E A x U l p y z H K X I z X C i E X V u N C V d M h R 4 d G J a H w f z R z B o 7 s 4 v q q X 7 e L p x P n R l A h J H B c d M 2 + V r B n g M r F T k g c p a o 4 5 7 f Z 9 H H L i K c y Q l B 3 b C l Q v Q k J R z E i c 6 4 a S B A i P 0 Z B 0 N P U Q J 7 I X z d 6 M 4 Y l W + n D g C 1 2 e g j P 1 9 0 S E u J Q T 7 u r O 5 E y 5 6 C X i f 1 4 n V I P L X k S 9 I F T E w / N F g 5 B B 5 c M k M 9 i n g m D F J p o g L K i + F e I R E g g r n W x O h 2 A v v r x M m m c l 2 y r Z t + f 5 y n U a R x Y c g W N Q A D a 4 A B V Q B T X Q A B g 8 g R c w B W / G s / F q v B s f 8 9 a M k c 4 c g j 8 w v r 4 B V e i m 6 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A G o + h I z c s E b D P G 8 P A s o z 4 R 2 / O + k = " > A A A C J n i c b V D L S s N A F J 3 U V 6 2 v q E s 3 g 0 V o Q U o i g o I U i m 6 6 r N g X t D V M p t N 2 6 E w S Z i Z C D f k a N / 6 K G x c V E X d + i p M 2 i L Y e u H A 4 5 1 7 u v c c N G J X K s j 6 N z M r q 2 v p G d j O 3 t b 2 z u 2 f u H z S l H w p M G t h n v m i 7 S B J G P d J Q V D H S D g R B 3 G W k 5 Y 5 v E r / 1 Q I S k v l d X k 4 D 0 O B p 6 d E A x U l p y z H K X I z X C i E X V u N C V d M h R 4 d G J a H w f z R z B o 7 s 4 v q q X 7 e L p x P n R l A h J H B c d M 2 + V r B n g M r F T k g c p a o 4 5 7 f Z 9 H H L i K c y Q l B 3 b C l Q v Q k J R z E i c

O + k = " > A A A C J n i c b V D L S s N A F J 3 U V 6 2 v q E s 3 g 0 V o Q U o i g o I U i m 6 6 r N g X t D V M p t N 2 6 E w S Z i Z C D f k a N / 6 K G x c V E X d + i p M 2 i L Y e u H A 4 5 1 7 u v c c N G J X K s j 6 N z M r q 2 v p G d j O 3 t b 2 z u 2 f u H z S l H w p M G t h n v m i 7 S B J G P d J Q V D H S D g R B 3 G W k 5 Y 5 v E r / 1 Q I S k v l d X k 4 D 0 O B p 6 d E A x U l p y z H K X I z X C i E X V u N C V d M h R 4 d G J a H w f z R z B o 7 s 4 v q q X 7 e L p x P n R l A h J H B c d M 2 + V r B n g M r F T k g c p a o 4 5 7 f Z 9 H H L i K c y Q l B 3 b C l Q v Q k J R z E i c

O + k = " > A A A C J n i c b V D L S s N A F J 3 U V 6 2 v q E s 3 g 0 V o Q U o i g o I U i m 6 6 r N g X t D V M p t N 2 6 E w S Z i Z C D f k a N / 6 K G x c V E X d + i p M 2 i L Y e u H A 4 5 1 7 u v c c N G J X K s j 6 N z M r q 2 v p G d j O 3 t b 2 z u 2 f u H z S l H w p M G t h n v m i 7 S B J G P d J Q V D H S D g R B 3 G W k 5 Y 5 v E r / 1 Q I S k v l d X k 4 D 0 O B p 6 d E A x U l p y z H K X I z X C i E X V u N C V d M h R 4 d G J a H w f z R z B o 7 s 4 v q q X 7 e L p x P n R l A h J H B c d M 2 + V r B n g M r F T k g c p a o 4 5 7 f Z 9 H H L i K c y Q l B 3 b C l Q v Q k J R z E i c

4 C h R m S s m O Z k X I S J B T F j K Q 5 O 5 Y k Q n i A e q S j a Y A 4 k U 4 y + T K F J 1 r p Q j 8 U u g I F J + r f i Q R x K Y f c 0 5 3 j G + W i N x b / 8 z q x 8 i + d h A Z R r E i A p 4 v 8 m E E V w n F k s E s F w Y o N N U F Y U H 0 r x H 0 k E F Y 6 2 J w O w V p 8 e Z k 0 z 8 q W W b b u z g u V m 1 k c W X A E j k E R W O A C V E A V 1 E A D Y P A C 3 s A H G B m v x r v x a X x N W z P G b O Y Q z M H 4 + Q X q g 6 a 7

< / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L w l 7 G U N U 3 j Figure 6 : Results on LeNet and its variants show that our method of combining distillation and noise injection improves noise robustness for different model architectures on MNIST. The benefit of our method is the most significant when the network struggles to learn with vanilla noise injection retraining method. This threshold noise level depends on the network architecture, as we have remarked for mutual information decay.

d j g v X f b J c x 2 l p O k f w = " > A A A C J X i c b V D L S s N A F J 3 U V 6 2 v q k s 3 g 0 V o Q U o i g o I u i m 6 6 r N g X N D F M p p N 2 6 E w S Z y Z C D f k Z N / 6 K G x c W E V z 5 K 0 4 f C 9 t 6 4 M L h n H u 5 9 x 4 v Y l Q q 0 / w 2 M i u r a + s b 2 c 3 c 1 v b O 7 l 5 + / 6 A p w 1 h g 0 s A h C 0 X b Q 5 I w G p C G o o q R d i Q I 4 h 4 j L W 9 w O / Z b T 0 R I G g Z 1 N Y y I w 1 E v o D 7 F S G n J z V / b H K k + R i y p p k V b 0 h 5 H x W c 3 o e l D M n E E T + 7 T 9 K p e O n 2 c V + t p W n L z B b N s T g C X i T U j B T B D z c 2 P 7 G 6 I Y 0 4 C h R m S s m O Z k X I S J B T F j K Q 5 O 5 Y k Q n i A e q S j a Y A 4 k U 4 y + T K F J 1 r p Q j 8 U u g I F J + r f i Q R x K Y f c 0 5 3 j G + W i N x b / 8 z q x 8 i + d h A Z R r E i A p 4 v 8 m E E V w n F k s E s F w Y o N N U F Y U H 0 r x H 0 k E F Y 6 2 J w O w V p 8 e Z k 0 z 8 q W W b b u z g u V m 1 k c W X A E j k E R W O A C V E A V 1 E A D Y P A C 3 s A H G B m v x r v x a X x N W z P G b O Y Q z M H 4 + Q X q g 6 a 7 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L w l 7 G U N U 3 j d j g v X f b J c x 2 l p O k f w = " > A A A C J X i c b V D L S s N A F J 3 U V 6 2 v q k s 3 g 0 V o Q U o i g o I u i m 6 6 r N g X N D F M p p N 2 6 E w S Z y Z C D f k Z N / 6 K G x c W E V z 5 K 0 4 f C 9 t 6 4 M L h n H u 5 9 x 4 v Y l Q q 0 / w 2 M i u r a + s b 2 c 3 c 1 v b O 7 l 5 + / 6 A p w 1 h g 0 s A h C 0 X b Q 5 I w G p C G o o q R d i Q I 4 h 4 j L W 9 w O / Z b T 0 R I G g Z 1 N Y y I w 1 E v o D 7 F S G n J z V / b H K k + R i y p p k V b 0 h 5 H x W c 3 o e l D M n E E T + 7 T 9 K p e O n 2 c V + t p W n L z B b N s T g C X i T U j B T B D z c 2 P 7 G 6 I Y 0 4 C h R m S s m O Z k X I S J B T F j K Q 5 O 5 Y k Q n i A e q S j a Y A 4 k U 4 y + T K F J 1 r p Q j 8 U u g I F J + r f i Q R x K Y f c 0 5 3 j G + W i N x b / 8 z q x 8 i + d h A Z R r E i A p 4 v 8 m E E V w n F k s E s F w Y o N N U F Y U H 0 r x H 0 k E F Y 6 2 J w O w V p 8 e Z k 0 z 8 q W W b b u z g u V m 1 k c W X A E j k E R W O A C V E A V 1 E A D Y P A C 3 s A H G B m v x r v x a X x N W z P G b O Y Q z M H 4 + Q X q g 6 a 7 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " L w l 7 G U N U 3 j d j g v X f b J c x 2 l p O k f w = " > A A A C J X i c b V D L S s N A F J 3 U V 6 2 v q k s 3 g 0 V o Q U o i g o I u i m 6 6 r N g X N D F M p p N 2 6 E w S Z y Z C D f k Z N / 6 K G x c W E V z 5 K 0 4 f C 9 t 6 4 M L h n H u 5 9 x 4 v Y l Q q 0 / w 2 M i u r a + s b 2 c 3 c 1 v b O 7 l 5 + / 6 A p w 1 h g 0 s A h C 0 X b Q 5 I w G p C G o o q R d i Q I 4 h 4 j L W 9 w O / Z b T 0 R I G g Z 1 N Y y I w 1 E v o D 7 F S G n J z V / b H K k + R i y p p k V b 0 h 5 H x W c 3 o e l D M n E E T + 7 T 9 K p e O n 2 c V + t p W n L z B b N s T g C X i T U j B T B D z c 2 P 7 G 6 I Y 0 4 C h R m S s m O Z k X I S J B T F j K Q 5 O 5 Y k Q n i A e q S j a Y A 4 k U 4 y + T K F J 1 r p Q j 8 U u g I F J + r f i Q R x K Y f c 0 5 3 j G + W i N x b / 8 z q x 8 i + d h A Z R r E i A p 4 v 8 m E E V w n F k s E s F w Y o N N U F Y U H 0 r x H 0 k E F Y 6 2 J w O w V p 8 e Z k 0 z 8 q W W b b u z g u V m 1 k c W X A E j k E R W O A C V E A V 1 E A D Y P A C 3 s A H G B m v x r v x a X x N W z P G b O Y Q z M H 4 + Q X q g 6 a 7 < / l a t e x i t > L(x; W S ; T ) = H( (z S i ; T = 1), y true ) + ↵T 2 H( (z S i ; T ), q T i ) + R(W S 0 ) < l a t e x i t s h a 1 _ b a s e 6 4 = " p U F i I L b d g h p n S p 6 1 F g M 8 D 2 I T I V U = " > A A A C y 3 i c j Z F N b x M x E I a 9 W z 5 K + G i A I x e r E V J W R N F u h Q R S V a l q L z m A V E r S V M q m q 1 n H m 1 i 1 d x f b S

G i A I x e r E V J W R N F u h Q R S V a l q L z m A V E r S V M q m q 1 n H m 1 i 1 d x f b S

:



Figures and Tables:

Figure 2: (a) Different loss terms on the test dataset and model test accuracy as a function of noise standard deviation, the losses are normalized to the pretrained model loss lpretrained , calculated using clean weights. Accuracy is calculated by performing the inference 100 times on the test set, error bars show the standard deviation.(b) Estimate of normalized mutual information between the input and output of the baseline LeNet and its variants as a function of noise standard deviation. A random subset of 200 training images are used for this estimate, with each inference repeated 100 times on a random realization of the network to estimate H(Y |X). Mutual information decays with rising noise, deeper and narrower networks are more susceptible to this decay.

Figure 3: (a) Test accuracy as a function of noise level, here we have η tr = η inf , error bars show the standard deviation of different training and inference runs. Our method with distillation achieves the best robustness. (b) Comparison of model performance at noise levels different from the training level.

Figure 4: (a) Test accuracy as a function of noise level for 4-bit ResNet-18, here we have η tr = η inf , error bars show the standard deviation of different training and inference runs. Retraining with distillation and noise injection achieves the best results with quantization. (b) Test accuracy of different models during retraining with noise level η = 0.057.

ground truth Noise injection at forward pass only H( (z S i ; T = 1), y true ) < l a t e x i t s h a 1 _ b a s e 6 4 = " A G o + h I z c s E b D P G 8 P A s o z 4 R 2 /

6 4 a S B A i P 0 Z B 0 N P U Q J 7 I X z d 6 M 4 Y l W + n D g C 1 2 e g j P 1 9 0 S E u J Q T 7 u r O 5 E y 5 6 C X i f 1 4 nV I P L X k S 9 I F T E w / N F g 5 B B 5 c M k M 9 i n g m D F J p o g L K i + F e I R E g g r n W x O h 2 A v v r x M m m c l 2 y r Z t + f 5 y n U a R x Y c g W N Q A D a 4 A B V Q B T X Q A B g 8 g R c w B W / G s / F q v B sf 8 9 a M k c 4 c g j 8 w v r 4 B V e i m 6 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A G o + h I z c s E b D P G 8 P A s o z 4 R 2 /

6 4 a S B A i P 0 Z B 0 N P U Q J 7 I X z d 6 M 4 Y l W + n D g C 1 2 e g j P 1 9 0 S E u J Q T 7 u r O 5 E y 5 6 C X i f 1 4 nV I P L X k S 9 I F T E w / N F g 5 B B 5 c M k M 9 i n g m D F J p o g L K i + F e I R E g g r n W x O h 2 A v v r x M m m c l 2 y r Z t + f 5 y n U a R x Y c g W N Q A D a 4 A B V Q B T X Q A B g 8 g R c w B W / G s / F q v B sf 8 9 a M k c 4 c g j 8 w v r 4 B V e i m 6 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " A G o + h I z c s E b D P G 8 P A s o z 4 R 2 /

6 4 a S B A i P 0 Z B 0 N P U Q J 7 I X z d 6 M 4 Y l W + n D g C 1 2 e g j P 1 9 0 S E u J Q T 7 u r O 5 E y 5 6 C X i f 1 4 nV I P L X k S 9 I F T E w / N F g 5 B B 5 c M k M 9 i n g m D F J p o g L K i + F e I R E g g r n W x O h 2 A v v r x M m m c l 2 y r Z t + f 5 y n U a R x Y c g W N Q A D a 4 A B V Q B T X Q A B g 8 g R c w B W / G s / F q v B s f 8 9 a M k c 4 c g j 8 w v r 4 B V e i m 6 A = = < / l a t e x i t > H( (z S i ; T ), q T i ) < l a t e x i t s h a 1 _ b a s e 6 4 = " L w l 7 G U N U 3 j d j g v X f b J c x 2 l p O k f w = " > A A A C J X i c b V D L S s N A F J 3 U V 6 2 v q k s 3 g 0 V o Q U o i g o I u i m 6 6 r N g X N D F M p p N 2 6 E w S Z y Z C D f k Z N / 6 K G x c W E V z 5 K 0 4 f C 9 t 6 4 M L h n H u 5 9 x 4 v Y l Q q 0 / w 2 M i u r a + s b 2 c 3 c 1 v b O 7 l 5 + / 6 A p w 1 h g 0 s A h C 0 X b Q 5 I w G p C G o o q R d i Q I 4 h 4 j L W 9 w O / Z b T 0 R I G g Z 1 N Y y I w 1 E v o D 7 F S G n J z V / b H K k + R iy p p k V b 0 h 5 H x W c 3 o e l D M n E E T + 7 T 9 K p e O n 2 c V + t p W n L z B b N s T g C X i T U j B T B D z c 2 P 7 G 6 I Y 0

5 s a H / m D 3 L j y S 3 C 2 a S E p B 0 a y 9 O i d e c d j T 1 p y p n QY / v T 8 j X v 3 H z z c f N R 4 / O T p s 6 3 m 8 x c n q q g k o Q N S 8 E K e p q A o Z z k d a K Y 5 P S 0 l B Z F y O k z P D x f 5 4 V c q F S v y v p 6 X d C x g m r O M E d B O S p q / Y g F 6 R o C b D 7 Z d c 5 q Z S 7 t 7 g 0 N 7 Z m q W w n y 2 d r c f x J 2 9 u H P r 6 j m X Y l M B 7 a v E s P X i v S j o z J N b T c u K W h u 8 i Y G X M 8 D 9 s x 3 8 n 4 2 C z p d V t V / 3 u T E f / 5 l 9 a J N w x R w k z V b Y D e v A d y F a Q g s t 4 y h p / o g n B a k E z T X h o N Q o C k s 9 N i A 1 I 5 z a R l w p W g I 5 h y k d O c x B U D U 2 9 S 4 s f u 2 U C c 4 K 6 U 6 u c a 3 + 7 T A g l J q L 1 F U u Z l T r u Y X 4r 9 y o 0 t n 7 s W F 5 W W m a k + u L s o p j X e D F Y v G E S U o 0 n z s A I p m b F Z M Z S C D a r b / h P i F a f / J d O N n p R m E 3 + v S 2 t X + w / I 5 N 9 A p t o z a K 0 D u 0 j 3 r o C A 0 Q 8 X p e 7 l 1 4 l / 5 H X / l X / r f r U t 9 b e l 6 i l f C / / w Y w w O O 9 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " p U F i I L b d g h p n S p 6 1 F g M 8 D 2 I T I V U = " > A A A C y 3 i c j Z F N b x M x E I a 9 W z 5 K +

Figure 5: Schematic of our retraining method combining distillation and noise injection.

ResNet-32 on CIFAR10 with analog non-idealities: our method of combining distillation and noise injection consistently achieves the best accuracy under different analog non-ideal effects.

Table content: <table><tr><td>Noise level</td><td/><td/><td>η = 0.05</td><td/><td/></tr><tr><td>Non-ideal fluctuation type</td><td>Temporal 10% Spatial 0%</td><td>Temporal 20% Spatial 0%</td><td>Temporal 0% Spatial 10%</td><td>Temporal 0% Spatial 20%</td><td>Temporal 20% Spatial 20%</td></tr><tr><td>No retraining</td><td>93% +/-0.14%</td><td>92.98% +/-0.18%</td><td>92.98% +/-0.15%</td><td>92.95% +/-0.15%</td><td>92.94% +/-0.15%</td></tr><tr><td>Noise injection</td><td>93.18% +/-0.13%</td><td>93.03% +/-0.15%</td><td>93.1% +/-0.14%</td><td>93.15% +/-0.15%</td><td>93.11% +/-0.13%</td></tr><tr><td>Distillation and noise injection</td><td>93.56% +/-0.12%</td><td>93.55% +/-0.11%</td><td>93.55% +/-0.13%</td><td>93.51% +/-0.12%</td><td>93.53% +/-0.12%</td></tr><tr><td>Noise level</td><td/><td/><td>η = 0.1</td><td/><td/></tr><tr><td>No retraining</td><td>90.46% +/-0.19%</td><td>90.22% +/-0.27%</td><td>90.5% +/-0.2%</td><td>90.4% +/-0.23%</td><td>90.1% +/-0.3%</td></tr><tr><td>Noise injection</td><td>91.87% +/-0.17%</td><td>91.93% +/-0.2%</td><td>91.91% +/-0.2%</td><td>91.79% +/-0.18%</td><td>91.81% +/-0.17%</td></tr><tr><td>Distillation and noise injection</td><td>92.83% +/-0.18%</td><td>92.77% +/-0.14%</td><td>92.88% +/-0.14%</td><td>92.89% +/-0.14%</td><td>92.86% +/-0.15%</td></tr></table>

ResNet-50 on ImageNet at different noise levels, showing the Top-1 accuracy on the test dataset, with no quantization applied. Uncertainty is the standard deviation of different training and inference runs.

Table content: <table><tr><td>Training method</td><td>Noise level</td><td>η = 0</td><td>η = 0.02</td><td>η = 0.04</td><td>η = 0.06</td></tr><tr><td>No retraining</td><td/><td>74.942%</td><td>72.975% +/-0.095%</td><td>64.382% +/-0.121%</td><td>46.284% +/-0.179%</td></tr><tr><td colspan="2">Gaussian noise injection</td><td>74.942%</td><td>73.513% +/-0.091%</td><td>70.142% +/-0.129%</td><td>65.285% +/-0.168%</td></tr><tr><td colspan="2">Distillation and noise injection</td><td>74.942%</td><td>74.005% +/-0.096%</td><td>71.442% +/-0.111%</td><td>67.525% +/-0.162%</td></tr></table>

John Von Neumann. Probabilistic logics and the synthesis of reliable organisms from unreliable components. Automata studies, 34:43-98, 1956. Stefan Wager, Sida Wang, and Percy S Liang. Dropout training as adaptive regularization. In Advances in neural information processing systems, pp. 351-359, 2013.

Table content: <table><tr><td>A APPENDIX</td><td/><td/></tr><tr><td/><td>…….</td><td/></tr><tr><td>1 st</td><td>2 nd</td><td>l th</td></tr><tr><td/><td>…….</td><td/></tr><tr><td>1 st</td><td>2 nd</td><td/></tr></table>

Peer review (iclr2020):
The manuscript illustrates how a noisy neural network can reduce the learning capacity. To mitigate this loss, the authors propose a method that combines the method of *noise injection and *knowledge distillation*. However, from a conceptual point of view, their contribution (i.e. (10) in Section 5,) is unclear to me. Specifically, the authors are not precise about how do they merge the aforementioned previous ideas and come up with the new loss function (10). Minor comment: Please correct (7).