Noisy Machines: Understanding noisy neural networks and enhancing robustness to analog hardware errors using distillation

The success of deep learning has brought forth a wave of interest in computer hardware design to better meet the high demands of neural network inference . In particular , analog computing hardware has been heavily motivated specifically for accelerating neural networks , based on either electronic , optical or photonic devices , which may well achieve lower power consumption than conventional digital electronics . However , these proposed analog accelerators suffer from the intrinsic noise generated by their physical components , which makes it challenging to achieve high accuracy on deep neural networks . Hence , for successful deployment on analog accelerators , it is essential to be able to train deep neural networks to be robust to random continuous noise in the network weights , which is a somewhat new challenge in machine learning . In this paper , we advance the understanding of noisy neural networks . We outline how a noisy neural network has reduced learning capacity as a result of loss of mutual information between its input and output . To combat this , we propose using knowledge distillation combined with noise injection during training to achieve more noise robust networks , which is demonstrated experimentally across different networks and datasets , including ImageNet . Our method achieves models with as much as ∼ 2× greater noise tolerance compared with the previous best attempts , which is a significant step towards making analog hardware practical for deep learning . 1 INTRODUCTION . Deep neural networks ( DNNs ) have achieved unprecedented performance over a wide variety of tasks such as computer vision , speech recognition , and natural language processing . However , DNN inference is typically very demanding in terms of compute and memory resources . Consequently , larger models are often not well suited for large-scale deployment on edge devices , which typically have meagre performance and power budgets , especially battery powered mobile and IoT devices . To address these issues , the design of specialized hardware for DNN inference has drawn great interest , and is an extremely active area of research . To date , a plethora of techniques have been proposed for designing efficient neural network hardware ( Sze et al. , 2017 ) . In contrast to the current status quo of predominantly digital hardware , there is significant research interest in analog hardware for DNN inference . In this approach , digital values are represented by analog quantities such as electrical voltages or light pulses , and the computation itself ( e.g. , multiplication and addition ) proceeds in the analog domain , before eventually being converted back to digital . Analog accelerators take advantage of particular efficiencies of analog computation in exchange for losing the bit-exact precision of digital . In other words , analog compute is cheap but somewhat imprecise . Analog computation has been demonstrated in the context of DNN inference in both electronic ( Binas et al. , 2016 ) , photonic ( Shen et al. , 2017 ) and optical ( Lin et al. , 2018 ) systems . Analog accelerators promise to deliver at least two orders of magnitude better performance over a conventional digital processor for deep learning workloads in both speed ( Shen et al. , 2017 ) and energy efficiency ( Ni et al. , 2017 ) . Electronic analog DNN accelerators are arguably the most mature technology and hence will be our focus in this work . The most common approach to electronic analog DNN accelerator is in-memory computing , which typically uses non-volatile memory ( NVM ) crossbar arrays to encode the network weights as analog values . The NVM itself can be implemented with memristive devices , such as metal-oxide resistive random-access memory ( ReRAM ) ( Hu et al. , 2018 ) or phase-change memory ( PCM ) ( Le Gallo et al. , 2018 ; Boybat et al. , 2018 ; Ambrogio et al. , 2018 ) . The matrix-vector operations computed during inference are then performed in parallel inside the crossbar array , operating on analog quantities for weights and activations . For example , addition of two quantities encoded as electrical currents can be achieved by simply connecting the two wires together , whereby the currents will add linearly according to Kirchhoff ’ s current law . In this case , there is almost zero latency or energy dissipation for this operation . Similarly , multiplication with a weight can be achieved by programming the NVM cell conductance to the weight value , which is then used to convert an input activation encoded as a voltage into a scaled current , following Ohm ’ s law . Therefore , the analog approach promises significantly improved throughput and energy efficiency . However , the analog nature of the weights makes the compute noisy , which can limit inference accuracy . For example , a simple two-layer fully-connected network with a baseline accuracy of 91.7 % on digital hardware , achieves only 76.7 % when implemented on an analog photonic array ( Shen et al. , 2017 ) . This kind of accuracy degradation is not acceptable for most deep learning applications . Therefore , the challenge of imprecise analog hardware motivates us to study and understand noisy neural networks , in order to maintain inference accuracy under noisy analog computation . The question of how to effectively learn and compute with a noisy machine is a long-standing problem of interest in machine learning and computer science ( Stevenson et al. , 1990 ; Von Neumann , 1956 ) . In this paper , we study noisy neural networks to understand their inference performance . We also demonstrate how to train a neural network with distillation and noise injection to make it more resilient to computation noise , enabling higher inference accuracy for models deployed on analog hardware . We present empirical results that demonstrate state-of-the-art noise tolerance on multiple datasets , including ImageNet . The remainder of the paper is organized as follows . Section 2 gives an overview of related work . Section 3 outlines the problem statement . Section 4 presents a more formal analysis of noisy neural networks . Section 5 gives a distillation methodology for training noisy neural networks , with experimental results . Finally , Section 6 provides a brief discussion and Section 7 closes with concluding remarks . 2 RELATED WORK . Previous work broadly falls under the following categories : studying the effect of analog computation noise , analysis of noise-injection for DNNs , and use of distillation in model training . Analog Computation Noise Models In Rekhi et al . ( 2019 ) , the noise due to analog computation is modeled as additive parameter noise with zero-mean Gaussian distribution . The variance of this Gaussian is a function of the effective number of bits of the output of an analog computation . Similarly , the authors in Joshi et al . ( 2019 ) also model analog computation noise as additive Gaussian noise on the parameters , where the variance is proportional to the range of values that their PCM device can represent . Some noise models presented have included a more detailed account of device-level interactions , such as voltage drop across the analog array ( Jain et al. , 2018 ; Feinberg et al. , 2018 ) , but are beyond the scope of this paper . In this work , we consider an additive Gaussian noise model on the weights , similar to Rekhi et al . ( 2019 ) ; Joshi et al . ( 2019 ) and present a novel training method that outperforms the previous work in model noise resilience . Noise Injection for Neural Networks Several stochastic regularization techniques based on noise-injection and dropout ( Srivastava et al. , 2014 ; Noh et al. , 2017 ; Li & Liu , 2016 ) have been demonstrated to be highly effective at reducing overfitting . For generalized linear models , dropout and additive noise have been shown to be equivalent to adaptive L2 regularization to a first order ( Wager et al. , 2013 ) . Training networks with Gaussian noise added to the weights or activations can also increase robustness to variety of adversarial attacks ( Rakin et al. , 2018 ) . Bayesian neural networks replace deterministic weights with distributions in order to optimize over the posterior distribution of the weights ( Kingma & Welling , 2013 ) . Many of these methods use noise injection at inference time to approximate weight distribution ; in Gal & Ghahramani ( 2016 ) a link between Gaussian processes and dropout is established in an effort to model the uncertainty of the output of a network . A theoretical analysis by Stevenson et al . ( 1990 ) has shown that for neural networks with adaptive linear neurons , the probability of error of a noisy neural network classifier with weight noise increases with the number of layers , but largely independent of the number of weights per neuron or neurons per layer . Distillation in Training Knowledge distillation ( Hinton et al. , 2015 ) is a well known technique in which the soft labels produced by a teacher model are used to train a student model which typically has reduced capacity . Distillation has shown merit for improving model performance across a range of scenarios , including student models lacking access to portions of training data ( Micaelli & Storkey , 2019 ) , quantized low-precision networks ( Polino et al. , 2018 ; Mishra & Marr , 2017 ) , protection against adversarial attacks ( Papernot et al. , 2016 ; Goldblum et al. , 2019 ) , and in avoiding catastrophic forgetting for multi-task learning ( Schwarz et al. , 2018 ) . To the best of our knowledge , our work is the first to combine distillation with noise injection in training to enhance model noise robustness . 3 PROBLEM STATEMENT . Without loss of generality , we model a general noisy machine after a simple memristive crossbar array , similar to Shafiee et al . ( 2016 ) . Figure 1 illustrates how an arbitrary neural network layer , l , such as a typical 3× 3 convolution , can be mapped to this hardware substrate by first flattening the weights into a single large 2D matrix , Wl , and then programming each element of this matrix into a memristive cell in the crossbar array , which provides the required conductances Gl ( the reciprocal of resistance ) to perform analog multiplication following Ohm ’ s law , iout = vinG . Note that a pair of differential pair of NVM devices are typically used to represent a signed quantity in Gl . Subsequently , input activations , xl converted into continuous voltages , v ( xl ) , are streamed into the array rows from the left-hand side . The memristive devices connect row with columns , where the row voltages are converted into currents scaled by the programmed conductance , G , to generate the currents i ( yl ) , which are differential in order to represent both positive and negative quantites with unipolar signals . The currents from each memristive device essentially add up for free where they are connected in the columns , according to Kirchhoff ’ s current law . Finally , the differential currents are converted to bipolar voltages , v ( yl ) , which are they digitized before adding bias , and performing batch normalization and ReLU operations , which are not shown in Figure 1 . However , the analog inference hardware of Figure 1 is subject to real-world non-idealities , typically attributed to variations in : 1 ) manufacturing process , 2 ) supply voltage and 3 ) temperature , PVT variation collectively , all of which result in noise in the system . Below we discuss the two key components in terms of analog noise modeling . Data Converters . Digital-to-analog converter ( DAC ) and analog-to-digital converter ( ADC ) circuits are designed to be robust to PVT variation , but in practice these effects do degrade the resolution ( i.e . number of bits ) . Therefore , we consider effective number of bits ( ENOB ) , which is a lower bound on resolution in the presence of non-idealities . Hence , we use activation and weight quantization with ENOB data converters and no additional converter noise modeling . NVM cells . Due to their analog nature , memristive NVM cells have limited precision , due to the read and write circuitry ( Joshi et al. , 2019 ) . In between write and read operations , their stored value is prone to drift over time . Long-term drift can be corrected with periodic refresh operations . At shorter timescales , time-varying noise may be encountered . For most of the experiments in this paper , we model generic NVM cell noise as an additive zero-mean i.i.d . Gaussian error term on the weights of the model in each particular layer ∆Wl ∼ N ( ∆Wl ; 0 , σ2N , lI ) . This simple model , described more concretely in Section 5 , is similar to that used by Joshi et al . ( 2019 ) which was verified on real hardware . In addition , we also investigate spatially-varying and time-varying noise models in Section 5.2 ( Table 1 ) .

Summary: The manuscript illustrates how a noisy neural network can reduce the learning capacity. To mitigate this loss, the authors propose a method that combines the method of "noise injection and "knowledge distillation". However, from a conceptual point of view,  their contribution (i.e. (10) in Section 5,) is unclear to me. Specifically,  the authors are not precise about how do they merge the aforementioned previous ideas and come up with the new loss function (10).