Improving Analog Neural Network Robustness: A Noise-Agnostic Approach with Explainable Regularizations

Authors: alice  duque
pedro  freire
egor  manuylovich
dmitrii  stoliarov
jaroslaw  prilepsky
sergei  turitsyn
Venue: NeurIPS_cc_2024_Workshop_MLNCP
Published: 2024-09-11 23:42:06

Abstract: This work tackles the critical challenge of mitigating "hardware noise" in deep analog neural networks, a major obstacle in advancing analog signal processing devices. We propose a comprehensive, hardware-agnostic solution to address both correlated and uncorrelated noise affecting the activation layers of deep neural models. The novelty of our approach lies in its ability to demystify the "black box" nature of noise-resilient networks by revealing the underlying mechanisms that reduce sensitivity to noise. In doing so, we introduce a new explainable regularization framework that harnesses these mechanisms to significantly enhance noise robustness in deep neural architectures, obtaining over 53\% accuracy improvement in noisy environments, when compared to models with standard training.

Main paper: 
 introduction: 
 As the applications of artificial intelligence (AI) continue to surge, the computational power required to sustain this growth is leading to increasingly high energy consumption and a corresponding significant carbon footprint [1, 2] . This escalating environmental impact necessitates the exploration of innovative solutions to enhance computational efficiency [3] . Among these emerging approaches, analog neural networks have gained attention due to their potential to offer faster processing speeds with lower energy requirements, thereby addressing the critical need for sustainable AI development [4, 5] .However, analog circuits have a major drawback: they are naturally much more sensitive to noise compared to their digital counterparts. Noise in neural networks is not a particularly new topic of study, and extensive solutions have been proposed to overcome noisy inputs [6] and noisy labels [7] , [8] . However, since the majority of networks in literature run on digital devices, these types of noise are external to the network itself, failing to address a type of noise that greatly affects analog networks in particular: hardware noise.While it is well-recognized that hardware noise is a significant source of uncertainty in analog networks [9] , it remains a relatively underexplored central topic. Ref [10] offers a broad analysis of noise effects across various architectures. However, the proposed solutions involve architectural modifications that result in increased hardware complexity. In contrast, [11] provides valuable insights into hardware noise but focuses on device-level solutions, which limit their general applicability. Ref. [12] demonstrates promising results by employing a device-agnostic approach during training. Nonetheless, that study lacks detailed explanations regarding the mechanisms behind the observed improvements.In this work, we examine the impact of additive noise on feed-forward networks, along with the underlying mechanisms that contribute to the effectiveness of noise-aware training. In parallel, we introduce a novel regularization strategy that enhances the network's robustness to correlated and uncorrelated hardware noise. This strategy not only offers a mathematically supported explanation, but it is also empirically validated through probability density function (PDF) analysis, providing strong justification for its effectiveness. We believe that the analysis of PDF evolution in neural networks introduced and applied here is a useful and important technique for designing and optimising their performance. 
noise in neural network models: 
 Building upon the classification proposed by Ref. [10] , we categorize noise as either correlated or uncorrelated in this work. Noise is considered correlated when all neurons within a layer experience the same perturbation simultaneously, typically arising from variations in shared physical components, such as temperature, supply voltages, or laser sources. Conversely, uncorrelated noise refers to unique perturbations affecting each neuron independently, though these perturbations are drawn from the same underlying probability distribution. Consistent with other works in the literature, we model noise as being drawn from a zero-mean Gaussian distribution, with variance determined by the specific hardware characteristics. In this study, we focus on noise introduced after the activation functions (Figure 1 ) rather than within the weights. This is motivated by the fact that analog activation functions are in general more complex than passive connections and more likely to introduce noise. When noise is injected into a network trained to achieve high accuracy, a significant drop in performance is expected. This highlights the disparity between simulated and actual performance when these networks are deployed on noisy physical devices.Among the solutions discussed in the literature, noise-aware training [12] assumes that neural networks can develop resilience to noise if trained with exposure to it. For that purpose, noise similar to what is expected during inference is injected during training, resulting in enhanced robustness. This method has shown promising results, leading to nearly a four-fold improvement in forecasting accuracy for a time-series prediction task. However, there is still a lack of understanding regarding the mechanisms that drive these improvements. Specifically, it remains unclear how neural networks adjust to mitigate both correlated and uncorrelated additive noise so effectively. In the following sections, we present, to the best of our knowledge, for the first time, an analytical and mathematically explainable regularization strategy to achieve comparable noise mitigation results. Without loss of generality, we apply our approach to a fully connected multi-layer perceptron (MLP) with 2 hidden layers of 300 neurons each, with sigmoid as the activation function. 
designing explainable regularizers: 
 Consider activations a (l-1) j of j th neuron in layer l -1 are corrupted by noise n(l-1)j ∼ N (0, σ 2 n ).In this scenario, these activations interact with weights wEQUATIONwhich can be written asEQUATIONThe last term corresponds to the total noise contribution on pre-activation z(l)i , or the error factor. If noise is correlated, then nj = n for all j, so the error factor becomes:j w (l) ij n(l-1) j = n(l-1) j w (l) ij .(EQUATIONThis means that the error factor can be effectively eliminated if we make j w (l) ij = 0. In other words, correlated noise can be mitigated by ensuring that each row of the weight matrices sums to zero. This allows us to formulate the first explainable regularizer for additive correlated noise, which enforces this mathematical condition. By introducing a regularization term into the loss function during training, we can effectively impose this constraint, leading to improved noise robustness. The modified loss function becomes: For visual inspection, Fig. 2 illustrates the effects of regularization on the distribution of weights per row, compared to the original neural network model trained without noise. The graphs depict the mean (solid line) and standard deviation (shaded areas) of each row within the weight matrices, with layers ordered by ascending mean values. As predicted by Eq. 4, all rows-except those of the input weight matrix-have their mean values driven toward zero. This exception arises because noise only affects activations, meaning the input weights do not directly interact with it. In the following section, we will present evidence demonstrating the improvement in accuracy achieved through this straightforward regularization strategy.EQUATION) (a) (b)For uncorrelated noise, canceling the error factor becomes more complex. Instead of attempting to eliminate it entirely, we aim to mitigate its propagation to subsequent layers. Given that all individual noise contributions within layer l -1 have zero mean Gaussian PDF, the error factor is also Gaussian with zero mean. Furthermore, since for a known a i can be interpreted as a random Gaussian variable ẑ(l)i ∼ N (µEQUATIONFrom the known pdf p z (z) of ẑ(l) i , and given that f (•) is monotonic, one can derive the PDF p a (a) of aEQUATIONwhere f -1 (•) is the inverse of activation function. (l) i ∼ N (µ (l) ij , 0.2) for different values of µ (l)ij . It's observed that input distributions centered further from the origin lead to output distributions of much-reduced variance. Intuitively, this occurs because noise fluctuations in the saturated regions of the activation function (where the derivative is close to zero) are largely suppressed and not transmitted to the output. Thus, in analog neural networks, if a set of weights and biases can shift the operating point of pre-activations toward these saturated regions, noise-induced perturbations are less likely to propagate to subsequent layers. This mechanism effectively reduces the impact of uncorrelated noise, enhancing the network's robustness to such perturbations. This also can give some design ideas for the activation function shape when this freedom is available.It is important to point out that, in the output layer, no additional nonlinearities are available to counteract noise contributions, which will inevitably affect the readout values. However, we found that this impact can be mitigated by reducing the magnitude of the output weights. In typical scenarios, reducing weights alone does not enhance noise performance, as both the signal and noise are attenuated equally, resulting in no real improvement in the SNR. However, if the intermediate layers meet the condition where pre-activations operate predominantly in saturated regions, the activation values will also be highly saturated. In our case, which uses the sigmoid activation function, this implies that the majority of activation values will be either 0 or 1. In such a distribution, reducing the output weights does not lead to information loss for activations that are zero-these values constitute a significant portion of the activations in the saturated regime. Thus, noise contributions are reduced without loss of information from the zero-valued activations.Moreover, for non-zero activations, which are at the maximum possible value (1 in the case of sigmoid), the output remains resilient to noise. This dual effect-noise suppression for zero activations and noise insensitivity for saturated ones-explains how reducing output weights, when combined with saturated intermediate layers, can mitigate the impact of noise on the final output. We can incorporate these conditions into our training by including new regularizer terms into our loss function, which becomes:EQUATIONin which f ′ (z i ) is the activation function's derivative at z i and λ i are tunable hyperparameters.The first regularization term penalizes pre-activations that fall within regions of high derivative in the activation function, while the second term penalizes large weight magnitudes in all layers except the input layer. The constraint on the hidden layer weights addresses a practical issue encountered during training. When we enforced pre-activations to move away from the origin, the network initially complied by amplifying the weights in the hidden layers. Although this approach effectively shifted the operating point toward the saturated region, it inadvertently amplified noise contributions. This increased noise then "leaked" into the linear region of the activation function, propagating through subsequent layers and negating any performance improvement.To prevent this, we apply a light constraint on the hidden layer weights, ensuring that the network does not resort to this suboptimal strategy during training.  
results and discussion: 
 Having established the explainability of the proposed regularization, we now demonstrate that the key features of noise-resilient networks are successfully incorporated by our regularization techniques, yielding significant performance improvements. To validate these claims and assess the overall performance, we evaluate the regularized network on three computer vision tasks, using the MNIST [14] , Fashion MNIST [15] and QuickDraw [16] datasets. The results highlight the efficacy of the regularization in enhancing noise robustness while maintaining strong task performance. For the sake of reproducibility, all code used in these experiments is available in [17] .In all tasks, the application of regularization effectively eliminated the impact of correlated noise, as shown in Fig. 4a )-(c), resulting in a flat accuracy curve with no observable performance degradation. For uncorrelated noise, substantial improvements were also achieved. On the MNIST dataset, the network experienced only a 4.77% drop in accuracy when subjected to hardware noise with variance as high as σ 2 = 1.0, compared to its performance in a noiseless environment. By contrast, a standard network trained without any noise-aware techniques showed a dramatic 41.52% accuracy decline under the same conditions. Similarly, in the more challenging Fashion MNIST task, regularization reduced the accuracy drop to 7.22%, compared to 53.86% for the standard trained network. For QuickDraw, the accuracy drop went from 65.58% to 35.29%, when the proposed regularization was in place. It's worth mentioning that 20 classes were selected from the original QuickDraw dataset -twice as many as MNIST and Fashion MNIST -, which reflect in the poorer performance comparatively. Nonetheless, these results demonstrate a significant enhancement in noise robustness. However, this improvement comes with a slight penalty to the maximum achievable accuracy.For MNIST, a network with standard training achieved 98.06% accuracy in a noise-free environment, while the regularized network reached 95.62% under the same conditions. Similarly, for Fashion MNIST, the accuracy for the standard network was 87.78%, while the regularized network achieved 85.37%. For QuickDraw, the figures are 78.11% and 75.12%. It is important to note that this accuracy reduction is also observed in networks trained with noise exposure.The promising results from the treatment of correlated noise indicate that the imposed weight constraint is relatively mild, allowing the network to assimilate it without compromising overall performance. In contrast, the regularization designed for uncorrelated noise introduces a more stringent constraint, which imposes a notable trade-off during training and can limit the maximum achievable accuracy. Despite this, the overall performance improvements underscore the effectiveness of the method. However, a key challenge of the proposed approach lies in the difficulty of optimizing the training process. The custom regularization terms require a systematic and often repetitive search for the optimal hyperparameters λ i , typically through a trial-and-error process. This adds significant complexity to the training pipeline, substantially increasing both the training time and computational cost. Fine-tuning these regularization coefficients is critical for balancing the trade-offs between noise suppression and accuracy, but it also represents a bottleneck in terms of scalability, especially in larger models or more complex architectures. 
conclusion: 
 In this work, we explored the fundamental mechanisms that allow a neural network to achieve resilience to hardware noise without introducing additional architectural complexity, relying solely on optimized weight and bias distributions. Our novel approach, based on the analysis of the pdf evolution throughout the network, offers a valuable design and optimization tool for improving noise robustness. By presenting explainable regularization techniques, we demonstrated an over 53% improvement in accuracy under high levels of hardware noise, underscoring the effectiveness of the method. Importantly, the proposed technique is device-agnostic, offering broad applicability across various analog neural networks affected by additive noise. This generalizability, combined with the simplicity of implementation, positions our approach as a significant advancement for the design of high-performance, noise-resilient neural networks in real-world hardware applications. 
 
 Appendix: Submitted to the Second Workshop on Machine Learning with New Compute Paradigms at NeurIPS (MLNCP 2024). Do not distribute. 



Reviews: review: title: title: Interesting explanation but no gain compared to state of the art
main_review: main_review: The authors propose a new regularization function to reduce the impact of the activation noise and explain why it works. Zeroing the sum of row weights for correlated noise completely clears the noise. For uncorrelated noise, the authors attempt to concentrate the preactivation values in the saturated part of the activation function.

Results show a good impact on the accuracy compared to an approach with standard training. Nevertheless, a "noise-aware" training gives even better results. It is not completely clear what the authors mean by "noise-aware" training. Since the results seem better than the proposed regularization function, authors should provide details about their test bench and why they claim a contribution.

Here are a few other things that could enhance the paper :
1) Evaluate the proposed method on networks and datasets that are more complex than MNIST and MNIST fashion.
2) Double-check equation (7). It is unclear why you only seem to use the preactivation values of layer two, but you consider the square weight values of all layers.
3) Your proposed method for uncorrelated noise and Sharpness-Aware-Minimization (SAM) probably share some similarities since you want the activations to be in a low derivative region. Please discuss this.
title: title: A simple, yet interesting study of possible regularizations to make networks more noise robust
main_review: main_review: This submission explores possible regularizations to reduce the influence of gaussian activation noise on the network outputs. The authors give expressions for correlated noise as well as uncorrelated noise. Noise robustness is an important property of neural networks if they are to be run on analog, quantum, or other volatile systems.

The aim of the paper is to replace the need to train neural networks (assuming they are trained digitally) explicitly with noise which would require detailed knowledge of noise strength and correlation by introducing regularization terms. Another point of view of what this paper tries to achieve is that it seeks to explain what actually happens to the weights when networks are trained under noise. This explanatory perspective, however, is only empirically backed, not shown in detail or as direct comparison between noise-trained and regularization-trained networks.

For the case of correlated noise, the authors find a simple way to eliminate the influence of the noise which holds up in experiments.
Furthermore, for the case of uncorrelated noise, networks trained with these regularizations still experience a significant drop (4.77% for MNIST, 7.2% for Fashion-MNIST) in accuracy for strong noise. This stands in contrast to networks that are trained under noise which might still be the preferred option for high noise variances (if a detailed noise model is available).

The proposed mechanisms that networks can resort to in order to deal with uncorrelated noise such as driving the signal into the constant part of the non-linearity likely explains the overall lower achievable accuracy on MNIST/Fashion-MNIST.

The experimental section of the paper could be more extensive. The theory could be extended to transformers where activations in queries and keys are multiplied, leading to potentially different noise distributions in the resulting variables. I believe the scope is sufficient for a workshop.