Noise Injection Node Regularization for Robust Learning
Authors: Noam Itzhak Levi
Itay Mimouni Bloch
Marat Freytsis
Tomer Volansky
Venue: ICLR 2023 poster (2023)
Keywords: Reguralization
Input corruption
Noise Injection
Deep Learning

Abstract: We introduce Noise Injection Node Regularization (NINR), a method of injecting structured noise into Deep Neural Networks (DNN) during the training stage, resulting in an emergent regularizing effect. We present theoretical and empirical evidence for substantial improvement in robustness against various test data perturbations for feed-forward DNNs when trained under NINR. The novelty in our approach comes from the interplay of adaptive noise injection and initialization conditions such that noise is the dominant driver of dynamics at the start of training. As it simply requires the addition of external nodes without altering the existing network structure or optimization algorithms, this method can be easily incorporated into many standard problem specifications. We find improved stability against a number of data perturbations, including domain shifts, with the most dramatic improvement obtained for unstructured noise, where our technique outperforms other existing methods such as dropout or $L_2$ regularization, in some cases. We further show that desirable generalization properties on clean data are generally maintained.

Published as a conference paper at ICLR 2023
NOISE INJECTION NODE REGULARIZATION FOR Ro- BUST LEARNING
Noam Leviš & Tomer Volansky
Itay M. Bloch
Department of Physics Tel Aviv University
Berkeley Center for Theoretical Physics, University of California and
Tel Aviv, Israel {noam, tomerv) i1
Theoretical Physics Group,
Lawrence Berkeley National Laboratory, Berkeley, CA, U.S.A. itayblochmeberkelee..c
Marat Freytsis
NHETC, Department of Physics and Astronomy Rutgers University
Piscataway, NJ, U.S.A.
marat. marat.freytsi srrat..frettsiirruttggeee edu
ABSTRACT
We introduce Noise Injection Node Regularization (NINR), a method of inject- ing structured noise into Deep Neural Networks (DNN) during the training stage, resulting in an emergent regularizing effect. We present theoretical and empiri- cal evidence for substantial improvement in robustness against various test data DNNs when trained under NINR. The novelty in our approach comes from the interplay of adaptive noise injection and initializa- tion conditions such that noise is the dominant driver of dynamies the start of training. As simply requires the addition of external nodes without altering the existing network structure optimization algorithms, this method can be easily incorporated into many standard architectures. We find improved stability against number of data perturbations, including domain shifts, with the most dramatic improvement obtained for unstructured noise, where our technique outperforms existing methods such Dropout or L2 regularization, in some cases. Further, desirable generalization properties on clean data are generally maintained.
perturbations for
INTRODUCTION
Nonlinear systems often display dynamical instabilities which enhance small initial perturbations and lead to cumulative behavior that deviates dramatically from a steady-state solution. Such in- stabilities are prevalent across physical systems, from hydrodynamic turbulence atomic bombs (see Jeans & Darwin (1902); Parker (1958); Chandrasekhar (1961); Drazin & Reid (2004); Stro- gatz (2018) for just a few examples). In the context of deep learning (DL), DNNs, once optimized via stochastic gradient descent (SGD), suffer from similar instabilities as a function of their inputs. While remarkably successful able to perturbations in their input data as result (Szegedy et al., 2014). Concretely, after training, even small changes the inputs at deployment can result total predictive breakdown.
multitude of real world tasks. DNNs are often surprisingly vulner-
One may classify such perturbations with respect to the distribution from which training data is implicitly drawn. This data typically assumed have support over (the vicinity of) some low- dimensional submanifold of potential inputs, which is only learned approximately due to the discrete nature of the training set. To perform well during training, network need only have well-defined behavior on the data manifold, accomplished through training given data distribution. However, data seen on deployment can display other differences with respect to the training set, as illustrated
$ Equal contribution

---

Published as a conference paper at ICLR 2023
Smagge
Generalization
Commaall e,y)
laals,y)
Falt + Car
Domain Shift
Figure I: Illustration of perturbations to data inputs with respect to the joint probability distribution manifold of features and labels. Points indicate (sample, label] pairs (æ, y). where different colored points correspond samples drawn from different marginal distributions. Black points represent pairs from training dataset {æ., Yt Jim1, with the red spheres indicating corrupted inputs, determined by shifted distribution functions fcorrupted + 6, y). The gray arrow represents an adversarial attack, performed by ascending up the gradient of the network output to reach the closest decision boundary, while generalization from training to test data is depicted as interpolation from black to blue points. Finally, domain shift is a shift in the underlying distribution on the same manifold, depicted by the green arrow and points.
in Fig. 1. These distortions introduce vulnerabilities that are a crucial drawback of trained DNNs, making them susceptible to commonly occurring noise which is ubiquitous in real-world tasks. By studying how networks dynamically act mitigate the negative effects of input noise, identify novel dynamical regularization method starting in a noise-dominated regime, leading to more robust behavior for range of data perturbations. This is the central contribution of this work.
Regularization involves introducing additional constraints order to solve an ill- posed problem to prevent weer-fitting. In the context of DL problems, different regularization schemes have been proposed (for a review, see Kukačka et al. (2018) and references therein). These methods are designed to constrain the network parameters during training, thereby reducing sensi- tivity to irrelevant features in the input data, as well as avoiding overfitting. For instance, weight norm regularization (L2, L1 etc.) (Cortes & Vapnik, 1995; Zheng al., 2003) can be used to reduce overfitting to the training data, and is often found to improve generalization performance (Hinton, 1987: Krogh & Hertz, 1991; Zhang et al., 2018). Alternatively, introducing stochasticity during training (e.g., Dropout (Srivastava et al., 2014)), has become a standard addition to many DNN ar- chitectures, for similar reasons. These methods are mostly optimized reduce the generalization error from training to test data, under the assumption that both are sampled from the same underlying distribution (Srivastava et al., 2014). Here, we propose new method which is instead tailored for robustness. Our method relies on noise-injection, that actively reduces the sensitivity to uncorrelated input perturbations.
Background:
Our contribution: In this paper, employ Noise Injection Nodes (NINs), which feed random noise through designated optimizable weights, forcing the network to adapt to layer inputs which contain no useful information. Since the amount of injected noise is free parameter, at initialization we can set it to be anything from a minor perturbation to the dominant effect, leading to a system breakdown for extreme values. The general behavior of NINs and how they probe the network is the main goal of Levi et al. (2022), while we focus here on their regularizing properties in different noise injection regimes. The results of Levi et al. (2022) are explicitly recast in the context of regularization in App. C for a linear model, which captures the main insights.
Our study suggests that within certain range of noise injection parameter values, this procedure can substantially improve robustness against subsequent input corruption and partially against other forms of distributional shifts, where the maximal improvement occurs for large noise injection mag- nitudes approaching the boundary of this window, above which the training accuracy degrades to random guessing. To the best of our knowledge, this regime has not been previously explored.
In the following, we analyze how the addition of NINs produces regularization scheme which we call Noise Injection Node Regularization (NINR). The main features of NINR are enhanced stability, simplicity, and flexibility, without drastically compromising generalization performance. In order to demonstrate these features, we consider two types of feed-forward architectures: Fully Connected Networks (FCs) and Convolutional Neural Networks (CNNs), and use various datasets train the systems. We compare NINR robustness improvement with standard regularization methods, as well performance of these systems when using input corruption during training (CDT). Our results
2

---

Published as a conference paper at ICLR 2023
Fully Connected DNN
(b) Convolutional DNN
No Regabrrbuke
b--000--0a Oxae
Tew Accuracy
TRE-NNR- Dicay
60
60
TEE-AMR-Casqu
(2
60
40
Disps
COT-34
0.6
0.0 0.2
0.4
"mnion
Figure 2: Robustness against random input perturbations tested on FC network (left) and CNN (right). Test accuracy vs. the scale of imput noise corruption defined in Eq. (7) is shown for L2. Dropout, in-NINR, full- NINR and CDT with maiee 0.4. Shades indicate 2 standard deviations estimated over 10 distinct runs. For the imput-NINR (full-NINR) fully- fully-connected implementations we take a. = 51.8 (16.4) in the decay phase and O, 231.6 (51.8) in the catapult phase. Similarly, for the convolutional implementations we take a, = 2.8 (0.9) in the decay phase and o, 87.5 (62) in the catapult phase. This key result illustrates that NINR significantly increases the robustness of generic architectures trained on the FMNIST dataset, while marginally affecting generalization (anoiee 0). Comparing the CDT and input-NINR curves demonstrates the advantage of our regularization method. While both techniques perform similarly well on data corruption of muise = 0.4, CDT is significantly worse on clean data. This is result of the CDT network being forced to fit both noise and data, without the ability to suppress the latter, a crucial attribute of NINR. Here, the learning rate is fixed to r 0.05 with mini-batch size B 128. Each training run performed for 500 SGD training epochs in total, or until 98% training accuracy has been achieved. For further details, see Sec. and App. A.
can easily be generalized to other architectures and more complex NINR topologies. In Fig. 2, present our main results, comparing networks trained the FMNIST dataset and demonstrating improved robustness against input perturbations without compromising generalization on clean data.
The paper is organized as follows. In Sec. 2 we briefly review important analytical and empirical results that are explored in depth in the work of Levi et al. (2022), demonstrating in this work how NINs implicitly generate adaptive regularization terms in the loss function. In Sec. 3, we empirically study the effectiveness of NINR. We begin by evaluating its effect on robustness against perturbations, including domain shifts and those adversarially designed, demonstrating the enhanced performance of NINR. We then verify that generalization performance on clean data is not hindered by training with NINR. We discuss related work Sec. 4, finally concluding Sec. 5.
NOISE INJECTION NODES REGULARIZATION
In the following sections explain how an effective regularization scheme against input corruption naturally emerges as a consequence of adding a NIN to a DNN. First, we discuss how the NIN generates implicit regularization terms directly from computing the effective loss function. Then, we review the adaptive nature of these terms as they relate to Noise Injection Weight (NIW) dynamics during training, and discuss the expected robustness gains depending on the evolution of the NIWs.
2.1 EMERGENT REGULARIZATION TERMS
In order to see how NINs generate implicit regularization terms, we study a vanilla feed-forward DNN setup. Consider a supervised learning problem modeled by a neural network optimized under SGD, with associated single sample loss function, L: Rdi + R. The loss depends on the model parameters 0 = (W(O), b(0)e- O,..., N₁. 1}. where N₁, is the number of layers, and the weights and biases associated with given layer are ((()) E Redexde+1, 1,(5) E Rde+1, At each SGD iteration,
mini-batch B consists of of labeled examples, { y:) [el RRdin Rdiahnl, The addition of a NIN in a given layer, εNI, corresponds to random scalar input, E R, sampled repeatedly for each SGD training epoch from chosen distribution! connected via NIWs WNI E R 111 We define for given layer f, the preactivation 2 (() + ((0) therefore the addition of a NIN to dense layer results in translation to the preactivation at ENI. 2 z (fxx)) + cWNI-
'One may also generate only once, before training. We empirically find no difference hetween the two options, which is expected from large batch averaging. For [B]] S 10, differences begin to emerge.
3

---

Published as a conference paper at ICLR 2023
The batch-averaged loss function including NIN can be written as series expansion" in the noise translation parameter NII
L(O, WNI)
Σ [u.g.c]EB
C(O, ((,
e (0 C(0; x, y).
(1)
B
B
{..g,c)EB
Equation (1) follows from noting that the NIN induced translation can be written as an operator. For further details see App. B. Expanding in the parameter eWNI, we obtain an infinite series given by
L(O,, WI) = L()) + Σ Re WNI).
(2)
A==1
Here, L(O) is the loss function in the absence of any NIN, while Ra are batch-averaged derivatives of the loss function with respect to the preactivations at the noise injected layer.
1
(eW V y((x)))* )* L(O; x.)
Rx WNT)= =
(3)
B
k!
These functions are products of the moments of the injected noise, the values of the NIWs them- selves, and preactivation derivatives of the loss function in the absence of injected noise.
It is impossible to estimate when perturbative analysis in is valid without specifying L(0; æ. y), as all R2 may become equally important, or the series itself may not converge. Furthermore, since we will be interested in rather large values of e, where the effect of higher R₂ terms is noticeable, the validity of the perturbative calculation is called into question even further. However, in order to gain intuition, we first study how the training procedure is altered by the NIN in the limit of small << 1. To make further progress, we will later validate our analysis below using a combination of empirical tests, and an investigation of a linear toy model where R2 0 for k 2. For sufficiently small and analytic activation and loss functions, the series converges and the full loss is -epr-opppo by the first two leading terms in For the rest of this work, we consider noise sampled from distribution with zero mean, relaxing this assumption only for some empirical results in App. D.2. Under this assumption the first two terms can be cast into simple forms,
1 W WW WNI. R₂
(4) Here, batch averaging is denoted by (---), while geNi = Ve0t y) and Hesi V VeuNo Vi C(0, L(0, x. y) are the network-dependent local gradient and local Hessian, respectively. As proven in Levi et al. (2022), the magnitude of R1 can then be estimated using (cgeg)) ~ (g ( BB]) where (gt is the vector of the batch-averaged squared values of the local gradients and o is the variance of the injected noise", while R2 may be estimated using (e"Hext) Re o (HHra)
R1 W (Egex))
2
up to corrections scaling as O ((11/BB).
While R1 may take both positive and negative values, the sign of HeNT depends on the network architecture. Since the spectrum of the local Hessian is generally unknown, our analytical results are only valid for certain limiting cases. Particularly, we focus the case of Mean Squared Error (MSE) loss and linear activation functions, where we find that the local Hessian H a positive semi definite (PSD) matrix, implying that R2 is a strictly non-negative penalty term, and R2 terms with k > 2 vanish identically. For the motivated case of piecewise linear activations, was shown in Botev et al. (2017) that the local Hessian is PSD, aside from non-analytical points, hinting that R2 acts as a regularizer for these networks as well. This implies that for networks with piecewise linear activations and MSE loss, an analysis similar to ours below, which keeps only the first two terms in the expansion of Eq. (2), is expected to hold not only for small but also for large values. We will use this construction to understand how these terms evolve during training in the next section.
"In practice, piecewise analytic activation functions such ReLU are often used, and if the noise causes the crossing of non-analytic point, the above expansion receives corrections. Empirically, we find this subtlety to not change any of our qualitative conclusions.
"We note that the noisy loss function Eq. (1) is invariant under the simultaneous rescaling of uNi - Xusi and + X -'.. Nonetheless, the SGD optimization equations are not invariant under this transformation. implying. in particular, that the value of the injected noise variance, o?. is a relevant parameter, not degenerate with the initialization values of the NIW. As a consequence, in order to fully explore the parameter space of noise injection, the noise (or more precisely, its variance) cannot be assumed to be small, and large noise injection values must be considered.
4

---

Published as a conference paper at ICLR 2023
(a) Decoupled
(b) Decay
(c) Catapult
(d) Divergent
100
10
FMNIST -NS
metwork
10
0.5
0.5
2
100
10
0.2
0.5
0.3
epochs
opocias
epochs
epocis
Figure 3: NIW dynamics during training for the various phases discussed in Sec. 2.2, for single hidden layer FC network with ReLU activations trained the full FMNIST dataset, as specified in App. Here, we show the evolution of the NIWs norm (blue) as well as the hidden layer weights norm [W()x)+)) (red) against the training (violet) and test (green) loss (solid) and accuracy (dashed). Left to right: The NIN magnitude determines the phase of the system, ranging from the smallest amount in the decoupled phase, an overwhelming amount the divergent phase. The behavior displayed by the NIWs, as well as the loss corroborates the predictions discussed in Sec. 2.2 and App. C, with experimental details in App. A.
The interpretation of R1 and R2 can now be made clear: Ri induces constrained random walk for in the norm of noise injection weights as well as for the data weights at layers E >>> lNT, with step size that changes according to the local gradient during training. On the other hand, R2, which doesn't depend on [B), can be understood as straightforward regularization term for the local Hessian, working to reduce its eigenvalues. These results imply that in the limit of large batch size, and in particular full batch SGD (ie., gradient descent), regularization via R2 is dominant." 4.
Further understanding of why pushing the local Hessian to smaller eigenvalues expected to reduce the sensitivity to noise corruption comes by looking at the loss for corrupted inputs. Consider there- fore a network without a NIN but with corrupted inputs, described by the substitution, æ + x + 6, with ō a random vector. To arrive at similar expressions to Eqs. (1) to (4), one can transform the preactivations z
(0) + (0))) 8 to obtain,
(1 .00 L(O: x. y). {a.y)EB
( -
(5)
B
Under the assumption that the components of the vector 5 are drawn i.dd from N(0,0f] the first two terms above assume simple forms', similar to Eq. (4),
(6) As before, for sufficiently large [B], Ri is subdominant and the main regularization term due to the noise is dictated by Ho. Thus if NIN is inserted to the first layer. it will act to reduce Hy and thereby reduce the sensitivity to data corruption. Furthermore, since DNN structure in general, and loss function in particular, couples the input layer to all succeeding layers, Ho contains information about deeper layers and will benefit from reducing the local Hessian away from the input layer. In Sec. we show results for NINs coupled to the input layer or to all layers. The above readily generalizes in this setup, resulting multiple emergent regularization terms, which we briefly discuss in App. B.
R1 = (0)
ga), R₂ === Tr
Despite the similarities in their descriptions, we stress that a system trained on corrupted data and a system with NIN are not the same. In the former the noise cannot be dynamically reduced without dramatically altering the optimization trajectory, implying that the DNN is not expressive enough to memorize the full data information (Ziyin al., 2022). Conversely, in the latter, the noise has its own weights and the system can therefore improve by suppressing them without harming generalization. Nonetheless, both systems are driven towards regions with smaller local Hessian eigenvalues.
2.2 EVOLUTION OF NOISE INJECTION WEIGHTT
The dynamical nature of the NINR and the corresponding NIWs strongly depends the noise distribution, parameterized in this study by a.. While the NIWs are updated with each learning
'In fact, it is shown in Levi et al. (2022) that all odd-terms in the expansion Eq. (2), are suppressed the square root of the hatch size, while the
are 'We comment on the slight subtlety of biases in Eq. (6). In any reasonable scenario, biases would not be corrupted, but if bias is treated as the zeroth component of z. the zeroth component of 5 should he = O. Taking this into account, the trace operation of Eq. (6) should
over the zeroth dimension.
5

---

Published as a conference paper at ICLR 2023
step, only under certain conditions is their impact the network performance actively suppressed as the training progresses. Below we briefly describe four distinct phases of the NIWs. These are illustrated in Fig. 3, where we show the evolution of the relevant quantities (weights, loss, accuracy) for a model trained on FMNIST, demonstrating the different behavior in each phase. A complete treatment of these phases is discussed in Levi et al. (2022), while brief derivation relating them with regularization is given in App. C for a linear network.
Decoupled phase. For O. <> 1 one has Ri >> R2, and the correction the loss function may assume positive and negatíve contributions. As consequence, the NIWs follow a small-step random walk without substantially affecting the behavior of the network.
Decay phase. For larger but not too large de, one may ensure R₂ > R1 at initialization while the NIN can still be treated perturbatively. In this regime, the NIWs initially experience exponential decay until R2 ~ R1, at which point they evolve according to the stochastic gradient. It is in this phase that one can begin to see noticeable improvement in robustness, with only minor slowing of the training. Increasing o, boosts the improvement until another phase is encountered.
Catapult phase. The discrete nature of the training algorithm will result in stiff numerical regime at sufficiently large de- Above a critical value (for the linear network discussed in App. C we find Fe.cat 2d /1 where dess is the dimension of the NIN layer and ry is the learning rate), the effect of the NIN on the network is so significant that it causes an initial increase of the data weights, which in turn leads to an exponential increase for the loss function, followed by a recovery to a new minimum The improvement in robustness is most extreme in this phase; however, the convergence of the network is slowed somewhat, rendering the usefulness of this phase to only some applications. It is possible that a scheduled increase of the training rate after the recovery from the initial increase in the data weights could speed up the convergence. We leave such investigation to future work.
Divergent phase. Further increasing da leads the DNN to a breakdown of the dynamics, where the network is unable to suppress the NIN and thus cannot learn any information.
The above discussion of phases as a function of a, should be taken as schematic. Other hyperpa- rameters, such as the batch size, may also influence the phase diagram. Nonetheless, we empirically observe these phases repeating across multiple architectures and tasks, and find them to broadly cap- ture the evolution of the NIWs. Overall, the decay and catapult phases are expected to produce an increase in robustness against input perturbations, and we empirically verify this expectation in the following sections. While we only have an analytic prediction of de,eat for a simple linear network, in other architectures can also be obtained empirically using only the training data.
3 EXPERIMENTS
In this section, we empirically show the effect of NINR on robustness for the different phases of noise injection, following similar methodologies to Hoffman et al. (2019). After discussing the two different architectures used in this paper, we begin our investigation by demonstrating that in certain cases NINR provides a significant increase in robustness against corruption of input data by random perturbations. We then discuss the performance of NINR for domain shifts, demonstrating its effectiveness. Next, we verify that NINR does not drastically reduce the network accuracy at the original task (e.g., before corruption). This is equivalent to ensuring the generalization properties of the network are not harmed due to the addition of NINs. In the main text we present results mostly for the FMINST dataset (Xiao et al., 2017). These results also extend to more complex scenarios, demonstrated in similar experiments for the CIFAR-10 10 (Krizhevsky et al., 2014) dataset in App. E, while evidence for improvement against adversarial attacks is given in App. D as well as results for other noise distributions and optimizers beyond SGD.
Throughout this section, we compare NINR to both unregularized DNNs, and networks explicitly regularized using L2 or Dropout. We also compare NINR to implicit regularization by training with varying amounts of input data corruption. For all of our experiments, we use either an FC or a CNN (see Fig. 2 and App. A for full details). We optimize using vanilla SGD with cross-entropy loss. We preprocess the data by subtracting the mean and dividing by the variance of the training data, as is done for all subsequent datasets. The learning rate is fixed n = 0.05 with mini-batch
An analogous phase related the size of the training step was discussed in Lewkowyez et al. (2020).

---

Published as a conference paper at ICLR 2023
Node
fNNNNR
Noise Node
eNINR
Wau
xif
Wst
f Walf
Wilf
Whstlghae Wist
Data
Pre activatioo
WWW
Representation
Representation
Figure 4: Left: An illustration of fully connected NINR (feNINR) for which Noise Injection Node is appended to representation (0) Right: Implementation of NINR for convolutional network (cNINR), where the Noise Injection Node is connected pixel-wise the image representation (f be subsequently fed into a convolutional layer.
size 128. Each training run performed for 500 SGD training epochs in total, or until 98% training accuracy has been achieved, unless otherwise specified. All test accuracy evaluations are done with the NIN output set to 0, i.e., E 0. The model parameters 0, WNI are initialized at iteration 0 using normal distribution as 2e WN.o = 1/d2, 1/d The hyperparameters chosen match reference implementations: the L2 regularization coefficient (weight decay) is set to AwD = 5 10-1 and the dropout rate is set to Parop = 0.5. When using L22 or Dropout, they are applied at/after each layer. When using input CDT as regularization method, we corrupt the input data according to Eq. (7) below. We further stress that once a NIN has been added to the network, no further modifications the training algorithm or architecture are required, and after choosing where to connect the NIN, the only free parameter is the injected noise variance o².
3.1 REALIZATIONS IN DIFFERENT ARCHITECTURES
The way in which NINR implemented depends on the type of layer to which the NIN is connected. Here, we comment on the two different realizations of NINR used in our experiments and depicted in Fig. 4. For both realizations we consider two distinct topologies: either we add a NIN at the input layer (in-NINR)
couple the NIN to every hidden layer including the input one (full-NINR).
Fully Connected Layers In the case of dense FC layers, we implement NINR (which we denote by feNINR) by extending the input vector by an additional noisy pixel e, initialized randomly per sample at each training epoch, and densely connecting the modified input vector to the next layer. The theoretical discussion in Sec. was derived for realization of this type.
Convolutional Layers Connecting NIN at the input of convolutional layer raises the need for a procedure for Convolutional NINR (cNINR). Since FC layers are insensitive to the input image geometry, taking æ > {z, <} is tantamount to adding noise mask for the entire input. In CNN, the same interpretation can be maintained by adding the noise to the input directly in a pixel-wise fashion, (2) + x (?) + WNI-C e, which is subsequently fed into the convolutional layer. Importantly, this modification preserves the form of the original layer, while converging to the original (*) for either 0, - 0 or ||NTT|
O. This can also be thought of as adding auxiliary layer that is non-dynamic identity matrix from the perspective of all data weights, while being densely connected from the perspective of the NIN?.
3.2
ROBUSTNESS AGAINST DISTRIBUTIONAL SHIFTS
3.2. INPUT CORRUPTION
Often, training is done with examples taken in ideal conditions, which would not always exist in real-world data. This implies that the test data would be sampled from a distribution that identical to the one trained on, albeit with an added noise component. To test the stability of networks against natural corruption, we perturb each test input image according
o2 noise T, + n
7 As procedure for cNINR preserves the structure of the original (1), it can be casily applied for other architectures beyond convolutional layers, including densely connected layers.
7

---

Published as a conference paper at ICLR 2023
Table I: A test of domain shift, trained MNIST up 100% training accuracy and evaluated on the USPS test dataset, comparing L2. Dropout, in-NINR, and full-NINR as discussed the text. We exclude CDT from this table, as it is not as prevalent as the other regularization schemes. and is tailored for random noise. The fully-connected and CNN NINR noise magnitudes are those of Fig. 2. Errors indicate 20 confidence intervals over 10 distinct runs for full training. We find the test accuracy on the full clean MNIST datasct to be similar across all regularization schemes, with 97% and 98% for FC and CNN respectively.
in-NINR (Decay)
In-NINR (Catapult)
full-NINR (Decay)
full-NINR (Catapult)
None
L2
Dropout
FC(%)
82.3 + 2.0
82.5 + 1.2
87.5 1.2
82.1 + 1.6
84.1 2.0
82.0 + 2.0
83.3 + 1.4 89.7 + 2.0
CNN(S5)
80.3 + 4.8
80.5 + 3.8
83.8 + 4.6
82.6 + 4.6
88.5 = 3.4
82.3 + 4.0
where each component of the perturbation vector is drawn from (N0, 1). In all cases except CDT, the networks are trained using clean FMNIST training data, but their accuracy is evaluated on cor- rupted FMNIST testing data.
In Fig. 2, we demonstrate that models trained with NINR are more robust against the noise defined above compared those trained via other regularization methods. This verifies our expectation that in the decay phase, NINR improves stability, at least as well as CDT for large corruption, while (un- like CDT) it does not degrade generalization performance for small corruption or clean data. We also note that noise injection offers the best results for robustness within the catapult regime. However, arrive at the same accuracy on the clean test dataset, more training epochs are generally required. Lastly, we empirically observe that in in-NINR in the catapult phase offers the greatest improvement in stability against input corruption for both networks, as in-NINR most closely resembles input data corruption. We repeat this experiment for the CIFAR-10 dataset in App. E, where the similar trends persist, though at the cost of longer training time in the catapult phase.
3.2.2 DOMAIN SHIFT
Another test of the generalization properties induced by NINR can be realized by considering Do- main Shift problems. Here, we consider the generalization between two different datasets, rep- resenting different marginal distributions, by training models with NINR on the MNIST dataset, and testing their performance on data drawn from a new target domain distribution: the USPS test set (Hull, 1994). In order to match the input dimensions of the MNIST data, we follow the original rescaling and centering done in LeCun et al. (1998). The USPS images were size normalized fit a 20 x 20 pixel box while preserving their aspect ratio, and then centered in a 28 x 28 image field, followed by the standard preprocessing procedure.
The results are presented in Table 1. We observe generalization improvement for both FC and convolutional networks when using different regularization schemes, compared to unregularized networks. Of particular interest are the gains obtained when implementing both in- and full-NINR in the catapult phase, with the convolutional network. As the architecture becomes more complex, the improvements from NINR's 's adaptive scheme becomes more pronounced. The enbanced perfor- mance implies that NINR in the catapult phase could prove very beneficial for domain adaptation tasks. This is not entirely surprising as the USPS dataset is expected to lie close to the MNIST training set in distribution space, as there are no new correlated features such as several different digits in one image. Further experiments for domain shift adaptation on the MNIST-C dataset can be found in App. F. For other datasets, with large distributional shifts away from MNIST, and novel input correlations, we do not expect NINR to generically outperform other regularization methods.
3.2.3 GENERALIZATION TO TEST DATA
In this section report some effects of NINR generalization from training to test data. While we showed above that NINR can substantially improve network performance on corrupted data, is also important that it does not fundamentally impair the network's 's generalization properties.
Generically, introducing input corruption during training to increase robustness can be shown to have negative effect generalization clean data. This is unsurprising it appears that the network essentially memorizes the noise (Zhang et al., 2016), which is clearly not part of the true data distribution. As the learning process with NINR inherently leads to suppression of the noise
8

---

Published as a conference paper at ICLR 2023
Table 2: Generalization clean test data, evaluated the FMNIST dataset. Comparison is made between L2. Dropout, in-NINR, full-NINR and CDT. and we highlight regularization methods which worsen general- ization by italicizing. For CDT. two values (Unoise = 0.2, 0.4) for the amount of corrpption are considered. The fully-connected and CNN NINR noise parameters are those used in Fig. 2. Errors indicate 2ar confidence intervals over 10 distinct runs. The NINR implementations, except perhaps the in-NINR at the catapult phase, have comparable generalization performance with to the rest of the regularization schemes, aside from CDT, where performance diminished as the network learns the noisy distribution rather than the original one.
in-NINR in-NINR full-NINR full-NINR
CDT
CDT (0.4)
None
T
Dropout
(Decay)
(Catapult) (Decay) (Catapult)
(0.2)
FC(%) 87.7
2.6 89.9
88. 89.1
0.7 86.2 0.8 88.5 1.8 88.2 86.6
0.9 85.5 3.8
CNN(S) 91.0 + 1.0 0.7 91.7
+.1.010++1111110 91.0 + 1.2 89.0 0.6 0.8 90.0
0.. 84.6 + 2.6 84.1 6.4
during the late stages, its generalization capabilities are expected to be far less affected. We verify this by comparing the performance of network trained with NINR against networks trained with L2, Dropout, CDT, and against unregularized DNNs.
In Table 2 we show generalization performance on the FMNIST test set for the FC and CNN architectures using the full dataset, consisting of 60 training examples with a 60/40 train- ing/validation split. Our main observation is that optimizing with NINR in the decay phase, as with the commonly used L2 and dropout regularizers, leads performance domain test samples as least as good as the unregularized case. (In no case is the performance better statistically significant level.) We note that some degradation occurs when training with noise injection in the catapult phase, for a fixed number of training epochs. This degradation can be ame- liorated by training for longer period. Contrasting NINR with CDT, Table 2 clearly demonstrates that generalization is compromised for the latter, as the network cannot distinguish data from noise, learning the corrupted distribution. We further verify these results for CIFAR- 10 in App.
clean data with in-
4 RELATED WORK
Noise injection during training as a method of enhancing robustness has been proposed in various configurations in the literature. These include adding noises to input data (Hendrycks et al., 2019; Gao et al., 2020; Liu et al., 2021), activations, outputs, weights, gradients (Holmström & Koistinen, 1992; Reed & Marks, 1999; Neelakantan et al., 2015; You et al., 2019) and more. Most studics keep the amount of injected noise fixed, while allow the network reduce its effect during training.
Our study expands upon these works, consolidating empirical evidence with analytical insights. Our main contributions are twofold: We provide analytic expressions for the implicit regularization terms generated within our scheme, as well as estimating their effects during training. When applied to specific architectures, this allows us to predict when NINR is expected to be most effective. Additionally, we probe novel phase of learning, starting with large amount of noise injection and leading greater improvement in robustness against input corruption. Works by Rakin et al. (2018) and Xiao et al. (2021) follow similar reasoning, though both are limited, by construction, to small amount of noise injection, and are more empirically driven. Rakin et al. (2018) and Rusak al. (2020) also feature complex custom update steps which are less adaptable to other architectures.
CONCLUSIONS
In this paper, we motivated Noise Injection Node Regularization as a task-agnostic method to im- prove stability of models against perturbations to input data. Our method is simply implementable in any open source automatic differentiation system.
While we restricted this initial study to single Noise Injection Node added to various layers, with a fixed scale of noise injection during training, this restriction can be relaxed, leading to potential improvements to NINR. For instance, changing the amount of injected noise during training, similar to learning rate scheduling, could aid in convergence speed while still obtaining the advantages of a large amount of noise injection.

---

Published as a conference paper at ICLR 2023
6 ACKNOWLEDGEMENTS
We thank Yasaman Bahri, Kyle Cranmer, Guy Gur-Ari, and Sho Yaida for useful discussions and comments. NL would like thank the Milner Foundation for the award of Milner Fellowship. MF is supported by the DOE under grant DE-SC0010000 and the NSF under grant PHY1316222 MF would like to thank Tel Aviv University, the Aspen Center for Physics (supported by the U.S. National Science Foundation grant PHY-160761 1), and the Galileo Galilei Institute for their hospi- tality while this work was in progress. The work TV is supported by the Israel Science Foundation (grant No. 1862/21), by the Binational Science Foundation (grant No. 2020220) and by the Euro- pean Research Council (ERC) under the EU Horizon 2020 Programme (ERC-CoG-2015- Proposal 682676 LDMThExp).
REPRODUCIBILITY STATEMENT
In Sec. 2, we state our theoretical results, ensuring that state our assumptions and the limitations of the approximations we make at every step. In several instances, we rely on proofs given in other works, well as supplement our analyses in App. C; The models and tools used for analysis in our experiments are provided in the following anonymous link: https: // annnymous 4open. set ence/ r /No se I jnjecti onNodeCode 68, while explicit details regarding our experi- mental setup as well as complete description of the data processing steps for the datasets we used, are given in Sec. and App. A.
REFERENCES
Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical Gauss-Newton optimisation for deep learning, 2017. URL https://arxiv. http///rxii..rg/bb///a//aaasa 110.//101100000 03662.
Subrahmanyan Chandrasekhar. Hydrodynamic and hydromagnetic stability. Clarendon Press, Ox- ford, 1961.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273-297, 1995.
Philip Drazin and William H. Reid. Hydrodynamic stability. Cambridge University Press, Cam- bridge, 2nd edition, 2004.
Xiang Gao, Ripon K. Saha, Mukul R. Prasad, and Abhik Roychoudhury. Fuzz testing based data augmentation to improve robustness of deep neural networks. In 2020 IEEE/ACM 42nd Interna- tional Conference on Software Engineering (ICSE), pp. 1147-1158, 2020.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2014. URL https: /arxivv org abs// 1tclb///1111 4.1.665777
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty. arXiv e-prints, art. arXiv:1912.02.
December 2019.
Geoffrey Hinton, 2012.
URL http: : /////wwww /tt///wwww///////////////// cs. toronto.ooooooooo .ata:e/dddo////e tttp:tti..ieee/sseeeeeeeeeeeeeee
slides Atlidesleccture hetp//lcccteesseeeeee seree_ssies_____
pdf.
Geoffrey E. Hinton. Learning translation invariant recognition in massively parallel networks. In J. W. de Bakker, A. J. Nijman, and P. C. Treleaven (eds.), PARLE Parallel Architectures and Languages Europe, pp. 1-13, Berlin, Heidelberg, 1987. Springer Berlin Heidelberg. ISBN 978- 3-540-47144-8.
Judy Hoffman, Daniel A. Roberts, and Sho Yaida. Robust learning with Jacobian regularization, 2019.
back Lasse Holmström and Petri Koistinen. Using additive noise back- propagation training. JEEE Transactions Neural Networks, 3(1):24-38, 1992. doi: 10.1 109/72. 1054115.
10

---

Published as a conference paper at ICLR 2023
J. J. Hull. A database for handwritten text recognition research. IEEE Transactions Pattern Analysis and Machine Intelligence, 16(5):550-555 5544, 1994. doi: 10.1109/34.21144.
James H. Jeans and George H. Darwin. I. The stability of a spherical nebula. Philosophical Transactions of the Royal Society of London. Series Containing Papers of Mathematical
or Physical Charucter, 199612-320)11-53, 1902. doi: 10.1098/rsta.1100
URL https:
//
otg/doi/abs 10. 10901098///0 rssaa. 1902 0012.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL
org/ www.o/ggg////o/ooooeeoo//eeees/o 1192/1111..0 6980.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 (canadian institute for advanced research). online: http://www.c..ooontooooooo wwwontooeddukkriaccffaahnn 55(5), 2014.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. In J. Moody, S. Hanson, and R.P. Lippmann (eds.), Advances in Neural Information Processing Systems, vol- ume 4. Morgan-Kaufmann 1991. URL. ht https::/pproceediini 1991/ http///ffecffee9eeeee1111ee11e10
neur cc/ paper/ Paper. .df.
Jan Kukačka, Vladimir Golkov, and Daniel Cremers. Regularization for deep learning: A taxonomy, 2018. URL htt.tttps:/ooo http::////peeevvieeeee net. formm wtt.ooormmmmmm.kkdeeeemoe d=SkHke xAW.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world, 2016. URL htt.tp:://aaxxv.orrrg... /aarrg/abb//61// /01.//6600.00 02533.
Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11): 22244, 1998. doi: 10.1509/5. 726791.
Noam Levi, Itay Bloch, Marat Freytsis, and Tomer Volansky. Noise injection as probe of deep learning dynamics, 2022. URL https: http//aawx....aaaaaa/a/aaaaaaaaa
13599.
Aitor Lewkowyez, Yasaman Bahri, Ethan Dyer, Jaseha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.022/8, 2020.
Aishan Liu, Xianglong Liu, Hang Yu, Chongzhi Zhang, Qiang Liu, and Dacheng Tao. Training robust deep neural networks via adversarial noise propagation. IEEE Transactions Image Processing, 30:5769-5781, 2021. doi: 10. 109/TTP.2021 33022117..
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2017. URL https://arxti: org/abs/ 6 06083.
James Martens and Roger Grosse. Optimizing neural networks with kronecker- factorrd approximate curvature, 2015. URL https::t///.o.....t. /ttp/aaaxvvvv//////e//// org. abs 07671.
Norman Mu and Justin Gilmer. MNIST-C: A robustness benchmark for computer vision. CoRR, abs/1906.02333 2019. URL ht http//rxivvi//va////v///.../..as http://awwwvv...aaaaaaeaa wtt./oo/////aaoo/oo 02337.
Arvind Neelakantan, Luke Vilnis, Quoc V. Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks. arXiv e-prints, art. arXiv:1511.0680% November 2015.
Eugene N. Parker. Dynamical instability in an anisotropic ionized gas of low density. Phys. Rev, 18741874 Mar 1958. doi: 10.1103/PhysRev.100. org/doi/10. 11t.:/1113//Pph/
URL https: :ttes://linn aps.
109.1874.
Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack, 2018. URL https:/ // arxiv. hww.v.rrg/aao abs/ /////111..11 09310.
Russell Reed and Robert J. Marks, II. Neural smithing: supervised learning in feedoororrrr artificial neural networks. MIT Press, 1999.

---

Published as a conference paper at ICLR 2023
Evgenia Rusak, Lukas Schott, Roland S. Zimmermann, Julian Bitteroolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. Increasing the robustness of dnns against image cor- ruptions by playing the game of noise. CoRR, abs/2001.06057, 2020, URL https://arxii: org/abs/ 2001 06057.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv e-prints, art. arXiv:1409. September 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Reseurch, 15(56):192--1958 2014. URL
arivastavalla.hct
Steven H. Strogatz. Nonlinear dynamics and chaos: with applications physics, biology, chemistry, and engineering. CRC press, 2018.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, D. Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6199, 2014.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for bench- marking machine learning algorithms, 2017.
Li Xiao, Zeliang Zhang, and Yijie Peng. Noise optimization for artificial neural networks. arXiv e-prints, art. arXiv:2 102 044400 February 2021.
Zhonghui You, Jinmian Ye, Kunming Li, Zenglin Xu, and Ping Wang. Adversarial noise layer: Regularize neural network by adding noise. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 909-913, 2019. doi: 10. 109//CIP22019.88830000
Chiyuan Zhang, Samy Bengio. Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. CoRR, abs/1611.00552 2016. URL http:
//arxiv. abs//bb/1111 /11...11.... 03530.
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger B. Grosse. Three mechanisms of weight decay regularization. CoRR, abs/1810..2281 2018. URL htttt:::////xii.oor///aa/ea 12281.
Alice Zheng, Michael Jordan, Ben Liblit, and Alex Aiken. Statistical debugging of sampled pro- grams. In S. Thrun, L. Saul, and B. Schölkopf (eds.), Advances in Neural Information Process- ing Systems, volume 16. MIT Press, 2003, URL https: pttp//eeccceeeeesceeeeepppc neurips.com ca/ paper/2003/ 0tteer/2200/11eea555511393 /tt.///660..11111111666666666660 7660 Pap00-Paeepppppp pdf.
Liu Ziyin, Kangqiao Liu, Takashi Mori, and Masahito Ueda. Strength of minibatch noise in SGD. In International Conference on Learning Representations, 2022. URL /tL.tttsss//ppeereevie
pet/fot net/forum? itt.oormmiiduoooobwvvvv....
12

---

Published as a conference paper at ICLR 2023
A NETWORK ARCHITECTURE DETAILS
Here we describe the experimental settings specific to each of the figures in the paper. All the models have been trained with cross-entropy loss unless otherwise specified.
Fig. 2(a). Fully connected, three hidden layers with width d1,2,3 1024, weight initialization W(0 ~ (0, 1/d4), b O. ReL.U activation, trained using SGD (no momentum) on FMNIST, with a learning rate of 7) = 0.05 and batch size [B] = 128.
Fig. 2(b). CNN composed of 2 convolutional blocks, followed by dense ReLU layer with width
d3 = 2048 and a dense connection to the prediction layer, weight initialization W(() _NO,, 1/de), b 0. Each convolutional block is taken as Conv2D(2,2) - ELU - Batch-Norm -
(ax,o trained using SGD (no momentum) on FMNIST, with a learning rate of = 0.05
and [B] = 128.
Fig. Fully connected, one hidden layer with d, 1024, weight initialization W(() N/00 1/d,),
b = 0. ReI.U activation, trained using SGD (no momentum) on FMNIST. with learning rate 1)
0.01 and [3| 1000. From left to right, the injected noise a = {0, .in din/ /7, 1.8 d n) corresponding to the decoupled, decay, catapult, and divergent phases, respectively. Here, din 785 is the dimension of the input data including a single NIN.
B ADDITIONAL THEORETICAL DETAILS
B. DERIVATION OF THE NOISE TRANSLATED LOSS FUNCTION
Here we provide additional details on the theoretical analysis of the noise translated loss function, leading to Eq. (1). The introduction of NIN at a specific layer ENI translates the preactivation as ((6x1) WNIC. A single sample loss function describing the translated preactivation can be written as
C WN; w, y) = L(0; 2(x0)((x)) + WNIE, y).
(8)
Using the definition of the translation operator
f(x + a) et f(z),
(9)
explicitly compute the batch averaged loss function
L(O, WNI)
Σ
L(O WNI:
eW v (Nd) C(O: x x.
[
B
(E.
(B.C.
26 (cWT. NI
L(0) +
Σ
((())) ( L(0;2,,,,)
B
k!
{w.c,y)
EB
Expanding in powers of EWWI, we obtain an infinite series given by
L(e, WNI) L(0)
ΣE
Σ
E
(cW3i V=
(10)
[
[a.c.v]EBk=]
where we identify the k > terms in the expansion with the implicit regularization terms defined in Eq. (3).
B.2 NIN AT ALL LAYERS
Here we extend our theoretical derivations from the case of a NIN connected to a single layer, to a single NIN connected all layers (the full-NINR case).
Similar to Eq. (1), we may write down the loss using the translation operator,
(N--1
L(0, WNT)
1
Σ
II
(W-()) C
L( x, x,
(1I)
B
[m.
(N,-11 of respective dimensions Rlxd, ,..., BRlxdN Focusing once the leading terms in e, that the first order
where may expect, have NL vectors W0(0)
..... NI
13

---

Published as a conference paper at ICLR 2023
regularization term is simply
R1
No-1
Σ
(W)))
T
((gge)
(12)
Ni
t=
which is the sum of the regularization terms at each layer. The second order regularization is slightly more complex, and may be written
N
ML-1NN-1
(W (HHone)
R2
(13)
2 2=
with Heyt = ((1 ((22) L(,, 20 y)-
According to Botev al. (2017); Martens & Grosse (2015), the terms which mix different layers in the Hessian are expected to be small, thus leading us to a sum over the single-layer R28 of the each f. Therefore, may use the arguments used in the main text to estimate the scaling of the two terms with o, and [B]. Much like the single-layer case, therefore expect Ri & ae/ /B] and R2 o.
Demonstrating the emergence of the phases discussed in the main text (i.e. the decay phase, the catapult phase, etc.) a deep linear network for this full-NINR case is beyond the scope of this work. However, we do note that empirically they are found to be present much like in the single- layer NIN case.
C NINR IN A LINEAR ToY MODEL
Noise Node
E
WNI
w (1)
(0) + WNIE
ÿ
x0
(O))
Input
Hidden layer
Output
Figure 5: Illustration of the univariate linear DNN with single input scalar, noise node, and one hidden layer.
In order elucidate the interpretation of noise injection nodes emergent regularization scheme, combined with a form of constrained random walk, we employ an (over)simplified univariate linear model which captures the main features present in realistic networks. Consider a linear network (i.e., linear activation functions), with single hidden layer and no biases (b = 0). aiming to perform linear regression task. The data consists of a set of training samples {( Ba) E R²]_ Ja=1' taken by drawing the inputs from normal distribution of I. E X (' oz), where a. > 0. The corresponding outputs are then given by a linear transformation yo = M Ta with M e R.
The noise node is added the input, and its weight WNI, with the input's data weight being w() The hidden layer is directly connected the output, and has single weight associated with it, w(1) as illustrated in Fig. 5. We use Mean Squared Error (MSE) loss, and for simplicity take a full-batch gradient descent, and thus our loss function is
LMSE
Σ
(w WNICC) Va)
(14)
2|
aEB
Performing an explicit averaging, we can further simplify
LASE ~ 2
2w (2) WNi (w - M) az
+ (w (1) MY203 + (w01))222203
B
(15)
14

---

Published as a conference paper at ICLR 2023
where & is a random variable with zero mean and unit variance From Eq. (15), we can easily see
that the optimal solution achieved for the weights w:!h) (/ M,, WNI,- = 0. We can clearly read R1, off Eq. (4) by separating them from the unperturbed loss function
LMSE (WNi, 0) = LASF: () + R1 ( 0) + R2( 0), while Rs vanish for k > 2. The various terms in Eq. (16) are given by
(16)
LMSE () = 2
(w - M)
R1 (WNI, 0) = WNI wNI = (iwl (w0 - MM) 0r
(17)
/BB
1
R2(WNI, 0) = oo? HENI = 2
2
where we have identified the local gradient term which generates constrained random walk for WNI, which decreases as the network approaches its data driven minimum. We also note that in the limit of an infinite batch size [B] - it vanishes, leaving only an effective regularization term for the hidden layer weight, namely L2 {(w(1))² where the Lagrange multiplier À = decreases with time as the noise weight WNI is pushed to O.
We may glean further insights from this linear example by studying its training dynamics for small and large noise variances. Assuming full batch gradient descent in the infinite sample limit, we neglect the local gradient contribution and focus on the coupled equations for the hidden layer weight and the noise weight, given respectively by
(1 e+1 = ( (1 -o,,,o,,, - n(w?
uw w/ M) 02
(18)
WNL,++1 = WNII.(1 - 7000-700.10 ? (2)
Assuming M * 0, without the loss of generality, we may set M = 1 as the equations remain
invariant under reparameterizationn. In the limit of of < 1/murn,o the equations decouple, with
(1) (m w
(1) (0)
the data weight following the standard GD equation without noise, i.e., W241 w.
1 1)w! (0) og. while the noise weight decays exponentially as long as 0 < Imc (2) and (mo 2 > no? /2.
Clearly, the smaller a, is, the smaller the regularizing effect of the noise on the local Hessian, given by the square of the hidden layer weights in this simple model.
We expect that in this regime, the limit of continuous time GD should reproduce the correct dynamics as ?o? 0. yielding differential equation for the noise weight
(t) = -² (t)) wx)()),, (t),
(19)
where & = dr/ dt is the continuous time derivative. The noise weight can therefore only decay.
Conversely, taking the large noise variance limit we find that the dynamics are ignorant of the origi- nal learning objective, as the resulting equations become simply coupled
w = ( (1 - n
(20)
WNL++1 = WNI, (1 (1- (w (2)))
These equations describe a NN, trained using completely random data with no labels or learning objective, with an effective loss given by the last term in Eq. (15). In this case, we expect the continuous time limit to fail as complete description of the possible dynamics, as no? may be
large. We may demonstrate this failure by taking the continuous time limit, obtaining
(t) = -o? (t)))x((tt)
wt)(t),
(t))() -o2 (t) (t))
(21)
implying both weights decrease in magnitude. This means the network, even for arbitrarily large o. will not diverge. However, this is clearly not the case for the discrete Eq. (20), which may become stiff for sufficiently large noise variance. This numerical artifact entirely changes the weight behavior, opening up the possibility for the system to either diverge, or catapult, as discussed in Levi et al. (2022). To summarize, this simple example provides useful test case for our main analytical derivations appearing in the main text, displaying all the expected features of NINR in a fully calculable setting.
AAditiiona O / BB)) corrections coming from stochastic variations in the of term emerge from batch- averaging but are neglected.
Taking (1) Mw UNI + Muxi and a. + Ge/M leaves the equations invariant.
15

---

Published as a conference paper at ICLR 2023
D ADDITIONAL EXPERIMENTS
Throughout this section, train the FC and the CNN using the same specifications given in Fig. 2, unless otherwise specified. Training is performed for the minimum between 500 epochs, and the time it takes the network to reach 98% training accuracy. This done with the goal of demonstrating that NINR using a large amount of noise injection requires a longer period of train- ing, otherwise suffering from degraded generalization performance, as discussed in the main text.
D.1 ADVERSARIAL ATTACKS
In addition imput perturbations caused by deployment issues, natural degradation, and unexpected noise sources, targeted perturbations, meant to maximally impair the performance of a network while changing the data as little as possible, form conceptually different concern. Quantifying what corresponds to minimal distortion of the data is a domain-speciftt and somewhat subjective task. Nevertheless, standard approaches exist. One of the simplest known implementations for an adversarial attack is the white-box untargeted Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014), which transforms inputs according to
I + x + ÓFGSM X ai x; y)),
(22)
where ŌFGSM is a small positive parameter that controls the size of the perturbation. We also con- sider the Projected Gradient Descent (PGD) attack (Kurakin al., 2016; Madry et al., 2017), which iterates the FGSM attack k times, compounding its effect.
(b) CNN FCSM
CNN
Fxl--80t--Daay
Text
Toe
Digget
12
Srate
Figure 6: Robustness against adversarial attacks. Left: FC network, Right: CNN (detailed specifications are in App. A). This key result illustrates that NINR significantly increases robustness generic model architectures trained on the FMNIST dataset. Shades indicate 2 standard deviations estimated over 10 distinct runs.
In Fig. we compare the performance of standard regularization schemes with NINR against FGSM and PGD type adversarial attacks. We find that NINR displays superior performance over Le and un-regularized nets. For FGSM attacks dropout performs best among the options tested, while for PGD attacks NINR outperforms.
These preliminary results suggest potential improvement against certain types of adversarial attacks when NINR is used. Further analysis is required to determine whether combining NINR with other regularization schemes, or changing the noise distribution during training could potentially produce a more successful scheme.
D.2 DIFFERENT NOISE DISTRIBUTIONS
Here, we examine the effects of sampling the NINs from different noise distributions on the perfor- mance of NINR. For each different noise distribution, we repeat the tests used to produce Fig. 2, demonstrating robustness against corrupted inputs. We compare results using uniform distribution e ~ U(-oe, o.) and an asymmetric (double Gaussian peaked at +o.)) distribution, for fcNINR and cNINR using DNNs trained the FMNIST dataset.
In Fig. 7, we see that varying the noise distribution has a minimal effect on NINR as a regularization scheme, aside from the asymmetric distribution for the catapult phase. We attribute this behavior to an extreme choice of noise injection scale, where a much longer training time is required to obtain good performance for NINR.
16

---

Published as a conference paper at ICLR 2023
(0) FC- Agymmtri: Netm Detributtoon
(0) EC- Uniten Nets Disribation
No Regabroutie
Ix In-NNE-Crapdd
Ndi-AM--Usty
PL--NE Cilppa
Ø
Asymmmettic Natao
Dunnllationn
CDT-64
CDT-66
03
02
0.2
0.5 06
0.6
1.3 3. 3.3 1.0
Figure 7: Robustness against random input perturbations for FC (top row) and convolutional (bottom row) networks using NINR training with different noise distributions (detailed specifications are in App. A). Left: Asymmetric double gaussian distribution peaked at to. Right: Uniform distribution U(- oz) The
fully-connected and CNN NINR noise magnitudes those Fig. 2. Shades indicate standard deviations estimated over distinct runs.
D.3 DIFFERENT OPTIMIZERS
Here, examine the effects of changing the optimization algorithm, beyond SGD, on the perfor- mance of NINR. For each different optimizer, we repeat the tests used to produce Fig. 2, demon- strating robustness against corrupted inputs. We compare results using RMSprop (Hinton, 2012) and Adam (Kingma & Ba, 2014), for fcNINR and cNINR using DNNs trained on the FMNIST dataset. Here,
different parameters for the different architectures and optimizers. Namely, 0.001 for CNN. Adam 31 0.9, B2 0.999, e 10-7 and 1j 0.01 for both FC and CNN, with noise injection magnitudes given in App. D.3.
RMSprop 0.9, = 10-7 and y = 0.0001 for FC and
(n) FC Adam Optininor
(b) FC RMBooup Optiniser
00
87
8.4
1.6
100
00
0.2
0.4
0.5
1.0
(c)
CNN Adus Optinissr
(c) CNX RM5pmg Optinlen
The Averncy
00
24 3.0
Figure 8: Robustness against random imput perturbations for FCs (top row) and CNNs (bottom row) in App. A). Left: Adam,
ing NINR training under different optimization schemes (detailed specifications trained with y 0.01 for both FC and CNN. Right: RMSprop, trained with 0.0001 for FC and 7 0.001 for CNN. The fully-com conneccon and CNN NINR noise magnitudes are those of App. D.3. Shades indicate stan- dard deviations estimated over 5 distinct runs.
17

---

Published as a conference paper at ICLR 2023
Table Amount of noise injection (o. for the different architectures using RMSprop and Adam
in-NINR decay (catapult) foll-NINR decaa (catapult)
RMSprop
FC
1158.2 (6179.9)
(1158.2)
(619.1)
6.19 (437.8)
in-NINR decay (catapult)
decay (catapult)
Adam
PC
116.8 (518)
36.6 (115.8)
CNN
6.2 (196.8
1.96 (138.4)
E RESULTS FOR CIFAR-10
Here, we implement cNINR, working with CNN based on VGG style blocks described in Si- monyan & Zisserman (2014). The CIFAR-10 dataset consists of color images of objects divided into 10 categories, with x 32 pixels in 3 color channels, each pixel intensity the range [0, 1]. partitioned into 50 000 training and 10 000 test samples, which are then preprocessed similarly to the FMNIST dataset.
The network used to test NINR performance is constructed by connecting the following blocks
C3n,2D(32,333) + ReLU + Batch Norm + Conv2D(32,3,3) + ReLU 7
Batch Norm + MaxPool(2.2) + Propout(parop = 0.2).
(64333(64,333) + ReLU + Batch Norm + Conv2D(64.3,3)
ReLU
Batch Norm MaxPool(2,2) + Dropout(parap = 0.3).
+Conv2D(128.3.3) + ReLU - Batch Norm Conv2D(128,3,3) Batch Norm + Max,)ol((222) + propout(pprop = 0.4).
ReLU
Dense ReLU Layer(500) > Linear Layer(10).
Optimization done using SGD without momentum with the learning rate fixed to n 0.05 and mini-batch size B 128. Each training is performed for 500 SGD training epochs in total, or until 98 % training accuracy has been achieved.
We provide preliminary results for robustness against input-data corruption in Fig. 9. In contrast to the previous sections, the CNN used to train on CIFAR-10 10 contains Dropout and L2 as part of its show results
architecture, making comparison between NINR and the two redundant. Therefore, for the same network with and without NINR, as well as CDT with different input corruption scales. The success of NINR is retained for in-NINR in the decay phase, while the catapult phase requires longer than 500 epochs to obtain similar generalization properties.
CIF ARIO
100
80
Test. Accuracy
60
Badie
Is NNE. Docr
40
NNR- Calech
CDT-42
CDT-64
20
CDT-06
0.0
0.2
0.4
0.6
0.8
1.0
noise
Figure 9: Robustness against random input perturbations for the network described in App. E using NINR training CIFAR 10. Here, we the in-NINR CNN implementation, taking o. 17.5 in the decay phase and a. 55.4 the catapult phase.
E'aach convolutional layer admits L, weight decay regularization (AwD 10-4).
18

---

Published as a conference paper at ICLR 2023
F RESULTS FOR MNIST-C
Here, we show some additional results for the same architectures and NINR parameters used in Sec. 3, trained on the MNIST data-set and tested on several classes of images from MNIST- C. The MNIST-C dataset (Mu & Gilmer, 2019) consists of 15 types of corruption applied to the MNIST test set, for benchmarking out-of-distribution robustness in computer vision. For testing purposes, the data is preprocessed similarly to the FMNIST dataset.
Table 4: Domain shift performance the MNIST-C test data, for networks trained the MNIST dataset. Comparison made between T22 Dropout, in-NINR, full-NINR and CDT. For CDT, values (Fnoise 0.2, 0.4) the amount of corruption are considered. The fully-connected and CNN NINR noise parameters are those used in Fig. 2. The NINR implementations improve performance for data transformations which are most closely related to gaussian noise injection, can be expected.
Fog transformation
in-NINR in-NINR full-NINR full-NINR CDT CDT
None L2 Dropout (Decay) (Catapult) (Decay)
(Catapult) (0.2) (0.4)
FC(%)
53.2
52.3
61.9
59.4
63.8
57.2
62.8
52.9 57.3
CNN(%) 71.8
67.3
61.8
62.5
62.9
57.8
57.8 62.7
Brightness transformation
in-NINR in-NINR full-NINR full-NINR CDT CDT
None L2 Dropout (Decay) (Catapult) (Decay)
(Catapult) (0.2) (0.4)
FC(%) 97.7
97.6
98.4
98.0
97.4
97.9
98.1
97.7 97.2
CNN(%) 98.9
98.7
98.7
98.8
98.8
98.6
98.1 97.5
Glass Blur transformation
in-NINR in-NINR full-NINR full-NINR CDT CDT
None L2 Dropout (Decay) (Catapult) (Decay) (Catapult) (0.2) (0.4)
FC(%) 93.7
93.2
96.0
94.7
93.4
94.4
94.3
94.6 94.3
CNN(%) 65.1
54.6
60.2
95.2
61.3
95.6
80.6 90.2
Impulse Noise transformation
in in-NINR in-NINR full-NINR full-NINR CDT CDT
None L2 Dropout (Decay) (Catapult) (Decay) (Catapult) (0.2) (0.4)
FC(%) CNN(%) 51.1 28.8
84.3 84.3
94.6
93.6
94.1
89.7
96.1
86.7 89.7
62.8
97.8
57.1
96.5
75.3 86.8
The results shown in Table indicate improved performance when the type of image corruption ap- plied to the MNIST images most closely resembles the injected noise. It can therefore be intuitively understood why the most dramatic performance enhancement is found for the Impulse Noise corrup- tion transformation, while other corruption transformation may not benefit much from NINR. We stress that NINR can be readily modified deal with different types of corruption by changing the
19

---

Published as a conference paper at ICLR 2023
noise injection distribution, well as incorporated with other regularization methods to compound their robustness enhancing effects.
G CONSTANT NOISE INJECTION
Here, we reproduce the results shown in Fig. 2. including an additional curve representing a con- stant imput noise injection. We implement this experiment by applying dNINR at the input layer (connecting to the first hidden layer) using the large NIN variance value used for the the "catapult" phase, but keeping the NIWs statie, fixed to their initialization values.
Fully Connected DNN
100
80
Test Accuracy
Baseline
-
60
M--NNR-Catpuul
Bull-NNR-BE Decay
40
12
Drepout
CDT-004
20
Constont Input Noise
0.0
0.2
0.4
0.6
0.8
1.0
0 noise
Figure 10: Rohustness against random input perturbations with the parameters used in Fig. 2. The ad- ditional orange curve represents In-NINR with a, 231.6 but with fixed NIW values, which essentially constant noise injection to the pre-activation at the first hidden layer.
20


Reviews:
[{"review_id": "_HYctRdhMj", "reviewer": "Reviewer_q36r", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on robust representation learning and attempts to propose a method that is robust to input perturbations. To achieve this goal, the authors propose Noise Injection Node Regularization (NINR) which severs as a “regularizer”. Experimental results demonstrate in some cases, this proposed method outperforms models trained with Dropout or L2 regularization.", "summary_of_the_review": "Based on what I have mentioned in the [Strength And Weaknesses] part, I have concerns about the novelty of this paper and also its experimental evaluation does not sound comprehensive.\n\n", "strength_and_weaknesses": "Strength: \nThe paper covers an interesting topic as robust representation learning is a challenging but practical problem, especially for robust image classification. The overall presentation sounds good.\n\nWeaknesses:\n\n1) I doubt the novelty of this paper. Injecting noises to the input or to the learned representation is not a new thing. There are already several papers covering similar topics, such as Ref1 and Ref2.\n\n2) All experiments are carried out on very simple and easy datasets such as MNIST and FMNIST. Thus, it is not quite sure how reliable these results are and whether it is a safe way to draw conclusions from them.\n\n3) Also for the comparisons, the authors only compare models with Dropout and L2 regularization. However, as the paper claims to learn a robust representation, the authors should consider comparing the proposed method with other robust models and other more complicated datasets such as MNIST-C and ImageNet-C for example.\n\n4) Even on the simple dataset, the proposed method cannot consistently outperform models trained with Dropout. Thus, I do worry about the benefits of this proposed method can bring us.\n\n5) There are some typos in the writing.\n\n\nRef1:  Liu, Aishan, et al. \"Training robust deep neural networks via adversarial noise propagation.\" IEEE Transactions on Image Processing 30 (2021): 5769-5781.\n\nRef2: Rusak, Evgenia, et al. \"A simple way to make neural networks robust against diverse image corruptions.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VzaewBg-trt", "reviewer": "Reviewer_KZHy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This works studies the noise injection node approach and how it improves the noise-resistance to various types of data perturbations. ", "summary_of_the_review": "Due to the issue of self-containess, I cannot give the acceptance to this paper. ", "strength_and_weaknesses": "The major problem of this manuscript is its writing, which cites a lot of analytical results from Anonymous (2022). Unfortunately, I cannot find this reference from the web at all. This makes the current version of  this paper lack of self-containess. Thus, frankly, I cannot judge the novelty of this work. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PbcfuelDoK5", "reviewer": "Reviewer_27TF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper looks at a new method for adding gaussian noise to nodes during training such that the amount of noise added is a learned parameter. The paper, the does some theoretical analysis, to obtain some heuristics about the effect of the noise. This followed up by empirical work looking at the evolution of the norms of noise scaling parameter during training for different noise variances. Finally, the paper tests the robust of such networks to input data corruption. ", "summary_of_the_review": "In summary, I like the idea, the analysis and the experiment, but I am concerned about the novelty and the presentation. ", "strength_and_weaknesses": "**Strengths**\n---\n\n1) I like the study on the norm of the scaling during training versus the noise variance this provides nice insights into how to set the noise. The theoretical underpinning for this comes from the Taylor expansion of the loss function with respect to the scaling parameters. The paper looks at the terms with the first derivative and the second derivative and in some simple case (such as 2 layer linear networks) derive heuristics for estimating these terms. Using the heuristics, the paper divides the variance into various regimes depending on whether the First derivative term or the second derivative dominates. \n\n2) The proposed method seems to drastically increase robustness of Convolutional neural networks. \n\n3) The idea of having adaptive noise is interesting. I will list this as a strength (because I don't fully know the literature) but two papers cited by the paper (Rakin et al. (2018) and Xiao et al. (2021)) might have already proposed something similar. \n\n**Weakness**\n---\n\nI think the major weakness of the work is putting itself in context of the existing literature. Specifically, the works on adding noise to the input data being equivalent to regularization is known (Bishop 1995). Further, based on a google search there is a lot of work analyzing constant noise injection.  For example, Camuto, Willetts, Simsekli, Roberts, Holmes 2020; Poole, Dickstein, Ganguli 2014; and  Cohen, Resnfeld, Kolter 2019. While the authors do not need to cite these particular work, but more discussion on work related to constant level noise injection would be appreciated. \n\nFurther, in the experimental evaluation the authors do not compare against constant level noise injection. Since the main novelty of the paper seems to be the adaptability of the noise level, I feel that the constant noise level is a crucial baseline. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0Ahik5EJU7", "reviewer": "Reviewer_J5gg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a regularization method for neural networks, namely Noise Injection node Regularization (NINR). The high-level idea is to inject random noise into the network’s training at a certain layer via a learnable weight. The authors provide analyses both theoretically and empirically to show how NINR could improve the robustness.\n\n", "summary_of_the_review": "I think this work has some merits, but I also have some concerns as given above.", "strength_and_weaknesses": "Neural network’s robustness is one of the most important topics in deep learning, so searching for a new regularization method that makes neural networks more robust to various kinds of scenarios such as distribution shift, adversarial attacks is definitely worthwhile. The paper presents an intuitively simple yet interesting idea motivated by mathematical and empirical insights. The experimental results are also extensive. \n\nRegarding the weaknesses, I have some comments:\n\n1. The novelty of this work compared to Anonymous, (2022), which I do not have the full context into.\n2. It is unclear from the experimental results whether this regularization approach is more useful than other well adapted ones. For example, in Table 1, Dropout is much better than NINR-based regularization for FC. In Table 2, L2 outperforms the rest for both FC and CNN.\n3. I may be mistaken, but it is unclear which layer of the neural network one should apply the NIN regularization. If applied on multiple layers, would the same analysis in Section 2.1 follow and how?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster