ON INJECTING NOISE DURING INFERENCE
Authors: Milad Khademi Nori
Yiqun GE
IL MIN KIM
Venue: Submitted to ICLR 2023 (2023)
Keywords: activation noise
energy-based modeling

Abstract: We study activation noise in a generative energy-based modeling setting during training for the purpose of regularization. We prove that activation noise is a general form of dropout. Then, we analyze the role of activation noise at inference time and demonstrate it to be utilizing sampling. Thanks to the activation noise we observe about 200% improvement in performance (classification accuracy). Later, we not only discover, but also prove that the best performance is achieved when the activation noise follows the same distribution both during training and inference. To explicate this phenomenon, we provide theoretical results that illuminate the roles of activation noise during training, inference, and their mutual influence on the performance. To further confirm our theoretical results, we conduct experiments for five datasets and seven distributions of activation noise.

Under review as a conference paper at ICLR 2023
ACTIVATION NOISE FOR EBM CLASSIFICATION
Anonymous authors
Paper under double-blind review
ABSTRACT
We study activation noise in a generative energy-based modeling setting during training for the purpose of regularization. We prove that activation noise is gen- eral form of dropout. Then, we analyze the role of activation noise at inference time and demonstrate it to be utilizing sampling. Thanks to the activation noise we observe about 200% improvement in performance (classification accuracy). Later, we not only discover, but also prove that the best performance is achieved when the activation noise follows the same distribution both during training and infer- ence. To explicate this phenomenon, we provide theoretical results that illuminate the roles of activation noise during training, inference, and their mutual influence on the performance. To further confirm our theoretical results, we conduct exper- iments for five datasets and seven distributions of activation noise.
INTRODUCTION
Whether is for performing regularization (Moradi et al. 2020) to mitigate overfitting (Kukackal et al. 2017) for ameliorating the saturation behavior of the activation functions, thereby aiding the optimization procedure (uucceree et al. (206)). injecting noise to the activation function of neural networks has been shown effective (Xu et al. (2012). Such activation noise denoted by z, is added to the output of each neuron of the network Tian & Zhang 2022) as follows:
t=s+z= f(
w + b) + az
(1)
where Wss F4. s, f()), a, 2, and stand for the ith element of weights, the ith element of input signal, the bias, the raw/un-noisy activation signal, the activation function, the noise scalar, the normalized noise (divorced from its scalar a) originating from any distribution, and the noisy output, respectively. Studying this setting inflicted by noisy units is of significance, because it resembles how neurons of the brain in the presence of noise learn and perform inference (Wu et al. 2001).
In the literature, training with input/activation noise has been shown to be equivalent to loss regu- larization: a well-studied regularization scheme in which an extra penalty term is appended to the loss function. Also, injecting noise has been shown to keep the weights of the neural network small, which is reminiscent of other practices of regularization that directly limit the range of weights (Bishop) 1995). Furthermore, injecting noise to input samples (or activation functions) is an in- stance of data augmentation (Goodfellow et al.) 2016). Injecting noise practically expands the size of the training dataset, because each time training samples are exposed to the model, random noise is added to the input/latent variables rendering them different every time they are fed to the model. Noisy samples therefore can be deemed as new samples which are drawn from the domain in the vicinity of the known samples: they make the structure of the input space smooth, thereby mitigating the curse of dimensionality and its consequent patchiness/sparsity of the datasets. This smoothing makes it easier for the neural network to learn the mapping function (Vincent et al. 2010))
In the existing works, however, the impacts of activation noise have been neither fully understood during training time nor broached at the inference time, not to mention the lack of study on the relationship between having activation noise at training and inference, especially for the generative energy-based modeling (EBM). In this paper, we study those issues: for the EBM setting, for the first time, we study the empirical and theoretical aspects of activation noise not only during training time but also at inference and discuss how these two roles relate to each other. We prove that, during training, activation noise (Gulcehre et al. 2016) is a general form of dropout (Srivastava et al. 2014). This is interesting because dropout has been widely adopted as regularization scheme. We then

---

Under review as a conference paper at ICLR 2023
formulate and discuss the relationship between activation noise and two other key regularization schemes: loss regularization and data augmentation.
We also prove that, during inference, adopting activation noise can be interpreted as sampling the neural network. Accordingly, with activation noise during inference, we estimate the energy of the EBM. Surprisingly, discover that there is very strong interrelation between the distribution of activation noise during training and inference: the performance is optimized when those two follow the same distributions. We also prove how find the distribution of the noise during inference to minimize the inference error, thereby maximizing the performance as high as 200%. Overall, our main contributions in this paper are as follows:
We prove that, during training, activation noise is general form of dropout. Afterward, we establish the connections between activation noise and loss regularizationn/aata augmen- tation. With activation noise during inference as well as training, we observe about 200% improvement in performance (classification accuracy), which is unprecedented. Also, we discover/prove that the performance is maximized when the noise in activation functions follow the same distribution during both training and inference.
To explain this phenomenon, we provide theoretical results that illuminate the two strik- ingly distinct roles of activation noise during training and inference. We later discuss their mutual influence on the performance. To examine our theoretical results, we provide ex- tensive experiments for five datasets, many noise distributions, various noise values for the noise scalar a, and different number of samples.
2 RELATED WORKS
Our study touches upon multiple domains: (i) neuroscience, (ii) regularization in machine learning, (iii) generative energy-based modeling, and (iv) anomaly detection and one-class classification.
(i) Studying the impact of noise in artificial neural networks (ANNs) can aid neuroscience to under- stand the brain's 's operation (Lindsay 2020 Richards et al. 2019). From neuroscience, we know that neurons of the brain (as formulated by Eq. never produce the same output twice even when the same stimuli are presented because of their internal noisy biological processes (Ruda et al.) 2020 Wu et al., 2001;; Romo et al., 2003).. Having noisy population of neurons if anything seems like a vistavantage (Averbeck et al.] 2006; Abbott & Dayan 1999): then, how does the brain thwart the inevitable and omnipresent noise (Dan et 1999)? We provide new results on top of current evidence that noise can indeed enhance both the training (via regularization) and inference (by error minimization) Zylberberg et al. 2016: Zohary et al. 1994).
(ii) Injecting noise to neural networks is known to be a regularization scheme: regularization is broadly defined as any modification made to a learning algorithm that is intended to mitigate over- fitting: reducing the generalization error but not its training error (Kukacka et al., 2017). Regular- ization schemes often seek to reduce overfitting (reduce generalization error) by keeping weights of neural networks small (Xu et al.) 2012). Hence, the simplest and most common regularization is to append penalty the loss function which increases in proportion to the size of the weights of the model. However, regularization schemes are diverse (Moradi et al. 2020); in the following, we re- view the popular regularization schemes: weight regularization (weight decay) (Gitman & Ginsburg 2017) penalizes the model during training based on the magnitude of the weights (Van Laarhoven, 2017 This encourages the model to map the inputs the outputs of the training dataset such that the weights of the model are kept small (Salimans & Kingma, 2016). Batch-normalization reg- ularizes the network by reducing the internal covariate shift: it scales the output of the layer, by standardizing the activations of each input variable per mini-batch (Toffe & Szegedy 2015).
Ensemble learning [Zhou 202T) trains multiple models (with heterogeneous architectures) and aver- ages the predictions of all of them (Breiman 1996). Activity regularization (Kilinc & Uysal 2018b) penalizes the model during training based on the magnitude of the activations (Deng et al. 2019 Kilinc & Uysal| 20). Weight constraint limits the magnitude of weights to be within range (ree-
bro & Shraibman| (205).. Dropout (Srivastava et al. 2014) probabilistically removes inputs during training: dropout relies the rationale of ensemble learning that trains multiple models. However, training and maintaining multiple models in parallel infficts heavy computationallmmeory expenses. Alternatively, dropout proposes that single model can be leveraged to simulate training expo-
2

---

Under review as a conference paper at ICLR 2023
nential number of different network architectures concurrently by randomly dropping out nodes during training [Goodfellow et al. 2016). Early stopping (Yao et al., 2007) monitors the model's performance on validation set and stops training when performance starts to degrade (Goodfellow et al. 2016). Data augmentation. arguably the best regularization scheme, creates fake data and aug- ments the training set (Hermandez-Garcia & Konig 2018). Label smoothing (Lukasik et al.) 2020) is commonly used in training deep learning (DL) models, where one-hot training labels are mixed with uniform label vectors (Meister et al. 2020). Smoothing (Xu al., 2020) has been shown to improve both predictive performance and model calibration (Li et al. [020b] Yuan et al. 22200).
Noise schemes inject (usually Gaussian) noise to various components of the machine learning (ML) systems: activations, weights, gradients, and outputs (targets/labels) (Poole et al.) (014)). In that, noise schemes provide a more generic and therefore more applicable approach to regularization that is invariant to the architectures, losses, activations of the MI. systems, and even the type of problem hand to be addressed (Holmstrom & Koistinen 1992). As such, noise has been shown effective for generalization well as robustness of variety of ML systems (Neelakantan et al. 2015).
(iii) Our simulation setting in this paper follows that Generative EBM (we briefly say EBM hence- forth) (LeCun et al., 2006). EBM (Nijkamp et al. 2020 is class of maximum likelihood model that maps each input to an un- normalizeeed scalar value named energy. EBM a powerful model that has been applied to many different domains, such as structured prediction (Belanger & McCallum 2016), machine translation (Tu et al.) 2020), text generation (Deng et al. 2020), reinforcement learn- ing (Haarnoja al.) (017), image generation (Xie et al., 2016), memory modeling (Bartunov al.)
2019), classification (Grathwohl et al. 2019), continual learning (Li et al. 2020a), and biologically-
plausible training (Scellier& & Bengio| 2017). (iv) Our EBM setting leverages separate autoencoders
(Chen et al. 2018) for classification, in that it resembles anomaly detection scenarios (Zhou & Par-
fenroth 201 An Cho 2013) and also one-class classification (Ruff al. 2018 Liznerski al.
2020
Perera & Patel 2019; Sohn et al. 2020).
3 EBM AND ACTIVATION NOISE DURING TRAINING AND INFERENCE
EBM is class of maximum likelihood model that determines the likelihood of a data point æ X C TeD using the Boltzmann distribution:
po exp(-Eo(x)) 2(0)= - JæEN exp(-Eo(z))ddz (2())
where (x) RDD - R. known as the energy function, is a neural network parameterized by 0, that maps each data point æ to scalar energy value, and 12(0) is the partition function. To solve the classification task in EBM utilizing activation noise, we adjust the general formulation of EBM in Eq. 2 as follows: given a class label y discrete set y and the activation noise during training, for each input we use the Boltzmann distribution to define the conditional likelihood as follows:
(e v. = exp (-E Ex (x
y. z))
0 v)
exp E-Ee (x y. )) dæ
(3)
S2(0 y. x)
JEEX
where (ar y, (TRD N, R. R is the energy function that maps input given label and noise to a scalar energy value L' [æ y, x). and (2(0) y, z) is the normalization function.
3.1 TRAINING WITH ACTIVATION NOISE
During training, we want the distribution defined by F. to model the data distribution PD PD y), which we achieve by minimizing the negative log likelihood (0, {())) of the data follows:
(0" q" (x)) = arg min (M q(2)), (M q())) = (< z=q(=) |- log (e y, x) 0.q(=)
where q(2) is the distribution of the activation noise z during training.
We explicate the relationships between activation noise during training and (i) dropout, (ii) loss regularization, and (iii) data augmentation. We start by presenting theoretical results illuminating that activation noise is general form of dropout. For that, define the negate distribution of
3

---

Under review as a conference paper at ICLR 2023
18]
Crspped
81
21
81 2g
f2 85 23
Figure I: To mitigate overfitting, dropout proposes that a single model can simulate an exponen- tial number of different network architectures concurrently by randomly dropping out nodes during training. As visualized, the neural network in Fig. (a) approximates many neural sub-networks such as those in Figs. (b), (c), and (d) where dropped neurons are painted red. Via this virtual ensem- ble learning, dropout mitigates overfitting, In this paper, we prove that activation noise generalizes dropout. shown Fig. (e), for activation noise at each neuron, the activation signal 8; (shown with blue) is added by noise 2; (shown with yellow) which could come from a variety of distribu- tions. When the noise distribution is the negate of the signal distribution for a neuron, that neuron is dropped (shown with red color), and in this case, the activation noise reduces to dropout.
primary distribution, based on which we derive Activation Noise Generality Proposition, stating that dropout is a special case of activation noise.
Definition (Negate Random Variable): We define random variable W E E the negate of random variable X A + F, denoted by X R W (where Σ, A, E, F C if the outcome of W negates the outcome of X; mathematically speaking, x +
Proposition 1 (Activation Noise Generality Proposition): When the noise at a given neuron comes from the negate random variable with respect to the signal S, i.e., * S, the activity noise drops out (the signal of) that neuron. Specifically, the summation of signal and noise becomes zero, for all outcomes.
s + z =
Proof: The proof follows from the definition of the negate random variable.
This theoretical result implies that activation noise can be considered as a general form of dropout: Fig. [ visualizes how the activation noise reduces to dropout. In Fig. 2 we compare the performances of activation noise with dropout with the simulation setting as for Fig.
Now explain how activation noise relates to loss regularization and data augmentation. To that end, we consider the EBM setting leveraging multiple distinct autoencoders for classification: one autoencoder is used for each class. We first write the empirical error without noise in the form of the mean squared error (MSE) loss function as follows:
;5=x 16=-(
(x y) a x (2,, y)dx VEY
ΣΣ [50(x (1) =x]? pD y)
VEY
where fe y) denotes the model parameterized by 0 given label y, and the energy is determined by (x flfe (x v) 2 Mcanwhile, fo x y) and a refer to the kth element of the output of the model and desired target (i.e., the original input), respectively. With activation noise, however, have
10 = T-=E// yEY
[f v.2) xx]² pD (a g)q(z)dr
(6)
where (x v) denotes the kth element of the output of the noisy model. The noise comes
from distribution q(z) during training. Expanding the network function fåt (x: y, 2) into the signal response (xx v) and the noisy response h v y, =) to give
(x y, 2)
f (a: (w) n (a: y. z):

---

Under review as a conference paper at ICLR 2023
CIFAR 10 Dataset
CFAR 20 Dataset
30
AcCuricy 60
25
Dropout
Dropout
1.00
Figure 2: Comparison between activation noise and dropout for CIFAR-10 and CIFAR-20 datasets as the value of noise scalar ar and dropout scalar e increase (for n 1000).
we can write Eq.
6 as lg = Ig + Ig. where Lā encapsulates the noisy portion of the loss given by
- - VEV
[hā(2 1) 2xx (x) (f 1v) =w)] pn(æ, v)g(z)drzdzs (8)
The term I can be viewed as the loss regularizer, specifically Tikhonov regularizer, has been shown in (Bishop) 1995). In general, not just for MSE as we have done, but for any arbitrary loss function v (x y. 2).z), .) we can re-write it as a combination of the losses pertaining to the signal component plus noise component and then the same result would hold. As suggested before, the second term of eиror, besides being regularizer, that is interleaved with the primary dataset during training. Hence, this way it can be said that the noise augments the dataset (Goodfellow et al. 2016).
also be
the loss for an auxiliary dataset
This section concerned how activation noise relates to dropout, loss regularization, and data aug- mentation. As alluded to. activation noise during training is beneficial for the performance; but, what we used activation noise also during inference? We will now answer this question: for EBM, activation noise be also beneficial in inference. Specifically, activation noise during inference in the EBM framework and then present the experiments yielding surprising results demonstrating the effectiveness of activation noise during inference.
will first present the role of
3.2 INFERENCE WITH ACTIVATION NOISE
Consider the inference phase assuming training has been done (i.e., when 0" has been determined). Given a test data point ær, we estimate the energy of our trained model Eeg (a: y: 2) with many different noise realizations following inference noise distribution r(z) which are averaged over to produce the energy. Therefore, the noise distribution r(z) ) at the inference can be considered as a sampler. Probabilistically speaking, we measure the expectation of the energy Eg (æ y) with respect to distribution r(z) as follows:
Eg (x) y) Ez EEe- y. x) Eg (æ V.z)r(z)dz.
(9)
The variance of Eg (a y, z), is determined by o² = { Eg- (x y, r)(()dd - Ég (æ y)² In practice, because the calculation of the integral in Eq. Is intractable, we perform the inference via Eg (x y) 1 Σ Eg- (æ y, (0) where the integral in Eq. 9 numerically approximated sampling from the distribution r(z) which generates the noise samples (2)0). Finally, given an input æ, the class label predicted by our EBM is the class with the smallest energy at we find the target class via arg min Ég- (x y²)
''EY
This approach of classification is common formulation for making inference which is derived from Bayes' rule. There is one difference, however, and that in EBM classification we seek the class whose energy is the minimum the class that the input data belongs to.
In the end note that, as discussed in this section, activation noise not only generalizes dropout during training, regularization scheme, but also offers the opportunity of sampling the model during inference (to minimize the inference error) possibly using a wide range of noise distributions as the sampler; this is advantageous dropout that is only applicable during the training. In the next section, we will first present the simulation settings detailing the architectures used to conduct the experiments and then the consequent results.
5

---

Under review as a conference paper at ICLR 2023
4 SIMULATION SETTING AND RESULTS
4.1 SIMPLE REPRESENTATIVE SIMULATION RESULTS
For the purpose of illustration, we first report only a part of our simulation setting pertaining to only one dataset, a part that is representative of the general theme of our results in this paper. This suffices for initiating the motivation required for further theoretical and empirical studies. In the next subsections, we will provide a comprehensive account of our experiments.
In this simulation, we trained 10 autoencoders, one for each class of CIFAR-IO dataset. Our au- toencoders incorporate the noise Σ which is a random variable following the standard Gaussian distribution (zero mean and unit variance) that is multiplied by the scalar a for each neuron of each layer of the model presented in Eq. | We use convolutional neural networks (CNNs) with channel numbers of [30, 73, 100] for the encoder, and the mirror of it for the decoder. The stride is 2, padding is 1, and the window size is 4x4. We used MSE as the loss function although binary cross-entropy (BCE) has also been tried and produced similar results. The number of samples n. is set to 1,000. This simulation setting is of interest in incremental learning where only one few classes present at a time (van de Ven et al. 2021) Moreover, doing classification via generative modeling fits in the framework of predictive coding which is model of visual processing which proposes that the brain possesses a generative model of input signal with prediction loss contributing as a mean for both learning and attention (Keller Welling 202T.
Fig. 3 shows the performances (classification accuracies) in terms of the scalar of Gaussian noise during both training and test. The omnipresent activation noise which present in both training and inference exhibits two surprising behaviors in our simulations: (i) it significantly improves the performance (about 200%), which unprecedented among similar studies the literature. (ii) The other interesting observation is that always being on or close to the diagonal leads to the best results. These observations will be discussed.
4.2 THEORIES FOR THE SIMULATION RESULTS
We first ask the question that why is noise so effective in improving the performance? For that, we need reiterate that noise is playing different roles (i) training and (ii) inference. For the first role, as alluded to before, what actually activation noise is doing during training is regularization. We suggested that indeed activation noise general case of dropout. We later demonstrated that activation noise is a form of loss regularization, and also an instance of data augmentation. All these categories fall under the umbrella of regularization. Meanwhile, we learned that, during the test phase, activation noise does sampling which reduces the inference error. In aggregation, therefore, regularization hand in hand with sampling account for this performance boost (about 200%).
The other interesting observation in our simulations is that to attain the highest performance, it is best for the noise scalar a (and in general for the distributions) of training and test noise to be equal. This can be clearly observed in Fig. B (and also in all other numerical results that we will present later). We ask the question that why is there such a strong correlation? Shouldn't the two roles of noise during training and test be mutually independent? Where and how do these two roles link to each other? Upon empirical and theoretical investigations, we concluded that the inter-relation between noise during training and test can be characterized and proved in the following theorem.
Theorem (Noise Distribution Must be the Same During Training and Test): Via having the infer- ence noise (i.e., the sampler) r(z) following the same distribution of noise during training q" (z), ie., r" r²(2)= = (z), the loss of the inference is minimized.
Proof: During the inference, to optimize the performance of the EBM model, the objective is to find the distribution r" (z) for the activation noise that minimizes the loss specified as follows:
r" (z) arg min (0" r(z)). CML (0", r())) E(n.g)>po: z<r(z) [- log pa- (x) y. x)]
r(z)
(10)
Meanwhile, from Eq. 4 we know that training is performed to minimize the loss (0 . (z)) with distribution q" (z) as the activation noise. Therefore, during inference, if we set the activation noise r" (=() the same as q² (z() in Eq. 10, the loss again will be minimal.
6

---

Under review as a conference paper at ICLR 2023
0.0
0.1
26.15
30.10
26.95
20.76
22,82
23.05
20.39
25.21
28.2
29.33
29.95
13.75
30.72
33.21
$1.02
32.07
52.49
36.5
32.35
15.0
32.52
34.11
27.76
31,94
28.92
30,02
29.22
29.52
35.24
31.31
35.71
0.3
16.67
25.76
37.27
46.17
9691
10.90
25.77
25,38
29.06 36.1
32.6 12.05
16.56
15.17
92.21
53.17
36.54
35449
0.4
14.17
24.36
23.32
32.49
50.92
48.2
40.25
38.61
25.45
28.6 33.05
33.86
3115
34.90
35.24
34.77
33.9
31.89
25.43
20.08
0.3
12:26
52.13
33:45
0.6
11.02
14.34
14.1
15.71
24.01
43.20
$1.01
DO.27
40.73
39.32
38.19
34.33
30.07
$3.7
57.85
35.00
50.59
39.3
35.11
21.63
0.8
11:77
24.04
22.21
21.19
15.3
26.12
52.27
00.05
02
45.27
19.09
17.23
16.44
15.15
15-48
Text Moue Nolee Scalaraa
20.19
17.29
20.25
17.26
53.3
41-48
40.05
40.06
39.44
58.07
37.68
1.0
16.75
10.12
31.75
1.1
10.02
10:35
13.45
10.20
10.99
11,02
10.09
11.55
21.36
34.33
$3.08
60.81
58.17
52.14
68.54
63.57
63.99
39.88
38.2
37.54
1.3
9:05
11.46
12.21
15.52
9.73
3.96
9.86
10.17
14.63
14.18
23.63
41.49
96.23
$9.93
57.19
52.57
49:28
46.52
41.89
42.42
10.45
11:4
12.85
11.29
10.20
10.1
12.07
15.75
27.57
42:34
$8.19
60.00
56.70
52.84
A8.11
44.59
15-
10:03
16.83
13:03
10:22
10.95
13.91
17.88
47.25
16.26
55.93
55.25
46.71
1.4
7.77
10:01
14.01
12:95
10.07
10.75
9.84
15:05
10.00
11:45
10.89
14.08
20.34
14.71
56.19
55.11
57.34
56.18
56.71
64.21
1.8
9.6
11.86
14.82
9.47
15.58
11.08
11.4
10.52
11.63
10:67 10.45
13.65
1538
19.5
27.19
38.69
49.94
54.79
57.00
54.84
10:79
10:31
10:35
06
18.5
29.03
49.26
54.75
55.36
0.1
0.5
0.4
0.6
0.6
1.3
1.8
Scalor
Figure 3: Different values for the scalar a of activation noise during training and test. When the activation noise during training and test are zero (no noise), the classification accuracy is 20.48% whereas at noise coordinate (0.9, 0.9) it 64.55%. This is more than three times increase or 200% improvement in performance. The first and second elements of the noise coordinate represent the values of noise scalar, a, the training and inference, respectively.
This implies that, during training, the distribution of our trained model is impacted by the distribution of noise q(z) via the second term of loss function Eqs. where the network function learns the distribution of signal in presence of certain noise distribution. During inference, when the noise distribution is the same as in the training, the performance is the best, and as soon as there is discrepancy between the two distributions, the performance degrades.
4.3 COMPREHENSIVE SIMULATION RESULTS FOR GAUSSIAN NOISE
We present extensive simulation results for various datasets (using Google Colab Pro+) considering Gaussian activation noise. We used the same architecture the one presented in Section 4.1 for all of our five datasets: CIFAR-10, CIFAR- 100 Krizhevsky et al.) 2009). CalTech- 101 (Fei-Fei al., 2004). Flower-102 (Nilsback & Zisserman 2008), and CUB-200 (aa et al.) (011). Given that the volume of our experiments is large, for computational tractability we slimmed down the larger datasets from about 100 (or 200) classes to 20 classes.
Our comprehensive simulation results explore the performances of our proposed EBM framework across five dimensions: (i) for five datasets examined the joint impacts of various noise distribu- tions. The noise scalar a varies during both (ii) training and (iii) test time within the range [0, 2) for 20 values with the step size of 0.1. However, do report all of them due the page limit;

---

Under review as a conference paper at ICLR 2023
Table I: Accuracies for various datasets, noise scalars a, and number of samples n. We observe that the performance on all datasets (except for CIFAR-20) peak at the noise scalars (0.9, 0.9). Meanwhile, the number of samples is set to 10 except for a during inference.
Scalars,
Samples
CIFAR-10
CIFAR-20
CalTech-20
Flower-20
CUB-20
(0.0,0.0)
1
20.48+2.03
14.11+2.15
16.47=2.13
19.6142.21
18.53
(0.0, 0.5)
n 10
11.6341.74
18.23
24.93=1.55
27.01
21.9441.66
(0.5, 0.0)
n 1
23.99+1.34
15.91
17.03=2.01
20.08=2.01
19.39
0.55
n 10
58.7241.95
21.4441.78
25.32=1.48
26.05=1.53
23.85#1.41
0.95
n 10
15.7141.92
21.97
20.31#1.69
23.01+1.21
22.03
1..5,
10
11.38
9.64
10.75
20.12:1.55
19.35
0.5).
n 10
31.38+1.87
18.98
19.13=1.02
25.98=1.34
21.61
(0.9, 0.9)
10
65.88+1.10
26.43+22.08
30.32
31.05+1.29
29.83+1.37
1.5)
п 10
14.31+2.01
19.87
21.32=1.67
23.05#1.29
22.43+11.36
01.5,0.5
N 104
36.3211.84
19.6311161
15.25=0.63
23.71:1.54
22.4611.39
0..5
n 10
40.14
22.33
23.43=1.43
22.33
21.49
(1.5, 1.5)
7 = 10
59.23+2241
28.09
27.68=1.44
28.42#1.66
24.38+1.60
instead, selectively present values that representative of the two key observations which intend to highlight: these values are (0.5, (0.5, 0.9). (0.5, 1.5), (0.9, 0.5), (0.9, 0.9), (0.9, 1.5), and (1.5, 1.5), where the pair (t denotes the sealar a of noise at training and test. (iv) The num- ber of samples n is set to 10 except for the cases of no/zero test noise in which n = 1, because when the scalar a of noise during test is 0 (no noise) such as in rows (0.0, 0.0) and (0.5, 0.0), the corresponding accuracy would not vary. Finally, (v) we assess the performances for different noise distributions via exploring all of their permutations.
In our simulations we resize the resolution of all images for our datasets to 32x32. The number of epochs is set to 100. For optimizer, Adam is chosen with the default learning rate of 0.001 and default momentum (Kingma & Ba 2014). The minibatch size is 256. We ran 10 experiments for each reported number in Table 1 and we present the means as well as standard error of the means (SEMs) over these runs.
The first observation we intend emphasize is that this very activation noise can contribute to considerable improvement in performance as high as 200%. This can be discerned in Table [ when comparing the row (0.0, 0.0) pertaining to the case with no noise, with the row (0.9, 0.9). When going from the former case to the latter case, the accuracy jumps from 20.48% to 65.88% for CIFAR-10 10 dataset with n = 10, 000 samples.
The same phenomenon can also be observed for all other datasets. Compared with the previous studies in the literature on the effectiveness of noise, this level of performance enhancement that is acquired by injecting simple noise is unprecedented if not uninaginable. This performance enhancement, as we theorized and proved in Theorem 1, is a result of the combination of both regularization and sampling, not in independent way, but indeed in complicated interdependent fashion (see Fig. to discern the interdependency).
The second observation signities the importance of having balance in injecting noise at training and test: the noise distribution at test r(z) ought to follow the distribution of the noise during training g(z) so that the loss of the inference minimized discussed in Theorem 1. This can be noticed when contrasting balanced rows (0.5, 0.5), (0.9, 0.9), and (1.5, 1.5) with the rest which are unbalanced. Interestingly, even the row (0.0, 0.0) which enjoys no noise outperforms unbalanced rows (0.0, 0.5), (0.5, 0.9). (0.5, 1.5), and (0.9, 1.5).
The third observation the asymmetry between having unbalanced noise for training versus test. Too large scalar for noise during test has far more negative impact on the performance than otherwise. For example, consider rows (0.5, and (0.0, 0.5). the former (large noise scalar a at training) delivers about 100% higher accuracies than the latter which bears high noise scalar a at test. This pattern, with no exception, repeats in all rows where the noise scalar during test is larger than that of the training noise such as (0.0, 0.5), (0.5, 0.9). (0.5, 1.5), and (0.9, 1.5). All these cases yield poorest performances even worse than (0.0, 0.0). This observation (as well as the preceding two observations) are better discerned in Fig. which portrays the heatmap of performance results.
8

---

Under review as a conference paper at ICLR 2023
1.0
Caudhy y
0.8
0.6
(m-4)
50
0.4
Ingnomal (1- 0m))
40
Idizrbbdon 21
0.2
20
10.0
-75
-5.0
-2.5
0.0
2.5
5.0
7.5
10.0
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Noise Scalar,
Figure 4: On the left side, we see different profiles pertaining to various noise distributions. On the right side, the accuracies for different noise distributions are shown as the scalar increases.
4.4 INVESTIGATING OTHER NOISE DISTRIBUTIONS
We now evaluate the performances alternative noise distributions. consider both types of distributions, symmetric and asymmetrie: (i) Cauchy, (ii) uniform, (iii) Chi, (iv) lognormal, (v) t-Distribution, and (vi) exponential. For noise distributions that are capable of becoming either symmetric not (e.g., uniform and Cauchy), explore parameters that keep them symmetric around zero; because based on our experiments (as we will see) we are convinced that symmetric distributions (i.e., symmetric about zero) consistently yield higher performances.
For different noise distributions, the underlying rule that justifies the results can be summarized as follows: when the activation noise 2 is added the signal it is conducive the performance as far as it does not entirely distort the signal. In other words, the Signal-to-Noise Ratio (SNR) can be beneficially decreased to threshold that delivers the peak performance; after that the performance begins to degrade. Given that = s + determine the value of the SNR (given by var[s]/ var[z]): narrowly (or broadly) the density of the random variable is distributed (i.c., var[3];; (ii) how large the scalar of is. Fig- presents the profile of different probability distributions: note the breadth of various probability distributions as they play vital role in how well they perform. In the following we discuss our results with respect to the value of the SNR.
+ až, and is constant, there are two factors that
the distribution of 8, particularly how
As we can see in Fig. (i) adopting symmetric probability distributions as activation noise sub- stantially outperforms asymmetric ones: hence, we observe that Cauchy, Gaussian, uniform, and t distribution consistently yield higher accuracies than Chi, exponential, and lognormal. This is be- cause having a symmetric sampling can more effectively explore the learned manifold the neural network. (ii) The performances of the different noise distributions the value the SNR as follows: as we reduce the SNR via increasing the noise scalar a, the per- formance rises up peak and then falls; this is valid for all the probability distributions. The differences in the slope of various distributions (in Fig. 4 on right) originate from the difference in the profile of the noise distributions: the wider the noise profile is (the larger var(ž)). the sooner the SNR will drop, and the earlier the pattern of rise and fall will occur. For example, in Fig. 4 the Gaussian distribution thinner than the Cauchy, its pattern of rise and fall happens with a small delay since its SNR drops with latency.
be justified with respect to
CONCLUSION
In this paper, specifically for an EBM setting, we studied the impacts of activation noise during training time, broached its role at the inference time, and scrutinized the relationship between acti- vation noise at training and inference. We proved that, during training, activation noise general form of dropout; we discussed the relationship between activation noise and loss regularization/datt augmentation. We studied the empirrical/theorticall aspects activation noise at inference time and proved that adopting noisy activation functions, during inference, can be interpreted as sampling the model. We proved/demonsstrated that when the sampler follows the same distribution as the noise during training, the loss of the inference is minimized and therefore the performance is optimized. To assess our theoretical results, we performed extensive experiments exploring different datasets, noise scalars during training and inference, and different noise distributions.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Larry F Abbott and Peter Dayan. The effect of correlated variability on the accuracy of a population code. Neural computation, 11(1):91-10,, 1999.
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruc- tion probability. Special Lecture on IE, 2(1):1-18, 2015.
Bruno B Averbeck, Peter E Latham, and Alexandre Pouget. Neural correlations, population coding and computation. Nature reviews neuroxcience, 7(5):358-366, 2006.
Sergey Bartunov, Jack W Rae, Simon Osindero, and Timothy P Lillicrap. Meta-learning deep energy- based memory models. arXiv:1910.02220, 2019.
David Belanger and Andrew McCallum. Structured prediction energy networks. In International Conference on Machine Learning, pp. 983-992, PMLR, 2016.
Chris M Bishop. Training with noise is equivalent to Tikhonov regularization. Neural computation, 7(1):108-116, 1995.
Leo Breiman. Bagging predictors. Machine learning, 24(2):1231140, 1996.
Zhaomin Chen, Chai Kiat Yeo, Bu Sung Lee, and Chiew Tong Lau. autoencoder-based network anomaly detection. In 2018 Wireless Telecommunications Symposium (WTS), pp. 1-5. IEEE, 2018.
Yang Dan, Jose-Maelll Alonso, W Martin Usrey, and R Clay Reid. Coding of visual information by precisely correlated spikes in the lateral geniculate nucleus. Nature neuroscience, 1(6):501-507 1998.
Lei Deng, Yujie Wu, Yifan Hu, Ling Liang, Guoqi Li, Xing Hu, Yufci Ding, Peng Li, and Yuan Xie. Comprehensive SNN compression using ADMM optimization and activity regularization. arXiv:1911.00822. 2019.
Yuntian Deng, Anton Bakhtin, Myle Oit, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy-based models for text generation. arXiv:2004.11144 2020.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 Conference on Computer Vision and Pattern Recognition Workshop, pp. 178-178. IEEE, 2004.
Igor Gitman and Boris Ginsburg. Comparison of batch normalization and weight normalization algorithms for the large-scale image classification. arXiv:1709.08145 2017.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Will Grathwohl, Kuan-Chieh Wang, Jorn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv:1912.03263: 2019.
Caglar Gulcehre, Marcin Moczulski, Misha Denil, and Yoshua Bengio. Noisy activation functions. In International Conference on Machine Learning, pp. 3059-3068. 2016.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pp. 1352-1361. PMLR, 2017.
Alex Hernandez- Gareia and Peter Konig. Data augmentation instead of explicit regularization. arXiv:1806.03888 2018.
Lasse Holmstrom and Petri Koistinen. Using additive noise in pack-propagation training. 1EEE Trans. on Neural Networks, 3(1):24-38, 1992.
Kurt Horiik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural nefworks, 2(5):359-366, 1989.
10

---

Under review as a conference paper at ICLR 2023
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448 456, 2015.
Anderson Keller and Max Welling. Predictive coding with topographic variational autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1086-1091, 2021.
Ozsel Kilinc and Ismail Uysal. GAR: an efficient and scalable graph- based activity regularization for semi-supervised learning. Neurocomputing. 296:46-55 2018: Sa.
Ozsel Kiline and Ismail Uysal. Learning latent representations in neural networks for clustering through pseudo supervision and graph- based activity regularization. arXiv:/802.03063 2018b.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Kukacka, Vladimir Golkov, and Daniel Cremers. Regularization for deep learning: A taxonomy. arXiv:1710.10686, 2017.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. tutorial energy-based learning. Predicting structured data, 1(0), 2006.
Shuang Li, Yilun Du, Gido M van de Ven, and Igor Mordatch. Energy-based models for continual learning. arXiv:2011.12216, 2020a.
Weizhi Li, Gautam Dasarathy, and Visar Berisha. Regularization via structural label smoothing. In International Conference Artificial Intelligence and Statistics, pp. 1453-1463, 2020b.
Grace W. Lindsay. Testing the tools of systems neuroscience on artificial neural networks. arXiv:2202.07033, 2020.
Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus- Robert Müller. Explainable deep one-class classification. arXiv:2007.01700 2020.
Michal Lukasik, Srinadh Bhojanapalli. Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pp. 6448-6458, 2020.
Clara Meister, Elizabeth Salesky, and Ryan Cotterell. Generalized entropy regularization or: There's nothing special about label smoothing. 20Xi::2005.00820, 2020.
Reza Moradi, Reza Berangi, and Behrouz Minaei. A survey of regularization strategies for deep models. Artificial Intelligence Review, 53(6))3947-3988) 2020.
Arvind Neelakantan, Luke Vilnis, Quoe V Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks. arXiv:1511.06807 2015.
Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of meme-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5272-5280, 2020.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics Image Processing, pp. 722-729. IEEE, 2008
Pramuditha Perera and Vishal M Patel. Learning deep features for one-class classification. IEEE Trans. on Image Processing, 28(1 1)55400-54633 2019.
Ben Poole, Jascha Sohl-Dicksteen, and Surya Ganguli. Analyzing noise autoencoders and deep networks. arXiv: 1831, 2014.
11

---

Under review as a conference paper at ICLR 2023
Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22(11):1611-177, 2019.
Ranulfo Romo, Adrián Hernández, Antonio Zainos, and Emilio Salinas. Correlated neuronal dis- charges that increase coding efficiency during perceptual discrimination. Neuron, 38(4):649--0) 657, 2003.
Kiersten Ruda, Joel Zylberberg, and Greg D Field. Ignoring correlated activity causes a failure of retinal population codes. Nature communications, 11(1):1-15, 2020.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International Conference on Machine Learning, pp. 4393-4402. PMLR, 2018.
Tim Salimans and Durk Kingma, Weight normalization: simple reparameterization accelerate training of deep neural networks. Advances in Neural Information Processing Systems, 29:901- 909, 2016.
Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- based models and backpropagation. Frontiers in computational neuroscience, 11:24, 2017.
Kihyuk Sohn, Chun-Liang Li, Jinsung Yoon. Minho Jin, and Tomas Pfister, Learning and evaluating representations for deep one-class classification. arXiv:2011.02578 2020.
Nathan Srebro and Adi Shraibman. Rank, trace-norm and max-norm. In International Conference on Computational Learning Theory, pp. 545-560, 2005.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(9):1929-1988 2014.
Yingjie Tian and Yuqi Zhang. A comprehensive survey regularization strategies in machine learning. Information Fusion, 80:146-166, 2022.
Lifu Tu, Richard Yuanzhe Pang, Sam Wiseman, and Kevin Gimpel. Engine: Energy-based inference networks for non-autoregressive machine translation. arXiv:2005.0088 2020.
Gido M. van de Ven, Zhe Li, and Andreas S. Tolias. Class-incremental learning with generative classifiers. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 3611-3620, 2021.
Twan Van Laarhoven. L.2 regularization versus batch and weight normalization. arXiv: 2017.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Leon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with local denoising criterion. Journal of machine learning research, 1 11(1)), 2010.
C. Wah, S. Branson, P. Welinder, Perona, and Belongie. The Caltech-UCSD Birds-200-200 Dataset. Technical report, 2011.
Si Wu, Hiroyuki Nakahara, and Shun-ichi Amari. Population coding with correlation and un- faithful model. Neural computation, 13(4):775 797, 2001.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference Machine Learning, pp. 2635-2644. PMLR, 2016.
Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, and Rong Jin. Towards understanding label smoothing. arXiv:2006. 1653, 2020.
Zongben Xu, Xiangyu Chang, Fengmin Xu, and Hai Zhang. L.112) regularization: A thresholding representation theory and a fast solver. IEEE Transactions on neural networks and learning systems, 2317)2101- 2012.
12

---

Under review as a conference paper at ICLR 2023
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learn- ing. Constructive Approximation, 26(2):289-315 2007.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3903-3911, 2020.
Chong Zhou and Randy C Paffenroth. Anomaly detection with robust deep autoencoders. In Pro- ceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 665 674, 2017.
Zhi-Hua Zhou. Ensemble learning. In Machine learning, pp. 181-210. 2021.
Ehud Zohary, Michael N Shadlen, and William T Newsome. Correlated neuronal discharge rate and its implications for psychophysical performance. Nature, 170064855:140-11444 1994.
Joel Zylberberg, Jon Cafaro, Maxwell Turner, Eric Shea-Brown and Fred Rieke. Direction- selective circuits shape noise to ensure a precise population code. Neuron, 89(2):699-383, 2016.
13

---

Under review as a conference paper at ICLR 2023
Appendix
With all that has been said on activation noise, however, there still remains a number of results unreported and many critical questions left unanswered due to the space limit: in this supplementary material, (a) we report some leftover results elaborating on those results given Section (b) We answer the question that what is the impact of the number of samples
the performance? (c) In
the main text, we used the same noise distribution during both training and inference but what if the noise distributions differ during training and inference? (d) We applied activation noise to all layers of the network but what if only portion of layers are injected with noise? How about exploring the case where both the noise scalar a and distribution vary at different layers? (e) We studied the effectiveness of activation noise specifically for a generative EBM setting; but, what about the typical discriminative modeling image classification scenario? Is activation noise also effective for that? We will answer all these questions comprehensively in this supplementary material exactly in the alphabetical order that they were presented above: we perform extensive experiments exploring many simulation scenarios pertaining to questions asked above.
A MORE DETAILS FOR THE PERFORMANCE RESULTS OF VARIOUS NOISE DISTRIBUTIONS
In the following provide the specifications of our probability distributions: as shown in Fig. 5 Cauchy distribution g(;; 2n. 7), we set 2n, which characterizes the mean of the Cauchy distribution, equal zero; and explore different values of 7. The same way for uniform distribution, the mean is set to zero: while uniform distribution originally has two parameters characterizing the start and end of the random variable, our simulation, because the uniform noise is desired to be symmetric, we define only one parameter characterizing the uniform distribution denoted by referring to its width. For Chi distribution g(ē; k) we assess its performance with different values of whereas for lognormal distribution g(ž; jr, o), the value of μ is set to zero and is varied. For t-distribution g(5; v). different values of y are explored. Worth mentioning that for t-distribution when v 1, the distribution reduces to Cauchy distribution. Finally, for exponential distribution g(ž; A), evaluate the performance for different values of X. Our experiments are performed 10 times via multiple random seeds. We report the means (= SEMs) over these experiments.
Overall, based on the performances across all values of the noise scalar a (as shown in Fig. 6 conclude that the best noise distribution is also the most popular one: Gaussian distribution; perhaps because it has the largest area under the curve for different values of the noise scalar a. Furthermore, it is clear that the distributions that are similar to the Gaussian distribution deliver best performances whereas for dissimilar ones the performance becomes worse. Meanwhile, we realized that various parameters (standard deviations σ) for the Gaussian distribution do not offer convincing improvements and only hasten/delay the occurrence of the rise and fall pattern for classification accuracy.
B IMPACT OF THE NUMBER OF SAMPLES
In Fig. we demonstrate the impact of the number of samples for the standard Gaussian noise ž with different noise scalar values a during training and test. It can be seen that the accuracy rises almost logarithmically the number of samples increases. Note that these results pertain CIFAR-I0 dataset. In Table 2 we provide the comprehensive form of the Table including the classification accuracies for different number of samples
C DIFFERENT NOISE DISTRIBUTIONS DURING TRAINING AND INFERENCE
We examine the performances pertaining to the combination of nonidentical noise distributions: adopting two different distributions, one for training and the other for inference. The parameters of different noise distributions outlined in Table 3 The noise scalar is set to one for training and test. The results of the experiment are displayed in Fig. g that demonstrates the performances for each two noise distributions in 7x7 heatmap grid: the empirical results confirm theoretical
14

---

Under review as a conference paper at ICLR 2023
0.30
Caucty (um 1
Geutean (gm c=11
Couchy y
Buusanip-00
Caucty (u y
0.3
Guussan
0.20
0.10
0.3
10.0
7.5
5.0
2.5
0.0
2.5
5.0
7.5
10.0
30.0
7.5
5.0
2.5
0.0
2.5
5.0
7.5
10.0
0.5
Unifom
Exponential (A 2)
fu
0.8
Exsomential 3)
Unifom (ur 51
0.4
0.2
0.8
Chi ie 11
Lognarmal (1 o-11
Chi (ice 21
Lognormal (gr o=
(k- 31
3-
0.4
0.0
10.0
-25
-10.0
-75
-2.5
10.0
distribation ty
Cauchy (g- y=1)
..0
Expsnamaaa(A11
0.6
Unform (u 4)
Chilk-
(y-
0.2
-10.0 -7.5
-5.0
-2.5
0.0
2.5
5.0
7.5
10.0
-10.0
-75
-5.0
-25
0.0
2.5
5.0
7.5
10.
Figure 5: This figure shows and juxtaposes different probability distributions that were chosen for activation noise. Barring the uniform distribution, all other distributions have heavier tails than that of Gaussian distribution. We believe having a heavy tail hastens the pattern of rise and fall of the classification accuracy it increases the SNR value. Meanwhile, symmetric distributions with high margin outperform asymmetric ones.
conclusions that was proposed in Theorem 1, proving that the optimal performance during inference is acquired if we have the same noise distributions during training and inference.
four symmetric noise distributions, whereas the worst are for when asymmetric noise is adopted for training and symmetric noise for inference (to sample). This result in accordance with our third observation discussed in Section 4.3 and also further corroborates Theorem 1. In Fig. g we can see that if we use symmetric noise during training and asymmetric noise during inference the performance will not be as poor as otherwise. This was also observed and explained in Section 4.4
In Fig. can be seen that the best results pertain
D LAYER-SELECTIVE NOISE FOR THE EBM SETTING
So far, we have injected noise to all the layers of our neural networks. this section, we study the case where inject noise only to portion of layers of the generative EBM model. We design and compare five schemes for noise injection:
(i) full noise (the default) where noise is injected to all layers (denoted by n-full); (ii) and (iii) odd/even noise where noise injected to all odd/even
15

---

Under review as a conference paper at ICLR 2023
60
50
Apccawery
Coudly la- y=
Gaussan c-
Caucty lu 9, y=
Guussanlp 2=
Couchy (u- v=3)
Gaussnn d-
0.00
0.25
0.50
0.75
1.30
1.25
1.50
1.75
0.02
0.5
0.50
0.75
1.00
1.25
1.50
1.75
holee Scalor:
Natso Scalorr
35.0
60
30.0
Accunecy
-50
100
25.0
Exponentail =1
22.5
Exponential 2
30
20
0.50
0.75
1.50
1.25
1.75
0.25
0.75
1.25
1.50
1.75
Noaa
Noiue Scaler,
(hi
Lognarmal (1- --11
Chi
Lognarmal (g o=
52.5
Chi (h)
(1 p-
30.0
Accurery
27.5
25.0
20
60
Icuncy
Accuraryy
Couchy (g-
Gaussian (u- dw Exponential Unilonm (u=4)
distr busion 11
de a)
dit butoon (+=2)
Lognurmal D, dittibution (vm
bution (p=3)
0.00
0.25
0.30
0.75
1.20
1.25
130
1.75
2.00
0.25
0.50
0.75
1.00
1.25
1.30
1.75
hoise Scalar,
Noisa Scalar.
Figure 6: In this figure we demonstrate the performances of different distributions both symmetric and asymmetric for three different cases of parameters in each individual figure. Then at the bottom- right figure we compare the best performances pertaining to the best parameters each individual figure.
layers (denoted by n-odd and n-even); (iv) and (v) encoder/decoder noise where noise is injected encoder/decoder layers (denoted by n-ene and n-dec). Worth mentioning that we used standard Gaussian noise with scalar = pertaining to the peak performance for both training and test.
Fig. presents the accuracies for neural network injected with all the noise schemes presented above. As it can be seen, injecting noise to all layers yields higher performances than the alternatives; n-even and n-odd are at the next rank while n-ene and n-dec are the worse. Fig. also jointly investigates the impact of injecting noise to different layers of neural network at both training and inference. We observe the results akin to the previous results Fig. 3 once again but for each layer: according to our results the noise injected to a specific layer during training and inference better to follow the same distribution. For example, when noise injected to even layers (n-even) of the neural network during training, none of the other noise injection schemes during inference could exceed the performance of n-even noise injection. Also, not injecting noise during training for specific layer while injecting noise during inference for that layer significantly reduces the performance. Accordingly, inspired by above results we can propose Corollary that generalizes Theorem 1.
16

---

Under review as a conference paper at ICLR 2023
Table 2: Accuracies for various datasets, different values of scalar noise a, and range of value for the number of samples n. The pair (-,+) in the left-most column denotes the scalar a of noise at training and test.
Scalars, a (0.0, 0.0)
Samples
CIFAR-10
CIFAR-20
CalTech-20
Flower-20
CUB-20
n
20.4842.0s
22.11
16.47=2.13
19.61
18.53
9.1242.25
13.6342.49
15.4322.18
19.45:2.25
18.3212.27
10
9.7612.22
15.95+2.47
18.57=2.05
21.71
19.13+2.19
(0.0,0.5) 0.5)
7 = 10
9.98
16.43
20.03
23.81
19.81
10
10.8
17.05+2..7
22.59
25.32+2.07
20.32+1.83
n 10a
11.63
18.23+2.11
24.93
27.01+2.00
21.94+11.6
(0.5, 0.0)
1
23.9
15.9142.38
17.03
20.08:2.01
19.39
23.25+2.32
15.9342.44
16.09-1.00
19.7742.07
18.36+2.15
n 10
36.0442.19
17.6322.28
19.01
21.04:1.94
12.09
(0.5, 0.5)
10
42.75+2.14
19.84
21.37=1.74
23.07:1.74
21.2621.98
n 10a
56.23
20.93
23.02
24.85
22.4911.77
N = 10
18.72
21.4411.78
25.32
26.05
13.85
12.25+2.24
17.7742.27
14.45=2.00
19.08=2.10
18.43+2.28
10
12.9412.22
18.89
16.91
20.94
19.49 2:208
0.99
N 10a
13.8212.14
20.45:12.19
18.03:2.03
21.5111.64
20.51
10a
14.652.01
21.344+.08
19.42=1.79
22.95+1.34
21.38+1.74
7 = 10 1
15.71+1.92
21.97+1.60
20.31
23.01+1.21
22.03+1.37
9.3712.43
7,6611.05
7.93
17.6112.21
16,5312.08
10
9.7942.34
8.03+1.03
8.52
18.13
17.07+1.95
(0.5, 1.5)
a 10
10.01+2.13
8.89+1.01
9.04
18.98#1.94
18.39+1.88
10a
10.73+2.04
9.36+0.80
9.93=0.98
19.53+1.63
18.71+1.92
n 10
11.3811.76
9.64:10.84
10. 7.92..99
20.12:1.55
19.35
1
25.98+2.34
16.19+2.24
11.22
19.91
18.78 18.78+22.0
10
27.50+2.17
16.47+2.18
13.87=1.36
21.40#1.99
20.00+1.87
00.9,0.5)
n 10a
28.8112.11
17.33:12.1
15.35
23.33
21.56:11.48
10a
29.6212.09
18,05.12.99
17.61=1.06
24.7611.56
23.28
n 10
31.38+1.87
18.9841.78
19.13=1.02
25.98#1.34
24.61+1.41
n = 1
25.2512.40
22.54:10.24
24.03=2.11
22.52:2.10
21.13::1.99
10
42.0412.37
23.2112:28
26.4112.03
24.94:1204
23.45
(0.9,0.9)
n 102
53.02+1.34
24.37+2.19
28.44+1.99
26.58+1.89
25.6241.56
N 10a
64.555+.115
25.25+2.11
29.32
29.45+1.35
28. 1..45
n 10a
65.88+1.10
26.43++.08
30.32
31.05+1.29
20.83+1.37
N =
I1.37+2.40
17.11+2.14
16.34=2.21
19.88+2.10
19.03+2.04
10
12.24+
17.41+2.11
17.54
20.24+:2.04
19.59
1.5)
n 10a
13.01+2.44
18.13+22.01
18.
18.66#2.02
21.0+1.84
20.72+1.75
n = 10a
13.91
19.01
20.07
22.35+11.0
21.61
n 10
14.3142.11
19.87+1.58
1..32=1.67
23.05
22.43
N = 1
18.75+221
15.73+-.116
9.35++1.33
18.61+2.21
17.38+2211
10
33.5142.18
16.47+2.11
11.56=1.21
20.13:2.05
19.28+2.02
0.5)
n 10
34.1842.07
17.3512.01
13.03
21.3441.93
20.52+1.86
n = 10a
35.95+2004
18.75+11.9
14.78=u.50
22.13+1.74
21.37+1.08
1
36.32+184
19.63+1.61
15.25=0.8a
23.71+1.54
22.46+1.39
n = 1
31.4410.25
18.9210.20
17.07=2.08
18.9242.20
17.57+2.07
10
36.7510.10
19.47+2.03
19.03
19.47
18.73+1..8
10
37.2440.10
20.34+1.93
21.43+1.79
20.34+1.98
19.61+1.72
0.9)
n = 3
39.44+181
21.15+1..b
22.12+1..5
11.15
20,33++1.68
10a
40.1441.79
22.33
23.43=1.43
22.33+1.55
21.49:1.35
N = 1
31.53+2.29
24.37+2.15
23.68=2.05
20.28
19.3942.31
n = 10
38.72+219
25.1541.91
24.34*1.9
21.3142.01
20.47
1
43.9842.55
26.3611183
25.62=1.81
23.2741.97
22.01
(1.5, 1.5)
n 10
58.9342.47
27.15:1178
26.37=1.69
25.5941.74
23.85
n 10a
59.2311241
28.09
27.68#1.44
28.42+1.65
24.38
17

---

Under review as a conference paper at ICLR 2023
Accuracy
30
Noise Scalars (0.5, 0.5)
Noise Scalars [0.5, 0.9)
10
Noise Scalars - 00.9.0.5) 0.5)
Noise Scalars (0.9, 0.9)
10
100
1000
10000
Number Samples
Figure 7: The impact of five different number of samples on the classification accuracy for differ- ent noise coordinates: the pair (.,)) in the legendbox denotes the scalar a of noise at training and test. It appears that the accuracy rises almost logarithmically as the number of samples increases. This can also be seen in Table
Table 3: The parameters used in the experiment pertaining to Fig. We chose the parameters that lead to the best performances different distributions. The noise scalar a set to one for training and test.
Gaussian
Cauchy
uniform
lognormal
Chi
exponential
Parameters
# == 0,
== 0
p= 0,
k:
==1
Corollary 1 (General Noise Distribution for the Sampler During Test): Via having the sampler r(2,v) following both the same placement and distribution of noise during training q" (z, v), i.e, (2) q" (2( where encodes the placement of noise among the layers, the loss of the inference is minimized.
Clearly, when the placement parameter e is selected such that all layers included for noise injec- tion, Corollary 1 reduces to Theorem
E ACTIVATION NOISE IN DISCRIMINATIVE MODELING
So far, we have considered only the generative EBM modeling. In this section, we examine the impact of activation noise the performance of classifiers relying on discriminative modeling: conduct our study with two popular datasets, CIFAR- 10 and CIFAR- on seventeen architectures In this experiment, we train for 30 epochs, the optimizer was Adam with
as listed in Table learning rate 0.001 and the default momentum, and the learning rate scheduler is cosine annealing similarity. Fig. 10 demonstrates the results of our experiments: the results of the CIFAR-10 10 dataset for different architectures, the performances reliably deteriorate as we increase scalar of the omnipresent (standard Gaussian) activation noise even with balanced noise enjoying n 1000 samples.
we can see in Fig. 10 exhibiting
Meanwhile, in Table 5 we report the performances of architectures for both CIFAR-10 and CIFAR-100 with three noise levels: (i) noise 0.0, (ii) mild noise 0.5, and (iii) strong noise a = 1.0. Based on our observations, we are convinced that adding even a mild level of bal- anced noise is detrimental the performance of the discriminative modeling scenario for different architectures and datasets.
To also investigate the scenario for unbalanced noise injection, provide Fig. I that shows the performance heatmap of the discriminative modeling classifier using ResNet50 architecture on CIFAR-10 10 dataset as increase the noise scalar a during both the training and inference. We can
18

---

Under review as a conference paper at ICLR 2023
Gaussian
Cauchy
62.4
63.0
63.0
61.0
19.7
20.2
19.0
Distributions the Test Noise
uniform
56.6
57.0
56.3
64.8
18.1
19.0
18.5
lognormal
29.7
Chi
31.3
30.9
31.1
31.4
29.5
35.2
33.5
exponential
35.1
35.4
35.0
35.6
33.9
Gaussi
Cauchy
uniform lognoraal Distributions the Training Neise
Chi
exponential
Figure 8: Accuracy of different noise distributions for the unbalanced case. The horizontal axis pertains to the noise distributions during training while the vertical axis corresponds to the noise distributions during inference. We that the distribution better be kept the same the training during inference to guarantee the optimal performance.
n-full 64.0
43.4
42.5
39.8
38.4
Injection
58.4
61.0
60.0
40.0
39.7
n-even
58.1
57.9
61.3
41.8
37.9
52.6
50.0
56.3
57.8
43.1
n-dec
53.7
50.9
46.0
46.6
57.4
Schemes of Noise Injection
Figure 9: Accuracy of the selective noise injection for the balanced case.
Table 4: Architectures for studying the impact of activation noise discriminative modeling.
VGG11 ResNet50 ResNeX129(32x40)
VGG13
VGG16 CegNetX-200MM SimpleDLA
VGG19 RegNetX-400MM DenseNet121
ResNet18 MobileNetV2 PreActResNet18
ResNet101
ResNeX(29(2x64d)
DPN92
DLA
19

---

Under review as a conference paper at ICLR 2023
93
VGG13
VG616
Rechee18
Accuraoy
ResNet50
Reshee101
ReglietX, 200MT
Regleet
MobleNatV2
9132xx1
ResNeX123(2x644)
SimpleDLA
DerseNet1221
Neet1
DPN92
Noise Scalar,
Figure 10:
The classification accuracy of different architectures as noise scalar increases. The scalar for the standard Gaussian noise is set the same during both training and inference.
Table Accuracies for CIFAR-10 and CIFAR-I00 datasets, for various balanced noise scalars, and different architectures. The activation noise was injected both during training and inference. The scalar a the standard Gaussian noise set the same during both training and inference.
Noise Scalars
0.0
(No Noise)
a = 0.5 (Mild Noise)
a = 1.0 (Strong Noise)
Architectures
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR- 100
CIFAR-10
CIFAR-100
VGGI1
91.21+t.03
70.48+90.3
91.01+0.05
70.16=a.0r
90. 90.72+0.00
69.92=0.05
VGG13
91.76:t0.08
71.02+0.08
91.41+0.04
70.69=0.0r
91.23+b.07
70.45=0.00
VGG16
92.6440.08
71.86+0.04
92.39+0.01
71.53=0.0z
92.13+0.05
71.23=0.08
VGG19
92.9110004
72.1810.00
92.75
71.89=0.05
92.4510.07
70.64:0.00
ResNet18
93.02#0.04
72.59+00.3
92.82+0.04
72.27=0.05
92.57+0.03
71.72=0.04
ResNe150
93.624:000
73.06=0.01
93.46+0.05
72.78=0004
93.11
72.45=0.00
ResNet101
93.75
73.19
93.56+0.04
72.91=0.03
93.17+0.04
72.60=0.06
RegNe Regoeee
94.2441.03
73.63:20.08
94.03
73.34=0.05
93.78=20.66
73.0020.04
RegNeXtuo
94.29.1..0
#0002
94.11+0.05
73.39=0.04
93.80=0.07
73.07=0.04
MobileNetV2
94.43.10.02
73.83
94.43
73.58=0.03
93.93
73.2222000
Res
94.73++00
74.11+t.00
94.55+0.04
73.83=0.0r
94.25+
73.55+0.000
ResNeX129
94.82++0.04
74.39 +0.01
94.63+0.00
74.06=0.us
94.26++0.07
73.75++0.04
SimpleDLA
94.8911002
74.43:1000
94.77=0.05
74.11=0.03
94.30+0.07
73.85
DenseNet121
95.0440.04
74.62+t0.04
94.83+0.00
74.29
94.54+
74.00
PreActResNetrs
95.1140.04
74.88+D.02
94.95+0.uG
74.51+a.u5
94.633+0.07
74.66=
DPN92
95.16440.03
74.95
94.99
74.66=
94.75++0.05
74.33
DLA
95.47 1:0.05
75.3110.05
95.2310.04
75.00=0.03
95.00
74.73.0000
discern that in the unbalanced noise case, the performance drop is more severe than the balanced case. Moreover, we made three observations Fig. that are noteworthy; these observations are reminiscent of the observations that were made in Section
(i) In Fig. the diagonal represents the balanced noise injection. can seen that the perfor- mance consistently deteriorates when moving from top-left entry (no noise) to bottom-right entry (intense noise); we recall, this is exactly the opposite of the observation made in Fig. 3 there, injecting balanced noise reliably and significantly increased the performance. (ii) We observe that in Fig. unbalanced noise worsens the performance which is in accordance with the observation in Fig. Finally, (iii) just in Fig, 3 there is asymmetry in performance for having
balanced noise: strong noise during inference with weak noise during training delivers noticeably worse performances than otherwise.
20

---

Under review as a conference paper at ICLR 2023
2362
72.06
22.03
23.53
23-43
25.22
22.30
22:04 13.02 12.00 12333 92.33
12.87 22.02
92.01 22.73
92.37
52.73
52-3
5303
43.92 93.63 93.8 93.32
0321
31.1 $1.07 93.01 12.90 92.33 92.4 12.00 97.94 9788 92.00 92.11
92.24
6274
63
53.6
41
93.61 93 93.88 013) 24 47.15 93.02 92.91 92.96 9263 $7.05 90. 07.09 92.83 92.79
9287
92
63
53.3
13.52
23.05
53.53 53.22 13.
53.42
23.35
23.23
90.03 12,52
92.13
92.73
12.92
90.07
92.74 72.03
92.03
52.00
22.53
84
9921
53.04
94.
55
9331 93.87 99.31
53.44
41
15.23 9107 92.3
92.95
92.29
92.09
92.14
9281
92.87
92.99
9282
92.7
63
92.00
22.01
23:01
13.14
20.24
93:44
53-10
23.20
03 12.99 12.30 92.03
72.02
92.00
92.00
00.77
92.01 $2.03
68-
93.73
22:33
22.00
02.93
53.20 93.42
23.36
93:21
12 93.05 92:03 92.03
92.03
92.78
90.02
$2.81
9289
5257
96/66
92.91
75.11
15/15 76:8 9092
7691
90.05
12.9 92.89
52.5
93.51
62.07
92.
68 62-73 97.02
97,91
90.03 97.00
63.3 93.01 $2.09
90.57
92.55
56
62.30
9251
22.33
92.35
8253 1283 92.64 92.75
72.16
92.08 23.05 28.33 29AN
72.04 92.71
9285
12.01
9132
22.46
92.45
124 1256 92.88
72.72 92.81 72.08 15.23 25.0
72.12
94.79
9282
12.74
92.92
52:30
$1,17
91.67
41.13
91.44
92-11 97.50 00.54
71 47.61 77 90.06 90.91 9011
$3.00
01.08
91
92.67
90.53
62
62.14
1,5
$2.48
51.14
92.88
22.00
52.12
22.31
52.47 22.65 23.74 92.73 22.79 22.03 9292
12.87
91.05 95.04
72.81
92.76
5239
14
52A5
20.75
92.11
21.32
F1.87
21.75 52.01 21 22.43 92.73 22,00 22.03 92.3 $2.83 91.34 92.00
22.09
91.05
5283
22.70
1,7
43.38
40
43.94
41.11
41.64 60.09
97.10
99.94 92.55
42.00
40.40
144
5337
00.22
25:15
50.32
50.58
22.41
01.52 21.03
21.42 22.70 22.02 0202
22.15
90.33
91.00 22.94 92.79 5292 52.54
30
01
1.3
Figure 11: The classification accuracy of ResNer50 architecture on CIFAR-10 dataset as the noise scalar a increases both during training and inference for the standard Gaussian noise
Our results for CIFAR-10 10 dataset in Figs. 10 and 11 carry over to CIFAR-100 dataset; hence, refrain from reporting the results pertaining to CIFAR- 100 in separate figures. From the results that have been reported in Figs. TO and 11, and also in Table it becomes clear that, in contrast to generative modeling, in discriminative modeling, the activation noise (injected to all-layers both training and inference) indeed worsens the performance.
F ON WHY ACTIVATION NOISE IS EFFECTIVE FOR GENERATIVE MODELING BUT NOT DISCRIMINATIVE MODELING
We do not have any rigorous mathematical proof showing why activation noise is effective for gen- crative modeling but not discriminative modeling. However, we can conjecture the reason. To this end, we need take step back and ponder about the function of neural networks. As we know, in the broad mathematical sense, neural networks are best understood as approximating often non- linear target function j() that maps input variable X an output variable Y. where X and Y come from training data D. Specifically, the aim is learning the target function j(.) from training data D via having j(-) to perform furve-iitting on the training data D.
For both the discriminative and generative modeling scenario, the dimensions of the input space X is the same; however, when comes to the output space the difference emerges: in a discriminative modeling setting, the output space has far fewer dimensions than that of generative modeling, For example, in the CIFAR-10 dataset, the output space Y for the generative model has 32x32-1024 dimensions whereas the discriminative model has only 10 dimensions. Meanwhile, we know that as soon as the number of dimensions grows, the curse of dimensionality comes to play: our training data D now becomes exponentially insufficient; the data becomes sparsely distributed in the space,
21

---

Under review as a conference paper at ICLR 2023
and instead of having smooth continuous manifold of the data which is desired, we will have patches of data scattered in the space. This makes hard for our j(.) to map the imput to the output because neural networks, the strict sense cannot, and in the practical sense, severely suffer when approximating functions whose samples in the input space are not smooth.
hinted in Universal Approximation Theorem (Horiik al. [989), in
The activation noise, as proved in the main text where we linked activation noise with data aug- mentation, imputes the primary dataset D with an auxiliary dataset DD' smooth the data manifold, thereby mitigating the curse of dimensionality; however, when the output has fewer dimensions, as in the case of discriminative modeling, this problem less pronounced. That said, one question that might arise is that these explanations delivered so far justify adopting activation noise during the training as an regularization scheme; how activation noise during inference comes to play and interact with the activation noise during training?
When we make the manifold of our dataset D more continuous and smooth during training by augmenting it with D', we are instructing the neural network to learn enhanced manifold instead of the original one: new one that is smoother, more continuous, and stretched. During inference we compute the likelihood of a given sample under our model; if the noise scalar is considerably larger than that of training time, causes to stretch the cloud our sample points; then clearly the manifold that is learned by neural networks has not seen the far points of the cloud and included them in itself during training; hence, the neural network would produce unreliable noisy likelihoods for these outlier points of the cloud. This is why the performance severely deteriorates when the noise in inference has larger scalar than that of training (even worse than no sampling/noise), In other words, larger noise scalar a at inference causes the cloud of noise samples fall outside of the convex hull of the dataset on which the neural network is trained. Eventually, as we presented in Theorem 1, the sampling cloud must be of the same shape/distribution the training cloud of noise that is used to smooth the dataset D.
22


Reviews:
[{"review_id": "hLw8oDPMtg", "reviewer": "Reviewer_Ysne", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method to add noise in EBM in classification tasks. The likelihood and loss functions are derived based on the injected noise $z$. The paper claims adding noise during training includes dropout as a special case, and propose to add the same amount of noise during inference. On image classification tasks, the paper shows increased accuracy when adding the proper amount of noise. ", "summary_of_the_review": "I think the paper needs significant improvement on writing and technical results. I do not think this draft can be accepted, but my decision might change based on authors' reply and revised version.", "strength_and_weaknesses": "Strengths:\n\n- The paper has a good overview of literature. \n- Adding noise to EBM seems to be a novel approach.\n- Experiments show huge improvement on classification results. \n\nWeaknesses:\n\n- The title is too big and does not match the paper. Noise is added in both training and inference. Also, since the paper focuses on EBM applied to classification tasks, the title must reflect that. \n- The presentation of methodology is confusing. \n  - It would be nice to have a detailed paragraph on how to mathematically perform classification with EBM (or ten AEs for these classes). I notice that you perform $\\arg\\min_y E(x|y)$ in the paragraph after eq(10). Is it just intuition or derived from Bayes rule applied to $p(y|x)$? \n  - Def 1 and Prop 1 seems straightforward and do not convey much insights. In practice we do not expect the dropout case to happen. What theory do you have for other types of noises? \n  - Eq (5): why use MSE for energy? Why can you decompose $x$ with independent $x_k$? \n  - Thm 1 should be in methodology section rather than experiment section. A theorem concluding empirical findings is much weaker than one that predicts. In addition, this theorem seems straightforward and I do not find anything interesting. \n- The experiments do not show interesting results.\n  - How is the task different from standard classification tasks? If not, why are we interested in this approach, instead of just using standard methods? \n  - What are some interesting applications of the proposed method? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "_79b-zjwkmD", "reviewer": "Reviewer_Zh4D", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper generalizes and studies activation noise for both training and inference for the specific case of energy-based models. Core contributions include 1) using activation noise helping at inference time (typically dropout is not used at inference time) and 2) through study of which pairs of distributions of noise works best at training and inference time ", "summary_of_the_review": "While the paper presents interesting observations using thorough experiments, the practical significance of these results is not apparent.", "strength_and_weaknesses": "Strengths\n- This paper generalizes dropout to activation noise and studies it thoroughly using controlled set of experiments\n- Using activation noise at inference time seems novel (I have not seen it before) and likely generally useful\n- The paper also presents an interesting negative result that activation noise is not effective for discriminative modeling.\n\nWeakness\n- Caption for Table 1 can be made more descriptive so that the reader can look at it in a self-contained way.\n- While the phenomena is novel, the most impressive results are obtained using 10^4 samples on all the datasets. I'm unsure as a reader if this has practical benefit even on a moderately larger models than the paper considers. I think at least a discussion how the insights can be used would make the paper stronger.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R9a97ywzovX", "reviewer": "Reviewer_BpSj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the effect of noise injection in the form of additive noise during training and inference of a specific class of neural network models energy based generative models, which are roughly variational encoders whose output is interpreted as energy: smaller energy corresponds to the correct class prediction. Several claims about properties of additive noise are made and multiple experimental results are provided. In particular, it is claimed that additive noise is equivalent to dropout and data augmentation, also that additive noise during inference improves accuracy. Experiments compare multiple different setups of the proposed noisy model on different datasets.", "summary_of_the_review": "The paper investigates an interesting matter of additive noise injection during training and inference. The problem is considered in a particular setup of energy based models. The theoretical claims made in the paper are not significantly novel and some are incorrect. The practical results are presenting an extensive evaluation of the different setups of the model, without any benchmarks or comparisons.\nIn the current state the paper is not ready for publication. \n\n----\nI thank the authors for the reply. Nevertheless, I believe that in its current form the paper has flaws that prevent it from being published. Therefore I stick to my current score.", "strength_and_weaknesses": "The paper demonstrates results of a very large amount of experiments, performed for different setups and with standard deviation of the run results. Unfortunately, there are no comparisons to any other regularizations and no reports of state-of-the-art results in the field, so it is very hard to judge how beneficial the proposed scheme is.\n\nThe paper is greatly over-claiming the obtained results.\nThe proposed proof of the equivalency of dropout to additive noise is based on the introduced in the paper negate random variable, which is supposed to follow the distribution of activations but being negative of the value. First, I am not sure that constructing such a variable possible in general. Second, this does not prove equivalence to binary dropout, because a mask is sampled anew every time - this will mean that additive noise has to change distribution on every step to correspond to the new mask. Moreover, dropout can be not only binary, but also continuous. \nThe announced in the abstract 200% improvement of the accuracy is seen only in the particular setup considered. It is not specified if the initial model (without noise) has been fine-tuned to perform best on the problem. In such setup adding regularization can obviously lead to very large positive changes, not confirming that particularly additive noise can improve a model twice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QqPb5e8Nw1", "reviewer": "Reviewer_qks6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors investigate the effect of activation noise (additive noise applied after the activation function) on an EBM based classification approach.\nThey mathematically describe activation noise during training and inference and come to the conclusion that (i) activation noise can be understood as a generalisation of dropout and (ii) that the loss at inference is minimised when the distribution of noise is used during training and inference.\nIn their experiments, the  authors demonstrate the latter point (ii) empirically and compare the effect of different activation noise distributions on classification accuracy.\nThey find that symmetric noise distributions are to be preferred.\n\n ", "summary_of_the_review": "All-in-all, I think this is an interesting paper on an important topic.\nHowever, there are parts of the theory that I cannot follow.\nI believe this has to be clarified to be ready for publication.", "strength_and_weaknesses": "**Strengths:**\n* I believe the topic is important and the community can benefit from papers taking a theoretical look at regularisation methods like this.\n* The paper is well written and included some interesting insights.\n* I appreciate the experiments done by the authors, especially that the benefits of noise during inference are investigated.\n\n\n**Weaknesses:**\n* Unfortunately, some of the theory is not completely clear to me and I am not sure I agree. See below.\n* The authors say that that dropout is a special case, but they do not include it in their experiments. This is unfortunate, because dropout is so widely used\n* I think the authors miss a point about the connection between dropout and activation noise.\n\n**Detailed comments:**\nMost importantly, I think the formulation in Eq. 4 might be problematic.\nThe authors give their objective for jointly optimise the network parameters theta and noise distribution q(z) to maximise the conditional log likelihood of their training data drawn form the distribution p_D(x,y). \nHowever, their model p_theta(x|y,z) is conditioned on the noise z instead of integrating over it.\nThis objective essentially tries to make p_theta(x|y,z) approximate p_D(x|y) (which could be derived from p_D(x,y)) for each z sampled form q(z). \nConsidering now that p_D(x|y) does not depend on z, the optimal solution for this objective would be to make q(z) infinitley concentrated on 0, as z cannot contribute to better approximate p_D(x|y), right?\n\nLater in Eq. 9, when the authors discuss inference, they use a different formulation, integrating the energy over the noise and then choosing the class with the lowest integrated energy. Here, the noise becomes part of the model.\nI think this approach is more sensible, but it is in conflict with Eq. 4, where we are integrating the log probabilities.\nThis is different, because of the partition function, right?\n\nRegarding viewing dropout as special case of activation noise, I agree with this perspective.\nHowever, in contrast to the perspective taken by the authors, the activation noise distribution that would result in dropout behaviour, would have to depend on the input x, as it has to (with a certain probability) exactly cancel out the activation of the neuron.\nThe types of activation noise considered in the paper generally don't depend on x.\nDoes the finding that we should use noise during inference hold for dropout?\n\n\n\n\n\n \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject