Simple and Effective Stochastic Neural Networks
Authors: Tianyuan Yu
Yongxin Yang
Da Li
Timothy Hospedales
Tao Xiang
Venue:  (2020)
Keywords: stochastic neural networks
pruning
adversarial defence
label noise

Abstract: Stochastic neural networks (SNNs) are currently topical, with several paradigms being actively investigated including dropout, Bayesian neural networks, variational information bottleneck (VIB) and noise regularized learning. These neural network variants impact several major considerations, including generalization, network compression, and robustness against adversarial attack and label noise. However, many existing networks are complicated and expensive to train, and/or only address one or two of these practical considerations. In this paper we propose a simple and effective stochastic neural network (SE-SNN) architecture for discriminative learning by directly modeling activation uncertainty and encouraging high activation variability. Compared to existing SNNs, our SE-SNN is simpler to implement and faster to train, and produces state of the art results on network compression by pruning,  adversarial defense and learning with label noise.

Under review as a conference paper at ICLR 2020
SIMPLE AND EFFECTIVE STOCHASTIC NEURAL NET- WORKS
Anonymous authors Paper under double-bling review
ABSTRACT
Stochastic neural networks (SNNs) are currently topical, with several paradigms being actively investigated including dropout, Bayesian neural networks, varia- tional information bottleneck (VIB) and noise regularized learning. These neural network variants impact several major considerations, including generalization, network compression, and robustness against adversarial attack and label noise. However, many existing networks are complicated and expensive to train, and/or only address one two of these practical considerations. In this paper propose a simple and effective stochastic neural network (SE-SNN) architecture for dis- criminative learning by directly modeling activation uncertainty and encouraging high activation variability. Compared to existing SNNs, our SE-SNN is simpler to implement and faster to train, and produces state of the art results network compression by pruning, adversarial defense and learning with label noise.
INTRODUCTION
Stochastic neural networks (SNNs) have a long history. Recently various stochastic neural network instantiations have been topical in their applications reducing overfitting (Gal & Ghahramani (016)]
Neelakantan et al. 2015) and training data requirements (Garnelo et al. 2018), providing confidence estimates predictions (Gal & Ghahramani] 2016), enabling network compression (Dai et al., 22018)
improving robustness to adversarial attack (Alemi et al. (0117) improving optimization (Neelakantan
al., 2015), generative modeling (Kingma & Welling 2014). and inputting or producing probability
distributions (de Bie et al. 2019 Frogner et al. 2019).
One of the most theoretically appealing stochastic neural network formulations Bayesian neural networks, which place a prior distribution on the weights of the network (Graves [201] Blundell et al., 2015 et al. 2018). However this usually necessitates more complex learning and inference procedures that rely on variational approximations or sampling. Another group of works instead focus on modeling the uncertainty in neural network activations. Notably the variational information bottle- neck (VIB) approach (Alemi et al. 2017) is motivated by information theoretic considerations (Tishby ct al. 1999) to learn a hidden representation that carries maximum information about the output and minimum information about the input. Evaluating the required mutual information terms requires modeling probability distributions over activations rather than weights. Deep VIB (Alemi et al.. 2017) leads improved generalization, adversarial robustness and model compression algorithms Dai et al. 2018). Furthermore, modeling stochastic activations through noise is often practically useful for improving exploration and local minima escape during optimization, generalization and adversarial robustness (You et al. 20188] Noh et al. 2017 Bishop 1995, Gulcehre et al., 2016), and in some cases can be linked back Bayesian models of weights (Noh et al.[ Gal & Ghahramani 2016) when the noise added at each activation can be considered as a result of different samples from the weight posterior.
In this paper we propose simple and effective stochastic neural network (SE-SNN) that models activation uncertainty through predicting a Gaussian mean and variance at each layer, which is then sampled during the forward pass. This is similar to the strategy used to model activation distributions in VAE (Kingma Welling 2014) and VIB (Alemi et al., 2017). Differently, we then place a non-informative prior activation distribution and derive activation regularizer that directly encourages high activation variability via preferring high-entropy activations. In conjunction with a discriminative learning loss this means that the network is optimized for activation patterns that have

---

Under review as a conference paper at ICLR 2020
Layer
Layer
fucha-23
(P(h
hQ-1)
("(h--1))
(*
Figure I: An illustration of the stochastic learning module in a SE-SNN. The output of layer I is sampled from the learned distribution defined by f (1(=") and / (1"
high uncertainty while simultaneously being predictive of the target variable. The interplay between these two objectives leads to several appealing capabilities in pruning, adversarial defense and learning with label noise. Pruning: Optimizing for high per-activation variabilitylunceraainyy and predictive accuracy simultaneously lead to the network packing more entropy into the least significant neurons so that the most crucial neurons are free to operate unperturbed. This leads to a simple pruning criterion based each neuron's 's entropy value. Adversarial defense: By optimizing for both per- activation uncertainty and the network's ' predictive accuracy. a representation--ee policy trained that perturbs the internal features during training for increased robustness (Alemi et al. 2017 You et al. 2018). Label noise: With SE-SNN, per-activation uncertainty can be easily aggregated to produce per-instance uncertainty. By optimizing for per-instance uncertainty and predictive accuracy, the network allocates the uncertainty to spread the representation and prediction of the hard-to-classify instances as downweight their influence on parameter learning. The result is a model robust to label noise well outlying training samples.
data augmentation
To summarize, our contributions are: (1) A new simple yet effective stochastic neural network formulation. (2) We show that our SE-SNN has connections to VIB (Alemi et al.] (2017), Dropout (Srivastava et al. 2014) and non-informative activation priors while being simpler to implement and faster to train, as well as impactful on variety of practical problems. (3) Comprehensive evaluations show excellent performance on pruning-bac model compression, adversarial defense, and label noise robust learning.
METHODOLOGY
Stochastic Layers We consider a neural network discriminatively trained for predictive task such as object recognition. Instead computing fixed point estimates of feature vectors, we propose to use stochastic neurons. More specifically, for an input h, a layer will output a series of univariate distributions. By sampling from those distributions independently, we get random output 2. Finally we apply the non-linear activation function 16(-) to and get the input for the next layer. In this study, we choose to use Gaussian distribution with parameterized mean and variance, which has been popularized by VAE (Kingma & Welling 2014) and VIB (Alemi et al. (207)) due to the ease of reparameterization. Formally, for the 1-th layer, this forward-pin wwaaddppass process can be written as (omitting neuron index for notation simplicity).
1 = (1-
( = (1 20 N (1 ,
0000 ())
(I)
This process is illustrated Figure 1 where each standard deviation predictor fo comes with softplus activation f(x) logg1 + exp(x)) to ensure negativiiy.
Supervised Learning Loss We can choose to replace some or all intermediate layers a vanilla neural network with such stochastic layers. For the final layer (i.e.. the classifier) we opt for standard linear layer, and the classification loss is the same as that of a vanilla neural network, e.g., cross- Since Gaussian distribution is fully reparameterizable, the network can be trained end-to-end as long as the sampling process 2 ~ NNp, σ) is realized by z = f + e o where (O, 1).
2

---

Under review as a conference paper at ICLR 2020
Max-entropy Regularization We place non-informative prior the produced Gaussian (de- noted as ((11101) The non-informative prior is a Gaussian with arbitrary mean (f1) and infinite variance (. This reflects the prior that none of neurons is meaningful for predictive purposes. The non-informative prior leads to regularization term that minimizes the KI. divergence of the produced Gaussian (N(12, and the infinite-variance Gaussian
min lim KT(N(HI, (1)/ (122, 02)));
Y/12 E IR
p1.0 02-200
lim (log 22 + of + (11 - -22)² 720
1
min
-)
1/12 E IR
11.01
203
lim (log - )) = min (1
min
(2)
Eq. 2 suggests that we simply need to maximize the predicted standard deviation, or equivalently the entropy of the predicted Gaussian. Thus we call it a max-entrupy regularizer Q. can be easily used in any existing neural network architecture:
min - (x(())
0
where or (h100 denotes the predicted standard deviation of hidden unit h given the neuron uncertainty prediction parameter 0. For numerical safety, we introduce a margin the loss,
min (b (o]((((())))
This means that the regularization does not punish the model as long as the entropy is larger than a threshold b. Note that, this loss design is not absolutely necessary as the increment for a shrinks (since the gradient is - ) during training, and thus never reaches infinity with finite number of updates. However, one can think of it as an early-stopping mechanism for this regularization term.
So far we have introduced the stochasticity to the smallest unit of a network, i.e., single neuron. To compute the value of the regularizer over mini-batch consisting of N training samples, we need to aggregate the entropy of multiple neurons and set the margin b on the aggregation. How to aggregate exactly is task-dependent, and we next provide some suggestions for three tasks including network pruning, adversarial defense, and label noise defense.
Pruning For network pruning, we aggregate entropy over samples for each neuron, and then penalize if that neuron's aggregated entropy is low. To this end, the regularizer formulated as:
K
2(())
= K
b
N
Σ log(o,.))) 1=1
(5)
j=1
where i = 1... N indexes the training samples, and jndj 1 channels). This regularizer aims make neurons very stochastic, to the point of compromising their reliability for computing a supervised learning task. Thus only those neurons that are most useful for the task get their entropy lowered and thus pay the regularization cost. Less useful neurons get their entropy maximized, allowing them to be detected and pruned after training.
K neurons in a layer (i.e., feature
Label Noise Different from pruning, we aim identify uncertain samples. Therefore, we aggregate entropy over neurons for each sample, and prefer high sample-wise entropy, leading to
N
K
2(0)) N b
Σ K
1
Σ
log log(o:)))+
With this regularizer, inlier samples pay the entropy cost in order to produce clean feature for classification to satisfy the supervised learning loss. Outlier samples, caused by either label noise or being out- distof-distriuution, are anyway hard to classify; the regularizer naturally inflates their entropy since doing so does not impact the supervised learning loss. This high-variance representation in turn reduces their (negative) impact on network fitting.
Adversarial Defense To defend against adversarial samples, we aim to inflate entropy over both the neuron- and sample-axes produce highly stochastic model:
1 200) N
Σ
Σ (6 (0-lo((o.)))
K
3

---

Under review as a conference paper at ICLR 2020
This looks similar to VIB's adversarial defense (Alemi et al. 2017). J. but with the vital difference that our entropy regularizer is not restricted to the (0, 1) prior used in VIB. It can be seen as learning a layer-wise data augmentation policy, which turns out to be very useful for adversarial defense in practice (see Sec.
3
RELATED WORK
Connection to VIB and Sparse VD Though we derive the max-entropy regularizer from the perspective of a non-informative activation prior, our work is closely related VIB (Alemi et al. (017) and sparse variational dropout (VD) (Molchanov et al. 2017a), despite their different perspectives. Specifically, if we replace our infinitevvariane Gaussian with a standard Gaussian, it becomes VIB (see Eq. 17 in (Alemi al. (017)). The max-entropy regularizer is also linked to Eq. 14 in Sparse VD (Molchanov et al. 7a). which also encourages large variance/entropy (at a different rate). But again, Sparse VD (Molchanov et al. (2017a) is derived with completely different motivation: It has an intuitive explanation that the regularizer corresponds to sparsity prior on the weights. We note that enforcing uncertainty on activations rather than weights has number of advantages: (i) The weight prior is intractable analytically, which leads to the fact that Sparse VD regularization is itself an approximation.
(ii) Deriving the regularizer from a weight prior is unnecessarily complicated for the purpose of sparsifying the model compared to ours. In contrast, our approach sidesteps the need to sample weights and avoids keeping multiple copies of the network, which can potentially improve efficiency (e.g., in memory usage).
Note that stochastic layers have been used in several other works in order to achieve better classifi- cation regression accuracy. KKngmma al. (2015) proposes generalization of Gaussian dropout where the dropout rates are learned, leading to higher classification accuracy. Natural-pant Natural-parameter networks (NPN) (Wang et al.] 2016) is class of probabilistic neural networks where the input, target output, weights, and neurons can all be modeled by arbitrary Poisson distributions for word counts) instead of being limited to Gaussian distributions, achieving late-off-the-ar performance on classification, regression, and representation learning tasks. To reduce computational cost, Postels et al. (2019) approximates uncertainty estimates using sampling-free approach and obtains better results on classification and regression tasks.
pioonential-famill distributions (e.g..
Network Compression Network compression based on pruning typically uses heuristics based on pruning low-importance weights or low-activation neurons (Molchanov et al. 201 7b) Wen et al. 2016), often assisted by sparsity-enhancing priors such as lasso (Wen et al.J 2016). We avoid the complication of Bayesian learning of weights by proposing a simpler and direct activation prior that predisposes neurons towards deactivation unless necessary to solve the supervised task.
Adversarial Defense Our method is related to existing randomization based methods for adver- sarial defense (Liu al., 2018 Xie et al., 2018 Alemi al. 2017: Ye & Zhu| [0088: Liu al.)
2019). However unlike these studies which use a fixed distribution for noise (Liu et al. [0188] Alemi
eet al. 2017). learned model distribution for effective randomization (Liu et al. (219)) image
perturbations (Xie et al.) 2018) or a learned adversarial data-generating distribution (Ye & Zhu 2018).
our randomization- based defense is both learned, and data-dependent since the variance at each layer is generated based on the output of the previous layer.
Label Noise Robustness A number of existing label noise-robust deep learning approaches require a subset of noisy data to be reliably re-annotated (cleaned) to verify which samples contain noise (Lee) al. 2017 Jiang et al., 2018). In contrast, some others including SE-SNN do not rely on additional human noise annotation. These methods address label noise by either iterative label correction via bootstrapping (Reed et al. 2015), adding additional layers on top of a softmax classification layer to estimate the noise pattern (Sukhbaatar et al.) 2015; Goldberger & Ren-Reuven] 2017), or loss correlation (Patrini et al., 2017). By allocating large uncertainty to outlying samples, SE-SNN can handle both label noise and oft-of-distributin noise robustness is appealingly simple in that requires neither explicit detection of noisy samples, nor additional annotation. However label noise robustness has been largely ignored by existing SNNs.
samples with correct labels. This approach to label
4

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
Experiments are carried out to evaluate the efficacy of the proposed framework in three applications: neural network pruning, adversarial attack defense and learning with label noise.
4.1 NEURAL NETWORK PRUNING
Competitors We follow the architectureeddatsse combinations used in most recent neural network pruning studies, including LeNet-5-Caffe network network on MNIST (LeCun 1998). 1GG-16 (Simonyan & Zisserman (2055) on CIFAR10 (Krizhevsky & Hinton 2009) and variant of VGG-16 on
CIFARTOO. Under these settings, the proposed method is compared with the following contemporary state-of-the-art methods including Generalized Dropout (GD) (Srinivas & Babu 2016). Group
Lasso (GL) (Wen et al. 2016), Sparse Variational Dropout (VD) (Molchanov et al. 2017a), Structured
Bayesian Pruning (SBP) (Neklyudov al., 2017), Bayesian Compression with Group Normal
Jeffreys Prior (BC-GNJ) and Group Horseshoe Prior (BC-GHS) (Louizos et al. (217)) Sparse 10
Regularization (LO) and LO with separate À for each layer (LO-sep) (Louizos et al. 2018), Variational
Information Bottleneck (VIBNet) (Dai et al. 2018), and Network Slimming (NS) (Liu et al. 2017).
Evaluation Metrics Following the majority of the existing evaluations, monitor the test error
while focusing on the following three compression/complexii
metrics: (a) Model size: The ratio of
nonzero weights in the compressed network versus the original model. (b) FLOPs: The number of
floating point operations required to predict a label from an input image during tes 2 (c) Run-time
memory footprint: The ratio of the space for storing hidden feature maps during run-time in the pruned network versus the original network.
Training While many existing studies remove redundant weights, remove redundant neurons during compression. Therefore, we can use the pipeline proposed in MMochanov et al.] (2017b). Specifically,, after training the neural network, an initial batch pruning stage followed by a loop of removing the least important neuron and fine-tuning, Since SE-SNN is designed to discount unimportant neurons through inflating their pre-activation variance, we find that the network achieves this by simultaneously assigning high- redundant neurons never activate their RELU non-linearity. In the initial batch pruning stage, neuron inactivity thus provides single-step pruning criterion before the iterative pruning begins (and one that is guaranteed not to affect the test accuracy since these neurons propagate no information). The pruning then enters the second stage where the least important neuron removal fine-tuning loop continues until reaching the target trade-off between accuracy and model compression objectives. Since one neuron/channel is removed each iteration, the accuracy never drops sharply. We set the uncertainty loss/regularizer weighting factor and margin (Eq. as 0.0001 and 4, respectively.
and negative mean. As a result, large portion of
Testing During testing, only the mean of the learned distribution is passed between layers. So there are no additional parameters and inference cost compared to the network's s deterministic counterpart. The distribution generation branches only used during training identify redundant neurons.
Results on MNIST The most commonly used benchmark and architecture is MNIST with LeNet- 5-Caffe. We follow the standard training and testing protocols. The results shown in Table T It is clear that SE-SNN achieves the best performance on FLOPs, run-time memory footprint, and test error. In terms of model size, model is only comparable with the state of art. This is because it does not prune the linear layer as much as other methods. Instead, it focuses on pruning the convolutional filters, hence the excellent performance on FLOPSS and memory footprint.
Results on CIFAR10 For CIFARIO, several VGG16 variants and training protocols have been proposed in different works. We use the standard VGG16 architecture but change the dimension of linear layers from 4096 to 512, as in (Louizos et al. 2017 Dai et al. 2018). The results in Table compare our method with (Louizos al.) 2017: Dai et al. 2018). The error rate for VIBNet in parentheses was obtained by further fine-tuning the pruned architecture. Our model achieves the best performance across all the evaluation metrics and error rates.
hhtps://github..o com Wt.m/BVIC/ccafiiie w.w.effirceemsttteeeaamp////mmaa Following the setting in (Dai et al. 2018). we count each multiplication as single FLOPP and exclude additions, which is also consistent with most prior work
5

---

Under review as a conference paper at ICLR 2020
Methods
Model size (%)
HLOPs (Mil)
Memory (%)
Error (%)
GD (Srinivas & Babu
2016
1.38
0.250
32.00
1.1
GL Wen et al. 2016
23.69
0.201
19.35
1.0
VD Molchanov al.
2017a
9.29
0.660
60,78
1.0
SBP (Neklyudov et al.
2017
19.66
0.213
21.15
0.9
BC-GNJ (Louizos et al.
2017
0.95
0.283
35.03
1.0
BC-GHS (ouizos ct al.
2017
0.64
0.153
22,80
1.0
L. (Louizos et 2018
8.92
1.113
85.82
0.9
LO-sep (Louizos et al. 2018)
1.08
0.389
40.36
1.0
VIBNet (Dai et al. 2018)
0.83
0.094
15.55
1.0
SE-SNN
2.35
0.061
11.08
0.9
Table 1: Compression results on MNIST using LeNet-5-Caffe.
Methods
Model size (%)
FLOPs (Mil)
Memory (%)
Error (%)
BC-GNJ (Louizos et al. 2017)
6.57
141.50
81.68
8.6
BC-GHS (ounzoos et al. 2017)
5.40
121.90
74.82
9.0
VIBNet (Dai et al. 2018
5.30
70.63
49.57
8.8(8.5)
SE-SNN
2.57
53.61
49.41
8.0
Table 2: Compression results on CIFAR10 using VGG16.
Results on CIFAR100 We compare with the study in (Liu et al. 2017). which uses the same VGG16 variant that replaces two fully connected layers with three convolutional layers. This architecture improves accuracy at the expense of FLOPs and memory. The results in Table that our model produces the best compression result while maintaining comparable accuracy. Note that the 26.2% error achieved by our model is identical that of the original network. So if the accuracy drop is used as a compression metric, our model is as good as any competitor.
4.2 ADVERSARIAL DEFENSE
Experimental Setting Following Alemi et al. (2017), we focus on two types of adversarial attacks: Fast Gradient Sign (FGS) Goodiellow et al. 2015) and an optimization-based attack
CW-L2 (Carlini& & Wagner 2017). We evaluate untargeted FGS attacks with attack magnitude c ranging from 0.0 to 0.5 and untargeted CW-L2 attack on models trained on MNIST. We use popular architecture including three FC layers with 1024, 1024 and 256 output neurons respectively. The third stochastic layer. We set the uncertainty loss/regularizer weighting factor
FC layer is implemented and margin as 0.1 and 16, respectively. Results averaged over 20 runs are reported. We compare against the original (undefended) network, termed 'Bascline', Deep VIB (Alemi et al., 2017) using the variational information bottleneck, Bayesian Adversarial Learning (BAL) (Ye putting distribution
(Ye
Zhu 2018
the adversarial data-generating process and Adv-BNN (Liu et al.
2019
learning a BNN to incorporate the effective randomness and using adversarial training to seek the best model distribution. Since our model stochastic, follow the best practice recommended in (Athalye et al. 2018) for attacking stochastic models: We compute the expected gradient over multiple stochastic samples for each input when constructing attacks. This because using the expected gradient over multiple posterior samples produces better gradient estimator for the attacker, allowing it to generate samples that are much harder to defend against (Athalye al. 2018). For the FGS attack, we evaluate two settings, namely normal training and adversarial training, the latter of which generates and uses FGS attack samples during training increase adversarial robustness.
Results From the comparative results shown in Figure 2al we can see that our SE-SNN outperforms the competing defense methods over range of FGS attack strengths when trained with adversarial attack samples (Figure 2a[R) or with only normal samples (Figure (a The advantage of SE-SNN particularly pronounced when the attack magnitude is large. We can also see from Figure 2b that under the stronger CW attacks the Baseline now fails completely. In this case SE-SNN provides the most effective defense. It is worth pointing out that, unlike BAL (Ye & Zhu 2018), which learns their models with explicit adversarial sampling, our SE-SNN can also work with training with only normal samples Figure 2 suggests that our SE-SNN beats BAL even without being trained with adversarial attack samples (comparing SE-SNN in Figure 2 BAL in Figure 2a R)).

---

Under review as a conference paper at ICLR 2020
Methods
Model size (%)
FLOPs (Mil)
Memory (%)
Error (%)
NS-Single (Lu et al. 2017)
24.90
250.50
26.5
NS-Best (Liu et al. 2017)
20.80
214.80
26.0
VIBNet (Dar al. 2018)
15.08
203.10
73.80
25.9(25.7)
SE-SNN
14.93
181.31
70.16
26.2
Table 3: Compression results on CIFAR100 using a VGG16 variant.
1.00
1.00
0.75
0.75
0.4
Baseline
0.50
Baseline
0.50
Adv-ONN
Adv
0.2
0.25
Deep VIB
0.25
Deep VIB
0.00
0.0
0.1
0.2
0.3
0.4
0.5
0.00
0.0
0.1
02
0.3
0.4
0.5
0.0
Baseline
BAL
Deep VIB SE SNN
Untargeted CW Attack
(a) Untargeted FGS (L: normal training, R: adversarial training)
(b) Untargeted CW-L2
Figure 2: Adversarial defense accuracy (mean standard deviation) under untargeted FGS and CW
attacks on MNIST.
4.3 ROBUSTNESS AGAINST LABEL NOISE
In this section, tasks including digit recognition and person re-identification (ReID) are considered. Compared to Digit recognition, RelD is much more challenging. Particularly, ReID is an instance recognition task, which aims match people under disjoint camera views. Although the performance of trate-of-the-at RelD models on public benchmarks approaches saturation, RelD with label noise remains an unsolved and under-studied problem.
Datasets The label noise robustness of our SE-SNN is evaluated for digit recognition using MNIST and ReID using Market-1501 (Zheng et al. 2015) benchmark. Market1501 collected by cameras and contains 751 training identities (12,936 images) and 750 test identities (19,281 images). The test set is organized into query set and gallery set. Cumulative Matching Characteristics (CMC) ranks and mean Average Precision are used as the performance measure.
Noise Generation We consider patterned label noise which more common in practice. Specif- ically, for MNIST, one class label will be flipped to a different one at strength p. The flipping pattern is fixed for each run but varies across different runs following (Hendrycks et al. 2018)) For Market- 1501, we use the commonly used ResNet-50 (He et al. (016) trained on the clean data to obtain the feature of each training sample and find the most visually similar samples using feature Euclidean distance. Then for randomly selected training samples, their identity labels assigned to that of the most similar sample that has different identity, imitating human annotation errors.
Competitors Baseline: The main baseline is the original network without adding our stochastic layers. For a fair comparison, we add a parallel layer to the penultimate (feature) layer in the baseline to guarantee the two networks have the same number of parameters. The input of the added layer the same as the feature layer, and its output is element-wise added along with the output of the feature layer to get the final feature, which is then fed fully connected layer for computing the classification loss. Bootstrup_ hard and Bootstrap_sol (Reed et al.) 2015):: Both iteratively use the model-predicted labels to refine the original labels that potentially corrupted by noise. They differ in whether the updated label is binary or continuous. Forward Correction (Patrini et al. 2017]: This model predicts the label corruption matrix (capturing the label flipping pattern) by first training a classifier on the noisy labels followed by corruption matrix estimation using the resulting softmax probabilities. The estimated matrix then employed to regularize the retraining of the model. For MNIST, we stick to the original setting, which uses the ar argmaz at the 97th percentile of softmax probabilities for label noise detection. For the ReID experiment, we replace this with the argmaæ over all softmax probabilities for given class following what Patrini et al.) (2017) did on datasets of more classes such as CIFAR100. CleanNer (Lee et al., 2017): different from other models, this requires a subset of noisy training samples to be e-nontttte by more reliable source (i.e., cleaned). then learns the similarity between class and query-embedding vectors, which is further used to detect noisy samples for sample pruning. Having cleaned subset gives CleanNet
7

---

Under review as a conference paper at ICLR 2020
1,0
Accuncy
Beseline
6000
Clean data
Noisy data
Bootstrap
Bootstrap soft
0.6
Forward
SE-SNN
4500
Forwardd
0.4
0.0
0.2
0.4
4000
20%
Corruption Strength
Noise percentage
(a) Results on MNIST
(h) Variance comparison Market (c) Samples with extreme variance
Figure 3: Label noise results on MINIST (a) and ablation study results on Market- 1501 (b,c), (c), the top row shows samples with the highest variance and bottom row those with the lowest when there is no label noise. The high variance uncertain samples correspond to poor detections and occlusions.
Strengths Models
10%
20%
50%
mAP
Rank
Rank5
Rank10
mAP
Rankl
Rank5
Rank
mAP
Rankl
Rank5
Rank
25.87
51.46
70.21
76.81
23.49
48
67.85
74.67
20.74
44.04
64.32
72.27
H
25.76
51.08
70.11
77.06
23.40
48.25
67.34
74.40
19.87
42.87
63.37
71.40
S 25.50
50.47
69.43
76.32
24.08
49.51
68.77
75.62
20.72
43.86
64.49
72.55
C 26.64
52.47
70.90
77.41
24.28
49.54
68.35
75.33
19.90
43.01
63.13
71.34
F 21.17
45.88
64.79
71.69
18.91
42.11
62.12
15.34
36.06
56.23
64.55
SE-SNN
27.30
53.21
71.03
77.38
24.72
49.87
68.90
76.10
21.32
44.86
65.03
72.93
Table 4: Re-ID results Market- 1501 with label-noise. Model abbreviations: B: ResNet-Baseline H: Bootstrap _ard (Reed et al.
2015), S: Bootstrap_son (Reed et al. 2015). C: CleanNet (Lee et al.
2017), F: Forward Correction Patrini et al. (017)).
an unfair advantage over other compared models. The model requires at least 5 FC layers so cannot be implemented for the MNIST backbone. For person ReID, 10% of the training set without noise is used as a clean reference set to train CleanNet using the author-provided code. After training, 20% of the whole training set deemed most likely to be noisy are removed before the final RelD model is trained on the remainder. Note that unlike the four competitors, our SE-SNN does not have any additional steps refine the label prune the training samples. As explained earlier, works by discountin//neutraliingg the negative influence of noisy samples (hence also works on out-o--disttibution samples with correct labels). Having said that, can be easily combined with a label refinement procedure such as the corruption matrix based one in Forward Correction (Patrini ct al. 2017).
Results
The results on MNIST are shown in Figure 3a We can see that even without any label correction/refinemmnt procedure, our SE-SNN is already very competitive
only beaten by Forward Correction when the corruption strength (percentage of samples with label noise) is large (>0.2). Once our SE-SNN combined with the same procedure adopted by Forward Correction (SE SE-SNNNFForward) (Table 4. our SE-SNN also achieves the best overall performance even beating CleanNet which needs an additional cleaned training subset. Note that the strongest competitor on MNIST, Forward Correction now is the weakest. This is because, as admitted in (Patrini et al.) (017). it struggles with estimating an accurate correction matrix given a small number of training samples per class (c.g.. 500 per class in CIFAR 100). The < 20 class size in Market-1501 means that the label correction procedure only has detrimental effect. Figure 33b compares the average variance/uncertainng inferred for clean and noisy data in Market-1500 than that of clean data on average. We also notice that even when there is no label noise, SE-SNN beats the baseline by a clear margin ( (70.20% mAp vs. 67.66%) ). This suggests that the mechanism of neutralizing outlying samples is still in play when the labels are clean. To validate this, we show some examples of outliers (those with the largest variance) in Figure 3e when there is no label noise. is clear that these are mostly caused by either poor person detection or occlusion (i.e., outliers). In contrast, images with the smallest variance mostly contain people detected perfectly and without occlusion the model is thus most confident about the feature representations for these.
achieve the best result. On the more challenging ReID task in Market-1500
We can see that, indeed the variance of noisy data is larger
More SE-SNN implementation details for the label noise robust learning and more results can be found in the Appendix A. Further analyses across all tasks including hyperparameter analysis, solving multiple tasks in one model, effects of the number of stochastic layers are given in Appendix B.
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
We proposed a simple and effective stochastic neural network framework. Our model is related to VIB and variational dropout, but provides simpler and more direct realization via neuron regularization by non-informative activation prior. Our extensive experiments show that this simple framework has diverse benefits for network pruning, adversarial defense and label noise robust learning.
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In ICLR, 2017.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018.
Chris M. Bishop. Training with noise equivalent to tikhonov regularization. Neural Computation, 7(1):108-116, 1995.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In ICML, 2015.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEESSP, 2017.
Bin Dai, Chen Zhu, Baining Guo, and David Wipf. Compressing neural networks using the variational information bottleneck. In /CML, 2018.
Gwendoline de Bie, Gabriel Peyré, and Marco Cuturi. Stochastic deep networks. In ICML, 2019.
Charlie Frogner, Farzaneh Mirzazadeh, and Justin Solomon. Learning embeddings into entropic wasserstein spaces. In ICLR, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout Bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In ICML, 2018.
Jacob Goldberger and Ehud Ben-Reuven. Training deep neural-networks using noise adaptation layer. In ICLR, 2017.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Alex Graves. Practical variational inference for neural networks. In NIPS, 2011.
Caglar Gulcehre, Marcin Moczulski, Misha Denil, and Yoshua Bengio. Noisy activation functions. In ICML, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. NIPS, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks corrupted labels. In ICML, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.

---

Under review as a conference paper at ICLR 2020
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterizzatoon trick. In NIPS, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun. The mnist database of handwritten digits. hutp://yann. lecun. com/exdb/mmist/, 1998.
Kuang-Huei Lee, Xiaodong He, Lei Zhang, and Linjun Yang. Cleannet: Transfer learning for scalable image classifier training with label noise. In ICCV, 2017.
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. ECCV, 2018.
Xuanqing Liu, Yao Li, Chongruo Wu, and Cho-Jui Hsieh. Adv-bnn: Improved adversarial defense through robust bayesian neural network. In ICLR, 2019.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017.
Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning. In NIPS, 2017.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. In ICLR, 2018.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In ICML, 2017a.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In ICLR, 2017b.
Arvind Neelakantan, Luke Vilnis, Quoe V. Le, Lukasz Kaiser, Karol Kurach, Ilya Sutskever, and James Martens. Adding gradient noise improves learning for very deep networks. CoRR, abs/1511 1,,511,0607,,0000
Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov. Structured bayesian pruning via log-normal multiplicative noise. In NIPS, 2017.
Hyeonwoo Noh, Tackgeun You, Jonghwan Mun, and Bohyung Han. Regularizing deep neural networks by noise: Its interpretation and optimization. In NIPS, 2017.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In CVPR, 2017.
Janis Postels, Francesco Ferroni, Huseyin Coskun, Nassir Navab, and Federico Tombari. Sampling- free epistemic uncertainty estimation using approximated variance propagation. In ICCV, 2019.
Scott F. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR Workshop. 2015.
Hippolyt Ritter, Aleksandar Botev, and David Barber. scalable laplace approximation for neural networks. In ICIR, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Suraj Srinivas and R Venkatesh Babu. Generalized dropout. CoRR, abs//611 1.66991 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 15:1929-1958, 2014.
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. In ICLR, 2015.
10

---

Under review as a conference paper at ICLR 2020
Strengths Models
10%
20%
50%
mAP
Rank
Rank5
Rank
mAP
Rank1
Rank5
Rank 10
mAP
Rank1
Rank5
Rank TO
B
0.41
0.83
0.75
0.98
0.56
0.82
0.74
0.62
1.25
2.04
1.48
1.56
0.60
0.76
1.34
1.12
0.70
0.97
0.57
0.77
0.93
1.82
1.55
0.97
0.78
0.93
0.95
0.51
1.11
0.61
C
0.83
1.45
0.98
0.90
0.44
0.53
0,96
0.58
1.08
1.68
1.85
1.66
F
1.03
1.64
1.98
1.53
0.82
1.50
1.12
0.86
0.56
1.48
1.63
1.97
SE-SNN
0.43
1.11
0.88
0.74
0.24
0.68
0.67
0.44
0.94
1.46
1.42
1.11
Table A.1: Standard deviation value of Re-ID results on Market-1501 with label-noise.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. In Allerton, 1999.
Hao Wang, SHI Xingjian, and Dit- Yan Yeung. Natural-parameter networks: A class of probabilistic neural networks. In NIPS, 2016.
Wei Wen, Chumpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In NIPS, 2016.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan L. Yuille. Mitigating adversarial effects through randomization. In ICLR, 2018.
Nanyang Ye and Zhanxing Zhu. Bayesian adversarial learning. In NIPS, 2018.
Zhonghui You, Jinmian Ye, Kunming Li, and Ping Wang. Adversarial noise layer: Regularize neural network by adding noise. CoRR, abs/1 abs/1805.08000 805.08000, 2018.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In ICCV, 2015.
A EXPERIMENTAL DETAILS OF LABEL NOISE
For the experiments MNIST, follow (Patrini al., 2017) train neural network with two FC layers of dimension 128, where both FC layers are implemented as stochastic layers. For Market-1501, following most recent state-of-the-art RelD models we use ResNet-50 (He al. (0166 as the backbone. In this model, only the last unit of the final residual block is implemented as stochastic layer.
For Market-1501, we train our model for two stages: (1) We fine-tune ImageNet pretrained ResNet-50 (for identity classification) on Market- 1501 training set for 60, 000 iterations. We use batch size 32 and Adam optimizer (Kingma & Ba 2015) with learning rate 3.5 x 10-1, (2) We retain the model parameters trained in stage (1) and extend the last unit stochastic layer with randomly initialized layer. Then, we train the stochastic layer and the classifier for another 20, 000 iterations with lower learning rate 5 x 10-4, The margin is set as the max-entropy regularization loss (Eq. 6 and loss weight 10 is assigned.
For both benchmarks, due to the randomness of noisy samples in selection and label reassignment, multiple runs of experiments are conducted. The standard deviation values of Re-ID results Market-1501 with label-noise are shown A.
B FURTHER ANALYSIS
B.1 HYPER PARAMETERS ANALYSIS
In this section, analyze the impact of loss weight, which is denoted and the margin b for regularizers of all tasks on MNIST. We report the FLOPs of the compressed model (Network pruning), adversarial defense accuracy under the FGSM attack (Adversarial defense) and test accuracy of model trained with 40% label noise (Label noise) in Figure [.
In Figure B. top), evaluate the effects of having different values of from [10 10-2, 10-3] 3.
We can see that, for all three values and across the three tasks, the performance our method is only marginally impacted by and the overall best performance is achieved when 10-2
11

---

Under review as a conference paper at ICLR 2020
0.35
10
0.8
Accuracy
0.25
Adv Acc
0.9
0.6
0.15
0.4
6-10-2
w-10
0.8
w=10
0.05
10
10-2
10
0.2
0.0
0.1
0.3
0.4
0.5
0.7
10
10
10
0.35
1.0
b-log(3) b=log(4)
0.8
accuracy
0.25
AdV Ace
b-log(36)
0.9
0.15
0.8
0.2
0.05
10g
1og
16
0.7
109201
10g:
16g::16
Network pruning
(b) Adversarial defense
(c) Label noise
Figure B.1: Impact analysis of loss weight w (top) and margin b (bottom) on MNIST.
2.0
Text Aucurac
0.9
6.8
0.8
1.0
Text
0.5
05L
0.4
Baselow
SE-SAW
SHUnnggning Methoos
0.0
0.2
0.3
0.5
Corruption Strongth
Epocha
Figure B.2: Test accuracy of different methods trained MNIST with label noise.
Figure B.3: Effect of the number of stochastic layers MNIST with label noise.
Figure B.4: Convergence
Curve on MNIST with label noise.
In Figure [B [bottom), we investigate the sensitivity of our model to the values margin Three values, [log( (I), log(4)) log(16)] are considered. Recall that our SE-SNN uses non-informative prior to regularize the uncertainty of neuron activations, and a margin b is employed to upper-bound the standard deviation of the stochastic layers in SE-SNN. Therefore, larger values of will lead to higher degree of uncertainty of stochastic layers. From Figure B.rreeBBBBBbo [bottom) we can see that increasing the value improves the performance of our method on adversarial defense and label noise. However, for network pruning, smaller & is preferred. Overall, our model is insensitive to the value of
B.2 COMBINATION OF NETWORK PRUNING AND LABEL NOISE
The effectiveness of our SE-SNN has been demonstrated the three tasks separately, but there is no reason why it cannot be deployed to deal with multiple tasks simultaneously. To validate this, in this experiment,
train SE-SNN on MNIST with 40% label noise while conducting model pruning. We use the same architecture with three FC layers (784-128-128) 128-10) as the previous label noise experiments, and make the first two layers stochastic layers. We compare the trained model with two competitors, including the baseline model (without stochastic layers) and our normal SE-SNN (without pruning) both trained with label noise.
From the results in Figure B.2 we can see that we get a pruned SE-SNN (145-26-15-10) trained with label noise achieving 95. 20% test accuracy, which even slightly better than our normal SE-SNN (95.17%) and much better than the baseline (81. (81.0%%).
B.3 EFFECT OF THE NUMBER OF STOCHASTIC LAYERS
In this section, we study the effect the number of used stochastic layers (SLs). The experiments are on the label noise robust learning task with network comprising FC layers. For the network
12

---

Under review as a conference paper at ICLR 2020
with SL, we make the penultimate layer stochastic. Note that the network with zero SL is the baseline. Figure [B33reBB.3aar0 shows the test accuracy models with different numbers of SLs used. We can see that increasing the number of SLs helps improve the model's robustness to label noise.
B.4 CONVERGENCE CURVE
The training efficiency of the proposed method is studied here by examining how fast the algorithm converges. The experiments are rum on the label noise robust learning task. Figure [.4 shows the losses of different methods during 10 training epochs. It is observed that Bootstrap and Forward have larger loss values than the rest. This is because they use losses in addition the classification loss. Overall our method is shown to be efficient other methods, despite the fact that we have introduced stochastic layers and used a reparamcterization trick during training.
13


Reviews:
[{"review_id": "rkgLYY6cKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper presents an efficient stochastic neural network architecture by directly modeling activation uncertainty and adding a regularization term to encourage high activation variability by maximizing the entropy of stochastic neurons. Compared with other existing approaches, such as Bayesian neural networks and variational information bottleneck, the proposed architecture is simpler to implement and faster to train. The authors also achieve state of the art results in various fields, including network compression by pruning, adversarial defense and learning with label noise.\n\nMajor comments: \n- Overall, I find the paper is easy to follow and the experimental evaluation shows promising results, but my major concern is about the novelty of this work, given the fact that the structure of the proposed stochastic layers is quite similar to VIBNet.\n- The derivation of the max-entropy term is somewhat unclear and I think the paper needs a major revision on this part. The authors suggest using a Gaussian with a finite mean and an infinite variance as the non-informative prior for the produced Gaussian random variable z (in Eq. 1), and then minimize the KL divergence between the produced Gaussian and the infinite-variance Gaussian. However, this may be questionable from a Bayesian viewpoint, in the sense that the infinite variance leads to an improper prior as the variance increases without bound and thus may produce an improper posterior distribution. This is not discussed and needs to be clarified in Sect. 2 (Max-entropy Regularization).\n- There are things unclear in the derivation (last line of Eq. 2), since log(\\sigma_2) trends to infinity.\n- In addition, the penalty terms for different types of tasks are directly given only based on some of the assumptions that the authors have made, there does not seem to be any theoretical justification for such choices.\n\nMinor comments:\n- Some of the notations used in this paper seem a bit confusing, which may hinder readability. For example, on page 3, in “The non-informative prior is a Gaussian with arbitrary mean (\\mu_1) and infinite variance (\\sigma_1)”, I guess \\sigma means the standard deviation? I would like to recommend using N(\\mu, \\sigma^2) to denote a Gaussian distribution, where \\sigma means the standard deviation and \\sigma^2 the variance.\n- On page 3, in “where \\sigma(h|\\theta) denotes the predicted standard deviation of hidden unit h given the neuron uncertainty prediction parameter \\theta”, there is no discussion on the neuron uncertainty prediction parameter. Does it means the predicted standard deviation is again parameterized by \\theta?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgsg7xnYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed SE-SNN, a type of stochastic neural networks that maximize the entropy in stochastic neurons along with the prediction accuracy. The authors argue that maximizing the entropy operates as a form of regularization to force the entropy into the least significant neurons, and that Increasing the diversity/randomness results in more robust models. Experiments show that SE-SNN outperform several baselines tasks such as network pruning, adversarial defense, and learning with noisy lables.\n\nSeveral closely related references are missing. The idea of producing distributions in each layer (i.e., using stochastic layers) is not new and is closely related to the work on local reparameterization trick and variational dropout [1] (predecessor of the cited sparse variational dropout), and various works that directly model neurons as distribution [2, 3]. \n\nBuilt on top of the reparameterization trick, the idea of maximizing the entropy of neurons to regularize the network is interesting. Such regularization is somewhat similar to sparsity regularization on neurons, which forces the information to concentrate on a small portion of the neurons. \n\nThe numbers for different methods in Table 1 are very close to each other. Without standard deviation it is difficult to evaluate the performance.\n\nNote that in Table 1 the best accuracy is actually achieved by SBP and L0. However, in Table 2 and 3 (experiments on CIFAR-10 and CIFAR-100), these two baselines are missing. Is there a reason why these baselines are excluded?\n\nAlso, looking at the results from Louizos et al., 2018, which is the most recent baseline among those chosen by the authors, they actually use WideResNet rather than VGG. Comparing to VGG, WRN performs significantly better. For example, the compressed network by Louizos et al., 2018 achieves an error rate of 3.83% on CIFAR-10, versus 8% from SE-SNN pruned VGG. \n\nFor the experiments on adversarial defense, the two attacks used seem rather weak (FGS from 2015 and CW-L2 from 2017). It may be the state-of-the-art attack around the time of Alemi et al., 2017, which the author claim to follow, but not now.\n\nMinor:\n\nP2: instead -> instead of\n\n[1] Variational Dropout and the Local Reparameterization Trick, NIPS 2015\n[2] Natural-Parameter Networks: A Class of Probabilistic Neural Networks, NIPS 2016\n[3] Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation, ICCV 2019", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkeJQaCz9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a simple stochastic neural network, which makes each neuron output Gaussian random variables. The model is trained with reparameterization trick. The authors advocates the adoptation of a non-informative prior, and shows that learning with the prior equals with an entropy-maximization regularization term. The paper presents the design of the regularization term for pruning, learning with label noise, and defensing with adversarial examples. The claims are well supported: the model is indeed simple, and the effectiveness is well supported by experimental results. \n\nI however think *efficiency* of the proposed model needs to be studied as well. Since the proposed algorithm is an approximate inference algorithm via reparametrization trick, it is necessary to see how fast does the approximate algorithm converge. The experiments don't report any convergence curve, or performance under limited time budget. I think there need to be some related results.\n\nAnother question is, what is the original inference problem of the designed regularization for pruning, label noise, and adversarial attack, respectively?\n\nUpdate\n====\n\nThanks for the additional experiments! After reading the rebuttal and other reviewer's comments I decide to keep my score. (Though I am less positive due to the concern of novelty raised by other reviewers.)", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use stacked layers of Gaussian latent variables with a maxent objective function as a regulariser. I agree with the reviewers that there is very little novelty and the experiments are not very convincing.
Decision: Reject