Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a simple stochastic neural network, which makes each neuron output Gaussian random variables. The model is trained with reparameterization trick. The authors advocates the adoptation of a non-informative prior, and shows that learning with the prior equals with an entropy-maximization regularization term. The paper presents the design of the regularization term for pruning, learning with label noise, and defensing with adversarial examples. The claims are well supported: the model is indeed simple, and the effectiveness is well supported by experimental results. 

I however think *efficiency* of the proposed model needs to be studied as well. Since the proposed algorithm is an approximate inference algorithm via reparametrization trick, it is necessary to see how fast does the approximate algorithm converge. The experiments don't report any convergence curve, or performance under limited time budget. I think there need to be some related results.

Another question is, what is the original inference problem of the designed regularization for pruning, label noise, and adversarial attack, respectively?

Update
====

Thanks for the additional experiments! After reading the rebuttal and other reviewer's comments I decide to keep my score. (Though I am less positive due to the concern of novelty raised by other reviewers.)
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposed SE-SNN, a type of stochastic neural networks that maximize the entropy in stochastic neurons along with the prediction accuracy. The authors argue that maximizing the entropy operates as a form of regularization to force the entropy into the least significant neurons, and that Increasing the diversity/randomness results in more robust models. Experiments show that SE-SNN outperform several baselines tasks such as network pruning, adversarial defense, and learning with noisy lables.

Several closely related references are missing. The idea of producing distributions in each layer (i.e., using stochastic layers) is not new and is closely related to the work on local reparameterization trick and variational dropout [1] (predecessor of the cited sparse variational dropout), and various works that directly model neurons as distribution [2, 3]. 

Built on top of the reparameterization trick, the idea of maximizing the entropy of neurons to regularize the network is interesting. Such regularization is somewhat similar to sparsity regularization on neurons, which forces the information to concentrate on a small portion of the neurons. 

The numbers for different methods in Table 1 are very close to each other. Without standard deviation it is difficult to evaluate the performance.

Note that in Table 1 the best accuracy is actually achieved by SBP and L0. However, in Table 2 and 3 (experiments on CIFAR-10 and CIFAR-100), these two baselines are missing. Is there a reason why these baselines are excluded?

Also, looking at the results from Louizos et al., 2018, which is the most recent baseline among those chosen by the authors, they actually use WideResNet rather than VGG. Comparing to VGG, WRN performs significantly better. For example, the compressed network by Louizos et al., 2018 achieves an error rate of 3.83% on CIFAR-10, versus 8% from SE-SNN pruned VGG. 

For the experiments on adversarial defense, the two attacks used seem rather weak (FGS from 2015 and CW-L2 from 2017). It may be the state-of-the-art attack around the time of Alemi et al., 2017, which the author claim to follow, but not now.

Minor:

P2: instead -> instead of

[1] Variational Dropout and the Local Reparameterization Trick, NIPS 2015
[2] Natural-Parameter Networks: A Class of Probabilistic Neural Networks, NIPS 2016
[3] Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation, ICCV 2019
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary: 
This paper presents an efficient stochastic neural network architecture by directly modeling activation uncertainty and adding a regularization term to encourage high activation variability by maximizing the entropy of stochastic neurons. Compared with other existing approaches, such as Bayesian neural networks and variational information bottleneck, the proposed architecture is simpler to implement and faster to train. The authors also achieve state of the art results in various fields, including network compression by pruning, adversarial defense and learning with label noise.

Major comments: 
- Overall, I find the paper is easy to follow and the experimental evaluation shows promising results, but my major concern is about the novelty of this work, given the fact that the structure of the proposed stochastic layers is quite similar to VIBNet.
- The derivation of the max-entropy term is somewhat unclear and I think the paper needs a major revision on this part. The authors suggest using a Gaussian with a finite mean and an infinite variance as the non-informative prior for the produced Gaussian random variable z (in Eq. 1), and then minimize the KL divergence between the produced Gaussian and the infinite-variance Gaussian. However, this may be questionable from a Bayesian viewpoint, in the sense that the infinite variance leads to an improper prior as the variance increases without bound and thus may produce an improper posterior distribution. This is not discussed and needs to be clarified in Sect. 2 (Max-entropy Regularization).
- There are things unclear in the derivation (last line of Eq. 2), since log(\sigma_2) trends to infinity.
- In addition, the penalty terms for different types of tasks are directly given only based on some of the assumptions that the authors have made, there does not seem to be any theoretical justification for such choices.

Minor comments:
- Some of the notations used in this paper seem a bit confusing, which may hinder readability. For example, on page 3, in “The non-informative prior is a Gaussian with arbitrary mean (\mu_1) and infinite variance (\sigma_1)”, I guess \sigma means the standard deviation? I would like to recommend using N(\mu, \sigma^2) to denote a Gaussian distribution, where \sigma means the standard deviation and \sigma^2 the variance.
- On page 3, in “where \sigma(h|\theta) denotes the predicted standard deviation of hidden unit h given the neuron uncertainty prediction parameter \theta”, there is no discussion on the neuron uncertainty prediction parameter. Does it means the predicted standard deviation is again parameterized by \theta?
review_title: Official Blind Review #3

Meta-review: This paper proposes to use stacked layers of Gaussian latent variables with a maxent objective function as a regulariser. I agree with the reviewers that there is very little novelty and the experiments are not very convincing.
Decision: Reject