Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel MCMC algorithm (ATMC) that estimates and samples from the posterior distribution of neural network weights. The motivation for this approach is that applying Bayesian inference to deep learning should lead to less overfitting and better uncertainty-calibrated models. Unlike previous work, the proposed method scales to large models (ResNet) and data sets (ImageNet).

In addition to the main contribution, this work improves an existing numerical integrator necessary for the ATMC sampler. Moreover, in order to apply this method to ResNet, the authors use a modified version of ResNet without stochastic regularization (batch normalization and dropout).

Empirical results show that the proposed method outperforms baselines on accuracy, log-likelihood, and uncertainty calibration. The single-sample posterior already yields decent results, beating a batchnorm-free version of ResNet.

Overall, the paper is well-written, fluently readable, and provides a clear presentation of motivations and methods. Experiments, although not very extensive, are described in sufficient detail and corroborate the claims. Related previous work is cited throughout the paper, although there is no explicit section for it.


Weaknesses:

While the authors claim that the need for hyperparameter tuning is reduced, they use a cyclic step size with parameter n, a Laplace prior with parameter b, a momentum noise with parameter 0.9, and dataset-specific parameters h0, m, and c. This weakens (or even contradicts) the claim, and raises the question of how much the choice of these hyperparameters affects performance.

On ImageNet, ATMC doesn't seem to be well calibrated, and the authors do not discuss the fact that the calibration is basically on par with SGNHT.


Minor:
- page 4: practise -> practice
- page 7: extra space after "compatible with MCMC methods"
- page 7: problem specific -> problem-specific
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The authors propose the adaptive thermostat Monte Carlo sampler for feedforward neural networks. The proposed approach dynamically adjust the amount of momentum and noisy applied to each model parameter during updates. ResNet++ (ResNet without batchnorm/dropout but adding SELU, fixup and weight normalization) is introduced. Further, the authors claim that the need for hyperparameter setup is reduced provided that early stopping, stochastic regularization and carefully tuned learning rate schedules are not required.

The authors highlight some practical issues with the Nose-hoover thermostat, however, recognize its mathematical soundness. When ATMC is described (temperature stages), a motivation is provided but not justified theoretically.

In (10) and (11), \gamma_1(), \gamma_2(), \eta_t and a are introduced without definition or further explanation.

The authors claim that the need for hyperparameter setup is reduced, however, in the experiments they use a cyclic step size with length n=50 (20 for ImageNet), a Laplace prior with parameter b=5, momentum noise with parameter 0.9, pre-conditioner parameter 0.0003 and c parameter 0.001. The impact of these choices on performance is not described. Further, the number of filters is doubled relative to ResNet-56 without explanation.

The calibration curves in Figure 3 are underwhelming. ATMC is better than SGD but not necessarily well calibrated. Also, note that x and y scales are heavily biased toward 1.

In summary, the proposed approach needs to be described in more detail and the experiments are not very satisfying given the claims made by the authors in the Introduction.

Minor:
- In (1) W is not defined.
- In (1) the dimensionality of D, Q and \Gamma is not defined but their elements are used.
- In (2) m is only defined after (3), in fact, only called by its name, pre-conditioner, in Algorithm 1.
- In Section 2.3 there is a reference to the step size, though not introduced until discretization later in Section 3.
- In (7) \beta() is a function of p, but not in other instances, e.g., (4), (10) and (11).
- Move Algorithm 1 closer to definition.
- In (13), d is not defined.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary:

The main idea of the paper is the introduction of ATMC, an adaptive noise MCMC algorithm that dynamically adjusts the momentum and noise applied to each parameter update while sampling from the posterior of a neural network. A modified version of ResNet architecture called ResNet++ has been introduced in the paper and later used to train it on ImageNet using MCMC which is great work considering the complexities associated. Furthermore, the authors claim ATMC to be robust to overfitting and it provides a measure of uncertainty with predictions. The paper is well written with clear flow and good mathematical explanation. 

Questions from authors:

1. The approach of Stochastic Gradient MCMC (SG-MCMC) has been there for quite some time and in this paper, there is no clear explanation of the advantages of ATMC over SG-MCMC methods. The authors can argue that the introduction of dynamically adjustable momentum and noise can make the paper novel and unique. However, in Stochastic gradient Markov chain Monte Carlo (SG-MCMC) [1], the authors used a meta-learning algorithm to learn Hamiltonian dynamics with state-dependent drift and diffusion which makes the system scalable to large datasets, very similar to this work. Also, changing the momentum variable in Hamiltonian dynamics with thermostat variable in stochastic gradient thermostats, the meta learner algorithm of [1] can be used to make the stochastic gradient thermostats approach adaptive. What are the advantages of the approach proposed in this paper over [1]?

2.  The paper follows the approach of stochastic gradient thermostats as introduced in the paper  [2] and builds upon it to make it adaptive to improve stability and for better convergence.  The advantages are clearly mentioned in the paper except for the case when 0 < ξ < D_c the total amount of noise added to the momentum is D_c and the friction coefficient β(ξ) = D_c. At this stage, the authors claim that 'the stochastic gradient noise is compensated for by adding less noise to the momentum update.' How is this done? Can authors please explain it a bit more.

3.  For the experiments, the authors introduced ResNet++ architecture which is based on ResNet architecture but is novel in its design with the use of SELU and removal of BatchNorm and with different initialization schemes. The design idea is well explained by the authors. The experiments were performed on CIFAR10 and ImageNet dataset to show the large scale scaling of the approach. A comparison with SGNHT is provided in the paper but a fair comparison with other approaches like SGHMC [3], PSGLD [4] and MCMC [1] is missing which might have proved the effectiveness of the approach when compared to other methods. Any comments on why the experiments were restricted to the ones mentioned in the paper?

4. Lastly, the ResNet++ network is trained on ImageNet and CIFAR10 datasets. There is no clear mention of the time duration it took for training and evaluating the network. The authors mentioned ‘BatchNorm did result in an overhead of roughly 20% compared to the ResNet++ model’, which is not clear. I would urge the authors to explain the above line and provide an estimate of the training time and testing time. 

A closing remark: The mention of the acknowledgement section in a double-blind review is not advisable and in future please refrain from doing so.

References:
[1] https://openreview.net/pdf?id=HkeoOo09YX#page=11&zoom=100,0,754
[2] http://people.ee.duke.edu/~lcarin/sgnht-4.pdf
[3] https://arxiv.org/pdf/1402.4102.pdf
[4] https://arxiv.org/abs/1512.07666






review_title: Official Blind Review #2

Meta-review: This paper proposes a variant of Hamiltonian Monte Carlo for Bayesian inference in deep learning.

Although the reviewers acknowledge the ambition, scope and novelty of the paper they still have a number of reservations regarding experimental results and claims (regarding need for hyperparameter tuning). The overall score consequently falls below acceptance.

Rejection is recommended. These reservations made by the referees should definitely be addressable before next conference deadline so looking forward to see the paper published asap.
Decision: Reject