Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The submitted paper presents a method of approximating the posterior distribution over the DNN parameters based on a Laplace Approximation scheme. It extends on the previous work by adding a diagonal correction term to the Kronecker-factored eigenbasis and also suggests a low-rank representation of Kronecker-factored eigendecomposition. Empirical evaluations were done to show that the proposed method has a more accurate uncertainty estimation compared to the previous work.

Overall, the paper is well-organized and easy to follow, although mathematical notations are not consistent throughout the paper. It is well-referenced, and the derivations look mostly correct (explained in minor comments). The main idea of the paper is convincing and well-motivated. To my knowledge, the proposed method of adding a correction term has not been introduced before. However, it is more of an incremental contribution to the existing works. In that sense, I am slightly concerned that its novelty is limited.

The experiments are not comprehensive. For the toy regression problem, a comparison to Hamiltonian Monte Carlo would be more informative. Moreover, it would be helpful to report the comparison with factorized variational methods (e.g. Graves, 2011) and experiment on modern architectures. I would be also interested to see what the additional time complexity by adding a diagonal correction term is and how more efficient it gets by low-rank approximation in experimental details.
 
Minor comments:
*Page 4: It would have been easier to understand if there had been a notational distinction between exact eigenbasis and K-FAC eigenbasis in defining V. Also, “In equation equation 10” -> “In equation 10”.
*Page 7: In Lemma 2, the order in describing the low-rank estimate does not appear to be correct. Also, in Lemma 4, shouldn’t there be a hat on I_{efb} and I_{kfac}?
*Page 9: The colour scheme in Figure 3 looks visually harder to read.
*Page 14: In equation 15 and 16, there should be a bracket for (2 \pi), and in Appendix B, some variables (e.g. p, k, R) are left unexplained.
*Page 14, 15: In equation 23 and proposition 1, \mathcal is missing, and vec operator is missing in equation 22.
*Page 3: In equation 2 and 4, is it A_{i-1} or A_i?
*Page 16: In equation 27, the dimension of X \in \mathcal{R}^{m \times m} seems incorrect. Also, in the last line, doesn’t X \odot D have different dimensions (similarly in equation 22)? 
review_title: Official Blind Review #5

final_score: rating: 3: Weak Reject
review_content: 
The paper studies the Laplace approximation for Bayesian inference of a neural network. Specifically, it proposes a diagonal correction and a further low-rank approximation to the Kronecker-factored eigenbasis for more accurate approximation of the Fisher information matrix and better scalablility, respectively. The proposed diagonal correction is shown to have a smaller residual error in F-norm. Experiments are given to show that the proposed Laplace approximation makes more accurate uncertainty estimations. 

The paper makes a certain contribution to existing Laplace approximations for the task in terms of accuracy and scalability. However, it is incremental and the novelty is a bit low, compared to many recent closely related works, for example,

Optimizing Neural Networks with Kronecker-factored Approximate Curvature. 2015
Practical Gauss-Newton Optimisation for Deep Learning. 2017
Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis. 2018
A scalable Laplace approximation for neural networks. 2018
Eigenvalue Corrected Noisy Natural Gradient. 2018

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Laplace approximation has been an important tool for obtaining uncertainty estimation for deterministic models. To efficiently approximate the Hessian matrix for neural networks, Ritter et. al (2018) proposes to use K-FAC . Motivated by the relatively inaccurate approximations of K-FAC, this paper proposes to improve KFAC approximations by combining eigen-basis corrections (EK-FAC, George et al, 2018) and diagonal corrections. The paper shows that the proposed method has smaller Frobenius approximate errors compared to K-FAC and EK-FAC. To further reduce the computational costs, the paper proposes a low-rank approximation by keeping only the L largest eigenvalues. Empirically, the paper demonstrates improved calibration and out-of-distribution entropies compared to previous approaches. 

# Diagonal Corrections. 
With the diagonal correction approximating the Fisher better, the paper shows the computation can still be conducted in the scale of W, which is similar to K-FAC. However, the method requires the diagonal correction matrix D is always positive, which might not be true. Moreover, because the computation requires D^{-1}, clipping D to a small constant will bring up stability issues. I wonder how this problem is tackled in this paper. 

# Writing 
The paper's notations are messy, which requires a lot of intellectual guesses to understand the conveyed idea. 
1) Notations are not introduced, such as $\delta \theta$, $W_{map}^{IV}$, the MN distribution in the appendix.
2) Notations are typoed. Eq(3) $N(0, A^{-1} \otimes G^{-1}) = MN(0, G^{-1}, A^{-1})$; The bottom paragraph in Page 4; eq(8).
3) Notations are abused. In particular for $V$ and $\Lambda$ when introducing EK-FAC. The paper uses $V$ for both true eigenbasis and the EK-FAC eigenbasis. In addition, the EK-FAC part should be moved to the background section.
4) The paragraph below Corollary 1 says the author can prove the proposed method also has closer approximations in terms of the Fisher inverse. But no proofs are given.
5) Caption of Figure4.

# Laplace Approximation 
For eq(4, 8, 12), The Hessian in Laplace approximations should be divided by $N$. Although the paper also mentions the scaling below those equations. But technically eq(4, 8, 12) are wrong and I don't know whether the experiments really did the scaling or not. 

# Low-rank Approximation
1) It is not clear why the low-rank approximation is necessary. The computational costs are inevitable to compute the eigen-system of A and G. Why do we need the low-rank approximation after that ? K-FAC is not a computationally expensive method either. 
2) I cannot understand the proofs of Lemma 2. In fact, I don't know what $ I_{1:L}^{top}, I_{1:K}^{top}$ means. More explicit formulas should be given for clarity. 
3) Lemma 4 states $I_ii = (\hat{I}_{def})_ii$. Although the diagonal correction makes $I_ii = (I_{def})_ii$, the low-rank approximation makes them unequal again. Or I guess you use a different D in eq(13) from the D in eq(10) ? 

# Experiments 
The paper needs more experiments to validate the proposed method. Firstly, for the MNIST experiments, it is better to use the same architecture as in Ritter et al (2018) for direct comparisons. Beyond that, the adversarial attack experiment and the mis-classification uncertainty experiment in Ritter et al (2018) seem to be good choices as well. 

# Overall
The paper proposes a diagonal corrected EK-FAC, achieving better approximation of the Fisher matrix. Interestingly, the paper shows that this corrections doesn't add too much computations.  However, the proposed low-rank approximation doesn't seem necessary. And the paper's notations and presentations are too messy to be an accepted paper.
review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: The contribution of the paper is marginal, as the principle of imposing Gaussians on the network to perform Bayes is not new. The Laplace-based approximation is half-baked and certainly much better techniques for Bayes exist in the recent literature; furthermore, it's selection is not substantiated enough, and, of course, it represents no novelty. The experimental results are not convincing, as both the considered scenarios are limited and the comparisons are too poor (no consideration of state of the art alternatives). The provided corollaries are not actually helpful and should be put in the appendix. 
review_title: Official Blind Review #2

Meta-review: This paper presents a variant of recently developed Kronecker-factored approximations to BNN posteriors. It corrects the diagonal entries of the approximate Hessian, and in order to make this scalable, approximates the Kronecker factors as low-rank.

The approach seems reasonable, and is a natural thing to try. The novelty is fairly limited, however, and the calculations are mostly routine. In terms of the experiments: it seems like it improved the Frobenius norm of the error, though it's not clear to me that this would be a good measure of practical effectiveness. On the toy regression experiment, it's hard for me to tell the difference from the other variational methods. It looks like it helped a bit in the quantitative comparisons, though the improvement over K-FAC doesn't seem significant enough to justify acceptance purely based on the results.

Reviewers felt like there was a potentially useful idea here and didn't spot any serious red flags, but didn't feel like the novelty or the experimental results were enough to justify acceptance. I tend to agree with this assessment.

Decision: Reject