Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: 

# summary

This paper proposed a method on uncertainty estimation in deep neural
networks. Compared with BNN and deep ensemble, the proposed approach in this
work has a storage advantage. Furthermore, this work provides a better
trade-off between accuracy and calibration.


# pros

1.  The approach in this work is quite interesting. The idea of augmenting
    weights with auxiliary low-dimensional latent variables in a deep neural
    network seems natural at first sight, but this approach is novel as far as
    my knowledge is concerned. Although VI with local latent variables is an
    old technique, this application in deep neural network is novel.
2.  The authors also proposed an efficient approach that can sample from the
    variational approximation conditioning on the latent variable. Since the
    original weight is large, such a sampling is necessary.
3.  This paper provides extensive empirical results and sufficient theoretical
    results. Experimental results show the proposed approach achieve a good
    balance between accuracy, calibration and memory requirements.


# cons

1.  My major concern is all experiments are conducted on ResNet18, and this is
    not a typical choice for practical problems. I think experiments on other
    larger net such as ResNet56 on CIFAR10 will make this paper more
    convincing.
2.  It is not clear to me how the authors choose the "mixture of delta
    measures", i.e. how to choose U<sup>k</sup>? Can the authors comment on this? In
    this case, q(U) is a categorical distribution. It seems better if  q(U)
    also depends on input data, however, the authors choose to use a fixed
    q(U). Can the authors comment on this choice?

Overall speaking, the idea and the presentation of this work are great in my
opinion.




review_title: Official Blind Review #1 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Bayesian deep learning attempts to incorporate uncertainty estimation in the modern neural network models. However, common approaches such as Bayesian neural networks and Deep Ensembles incur large memory overhead because of the increased parameter sizes. Motivated by the inducing points approach in the sparse variational Gaussian Processes area, this paper proposes the Gaussian variational posterior relying on the compact *inducing weights* $U$ and the conditional distribution $p(W|U)$. Because the variational parameter shifts from $W$ to the smaller $U$, the resulting model potentially uses even fewer parameters than a deterministic counterpart. In general, I think the paper proposes an interesting approach that could help addressing the storage issues of current Bayesian deep learning models. 

**Novelty**
The proposed approach is both novel and elegant. Given the high dimensionality of the parameters, many BNN approaches consider structured covariance for the variational posterior, which involves the balancing between computational costs and approximating capacity. And these approaches all maintain a "mean parameter", thus the memory costs are at least as large as the original network. In comparison, this paper turns to an augmented space using the joint Gaussian distribution, and instead model the variational posterior for the compact *inducing weights*. In consequence, the resulting model could potentially have even fewer parameters than a deterministic counterpart, while being able to conduct uncertainty quantification. Given that the inducing points approach has achieved big successes for scalable GPs, I think the proposed method could be impactful for uncertainty modelling in deep learning models. 

**Experiments**
This paper conducts experiments covering both image classification and out-of-distribution detection. Empirically, their model has increased calibration compared to the deterministic network while using $\leq 47.9\%$ of parameters. However, as shown in Figure 4, the proposed approach incurs large computational overheads especially when using small number of samples. 

**Questions**
1. What are the inducing weights for the conv layer? Are they matrices of shape [128, 128, kern_size, kern_size]?
2. The Ensemble-U approach uses a dirac measure $\sum_k \delta_{U^k_l}$ in each layer and drops the KL penalty. Are dirac measures of different layers independent, or like deep ensemble, the particles are tied into $K$ disjoint groups across layers? If it was the former, I would reckon that the particles of the same layer would converge to the same place; if it was the latter, it should be made clearer in the paper.
3. How does the sample size $K$ influence the performance? In practice, I think only a small number of $K$ should be used.
4. A hyperparameter $\sigma_{max}^2$ is set for the variance of $q(W)$. And the similar hyperparameter $\lambda_{\max}$ is set for the variational posterior. Especially for $\lambda_{max}$, increasing it from $0.1$ to $0.3$ observes large performance drop. Is this an issue due to the expressiveness of the proposed posterior ? 

**Clarity** 
In spite of my questions regarding the model details, the paper well presents its main methods. Besides, I think Sec3.3 is worthy of more polishing, since it looks confusing when you start by studying $U_c$ instead of $U_r$ or $U$. And a few typos to be corrected, 1. Abstract: whichenable 2. Last paragraph of introduction: our approach achieve 3) Related works: function-space inference is appealing to ... 
review_title: A compact variational BNN posterior for memory efficiency

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
This work proposes a specific parametrisation for the Gaussian prior and approximate posterior distribution in variational Bayesian neural networks in terms of inducing weights. The general idea is an instance of the sparse variational inference scheme for GPs proposed by Titsias back in 2009; for a given model with a prior p(W) perform variational inference on an extended model with a hierarchical prior p(U) p(W | U), that has the same marginal p(W) = \int p(U)p(W | U)dU as the original model. The authors then consider “U” to be auxiliary weights that are jointly Gaussian with the actual weights “W” and then use the decomposition p(W|U)p(U), q(W|U)q(U) for the prior and approximate posterior (which can easily be computed via the conditional Gaussian rules). Furthermore, they “tie” (almost) all of the parameters between q(W|U) and p(W|U) (similarly to Titsias, 2009). The main benefit from these two things is that since the mean and covariance of the Gaussian distribution over W conditioned on U can be efficiently represented as functions of U, whenever dim(U) << dim(W) we get reductions in memory for storing the distributions over the parameters in the network. The authors furthermore, discuss how to efficiently parametrize the joint distribution over W, U, discuss different choices for q(U) (that can lead to either traditional VI or something like deep ensembles). In addition, they also discuss how more efficient sampling from q(W|U) can be realised via an extension of the Matheron’s rule to the case of matrix random variables. Finally, they evaluate their method against traditional mean field variational Bayesian neural networks and deep ensembles on several tasks that include regression, classification, calibration and OOD performance. 

### Pros
- The method provides a novel way to induce parameter efficiency in variational bayesian neural networks
- It connects to the sparse GP literature and the trick seems to be general enough, in that it can be applied to any distribution that admits some parametrisation in terms of a Gaussian random variable.
- The extension of the Matheron’s rule can be of independent interest
- Extensive set of experimental tasks, with a nice ablation study for lambda_max and sigma_max

### Cons
- Comparison against alternative approaches that induce parameter efficiency are missing
- Results are mixed and sometimes not very convincing


### Recommendation 
While I find the main idea very interesting and the presentation of it relatively clear, I unfortunately cannot recommend acceptance of this work as is. The main motivation behind improving parameter efficiency can also be performed with other, perhaps much simpler, ways and comparisons against such approaches is missing. Furthermore, the results, at least in their current state, are a bit mixed and thus not very convincing. 

### Detailed feedback
Overall, this work is interesting and relatively easy to follow. Using inducing variables in neural networks in not a new concept, as it has been used before in, e.g., [1, 2], but the motivation of using them as a means towards reducing the number of parameters of each distribution is new. The authors explain the main idea behind them in a clear manner. Furthermore, they clearly explain the need for using matrix normal distributions, explaining how the Kronecker product factorisation of the covariance improves the parametrization efficiency, along with how efficient sampling from q(W|U) can be performed. In addition, performing parameter efficient deep ensembles in U space is a nice bonus of this formulation. Section 3.3 however is dense and can be a bit hard to follow (whereas Appendix D is clearer). I would therefore suggest that the authors briefly describe the main idea in a couple of sentences and refer the readers to Appendix D instead.

My main point for feedback is for the experimental section. The authors argue at the beginning of section 4 that “the goal is to demonstrate competitive performance …. While remaining computationally efficient.” Is that claim for the training or evaluation phases? From what I see, during the training phase, the inducing weights framework is not faster or more efficient than the baselines. The main advantage of the inducing weight is instead memory reduction in terms of parameters which translates to “memory efficiency” and not “computational “efficiency”. If then memory efficiency is the main target, I believe that some reasonable baselines are missing. For the case of a variational Bayesian neural network a simple baseline that performs pruning post-hoc (e.g., the one presented at the FFG-W paper or the one from [3]) in order to reduce the parameter count; remove weights by either setting them to exact zero or equal to the prior and then use the variational posterior for those that survive. It would be interesting to see how such an approach would fare not just on accuracy, but also on the ECE and OOD. Similarly for deep ensembles, a baseline where you perform pruning (e.g., simple magnitude based) would also serve as a better baseline for the Ensemble-U.  Both of these baselines would provide a more complete picture and would be a better signal in understanding whether the inducing weights framework is a better choice overall.

As for other points

- At figure 3 you only show FCG-U and not the FFG-U which is used in all of the other experiments. How does the uncertainty look like with FFG-U? 
- Sometimes, both FFG-U and Ensemble-U underperform compared to FFG-W and Ensemble-W and it would be good to know if it is due to less free parameters to optimise or due to the model definition itself. How does FFG-U perform when increasing dim(U) such that the parameter count is the same as FFG-W? Similarly for Ensemble-U vs Ensemble-W.
- The second point at the end of page 2 only makes sense in hindsight (i.e., after one reads section 3). Perhaps the authors could expand on them a bit better so that it is self-contained (e.g., explain what you mean with “q(theta|alpha) can be efficiently parametrised”).


[1] Structured and efficient variational deep learning with matrix Gaussian posteriors, C. Louizos & M. Welling, 2016
[2] Global inducing point variational posteriors for Bayesian neural networks and deep gaussian processes, S. Ober & L. Aitchison, 2020
[3] Practical Variational Inference for Neural Networks, A. Graves, 2011

review_title: Interesting work, some key baselines are missing

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes to define the weights approximate posterior of a NN with inducing variables.

The main benefit of the approach lies in the compactness of the description (Fig4, right). However, if memory size is the main issue, entropy should be considered instead of just a count of parameters. Moreover, if complexity and memory footprint is the main concern, MC dropout sounds like a reasonable alternative to Bayesian NN. Why do the authors do not compare their proposal to a conventional MC dropout approach?

Overall, the contribution of the paper appears to be valid but relatively limited in scope compared to what exists in the literature (inducing variables are known, Bayesian NN are known, ensemble methods are known). What is the gain in knowledge brought by the paper? I might have missed a piece of the contribution, but the way the paper is written does not help. It is not self-contained (many previous works have to be read to follow the story), and remains relatively opaque to the non-expert. It lacks a clear and eye-bird picture of the approach (including both training and inference steps), to position and compare it to existing works.  

The overhead in optimizing (1) or (2), compared to training a deterministic NN, is not discussed. 

Variables d_in^l is used in Section 2, but only defined in Section 3. 

review_title: Unclear gain in knowledge

Meta-review: The scores for this paper have been borderline, however the decision has been greatly facilitated by the participation of the authors and reviewers to the discussion and, more importantly, by active private discussion among reviewers and AC. Specifically, from the private discussion it seems that the reviewers find interesting ideas in this paper, but are overall are not entirely convinced about its significance, at least in the way the paper is currently positioned and motivated. 

More specifically, the reviewers found the main idea of using inducing weights interesting, both technically (e.g. associated sampling scheme) but also in terms of application (sparsity). The results are insightful from a theoretical perspective. That is, the inducing weights and their treatment does seem to result in interesting and potentially useful statistical properties for the model. On the other hand, it is important to note that the high-level idea of variational inducing weights, with usage of matrix normals in this setting, as well as connection to deep GPs has been studied before, as pointed out by R2 (refs [1,2]). Furthermore, even after discussions the motivation is still not entirely convincing, especially in conjunction with the experiments. Although various interesting ideas exist in the paper, both R2 and R3 in particular remain unconvinced about what is the main benefit (e.g.  pruning or runtime efficiency) stemming out of the proposed idea. Another reviewer agreed with this point in the private discussions.  Apart from overall clearer positioning of the paper, the claimed benefit would need to be supported by experiments tailored to illustrate this main point. The authors argued against some of the suggested comparisons (e.g. past pruning methods), and further discuss that there is no established experimental benchmark for the parameter efficiency of BNNs. I indeed sympathize with both of these arguments; however, I believe that if the reviewers' suggestions for extra experiments are rejected, it remains the responsibility of the authors to find a slightly different way of motivating their work and demonstrating its efficiency in some convincing, meaningful and more well-defined setting with the appropriate benchmarks.  

Decision: Reject