Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces Variational Structured Dropout (VSD), a scalable method building on previous Variational Dropout methods, and with a structured representation of the variational noise. The paper derives the method, introduces a hierarchical prior, and provides results on many benchmarks while comparing to many baselines. VSD performs reasonably well in all benchmarks on uncertainty metrics and accuracy. VSD also performs well on out-of-distribution uncertainty experiments.
main_review: I think this is a good paper and a nice method. There is good motivation, good explanations and a thorough experiments section, where VSD is seen to perform well. Overall I like the paper, and so I focus the rest of the review on a few questions that, if answered, could increase my score:

1. In Equation 4, the authors add a mutual information term to the variational lower bound. Is this new objective still a valid lower bound of the model evidence? It apears to me like it might not be due to the mutual information term. If indeed it is no longer a valid lower bound, I think the authors should make it clearer in the text.
- Also, in Appendix I.1, the authors describe the KL annealing that they use, where they weigh the KL term by $\lambda$. For $\lambda \neq 1$, the objective is no longer a valid lower bound. I think it is fine to do this (as it has been found to help in the past), but I think this additional hyperparameter should be explicitly mentioned in the main text somewhere.
2. The results are all means over 5 random seeds. I would like to see standard deviations too, either in the main text or in additional tables in the Appendix.

Additional, more minor points:

3. I think it would be nice to have some further intuition on what T=1 vs T=2 means.
4. The authors could also compare to natural-gradient VI methods like VOGN (Osawa et al., 2019: "Practical Deep Learning with Bayesian Principles").


---------- After author rebuttal -----------
Thanks to the authors for responding to my questions and concerns. They answered many of my questions. However, given very valid concerns from other reviewers (particularly the discussion with Reviewer VNbM), I have decided to keep my score and not increase it. I do this with the expectation that the authors will fix/clarify many of Reviewer VNbM's concerns, such as:
- Being explicit in the text about the potential limitation that the new ELBO has a relatively poor regularisation, which may be the reason that the hyperprior is necessary and works well.
- More importantly, being explicit in the text regarding *exactly* which implementation is used for which experiment, with key differences in the main paper, and smaller settings in an Appendix. Reviewer VNbM brought up a very important point regarding implementations and this should really have been in the submitted draft of the paper itself.
limitations_and_societal_impact: The authors could add a line or two on potential societal impacts, but this is not necessary.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes Variational Structured Dropout, an extension to variational dropout that uses a full covariance matrix to model the posterior at the pre-activations. This covariance matrix, after spectral decomposition, is parameterized as the product of T Householder matrices.

The new parameterization of the posterior does not suffer from the singularity issue of Variational Dropout. When using a diagonal posterior approximation, the singularity issue is that the approximate posterior has zero measure, which results in the KL-term being undefined. VSD avoids this by using a full covariance matrix thus ensuring that the approximate posterior has non-zero measure.

To train the model using the new parameterization, the authors optimize a new variational lower bound. This lower bound is a lower bound to the ELBO.

Regarding the prior, the paper advocates for using a hierarchical prior distribution where the prior is a zero mean Gaussian distribution over the weights, with the hyperprior determining the standard deviation.

The experiments first examine the impact of T, the number of matrices used to parameterize the approximate posterior. Next, large scale experiments on deep neural networks demonstrate the performance of VSD against a number of baseline Bayesian inference methods measuring both predictive performance and predictive entropy (for OOD detection).

main_review: ### Originality:

I find the idea of VSD quite interesting and it is, to my knowledge, novel. It is an excellent observation that one can address the singularity issues using a full covariance approximate posterior. Moreover, it can also be understood as a novel type of low rank parameterization of the approximate posterior.

Regarding the hierarchical prior, I have not seen a hierarchical prior over the weights in this form, although it is quite possible that similar ideas have been proposed before.

### Quality:

The work is high quality. It is well motivated, it addresses the concerns with variational dropout and it also improves performance.

I have a slight concern regarding the new variational lower bound $L_{MI}$ . it is unclear to me what the effect of discarding the mutual information is. I would expect that with $L_{MI}$, we get highly correlated weights posteriors. Why doesn’t this training objective result in the weight posterior having high correlations, thus having very low probability mass?

The experimental section of the paper could be improved. It is good to see that numerous benchmarks and baseline methods are used, but all the methods seem to be underperforming. For instance, the ResNet-18 results are lagging significantly behind the reference implementations available at https://github.com/google/uncertainty-baselines/tree/main/baselines/cifar . Apart from this, the experiments look at both predictive performance and predictive entropy (for OOD samples), and they show very promising results.

Regarding the derivations in the supplementary material: I read the results, but I did not carefully check them.

### Clarity:

The paper does an excellent job at presenting the method, explaining the motivation and describing related works. The paper flows very well, each section leading into the next. 

### Significance:

The work has potential impact for both researchers and deep learning practitioners. For researchers, it is certainly interesting how the work addresses the theoretical concerns with VD. For practitioners, VSD could be an effective approach at capturing model uncertainty depending on the computational costs.


### Further questions:

* The paper mentions that using T > 2 has high computational cost. Is this because pre-activations are costly to compute, or that the method converges too slowly? What is the computational cost and complexity associated with the VSD parameterization?
* How many samples (or forward passes) are used for prediction?
* What hyperparameter values $\alpha$ and $\beta$ were used in the experiments?


### Minor:

* Line 235 convolutional instead of convolution?
* Many of the references, such as [26] and many others cite Arxiv versions of papers instead of the versions that are published at ML conferences. Also names such as Bernoulli are sometimes not capitalized correctly.


limitations_and_societal_impact: The limits and societal impacts are addressed adequately.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a new extension to the well-known and widely used Monte-Carlo dropout scheme as variational approximation to the Bayesian posterior.

main_review: The paper is overall difficult to read and understand, especially the discussion on the different parameterization choices.
Inspired by the MC-dropout, the Authors end up with a very involved parameterization of the posterior and to achieve this they propose a lot of incremental improvements and changes, which are not well motivated, in my opinion. 

Below some detailed comments to justify my overall score:

- From the presentation of the Householder parameterization in Sec. 3.1 (and around line 136), it seems that the $\Sigma$ is full-rank.
This is possible only when the number of transformations via $H$ (T) is equal to the dimensions of $\Sigma$ (K). 
In other cases, this could be considered as low-rank parameterization of the covariance.
- The parameterization of the vectors $v_t$ as a flow (i.e. $v_t = f(v_{t-1})$) is not discussed in the paper, but only after line 528 in the Appendix. 
This is an important modeling choice that should have been present in the main paper. 
- The Authors propose to run an empirical Bayes procedure for choosing the prior parameters but the effect of this choice is not assessed. Minor: From the discussion around line 192 it might be confusing for a reader why $\beta$ does not appear in Eq. 5; maybe it would be better to first show the full KL (Eq. 11 in Appendix) and then show the optimized one. As an alternative, you could use a different symbol for the optimized KL (like $\tilde{\mathbb{D}}_{KL}$).
- Still regarding the empirical Bayes, given that the $\beta$ is a matrix with same dimensions of $\mathbf W$, would it be possible for the KL to collapse to 0? In the minimization of the variational objective, all it needs is for $\alpha_j \rightarrow 0$ and it's not clear how this could be prevented. Did you experienced this phenomenon in your experiments?
- Connected with the point above, I can see why the Authors needed a hierarchical parameterization of the prior to enforce a stronger regularization. Regarding the hierarchical prior, the main paper doesn't discuss the parameterization of $p(z)$ and $q(z)$ (some details are present in the Appendix but should be reported also here).
Minor: For Figure 2 with the double y-axis it's not possible to understand which axis is corresponding to which dataset.

Two important remarks/contradictions after checking the submitted code, which are not discussed anywhere:

- From the presentation of Sec. 4.2 I seems that the Authors used this approach also for convolutional layers. This indeed is not what is happening, or at least this is not what is implemented in the submitted code, where the weights of the convolutional layers are simply optimized. The extension from fully-connected layers to convolutional ones should be fairly simple.
- From the interpretation of line 233, I seems that the Authors will finally consider only the version with the hierarchical prior as their final proposal but, again, the code implements only the previous one (see for example classification/model.py:162, which implements only Eq. 5)



limitations_and_societal_impact: Limitations and social impact are not discussed.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a novel variational structured approximation of the posterior distribution of Bayesian neural networks by the Bayesian interpretation of Dropout regularization.   An orthogonal transformation is used to learn a structured representation of the variational noise and consequently induces statistical dependencies in the approximate posterior. 
main_review: Originality: The method is new and interesting. 

Quality: Technically sound with extensive experiments. 

Clarity: Clearly written. 

Significance: Compared with previous variational-based methods, the proposed VSD achieves good results by incurring structure into the covariance, which is expected and reasonable.  The main concern is its importance of practical use. For example, in comparison with the standard non-variational baseline such as SWAG, which serves as a simple baseline for Bayesian uncertainty in deep learning,  the results in Table 2 and Table 3 show that in most cases the proposed VSD performs worse than SWAG. It then raises the long-standing question that why the Bayesian methods, though with high complexity,  do not even yield competitive results as compared with simpler methods. It would be appreciated if a comprehensive comparison of VSD and the baseline SWAG, which is simple modification of non-bayesian methods to obtain uncertainty estimates,  is further clarified. 
limitations_and_societal_impact: 1. There is a lack of comparison with non-variational baselines SWAG and SWAG in terms of predictive uncertainty in the out-of-distribution settings, as shown in Figure 3. 

2. The results in Table 1-3 lacks the variances for the random runs.
review_title: null

Meta-review: This paper proposes structural variational dropout for BNNs. 

Reviewers think the proposed methodology is novel, and the paper's theoretical contribution is significant, in the sense that it addresses a well-known theoretical issue of variational dropout & MC-dropout. Given that MC-dropout is quite often used in practice, such theoretical contribution is useful.

Still the paper has the problem that, some reviewers are concerned with the experiments using implementations that are different from the version used in theoretical analysis. So in revision, either this difference needs to be explained (if it is a confusion), or, the practical approach needs to be justified better to so that they do not have the infinite KL issues of MC-dropout.
Decision: Accept (Poster)