Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presented an iteratively refined variational inference for Gaussian latent variables. The intuition is straightforward and makes sense to me. However, I have some concerns.

Detailed comments:
1. In theoretical justification, only K=2 is discussed. My intuition is that as K increases, the approximation of the true posterior should be closer. The summation of multiple Gaussian distributions can arbitrarily approximate any distribution given enough base distributions. I would like to see some theoretical discussion about K. At least in the experiment, the author should provide the performance of different Ks.
2. The toy example in the paper is simply 1D Gaussian. I want to see more discussion for high dimensional latent variables. So in the experiments, how you parameterized the distribution for each weight? Totally independent? or allowing structural correlations? I am not sure the details of the implementation in this paper, but I also have a naive question for high dimensional Gaussian. Does it require to compute the matrix inverse when sampling a_k?
3. Another related paper "Guo, Fangjian, et al. "Boosting variational inference." arXiv preprint arXiv:1611.05559 (2016)." should be discussed as well.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes to improve standard variational inference by increasing the flexibility of the variational posterior by introducing a finite set of auxiliary variables. Motivated by the limited expressivity of mean field variational inference the author suggests to iteratively refine a ‘proposal’ posterior by conditioning on a sequence of auxiliary variables with decreasing variance. The key requirement to set the variance of the auxiliary variables such that the integrating over them leaves the original model unchanged. As noted by the authors this is a variant of auxiliary variables introduced by Barber & Agakov. The motivation and theoretical sections seems sound and the experimental results are encouraging, however maybe not completely supporting the claim of new ‘state of the art’ on uncertainty prediction. 

Overall i find the motivation and theoretical contribution interesting. However I do not find the experimental section completely comprehensive why I currently think the paper is borderline acceptance. 

Comments
1) The computational demand using the method seems quite large by adding O(NumSamples * NumAuxiliary) additional computational cost on top of the standard VI. Here each factor M is quite large e.g. 200 epochs for CIFAR10 (if i understand the algorithm correctly?)
2) For the UCI experiments the comparison is only made against DeepEnsembels or other VI methods, however to the best of my knowledge MCMC methods are superior in this setting given the small dataset size? 
3) The results on CIFAR10 do seem to demonstrate that the proposed method is superior to DeepEnsembles and standard VI in one particular setting where VI is only performed over a small subset of layers in a ResNet (why doesn’t it work for when doing VI on all the parameters?). However generally looking at the best obtained results of ~86% acc this is quite far from current best probabilistic models (see e.g. Heek2019 that gets 94% acc). Some of this can probably be attributed to differences in data-augmentation and model architecture however in general it makes it very hard to compare with other methods when the baselines are not competitive. 

Minor Comments:
 In relation to comment 3) above I think you should reword the sentence “It sets a new state-of-the-art in uncertainty estimation at ResNet scale on CIFAR10” in the conclusion.


“In  order  to  get  independent  samples  from  the  variational  posterior,we have to repeat the iterative refinement for each ensemble member”: Does this imply that if we want M samples we first have to optimize using the standard VI and then to M optimizations to get q_k(w)?


How sensitive is the method to sequence of variances for a?

[Heek2019]: Bayesian Inference for Large Scale Image Classification

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The paper proposes a new way to improving the variational posterior. The q(w) is an integral over multiple auxiliary variables. These auxiliary variables can be specified at different scales that can be refined to match different scales of details of the true posterior. They show better performance regression and classification benchmark datasets. They also show that the training time is at a reasonable scale when being parallelized.

I think the idea is quite interesting. They did a good illustration of the difference between their model and related works. They also compare with the state-of-art variational approximation methods. 

One concern I have is the complexity. I don't think it's just O(MK) since it has to optimize for each phi_k for each posterior sample w. This could be quite large depending on problems. 

Also is the refining only run for once or run after each update of the mean field q(w)? If it's the latter, the overhead would be much larger.

When w is very high-dimensional, the number of auxiliary variables should be exponentially larger. Is that true? Or it's actually invariant to the dimensionality of the posterior distribution?

The paper proves that the refined ELBO is larger than the auxiliary ELBO which is larger than the initial mean field. But the initial ELBO should be a tight lower bound of the true log likelihood. Would that be a problem that ELBO_ref actually spill over the true log likelihood which is a dangerous sign?

Overall I think the paper is well written. The experiments are carefully designed. The idea is interesting and useful.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary.

This paper describes a method for training flexible variational posterior distributions, which consists in making iterative locale refinements to an initial mean-field approximation, using auxiliary variables. The focus is on Gaussian latent variables, and the method is applied to Bayesian neural nets to perform variational inference (VI) over the weights. Empirical results show improvements upon the performance of the mean-field approach and some other baselines, on classification and regression tasks.

Main comments.
 
Overall, this paper is well written and easy to follow. It tackles an important topic in VI and proposes an interesting idea to improve the flexibility of the approximate distribution. I have the following comments/questions.

- On the guarantee of improvement. I still have some doubts regarding the inequality “ELBO_aux >=  ELBO_init”. Can you please elaborate more on this and provide a detailed formal proof? Figure 2 shows that ELBO_aux can go below ELBO_init.
- The focus of the paper is on Gaussian variables and a configuration where some key distributions, q(a_1) and q(w|a_1), are accessible in closed from. The generalization of the proposed method beyond these settings should be discussed and explored in experiments. 
- Important baselines are missing in the experiments. I would recommend including at least the other VI techniques relying on auxiliary variables to build flexible variational families [1,2]. This would help to better assess the impact/importance of the proposed method.

[1] Ranganath, Rajesh, Dustin Tran, and David Blei. "Hierarchical variational models." ICML. (2016).    
[2] Maaløe, Lars, et al. "Auxiliary deep generative models." ICML (2016).

review_title: Official Blind Review #3

Meta-review: In this paper a method for refining the variational approximation is proposed.

The reviewers liked the contribution but a number reservations such as missing reference made the paper drop below the acceptance threshold. The authors are encouraged to modify paper and send to next conference.

Reject. 
Decision: Reject