Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a learning method for Gaussian variational auto-encoders based on Taylor expansion approximation. This allows an analytical formula for ELBO and therefore avoid the requirement of sampling during training. 
main_review: *Update after rebuttal* I have read the authors’ response and other reviews. I would like to thank the authors for their response. They have addressed my concerns - limited experiments and poor presentation. Although, the related work section could still use some work in terms of writing (for example, to avoid presentation in a form of a list “A did this, B did that”, but it has been improved. I am therefore increasing my score.

————————————————————

Strong points:
1. Simple yet novel approach for approximating ELBO that allows analytical solution
2. Sampling-free training of Gaussian VAEs that scales to large networks
3. Rather thorough experiments in terms of covering the question from different perspectives, i.e. assessment of Taylor approximation, quantitative performance, posterior collapse discussion

Weak points:
1. Limited experiments in terms of considering other baselines. Related works cover a lot of different VAEs and their training methods, yet almost all the experiments are on contrasting with sampling VAE (without the proper specification which specific sampling VAE is used)
2. Motivation for the sampling free training for VAE in general as a concept is not very well provided considering that it comes with the computational overhead. The text implicitly assumes that sampling free training is obviously better and only additionally support it with better performance
3. Some writing issues. The related work section in particular could see a lot of improvement. Considering the context of the paper, one would expect the related work section to focus on the discussion of the training methods for VAEs and on contrasting sampling vs sampling free methods. Instead a reader finds a generic overview of different VAEs, with some works that do not seem to be really related to the submission apart from being devoted to VAEs, e.g. Tolstikhin et al, or Rezende et al.

I am voting for a weak acceptance of the paper as I found the general idea of the paper to be appealing and worth attention of the community, however the above mentioned weaknesses stop me from higher score. I believe the paper can be made very strong with some work as thorough empirical support and better focussed presentation.

Specific comments/suggestions (not necessarily important for assessment, but points to improve the paper):
1. Mix of reference styles is used with Author and [Number] mixed
2. Related work, first paragraph. 3 layer network, specific number of hidden units and latent dimension, MNIST dataset - these may be too much details for related work description. 
3. Related work, last sentence - Not very clear what this is for in this section
4. f_{\theta} is not defined
5. I_m is not defined
6. Figure 1. Not clear what sampling and sampling-free mean here as it seems that sampling one should not depend on \gamma, a hyperparameter of the proposed sampling-free method
7. Figure 4. Why there is no shaded area for log ML plot? 
8. It would be interesting to see a comparison between the proposed approximated sampling free method and the exact sampling free method, even though it would mean to rely on a small scale experiment

Minor: 
1. Section 3.3. "that works independentLY"


summary_of_the_review: The paper seems to provide an interesting idea for training a VAE that allows an analytical solution and therefore does not need sampling, but the empirical evaluation of the method is somewhat limited and presentation of the paper could be significantly improved. Therefore, I am voting for the weak acceptance as I believe the idea is worth to be known by the community but the existing drawbacks stop me from giving a higher score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
In this paper, the authors propose a closed-form approximation for the Gaussian decoder distribution in VAE.  The closed-form is achieved by matching the mean and covariance matrix with the Taylor expansion of the nonlinear mapping. 
main_review: 

Strengths 

1. The closed-form approximation reduces the computation cost for computing the ELBO of exact Gaussian VAE. 
2. The paper is well organized and clearly written. 


Weakness.

1. The proposed method is similar to the well-known moment matching (MM) methods. However,  a discussion of the relationship between the proposed method and MM is missing. In addition, it is better to include a comparison with MM to justify the benefit of the proposed method. 

2. The proposed method is limited to Gaussian VAE.  It may not apply to VAE with other distribution. 

3.  In experiment 4.1,  how to set the hyper-parameter $\gamma$ in Eq.(21).  It is better to set $\gamma=1$ to justify the theoretical approximation quality. 
summary_of_the_review: 
Overall,  I think the proposed method is similar to the moment matching method. It is marginal novel and limited to the Gaussian case. Thus, in my opinion,  this paper is marginally below the acceptance threshold.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approximation that frees the calculation of ELBOs of Gaussian variational autoencoders from sampling. To achieve this, it utilizes Taylor expansion on the decoder networks. The proposed method was evaluated on three typical datasets. The authors also found that the approximation alleviated the issue of posterior collapse to the sampling VAEs.
main_review: Post-rebuttal: Thank you for the rebuttal. The authors addressed some of my concerns, added experimental comparison and updated the text. However, the overall novelty and contributions are still not adequate for ICLR in my opinion. I will not change the score.

######

Pros: 
 
+ The analytical approximation would help with reducing the variance and computational expense of the ELBO. 
 
+ The proposed method achieves similar or better performance than its sampling counterpart.

+ The proposed method is found alleviated the issue of posterior collapse that the sampling VAEs suffers.


Cons: 

- The writing should be substantially improved. The words and sentences are not so fluent. There are obvious grammar errors and typos. I believe that it is critical for the readers to correctly understand and evaluate the contributions. The figures are labeled in a random order.

- The advantages in performance looks marginal comparing to the sampling VAE. The proposed method beats sampling VAE for small number of samples. I don't mean to criticize this point since it's an approximation while the sampling ELBO is an unbiased estimate. Instead, the claimed advantages are not well demonstrated. For example, the authors could show the wall time and variance of ELBO. Sec 4.4 is qualitative.

- The proposed approximation is limited to Gaussian encoders and ReLU decoder networks. The Taylor expansion itself should be applicable to other types of networks, but the performance then needs further evaluations. 

- Posterior collapse part could be more elaborated. Can the authors address more how this approximation could solve the issue?

Concerns:

* What's its connection to the Laplace method in variational inference.
* Strictly speaking, the approximation is not the exact ELBO using in the sampling VAE and thus different objective in fact. Futhermore, it's not necessarily a lower bound to the marginal likelihood. What is the gap between the approximation and the ELBO or true marginal likelihood?
* The covariance of reconstructions evaluated in the paper is not clearly defined? For approximate VAE and sampling VAE, the output is different. The former is a distribution but the latter is a sample.
* Sec 3.2: Why is the intractability due to exponential complexity rather than integrability?

Minor:

* The computional complexity better be in form of big O notion.
* The layout of Eq. 16 better be topdown.
* Figure 5 ref before Figure 1
summary_of_the_review: Overall, I vote for rejecting. I like the idea that analytical approximate ELBO would help the computation. The finding on posterior collapse is also interesting. However, the method and evaluation are not adequately demonstrated. The writing can be largely improved too.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a sampling-free approximation to the ELBO of a variational auto-encoder with Gaussian likelihood and mean-field Gaussian variational posterior. The approach relies on a Taylor series around the mean of the posterior, which allows for an evaluation of the ELBO's expectation, rather than relying on Monte Carlo Sampling.

main_review: _Edit: Thank you for the rebuttal. The authors clarify some of my concerns and include some minor improvements in the experimental comparisons, which is why I slightly increase the score (3 $\to$ 5). However, the overall contribution still remains minor in my opinion which is why I can't recommend acceptance._
__________


### Strengths
- The paper is overall well written, and the approach is evaluated on several data sets.
- The computational cost introduced by the approach is explicitly discussed and not hidden in an appendix. 

### Weaknesses
- The contribution itself is very minor, as Taylor approximations to solve expectations are a common approach in the literature. (See, e.g. Wang and Blei (2013); Blei et al. (2017) and the references therein for further pointers, or the keyword delta method.)
- The sampling-free method aims to offer a better approximation to the ELBO. However, the ELBO in itself is only of limited use and mainly serves as a gradient signal. While the experiments seem to show that the approach improves upon a naive sampling, the paper lacks a comparison (and a discussion) to other methods that aim to stabilize the gradient e, e.g. Roeder et al. (2017) and similar methods. 
- Similarly, a comparison with other approaches of similar network depths to properly place, e.g. its FID into relation with the current literature, is missing.

### Minor points & Questions
- How to read Figure 1 and the $\gamma$ discussion? Sec 3.3 first mentions the exploration of three $\gamma$ values (0.5,0.9,0.99), while Figure 1 explores a different set of values, which influence only the orange sampling ELBO, not the blue sampling-free ELBO. Is the legend broken here? Similarly, what does the number of iterations refer to, as the blue line seems to have converged from the beginning?
- Figures 7,8 in the appendix seem to be broken for the Autoencoder. Is that an error in the figure or due to problems with the underlying model? 
- Sec 4.1 mentions the "true output... via a large number of samples". What is large here? _(I might have overlooked the hyperparameter in the appendix)_
- What is the precise architecture used in the experiments? Sec 4.2 first claims to follow Burgess et al.'s architecture, then later mentions a 5 conv layers + 1 fully connected layer structure. Appendix B.1 switches to claiming to follow Higgins et al. instead but mentions again five convolutional layers, which is not what Higgins et al. report to be using in their appendix (they claim four convolutional layers). (Burgess et al. also use four convolutional layers but a different number of channels compared to Higgins.)
- Are the minimum/maximum missing from the right plot in Figure 4, or so close to the mean to not be visible?
- Table 2 gives the average over multiple runs. What is the standard deviation for the results?


__________
Wang and Blei, Variational Inference in Nonconjugate Models, JMLR 2013  
Blei et al., Variational Inference: A Review for Statisticians, JASA, 2017  
Roeder et al., Sticking the landing: Simple, lower-variance gradient estimators for variational inference, NeuRIPS 2017

summary_of_the_review: The paper demonstrates the usefulness of a Taylor approximation compared to a sampling-based approach. However, that in itself is of limited novelty, and the paper lacks a proper comparison to other methods and the broader literature to classify it properly.
review_title: null

Meta-review: This paper proposes a way to train Gaussian variational autoencoders that does not require the computation of empirical expectations but instead approximates the decoder network by its Taylor series. Results on 3 datasets show the competitiveness of the approach.

Based on the limited novelty of the approach, three (out of 4) knowledgeable reviewers recommend rejection and I agree. Variational autoencoders are simply doing variational inference in a specific model and, as one of the reviewers has pointed out, these types of approximations have been exploited in the inference world (before the popularization of the reparameterization trick)  for many years. Methods, where we replace a term in the joint distribution with a simpler function, are known in the variational inference world as local variational approximations, see, e.g. Murphy’s book (Machine Learning: A Probabilistic Perspective, 2012, Sec. 21.8) as a reference. The community has departed from such approaches as using the re-parameterization trick is unbiased, more general (e.g., not limited to Gaussian encoders) and allows for highly automated methods (no need to do derivations on a case-by-case basis).  Nevertheless, I encourage the authors to thoroughly explore the literature on variational inference with regards to these types of approximations. It may well be the case that, in the future, we revert back to these methods if they perform well in practice with modern architectures. For this, more comprehensive evaluations and comparisons are needed.
Decision: Reject