Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors suggest an alternative probabilistic generative model that assumes a prior distribution on the parameters of a classical autoencoder. The authors use empirical Bayes on a separate set of data to determine hyperparameters, use stochastic gradient HMC for posterior inference, and fit a hierarchical latent variable model to the latent codes in order to be able to sample from the latent space. 
main_review: I think this is a well-written paper that constitutes a potentially welcome addition to the increasing arsenal of probabilistic generative models in machine learning. The motivation and presentation is clear, the authors' solution to potential challenges of their methodology is parsimonious, and their results are of interest for other researchers in the field. However, there are also some issues that needs to be resolved before the final decision is made.

Figure 1 summarizes one of my main concerns with the authors' argumentation. From my perspective, this figure demonstrates a result that is completely trivial/expected. Their hyperparameter determination method is an empirical Bayesian approach, so therefore it would of course create a better reconstruction of the data without any training. Actually contrary to authors' presentation, empirical Bayesian approaches are not without their disputes in the field due to their overreliance of the data (the double-dipping problem). So, Figure 1 is not only trivial, but it also represents a potential weakness of the empirical Bayesian methods, which would be overreliance on the data at the expense of generalizability or out-of-distribution performance. Therefore one could expect a sharper decline in reconstruction performance as the data distribution shifts away from the original distribution, or a more drastic loss in local smoothness in the latent space (slight perturbations leading to more drastic changes in the reconstruced output). 

The authors' lapse in the presentation of empirical Bayes methods (contrasting them with the 'pathologies' of standard priors) are accompanied by lack of significant experiments that would probe the potential weaknesses of an empirical Bayesian approach. I would love to see how the authors would decide to test these potential weaknesses of their methods. I am not necessarily in favor or against empirical Bayes methods, but if they are used, their potential failure cases must be well examined.

In connection with my previous comments, since there are no theoretical reasoning for setting the prior sample size, I would be very curious as to what kind of results we would get when for authors' method we used all the available data to learn the hyperparameters, and use samples from the 'prior' to conduct inference. Another informative experiment would be to use more uninformative priors such as N(0, 10), so that we can see whether just using more uninformative priors can produce similar results to authors' empirical Bayesian approach.

I also think that the authors' work would benefit a more nuanced discussion of their selection of the DSWD. We do not necessarily know what kind of probabilistic inference this optimization method implicitly corresponds to. Since this is a very challenging issue, this would be fine as long as the authors discuss why DSWD would be preferable over another sample based metric of distributional distance. 
limitations_and_societal_impact: For limitations see my comments above. I do not think the paper requires a societal impact discussion - at least in its current incarnation.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors outline 3 limitations of BAEs: 1) lack of generative modeling 2) inference intractability 3) picking good priors over model parameters. They address 1 with latent space density estimation, 2 with MCMC methods, and 3 by replacing the KL between the BAEs induced distribution and the data generating distribution with the distributional sliced-Wasserstein distance.
main_review: ### Originality:
- To the best of my knowledge, the proposed objective is novel.

### Quality:
- The continuous Bernoulli is meant to approximate the Bernouli and is appropriate to model (approximately) continuous data in [0,1] with one critical assumption: most of the data is approximately either 0 or 1. Indeed, the continuous Bernoulli can only achieve modes of either 0 or 1. This assumption is true for MNIST, where most pixels are either near 0 or 1. However, using this distribution for grayscale images and/or RGB images where one reasonably expects modes other than 0 and 1 seems drastically inappropriate. Could the author(s) please discuss?
- Why is figure 1 significant? Using a N(0,1) prior on weights amounts to randomly initializing a neural network. What is the expectation here? Ridge regression places N(0,1) on linear coefficients, yet one would never expect anything sensible by sampling coefficients from N(0,1). I think I am missing something. Can the author(s) help me out here?

### Clarity:
- The paper is generally well written.

### Significance:
I am deeply concerned by the choice of likelihood (the continuous Bernoulli). Looking at figure 6's reconstructions for the baseline methods, one can tell pixel intensities for each RGB are close to 0 and 1, which limits the observed colors to red, blue, green, magenta (red + blue), yellow (red + green), cyan (blue + green), white, and black. The proposed method seemingly does a good job of overcoming likelihood misspecification (i.e. interpolating between these 8 color modes). But it must be asked: do the performance benefits hold when the likelihood model is more appropriately selected? If not, I am concerned the complexity of this approach is unwarranted. In other words, does integrating model parameters over a well-selected prior simply provide the proposed methods a way to interpolate between the colors available under the likelihood? If the author(s) could show a tangible benefit for a more appropriate likelihood model, then I would be willing to increase my score significantly. 
limitations_and_societal_impact: The author(s) adequately discuss limitations and potential negative societal impact in the last paragraph of their article.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The goal of this paper is to train a Bayesian autoencoder (BAE), where the AE parameters are specified as random variables with trainable hyperparameters. Three contributions are claimed: (1) the distributional sliced-Wasserstein (DSW) distance [25] is leveraged to enable sample-based training of BAE hyperparameters (which model the distribution of AE parameters); (2) to instantiate the AE parameters, the SGHMC [9] is used; (3) to sample from a BAE, the authors propose to learn an ex-post density to model the distribution of the projected latent codes, mimicking [12,8,16]. Experiments on MNIST, FREY-YALE, and CELEBA are conducted.
main_review: Originality. Existing techniques are combined to train BAEs. However, the presented method does not seem to work much better than existing methods. Related work is believed adequately cited.

Quality. The presented method might not be sound. 
It's not rigorous/right to define p(x|w) as in Eq (2), because the lambda is encoded with x and there is a cycle within the definition. Similarly for Eq (3); rigorously, this is not a marginal distribution, which means that it might not be right to minimize the DSW distance between pi and p_phi (in Section 3.2).
The experimental results are not strong enough to support it. Besides, the demonstrated results are not convincing and there might be bugs in the code; for example, the reconstructed/generated images show distinct hue characteristics from the training images, which might be caused by the batch-norm layers in the model.

Clarity. Overall, the submission is not in a good shape. It's challenging to figure out what's actually done. Please refer to other comments below.

Significance. The main research direction of Bayesian neural networks is very important. However, the presented method is not convincing.

Other comments:

The word "prior" is confusing; sometimes it means the prior for AE parameters, while sometimes it means the prior for the projected latent codes.

In the Inference paragraph starting from Line 109, it's not clear at all that the inference is to instantiate the AE parameters w from BAE hyperparameters. Similarly for the reasons why we need to do inference.

It's not clear what are the main novelties, as most of the equations in Sections 2 and 3 are from existing papers. 

In Lines 200-202, since the sampling is enabled after training (i.e., via the ex-post density estimation of projected latent codes), how do you sample from p_phi (x) for training with DSW distance?

The experimental settings of Section 4.1 are very confusing. I spent a lot of time here, but I failed to understand those settings. For example, what's the meaning of "tuning the prior?"
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The author(s) face the design of Bayesian autoencoders (BAE) and, in particular, the model selection problem. A likelihood model is proposed, then to optimize it the prior of the weights is also needed, a normal distribution is proposed where mean and variances are to be learned. Marginal likelihood estimation is brought here to learn these hyperparameters, but being intractable is transformed into an approximated expression using distributional sliced-Wassersteing distance (DSWD). This expression is estimated with Monte Carlo integration and optimized via gradient descent. In the end, the prior is found by matching the true data distribution and the parametrized marginal distribution of the joint probability of the data and the weights.

The contribution of the manuscript is to develop the mathematical framework to optimize the hyperparameters of a prior for the weights in an autoencoder using DSWD. The solution is designed for a normal prior with to-be-learned mean and variance and continuous Bernoulli likelihood for the probability of the data conditioned to the weights.
main_review: Although the paper deals with quite involved expressions and methods, it is well written and can be followed. The proposal is explained step by step, with some summaries. Details have been included in the supplementary materials.

The efforts to make the proposal work are remarkable, and results in the experiments are of interest. Convergence times are affordable, see Fig. 3. Results on test log-likelihood, FID and generated samples are reported. In particular, generated samples are of high quality. And results such as Figure 6 in the supplementary material are interesting.

Overall I find the proposal principled, of quality and significant.
limitations_and_societal_impact: They have addressed this issue. Being a general approach to learn how to generate samples, it has not immediate societal impact.
review_title: null

Meta-review: This paper introduces a model selection method for Bayesian autoencoders, based on a Bayesian formulation. Learning is achieved by optimization of the distributional sliced-Wasserstein distance (DSWD) and posterior estimation is carried out via stochastic gradient Hamiltonian Monte Carlo, useful for ​representation learning with uncertainty. 

The paper has been perceived quite positively. The address challenges in working with BAE 1) lack of generative modeling 2) inference intractability 3) picking a prior over model parameters. They address 1 with latent space density estimation, 2 with MCMC methods, and 3 by optimizing the distributional sliced-Wasserstein distance of BAEs induced distribution and the data generating distribution. The authors also provide solid and convincing experimental results.

However, there is also a potential misleading problem with the notation, that is related to the issue raised by Reviewer 7aYu, the apparent cyclic definition in the probabilistic model. In this respect, it is important to highlight that an autoencoder (or its Bayesian treatment BAE) should be viewed as a supervised models This is in contrast to a VAE where the encoder is an inference distribution so strictly speaking, the statistical model is the decoder only. 

To highlight the supervised nature of the problem, one remedy could be to introduce $y$ as a supervised target
and let $y_n = x_n$.  Eq (1) can be expressed in the familiar form as in regular Bayesian NNs:
$p(w|x,y) = p(y|w,x) p(w) / p(y|x)$.

Starting with this notation seems to solve the potential confusion (raised as a concern by reviewer 7aYu) that I feel is not fully resolved during the discussion and must be addressed in the final manuscript. I suggest that the authors do not hide the notation and clearly define the Bayesian hierarchical model, and explicitly spell out the conditional in (2). 
It could be also helpful to shift the discussion in the experimental section, that starts from line 214 about the key differences between a VAE to the introduction. 

I believe that the authors can address this in the final manuscript, hence I am suggesting acceptance as a poster, acknowledging positive points raised by the reviewers. 

Decision: Accept (Poster)