Model Selection for Bayesian Autoencoders
Authors: Ba-Hien Tran
Simone Rossi
Dimitrios Milios
Pietro Michiardi
Edwin V Bonilla
Maurizio Filippone
Venue: NeurIPS 2021 Poster (2021)
Keywords: Bayesian methods
Probabilistic models
Model selection
Bayesian Autoencoders

Abstract: We develop a novel method for carrying out model selection for Bayesian autoencoders (BAEs) by means of prior hyper-parameter optimization. Inspired by the common practice of type-II maximum likelihood optimization and its equivalence to Kullback-Leibler divergence minimization, we propose to optimize the distributional sliced-Wasserstein distance (DSWD) between the output of the autoencoder and the empirical data distribution. The advantages of this formulation are that we can estimate the DSWD based on samples and handle high-dimensional problems. We carry out posterior estimation of the BAE parameters via stochastic gradient Hamiltonian Monte Carlo and turn our BAE into a generative model by fitting a flexible Dirichlet mixture model in the latent space. Thanks to this approach, we obtain a powerful alternative to variational autoencoders, which are the preferred choice in modern application of autoencoders for representation learning with uncertainty. We evaluate our approach qualitatively and quantitatively using a vast experimental campaign on a number of unsupervised learning tasks and show that, in small-data regimes where priors matter,  our approach provides state-of-the-art results, outperforming multiple competitive baselines.

Model Selection for Bayesian Autoencoders
Ba-Hien Tran EURECOM (France)
Simone Rossi EURECOM (France)
Dimitrios Milios
EURECOM (France)
Pietro Michiardi
Edwin V. Bonilll
Maurizio Filippone EURECOM (France)
EURECOM (France)
CSIRO's Data61
The Australian National University
The University of Sydney (Australia)
Abstract
We develop novel method for carrying model selection for Bayesian autoen- coders (BAEs) by means of prior nyper-parameter optimization. Inspired by the common practice of type-Il maximum likelihood optimization and its equivalence to Kullback-Leibler divergence minimization, we propose to optimize the distribu- tional iased--aasersttenn distance (DSWD) between the output of the autoencoder and the empirical data distribution. The advantages of this formulation are that we can estimate the DSWD based on samples and handle nigh-dimensional problems. We carry out posterior estimation of the BAE parameters via stochastic gradient Hamiltonian Monte Carlo and turn our BAE into generative model by fitting a flexible Dirichlet mixture model in the latent space. Consequently, we obtain a powerful alternative variational autoencoders, which are the preferred choice modern applications of autoencoders for representation learning with uncertainty. We evaluate our approach qualitatively and quantitatively using vast experimental campaign number of unsupervised learning tasks and show that, in small- data regimes where priors matter, our approach provides state-of-the-art results, outperforming multiple competitive baselines.
Introduction
The problem of learning useful representations of data that facilitate the solution of downstream tasks such clustering, generative modeling and classification, is at the crux of the success of many machine learning applications [see, e.g., 5, and references therein]. From a plethora of potential solutions to this problem, unsupervised approaches based autoencoders [10] are particularly appealing as, by definition, they do not require label information and have proved effective in tasks such dimensionality reduction and information retrieval [20]. Autoencoders are neural network models composed of two parts, usually referred as the encoder and the decoder. The encoder maps input data to set of lower-dimensional latent variables. The decoder maps the latent variables back to the observations. The bottleneck introduced by the low-dimensional latent space is what characterizes the compression and representation learning capabilities of autoencoders.
not surprising that these models have connections with principal component analysis [4], factor analysis and density networks [33], and latent variable models [28].
In applications where quantification of uncertainty is primary requirement where data is scarce, it is important to carry out a Bayesian treatment of these models by specifying prior distribution over their parameters, i.e., the weights of the encoder/decoder. However, estimating the posterior
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

distribution over the parameters of these models, which we refer to as Bayesian autoencoders (BAES), generally intractable and requires approximations. Furthermore, the need specify priors for large number of parameters, coupled with the fact that autoencoders are not generative models, has motivated the development of Variational Autoencoders (VAES) as an alternative that can overcome these limitations [25]. Indeed, VAES have found tremendous success and have become one of the preferred methods in modern machine-learning applications [see, e.g., 26, and references therein].
To recap, three potential limitations of BAES hinder their widespread applicability in order to achieve a similar or superior adoption their variational counterpart: (i) lack of generative modeling capabilities; (ii) intractability of inference and (iii) difficulty of setting sensible priors over their parameters. In this work we revisit BAES and deal with these limitations in a principled way. In particular, we address the first limitation in (i) by employing density estimation in the latent space. Furthermore, we deal with the second limitation in (ii) by exploiting recent advances in Markov chain Monte Carlo (MCMC) and, in particular, stochastic gradient Hamiltonian Monte Carlo (SGHMC) [9]. Finally, we believe that the third limitation (iii), which we refer to as the difficulty of carrying out model selection, requires more detailed treatment because choosing sensible priors for Bayesian neural networks is an extremely difficult problem, and this is the main focus this work.
Contributions. Specifically, in this paper we provide novel, practical, and elegant way of per- forming model selection for BAEs, which allows us to revisit these models for applications where VAFS are currently the primary choice. We start by considering the common practice of estimating prior Jtapae--parameters via type-Il maximum likelihood, which is equivalent to minimizing the Kullback-Leibler divergence (KL) between the distribution induced by the BAE and the data generat- ing distribution. Because of the intractability of this objective and the difficulty to estimate it through samples, we resort to an alternative formulation where we replace the KL with the distributional sliced- Wasserstein distance (DSWD) between these two distributions. The advantages of this formu- lation are that large dimensional problems. Once BAE distribution over the BAE parameters via SGHMC [9]. which is a powerful sampler that operates on mini-batches and has proven effective for Bayesian deep/convolutional networks [45, 52, 23). Furthermore, turn our BAE into a generative model by fitting flexible mixture model in the latent space, namely the Dirichlet Process Mixture Model (DPMM). We evaluate our approach qualitatively and quantitatively using a vast experimental campaign on a number of unsupervised learning tasks, with particular emphasis on the challenging task of generative modeling when the number of observations is small.
estimate the DSWD based on samples and, thanks to the slicing, we can handle
are optimized. we estimate the posterior
2 Preliminaries on Bayesian Autoencoders
An autoencoder (AE) is a neural network parameterized by a set of parameters w, which transforms an unlabelled dataset, def {x, ) into set of reconstructions f def {f, J.N. with x,, f, R An AE is composed of two parts: (1) an encoder fene which maps an input sample x,, to a latent code Zn E K K D; and (2) decoder faee which maps the latent code a reconstructed datapoint f.. In short, f = f(x; w) = (face o (x), where we denote w == {Wene, 3 the union of parameters of the encoder and decoder.
The Bayesian treatment of AES dictates that a prior distribution p(w) is placed over all parameters of fene and fae: and the posterior is inferred using Bayes' rule. For the sake of presentation. can consider a BAB as a classic Bayesian neural network (BNN) for a supervised learning task [34, 37], where we have labels y def (yn J.N. associated with each input point x,. From this perspective, can write the posterior on w as follows:
ply| w. w.x))(()
p(w yy.)
ply
where p(y w, is the likelihood defined by the network architecture and the denominator p(y x) constitutes the marginal likelihood. In order to keep the notation uncluttered, we can drop the explicit dependency on x as input, which leads to:
p(w ly) ply w w)p(w)
(2)
ply)
2

---

We observe that for an AE the labels y are the same as the datapoints x, meaning that the likelihood computed in x.
Likelihood model. Before giving an in-depth treatment on priors for BAES in the next section, we briefly discuss the likelihood, which can be chosen according to the type of data. Firstly, we assume factorization of the likelihood on the datapoints, i.c. p(y w) II, p(y,, w). Secondly, given that our experiments mainly focuses on image datasets where pixel values are normalized in the [o, 1] range, we will use the continuous Bernoulli distribution [31]:
D plyn w) = II K(AN) A.)
ynit =
plyn (f,),
(3)
where (A,) is a properly defined normalization constant [31] and X = fi =n. E (0, 1] is the i-th output from the BAE given the imput Xn- We note that, as fn depends deterministically on w, we will use the above expression to refer both p(y,, w) and p(y,. f,). where the latter term will be of crucial importance when we define the functional prior induced over the reconstruction f. Finally, we remark that in the Bayesian scheme both the prior and likelihood are modeling choices. In fact, in our experiments we explore an additional likelihood model, namely the truncated Gaussian, and we show how the problem of selecting good priors is orthogonal to the choice of the likelihood.
Inference. Although the posterior over the BAE parameters is analytically intractable, it can be approximated by variational methods or using MCMC sampling. Within the large family of approximate Bayesian inference schemes, SGHMC [9] allows us to sample from the true posterior by efficiently simulating Hamiltonian system [38]. Unlike more traditional methods, SGHMC can scale up to large datasets by relying on noisy but unbiased estimates of the potential energy function U(w)= = -- Em loy w) - log p(w). These can be computed by considering a mini-batch of size M of the data and approximating Σx=1 ply, w) R
N M Σjer, log p(y; [w), where IM is a set of M random indices. More details on SGHMC can be found in the Appendix.
Pathologies of standard priors The choice of the prior is important for the Bayesian treat- ment of any model it characterizes the hy- pothesis space [34, 36]. Specifically for BAES, one should note that placing prior on the pa- rameters of the encoder and decoder has an im- plicit effect on the prior over the network output (i.c. the reconstruction). In addition, the highly nonlinear nature of these models implies that interpreting the effect of the architecture is theo- retically intractable and practically challenging. Several works argue that vague prior such as (O, 1) good enough for some tasks and mod- els, like classification with convolutional neural networks (CNNS) [49].
Ouput with AV 13 Prior
Output with Optimized Prior
Input
MNIST
7
3 !
00D
CELEBA
COD
Figure 1: Realizations sampled from different pri- ors given input image. OOD stands for out-of- distribution.
However, for BAES this not enough, as illustrated in Fig. 1. The realizations obtained by sampling weights/biases from a ((( 1) prior indicate that this choice provides poor inductive bias. Meanwhile, by encoding better beliefs via an optimized prior, which is the focus of the next section, the samples can capture main characteristics intrinsic to the data, even when the model is fed with out-of- distribution inputs.
3 Model Selection for Bayesian Autoencoders via Prior Optimization
One of the main advantages of the Bayesian paradigm is that we can incorporate prior knowledge into the model a principled way. Let us assume a prior distribution (ww)) on the parameters of that determine the prior,
the AE network, where now we are explicit on the set of (hyper-)paramett i.e., zb. Specifying the prior is easy, e.g., a Gaussian. Determining the effective functional prior, i.e., the prior over the network output f is not trivial due to the complex nonlinear forms of fenc and faxe.
3

---

which induce a non-trivial effect on the output (functional) prior:
Ps (f) = f(x: w) (w)
(4)
where, as before, f(; w) is the functional output of the BAE. Although (+ cannot be evaluated analytically, it is possible to draw samples from it.
Prior parameterization. The only two requirements needed to design a parameterization for the prior are: to be able to (1) draw samples from it and (2) compute its log-density at any point. The latter is required by many inference algorithms such as SGHMC. We consider a fully-factorized Gaussian prior over weights and biases at layer I:
p(wr) = (wi; Ml o..) p(b:) (brr pi, of))
Notice that, as we shall see in 3 3.2 and 3.3, in order to estimate our prior hyper-pane parameters. we will require gradient back proppppppagttion through the stochastic variables wy and by. Thus, we treat these parameters in deterministic manner by means of the reparameterization trick [42, 25].
3.1 Another route for Bayesian Occam's razor
common way to estimate hyper-parameters (i.e., prior parameters v/) is to rely on the Bayesian Occam's 's razor (a.k.a. empirical Bayes), which dictates that the marginal likelihood (y) should be optimized with respect to tb. There are countless examples where such simple procedure succeeds in practice [see, c.g., 41, 21J. We note however that marginal likelihood maximization for large number of byper-paramettes can suffer from overfitting [41, 40J. Nevertheless, we do not expect significant overfitting issues in our setting, as we focus on data that are characterized by high level of structure (i.e. images). As we have seen, regular choices for the prior completely fail capture the properties of such struttrtt outputs.
The marginal likelihood is obtained by marginalizing out the outputs f and the model parameters w.
pv(y) = p(y (f)df,
where ply f) and Pvr (f) are given by Eq. 3 and Eq. 4, respectively. Unfortunately, in our context it is impossible to carry out this optimization due to the intractability of Eq. 6.
Classic results in the statistics literature draw parallels between maximum likelihood estimation (MLE) and KL minimization [2],
/ m(())
T(y) dy
arg max
((yy)logPeel log 1 arg min +
(yy) log pep (y)
+
kI((T(y) p (x))
where π (y) ) is the true data distribution. This equivalence provides us with an interesting insight on an alternative view of marginal likelihood optimization as minimization of the divergence between the true data distribution and the marginal (y)).:
This alternative view allows one to obtain a viable optimization strategy that relies estimate of the data distribution (y)). This presents additional challenges however, as the empirical evaluation and optimization of KI. divergences remains a well-known challenging problem [13]. Although it is possible to evaluate KL (or any other f f-divergence) empirically by leveraging results from convex analysis [39], have opted to substitute KL with alternative metric that is more convenient from a computational perspective. We are inspired by recent works on generative adversarial networks [3, 16] and Bayesian neural networks [45], where is shown that the Wasserstein distance can be estimated efficiently using samples only, even for high-dimensional distributions. We thus employ the Wasserstein distance surrogate for KL divergence, so that avoid the challenges of empirical KL estimation.
empirical
To summarize: (1) we would like to do prior selection by carrying out type-Il MLE; (2) the MLE objective is analytically intractable but the connection with KI. minimization allows us to (3) swap the divergence with the Wasserstein distance, yielding practical framework for choosing priors.

---

3.2 Matching the marginal distribution to the data distribution via Wasserstein distance minimization
Given the two probability measures π and Pus both defined on Rd for simplicity, the p- Wasserstein distance between и and Py is given by
/ |ly
(T
inf
y')ddyy,
(8)
-Er((.pp))
where 1'(T,, Pv) is the set of all possible distributions 7(y, y") such that the marginals are (y) and Pe (y") [47]. While usually analytically unavailable computationally intractable, for D = 1 the distance has a simple closed form solution, that can be easily estimated using samples only [27].
The distributional sliced- Wasserstein distance (DSWD) takes advantage of this result by projecting the estimation of distances for high-dimensional distributions into simpler estimation of multiple distances in one dimension. The projection is done using the Radon transform R. an operator that maps a generic density function yp defined in R.D to the set of its integrals over hyperplanes in RRD.
(t, 0) = (rr))(( - (el(t-rToo)cc Od)rr, AL R, AO E S
where SD-1 is the unit sphere in R and ō(-) is the Dirac delta [17]. Using the Radon transform, for a given direction (or slice) 0 we can project the two densities π and p into one dimension and we can solve the optimal transport problem in this projected space. Furthermore, to avoid unnecessary computations, instead of considering all possible directions in SP-1, DSWD proposes to find the optimal probability measure of slices o (0) on the unit sphere SD-1,
(Fo(o) WP (1,(1,)) Rpp(1, 1/p
DSW ( (T, Pv) == sup
(10)
aEMe
where, for C O, Mc is the set of probability measures a such that Eg,or~a [0 S (a constraint that aims to avoid directions to lie in only one small area). The direct computation of DSW, in Eq. 10 is still challenging but admits an equivalent dual form,
(Ex(o) [WF (Ra (t,, h())), Rp (t, ((O))))) AcTE, ~a ~o [M() ()(O)T(('))))) [O + ACC, (11)
sup keH
where ō is uniform distribution in SD-1, H is the of functions h gD-1 - SP-1 and A is regularization hyper-parameter. The formulation in Eq. 11 is obtained by employing the Lagrangian duality theorem and by reparameterizing (0) as push-forward transformation of uniform measure in SD-1 via h. Now, by parameterizing h using deep neural network with parameters Φ. defined as he. Eq. 11 becomes an optimization problem with respect to the network parameters. The final step is to approximate the analytically intractable expectations with Monte Carlo integration,
{[
K
1/p
max +
Σ
[ (RT(RT(,, (0:)). ho
AC K2
(ho(O,) (o) AoO
K
with 0, ~ (0). Finally, we can use stochastic gradient methods to update & and then use the resulting optima for the estimation of the original distance. We encourage the reader to check the detailed explanation of this formulation, including its derivation and some practical considerations for implementation, available in the Appendíx.
3.3 Summary
We aim at learning the prior on the BAE parameters by optimizing the marginal (y) obtained after integrating out the weights from the joint (y w). The connection with empirical Bayes and KL minimization suggests that we can find the optimal 26" by minimizing the KL between the true data distribution лt (y) and the marginal p (y). However, matching these two distributions is non-trivial due to their high dimensionality and the unavailability of their densities. To overcome this problem, we propose sample-based approach using the distributional sliced 2- Wasserstein distance (Eq. 11) as objective:
26 = = min (pSW)(p((s)
(12)
5

---

This objective function is flexible and does not require the closed-form of either pe (y) or (ty). The only requirement is that we can draw samples from these two distributions. Note that we can sample from (y).). (.y.)b by first computing f after sampling from (w(w) and then perturbing the generated f by sampling from the likelihood p(y [f). For the continuous Bernoulli likelihood this operation can be implemented by using the reparameterization form that allows to backpropagate gradients [31].
Experiments
Competing approaches. We compare our proposal with a wide selection of methods from the literature. For autoencoding methods, we choose the vanilla VAE [25], the B-VAE [19] and WAE (Wasserstein AE) [43]. In addition, we consider models with more complex encoders (VAE +
Sylvester flows [46]). generators (2-stage VAE [11]), and priors (VAE VampPrior [44]). For CELEBA we also include a comparison with Generative Adversarial Networks (GANS), with the vanilla setup of NS-GAN [15, 32] and the more recent DiffAugment-GAr [53, 24). Finally, we also compare against BAE with the standard N (0, 1) prior. Unless otherwise stated, all mmodels-include ours share the same latent dimensionality (K" 50). We defer more detailed description of these models and architectures to the Appendix.
Generative process. Differently from VAES and other methods, deterministic and Bayesian AES are not generative models. To generate new samples with BAES we employ ex-post density estimation over the learned latent space, by fitting a density estimator (z( to (z, E Wese Iy) [S (x;; w )) In this work, we employ a nonparametric model for density estimation based on Dirichlet Process Mixture Model (DPMM) [7], so that its complexity is automatically adapted to the data; see also [6] for alternative ways to turn AES into generative models. After estimating pe (z), a new sample can be generated by drawing neew from (z) and few Ep(wac yy) f (Zuew; Waec)). )-
Evaluation metrics. To evaluate the reconstruction quality, we use the test log- oog-likllhood (LLL), which tells us how likely the test targets are generated by the corresponding model. The predictive log-likelihood is a proper scoring rule that depends on both the accuracy of predictions and their uncertainty [14]. To assess the quality of the generated images, instead, we employ the widely used Fréchet Inception Distance (HID) [18]. We note that, as GANS are not inherently equipped with explicit likelihood model, we only report their FID scores. Finally, all our experiments and evaluations are repeated four times, with different random training splits.
4.1 Analysis of the effect of the prior
To demonstrate the effect of our model selection strategy, we consider scenarios in the small-data regime where the prior might not be necessarily tuned on the training set. In this way are able to impose inductive bias beyond what is available in the training data. We investigate two cases:
MNIST [29]: We use 100 examples of the 0 digits to tune the prior. The training set consists of examples of 1-9 digits, whereas the test set contains 10 000 instances of all digits. We aim to demonstrate the ability of our approach to incorporate prior knowledge about completely unseen data with different characteristics into the model.
FREY-YA YALE [12]: We use 1 956 examples of FREY faces to optimize the prior. The training set and test set are comprised of YALE faces. We demonstrate the benefit of using a different dataset but from the same domain (e.g. face images) to specify the prior distribution.
Visual inspection. Fig. 2 shows some qualitative results (additional images are available in the Appendix), while Fig. 3 shows the convergence of the Wasserstein distance during prior optimization in our proposal. From a visual inspection we see that, on MNIST, by encoding knowledge about the "0" digit into the prior, the BAE can reconstruct this digit fairly well although we only use "1" to "g" digits for inference (differently from the BAE with standard prior). Similarly, on FREY- we see that by encoding knowledge from another dataset in the same domain, the optimized prior can impose a softer constraint compared to using directly this dataset for inference. the union of FREY and YALE faces for training (methods denoted with a *), VAE yields images that are similar to FRBY instead of YALE faces, while generated images from BAE with A/0 1) prior are of lower quality. This again highlights the advantage of our approach to specifying informative prior compared to using that data for training. Another important benefit of our Bayesian treatment of AES is that
addition, if we use directly
quantify the uncertainty for both reconstructed and generated images. The last

---

MNIST =N 200)
FREY-YALE 500)
Reconstructed
Generated
Reconstructed
Generated
Wasserstein dist.
200
8
O
Ground Truth
7
4
0
3
3
3
100
10n
VAE
7 Q 0
0
6
2 2 9
0 2
500
1000 1500 2000
7
0
6
FREY
A00, BAE
(o 1) BAB 700
2
3 8
Wassersten
200-
800
3
100-
ram
BAE
Optim. prior
0
3
8
8
1000 1500 2000 500
Uncortainty
Iteration
Figure Qualitative evaluation for MNIST and YALE. Here, indicates using union of the training data
Figure 3: Convergence of the proposed Wasserstein minimization scheme.
data used optimize prior train the model. The last row depicts standard deviation of reconstructed/generaae images
estimated by BAE using the optimized prior.
MNIST Test (1)
YALE Test (1)
Ablation study on latent dimensionally (MNIST TL)
1900
800
1900
pains
1800
700
1700-
1700
1600
1000 Ios rang powes
200
500
1000
2000
50
100
200
500
10
50
100
10
20
100
Training
Training
latent space
VAE
3-VAE
BAE +V 1) Prior
BAE Optim. Prior
Figure 4: Test log- likelihood (LL) of MNIST and YALE. Left: test LL a function of training size; Right: test LL function of latent dimensionality.
row of Fig. illustrates the uncertainty estimate corresponding to the BAE with optimized prior MNIST and YALE datasets. Our model exhibits increased uncertainty for semantically and visually challenging pixels such as the left part of the second "0" digit image in the MNIST example. We also observe that the uncertainty is greater for generated images compared to reconstructed images as illustrated in the YALE example. This is reasonable because the reconstruction process is guided by the input data rather than synthesizing new data according to a random latent code.
Visualization of inductive bias on MNIST. To have an intuition of the inductive bias induced by the optimized prior, we visualize low-dimensional projection of parame- ters sampled from the prior and the posterior [22]. As we see in Fig. 5, the hypothesis space induced by the (0, 1) prior is huge, compared to where the true solution should lie. Effectively this is another visualization of the famous Bayesian Occam's 's razor plot by David MacKay [35], where the model has very high complexity and poor inductive bi- ases. On the other hand, by considering our proposal to do model selection, the hypothesis space of the optimized prior reduced to regions close to the full posterior. Additional visualizations available in the Appendix.
-00 -50 40 -30
Ni(o, 1) Prior Samples from true postertor Optm. Prior
Figure 5: Visualization in 2D of samples from priors and posteriors of BAE's param- eters. The setup the same before with MNIST.
Quantitative evaluation. For quantitative analysis we rely on Fig. 4, where we study the effect on the reconstruction quality of different training sizes (on the left) and different latent dimensions (on the right). Since we observed that the results of VAE variants are not significantly different, we only show the results for 8-VAE and we leave the extended results to the Appendix. From this experiment can draw important conclusions. The BAE with optimized priors clearly outperforms
7

---

Reconstructios
Gonerated Samples
Test log likelhoodd (11)
Ground Truth
WAE (43)
7000-
VAE [25]
6500
A-VAE [19]
VAE Sylveser Flows (46)
(4)
VAE VampPrior [M]
300-
Stage [11)
200
BAE (0, 1) Prior
BAE Optim. Prior (Ours)
100
NS GAN [15]
DitAugment GAN [53)
sdo
1000
2000
4000
Training size
Figure 6: Qualitative (left) and quantitative evaluation (right) on CELEBA. The markers and bars represent the means and one standard deviations, respectively. In the (left) figure, the sizes of training data and the data for optimizing prior are 500 and 1000, respectively. The higher the log- log-likelihood (LL) and the lower FID the better.
the competing methods (and the BAE with standard prior) in the inference task for all training sizes, with slightly diminishing effect for larger sets, as expected. This pattern also holds when looking at different latent dimensions (Fig, 4, right), where regardless of the dimensionality of the latent space, BAES with optimized priors achieve the best performance.
4.2 Reconstruction and generation of CELEBA
We now look at a more challenging benchmark, the CELEBA dataset [30]. For our proposal, we use 1 000 examples that are randomly chosen from the original training set to learn the prior distribution. The test set consists of about 20 000 images. The goal of this experiment is to evaluate whether sacrificing part of the training data to specify good prior is beneficial when compared to using that data for training the model. Fig. shows qualitative results for the competing methods, their corresponding test LLS and FIDs for different training dataset sizes. In terms of test logllioel (LLs) (Fig. 6, top right), we observe two clear patterns: (i) that BAE approaches perform considerably better than other methods and (ii) the VAE with Sylvester flows performs consistently poor across dataset sizes. This latter observation indicates that having more expressive posterior for the encoder not helpful when considering the small training sizes used in our experiments. More importantly, we see that the BAE using the optimized prior significantly outperforms other methods despite using less data for inference. These results largely agree with the quality of the reconstructions (first column of images in Fig. 6, left) in that BAE methods provide more visually appealing reconstructions when compared to other approaches.
We now evaluate the quality of the generated images (second column of images in Fig. 6, left) along with their FID scores [18]. Visually, it is clear that images generated from VAES (standard, B, Sylvester and Wasserstein Autoencoder (WAH)) are very poor. This failure may originate from the fact that the aggregated posterior distribution of the encoder is not aligned with the prior on the latent space. This problem is more prominent in the case of small training data, where the encoder is not well-trained. The VampPrior tackles this problem by explicitly modeling the aggregated posterior, while 2-stage VAE uses another VAE to estimate the density of the learned latent space. By reducing the effect of the aggregated posterior mismatch, these strategies improve the quality of the generated images remarkably. These results are consistent with their corresponding FID scores (Fig. 6, bottom right) where we also see that BAE using the optimized prior consistently outperforms all variants of VAES and NS-GAN. Finally, we see that DiffAugment- size of 500, yields better FID scores. However, this not surprising as this model uses much more complex network architectures [24], combined with powerful differentiable augmentation scheme.
with the exception of using a training
8

---

VAE (FID: 298.73 = 5.21)
VAE Sylvester Flows (FID: 238.95 16.95)
VAE VampPrior (FID: 127.05 8.18)
2-Stage VAE (71D:87.77 +: 1,01)
BAE with (/, 1) Prior (FID: 84.11 +: 4.09)
BAE with Optim. Prior (FID: 62.75 5.61)
Figure Qualitative and quantitative evaluation of generated samples with the truncated Gaussian likelihood [8]. Here, we use CELEBA samples for inference.
More importantly, it is clear that with few training samples method generates more semantically meaningful images then all other approaches, including DiffAugment-GAN.
The effect of the likelihood. As previously mentioned, in the Bayesian paradigm, the likelihood represents a modeling choice to be made in addition to the choice of the prior. Our empirical evaluation was predominantly conducted with the continuous Bernoulli likelihood. This likelihood maybe not be an ideal choice for colored images because it biases pixel values to the extremes, which results in saturated images. For the CELEBB dataset, we additionally consider the truncated Gaussian likelihood [8], which is another valid alternative for [0, 1 1J-valued data (results summarized in Fig. 7). We observe that indeed the colors in the images generated by this likelihood are more realistic and less saturated compared to those generated by the continuous Bernoulli. Still the problem of selecting a good prior is present, as can be seen for methods like e.g. VAE. For our proposed approach we haven't 't made strong assumptions flexible and not tied a specific choice. These results show that our method is not only quantitatively but also qualitatively better than the competing approaches and confirm that the benefits from framework are independent of the choice of likelihood. In Appendix, we show more qualitative and numerical results of reconstruction and generation with the truncated Gaussian likelihood.
the likelihood (just the ability sample from it) and as such it is
4.3 Prior adjustment versus posterior tempering
We have shown that the proposed framework for adjusting the prior is compatible with standard Bayesian practices, as emulates type-Il maximum likelihood. In other words, the distribution fitting that we induce by means of Wasserstein distance minimization relates to the marginal output of BAES, very much in the same spirit of marginal likelihood maximiza- tion. The distribution is fit considering all possible functions, when marginalized through the likelihood, creating implicit regularization effect. Our scheme does not give more weight to particular training instances. but it simply restricts the hypothe- sis space. This is unlike posterior tempering [51, 22, 48, 1, 50], which is commonly defined (r(w y) p(y [w p(w), where 7 > 0 a temperature value. With known to improve performance in the case of small training data and using miss-specified priors, but it corresponds to ar- tificially sharpening the posterior by over-counting the data 7 times.
Model selection ve posterior tempering
Teet pee.prediion vesterce
100 1000
10000 100000
10-
100
500 1000
soob sóooa 50000
Effective num. data points
Figure 8: Average test predictive vari- ance function of the number of data points used to optimize the prior, and the temperature (i.e. how many times the data points are over-counted).
1, tempering is

---

Performance diferret temperatures
To demonstrate the differences with our proposal, setup a comparison on MNIST. the empirical comparison of Fig. 8, we consider different temperatures and different sets of data points used to optimize the prior. As expected, the tempered posterior quickly collapses the mode, while the posterior after our treatment retains a sufficiently constant variance, regardless of the number of data points used. It is also interesting to notice that with the (0,, 1) prior, the best temperature is т 0.1, while for our approach that optimizes the prior is = 1, further confirming that the model now is well specified (Fig. 9).
1600
Test
1500
0.1 0.5
BAE + (OD 1) Prior BAE Optim. Prior
Figure 9: Test performance for temper- ature scaling with different priors. The dotted lines indicate the best performance.
5 Conclusions
In this work, we have reconsidered the Bayesian treatment of autoencoders (AE) in light of recent advances in Bayesian neural networks. We have addressed the main challenge of BAES, so that they can be rendered as viable alternative to generative models such as VAES. More specifically, we have found that the main limitation of BAES lies in the difficulty of specifying meaningful priors in the context of highly-structured data, which is ubiquitous in modern machine learning applications. Consequently, we have proposed to specify priors over the autoencoder weights by means of a novel optimization of prior hyper-parameters. Inspired by connections with marginal likelihood optimization, we derived a practical and efficient optimization framework, based on the minimization of the distributional sliced-Wasserstedd Wasserstein distance between the distribution induced by the BAE and the data generating distribution. The resulting to perform model selection for BAES, and we showed its advantages in extensive experimental campaign.
optimization strategy leads to a novel way
Limitations and ethical concerns. Even if theoretically justified and empirically verified with extensive experimentation, proposal for model selection still remains proxy to the true marginal likelihood maximization. The DSWD formulation has nice properties and computational tractability, but it may represent only of the possible solutions. At the same time, we stress that the current literature does not cover this problem of BAES at all, and we believe approach is a considerable step towards the development of practical Bayesian methods for representation learning in modern applications characterized by large-scale structured data (including tabular and graph data, which currently covered). At the same time, the accessibility to these models to a wider audience and different kind of data might help to widespread harmful applications, which concern shared among all generative modeling approaches. An ethical analysis of the consequences of Bayesian priors in unsupervised learning scenarios is also worth an in-depth investigation, which goes beyond the scope of this work.
asymptotic convergence
Acknowledgments and Disclosure of Funding
MF gratefully acknowledges support from the AXA Research Fund and the Agence Nationale de la and
Recherche (grant
References
[1] L. Aitchison. A Statistical Theory of Cold Posteriors in Deep Neural Networks. In International Conference Learning Representations, 2021.
[2] H. Akaike. Information Theory and an Extension of the Maximum Likelihood Principle. In 2nd Inter- national Symposium on Information Theory, 1973, pages 268-20 281. Publishing House of the Hungarian Academy of Sciences, 1973.
[3] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein Generative Adversarial Networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, volume 70 of Proceedings of Machine Learning Research, pages 214-223 Sydney, NSW, Australia, 6-11 August 2017. PMLR.
10

---

[4] P. Baldi and K. Hornik. Neural Networks and Principal Component Analysis: Learning from Examples Without Local Minima. Neural networks, 2(1):53-58, 1989.
[5] Y. Bengio, A. Courville, and P. Vincent. Representation Learning: A Review and New Perspectives. JEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798 1828, 2013.
[6] Y. Bengio, L. Yao, G. Alain, and P. Vincent. Generalized Denoising auto--Encderr as Generative Models. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 899-907, 2013.
[7] D. M. Blei and M. Jordan. Variational Inference for Dirichlet Process Mixtures. Bayesian Analysis, 1():121 143, 2006.
[8] J. Burkardt. The Truncated Normal Distribution. Department of Scientific Computing Website, Florida State University, pages 1-35, 2014.
[9] T. Chen, E. Fox, and C. Guestrin. Stochastic Gradient Hamiltonian Monte Carlo. In Proceedings of the 31st International Conference on Machine Learning, ICML 2014, Proceedings of Machine Learning Research, pages 1683-1691, Bejing, China, 22 24 Jun 2014. PMLR.
[10] G. W. Cotrrell, P. Munro, and D. Zipser. Image Compression by Back Propagation: A Demonstration of Extensional Programming. Models of Cognition, pages 208-240, 1989.
[11] B. Dai and D. Wipf. Diagnosing and Enhancing VAF Models. In International Conference Learning Representations. 2019.
[12] Z. Dai, A. C. Damianou, J. González, and N. D. Lawrence. Variational Auto-encoded Deep Gaussian Processes. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[13] D. Plam-Shepherd, Requeima, and D. Duvenaud, Mapping Gaussian Process Priors Bayesian Neural Networks. In NeurIPS workshop on Bayesian Deep Learning, 2017.
[14] T. Gneiting and A. F. Raftery. Strietly Proper Scoring Rules, Prediction, and Estimation. Journal of the American statistical Association, 102(477):359-375 2007.
[15] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Nets. Advances Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[16] Gulrajani, Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved Training of Wasserstein GANs. In Advances in Neural Information Processing Systems, volume 30, pages Associates, Inc., 2017.
Curran
[17] S. Helgason. Integral Geometry and Radon Transforms. Springer Science Business Media, 2010.
[18] Heusel, H. Ramsauer, Unterthiner, B. Nessler, and Hochreiter. GANs Trained by Two Time- Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6626-6637, 2017.
[19] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. VeE: Learning Basic Visual Concepts with a Constrained Variational Framework. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24- 2017, Conference Track Proceedings. OpenReview.net, 2017.
[20] G. E. Hinton and R. R. Salakhutdinov. Reducing the Dimensionality of Data with Neural Networks. Science, 313(5766):504-5000 2006.
[21] A. Immer, M. Bauer, V. Fortuin, G. Rätsch, and M. F. Khan, Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning. In Proceedings of the 38th International Conference on Machine Learning, ICMI. 1021, 18-24 July 2021, Virtual Event, volume 139 Proceedings of Machine Learning Research, pages 4573. PMLR, 2021.
[22] P. Izmailov. W. Maddox, Kirichenko, Т. Garipov, D. Vetrov, and G. Wilson. Subspace Inference for Bayesian Deep Learning. In Proceedings of the Shirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, volume 115 of Proceedings of Machine Learning Research, pages 1169 1--111177 1179. AUAI Press, 2019.

---

[23] P. Izmailov, S. Vikram, M. D. Hoffman, and A. G. Wilson. What Are Bayesian Neural Network Posteriors Really Like? In Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 June 2021, Virtual Event, 2021.
[24] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and Improving the Image Quality StyleGAN. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 8107-8116. IEEE, 2020.
[25] D. Kingma and M. Welling. Auto- Buto-Ene Variational Bayes. In International Conference on Learning Representations, 2014.
[26] D. Kingma and M. Welling. An Introduction to Variational Autocncoders. Foundations and Trends Machine Learning, 12(4):307-39) -3027 2019.
[27] S. Kolouri, K. Nadjahi, U. Simsekli, R. Badeau, and Rohde. Generalized Sliced Wasserstein Distances. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 261-272, 2019.
[28] N. D. Lawrence. Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models. Journal of Machine Learning Research, 6:1783-1816, 2005.
[29] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- Learning Applied to Document Recognition. Proce Proceedings of the IEEE, 86( 1):2278-2324, 1998.
[30] 7. Liu, P. Luo, X. Wang, and X. Tang. Deep Learning Face Attributes in the Wild, In 2015 IEEE International Conference on Computer Vision, 2015 Santiago, Chile, December 7-13, 2015, pages 3730-3738. IEEE Computer Society, 2015.
[31] G. LLaiza--Gaeen Ganem and J. P. Cunningham. The Continuous Bernoulli: Fixing Pervasive Error in Variational Autoencoders. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8 8-14, 2019, Vancouver, BC, Canada, pages 13266-13333 13276, 2019.
[32] M. Lucic, K. Kurach, M. Michalski, Gelly, and O Bousquet. Are GANs Created Equal? A Large- Scale Study. In Advances in Neural Information Processing Systems 31: Anmual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 698-707, 2018.
[33] D. J. MacKay and M. N. Gibbs. Density Networks. Statistics and Neural Networks: Advances at the Interface, pages 129-144, 1999.
[34] D. J. C. Mackay. Bayesian Methods for Adaptive Models. PhD thesis, California Institute of Technology, USA, 1992. UMI Order No. GAX92-32200.
[35] D. J. C. Mackay. Information Theory, Inference and Learning Algorithms. Cambridge University Press, first edition edition, June 2003.
[36] I. Murray and 7. Ghahramani. A Note on the Evidence and Bayesian Occam's 's Razor. Technical Report GCNU-TR 2005-003, Gatsby Computational Neuroscience Unit, University College London, 2005.
[37] R. M. Neal. Bayesian Learning for Neural Networks (Lecture Notes in Statistics). Springer, edition, Aug. 1996.
[38] R. M. Neal. MCMC Using Hamiltonian Dynamics, chapter 5. CRC Press, 2011.
[39] X. Nguyen, M. J. Wainwright, and M. I. Jordan. Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk Minimization. IEEE Transactions on Information Theory, 56(11):5847-533) -861, 2010.
[40] S. Ober, C. E. Rasmussen, and M. van der Wilk. The Promises and Pitfalls Deep Kernel Learning. In Proceedings of the 37th Conference on Uncertainty in Artificial Intelligence, UAI 2021, July 27-30, 2021, Virtual Event, 2021.
[41] C. E. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
[42] D. Rezende, S. Mohamed, and D. Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, volume 32 of Proceeding of Machine Learning Research, pages 1278-1286, Beijing, China, 21-26 June 2014. PMLR.
12

---

[43] I. Tolstikhin, O. Bousquet, S. Gelly, and B. Schoelkopf. Wasserstein Auto-Encoders. In International Conference Learning Representations, 2018.
[44] J. M. Tomczak and M. Welling. VAE with VampPrior. In International Conference Artificial Intelligence and Statistics, AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, volume 84 of Proceedings of Machine Learning Research, pages 1214-1223. PMLR, 2018.
[45] B.-H. Tran, S. Rossi, D. Milios, and M. Filippone. All You Need Good Functional Prior for Bayesian Deep Learning, arXiv preprint Xiv::v20
2020.
[46] R. den Berg, L. Hasenclever. J. M. Tomezak, and M. Welling, Sylvester Normalizing Flows for Varia- tional Inference. In Proceedings of the Thirty- Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pages 393-402. AUAI Press, 2018.
[47] C. Villani. Optimal Transport: Old and New, volume 338, Springer Science & Business Media, 2008
[48] F. Wenzel, K. Roth, B. S. Veeling, J. Swiatkowski, L. Tran, S. Mandt, J. Snoek, T. Salimans, R. Jenatton, and S. Nowozin. How Good is the Bayes Posterior in Deep Ncural Networks Really? In Proceeding of the
37th International Conference on Machine Learning, ICML 2020, Virtual, 25-30 June 2020.
[49] A. G. Wilson and P. Izmailov. Bayesian Deep Learning and a Probabilistic Perspective of Generalization. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[50] C. Zeno, I. Golan, A. Pakman, and D. Soudry. Why Cold Posteriors? On the Suboptimal Generalization of Optimal Bayes Estimates. In Third Symposiun on Advances Approximate Bayesian Inference, 2021.
[51] G. Zhang, S. Sun, D. Duvenaud, and R. B. Grosse. Noisy Natural Gradient Variational Inference. In Proceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmässonn Stockholm, Sweden, July 10-15, 2018, volume 80 Proceedings of Machine Learning Research, pages 5847-5856. PMLR, 2018.
[52] R. Zhang, C. Li, J. Zhang, C. Chen, and A. G. Wilson, Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[53] S. Zhao, Z. Liu, J. Lin, J. Zhu, and S. Han. Differentiable Augmentation for Data-Efficient GAN Training. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual, 2020.
13


Reviews:
[{"review_id": "TsUEvEtXjv", "reviewer": "Reviewer_7aYu", "summary": "The goal of this paper is to train a Bayesian autoencoder (BAE), where the AE parameters are specified as random variables with trainable hyperparameters. Three contributions are claimed: (1) the distributional sliced-Wasserstein (DSW) distance [25] is leveraged to enable sample-based training of BAE hyperparameters (which model the distribution of AE parameters); (2) to instantiate the AE parameters, the SGHMC [9] is used; (3) to sample from a BAE, the authors propose to learn an ex-post density to model the distribution of the projected latent codes, mimicking [12,8,16]. Experiments on MNIST, FREY-YALE, and CELEBA are conducted.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality. Existing techniques are combined to train BAEs. However, the presented method does not seem to work much better than existing methods. Related work is believed adequately cited.\n\nQuality. The presented method might not be sound. \nIt's not rigorous/right to define p(x|w) as in Eq (2), because the lambda is encoded with x and there is a cycle within the definition. Similarly for Eq (3); rigorously, this is not a marginal distribution, which means that it might not be right to minimize the DSW distance between pi and p_phi (in Section 3.2).\nThe experimental results are not strong enough to support it. Besides, the demonstrated results are not convincing and there might be bugs in the code; for example, the reconstructed/generated images show distinct hue characteristics from the training images, which might be caused by the batch-norm layers in the model.\n\nClarity. Overall, the submission is not in a good shape. It's challenging to figure out what's actually done. Please refer to other comments below.\n\nSignificance. The main research direction of Bayesian neural networks is very important. However, the presented method is not convincing.\n\nOther comments:\n\nThe word \"prior\" is confusing; sometimes it means the prior for AE parameters, while sometimes it means the prior for the projected latent codes.\n\nIn the Inference paragraph starting from Line 109, it's not clear at all that the inference is to instantiate the AE parameters w from BAE hyperparameters. Similarly for the reasons why we need to do inference.\n\nIt's not clear what are the main novelties, as most of the equations in Sections 2 and 3 are from existing papers. \n\nIn Lines 200-202, since the sampling is enabled after training (i.e., via the ex-post density estimation of projected latent codes), how do you sample from p_phi (x) for training with DSW distance?\n\nThe experimental settings of Section 4.1 are very confusing. I spent a lot of time here, but I failed to understand those settings. For example, what's the meaning of \"tuning the prior?\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "TPtb1tr3fDe", "reviewer": "Reviewer_r735", "summary": "The author(s) face the design of Bayesian autoencoders (BAE) and, in particular, the model selection problem. A likelihood model is proposed, then to optimize it the prior of the weights is also needed, a normal distribution is proposed where mean and variances are to be learned. Marginal likelihood estimation is brought here to learn these hyperparameters, but being intractable is transformed into an approximated expression using distributional sliced-Wassersteing distance (DSWD). This expression is estimated with Monte Carlo integration and optimized via gradient descent. In the end, the prior is found by matching the true data distribution and the parametrized marginal distribution of the joint probability of the data and the weights.\n\nThe contribution of the manuscript is to develop the mathematical framework to optimize the hyperparameters of a prior for the weights in an autoencoder using DSWD. The solution is designed for a normal prior with to-be-learned mean and variance and continuous Bernoulli likelihood for the probability of the data conditioned to the weights.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the paper deals with quite involved expressions and methods, it is well written and can be followed. The proposal is explained step by step, with some summaries. Details have been included in the supplementary materials.\n\nThe efforts to make the proposal work are remarkable, and results in the experiments are of interest. Convergence times are affordable, see Fig. 3. Results on test log-likelihood, FID and generated samples are reported. In particular, generated samples are of high quality. And results such as Figure 6 in the supplementary material are interesting.\n\nOverall I find the proposal principled, of quality and significant.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They have addressed this issue. Being a general approach to learn how to generate samples, it has not immediate societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns, beyond the future application of generative modeling to non-ethical applications.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "BZ0DRV7xufD", "reviewer": "Reviewer_Lfxg", "summary": "The authors outline 3 limitations of BAEs: 1) lack of generative modeling 2) inference intractability 3) picking good priors over model parameters. They address 1 with latent space density estimation, 2 with MCMC methods, and 3 by replacing the KL between the BAEs induced distribution and the data generating distribution with the distributional sliced-Wasserstein distance.", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality:\n- To the best of my knowledge, the proposed objective is novel.\n\n### Quality:\n- The continuous Bernoulli is meant to approximate the Bernouli and is appropriate to model (approximately) continuous data in [0,1] with one critical assumption: most of the data is approximately either 0 or 1. Indeed, the continuous Bernoulli can only achieve modes of either 0 or 1. This assumption is true for MNIST, where most pixels are either near 0 or 1. However, using this distribution for grayscale images and/or RGB images where one reasonably expects modes other than 0 and 1 seems drastically inappropriate. Could the author(s) please discuss?\n- Why is figure 1 significant? Using a N(0,1) prior on weights amounts to randomly initializing a neural network. What is the expectation here? Ridge regression places N(0,1) on linear coefficients, yet one would never expect anything sensible by sampling coefficients from N(0,1). I think I am missing something. Can the author(s) help me out here?\n\n### Clarity:\n- The paper is generally well written.\n\n### Significance:\nI am deeply concerned by the choice of likelihood (the continuous Bernoulli). Looking at figure 6's reconstructions for the baseline methods, one can tell pixel intensities for each RGB are close to 0 and 1, which limits the observed colors to red, blue, green, magenta (red + blue), yellow (red + green), cyan (blue + green), white, and black. The proposed method seemingly does a good job of overcoming likelihood misspecification (i.e. interpolating between these 8 color modes). But it must be asked: do the performance benefits hold when the likelihood model is more appropriately selected? If not, I am concerned the complexity of this approach is unwarranted. In other words, does integrating model parameters over a well-selected prior simply provide the proposed methods a way to interpolate between the colors available under the likelihood? If the author(s) could show a tangible benefit for a more appropriate likelihood model, then I would be willing to increase my score significantly. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author(s) adequately discuss limitations and potential negative societal impact in the last paragraph of their article. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "4-1F6aUGg6p", "reviewer": "Reviewer_kcwF", "summary": "The authors suggest an alternative probabilistic generative model that assumes a prior distribution on the parameters of a classical autoencoder. The authors use empirical Bayes on a separate set of data to determine hyperparameters, use stochastic gradient HMC for posterior inference, and fit a hierarchical latent variable model to the latent codes in order to be able to sample from the latent space. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this is a well-written paper that constitutes a potentially welcome addition to the increasing arsenal of probabilistic generative models in machine learning. The motivation and presentation is clear, the authors' solution to potential challenges of their methodology is parsimonious, and their results are of interest for other researchers in the field. However, there are also some issues that needs to be resolved before the final decision is made.\n\nFigure 1 summarizes one of my main concerns with the authors' argumentation. From my perspective, this figure demonstrates a result that is completely trivial/expected. Their hyperparameter determination method is an empirical Bayesian approach, so therefore it would of course create a better reconstruction of the data without any training. Actually contrary to authors' presentation, empirical Bayesian approaches are not without their disputes in the field due to their overreliance of the data (the double-dipping problem). So, Figure 1 is not only trivial, but it also represents a potential weakness of the empirical Bayesian methods, which would be overreliance on the data at the expense of generalizability or out-of-distribution performance. Therefore one could expect a sharper decline in reconstruction performance as the data distribution shifts away from the original distribution, or a more drastic loss in local smoothness in the latent space (slight perturbations leading to more drastic changes in the reconstruced output). \n\nThe authors' lapse in the presentation of empirical Bayes methods (contrasting them with the 'pathologies' of standard priors) are accompanied by lack of significant experiments that would probe the potential weaknesses of an empirical Bayesian approach. I would love to see how the authors would decide to test these potential weaknesses of their methods. I am not necessarily in favor or against empirical Bayes methods, but if they are used, their potential failure cases must be well examined.\n\nIn connection with my previous comments, since there are no theoretical reasoning for setting the prior sample size, I would be very curious as to what kind of results we would get when for authors' method we used all the available data to learn the hyperparameters, and use samples from the 'prior' to conduct inference. Another informative experiment would be to use more uninformative priors such as N(0, 10), so that we can see whether just using more uninformative priors can produce similar results to authors' empirical Bayesian approach.\n\nI also think that the authors' work would benefit a more nuanced discussion of their selection of the DSWD. We do not necessarily know what kind of probabilistic inference this optimization method implicitly corresponds to. Since this is a very challenging issue, this would be fine as long as the authors discuss why DSWD would be preferable over another sample based metric of distributional distance. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "For limitations see my comments above. I do not think the paper requires a societal impact discussion - at least in its current incarnation.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a model selection method for Bayesian autoencoders, based on a Bayesian formulation. Learning is achieved by optimization of the distributional sliced-Wasserstein distance (DSWD) and posterior estimation is carried out via stochastic gradient Hamiltonian Monte Carlo, useful for ​representation learning with uncertainty. 

The paper has been perceived quite positively. The address challenges in working with BAE 1) lack of generative modeling 2) inference intractability 3) picking a prior over model parameters. They address 1 with latent space density estimation, 2 with MCMC methods, and 3 by optimizing the distributional sliced-Wasserstein distance of BAEs induced distribution and the data generating distribution. The authors also provide solid and convincing experimental results.

However, there is also a potential misleading problem with the notation, that is related to the issue raised by Reviewer 7aYu, the apparent cyclic definition in the probabilistic model. In this respect, it is important to highlight that an autoencoder (or its Bayesian treatment BAE) should be viewed as a supervised models This is in contrast to a VAE where the encoder is an inference distribution so strictly speaking, the statistical model is the decoder only. 

To highlight the supervised nature of the problem, one remedy could be to introduce $y$ as a supervised target
and let $y_n = x_n$.  Eq (1) can be expressed in the familiar form as in regular Bayesian NNs:
$p(w|x,y) = p(y|w,x) p(w) / p(y|x)$.

Starting with this notation seems to solve the potential confusion (raised as a concern by reviewer 7aYu) that I feel is not fully resolved during the discussion and must be addressed in the final manuscript. I suggest that the authors do not hide the notation and clearly define the Bayesian hierarchical model, and explicitly spell out the conditional in (2). 
It could be also helpful to shift the discussion in the experimental section, that starts from line 214 about the key differences between a VAE to the introduction. 

I believe that the authors can address this in the final manuscript, hence I am suggesting acceptance as a poster, acknowledging positive points raised by the reviewers. 

Decision: Accept (Poster)