Probabilistic Autoencoder
Authors: Vanessa M Boehm
Uros Seljak
Venue: Accepted by TMLR ()

Abstract: Principal Component Analysis (PCA) minimizes the reconstruction error given a class of linear models of fixed component dimensionality. Probabilistic PCA adds a probabilistic structure by learning the probability distribution of the PCA latent space weights, thus creating a generative model. Autoencoders (AE) minimize the reconstruction error in a class of nonlinear models of fixed latent space dimensionality and outperform PCA at fixed dimensionality. Here, we introduce the Probabilistic Autoencoder (PAE) that learns the probability distribution of the AE latent space weights using a normalizing flow (NF). The PAE is fast and easy to train and achieves small reconstruction errors, high sample quality, and good performance in downstream tasks. We compare the PAE to Variational AE (VAE), showing that the PAE trains faster, reaches a lower reconstruction error, and produces good sample quality without requiring special tuning parameters or training procedures. 
We further demonstrate that the PAE is a powerful model for performing the downstream tasks of probabilistic image reconstruction in the context of Bayesian inference of inverse problems for inpainting and denoising applications. Finally, we identify latent space density from NF as a promising outlier detection metric.

Published in Transactions on Machine Learning Research (09/2022)
Probabilistic Autoencoder
Vanessa Böhm
wbochm@berkeleyycoom edu
Berkeley Center for Cosmological Physics
Department of Physics
University of California Berkeley, CA, USA
Laurence Berkeley National Laboratory
Uroš Seljak
useljak-berkkley edu
Berkeley Center for Cosmological Physics
Departmend of Physics
University of California Berkeley, California, USA
Lawrence Berkeley National Laboratory
Reviewed on OpenReview: https: // openreview.net/ hetpeeewwwwwweeeeeoooeeeeee forum? &d id- AEo JuJKVA
Abstract
Principal Component Analysis (PCA) minimizes the reconstruction error given a class of linear models of fixed component. dimensionality. Probabilistic PCA adds a probabilistic structure by learning the probability distribution of the PCA latent space weights, thus creating generative model. Autoencoders (AE) minimize the reconstruction error in class of nonlinear models of fixed latent space dimensionality and out ourperfor PCA al fixed dimensionality. Here, we introduce the Probabilistic Autoencoder (PAE) that learns the probability distribution of the AE latent space weights using a normalizing flow (NF). The PAE is fast and easy to train and achieves small reconstruction errors, high sample quality, and good performance downstream tasks. We compare the PAF Variational AE (VAE), showing that the PAE trains faster, reaches lower reconstruction error, and produces good sample quality without requiring special tuning parameters training procedures. We further demonstrate that the PAE is a powerful model for performing the downstream tasks of probabilistic image reconstruction in the context of Bayesian inference of inverse problems for inpainting and denoising applications. Finally, we identify latent space density from NF as a promising outlier detection metric.
1 Introduction
Deep generative models are powerful machine learning models that can learn complex, higb-dimensional data likelihoods and generate samples from them. Because their probabilistic formulation, generative models are becoming an indispensable tool for scientific data analysis a range of domains including particle physics (Paganini ct al., 2018; Stein al., 2020) and cosmology (Thorne et al., 2021; Reiman et al., 2020).
Variational Autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) among the most popular generative models. VAËs project the data to a lower dimensional latent space and reformulate the data likelihood estimation variational inference problem. Their training objective is the Evidence Lower BOund (ELBO), which approximates the true data likelihood with a variational ansatz from below. VAEs can be built with expressive architectures, enjoy the benefits of regularization through data compression and have a firm theoretical foundation. Different generative adversarial networks (Goodfellow al., 2014), another popular class of generative models, VAEs provide an estimator for the data likelihood and a posterior distribution for the latent variables.

---

Published in Transactions on Machine Learning Research (09/2022)
Despite their popularity, variational autoencoders have well known practical limitations. Successful VAE training requires to find a delicate balance between the two contributing terms to the ELBO: The distortion term, which encourages high quality reconstructions, and the rate term, which controls the sample quality by matching the aggregate posterior with a chosen prior distribution (Alemi et al., 2018). Whether the VAE training process succeeds in striking this balance depends on a number of factors, including the network architectures, the chosen prior and the class of allowed posterior distributions (Hoffman & Johnson, 2016). In some cases, too powerful decoders can decouple the latent space from the input (Bowman al., 2016; Chen et al., 2017) and lead to posterior collapse (van den Oord et al., 2017).
A long list of works have dissected and studied the training behavior of VAEs (Alemi et al., 2018; Hoffman & Johnson, 2016) and suggested modifications to remedy common issues. Many fixes add complexity to the VAE model, e.g. by modifying or annealing the ELBO objective (Bowman et al., 2016; Alemi et al., 2017; Higgins et al., 2017; Makhzani et al., 2015), choosing more expressive posterior distributions (Kingma et al., 2016; Rezende & Mohamed, 2015; Salimans et al., 2015; Tran al., 2016), or using more flexible priors (Bauer& & Mnih, 2019; Chen et. al., 2017; Tomczak & Welling, 2018).
In this work we take different approach. We give up on the variational ansatz that lies at the heart of VAEs and instead suggest a conceptually very simple model with very stable training properties. The Probabilistic Autoencoder (PAE) is motivated by probabilistic principal component analysis (Tipping & Bishop, 1999) and consists of an Autoencoder (AE), which is interpreted probabilistically after training by means of a Normalizing Flow (NF). Both of these components are comparably easy to set up and train and this two-stage set up allows the practitioner optimize their ayper-parameters (model architecture, training procedure, etc.) independently. We claim that the PAE is viable alternative to VAFs despite its conceptual simplicity. We back this claim empirically through ablation studies. Specifically, we compare the performance of the PAE to that of equivalent VAEs in a number of tasks which we think are specifically relevant for practical applications: data compression (reconstruction quality), data generation, anomaly detection and probabilistic data denoising and imputation.
Our primary contributions are: 1) a simple generative model designed with ease-of-use and training mind 2) a quantitative comparison of this model to variational autoencoders, showing that it performs relevant tasks at comparable quality and accuracy without variational inference 3) new anomaly detection metric through NF density estimation in latent space, which is byproduct of the PAE, but can also be used within the VAE framework. We make all of our code publicly available."
2 Motivation: Probabilistic PCA
The probabilistic autoencoder is motivated by linear Principal Component Analysis (PCA) and its proba- bilistic interpretation, probabilistic principal component analysis (Tipping & Bishop, 1999), which provides a PCA-based data likelihood estimate.
A principal component analysis of data x E RN at fixed latent space dimensionality K (K<N) finds the orthogonal linear transformation, O,
O: RRKKK 2 Oz, -0"=1x
2
(1)
that maximizes the data variance in the latent space. Maximizing the variance of the transformed data is equivalent to minimizing the average reconstruction error (the residual variance in data space). The PCA problem can be solved analytically and the principal components are given by the cigenvectors of the data covariance matrix.
A suitable latent space dimensionality, K, is chosen by inspecting the eigenvalues, d, of the data covariance and keeping only the eigenvectors that correspond to the largest eigenvalues. The average reconstruction error that originates from the discarded eigenvalues is - = N
Em= A
The data model under a PCA is
T = Oz +
(2)
2

---

Published in Transactions on Machine Learning Research (09/2022)
where O is constructed from the eigenvectors that correspond to the largest eigenvectors and e is the residual not captured by the PCA transformation.
In probabilistic PCA (PPCA) the residuals are assumed to follow a Gaussian distribution. The implicit likelihood then,
1
In p[]]2)) = - [N (n( + det + (a: 0z) 0z)]
Under the approximation that the reconstruction error is uncorrelated and isotropic, Σ is a diagonal matrix with necon along its diagonal, Σ = oecon In.
The implicit likelihood in equation 3 alone is not yet a probabilistic model for the data. A fully probabilistic structure requires a prior over the latent space. PPCA (Tipping Bishop, 1999) assumes that the latent variables follow a Gaussian distribution with mean zero and covariance A, where is diagonal matrix with the rank-ordered eigenvalues X; along its diagonal.
The Gaussian prior allows us to analytically compute the marginal,
In ((x) - [n + det C + +
(4)
with C = OAO E.
In summary, probabilistic PCA constructs a probabilistic model by first finding a basis which minimizes the reconstruction crror in a class of models, followed by using the probability distribution of the latent variables as the prior. With the probabilistic autoencoder we generalize this approach to non-linear models. The PCA is replaced by an autoencoder trained to minimize the reconstruction error, and the Gaussian ansatz for the prior is replaced by a normalizing flow.
3
The probabilistic autoencoder
3.1 PAE training
In analogy to PPCA the PAE is constructed in two stages. Stage 1 is an autoencoder with encoder and decoder g, both deep neural networks with respective trainable parameters and 0,
f : R.N R x + fa(a), 9o R - R.N 2 + -o
(5)
The training objective of the AE is the reconstruction error Læ-distance,
=AE= = Ep(es || (o(s(()))1
(6)
The autoencoder is not probabilistic model. To construct the PAE, interpret it probabilisticaly with second stage. We approximate the latent space prior, p(=), by performing density estimation on the AE-encoded training data. In PPCA the latent space density is modeled with a Gaussian. The PAE employs a more flexible density estimator for modeling the prior, a normalizing flow.
Normalizing flows (Rippel & Adams, 2013; Dinh al., 2015; 2017; Kingma & Dhariwal, 2018; Grathwohl et al., 2019) address the task of modeling the density distribution p(z) of input data z by introducing a bijective mapping, by(2) = u, from the data to an underlying latent representation u,
by K + RK ヒ u=b(zz
Requiring the latent variables follow a given prior distribution qu),, one can write the modeled data probability density using conservation of probability,
(z) V=b))
(8)
Here, quu) is some simple normalized latent space probability density, usually a Gaussian N/ I), and V (z), is the Jacobian determinant of the transformation b,(2). The NF is parametrized by parameters
3

---

Published in Transactions on Machine Learning Research (09/2022)
Y and training takes the form of maximizing the data likelihood (x(2) with respect y. Equation 8 requires evaluation of the Jacobian. NF architectures have forms for which this is simple and fast. The architectural constraints originating from this requirement also introduce beneficial regularizing properties and prevent overfitring. schematic diagram of the PAE is shown in figure 1.
In the PAE, an NF trained as a mapping from the latent space of the AE, z, to the Gaussian latent space of the normalizing flow, u. The training objective of the NF is the negative log likelihood of the encoded samples, fø()),
LNF Ep(z)| In pn)] Ep(z) -Inp(w) In del 2b (z)
Su u=b,(
The normalizing flow maps the potentially very irregular latent space distribution of the AE to a Gaussian distribution.
To sample from the PAE we draw a sample, w~N(0, 1), from the NF latent distribution and pass it through both the NF and AE generators (left panel in figure 1),
x (b, (u)).
(10)
Just as in PPCA, density estimation the encoded data only provides approximate prior. Formally, for fully probabilistic model, prior would be given by the aggregate posterior,
Pmodel = f f p(z))
Pmode p = F F(()) (
(11)
meaning that the density estimation should be conducted on samples from the posteriors. Since our first stage is a non- pon-probabillitic autoencoder, there is no notion of a posterior. Our choice to fit an approximate prior on the encoded data, however, is not completely unjustified: we know that small reconstruction errors (which can be easily achieved with an AE) generally result in very narrow posteriors centered on latent space points which are well determined through the projection of the data into the latent space. The position of these points are hardly influenced by the prior. By replacing samples from the posterior with the encoded samples, we replace the narrow posteriors by Dirac delta distributions. By fitting on the AE encoded data we approximate the maximum posteriori (MAP) solution with the AE encoded position. This is not a formal mathematical derivation, but can serve as a reasoning for why the PAE is able to compete with fully probabilistic models even in probabilistic lasks.
The generalization and regularization properties of NFs are another important ingredient for enabling the use of approximate MAP positions instead of samples from the posterior. NFs are unlikely to fit delta functions their training points, but smoothly interpolate bet ween them. The success of NF-based density estimation is proof of this: many datasets NE's achieve the highest validation log data likelihoods (Durkan et al., 2019). If the NF does not provide sufficient regularization, it will become apparent overfitting (lower density estimates on validation data), which can be controlled by simplifying the architecture or early stopping.
The AE latent space is usually of relatively low dimensionality, K K&N, which allows for computationally tractable density estimation: the NF models do not require complex deep architectures and are fast to train, which enables efficient syper-parametee optimization.
3.2 Comparison to VAE
Different to the PAE a variational autoencoder is trained on a fully probabilistic objective, the Evidence Lower BOund (ELBO),
ELBO- LVAE (p((z) [[))())))) [==])()))) DKL [=]
(12)
where qá is an approximate, parametrized variational posterior, usually Gaussian with diagonal covariance. Typical choices for the parametrized implicit likelihood po (z)]2) are Bernoulli distribution for binary valued
4

---

Published in Transactions on Machine Learning Research (09/2022)
pl z)
f
9o
l0 ( (x))
p
p(z)
b
1 b y by
u
go
p(u)=N((0.1
Schematic diagram of the PAE (left panel) and an illustration of the sampling procedure from the PAE (right panel). The autoencoder networks are depicted as gray trapezia, the normalizing flow is represented by black arrows and the latent spaces of the autoencoder and normalizing flow are shown in red and blue, respectively.
Figure 1:
data or a diagonal Gaussian distribution for continuous data. The ELBO is guaranteed to bound the true evidence p(x) from below. During the VAE training equation is evaluated stochastically on samples from the approximate posterior. Equation 12 shows that the VAE objective balances the average reconstruction error (the likelihood term or distortion) with the sample quality (the KL term rate). If the former dominates the loss during training, the encoded distribution and prior do not match well. If this is the case, samples from the prior can land outside of the encoded domain resulting in low sample quality. If the KL term dominates, some latent dimensions will solely be used to satisfy the second term and not encode any information about the input data, a problem known as posterior collapse (Alemi et al., 2018). Balancing the two terms can be controlled by an additional paramcters B,
Cy-VAE = - E)(x)) (212)) [In [anpp(())]] - DKL [=<(=|=)|p<<3]]]
(13)
This and related modificat ion are known as 3-VAEs (Bowman et al., 2016; Alemi et al., 2017; Higgins el al., 2017; Makhzani et al., 2015). Training on equation 13 usually involves a grid search in order to find an optimal value for 3 and anncaling schodules.
The PAE optimizes the reconstruction and sample quality individually. Training of stage reaches optimal reconstruction error. The latter is then left unchanged in the training of stage 2, which can focus entirely on matching the latent space distribution. We test in our experiments whether this procedure results in an advantage in reconstruction error and sample quality. A practical advantage of this procedure is that it facilitates the prper-parameter search over model architecture and training schedule. Instead of having to iterate over encoder prroder/decoder, decoder and flow architecture and the balance between rate and distortion term, each step can be optimized individually and towards single objective.
For comparisons between VAE and PAE to be fair, we allow the VAE prior, which is typically a standard normal distribution, to be more flexible. In analogy to the PAE, model it with normalizing flow,
LRoW FAO VAE p(() [E (2)00) [=]]=))) DEL (=))
(14)
4 Downstream Tasks
In our experiments, we test the PAE performance not only in terms of sample and reconstruction quality, but also terms of anomaly detection, a highly relevant downstream task of generative models. appendix E, further show how the PAE be used for posterior-based probabilistic image inputation.
4.1 Anomaly detection
One application generative models is anomaly or ant-of-distribution (OoD) detection. This is often based the assumption that density estimator should return smaller probability densities for out-of-listribution
5

---

Published in Transactions on Machine Learning Research (09/2022)
data than in-distribution (ID) data. However, this is assumption is not always satisfied and generative model based density estimators have been reported to perform poorly in some OoD detection problems. OoD detection with VAEs, NFs (Kingma & Dhariwal, 2018) and PixeICNNs (van den Oord et al., 2016) can exhibit catastrophic outlier detection failures (Nalisnick et al., 2019).
The PAE model not trained to maximize the data likelihood, nor does it provide an estimate of it. While we could attempt to perform marginalization over the latent space (after introducing an approximate suggest much simpler outlier detection metric:
implicit likelihood) in order to obtain such an estimate, the estimated density in latent space. We find in our experiments that the NF estimated latent space density is an excellent OoD detection metric for outlier detection problems that have been identified as problematic in the literature.
Our OoD metric is again motivated by PPCA: The PPCA model is a useful toy model to understand how dimensionality reduction prior density estimation can be beneficial. The PPCA density estimate latent space is given by
K K (2 + Σ Σ
K In d +
1 In p(2) - 2
2
(15)
Equation 15 diverges for d - O. This suggests that p(z) and henceforth p(æ) (Equation 4) can be dominated by small cigenvalues. These small cigenvalue components are well known to be difficult to estimate from limited amount of data, i.c. they are prone to overfitting. This has led to the development of special co- variance estimation regularization techniques known as shrinkage methods in the statistics literature (Chen et al., 2010; Ledoit & Wolf, 2004). Dimensionality reduction keeping the largest eigenvalues removes dimen- sions with vanishing variance and hence cures the cstimator's sensitivity to small and likely mis-ostimated eigenvalues. In PPCA f(x)), the data covariance C = O oO + Σ is regularized by the noise covariance, E, which reinterprets the discarded and mis-estimated eigenvalues as noise. Noisc is not informative for anomaly detection, which suggests that PPCA data space density estimation has no advantage over density estimation in latent space for this task.
We illustrate this on an outlier detection problem between the FashionMNIST (Xiao et al., 2017) and MNIST (Lecun et al., 1998) data sets in figure 2. Both of these data sets have a data covariance matrix with a high condition number. Only 86 PCA components are required to capture 90% of the data variance in MNIST (compared to N=784) and the smallest eigenvalues are evidently singular. Similar applics to F-MNIST, where a PCA captures 90% of the data variance with 83 components. These data are known to produce rophic failures in OoD detection, specifically when presenting samples from the MNIST data set to models trained F-MNIST (Nalisnick et al., 2019). We construct probabilistic PCA model for F-MNIST (from training data) and use equation 4 and equation 15 as outlier detection metrics to separate F-MNIST in- Distributtinn (iD) test data from MNIST Out-of- Distribution (OoD) data. In figure 2 we show the Area Under Receiver Operator Curve for this outlier detection task as a function of the number of PCA components. The highest outlier detection accuracy based on latent space density estimation (equation 15) is AUROC- 0.980 and it reached at a relatively low number of PCA components of 127. The highest accuracy for OoD detection based on data space density estimation (equation 4) is AUROC = 0.974 at 37 components. We argue analogously that the PAE latent space density is better for OoD detection than using a full dimensionality NF. In our experiments, we find that the PAE latent space density is superior anomaly detector than the ELBO or full dimensionality NF.
5 Related Work
Generative moment matching networks (Li et al., 2015) have been proposed to be used in a 2-stage PAE-like set up, consisting of an auloencoder and a mapping of a Gaussian to the encoded distribution. The second stage is non-invertible and trained with a noment-matchiin et al., 2018) employ a training objective based on the Wasserstein distance to match the encodod distribution to a given prior. WAEs achieve high sample quality, but do not provide a density estimate. Generative latent flows (Xiao et al., 2019) are similar conjunction of AE and NF and achieve high sample quality, but the
objective. Wasserstein autoencoders (Tolstikhin
6

---

Published in Transactions on Machine Learning Research (09/2022)
fraction of total variance
1.0
value of elgenvalue
10
0.8
0.8
10a
0.6
AUROC
0.6
10-4
0.4
0.4
10-1
MNIST
FMNIST
0.2
MNIST
FMNIST
0.2
xlx
200
400
600
800
0
200
400
600
800
200
400
600
800
PCA component
PCA companents
components
Figure 2: Principal component analysis of FashionMNIST and MNIST data sets and outlier detection accuracy (in-distribution: Fashion MNIST, out-of-distribution: MNTST) with equation 4 and equation 15 a function of included number of PCA components. A higher AUROC value corresponds to better separation between in- and out-of-distribution data.
authors do not perform controlled ablation studies to compare their approach with ELBO-based training objectives, nor do they explore the probabilistic interpretation for downstream tasks.
Giving more fiexibility to the prior distribution has also been suggested in the context of VAEs, e.g. by modeling it with an NF (Bauer & Mnih, 2019; Chen et al., 2017; Tomczak & Welling, 2018). This has sometimes been deemed prone to overfitting (Tomczak & Welling, 2018) and many works suggest using more expressive variational distributions instead (Kingma et al., 2016; Rezende & Mohamed, 2015; Salimans et al., 2015; Tran et al., 2016). In our experiments, we compare our PAE with a VAE with an NF prior because it allows us to compare models with the same architecture. We pay special attention to overfitting, but do not observe il to be problem.
Other approaches that improve the VAE sample quality include 3-VAEs (Tishby et al., 2000; Alemi et al., 2017: Hledik et al., 2019; Higgins et al., 2017) and 2-Stage- VAEs (Dai & Wipf, 2019). VAES balance rate and distortion by means of an additional scalar parameters (8). 2-stage VAFs combine two consecutive VAE's, one for the purpose of data compression, where the KL term in the ELBO is suppressed and a second one for latent space density estimation. This two-stage approach achieves high quality samples. With the PAE we demonstrate that this success does not rely on the ELBO objective in the first stage and that the first stage can be replaced by an AE and the second stage by a powerful NF density estimator. High sample quality is also achieved by VQ-VAE (van den Oord et al., 2017; Razavi et al., 2019), another model that requires 2-stage training. It combines a more complicated first stage which includes lyper-ppramnee luning and discretization with a second stage in which an autoregressive model is trained to learn the prior. The successes of 2-stage models indicate that scparating the tasks of learning a lower-dimensional representation and learning its distribution beneficial for generative model performance.
Other non-ELBO approaches that address density estimation for data that is confined to lower dimensional manifold include Miey-fows (Brehmer & Cranmer, 2020) and relaxed injective probability flows (Kumar et al., 2020). Instead of separating the tasks of compression and density estimation in the lower dimensional manifold, these models regularize a flow itself. The authors of M(er-flows compare their model with the PAE and find that the PAE reaches the same low reconstruction error and higher accuracy in anomaly detection. The recently introduced regularized autoencoder is another deterministic VAE alternative based on an autoencoder, that produces comparable or even better sample quality than VAFs (Ghosh et al., 2020).
Downstream tasks: Out-of-distribution detection with generative models has recently attracted a lot of attention, triggered by the finding that state-of-the art generative models such VAE, GLOW (Kingma & Dhariwal, 2018) and MAF (Papamakarios et al., 2017) fail in this task on a number of standard data sets (Nalisnick et al., 2019). Our finding that dimensionality reduction combined with latent space density estimation results in reliable anomaly detection is in line with other works: e.g. Abati el al. (2019) use a single stage training with a free parameter that controls the relative contribution of reconstruction error and NF latent space density and requires tuning of the free parameter. They apply it to OoD, but not to other PAE tasks such as generating samples, denoising and inpainting. Other proposed solutions include the use likelihood ratios as OoD metric instead of the likelihood itself (Ghosh et al., 2020). More recently,
7

---

Published in Transactions on Machine Learning Research (09/2022)
reliable OoD detection reported with df method which leverages another density estimator on top of the density estimation (Morningstar et al., 2021).
6 Experiments
The aim of our experiments is to test whether training on the ELBO offers any measurable advantage over the PPCA-inspired PAE training approach. For this test, we construct and train equivalent PAF and VAE models and compare them in terms of reconstruction error, sample quality, outlier detection accuracy and probabilistic inpainting and denoising ability. We conduct our detailed comparisons between PAE and VAE models on the FashionMNIST data set. For anomaly detection we perform an additional comparison on MNIST and CIFAR10 with the anomaly detection method by Abati al. (2019) (Appendix C). We further train PAE model on the higher dimensional Celeb-A (Liu et al., 2015) data set (Appendix A).
6.1 Ablation studies
We compare the PAE to two ELBO-based alternatives:
1. A VAE with a normalizing flow prior, where the encoder/decoder pair is trained on equation 13 with B-0, ie. without the KL-Divergence term and the variance g(z) is kept constant. The normalizing flow trained a second stage on the encoded distribution. The ELBO with 3=1 and the normalizing flow prior is used as a density estimator. We call this model B.-VAE. The difference to the PAE lies in the noisy estimation of the likelihood during training. Comparing the PAE to the 30 VAE tests whether training the reconstruction error instead of the distortion term offers any advantage.
A VAE with a normalizing flow prior that is trained on the ELBO with 3=1. We call this model flow- VAE. The difference to the PAF lies in the training procedure. In the flow-VAR, the normalizing flow, encoder and decoder are trained jointly under the ELBO training objective. This means that the normalizing flow is trained on samples from the approximate posterior instead of the encoded samples, which are used in the PAE.
Table summarizes the different models and the parameters in which they differ. To allow for a fair com- parison, we use the same model architecture and training parameters for all experiments, except in cases where fixing them might disadvantage the VAE. Encoder and decoder networks are loosely based on the infoGAN architecture (Chen et al., 2016). Normalizing flows are constructed from RealNVP transforma- tions (Dinh et al., 2017), Neural Spline Flow (NSF) transformations (Durkan et al., 2019) and trainable permutations (Kingma & Dhariwal, 2018) (GLOW). The exact model architectures are listed in table F and the choice of parameters is detailed below in table 6.3. Because we use non- non-binarized data, we use Gaussian implicit likelihood in our VAE models (instead of a Bernoulli likelihood which would be the suitable choice for binarized data). The value of the scale parameter, a, of this implicit likelihood was set to a = 0.1. We found this to be be the optimal value in small ablation study (appendix G). We also found that the model performance is not overly sensitive to this choice. In addition to the experiments presented here, we trained vanilla. 3-VAE and show results in appendix B. We did not include this model in main text because it does not use a flow prior and requires parameter fine-tuning. We also find that it performs worse than all alternatives studied in the main text.
6.2 Data sets and preprocessing
We perform our ablation studies on the Fashion-MNISST (Xiao et al., 2017) data set, which we split into 50,000 training examples, 10,000 validation and 10,000 test samples. As outlier data sets we use MNIST (Lecun et al., 1998) and Omniglot (Lake et al., 2015), as well as horizontal and vertical flips of F-MNIST test data. We preprocess the data, by dequantizing (adding uniform noise E [-1/256, 1/256]) before rescaling pixel values to the interval [-0.5,0.5].
8

---

Published in Transactions on Machine Learning Research (09/2022)
model parameters
model name
B VAE
PAE
Hlw-VAL
training objective(s)
Cg-VAE, LNF
CAE,LNF
Lioow-VA VAE
flow prior
TRUE
TRUE
TRUE
2 stage training
TRUE 0
TRUE
FALSE 1
B
N/A
dropout rate
0.15 ELBO
0.15
N/A
OoD metric
logp(z)
ELBO
Table 1: Overview of the different models used in the ablation studies.
6.3 Parameter choice and training procedure
Training of the encoder/decodee pair: We used the samne encoder and decoder architecture for all of our experiments. We further fixed the latent space dimensionality to 40, the mumber training steps to 300,000 and used learning rate schedule which we keep the learning rate constant the initial value up to training step 100,000, then reduce lincarly down 1/10 of the initial rate over 50,000 steps. For the remaining 150,000 steps restarl. the learning rate and repeat the annealing scheme. We did not find our final loss to depend on the details of this annealing scheme, but found that annealing and restarting was beneficial. We used the ADAM optimizer (Kingma & Ba, 2015) with parameters B1 = 0.9, 32 = 0.999, c = 10-7 in all of our trainings (including for the normalizing flow).
The batch size, initial learning rate, sample size in the stochastic evaluation of the ELBO as well as the drop out rate were optimized to yicld the best possible reconstruction error on validation data on the 30 VAE model: we ran around 30 shorter trainings (100,000 steps) with different combinations of these parameters and used a Gaussian Process surrogate to determine the combination of parameter values that optimized the reconstruction error the validation data. The values obtained through this procedure are outlined in table 2 and were used in all experiments. The only parameter that we adapted in some of our experiments is the dropout rate. The dropout layer is a necessary regularization to prevent overfitting in autoencoder modols and models with = O. This regularization not necessary in VAE models with a flow prior and was not used in these models. We chose the B0- VAE for parameter optimization, because we think that it is fair middle ground between PAE and flow-VAr VAE sharing some properties with both. It was also chosen for convenience, because it allowed us to optimize the encoder decoder pair and and normalizing flow separately, without having to worry about the feedbacks of changing one the other. We point out that optimizing the flow-VA VAE would have suffered from this complication. To not disadvantage the flow-VAF, we also train flow-VAE with a different (simpler) NF' architecture.
common model parameters
batch sixe
256
initial learning rate
1.23E-03
learning rate annealing
TRUE
sample size
16
training steps latent size
300000
40
Table 2: Training parameters that were kept constant in all encoder/decoder pair trainings. The parameters were optimized to achieve the lowest reconstruction error on validation data for the 80 VAE model.
Training of normalizing flows: We construct normalizing flows from realNVP, NSF and GLOW building blocks. We optimized the number of building blocks of each kind to achieve maximal log (x) on the validation data encoded with the 3A- model. In models where the NF was trained separately, we used a learning schedule consisting of 120 epochs during which both learning rate and batch size were annealed and restarted. We found that restarting learning rate was especially helpful when using NSF transformation
9

---

Published in Transactions on Machine Learning Research (09/2022)
Lod
Figure 3: Visual comparison of F-MNIST reconstructions with different models. From left to right: original data, 3u-VAL -AE, PAE, flow-VAE, flow-V Vs)().
layers, but that the final loss did not strongly depend on the details of the annealing and restarting scheme. Training of VAE with normalizing flow prior: In the Flow-VAEs, encoder, decoder and NF are trained jointly on the ELBO objective with the same training paramcters and architectures that were used for fo- VAE and PAE. The only parameter that adapted was the dropout rate, which set to zero. To test. the dependence the flow architecture, we had found to work best on the VAE encoded data, and another experiment, in which we simplified the flow architecture (low-VAE (s)). We found the deeper architecture lo be superior. All NF archilectures are detailed in appendix F.
two experiments. One in which adapted the flow architecture
For every model (PAE, VAE, flow- VAE) we repeated every training run three times starting from different initial network parameter values. We report results for the trained models that achieved the lowest training loss validation data out of these three We found very little seatter between different trainings of the same model.
Robustness of results:
6.4 Results
Reconstruction Quality: We measure the reconstruction quality in all models by means of the average reconstruction error the test data set,
N
11 o recon MN
1
Σ
Σ Eis golf (z(f((,))))
0 recon.j
(16)
M
where j labels the image and the pixel. In the VAE models we evaluate the reconstruction error using the mean of the variational posterior. We also report the 95 percentile of the image-wise reconstruction errors Poss (ocom)) which better characterizes the large error tail of the distribution. The results are listed in table 3, the reported errors were obtained through boolstrapping on the test data.
reconstruction quality
model name
VAE
PAE
flow VAE
flow- VAE(s)
Peoon [x10-3] (1)
6.24 0.06
5.8710.06
6.22 0.07
6.66 + 0.06
Pa0% ?ocoo) [x10-3] 31 (4)
29.6 0.3
27.6 + 0.3
35.7 0.4
31.9 + 0.3
Table 3: Comparison of F-MNIST models in terms of reconstruction quality. The PAE model achieves the lowest reconstruction errors.
We find that the PAE model has consistently and significantly lower mean and 95 percentile reconstruction error than ELBO--aseed models. In figure 3, we show reconstructions of test data points for each model.
Sample Quality: How to best measure image generation quality in a way that quantifies both image quality (are the samples visually compelling?) and diversity (is the model sampling from the full distribution?), is still a question of active rescarch. Here, we mcasure the sample quality in terms of the well established Frechet- Inceett-nception Distance (FID) score (Heusel al., 2017), which known correlate well with human perception of image quality. The FID score takes sample of generated images and sample of real
10

---

Published in Transactions on Machine Learning Research (09/2022)
Figure 4: Visual comparison of random samples. From left to right: Bu- VAE, PAE, flow-VAE, flow-VA (s).
images, passes each of them through the pre-trained Inception network (Szegedy et al., 2016) and extracts features the outputs of one of the last layers. Each set of features is then fitted with multivariate Gaussian distribution, (alu,, Σ). The FID score is defined as the Fréchet distance between these two Gaussians
FID Tgen) = || Mgeal + Tr (Coris + Egen 2
2(Eorig Dguu)
(17)
In figure 4 we show generated images from each model. While visually comparable, we get the lowest FID scores for the flow-VAL model (table 4). A concern sometimes raised with models that use flow priors is
sample quality
model name
BAE
PAE
flow- VAE
flow- VAE (s)
FID score (1)
32.8 + 0.4
28.4 + 0.3 22.8 + 0.3
33.9 + 0.4
Table 4: Sample quality, measured in terms of the FTD score. Lower values are better. Error estimates obtained through bootstrapping.
that the added expressiveness of the model results in overfitting or memorization of training data. We pay special attention to overfitting during training by monitoring the loss on validation data. Outlier Detection Accuracy: For outlier delection, we use the outlier detection metries listed in table 1 and report the outlier detection accuracy for each model in terms of the Area Under Receiver Operator Curve (AUROC). An AUROC of corresponds a perfect scparation between the out-of-distribution and in-distribution data. For the latter we use the F-MNIST test data set. The ELBO of the VAE evaluated using 10 samples from the variational posterior and we verified that using more samples does not achieve better AUROC values. The results are listed in 5. The latent space density estimation of the PAE model outperforms the ELBO of our VAE models in all but one of our experiments. However, we do not observe a catastrophic failure of OoD detection with the ELBO as has been reported for this data set (Nalisnick et al., 2019). Following our discussion in 4.1, which suggests that latent space density estimation superior
OoD detection accuracy measured in terms of AUROC t
OoD data
Bp VAE
PAE (AE+flow)
flow- VAE
flow- (s)
MNIST
0.9086 0.0023
0.9970 + 0.0003
0.9633 0.0014
0.9230 + 0.0022
Omniglot
0.8668 + 0.0025
0.9736 + 0.0011
0.9221 + 0.0020
0.8753 + 0.0021
F-MNIST hor.
0.6790 + 0.0036
0.6883 + 0.0038
0.6758 + 0.0036
0.6773 + 0.0035
F-MNIST vert.
0.8891 0.0022
0.8789 + 0.0024
0.8994 + 0.0022
0.8964 + 0.0019
Table 5: Out of distribution detection with different models. The outlier detection accuracy is measured in terms of the AUROCE [0, Higher values are betler. Error estimates obtained through bootstrapping.
data space density estimation in many cases, we disseminate the ELBO further and take a look at the different contributing terms of the ELBOO. Specifically, we split the ELBO into three terms, which we identify
11

---

Published in Transactions on Machine Learning Research (09/2022)
distortion (first term), entropy (second term) and cross entropy (last term),
ELBO = Ep(x)) (2))()) [=)|)()))) - (0(z)z) pr
(18)
and measure the outlier detection accuracy with each of these terms individually. The results are listed in table 6.
We find that the cross entropy term, which measures a stochastic mean over the latent space density and is therefore closely related the latent space density, a very reliable outlier detection metric for models trained on the ELBO objective.
OoD detection with flow-VAE, AUROC(1))
OoD data
distortion
rate
entropy
cross entropy
MNIST
0.9620 + 0.0014
0.9965 + 0.0003
0.9659 + 0.0012
0.9963 + 0.0004
Omniglot
0.9176 + 0.0023
0.9786 + 0.0009
0.8965 + 0.0023
0.9795 + 0.0009
F-MNIST hor.
0.6751 + 0.0042
0.6728 + 0.0038
0.6274 0.0041
0.6619 + 0.0038
F-MNIST vert.
0.8977 + 0.0023
0.8981 + 0.0022
0.7415 + 0.0041
0.8921 0.0024
Table 6: Dissecting out-of-distribution detection with the ELBO. Error estimates obtained through boot- strapping.
7 Discussion and conclusion
We have introduced the probabilistic autoencoder, simple generative model with lower dimensional latent space that is motivated by probabilistic PCA. Different to variational autoencoders, it builds the probabilistic structure after the first stage of training, but has the advantage of being simple and straightforward to set up and train. Because it is first trained to achieve optimal reconstruction error and then, a second stage, to produce optimal samples, it performs both tasks reliably and well. We further test its performance in two additional downstream tasks, which we think are particularly relevant for practical applications: anomaly detection and probabilistic image denoising and inputation. We find that the PAE performs all considered tasks at comparable quality equivalent VAE models, suggesting that ELBO-based variational optimization is not an essential component of this class of models. We find that our proposed OoD metric of NF density in latent space, while a natural byproduct of PAE, can also be used in VAEs they are complemented with an NF prior.
References
Davide Abati, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Latent space autoregression for novelty detection. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 481-490. Computer Vision Foundation IEEE, 2019. URL http:///penaacees. httpc//cssseeeee.....c//cceetete CVPR) CVPR|__019//ttttmm 2019/html/
doi: (11109/CCP.22019/0 Abati \_ Beatenti_Sppaceee L_a_aaeetttssppceeattttreerrrrre Autorrgreesiion ff Betoovetyy_Detectttto (ateecicoo__oPP__21111_pppppp html.
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreeiiv..ee forum?id-HyxQBBcee.
Alexander A. Alemi, Ben Poole, Ian Fischer, Joshua V. Dillon, Rif A. Saurous, and Kevin Murphy. Fix- ing a broken ELBO. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th Interna- tional Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10- 15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 159-168. PMLR, 2018. URL momi188.htmm
12

---

Published in Transactions on Machine Learning Research (09/2022)
Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), The 22nd International Conference on Artificial Intelliennce and Stalistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pp. 66-75. PMLR, 2019. URL http::/proceediigg http:///mmm...ppeeee/////////aaa
19m.
Vanessa Böhm, François Lanusse, and Uros Seljak. Uncertainty quantification with generative models. CoRR, abs/1910.100445 2019. URL http:://arx.v.orgg/bbb//0......
Ashish Bora, Eric Price, and Alexandros G. Dimakis. Ambientgan: Generative models from lossy mea- surements. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview. net, 2018. URL https: https://openreviiee wtt.errrwww..etttooooo/t//t/////
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. Gener- ating sentences from a continuous space. Proceedings of the 20th STGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016, pp. 10 21, 2016. URL 1002.pdf.
http://aclweb.
Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estimation. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Infor- mation Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: /
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abbeel. Infogan: Inter- pretable representation learning by information maximizing generative adversarial nets. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Newral Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spoin, pp. 2172-2180, 2016. URL attps://proceedings.eee neurips..e Abattaachbhbttt
Xi Chen, Diederik Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Nya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In 5th International Conference on Learning Represen- tations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview .eet,, 2017. URL
Y. Chen, A. Wiesel, Y. C. Eldar, and A. O. Hero. Shrinkage algorithms for mmsee covariance estimation. IEEE Trunsactions Signal Processing, 58(10):5016 5029, Oct 2010. ISSN 1941-0476. doi: 10.1109/ TSP. 2010.2053029.
Bin Dai and David P. Wipf. Diagnosing and enhancing VAE models. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
URL http:///ppperreeeeeeeee/tr/re/ee
Laurent Dinh David Krueger, and Yoshua Bengio. NICE: non-linear independent components estimation. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL http: www.rvv.orrgbb//111...m
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th Interna- lional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.nee net, 2017. URL https: httpp://pppaaaeiiissssss/oooorrp
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep con- volutional networks. IEEE Trans. Pattern Anal. Mach. Intell., 3852):205-307, 2016. doi: 10.1109/ URL https://aoi.orr/10.11000 2015
243. 2439281.
13

---

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Isabelle Guyon, UI- rike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neurol Information Processing Systems 30: Annual Conference on Neu- vul Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 6626 6637, a- Published in Transactions on Machine Learning Research (09/2022)
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer Florence d' A'Alch-BBuc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 7509-7520, 2019.
Partha Ghosh, Mchdi S. M. Sajjadi, Antonio Vergari, Michael J. Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. In 8th International Conference on Learning Representa- tions, TCLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: / hoppaareiiee.eesfffoorrrssssasss
Tan J. Goodfellow, Jean Pouget-Abadie,, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Sys- tems 27: Annual Conference on Newral Information Processing Systems 2014, December 8-13 2014, Mon- treal, Quebee, Canada, pp. 2672-2680, 2014. URL adversarial-nets.
Will Grathwohl, Ricky T. Q Chen, Jesse Bettencourt, Nya Sutskever, and David Duvenaud. FFJORD: free-form continuous dynamics for scalable reversible generative models. In 7th International Conference on Learning Represenlalionns ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.nel, 2019. URL
2017. URL http: http:://paaers.iipp..... converge-to---te aolatto--lllllaaaaee nashealll-ns---uuuliiiiiii
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.nel, 2017. URL http:: /tttps::/openrevierest forum?id-Sy22208g1.
Michal Hledík, Thomas R. Sokolowski, and Gasper Tkacik. A light. upper bound on mutual information. In 2019 IEEE Information Theory Workshop, ITW 2019, Visby, Sweden, August 25- 2019, pp. 1-5. IEEE, 2019. doi: 10.1109/ 4000://777777777191101979
URL
1tt.:/110//TA44777.00000.9090000 8989922..
Matthew D. Hoffman and Matthew J. Johnson. ELBO surgery: yet another way to carve up the variational evidence lower bound. In Advances in Approzimate Bayesian Inference, NIPS 2016 Workshop, 2016. URL
http:://aproximatteiineeeee www.nnffeeneeoorroo www.enceorrr/cccceteeeeeeeeeeeee
Kyong Hwan Jin, Michael T. McCann, Emmanuel Froustey, and Michael Unser. Deep convolutional neural network for inverse problems in imaging. IEEE Trans. Image Process., 26(9):4509-455) 4522, 2017. doi: 1t0p//IPP.011.2713000000 URL http::///ooo..oo///00000//...... 171.229977..7.999
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:://arxxv..rg/bbb//// ab/1//2..9/111.........
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annuol Conference on Neural Information Processing Systems 2018, NewrIPS 2018, 3-8 December 2018, Montréal, Canada, pp. 10236- 10245, 2018. URL ix1-convolutions
dlow-generative wowowwwthhiiveeenn
14

---

Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and lan J. Goodfellow. Adversarial autoencoders. wtt../////../.............. Published in Transactions on Machine Learning Research (09/2022)
Diederik P. Kingma and Max Welling. Auto-cncoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, TCLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL attp://arraivoogg/bb//111......
Diederik P. Kingma, Tim Salimans, and Max Welling. Improving variational inference with inverse autore- gressive flow. CoRR, abs/ 0ee//6666049344 2016. URL http://arxiv..org/aaa 1606.64934.
Abhishek Kumar, Ben Poole, and Kevin Murphy. Regularized autoencoders via relaxed injective prob- ability flow. In Silvia Chiappa and Roberto Calandra (eds.), The 23rd International Conference on Artificial Intelligence and Statisties, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy), volume 108 of Proceedings of Machine Learning Research, pp. 4292-4301. PMLR, 2020. URL http://ppoceeeding...k prrpeaa/vvv00/uumm2a......l
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266) 1338, 2015. ISSN 0036-8075. doi: 10.1126/ science.aab3050 URL
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. In Proceedings of the IEEE, pp. 2278-2324 1998.
Olivier Ledoit and Michael Wolf. well-conditioned estimator for large-dimensional covariance matrices. J. Multivor. Anal., 88(2):365-4111 February 2004. ISSN 0047-259X. doi: 10.1016 (0.1016/50047-250000))00000 URL httpp://bbd..rr//0.000///0000
(03) 00096-4.
Yujia Li, Kevin Swersky, and Richard S. Zemel. Generative moment matching net works... In Francis R. Bach and David M. Blei (ods.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, Pp. 1718 1727. JMLR.org, 2015. URL
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
CoRR, abs/ abs/1511.05644, 2015. URL
Pierre-Alexandre Mattei and Jes Frellsen. Leveraging the exact likelihood of deep latent variable models. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: An- rual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 3859-3870, 2018. URL https: h/tppe///proccdinggss neur cwurrpp..oo hptp.ccppaarr//1//aaaaaas 6009155H33h1140088464caa2222 A.aa444tttbttttt.htm....
Plerre-Alexandre Mattei and Jes Frellsen. MIWAE: deep generative modelling and imputation of incom- plete data sets. Tn Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- ternational Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 4413 4423. PMLR, 2019. URL attp://proceedings..... wtw.:/iiinn......pres//s/////e/e httpa////////aateee...tttte
Warren R. Morningstar, Cusuh Ham, Andrew G. Gallagher, Balaji Lakshminarayanan, Alex Alemi, and Joshua V. Dillon. Density of states estimation for out of distribution detection. Tn Arindam Banerjee and Kenji Fukumizu (eds.), The 24th International Conference on Artificial Intelligence and Statistics, AIS- TATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Research, pp. 3232 3240. PMLR, 2021. URL http:// attp:///proceedings mlr www..rrpres/vii/e... preess//130//mmmiigttaaa.a....
Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, and Balaji Lakshminarayanan. Do doep generative models know what they don't know? In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL http::///ppneeeeieraaa//mmmmmiip
15

---

Published in Transactions on Machine Learning Research (09/2022)
Michela Paganini, Luke de Oliveira, and Benjamin Nachman. Accelerating Science with Generative Ad- versarial Networks: An Application to 3D Particle Showers in Multilayer Calorimeters. Physical Review Letters, 12044)) January 2018. doi:
George Papamakarios, Jain Murray, and Theo Pavlakou. Masked autoregressive flow for density es- timation. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Sys- tems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 2338- 2017. URL https:// htw.cceciinns.uurrp..c////p//ppp /20h///ah//////////////// htt.:/oott....aaaaaaaattt..tt..a Abstract.html
Patrick Putzky and Max Welling. Recurrent Inference Machines for Solving Inverse Problems, arXiv e-prints, art arXiv:1706.04000 Jun 2017.
Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ- VAE-2. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence FAlché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neurol Information Processing Systems 32: Annual Con- ference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 14837 14847, 2019. URL https://pret http::///proceedng..eeeee seeerrpp.cccpaape/20199
David M. Reiman, John Tamanas, J. Xavier Prochaska, and Dominika Durovčíková. Fully probabilistic quasar continua predictions near Lyman-alpha with conditional neural splinc flows. arXiv e-prints, art. 20Xiv:2006006155, May 2020.
Danilo Jimenez Rezende and Shakir Mohamed, Variational inference with normalizing flows. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, httpriinnmm...ppeeer///////eeeee
pp. 1530 1538. JMLR.org, 2015. URL
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, pp. 1278-1286, 2014. URL proceeedinggg/pperr//////// wttpe///e.//eeeeeee......m.
Oren Rippel and Ryan Prescott Adams. High- dimensional probability estimation with deep density models, CoRR, abs/1300 1bb/1332.5125, 2013. URL http://aatw...rr///b//3.0..5.. 55025.5..1115
Tim Salimans, Diederik P. Kingma, and Max Welling, Markov chain monte carlo and variational inference: Bridging the gap. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11JJul 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1218-1226. JMLR.org, 2015. URL UtL.http::/proceeddii mlr. http://nsssmmm..pes///ss///// press/v37/ sal 1.mama11.html
Uros Seljak and Byeonghee Yu. Posterior inference unchained with EL 20. arXiv e-prints, art arXiv:1901.0444 Jan 2019.
George Stein, Uros Seljak, and Biwei Dai. Unsupervised in-distribution anomaly detection of new physics through conditional density estimation. arXiv e-prints, art arXiv:2012.11638 December 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEFE Conference on Computer Vision and Patlern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2818 2826. IEEE Computer Society, 2016. doi: 10.1109/CVVP CVP..22016.23. URL http:://ooi.orgg//011////////
Ben Thorne, Lloyd Knox, and Karthik Prabhu. A generative model of galactic dust emission using variational autoencoders. MNRAS, 504(())2603 2613, June 2021. doi: 10.1093 1..1008/mmraa/stab10011
16

---

Published in Transactions on Machine Learning Research (09/2022)
Michael E. Tipping and Christopher M. Bishop. Probabilistic principal component analysis. Journal of the Royal Stalistical Sociely: Series B (Statistionl Melhodologyy,, 61 (3):611 12262222,,2222111 1999. doi: 10.1111/1467- 9868.00196. URL
Naftali Tishby, Fernando C. N. Pereira, and William Bialek. The information bottleneck method. CoRR, physics/0004057,, 2000. URL http://sssssslaallaalr/////////s
Nya O. Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. Wasserstein auto-encoders. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net,, 2018. URL https://openrrviee..eet)
Jakub M. Tomczak and Max Welling. VAE with a vampprior. In Amos J. Storkey and Fernando Pérez-Crus (eds.); International Conference on Artificial Intelligence and Statistics, AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, volume 84 of Proceedings of Machine Learning Research, html
pp. 1214-1223. PMLR, 2018. URL http:/ attp::/proceet http::///poceeeng...ilee waa.ingsss........ paress//84/tomeeee
Dustin Tran, Rajesh Ranganath, and David M. Blei. Variational gaussian process. In Yoshua Bengio and Yann LeCun (ods.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerlo Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiw.org/bbbl 1511.06499.
Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Deep image prior. Int. Comput. Vis., 128 (7):1867-1888, 2020. doi: (00000//11233-00--00 01303-4.
URL http:://0l...rg/10100///////////
Aäron den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukeuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. In Daniel D. Lec, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnelt (eds.), Advances in Neural Information Pro- cessing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5- 10, 2016, Barcelona, Spain, pp. 4790-4798, 2016. URL bttp:::/pproceedngge Abt.ro0tt...bttra...tt.....
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Isabelle Guyon, Ulrke von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Sys- tems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6306-6315, 2017. URL https: 7a98aff Ta30B107766600000222600099220 www.::tbbc.-bttttt.t..aa..
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image datasel for benchmarking machine learning algorithms. CoRR, abs/ Abb/1708.07747, 2017. URL http:://arxiv.orgaaa///ooo0
Zhisheng Xiao, Qing Yan, Yi-an Chen, and Yali Amit. Generative latent flow: A framework for non- adversarial image generation. CoRR, abs/1995.10488 2019. URL http:///arxxv.orr//////00555
A PAE on Celeb-A
To demonstrate the feasibility of the PAE approach on higher dimensional, more complex data, we train PAE on Celeb-A. The celeb-A samples are cropped to the central 128x128 pixels and then downsampled to 64x64 pixels. We used the same preprocessing, architecture and training procedure as for the F-MNIST experiments and a latent space dimensionality of K 64. Samples from this model are shown in figure 5 on the left. On the right we show interpolations between images, produced by projecting two images from the test data set into the latent space of the NE, connecting them in the NF latent space by linear interpolation and sampling along the connecting line equally spaced intervals. The samples then get forward modeled into data space to produce the images shown. The smooth transitions indicates that the PAE produces a continuous latent space without any holes.
17

---

Published in Transactions on Machine Learning Research (09/2022)
Figure 5: PAE performance Celeb-A at K-64. Samples (left) reach FID-49: 49.2 (reconstructions
F1D=44.0). Right: Interpolations between samples from the test set.
Lee
Figure 6: From left to right: Original test images, their reconstructions and samples generated with the vanilla 3-VAE described in Appendix B.
B Comparison to vanilla VAE
For completeness, we include here results for a conventional VAE, which shares the same architecture and training parameters as the models used in the other experiments, but uses a standard normal distribution as a prior. We trained the VAE on equation 13 with initial value of , 100, which was then linearly annealed during the first 100,000 steps. We show samples and reconstructions of this model in Fig 6 and list reconstruction error and FID scores in Table 7. In all cases the results and visual images are inferior to flow-VAL VAE and PAE described in the main text.
0² recon [x10-3 (1) Ponx (o ( [x10-3 3 (1) 68.00 0.60
FID Score (4)
12.90 + 0.10
32.3 + 0.3
Table 7: Reconstruction errors and FID scores vanilla 3- VAE.
C Comparison of anomaly detection with Abati et al.
We compare anomaly detection with the PAE to the results presented in Abati et al. (2019). Their set up similar, but instead of a two-stage training, they propose a joint training of autoencoder and density estimator. Their loss function combination of reconstruction error and estimated density of encoded data. The contribution of the density term the loss function is controlled by a tunable scalar parameter X.
18

---

Published in Transactions on Machine Learning Research (09/2022)
Using the same autoencoder architecture latent space dimension, and batch size?, we perform anomaly detection experiments as Abati et al. (2019) but with a PAF-style two-stage training: We train models on each class of the CIFARIO and MNIST datasets separately and then evaluate the outlier detection accuracy when each these models is applied to the full test dataset containing all classes (considering all other classes outliers). Results of these experiments and the results from Abati et al. (2019) are shown lables 8 and 9.
Class
2
3
4
9
mcan
Abati et al.
0.993
0.999
0.959
0.966
0.956
0.964
0.994 0.991
0.98 0.974
0.953
0.981
0.975
PAE
0.988
0.999
0.972
0.964
0.97
0.955
0.955
0.978
0.975
Table
Comparison of MNIST anomaly detection results.
Class
0
2
3
4
5
7
8
9
mean
Abati et al.
0.735
0.58
0.69
0.542
0.761
0.546
0.751
0.535 0.529
0.717
0.548
0.608
PAE
0.737
0.472
0.684
0.52
0.749
0.506
0.758
0.682
0.446
0.604
Table 9: Comparison of CIFAR anomaly detection results.
D Interpolation studies
We expand on the image interpolation results shown in Appendix A by comparing PAE image interpolation with AE image interpolations and adding pixel-level interpolation (lincar interpolation between pixel values) as a baseline. Results are shown in figure 7. While both AE and PAE produce fairly well int interpolated images of high quality, PAE tends to produce more natural looking interpolations with fewer artifacts.
E PAE posterior and application to data inputation
Large-scale data acquisition often results in noisy and incomplete data. In most applications, e.g. when one is interested in finding certain rare features in image, the aim of data restoration is not only to obtain the most probable uncorrupted image, but also to obtain an estimate of its fidelity. A plethora of generative model based approaches for image reconstruction have been suggested in the literature (Rezende et al., 2014; Mattei & Frellsen, 2018; Dong et al., 2016; Jin et al., 2017; Putzky & Welling, 2017; Ulyanov et al., 2020; Bora et al., 2018; Mattei & Frellsen, 2019), but few of them enable uncortainty quantification (Böhm et al., 2019). With the PAE, we can perform sound posterior-based data restoration. This does not only enable uncertainty quantification, it also provides framework for consistently including analytical data models, such physics models.
A latent space posterior for a corrupted data point ã Mæ+n where M is pixel-wise mask and denotes the noise, consists of an implicit likelihood and a prior. The form of the implicit likelihood is determined by the noise properties. For Gaussian noise, n, with noise covariance noise (typically a diagonal matrix) and a generative model gø, trained on uncorrupted data, the implicit likelihood is given by
( x, M, nolse) = N (a) o2 2 o o
Pnoise)
(19)
Note that the covariance of this Gaussian likelihood composed of the generative model's reconstruction error, 2 pocon and the noise level in the corrupted data, o naise. For sufficiently high latent space dimensional- ities the latter dominates, Trecon, nois,,, ensuring that the likelihood is well approximated by Gaussian. By replacing lateni space of the generalive model. Posterior analysis of high-dimensional data becomes computationally tractable in this lower dimensional space.
by its generative process, (z),), we bring the inference problem to the low dimensional
Pobtained from
19

---

Published in Transactions on Machine Learning Research (09/2022)
Figure 7: Linear interpolations between images. in each panel we show in the top row the reconstructions from linear interpolations in PAE latent space, in the middle row from AE latent space, and in the bottom row linear interpolations in data space (lincar interpolation between pixel values).
20

---

Published in Transactions on Machine Learning Research (09/2022)
The prior is modeled by the generative models' latent space distribution. Combining the two, we obtain the log posterior In pop (z| M, noise) = In pa z, M, a noise) + (z) - const.
(20)
To denoise and inpaint a corrupted image one performs latent space posterior analysis. A point estimate is given by the MAP; =', the maximum of equation 20, which forward modeled into data space, Breoon = g yields the most likely underlying image. A full posterior analysis can be performed with many techniques, including Laplace approximation, VI or MCMC sampling. Given the multi-modal posterior of some of our examples we fit full rank Gaussian mixture model to equation 20 following (Seljak & Yu, 2019) in our experiments. We can then sample from this model to obtain other solutions that are compatible with the data.
We examine probabilistic reconstruction with both PAE and dow-VAB on three examples created by cor- rupting test data with uncorrelated Gaussian noise (mn=0, 0011.) and masks. The true underlying images are shown in the first column of figure 8 and the corrupted input data to the reconstructions in the sec- ond column. The masked areas were chosen to allow for several plausible inpainting solutions. To obtain reconstructions we minimize the negative of the log posterior in equation 20. Since expect multimodal posteriors we run 20 minimizations starting from random points which we draw from the prior. We keep only those minimization results that are associated with a positive definile Hessian (Irue minima and no saddle points). In the third and fourth column we show the forward modeled deepest minimum found by this procedure for the PAE (third column) and VlO model (fourth column). The reconstructed examples are well denoised, but flow-VAE and PAE find slightly different inpainting solutions for the masked areas. This is most visible for the first example, where the flow-VAE prefers a clasp on the bag, while the VAE reconstructs a dint the top. While this suggests that the models have learned slightly different priors and forward models, we find that all reconstructions are very plausible and that the PAE does not seem to perform worse at this task than the flow- In fact, in the first example, the PAE seems to better reconstruct the area outside of the masked area (e.g. the ribbon on the left), a consequence of the lower recoustruction error of the PAE model.
To obtain uncertainty estimates we fit a Gaussian mixture model using the local minima associated with the largest posterior mass. We use the thus constructed posterior approximation for uncertainty estimation by generating samples from this posterior. The samples are shown in figure 9, with samples from the PAE posterior in the top row and samples from the VAE in the bottom row. We observe some variety in these samples, which we encouraged by posing problems that should have multiple plausible solutions. For example the PAE finds that handbags with different dint depths are compatible with the data. The fAow-VAE mostly prefers handbags with clasps, but it also produces two samples without a clasp. It seerss that neither of the two models explores the full variety of potential solutions (the flow- VAE missing the dints and the PAE missing the clasps). In the second and third example the PAE model seems to provide some greater variety in the samples. This could be an indicator that the flow- posterior is more complex and wasn't 't explored properly by our poterior analysis. It is not in scope of this work to explore these possibilities in great detail, instead, we note that flow-VAE and PAE perform comparably well at this task.
F Model architectures and training procedures
We use the same encoder and decoder architecture in all experiments. Details are given in table 10 and table 11. The only model parameter that is varied the dropout rate, which was set to 0.15 in models where it was necessary to avoid overfitting and to zero otherwise. The choice of hyperparameters is described in detail in section 6.3.
The normalizing flow(s) we use are composed of the same building blocks in all experiments but vary in how often the building blocks are repeated. All blocks have the same structure and only differ in the lype of transformation that is performed. A block is made up of a transformation of the first half of data variables, a swapping of the first half of data variables with the second half, a transformation on the other half of variables and finally a trainable permutation of the variables. Block 1 applies a neural spline flow transformation, Block 2 realNVP transformation with shifting and rescaling, Block 3 realNVP transformation with only
21

---

Published in Transactions on Machine Learning Research (09/2022)
Figure 8: Underlying true data (left column), corrupted data (second column) and most likely reconstructions
with the PAE (third column) and flow-VAE (fourth column).
Figure 9: Samples from the reconstruction posteriors (PAE: top row, flow-VAt VAE: bottom row).
22

---

Published in Transactions on Machine Learning Research (09/2022)
encoder
layer input
details
normalized to -0.5,0.5), de-quantized with uniform noise |1/2256, 1/256]
convolutional
kernel sizc=[4,4] strides=(2:2), filters=64, padding='SAME"
leakyReLL
a=0.2
convolutional
kernel size= strides= filters=122 128, padding-'AAME" momentum=009999, c=le-5
batch norm
dropout
dropout rate dependent on model, see 2 a=0.2
leakyReL.U
reshape
flatten
linear
output faze=1024
batch norm
momentumm009999 c=le-5
dropout
dropout rate dependent on model, see 2 a=0.2
leakyReLU
linear
2x latent size
Table 10: The layout of the encoder network used in all experiments.
decoder
layer
details
input
encoded data, latent size=40 output size=1024
linear
batch norm
momentum=0.999, e=le-5
leakyReLU
a= 0.2
linear
output 1ize=128x28/445 camentuum-0999. c=1e-5 a=0.2
batch norm
leakyReLU
dropout
dropout rate dependent on model, see 2
reshape
output (tze=[22/4,28/4,.1 output (2tapee222/2,2//22 kernel size =[4,4] strides- [2r22
transpose convolution
batch norm
momentum=0.9999 e=le-5
leakyReLU
a=0.2 dropout rate dependent on model, see 2 output shape=[28,28,1] kernel size =[4,4] strides = [2,2] subtract 0.5 to match input data.
dropout
transpose convolution
sigmoid
Table 11: The layout of the decoder network used in all experiments.
shifting. The architectural details are given in table 12. We use two different layouts in our experiments, deeper flow with [2x block 1, 4x block 2 and 4x block 3] and a simpler flow with [1x block 1, 2x block 2 and Ix block 3). For our Celeb-A experiments, we use a normalizing flow consisting of [Ix block 1, 2x block 2 and 2x block 3). Every flow also features a re-scale operation that ensures that the encoded training samples within the range 2 E 1_.
The neural spline flow uses two networks to determine the bin widths and slopes of the rational quadratic splines. The layouts of these networks listed in table 13.
23

---

Published in Transactions on Machine Learning Research (09/2022)
normalizing flow
blocks
layer
details
rescale
scale dependent on model, see 14
1
neural spline transform
bins=36, bins: bin network, slopes: slope network
swap permutation
swap first half of dimensions with second half bins=36, bins: bin network, slopes: slope network
neural spline transform
trainable permutation
GLOW-style LU decomposition of orthogonal matrix trainable shift and rescale
2
real NVP transform
swap permutation
swap first half of dimensions with second half trainable shift and rescale
real NVP transform
trainable permutation
GLOW-style LU decomposition of orthogonal matrix trainable shift swap first half of dimensions with second half trainable shift
3
real NVP transform
swap permutation
real NVP transform
trainable permutation
GLOW-style LU decomposition of orthogonal matrix 1/scale dependent model, table 14
rescale
Table 12: Building blocks of the normalizing flows used in all experiments.
bin network
slope network
layers
details
layers dense
details
dense
output size=latent size/2
output size-latent size/ /22
leakyReLU
a=0.2
LeakyReLU
a=0.2
dense
output size=latent size/2
dense
output size=bins-1
leakyReLU
a=0.2
reshape
[latent sizc/2, bins-1]
dense
output size=bins
softplus
softplussx)+1e-2
reshape
[latent size/2, bins]
softmax
softmax(x)(2-1e-2 bins)+
Table 13: Networks used determine bin widths and slopes in the rational quadratic spline of the neural spline flow.
G Flow-VAE ablation studies
We ran several flw-VAB models with different values of a, with being the scale parameter in the implicit ELBO likelihood, a multivariate Gaussian with f=fo(=) and E=ol,
Po (x]2)) = G(z)fo(z),
(21)
The objective of this ablation study is to test whether the flow-VAE performance depends on this parameter. Similar to the 3-parameter, a could have an influence on the relative contribution of distortion and rate term to satisfying the ELBO objective during training. The results of this ablation study (in terms of roconstruction error and sample quality) are listed in table 15. We do not find a strong dependence of neither reconstruction error nor sample quality on a. Somewhat counter-inluitively we get slightly higher reconstruction errors for lower values of a, an indication that the model is starting overfit. A test run with 0.05 (not listed in the table) resulted in catastrophic overfitting. A slightly higher value, seems to be sweet spot, with the lowest reconstruction error, signs of overfitting and good FID score. This is the model we use in the result section. We note that the other values of a we tried would not change our results in a way that invalidates our comparison or conclusions. We also note that such an ablation study/ parameter optimization not needed for the PAE training, which highlights an advantage of the PAE approach.
0.1,
24

---

Published in Transactions on Machine Learning Research (09/2022)
normalizing flow layouts for each model
model name
30- VAE
PAE & flow- VAE
flow- (ow()AE(s)
repeats flow block 2
2
1
repeats flow block 4
4
2
repeats flow block 4
4
1
flow rescale scale
344
3.44
1
Table 14: Normalizing flow layouts of different models. Blocks are described in table 12.
a-ablation study
0.08
0.1
0.12
Tecon 1x10-3 3
(4)
7.85 + 0.09
6.22 + 0.07
8.35 + 0.09
Pos% _
recon
x 10 $1 (L)
3.3
35.7 + 0.4
39.9 0.5
sample FID score (1)
22.5 + 0.2
22.8 + 0.3
22.8 + 0.3
Table 15: Flow-VAE ablation study: reconstruction error and sample quality as a function of a, the width of the Gaussian likelihood entering the distortion term in the ELBO.
25


Reviews:
[{"review_id": "LuXPPPvbzW", "reviewer": "Reviewer_3jv7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The paper explores the impact of using a two-stage training process for VAE-style models . While other works have proposed similar training approaches, it is useful to directly compare the two-stage process to the more standard joint training with the ELBO objective in a controlled setting.\n*  The paper also shows how such a model can be used for anomaly detection, by doing density estimation either in latent space, or in data space. The experiment and analysis with PPCA motivates the use of latent space based anomaly detection. In particular the line \"The very small eigenvalues are likely to be mis-estimated and not informative for the outlier detection task\" seems potentially quite insightful (but should be expanded for clarity).  \n\nWeaknesses:\n\n* \"this noise component is uninformative for the outlier detection task and might obscure the results\" - This is a strong statement that makes some underlying assumptions about the nature of outliers. Maybe it is a good assumption, but that should be made explicit.\n* \"In our experiments, we find that the PAE latent space density is a superior anomaly detector than the ELBO\" - because of the max value of the AUROC curves? The difference between AUROC = 0.980 and AUROC = 0.974 is pretty tiny. It does seem like the p(z) anomaly detector is more robust to the number of PCA components, but this is not discussed in the text.\n* Anomaly detection: while the experiments are interesting, I wanted to see more discussion of other anomaly detection methods. The paper answers the question of: Which AE-style model (and which terms in the ELBO) is more effective as an anomaly detector, but it doesn't say in how useful the best presented anomaly detector is compared to other methods in the literature. For those who want to build the most effective anomaly detectors this is an important question to answer.\n* \"test data set into the latent space of the NF, connecting them in the NF latent space by a straight line and\nsampling along this line in equally spaced intervals\" - I'm unsure whether this refers to a simple linear interpolation in the latent space, or in a way that somehow reflects the probability density associated with the NF (e.g. projected onto a manifold of high probability latents). I think this isn't quite clear enough.\n* \"Comparing the PAE to the β0-VAE tests whether training on the reconstruction error instead of the distortion term\noffers any advantage\" - there is an additional confounding factor which is that the beta0-VAE samples from the inference distribution q(z ; x) with a fixed variance, wheras PAE deterministically regresses a point z given x. This extra noise will typically incentivize the model to \"spread out\" z-space more, and may impact flow-based density estimation in the second stage of training. \n* If the intent with B0-VAE is to \"tests whether training on the reconstruction error instead of the distortion term offers any advantage\", then why not just replace the reconstruction term in PAE? Although it shouldn't make any difference as the Gaussian reconstruction log-likelihood (with fixed variance) is equal up to a constant to the MSE used to train the PAE. \n* The lack of discussion or comparisons to \"Latent space autoregression for novelty detection.\" [1] is a bit of a problem, as it is a well cited paper that proposes a similar strategy for novelty detection. \n\n[1] Abati, Davide, et al. \"Latent space autoregression for novelty detection.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission presents Probabilistic Autoencoder (PAE), a latent-variable generative model trained in a two-stage process. In the first stage an autoencoder is trained with a simple reconstruction objective. In the second stage a flow-based model is used to estimate the density of input data that has been encoded in latent space. The model is compared to a variety of VAE-based baselines in reconstruction, generative modelling, and anomaly detection tasks on image datasets. \n\nThe notion of training a prior over latent variables after first training an auto-encoder has been introduced previously (e.g. in VQ-VAE [1]), but this paper provides a detailed explanation of the process, as well as a range of experiments that reveal the experimental benefits of the approach compared to more typical VAE-style baselines.\n\nUsing probabilistic PCA as a motivating example, the paper explores the impact of latent dimensionality on anomaly detection, both using data and latent space density estimation-based detectors.  Again, this is not the first example of latent-space based anomaly detection (e.g. [2]), but the connections to PPCA, and the empirical exploration in this context are useful contributions.\n\n[1] Van Den Oord, Aaron, and Oriol Vinyals. \"Neural discrete representation learning.\" Advances in neural information processing systems 30 (2017).\n[2] Abati, Davide, et al. \"Latent space autoregression for novelty detection.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "* [strengthen] An expansion of the discussion in the PPCA-based anomaly detection section to clarify statements like \"this noise component is uninformative for the outlier detection task and might obscure the results\".\n* [strengthen] An more thorough explanation of how the PPCA anomaly detection results motivate the use of latent spaces for anomaly detection. (See the point in the Weaknesses for more info)\n* [critical] A comparison to existing methods in the literature for anomaly detection. In particular to \"Latent space autoregression for novelty detection.\" At the very least this method should be discussed, and ideally the authors would report AUROC scores for their methods on the MNIST and CIFAR eval regimes described in that paper. Otherwise we simply don't know how useful the proposed method is in relation to other existing methods.\n \n", "broader_impact_concerns": "I think the ethical implications of the work are quite limited and don't warrant a broader impact statement. The work presents a generative image model and it inherits the ethical concerns for this application such as privacy (deep-fakes) and emebdded bias. However it is far from a state of the art model, and is unlikely to be applied in its proposed form for any harmful purpose.   ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "04ub14UDhM", "reviewer": "Reviewer_RVev", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**: The paper proposes a fairly simple approach and shows that it is competitive with VAEs. In particular, the proposed approach does not use any kind of variational inference.\n\n**Weakness?** The main advantage of the proposed approach appears to be simplicity. However, simplicity is debatable. In particular, unlike VAEs the proposed method requires a two-stage approach, where the components of the model are trained sequentially. The VAE model is trained jointly with a single optimization process. Moreover, standard VAEs with a Gaussian latent prior (not using NFs) are arguably simpler than the proposed model.\n\n**Strength:** The experiments are done carefully. In particular, the authors present a very detailed description of the hyper-parameters and the tuning procedure. The authors developed a non-trivial $\\beta_0$-VAE baseline to ensure a fair comparison.\n\n**Weakness:** Generally, the experiments are not particularly conclusive. On the reconstruction error, PAE achieves the best results (which is expected given that unlike the baselines PAE does not include regularization, e.g. KL or noise in the encoder). However, flow-VAE achieves better FID scores. \n\n**Strength:** The proposed method achieves good out-of-distribution detection performance, improving upon VAE baselines.\n\n**Weakness:** The OOD detection experiments are somewhat limited, as the authors only consider the FashionMNIST in-distribution dataset. Moreover, multiple prior works also indicated that normalizing flows applied to embeddings can achieve strong OOD detection results [e.g. 1-3]. Moreover, it is not clear how PAE compares to specialized OOD detection methods.\n\nGenerally, the main takeaway from the paper is that PAE performs comparably to VAEs with NF latent space prior, while arguably being simpler. While this is an interesting result, it is not clear whether PAEs will be useful in practice, compared to other generative models and even VAEs with regular Gaussian priors. \n\n\n[1] *Deep Residual Flow for Out of Distribution Detection*; Ev Zisselman, Aviv Tamar\n\n[2] *Why Normalizing Flows Fail to Detect Out-of-Distribution Data*; Polina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson\n\n[3] *Hybrid Models for Open Set Recognition*; Hongjie Zhang, Ang Li, Jie Guo, Yanwen Guo", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an approach to generative modeling, which consists of two stages:\n1. Train a deterministic autoencoder model, minimizing the reconstruction error\n2. Train a normalizing flow model to estimate a density in the autoencoder embedding space.\n\nThe authors show that this approach is competitive with VAEs of the same architecture, where the flow serves as a prior in the latent space.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "**Important**:\n- Could you add a baseline using a VAE with a regular Gaussian likelihood and the same encoder-decoder architecture to the experiments?\n\nGenerally, I would feel much more comfortable recommending acceptance if the authors could demonstrate that PAEs are useful relative to other generative models, and not just comparable to specifically VAEs with NF priors. Is there an application where you would expect PAEs to be particularly useful?\n\n**Typos and minor suggestions**:\n- In Eq. (1) you say that $O \\in R^{N \\times K}$, $K < N$, and $O O^T = 1_N$. Wouldn't $O O^T$ be low-rank, as $K < N$?\n- \"this is assumption is not always satisfied\" (page 5)\n- \"lowest training loss on validation data\" (page 9)", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jbXbiLNAbm", "reviewer": "Reviewer_kA9U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Simple approach that works well in practice\n- Well written manuscript that is easy to follow\n- Good experimental evaluation\n- Thorough explanation of the motivation and extensive ablation studies\n- New metric for OOD works quite well and can be applicable to VAEs as well\n\nWeaknesses:\n- Incremental given other works in the field; this two step procedure has been explored before, e.g., in VQ-VAEs (Razavi et al., 2019) and RAEs (Ghosh et al., 2020). The main difference with, e.g., RAEs, seems to be that the authors consider a more flexible latent density model (i.e., a normalizing flow) instead of a mixture of Gaussians. \n- Some claims are not backed up by experimental evaluation (e.g., compression capabilities)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes probabilistic Autoencoder (PAE), a simple procedure motivated from probabilistic PCA. It is comprised from two stages; on the first stage a standard deterministic Autoencoder is trained on a data set and on the second stage a normalising flow prior is trained to obtain a density over the latent space of the encoder. Armed with this normalising flow prior, the authors propose to use PAE for several tasks associated usually with generative models. The authors also propose a new metric for out-of-distribution detection, namely the log probability of the latent code under the (learned) prior. There are relatively extensive ablation studies and experimental evaluation. Therefore, the contributions of this work are:\n\n1. The PAE 2-stage framework, which is simple and seems to work reasonably well in practice\n2. The new out-of-distribution metric which relies on density estimation in the latent space\n3. The several experiments done to compare PAE to VAE in a variety of tasks", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. The authors argue about the compression properties of the PAE model, but no experiments are done to show its performance. Given that they train the normalizing flow prior, I would have expected at least an estimate of the log-likelihood of the test data. The latter would serve as a reasonable proxy for the compression capabilities of PAE versus, e.g., the VAE baseline. This is a critical experiment for the data compression claim (i.e., the one at the conclusion).\n2. Given the similarity with RAEs (Ghosh et al., 2020), I would have expected that the authors did a comparison against that method as well. This would highlight whether a more flexible latent density is needed over the simpler mixture of Gaussians baseline. I would say that this is a medium importance point, given that there is also some novelty on the latent space OOD metric of this work.\n3. The data modelled in this work are discrete / have bounded support, however a Gaussian likelihood is assumed for both the VAE and PAE. What happens when a more appropriate likelihood is used? For example, on MNIST / FMNIST, you could try binarising the data and using a Bernoulli likelihood, or you could use, e.g., a discretised logistic [1] on the pixel values directly as they are 8-bit grayscale values [2]. This is not a critical experiment, but it would strengthen this work.\n4. I would appreciate if the authors discuss how is the interpolation behaviour of the standard AE that the PAE is based upon. This would highlight whether the normalising flow “fixes” potential holes in the latent space of the AE, or whether these holes are not there to begin with (and thus the normalising flow does not bring much extra compared to the standard AE). This is not a critical experiment, but it would strengthen this work.\n\nMinor points\n- In eq. 9, I take that the $p(z)$ refers to the empirical distribution of the latent codes. I would advise that the authors change this to, e.g., $\\tilde{p}(z)$ in order to distinguish it from the latent space prior (which is also denoted as $p(z)$). \n- It would be good if the authors mention how the VAE is evaluated for all the metrics. Are they averaged metrics over multiple samples from $q(z|x)$ or is only the mean of the variational posterior propagated (i.e., which is closer to what a standard AE does)? \n- At page 10, in the first sentence after “Outlier detection accuracy”, the authors mention the “outlier detection metrics listed in 1”; what is “1”? \n\n[1] PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications, Salimans et al., 2017\n\n[2] Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms, Xiao et al., 2017\n", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores a two-step approach to generative modelling: first a non-linear autoencoder is trained to reconstruct the data, then a density model (here a normalizing flow) is trained to model the latent-space distribution. This constitutes an alternative to variational autoencoders (VAEs), and the paper conducts a thorough comparison between the two.

Although the two-step approach is not new (for example VQ-VAE [1] and RAE [2] follow it too), the paper provides a number of useful insights and a thought-provoking and careful comparison to VAEs. After some requested additional experiments that the authors supplied during the discussion period, the reviewers are satisfied that the paper's claims are adequately supported by empirical results. Therefore, the paper meets the criteria for acceptance to TMLR.

[1] van den Oord et al., 2017; Razavi et al., 2019

[2] Ghosh et al., 2020

Nonetheless, the paper requires a minor revision to rephrase the main claims and bring them in line with what the experiments have demonstrated. In particular, the experiments substantiate the claims that the proposed approach has advantages compared to equivalent VAE models, but they don't support claims that the proposed model achieves state-of-the-art sample quality or outlier-detection performance. Therefore, the following statements in the abstract must be rephrased:

"the PAE [..] produces state-of-the-art sample quality"

"[...] achieves state-of-the-art results and outperforms other likelihood-based estimators"

Additional minor points to consider revising:
- In Appendix B, "batch sizefootnote" looks like a typo.
- Appendix B contains important results that would be worth highlighting in the main body (either by moving them to the main body or linking to them from the main body).
Decision: Accept (poster)