Structured Uncertainty in the Observation Space of Variational Autoencoders
Authors: James Langley
Miguel Monteiro
Charles Jones
Nick Pawlowski
Ben Glocker
Venue: ICLR 2022 Submitted (2022)

Abstract: Variational autoencoders (VAEs) are a popular class of deep generative models with many variants and a wide range of applications. Improvements upon the standard VAE mostly focus on the modelling of the posterior distribution over the latent space and the properties of the neural network decoder. In contrast, improving the model for the observational distribution is rarely considered and typically defaults to a pixel-wise independent categorical or normal distribution. In image synthesis, sampling from such distributions produces spatially-incoherent results with uncorrelated pixel noise, resulting in only the sample mean being somewhat useful as an output prediction. In this paper, we aim to stay true to VAE theory by improving the samples from the observational distribution. We propose an alternative model for the observation space, encoding spatial dependencies via a low-rank parameterization. We demonstrate that this new observational distribution has the ability to capture relevant covariance between pixels, resulting in spatially-coherent samples. In contrast to pixel-wise independent distributions, our samples seem to contain semantically meaningful variations from the mean allowing the prediction of multiple plausible outputs with a single forward pass.

Under review as a conference paper at ICLR 2022
STRUCTURED UNCERTAINTY IN THE OBSERVATION SPACE OF VARIATIONAL AUTOENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
Variational autoencoders (VAEs) are a popular class of deep generative models with many variants and wide range of applications. Improvements upon the standard VAE mostly focus on the modelling of the posterior distribution over the latent space and the properties of the neural network decoder. contrast, improv- ing the model for the observational distribution is rarely considered and typically defaults to a pixel-wise independent categorical or normal distribution. image synthesis, sampling from such distributions produces aantiall-incoherent results with uncorrelated pixel noise, resulting in only the sample mean being somewhat useful output prediction. In this paper, aim to stay true to VAE the- ory by improving the samples from the observational distribution. We propose alternative model for the observation space, encoding spatial dependencies via a low-rank parameterisation. We demonstrate that this new observational distri- bution has the ability to capture relevant covariance between pixels, resulting spatially-cohere samples. In contrast to pixel-wise independent distributions, our samples seem to contain semantically meaningful variations from the mean allowing the prediction of multiple plausible outputs with single forward pass.
INTRODUCTION
Generative modelling is one of the cornerstones of modern ma- chine learning. One of the most used and widespread classes of generative models is the Variational Autoencoder (VAF) (Kingma Welling, 2014; 2019). VAEs explicitly model the distribution of observations by assuming latent variable model with low-dimensional latent space and using a simple parametric distribution in observation space. Using neural network, VAEs decode the latent space into arbitrarily com- plex observational distributions.
Despite many improvements on the VAE model, one often- overlooked aspect is the choice of observational distribution. As an explicit likelihood model, the VAE assumes a distribu- tion in observation space using delta distribution would allow gradient based optimisation. Most current implemen- tations, however, employ only simple models, such pixel- wise independent normal distributions, which eases optimisa- tion but limits expressivity. Else, the likelihood term is of- ten replaced by reconstruction loss which, in the case of an T-2 loss, implicitly assumes an independent normal distri- bution. Following this implicit assumption, samples are then generated by only predicting the mean, rather than sampling in observation space.
Figure I: Top: Samples gener- ated with standard VAE exhibit- ing pixel-wise independent noise. Bottom: Samples with struc- tured observation space VAE are realistic and spatially coherent.
An application where this disconnect becomes apparent is im- age synthesis. The common choices for observational distri- butions are pixel-wise independent categorical or normal distributions. For pixel-wise independent distributions, regardless of other model choices, sampling from the joint distribution pixels will

---

Under review as a conference paper at ICLR 2022
result in ppatially-incoherre samples due to independent pixel noise (cf. Figure 1). To address this problem, researchers use the predicted distributions to calculate the log-likelihood in the objec- tive but then discard them in favour of the mean when generating samples or reconstructing inputs. However, this solution is akin to ignoring the issue rather than attempting to solve it.
In this work, we explore what happens when strictly follow VAE theory and sample from the predicted observational distributions. We illustrate the problem of spatial incoherence that arises from using pixel-wise independent distributions. We propose using a spatially dependent joint dis- tribution over the observation space and compare it to the previous scenario. We further compare the samples to the mean of the predicted observational distribution, which is typically used when synthesising images. We note that, in this work. we are not focusing on absolute image quality. Instead, we aim to point to an issue often overlooked in VAE theory and application, which affects most state-of-the-art methods. Thus we analyse the relative difference between using and not using a joint pixel dependent observational distribution for basic VAE. Yet, our findings are of broad relevance and our proposed model can be used in more advanced VAE variants.
2 RELATED WORK
Modern generative models can be divided into two classes, implicit likelihood models, such as Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) and diffusion models (Song & Durkan, 2021), and explicit likelihood models, such as VAEs (Kingma & Welling, 2014; Rezende et al., 2014; Kingma & Welling, 2019), flow models (Dinh et al., 2017: Kingma & Dhariwal, 2018; Dinh et al., 2015) and auto-regressive models (Van Den Oord et al., 20;;;;;; Salimans et al., 2017). Despite implicit likelihood models having achieved impressive results in terms of sample quality without the need for explicitly modelling the observation space (Karras et al., 2020; Brock et al., 2019), interest in explicit likelihood models have prevailed due to their appealing properties, such as ease of likelihood estimation.
One of the most popular and successful explicit likelihood models is the VAE (Kingma & Welling, 2014; Rezende al., 2014; Kingma & Welling, 2019). Since its introduction, there have been numerous extensions. For example, Van Den Oord et al. (2017); Razavi et al. (2019) quantize the latent space to achieve better image quality; Higgins et al. (2017); Chen et al. (2017) modify the latent posterior to obtain disentangled and interpretable representations; and Vahdat & Kautz (2020); Sønderby et al. (2016) use hierarchical architectures to improve sample quality.
Like most other explicit likelihood models, the VAE requires the choice of a parametric observa- tional distribution. This choice often pixel-wise independent. As a result, practitioners use the distribution to calculate the likelihood but use its expected value when sampling, as the samples themselves, are noisy and of limited use in most applications. However, according to the theory and as previously pointed out by Stirn & Knowles (2020) and Detlefsen et al. (2019), a latent space sample should entail a distribution over observations and not a single point. Despite attempts to enforce spatial dependencies in the decoder architecture (Miladinovic et al., 2021), without pixel- dependent joint likelihood, the observation samples will remain noisy. Notable exceptions are auto- regressive models and auto-regressive VAE decoders ( (Van Den Oord et al., 2017: Razavi et al., 2019; Gulrajani et al., 2017; Nash et al., 2021). Unlike other explicit likelihood models, auto-regressive models jointly model the observational distribution by sequentially decoding pixels while condi- tioning on previously decoded values. While this sampling procedure results in spatially coherent samples, is computationally expensive and uncertainty estimation is not trivial.
In the context of non-auto-regressive VAE decoders, work focusing on modelling a joint obser- vational distribution that accounts for pixel dependencies limited. Monteiro et al. (2020) use a low-rank multivariate normal distribution to produce spatially consistent samples in a segmenta- tion setting. However, they focus on discriminative models only. For generative models, a notable exception is a work by Dorta et al. (2018) which, similarly to our proposed method, employs a non-diagonal multivariate normal distribution over observation space. The key difference is the choice of parameterisation used for the covariance matrix. Dorta et al. (2018) predict the Cholesky- decomposed precision matrix, which grows quadratically with the size of the image. To address this computational constraint, the authors use sparse decomposition. This decomposition siders only a local neighbourhood of pixels, which limits its ability to capture long-range spatial dependencies. In contrast, our approach uses global parameterisation.
2

---

Under review as a conference paper at ICLR 2022
3 METHODS
3.1 VARIATIONAL AUTOENCODERS
We briefly revisit the theory of standard VAEs as proposed by Kingma & Welling (2014); Rezende et al. (2014). We assume prior distribution over the latent variables, p(z), probabilistic encoder of the posterior, po (z]x), and a probabilistic decoder of the likelihood, Po(x|z) In practice, the probabilistic encoder describes intractable posterior. Using variational inference, we approximate this posterior with a distribution, I (z) (z|x) given parameters Φ. Here, VAEs provide the algorithm to jointly learn the parameters & and (Kingma & Welling, 2014). Considering some dataset X [x the VAE objective is given by maximising the evidence lower bound with respect to the parameters and Φ (Bank et al., 2020; Kingma & Welling, 2014):
[(O, oi (0) - DKL 2 [zo())(())[paa)
+ (ag(atptt [og [2)
(I)
Since the derivative of the lower bound w.r.t & is problematic due to the stochastic expectation operator, the re-parameterisation trick used to yield the Stochastic Gradient Variational Bayes (SGVB) estimator (Kingma& & Welling, 2014).
Note, from the second term equation 1, that the optimisation objective requires the choice of an observational distribution to calculate the likelihood that the data comes from the predicted distri- bution: po X
(x z). Hence, follows that a latent sample entails distribution over observations. The predicted distribution should be as close as possible to the observed distribution. Its samples should look like real observations. However, in the case of highly structured data such as images, this will not be the case when using the commonly-employed pixel-independent joint distribution.
3.2 STRUCTURED OBSERVATION SPACE VARIATIONAL. AUTOENCODERS
With the commonly assumed model for the observation space, where the predicted joint distribution pixel-wise independent, samples can only add noise to the predicted mean, as shown in the exam- ples in section 4.1. By incorporating spatial dependencies in the predicted distribution, we aim to overcome this limitation and generate more realistic samples under the observed distribution.
Our solution is to replace the joint independent distribution predicted by the decoder with one that explicitly models dependencies between outputs. Specifically, we modify the final layer of the decoder to predict low-rank parameterisation of fully populated covariance matrix for in multivariate normal distribution, po(x ~ NH, E). This small modification can be applied to most existing VAE architectures.
Inspired by a recent discriminative model (Monteiro et al., 2020), we use an efficient parameterisa- tion, Σ = PPP +D, to model covariance globally, albeit at low-rank. This yields compact model it and a covariance diagonal, D = diag(()) E R(SxC), with diagonal elements, d. Here, S H W is the number of pixels, C' is the number of channels, and R is the rank of the parameterisation. Since we have only modified the distribution of the likelihood, the SGVB estimator (equation 1) is applicable without modification.
with covariance factor, P E IR x
However, we found optimising the SGVB estimator with a non de-dddagonl covariance to be unsta- ble regarding the variance. The instability comes from the fact that two routes can optimise the likelihood: to find the correct mean and appropriate variance around it or to keep increasing the variance (uncertainty about the mean). The second direction is obviously undesirable and results in implausible samples (with overly bright colours and high contrast; ef. Appendix A), Monteiro et al. (2020) observed a similar phenomenon and pre-trained on the mean to avoid it. This problem is not unique to implementation; the stability of variance networks has been discussed before (Stim& & Knowles, 2020; Detlefsen et al., 2019).
In our case, we found pre-training the mean to be beneficial but insufficient. To further mitigate the problem, our solution includes weight initialisation, fixing the covariance diagonal to a small positive scalar: D eI and constraining the entropy of the predicted distribution. Constraining the entropy constrains the variance indirectly, thus giving preference to low-variance solutions. We compute
We found 10 yield good results.
3

---

Under review as a conference paper at ICLR 2022
the entropy of the normal distribution in closed form and add it the objective function. We employ soft-constraints for the entropy and KL divergence using the modified differential method of multipliers (Platt & Barr, 1988), for its agreeable convergence and stability properties, which results in the following Lagrangian formulation:
1 M
[(0, o: x0))
Σ m=1
log po (x((0(12((
(2)
M
B DKL [ - EKL
An H((x()))) -E
where 3 and EKZ are the Lagrangian multiplier and the slack variable, respectively, for the B- 3-VAE
constraint; (xx(()) (2) is the entropy of the predicted distribution and Xи and &or are the Lagrangian multiplier and the slack variable, respectively, for the new constraint.
4 EXPERIMENTS AND RESULTS
4.1 STANDARD VAE VS. STRUCTURED OBSERVATION SPACE VAE
We start by comparing standard VAE, with a pixel-wise independent normal observational distri- bution, to the proposed method, with low-rank multivariate normal observational distribution. We perform the comparison in two datasets: the CELEBA dataset (Liu et al., 2015) and the UK Biobank (UKBB) Brain Imaging dataset (Miller et al., 2016). For all models, we use latent space of dimen- sion 128. and a target KL loss. For the low-rank model, we use rank of 25. For the CELEBA dataset we target loss, EKL, of 45 for both models and &μ 504750 for our model. For the UKBB dataset we use a target KL loss LKL. of 15 for both models and &, 198900 for our model. Figures 2a & 2b and 2c 2d show the qualitative results for the comparison.
In Figures 2a and 2c, we see that the samples of the standard VAE exhibit uncorrelated pixel noise around the mean, resulting from the pixel-wise independent joint observational distribution. In contrast, in Figures 2b and 2d, we see that the samples produced by our method contain semantically meaningful variations around the mean and are spatially coherent, as illustrated in the difference (row 3) between the mean (row 1) and the sample (row 2). Looking at the variance of the two methods (row 4), we see a significant difference in the regions where each model is uncertain, highlighting the difference in behaviour between the two predicted distributions. The predicted covariance (rows and 6) for the low-rank model contains a structure that pertains to the image content. These figure rows represent positive and negative covariance to the central pixel indicating global covariance can is modelled. This structure results in spatially coherent samples as opposed the noisy samples of the standard VAE, which are consequence of the diagonal covariance. Interestingly, we observe more variation in the means of the standard VAF, suggesting that as more variation can be modelled in the observation space, less needs to be modelled in the latent space.
Quantitative evaluation of generative modelling is an inherently difficult task due to its subjective nature. While measuring the log-likelihood is the obvious choice, it often not indicative of sample quality (Theis et al., 2016; Borji, 2019). The Fréchet Inception Distance (FID) (Heusel et al., 2017) the current standard choice of metric due its consistency with human perception (Borji, 2019). We note this metric is not without its criticisms (Borji, 2019; Razavi et al., 2019). regardless, we use it evaluate our generative models and report the results in Table The results do not represent the absolute performance of the proposed method, but rather the relative difference when compared to a standard VAE with a pixel-wise independent observational distribution while everything else is constant. We emphasise, the proposed method is compatible with generative models from other works. We observe that, for both datasets, the proposed method achieves a lower FID score than the standard VAE. Notably, the samples from the model with low-rank multivariate normal obser- vational distribution outperform the means standard model, indicating that sampling from the observational distribution, theory entails, does not reduce image quality.
4

---

Under review as a conference paper at ICLR 2022
(a) Standard VAE on CELEBA
Structured Observation Space VAE CELEBA
(c) Standard VAF UKBB Brain Scans
(d) Structured Observation Space VAE UKBB
Figure 2: Qualitative results comparing a standard VAE (2a) and the proposed VAE (2b) on the CELEBA dataset. Same comparison on the UKBB dataset (2c 2d). The rows from top bottom: the mean of the observational distribution, a sample from the observational distribution, the differ- ence between the mean and the given sample, the pixel-wise independent variance per pixel, a slice of the covariance matrix: positive covariance and negative covariance to the central pixel.
Table 1:
FID metric results for a standard VAE with a pixel-wise independent observational distri- bution and our modified VAE. Lower FID scores represent better performance.
Method
Dataset
FID J
Standard VAE (means)
121.65
Standard VAE (samples)
196.40
CELEBA
Our VAE (means)
132,93
Our VAE (samples)
104.62
Standard VAE (means)
211.24
Standard VAE (samples)
332.89
UKBB
Our VAE (means)
141.87
Our VAE (samples)
79.712
4.2 INTERPOLATION IN THE OBSERVATION SPACE
To explore the expressiveness of the representations captured in the observation space, visualise a continuous range samples from the proposed method. We perform spherical linear interpola- tion over the observation space capture range of plausible images between initial samples,
5

---

Under review as a conference paper at ICLR 2022
Figure 3: Spherical linear interpolation between auxiliary noise variables wp. The four corners are random samples from predicted distribution with all intermediate steps as interpolations be- tween them. The two images represent interpolations in the observation space for two observational distributions predicted from different latent codes
Ya and yn- This is shown in equation 3, where Wp E IR R and Wa (SxC)) are both auxiliary noise variables, slerp is the spherical interpolation function (see appendix D) and E [0, 1] is the interpolation factor.
= + Pwp. + Vews
(3)
where wp, slerp(wp., wp,
It is important to note that we only interpolate over wp because it restricts the dimensionality of the hyper-sphere to size R: the v(wa wa term only adds small amount of uncorrelated noise, so setting as a constant has negligible effect. Figure 3 uses this technique to visualise the variation contained the observation space, demonstrating that the distribution captures semantically relevant features, such as hair colour, skin tone and background colour for the CELEBA dataset. Interpolation between differences in such features would typically entail interpolating latent variables and a forward pass through the decoder for each interval, which is not required here.
4.3 INTERACTIVE SAMPLING FROM THE OBSERVATION SPACE
We have demonstrated that low-rank multivariate normal observational distribution can model a range of features. However, it would be useful if could synthesise images with semantically meaningful human input. A step in this direction entails fixing the auxiliary noise variables asso- ciated with each sample and scaling the principal components of our covariance factor, P. This is demonstrated in equation 4: using the singular value decomposition (SVD) of P and introducing diagonal matrix of scaling coefficients, A E (RxR)
(4) Adjusting the scaling coefficients A allows us to tune spatially correlated features in the sam- ple image. The use of SVD often makes the effect of each coefficient separable and semantically relevant to the image domain. Images generated through this method for CELEBA are shown in Figure 4, where each row demonstrates the effect of scaling a different principal component. This figure shows the effect on the first ten principal components. The effect on all components and addi- tional results the UKBB data given in the Appendix B. Since this manipulation is using only the observational distribution, manipulation of these samples can be achieved without performing additional forward passes the model.
P - U(SA)V

---

Under review as a conference paper at ICLR 2022
Figure 4: The effect of scaling each of the ten most principal components (each row), from top to bottom for a fixed auxiliary noise variable. The scale factor for each component ranges from -5 to +5 with intervals of 0.5.
4.4 INTERACTIVE EDITING OF PREDICTIONS
One of the benefits of modelling spatial correlations in the observational distribution that this information can be leveraged interactively edit predictions. This involves manually editing part of the prediction and calculating the conditional distribution of the remaining pixels; for an arbitrary multivariate normal distribution, this is expressed in equations 5 and where the edited pixels are modelled by Y2 and the remaining pixels are modelled by y1- We use the mean of the conditional distribution (ji) as the corrected image. The updated covariance matrix (E) is not needed, so we avoid evaluating to reduce the computational cost of this process.
y1 Y2 μ
[1 122
En
E
E
(5)
Σz1
p(y1ly2 = b) ~ NNù, Σ
(6)
where j1 11 + E En bb- 142)
and Σ E11 Σ₁ E21
Interactive editing is demonstrated Figure 5 (with additional examples in Appendix C), where prediction from our model, trained on the CELEBA dataset, is sequentially edited to alter the hair colour and skin tone. This demonstrates the power of the method: we can manually edit small number of pixels and automatically update the remainder of the image coherently with the manual edit.
4.5 OBSERVATIONAL. DISTRIBUTION WITHOUT DEEP LEARNING
Typical VAEs rely their deep learning components model the features for their output. In contrast, since we use a low-rank parameterisation of a full covariance matrix, our observational distribution model spatially-correlate
features its own. As result, the ability to model these features is not solely left to the deep learning components of the VAE; it is shared with the linear transformations that compose the low-rank multivariate normal distribution.
To understand the expressiveness of our observation space model, we carried out an experiment in which the VAE architecture is replaced with the parameters for the low-rank multivariate normal
7

---

Under review as a conference paper at ICLR 2022
Figure 5: Sequentially editing hair colour and skin tone interactively. From left to right: a predicted image with small coloured edit made the hair, the image after the conditional distribution has been calculated, the image with further edit to the skin tone. the image after the conditional distri- bution has been recalculated. N.B: The red circles highlighting the manual edits are for illustration purposes only and serve no computational purpose.
Figure 6: qualitative comparison between 100 samples from the learnt observation space at rank 25 with no deep learning components (left) and 100 samples from linear PCA model with 25 features (right). In both cases, the data used for fitting is the same random subset of 10000 images from the CELEBA dataset.
distribution with no deep learning layers or latent space representation involved. We then train this simple model as described in section 3.2. This allows us to examine the capability of the model to capture the observational distribution over the dataset, without deep learning.
This modelling reminiscent of principal component analysis (PCA), particularly given that we low-rank parameterisation, akin to the dimensionality reduction of PCA. Figure 6 compares samples from the distribution alongside samples from a linear PCA model equivalent feature reduction after fitting to the CELEBA dataset (Liu et al., 2015).
There is little perceivable difference between the samples of the two methods, so we conclude that low-rank multivariate normal distribution is comparably expressive to PCA for feature reduction. Furthermore, this experiment confirms the ability to learn the parameters of our distribution through backpropagation.
8

---

Under review as a conference paper at ICLR 2022
5 DISCUSSION
This paper highlights an often-overlooked aspect of the VAE architecture the observational distri- bution. We have confirmed that pixel-wise independent observational distributions produce samples with uncorrelated pixel noise. We have introduced a low-rank multivariate normal distribution as a choice for the observational distribution of a VAE, able to model covariance between pixels and produce multiple spatially coherent samples with single forward pass of the decoder. Our method introduces stability issues that are otherwise not present, but that we are able to resolve with an en- tropy constraint. Our results indicate that our choice of observational distribution is beneficial when compared to a pixel-wise independent distribution, as well as allowing sampling to be the primary method of image synthesis without reduction in quality, theory would entail. Our method is com- patible with many VAE architectures and may be applied to state-of-theart models. We find that a low-rank multivariate observational distribution interactive manipulation of samples with a single decoder forward pass from single latent variable.
be interpolated within, and allows for semantic,
Introducing an expressive observational distribution that is able model features on its own, as we have, promotes discussion comparing the features modelled in the observation space to those modelled in the latent space, In section 4.5, we observe our observational distribution's 's ability to model features on its own and in section 4.1 we observe decrease in variation of the predicted means for our model compared a standard VAE. Assuming dataset containing finite uncertainty, we deduce that the modelling of this uncertainty is split between the latent space and the observation space. Understanding where this split lies and what influences this is an open question left for future work.
ETHICS STATEMENT
Generative modelling is subject dataset-inherited bias and our contributions are also susceptible. Whilst methods such some cases a desirable tool, typical usage may expose an undesired bias, particularly after training on the CELEBA dataset, which has larger societal impact. We acknowledge that these biases, such as lack of diversity, are present but state these as artefacts of the chosen dataset and not of our own design, opinions or beliefs. There is clear need for more diverse and representative datasets that would allow a more complete picture of the abilities and limitations of generative models to be obtained.
allow us to explore the biases that exist within a dataset, which in
Our method allows for the synthesis of multiple plausible samples as well as manipulation of sam- ples with single forward pass of the model, where other methods require multiple forward passes. This reduction in computational burden could provide access to machine learning models for those with low-powered devices as well as reducing energy consumption and computational costs.
REPRODUCIBILITY
All our code will be made publicly available a dedicated GitHub repository. The CELEBA data publicly available and we will ensure that these results are fully reproducible with the provided code. Access to the brain imaging data can be requested via a data access application to the UK Biobank Study (https: : www. ukbi obank. ac. uk/).
REFERENCES
Dor Bank, Noam Koenigstein, and Raja Giryes. Autoencoders. CoRR, abs/2003.05991, 2020. URL
https: petpss//arkerr //arxiv...ivve org/ abs 2203 05991.
Ali Borji. Pros and cons of GAN evaluation measures. Comput. Vis. Image Underst., 179:41- 65, 2019. doi: 10.10166..cvi..2018.. 10.009. URL httph::t///ttt http::://do...o///1/// 101600.......11 cvii. 2018.10.005
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fi- delity natural image synthesis. In 7th International Conference Learning Representations,

---

Under review as a conference paper at ICLR 2022
ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: wetorrmmiiiBBlee
// //openeeview.... htt.://rrrmmmmiilllsssss0------ x
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In 5th International Confer- ence on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:/ BysvGP5ee.
/ttp.ttt////ppeeeiiieett /wwp/rpppe/eww..//////// net / forum?id=
Nicki Skafte Detlefsen, Martin Jørgensen, and Søren Hauberg. Reliable training and esti- mation of variance networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alcch-Buue, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pp- 2019. URL https: htt.::::/ppooceediie Abstract html.
neuri wwwuripps.... pss, htt/ppaarr/211//astt//s
07211 (121166888899999594499
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components es- timation. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL http: / /aa/aaiivvv.rr..///aa wtw./aaaa//aaooooooeegog/a/eaaea abs 8516..
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:::p///.......t. /owpr//iip..r...e..emmm net / Eww.eoirnniiii............ -iiidkdkKK999999
G. Dorta, S. Vicente, L. Agapito, N. D. F. Campbell, and I. Simpson. Structured uncertainty predic- tion networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 5477-5485, 2018. doi: 1111009/CVPR.2118000007
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vázquez, and Aaron C. Courville. Pixelvae: A latent variable model for natural images. In 5th Interna- tional Conference on Learning Representations, ICLR 20/7, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: https:://ppeervieee.e net/ B.-umm?dd-JKKvt5....
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vish- wanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6626-6637, 2017. URL https http:///ooccedgeiccooo nwwrips.crirooooooooooo oa/ paper/2017 http//0001hhhhh/////
Abst html.
I. Higgins, Loic Matthey, A. Pal, C. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In Proc. CVPR, 2020.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL htlttttp//axxaa org/
bbss1312, 6114.
Diederik P. Kingma and Max Welling. An introduction to variational autoencoders. Foun- dations and Trends@ in Machine Learning, 12(4):307-392, 2019. ISSN 1935-8245. doi: 10.1561/2200000566 URL httptt//////xx//// doi.
10

---

Under review as a conference paper at ICLR 2022
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolu- tions. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL https: (.9db6a2333 cet
and R. Gar-
wtw.eeeediininneeeeeeeeeeeeeeeee neurips www.eeew.....aaappeeeeee pwtpppper/211//1111/ pdf.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Dorde Miladinovic, Aleksandar Stanic, Stefan Bauer, Jürgen Schmidhuber, and Joachim M. Buh- mann. Spatial dependency networks: Neural layers for improved generative image modeling. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Aus- tria, May 3-7, 2021. OpenReview.net, 2021. URL https: wpw.reveeeeeeeeeeeeeeeeeee / net id=I4c4K9
forum?
Miller, F Alfaro-Almarry N K Bangerter, Thomas, Yacoub, Xu, Bartsch, Jbabdi, S N Sotiropoulos, J L R Andersson, L Griffanti, G Douaud, T W Okell, P Weale, I Dragonu, Garratt, Hudson, R Collins, Jenkinson, P Matthews, and S Smith. Multimodal population brain imaging in the uk biobank prospective epidemiological study, 2016.
Miguel Monteiro, Loic Le Folgoe, Daniel Coelho de Castro, Nick Pawlowski, Bernardo Marques, Konstantinos Kamnitsas, Mark van der Wilk, and Ben Glocker. Stochas-
tic segmentation networks: Modelling spatially correlated aleatoric uncertainty. In Hugo
Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hiuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL pettps://proceeeee iww.oceeddings.nerii..... neurips www.:/ps.c.ppppp////...... 95f8d9901caB
hec/paperr2220/hase0 Abstrratt....tt.tt.............. html.
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W. Battaglia. Generating images with sparse representations. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Inter- national Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, vol- ume 139 of Proceedings of Machine Learning Research, pp. 7958-7968. PMLR, 2021. URL .mlr. www.ppeessvlli wtw.s/hla...a.......eaa
John Platt and Alan Barr. Constrained differential optimization. son (ed.), Neural Information Processing Systems. American Institute of Physics, 1988.
D. Ander-
URL https://prroeeeingg..ot neurips https//ewi...ccccc////e//eeeeee wwtpcpappe//////E1e.. a87 F
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnet (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 14866-14876. Cur- ran Associates, Inc., 2019. URL https::/prooeeddigg neurips wwtrrisscccppaeaee /pcpppppe//219//////// pdf.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on International Conference on Machine Learning Volume 32, ICML'14 14, pp- 11-1278-11-1286 JMLR.org. 2014.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcnn++: A pixelcnn imple- mentation with discretized logistic mixture likelihood and other modifications. In ICLR, 2017.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Lad- der variational autoencoders. In Proceedings of the 30th International Conference Neural Information Processing Systems, NIPS' 16, pp. Associates Inc. ISBN 9781510838819.
Red Hook, NY, USA, 2016. Curran
Yang Song and Conor Durkan. Maximum likelihood training of score-based generative models. CoRR, abs/2101 00925, 2021. URL https: /ttpas//////iv.///eeeeee
/ttp://wwwwwwwww///////////////e www.:oss//////oeeo 01. 09258.

---

Under review as a conference paper at ICLR 2022
Andrew Stim and David A. Knowles. Variational variance: Simple and reliable predictive variance parameterization. CoRR, abs/2006.04910, 2020. URL battpa://arxiv... http//awiiii......a.a/oooo abs/ 04910.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceed- ings, 2016. URL http: www.v.oroo...........a...o..aa.a abs 15114.1844.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Infor- mation Processing Systems (NeurIPS), 2020.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference Machine Learning Volume 48, ICML'16, pp. 1747-1756. JMLR.org, 201
Aäron Van Den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelenn decoders. Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 4797-4805, Red Hook, NY, USA, 2016b. Curran Associates Inc. ISBN 9781510838819.
Aäron Van Den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 6306-6315, Cur- ran Associates, Inc., 2017. URL https://proceediing www.ins. peeurips.cc/paaee/2077/ aat. Pastocc-pppppppp.p
12

---

Under review as a conference paper at ICLR 2022
A
OUT-OF-THE-BOX SAMPLES WITHOUT STABILISING
Figure 7: The results after training with a pre-training phase and weight initialisation, but without the fixed component, D cI, or the entropy constraint. The first row represents the predicted means and all other rows represent samples from the predicted distribution outputted from the probabilistic decoder. Each column is a new sample from the latent prior decoded to predict distributions over the observation space. Model trained for 100 epochs on random subset of 10000 images from the CELEBA dataset, Latent dimensionality: I 128, rank: R 25, target KL loss: εKL = 45.
13

---

Under review as a conference paper at ICLR 2022
B SCALING OF ALL INDIVIDUAL PCA COMPONENTS FOR CELEBA AND UKBB
Figure 8: The effect of scaling each of the principal components (each row), from top to bottom for a fixed auxiliary noise variable and a rank 25 parameterisation. The scale factor for each component ranges from -5 to +5 with intervals of 0.5. Observational distribution predicted by our VAE after training on the CELEBA dataset.
14

---

Under review as a conference paper at ICLR 2022
Figure 9: The effect of scaling each of the principal components (each row), from top to bottom for a fixed auxiliary noise variable and rank 25 parameterisation. The scale factor for each component ranges from -5 to +5 with intervals of 0.5. Observational distribution predicted by our VAE after training on the UKBB dataset.
15

---

Under review as a conference paper at ICLR 2022
C ADDITIONAL EXAMPLES FOR INTERACTIVE EDITING
Figure 10: Sequential interactive editing. From left to right: predicted image with a small coloured edit made the hair, the image after the conditional distribution has been calculated, the image with further edit to the skin tone, the image after the conditional distribution has been recalculated. N.B: The red circles highlighting the manual edits are for illustration purposes only and serve no computational purpose.
Figure 11: Interactive editing. Left: predicted image with single-pixel manual coloured edit over the hair. Right: the mean of the calculated conditional distribution. N.B: The red circle highlighting the manual edit is for illustration purposes only and serves no computational purpose.
Interactive editing. Left: predicted image with manual coloured edit over the skin. Right: the mean of the calculated conditional distribution. N.B: The red circle highlighting the manual edit is for illustration purposes only and serves no computational purpose.
Figure 12:
Figure 13: Interactive editing. Left: predicted image with a manual coloured edit over the back- ground. Right: the mean of the calculated conditional distribution.
16

---

Under review as a conference paper at ICLR 2022
D SPHERICAL INTERPOLATION
For completeness, we include the spherical interpolation formula used in section 4.2.
slcrp(a, b,t) = sin t)w) a +
sin(tw)
(7)
sin w
sin w
where w cos ([a) - [b|)
17


Reviews:
[{"review_id": "wU3wSDRlVij", "reviewer": "Reviewer_cypB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem the authors tackle in this paper and the proposed solution are interesting. The submission is technically sound and the paper is well written.\n\nHowever, I have some major concerns about the originality of the work and feel that further comparison with other methods would be needed to show improvements over the state-of-the-art methods. Specifically:\n1) this work is closely related to Monteiro et al. (2020), who used a similar model in a segmentation setting. Is the model proposed here the same considered in Monteiro et al? If not, the authors should highlight methodological differences. If yes, the authors should highlight the conceptual contributions of their work.\n2) Dorta et al. (2018) considered a closely-related model, but making a different modeling choice for the covariance matrix. The authors should compare their results with those obtained with the model from Dorta et al (e.g., comparison of FID scores of generated samples).\n\nOther questions / concerns:\n- I find it hard to judge whether the means from the structured-observation-space VAE look visually more realistic than the mean from the canonical VAE. Can the author comment on this? Also, does not \"fixing the covariance diagonal to a small positive scalar\" artifactual reduce independent noise in the samples? I wonder if a visual comparison with canonical VAE is fair in this setting.\n- Are there other potential use cases for the observational model beyond pixel editing?\n- I think another way to improve upon the canonical VAE likelihood is to consider an iid gaussian likelihood on perceptual features [1]. The authors should consider this method in the related method section and potentially include this technique in the comparison.\n\n[1] Hou X, Shen L, Sun K, Qiu G. Deep feature consistent variational autoencoder. In2017 IEEE Winter Conference on Applications of Computer Vision (WACV) 2017 Mar 24 (pp. 1133-1141). IEEE.", "summary_of_the_paper": "The authors aim at improving the canonical VAE model by replacing the standard iid Gaussian likelihood with a multivariate Gaussian with (low-rank + diagonal) covariance.\n\nIn applications to CelebA and a brain MRI dataset from UK Biobank, the authors compare the proposed structured-observation-space VAE with a canonical VAE, showing that the samples from their model have lower Fréchet Inception Distance (FID) scores than both means and samples from the canonical VAE. The authors then evaluate the expressiveness of the representations learned in the observation space by visually evaluating interpolations in the observation space, and images obtained by rescaling the principal components of the observational covariance matrix. Finally, the authors show how their model can be used for interactive editing—i.e., editing a small number of pixels and using the conditional distribution to infer coherent edits in the remainder of the image.", "summary_of_the_review": "The paper is sound, well written and tackles an interesting problem. However, more work is needed to demonstrate improvement upon state of the art methods (specially, Dorta et al, 2020, and potentially Hou et al, 2017).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qf01PjnvbSo", "reviewer": "Reviewer_DqfM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "More quantitative results are needed to demonstrate possible advantages of the proposed method. Overall the only contrubution of this paper is to transfer the method of Monteiro et al. into the variational autoencder framework. Therefore this paper should clearly demonstrate the benefits of the proposed method over the established practice of using the predicted mean output vector of the decoder.\nFig. 4 a) and b) compare the proposed method with the established practice of using the predicted mean output vector of the decoder. However in Fig. b) the samples from the proposed method shows some unfavourable color shift. Thus the mean vector creates the more realistic image and would still be the preferable option.\n\nMore quantitative results are needed:\n\n- \"Structured uncertainty\": Is the uncertainty calibrated? Please provide experiments on synthetic data, for example sampled from a multivariate distribution with known mean and variance to demonstrate that the framework actually models the uncertainty of the data distribution.\n- Use of constraints in Eq. 1: Is convergence of the model affected? Are the Lagrangian multupliers maximized during optimization? Does this affect the numerical stabiilty during training?\n- How were the dataset-specific slack variables in Eq, 1 chosen? \n\nReferences\n\n[Monteiro at al. 2020] Monteiro, Miguel, et al. \"Stochastic segmentation networks: Modelling spatially correlated aleatoric uncertainty, Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty, Neurips 2020\n", "summary_of_the_paper": "In the standard Variational Autoencoder framework the statistics of the decoder output are assumed to be pixel-independent Gaussian which can lead to problems when sampling from the model when covariances are missing. To overcome these limitations, the authors propose to use the network architecture proposed by Monteiro at al. 2020 in the decoder of a variational autoencoder. In the approacah of \nMonteiro at al. 2020, the output distribution is modeled as low-rank multivariate normal. Qualitative results on the CelebA dataset demonstrate that samples from the proposed modified variational autoencoder capture covariances between different parts of the image.", "summary_of_the_review": "The contribution of this paper is limited, it merely takes an existing method to capture covariances of model outputs and incoroporates it into the decoder of a variational autoencoder. More quantitative results are needed to demonstrate possible advantages of the proposed method and calibration of the estimated covariances. Further details on parameter optimization and convergence are needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XW5XCEEyZyM", "reviewer": "Reviewer_uSoj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written, easy to follow, and well motivated. I agree with the authors that having more flexible decoder distributions, and not just more flexible neural networks (mapping to parameters of simple distributions) is an understudied area in VAEs that deserves work. The paper is very simple and relies mostly on empirical results, but I do think that the authors properly show improvements and added benefits over using diagonal covariance matrices.\n\nOne concern I have though is that it seems like adding this structure to the covariance matrix come with its share of training instabilities. The authors need to add regularizers (eq. 2), and some of the parameters used look incredibly specific (e.g. $\\xi_H=-504750$ and $\\xi_H=-199250$ for different datasets). How were the values of $\\xi_H$ chosen? How robust are the results to changes in this hyperparameter? I think a careful answer to these questions should be presented, at least in the appendix to strengthen the paper.\n\nAlso, while I think the experiments do suggest improvements over diagonal covariance matrices, I think there are relevant additional experiments that are left out. Firstly, as the authors correctly point out, they are not the first to use a structured covariance matrix in the decoder of a Gaussian VAE [1], although this previous work uses sparse covariance matrices rather than low-rank plus diagonal ones. The absence of comparisons against this other structure significantly weakens the paper in my opinion: the baseline should not be diagonal covariance, it should be sparse covariances based on pixel neighbouring structure (as in [1]); and the reader should be convinced that low-rank plus diagonal is a more sensible choice than sparse. In other words, the authors claim \"([1]) limits its ability to capture long-range spatial dependencies\", and they should empirically verify that this actually hurts performance. Some more minor experimental things that I believe are also left out are: The authors should report \"Our VAE (means)\" in Table 1 as well. I imagine these results would be worse that the standard VAE's given for example the grey backgrounds in CELEBA, but I think this is actually interesting! It means the decoder is actually being used as a distribution in the generative model. I also believe that log-likelihoods (or ELBOs) should be reported in Table 1.\n\nI will consider increasing my score if my main issues (hyperparameter instability, and comparisons against [1]) are adequately addressed during the rebuttal period.\n\nMinor things:\n\n-Citation links seem to be formatted in a different way than other papers I reviewed (they appear in blue in other papers).\n\n-The notation S = H x W seems a bit weird: why collapse only height and width, but not channels C into a single variable? That is, S could be defined as S = H x W x C and notation would be simplified.\n\n-Above eq.3, $t \\in [0..1]$ should be $t \\in [0,1]$ I believe.\n\n-The slerp function in eq.3 should be explicitly written down, at least in the appendix.\n\n[1] Structured uncertainty prediction networks, Dorta et al. 2018\n\n========================================================================================================\n\nUPDATE 1 AFTER REBITTAL\n\n========================================================================================================\n\nI have read the author's rebuttal, and while I understand that adding the comparisons against Dorta et al. is a significant amount of work, I am not willing to increase my score until definitive comparisons have been done, even if agree with the authors that the method of Dorta should not add large-scale noise, as ELBO/FID comparisons could remain interesting.\n\n========================================================================================================\n\nUPDATE 2 AFTER REBITTAL\n\n========================================================================================================\n\nI have read the updated author's rebuttal with comparisons against Dorta et al.'s method. I decided to increase my score, and still want to encourage the authors to include ELBO comparisons in an updated version of the manuscript.", "summary_of_the_paper": "This paper proposes to use a low-rank plus diagonal covariance matrix, rather than the usual diagonal ones, in the decoder of Gaussian VAEs. The authors empirically show the advantages of using more expressive covariance matrices in the decoder.", "summary_of_the_review": "While I think this paper proposes a sensible idea and shows improvements over using diagonal covariances in Gaussian VAE decoders, I have concerns about the stability of the proposed method; and I believe that the authors should compare against a stronger baseline than the one they use for the experiments to be fully convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Dq9wbfEFeWD", "reviewer": "Reviewer_2FaS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The authors provide an adequate introduction and summary of the literature, and highlight how their proposed method could be plugged into existing models.\n\nWeaknesses:\n- The approach is not radically novel and the paper could improve further by providing all details to allow the reader to fully reproduce their results. This could be included in an extended supplementary section.\n- Another part that could greatly improve the paper is some analysis of the scale of the variables. \n- The paper would greatly benefit of analysis showing how the approach extended into a more complex VAE framework further improve any downstream task. \n- How should the reader interpret the scales of the Lagrangian multiplier and the slack variable in Eq (2)? In section 4.1 these values are reported for (2) datasets. Adding additional analysis/explanation for these variables would benefit the paper and help persuade any reader about the robustness of  the results presented.\n- Figure 2 would also benefit from further explanation to better understand how these results support the claim of the authors.", "summary_of_the_paper": "The paper proposes to include some structure in the observation model of the standard VAE, typically implemented with a normal distribution with diagonal covariance matrix. The authors follow the work in [Monteiro 2020] which include low rank structure structure in the covariance matrix for the observation model in another generative framework.\n", "summary_of_the_review": "\nThe paper provides an extension of the work in [Monteiro et al. 2020] to the case of VAEs. The paper provides adequate references but it would greatly benefit for more downstream experiments and detailed explanation as suggested in the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In order to evaluate the evidence lower bound (ELBO), VAEs typically use a parametric distribution-based decoder $p(x|z)$. If the data is continuous, one often considers a Gaussian VAE, where the canonical setting is to assume a diagonal covariance matrix $p(x|z) = N(x; \mu(z), \sigma^2 \mathbf{I})$. In this paper, the authors suggest replacing the diagonal covariance matrix with a structured covariance matrix (low-rank + diagonal). As this only amounts to a minor change to a canonical Gaussian VAE, strong empirical results are expected to justify its acceptance. However, the image generation results presented in the paper are not comparable to the state-of-the-art VAE results (e.g., Arash Vahdat, and Jan Kautz. "NVAE: A Deep Hierarchical Variational Autoencoder." Neural Information Processing Systems (NeurIPS), 2020).
Decision: Reject