On Incorporating Inductive Biases into VAEs
Authors: Ning Miao
Emile Mathieu
Siddharth N
Yee Whye Teh
Tom Rainforth
Venue: ICLR 2022 Poster (2022)
Keywords: VAEs
Variational autoencoders
Variational auto-encoders
Representation learning
Inductive biases

Abstract: We explain why directly changing the prior can be a surprisingly ineffective mechanism for incorporating inductive biases into variational auto-encoders (VAEs), and introduce a simple and effective alternative approach: Intermediary Latent Space VAEs (InteL-VAEs). InteL-VAEs use an intermediary set of latent variables to control the stochasticity of the encoding process, before mapping these in turn to the latent representation using a parametric function that encapsulates our desired inductive bias(es). This allows us to impose properties like sparsity or clustering on learned representations, and incorporate human knowledge into the generative model. Whereas changing the prior only indirectly encourages behavior through regularizing the encoder, InteL-VAEs are able to directly enforce desired characteristics. Moreover, they bypass the computation and encoder design issues caused by non-Gaussian priors, while allowing for additional flexibility through training of the parametric mapping function. We show that these advantages, in turn, lead to both better generative models and better representations being learned.

Published as a conference paper at ICLR 2022
ON INCORPORATING INDUCTIVE BIASES INTO VAES
Ning Miao! 1*
Emile Mathieu'
N. Siddharth
Yee Whye Teh Tom Rainforth"
ABSTRACT
We explain why directly changing the prior can be surprisingly ineffective mech- anism for incorporating inductive biases into variational auto-encoders (VAEs), and introduce a simple and effective alternative approach: Intermediary Latent Space VAEs (InteL.-VAEs) InteL-VAEs use an intermediary set of latent variables to control the stochasticity of the encoding process, before mapping these turn to desired the latent representation using parametric function that encapsulates inductive bias(es). This allows us to impose properties like sparsity or clustering learned representations, and incorporate human knowledge into the generative model. Whereas changing the prior only indirectly encourages behavior through regularizing the encoder, -nteL-VAEs are able to directly enforce desired character- istics. Moreover, they bypass the computation and encoder design issues caused by Gaussian priors, while allowing for additional flexibility through training of the parametric mapping function. We show that these ad advantages, in turn, lead to both better generative models and better representations being learned.
INTRODUCTION
VAΣs provide a rich class of deep generative models (DGMs) with many variants (Kingma & Welling, 2014; Rezende & Mohamed, 2015; Burda et al., 2016; Guirajani et al., 2016; Vahdat & Kautz, 2020). Based on an encoder-decoder structure, VAFs encode datapoints into latent embeddings before decoding them back to data space. By parameterizing the encoder and decoder using expressive neural networks, VAEs provide a powerful basis for learning both generative models and representations.
The standard VAE framework assumes isotropic Gaussian prior. However, this can cause issues, such as when one desires the learned representations to exhibit some properties of interest, for example sparsity (Tonolini al., 2020) clustering (Dilokthanakul et al., 2016), when the data distribution has very different topological properties from a Gaussian, for example multi-modality (Shi et al., 2020) or group structure (Falorsi al., 2018). Therefore, variety of recent works have looked to use non-Gar Gaussian priors (van den Oord et al., 2017; Tomczak & Welling, 2018; Casale et al., 2018; Razavi al., 2019; Bauer & Mnih, 2019), often with the motivation of adding inductive biases into the model (Davidson et al., 2018b; Mathieu et al., 2019b; Nagano et al., 2019; Skopek et al., 2019).
In this work, we argue that this approach of using non- non-Gaussian priors can be a problematic, and even ineffective, mechanism for adding inductive biases into VAEs. Firstly, non Gon-Gaussian priors will often necessitate complex encoder models to maintain consistency with the prior's 's shape and dependency structure (Webb et al., 2018), which typically no longer permit simple parameterization, Secondly, the latent encodings are still not guaranteed to follow the desired structure because the 'prior' only regularizer the encoder. Indeed, Mathicu et al. (2019b) find
appears in the training objective that changing the prior is typically insufficient in practice to learn the desired representations at a population level, with mismatches occurring between the data distribution and learned model.
To provide an alternative, more effective, approach that does not suffer from these pathologies. introduce Intermediary Latent Space VAEs (AnteL-VAEs), an extension to the standard VAE framework that allows a wide range of powerful inductive biases to be incorporated while maintaining an isotropic Gaussian prior. This is achieved by introducing intermediary that deal with the stochasticity of the encoding process before incorporating the desired inductive biases via parametric function that maps these intermediary latents the latent representation itself, with the decoder taking this final representation as input. See Fig. for an example.
of latent variables
'epprrmeent of Statistics, University of Oxford, UUniversity of Edinburgh Correspondence to: Ning Miao maing.mian&e ox.as..x.cc.kk,, Tom Rainforth <rainforth @ stats.ox.ac.ukk

---

Published as a conference paper at ICLR 2022
Representation
Generation
4
Pa
X
y
Z
X
Figure 1: Example InteL-VAE with star-like data. We consider the auto-encoding for two example datapoints (I1 and I2, shown in green), which are first stochastically mapped to ) using a Gaussian encoder. This embedding is then pushed forward to using the non-stochastic mapping ge. which a radial mapping to enforce a spherical distribution. Decoding is then done in the standard way from 2, with the complexity of the decoder mapping simplified by the induced structural properties of Z.
The VAE framework provides a variety of advantages over directly replacing the prior. Firstly, it directly enforces our inductive biases the representations, rather than relying on the regularizing effect of the prior to encourage this implicitly. Secondly, it provides a natural congruence between the generative and representational models via sharing of the mapping function, side-stepping the issues that non-Gaussian priors can cause for the inference model. Finally, allows for more general and more flexible inductive biases to be incorporated, by removing the need to express them with an explicit density function and allowing for parts of the mapping to be learned during training.
To further introduce a number of novel specific realizations of the InteL.-VAE framework, showing how they can be used to incorporate various inductive biases, enforcing latent representations that are, for example, multiply connected, multi-modal, sparse, hierarchical. Experimental results show their superiority compared with baseline methods in both generation and feature quality, most notably providing state-of of-the-art performance for learning sparse representations in the VAE framework.
To summarize, we a) highlight the need for inductive biases in VAEs and explain why directly changing the prior a suboptimal means for incorporating them; b) propose InteL- VAEs simple but effective general framework to introduce inductive biases; and c) introduce specific InteL--VAE variants which can learn improved generative models and representations over existing baselines on a number of tasks. Accompanying code is provided at
con/ NingMiao/ VAG.
THE NEED FOR INDUCTIVE BIASES IN VAES
Variational auto-encoders (VAEs) are deep stochastic auto-encoders that can be used for learning both deep generative models and low-dimensional representations of complex data. Their key components
encoder, g (2(r), which probabilistically maps from data x E X' to latents E Z; a decoder, p0(x)22, 2), which probabilistically maps from latents to data; and a prior, p(z), that completes the generative model, p(=) ((=)po pa (arl=), and regularizes the encoder during training. The encoder and decoder are parameterized by deep neural networks and are simultaneously trained using a dataset {#1, X2, ..., NN and variational lower bound on the log-likelihood, most commonly,
(1) Namely, optimize C(O, Φ) := Ez-pon (x) [C(x, 0, 6)]. where Pdata (x) represents the empirical data distribution. Here the prior is typically fixed to a standard Gaussian, i.e. p(z) (2; O.
[(x, 0, Φ) = (22qg(<))) [logPo(x))
Po (x/[²) DKL. (1>(=|a) p(=)).
While it is well documented that this standard VAE setup with a 'Gaussian' latent space can be suboptimal (Davidson et al., 201 Sa; Mathieu et al., 2019b; Tomezak & Welling, 2018: Bauer & Mnih, 2019; Tonolini et al., 2020), there perhaps less of a unified high-level view on exactly when, why, and how one should change it to incorporate inductive biases. Note here that the prior does not play the same role as in a Bayesian model: because the latents themselves are somewhat arbitrary and the model is learned from data, does not encapsulate our initial beliefs in the way one might expect.
We argue that there are two core reasons why inductive biases can be important for VAEs: (a) standard VAEs can fail to encourage, and even prohibit, desired structure in the representations learn; and (b) standard VAEs do not allow one to impart prior information or desired topological characteristic into the generative model.
2

---

Published as a conference paper at ICLR 2022
Considering the former, often has some a priori desired characteristics, or constraints, the representations learned (Bengio et al., 2013). For example, sparse features can desirable because they can improve data efficiency (Yip & Sussman, 1997), and provide robustness to noise (Wright et al., 2009; Ahmad & Scheinkman, 2019) and attacks (Gopalakrishnan et al., 2018). In other settings one might desire clustered (Jiang et al., 2017), 7), disentangled (Ansari & Soh, 2019; Kim & Mnih, 2018; Higgins et al., 2018) or hierarchical representations (Song & Li, 2013; Sønderby et al., 2016; Zhao al., 2017). The KL-divergence term Eq. (1) regularizes the encoding distribution towards the prior and, as standard Gaussian distribution typically does not exhibit our desired characteristics, this regularization can significantly hinder our ability to learn representations with the desired properties.
Not only can this be problematic at an individual sample level, it can cause even more pronounced issues at the population level: desired structural characteristies of our representations often relate to the pushforward distribution of the data in the latent space, (=(=) := lE (x00 which is both difficult to control and only implicitly regularized to the prior (Hoffman & Johnson, 2016).
Inductive biases can also be essential to the generation quality VAEs: because the generation process of stan- dard VAEs is essentially mushing-forwart the Gaussian prior on Z to data space X' by a 'smooth' decoder, there underlying inductive bias that standard VAEs prefer sample distributions with similar topology structures to Gaussians. As result, VAEs can perform poorly when the data manifold exhibits certain different topological properties (Caterini et al., 2020). For example, they can struggle when data is clustered into unconnected com- ponents as shown in Fig. 2, when data is not simply-connecte mappings using finite datasets and conventional architectures (potentially prohibitively) difficult. In particular, can necessitate large Lipschitz constants in the decoder, causing knock-on issues like unstable training and brittle models (Scaman & Virmaux, 2018) as well as posterior collapse (van den Oord et al., 2017: Alemi et al., 2018). In short, the Gaussian prior of a standard VAE can induce fundamental topological differences to the true data distribution (Falorsi et al., 2018; Shi et al., 2020).
(a) Data
(b) VAE
Figure 2: VAE learned generative distri- bution Ep(a) [Po (x|=)] for mixture data. This renders learning effective
3 SHORTFALLS OF VAES WITH VON-GAUSSIAN PRIORS
Though directly replacing the Gaussian prior with a different prior sounds like simple so- lution, effectively introducing inductive biases can, unfortunately, be more complicated.
Firstly, the only influence of the prior dur- ing training as a regularizer on the encoder through the DKL (g+(2(z) || p(2)) term. This regularization is always competing with the need for effective reconstructions and only has an indirect influence on (=). As such, simply replacing the prior can be an ineffective way of inducing desired structure at the population level (Mathieu
(a) Directly replacing p(x)
(b) InteL-VAE
Figure 3: Prior-encoder mismatch. We train (a) VAE with sparse prior and (b) MAtE with a sparse inductive bias on 2 dimensional sparse data. Figure shows target latent distribution p(2) (blue), learned variational embeddings Zo (=(2))) of exemplar data (green), and data pushforward qa (=()) (red shadow) for each method. Simply replacing the prior does not help the VAE match prior structure on either a per-sample or population level, whereas InteL- VAE produces an effective match.
al., 2019b), particularly if p(z) is complex distribution that it is difficult fit (see, e.g.. Fig. 3a). Mismatches between (=)2) and p(=) can also have further deleteri- ous effects on the learned generative model: the former represents the distribution of the data in latent space during training, while the latter is what is used by the learned generative model, leading to unrepresentative generations if there is mismatch.
Secondly, it can be extremely difficult to construct appropriate encoder mappings and distributions for non Gaussian priors. While the typical choice of a mean-field Gaussian for the encoder distribution is simple, easy to train, and often effective for Gaussian priors, it is often inappropriate for other choices of prior. For example, in Fig. 3, we consider replacement with a sparse prior. VAE with a Gaussian encoder struggles to encode points in manner that even remotely matches the prior. One
3

---

Published as a conference paper at ICLR 2022
might suggest replacing the encoder distribution as well, but this has its own issues, most notably that other distributions can be hard to effectively parameterize or train. In particular, the form of the required encoding noise might become heavily spatially variant; in sparse example, the noise must be elongated in a particular direction depending on where the mean embedding is. If the prior has constraints or topological properties distinct from the data, it can even be difficult to learn a mean encoder mapping that respects these, due to the continuous nature of neural networks.
THE INTEL- VAE FRAMEWORK
To solve the issues highlighted in the previous section, and provide a principled and effective method for adding inductive biases to VAEs, we propose Intermediary Latent Space VAEs (InteL-VAEs)) The key idea behind Intel.-VAEs is to introduce an intermediary set of latent variables y E V used as stepping stone in the construction of the representation 2 E 2. Data is initially encoded in y using a conventional VAE encoder (e.g. mean-field Gaussian) before being passed through non-stochastic mapping gv : y + 2 that incorporates our desired inductive biases and which can be trained, needed, through its parameters t. The prior is defined on Y' and taken to be standard Gaussian, p(y) = -N (y; 0, I). while our representations, 2 = gv (v), correspond to a pushforward of y. By first encoding datapoints y. rather than directly, we can deal with all the encoder and prior stochasticity in this first, well-behaved, latent space, while maintaining as our representation and using it for the decoder po (ar) z)) In principle, Ps can be any arbitrary parametric (or fixed) mapping, including non non-differentiable or even discontinuous functions. However, to allow for reparameterized gradient estimators (Kingma & Welling, 2014; Rezende & Mohamed, 2015), we will restrict ourselves to go that are sub- diffeffffreet (and thus continuous) with respect to both their inputs and parameters. Note that setting 9y to the identity mapping recovers a conventional VAE.
As shown in Fig. 1, the auto-encoding process is now X' "5 y 24 Z 22, X. This three-step process no longer unambiguously fits into the encoder-decoder terminology of the standard VAE and permits a variety of interpretations; for now we take the convention of calling go (y(zz) the encoder and (zrl2) the decoder, but also discuss some alternative interpretations below. We emphasize here that these no longer respectively match up with our representation model-which corresponds to passing an input into the encoder and then mapping the resulting encoding using Pv -nd our generative model-which corresponds to N Nyy; 0, (z/2 = (w)), such that we sample a y from the prior and then pass this through through Iy and the decoder in turn.
The mapping de introduces inductive biases into both the generative model and our representations by imposing a particular form on z, such as the spherical structure enforced in Fig. 1 (see also Sec. 6). It can be viewed as shared module between them, ensuring congruence between the two. This congruence allows us to more directly introduce inductive biases through careful construction of gø- without complicating the process of learning an effective inference network. In particular, because ) is treated as our latent space for the purposes of training, we sidestep the inference issues that non-Gaussian priors usually cause. Moreover, because all samples must explicitly pass through Po during both training and generation, we can more directly ensure the desired structure is enforced without causing mismatch in the latent distribution between training and deployment.
Training As with standard VAEs. training of an InteL-VAE is done by maximizing a variational lower bound (ELBO) on the log evidence, which we denote Cy. Most simply, we have
[o(a (o) (1
log Po (x) := log (Epiw) (o(())9(()))) = log (gg m)
2o (y(x))
(2)
2 EEgo(ole)[ogg po po (1)) DKL. (2o || (W: I)) =: Ly 0, v).
Note that the regularization is on y. but our representation corresponds to corresponds to the optimization arg max RF [Ly( 0, Φb v)), which can be performed using stochastic gradient ascent with reparameterized gradients in the standard manner. Although inductive biases are introduced, the calculation, and optimization, of Cy is thus equivalent to the standard ELBO. In particular, parameterizing qa (ylaz) with a Gaussian distribution still yields an analytical DKL (go (v/2)) II N ((; 0, I)) term.
(v))). Training
Alternative Interpretations It is interesting to note that our representation, (v)),, only appears in the context of the decoder in this training objective. As such, we see that an important alternative interpretation of Intel--VAR is consider dg as being a customized first layer in the decoder, and our
4

---

Published as a conference paper at ICLR 2022
time representations partial decodings of the latents y. This viewpoint allows it to be applied with more general bounds and VAE variants (e.g. Burda et al. (2016); Le et al. (2018); Maddison et al. (2017); Naesseth et al. (2018); Zhao et al. (2019)), as it requires only carefully customized decoder architecture during training and an adjusted mechanism for constructing representations at test-time.
Yet another interpretation is to think about Intel. VAES implicitly defining conventional VAE with latents 2, but where both the non-Gaussian prior, (2), and our encoder distribution, Is. (z)zr, are themselves defined implicitly as pushforwards along Iv, which acts as shared module that instills a natural compatibility between the two. Formally we have the following theorem.
Theorem 1. Let (z()) and (0.((z)z) represent the respective pushforward distributions of 10, and g (2)))) induced by the mapping I y + Z. The following holds for all measurable gø:
DEL (2e,0(2)x) |l Px()))
- DxL (gølu<z) (1; 0, I))-
If ge is also an invertible function then the above becomes an equality and Ly equals the standard ELBO on the space of 2 as follows
Ly(x, 0, o. v)- = Poo - Dxr. (2|ar) pp()))
(4)
The proof is given in Appendix A. Here, (3) shows that the divergence in our representation space 2 is never more than that in V, or equivalently that the implied ELBO on the space of 2 is always at least as tight as that on y:: (4) shows they are exactly equal if gy is invertible. As the magnitude of DKL (go(ylæ) N3y 0, 1)) in InteL-VAE will remain comparable to the KL divergence in
standard Gaussian prior VAE setup, this, in turn, ensures that DRL (go.s (2|ær) || (z)) does not be- come overly large. This is in stark contrast to the conventional non-Gaussian prior setup, where it can be difficult to avoid DKL (2)(2 || Pz) exploding without undermining reconstruction (Mathieu al., 2019b). The intuition here that having the stochasticity in the encoder before it is passed through by ensures that the form of the noise in the embedding is inherently appropriate for the space: the same mapping is used to warp this noise as to define the generative model in the first place. For example, when 9y is sparse mapping, the Gaussian noise in (3(xz) will be compressed to a sparse subspace by Iw: leading to a sparse variational posterior a (2|z) as shown in Fig. 3b. In particular, 2o (y) does not need to learn any complex spatial variations that result from properties of 2. In turn, InteL-VARE further alleviate issues of mismatch between pv (2) and (2).
Further Benefits A key benefit of InteL-VAEE is that the extracted features are guaranteed to have the desired structure. Take the spherical case for example, all extracted features gy (Ma(2)) lie within small neighborhood of the unit sphere. By comparison, methods based on training loss modifications, e.g. Mathicu et al. (2019b), often fail to generate features with the targeted properties.
A more subtle advantage is that we do not need to explicitly specify (2). This can be extremely helpful when we want to specify complex inductive biases: designing a non-stochastic mapping is typically much easier than density function, particularly for complex spaces. Further, this can make it much easier to parameterize and learn aspects of p (z) in a data-driven manner (see e.g. Sec. 6.3).
RELATED WORK
Inductive biases There is much prior work on introducing human knowledge to deep learning models by structural design, such as CNNs (LeCun et al., 1989), RNNs (Hochreiter & Schmidhuber, 1997) and transformers (Vaswani et al., 2017). However, most of these designs are on the sample level, utilizing low-level information such as transformation invariances or internal correlations in each sample. By contrast, InteL- VAES provide convenient way to incorporate population level nhowlegge-informaato information about the global properties of data distributions can be effectively utilized.
Non-Gaussian priors There is an abundance of prior work utilizing non-Gaussian priors to improve the fit and generation capabilities of VAEs, including MoG priors (Dilokthanakul et al., 2016; Shi et al., 2020), sparse priors (Mathieu et al., 2019b; Tonolini et al., 2020; Barello et al., 2018), Gaussian- process priors (Casale et al., 2018) and autoregressive priors (Razavi et al., 2019; van den Oord et al., 2017). However, these methods often require specialized algorithms to train and are primarily applicable only to specific kinds of data. Moreover, as we have explained, changing the prior alone often provides insufficient pressure its own to induce the desired characteristics. Others have proposed non-Gaussian priors to reduce the Welling, 2018) and LARS (Bauer & Mnih, 2019), but these are tangential our inductive bias aims.
gap, such as Vamp- VAE (Tomczak &
5

---

Published as a conference paper at ICLR 2022
Non-Euclidean latents A related line of work has focused non-Euclidean latent spaces. For instance Davidson et al. (2018a) leveraged a von Mises-Fisher distribution on a hyperspherical latent space, Falorsi et al. (2018) endowed the latent space with SO(3) group structure, and Mathieu et al. (2019a); Ovinnikov (2019); Nagano et al. (2019) with hyperbolic geometry. Other spaces like product of constant curvature spaces (Skopek al., 2019) and embedded manifolds (Rey al., 2019) have also been considered. However, these works generally require careful design and training.
Our of non-stochastic mapping shares some interesting links normalizing flows (NFs) (Rezende & Mohamed, 2015; Papamakarios et al., 2019; Grathwohl et al.. 2018; Dinh al., 2017: Huang et al., 2018; Papamakarios al., 2018). Indeed NF would be a valid choice for Jv. albeit an unlikely one due to their architectural constraints. However, unlike previous use of NFs in VAFs, our Gs is crucially shared between the generative and representational models, rather than just being used in the encoder, while the KI. divergence in our framework is taken before, not after, the mapping. Moreover, the underlying motivation, and type of mapping typically used, differs substantially: our mapping is used to introduce inductive biases, not purely to improve inference. Our mapping is also more general than a NF (e.g. it need not be invertible) and does not introduce additional constraints computational issues.
Normalizing flows
6 SPECIFIC REALIZATIONS OF THE SITE-VVE FRAMEWORK
We now present several novel example InteL-VAEs introducing various inductive biases through different choices of Iw- We will start with artificial, but surprisingly challenging, examples where some precise topological properties of the target distributions are known, incorporating them directly through fixed fø- We will then move onto experiments where impose fixed clustering inductive bias when training on image data, allowing us to learn InteL-VAES that account effectively for multi-modalle in the data distribution. Finally, we consider the example of learning sparse representations of high dige--imensioall data. Here we will see that it is imperative to exploit the ability of Intel. VAEs learn aspects of 9r during training, providing flexible inductive bias framework, rather than pre-fixed mapping. By comparing InteL-VAES with strong baselines, we show that InteL-VAEs are effective introducing these desired inductive biases, and consequently both improve generation quality and learn better data representations for downstream tasks. One note of particular importance is that we find that InteL- VAEs provide of-te-of-the-an performance for learning sparse VAE representations. A further example of using InteL- VAEs to learn hierarchical representations is presented in Appendix B, while full details on the various examples are given in Appendix C.
6.1 MULTIPLE- CONIPLLE-CONECTTVIIY
Data is often most naturally described Euclidean spaces such as circles, e.g. wind direc- tions (Mardia & Jupp, 2000), and other multiply- connected shapes, e.g. holes in disease databases (Liu al., 1997). For reasons previously explained in Sec. 2, standard VAEs cannot practically model such topologies, which prevents them from learning gen- erative models which match even the simplest data distributions with non-trivial topological structures, as shown in Fig. 4b.
(a) Data
(b) VAE
(c) InteL-VAE
Luckily, by designing gy to map the Gaussian prior to simple representative distribution in topolog- ical class, we can easily equip mnte-VAEE with the knowledge to approximate any data distribu- tions with similar topological properties. Specifically, by defining De the orthogonal projection 8². (2) = 2/ + c). we map the Gaussian prior approximately to a uniform distribution to S!, where is small positive constant to ensure the continuity of g near the origin. From Rows and 2 of Fig. 4, find that this inductive bias gives Intel.-VAHR the ability to learn various distributions with hole.
Figure 4: Training data and samples from learned generative models of vanilla-VAE and InteL-VAE for multiply-connected and clustered distributions. InteL-VAE uses [Rows 1,2] circular prior with one hole, [Row 3] multiply-connected prior with two holes, and [Row 4] clustered prior. Vamp- be- haves similarly to a vanilla VAE; its results are presented in Fig. 4.

---

Published as a conference paper at ICLR 2022
We can add further holes by simply "gluing' point pairs. For example, for two holes we can use
(2( = Concat (91 (v))),, 91 (1()):21 / (4/3 - (1 - 19 (v) (z) - 1/v3),
(5) which first map to approximately S and then glues (0, 1) and (0, -1) together to create new holes (see Fig. C. for an illustration). Furthermore, we can continue glue points together to achieve higher number of holes h. and thus more complex connectivity. Row 3 of Fig. 4 gives an example of learning an infinity sign by introducing a 'two-hole' inductive bias.
Compared with vanilla- and VAE, which try to find a convex hull for real data distributions, InteL-VAEEs can deal with distributions with highly non-convex and very non-smooth supports (see Fig. 4 and Appendix C.1). We emphasize here that our inductive bias does not contain the information about the precise shape of the data, only the number of holes. We thus see that InteL-VABS can provide substantial improvements in performance by incorporating only basic prior information about the topological properties of the data, which point out a way to approximate distributions on more complex structures, such linear groups (Gupta & Mishra, 2018).
6.2 MUDTI-MODALITY
Many real-world datasets exhibit multi-modality. For example, data with distinct classes are often naturally clustered into (nearly) disconnected components representing each class. However, vanilla VAEs generally fail to fit modtl data due to the topological issues explained in Sec. 2. Previous work (Johnson et al., 2017; Mathieu et al., 2019b) has thus proposed the use of multi-modal prior, such as a mixture of Gaussian (MoG) distribution, so as to capture all components of the data. Nonetheless, VAEs with such priors often still struggle to model multi-modal data because of mismatch between (z) and p(z) or training instability issues.
We tackle this problem by using mapping go which contains a clustering inductive bias. The high-level idea is to design a mapping Pe with a localized high Lipschitz constant that 'splits' the continuous Gaus- sian distribution into disconnected parts and then pushes them away from each other. In particular, we split Y it into K equally sized sectors using its first two dimensions (noting it is not needed to split on all dimensions to form clusters), as shown in Fig. 5. For any point y. we can easily get the center direction r(y) of the sector that y belongs to and the distance dis(y) between y and the sector boundary. Then we define ge (v) as:
disco
(a)
(b)
(C)
Figure 5: Illustration of clustered mapping where K 3. The circle represents density isoline of Gaussian. Note that not all points in the sector are moved equally: points close to the boundaries between sectors are moved less, with points on the boundary themselves not moved at all.
9 (v) y cidis)))?r(yy),
where G and 02 are empirical constants. We can see that although Iø has very different function on different sectors, it is still continuous the whole plane with (v y sector boundaries, which is desirable for gradient-based training. See Appendix C.2 for more details.
To assess the performance of our approach, we first consider simple 2-component MoG synthetic dataset in the last row of Fig. 4. We see that the vanilla VAE fails to learn a clustered distribution that fits the data, while the InteL-VAt sorts this issue and fits the data well.
To provide a more roaldworld example, we train an InteL-VAE and variety of baselines on the MNIST dataset, comparing the generation quality of the learned models using the FID score (Heusel et al., 2017) in Table 1. We find that the GM-VAE (Dilok- thanakul et al., 2016) and MoG-VAE (VAE with a fixed MoG prior) achieve performance gains by using non-Gaussian priors. The Vamp- VAE (Tomezak Welling, 2018) and VAE with a Sylvester Normal- izing Flow (Berg et al., 2018) encoder provide further gains by making the prior and encoder distributions more flexible respectively. However, the InteL-VAr VAA comfortably outperforms all of them.
Method
FID Score (L)
VAE
42.0 + 1.1
GM-VAE
41.0 - 4.7
MoG-VAE
41.2 3.3
Vamp- VAE
38.8 + 2.4
VAE with Sylvester NF InteL-VAE
35.0 + 0.9
32.2 + 1.5
Table 1:
Generation quality on MNIST. Shown is mean FID score (lower better) + standard deviation over 10 runs.
7

---

Published as a conference paper at ICLR 2022
Method
Data
VAE
GM VAE
MoG-VAE
Vamp VAE
Flow
Intel. VAE
Uncertainty(
0.2 +
3.5 + 1.8
4.5 0.8
24
16.2
2.1
0.9 0.8
'I' proportion( (3))
50.0 +: 0.2
48.8 1: 0.2
48.1 1
0.3
47.7 4..7.00.4
48.8 1 0.1
42.5 1
1.0
49.5 1: 0.4
Table 2: Quantitative results on MN Uncertainty is the proportion of images whose labels are "indistinguishable" by the pre-trained classifier, defined as having prediction confidence < 80%. '1' proportion is the proportion of images classified as '1".
To gain insight into how InteL-VAAE achieve superior generation quality, we perform analysis on simpli- fied setting where select only the '0' and '1' digits from the MNIST dataset to form strongly clustered dataset, MNIST-01. We further decrease the latent dimension to 1 to make the problem more challeng- ing. Fig. 6 shows that here the vanilla VAE generates Figure 6: Generated samples for MNIST-01.
8
0
0
O O
O
0
/
/
101
1
0
O
0
D
C
(a) VAE (b) MoG- MAG-VAE VAE (c) InteL-VAE
some samples which look like interpolations between "O' and "I', meaning that it still tries to learn connected distribution containing 'O' and '1'. Further, the general generation quality is poor, with blurred images and lack of diversity in generated samples (e.g. all the 'I's have the same slant). Despite using a clustered prior, the MoG- VAE still produces unwanted interpolations between the classes. By contrast, VAE generates digits that are unambiguous and crisper.
To quantify these results, we further train a logistic classi- fier on MŇIST-01 and use it to classify images generated by each method. For each method, we calculate the pro- portion of samples produced by the generative model that are assigned to each class by this pre-trained classifier, as well as the proportion of samples for which the clas- sifier is uncertain. From Table 2 we see that InteL-VAt significantly outperforms its competitors the ability to generate balanced and unambiguous digits. To extend this example further, and show the ability of InteL-VVAEs to learn aspects of 9v during training, we further consider parameterizing and then learning the relative size the clusters. Table 3 shows that this can be successfully learned by InteL-VAEs on MNIST-01.
True Prop.
Learned Prop.
0.5
0.47 0.01
0.4
0,36 0.10
0.25
0,25 + 0.08
0.2
0.16 1: 0.11
0
0.02 0.01
Table 3: Learned proportions of '0's on MNIST-01 for different ground truths. Error bars are std. dev. from 10 runs.
6.3 SPARSITY
Sparse features are often well-suited to data efficiency on downstream tasks (Huang & Aviyente, 2006), in addition to being naturally casier visualize and manipulate than dense features (Ng et al., 2011). However, existing VAE models for sparse representations trade off generation quality to achieve this sparsity (Mathieu et al., 2019b; Tonolini et al., 2020; Barello et al., 2018). Here, show that InteL-VVAEs can instead simultaneously increase feature sparsity and generation quality. Moreover, they are able to achieve tfte-of-the-ar scores on sparsity metrics.
Compared with our previous examples, the g6 here needs to be more flexible so that it can learn to map points in a data-specific way and induce sparsity without unduly harming reconstruction. To achieve this, we use the simple form for the mapping: ge (y) multiplication, and DS a 'dimension selector' network that selects dimensions to deactivate given y. DS outputs values between [0, 1] for each dimension, with 0 being fully deactivated and 1 fully activated; the more dimensions we deactivate, the sparser the representation. By learning DS during training, this setup allows us to learn a sparse representation in data-driven manner. To control the degree of sparsity, we add sparsity regularizer, Lap, to the ELBO with weighting parameter ? (higher corresponds to more sparsity). Namely, we optimize Ly d, v) + 7 Lap(O, v)), where
@ (v) where @ is pointwise
M (H (DS(g)))
M
1
Cop(d. = E
M M
H.
Σ DS( i=1
M
((v)= = - (1s/|o/]i) log (v; ((]l) is the normalized entropy of an positive vector v, and the expectation over drawing a minibatch of samples ii .., UM and then sampling each corresponding
TMM
Yi ~ (- = Iz). Cap encourages DS to deactivate more dimensions, while also encouraging diversity in which dimensions are activated for different data points, improving utilization of the latent space. Please see Appendix C.3 for more details and intuitions. Initial qualitative results are shown in Fig. 8, where we that our Intel-VAAt is able to learn sparse and intuitive representations.
8

---

Published as a conference paper at ICLR 2022
300
0-
100
Ours
0.8
250
1000
Sparse-VAE VanillaVVAE
DD
0.6
200
0.4
FID
150
Sir 0.11
100
2 0.0.2
-0.0
00..1..0 90
0.2
cum1y-10..
cum/y-100.31
0.2
0.4
0.6
100
500 1000 2000
Sparse scores
#Data
Figure 7: Results on Fashion-MNIST. The left figure shows FID and sparsity scores. Lower FID scores (J) represent better sample quality while higher sparse scores (->) indicate sparser features. The right figure shows the performance of sparse features from Intel.-VAF on downstream classification tasks. See Appendix C.3 for details and results for MNIST.
To quantitatively assess the ability of our ap- proach to yield sparse representations and good quality generations, we compare against vanilla VAEs, the specially customized sparse- VAE of Tonolini et al. (2020), and the sparse ver- sion of Mathieu et al. (2019b) (DD) on Fashion- MNIST (Xiao et al., 2017) and MNIST. As shown in Fig. 7 (leff), we find that InteL- VAES increase sparsity of the representations measured by the Hoyer metric (Hurley & Rickard, - increasing generative sample quality the same time. Indeed, the FID score obtained by InteL-VAE outperforms the vanilla VAE when y 3.0, while the sparsity score substantially increases with 7/ reaching ex- tremely high levels. By comparison, DD signifi- cantly degrades generation quality and only pro- vides a more modest increase sparsity, while its sparsity also drops if the regularization co- efficient is set too high. The level of sparsity achieved by sparse- VAEs was substantially less than both DD and IAAEL.VAES.
Avg. magnttue
Trouser Coal
Sneaker
0.5
0.0
20
Latent dimension
(a)
(b)
(c)
Figure 8: Qualitative evaluation of sparsity. [Top] Average magnitude of each latent dimension for three example classes in Fashion-MNIST: less than 10% dimensions are activated for each class. [Bottom] Activated dimensions are different be- tween classes: (a-c) show the results of separately manipulating an activated dimension for each class. (a) Trouser separation (Dim 18). (b) Coat length (Dim 46). (c) Shoe style (formal/sport, Dim 25).
To further evaluate the quality of the learned features for downstream tasks, we trained a clas-
sifier predict class labels from the latent representations. For this, we choose a random for- est (Breiman, 2001) with maximum depth 4 as it is well-suited for sparse features. We vary the size of training data given to the classifier measure the data efficiency of each model. Fig. 7 (right) shows that VAE typically outperforms other the models, especially in few-shot scenarios.
Finally, to verify InteL-VAA''s effectiveness on larger and higher-resolution datasets, we also make comparisons on CelebA (Liu et al., 2015).
Method
FID (L)
Sparsity (t)
VAE
68.6::1.1
0.22-0.01
From Table 4, we can see that InteL-VAE in- crease sparse scores to 0.46 without sacrific- ing generation quality. By comparison, the maximal sparse score that sparse- VAE gets is 0.30, with unacceptable sample quality. Inter- estingly, VAteL-VAEs with ly low regulation ? achieved particularly good generative sample quality, outperforming even the Vamp- VAE and a VAE with a Sylvester NF encoder.
Vamp VAE
67.5+1.1
0.22-0001
VAE with Sylvester NF
66.3:10.4
0.22-10011
Sparse-VAr VAE (a: = 0.01)
328++10.1
0.25
Sparse- VAE
0.2)
337-+8.1
0.28
InteL-VAr VAE (7 30)
64.910.4
0.25-00.01
InteL-VAt VAE (7 = 70)
68.010.6
0.46+00.02
Table Generation results on CelebA.
Conclusions In this paper, proposed InteL-VAEs general schema for incorporating inductive biases into VAEs. Experiments show that InteL-VAFs can both provide representations with desired properties and improve generation quality, outperforming variety of baselines such as directly changing the prior. This is achieved while maintaining the simplicity and stability of standard VAEs.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
We do not believe that there are direct ethical concerns regarding our paper: the datasets we consider all already well established and do not contain sensitive information, while the methods and ideas we introduce have no clear direct potential negative societal impacts of their own. From a bigger picture perspective, work like ours that looks to permit more effective incorporation of inductive biases into models can be thought of as allowing more direct human control on how models will behave after training. While this will typically be force for good, for example by encouraging model interpretability and providing mechanisms to try and induce positive characteristics like fairness, in rare circumstances there may also be the potential for this to used nefariously by deliberately encouraging undesirable behavior. However, we do not believe that our work is any more prone to such exploitation than existing methods or that the risk of it being used in such as a way is significant.
REPRODUCIBILITY STATEMENT
Full experimental details are given in Appendix C, while anonymized source code for reproducing all our experiments directly is provided https://gihub.ccm///// these should make it straightforward for others to reproduce our empirical results. We have been careful to provide quantitative metrics of performance whenever possible, rather than just relying on qualitative or anecdotal evidence. Repeat runs and error bars are provided whenever this is feasible, with the level of variability always found to be sufficiently small to draw reliable and statistically sound conclusions. In fact, the training stability and consistent performance of our general approach under retraining provides a clear advantage itself compared many of the baseline methods. Full formal proof for our only theoretical result is given in Appendix A, while the assumptions it makes are clearly stated and easily verifiable.
Together
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowiez, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: /www://wwww////////////// wwwsor flww. org/. Software available from tensorflow.org.
Olah,
Subutai Ahmad and Luiz Scheinkman. How can we be so dense? the benefits of using highly sparse representations. arXiv preprint arXiv: 2019.
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy, Fixing a broken elbo. In International Conference on Machine Learning, pp. 155-168. PMER, 2018.
Abdul Fatir Ansari and Harold Soh. Hyperprior induced unsupervised disentanglement of latent representations. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3175-3182, 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409 14090447, 2014.
Gabriel Barello, Adam S. Charles, and Jonathan W. Pillow. Sparse-coding variational auto-encoders. Preprint, Neuroscience, August 2018
Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 66-75. PMLR, 2019.
Mohamed Ishmael Belghazi, Sai Rajeswar, Olivier Mastropietro, Negar Rostamzadeh, Jovana Mitro- vic, Aaron Courville, and AI Element. Hierarchical adversarially learned inference. stat, 1050:4, 2018.
10

---

Published as a conference paper at ICLR 2022
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence. 35(8):1798- 2013.
Rianne van den Berg, Leonard Hasenclever, Jakub M Tomezak, and Max Welling. Sylvester normalizing flows for variational inference. arXiv preprint FrXxi:/803.05649 2018.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Yuri Burda, Roger B Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In ICLR (Poster), 2016.
Francesco Paolo Casale, Adrian V Dalca, Luca Saglietti. Jennifer Listgarten, and Nicoló Fusi. Gaussian process prior variational autoencoders. NeurlPS, 2018.
Anthony L Caterini, Robert Cornish, Dino Sejdinovic, and Arnaud Doucet. Variational inference with continuously-indexed normalizing flows. 2020.
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomezak. Hyperspherical variational auto-encoders. arX4v:1804.0000 [cs, stat], September 2018a. URL http:: org/abs /8044 00891.
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub Tomezak. Hyperspherical variational auto-encoders. 34th Conference on Uncertainty in Artificial Intelligence (UAI-18), 2018b.
Alfredo De la Fuente and Robert Aduviri. Replication/maccine learning. 2019.
Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders. arXiv preprint arXiv:1611.02644 2016.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. 1 [cs, stat], February 2017. URL U.R.ittp:://rrii.org//bbss 605. 08803.
Luca Falorsi, Pim de Haan, Tim R. Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S. Cohen. Explorations in homeomorphic variational auto-encoding. arXiv:/807.044 [cs, stat], July 2018. URL http: http///axxvv.rr../. / 1878 . 04689.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 315-323. JMLR Workshop and Conference Proceedings, 2011.
Soorya Gopalakrishnan, Zhinus Marzi, Upamanyu Madhow, and Ramtin Pedarsani. Combating adversarial attacks using sparse representations. arXiv preprint /80.0:8003.03880, 2018.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, llya Sutskever, and David Duvenaud. FFJORD: Free-form continuous dynamics for scalable reversible generative models. arX1v:1810.01367 [cs, stat], October 2018. URL http: //axtp://axivv... org/ abs /aabbs//810 1810.111000 01367.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: latent variable model for natural images. arXiv preprint arXiv:1611.05013. 2016.
Ved Prakash Gupta and Mukund Madhav Mishra. On the topology of centain matrix groups. THE MATHEMATICS STUDENT, pp. 61, 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Proceedings of the 3 International Conference Neural Information Processing Systems, pp. 6629-6640, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv: 1812. 02230, 2018.
11

---

Published as a conference paper at ICLR 2022
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Matthew D Hoffmann and Matthew J Johnson. Elbo surgery; yet another way to carve up the variational evidence lower bound. 2016.
Xianxu Hou, Linlin Shen, Ke Sun, and Guoping Qiu. Deep feature consistent variational autoencoder. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1133-1141. IEEE, 2017.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. pp. 10, 2018.
Ke Huang and Selin Aviyente. Sparse representation for signal classification. Advances in neural information processing systems, 19:609-616, 2006.
Niall Hurley and Scott Rickard. Comparing measures of sparsity. IEEE Transactions on Information Theory, 55:4723-4741, 2009.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. In IJCAI, 2017.
Matthew J. Johnson, David Duvenaud, Alexander B. Wiltschko, Sandeep R. Datta, and Ryan P. Adams. Composing graphical models with neural networks for structured representations and fast inference. arXiv:1603.06277 [stat], July 2017. URL http: http///xxi...ggaa///////// 1660..0677.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2649-2658. PMLR, 10-15 Jul 2018. URL 8b. html.
http:/ http:://proceeee /ptpppeeeeinggss.e/eeppppp mmlr. ptessppess///e wwt.:/rwesss/eeeee
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, and Patrick van der Smagt. Learning hierarchical priors in vaes.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018.
Tuan Anh Le, Maximilian Igl, Tom Rainforth, Tom Jin, and Frank Wood. Auto-encoding sequential monte carlo. In International Conference on Learning Representations, 2018.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 144::541-551 1989.
Bing Liu, Liang-Ping Ku, and Wynne Hsu. Discovering interesting holes in data. In Proceedings of the Fifteenth international joint conference on Artifical intelligence- 2, pp. 930-935, 1997.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Chris Maddison, John Lawson George Tucker, Nicolas Heess, Mohammad Norouzi, Andriy Mnih, Arnaud Doucet, and Yee Whye Teh. Filtering variational objectives. In NIPS, 2017.
K. V. Mardia and Peter E. Jupp. Directional Statistics. Wiley Series in Probability and Statistics. J. Wiley, Chichester: New York, 2000, ISBN 978-0-471-99333-3..
Emile Mathieu, Charline Le Lan, Chris J. Maddison, Ryota Tomioka, and Yee Whye Teh. Continuous hierarchical representations with poincarl'e variational auto-encoders. January 2019a. URL pttts://arrtie wrg//bo//////ooooooooooo 901 06033v3
12

---

Published as a conference paper at ICLR 2022
Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Tch. Disentangling disentanglement in variational autoencoders. International Conference on Machine Learning, pp. 1402-4412 PMLR, 2019b.
Christian Naesseth, Scott Linderman, Rajesh Ranganath, and David Blei. Variational sequential monte carlo. In International Conference on Artificial Intelligence and Statistics, pp. 968-977,
PMLR, 2018.
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A wrapped normal distribution on hyperbolic space for gradient-based learning. arXiv:1902.02999 [cs, stat], May 2019. URL http: pett::/arree /arxivv./rivvvv//a org/ abss / 1902. 02992.
Andrew Ng al. Sparse autoencoder. CS294A Lecture notes, 72(2011):1-19, 2011. 2011.
Ivan Ovinnikov. Poincarl'e wasserstein autoencoder. January 2019. URL wttptt:tttaat....ooo
abs/1901. 01427v2.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. arXiv:770.007057 [es, stat], June 2018. URL MRLLttpp/aaxxxv.rrr abg20/0 07057.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.02762, 2019.
Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In International Conference on Machine Learning, pp. 324-333. PMLR, 2016.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In NIPS, 2019.
Luis A. Pérez Rey, Vlado Menkovski, and Jacobus W. Portegies. Diffusion variational autoencoders. 19X1v:191100891 [es, stat), March 2019. URL htttt: wttp:///..rraab///////aaa 08 991.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference Machine Learning, pp. 1530-1553 1538. PMLR, 2015.
Igal Sason. On data-processing and majorization inequalities for f-divergences with applications. Entropy, (10))): 1022, October 2019. ISSN 1099-4300. doi: 10.3390/e21 101022.
Kevin Scaman and Aladin Virmaux. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 3839-3848, 2018
Wenxian Shi, Hao Zhou, Ning Miao, and Lei Li. Dispersed exponential family mixture vaes for interpretable text generation. International Conference on Machine Learning, pp. PMLR, 2020.
Ondrej Skopek, Detavian-Fugen Ganea, and Gary Bécigneul. Mixed-curvature variational autoen- coders. November 2019. URL https: /ttess///axiiii http//axvvvv.///////e/////a org/ http///ss11111 /ttp://wwwwrg///////1...../ 08411v2.
Casper Kaae Sønderby, Tapani Raiko, Lars Maalge, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In NIPS, 2016.
Tiecheng Song and Hongliang Li. Wavelbp based hierarchical features for image classification. Pattern Recognition Letters, 34(12):1323-13333 2013.
Jakub Tomezak and Max Welling. Vae with a vampprior. 21st International Conference on Artificial Intelligence and Statistics, AISTATS 2018, 2018.
Francesco Tonolini, Bjørn Sand Jensen, and Roderick Murray-Smtth. Variational sparse coding, In Uncertainty in Artificial Intelligence, pp. 690-700. PMLR, 2020.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint rxXi::2077003895 2020.
13

---

Published as a conference paper at ICLR 2022
Aaron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1309-6318, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Stefan Webb, Adam Golinski, Robert Zinkov, Siddharth Narayanaswamy. Tom Rainforth, Yee Whye Teh, and Frank Wood, Faithful inversion of generative models for effective amortized inference. In NeurIPS, 2018.
John Wright, Allen Y. Yang, Arvind Ganesh, S. Shankar Sastry, and Yi Ma. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31)) 210-227, 2009. doi: 10. 109/TPAMM..208.77.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
Kenneth Yip and Gerald Jay Sussman. Sparse representations for fast, one-shot learning. In Proceedings of the fourteenth national conference on artificial intelligence and ninth conference on Innovative applications of artificial intelligence, pp. 521-527, 1997.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from generative models. In International Conference on Machine Learning, 2017.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Balancing learning and inference in variational autoencoders. In Proceedings of the aaai conference on artificial intelligence. volume 33, pp. 5885-5892, 2019.
14

---

Published as a conference paper at ICLR 2022
APPENDIX A PROOFS
Theorem 1. Let pi)(2) and (o.0(2(z) represent the respective pushforward distributions of N
1)
and (z))) induced by the mapping Iv y 1y+> + Z. The following holds for all measurable I
DEL (2)x)())z)) |l Pv(2)) I DxL (goly)z) I| Ny: I)).
(3)
If ge also an invertible function then the above becomes an equality and Ly equals the standard ELBO on the space of 2 as follows
Ly(x, 0 O, xb) (ela) log (log (o)))) DKL (26. Pp()).
(4)
Proof. We first prove the inequality from Eq. (3), then we show that Eq. (3) is actually an equality when Iy is invertible, and finally prove that the reconstruction term is unchanged by Je-
Let us denote by F and 9 the measurable map D (), + (2,9). We can actually define the measurable space (2,9) as the image of (V, by Iv. then Io is automatically both surjective and measurable.' We also assume that there exists a measure on V. which we denote ç. and denote with v the corresponding pushforward measure by Oe on Z. We further have (AA {(9 (A)) for any A g.²
of respectively y' and Z, and we have by construction
We start by proving Eq. (3), where the Kullback-Leibee (KL) divergence between the two push- forward measures' & 2 qs 9g) and P p 9p is upper bounded by DxL (as || p(y)), where here we have p(y) (y: 0, I) but we will p as a convenient shorthand. At high- we essentially have that Eq. (3) follows directly the data processing inequality (Sason, 2019) with a deterministic kernel
(y)) Nonetheless, we develop in what follows proof which addition- ally gives sufficient conditions for when this inequality becomes non-strict. We can assume that DKL (g)()) II N (y; 0, 1)) is finite, as otherwise the result is trivially true, which in turn implies 9o < p. For any A E G. we have that if (A) p o g (A) p(g (A)) 0 then this implies 4s (9 (A)) = Q 9pp (A) Ts (A) 0. As such, we have that go. < ps and so the DEL (2(x) Pv (=)) is also defined.
Our next significant step is to show that
96 9 о Iv.
E
Ep(s)
(9+))
(A.1)
-I
p
p o Fg
where (gg) denotes the sigma-algebra generated by the function PV- To do this, let h (2,9) - = h 9v- To show this, we will
(R,, (). be measurable function s.t. Ep(v) demonstrate that they lead to equivalent measures when integrated over any arbitrary set A E G:
[ p
((
A 96 g I po gx -1 du =
1 A o dv= = / La d(as og
p 9
z
(a)
( dgp ~/100 (w) gw) ao de
@
20 d (IA (I
p
@
(I (I1 Epis)
[go [(ow)) p d
p
(
(I (h p / (II (w) go) (h
gv) dp
(c) / LA h = 1 h (p g5 dv,
( (A) F. We recall that Io is said to be measurable if and only if for any A E 9. g.
Thhe notation 95 (AA does not imply that go is invertible, but denotes the preimage of which is defined as 9 ( (A) = {y E y (v) A).
We denote the pushforward of a probability measure x along map 9 by x
'We denote the absolute continuity of measures with <, where μ is said to be absolutely continuous w.r.t. v, i.c. < v, if for any measurable set A, p(A) 0 implies (((A) = 0.
15

---

Published as a conference paper at ICLR 2022
where we have leveraged the definition of pushforward measures in (a & e); the absolute continuity of 9s w.r.t p in (b); the conditional expectation definition in (c); and the definition of h in (d). By equating terms, we have that Os o - /p 96 h, almost-surely with respect to qø o g and thus that Eq. (A.1) is verified.
Let define f + x log(x), which is strictly convex on [0, oo) (as it can be prolonged with f(0) = 0). We have the following
DEL (2).) (x))
(0)
log
4d.1
9s.v dv
z
Pv
(b)
9d..v
Io.
log
pe du
Pv
Pob
(c)
/
l0.v
Py du
f
d(p
(pogg'))
2
PV
2
Po
(4)
Lo.0
[
4á o
Fv
I
dp
f
ge
de
Pb
v)
pog
Ep(s)
90
((
p
(
Bp(w)
f
9d
|
p d
(0)
p de
(
/
log
9
ao p dd
(2(()lr)
log
p(y)
DKL. (gølu)æ) p(u)).
where we leveraged the definition of the KL divergence in (a & j); the absolute continuity of go w.r.t. p in & i); the definition of (c & h); the defnition of the pushforward measure in (d); Eq. (A.1) in (e); the conditional Jensen inequality in (f) and the law of total expectation in (g). Note that this proof only holds for the KI divergence, but for any f-divergences as they are defined as in (b) with f convex.
To prove Eq. (4), we now need show that line (f) above becomes an equality when gø invertible. As f strictly convex, this happens if and only if Ep(s) 2s a(gw)) A sufficient condition
for this to be true for ge to be measurable w.r.t. a (go) which satisfied when gv: y + Z p invertible as o (gw) 2 F, as required. We have thus shown that the KL divergences are equal when using an invertible 9v-
For the reconstruction term, we instead have
log po (eop(
E Pog
Eq. (4) now follows from the fact that both the reconstruction and KI. terms are equal.
16

---

Published as a conference paper at ICLR 2022
APPENDIX B HIERARCHICAL REPRESENTATIONS
The isotropic Gaussian prior in standard VAEs as- sumes that representations are independent across dimen- sions (Kumar al., 2018). However, this assumption is often unrealistic (Belghazi et al., 2018; Mathieu et al., 2019b). For example, Fashion-MNIST,, high-level fea- tures such as object category, may affect low-level features such as shape or height. Separately extracting such global and local information can be beneficial for visualization and data manipulation (Zhao et al., 2017). To try and cap- ture this, introduce inductive bias that is tailored to model and learn hierarchical features. We note here that aim is cal VAE approach, as a wide variety of cuttll--cutomizeed and powerful approaches are already well-established, but to show how easily the Intel- VAE framework can be used induce hierarchical representations simple, lightweight, manner.
9
Yo
20
y1
Z1
46
P8
X
X
Y2
2
YN
Z
try and provide state-of-the-art hierarchi-
Figure B.1: Graphical model for hierar- chical InteL-VAE
Mapping design Following existing ideas from hierarchical VAEs (Sønderby al., 2016; Zhao et al., 2017), we propose a hierarchical mapping Pv- As shown in Fig. B.I, the intermediary Gaussian variable y is first split into set of N layers [No, Vl, .... yN]. The mapping = - gy) then recursively defined as 2; NN:(zi-1: Mi). where NN, is a neural network combining information from higher-level feature 2,-1 and new information from ys- As a result, we get a hierarchical encoding = = [Ao, 21, ..., ZN]. where high-level features influence low-level ones but not vice-versa. This By thus endows Intel.- VAEs with hierarchical representations.
Experiments While conventional hierarchical VAEs, e.g. (Sønderby et al., 2016; Zhao et al., 2017; Vahdat & Kautz, 2020), use hierarchies try and improve generation quality, our usage is explicitly from the representation perspective, with our experiments set up accordingly. Fig. B.2 shows some hierarchical features learned by InteL-VAE on Fashion-MNIST. We observe that high-level information such as categories have indeed been learned in the top-level features, while low-level features control more detailed aspects.
(a)
(b)
(d)
To provide more quantitative investigation, we also consider the CelebA dataset (Liu et al., 2015) and investigate performance paring to vanilla- VAEs with different latent dimen- sions. For this, train linear classifier predict all 40 binary labels from the learned features for each method. In order to eliminate the effect of latent di- mensions, we compare InteL-VAE (with fixed latent dimension 128) and vanilla VAE with different latent dimensions (1, 2, 4, 8, 16, 32, 64, 128). We show ex- periment results on some labels as well as the average accuracy on all labels in Table B. and Fig. B.3. We first find that the optimal latent dimension increases with the number of data points for the vanilla-VARR but is always worse than the InteL-VAE. Notably, the accuracy with InteL-VAE is quite robust, even as the number of data points gets dramatically low, indicating high data efficiency. To the best of our knowledge, this is the first result showing that hierarchical inductive bias in VAE is beneficial to feature quality.
(e)
downstream tasks, com-
Figure B.2: Manipulating representations of a hierarchical InteL-VAE. The features are split into 5 levels, with each of (a) [highest] to (e) [lowest] corresponding to an example feature from each. We see that high-level features control more complex properties, such as class label or topological structure, while low-level features control simpler de- tails, (e.g. (d) controls collar shape).
Related work Hierarchical VAES (Vahdat & Kautz, 2020; Ranganath et al., 2016: Sønderby et al., 2016; Klushyn et al.; Zhao et al., 2017) seek to improve the fit and generation quality of VAEs by recursively correcting the generative distributions. However, they require careful design of neural
17

---

Published as a conference paper at ICLR 2022
Model
Latent dim
Data size
50
100
500
1000
5000
10000
VAE
0.791
0.799
0.814
0.815
0.819
0.819
0.788
0.801
0.820
0.824
0.829
0.831
32
0.769
0.795
0.825
0.832
0.842
0.846
64
0.767
0.794
0.826
0.832
0.849
0.855
128
0.722
0.765
0.817
0.825
0.830
0.852
InteL-VAE 64
0.817
0.824
0.841
0.846
0.854
0.857
Table B.1: Average accuracy in predicting all 40 binary labels of CelebA. Overall best accuracy is shown in bold and best results of vanilla- VAEs are underlined for comparison. Each experiment is repeated 10 times and differences are significant the 5% level for data size < 1000.
layers, and the hierarchical KL divergence makes training deep hierarchical VAEs unstable (Vahdat & Kautz, 2020). In comparison, InteL-VAE with hierarchical mappings is extremely easy to implement without causing any computational instabilities, while its aims also differ noticeably: approach successfully learns hierarchical
that is rarely mentioned in prior works.
18

---

Published as a conference paper at ICLR 2022
5 Clock Sha _lack_Shadoo
Attractive
Bald
Bangs
Black Hair
Blond Hair
0 0.00
620
0.5
#Dala
0.00
655
522
6.4
0.00
0.30
0.04
150
200
0.90
#Dola =
8.66
0.900
0.90
179
#Data E 500
0.00
3.44
0000
0.35
#Data = 1000
0.0
0.00
0.50
0.90
3.7
0.80
0.09
#Data 2000
0.70
8:58
033
0.99
3.4
0.70
*Data 5000
3.00
8.50
5.50
0.90
0370
Dbdo 10000
3.00
0.90
0.30
376
0,00
10 12 20135
1 12.22 20333
2. 10 20135
15 20333
10 2112
Figure B.3: VAL'ss performance of attribute prediction CelebA dataset. Each column shows results on the same feature with different data sizes and each column shows results on different features. In each graph, test accuracy of vanilla-VAB with different latent dimensions in blue line. And results of InteL-VAE with hierarchical prior are shown red. We find that our method (red line) achieves comparable even better results compared with vanilla- VAE with all latent dimensions.
shown
19

---

Published as a conference paper at ICLR 2022
APPENDIX C FULL METHOD AND EXPERIMENT DETAILS
In this section, we first provide complete details of the mapping designs used for our different InteL- VAE realizations along with some additional experiments. We then provide other general information about datasets, network structures, and experiment settings to facilitate results reproduction.
C.I MULTIPLE-
Mapping design Full details for this mapping were given in the main paper. Fig. C.I provides further illustration of the gluing process. Additional resulting including the Vamp-VAE are given Fig. 4.
1
-1
i
0
-1
1
(a) Circular prior with
(b) Glue point pair
(c) Implied prior with / =
Figure C.1: An illustration of the glue function in multiply-connected mappings.
C.2 MULTI-MODALITT
Mapping design In Sec. 6.2, we see the general idea of designing clustered mappings. In this part, we delve into the details of mapping design as well as extending dimensional cases. For simplicity's ss sake let us temporarily assume that the dimension of ) is 2. Our approach is based on splitting the original space into K equally sized sectors, where K is the number of clusters we wish to create, shown in Fig. 5b. For any point y, can get its component (sector) index ci(y) as well as its distance from the sector boundary dis(y). By further defining the radius direction for the k-th sector (cf Fig. 5c)
dimensional and high-
2T
k: + 1 )) sin
2
+ 1)))
(k)
cos
Wk (1,..., K}.
K
K
we can in turn define g(y) as:
r(y) = A (cii())).
(C.1)
( = y c)))s)))
(C.2)
where ej and 62 are constants, which are set to 5 and 0.2 in our experiments. we make sure still continuous by keeping g(y) y boundaries.
When dimension of y is greater than 2, we have more diverse choice for g. When K is decomposable, ie., K II, K,, can separately cut the plane expanded by V2 and 22+1 into K; sectors by the Eq. (C.1). As a result, ) split into K = II k; clusters. When = 2, we find that g only changes the 1-st dimension of V, so it can be applied cases where latent dimension is 1.
Learnable proportions We can also make the mapping more flexible by learning rather than assigning the cluster proportions. To do so, we keep learnable value uę for each cluster and set the angle the i-th sector as 2r Things are simpler for the 1-dimensional case where we can uniformly translate y by learnable bias 6 before splitting the space from the origin.
C.3 SPARSITY
Relationship to soft attention We note that our setup for the sparsity mapping shares some similarities with soft attention layer (Bahdanau al., 2014). However, there also some
20

---

Published as a conference paper at ICLR 2022
(a) Real distribution
(b) VAE
(c) Vamp-Vic VAE
(d) InteL-VAE
Figure C.2: Extension of Fig. showing Vamp- VAE baseline and additional circular target distribution (top row, uses the same single hole dæ the second and third rows).
important points of difference. Firstly, soft attention aims to find the weights to blend features from different time steps (for sequential data) or different positions (for image data). contrast, the dimension selector (DS) selects which dimensions activate deactivate for the same latent vector. Secondly, the weights of features are usually calculated by inner products features for soft attention, while DS relies network to directly output the logits.
Sparsity regularizer Our sparsity regularizer term, Lep. is used to encourage our dimensionality selector network (DS) to produce sparse mappings. It is defined using a mini-batch of samples (y.), drawn during training as per (7). During training, the first term of C.p decreases the number of activated dimensions for each sample, while the second term prevents the samples from all using the same of activated dimensions, which would cause the model degenerate to vanilla VAE with lower latent dimensionality.
note that Cap alone not expected to induce sparsity without also using the carefully constructed Jo of the suggested InteL-VAE. We confirm this empirically by performing an ablation study on MNIST where we apply this regularization directly to a vanilla VAE. We find that even when using very large values of > 30.0 Moreover, unlikely for the InteL-VAL VAE, this substantially deteriorates generation quality, with the FID score raising to more than 80.0 the same time.
only slightly increase the sparsity score (0.230 - 0.235).
Sparse metric We use the Hoyer extrinsic metrie (Hurley & Rickard, 2009) measure the sparsity of representations. For representation Ii
Hzyer(z) VD - |||||/||21|2
(C.3)
VD
Here, following Mathieu et al. (2019b), we crucially first normalized each dimension to have standard deviation 1, 14 za /d. ensure that only measure sparsity that varies between data
21

---

Published as a conference paper at ICLR 2022
0.9
300
0
200
Ours
1600
500
DO
250
Sparse-VAE
Accuracy (
0.7
Vanilla-VAE
200
FID
150
0.5
ViEig 3.01
50
a.
0:50
K-00.11
00.1
0..
20.0
0.3
Ourstre 13:0:
Ountyy 32:5
Dustyy 123.0)
0.4
0.6
0.8
1.0
1000 2000 5000
Sparse scores (-)
DDaa
Figure C.3: Results on MNIST. The left figure shows FID and sparsity scores. Lower FID scores (L) represent better sample quality while higher sparse scores (->) indicate sparser features. The right figure shows the performance sparse features from Intel. on downstream classification tasks. See Sec. 6.3 for details and results for MNIST.
Parameters
Synthetic
MNIST
Fashion-MNIST
MNIST-01
CelebA
Dataset sizes
Unlimited
55k/5k/10k
55k/5k/10k
10k/Ik/2k
163k/20k/20k
Input space
R
Binary 28x28
Binary 28x28
Binary 28x28
RGB 64x64x3
Encoder net
MLP
CNN
CNN
CNN CNN
CNN
Decoder net
MLP
CNN
CNN
CNN
Latent dimension
2-10
50
50
1-10
1-128
Batch size
10-500
100
100
100
100
Optimizer
Adam
Adam
Adam
Adam
Adam
Learning rate
le-3
1e-3
le-3
le-3
le-3
Table C.1: Hyperparameters used for different experiments.
points (as is desired), rather than any tendency to uniformly 'switch off" certain latent dimensions (which is tangential to our aims). In other words, this normalization is necessary to avoid giving high scores to representations whose length scales vary between dimensions, but which are not really sparse.
By averaging Hoyer(=) over all representations, we can get the sparse score method. For the sparsest case, where each representation has single activated dimension, the sparse score is 1. And when the representations get denser, || get smaller compared with | leading to smaller sparse scores.
Reproduction of Sparse VAI We tried two different code bases for Sparse- VAE (Tonolini et al., 2020). The official code base gives higher sparse scores for MNIST and FashionMNIST (though still lower than VAEe,.VAE), but is very unstable during training, with runs regularly failing after diverging and producing NaNs. This issue gets even more severe CelebA which occurs after only a few training steps, undermining our ability to train anything meaningful at all. To account for this, switched to the codebase" from De la Fuente & Aduviri (2019) that looked to replicate the results of the original paper. We report the results from this code base because it solves the instability issue and achieves reasonable results CelebA. Interestingly, though its generation quality is good MNIST and Fashion-MNIST, it fails to achieve sparse score significantly higher than vanilla-VAF. As the original paper does not provide any quantitative evaluation of the achieved sparsity, it is difficult to know if this behavior is expected. We note though that the qualitative results shown in the paper appear to be substantially less sparse than those we show for the InteL-VAE. ef their Figure compared to the top row of our Fig. 8. In particular, their representation seems to mostly 'switch off" some latents entirely, rather than having diversity between datapoints that needed to score well under the Hoyer metric.
'https: thub. fo51/3 ona ooaooonanaaaaaa -Sparsprreeeeee seterre-oodnn
22

---

Published as a conference paper at ICLR 2022
Encoder
Decoder
Input 64 64 3
Input dim
4x4 conv. 64 stride BN LReLU 4x4 conv. 128 stride & BN & LReLU
Dense (8x8x256)
ReLU
4x4 upconv. 256 stride & BN & ReLU
4x4 conv. 256 stride 2 & BN & LReLU
4x4 upconv. 128 stride & BN & ReLU
Dense (dim)
4x4 upconv. 3 stride
Table C.2: Encoder and Decoder structures for CelebA, where dim is the latent dimension.
C.4 ADDITIONAL EXPERIMENT DETAILS
Both synthetic and real datasets are used in this paper. All synthetic datasets (sphere, square, star, and mixture of Gaussian) are generated by generators provided our codes. For real datasets, We load MNIST, Fashion-MNIST, and CelebA directly from Tensorflow (Abadi et al., 2015), and we resize images from CelebA to 64x64 following Hou et al. (2017). For experiments with specified number of training samples, randomly select subset of the training data. We use the same random seed for each model the same experiment and different random seeds when repeating experiments.
Datasets
Model structure For low-dimensional data, the encoder and decoder are both simple multilayer perceptrons with hidden layers (10-10-10) and ReLU (Glorot et al., 201 1) activation. For MNIST and FaShion-MNIST, we use the same encoder and decoder as Mathicu et al. (2019b). For CelebA, the structure of convolutional networks shown in Table C.2.
Experiment settings GTX CTTX-1080-Ti GPU.
Other hyperparameters are shown in Table C.I. All experiments are run on
23


Reviews:
[{"review_id": "W5VEHWCP58", "reviewer": "Reviewer_9rm9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The technique is reasonable, and the experiments are with a decent comparison with baseline methods. Overall, I think this is solid work. That being said, some reasoning around the proposed framework sounds awkward, due to which I think the technical contribution of the paper is not clearly presented.\n\nMy concern is on the novelty of the abstract proposed framework, InteL-VAE. Basically, the transformation of the Gaussian latent variable, $g_\\psi$, is just a part of a specific design of the decoder. The authors emphasize that $g_\\psi$ is \"shared\" by both decoder and encoder, and I do understand the intention of such a stance (i.e., the output of $g_\\psi$ can be regarded as a transformed latent variable). However, I do not think it makes much sense because in a technical point of view, $g_\\psi$ is nothing more than a part of the decoder. Designing a decoder based on some prior knowledge of data is a common practice, and thus the abstract framework of the proposed method can hardly claim significant technical novelty.\n\nI think the technical contribution of the paper rather lies in the particular definitions of $g_\\psi$, some of which are presented in the main text, and more are in the appendix. For example, the multi-modality case in section C.2 is very interesting (yet looks ad-hoc). So, I would suggest pivoting the main claim of the paper on these particular instances of $g_\\psi$, rather than on the abstract framework of InteL-VAE. For example, it would be great if the multi-modality $g_\\psi$ is enough discussed in the main text, rather than being deferred to the appendix.", "summary_of_the_paper": "The authors propose to introduce some transformation of the Gaussian latent variable of VAEs in order to incorporate specific inductive biases on the semantics of the latent representation. They showcase several examples of such transformation and empirically examine the performance with comparison to baseline methods.", "summary_of_the_review": "While I think this is solid work, the current presentation is not necessarily kind for readers to understand the technical contribution specific to this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OTQTa_coBZ7", "reviewer": "Reviewer_NaAZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The idea of modifying decoders is interesting.\n- The presented mappings seem to be indeed helpful in most of the cases presented in the experimental section.\n\nWeaknesses:\n- I do not agree with the statement that the prior in the VAE framework is about regularizing. First, the VAE is not a Bayesian model (i.e., we still optimize the likelihood function, not the marginal likelihood). Second, the prior (or rather the marginal distribution over z’s) could be trained and then it serves a different purpose, namely, fitting a model to match the aggregated posterior distribution. Therefore, I find the paragraphs on the first page not fully convincing.\n- From the presentation perspective, it is rather a peculiar choice to refer to a figure (Fig. 3) that could be found on page 4. It breaks the flow of reading the paper.\n- It is unclear to me why the deterministic variable z is obtained by taking the mean value of y, namely, $z = g_{\\psi}(\\mu_{\\phi}(x))$ (page 4). Shouldn’t it be $z = g_{\\psi}(y)$? Otherwise, what’s the point in having a stochastic y?\n- Following the previous point, in Eq. 2, it is mentioned that $g_{\\psi}(y)$ is used. I start doubting whether the ELBO is correctly calculated.\n- I do not see how the proposed idea is different from having a more sophisticated decoder in which we focus on its specific part (i.e., first layers). In other words, I do not see why the proposed analysis in Sect. 4.1 and 4.2 cannot be applied to any decoder.\n- The whole paper is written in a pretty confusing manner. One can have an impression that z’s are separate random variables (the authors constantly repeat that the resulting distribution over z’s after applying a mapping g is, e.g., approximately spherical) but they are not used anywhere in the objective function explicitly. I fully understand that it is interesting to see how the inductive biases could be used and that the resulting distributions are of a specific form. However, it does not help a reader to follow the paper. Especially that the stochastic latent variables are y and they are used in the ELBO.\n- I tried to go through the code but after spending 30min I gave up. The code is not necessarily helpful to figure out some missing parts of the paper.\n- I think the paper presents some interesting ideas, however, it is lost by not necessarily a good way of presenting it.\n", "summary_of_the_paper": "The paper proposes to incorporate some inductive biases into VAEs by modifying decoders. The idea is to first transform stochastic latent variables using some specific mapping and then apply a deep neural network. The expected result is that the encoder will be forced to assign z’s in such a way that the mapping can turn them into a manifold of a given form.", "summary_of_the_review": "The paper shows that modifying a decoder in a VAE could be helpful and allows to still use a simple marginal distribution over z’s (e.g., a standard Gaussian distribution). However, the presentation of the paper is somehow misleading in many places that making it difficult to follow the flow of thoughts. Therefore, I believe the paper is ready to be accepted.\n\n=== UPDATE ===\nI would like to thank the authors for their rebuttal. Indeed, some of my concerns were fully addressed and the paper looks better now. However, I am still not fully convinced how to properly choose the transformation $g$ and what is the recipe for that. Anyway, I decided to increase my score to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F9b2bwWWMNu", "reviewer": "Reviewer_tE7T", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Firstly, I'd like to thank the authors for an exceptionally well-written paper - it was a joy to read! The well-made figures and plots really help to show the benefits of the proposed model.\n\nThe problem of how to effectively introduce biases in VAEs is nicely introduced and discussed. In particular, I found the discussion on the shortcomings of prior modification enlightening and convincing. The proposed model is both simple and intuitive, and I like the connection to normalising flows that the authors make a couple of times, since it gives a nice, mental picture. I would have liked, however, to see some of the equations relating to the inductive biases from the supplementary in the main paper, for instance equations (C.3) through (C.5). I think they are quite important for understanding how the model would work in practice.\n\nI am also not entirely convinced that the transformation should be seen as being shared between the encoder and decoder. I understand why the authors would like to push this view, since the introduced inductive bias, by definition, should improve the learnt representations, which may then be used for downstream tasks. However, my understanding of the model is that the transformation is simply the first layer of the decoder, which then happens to be handcrafted to introduce the appropriate inductive bias. I don't think this view reduces the significance of the proposed model; on the contrary, I find it cleaner to separate the encoder and the decoder this way. (Indeed, some VAE fundamentalists would argue that only the decoder is the model, so decoupling the encoder and the decoder would also please them :) ).\nIn this light, I don't think theorem 1 is necessary to have in the main paper and, personally, I find the discussion of how to construct inductive biases for different problems both more interesting and practically useful. Of course, if I have misunderstood the model I'm happy to be corrected by the authors.\n\nThe experiments are well-thought-out and show increasingly complex aspects of the model - from learning to represent toy data to learning sparse representations of CelebA images. I like the progression of the section, even though the supplementary material is needed to make it completely reproducible.\nThe point that the authors make that only the topological properties of the data distribution need to be matched by the inductive bias is particularly fascinating. I would very much like to see this discussed further somewhere, though it may be outside the scope of the paper.\n\nI didn't find any technical or mathematical issues in the paper, but there aren't many equations either. This isn't meant as a criticism - I like the simplicity of the model, and it highlights how little is needed to improve the decoder. In that sense, it's a nice reminder of how ineffective neural networks can be.\n\n\n**Questions**\n\nIn the discussion of the links to normalising flows, you mention that a normalising flow would be an unlikely choice for the transformation. I assume that you are referring to the architectural constraints and computational costs of normalising flows, but are there other reasons for not using a normalising flow as the transformation?\n\n\n**Typos**\n\n* Page 5, fourth paragraph from below: \"in start contrast\" -> \"in stark contrast\"\n* Page 9, first paragraph from above: \"DD significantly degrades generation ~generation~ quality\"\n\n\n", "summary_of_the_paper": "The paper presents a novel method for introducing inductive biases into variational auto-encoders (VAEs). The inductive biases are specified through a deterministic transformation in the latent space, which is shared between the encoder and the decoder. Through experiments on toy data, the authors demonstrate how to design appropriate inductive biases by matching the topological properties of the transformed representations to those of the real data distributions. They further quantify the performance of the model's ability to learn informative representations through experiments on MNIST, Fashion-MNIST, and CelebA, and show that it outperforms competitors both in terms of the generation quality with and without sparsity constraints as well as terms of downstream classification based on the representations.", "summary_of_the_review": "The proposed model is clearly of interest to the ICLR community. Both the discussion of the issues with modifying the prior and the simple and effective solution of introducing a latent transformation should provoke some interesting discussions and further work. The paper is exceptionally well-written and presented, and while it would be a stretch to call it a groundbreaking paper, it is a solid \"accept\".\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0fR0K8N7KFb", "reviewer": "Reviewer_pbbP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Incorporating human knowledge into deep learning architectures is becoming a very relevant research area as the ML community is integrating with science and engineering. The paper offers a relevant contribution to this booming field of research. \n- The paper is very well written and it contains sound and convincing arguments in favor of using user designed inductive biases and against the use of the VAE prior in order to induce those constraints. \n- While the general approach is not very novel, the specific layers used to induce topological constraints are innovative, clever and useful. In my opinion the main contribution comes from these layers rather than the general technique. \n\nWeaknesses:\n- From a technical point of view, the main contribution is hardly novel as it simply entails the use of a user designed fist encoder layer. However, the approach is interesting from a conceptual and  interpretability standpoint.\n- The sparsity application is hardly novel as it is nothing more than a simple soft attention layer. Soft attention is already widely used in the generative modeling literature. \n\nSuggestions:\n- I would recoment the authors to de-emphasize the general approach and to leave more space for the details of the topological layers used as in my opinion they are the main innovation of this work. ", "summary_of_the_paper": "This work introduces a new method to embed used-designed inductive-biases into VAE architectures. the basic idea is straightforward: a fixed user-defined deterministic transformation is used to map the latent z into a \"structured latent\" y. The transformation is used to induces topological and/or sparsity constraints. Technically, this is identical to using the transformation as the first layer of the decoder architecture. Besides of the general idea, the authors introduce a variety of simple but clever transformations used to enforce latent constraints. The approach is shown to offer an interpretable latent space and attractive performance in a large class of density estimation tasks.", "summary_of_the_review": "The paper is well written and convincing. The new framework is not novel from a technical point of view but the details of the layers introduced to model complex topologies are. There is a major need of methods for embedding human knowledge into deep learning architectures and this paper is on a good direction. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: I am recommending a poster for this paper.  There was considerable discussion and much author response.  The reviews were good (after taking author response and paper revision into account) with one out of three being enthusiastic. There was a concern that the basic idea was technically mis-represented as the inductive bias is being placed in the decoder rather than prior. But I am convinced that it is a reasonable idea to place bias in the decoder and that idea is worth publication.

Personally I think the paper would be much stronger with better empirical evaluation.  I find a focus on MNIST (or fashion MNIST) unconvincing. Results on CelebA should be accompanied by sample image generations.  I would rather see downstream task metrics based on learned features.  This paper cannot be put in the same class as recent results on unsupervised learning of image features for downstream tasks. It remains an open question as to whether this paper provides any contribution in that arena.
Decision: Accept (Poster)