Prior Learning in Introspective VAEs
Authors: Ioannis Athanasiadis
Fredrik Lindsten
Michael Felsberg
Venue: Accepted by TMLR ()

Abstract: Variational Autoencoders (VAEs) are a popular framework for unsupervised learning and data generation. A plethora of methods have been proposed focusing on improving VAEs, with the incorporation of adversarial objectives and the integration of prior learning mechanisms being prominent directions. When it comes to the former, an indicative instance is the recently introduced family of Introspective VAEs aiming at ensuring that a low likelihood is assigned to unrealistic samples. In this study, we focus on the Soft-IntroVAE (S-IntroVAE), one of only two members of the Introspective VAE family, the other being the original IntroVAE. We select S-IntroVAE for its state-of-the-art status and its training stability. In particular, we investigate the implication of incorporating a multimodal and trainable prior into this S-IntroVAE. Namely, we formulate the prior as a third player and show that when trained in cooperation with the decoder constitutes an effective way for prior learning, which shares the Nash Equilibrium with the vanilla S-IntroVAE. Furthermore, based on a modified formulation of the optimal ELBO in S-IntroVAE, we develop theoretically motivated regularizations, namely (i) adaptive variance clipping to stabilize training when learning the prior and (ii) responsibility regularization to discourage the formation of inactive prior modes. Finally, we perform a series of targeted experiments on a 2D density estimation benchmark and in an image generation setting comprised of the (F)-MNIST and CIFAR-10 datasets demonstrating the effect of prior learning in S-IntroVAE in generation and representation learning.

Published in Transactions on Machine Learning Research (06/2025)
Prior Learning in Introspective VAEs
Ioannis Athanasiadis
fommniss.thhaaasiiin &liu.se
Department of Electrical Engineering
Linköping University
Fredrik Lindsten
fredrik. lindstenn @liu.se
Department Computer and Information Science Linköping Universiby
Michael Felsberg
michael. Gliu.se
Department of Electrionl Engineering
Linköping University
Reviewed OpenReview: https://openrentt net/ forum? id=u4YDVFodyX
Abstract
Variational Autoencoders (VAEs) popular framework for unsupervised learning and data generation. A plethora of methods have been proposed focusing improving VAEs, with the incorporation of adversarial objectives and the integration of prior learning mecha- nisms being prominent directions. When it comes to the former, an indicative instance is the recently introduced family Introspective VAEs aiming at ensuring that a low likelihood assigned unrcalistic samples. In this study, we focus on the VAE (S-IntroVAE)) one of only two members of the Introspective VAE family, the other being the original IntroVAF. We select S-IntroVAE for its state-of-the-art status and its training stability. In particular, we investigate the implication of incorporating multimodal and trainable prior into this I-trooVAE. Namely, we formulate the prior as a third player and show that when trained in cooperation with the decoder constitutes an effective way for prior learn- ing, which shares the Nash Equilibrium with the vanilla S-IntroVAE. Furthermore, based on a modified formulation the optimal ELBO S-IntroVAE, we develop theoretically motivated regularizations, namely (i) adaptive variance clipping stabilize training when learning the prior and (fi) responsibility regularization to discourage the formation of in- active prior modes. Finally, perform series of targeted experiments on 2D density estimation benchmark and in an image generation setting comprised of the (F)-MNIST and CIFAR-10 10 datasets demonstrating the effect of prior learning in S-IntroVAL in generation and representation learning.
1 Introduction
Variational Autoencoders (VAFs) RRzznnde et al.) 2014 Kingma Welling 2013) constitute a popular
gencrative framework where variational inference utilized learn low-dimensional embeddings by modcling the density of the high-dimensional data. VAEs enjoy plethora of applications, ranging from anomaly detection (Chauhan et al. 2022 to representation disentanglement (Higgins et al. 2017) and high-resolution
image generation [Razavi et al. 2019)
From a representation learning perspective, VAEs produce structured latent spaces due to the regularization imposed by fitting a prior distribution. This contrasts with unregularized autoencoders, which often lack such structure (Shrivastava et al. 2024 Oring 2021 Leeb al., (2020). These structured representations are particularly valuable in domains like scientific discovery, where understanding the underlying data structure
is critical (Wang al. 2023).

---

Published in Transactions on Machine Learning Research (06/2025)
log(p(z(s))
log(p(æ) a))
Encoder Update
Decoder Update
Enc.
Dac.
A)
KL[=pp(2]
Prior
KL[2|p(2)
Fixed Prior
Prior Update
Figure Prior player realizations within Introspective VAEs. The prior component can be regarded as a third player that can actively participate in the adversarial game along with the encoder and the decoder. The overlayed snowflake indicates that the component not updated.
Despite VAEs falling short of other popular generative paradigms, such the Generative Adversarial Net- works (GANs) Goodfellow et al. 2020 and diffusion models (Ho et al. 2020) in terms of generation quality, they are distinctive in the sense of simultancously providing the amortized inference and generation modeling Gatoooolos & Tomczak 2021 Building upon that, combining VAFs with these frameworks has been a popular research direction aiming at retaining its merits while mitigating its limitations (Makhzani et al. 2016). With diffusion models emerging as the stato-of-tho-art framework for image synthesis Yangg et al. 2022), there are recent works on VAE/diffusion hybrids (Preechakul al. 2022 Rey et al. 2019). Addition- ally, the VAE/GAN hybrid literature is also established sub-field targeted improving the poor training stability of GANs (Mescheder et al. 2018) and generation diversity (Huang et al. 2018 while addressing the blurry generation of VAEs.
Another independent direction to improving VAEs is learning the prior as opposed to fitting into a fixed onc, commonly the standard Gaussian. Trainable priors allow for identifying structure within the data, which is of high interest in the unsupervised and semi-supervised learning setups. Moreover, sufficiently expressive priors are needed for generating realistic data from complex distributions (Lavda et al.) 2019 Dilokthanakul et al. 2016 Additionally, utilizing the structured capture in the latent space (Lavda et al. 2019 can benefit the generation performance well as provide control over the semantics of the generated samples even in the absence of labels.
Motivated by the prospect of combining the strength two distinct and conceptually different directions for enhancing VAFs, we consider the problem of incorporating prior learning in the S-IntroVAF (Daniel & Tamar 2021) framework. Our intuition is that the appcaling features of roducing over-regularization and holes enabled by prior learning
sufficient for realistic sample generation. On the other hand, although adversarially trained VAEs possess higher quality generation capabilities, they still subject to the problem associated with assuming an over-simplistcc prior.
Based on these, we formulate the prior as an additional player in S-IntroVAE, which participates in the adversarial training. More specifically, we extend the original analysis provided by Daniel & Tamar (202I) and conclude that the prior-decoder cooperation scheme is viable option for learning the prior while remaining faithful to the Nash Equilibrium (NE) of the vanilla S-IntroAEE. Our work is partly related the CS-Intro VAE. Yuu et al. 2023), where fixed three-component Mixture of Gaussian (MoG) prior
2

---

Published in Transactions on Machine Learning Research (06/2025)
integrated into S-IntroVAE by replacing the Kullback-Leible (KL) with the Cauchy-Schward divergence to allow for closed-form divergence computation. Notably, in our work, we follow the original variational analysis provided in Daniel & Tamar (2021), using the KL divergence and its theoretical properties, thereby investigating the effect of using a multimodal prior, including its trainable form, in isolation. Formally our contributions are:
extending the original S-Intro VAE under the preoder cooperation scheme.
two theoretically motivated regularization (i) adaptive variance clipping and (ii) responsibilities
entropy, which enable robust prior lcarning.
the experiments on a synthetic 2D density estimation and an image generation task demonstrating the effect of prior learning in S-IntroVAE in generation and representation learning.
2 Related Work
autoencoder- structure is utilized, along with variational inference, to maximize a lower bound the marginal log-likelihood of the data (the evidence lower bound, ELBO). More specifically, this resorts to simultaneously minimizing the sum of the empirical reconstruction error and the Kullback-Leibler (KT) divergence between the extracted latent representations and an assumed prior (typically the standard Gaussian distribution). A tighter ELBO was proposed by [urrda al. (2015), based on an importance weighting scheme, providing more fiexibility during training by being more forgiving of inaccurate posterior estimates. Hierarchical variations of VAEs (Vahdat & Kautz 2020: Sønderby et al., 2016) rely on multiple stochastic layers where each of them is conditioned on the previous one, resulting in more efficient representation learning (Child 2020 Zhao et al. 2017
VAEs: In VA RRezende et al. 2014 Kingma & Welling, 2013
Prior Assumption in VAEs: Several studies suggest that assuming an over-simplistic prior can over-
regularize the VAEs hindering their performance (Lin & Clark 2020 Tomczak & Welling 2018 Hoffman
& Johnson 2016). Goyal et al. (2017) argue that assuming a standard Gaussian prior can onill meaningful semantic information in the latent representation. Moreover an over-simplistic prior introduces holes in the prior negatively affecting the generation capabilities of VAËs (Aneja et al.] 2021 Rezende & Viola 2018 Towards addressing this shortcoming, Tomezak & Welling (0088) proposed the VampPrior where trainable pseudo-inputs are fed into the encoder providing the parameters of a. MoG distribution to replace the standard one. Connor et al. (2021) adopt a manifold-learning approach to define an MoG prior, which is better crafted for the latent space of the data. Kalatzis et al. (2020) assunne a Riemannian latent space where the prior is inferred from the data, replacing the standard Gaussian with a Brownian motion prior.
Adversarial Objectives in VAEs: In Adversarial Autoencoders (AEEs) Makhzani et al. 2016) the latent space is regularized into following the assumed prior through min-max game between the encoder and discriminator module. The VAE/GAN hybrid was proposed by Larsen al. (2016) where the similarity distance, for mcasuring the reconstruction error, is implicitly learned through an adversarial game in which the decoder network serves as both a VAE decoder and the generator of GAN. In the seminal IntroVAE (Huang et al., 2018), the VAFs are framed as an adversarial game between the encoder and the decoder by considering the KL divergence as an energy function. The I-IntroVAB improves the training stability of VAEE while also providing the thcoretical analysis suggesting that Introspoctive VAEs constitute a variational instance of GANs. In CS-IntroVAE (Yu al.) 2023) the KL was replaced by Cauchy- Schwarz divergence while using fixed three-component MoG in I-IrroVAR leading to improved generation perfor- mance.
3 Background
Our work builds upon the framework proposed by Daniel & Tamar (022)). To avoid confusion we adopt, whenever possible, identical notations as presented in their work. Let x ~ Paaca (æ) be a data sample and a latent representation. A VAE aims at learning a parametric model Pa, (x, 2) Pa, (#lz)p. such that the marginal log- likeliiddoood of the data maximized. Due to the intractability of that likelihood (Kingma & Welling 2013), we resort to maximizing the E.B.O. Assuming a prior Pz the latent space, an encoder go
3

---

Published in Transactions on Machine Learning Research (06/2025)
providing the approximating posterior and decoder i parametrized by ø and 0 respectively, evaluate the ELBO, denoted as W, at point x as:
(a, (=)) (aaee)))exaaaaaalll (x|=)] S log p(g)
(1)
with [T[[-I-] denoting the KL, divergence. In practice, the encoder and the decoder are typically realized through neural networks with parameters ó and 0, respectively. The 3-VAE reformulates the ELBO by weighting the relative contribution of the KL term using the 3 hyperparameter, that is:
W (1; Uo. Paos 3) = -B (T]oo()/<))lp<(0)) + (ere Pa. (x/|2)]-
(2)
Note that, from an optimization perspective, the 8-VAE ELBO formulation is equivalent to using independent weighting hyperparameters for each of its constituting terms, such that W(x; qs. Pd. Srees SKL) = - 0KL KL (a [=)]][=)] + Bree [Ez=go(=)=) [log Pa, (x/z)) This ELBO formulation is convenient for tuning the S- IntroVAE and therefore is the adopted ELBO formulation. Additionally, when clear from the context, we omit expressing the ELBO with respect to the 3KL. and Pree to enhance clarity.
3.1 Learning the optimal prior
In light of the previously discussed implication of imposing a simple prior in VAE, the question arises: what is the optimal prior ? In this aspect, an insightful reformulation of the empirical ELBO is provided by
Tomezak
)022
Eg (1) (a) (W:: Pdo Brec, BKL)) = Brec -Er-naaua (e) (elo Pa. Pa. (x|²) + 3KL (H( CE) (=))).
(3)
with H-|- and CEI-II-] denoting the Shannon and the cross-entropies, respectively, and 96(2) = E Ex-psaut (=(2)²)) is the aggregated posterior. The formulation above suggests that the optimal prior can be found as the maximizer of the ELBO, namely (=(=) (2(),, as this is when the negative cross entropy term maximized (Gibbs' inequality). Towards this, utilizing trainable MoG prior emerges as a relevant alternative to the standard Gaussian. A prior encoder pairing was realized by Tomczak & Welling (2018), termed VampPrior, leading to better separation in latent space. Formally, the MoG and Vamp M-modal priors, denoted by px(z) and (2) respectively, are parametrized as:
M (x = Σ wi -N(2lu,, and p²())= = Σ w. gg(2),;)
M
(4)
with Σ w, and wi the contribution of each component, #4 and o, the means and variances of the MoG prior and x, pseudo-inputs for the VampPrior.
3.2 S-IntroVAE
correln typical VAEs, the encoder and the decoder are updated simultaneously in a single backpropagation stage. Motivated by the observation that assigning high likelihood for the real data does not necessarily imply assigning low likelihood for the unlikely ones, the Introspective VAEs family (Daniel & Tamar 2021
Huang et al.) 2018) formulates an adversarial game between the encoder and the decoder. In S- DDaniel & Tamar 2021), the ELBO is regarded as an energy function, and on that basis, the encoder is induced to assign high energy to real and low energy to generated data. On the contrary, the decoder aims at generating data (i.c., reconstructed and generated samples) that resemble those of the real data distribution to fool the encoder. The above setup constitutes an adversarial game between the encoder and the decoder similar to the GAN (Goodfellow et. al.) 2020) paradigm.
4

---

Published in Transactions on Machine Learning Research (06/2025)
For notational brevity in the derivations below we drop the dependence the parameters and ø and simply write d for the decoder and q for the encoder, while we henceforth refer to Bu--(a)) simply as Rp[] when clear from the context. Formally, given the empirical Pasta(z) and pa(x) = (=) (P)](((]])) the generated data distribution, the encoder q and decoder are alternately updated towards maximizing their respective objectives (g d) and La(g, d) defined as:
Lg d) = Epanta Wx:qq,d) - Epul" cxp(aW(r.,.))), d)),
(5)
La(g,d) Epast [W q, d)) + Epu (W))g,
where a 2 and y 2 0 are hyperparaneters. Daniel Tamar (2021) show that there is a NE for this two-player game. Specifically, define d' as:
d' E arg arg min (1)llpaaa(=))laaaaa + y [[]p(()])
(6)
d
Assumption 1 (Modified DDniel & Tamar 2021 ). For all x such that Pdata (2) 2 0 we have that (pa (z)]a++ S (x).
Theorem (Daniel & Tamar, 2021 Under the Assumplion the pair of optimal q" = Pa- (z/ar and d as defined in (6) constitutes a NE of the game (5)
Remark 1. The Assumption 1 is a modified version of the used by Daniel & Tamar 2021 and es- sentially suggests that (z)( has to be sufficiently enclosed by the true data distribution. This modification corrects a seemingly minor oversight that, however, has imporlant implications for the interprelation of the theorem. In particular, the proof of Theorem requires expressing the condition under which the optimal q", which marimizes Lyg satisfies KL[q" (2/=)]]p[[=(x)] =. Importantly, if a sample x lies outside of the support of (zata(z) but within the support of pz(z), then the optimal qº satisfies KL pa(2<=)] cc. The original assumption in (Daniel 8 Tamar 2021) does not separate this case, whereas the modified assumption properly accounts for it. See B. for detailed explanation of this malter.
We refer the readers to the original work of Daniel & Tamar (2021) for the proof that for every Pdata there always exists 7 2 0 such that Assumption holds for Pa- Theorem D suggests that, at convergence, the S-Intro formulation leads to optimal inference capabilities (i.c., the approximated posterior equals the true one) while the generated data distribution converges to an entropy-regularizedd version of the true dala distribution.
4 Prior Learning in S-IntroVAE
4.1 Theoretical analysis
In this section, we extend S-IntroVAF by introducing a third player dedicated to modeling the prior. Our formulation draws inspiration from DeLiGAN (Gurumurthy et al. 2017] where the noise in GANs was parametrized by a learnable MoG. In contrast to DeLiGAN, in our setting the prior (which is similar to the noise in GANs) has dual role as (i) the source of the generated data distribution and (ii) the target basod on which the adversarial training is performed. We theorctically analyze the implication of training the prior within the S-IntroVAE and conclude that learning it in cooperation with the decoder constitutes viable option for prior learning.
In our three- player setup the encoder g, the decoder d, and the prior À are all flexible. We denote the generated data distribution as v²)x) = [px(s)[p<(>> | =)] to highlight its dependence on both the decoder d and the prior players. In that case, the adversarial game of (5) becomes:
[ exp(a W ((;
Lg(A, q. d) Eprata W(: d)) - Eg
(7)
Taa(), q. d) =Epuau [W(x; d)) + . Kpg [Wxxd,,
5

---

Published in Transactions on Machine Learning Research (06/2025)
The encoder is trained to maximize the Lg whereas the prior and the decoder maximize the La objective (i.e., prior-decoder cooperation). Below we show that prior-decoder cooperation is a viable option for prior learning which retains NE from the original S-IntroVAIl formulation.
We modify (6 to support our learnable prior setup. Let denote the set of possible parameterizations of the prior and A, we define:
(X", E arg min (=)]))aa + Hlp)()))
X,d
Let us also extend Assumption to account for the prior being learnable.
Assumption 2. For all such that Pdota (z) 2 we have that [p]- (x)]a S Pdota(x)..
Corollary 1. Under the Assumption 2 when training the prior player in cooperation with the decoder
player then the triplet q" = p.- (z|x), and dr as defined in constitutes a NE of the game
Sketch of proof. The proof of Corollary follows from Theorem under the modified Assumption (see
Remark D. Analogous to Theorem DDanill Tamar 2021 proving Corollary entails first showing
that the optimal encoder converges the true posterior under the Assumption 2 Then, given that the
encoder has converged, the prior and the decoder as defined in (8) maximize the La objective defined in
This concludes the proof of Corollary 1 For the complete proof, we refer readers to B.2.2
Our three-player formulation similar in nature to S-IntroVAE with the encoder converging the true posterior while the generated data distribution converges to mntropy-regularine version of the real data distribution. The key difference, however, lies in the fact that our formulation allows for trainable prior, unlocking the merits of prior learning such as mitigating the prior hole problem, unsupervised clustering (Dilokthanakul et al., 2016),, explainability (Klushyn et al. 2019). and more controllable generation (Lavda et al. 2019). More specifically, for fixed encoder q and decoder d, given a batch of real and generated data respectively, the prior update socks (i) to support a linear combination (controlled by the y hyperparameter) of the empirical real and fake aggregated posterior and (ii) be idempotent under the projection by d.
4.1.1 Optimal ELBO in the assumption- setting
Corollary [ requires Assumption 2 to hold, however, in practice this might not be the case, especially carly in training. For instance, having a (z) generating (i) out-of-distribution data or (ii) realistic samples al a disproportionately higher rate compared to the real distribution, are two obvious cases where such an assumption is violated. Analyzing the behavior of the encoder in these cases provides an intuitive connection to regularly trained VAEs and motivates some of implementation choices. Let X = {a|2 Pdata (z) >> OU p (x) >> 0} (i.e., the set of all possible samples in the union of real and generated data supports), we define the ELBO W X, d) as:
00, fx E X
Paata (x) = 0}
W(x:A,g',d)>
(aaia(x) 1 log ma(a))
x E for E Paata (④) > on (x) > pdata
(9)
(o), x (o X (x)sa()) > on (x) S Pdataa (d))a()))))
Proposition Given a fiaed generated data distribution (x) the q" mazimizing Lg d, in Eq. 7 such that the ELBO W W(z; X, ,-, dd) satisfies Eq. g
The proposition (for the proof see B.3 above suggests that under the Assumption 2 the encoder in S- VAE behaves similar to the one in regular VAEs. Alternatively, as a consequence of the repelling objective acting the generated data, the encoder in S-TntroVAF diverges from its VAF-optimal state. This

---

Published in Transactions on Machine Learning Research (06/2025)
divergence depends the sample-wise mismatch between (a( and (data()). Interestingly, it also appears that the optimal ELBO with respect to the encoder is a continuous function of the (d)ta measure.
4.1.2 Practical implications in the assumption-free free setting
Let us now investigate how the theoretical claims suggested by Proposition 1 are realized in practice. For this purpose, the image generation setting was deemed an appropriate testbed due to being easy interpret while the same time sufficiently complex allowing sition only concerns the optimal encoder given fixed real and generated distributions. Based on that, we employ well-trained S-Intro VAE VAE In this regard, having the prior and the decoder fixed translates to having a fixed generated data distribution.
draw generalizable conclusions. Note that propo-
and overfit the encoder network while keeping the prior and decoder fixed.
Bueg
Bueg 256
Pd
2
data 8yea
REC
NELBO
200
150
syu P data (z
REC
NELBO
1006
2000
3000
1000
2000
3000
1000
2000
3000
120
2003
100
800
(aa))
Pd
REC
KL
NELBO
REC
KL
NELBO
1000
2000
1000 2000
1000 2000
1.00
Figure 2: Overfitting the encoder given a fixed generated data distribution across three different configura- tions (rows). The experiment was conducted both under the thcoretically faithful hyperparamcter setting (Bneg = 1- left) and the one used in practice (Borg 256 right). The first. line in the REC, KL and NELBO plots refers to the real data distribution whereas the second one refers to the generated data distribution. The image data figures, each configuration, correspond to the real data distribution, the reconstructed real data distribution, the generated data distribution and the reconstructed generated data distribution from top to bottom. The figures above generated by utilizing trained S-IntroVAB trained under fixed 10-modal MoG prior.

---

Published in Transactions on Machine Learning Research (06/2025)
As outlined by the proposition, the encoder treats each sample (i.e., image in this context) differently depending on the likelihood ratio between Pdata(z) and (2).. Unfortunately, to this end, we can not make use of the Proposition | as we do not have access to the analytical densities of either of these distributions. To overcome the aforementioned challenge we use a subset of the real data distribution to construct synthetic real and generated data distributions, denoted as Paata syn (x) and pa (x) respectively. Additionally, it was also nccessary, to use a batch size of 1 to avoid leaking information between samples inside and outside of the support of real data distribution due to the batch normalization layers. Based on these experiment can use them proxies for testing the proposition. Specifically,
synthetic distributions,
with three distinct configurations with different properties: (i) Paata(() (a)) = pá (x)) where both distributions consist of multiple different samples (ii) Pdata (z) consisting of single sample 2o, whereas pa p²a""(2))
(zx consists of multiple samples, including the zo of the Palta (x) and (iii) the reversed (ii) where Palta(a) (x) and p/"'" (x)
payn
distributions are swapped. We used 10 samples, one for each class, to construct the synthetic distributions.
In its theoretical faithful realization the results for (i), (ii) and (iii) are displayed on the left side of Figure under Sneg = 1. These closely align with what has been suggested by the proposition where when the likelihood of generating a sample is sufficienlly enclosed by the likelihood of observing that sample in the real data distribution then the encoder pushes the ELBO towards VAF-optimal levels. On the other hand, in cases where there is a significant likelihood mismatch the encoder can afford to either push the ELBO to its optimal level or diverge from that depending on whether the mismatch appears with respect. to the real the fake data distribution. For instance, when looking at configuration (ii) (2nd row) the encoder minimizes the NELBO (negative ELBO) for the image that is 10 times more likely under the real distribution compared to the fake distribution, whereas the NELBO increases for the samples outside the support of the real data distribution.
In practice, when computing the loss corresponding to maximizing Ly objective, the real and fake ELBOs use different weights for the reconstruction and the KL. losses, in particular for CIFAR-10 the Baeg: corresponding to the 3KL. for the fake E.BBO, was set to 256 while the remaining I's were set to 1. Using Aneg 256 essentially prompts the encoder to focus more the KL compared to the reconstruction loss when repelling the fake data. However, even in this case, where the hyperparameter configuration diverges from the one theoretically accounted for, we observe that similar patterns emerge.
4.2 Implementation
In this section, we outline the implementation choices as well as the motivation behind them enabling prior lcarning in S-IntroVAE in a prior-decoder cooperation manner. Pseudo-code for the prior learning in S- Intr is provided in Algorithm [
4.2.1 Prior as source and target
In the prior- cooperation setting the prior player À maximizes La(A, q, d). In practice, given a real Freal and 2, ~ px(2). the prior minimizes the loss [x(ereals2, 2x) given by:
Lp(Ereal, 2s) -Brec (Freaar)al) + BKI. LKI((real) (Freat) + (7r Broe (s((((())))
(10)
+ 3KL (Dx)))),
where D(2x) is the fake sample generated from decoding the latent 2,, while Lree and LkL the reconstruction and the KL losses respectively. We remained consistent with the S-IntroVAE,, where the reconstruction of fake data was scaled by 7 = 10-8 and the stop-gradient (sg) operator was applied when generating fake sample before computing its reconstruction loss. Additionally, we observe that the reconstruction loss for the real sample is not affected by the prior. In light of these, the prior player is trained both as a larget for the real and fake posterior and as a source of fake samples. Based on that, a subtle issue arises when minimizing the (DL=)) term, since the prior can minimize it by either becoming a good source for generating realistic data or a good target that supports the posterior of generated data of low quality. The latter case is particularly problematic during the early stages of training, when the generated data lie outside the support of the real data, causing the encoder to assign suboptimal posterior, described in
8

---

Published in Transactions on Machine Learning Research (06/2025)
cipping
Arasy 58
11.
10
4-
100
Heraaton in thousands
Will chopre
15.0
12.5
0.4
10.3
0.-
25
200
325 150
175
50
100 150 200 230
300
(a)
(b)
Figure 3: Regularization for robust prior learning in S-IntroVAE. (a) Clipping the prior log- variance is crucial for maintaining the stability of S-IntroVAL under the prior decoder cooperation. Two models were trained CIFAR-10 for 120 epochs until crashing. Without clipping, the log- log-variance tends to explode, leading to an increase in the FID metric and ultimately causing the model to generate indiscernible patterns before crashing. A unimodal trainable prior was used for both models differing only by the log- lor-variance clipping, (b) Emerging mode collapse due to unconstrained gencration. We trained two models on CIFAR-10 for 200 epochs under a 10-modal MoG (log- variance clipped), differing only in the amount of entropy regularization. The red border indicates samples generated by inactive modes (i.c., average responsibility smaller than 10-2). Note that not regularizing the responsibility entropy quickly degenerates into a unimodal prior setting where single mode is responsible for supporting the aggregated posterior. The unimodal collapse eventually leads to mode collapse and an increase in FID due to the unconstrained generation originating from the inactive modes. On the contrary, regularizing the responsibility entropy maintains more uniform responsibility allocation among the modes and addresses the mode collapse issue.
Proposition] [ To address this, we follow Shocher et al. 2023) and apply the ag operator to the prior as the target while allowing gradient flow for the prior as the source when computing LKL for the fake samples. We henceforth refer to this modified LKI. as eg which replaces the original when computing the KT. loss of the D(z,) in 100
4.2.2 Adaptive variance soft-clipping
Although theoretically sound, the prior decoder cooperation scheme led to instabilities. In particular when parameterizing the prior as MoG prior these instabilities manifested as exploding prior log- varviniissess (sce
Fig. 3ab that became evident as the real data distribution became more complex (e.g. CIFAR-10 images vs 2D data). We attribute the aforementioned behavior to the interplay of three aspects: (i) the encoder pushing to suboptimal ELBOs (i.c., suboptimal roconstruction and KL losses) for those samples whose likelihood in fake data distribution is not sufficiently enclosed by the real one (see Proposition implication in 4.1.2. (ii) hyperparameter-tunnng caveats where good results generally required setting the
D and its practical
9

---

Published in Transactions on Machine Learning Research (06/2025)
SKL of the fake ELBO (termed as Queg) to be an order of magnitude of the latent dimension Daniel & Tamar 2021 and (iii) the behavior of the target distribution in KT. minimization where the target variance
increases when the source posterior is unlikely under the target distribution (see C.4b. Notably, (i) and (ii) promote the posterior of the real samples that overlap with insufficiently enclosed fake ones to diverge from the prior whereas (iii) increases the variance of the prior in an atlempt to support a diverging aggregated posterior, which can lead to exploding log-variance in severe cases of (i) corresponding the second row of Eliminating (i) or (ii) requires extensive hyper-parameeer tuning for each Pdata, assumiin that such parer-erreettrr set even exists. Instead, opted to address the issue of exploding log- vagvaaiassess by tackling (iii). Namely, we employed an adapting soft-clipping scheme inspired by Chua et al. (2018) ; Chang et al. (0023) where instabilities were also observed when learning log- log-variances. Concretely, for each latent dimension j, the prior log-variance is clipped to the range [aj. bj] using the function Sa, defined as:
1 1 exp(33 (a, (1y=x)) 3j
fa(x)= = x
log
(11)
+ cxp(3) (x - b;))"
with B, = by- and K a positive hyperparameter. The formulation above allows for controlling the steepness of clipping in a unified way using a single hyperparameter K for all latent dimensions. We elaborate further on this choice in C.T For our all our experiments we used = 10.
4.2.3 Responsibilities regularization
Duc to the nature of the Lg objective inducing the encoder to act discriminator between real and fake data, it is evident that the posterior can diverge arbitrarily from the prior (see Proposition 1
In practice, we observed that such behavior can cause certain prior components, of a MoG prior, to become more dominant than others in terms of the responsibilities of prior modes to posterior, leading to the formation of inactive prior modes and vanishing gradients (see C.3.2p. Consequently, as the aggregated posterior only supported by portion of the prior modes, there are not multiple real samples competing for the same region of the latent space leading to unconstrainted generation when sampling for those inactive prior modes. Note that the issue of inactive prior modes formation is applicable both when having a trainable (prior-dccoder cooperation) and fixed MoG prior. To alleviate this we employ an entropy regularization on the responsibilities of each prior component discouraging inactive modes from forming. Concretely, the responsibility & corresponding to the ith mode is computed as:
w; -N(elu,,
C = (x) (x) (zyng()la)
(12)
1=1 wi N(221,,
M
Finally, we define the responsibility vector as:
C = [c1, ,a,,e,,....cc cMr,
(13)
and compute its normalized entropy H. (C). The (C) weighted by a non-negative hyperparameter Tentropy: added to the Ly objective. Notably, our responsibility regularization is closely related to the mean entropy maximization regularizer used by Assran et al. 2022 Joulin & Bach (0022) regularizing the mode assignments instead of cluster assignments. Ultimately, encouraging uniform responsibilities accounts for the vanishing gradient issue. We provide the derivation for the prior mode responsibilities in C.3 Fig 3b illustrates a representative case of responsibility entropy development when left unregularized. Note that the responsibility regularization is relevant only for multi-moda priors and therefore not needed in the original S-IntroVAE under the standard Gaussian prior.
TTee entropy normalized dividing by the maximum entropy given M possible assignments, where M the number of prior components in the MoG.
10

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm 1 Prior Learning in S-IntroVAA (Daniel & Tamar 2021). The oad-highlighted segments indicate the parts that differ from the standard Gaussian S-IntroVAE. The Lree and the LKL refer to the reconstruction loss and the KL divergence between the posterior and the prior target respoctively, whereas the 48 is a LKL
modified KL divergence that applies the stop-gradient ag operator on the prior as target.
Require: Brec.9KL. Pneg: 7 9. Fantropy;
1: ΦE-PA,Op 0p & Initialize network parameters
2: s 1/input dim
b Scaling constant Scaling parameter for fake data reconstruction A VAE training stage precedes adversarial training
3: "rr + -00-
4: a + Clipping ranged found after the VAE training stage
5: while not converged do
6: Treal Random mini-batch from dataset
7: Euy Plogvar, Get MoG prior parameters from PA
8: llggar CLuPLOGYARIANCCH(2Ogaaaa (Jlgvva a,
9:
2s SAMPLaFFRoNMMoG//zy.... = Togvar w)
10: UPDATTEENCCODER((cca) ØE. Breci PKL, Ineg: Pentropy: 7))
11:
UnDxEE/hion.NDDEECOO((e..aal 2s, PAi OD. Brec, SKL, Y, ()) while
12:
13: procedure UPONTEENCODER((pell 2x, PE, Brees PKL, Progg Fentropys 7)
14:
W < -s (Brec (reel()aat) + SKT. LKI. (at Wy s (Srec (rre((((z)) Sueg LKL (D(z.)
15:
16:
exp W <- 0.5 exp(2 Wr)
17:
C= COXPUTTRRESPONSIILIIIISSS
18:
Entropy = NORMAUIZED
19:
Lg t W exp Wsss Tentropy Entropye
20:
dg ég + 7/V on (Læ)
>. Adam update
21:
end procedure
22:
procedure UTODSTTPPRORAANNDDDODOOE
Bree: 9:
23:
W' -8. (Brec Liee (Zreal) + SKI. LKC. (2roal)) W -8 (7) Broe (eee)) AKI LK (D (D(2.))
24:
25:
LPD W 77- W.WW
26:
@p 0D + Vep (LPD) (LpD)
Adam update
27:
PA < PA n P
b Adam update
28:
procedure
29:
function Currl.cwAAAANNCEEHHeeeea a, K)
30:
"logvar <- Clipping the log-variance
C
b Eq. 11
31:
return "logvar
CC
32: end function
33: function SAMPLEF SAMPLEFROMMOGE ROMMOG(E Blogaars w)
34: it Samples mode index from Categorical(w)
_(] 35: Fead + exp 0.5 "logvar
(+)
36: return 2s
Samples from (( sta
(6) (0)
37:
38:
end function
39: function COMPUTERRSSONNIILLUISSE
40:
Compute the expected responsibilities for each mixture component
Eq. Eq.
41:
Construct the responsibility vector
42:
return
43:
function
44:
function NORMALLMEDETTRPPP(()
45:
Compute the cutropy of responsibility voctor Normalize the entropy
46:
Footnote
47:
return Entropyc
48: end function
11

---

Published in Transactions on Machine Learning Research (06/2025)
5 Experiments
In this section we investigate the impact of learning the prior in S-IntroVAE. Our testbed consists of a 2D density estimation benchmark alongside three image datasets of varying complexity. To crystallize the effect of prior learning we compare multiple key prior configurations with varying levels of flexibility. Namely, we considered the standard Gaussian, the fixed multi-modal MoG and the trainable multi-modal MoG priors while also ablate over learnable and uniform mixture contributions, when relevant. Concretely the prior configurations considered in experiments
Standard Gaussian: The commonly used isotropic Gaussian prior N(0, 7)-
Fixed MoG with uniform component contributions: A VampPrior with uniform contribution weights that is trained during the VAE stage, turned into MoG 2 and remained fixed during the adversarial training.
Trainable MoG with uniform component contributions: A VampPrior with uniform contri- bution weights that is trained during the VAE stage, turned into MoG and continue being trained throughout the adversarial training.
Fixed MoG with learnable component contributions: VampPrior with learnable compo- nent contributions weights that is trained during the VAE stage, turned into MoG and remained fixed during the adversarial training.
Trainable MoG with learnable component contributions: A VampPrior with learnable com- ponent contributions weights that is trained during the VAE stage, turned into a MoG and continue being trained throughout the adversarial training.
Importantly, any argument in favor of prior learning (i..,, trainable MoG) should be supported by perfor- mance improvements over both the standard Gaussian and the fixed MoG configurations.
5.1 2D density estimation
Generated Datas
Roal Data
EGuunsian
Mapirals
Latent Space
Donsity
Chactenbeadd
Ringe
(b)
Figure 4: (a) Real data (b) Qualitative results density cstimation, within each dataset provide, from left to right, the results under the standard Gaussian, fixed and trainable MoG with learnable contributions corresponding to the 2nd 5th and 6th columns in Table respectively.
For the Density estimation benchmark, adopt the same evaluation scheme originally used in S-Intro VAE (Daniel & Tamar 2021), namely, we use the gnELBO (grid-normalized ELBO) and the histogram-based KL. and JSD (Jensen Shannon divergence) divergences as measures of the inference, the forward and reverse generation capabilities, respectively.
To understand how modcling the prior as third player affects S-IntroVAE we compare throe discrcte prior settings, namely (i) standard Gaussian, (ii) fixed MoG and (iii) trainable MoG in decoder-cooperation,
2The VampPrior was turned into MoG by loading the aggregated posterior pseudoinputs into parameters MoG, ultimately breaking the prioreencoder pairing.
12

---

Published in Transactions on Machine Learning Research (06/2025)
termed as Intro-Prion (IP) (sce Fig. priors we experimented with both uniform and learnable contribution (T.C) of each mode while modeled the multi-modal prior using 64 components. More specifically, for the LC configuration, the contributions were learnable for both (ii) and (iii) during the VAE pre-training stage whereas during the adversarial training remained learnable only for (iii). The VampPrior was used during the VAE stage due to its benefits in latent space structuring over the MoG (Tomczak & Welling 2018). The latter was turned into a MoG (see Footnote 2) during the adversarial training to ensure prior decoder cooperation and to exploit the properties of its NE as given by Corollary I More specifically, we note that under the VampPrior, the prior is paired with the encoder establishing a prior-encoder cooperation. As analyzed in B.2.1 this cooperation leads to the prior and the decoder pulling the generated data distribution toward potentially incompatible objectives. When it comes to the regularization, the bela-adapling log-variance clipping was used for IP with K = 10 and [aj, by] set to the minimum and maximum log- oogvvariance in each latent dimension j as found during the VAE warm-up while the mode responsibilities were left unrogularized (i.c., Teutropy = 0). For all prior settings, we used 100 Monte Carlo samples to approximate the KL divergence between uni- and modti- Gaussian distributions.
I for conceptual visualization of the three settings). When utilizing MoG
Model -
VAF
S-IntroVA
S-hitmVAR
In line with Daniel & Tamar (2021), we identified the optimal hyperparametert (i.e., Bree, 3KL and Bueg) by performing an extensive grid-search while we used a = 2 and 7 y=
Prior Type +
SG
McG(64)
EXT Flag
N/A
N/A
IP Flag
N/A
NJA
/
In Table D we report the average (mean + standard error) performance across five seeds. As already re- ported by Daniel & Tamar (2221 the VAE formula- tion lags behind the S-Intro VAE across all metrics. Regarding prior learning in S-LntroVAE, the quanti- tative results suggest that in most cases, IP improves the generation performance compared to when using the SG prior or the fixed MoG. In particular, this is more evident when looking at the histogram-based KL metric. The observation above aligns with our intuition as according to Corollary 1 both the prior and the decoder players cooperate towards minimiz- ing the [L/paata(>) (=) [P2 term boosting the for- ward generation per performance. An exception to this trend is observed on the 8Gaussian dataset, where training under the standard Gaussian prior achieves the best generation results. Since the 8Gaussian dis- plays the most multi-modal structure (i.e., large ar- cas of low density) we attribute this deviation to the trade-off between stability and modeling multi-modal distribution with push-forward models as discussed in (Salmona et al. )022).
&Cecoiim
gaELDO
7.48
0,51 ixin
3,02 ino
4,8
0.25 Lsve
0,25 Ja:
KL. 1
BS4 eam
1.28 s4.
4.46 saas
2.95 sea
1.94 eea
2,21 1011
35D
17.41 2019
1.01 20.4
1.77 axt
1.70 2200
1:15 s0
1.08 21
gaLLDO
6:20 Jaon
0,41 1m:
6:41 200
6,04 1mw
5.81 ins
0.47
25pirale
KL. 4
10:18 enas
9.5 04.0
8.61 aaa
8.31 aea
0:45 moa
5.02 18
3SD U
4.54 p01
4.21 pm:
3,76 2014
3.58 2014 3.89 2018
3.61
Clackertward
gaDLBO 1
8:62 2ass
7:21 20.5
1.00 2x1a
7.81 20m9
7:67
kL. 4
20.79 ande
19.32 am
18.58 14::
18.72 exr
19,04 maa
17.7 103
JSD L
9.97 2018
8:57 an
8.65 2014
8.71 201
8.9 2m
8.46 s0xr
gabLBO J
6:37 eam
6.03 ua.s
6.75 mna
6.86 s.1a
6.4 sas
6.65 ..14
Rings
KL. 4
13.3 aac
9.50 1227
10,07 1433
9.77 oa.
11.31 mad
10.31 amm
JSD U
7.4 mM
4.05
20
2004
5.19 seas
4.33
Table 1: Quantitative performance on the four 2D datasets was evaluated. The L.C flag refers to the com- ponent contributions being Icarnable while the IP flag refers to training the prior (i.e., prior decoder cooper- ation scheme). Reported values are mean + standard error over five runs.
Additionally, when evaluating the qualitative performance as depicted in Fig. 4 formulation lends to give rise to better-separated clusters in the latent space, more intuitive support of the aggregated posterior, and fewer samples in between the modes.
we observe that the IP
5.2 Image generation
We investigate whether and to which extent prior learning improves the generation performance and the representation learned using the (F)-MNIST and CIFAR-10 datasets. We evaluate the generation quality using the FID metric for samples generated from sampling from the prior and the aggregated posterior denoted as FID(GEN) and FID(REC) respectivcly. To get better, more holistic view of how prior lcarning impacts the generation, we also report the recall and precision metries (Kynkäänniemi et al. 2019), denoted as (GEN) and precision respectively.
13

---

Published in Transactions on Machine Learning Research (06/2025)
The quality of the representations learned by the encoder was evaluated by fitting linear SVM, similar to Kviman et al. (2023). using 2K-SVM and 10K-SVM iterations as well as utilizing k-nearest neighbor classifier (k-NN) using 5-NN or 100-NN (Caron et al. 2021).
We use the default training hyperparameters and architectures as provided by Daniel & Tamar 2021 to train the S-IntroVAF, except that the first epochs were used as a VAE training warm-up. We conduct experiments using the same configuration used for the 2D data, while we employ the Tentropy regularization with value chosen from {0, 1. 10, 100} and report the quantitative results for the one that led to the optimal FID(GEN) for each prior setting. The prior modeled using 10 and 100 components and found that the latter is superior across all metries, whether using the fixed or trainable MoG configurations, which is an indication that using a sufficiently large number of components is essential. The results provided in Table suggest that replacing the standard Gaussian with a MoG prior (either fixed or trainable) can benefit both the quality of the generation and the learned representation, however, the benefit is less profound CIFAR-10 compared to the (F)-MNIST datasets. We attribute this behavior CIFAR-10 potentially being opposed to (F)-MNIST which more likely to
(close-to) uni-modal distribution (Salmona al. 2022 be multi-modal.
At this stage, it is natural to question whether the increased generation performance of the MoG configura- tions a byproduct of memorized samples in the mixture modes. In this regard, a high precision accompanied with low recall would be an indication of model memorizing specific training samples at the expense of dis- tribution coverage. The results shown Table 2 do not hint such sample memorization behavior, that is, the relative relationship between recall and precision is similar across all settings.
When comparing fixed (w/o IP) to trainable (w/ IP) MoG priors, observe trend where the IP achieves optimal generation perfor- mance two out of the three image bench- marks. When it comes to linear separabil- ity, the IP significantly improves over the fixed MoG for two out of the three benchmarks. In- tcrestingly, lcarning the prior significantly im- proves the classification performance under the k-NN model across all datasets. This suggests that prior learning in E-IntroVAB leads to a more defined class separation and more inter- pretable latent space, where similar samples are more effectively clustered together.
50
WNo F a/o
PP wo
No P PWLL
IP WALC
A qualitative inspection of the latent space (see Fig. 5) reveals that modeling the prior as mixture of MoG results in better-scparated clusters compared to a standard Gaussian. When comparing a fixed MoG to a train- able MoG, the improvement class scparation is less pronounced but still noticeable which aligns with the quantitative results shown in Table [ For the complete results and latent space visualization, we refer the readers to D.2 and D.4 respectively.
CRAPHE
Figure 5: Visualizing the first two latent dimensions of the latent space and the i-SNE 2D embeddings of the full latent space. The columns correspond to those in Tab. 2 Differ- ent colors correspond to diffcrent classes. The black dots refer to the means of the prior components and their size corresponds to their contribution weight.
Finally, it is worth noting how the entropy regularization behaves differently based on the training hyperparameters, dataset. complexity and prior learning configuration. In this gard, we observe that a higher T Tentropy nec- essary to achieve the optimal performance on CIFAR- 1- compared to the (F)-MNIST datasets under the IP
14

---

Published in Transactions on Machine Learning Research (06/2025)
Model
S-IntroVAE
S-IntroVAE
Prior Type-
SG
M1G()00)
LC Flag
N/A
X
X
IP Flag +
N/A
/
Yeutropy
0
10
10
10
Entr.
0
0.892
=0.002
0.882
#0.001
0.882 #0.002
0.853 +0,034
FID (GEN)
1.414 =0.025
1.322
=0.025
1.352
#0,052
1.32 #0.001
1.309 #0.027
FID (REC)
1.503 =0.081
1.342 =0.05
1,473
1.363 +0.075
1.385 #0.051
MNIST
Recall (GEN)T
0.565 20.003
(.562
20.005
0.553 2.0.00
0.556 10.003
0.557 1:000
Precision (GEN)
0.522 20 000
0.55
20 005
0.556 20.001
0.561 20.005
0.562 20.005
2K-SVM1
0.93 #0.001
0.961
#0.001
0.97 =0.004
0.962 +0.002
0.972 #0.002
10K-SVM1
0.93 #0.001
0,961
=0.001
0.97 =0.004
0,962 #0.002
0.972 +0.002
5-NN1
0.763 =0.003
0.916 =0.004
0.947 #0.011
0.92 =0.001
0.957
#0.004
100-NN1
0.87 10/00
(.934
20.002
0.953
20.007
0.935 1:0000
0.958
1.0.002
Tentropy
0
10
10
10
Entr.
0.931 #0.003
0.931 #0.001
0.944 #0.001
0.903 +0.005
FID (GEN) 4
3.326 =0
2.785 20.031
3.025
2.727
2.831
FID (REC)
3.76 097
2.994 10000
3.129
20.005
3.185
40.101
3.511 4.0074
FMNIST
Recall (GEN)T
0.314 =0.012
0.35 200000
0.336 20.007
0.346 100004
0.3-1 4:0008
Precision (GEN))
0.518 -00000
0.553 20.005
0.558 20.001
0.576 10.006
0.574 10.003
2K-SVM1
0.681 =0.001
0.731 #0.008
0.695 #0.007
0.712 #0.005
0,696 +0,008
10K-SVM1
0.731 20 006
0.78 20.0002
0.772 20.000
0.778 20.002
0.773 100000
5-NN1
0.425
=0 00N
0.683 20 000
0.693 20.005
0.678 10.006
0.707 40.005
100-NNI
0.606 =0.014
0.736 =6.005
0.729 20.005
0.731 +0.005
0.739 #0.004
Ventropy
100
100
10
Entr.
0
0.839 20.000
0.94 =0.002
0.929 20.003
0.511
FID (GEN) 4
4.424 =0.064
1.465
20.038
4.385
20.140
4.117 :0.051
4.591 1:0.255
FID (REC)
4.13 1:0066
1.205
10.091
4.084
10.006
1.141 1:00300
4.585 1:0.733
CIFAR-10
Recall (GEN)T
0.283 #0.003
0.281 =0.001
0,283
#0.008
0.282 +0.008
0.264 #0.012
Precision (GEN)
0.685 #0.004
0.676 =0.002
0.679
#0.004
0,677 +0.007
0.685 #0.006
2K-SVM+
0.245 =0 001
0.25 20 002
0.271
20.006
0.26 20.002
0.256 20,000
10K-SVMI
0.391
0.396 20.003
0.407
20.007
0.401 40.002
0.396 40.002
5-NN1
0.206 20.001
0.189 1:0
0.239 20.005
0.196 10.001
0.219 1:0000
100-NN1
0.308 =0.007
0.216 =0.008
0.32 #0.005
0.259 +0,003
0.273 #0.004
Table 2: Quantitative performance on the images datasets. The LC flag refers to the component contribu- tions being learnable while the IP flag refers to training the prior (i.e., prcorddecoder cooperation scheme). Reported values are mean + standard error over three runs. The Tentropy row corresponds to the regular- ization used to obtain the optimal FID(GEN) for each training configuration, where the Entr. row refers to the normalized entropy of the responsibilities where the closer to one its value the more uniformly the aggregated posterior is supported by the prior components.
configuration. Additionally, allowing for learnable contributions under the IP configuration tends to decrease the normalized entropy of the responsibilitics suggesting that contributions tend to vanish as soon as they no longer support the aggregated posterior which advocates for the importance of taking measures (e.g., using
15

---

Published in Transactions on Machine Learning Research (06/2025)
the rentropy) to utilize all the components when performing the discrimination (i.e., updating the encoder). For the full ablation on the Tentropy parameter we refer readers to D.3
6 Conclusions
In this study, we have proposed a prior-decoder cooperation scheme as a theorctically sound approach to prior learning in S-IntroVAE, marking the first successful integration of prior learning in Introspective VAEs. Our approach aims to combine two independent directions for improving VAžs: prior learning and the incorporation of adversarial objectives. To realize our proposed scheme, we identified several challenges, which we addressed with theoretically motivated regularization techniques, specifically (i) adaptive log- variance clipping and (ii) responsibility regularization. Our experimental results conducted on 2D and nigh-diiensional image settings demonstrate the effects of learning the prior in S-IntroVAE: These include a better-structured and more explainable latent space and, in most cases, improved generation performance. We firmly believe that our theoretical insights, coupled with the empirical results, pave the way towards a better understanding of Introspoctive VAEs and their connoction to their VAEs and GANs counterparts. Finally, owing to the unique nature of the problem where a multimodal distribution constitutes both the source and the target, we hope that analyses enjoy practical use in other areas that deal with problems of similar characteristics e.g., Idempotent Generative Networks (Shocher et al. 2023 or adversarially robust clustering Yang et al. 2020
Acknowledgements
We thank Emanuel Sanchez Aimar and Shashi Nagarajan for the discussions during the early stage of the study. This work was supported by the Wallenberg Artificial Intelligence, Autonomous Systems and Softwarc Program (WASP), funded by the Knut and Alice Wallenberg Foundation. The computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NATSS) at C3SE, partially funded by the Swedish Research Council through grant agreement no. 2022-06725.
References
Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdai. A contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems, 34:480-499 19480-993,-00 2021.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In European Conference on Computer Vision, pp. 456-473. Springer, 2022.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencodors. arXiv preprint arXiv:1509.c 00519, 2015.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 9650-9660, 2021.
Bo Chang, Alexandros Karatzoglou, Yuyan Wang, Can Xu, Ed H Chi, and Minmin Chen. Latent user intent modeling for sequential recommenders. In Companion Proceedings of the ACM Web Conference 2023, pp. 427 431, 2023.
Kushal Chauhan, Pradeep Shenoy, Manish Gupla, Devarajan Sridharan, el al. Robust outlier detection by bi-bii vae likelihoods. In Proceedings of the IEEE/CU OEE//CVF Conference on Computer Vision and Pattern Recognition, pp. 9881-9890, 2022.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. arXiv preprinl Xix::200 10600 2020.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. Advances in neural information processing systems. 31, 2018.
Marissa Connor, Gregory Canal, and Christopher Rozell. Variational autoencoder with learned latent struc- ture. In International Conference on Arlificial Intelligence and Statislics, pp. 2359 2367. PMLR, 2021.
Tal Daniel and Aviv Tamar. Soft Soft-introvae: Analyzing and improving the introspective variational autoen- coder. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4391 4400, 2021.
Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulku- maran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoen- coders. Xi preprint 1rXiu:1611.02646 1.06648, 2016.
loannis Gatopoulos and Jakub M Tomczak. Self-supervised variational auto-encoders. Entropy, (3( 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Couvvill, and Yoshua Bengio. Generative adversarial networks. Commumications of the ACM, 63(11): 139-144, 2020.
Prasoon Goyal, Zhiting Hu, Xiaodan Liang, Chenyu Wang, and Eric P Xing, Nonparametric variational auto- encoders for hicrarchical representation learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1094-5102 2017.
Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 166-174, 2017.
Trina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 6851, 2020.
Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advonces in Approzimate Bayesion Inference, NIPS, volume 1, 2016.
Huaibo Huang, Ran He, Zhenan Sun, Tieniu Tan, et al. Introvae: Introspective variational autoencoders for photographic image synthesis. Advances in neural information processing systems, 31, 2018
Xu Ji, Lena Jehale-Fazine, and Maksym Korablyov. Properties of minimizing entropy. arXin preprinl arXiv:2112.0311 2021.
Armand Joulin and Francis Bach. A convex relaxation for weakly supervised classifiers. arXiv preprint arXiv:1206.66 6413, 2012.
Dimitris Kalatzis, David Eklund, Georgios Arvanitidis, and Søren Hauberg. Variational autoencoders with riemannian brownian motion priors. arXiv preprint arXiv:2002.05227 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes arXiv preprint arXiv:1312.6114, 2013.
Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, and Patrick van der Smagt. Learning hierarchical priors in vaes. Advances in newral information processing systems, 32, 2019.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Oskar Kviman, Ricky Molén, Alexandra Hotti, Semih Kurt, Victor Elvira, and Jens Lagergren. Cooper- ation in the latent space: The benefits of adding mixture components in variational autoencoders. Tn International Conference on Machine Learning, pp. 18008 18022. PMLR, 2023.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advonces in neural information processing systems, 32, 2019.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In International conference on machine learning, pp- 1558-1566. PMLR, 2016.
Frantzeska Lavda, Magda Gregorová, and Alexandros Kalousis. Improving vae generations of multimodal data through data-dependent conditional priors. arXiv preprint orXis:1911.1088. 2019.
Felix Leeb, Guilia Lanzillotta, Yashas Annadani, Michel Besserve, Stefan Bauer, and Bernhard Schölkopf. Structure by architecture: Structured representations without regularization. ar preprint arXiv:2006.0771 077996 2020.
Shuyu Lin and Ronald Clark. Ladder: Latent data distribution modelling with generative prior. arXiv preprint arXit:2009.0008, 2020.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian Goodfellow. Adversarial autoencoders. In International Conference on Learning Representations, 2016. URL http:// wwtp:////w.aabaa/...../. http:://araav.orrr/bb//5.......
05644 05644
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference machine learning, pp. 3481-3490. PMLR, 2018.
Alon Oring. Autoencoder image interpolation by shaping the latent space. Master's thesis, Reichman University (Isracl), 2021.
Konpat Precchakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward meaningful and decodable representation. In Proceedings of the IRRE/CVP Conference Computer Vision and Pattern Recognition, pp. 10619 10619-10629, 2022.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Luis A Pérez Rey, Vlado Menkovski, and Jacobus W Portegies. Diffusion variational autoencoders. arXiv preprint XrX1t:1901.0891,, 2019.
Danilo Jimenez Rezende and Fabio Viola. Taming vaes. arXiv preprint arXiv:1810.00597 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1278 1286. PMLR, 2014.
Antoine Salmona, Valentin De Bortoli, Julie Delon, and Agnès Desolneux. Can fush-forward generative models fit. multimodal distributions? Advances in Neural Information Processing Systems, 35:10766- 10779, 2022.
Assaf Shocher, Amil Dravid, Yossi Gandelsman, Inbar Mosseri, Michael Rubinstein, and Alexei A Efros. Idempotent generative network. arXiv preprint arXiw:2311.01402, 2023.
Anika Shrivastava, Renu Rameshan, and Samar Agnihotri. Latent space characterization of autoencoder variants. arXiv preprint 1110:2442. 2024.
Casper Kaac Sonderby, Tapani Raiko, Lars Maaloe, Søren Kaae Sonderby, and Ole Winther. Ladder varia- tional autoencoders. Advances in neural information processing systems, 29, 2016.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Jakub Tomezak and Max Welling. Vae with vampprior. In International Conference on Artificial Intelli- gence and Stalistics, pp. 1214-1223. PMLR, 2018.
Jakub M Tomezak. Deep generative modeling. Springer, 2022.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in Newral Information Processing Systems, 33:19667-19679, 2020.
Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972))47 60, 2023.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming- Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. Xiv preprinl ar Xiv:22200.000 00796, 2022.
Xu Yang, Cheng Deng, Kun Wei, Junchi Yan, and Wei Liu. Adversarial learning for robust deep clustering. Advances in Neural Information Processing Systems, 33:9098 9108, 2020.
Zilong Yu, Yunyun Yang, Yonghin Zhu, Bixue Guo, and Chun Li. Cs-introvae: Cauchy-schwarz divergence- based introspective variational autoencoder. IEEE Transactions on Multimedia, 2023.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep generative models. In International Conference on Machine Learning, pp. 4091-4099. PMLR, 2017.
19

---

Published in Transactions on Machine Learning Research (06/2025)
A Preliminaries
The ELBO, given sample can be formulated as:
W(z
Eg aa [[gpa(a[=)] - KL(o(z|>)|l(())
Re Pa (ar|=] E 1og ((2(x))
p=
(=(2)
E i [log Pa(2 pd((x) g(2(x))
(14)
log
(=(z)
(z(2)
Es-a [log pa([a(())) + log log [(][x)]
logpa(>) [Llg(z log pa()),
with KL[-II-] denoting the Kullback-Leibler (KL) divergence.
B Nash Equilibrium in S-IntroVAE
In this section, provide the theorems based on which the prior decoder cooperation emerges viable option for lcarning the prior in S-nntroVAE. First, we revisit the derivation of the Nash Equilibrium (NE), under the fixed prior case (originally provided by Daniel & Tamar )) which we modify to account for samples outside the support of the real data distribution. The details and the motivation behind the aforementioned modification are provided in Section B.I
For simplicity, analysis is conducted in the discrete domain which is in practice sufficiently re- vealing we deal with finite data. From theoretical standpoint, we can rely on continuity arguments under the assumption of Leibniz's continuity.
B.1 S-IntroVAE under a fixed prior (DDnniel & Tamar 2021
The adversarial game defined by Daniel & Tamar (2021
1 exp(a W(x;
Lg(g,d) d) = =Epaa. [W(r; d)] pa
(15)
La(g,d) = =Epnata ([r;q, d)] + Byu [W(z:q.d)
where a 2 1, 7 2 and pa(x) E [p(e)(pa(<)>)) with p(z) fixed prior distribution. Note that although originally, standard Gaussian (SG) prior was used the derivation extends any prior distribution long as it is fixed. For notational brevity, will henceforth refer to the expectation over the real data distribution Epppaan []] simply as Epascall [], the same applies to the generated data distribution as well.
Lemma 1. Assuming that [pa(=)]=+1 S (z) for all such that P 2 0, the q² marimizing the Lg satisfies q" (d))(z))) = pa(z|z)
Remark 2.
The assumption used in Lemma 1 is modified version of the one used in (Daniel Tomar 2021) in order to account for samples outside of the support of the Pdota(x). Specifically we require the assumption [pa( < (x) to hold for all such that Páata (x) 2 instend Paata (x) > The ulility of this modification is revealed in the proof below.
Proof. Using the E.BBO reformulation provided in 14 we develop the Iggg,d objective as:
20

---

Published in Transactions on Machine Learning Research (06/2025)
1
T d) R Popaaa [W Pps
exp(a ((
R Pataa [loga(<> KTllg()a))lpu((te]]]
Eps
Pana [logp((x) [Klo@=/=))]]][((]]]]]]]
Epg [x) exp (iggllooKKi(((ll]lloddtatta)))))
Σ Panta (logpa(z) ( [pa(x)]a~1 exp(-a KIlg(2[=)]lnd<(]]))
- [Kop))/o)) [xa(z))=
Paata
G(g,d), I (x)ata(x) > 0}
Σ [pa(x))att -xpll=teeeppppp_____ ( 2
Ela,,) d), a (pdata 0)
(16)
The optimal qº for each x can be found as the maximizer of the Lg(g.d) d).
Given x such that Pdata (x) the optimal g² can be found as the maximizer of the function G(g, d). In that case, observe that q contributes to G(g, d) only via the KL term. Based on that, the saddle point can be found by analyzing the derivative of G(q, d) with respect to the KL.
0Gig. d)
(-1+ [pa(x)]<+1 Pdtaaa (x)
OKLLg())=)llp<((b>)) (a)
exp(-a (Llg(t)z))laa((=)))
(17)
For such that (aata()) 0 and [ (a)ta(ar) < 1 observe that the (KL 0 for e [0, oo) (KT, non negative), that is the G(q, monotonically decreases with respect (Ilg(el)))llex((Ie))
8G(g.))
For such that (z) 0 and [a(2))a= when ([l(() 0.
aGig.t 1 we observe that the [KL[otele)]|na[ee]]]]
0 only
Pdaaa (x)
Additionally 3KKT
8Glg.d
(a)ta( (x)) (-a-
[2a(z))=11
Palata (ar) exp(-a (LLa(s) s S..
Based on these two cases above, we conclude that (sl(((a()) = 0 is global maxima of Lg d) for x such that Pdata (x) > 0 and [pa(x))==+1 Pdata (x) < 1
For such that Pdata 0 the optimal 4- can be found as the maximizer of the function Q(g, d).
aQ d)
[KL[g(z|>) [p([ [pa())]ot1 exp(-a KLlg(ale)]l<<(212)))
(18)
We observe that [K]. O, given that (L(g(2\<))Ipa((Io)) E [0, co) we conclude g" (=|() such that (=())(()]]]p((([]))) global maxima of (1 d) for x such that Paata(x) 0. The result above
aQ(q.d)
21

---

Published in Transactions on Machine Learning Research (06/2025)
contradicts what has been argued in (Daniel & Tamar 2021 and is the motivation behind extending the assumption used in Lemma 1 to account for samples outside of the support of Pduta( (i.e. Pdata 2 0 instead of Pdeta (x) > used in Daniel & Tamar 2021)). Under the modified assumption, for x such that Pdata (2) = 0 we also have pa(z) = 0. In this case samples outside the support of the real data distribution not contribute to the lgg(gdd) d) objective and therefore do not influence the optimal q²"
Given that the KL is a proper divergence and under the assumption that [pa()))=-1 S Pdata (x) holds for all such that Pdtta (I) 2 conclude that q" (2(zx Pa(212)) is the global maxima of the (g d), that is:
Lg(g(d), Lg(a" (d). for all q.
(19)
Let us define d' as:
d' E arg min (AL[ppaaa(=))lppat + H[pp())]]
(20)
Assumption 3 (Modified DDaniel & Tamar 2021 ). For all such that Pdata (x) 2 0 have that pps (x)]at1 S (d)tal.).
Theorem (Daniel & Tamar 2021 ). Under the Assumption the pair of optimal q² Pa- Pa- (2(1z) and d
defined in 20 conslitules NE of the game 15
Proof. First, develop the La(g, d) as:
La(g. d) = E paata
[W d)] + 7 Bp, IW (z:
E Exaaa
[log (og))(() - ([]g(=]=)\|xx<<(]]])
+ 7 Epp
[log Pa(x)) (L]g(s))))lpxx<(l))))
log pa(x:)
Epanta
+ log Paata (z) (KLI((|=)]lpa((loo)))
Pdata (x)
(21)
+ y Epp MogPa(as
KL [[
Pppaaa log Pdata (x:)
KL[PPatta (z) llpa) - -([p(=))
Epencal [KL [2)
- 7 Ep [KL])
with H[-] denoting the Shannon entropy. Note that since [L]g(> 2 0 = KL[g" (=]x)]lpa<((=]) the
d= maximizing the La(g, d) can be found as the maximizer of La(g", d). Based on that we set = q (²(d) in and find the expression of that maximizes the objective La(q" (d) d) as:
La(g" (d), Epancs [log Petata (x)
KL[paata (I) [pa(x)] - 7 H.H.HHHPaaa [p(())
(22)
Epa KL [IIg - Ep. [KL [
the Epaaa loo (x)] is fixed given distribution (a)ta(x) while the KL[-11-] and H[| are non-negative, we can derive the maximizer d" according to (20). Based on that and according to Lemma
L,(g(d"), < La(a" for all g, La(g" d) S (d") for all d,
(23)
22

---

Published in Transactions on Machine Learning Research (06/2025)
and therefore conclude that the pair qr and d' such that:
q' (2(2() Pa (z|z),
(24)
min mig d' E arg
+y H]]p)())])
d
is a NE of the 15
We refer the readers to the original work by Daniel & Tamar [2021] for the proof that for any Pdata (x) there always exists y > 0 such that the assumption 3 holds for Pa- (z).
B.2 S-IntroVAE under a trainable prior
Let denote the set of possible parameterizations of the prior distributions. We now assume that the prior Pz(z) is learnable and henceforth is denoted as px(z) with E A while the generated distribution under that prior is (x)) (px(s))a(x)22) Consequently the adversarial game 15 is modified as:
Lg q, d) Paaat [W(x;), Ep2 L exp(a W 12,
(25)
La(A, q. Epaata [W(x; d)) + + Ep3 [
B.2.1 Prior-encoder cooperation
Here we conjecture the infeasibility of learning the prior in collaboration with the encoder while maintaining the same NE of the S-IntroVAE. Intuitively, this formulation seeks to find the optimal prior as the balance between maximizing the real ELBO and minimizing the fake exp(ELBO).
Similarly, the definition in 20 is modified as:
d'(A) E arg min (x)))) + HIp2(=)))
(26)
d
to account for the parameterized prior. Let (x) a discrete distribution of sample size N and e's non-negative real numbers realizing the unnormalized probability masses of (x) distribution such that the likelihood of sample dy. is calculated as:
px(x))
Ek
(27)
N
==1
Let us define the entropy (x)) and the a-order regularization
A (x)) as:
N
Hp) Σ (a(.)) log (p²().)),
(28a)
N
Alpa(=)) = Σ px())) (x:))
(28b)
3Te hyperparameter the used
23

---

Published in Transactions on Machine Learning Research (06/2025)
Lemma 2. Minimizing (c) with respect to moss e requires positive(negative) update if log en is larger(smaller) than e]
According to the definitions Eqs. 28a probability masses e's as:
Proof.
and 27
the entropy can be developed with respect to the
Ci
Ci
Hi[e]
log
(29)
N
Cj
Based on
29
the derivative of Hi[e] with respect to the mass can be computed as:
N
0H
a
Ej
[c]
log
Dek
DEk
N
N
i=1
ej
Σ
Cjj
1=1
1=1
N
Ck
log
Σ
C
log
N
N
Σ Ej
Σ
Ej
Σ
Σ Cj
1=1
j=1
1=1
N
Σ
Cj
ek
Σ Ej
Ck
1=1
==1
log
log
N
N
N
N
N
Σ
e₂)²
Σ
Ej
Σ
e2)²
Σ e;
ej
ej)²
1=1
Σ Cj
N
ΣN2
N
Σ E
N
loog
Ci
log
171
(30)
N
N
Σ
e2)
Ej
e2)
e,
ej
e₁)
1=1
1=1
1=1
N
Σ
log
ck
log
Σ
N
e2
Ei
)2
ek
1=1
1=1
Ei
log
log
N Σ 1=1
1=1
,2
)2
N
Ei
ei
log
log es
log ek
N
N
ck
N
N e
Σ
1=1
Ej
1=1
The update towards minimizing the entropy regularization reads as eh = (ex - 17 Bes (e)) According to 30p. the update - deg [e] of mass eg positive if log e larger than Ellog e] and vice versa.
88
0M
The proof was originally provided Ji et
2021
24

---

Published in Transactions on Machine Learning Research (06/2025)
Lemma 3. Minimizing (e)] with respect to mass e requires negative(posittee) update if ej is larger(smaller) than Bleg)-
Proof. According to the definitions Eqs. respect to the probability masses e's as:
28b and 27
the a-order regularization can be developed with
a
(a+1)
Ale]
Σ
Ei
(31)
N
N
N
Σ
Ej
Based on 31
the derivative of Alel with respect to the mass can be computed as:
(a+1)
(a+1)
(a+1)
0
[e]
i
ck
Dek
Dek
N
DEN
N
N
Σ
ej
Σ
ej
(a+1)..6) og
ej) (a+))
(a + 1). e
(atl)
e3)
(a 1) e,
(a+1)
ez)a
N
jml
N
Σ
ej (((+11)
Σ e (()21)))))
22(++1)
1=l
F=l
==1
c
e
(a+1)
(a+1)
(a+
ck
N Σ
N
(a+1)
E
(a+2)
Σ (a+1)
(32)
Similarly to the entropy minimization case, the update towards minimizzing the a-order regularization reads 86
as ea = (es - 17 des [e)) According to 332 the update -n Deg
0A
[e] of mass es is negative if eg is larger
than E[e"] and vice versa.
d' (A) E argmin min [KI. (x) (2) +y Hx))
26 revisited)
Lemma 4. For q" = (2)x), the d' mazimizing the La(d, q², d) satisfies 26
Proof. Similar to Thcorem we develop the La(d, q, d) as:
La(),q,d) Epanca [log Pdata (x)]
KL KL[pdata (I)) Ipp (x)] - (x)]
(33)
Epinta (KL((())))ll) - Ey3 [KL[(]e))))]]]] |
with [[-|] denoting the Shannon entropy. Note that since [L (2) (2|=)) d' maximizing the La(d, 9, d) can be found as the maximizer of La( g", d). Based on that we set q q" q²(,,d) in 33 and find the d' that maximizes the objective TddA,, q² (A, d), d) as:
(=|=) the
25

---

Published in Transactions on Machine Learning Research (06/2025)
La(A, q² d), paa [og Pdata (x)
KL Paataa (z)]lp - (-)
(34)
E Pasnn
-y Ep, [e
Based on 34
can derive the maximizer d' according to 26
Let us now define:
(d) E arg min KL Paata (x))) + Alpd(a)]
(35)
Lemma 5. Assuming that (x) < Paatal()) for all x such that Paata (2) 2 0, for q- = =2(()))), the X
mazimizing the Lg(A, q², satisfies 33
Proof. Given the trainable prior px()) the Lg(d, q, becomes:
Lg q, d) = Σ Pdata (x) (zog - KL(((s))))lo)(()))))
(36)
1 [v) exp(-a KLIg(E
Let ('(x/x) (2)22) the objective Lg(d, dd) reads as:
(,(,"""d) Σ Pdata (x) (logp2(x) KLlg
1
[p)(x)]aat aKg_:/tLLeepp(LLLaa(((aaaaaa a
Σ Pdlaaa (r) log (x) [v)
Σ (d) (log pd (z) log Pdata (z) + log Paata (x)) a
-b((x))
Σ Paata log (x(
(37)
Pdata (r)
Σ Pdata (x) log Pdgta (x)
Σ ())()))
Σ Pdiata (xr) log Pdaaa (x)
p (x)
1 Exd(r) -v(x))"
Pdata (x: log Priata (x)
KL -KKT/paaux)) (1)
Epasca [log Panta (x) (a)
Based on 37
we observe that the Epasta [log Paata fixed given Paata while A[-] is non-negative, therefore
derive the maximizer X according to 35
26

---

Published in Transactions on Machine Learning Research (06/2025)
d' (A) E arg min (tTT(paaa(()))lp25 +o Hl)p())))
26
revisited)
d
Lemmas and suggest that minimizing the entropy and the a-order push towards the Dirac and uniform distribution respectively. Based on that and the minimization objectives of the and À players, we formulate conjecture on the incompatibility of prior encoder cooperation.
Conjecture 1. When training the prior player in cooperation with the encoder player (t.e. to mazimize the same LA d.d) objective), there does not erist X such that the triplet X*, q" satisfiying (x(2()) = pa: (z/2) and d' as defined in 26 constitutes a NE of the game 250, under the assumption that pa- (z, + Pa (1) pd (z).
Remark 3. The Conjecture suggests that the prior encoder cooperation scheme not. a variable option for prior learning in S-Intro VA in the sense that it does not share the same NE with its fixed prior counterport.
B.2.2 Prior-decoder cooperation
Here, we consider the same game defined in (25) but under a prior-decoder decoder cooperation scheme where both the prior À and decoder d players maximize the same objective (,g 9, d). First, let us extend Lemma for the trainable prior case.
Lemma 6. Assuming that (p)(x)) S Pdata(() (x) for all æ such that Pdata(x) (x) 2 O, the q" marimizing the q,d) salisfies q" (=)o) (=(1)))
Proof. We develop the (A q, d) objective as:
[W(x; A, d)] Ep, [_ exp(a W(x d))
(A q.d) d) = Eaasa
E Paaaa [logp) -[[(<]]))l]]]]]]]]]]]]]]]
a E3 [exp(logla (igeeeollell(
Eppar
(1ogp)(=) -L(())))((((]+])])(]+])))))
-_.B, [Ir](r)]* exp (iggtblld)(((
Σ (a)ata()) (log ([lي() 1 [x(()) (xpppaaaxxppaa exp(-a K[]g(e[=))lAλ<(<=)]]
(38)
We follow the same reasoning used in Lemma 1 and conclude that under the assumption that (x)) < Pdata (z) holds for all x such that Paata (x) 2 0 g (2|æ) pa(2[=)) is the global maxima of the [g d), that
is:
L,(A,,(A,d)) (2, for all
(39)
Let us define X and d² as:
(A*,d") E arg min (x)) + H[p)(x))
(40)
A,
Now we also modify the Assumption B as:
Assumption 4. For all such that Paata(at (x:) 2 have that 1n (x))a+1 Pdaaaa
27

---

Published in Transactions on Machine Learning Research (06/2025)
Corollary 2. Under the Assumption when training the prior player X in cooperation with the decoder player d then the triplet q' = p2- (z|x), X and d² as defined in 440 constitules NE of the game 25
Proof. Similar to Theorem
we develop the La(A, d) as:
La(A, d)
Epaat [log Pdata (x))
KL[paata (x) [[pa(() - H[pp(=)]
(41)
-ppanaa [TL [
with [-]-] denoting the Shannon entropy. Note that since KI[g(e\=))l 2 o (Llg"el)))
the (X*, (X*,d") maximizing the La(A, q, d) can be found as the maximizer of La(d, q,, d). Based on that we set
q o" (2, d) in 41 and find the (X* ,d") that maximizes the objective La(A, o" (A,d), , d) as:
La(d, (A,
paann [log Pdata (x)]
KL [Paaaa (x))lp((x)) r H[pa(=)]
(42)
Epare [KLaa pa(z|=)]] - (eL
We can now derive the maximizer (X* d according to 44 Based on that and according to Lemma
Lg(,, ,d),, S (A, for all (A,d), Lx(1 (X","""""'")) ,d"), for all and d,
(43)
and therefore we conclude that the triplet X", qº and d' such that:
g- (=(x|x) p2- (2(x),
(X*, ) arg min (xTl)()aa(()))l)((())) + H[pa(=)))- A,
(44)
is a NE of the 25
As the proof of the existence of the does not assume the nature of the prior, the proof provided by Daniel & Tamar 2021)) | can be trivially extended for our case of p to show that there exists y such that the pz with (X*, d²) as defined
40 satisfies the Assumption
B.3 Optimal ELBO in the assumption-free setting
In the previous section, the NE of the S-IntroVAE under the prior decoder cooperation scheme 25 was analyzed under the Assumptions satisfied, particularly in the early stages of training. For instance, for the generator/decode to generate samples of very low quality (i.e. outside of the support of real data distribution) or to experience mode-collapse (i.e. generating some realistic samples at disproportionately higher frequency compared to the real data distribution). Evidently, both these cases might lead to violations of said assumption.
In practice, however, such an assumption might not always be
common adversarial training
Analyzing the behavior of the encoder in the assumption setting provides insights into the train- ing dynamics of S-IntroVAE, enabling a better understanding of the method and its relationship traditional VAEs. Furthermore conducting the analysis with respect the ELBO W(z; X. d) offers a practical tool since the E.BBO is comprised of the reconstruction and the KL divergence losses opposed
28

---

Published in Transactions on Machine Learning Research (06/2025)
to the (I]])) term (used in Lemma which is intractable.
Let X = ( E (x) (2) > 0} we define the ELBO W(;;A, d) as:
-00
{ E X
Pdata (z) 0}
1 log
Pdata (x)
I E { e X Pdata (x) > 0 (2) a+ > Praaa(
(45)
(z(
log p2 (2),
{ e X Pdata (z) 0n (x) (
Proposition 2. Given a fized generated data distribution (x) the q" maximizing (, d. 9) in 225 is such that the ELBO ((x; x.q". dd) satisfies 45
Proof. Similarly to Lemma develop Lg q, d) as:
Lg q, d)
exp(-a KLlg(tl)))od((]aa]]] Σ Pdata (x) (logpà(x) - 1z(()) exp(-a
Σ Pdata (I)
Pautu(a)
= E((,,, x (x) > 0}
Σ (-1 exp(--
exp(-a KLLg(sl=))lp
E d), a (pdata(x) = 0}
(46)
Again, we can find the q¹ maximizing (A, q, d) by analyzing the derivatives of the functions G(A, q, d) and Q,), d). In particular, we identify four cases.
x E fo E X (x)ta(x) > 0 (=)]= > Paaua()))
In this case, the q¹ can be found as:
aG q, d)
0
]KL[)(=|=)II (=|=)
Pdata (x)) (1+ (x)
exp(-- [L]g((//))l>> = +>
(d)ta(x)
exp(-a KL(g(2 = Pdata (z)
[pa(z)|at
Pdata (I)
KI[g(al=)1lAaate)) log
(47)
(x))
log () - KLlg(sl=)1I>(<[<]] log (
+ log (x
(x)
W(z:Agg 1 log (x)
1 loglp> +
+
W(z;g,d) log Paita(
1
pd
Note that aG(A.g.)
[KLL Pdata Pdata(x)) (x) (-a (e)(2))=t exp(-a KL (v) 50 there-
fore the g" such that W(x; x, ,0,"" d) = 1 log (alaa(a)) is the maximizer of L (A, q, d) for (x x
(x)
(a)ta( 0n (p)( (x))
29

---

Published in Transactions on Machine Learning Research (06/2025)
x {x X (x) > on [p(()))(+1 I Pdata
In this case, the maximizer of L, was found in Lemma 6 as the q" such that
= O. Substracting (z) to both sides and using 14 we get that, the q¹ such that W(r;,, = (o) the maximizer of L q, q, for x E (x E x Pdata (x) on (=))+t)++ S (d))
x (x E Pduta (x)= =
In this case, the maximizer of Lg(A.q.d) was found in Lemma as the q" such that KL(9" (=/=)|lp(=(z31
se. Substracting log pxx) to both sides, using [4] and given that log p2 (a:) I O we get that the q" such that W d) -00 is the maximizer of (A q, d) for E {x X Pdasa (2) = 0}.
E (x X Pdasa (x) (2) = 03=0 =
Note that the (P) 0 (x)) 0} set refers to samples z outside of the support of both real and generated data distributions which are of no practical relevance. In practice, the encoder maximizes the Lg over the expectation of empirical real and generated data distributions, motivating the definition of X as the union of their supports.
Interestingly, the ELBO W(;; X, o", d) at the optimal q² is continuous function with respect to Pdata (I). Additionally, it is revealed that the higher the sample-wise likelihood mismatch between the real Pdata (x) and generated (x) data distribution, the lower (more negative) the ELBO W W(r; g" is. The aforementioned behavior aligns with our intuition as the encoder S-IntroVAE acts as a discriminator.
On the other hand, given fixed (z), it can be trivially shown that the encoder of regularly trained VAEs converges to true posterior which is equivalent to WvAE A. d) log p² Naturally, these two observations relate the behavior of the encoders of VAEs and S-IntroVAEs where the latter behaves similarly to the former only if (2) is sufficiently enclosed by the Pr)t Given a Pdata (x), the enclosed term refers to the generated data distribution (z) for which the Assumption holds.
C Implementation
In this section, we provided the details behind some implementation choices.
C.1 Adaptive variance soft-clipping
The Chang et al. 2023 Chua et al. 2018 works realize log variance soft-clipping as:
fe (logvar)
logvar softplus(logvar-) + softplus(a logvar)
logvar log(1 + exp( (logvar b)))
(48)
+ log(1 + exp(B (a logvar))).
where fallogvrr) the soft-clipped output, [a, b) is the clipping interval and B positive hyperparamcter
controlling the steepness of softplus function. In these works a pre-specified [a, bj range was used and
We used this notation distinguish between the ELRO of the VAL. which we still refer that simply as W.
30

---

Published in Transactions on Machine Learning Research (06/2025)
24l
Figure The behavior of soft-clipping depends on the clipping range. Note that when using identical I's (c.g. == 1) the clipping behavior changes depending on the range (rows). On the other hand, when formulating the 3 as function of the range and the K hyperparameter the behavior remains consistent. Increasing the K (columns) leads to retaining bigger portion of the original clipping range.
naturally finding the optimal 3 hyperparameter for softplus is subject to proper fino-tuning. In practice, the default option of 3 1 was used in both studies.
In our case, different clipping intervals are applied to each latent dimension, that is [aj, by] for each j latent dimension. The [ay. interval was determined based on the minimum and maximum variance the prior's modes in each latent dimension as emerged during the VAE pre-training stage. Based on that, identifying the optimal By's through manual fine-tuning not a feasible option. Towards overcoming this challenge we model the By's as:
K
Bg bj aj
(49)
with K being controllable hyperparameter. Based on these, derive the K such that:
fa(b,) - fclag) p. by - aj
(50)
with p E (0, 1). Intuitively the (50) suggests that the initial range should be proportionally maintained post
the soft-clipping. The maintained proportion controlled by p. Developing (5)) based on the soft-clipping function defined in (88) we get:
31

---

Published in Transactions on Machine Learning Research (06/2025)
folb,) fc(az) 2 p. (b3 - az)
by
log(1 + exp(B3 (bi by]))
loog1 + exp(B3 (ay - bj]))
Bj
3j
1
a; +
log + exp( (aj bU))) log(1 + exp(3y (a, - a,))) 2 p (b, - a,)
Bj
3j
1
bi
(2) +
-oog(1 + cxp(8; (ay - -by))) - aj + 8j logg(1 + cxp(3; (1133-(aa) (a, - bR])) Bj log(()
Bj
B5
2 (b: - -aj)
2
2 -log11 + exp(3; (a, b;)))
(by - az)
log(2)
3j
2 p (by - aj)
B (1- p)- az)) 2 log(4) 2 log(1 + exp(3) (a; bj))).
(51)
We can derive K using the formulation defined
49 as:
(1 p) K 2 log( ()) 21gg(1 + exp(-K)).
(52)
Note that the K only depends on P and therefore can be tuned for all latent dimensions simultaneously irrespectively of the soft-clipping range [a..by] (see Fig. 6). In our study, we used a of 0.85 and found that satisfies the condition (5 In other words, having an adapting 83 by-a, guarantees that at least 85% of the initial range is maintained, post-clipping, in all latent dimensions. Alternatively, our B-adapting formulation can be interpreted as mechanism where the soft-clipping function maintains the same average rate of change in all latent dimensions, as suggested by (50). Finally, the adaptive clipping function fe becomes:
10
fc felogvar,, = logvar,
llog(1 + exp(3y- (logvar, - b;)))
3
(53)
3 log(1 + exp(B3 (a, - logvar,))):
C.2 Losses in S-IntroVAE with trainable prior
Let Freal a real sample and z ~ Px().. The encoder, the decoder, and the prior players minimize the Lg, Lp and Lp losses respectively which write as:
LE(ereal, 2) Brec Lree (Treal) + SKL LKL (Freat) 1 exp(-a (Brec Lree (D(xx)) + Boeg LKL (DLx())))) a
LDp 2x) = Broe Trree (Frent) + I + (Np Broe Turec (re((s)()((2))) 3KI. (D(x)))
Lp(Ireal, Bree Lanattt + SKL. UKI((rraa) + (Brec Lnnn( + 8KI. LIKT D(xx)))
(54)
When computing this particular KT. we only propagate the gradient for prior as while applying the ag operator prior as target.
32

---

Published in Transactions on Machine Learning Research (06/2025)
where (2x) is the fake sample generated from decoding the latent 2 while Liec and LKL the reconstruc- tion and the KT, losses respectively. Both LE and I.D are identical to the original S-IntroVAR (Daniel & Tamar 2021 with % a lyperparameter also set to le Note that the crossed-out terms do not affect the optimization, as they are constant with respect to the network being updated (c.g., the reconstruction losses are constant with respect to the prior when minimizing the Lp).
C.3 Responsibilities regularization
In this subsection, we provide the thcoretical motivation behind the responsibilities regularization which we utilize to discourage the formation of inactive prior modes. The notion of inactivity describes a prior mode that contributes negligibly in supporting the aggregated posterior compared to other more dominant modes. Sampling from inactive prior modes leads to unconstrained generation, which may negatively impact gencration performance. To this end, analyzing the minimization behavior of the (Frea)))) terms in (54) is key to avoiding and/or eliminating the inactive prior modes these are the terms that induce fitness between the real aggregated posterior and the prior.
be the posterior distribution of the a sample Ta. an M-modal prior distribution (x(=) Σ wi NN(zuu and uni-modal prior distribution p;(z) = -N(zlus, 13 corresponding to the
Let ((11,,)) (x]lus 1)
M
1=1
ith mode of (z() distribution.
According to the notation defined above, the (xL(F,) approximates the KL divergence between the uni- modal posterior g(=[x,,) and the multi-modal prior (2(2) (i.e., (Tll(()).))lxxx())) as:
T
(<>/(0s) (xL( px(zt)
{Llg()l=))lpxx<3)]== =
Σ
log
(55)
T
using 7' MC samples with = ~ N(2lu,, o2). Similarly define the Like (xx) as the approximation of the KL divergence between the uni-modal posterior ((((x,) and the ith prior component pi(z) (i.e., (L]x())==))lxx<()]])
as:
Σ log (<(((x,) (x.), p((zf)
[L) ~ T
(56)
For simplicity, we now assume that T' and drop the index for notational brevity, that we refer to the = simply as zg.
C.3.1 Responsibilities computation encoder update
First, let us analyze the minimization behavior from the encoder's perspective. For single MC sample Zel the (xL( can be computed as:
[xI(xx) = log (0g4(22(2.)) log (x())
(57)
loggN 021) llog Σ wi ((zzsuu,,
Based that, now compute the derivative of T.L. (x.) above with respect to 2, as:
33

---

Published in Transactions on Machine Learning Research (06/2025)
(LK)(
Ma T.T
Me 2,
wi NNazll 03.)
111 -2
Oz.
0
M
wi
(2, of)
M
2
Pi 28
(58)
0
28
Mi 28
0
c
o?
with of w.-N(zalm.ss))
denoting the responsibility of mode to 2, of the sample Ts.
Σ (F
Similarly we can calculate the derivative of Lig, (2.)) with respect to 2n as:
aL (1..)
Ms - 2s
11
2x
(59)
O2,
o²
Based on Eqs.
and 59 we conclude that:
M (xa))))))
cf
(60)
Oz.
Oz.
The decompasition provided above reveals the effect that responsibilities of each prior component have when fitting uni-modal posterior into multi-modal prior distributions. More specifically, it is shown that minimizes the LKL by seeking the prior modes according to the responsibilities of. Motivated by this, we define the expected responsibility of mode i to the real aggregated posterior as:
w; W(2lpa,0f1)
C₁ (x Bxwnoo(())
(61)
M 1- un (((22
C.3.2 Inactive modes and vanishing gradients prior update
When computing the derivative of the (x)( concerning the contribution w; we will need to take into account that sum of all contributions has to be 1. To case the computation we can model wy
where 6 non negatiee real number realizing the unormalized probability mass of the th component, and compute the derivative with respect the normalized energy e₁- Based on that
34

---

Published in Transactions on Machine Learning Research (06/2025)
M Σ
i log i=1
(LKI(
de:
de:
(1 - wi) (V(2ll14,03)
M
Wi N(22alu,,
Σ 69
M
wi (((llm),01
C1
(62)
1
1
(N(2slut,
M
M
Σ e
N(zzluu o/1)
2
M
61
M
Σ
((zs)11,of1)
1=1
wi ((zsllu, (zalM);0011)
M Σ e; ((zslp
The result above aligns with intuition it suggests that given latent 2, the energy e, corresponding to the unnormalized contribution of ith component increases if it is more likely to have been sampled from that mode compared to the MoG prior, and vice versa.
Similarly, we compute the derivatives of TRI. (xx) with respect with respect to 14 and as corresponding the mean and the standard deviation of the tth prior component respectively. In this case the gradient steps write as:
(LKL(Fs)
24 #i -G
and
Opci
0,
(63)
(LKL(W) (z. u1) o²
Oo,
The derivatives above reveal the behavior of the individual prior components in the presence of inactive modes. In particular, an inactive mode i manifests as low ç; responsibility (i.e., o close to zero for all real sample x.), due to insufficiently supporting the real aggregated posterior relative to other, more dominant modes. Consequently, a vanishing gradient issue arises, where the mean and the standard deviation of the inactive mode i are not updated (towards supporting the posterior) as indicated by 63 On the other hand, the unnormalized contributions of the inactive modes tend to vanish in favor of other more dominant modes 62 suggests. Based these observations, it is clear that in the presence of inactive modes, allowing for learnable contributions enables the prior player to climinate inactive modes. Conversely, not allowing learnable contributions leaves the prior with inactive modes that cannot adapt to the aggregated posterior, due to their low responsibility and consequently vanished gradients rendering the model prone to unconstrained generation.
C.4 Exploding variance prior update
In this subsection, we provide the theoretical motivation behind the log-variance clipping that was used stabilize training. We now assume a posterior 2, such that:
35

---

Published in Transactions on Machine Learning Research (06/2025)
1 H4l = t..... with 1> 1.
(64)
The formulation above suggests that the sample 2, is highly unlikely under the Gaussian distribution defined by the jth prior component, with the parameter controlling the degree of unlikeliness. Under the assumption of (64) the magnitude of the update rules derived in 63 write as:
(xs) Oμ4
t
c
and
0
(65)
(LKI(Z)))
t o,
o
t --1
t
C
0o;
0,
0₂
0i
The derivation above suggests that the gradient magnitudes of the μ4 and 0; parameters scale linearly and quadratically with f, respectively. Based on this, it is evident that the standard deviation, and therefore the variance, of the prior more sensitive to explosions in the presence highly unlikely posterior samples 2ar compared to the mean. Note that since the contributions of are computed relative all prior modes, is possible for 2g be unlikely under the ith prior component while of remains non-negligible. Finally given that (64) holds, sign( (L.) = -1 suggesting that the explosion of variance in the presence unlikely de.
posterior samples results in increasing the variance towards minimizing the KL divergence.
Additional Details and Results
D.1 Baseline reproduction for 2D datasets
Due to the inherent randomness involved in evaluating the generation quality, the grid-search-based hyper- parameter tuning and the computation of KL-divergence in Monte-Carle fashion, in table we compare the baseline performance across five key settings. Namely, the baseline as (i) reported in (Daniel & Tamar 2021) (ii) reproduced by us using the official code-base (Daniel & Tamar 2021 reproduced by our code-base computing KL both (iii) in closed-form (c) and (iv) in Monte-Carlo (s) manner and finally (v) replicated by our full pipeline of hyperparameter tuning which can result in selecting different optimal hyperparameter for each dataset (compared to those provided by Daniel & Tamar 2021 ). Although computing the KI. divergence in a Monte-Carlo fashion unnecessary for the uni-modal prior (baseline) is important to verify that both closed and Monte-Carlo-basse KL computation lead to comparable performance.
2D Dataset
Bree
Sneg
SGaussian
0.2 (0.2)
0.3 (0.3)
0.9 (0.9)
2Spirals
0.2 (0.2)
0.05 (0.5)
0.2 (1)
Checkerboard 0.05 (0.2)
0.2 (0.1)
0.8 (0.2)
Rings
0.2 (0.2)
0.2 (0.2)
0.6 (1)
Table 3: Optimal hyperparameter under the standard Gaussian prior for each dataset as found using grid- search and as reported by Daniel & Tamar (2021) (in parenthesis).
36

---

Published in Transactions on Machine Learning Research (06/2025)
S-IntroVAE (SG)
Source
reported
official (reproduced)
ours (reproduced)
ours (replicated)
KL Calculation
s
Mode
8Gaussian
gnELBO
1.25 20.35
0.62 20.13
0.52 =0.09
0.51
10.15
0.51 10.15
KL +
1.25 20.11
1.33
#0.52
1.36
=0.41
1.23
40.11
1.23
40.11
JSD +
0.96 20.15
1.16
10.15
1.16
20.11
1.01
20.18
1.01
40.18
2Spirals
gnELBO
+ 5.21
20.04
5.47
= 0.05
5.47
+ 0.06
5.47
#0.14
6.41
#0.61
KL
8.13
+0.3
10.21
40.39
10.66
#0.19
10.26
#0.39
9.5
#1.23
JSD 1
3.37
#0.04
4.03
#0.1
1.11
=0.16
4.08
#0.06
1.21 #0.5
Checkerboord
gnELBO
4.47
10.29
6.28
6.22 =0.80
6.33
#:075
7.21
4:0.12
KL +
20.27 20:21
19.72
+0.23
19.99
#0.28
19.94
20.37
19.62
+0.57
JSD 4
9.06 20.15
9.04
=0.19
9.34
+0.19
9.19
#0.17
8.87
#0.15
gnELBO
6.3 #0.08
5.81
=0.06
5.8
#0.05
5.85
#0.13
6.03
4:0.12
Rings
KL +
9.18 10.33
10.67 =0.5
10.75
1:0.9
10.89
#0.45
9.99 4:0.99
JSD +
4.13 40.09
4.37
20.12
4.35
20.12
4.2 40.11
4.05 10.15
Table 4: Baseline performance across five key settings. For each setting, report the performance (mean + standard deviation) over five runs. When reporting the performance for columns 2 to 4 we used the optimal hyperparameters as provided (reproduced) by Daniel & Tamar 2021] (also found in parenthesis in Table 3 whereas, for the 5th column, we used the optimal hyperparameters found by our grid-search implementation (replicated). Note that for the 8Gaussian dataset, we found the same optimal hyperparameters leading to identical performance between the 4th and the 5th columns. The 'c' and 's' refer closed-form and sample- based computation of KL divergence.
D.2 MoG ablation on the image experiments
In Table we provide the full ablation on the image generation benchmark suggesting that utilizing a sufficient number of prior modes is crucial for achieving optimal generation and representation learning performance.
37

---

Published in Transactions on Machine Learning Research (06/2025)
Model -
S-IntroVAE
S-IntroVAE
Prior Type +
SG
MoG[10)
MoG (100)
LC Flug- +
N/A
X
X
/
X
X
IP Flag
N/A
X
/
X
/
X
V
x
Festagg
10
0
100
10
10
10
Entr.
0.966 40.0
0.952 snmi
0.948 s0.30
0.988 10.000
0.892 me
0.852 60.00
0.882 64.003
0.853 60.004
FID (GEN)
1-114
1:38
1.356
1.127
1.365
1.322
1.352
1.32
1.309
FID (REC)
55503 20.00
14488
1.51 25000
1.629 20.373
1.472 2b:00s
1.342 20.00
1.473 2011
1.363 SEArS
1.385 20.003
MNIST
Rocall (GEN)
0.565 2o.
0.569 20.000
0.545
a0 000
1.554 20:307
0.352 23:093
0.562
2:0 000
0.553 de 000
0.556 =f 013
0.587 20:101
Precision (GEN)
0.522 20.004
0.533 200000
0.563
20.003
0.54 2um
0.553
#2:000
0.55 s0.00
0.356 20
0.561 =000
0. D.5622 =0.000
2K-SVM 1
0.93 200000
0. 0.956 20
0.972
20.001
0.967 20:00
0.950
20/03
0.961
2: 001
0.97 10
0.962 at
0.972
20.00
10K-SVM 1
0.93 6:0.00
0.957 10.00
0.972
10.001
0.957 20,32
0.958
50.003
0.961
10 001
0.97 10.000
0.962 64.003
0.972
=0.002
5-NN f
0.763
0
0,913
0.876 20007
0.812
0.916
0.917
0.92
0.957
100-NN
0.87 200
0.897 m0.0
0.9:9 240
0. 0.907 20.00s
0.885
20.000
0.934
200
0.953 20
0.933
20.003
0.958
-0.000
Testagg
10
10
0
0
10
10
10
Entr.
0.978
0.982
0.951
0,82
0.931
0,931
0.944
0.903
00.005
FID (GEN)
3.326 a0.03s
2.778 20.01
3.019 20100
2.836
anuss
2.987
2.785
20.00
3.025
2.727
20.079
2.831 20.3
FID (REC) 1
3.76 60.007
3.102 ano0a
3,406 a0.0s
3.189 a0.392
3.339
2.994 10.00
3.129 aa.ss
3.185
3.511 80.074
FMNIST
Recall (GEN)* 7
0.311 20.022
0.348 20.0
0.327 2000
0.338 20.324
0.336
20.004
0.35
=0.008
0 20
0.316
=000044
0.311 =0.006
Precision (GEN)
0.518 c0.ce
0. 0,556 20.00
0.551 200
0. 1.558 20.307
0.560
20/004
0.553
2: 000
0.558 101
0.576
20.000
0.574 20.005
2K-SVM 1
0.681 20.00
0.703 20001
0.681 10.000
0.715 26.e
0.68 e0o02
0.731
10.003
0.695 20
0.712 15.005
0.696 20.305
10K-SVM
0.731
0.771
0.763
0.775
0.763
0.78
0.772
0.778
0.773
5-NN t
0.425 20.000
0.591 20.00
0.6:1 20002
0. 0.601 20.325
0.618
=0.033
0.683
0.693 20
0.678 =0.008
0.707
=0.005
100-NN
0.606 a8.c24
0.682 am.04
0.691
10.000
0.69 aawi
0.650
wn.000
0.736
000
80.000
0.731
60.003
0.739
.0.004
Tertogy
10
10
10
0
10
100
100
10
Ente.
0.895 an.0e
0.886 20.00
0.914 a0.312
0.839
20.000
20002
0.929 20.003
0.511
20.043
FID (GEN)
4.424 40.064
4.538 10.1
4.876 40.0
4.547 20.079
4.595
4.465
1n noa
4.385
10.140
4.417
4.594
60.330
FID (REC) 1
1.13 20.000
4.379 a0.0s
1.686 20.14
4.539 aw.ss
4.519
s0oss
4,205 100000
4.084
10.006
4,141
4.585 20.373
CIFAR-10
Recall (GEN) 1
0.283 20.000
0.26 1,266 21.0007
0.253 251
0. 0,268 20.002
0.267
0.281
0.283
10.00
0.282
=0.005
0.261 20.032
Precision (GEN)
0.685 re4
0.687
20
0.689 200
0.69 20.00x
0.68 gm
0.676
0.679 20
0.677
007
0.685
2K-SVM 1
0.245
0.241
0.264
0.246
0.224
0.25
0.271
0:26
0.256
10K-SVM
0.391
0.385
0.379
0.387
0.365 28/13
0.396
0.407 20.00
0.401
0.396
5.NN 1
0.206 40.000
0.175
10.00
0.238 10.000
0.175 20.303
0.174
20:004
0.189 =0
0.239
10.005
0.196
s0.001
0.219 #0.300
100-NN 7
0.308 20.007
0.192 20.000
0.305 20.011
0. 0.186 a0.0s
0.219 20015
0.216 10 c0s
0.32 20.005
0.259 s0.003
0.273 20.004
Table 5: Quantitative performance on the images datasets. The LC flag refers to mixture component contributions being learnable while the IP flag refers to training the prior (i.e., prior-decoder cooperation scheme). Reported values are mean + standard error over three runs. The Teatropy row corresponds to the rogularization used to obtain the optimal FID(GEN) for each training configuratio,, where the Entr. row refers to the normalized entropy of the responsibilities where the closer to one its value the more uniformly the aggregated posterior is supported by the prior components.
38

---

Published in Transactions on Machine Learning Research (06/2025)
D.3 Illustrating the effect of regularizing the entropy of the responsibilities
Regularizing the entropy of the responsibilities, described in C was essential for avoiding the formation of inactive modes in our decodee cooperation scheme. Here we provide empirical evidence for that choice by analyzing the curves of normalized entropy the responsibilities for different regularization intensities controlled by the Teutropy hyperparameter and the corresponding FID(GEN) measuring the generation quality. Towards identifying the optimal value we experimented with Tentropy E [0, 1, 10, 100). however, to enhance the readability of Fig. 7 we omitted the curves for Tentropy 1 they displayed similar behavior to Feutropy 0.
MoG (10)
McG (10)
PRNNIGT
MoG (103) wol IP
MoG (105) wi
MNNST
PANNST
CIAAN-1
Figure 7: The effect of regularizing the entropy of the responsibilities under the 10- and 100-modal MoG priors.
Inspecting Fig. f reveals interesting insights into the effect responsibilities' regularization. First, it can be seen that different optimal Fentropy are be expected depending on the prior learning configuration and the datasets as indicated by the FID(curves). Additionally, we observe that the issue of inactive prior mode formation is more pronounced under the IP formulation. The blue lines, representing unregularized responsibilittis, lend to converge to a lower level compared to the fixed MoG prior setting, We altribute this
39

---

Published in Transactions on Machine Learning Research (06/2025)
behavior the prior modes being updated to support the aggregated posterior, which adapts according to discriminating objective. Interestingly, we also observe that when allowing for learnable contribution (i.e. LC) under the IP generally decreases the entropy of the responsibilities. This observation can be explained by the derivative KL with respect to the energy contributions as given by 62 More specifically was shown that the contributions of inactive prior modes tend decrease in favor of more dominant ones, fur- ther reducing the normalized entropy of the responsibilitics. Finally, the FID FID(GEN) curves corresponding CIFAR-10 dataset highlight the detrimental effect of generating samples from inactive modes. More specif- ically, when the responsibilities' entropy approaches zero (blue curves in CIFAR-10) the FID(GEN) tends to increase when not allowing for learnable contributions (dotted blue curves). In other words, generating samples from modes that do not support the aggregated posterior (i.e., inactive modes) leads to degraded generalion quality.
D.4 Latent Space Inspection
Here, we provide visualizations of the latent space of S-IntroVAE under the different configurations considered for the image generation task. More specifically we are interested in understanding how allowing for a trainable prior affects the latent space learned in S-IntroVAEs. Overall, the quantitative results suggest that learning the prior during the adversarial training leads significantly different latent space. In particular, we observe that the prior components are spread more evenly when allowing for trainable
prior comparod to when fixing it (sce Fig.
MaG (10)
Mod
MaG (18)
(100)
Mod (188)
MAG (100)
MaG (100)
Figure 8: Visualizing the first 2 latent dimensions of S-IntroVAE under different prior configurations along with samples from the aggregated posterior. Different colors correspond to different classes. Note that learning the prior during the adversarial learning leads to significantly different. latent space. The black dots refer to the means of the prior components, when applicable (i.e. w/ LC) the size of these dots refers to the contribution of this component in the MoG (c.g. the smaller the size the lower the contribution).
We also employed the t-SNE dimensionality reduction technique to visually inspect how prior learning affects the high-dimensional latent space. The quantitative results indicate that prior learning tends to create better- separated clusters. Although the separation effect less pronounced when modeling the prior with many components (e.g. 100 10 components), it remains noticeable (see Fig. 9h.
40

---

Published in Transactions on Machine Learning Research (06/2025)
(10) IPo
(18)
(100)
Med (100)
NaG (100)
Figure 9: Visualizing the high-dimensional latent space of the aggregated posterior using t-SNE dimension- ality reduction technique. Note that learning the prior during the adversarial learning generally leads better-separated clusters in the latent space. Different colors correspond to different classes.
41


Reviews:
[{"review_id": "Y4uim529G7", "reviewer": "Reviewer_5Qnh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Sound approach\n* It is clear what the authors want to do\n\nWeaknesses:\n* In a sense, the idea of the prior in Bayesian inference, is to inject prior knowledge and combine it with data. If the prior is learned, I wonder how much is one Bayesian? If the prior is learned, what is the advantage of VAE approach in principle, and and why one wouldn't simply train a regular AE / Masked AE for representation learning. I feel like this part of the paper is rather weak and needs elaboration. \n* Another weakness of this paper is toy data. It is a bit hard to get a feeling whether the proposed addition to the introspective VAE is actually helpful. \n* Clarity of the result section can be improved\n* Statistical comparison is lacking depth", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes prior learning for introspective VAE, the idea of which is to assign low likelihood to unrealistic datapoints. The authors add a modification to the general algorithm of introspective VAEs by \n\n1. Adding a Mixture of Gaussians as a form of prior\n2. Modification of the original S-IntroVAE adversarial training objective", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please divide the standard deviations by the $\\sqrt{N_{runs}}$ and report the standard error. Reporting standard deviation is not relevant here. \n2. I think that something should be done to improve the readability of the results. It is very hard to grasp what the authors are really trying to say from the tables. I believe that among the minimum here is to add a caption telling how to read the results (e.g. row-wise or column-wise). \n3. The of Algorithm 1 needs work. Currently it reads as if one defines the procedures in the code, but not one seems to be calling those. Please, revise to make it clear what is called when (I am talking about lines e.g. 7-15. You define a function, but never call).\n4. The paper is really missing more convincing benchmarks. The original SoftIntroVAE paper looks at generating high-fidelity images, such as faces. This particular paper looks at CIFAR / MNIST, and Toy data. I request the authors do a proper experimentation and back-to-back comparison between the proposed method and the original SoftIntroVAE.\n5. What made me curious, like all works on learning the prior is the capability of the resulting approach to overfit and memorize training data. I suggest the authors to also run the experiment showing whether prior learn does any harm in terms of sample memorization. This is related to my first point \"being Bayesian\".", "broader_impact_concerns": "There are not many concerns in relation to this paper, except it contributes to the generation of fake faces etc.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4L5UPwJDIz", "reviewer": "Reviewer_NfCx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Studies a relevant problem\n- Clear exposition of prior work\n- Experimental settings and scope appear appropriate to potentially support the claims\n\n## Weaknesses\n- The biggest weakness is that the construction and presentation of the experiments is too difficult to follow. As a result, I cannot confidently interpret the experimental results, so I cannot confidently say that the experiments support the claims of the paper. \n- There are some vagueness in the writing that makes the paper a bit difficult to follow \n- The experimental comparison could be more enlightening by comparing to other representation learning approaches besides VAEs (e.g. noise contrastive estimation)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an alternative training scheme for VAEs that learns the prior using an adversarial objective. The motivation is to improve over prior work that learns an VAE using an adversarial objective. Variants of the approach are compared to a standard VAE and a version of prior work that uses an adversarial objective. The experiments are conducted on several low-dimensional density estimation datasets, MNIST, FMNIST, and CIFAR-10. The experiments claim to show that the approach leads to superior density estimation performance and latent separation than relevant alternatives.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- First read of the abstract doesn't give a good sense of what the main difference between S-IntroVAE and Introspective VAEs is.\n- Introduce the term 'S-IntroVAE' with a reference in the introduction. After reading the abstract and the introduction, it's not clear where the S-IntroVAE is coming from, nor what it signifies. The reference first appears on page 3, despite the term being used on pages 1 and 2.\n- L25 in Alg. 1. is too vague, include a pointer to the section or equation that is relevant to implementing it.\n- Theorem 2 should be justified with a proof or proof sketch.\n- The experiments section should begin with a summary of the goals of the experiments section that connects to the paper's claims.\n- The font size on Table 1 is too small, please make it bigger.\n- The configuration of the experiments shown in Tables 1 and 2 is not clear. S5.1 introduces three discrete prior settings: SG, fixed MoG, and IP. However, Table 1 does not list them as separate settings. Instead, IP is a row of the table, whereas SG and MoG are columns of the table (and for some reason, SG is in two different columns in Table 1? It looks like maybe one of the columns is supposed to just be the standard VAE, and the rest are S-IntroVAE -- the table needs to make this clearer if so). Altogether this table is very confusing. If the prior has mutually exclusive settings, then they should appear as mutually exclusive values in a cell or some other configuration that shows them being mutually exclusive. I'm having to guess what the experiments actually show here (and the crossed out VAMP in the column title is not helping my understanding, either). Table 2 also appears to be missing the standard VAE results (I cannot tell for sure, the layout is different from Table 1). Fixing all of these issues would be necessary to secure an acceptance recommendation from me.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MCBLGi5vp2", "reviewer": "Reviewer_kyfR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- The scope of this work is well-defined.  \n- Each methodological component is clearly motivated and properly justified.  \n\n### Weaknesses:  \n- **Presentation of the Proposed Methodology (Section 4) Could Be Improved:**  \n  - The theoretical results in Section 4.1 do not provide much insight beyond defining the objective function. Unless I am missing key insights, Theorem 2 appears to be a trivial extension of Theorem 1, given that the proof of Theorem 1 is independent of the parametrization of \\( p_d \\). Additionally, the practical implementation of Proposition 1 deserves to be included in the main text.  \n  - Many important methodological details are either relegated to the appendix or only explained verbally without proper formulation. For instance, Section 4.2.2 focuses on addressing the prior variance explosion issue. However, it is unclear what specific variance is being referenced. Furthermore, the application of variance clipping (Equation 10) within Algorithm 1 is not explicitly stated.  \n\n- **Experimental Details:**  \n  - It would be helpful if the authors explicitly stated the prior choices of the competitor methods in the experiments.  \n\n- **Hyperparameter Tuning:**  \n  - The proposed training algorithm introduces additional hyperparameters beyond those already required in S-IntroVAE. A systematic approach to tuning these hyperparameters—such as \\( K \\) for variance clipping, \\( r \\) for entropy regularization, and the number of mixture components \\( M \\)—should be elaborated on.  \n\n- **Theoretical Claims:**  \n  - The claim that adaptive variance clipping and responsibility entropy are theoretically motivated seems somewhat overstated. The theoretical justification for these two components is unclear and should be explicitly addressed.\n\n- **Other rather minor Issues**:  \n  - The mathematical derivations in the appendix lack rigor. In particular, the expectation of certain statistics and their Monte Carlo estimates are often misused. For example, all equalities in Appendix C are mathematically incorrect—KL divergence in this context is not a function of \\( z_s \\).  \n  - Several capitalization and formatting errors are present in the References section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work focuses on enabling prior learning for the Soft-Introspective VAE (S-IntroVAE) method. Instead of adopting the standard Gaussian as the VAE prior, the authors employ a finite mixture of Gaussians. However, directly applying the training procedure used in S-IntroVAE can lead to unstable training dynamics and the collapse of the mixture prior, where the Gaussians merge into a single dominant component. To address these issues, the authors introduce adaptive prior variance clipping to stabilize training and mixture weight regularization to prevent prior degeneration.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please see the weakness part.", "broader_impact_concerns": "This work focuses on adapting prior distributions for VAE, which is not expected to raise broader social impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were torn on this paper. I have gone over their main points of critique and found that the revision fixes the majority of justified criticisms. A few minor issues would be good to improve upon, so I request a minor revision before accepting the paper.

1. One reviewer points out that it is confusing to have a headline on page 5 followed only by an algorithm. First, I propose to move the algorithm to a later stage in the paper (where it is actually discussed). Secondly, given the length of the algorithm, I would like to have it on its own page without any further content (including headlines).
2. I found remark 1 to raise more questions than it answers. If the implication is important, why is it not discussed in the paper? I would like to see either a bit more detail in the remark, or that the language of the remark is changed to not call the undescribed implication 'important'.
3. The proof sketch in Sec. 4.1 should include a reference to where the full proof can be found.
4. Fig. 2 contains a set of images, which are so small that they cannot be viewed without zooming. Please update the figure to make the images possible to look at.
Decision: Accept (poster)