$t^3$-Variational Autoencoder: Learning Heavy-tailed Data with Student's t and Power Divergence
Authors: Juno Kim
Jaehyuk Kwon
Mincheol Cho
Hyunjong Lee
Joong-Ho Won
Venue: ICLR 2024 poster (2024)
Keywords: Variational autoencoder
Information geometry
Heavy-tail learning
Generative model
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: The variational autoencoder (VAE) typically employs a standard normal prior as a regularizer for the probabilistic latent encoder. However, the Gaussian tail often decays too quickly to effectively accommodate the encoded points, failing to preserve crucial structures hidden in the data. In this paper, we explore the use of heavy-tailed models to combat over-regularization. Drawing upon insights from information geometry, we propose $t^3$VAE, a modified VAE framework that incorporates Student's t-distributions for the prior, encoder, and decoder. This results in a joint model distribution of a power form which we argue can better fit real-world datasets. We derive a new objective by reformulating the evidence lower bound as joint optimization of KL divergence between two statistical manifolds and replacing with $\gamma$-power divergence, a natural alternative for power families. $t^3$VAE demonstrates superior generation of low-density regions when trained on heavy-tailed synthetic data. Furthermore, we show that $t^3$VAE significantly outperforms other models on CelebA and imbalanced CIFAR-100 datasets.

Published as a conference paper at ICLR 2024
- VARIATIONAL AUTOENCODER:
LEARNING HEAVY-TAILED DATA WITH STUDENT'S T AND POWER DIVERGENCE
Juno Kim Jaehyuk Kwon Mincheol Cho Hyunjong Lee
Joong-Ho Won'
Department of Mathematical Informatics, The University of Tokyo
2 Centter for Advanced Intelligence Project, RIKEN
3 DDpartment of Statistics, Seoul National University
junokimeg.o www.ecc...ttoo ucc.u-tokyo.aco (jh19984, code 478,, hywnnoogg226eeesm...e.
wonjestats. ac.
ABSTRACT
The variational autoencoder (VAE) typically employs a standard normal prior as regularizer for the probabilistic latent encoder. However, the Gaussian tail often decays too quickly effectively accommodate the encoded points, failing to preserve crucial structures hidden in the data. In this paper, we explore the use of heavy-tailed models to combat over-regularization. Drawing upon insights from information geometry, we propose VAV, a modified VAE framework that incorporates Student's t-distributions for the prior, encoder, and decoder. This results in joint model distribution of power form which we argue can better fit real-world datasets. We derive a new objective by reformulating the evidence lower bound as joint optimization of KL divergence between two statistical manifolds and replacing with 7-power divergence, a natural alternative for power families. VAVAE demonstrates superior generation of low-density regions when trained on heavy- tailed synthetic data. Furthermore, we show that VAV significantly outperforms other models on CelebA and imbalanced CIFAR-I00 datasets.
INTRODUCTION
The variational autoencoder (VAE, Kingma & Welling, 2013) is popular probabilistic generative model for learning compact latent data representations. The VAE consists of two conditional models: an encoder which models the posterior distribution of the latent variable given observation æ, and decoder which infers the observation from its latent representation, which are jointly trained by optimizing the evidence lower bound (ELBO). The VAE framework a priori does not require the prior, encoder or decoder to be a particular probability distribution; the usual choice of Gaussian is mainly due to feasibility of the reparametrization trick and closed-form computation of divergence.
However, weal-world data frequently exhibits outlier-heavy heavy-tailed behavior which is better captured by models of similar nature. Recently, Floto al. (2023) showed that the Gaussian VAE encodes many points in low-density regions of the prior; the distribution is too tight effectively fit complex latent representations. We argue that distributing more mass the tails allows encoded points spread out casily, leading to adopt Student's t-distributed prior. We also incorporate i-distributed decoder which amplifies variability of data generated from low-density regions. From Bayesian perspective, this is equivalent to incorporating latent precision affecting both Together with the prior, this results in a joint model Po.v [ 2) of a power form, generalizing the exponential form of the original VAE analogously to how the t-distribution generalizes the Gaussian.
Changing the distributions usually necessitates numerical integration to estimate the ELBO. We provide novel alternative based recent theoretical insights from information geometry. Han et al. (2020) showed that the ELOO can be reformulated as minimization of KI. divergence between two statistical manifolds. Separately, Eguchi (2021) has developed a theory of 7-power divergence that parallels KL divergence. In this new geometry, power families play the role of exponential families,
Equal contribution.

---

Published as a conference paper at ICLR 2024
providing a natural setting for joint optimization of heavy-tailed models. By minimizing "y-power instead of KL divergence, we construct a general- purpose framework implementing t-distributions for the prior, encoder and decoder and a new objective called y-loss, which we call the V'E.
VAE requires a single hyperparameter V which is coupled to the degrees of freedom of the t- distributions and controls as a balance between reconstruction and regularization. In particular, VAE encompasses the Gaussian VAE and ordinary autoencoder as the limiting cases V - 0o and - 2, respectively. The 7-loss has an approximate closed form analogous to the ELBO and can be optimized via a t-reparametrization trick. We empirically demonstrate that e VAE can successfully approximate low-density regions of heavy-tailed datasets. Furthermore, VAE is able to learn and generate high- -igh-dimessoonal images in richer detail compared to various alternative VAEs. Finally, we extend our model a hierarchical architecture, the HVVA, which is able to reconstruct high-resolution images with more sophistication.
1.1 RELATED WORKS
Many authors point out the standard normal prior can induce over-regularization, losing valuable semantics hidden in the data. Alternative priors based on Gaussian mixtures (Tomczak & Welling, 2018: Dilokthanakul et al., 2016), the Dirichlet distribution (Joo et al., 2020), the von Mises-Fisher distribution (Davidson et al., 2018), normalizing flows (Rezende & Mohamed, 2015) stochastic processes (Goyal et al., 2017; Nalisnick & Smyth, 2017) have been proposed to mitigate this influence.
Another line of research has focused on modifying the ELBO using different divergences (Li & Turner, 2016; Deasy et al., 2021) or weights (Higgins et al., 2017). Instead of only changing the KL regularizer, take advantage of the joint KL divergence formulation for the first time. The 7-power and similar divergences have been studied before in the robust statistics literature. Some examples are density power divergence (Basu al., 1998), logarithmic 7-divergence (Fujisawa & Eguchi, 2008), and robust variational inference (Futami et al., 2018).
Some previous works have studied VAEs incorporating the t-distribution, namely the Student-t VAE with a I-decoder (Takahashi et al., 2018), the VAE-st with a t-prior and t-encoder (Abiri & Ohlsson, 2020), and the "disentangled" Unlike t² VAE, none of these stray from the ELBO framework, and the latter two require numerical integration to compute the KI regularizer. We also point out that these models all use a product of univariate t-distributions while VAA uses the multivariate t-distribution. We implement and compare with these models our experiments.
DE- DE-VAE with a product of t-priors (Mathieu et al., 2019).
Finally, heavy-tailed distributions have also been used as base densities in the normalizing flow literature (Alexanderson & Henter, 2020; Jaini et al., 2020; Laszkiewicz et al., 2022; Amiri et al., 2022), where it has been argued that t-distributions lead to improved robustness and generalization. We take a step further by enforcing a power form on the joint model, which is key to
t VAE''s success.
THEORETICAL BACKGROUND
In this section, summarize key aspects of the motivating theories of variational inference and information geometry. Details and proofs are deferred to Appendix A.
2.1 VAE AS JOINT MINIMIZATION
Formally, a VAE models the distribution (2) of the observed variable R" by jointly learning a stochastic latent variable 2 E RT Generation is performed by sampling = from the prior pz (z), then sampling æ according
probabilistic decoder po(x]²) parametrized by 0 E O. The observed likelihood pe (xx) ) = f po [ p2) (z)d2 is intractable, so we instead aim to approximate the posterior Po(2(2) with a parametrized encoder q (2(x) by minimizing their KL divergence. This leads to maximizing the evidence lower bound (ELBO) of the logelikel, defined as
L(x; 8, Ø) : = log (x - DKL (90 (2|x) || (=(x)))
E (g-ggst(a) (a) [log po (z/2)] - DKL (2)()) [I))))).
(2)
Since -Eg (11) [log po (x]==) is the cross-entropy reconstruction loss of the original data, the ELBO can be understood adding KI. regularization term, forcing (=(x)) close to the prior
2

---

Published as a conference paper at ICLR 2024
in order to ensure that the latent state encodes only useful information. In practice, the prior is typically standard normal and the encoder and decoder distributions are parametrized Gaussians,
pz(z) ~ Nm (0, 1), (=(2)²) ~ Nn (1), Ex(x)), pa(=]z) ~ N (=((uo(2),
(3)
in which case the reconstruction loss equal to the mean squared error (MSE) between ær and the decoded mean up to constant,
E (-lge(llu) [og](()))2]] (11x))))))))
2 || (=)] const.
The encoder covariance is usually taken as (but not assumed to be) diagonal, Eş(x) = diag (x).
Han al. (2020) point out that the VAE can be reinterpreted as a joint minimization process between two statistical manifolds, Let P = {po(I, 2) p=(x)=)pzz (z) 0 E Q be the model distribution manifold, and Q == {go (z, 2) Pdata (=))(((())) Φ E } the data distribution manifold, both submanifolds of the space of joint distributions. The divergence between points on P, Q may be recast (Appendix A.1) as
( 2) (z, 2)) -Ezopaara [c(x; 0, Φ) H (Pdsta),
(5)
where H (Pdata) denotes the differential entropy of the data distribution. Thus, maximizing the ELBO is equivalent to finding the joint minimizer
(pa- = argmin (x
(6)
7EP,
2.2 INFORMATION GEOMETRY AND POWER DIVERGENCE
Any divergence D statistical manifold M of probability distributions automatically induces Riemannian metric g and two affine connections r, D on M dually coupled with respect to (Amari, 2016, Chapter 6). For KL divergence, g is the Fisher information metric and the I and aatoapapallll curves connecting two points p. are the m- (miitutture) geodesic (z(r) = (1 t)p(z) + tq(x) and the e-(exponential) geodesic (x( x exp((1 t) log p(z) + t log g(z)), respectively.
particular, the -flat submanifolds consist of the exponential families. The problem (6) may then be solved by
applying the information geometric em-algorithm (Csiszár, 1984; Han et al., 2020).
Paralleling the KL case, Eguchi (2021) defines the 7-power entropy and cross-entropy for probability distributions q. p with power y as
# C,(9,)) = -aa.)):--oa ((())) p(x)
(p) = - 1lpll = (llo)ttaa))
-
dx
Tlpll
and the 7-power divergence as (glp)) = C. - Hg) In our paper, we introduce an additional factor of 1/7 in order to extend to the case -1 < <1><<0 0,"
(glp)) : 7c - (z,
(8)
We show that D., is indeed divergence and derive its induced metric in Appendix A.2. Computing the dual connections yields that the totally I"* geodessi (or y-flat) submanifolds consist of power families of the form
S. (po x (1 + 7 s(x)) : E 9).
In particular, the family of d-variate Student's t-distributions with variable mean 14, scale matrix Σ and fixed degrees of freedom
(r E, = Cu.d [Σ| (1+ 1 (()=--)TTT) (1+-(0-1)T=-10 x -1) ())
24d
r(
Cu.d
(10)
((x)
is flat when 7 2 y-power divergence from qu La Eg, v) pa ta(j1, Σ1, v) is finite when > 2 and can be
utd. which assume in the remainder of this Section. Furthermore, the
'gucch (2021) also points out that the t-distribution naturally emerges as the maximal entropy distribution when = see Proposition 2. However, his original definition of 7-power divergence is erroncously non-negative only for 7 > 0, necessitating our modification.
3

---

Published as a conference paper at ICLR 2024
expressed in closed-form (Proposition 3) as
11 + wー2)
T [- - 1=000
1-7
शपो (1 -
(11)
2-2 tr (ET Eo) + (14 - 11) T E 1 m))]
E1 X0+
KL divergence may be retrieved from "-power divergence as limy-0 see Proposition Moreover, Equation (11) converges to the KL divergence between the limiting Gaussian distributions goo ~ NNa(Ho Mo) and Poe Na(111, Σ1) as v oc.
[Ip) DKL (g pl.
3 THE t VARIATIONAL AUTOENCODER
3.1 STRUCTURE OF THE VAE
Throughout this section, we present definitions and thcoretical properties of our VAE model and 7-loss function. Full derivations are provided in Appendix B. When the prior, encoder and decoder are normally distributed as in Equation (3), the joint model distribution po 2) E P takes the form
exp -
|| [1=1]+ + 0 | mo(2)ll )]
Pe 2) x
(12)
Since the tail is exponentially bounded, its capacity to approximate the data distribution manifold corresponding to real-world data is limited. To mitigate this problem, we propose a heavy-tailed model p (x, z) of a power form, parametrized by the degrees of freedom v > 2,
| (1e2)
-/ (il=-m(()1)))
(a, 2) x x
(13)
o²
which is obtained from the prior and decoder distributions
(z) t. (2)0,1,
(14)
P (x]22) t. (x (o(2)) Itv
1+v ||=1|2 2
(15)
Since the true posterior 2/2 is t-distributed with degrees of freedom + n when the decoder is shallow (discussed in Appendix B.2), are also motivated to incorporate t-distributed encoder
(2 mm (1)) + - -n) - (x)) u+)
(16)
Hence, the VAE model and data distribution manifolds are P, = (po 2) = (=)=),)
E } and Q, = (x, z) = (zata( (z)(). : o E 4). This generalizes the Gaussian VAE
in the sense that pa,, (2,, 2) and (-(r() uniformly converge to pa pa(zz z) and 40 (-(xx) as +
3.2 POWER DIVERGENCE LOSS
From the geometric relationship of y-power divergence and power families and the model equation (13), we motivated replace the KL objective in the joint minimization problem (6) with y-power divergence for 1 VAE,
(pe- v)
argmin (a P)
(17)
PEP.. 2
where y is coupled to / as 7 = レ---- The power divergence from Ze. E Q, to Pa. Py can be computed in closed-form after approximation of order 72 (Proposition 5), After rearranging constants, we obtain the y-loss objective which is amenable to Monte Carlo estimation,
(,
E Exrpa
E
an ( || 2=(2)]1
2
(18)
+ ||)(T)]1
n- tz) - 2C
1
[ -00-)))))))))))))))))))))))
C2
for constants C₁ p+m+n=2 (1+ vナカー
Cznm) and Cg = wー2
++m+n=22 0d vmmtu
See Appendix B.3 for proofs and analysis of error.
4

---

Published as a conference paper at ICLR 2024
1.0
10-
2.0
Regullattir
0.0
1.0
0.4
54
04
0.
00
0.0
0.00
0.25
0.50
0.75
1.00
1.50
10"7
10"1
132
10"7
10:1
107
207
Agtal
0--2
Figure 1: (a) Dependency of regularization on Eg (2) when m = n n 0,0= (left): (b) graph the alternative prior scale against (middle), (c) graph of the regularizer coefficient against (right).
The y-loss does not lower bound the data likelihood; its precise role will be made clear shortly. We emphasize that our framework is to view the ELBO not as bound of likelihood (which leads to modifying the KL regularizer), but as a divergence between joint distributions (leads to modifying the entire divergence). This change is further justified in Fujisawa & Eguchi (2008) which shows that 7-type divergence minimizers are asymptotically efficient M-cstimators, lending support to our approach solid alternative maximum likelihood methods for statistical inference.
3.3 CONTROLS REGULARIZATION STRENGTH
We now shine light on the meaning of the VAE hyperparameter v. Analogously to the ELBO, the y-loss (18) consists of MSE reconstruction error and additional terms which act as a regularizer. This can be made precise: in fact, the remaining terms equivalent (up to constants) to the 7-power divergence from the posterior 46 (z(æx) to the alternative prior
p=(2) = tm 12,0,
72
It (o x=zz)
(19)
derived in Appendix B.4. Equation (18) can then be rewritten as
(0, Ex-pusta
(z || m=(=)]] (as, [1+:)] + const.
(20)
Hence, y-loss can be interpreted similarly balance between reconstruction and regularization, and y controls both the target scale 72 and the regularizer coefficient a 205
7M
Figure ((a) plots the regularizer as a function of Eg (x) for a range of y when m = n 1. The KL graph verifies that (x(x) is forced towards 72
the Gaussian case, and this effect is particularly strong when Es(x)) (xx < <² For smaller degrees of freedom, becomes less than and the forcing effect for small (x) decreases as well, allowing the encoded distributions to be more point-like. The overall weakening of regularization consistent with our motivations for adopting heavy-tailed prior. This also separates VVVA from models which simply assign weight to the KL regularizer such as B- V-V (Higgins et al., 2017), where the target 72 remains constant.
Figure ((b).C) further show the dependency of т. against for E {1,4, 64). As {VVAE converges to the Gaussian VAE. As v + 2, theory both T, a - 0 so that regularization vanishes and VAE regresses to raw autoencoder. However, this does not occur practice in high dimensions as the model is much less sensitive to v near 2 due to the slow convergence rate. low dimensions, this regime does come into play and t VAE performs significantly worse if is very small. We therefore suggest selecting u - to be roughly logarithmically spaced to observe varying levels of regularization strength, and only few values needed for higher dimensional datasets.
3.4 SHVAE THE BAYESIAN HIERARCHY
The principle of VAE naturally extends to hierarchical VAEs (HVAEs), which have recently shown great success in generating high-quality images and learning structured features (Vahdat 2020: Child, 2021; Havtor et al., 2021). In Appendix B.1, prove SAAA may be viewed
& Kautz,

---

Published as a conference paper at ICLR 2024
Algorithm 1: Overview of VAE
Require data (1(1) ... x(k), x hyperparameter y compute 7, T; C1,
if TRAINING PHASE then
Take gradient step with V (0,
for i E {1....,.................... .......KKK do
using t-reparametrizztion trick
(24)
retrieve pd (z (x))))
if GENERATION PHASE then sample Zgen ~ ()) retrieve 140 (Zgen)
sample 200) (0 (2(x()) retrieve p=(2())
D (16)
(19)
sample Frecon P8, Pa (x)2(0))
D (15)
sample Zgen pe. ( Zgon)
(15)
CNNE 9.
EHAE 12:00
CNAE 15.00
ESAE ia 18
EMAE (+-00 21.01
P-VAE 0.11
DE-WAR Op-m 9.0)
WASee
Figure 2: Log-histograms of samples generated from VAE (v = 9, 12, 22, 15, 18, 21), Gaussian VAE,
3-VAE (3 0.1), Student-t VAE, DE-VAE and VAE-st. Solid lines illustrate the true density Pheavy.
Bayesian model by introducing latent precision affecting both x and #2:
A ~ (Ale), z ~ No. (=|0, (=10, =|=,> ~ Nu(crIn((E),
(21)
is then straightforward to add any number latent layers zj/Res, N (z (o(=s)) VA vA-
to obtain heavy-tailed hierarchical prior (21.... ... 21). In Appendix B.5, we develop two-level hierarchical VAVAE or HVAE, where the priors 21 and ø2| 21. decoder x ( 22) and encoders
zi jar and 22 (Ir, are all t-distributed. We also rederive the 7-loss from the information geometric formulation (17), showcasing the applicability of our approach.
4 EXPERIMENTS
In this section, we explore the empirical advantages that V'E offers over the Gaussian VAE and other alternative models by analyzing various aspects of performance synthetic and real datasets. summary of our framework provided Algorithm to assist with implementation. Experimental details including the reparametrization trick for & network architectures and further results documented in Appendix C.
4.1 LEARNING HEAVY-TAILED BIMODAL DISTRIBUTIONS
Univariate dataset. We begin by analyzing generative performance on a univariate heavy-tailed dataset. We compare VAE with E (9. 12, 15, 18, 21). Gaussian VAE and 3-VAE, as well as other I-based models: Student-t VAE (Takahashi et al., 2018), VAE (Mathieu et al., 2019) and VAE-st
(Abiri & Ohlsson, 2020), Each model is trained on 200K samples from the bimodal density
Pheavy (x) = 0.6 x (x) 2, 12, 5) + 0.4 x 4 (a 12,
(22)
We then generate 500K samples from each of the trained models and compare to Pheavy illustrated in Figure 2. We plot log-scale histograms in order to capture behavior in www-denstty regions. The Gaussian and 3- VAE completely fail to produce reliable tail samples, in particular generating none beyond the range of 100. In contrast, t-based models (with the exception of VAE-st) are capable of learning a much wider range of tail densities, establishing the efficacy of heavy-tailed models. For
14=11 "The factor 1+p am
the t-decoder (15) may hence understood incorporating information gained on the latent precision when is observed. For tail region values encoded with large magnitude, one infers smaller precision and thus increased variance for the output.

---

Published as a conference paper at ICLR 2024
Table I: p-values for the MMD test for (a) univariate and (b) bivariate synthetic heavy-tailed data. Rejected values are shown in red. Hyperparameters are tuned separately for each model and the best versions are reported; see Tables C1 and C2 for the full data.
Model
Full
Left tail
Right tail
Model
Full
Left tail
Right tail
VAB (v 18)
0.322
0.377
0.693
Student t-VAE
0.587
0.291
0.114
VAE
0.514
0.036
0.003
DE- VAE (v = 9)
0.943
0.424
0.814
B- VAE (3 = 0.1)
0.614
0.011
< 0.001
VAE-st (V = 12)
< 0.001
0.953
0,643
(a) MMD test p-values for univariate data.
Model
Full
Left tail
Right tail
Model
Full
Left tail
Right tail
VAE (v = 30)
0.276
0.214
0.213
Student (-VAE
0.530
< 0.001
< 0.001
VAE
0.116
0.004
< 0.001
DE-VAE (v 3)
0.624
0.002
0.057
8-VAE (3 0.5)
0.251
0.011
< 0.001
VAE-st (v
0.485
0.001
< 0.001
(b) MMD test p-values for bivariate data.
VAE,, we observe the best performance for 20; smaller leads to tail overestimation, while further increasing v loses mass and ultimately converges to the Gaussian case as + oc.
For quantitative comparison, we apply the maximum mean discrepancy (MMD) test (Gretton et al., 2012) to evaluate whether the generated distributions are distinguishable from the original. As the test is insensitive to tail behavior, we also truncate the data by removing all points with absolute value less than 6 and apply the MMD test to the resulting tails. Table (()) shows p- values for the full dataset, left (x < 66) and right tails (æ > 6). Testing with the full dataset fails to reject most models; restricted to low-density regions, however, MMD testing completely rejects the Gaussian VAE and tuned 3-V.F. In contrast, our model output is indistinguishable from Pheaay for moderate values of v.
Bivariate dataset. To distinguish t VAE from other t-based models, we further design a bivariate heavy-tailed dataset. We generate 200K samples x; from Pheavy (I) 0.7 x (2 - 2, 22, 5) 0.3 x t (ax) 2, 22, 5) and take y4 = xz + 2 sin (ra) with noise distributed as tz 0) I2, 6). We then run the MMD test for the full data and the left (x²2 + y² > 102, x < 0) and right (x² + y² > 102 x > 0) tail regions. The results are presented in Table (bb). All models approximate the true distribution well in high-density regions; however, for low-density generation, the null hypothesis is rejected in every case except for VAA,, demonstrating the utility of implementing the maltivariate t-distribution.
4.2 LEARNING HIGH-DIMENSIONAL IMAGES
We now showcase the effectiveness of our model nigh-dimensional data via both reconstruction and generation tasks in CelebA (Liu et al., 2015; 2018). The Fréchet inception distance (FID) score (Heusel al., 2017) is employed to evaluate image quality. In order comprehensively establish superiority, we also compare against a wide range of recent alternative VAE models.
natural question ask is whether heavy-tailedness is indeed the factor contributing to effective latent encoding. We address these concerns by comparing with simpler alternatives: Gaussian prior with increased variance Nn, (0, x² T) for и. > 1, and 33 with regularizer weighting 3 1. We also include Student-t VAE and DE-VAE as an ablation study of t-based model components. In addition, we implement two strong models which address latent structural collapse from different perspectives. The Tilted VAE (Floto et al., 2023), whose prior is designed to contain more mass in high-density rather than low-density regions; and FactorVAE (Kim Mnih, 2018), to assess the effect of disentanglement. Hyperparameters of all models are tuned separately for each experiment.
CelebA reconstruction. We evaluate the quality of reconstructed 64 x 64 images in Table 2(a), where 2 VAE consistently achieves the lowest FID out of all models. While Tilted VAE and 8- 3-VAE also show good performance, they still cannot match vVVAE in particular for rarer class labels. These images exist in low-density regions of the data distribution; therefore, we hypothesize that the sailedess of VAE makes better suited in particular to learning rare features. We note that
7

---

Published as a conference paper at ICLR 2024
Table 2: Reconstruction FID scores of CelebA and CIFARIOO-LT. In CclebA, both overall scores and selected classes are shown. Bald, Mustache (Mst), and Gray hair (Gray) are rare classes (less than 5% of the total), while No beard (No Bd) common (over 50%). In CIFAR FID is measured varying imbalance factor Complete results of tuning each model are included in Appendix C.3.
(a) CelebA
(b) CIFAR100-LT
Framework
Bald
Gray
No
Framework
p
AVAE 10)
39.4
66.5
61.5
67.2
40.1
( (vr 10)
97.5
102.8
108.3
128.7
VAE
57.9
85.8
79.7
58.4
VAE
256.1
267.2
277.4
267.3
VAE (K = 1.5)
73.2
105.3
96.4
114.5
73.8
VAE (ic 1.5)
274.2
290.5
296.7
297.7
8-VAE (8- 0.05)
40.4
69.3
62.7
71.1
40.9
3-VAE (33 = (111)
114.1
130.4
138.5
160.6
Student-A VAB
78.4
112.0
104.2
118.7
78.6
Student-f VAE
259.5
314.1
323.7
333.4
DE-VAE (1 5)
58.9
89.6
81.3
91.9
59.1
DE-VAE (V 2.5)
219.4
250.2
256.7
258.5
VAE (T 50)
73.0
73.7
42.9
Tihted VAE (r 50)
101.0
126.1
147.0
193.2
FactorVAB VAB (%h 5)
59.8
91.7
85.7
95.2
60.8
FactorVAE
232.3
272.5
275.6
270.1
25
Original
V'E
VAE
VAE (x = 1.5)
Tilted VAE
Figure 3: Original and reconstructed images by VAB (V 10), Gaussian VAE, VAE with 1.5, and Tilted VAE (T 50).
simply increasing the variance of Gaussian VAE results in images with reduced clarity. Moreover, disentangling does
to significantly alleviate over-regularization.
Investigating further, the images in Figure 3 display rare feature combinations; for example, the top left image belongs the intersection of the Male and Heavy Make-up classes, which constitute around 1 1% of all images. We see that the Gaussian VAE largely fails to learn the image and instead generates much more feminine face, evidenced by e.g. eyeshadow and lip color. In contrast, our model is able to more closely mirror the original image. Hence, VAE is capable of learning images more accurately, particular those with rare features.
Framework
FID
VVAE (v 10)
50.6
(VVA (v
10)
VAE
VAE
64.7
VAE (K 1.5)
79.6
3-VAE (3 = 0.05)
51.8
Student-t VAE
82.3
VAE 1.5)
8-VAE
DE-VAE (v = 2.5)
58.9
Tilted VAE (T 30)
59.2
FactorVAB (Tte = 2.5)
67.0
Student-t VAE
DE-VAE VAE
Table 3: Generation FID scores for CelebA.
Figure 4: Generated CelebA ex- ample images.
Tilted VAE
FactorVAl
CelebA generation. Works on VAE models often not consider the generative aspect due to difficulties in producing sharp images. Nevertheless, we find that VAE consistently generates high-quality samples we sample from the alternative t-prior (z).. As demonstrated in Table 3 and Figure 4, + VAE outperforms all other models in FID score and generates note that 3- 3-VAE is close contender but cannot surpass VAE even when 3 is fine-tuned; generation FID scores for various 3 values documented in Appendix C.3.
vivid images. We
8

---

Published as a conference paper at ICLR 2024
Original
*HVAA
HVAE
Figure 5: Original and reconstructed images by H'VAE (v = 10) and HVAE.
One interpretation of the heavy-tailedness of real-world data is when the occurrence frequencies of each class follow long-tailed distribution. For instance, CelebA comprises facial attribute labels with highly varying frequencies. Motivated by this observation, we conduct reconstruction experiments with the CIFARI00-LT dataset (Cao et al., 2019), which is a long-tailed version of the original CIFAR- 100 (Krizhevsky, 2009). We further consider varying degrees of imbalance, quantified by the imbalance factor p which defined the ratio of instances in the largest class to the smallest. In this experiment, we take E (1, 10, 50, 100) by linearly reducing the number of instances in each class.
Imbalanced CIFAR.
Table 2(b) reports reconstruction FID scores of each tuned model, where 1 VAE again achieves the lowest scores. This time, we draw the reader's attention to the comparative performance of ('VAE, 3-VAE and Tilted VAE as p varies. While the scores are relatively similar in the balanced case, the gaps become larger as p increases; that is, t VAE experiences less of a performance drop even for extremely lopsided datasets (p 100). In conclusion, verify that t VAE is especially strong with regard to imbalanced data with long tails.
HHVAE. We additionally implement the two-layer hierarchical (HHVAE constructed in Appendix B.5 and compare with the Gaussian HVAE on higher resolution 128 x 128 CelebA images. The reconstruction results are displayed in Figure 5, and corresponding FID scores are recorded in Table C3. We see that the increased hierarchical depth allows (3HVAE to learn more sophisticated images, again with substantially higher elarity and sharper detail compared the Gaussian HVAE. These results further justify the generality and effectiveness of our theoretical framework.
Training cost and hypepaaraeeter selection. Unlike models such as DE- VAE or VAE-st which require numerical integration to calculate KI. divergence between t-distributions for each data point, the explicit form of the 7-loss (18), as well as the t-based sampling and reparametrization processes, do not create any computational bottlenecks instability issues; the runtime of & was virtually identical to the corresponding Gaussian VAE. In addition, as shown in Figure and also corroborated by experiments, V''A''s performance remains consistent across different values of v for high- dimensional data. Thus another unique strength of our model is that there is no need to extensively tune the hyperparameter, only a couple of trials suffice.
CONCLUSION
Motivated by topies in regulaaaaaiii and information geometry, we establish a novel VAE framework for approximating heavy-tailed densities that uses Student's t-distributions for prior, encoder and decoder. We also derive the 7-loss objective by replacing the KI. divergence joint optimization formulation of ELBO with y-power divergence, and study the effects of its regularizer. The generative and tail-modeling capabilities of VAE are demonstrated on heavy-tailed synthetic data. Finally, we show that VAE outperforms various alternative VAE models and can Iearn images in richer detail, especially in the presense of rare classes or imbalance. Our ideas may hopefully be extended to other probabilistic or divergence- based inference models in future works.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the AI-Bio Research Grant through Seoul National University and by the Institute of Information & Communications Technology Planning & Evaluation (ITPP) grant funded by the South Korean government (MSIT) 1N0.2021-0001345-0004 Artificial Intelligence Graduate School Program (Seoul National University)).
REFERENCES
N. Abiri and M. Ohlsson. Variational auto-encoders with Student's t-prior. Proceedings of 27th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, pp. 415-420, 2020.
S. Alexanderson and G. E. Henter. Robust model training and generalisation with Studentising flows. In Second ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2020.
S.-i. Amari. Information geometry and its applications, volume 194. Springer, 2016.
S. Amiri, E. Nalismick, A. Belloum, S. Klous, and L. Gommans. Generating heavy-tailed synthetic data with normalizing flows. In The 5th Workshop on Tractable Probabilistic Modeling, 2022.
A. Basu, I. R. Harris, N. L. Hjort, and M. Jones. Robust and efficient estimation by minimising a density power divergence. Biometrika, 85(3):549-555 559, 1998.
K. Cao. C. Wei, A. Gaidon, N. Arechiga, and T. Ma. Learning imbalanced datasets with label- distribution-aware margin loss. In Advances in Neural Information Processing Systems, 2019.
R. Child. Very deep VAEs generalize autoregressive models and can outperform them on images. In International Conference Learning Representations, 2021.
I. Csiszár. Information geometry and alternating minimization procedures. Statistics and Decisions, Dedewicz, 1:205-231 1984.
T. R. Davidson, L. Falorsi, N. De Cao, T. Kipf, and J. M. Tomczak. Hyperspherical variational auto-encoders. In 34th Conference on Uncertainty in Artificial Intelligence 2018, pp. 856 865. AUAI, 2018.
J. Deasy, T. A. Mclver, N. Simidjievski, and P. Lio. a- G-VAEs Optimising variational inference by learning data-dependent divergence skew. In ICML Workshop Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
N. Dilokthanakul, P. A. Mediano, M. Garnelo, M. C. Lee, H. Salimbeni, K. Arulkumaran, and M. Shanahan. Deep unsupervised clustering with Gaussian mixture variational autoencoders. arXiv preprint arXiv: 1.02648, 2016.
Ding. On the conditional distribution of the multivariate distribution. The American Statistician, 703))229,- 2016.
S. Eguchi. Pythagoras theorem in information geometry and applications to generalized linear models. In Handbook of Statistics, volume 45, pp. 15-42. Elsevier, 2021.
G. Floto, S. Kremer, and M. Nica. The Tilted variational autoencoder: improving out-of- distribution detection. In The Eleventh International Conference Learning Representations, 2023.
H. Fujisawa and S. Eguchi. Robust parameter estimation with small bias against heavy contamina- tion. Journal of Multivariate Analysis, 99(9):2053-25 2081,, 2008
Futami, I. Sato, and M. Sugiyama. Variational inference based on robust divergences. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 813- 822. PMLR, 2018.
P. Goyal, Z. Hu, X. Liang, C. Wang, and E. P. Xing. Nonparametric variational auto-encoders for hierarchical representation learning. Proceedings of the IEEE International Conference on Computer Vision, pp. 5094-5102, 2017.
10

---

Published as a conference paper at ICLR 2024
A. Gretton, K. M. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
T. Han, J. Zhang, and Y. N. Wu. From em-projections to variational auto-encoder. In NeurlPS 2020 Workshop: Deep Learning through Information Geometry, 2020.
J. D. Havtorn, J. Frellsen, S. Hauberg, and L. Maalee. Hierarchical VAEs know what they dont know. In International Conference on Machine Learning, 2021.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by two time-scale update rule converge to a local Nash equilibrium. Advances Neural Information Processing Systems, 30, 2017.
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, Mohamed, and A. Lerchner. beta-VAE: VAE: learning basic visual concepts with a constrained variational framework. In The Fifth International Conference on Learning Representations, 2017.
P. Jaini, I. Kobyzev, Y. Yu, and M. A. Brubaker. Tails of Lipschitz triangular flows. In Proceedings of the 37th International Conference on Machine Learning, volume 119. PMLR, 2020.
W. Joo, W. Lee, S. Park, and L.-C. Moon. Dirichlet variational autoencoder. Pattern Recognition, 107: 107-514, 2020.
H. Kim and A. Mnih. Disentangling by factorising. In The 25th International Conference on Machine Learning, pp. 1649-2658.. PMLR, 2018.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. 2014. arXiv:1412.6980.
arXiv preprint
D. P. Kingma and M. Welling, Auto-encoding variational bayes. 2013. arXiv preprint arXiv:1312.6114.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
M. Laszkiewicz. J. Lederer, and A. Fischer. Marginal tail-adaptive normalizing flows. In Proceedings of the 39th International Conference on Machine Learning, volume 162. PMLR, 2022.
Y. Li and R. E. Turner. Rényi divergence variational inference. Advances in Neural Information Processing Systems, 29, 2016.
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision, December 2015.
Z. Liu, P. Luo, X. Wang, and X. Tang. Large-scale CelebFaces Attributes (CelebA) dataset. Retrieved August, (5(2018):11, 2018.
E. Mathieu, T. Rainforth, N. Siddharth, and Y. W. Teh. Disentangling disentanglement in variational autoencoders. In International conference on machine learning, pp. 4402 4412. PMLR, 2019.
E. Nalisnick and P. Smyth. Stick-breaking variational autoencoders. In The Fifth International Conference on Learning Representations, 2017.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin. N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, hirh-performancc deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
D. Rezende and Mohamed. Variational inference with normalizing flows. International Conference on Machine Learning, 1530-1538, PMLR, 2015.
C. Sønderby, T. Raiko, I.. Maaløe, S. Sønderby, and O. Winther. Ladder variational autoencoders. In Advances in Neural Information Processing Systems, 3745-3753, 2016.
A. Subramanian. PyTorch-Veee VAE. httpp:://ghhhbb.ccm///tt/ttttttt 2020.

---

Published as a conference paper at ICLR 2024
H. Takahashi, T. Iwata, Y. Yamanaka, M. Yamada, and S. Yagi. Student-t variational autoencoder for robust density estimation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 26962702, 2018
Tomezak and M. Welling. VAE with VampPrior. In The 21st International Conference on Artificial Intelligence and Statistics, pp. 1214-1223, PMLR, 2018.
A. Vahdat and J. Kautz. NVAE: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 13:19667-19677 2020.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
In Appendices A and we give precise statements, proofs and more details for the theory presented in Sections 2 and 3. The argument in A.1 is due to Han et al. (2020). The material in A.2 up to Proposition 2 is adapted from Eguchi (2021), while the general theory information geometry is presented in Amari (2016). All subsequent material is our own original work. In Appendix C, provide implementation and training details for the experiments conducted in our paper.
A THEORETICAL DETAILS
A. REWRITING THE ELBO AS KL DIVERGENCE
Equation (5) states that minimizing the divergence between points the model and data distribution manifolds equivalent to maximizing the expected ELBO. This can be shown by performing the following algebraic manipulation:
// Pieara (2))) log
Panta (2) dzdæ
(KL(qs(x, 2) po(ox, 2))
po pa(x)po
f Palata (I) log
Pdata (I) dx + f (aaca(())
[f (=(()x)) log (=(2)2))
go go(2|z) dz dr
po (a:)
Expopasta [- lo (x (2((9)(2)z) po(2|x)))] H (Paata) Egropaata [xx; [)] (Paata))
This formulation opens the door to various VAE modifications by replacing the joint model KL divergence with other divergences, assuming closed-form computation is possible and results in an estimable objective.
A.2 INFORMATION GEOMETRY OF P-POWER DIVERGENCE
Let M = (x)( 0 E θ} be a statistical manifold of probability distributions on a probability space X parametrized by local coordinates D M M + Rz0 satisfying
(f.,....,.)) ,0a)". divergence on M is C function
1. D(g (1)) 2 0 for all p. q E M.
2. D(g pp) if and only if = 9.
3. At any point Ps E M, D(pe ) is positive-definit quadratic form for infinitesimal displacements 6' + dB,
D(po por)
Σ
(o)(O)d ((1d0 O
i,j=1
for symmetric positive-definite matrix g(0) = (gt (0))
Proposition 1. 7-power divergence as defined in Equations (7)- E (01,0) is
divergence the finite y-entropy submanifold {p M: ||p|l < 00) M.
Proof. When 7 O, we have by Hölder's inequality
i (<))p(>)ddr < |alli I1 Iall Ileli
with equality iff p q. When -1 < < 0, we have by the reverse Hölder inequality
/ q(=)p(x))dz 2 ||a|l Ilp"ll lall IIell
again with equality iff q. In both cases conclude that
p.(allp) = - ((x)
p(x)
1 I|lll 2 O. dr +
13

---

Published as a conference paper at ICLR 2024
The 1st and 2nd order local expansion terms of Dy (pe por) around 6' 0 can be computed
0
Dpo por) = -- /
Öpo
po (x
? l ipollity /
1
Opo 00, (z)
(x)
dr
po(x))
00: p/=8
00,
lppelli+>
(Po Per) = / px(x)
px(x)
(x( dx 0,
00%
[ 1-7
a0;
Ipoll1t>
1'=00
and
02
9iy
(y(pe por)
00,00'
7-1
Ope
(x)
po(x)
(x)
dx
00,
|lpel 111t>
08,
|Ipoll
[//
(/
Ape
Ope
-1lpolli-> 1-22
po po(x))+7
po (-=))=--111
(x)
(x)dx
a0:
00,
- -(f po (x))
Opo (s))drr
(ʃ
Opo
po (2))
(3r)dz)
00:
00,
from which we can check that (0) (0)
Any divergence D on M thus naturally induces a Riemannian metric g M. For KL divergence, this is the Fisher information metric. DD also induces two affine connections r, D* on M as
8
(0) 00,00,08, 10-00
D(p [p
T (0)
D(po por)
a8; 00 o0-0
The connections are dually coupled with respect to g. i.e. the parallel transport of any two vectors, each by and r*, preserves their inner product. For 7-power divergence, it is straightforward to check that r (0) vanish identically for mixture families
Σ {1-E))
(1-
Po (x)
Σ (x))))
(00)
Hence, the r-autoparallel curve connecting two points is equivalent to the linearly interpolating m-geodesic, identically to KL divergence. Moreover, the dual symbols
Öpe ae;
pa (a(x) (x)
r (0)
(x)
dx
08 00
08,00
Po 117
vanish when (x)) is linear in 0. Hence the r' autooparall curve from distribution p to is given by the y-power geodesic
)
(+(x)) x (1-
p(x)
((x)
LE [0,
Moreover, the totally - -g-gddessi submanifolds consist of power families (9) and in particular i-distributions with distributions.
vtd The power form can also be recovered as the maximal 7-entropy
Proposition 2. For any statistic sa(x), a maximal y-entropy distribution satisfying the mean constraint Eg-p [su(=)] is of the form of Equation (9).
Proof. Introducing multiplier A, we seek the stationary points of the Lagrangian functional
J(p, A) = p(x) p(a) + / (50(2z) - 4)p(z)dxx.
The corresponding Euler- Lagrange equation (1 + ))((=)" + (so(2)) -1) = 0,
which yields Equation (9) after suitable scaling with s(x) sn()).
14

---

Published as a conference paper at ICLR 2024
We proceed analyze the "Y-power divergence between two t-distributions.
Proposition The y-power divergence from qu ~ ta(/oo Eo, to Pv ta(12, 1) given by Equation (11) when 2 and y
2
v+d
Proof. We first evaluate the power integral
{ 4o (=())po()))dd Rerwg (po( (x)
1 -xx (-)mm))T ET (x: m1)) 1+ v
|Σ||
E Enwq [tr Lr(Ii mi) m1) 11]
Co,a |
+ tr E1 Eo Σx (mo mi)euo - m1)
1
1 |-11-3
|+ tr (Ir Eo) + (m) u1) Ej (m0 ui)
and similarly
/ (u()) m,d Eol -
/ pu(x))tada= Ch,d |Σ (
d
1+
1+
D
V Vー
The y-power entropy and cross-entropy may then be computed
Hglgaw) - 15-() (f (o)()))
-2,
[oo zain
1+ d
+
1/ -
(g = (f go (((()))( (f (()))ttdd)
-
-- 22100
1 + Vー
d 1
(+
1 wー tr [ + (11 p1) E
1
(w0 -
mi)
which combine to give Equation (11).
Proposition 4. 7-power divergence converges to KL divergence as 7 + in the following cases.
1. For any two fixed distributions p, q. lim, (g l1 p) = DKL
2. For qu ta(po, Co, v) and pa ~ ta(p1, Σi, v) with v -2 d and limiting distributions
lx ~Na(Po Σn) and Pso ~ Na(p1, E,), lim (a ll pv) = DKu Poc).
Proof. For may calculate the linearization of C. p) in y via differential coefficients,
lim (g.p)) +1 7-0
1
{
p(x)
((x)) lim
dr
||pl1
7
7
/ ((())
8
p(x)
da
On
l1
1=0
/ (xx) log p(x)dr = C(g.p)
where ((g.)) is the ordinary cross-entropy. The same relation holds for entropy when q. Thus, (g (g) +1 = C(g. p) - H(() = DKL(q p).
lim (g p) = lim 7000
7-0
For 2, it is straightforward to show directly that Equation (11) converges to
Σn
( |1 Poo)
log
d tr (Σ + (po m11) TE (HO m1)
2
Enl
by noting that Co,d (2m2v)- d/2 from Stirling's approximation and 7v -2.
15

---

Published as a conference paper at ICLR 2024
B VVE COMPUTATIONS
B.1 DERIVATION OF JOINT MODEL AND BAYESIAN VIEW
The joint model distribution (13) of VAE is explicitly defined as
(2, 2) Cumm+no
1 +
|| ||=|
- || =a(=))l))
We may retrieve the prior (14) by marginalizing out z, which also confirms pø,, is valid density:
(z) = / po, 2)
= =Cum+no
+
||21|2
11+ 1
1+ 11+11
1 v m
|| - M4 (=)
dx
1ー1 ||
(v + m o)o
1 1 ||=11²
we min
1+ |=
Cotmm
v
1 1-+u1um mm
Cvmm+n Cutmon (1+
m m)
&
1 (1 v ||
CCmm (1+ - 11:113
[1al1
= 4m (m)
where we have used the fact that
M-
1 m (UT)
CCttmnnuum (1+ m)
7
Cumntn T P 2
((V + m)m)
(
Consequently, the VAE decoder is derived
(1/2)) (2,
(z,((z)
-Cummnn
1 + v |21|
- (1+
(1+ 1+ || (v m))
[+
1 1++=-m m || 2=
+ -ー 1m
+ v ||
tn (0(2),
0² P1
1 1m
We now prove that the prior-decoder pair equivalent to the Bayesian hierarchical model (21). By integrating out the latent precision:
Nm
0,
x²
(Alp)dd
x
exp
x
||
//2
A
1
dd
2v
x
11
-
1
||
and
(
0 1
-11-) x x2(11)a
x 2
Nn
x
[=(2); -ー 021 Nm
x
exp
20o
||
1=(=)|12
2v
||=||2
A
15+3+3-14
16

---

Published as a conference paper at ICLR 2024
& 1+ ||]|
|| u=(2)11
VO
1 1 v |
x bn x po(z),
0²
+ m
1 Im
we recover the VAE architecture.
B.2 SHALLOW VVE
We consider the simplest case when the decoder mean [4(22) = W2 + is linear with parameters 0 (W, b). W R b E IR" which call the 'shallow" VAE For completeness, first describe the corresponding shallow Gaussian VAE as well. In this case, the joint model (12) with
[=) WN b is easily checked be normally distributed
(6) WWT +021 W
N Wm+n
and the true posterior can be obtained exactly as
z|æ Nm (WT (WWT o² 12 (s b), WT (WWT + WW)
On the other hand, the joint model (13) for the shallow VAE I-distributed:
("z 2
x (M=D)'
(-
I
x -- b)
x
1
-WT WTW+o'2
021)
2
DO²
x tm+n (0). (WWTT (WW( W
021 1 1)..)
W
Then the true posterior is in fact also t-distributed, but with degrees of freedom + n rather than (Ding, 2016): 2|2 tm (f(x), E(x), v + n) with mean
A(x) = T (WWT + tx b)
and scale matrix
1 +
b) (WWT + -22 b) WT (WWT -
Σ(x)
vn
This motivates the definition of the VAE encoder (16).
B.3 DERIVATION OF L-LOSS
Our goal is derive the 7-power divergence between point (I,, on the model distribution manifold Pv, given by Equation (13), and a point on the data distribution manifold 2,, given by
g (x, 2) = (x)) x tm (2 (p(()) I It Ex()),
We begin by computing the required double integrals. First,
/| pe, (z, 2) 1+ +zddz Eznpz, Eurppo (+12) (x,
= = 1"O Bemmtn1E-rpae Ee-pp. Bansspp....raaaee. Bempa,
1 +
||
+ 02 | -|0--H(<))))
1 1 vー ||
o Cr Eunpz.
1 +
|| 2
y tr
1/ m
1+ 0²1
/ -
= 0 C T EeNpzn
+ ||
n
(1+_ v
||:113 [|=a1p]
シ m -2
17

---

Published as a conference paper at ICLR 2024
= =o "o-""C
v.m+n
1 m
V 2
where we have repeatedly used the fact that
v Eanta(u || wll tr 1o - p) 11)
tr
-- 2
Next, using the entropy computations in the proof of Proposition 3,
/I g0,0(2, 2)1=
/ (f [
Pdata pdata(z) (z) dx
=C4nm (1 (1+ w) 2
I+ vvv+n=2 (1+v+m-2) Ex(x)) Palca( 4 dr.
m
Moreover, we have
/ʃ gowl 2I, 2 = Eurpisaa (emqs.o(11)) (P,,0(x,
- 1 Cum+t In Eamp Ezmgs.o((a) I
1 v
1+
||:1| + oz - - (o(2)]13)
-o C" vmm+n Co Earxpsata 1 V
[1
+ J Ex || no(=))²){
1/ v tr (x)
by utilizing the sum-of-square -f-uuressss decomposition ||~|| = lllb(x) + (2 (=()))) entropy of the joint data distribution is then obtained as
The 7-power
(J) 20,0(1, 2) dedz
147
=
= --City
u+n.m
(1 + 71 20t
+
1 V n2 -
m
(/ES(Ce) 3 Pdata (x) 1 (2)
and the y-power cross-entropy is Cy Pew) =
(// g0 2)po,u(z, 2), ddddz (J pe (( 23 dxdd))
17
-
=- -0 10 10- (,mmtn) ग
+
m +
V
-
C:C²
X Egaepassa
1 +
1
Esnga (1x) || (=)))] + [1la(x))1 +
E
+ - tr En(e))
02
Substituting the above two expressions in the definition of 7-power divergence (8) yields the formula
Ci 7
(/ /|Eg(())
+
C2
Dy po,))
(1)ta(x))11 dxr
7
x Expopaata
1+ V Bengenn mo(2))1 [1)((=))
1 (1 (g)
V
tr Io(er)
We now provide justification for the approximation
( [ Paata (x) 1 dx
+
スमन (x) H (pdata) '?
which is valid up to first order when hy < 1. Note that the linear coefficient H(Paata) is independent of parameters and o, leading to the -looss (18) after rearranging constants:
(0,0,) = 2C2
7V D9 po..)
7wC1,
H(Pdatu)
2C2
2
18

---

Published as a conference paper at ICLR 2024
Proposition 5. Let a be any positive continuous finntion,, Y (-1, and suppose the values
Hz. Hy,.e = Pdata (1)) +47 [log (x)ta((() da
are finite for each j {0, 13, k (1,2). Then for any compact set f2 C supp Pdatas
(1 (x)) "patta (2x) tx
-A1 (v)) [1)
147 panta (r))dr
=7 Paata(a)) log Pdata Pa (r) + O(72).
Remark. The assumption a is valid since variance is typically trained by setting log Eg(x) as neural network. The integrability condition on Pdata is slightly stronger than finite entropy and holds for e.g. d-dimensional Gaussian and t-distributions (when
and =-
Proof. Denote by min max the minimum and maximum of on Ω respectively. We begin by linearizing the first term. Set h(x)= == (<x) Pdata For each x E Q, we may Taylor expand
1 n² (log h(x))( h())))
h(x)7 + y log h(x) +
for some 70(x) E (7, 0), and the integral of the remainder is bounded since
(log)()))²((a)) (x))
Paaaaa (r)drr
(log (daa (1) + Santheuy (log h(z)))((())) Paaaa()da
< Jan(k21)
(log Paata log min) (amin))Paaaa P'Paaaa (x)dr
1 Jan(b<e1) (log Pdata (x) - loog Fmax Pdaata (P)ata(x) (
+ omix
< (log minu) )2 2 [log mmin Ho, 1 + Ho,2 + omax mlax))maxx² + 2 flog mmax H1,1 + H1,))
00,
Therefore, obtain the approximation
(Js ((x)) (z) T =1 =1+7 / log h(x)) Paata (x)dd 0(72)-
For the second term, setting h(x) = (x)) and linearizing with respect to = 2 1ty,
we have
[ ((xx) (I)dr= = (x)dd=1-1y L// log (zz) (x) + 0(72)
with the remainder bounded by 1/2/2- max{ (log mmax)? (log mmin)² max mlnn Plugging in ? + O()2) and subtracting both sides gives the desired approximation.
B.4 DERIVATION OF y REGULARIZRZRRR
We compute the 7-power divergence from the encoder distribution (16) the alternative prior (²) Im (=]0,~~1,, n2),
where the constant is yet to be determined. By Equation (11), have for 2+m=+n
(9 ll
(1 (1+ v n -
(
m
1
Ex(x))
171
v+n.m
n2
2mm
(1m((2))2
1/
tr (I) + + n0)
V/ ++
1- n -
19

---

Published as a conference paper at ICLR 2024
Pz(21)
po 22(22221)
po(ar 21, 22
22
20(21/2)
22(22/0r 21),
Figure B1: Structure of hierarchical VAE.
Note that the coefficient of tr (x) relative the || term is レイーーク aligning with the y-loss formula (18). remains choose T to match the coefficient
2C C2 of the [x)())] शाीन term:
(1+ 1TS
1
nI
2:147
1
m
vC
C2
=- V 2
n
(1
n2
2m
Cy v.tam
V + m + n2 2
V 2
0"
Cr
2,mtt
T
from which we obtain
"Cu,
o
1 (1+
y In
U 2
Furthermore, the precise multiplicative factor required to match the y-loss is (again looking at the term)
1
cib utnmm
m
x
1+
2
?
V + n 2
+
71
1+
m
1-
12
2015)
σ
Cr
(1+--
[1 v
2
+ n 2
V
7/U
+
1+ v-2
2
TV
2C
which is positive, implying that ( ll pt) indeed acts as regularizer. Finally, the additive constant in Equation (20) is equal to 2 (v + n)?2.
B.5 CONSTRUCTING THE SVVA
We develop the heavy-tailed version of the two-level hierarchical VAE (Sonderby et al., 2016) with latent variables (21, 22) E MR depicted Figure B1. In the generative phase, 21 sampled from the initial prior Pz.,, then 22 is drawn from the conditional prior (0.=(=2|=11) and ar is drawn from the decoder pa, (x| 22). the encoding phase, 21 and ž2 are successively drawn from the hierarchical encoders q (r) and (6. x, 21), respectively. We remark that the second level decoder and encoder can also depend through skip connections on 2j and æ. Moreover, we assume two levels only for simplicity, and HVAE can be readily expanded to any hierarchical depth.
The Bayesian scheme (21) naturally extends to
X (Av,) 21 Nm,, (2110,
22/21, A Nn, (=21G oo)))
of)), x/21, X (2 22), Vx 10
which likewise to the derivation in Appendix B.1 integrates to give the t-priors and t-decoder: pz.v (2,) Lm, (= I, I,P),
20

---

Published as a conference paper at ICLR 2024
vー 1+v1
1+
(2,2(22(21) lmg
22 Co(21);
0x v + m1
1 v- m
1 - 1| 1 0= 20=21110 2|21 (o(2))
(r, 22) in
|o(21, 22),
L.I. v mi + m²
+ v (ml + m2)
We also define the t-encoder distributions as
(2.0(21(2)) limi 21 (o(x), 1- ,a(z),
q0,u (22 tmo
22 (2((z
21( my +n)
1+v- + n)
Defining -+ma-ma+t the joint model distribution is derived as
(e, Z1; =2) P2, -P2,u(z1)Po. taa 22)
1 I+ v ||2111
CCym
m2
1+ 1 1
22 (a(2))1
X Cu4mi.ma
0²
1+ va?
14 v 1mmmm
+ 1 || 1?
1+ ||21| + 0. 1| (o(21)]1
X Cu4mi +migon
+ wー (m² + m22)
| 140(22)l )|
x 1 vo + v-1 |221|
+ v- 0= 11 (o(21))1
= Cuym Co+mi Cutmi +m2
m2 0
(1+
m1 m m
1
mi + m2
v
v
at +m
|21l2
|| C0(=1)11
| - u0(=1, 22)11²
1
vo?
vo
1|211
1| Co(=1)1? vo?
||x - u0(21, 22)11 20)1()
7
Cu.my +
m2 o
1+
vo
We proceed to derive the 7-power divergence between Pa, (I, 21, 21, 22) and the data distribution 2o ø1, 22) = 4d =1 (o), |x) [a)) analogously to Appendix B.3.
For the y--etropy of the joint data distribution, Proposition needs to be applied twice, first with respect to Paata (x)) then with respect to go (z.kx)
H. Hl, = (J] (x, 21, 22) dz2d2,dx
#
it
2o (22ll /2, 22) 41) (2.((1)2)) 147 (x) 1+7
dzj dx
( mi +
m22
1 11
Cutmytumm
V my
-2
=
(J (1, 20.0(11 "" +77
x
21

---

Published as a conference paper at ICLR 2024
(s( [x( (2)0(21)))) (2) ttd
x -
Pdata(x)dxx - H H(Peata) 7)
01 / /// [2x(2, 21)1 भमंग qe (21(2)d22
H H9 7) Piiasa (2)dz const.
E 21) Nमंक) log [As(x)) }
+ const.
Note that a new order O(7) correction term log ( (r) appears in the inner approximation due to the dependency of the differential entropy of qs. (z)
Next, we evaluate the required integrals in the y-power cross-entropy,
II] (o, 21, dzdzzdzi = Barpp, E Begrpu (ellt))BBrrrr Berpo [po, (2, 21, 22)
1+ || ||
+ || - (0(3)1
= = -0mm1-22ta
0 "" Bimpp, E P2
vo?
1/ m2 + m2
1 v || 2
+ v tw'oo 00, 11:2 - (0(21)ll 0²! 02!
tr
V m1 + 1112 -2 -2
1 x (mi + m2)
+ my m2 + 2
= = - m
mm oz
ym
V + mi + m2 -
1 | 2| (z12))
x Bei (earpo.o((1a))
vo?
m11 + 12 + ni 2
C - tn 2112
1
Furthermore,
J|] (0, 21, ( 21, dzzdzzde
Exeppana Esrngg ( Beanga
21, 22]
0m yma o
+ ||21| +
|| (0(2))11
+ | - m0 21
x Epcepata E a (1 ( Esg an a (1)
mo?
vo
= +mg -m
7m 0 - E Exapeann 1+ 1
[KKo(a)112
1 ツ+n-2 tr As(x)
v
+ yo?
E. no (112)
[Ima(1, (0(21)11++
v
tr 2x
21
11 V mi + -
+ y02
E us ( 11 (m))) || - Ma(21,
Therefore, the 7-power cross-entropy obtained as Cr(gs.,sP..))
(o, 21, 22), 21, 22)" dezdzzd dx (JJ)
-177
(1, 21, 22) +7 drdzzdez
1
+ m2 + n
(C) (Cr.my+my+a 0F 2 )
1
2
m:
X Epeepaata 1+ ++
[K(((0))1 + vtn-2 tr Ap(x)
22

---

Published as a conference paper at ICLR 2024
Bn ((12))
||us( 21) - (o(21)1)++
Ei (25, 21)
v + mi + n
En (leg. | - (01, PO
Putting everything together and tidying constants, finally obtain the y-loss for HVAE,
(0, Ra-paaaa
Bss
|| (0(21,
Pdaaa
d
reconstruction orrer
v
C
+ ||<6(2))11
tr (r)
+
log [As(x)]
n
2C²
regularizer for (0.0(2112)
1 + Ex Ber 0g
V vr Es
vogC
-(((o,z))-G(zzi)))
E5 21) 2049
V + m1 + n 2
O2
regularizer 90. (02/222,
(23)
Similarly to the 7-loss, we see that (, Φ) consists of the reconstruction error and a sum of regularizing terms for each of the first and second level encoders. It is also possible to replace the log [x)( with constant times A a पाो for consistency by converting the order O()) correction term in Proposition from differential entropy to 7-power entropyy
C EXPERIMENTAL DETAILS
All experiments are implemented via Python 3.8. 10 with PyTorch package (Paszke et al., 2019) version 11.l+cull 17, and run Linux Ubuntu 20.04 with Intel Xeon Silver 4114 @ 2.20GHz processors, an Nvidia Titan V GPU with 12GB memory, CUDA 11.3 and cuDNN 8.2. The code available Github.
C.I REPARAMETRIZATION TRICK FOR VAE
Recall that for the Gaussian VAE, given N data points (1) .... (N()), Monte Carlo estimate the observed ELBO is computed using samples 2(111) (i.LL drawn from (-(-)2(())
()
Σ L N
1
log po (2(00))))))) DKL (1)() (lo((())))))lll(((()
In order to backpropagate gradients through stochastic nodes, the reparametrization trick must be used wherein 2(i,)) (z(()))) + (1.) @ (f.f) and (1,2) is sampled from (o, T). Since the t-distribution behaves similarly under affine transformations, this mechanism is simple to implement for our model.
A multivariate t-distribution T ~ talp, Σ, v) may be constructed from multivariate centered Gaussian NN(0,, Σ) and independent chi-squared variable ~ x²) via
Z
T
+
VV/v
We use an encoder with diagonal covariance,
(2(2) = tm (1)))))
1 レー In diag ( +n))
23

---

Published as a conference paper at ICLR 2024
hence may obtain 214,) by drawing e..) Nm (0, and 5(1.)) x2 + n) independently and computing
1 oo(x(0)
2(6,0 = -Mo(x(0)
(86./)/ (1V n vI+u-ly (e(1))
(24)
- -mo(x(0) + V 8(1.8) ) (6.)
C.2 DETAILS ON SYNTHETIC DATASETS
We first generate 200K train data, 200K validation data, and 500K test data from the heavy-tailed bimodal distribution (22). We then construct decoder and encoder networks with a common multi- layer perceptron architecture, which consists of two fully connected layers each with 64 nodes and Leaky ReLU activations. In the training process,
use batch size of 128 and employ the Adam optimizer (Kingma & Ba, 2014) with a learning rate of x 10-3 and weight decay 1 x 10-4. Moreover, adapt early stopping using validation data with patience 15 to prevent overfitting. All models finished training within 80 epochs except VAE-st.
After completing training, we generate 500K samples from each model. The large sample size is necessary in order to accurately plot the far tails. For MMD hypothesis testing, we employ the fast linear time bootstrap test (Gretton al., 2012) with 100K subsamples from each distribution and IK repetitions. For the tail distributions, we employ the same test with appropriate subsampling to match sample sizes.
Moreover, we also compare the reconstruction loss of VAE and Gaussian VAE in Table C1. This comparison is valid since both the ELBO (2) and 7-loss (20) optimize the same MSE loss with different regularizers. We observe that models with smaller v exhibit smaller reconstruction loss, demonstrating the relaxation of regularization discussed in Section 3.2. Smaller losses can of course be achieved by 3-VAE by decreasing 3, which is close to simple autoencoder.
The empty blocks for 3-VAE (3 = 2.0) and VAE-st (v = 15.0) are due to no tail samples being generated at all. Also, the reconstruction loss of Student-t VAE cannot be numerically compared assumes learnable variance o².
C.3 DETAILS ON CELEBA AND IIFAR100-LA
The original CIFARI00 dataset is provided by Krizhevsky (2009). CIFARI00 consists of 60K 32 x 32 color images with 100 balanced classes, divided into 50K train images and 10K test images. The CelebA dataset is provided by Liu et al. (2015; 2018). The dataset consists of 202,599 face images from 10,177 celebrities, which are split into 182,637 training and 19,962 test images following the recommendation provided in the CelebA official documentation. Each image is subjected to uniform center-cropping, specifically by 148 x 148 pixels, followed by downsizing to 64x64 pixels.
All VAE models trained for 50 epochs using batch size of 128 and latent variable dimension of 64 with the Adam optimizer. We tune learning rates separately for each model within the range of 5 10 and present the best result. In particular, good results are obtained when the
5 x 10-5 learning rate is set to 8 x 10-5 ~ 1 x 10-4 in CelebA, and 2 x 10 ~ 4 x 10 in CIFAR CIFAR100-LT In the case of Factor VAE, a separate Adam optimizer is used for the discriminator with a learning rate of x 10
The basic encoder and decoder architectures for most models are based on the public GitHub repository (Subramanian, 2020) with modifications. The encoder network consists of 5 convolutional layers with hidden layer sizes of 128, 256, 512, 1024, 2048. Batch normalization and ReLU activation applied after each convolutional layer. Two fully connected layers for p²(x)) and Eg(x) then flatten the encoder output to the latent space. The decoder network is the reverse of the encoder architecture, with 5 transposed convolutional layers instead. The final part of the decoder is pair of transposed convolutional layers and a tanh activation.
Hierarchical models, HVAE and t HVAE, maintain the encoder and decoder networks of the original model while adding a second encoder (2. jr, 21) and a layer that estimates the mean of the second prior (21) using multi-layer perceptron. The dimension of 2g is set to half that of 21, and skip
24

---

Published as a conference paper at ICLR 2024
Table C1: Full results of MMD test p-values 1-dimensional synthetic data analysis. Rejected values shown in red.
MMD test p-values
Model
Parameter
Recon loss
Full
Left
Right tail
9.0
0.001
0.130
0.580
0.707
12.0
0.026
0.330
0.347
0.744
VAE
15.0
0.008
0.363
0.622
0,831
18.0
0.322
0.377
0.693
0.847
21.0
0.372
0.188
0.838
0.853
VAE
0.514
0.036
0.003
1.028
0.1
0.614
0.011
0.001
0.102
0.2
0.137
0.001
0.032
0.210
8-VAE
= 0.5
0.050
0.007
0.027
0.506
2.0
0.001
Student &-VAE
0.587
0.291
0.114
V 9.0
0.943
0.424
0.814
1.012
12.0
0.068
0.222
0.406
1.018
DE- VAB
V 15.0
0.481
0.046
0.443
0.997
18.0
0.763
0.219
0.411
0.984
V 21.0
0.597
0.376
0.146
0.985
9.0
0.001
0.179
0.460
1.062
12.0
0.001
0.953
0.643
1.057
VAE-st
15.0
0.001
0.363
1.045
V 18.0
0.001
0.925
< 0.001
0.981
21.0
0.001
0.326
0.001
1.094
Table C2: Full results of MMD test p-values of 2-dimensional synthetic data analysis. Rejected values are shown in red.
MMD test p-values
Model
Parameter
Full
Left tail
Right tail
30.0
0.276
0.214
0.213
U
40.0
0.716
0.130
0.001
VAE
50.0
0.766
0.142
0.004
F' = 60.0
0.665
0.060
0.002
V 70.0
0.773
0.100
0.002
VAE
0,116
0.004
0.001
8 = 0.1
0.631
0.001
0.001
3-VAE
8 0.2
0.359
0.001
0.001
8 0.5
0.251
0.001
0.015
Student t- VAE
0.530
0.001
0.001
V 3.0
0.624
0.002
0.057
VAE
V = 5.0
0.672
0.001
0.001
V 7.0
0.452
0.001
0.001
D = 9.0
0.539
0.001
0.001
3.0
0.485
0.001
0.001
VAE-st
D = 5.0
0.092
< 0.001
0.020
7.0
0.250
0.001
0.001
V 9.0
0.156
< 0.001
0.020
25

---

Published as a conference paper at ICLR 2024
connections are implemented by concatenating the connected input with the original input. After training. we load the best model of all epochs and evaluate FID scores for the whole dataset and each class separately.
The complete version of Table 2. In Tables C3 and C4, we provide the complete version of Table 2 including the FID scores with varying hyperparameters. We verify that VAV achieves the lowest FID scores, regardless of v. In addition, we include the FID scores of Hierarchical model at the bottom of Table emphasizing that HVAA consistently yields more favorable outcomes compared to the conventional model.
Table C3: CelebA FID scores for overall and selected attribute classes with varying hyperparameters for each model, including Pale Skin (Pale) and Double Chin (Chin), We also include the Gaussian HVAE and HVVA reconstruction scores.
Framework
Parameter
Overall
Bald
Mst
Gray
Pale
Chin
Bd
Young
v =
39.4
66.5
61.5
67.2
54.6
57.2
40.1
46.7
y 5
40.2
67.5
62.6
68.8
55.1
58.1
40.8
47.8
VVA
v 2.5
40.0
67.6
62.0
68.0
55.8
57.7
40.7
47.2
V = 2.1
40.4
68.5
62.5
68.0
55.8
58.4
41.0
47.6
K = 1
57.9
85.8
79.7
91.0
70.8
78.2
58.4
69.1
VAE
K 1.5
73.2
105.3
96.4
114.5
85.3
97.5
73.8
84.1
8 0.5
50.9
81.0
72.4
84.3
66.0
71.6
51.4
60.1
3-VAE
3 = 0.25
46.7
76.5
69.3
79.1
60.7
66.9
46.9
55.7
3 0.1
42.1
72.1
65.3
74.8
56.1
62.4
42.5
51.3
3 0.05
40.4
69.3
62.7
71.1
55.
59.8
40.9
49.0
Student-f VAE
78.4
112.0
104.2
118.7
91.7
100.7
78.6
88.9
V =
59.7
89.2
83.8
95.6
73.8
81.4
59.9
69.4
V 5
58.9
89.6
84.3
94.9
72.8
80.2
59.1
68.9
DE-VAE
V 2.5
59.0
84.6
81.3
93.1
72.1
78.2
59.4
68.1
V = 2.1
59.1
90.5
83.7
94.5
72.9
80.9
59.4
69.1
T 20
49.5
79.8
73.3
82.1
63.8
70.3
49.9
59.3
30
45.9
75.1
69.5
77.1
60.6
65.4
46.3
54.7
Tilted VAE
T =
45.0
74.4
68.1
76.6
59.9
65.1
45.2
54.1
F 50
42.6
73.0
65.4
73.7
57.8
62.1
42.9
50.8
Ne 20
61.2
96.7
88.7
100.1
77.1
84.8
61.3
71.8
7 10
60.8
87.2
101.2
75.6
85.5
60.8
72.4
Factor
Nie 5
59.8
91.7
85.7
95.2
74.2
81.5
59.9
70.0
Nhe 2.5
60.8
92.5
86.6
96.8
73.6
82.5
60.8
71.1
HVAE
56.9
85.4
78.9
88.1
68.0
89.1
56.7
67.2
V
37.5
65.9
59.6
67.7
52.8
56.1
37.8
45.2
РHVAE
V = 5
37.0
65.4
59.3
67.2
52.6
55.9
37.4
44.5
V 2.5
36.7
64.6
58.6
66.1
52.5
54.9
37.1
43.8
Comparing 3-VAE and VAE. For more accurate comparisons between 3- and VAE in CelebA generation, we vary 3 within a range from 0.001 to 1. We again note that the performance of (VAE is not sensitive v, fine-tuning is required. Figure C2 shows that 3- performs best when 3 is set to 0.05 and becomes less effective as 3 deviates further from 0.05.
This aspect is also considered in the reconstruction experiment design. If 3 is to ever smaller values, the reconstruction results may appear to improve over other models. However, this is because 3-VAE essentially regresses raw autoencoder as 3 - 0, so that while reconstruction performance can be improved, generation performance will simultancously deteriorate, We therefore do not tune for extremely small values of B.
26

---

Published as a conference paper at ICLR 2024
Table C4: FID scores (Total data and lower 10% of tailed class) on CIFAR100-LT with varying imbalance factor p.
Framework
Parameter
p = 1
p = 10
p =
50
p = 100
Total
Rare
Total
Rare
Total
Rare
Total
Rare
2.1
100.0
142.2
101.2
146.2
123.5
166.2
130.2
170.8
2.5
98.5
140.9
100.4
144.8
128.5
170.7
130.0
172.1
VAE
= 5
100.2
141.8
103.8
147.7
126.8
171.3
131.0
173.7
10
97.5
140.9
102.8
147.6
128.5
170.8
128.7
170.9
256.1
305.7
267.2
315.5
277.4
332.7
287.3
342.3
VAE
1.5
274.2
298.9
290.5
313.1
296.7
318.9
297.7
319.6
0.5
179.5
210.2
198.2
230.8
214.6
245.8
223.7
252.9
3-VAE
0.25
142.1
178.0
160.0
197.7
169.2
204.5
182.5
217.5
8 0.1
114.0
154.7
130.4
170.8
138.5
178.4
160.6
199.0
Student-t VAE
259.5
315.8
314.1
353.1
323.7
364.1
333.4
374.3
2.1
230.4
258.5
250.0
280.0
253.0
282.7
254.4
285.6
219.4
250.0
250.2
281.6
256.7
285.
258.5
303.2
DE-VAE
5
232.6
260.9
252.9
284.6
252.0
281.3
269.7
306.7
10
230.9
262.0
250.5
278.0
258.0
283.6
272.4
303.5
T 20
113.8
154.1
131.2
172.6
149.6
189.5
181.8
221.8
30
104.4
146.1
124.1
166.5
143.3
184.5
179.1
220.6
Tilted VAE
7 40
101.0
142.5
123.9
167.4
179.1
220.6
172.4
223.6
50
101.0
142.8
126.1
168.4
147.0
187.8
193.2
229.7
The 20
238.8
266.2
277.6
322.8
273.8
301.8
269.1
297.8
Tto 10
240.4
268.6
273.5
324.3
270.5
298.1
270.2
298.2
PactorVAE
5
232.3
263.3
272.5
323.6
275.6
306.3
270.1
296.7
Itc 2.5
236.0
264.4
275.7
328.9
269.2
298.2
269.8
297.8
betaetE
Frumework
Parameter
FID
Framework
Parameter
FID
50.6
e = 10
62.8
60-
50.9
61.6
PVAR
DE-VAE
2.5
50.6
2.5
58.9
P
50.7
F 2.1
60.4
0.5
58.4
VAV
0.25
54.7
Tilted VAE
30
59.2
0.1
52.4
T 40
60.8
0.05
79.6
Ne 2.5
67.1
VAE
1.5
64.7
FactorVAR
=
68.0
% 73.1
Student-2 VAE
82.3
0.11
0.02
0.03
8:04
0.03
5.05
0.2
0.23
20
78.6
beta
Figure C2: Generation FID scores against values for 3-VAE.
Table C5: CelebA FID scores for overall classes with varying hyperparameters for each model.
27


Reviews:
[{"review_id": "EB3crDuURD", "reviewer": "Reviewer_fEQe", "summary": "The authors develop a new variational autoencoder designed for heavy-tailed data by changing the prior, encoder, and decoder from Gaussian distributions to t-distributions, and the KL divergence to power divergence. The authors draw upon the EM perspective of VAEs developed by Han et al., 2021 to formulate a joint minimization objective, and use ideas from information geometry to tie together the exponent in the power divergence with the degrees of freedom in the t-distributions. The degrees of freedom hyperparameter is shown to affect the degree of regularization in the model. The model formulation is also described in terms of a Bayesian point of view. Four experiments are conducted comparing the performance of the proposed model with a selection of other competing VAEs: the first involving a univariate synthetic dataset (reporting histograms and MMD test p-values), the second involving a bivariate synthetic dataset (reporting MMD test p-values), the third involving the CelebA image dataset (reporting FID scores), and the fourth involving the CIFAR100-LT image dataset (reporting FID scores). In all cases, the model is shown to vastly outperform competing methods. All arguments are supported with ample theoretical derivations in the supplementary material.", "questions": "- Is the gamma-power divergence not the same as the Renyi divergence (up to constants)? How do they differ?\n- Can you provide an algorithm environment to show how this should be implemented at a glance? I appreciate the presentation and the derivation of the model, but any reader looking to quickly implement it is likely to have trouble if they do not thoroughly read the paper. \n- What do the MMD values themselves look like? These might be more useful to report than the p-values.\n- Since you know the densities explicitly in the synthetic tests, you could use KSD instead of MMD, as this will have much improved statistical power. Have you tried this?\n- Have you tried the model on any real datasets other than image sets?\n- Do you know how the model compares with other normalizing flow models incorporating t-distributions?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- An interesting and important extension on the VAE framework to meaningfully deal with heavy-tailed data.\n- A broad investigation and discussion into the underlying concepts, providing an excellent derivation of the method, and a good amount of detail extending into the appendices. \n- Hyperparameters and model choices are thought out and well-justified. The authors could have simply attached a few of the underlying ideas together without much thought, but have chosen to go the extra mile. \n- Model is shown to be highly effective compared to competitors, even on reasonably challenging image datasets where VAEs do not typically do well.\n- Very well-written paper; no grammatical issues or typos that I could detect.", "weaknesses": "- Reporting p-values is not exactly ideal, especially when metrics are available. \n- No general summary to assist with implementation.\n- No examples conducted on datasets where heavy tails are known to be especially relevant (e.g. economic datasets).\n- No discussion of, or comparisons to, similar developments in the normalizing flow literature (e.g. [1] and [2]).\n\n[1] Jaini, P., Kobyzev, I., Yu, Y., & Brubaker, M. (2020). Tails of Lipschitz triangular flows. In International Conference on Machine Learning (pp. 4673-4681). PMLR.\n\n[2] Liang, F., Mahoney, M., & Hodgkinson, L. (2022). Fat–Tailed Variational Inference with Anisotropic Tail Adaptive Flows. In International Conference on Machine Learning (pp. 13257-13270). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AAVREziQKt", "reviewer": "Reviewer_5yMY", "summary": "The authors address the limitation of the standard Variational Autoencoder (VAE) that employs a Gaussian prior which sometimes fails to capture the intricate structures in data due to its fast-decaying tails. They propose a novel framework by leveraging the heavy-tailed nature of the Student's t-distributions for the prior, encoder, and decoder. They argue that this new formulation can better fit real-world datasets.  Empirical results suggest that t3VAE performs better in modeling low-density regions, and yields superior performance on the CelebA and imbalanced CIFAR-100 datasets.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The use of Student's t-distributions as an alternative to the Gaussian prior in VAEs seem a fresh perspective. The heavy-tailed nature can indeed address some of the shortcomings of the standard Gaussian prior.\n\n- The modification of the evidence lower bound to incorporate γ-power divergence might seem appropriate given the nature of the power families.\n\n- Demonstrated superior performance on benchmark datasets, particularly on imbalanced/ longtailed version of CIFAR-100, which is a testament to the proposed model's robustness.", "weaknesses": "- The transition from Eq(5) to Eq(19) is not straightforward for me. While intuitively, adopting the γ-power divergence to the KL-divergence might seem appropriate, a central question arises: Does the γ-power divergence between the two manifolds still lead to the ELBO? This needs to be addressed for better clarity.\n\n- Given that both t3VAE and the Student-t VAE integrate the Student's t-distribution into the VAE framework, a more granular comparison in the related work section would be enlightening. Highlighting the distinct features and advantages of t3VAE over the Student-t VAE would give readers a clearer understanding of its contributions.\n\n-  I noticed that the hierarchical variants of the model weren't evaluated in the experimental section. Such an evaluation might provide insights into the model's performance in different configurations. I'd be keen to see these results.\n\n- Incorporating the Student's t-distributions inherently increases the model's complexity. This can potentially introduce challenges in training and optimization. I recommend the authors delve into these potential challenges, discussing potential remedies and considerations for practitioners aiming to implement t3VAE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sHlthYNJhQ", "reviewer": "Reviewer_1fkZ", "summary": "In this paper, the authors introduce a novel VAE-like generative model ($t^3$VAE) in which the underlying distributions (for the prior, encoder, and decoder) are assumed to be (multivariate) Student's t. This idea leads to a better approximation of heavy-tailed densities, which is confirmed by experiments on both synthetic and real-world datasets. The presented (solid) theoretical justification stems from viewing classical VAE as a joint minimization process between two statistical manifolds and relies on replacing the KL divergence with the $\\gamma$-power divergence.", "questions": "(1) Have you considered providing experimental results for hierarchical architectures?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The idea behind $t^3$VAE (although somewhat natural) seems original and significant since it allows us to overcome (to some extent) the limitations of classical VAE.\n\n(2) The proposed solution has (as the authors show) a solid theoretical background.\n\n(3) The experimental results prove the superiority of $t^3$VAE over the state-of-the-art.", "weaknesses": "(1) The authors claim that their idea is extendable to hierarchical models. Although there is a theoretical justification for this in the appendix, the paper would benefit from corresponding experimental studies.\n\n(2) Minor comments:\n\np. 2, l. 10 from bottom: the authors probably wanted to write \"likelihood\" (instead of \"log-likelihood\"),\n\np. 7, Tab. 3: $t^3$VAE $\\to$ $t^3$VAE ($\\nu=10$),\n\np. 9, l. 12: I suggest not to use the phrase \"scores highest\" (as in the case of FID \"lower is better\").", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)