Trading Information between Latents in Hierarchical Variational Autoencoders
Authors: Tim Z. Xiao
Robert Bamler
Venue: ICLR 2023 poster (2023)
Keywords: VAE
hierarchical VAE
rate distortion theory
information theory

Abstract: Variational Autoencoders (VAEs) were originally motivated as probabilistic generative models in which one performs approximate Bayesian inference. The proposal of $\beta$-VAEs breaks this interpretation and generalizes VAEs to application domains beyond generative modeling (e.g., representation learning, clustering, or lossy data compression) by introducing an objective function that allows practitioners to trade off between the information content ("bit rate") of the latent representation and the distortion of reconstructed data. In this paper, we reconsider this rate/distortion trade-off in the context of hierarchical VAEs, i.e., VAEs with more than one layer of latent variables. We propose a method to control each layer's contribution to the rate independently. We identify the most general class of inference models to which our proposed method is applicable, and we derive theoretical bounds on the performance of downstream tasks as functions of the individual layers' rates. Our experiments demonstrate that the proposed method allows us to better tune hierarchical VAEs for a diverse set of practical use cases.

Published as a conference paper at ICLR 2023
TRADING INFORMATION BETWEEN LATENTS IN HIERARCHICAL VARIATIONAL AUTOENCODERS
Tim Z. Xiao
Robert Bamler
University of Tübingen IMPRS-IS
University of Tübingen robert wam.leeenii-tubingee
zhenzhong wia.euni-ttebingeeeeee de
ABSTRACT
Variational Autoencoders (VAEs) were originally motivated (Kingma & Welling, 2014) probabilistic generative models in which one performs approximate Bayesian inference. The proposal of 3-VAEs (Higgins et al., 2017) breaks this interpretation and generalizes VAEs to application domains beyond generative modeling (e.g., representation learning, clustering, or lossy data compression) by introducing objective function that allows practitioners to trade off between the information content ("bit rate") of the latent representation and the distor- tion of reconstructed data (Alemi al., 2018). In this paper, we reconsider this rate/distortion trade-off in the context of hierarchical VAEs, i.c.. VAEs with more than one layer of latent variables. We identify a general class of inference mod- els for which one can split the rate into contributions from each layer, which can then be tuned independently. We derive theoretical bounds on the performance of downstream tasks as functions of the individual layers' rates and verify our theoretical findings in large-scale experiments. Our results provide guidance for practitioners which region in rate-space to target for given application.
INTRODUCTION
Variational autoencoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) are class of deep generative models that are used, e.g., for density modeling (Takahashi et al., 2018), clustering (Jiang al., 2017), nonlinear dimensionality reduction of scientific measurements (Laloy al., 2017), data compression (Ballé et al., 2017), anomaly detection (Xu et al., 2018), and image generation (Razavi al.. 2019). VAEs (more precisely, 3-VAEs (Higgins et al., 2017)) span such diverse set of application domains in part because they can be tuned to a specific task without changing the network architecture, way that is well understood from information theory (Alemi al., 2018).
The original proposal of VAES (Kingma & Welling, 2014) motivates them from the perspective of generative probabilistic modeling and approximate Bayesian inference. However, the generalization to 3-VAEs breaks this interpretation they are longer trained by maximizing lower bound on the marginal data likelihood. These models are better described as neural networks that are trained to learn the identity function, i.e., to make their output resemble the input closely possible. This task made nontrivial by introducing a so-called (variational) information bottleneck (Alemi et al., 2017; Tishby& & Zaslavsky, 2015) one more layers, which restricts the information content that passes through these layers. The network activations at the information bottleneck are called latent representations (or simply "latents"), and they split the network into an encoder part (from input to latents) and a decoder part (from latents to output). This separation of the model into an encoder and decoder allows to categorize the wide variety of applications of VAEs into three domains:
data reconstruction tasks, i.e., applications that involve both the encoder and the decoder; these include various nonlincar inter- and extrapolations (c.g., image upscaling, denoising. or inpainting), and VAE-based methods for lossy data compression:
2. representation learning tasks, i.c.. applications that involve only the encoder; they serve a downstream task that operates on the (typically lower dimensional) latent representation, e.g., classification, regression, visualization, clustering, or anomaly detection; and 3. generative modeling tasks, i.c., applications that involve only the decoder are less com- mon but include generating new samples that resemble training data.

---

Published as a conference paper at ICLR 2023
codle
comrapunding 18:55
3.5
ur suin forattor 100---0000
2000
recomitruuti (PSNR)
2. rpp learning (socurscy)
generacion (15)
2. 2 S recomiii Incotttattion performance
(PSNR)
genorative modelnng perfomance
(incpptio score (15))
Jkg Ag)
Figure 1: Left: trade-off between performance in the three applications domains of VAEs, using GHVAE trained on the SVHN data set (details: Section 5) higher is better for all three metrics; gray dots on walls show 2d-projections. Right: color code, corresponding layer- rates (Eq. 7), and individual performance landscapes (size of dots x performance). The hyperparameters 32 and $1 allow us to tune the HVAE for best data reconstruction (4), best representation learning (o), or best generative modeling (O). Note that performance landscapes differ strongly across the three applications, and neither a standard VAE (32= 31 = 1; marked " in right panels) nor a conventional 3- (32 31; dashed red lines) result in optimal models for any of the three applications.
The information bottleneck incentivizes the VAE to encode information into the latents efficiently by removing any redundancies from the input. How agressively this is done can be controlled by tuning the strength 3 of the information bottleneck (Alemi et al., 2018). Unfortunately, information theory distinguishes relevant from redundant information only in a quantitative way that agnostic the qualitative features that each piece of information represents about some data point. In practice, many VAE-architectures (Deng et al., 2017; Yingzhen & Mandt, 2018: Ballé et al., 2018) try to separate qualitatively different features into different parts of the latent representation by making the model architecture reflect some prior assumptions about the semantic structure of the data. This allows downstream applications from the three domains discussed above to more precisely target specific qualitative aspects of the data by using or manipulating only the corresponding part of the latent representation. However, in this approach, the degree of detail to which each qualitative aspect is encoded in the latents can be controlled at most indirectly by tuning network layer sizes.
In this paper, argue both theoretically and empirically that the three different application domains of VAEs identified above require different trade-offs in the amount of information that is encoded in each part of the latent representation. We propose method to independently control the information content (or "rate") of each layer of latent representations, generalizing the rate/distortion theory of 3-VAEs (Alemi et al., 2018) for VAEs with more than one layer of latents ("hierarchical VAEs" or HVAEs for short). We identify the most general model architecture that is compatible with our pro- posal and analyze how both theoretical performance bounds and empirically measured performances in each of the above three application domains depend on how rate is distributed across layers.
Our approach is summarized in Figure 1. The 3d-plot shows empirically measured performance metrics (discussed in detail in Section 5.2) for the three application domains identified above. Each point on the colored surface corresponds to different layer-wise rates an HVAE with two layers of latents. Crucially, the rates that lead to optimal performance are different for each of the three application domains (see markers A, O, and in Figure 1), and none of these three optimal models coincide with conventional 3- 3-VAE (dashed red lines in right panels). Thus, being able to control each layer's individual rate allows practitioners to train VAEs that target a specific application.
The paper is structured as follows. Section 2 summarizes related work. Section 3 introduces the proposed information-tradin method. We then analyze how controlling individual layers' rates can be used to tune HVAEs for specific tasks, i..., how performance in each of the three application domains identified above depends on the allocation of rates across layers. This analysis is done theoretically in Section 4 and empirically in Section 5. Section 6 provides concluding remarks.
2

---

Published as a conference paper at ICLR 2023
p(za)
p(z))
p(zo)
22
22
ao(2sin) au
Po (2)
[2((21)22)
au(aikaa
Z1
Z1
po(xiz:)
(x(z)))
(a) bottom-up
(b) implicit top-down (e.g.. LVAE)
(c) generalized (explicit) top-down
Figure 2: Inference (dashed arrows) and generative (solid arrows) models for hierarchical VAEs (HVAEs) with two layers of latent variables. White/gray circles denote latent/observed random variables, respectively; the diamond dj in (b) is the result of a deterministic transformation of x.
RELATED WORK
We group related work into work on model architectures for hierarchical VAEs, and on B-VAEs.
Model Design for Hierarchical VAEs. The original VAE design (Kingma & Welling, 2014;
Rezende et al., 2014) has a single layer of latent variables, but recent works (Vahdat & Kautz, 2020; Child, 2021), found that increasing the number of stochastic layers in hierarchical VAEs (HVAEs) improves performance. HVAEs have various designs for their inference models. Sønderby et al. (2016) introduced Ladder VAE (LVAE) with top-down inference path rather than the naive bottom- up inference (see Section 3), whereas the Bidirectional-Infereme uses combination of top-down and bottom-up. Our proposed framework applies a large class of inference models (see Section 3) that includes the popular LVAE (Sønderby et al., 2016).
VAE (BIVA) (Maaløe et al., 2019)
Interpretations. Higgins et al. (2017) introduced an extra hyperparameter 3 in the objective of VAEs that tunes the strength of the information bottle- neck, and they observed that large / leads to disentangled latent representation. An information- theoretical interpretation of 3- BAEE was provided in (Alemi et al., 2018) by applying the concept of (variational) bottleneck (Tishby & Zaslavsky, 2015; Alemi et al., 2017) autoencoders. Due interetation, BAVAES are popular models for data compression (Ballé al., 2017: Minnen al., 2018; Yang al., 2020), where tuning allows trading off between the bit rate of compressed data and data distortion. In the present work, we generalize 3 B-VEEs when applied to HVAEs, and we introduce framework for tuning the rate of each latent layer individually.
3-VAEs And Their Information-Theoreel
to this information-theoretie
3 A HIERARCHICAL INFORMATION TRADING FRAMEWORK
We propose refinement of the rate/distortion theory of 3-VAEs (Alemi et al., 2018) that admits controlling individual layers' rates in VAEs with more than one layers of latents (hicrarchical VAEs).
3.1 CONVENTIONAL 3- VAE WITH HIERARCHICAL LATENT REPRESENTATIONS
We consider a hierarchical VAE (HVAE) for data with L layers of latent representations { Figure 2, discussed further in Section 3.2 below, illustrates various model architectures for the exam- ple of L 2. Solid arrows depict the generative model pe { æ), where 6 are model parameters (neural network weights). We assume that the implementation factorizes pa (zz), æ) as follows,
p=((z1}, a) - po (z).) po (z( 22 2/. 21.) po (2) [222) po (0 1
where the notation . for any is short for the collection of latents { \/2 (thus, ボ>1 and {zt) are synonymous), and the numbering of latents from L down to 1 follows the common convention in the literature (Sønderby et al., 2016; Guirajani et al., 2017; Child, 2021). The loss function of
3

---

Published as a conference paper at ICLR 2023
normal 3-VAE (Higgins et al., 2017) with this generic architecture would be Cp ④) Eanin ((s((==)) xe)
log po (axl(==)))) +-3 DKL. [go (zc) x)) po [
(2)
"distortion"
"rate" R
Here, (o((ze} a is the inference (or "encoder") model with parameteres Φ, Xaain is the training divergence, and the Lagrange parameter 3 > trades off
set, DKLL-11 denotes Kullback-Leibbe between a (total) rate R and distortion D (Alemi et al., 2018). Setting = 1 turns Eq. 2 into the negative ELBO objective of a regular VAE (Kingma & Welling, 2014). The rate R obtains its name as it measures the (total) information content that 4s encodes into the latent representations {zz), which would manifest itself in the expected bit rate when one optimally encodes a random draw {zz} ~ qo(( ((=) 2)) using po (=z)) an entropy model (Agustsson & Theis, 2020; Bennett al., 2002). An important observation pointed out in (Alemi et al., 2017) is that, regardless how rate R traded off against distortion D by tuning 3. their sum R + D is in expectation under any data distribution Pdata (x) allysss lower bounded by the entropy H (x)] == E pp) log Pdatal (x)),
Pen (a)) R + DJ 2 H [Pdata
V Pdata-
(3)
The rate R in Eq. 2 is property of the collection {ze} of all latents, which can limit its interpretability for some inference models. For example, the common convention of enumerating layers ze from f = L down to in Eq. 1 reminiscent of a naive architecture for the inference model that factorizes in reverse order compared to Eq. 1 ("bottom up", see dashed arrows in Figure 2(a)), i.e. (f({))] x) qo 21 xz) go (2) ..- qo (122/=zz-1 Using HVAE with such "bottom-up" inference model to reconstruct some given data point æ would map æ to 2 using 2o z
Limitations.
(æ>) and then map 21 back to the data space using po (=t(z)), 1). thus ignoring all latents Z₁ with & > 1. Yet, the rate term in Eq. 2 still depends on all latents, including the ones not needed to reconstruct any data (practical VAE-based compression methods using bits-back coding (Frey & Hinton, 1997) would, however, indeed use ze with E >>> 1 as auxiliary variables for computational efficiency).
3.2 TRADING INFORMATION BETWEEN LATENTS
Many HVAEs used in the literature allow us to resolve the limitations identified Section 3.1. For example, the popular LVAE architecture (Sønderby et al., 2016), (Figure 2(b)), uses an inference model (dashed arrows) that traverses the latents { in the same order as the generative model (solid arrows). We consider the following generalization of this architecture (see Figure 2(c)),
{0<{=>} } æ) = go (x) go (2 æ1, x)) go 11-1, Z1, x) .... (z 2222 x). (4) Formally, Eq. 4 is just the product rule of probability theory and therefore holds for arbitrary in- ference models go (fzz) z)) More practically, however, we make the assumption that the actual implementation of (=<)) (x) follows the structure in Eq. 4. This means that, using the trained model, the most efficient way to map a given data point æ to its reconstruction & now involves all latents ze (either drawing sample taking the mode at each step):
(=)
(=L 1|22.0)
To (21
2L
2L-1
Z2
(5)
Layer-wise Rates. We can interpret Eq. 5 in that it first maps x to a "crude" representation æl, which gets iteratively refined to 21. and finally to reconstruction â. Note that each factor (zz R2e+1, æ) of the inference model in Eq. is conditioned not only on the previous lay- Zz441 but also the original data œ. This allows the inference model to target each refinement step in Eq. 5 such that the reconstruction & becomes close to æ. More formally, we chose the infer- ence architecture in Eq. 4 such that it factorizes over {zr] in the same order the generative model (Eq. 1). This allows us to split the total rate R into a sum of layer-wise rates as follows,
R (=Bga((z))le) log (x(=)(a)) log (0(=1-1/z1,0)) (z1. R((z)) + ((Z))11=1) + R(zL-2 21 + ...
2 ( (212222
log po (21
po(21,-1/21)
(6)
Here,
R(zL) = DKL (go(z1 x)) [o()))] and
[ee]Drr quantify the information content of the highest-order latent representation 2,. and the (expected)
R(*1=2041) = Rg [a)] [DLL] [go(ze 22641, 2)) po [2(+1)]]
increase in information content in each refinement step 21+1 -> 2, in Eq. 5, respectively.
4

---

Published as a conference paper at ICLR 2023
Controlling Each Layer's Rate. Using Eqs. 6-7, generalize the rate/distortion trade-off from
Section 3.1 by introducing L individual Lagrange multipliers 3z. 3L-1..... .... 81. collectively denoted as boldface B. This leads to new loss function that generalizes Eq. 2 as follows,
Lo(0, o) = Egg [D + BLR(zL) B =1) + ... + 31
(8)
Setting all Bs to the same value recovers the conventional 3-VAE (Eq. 2), which trades off distortion against total information content in {=e]. Tuning each 5-hyperparamettr individually allows trading off information content across latents. (In a very deep HVAE (i.e., large L) it may be more practical to group layers into only few bins and to use the same 3-value for all layers within bin.) We analyze how to tune Is for various applications theoretically in Section 4 and empirically in Section 5.
4
PERFOR BOUNDS FOR HVAES
In this section, we analyze theoretically how various performance metrics for HVAES are restricted by the individual layers' rates R(zz,) and R(zz|=2/+1) identified in Eq. 7 for a HVAE with "top- down" inference model. Our analysis motivates the use of the information-trading loss function in Eq. 8 for training HVAEs, following the argument from the introduction that VAEs are commonly used for a vast variety of tasks. As we show, different tasks require different trade-offs that can be targeted by tuning the Lagrange multipliers 33 in Eq. 8. We group tasks into the application domains of (i) data reconstruction and manipulation, (ii) representation learning, and (iii) data generation.
Data Reconstruction and Manipulation. The most obvious class of application domains of VAEs includes tasks that combine encoder and decoder to map some data point æ to representations ( (zz) and then back to the data space. The simplest performance metric for such data reconstruction tasks is the expected distortion E Pau [D], which we can bound by combining Eq. 3 with Eqs. 6-7,
(es [D] 2 [p (x+) - (x)) [R(z1.) - R(z |zz.) + +... + R[21|222)]
(9)
Eq. 9 would suggest that higher rates (i.e., lower 3's) are always better for data reconstruction tasks. However, in many practical tasks (e.g.. image upscaling, denoising, or inpainting) the goal is not solely to reconstruct the original data but also to manipulate the latent representations [zz] in meaningful way. Here, lower rates can lead to more semantically meaningful representation spaces (see, e.g., Section 5.6 below). Controlling how rate is distributed across layers via Eq. 8 may allow practitioners to have a semantically meaningful high-level representation n with low rate R(zL) while still retaining a high total rate R, thus allowing for low distortion D without violating Eq. 9.
Representation Learning. In many practical applications, VAFs are used as nonlinear dimen- sionality reduction methods prepare some complicated high-dimensional data æ for downstream tasks such as classification, regression, visualization, clustering, or anomaly detection. We consider a classifier Pets (z) operating on the latents zạ some level L. We assume that the (unknown) true data generative process Pusta (y, x) = Py)a(()) Paua Paua(<)ly) generates data æ conditioned on some true label y. thus defining a Markov chain y Pian, x 25, Zz Pob,, ý where := arg max, Pets. Petx (y)zz). Classification accuracy is bounded (Meyen, 2016) by function of the mutual information (J; 2z),
log g0 x:)
(y: ze) S [,( ze) = E Ex)
E (x)
(10)
(z²)
log g0 ax)
Epen(e (2))
E (xel
DKL (20(2)) ) po(2))
po (zz)
log 2 (2(()(
< Epeas( (an)
E Eqg (x)
po (x(2))
BBgs
DKL. [gol (22441 ax, 2x) (1224+112)))
< Ep [((z)) +
+... R(ze x20+1)]
R(*s() (SR)
Here, (z)))) := (x) (x)//=²) and we identify R(zy)) as the rate accumulated in all layers
from 21. Zz. The first inequality in Eq. 10 comes from the data processing inequality (MacKay,
5

---

Published as a conference paper at ICLR 2023
2003), and the other two inequalities result from discarding the (nonnegative) KL-terms. The clas- sification accuracy is thus bounded by (Meyen, 2016) (see also proof in Appendix B)
class. accuracy 1 f ( (y: ze)) I f (=) (R(2())) ( "E"'(Epannn (Epeu(z)) [l))
(11)
where f-1 is the inverse of the monotonic function f(o) = [p [1)] aa
log M-T
1-a
with M being the number of classes and H [panta (y)] < log M the marginal label entropy. Eq. 11
suggests that the accuracy of an optimal classifier on would increase as the rate R(z2() accu- mulated from zL ze grows (i.e., B3c 0), and that the rate added in downstream layers #<g would be irrelevant. Practical classifiers, however, have limited expressiveness, which very high rate R(zze) might exceed by encoding too many details into ze that are not necessary for classifi- cation. We observe in Section 5.6 that, in such cases, increasing the rates of downstream layers zee improves classification accuracy as it allows keeping 2ạ simpler by deferring details to zet-
Data Generation. The original proposal of VAEs (Kingma & Welling, 2014) motivated them from a generative modeling perspective using that, for 3 = 1, the negative of the loss function in Eq. is lower bound on the log marginal data likelihood. This suggests setting all 3-hyperparameterss in Eq. to values close to 1 a HVAE is used primarily for its generative model pa-
In summary, our theoretical analysis suggests that optimally tuned layer-wise rates depend on whether a HVAE is used for data reconstruction, representation learning, or data generation. The next section tests our theoretical predictions empirically for the same three application domains.
5 EXPERIMENTS
To demonstrate the features of our hierarchical information trading framework, we run large-scale grid searches over a vro-dimensional rate space using two different implementations of HVAEs and three different data sets. Although the proposed framework is applicable for HVAEs with I, 2 2, we only use HVAEs with I, = 2 in our experiments for simplicity and visualization purpose.
5.1 EXPERIMENTAL SETUP
Data sets. We used the SVHN (Netzer al., 2011) and CIFAR-10 (Krizhevsky, 2009) data sets (both 32 x 32 pixel color images), and MNIST (LeCun et al., 1998) (28 x 28 binary pixel im- ages). SVHN consists of photographed house numbers from 0 to 9, which are geometrically simpler than the 10 classes of objects from CIFAR-10 10 but more complex than MNIST digits. Most results shown in the main paper SVHN; comprehensive results for CIFAR-10 and MNIST are shown in Appendix A.2 and tell a similar story except where explicitly discussed.
Model Architectures. For the generative model (Bq. 1), assume (fixed) standard Gaussian prior p(z2)= = (0,, I), and we use diagonal Gaussian models for po (22) = N (z)) 9n (22)²) and (for SVHN and CIFAR-10) (xx 21) = (21). TT (this is similar to, e.g., (Minnen et al., 2018)). Here, 9u- Io, and 9p' denote neural networks (see details below). Since MNIST has binary pixel values, we model with a Bernoulli distribution for (=(2)) = (21))), For the inference model, we also use diagonal Gaussian models for (2 [z) = (ff fo (x)²) and for to (2)) a., 22) = (far (z. 22), for (x,, 22) where fy. fc. fx', and for are again neural networks.
We examine both LVAE (Figure 2(b)) and our generalized top-down HVAES (GHVAEs; see Fig- ure 2(c)), using simple network architectures with only 2 to 3 convolutional and fully connected layers (see Appendix A. for details) so that we can scan a large rate-space efficiently. Note that are not trying to find the new state-of-the-art HVAEs. Results for LVAE are Appendix A.2.2.
We trained 441 different HVAEs for each data set/model combination, scanning the rate- hyperparameters (32, 31) over a 21 x 21 grid ranging from 0.1 to 10 on a log scale in both directions RTX-208011 GPU (~27 hours in total for each data set/model combination using 32 GPUs in parallel).
(see Figure 1 on page 2, right panels). Each model took about 2 hours to train
Baselines. Our proposed framework (Eq. 8) generalizes over both VAEs and 3-VAEs (Eq. 2), which we obtain in the cases 32 = 3 = 1 and B2 = 3,, respectively. These baselines are indicated as black "o" and red "o" circles, respectively, in Figures 3, 5, 6, and 7, discussed below.

---

Published as a conference paper at ICLR 2023
SVHN
CIFAR-10
150
28
80
28
60
100
NNSd
24
22
22
30
20
16 91
-18
B(ma)
20
[(22(1g)) (2g)
[2(xg)
R(gg)
(a) Rate/rate/distortion surface for SVHN.
(b) PSNR-rates comparison 2d.
Figure 3: PSNR-rate trade-off for GHVAEs trained on SVHN and CIFAR-10. Figure (a) visualizes
the same data as the left panel of (b) in 3d. Black circles "o" mark standard VAEs (B2 = 31 1), red circles "o" mark 3 3-VAEs (B2 (1)) ) and purple circles "o" mark optimal models along constant total rate (dashed diagonal lines) as defined in Section 5.3. Crosses point to columns in Figure 4.
Metries. Performance metrics for the three application domains of VAEs mentioned in the intro- duction are introduced the beginnings of the corresponding Sections 5.4-5.6. addition, we evaluate the individual rates R(z2) and R(z1 (Z2) (Eq. 7), which we report in nats (i.e., to base e).
5.2 THERE IS NO "ONE HVAE FITS ALL"
Figure on page summarizes our results. The 21 x21 GHVAEs trained with the grid of hyper- parameters 82 and 31 map out a surface in a 3d-space spanned by suitable metrics for the three application domains (metrics defined in Sections 5.4-5.6 below). The two upper right panels map colors on this surface to 3s used for training and to the resulting layer-wise rates, respectively. The lower right panels show performance landscapes and identify the optimal models for the three appli- cation domains of data reconstruction (A), representation learning (o), and generative modeling (O).
The figure shows that moving away from a conventional V-V (32 = 31: dashed red lines in Figure 1) allows us to find better models for given application domain as the three application domains favor vastly different regions in 3-space. Thus, there is no single HVAE that is optimal for all tasks, and HVAE that has been optimized for one task can perform poorly on different task.
5.3 DEFINITION OF THE OPTIMAL MODEL FOR A GIVEN TOTAL RATE
One of the questions we study in Sections 5.4-5.6 below is: "Which allocation of rates across layers results in best model performance if we keep the total rate R fixed". Unfortunately. it is difficult to keep R fixed training time since we control rates only indirectly via their Lagrange multipliers 32 and 31. We instead use the following definition, illustrated in Figure 6 for performance metric introduced in Section 5.6 below. The figure plots the performance metric over for all 21 x 21 3- settings and highlights with purple circles "o" all points on the upper convex hull. These highlighted models are optimal for a small interval total rates in the following sense: if we use the total rates R of all "o" to partition the horizontal axis into intervals then, by definition of the convex hull, each "o" represents the model with highest performance in either the interval to its left or the one to its right.
5.4 PERFORMANCE ON DATA RECONSTRUCTION
Reconstruction is popular task for VAEs, c.g., in the area of lossy compression (Ballé al., 2017 We measure reconstruction quality using the common peak signal-to-noise ratio (PSNR), which is equal to E_ log D) up to rescaling and shifting. Higher PSNR means better reconstruction.
Figure 3(a) shows a 3d-plot of PSNR as a function of both R( (1]) and R(zy) ) for SVHN thus generalizing the rate/distortion curve of conventional 3-VAE to rate/rate/distortioo surface. Fig- ure 3(b) introduces a more compact 2d-representation the same data that we use for all remaining metrics in the rest of this section and in Appendix A.2, and it also shows results for CIFAR-10.

---

Published as a conference paper at ICLR 2023
'T' (4==)=994--022))) (025: PSNR 29.1
22" R(2z)= 20.2 (4:=050))
PSNR=238
"3": R(es) =5.1 (0-=1.58)
21. 1S
= 011) 4524
R(s. = (..3.) 5M
R(a.)z)) (App=
samples
5
2
reconstructions
originals for reconstructions: 00
2
Figure 4: Samples (top) and reconstructions (bottom) from 3 different models (blue column labels "I", "2", and "3" from left to right correspond to crosses "1", "2", and "3" in Figures 3(b) 5). Consistent with PSNR and IS metrics, model "I" produces poorest samples but best reconstructions.
Inception Score (15)
Diveriity
Sharpness
1.0
RBF-SVM Accuracy
0.6
0.6
0.4
20
20
40
20
(l)e))
R(ss)
(((z))
Total Rate
Figure 5: Sample generation performance, measured in Inception Score (IS, see Eq. 12) and its factorization into diversity and sharp- ness a function of layer-wise rates for GHVAEs trained using SVHN data. Crosses in left panel correspond to samples shown in Figure 4. Markers "o", "o", and "o" same as in Figure 3.
Figure 6: RBF-SVM clas- sification accuracies on μ2- Dashed line shows theoret- ical bound (Eq. 11). Other markers as in Figure 3.
Unsurprisingly and consistent with Eq. 9, reconstruction performance improves as total rate grows. However, minimizing distortion without any constraints is not useful in practice as we can simply use the original data, which has no distortion. To simulate practical constraint in, e.g., a data- compression application, we consider models with optimal PSNR for a given total rate R (as defined in Section 5.3) which are marked as purple circles "o" Figure 3(b). We see for both SVHN and CIFAR-10 10 that conventional 3-VAEs (32 = 81; red circles) perform somewhat suboptimal for given total rate and can be improved by trading some rate in 22 for some rate in ø1. Reconstruction examples for the three models marked with crosses in Figure 3(b) are shown in Figure 4 (bottom). Visual reconstruction quality improves from "3" to "2" to "1"", consistent with reported PSNRs.
5.5 PERFORMANCE ON SAMPLE GENERATION
We next evaluate how tuning layer-wise rates affects the quality of samples from the generative model. We measure sample quality by the widely used Inception Score (IS) (Salimans et al., 2016).
IS exp (EEpo(e) [DKL [pes. (=) Pat.. (v)]] e (v)) (v)) EEggoon) [H[pa. (v [=]]
(12)
Here, Pe is the trained generative model (Eq. 1), Pels. (M is the predictive distribution of classifier trained on the same training set, and Pets. (3) = Epa(z) (w) Pets.
Pets (v/==)] The second equality in Eq. 12 follows Barratt & Sharma (2018) to split IS into product of a diversity score and a sharpness score. Higher is better for all scores. The classifier is ResNet-18 (He al., 2016) for SVHN (test accuracy 95.02 %) and a 121 (Huang et al., 2017) for CIFAR- 10 (test accuracy 94.34 %).
Figure (left) shows IS for GHVAEs trained on SVHN. Unlike the results for PSNR, here, higher rate does not always lead to better sample quality: for very high R(zz) and low R(z1 (Z2). IS even- tually drops. The region of high IS is in the area where 82 < 31, i.c., where [2(z2) is higher than in comparable conventional 8-VAE. The center and right panels of Figure 5 show diversity and sharpness, indicating that IS is mainly driven here by sharpness, which depends mostly on R(zz), possibly because 2y captures higher-level concepts than 21 that may be more important to the clas- sifier in Eq. 12. Samples from the the three models marked with crosses in Figure 5 are shown in Figure (top). Visual sample quality improves from "1" to "3" to "2", consistent with reported IS.
8

---

Published as a conference paper at ICLR 2023
MI
Loginic
Linear SVM
RBF SVM
KNN
SWIN
CIFNR-30
R/
Inser caumiiiers
nonlinsar classfiers
Figure 7: Mutual information (MI) Ly; Ø2) and classification accuracies of four classifiers (see column labels) function of layer-wise rates R(z2) and R(z |Z2). Classifiers are conditioned on f2 == arg maxzy (( |x) learned from GHVAEs trained with SVHN (top) and CIFAR-10 10 (bottom). Markers "o", "o", and "o" same as in Figure 3.
5.6 PERFORMANCE ON REPRESENTATION LEARNING FOR DOWNSTREAM CLASSIFICATION
VAEs are very popular for representation learning as they map complicated high dimensional data æ to typically lower dimensional representations {zt]. }. To measure the quality of learned representa- tions, we train two sets of classifiers on a labeled test set for each trained HVAE, each consisting of: logistic regression, a Support Vector Machine (SVM) (Boser al., 1992) with linear kernel, an SVM with RBF kernel, and k-nearest neighbors (kNN) with & = 5. One set of classifiers is conditioned on the mode μi of go (z2l(²) and the other one on the mode l1 of go [1) 222 x). where z2 ~ qo (z). We use the implementations from scikit-learn (Pedregosa et al., 2011) for all classifiers.
Figure 7 shows the classification accura- cies (columns 2-5) for all classifiers trained on 422- The first column shows the mu- tual information [y 22), which depends mainly on R(22) as expected from Eq. 10. As long the classifier is expressive enough (e.g., RBF-SVM or kNN) and the data is simple (SVHN; top row), higher mutual information (< higher R(22)) cor- responds to higher classification accuracies, consistent with Eq. 11. But for less expres- sive (e.g., linear) classifiers or more complex data (CIFAR-10; bottom row), increasing R(2, 22) improves classification accuracy (see purple circles "o" in corresponding panels), consistent with the discussion below Eq. 11. We see similar effect (Table I) for most classifier/data set combinations when replacing μ2 by 14i which has more information about æ but is also higher dimensional.
Table 1: Optimal classification accuracies (across all (02, )-sstttings) using either μl or f1.
Data Set
log. reg. lin. SVM
RBF SVM
kNN
SVHN (Ju2)
28.43 %
27.87%
77.60
64.25 %
SVHN (161)
45.77 %
49.81 %
59.28 %
56.49 %
CIFAR-10 (142)
47.36 %
46.95
53.15
44.20 %
CIFAR-10 10 (141)
43.27 %
42,55 %
45.60 %
39.25 %
CONCLUSIONS
We classified the various tasks that can be performed with Variational Autoencoders (VAEs) into three application domains and argued that each domain has different trade-offs, such that a good VAE for one domain is not necessarily good for another. This observation motivated to propose refinement of the rate/distortion theory of VAFS that allows trading off rates across individual layers of latents in hierarchical VAEs. We showed both theoretically and empirically that the proposal indeed provides practitioners better control for tuning VAEs for the three application domains. In the future, would be interesting explore adaptive schedules for the Lagrange parameters that would make it possible to target a specific given rate for each layer in a single training run, for example by using the method proposed by Rezende & Viola (2018).
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors would like to thank Johannes Zenn, Zicong Fan, Zhen Liu for their helpful discus- sion. Funded by the Deutsche Forschunsseemiincchat (DFG, German Research Foundation) under Germany's 's Excellence Strategy EXC number 2064/1 Project number 390727645. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S18039A. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Tim Z. Xiao.
Reproducibility Statement. All code necessary reproduce the results in this paper is available at htttps://gittiii www/tiowzz///cccccccc/ctccctctct
REFERENCES
Eirikur Agustsson and Lucas Theis. Universally quantized neural compression. Advances in Neural Information Processing Systems, 33: 13:77-1116 2020. 4
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy, Fixing broken elbo. In International Conference Machine Learning, pp. 159-168. PMLR, 2018. 1, 2, 3
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2017. 1, 3,
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. In International Conference on Learning Representationss 2017. 3,
Johannes Ballé, David Minnen, Saurabh Singh. Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In International Conference on Learning Represen- tations, 2018. 2
Shane Barratt and Rishi Sharma. note on the inception score. arXiv preprint arXiv:1801.0197
2018. 8
Charles H Bennett, Peter W Shor, John A Smolin, and Ashish V Thapliyal. Entanglement-assise capacity of quantum channel and the reverse shannon theorem. IEEE transactions on Informa- fion Theory, (1110)):637-2655 2002. 4
Bernhard Boser, Isabelle M Guyon, and Vladimir N Vapnik. A training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop Computational learning theory, pp. 144-152, 1992. 9
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021.
Zhiwei Deng, Rajitha Navarathna, Peter Carr, Stephan Mandt, Yisong Yue, Iain Matthews, and Greg Mori. Factorized variational autoencoders for modeling audience reactions movies. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2577-2586, 2017. 2
Brendan J Frey and GE Hinton. E cient stochastic source coding and application bayesian network source model. Computer Journal, 1997.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. In International Conference on Learning Representations, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016. 8
10

---

Published as a conference paper at ICLR 2023
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, 2017. 1, 3,
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 4700-4708, 2017. 8
Zhuxi Jiang, Yin Zheng. Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: an unsupervised and generative approach to clustering, In International Joint Con- ference on Artificial Intelligence, 2017. 1
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Reprexentations, 2014. 1,3,4, 4,
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. 6
Eric Laloy, Romain Hérault, John Lee, Diederik Jacques, and Niklas Linde. Inversion using new low-dimensional representation of complex binary geological media based on a deep neural network. Advances in water resources, 110:387-405, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11)2227--2322 1998. 6
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. Advances in Neural Information Processing Systems, 32, 2019. 3
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003. 5
Sascha Meyen. Relation between classification accuracy and mutual information in equally weighted classification tasks, Master's 's thesis, Universität Hamburg, 2016. 5, 6, 17
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances in Neural Information Processing Systems, 31, 2018. 3,6 6
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning, In NIPS Workshop Deep Learning and Unsupervised Feature Learning, 2011. 6
F. Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011. 9
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in Neural Information Processing Systems, 32, 2019.
Danilo Jimenez Rezende and Fabio Viola. Taming vaes. arXiv preprint arXiv:1810.00597 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. International conference on machine learning, pp. 1278-1286. PMLR, 2014. 1,3
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in Neural Information Processing Systems, 29, 2016. 8
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kane Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in Neural Information Processing Systems, 29, 2016. 3,
11

---

Published as a conference paper at ICLR 2023
Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Student- t variational autoencoder for robust density estimation. In International Joint Conference on Artificial Intelligence, 2018. 1
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pp. 1-5. IEEE, 2015. 1,
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 13:19667-19679 2020. 3
Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, al. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In Proceedings of the 2018 world wide web conference, pp- 187-196, 2018.
Yibo Yang, Robert Bamler, and Stephan Mandt. Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33:573-584, 2020. 3
Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder, In International Conference on Machine Learning, 2018. 2
12

---

Published as a conference paper at ICLR 2023
A EXPERIMENT SUPPLEMENTARIES
A.1 IMPLEMENTATION DETAILS
Table Model architecture details for generalized top-down HVAES (GHVAEs) used in Section 5. Conv and ConvTransp denote the convolutional and transposed convolutional layer. which has the corresponding input: input channel, output channel, kernel size, stride, padding. FC represents fully connected layer.
Data (=2/2))
( (2)
[(21
p 21)
a:: Conv(3, 32, 4, 1), Conv(32, 32, 2,
Share:
Conv(3, 32, 2, D), Conv(32, 32, 4, 1), Conv(32, 32, 4,
Shore:
For FC(In=32, Out=512)
FC(In=32, Out=256)
Conv ConnTransp(32.11 32,
SVHN/ CIFAR-10
For mean:
Conv ConvTrraspp322, 4, 1).
Share: ConvTTansp(644. 4,
For mean: FC(In=512, Out=32)
FC(I==256, Out=512)
Conv NunTraaspp32, 3, 4,
For variance:
For variance: FC(In=512, Out=32)
FC(In=256, Out=512)
For mean; Conv(32, 32, 3, I, I)
For variance; Conv(32, 32, 3, I,
Zi dimss: 512
22 dims: 32
Fx = 0.71
Total params: 475811
For a:: Conv(1, 16, 4, 1), Conv(16, 16, 1,
Share:
Conv(1, 16, 2, l), Conv(16. 16, 4, 1), Conv(16, 16, 4, 0)
Shore: FC(In=20, Out=128)
FC(In=20, Out=256)
ConvTransp166, 16, 0).
MNIST (Binary)
For mean:
ConvTranssple 16, 1). ConvTransp(16, 4,
Share:
For FC[In=256, Out=20)
FC(In=128, Out=256)
Conv Transp(32, 16,4, 1,
For variance:
For variance: FC(In=256, Out=20)
FC(In=128, Out=256)
For mean:
Conv(16,
For variance:
Conv(16, 16, 3, 1,
dims: 256
dims: 20
N/A
Total params: 122713
Table 3: Model architecture details for LVAES used in Section 5. Conv and Conv ConvTrunsp denote the convolutional and transposed convolutional layer, which has the corresponding input: input channel, output channel, kernel size, stride, padding. FC represents fully connected layer.
Data set ((22(2))
9 ((21/22,2)
p(21/22)
p( x11
Shane:
Conv(3, 4, I), Conv(32, 32, 4, 2, 1), Conv(32, 32, 4, 2,
Involve d: Conv(32, 32, 4, 2,
Share: FC(n==3,, Out=256)
Conv Man/Transp(32, 32, 2, 1),
SVHN/ CIFAR-10
For mean:
For mean:
Conv ConvTransp(32, 32, 2, 1), Conv ConvTransp(32, 3, 2, 1)
For mean: FC(In=512, Out=32)
Conv(32, 32, 3, 1)
FC(I==566, Out=512)
For variance: Conv(32, 32, 3, FC(In=266, Out=512)
For variance:
For variance: FC(In=512, Out=32)
21 dims: 512
2g dims:
62 0.71
Total params: 408131
13

---

Published as a conference paper at ICLR 2023
A.2 ADDITIONAL RESULTS
Here we attached the results for MNIST, as well as the full results for LVAE on SVHN and general- ized top-down HVAEs on CIFAR-10.
A.2.1 RESULTS FOR GENERALIZED TOP-DOWN HVAES ON MNIST
We also evaluate our proposed framework using generalized top-down HVAEs trained binary MNIST data (i.e., black and white images rather than grayscale).
We note that the inception score (IS) behaves different in MNIST models compared to SVHN (see Figure 5) in that optimal IS in MNIST occurs for high R(z1|z)) rather than high R(z2). This indicates that semantically low-level properties (hand-writte style) of MNIST might have more variation than high level properties (the digit), whereas SVHN images show variation in additional high-level properties such as the background color.
PSNR
Inception Score (IS)
MI
140
10M
40
Logistic
Linear SVM
RBF SVM
MNN
120
0.08
0.93
0.58
R00e [e2)
50
10
201 40
300
10
600 330
10
R(me))
R(x))
Figure 8: Trade-offs between rates and all metrics we used in Section from the generalized top- down HVAEs trained with MNIST. The results from the standard VAE (i.e. 83 B1 = 1) and the 3-VAE (i.e. B3 = 31) are marked with "o" and "o". The markers "o" highlight the optimal models selected using convex hull (see Figure 6 for details). The diagonal grid lines are references for equivalent total rates, i.c. points on the same line have the same total rates.
14

---

Published as a conference paper at ICLR 2023
A.2.2 RESULTS FOR LVAE ON SVHN
PSNR
Inception Score (15)
MI
60
Rlbe hn3
00
DOODOO
Logistic
Linear SVM
RBF SVM
KNN
60
R(n 301
20
R(as)
((xa]
[(ss)
((sa))
Figure 9: Trade-offs between rates and all metrics we used in Section from LVAE trained with
SVHN. The results from the standard VAE (i.e. 83 31 and the 3-VAE (i.e. Bg (1) marked with "o" and "o". The markers "o" highlight the optimal models selected using convex hull (see Figure for details). The diagonal grid lines are references for equivalent total rates, i.e. points on the same line have the same total rates.
15

---

Published as a conference paper at ICLR 2023
A.2.3 RESULTS FOR GENERALIZED TOP-DOWN HVAES ON CIFAR-10
PSNR
Inception Score (IS)
MI
125
100
20-1
1X00 00
Logistic
Linear SVM
RBF SVM
KNN
150
1.46
100
75
12
-00
002 201 00 0 201 131 20 00
[((as)
R(as]
Figure 10: Trade-offs between rates and all metries we used Section 5 from the generalized top- down HVAEs trained with CIFAR-10. 10. The results from the standard VAE (i.e. 332 31 1) and the 3-VAE (i.e. B2 = 31) are marked with "o" and "o". The markers "o" highlight the optimal models selected using convex hull (see Figure 6 for details). The diagonal grid lines are references for equivalent total rates, i.e. points on the same line have the same total rates.
16

---

Published as a conference paper at ICLR 2023
B PROOF OF THE BOUND ON CLASSIFICATION ACCURACY
This section provides proof of Eq. 11 by reformulating the proof of Proposition in the thesis by Meyen (2016) into the notation used in the present paper. We stress that this section contains no original contribution and is provided only as a convenience to the reader, motivated by reviewer feedback. All credits for this section belong to Meyen (2016).
We consider an (unknown) true data generative distribution (V x) for data æ with (unobserved) true labels y, and hierarchical VAE with an inference model qu( ( (ze) x)) of the form of Eq. 4.
Focusing on a single layer & of latents, we denote the joint probability over y, æ, and žy as
q(y. x xe) Pdata (y. x) 4s (Zz(az)
(13)
where the marginal go (0(=z (x) of go ( x) is defined as usual. We further consider a classifier Pots. (y)zz) that operates on zy. Denoting its top prediction as 1 == arg maxy Pets. (ess(y/z)) ze), the classifi- cation accuracy is a == Eg (oy.sl. where 5 is the Kronecker delta.
Theorem 1. The mutual information [g(v; zt) between the latent representation and the true label y under the distribution q defined in Eq. 13 is lower bounded as follows,
Ig(y: ze) 2 f(a) with
f(a) = H pa 99] - H2(a) - (1 a)) log(M 1)
(14)
where Hy(a)
-a loga - (1 - a)) log(1 aa) is the entropy of a Bernoulli distribution, H Pana [u] S log M is the marginal entropy of the true labels, and M denotes the number of classes.
Before we prove Theorem I, we note that the function f is strictly monotonically increasing on the relevant interval [max, Paana (y), 1]. Thus, f is invertible and we obtain the following corollary: Corollary 1. The classification accuracy is upper bounded as in Eq. 11 of the main text, i..., (15)
a S (I ze)) S f (Epan(()) [R(z)))).
The second inequality in Eq. 15 results from the bound [g ze) Bpen(()) [R(zg)] derived in Eq. 10, using the fact that f-1 monotonically increasing (since f is).
Proof of Theorem 1. We split the mutual information into two contributions,
[y: ze) = Ha.. [v] Hglv|ze] Hp.. [v] Exz=a(==) [ log [[)])))]]
(16)
where, as clarified in the second equality, H, [u] is the expectation over zς of the conditional entropy of y given ze, and q(zz) and ((v(zz) are marginals and conditionals of q (Eq. 13) as usual.
Since H [v] is fixed by the problem at hand, finding lower bound on (y; ze) for given clas- sification accuracy a is equivalent to finding an upper bound on the second term on the right-hand side of Eq. 16, Hylv) = (z(=)(=)) ( [By=q(v)ze) --) log ((y]==)]], with the constraint Ey(dg.sl a. We do this by upper bounding the conditional entropy Bx [- log (()]=)) of y given ze for all z₁ independently, and then taking the expectation over ž₁ ~ q(z)).
For fixed latent representation zz. we first split off the contribution to Eysg(sl=)) log q(v 2e)] from y V. where ý arg maxy Pets 22) is the label that our classifier would predict for zt.
Ey-o(vlee) log (v]()))) = - y log g(v =0
Σ log ツデツ
(17)
Here, the second term on the right-hand side resembles the entropy of a distribution over the re- maining (M 1) labels (y A g)) except that the probabilities sum to (1- p(y=0(=2) rather than
one. Thus, regardless of the value of q(y أ z2). this term is maximized if g 22) distributes the
remaining probability mass (1 40y= = 9 2z)) uniformly over the remaining (M 1) labels, i.e.
1 - q(y= 0 M -
Ey log (vvl [x]] < - 0| =2) log g(v )1=;) (1 g(v - - z1)) log
( ū()) (1 q(v U| 2 log(M - 1).
(18)
Plugging Eq. 18 back into Fq. 16, we obtain the bound
[n zz) H para [y] Ez(~n(=e) (2 =x)] E -x(=)==)) [1 q(v (| log(M (M 1).
(19)
We arrive at the proposition (Eq. 14) by pulling the concave function H2 out of the expectation using Jensen's 's inequality, and by then identifying Ez (Q q(v= = 9) =
17


Reviews:
[{"review_id": "tfPasIk19ss", "reviewer": "Reviewer_nhzt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to assign separate beta Lagrange factors to each level in hierarchical VAEs to control the rate for each level separately. They argue that different applications of VAEs have different trade-offs for the rate, e.g., classification, generation, and reconstruction. The experiments sweep over values of each beta in 2-level VAEs on MNIST, SVHN, and CIFAR10. They report various metrics for each setting, showing that different values of beta are useful in different settings.", "summary_of_the_review": "While the paper does not present any radically novel techniques or entirely surprising findings, the authors tackle a well-defined and practical question that, to the best of my knowledge, has not been addressed previously in the literature. The clarity of the presentation makes the paper approachable by readers with varying levels of background knowledge, and the empirical findings demonstrate a high degree of rigor (i.e., performing a large-scale sweep, multiple datasets, multiple metrics). For these reasons, I recommend acceptance.", "strength_and_weaknesses": "**Strengths**\n- **Clear presentation.** The authors clearly lay out the setting (hierarchical VAEs), three application domains (classification, generation, and reconstruction), and the main contribution (evaluating different beta trade-offs). Supporting diagrams in Figure 2 show the setup considered by the authors. Mathematical equations provide clear and consistent definitions of the quantities under consideration without introducing superfluous notation. And the experiment section is laid out with separate sections and plots for each setting.\n- **Great plots / empirical analysis.** Beyond the clear presentation, the empirical findings are presented extremely well. It’s apparent that a significant amount of time went into creating these plots. The plots present findings with a mix of color-coding, shapes, and 2D and 3D visualizations. The plots do a great job of presenting the distinction between VAE vs. beta-VAE vs. having multiple beta hyperparameters.\n- **Somewhat practical/relevant.** The paper tackles a rather practical and straightforward question of whether adjusting the rate at each hierarchical level in VAEs significantly impacts performance on various applications. This is a well-defined question, which the authors directly address by sweeping over hyperparameters (betas) and evaluating performance using relevant metrics. While I’m not sure whether a reader would come away from the paper with clear guidance on how to adjust these hyperparameters for any given architecture, the main message—that these hyperparameters can and should be separately adjusted—is clear.\n\n**Weaknesses**\n- **Limited novelty.** The primary novel aspect of the paper is in proposing to separately adjust different beta hyperparameters for each level in hierarchical VAEs, showing that this is relevant for various applications. I did not find this to be particularly novel, as the paper boils down to performing a hyperparameter sweep over multiple (in this case, two) dimensions rather than one. Likewise, the authors do not propose any form of method for automatically adjusting these hyperparameters for a given application. Rather, the paper presents a set of hyperparameter sweeps, showing that different regions are relevant for different applications. It’s unclear how these findings can readily be extended to hierarchies of arbitrary depth beyond simply performing hyperparameter sweeps in these larger spaces. The primary way of fixing this weakness would be to demonstrate some new capability afforded by this technique and/or some new method for targeting a particular application. For instance, rather than separately sweeping over both beta hyperparameters, one might consider using the constrained optimization formulation presented by Rezende & Viola, 2018, in which a target rate (or distortion) is satisfied.\n- **Simple-ish models and datasets, which may not be relevant for practical settings.** The authors explore their empirical question in the setting of somewhat small 2-level hierarchical VAEs using MNIST, SVHN, and CIFAR10, all of which are fairly small image datasets. While I entirely understand the need to manage the complexity of the empirical setting to make such analyses tractable, it does raise questions of whether these analyses offer practical guidance for a practitioner. I would have a hard time believing that any of the models and datasets in this paper would be a go-to choice for any of the applications (classification, generation, reconstruction) considered. I understand that the authors are not attempting to obtain SOTA performance in any of these domains. However, to address this weakness, it may be useful to target one particular application more directly, e.g., compression/reconstruction, and show that these findings are relevant with full-scale models and datasets.\n- **Framing may not be fully general/applicable.** The authors consider a stripped-down version of hierarchical VAEs, in which the generative process is a strict Markov chain. However, such models are not typically considered within the literature. Starting with the IAF (Kingma et al., 2016) hierarchical VAE architecture, most hierarchical models have introduced deterministic connections between higher-level variables and lower-levels (including the conditional likelihood). This is true of modern large-scale hierarchical VAEs like NVAE (Vahdat & Kautz, 2020) and VDVAE (Child, 2021). These deterministic connections allow gradients to flow directly from the reconstruction to the higher-level variables, which undoubtedly alters the gradients and learning dynamics. I understand that the authors restricted their analysis to Markov hierarchies for practical purposes, but the findings may be moot if they are not relevant in the more general setting of non-Markov hierarchies. Fixing this weakness would involve performing analyses in this more general setting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ieYwZLvfVc", "reviewer": "Reviewer_kj2j", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a relatively thorough study of the effect of $\\beta$ for the case of hierarchical VAEs. Specifically, the authors study the rate-distortion performance of two-layer VAEs with various combinations of $\\beta$s and show that different tasks require different settings of $\\beta$s.  The paper also presents a theoretical discussion that predicts the empirical results. ", "summary_of_the_review": "Overall, I enjoyed reading the paper as it is very well-written and the theoretical discussions conclusions are coherent with the empirical findings.  I particularly value the fact that the paper explicitly tries to provide useful guidelines for practitioners who may need to tune a hierarchical VAE for a specific task.  However as mentioned above, the paper does not present a novel idea, and is somewhat incomplete in experiments.", "strength_and_weaknesses": "**Strengths**\n* Very well-written paper with clear motivation, with an interesting generalization of $\\beta$ VAEs.\n* Thorough analysis and discussion.\n\n**Weaknesses**\n* Lack of empirical results for $L > 2$.  While I understand that it may not be feasible to perform a thorough grid search (as was done for the two-layer VAEs in this paper) with deeper VAEs, basing all empirical conclusions on $L=2$ experiments seems a little brittle.  While the theoretical discussions are sensible, it is unclear to me whether the empirical results presented would readily generalize to much larger values of $L$ -- especially given finite capacity models with possibly different optimization dynamics  in practice.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WIaDayVhqb", "reviewer": "Reviewer_xknb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors extend an existing method and develop an approach to control each layer’s contribution to the rate independently in the hierarchy VAE. They identify the most general class of inference models to which their proposed method is applicable. In the experiments, they demonstrate that the proposed method better tunes hierarchical VAEs. ", "summary_of_the_review": "The author could add more theoretical and experimental studies to make the paper stronger. For example, how does the distribution of $\\beta$s affect the performance? How many layers of H-VAEs do we need given specific datasets? \n", "strength_and_weaknesses": "Strength:\n\nThe paper was well written with good structures. \n\nWeakness\n\nThe paper is a simple extension of existing works, and the significance and novelty of the paper are not strong enough for publication at the moment. The paper could be stronger with additional formal theoretical study. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ItiTnerLUsR", "reviewer": "Reviewer_7gHR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper clarifies that the different applications in VAEs require different trade-offs in the IB formulation; that is, the single regularization coefficient of rates in IB cannot achieve the best performance in three application domains. Motivated by this analysis, the authors proposed to extend the existing HVAEs based on the IB formulation and introduce flexible regularization terms in HVAEs. The new model can show better trade-offs under different tasks thanks to this flexibility.", "summary_of_the_review": "I think the contributions of this work are two folds;\none is the analysis that the different tasks in VAEs require different levels of regularization about rates in IB problems. The other contribution is introducing the new HVAE that has flexible rate regularization.\n\nI think the first contribution is insightful and should be shared in the community. Still, the second contribution seems insufficient since I could not be convinced why using different beta is important numerically or theoretically.\nIt would be better to include a discussion about how using different betas leads to resolving the problem of the necessity of different rates among different tasks by introducing additional flexibility.", "strength_and_weaknesses": "# Strength\n- This paper explicitly discusses how the three commonly used tasks of VAEs are related to the rates in the IB formulation. Then the authors showed theoretically and empirically that the different rates are preferable for different tasks, which seems missed in existing work.\n- The authors extend the IB formulation of VAE to the different levels of regularization of rates in HVAE.\n- The authors studied how the rates affect the performance of different tasks by controlling the regularization in detail.\n\n# Weakness\n- I am not satisfied with the explanation of how the rates of the representation $Z$ and accuracies in downstream tasks are related to each other. In particular, the authors introduced the bound Eq.11, but I do not think the assumptions of Eq. 11 in Meyen 2016 are satisfied in VAEs.\n- Decomposing the rate term in IB and introducing different $\\beta$s in HVAEs are natural. Although using different $\\beta$s introduces additional flexibility and improves the performance, I do not think it is practical because we must choose $\\beta$s, and it is not clear how to tune them.\n- As the authors mentioned in Sec 6, it would be better to include the results about how we choose $\\beta$s or the results that show the merit of using $\\beta_1\\neq \\beta_2$ not $\\beta_1=\\beta_2$. In the current numerical experiments, I could not see that advantage.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster