Evaluating Lossy Compression Rates of Deep Generative Models
Authors: Sicong Huang
Alireza Makhzani
Yanshuai Cao
Roger Grosse
Venue:  (2020)
Keywords: Deep Learning
Generative Models
Information Theory
Rate Distortion Theory

Abstract: Deep generative models have achieved remarkable progress in recent years. Despite this progress, quantitative evaluation and comparison of generative models remains as one of the important challenges. One of the most popular metrics for evaluating generative models is the log-likelihood. While the direct computation of log-likelihood can be intractable, it has been recently shown that the log-likelihood of some of the most interesting generative models such as variational autoencoders (VAE) or generative adversarial networks (GAN) can be efficiently estimated using annealed importance sampling (AIS). In this work, we argue that the log-likelihood metric by itself cannot represent all the different performance characteristics of generative models, and propose to use rate distortion curves to evaluate and compare deep generative models. We show that we can approximate the entire rate distortion curve using one single run of AIS for roughly the same computational cost as a single log-likelihood estimate. We evaluate lossy compression rates of different deep generative models such as VAEs, GANs (and its variants) and adversarial autoencoders (AAE) on MNIST and CIFAR10, and arrive at a number of insights not obtainable from log-likelihoods alone.

Under review as a conference paper at ICLR 2020
EVALUATING LOsSY COMPRESSION RATES OF DEEP GENERATIVE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep generative models have achieved remarkable progress in recent years. De- spite this progress, quantitative evaluation and comparison of generative models remains as one of the important challenges. One of the most popular metrics for evaluating generative models is the log-likelihood. While the direct computa- tion of log- lilkliihhood can be intractable, it has been recently shown that the log- likelihood of some of the most interesting generative models such as variational autoencoders (VAE) generative udversarial networks (GAN) can be efficiently estimated using annealed importance sampling (AIS). this work, we argue that the log-likel metric by itself cannot represent all the different performance characteristics of generative models, and propose to use rate distortion curves to evaluate and compare deep generative models. We show that we approxi- mate the entire rate distortion curve using one single run of AIS for roughly the same computational cost a single log- likeeikoo estimate. We evaluate lossy compression rates of different deep generative models such as VAEs, GANs (and its variants) and adversarial autoencoders (AAE) MNIST and CIFAR10, and arrive at a number of insights not obtainable from log-likelihoods alone.
INTRODUCTION
Generative models of images represent one of the most exciting areas of rapid progress of AI (Brock et al., 2019; Karras et al., 201 8b;a). However, evaluating the performance of generative models remains significant challenge. Many of the most successful models, most notably Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), are implicit generative models for which computation of log-likelihoods is intractable or even undefined. Evaluation typically focuses on metrics such as the Inception score (Salimans et al., 2016) or the Fréchet Inception Distance (FID) score (Heusel et al., 2017), which do not have nearly the same degree of theoretical underpinning as likelihood based metrics.
Log-likelihoode are one of the most important measures of generative models. Their utility is evi- denced by the fact that likelihoods (or equivalent metrics such as perplexity bis-per-dimmensio)) are reported in nearly all cases where it's convenient to compute them. Unfortunately, computa- tion of log- likelihoods for implicit generative models remains difficult problem. Furthermore, log-likelihoods have important conceptual limitations. For continuous inputs in the image domain, the metric often dominated by the fine-grained distribution over pixels rather than the high-level structure. For models with low-dimensional support, one needs to assign an observation model, such as (rather arbitrary) isotropic Gaussian noise (Wu al., 2016). Lossless compression metrics for GANs often give absurdly large bits- biii-eer-iimeniion (e.g. 1024) which fails to reflect the true perfor- mance of the model (Grover al., 2018; Daníhelka et al., 2017). See Theis al. (2015) for more discussion of limitations of likelihood- based evaluation.
Typically, one is not interested in describing the pixels of an image directly, and would be sufficient to generate images close to the true data distribution in some metric such Euclidean distance. For this reason, there has been much interest in Wasserstein distance as a criterion for generative models, since the measure exploits precisely this metric structure (Arjovsky et al., 2017; Gulrajani et al., 2017: Salimans et al., 2018). However. Wasserstein distance remains difficult to approximate, and hence it is not routinely used to evaluate generative models.

---

Under review as a conference paper at ICLR 2020
We aim to achieve the best of both worlds by measuring lossy compression rates of deep generative models. In particular, we aim to estimate the rate distortion function, which measures the number of bits required to match distribution to within given distortion. Like Wasserstein distance, can exploit the metric structure of the observation space, but like log-likelihoods, it connects to the rich literature of probabilistic and information theoretic analysis of generative models. By focusing on different parts of the rate distortion curve, one can achieve different tradeoff's between thereby fixing the problem whereby
the description length and the fidelity of reconstruction lossless compression focuses on the details at the expense of high-level structure. It has the further advantage that the distortion metric need not have probabilistic interpretation; hence, one is free to use more perceptually valid distortion metries such as structural similarity (SSIM) (Wang et al., 2004) or distances between hidden representations of convolutional network (Huang et al., 2018).
Al Algorithmically, computing rate distortion functions raises similar challenges to estimating log- likelihoods. We show that the rate distortion curve can be computed by finding the normalizing constants of a family of unnormalized probability distributions over the noise variables z. Inter- estingly, when the distortion metric is squared error, these distributions correspond to the posterior distributions of for Gaussian observation models with different variances; hence, the rate distortion analysis generalizes the evaluation of log- liglliiels with Gaussian observation models.
Annealed Importance Sampling (AIS) (Neal, 2001) is currently the most effective general-purposs method for estimating log-likelihooss of implicit generative models, and was used by Wu et al. (2016) to compare log-likelihoods of a variety of implicit generative models. The algorithm is based on gradually interpolating between tractable initial distribution and an intractable target distri- bution. We show that when AIS is used estimate log-likelihoods under a Gaussian observation model, the sequence of intermediate distributions corresponds to precisely the distributions needed to compute the rate distortion curve. Since AIS maintains a stochastic lower bound on the nor- malizing constants of these distributions. it automatically produces an upper bound on the entire rate distortion curve. Furthermore, the tightness of the bound can be validated on simulated data using bidirectional Monte Carlo (BDMC) (Grosse et al., 2015; Wu et al., 2016). Hence, we can approximate the entire rate distortion curve for roughly the same computational cost as a single log-likelihood estimate.
We use our rate distortion approximations to study a variety of variational autoencoders (VAEs) (Kingma & Welling, 2013), GANs and adversarial autoencoders (AAE) (Makhzani et al., 2015), and arrive at a number of insights not obtainable from logliikelsooods alone. For instance, observe that VAEs and GANs have different rate distortion tradeoffs: While VAFS with larger code size can generally achieve better lossless compression rates, their performances drop at lossy compression in the low-rate regime. Conversely, expanding the capacity of GANs appears bring substantial reductions in distortion at the high-rate regime without any corresponding deterioration in quality in the low-rate regime. We find that increasing the capacity of GANs by increasing the code size (width) has a qualitatively different effect on the rate distortion tradeoffs than increasing the depth. We also find that that different GAN variants with the same code size achieve nearly identical RD curves, and that the code size dominates the performance differences between GANs.
BACKGROUND
2.1 ANNEALED IMPORTANCE SAMPLING
Annealed importance sampling (AIS) (Neal, 2001) is Monte Carlo algorithm based on constructing
a sequence of n intermediate distributions p& (z) = Zz where & E (O...., n), between a tractable
PL(z)
initial distribution Po (z) and the intractable target distribution p.. (z). At the the k-th state (0 < k <
n). the forward distribution 9 and the un-oormalized backward distribution q are
(2 .2)) pm(2 To() ... Tk- (RA (zE- 1),
To (z (z....., Z+) Py. (zx (T (Zu-1 (Zx) To 21),
(2)
where To is an MCMC kernel that leaves pa (z) invariant; and Th is its reverse kernel. We run M independent AIS chains, numbered i = 1,..., M. Let 2 be the k-th state of the i-th chain. The
2

---

Under review as a conference paper at ICLR 2020
importance weights and normalized importance weights are w (3
(.())....,,, 2
(z())) (2(2))
(x(x)) i0
(3)
Po (2) pi (22)
Pk (2,)
Σ
im W
M
At the k-th step, the unbiased partition function estimate of pı (z) is 2x =
Σm= w.. 1 M
At the ki-th step, we define the AIS distribution AIS (z) as the distribution obtained by first sampling
z,....,,, 2 M from the M parallel chains using the forward distribution (z(z)...., b,)) and then
ce-sampling these samples based on t.g. More formally, the AIS distribution is defined as follows:
aA (z) = -TTM, 97
vof0lz - -1]]
Bidirectional Monte Carlo. We know that the log partition function estimate log is a stochastic lower bound on log Z (Jensen's inequality). As the result, using the forward AIS distribution as the proposal distribution results in a lower bound on the data lig-likelihood. By running AIS in reverse, however, we obtain an upper bound on log Z. However, in order run the AIS in reverse, we need exact samples from the true posterior, which is only possible on the simulated data. The combination of the AIS lower bound and upper bound on the log partition function called bidirectional Monte Carlo (BDMC), and the gap between these bounds is called the BDMC gap (Grosse et al., 2015). We note that AIS combined with BDMC has been used to estimate log-likelihoods for implicit generative models (Wu et al., 2016). In this work, we validate our AIS experiments by using the BDMC gap to measure the accuracy of our partition function estimators.
2.2 RATE DISTORTION THEORY
Let x be random variable that comes from the data distribution Pa(x). Shannon's fundamental compression theorem states that we can compress this random variable losslessly at the rate of H(x). But if we allow lossy compression, using the code z, and have a lossy reconstruction x = f(z) with the distortion of D, given a distortion measure d(x,, x) = d(x, fz)). The rate distortion theory quantifies the trade-off between the lossy compression rate I and the distortion D. The rate distortion function R(D) D) is defined as the minimum number of bits per sample required achieve lossy compression of the data such that the average distortion measured by the distortion function is less than D. Shannon's 's rate distortion theorem states that RDD) equals the minimum of the following optimization problem:
compress at the rate of R, where R < H(x),
min I(z;x) s.t. Ex(x..) [d(x, fz)) $ D. ((2/x))
(5)
where the optimization is over the channel conditional distribution q(z|x). Suppose the data- distribution is px(x). The channel conditional q(z(x) induces the joint distribution ((z, = (x(x))(z)x). which defines the mutual information X(z; q(z) is the marginal distribution over z of the joint distribution g(2, x), and is called the output marginal distribution. We can rewrite the optimization of Eq. 5 using the method of Lagrange multipliers as follows:
min I(z; x) + E) [d(x, [())]] x()(x)
(6)
2.3 IMPLICIT GENERATIVE MODELS
The goal of generative modeling is to learn a model distribution p(x) to approximate the data distri- bution (x). Implicit generative models define the model distribution p(x) using a latent variable z with a fixed prior distribution p(z) such as Gaussian distribution, and a decoder or generator net- work which computes X = fz). In some cases (e.g. VAEs. AAEs). the generator explicitly param- eterizes a conditional distribution p(x|z), such as a Gaussian observation model N (xx; (z), I²I). But in implicit models such as GANs, the generator directly outputs x fzz). In order to treat VAEs and GANs under a consistent framework, we ignore the Gaussian observation model of VAEs (thereby treating the VAB decoder as implicit model), and use the squared error distortion of f(x)f(z) = ||x f() 12. However, we note that it is also possible to assume a Gaussian ob- servation model with fixed o² for GANs, and use the Gaussian negative log- likeeloodoo (NLL) as the distortion measure for both VAEs and GANs: d(x, fz)) = Log.N((x Vxxff((x)) f(z), I'I.. This is equivalent to squared error distortion up to linear transformation.
3

---

Under review as a conference paper at ICLR 2020
3 RATE-PRIOR DISTORTION FUNCTIONS
In this section, we describe the rate-prior distortion function, as variational upper bound on the true rate distortion function.
3.1 VARIATIONAL BOUNDS ON MUTUAL INFORMATION
We must modify the standard rate distortion formalism slightly in order to match the goals of gener- ative model evaluation. Specifically, we are interested in evaluating lossy compression with coding schemes corresponding to particular trained generative models, including the fixed prior p(z). For models such VAEs, (L(g(z) ())(())))ll(()) is standardly interpreted as the description length of z. Hence, we adjust the rate distortion formalism to use (xp(x) KL (2) (x)])((z)) in place of I(x, z), and refer to this as the rate-prior objective. The rate-prior objective upper bounds the standard rate:
I(x: z) S I(x:2) (L p(z)) ( (g
In the context of variational inference, ((((x) x) is the posterior, ((z) aggregated posterior (Makhzani et al., 2015), and p(z) is the prior. In the context of rate distortion theory, q(z is the channel conditional, (z) is the output marginal, and p(z) is the variational output marginal distribution. The inequality is tight when p(z) q(z), i.c., the variational output marginal (prior) is equal to the output marginal (aggregated posterior). We note that the upper bound of Eq. has been used in other algorithms such as the Blahut-Arimoto algorithm (Arimoto, 1972) or the variational information bottleneck algorithm (Alemi et al., 2016).
(x(x))(z)x)dx is the
3.2 RATE-PRIOR DISTORTION FUNCTIONS
Analogously to the rate distortion function, define the rate-prior distortion function (D) as the minimum value of the rate-prior objective for given distortion D. More precisely, R2 (D) is the solution of
min E (x) (L < xx)
s.t. (x) [d(x, f(z)) < D.
We rewrite the optimization of Eq. 8 using the method of Lagrange multipliers as follows:
(L)()))) + ((3)x)
min
Conveniently, the Lagrangian decomposes into independent optimization problems for each x, al- lowing to treat this as an optimization problem over (z)xx) for fixed x. We can compute the rate distortion curve by sweeping over |3 rather than by sweeping over D.
Now we describe some of the properties of the rate-prior distortion function R,,,), which are straightforward analogues of well-known properties the rate distortion function.
Proposition 1. Rpp has the following properties:
(a) (D) is non-increasing and convex function of D.
(b) We have R(D) = minp(x) (D). As a corollary, for any p(z), we have R(D) S (D).
(c) The rate-prior distortion optimization of Eq. 9 has unique global optimum which can be expressed as 4 (z|x) 2a(x) pzz) (xp(-5((x, fz))).
Proof. The proofs are provided in Appendix C.I.
Prop. lb states that for any prior p(z), (D) is variational upper-bound on R(D). More specif- ically, we have R(D) = minp(z) RDD). which implies that for any given B, there exists prior pī(z), for which the variational gap between rate distortion and rate-prior distortion functions at B is zero. Fig. la shows geometrical illustration of Prop. 1b for three values of 3 E {0.25, 1, 1). We can see this figure that all R,, curves are upper bounds on RDD), and for any given 3, Ryg (D) is tangent to both R2p and to the line with the slope of 3 passing through the optimal solution.
4

---

Under review as a conference paper at ICLR 2020
72
2,40)
R00
(a)
(b)
Figure 1: The rate-prion distortion function with (a) arbitrary distortion function d(x, f()) and (b)
negative log-liket logliiielhood distortion function of ((g/(x)))
3.3 RATE-PRIOR DISTORTION FUNCTIONS WITH NLL DISTORTION
If the decoder outputs probability distribution (as in VAE), we can define the distortion metric coincide with the negative log-likelihood (NLL): d(x, fz) some of the properties of the rate-prior distortion functions with NLL distortions.
log We now describe
Proposition The rate-prior distortion function (D) with NLL distortion - -xgg((x(z) has the following properties:
(a) R(D) is lower bounded by the linear function of Ha - D, and upper bounded by the rate-prior distortion function: Ha D < R(D) S (D)
(b) The global optimum of rate-prior distortion optimization (Eq. 9) be expressed as g (z x) 2z(x) p(z)p(((22) where Z = p(z)p(x|z)*dd
(c) At β 1, the negative summation of rate-prior and distortion is the true log- ligeliiel::ood:
Cp = (x)) [x)) - Ra D& 8-1
Proof. The proofs provided in Appendix C.2.
Fig. 1b shows the geometrical illustration of Prop. 2. We can see that according to Prop. 2a the rate distortion function R(D) is sandwiched between the linear function H. D and the rate-prior distortion R2 (D) At 3 = let L and Cp be the negative summation of rate and distortion on the rate distortion and rate-prior distortion curves respectively (shown in Fig. lb). From Prop. 2c know that Lp is the true log-likelihood of the generative model. From Prop. 1b, we can conclude that - max,(a) Cp. This reveals an important relationship between rate distortion theory and generative modeling that was also observed in Lastras (2019): for given generative model with fixed conditional pl 0(x(2) x())), the best log-likelihood C, that can achieved by optimizing the prior p(z) is the L", which can be found by solving the rate distortion problem. Furthermore, the corresponding optimal prior pz(z) is the output marginal the optimal channel conditional of the rate distortion problem at 3 1. Fig. 1b shows the rate-prior distortion function Ry (D) corresponding to (z). In a "good" generative model, where the model distribution is close to the lata-distribution, the negative log-likeihhood Cpp is close to the entropy of data Ha. and the variational gap between (D) and R(D) is tight.
4 BOUNDING RATE-PRIOS DISTORTION FUNCTIONS WITH AIS
In the previous section, we introduced the rate- vate-prior distortion function R, (D) and showed that upper bounds the rate distortion function R(D), However, evaluating R. (D) is also intractable. In this section, we show how we can upper bound (D) using a single run the AIS algorithm.
AIS Chain. We fix temperature schedule ( = So < B1 < < Bn = 2o. For the k-th intermediate distribution, use the optimal channel conditional qu (z)x) and partition function Zi (x), corresponding to points along (D) and derived Prop. lc:
1
(a(x)==(((a)x0) (z)x)d.. (10)
(z|x)=
Th (2)x),, where (z|x) p(z) exp( eexp(_Bxd(xx f(2))), Zz(x) = Ze
5

---

Under review as a conference paper at ICLR 2020
Conveniently, this choice coincides with geometric averages, the typical choice of intermediate dis- tributions for AIS, i., the kd step happens to be the optimal solutions for 3x. This chain is shown in Fig. 2. For the transition operator, k-th step, the rate-prior R4 (x) and the distortion (x(x) are
Hamiltonian Monte Carlo (Neal et al., 20.1). At the
(x)x) - (L(xx())))))
Dx(x) - (xn( f(z))]
(1I)
AIS Rate-Prior Distortion Curve. For each data point x, we run M independent AIS chains, numbered i = I...., M, in the forward direction. At the k-th state of the i-th chain, let z be the state, w be the AIS importance weights, and Wi be the normalized AIS importance weights. We denote the AIS distribution at the k-th step as the distribution obtained by first sampling from all
the M forward distributions (2), 2 [1)x i==:M and then re-sampling the samples based on their normalized importance weights unf (see Section 2.1 and Appendix C.4 for more details). More formally d (z/x)
M f(zz= z/)l-
ATS Ok
[z/x) = BTM (1) (x)
(12)
2=1
Using the AIS distribution dk (z) defined in Eq. 12, we now define the AIS distortion D (x) and the AIS rate-prior 12 (x) follows:
D& x = E (a/x)
R AIS (x) = KL(gi AIS (2|x)((xx()))))
(13)
We now define the AIS rate-prior distortion curve RS (D) (shown in Fig. 1b) as the RD cruve obtained by tracing pairs of (x)) DAIS (x)).
Proposition The AIS rate-prior distortion curve upper bounds the rate-prior distortion function: RAI((D Z (D).
Proof. The proof is provided Appendix C.4.
Estimated AIS Rate-Prior Distortion Curve. Although the AIS distribution can be easily sampled from, its density intractable to evaluate. As the result, evaluating RA) is also intractable. We now propose to evaluate an upper-bound on RAIS (D) by finding an upper bound for (x) and an unbiased estimate for DAIS(xx (x). We use the AIS distribution samples z and their corresponding weights w to obtain the following distortion and partition function estimates:
DA (x) = Σ dix
2 A
1 = M
Σ w..
(14)
Having found the estimates DAIS(X (x) and (x)), we propose to estimate the rate as follows:
R (x) log k
(x)) 33x DAIS
(15)
We define the estimated AIS rate-prior distortion curve R (D) (shown in Fig, 1b) as an RD curve obtained by tracing pairs rate distortion estimates [feis (x), DA (x)).
The estimated AIS rate-prior distortion curve upper bounds the AIS rate-prior dis- tortion curve in expectation: E E(RA (RR)) R. (D). More specifically, we have
Proposition 4.
E ATS (x)] 2 R (x),
E[D AIS (x)] (x()
(16)
Proof. The proof is provided in Appendix C.4.
In summary, from Prop. 1, Prop. 3 and Prop. 4, we can conclude that the estimated AIS rate-prior distortion curve upper bounds the true rate distortion curve in expectation (shown in Fig. 1b):
E (D) 2 (D) 2 (D) 2 R(D).
(17)
In all the experiments, we plot the estimated AIS rate-prior distortion function R (D).

---

Under review as a conference paper at ICLR 2020
(x v(0)
(r6els) - p())p(si))))
axle]x) 2(2)
ge(r))) p(rbe)
Ou de)x) si2 Rom)
2x(x)
Zu(a)
Re(x]
D/5) Byoll Saatala]
Figure 2: AIS chain for estimating the rate-prior distortion function with NLL distortion.
Accuracy of AIS Estimates. While the above discussion focuses obtaining upper bounds, note that AIS is one of the most accurate general-purpose methods for estimating partition functions, and therefore we believe our AIS upper bounds to be fairly tight in practice. In theory, for large number of intermediate distributions, the AIS variance is proportional to 1/M K' (Neal, 2001; 2005), where M is the number of AIS chains and K is the number of intermediate distributions. For the main experiments of our paper, we evaluate the tightness of the AIS estimate by computing the BDMC gap, and show that in practice our upper bounds are tight (Appendix D).
The Rate Distortion Tradeoff in the AIS Chain. Different values of 3 corresponds to differ- ent tradeoffss between the compression rate and the distortion
given generative model. / corresponds to the case where 20 (z|x) = p(z). In this case, the compression rate is zero, and the distortion would be large, since in order to reconstruct x, we simply sample from the prior and generate a random x that is completely independent of x. In this case, the distortion would be (x) Ep(x) [d(x, fz)))] In the case of probabilistic decoders with NLL distortion, another interesting intermediate distribution is 3e 1, where the optimal channel conditional is the true posterior of the generative model (((xx)= = p(z]x). In this case, as shown in Prop. 2c, the summa- tion of the rate-prior and the distortion term the negative of true log- logliikelhhood of the generative model. As 3n + co, the network cares more about the distortion and less about the compression rate. In this case, the optimal channel conditional would be qn (z|x) (x) arg min, d(x, f()).. In other words, since the network only cares about the distortion,
5(2 (x)),),, where
the optimal channel condition puts all its mass on ZyL which minimizes the distortion. However, the network would require infinitely many bits to precisely represent this delta function, and thus the rate goes to infinity.
5 RELATED WORKS
Evaluation of Implicit Generative Models. Quantitative evaluation of the performance of GANs has been a challenge for the field since the beginning. Many heuristic measures have been proposed, such as the Inception score (Salimans et al., 2016) and the Fréchet Inception Distance (FID) (Heusel et al., 2017). One of the main drawbacks of the IS or FID is that a model that simply memorizes the training dataset would obtain near-optimal score. Another, drawback of these methods is that they use a pretrained ImageNet classifier weights which makes them sensitive to the weights (Barratt & Sharma, 2018) of the classifier, and less applicable to other domains and datasets. Another evalua- tion method that sometimes is being used is the Parzen window estimate, which can be shown to be an instance of AIS with zero intermediate distributions, and thus has a very large variance. Another evaluation method of GANs that was proposed in Metz et al. (2018) is measuiring the ability of the generator network to reconstruct the samples from the data distribution. This metric is similar to the distortion obtained at the high-rate regime of our rate distortion framework when 3 + co. Another related work is GILBO (Alemi & Fischer, 2018), which similar our framework does not require the generative model to have tractable posterior and thus allows direct comparison of VAEs and GANs. However, GILBO can only evaluate the performance of the generative model on the simulated data and not the true data distribution.
Rate Distortion Theory and Generative Models. Perhaps the closest work to ours is "Fixing a Broken ELBO" (Alemi et al., 2018), which plots rate-prior distortion curves for VAEs. Our work is different than Alemi et al. (2018) in two key aspects. First, in Alemi et al. (2018) the rate-prior distortion function is evaluated by fixing the architecture of the neural network, and learning the distortion measure d(x, fz)) in addition to learning q(z(x). Whereas, in our definition of rate dis- tortion, assumed the distortion measure is fixed and given by trained generative model. As the result, we plot the rate-price aate-prior distortion curve for a particular generative model, rather than a par- ticular architecture. The second key difference is that, consistent with the Shannon's 's rate distortion
7

---

Under review as a conference paper at ICLR 2020
Deep and Shallow GANs
Comparing GANs on CIFARIO
DNSCOARS
OhOmg CAnto
W
TECOCSANDEO
DCGARSOE
397
15e
D Ostoott on ESquared Emto)
Datortion (Squared Error)
MNIST
(b) CIFAR-10
Figure 3: The rate distortion curves of GANs.
theorem, we find the optimal channel conditional (z(xx by using AIS; while in Alemi et al. (2018), (z|xx) is variational distribution that is restricted to variational family.
See Appendix A for a discussion of related works about practical compression schemes, distortion- perception tradeoffs, and precision-recall tradeoffs.
6 EXPERIMENTS
In this section, we use our rate distortion approximations to answer the following questions: How do different generative models such as VAEs, GANs and AAES perform at different lossy compression rates? What insights can we obtain from the rate distortion curves about different characteristics of generative models? What is the effect of the code size (width), depth of the network, or the learning algorithm on the rate distortion tradeoffs?
6.1 RATE DISTORTION CURVES OF DEEP GENERATIES MODELS
Rate Distortion Curves of GANs. Fig. 3 shows rate distortion curves for GANs trained on MNIST and CIFAR-10. We varied the dimension of the noise vector z, as well as the depth of the decoder. For the GAN experiments on MNIST (Fig. 3a), the label "deep" corresponds to three hidden layers of size 1024, and the label "shallow" corresponds to one hidden layer of size 1024. We trained shallow and deep GANs with Gradient Penalty (GAN-GP) (Gulrajani et al., 2017) with the code size d E [2, 5. 10, 100} on MNIST. For the GAN experiments on CIFAR-10 10 (Fig. 3b), we trained the DCGAN (Radford et al., 2015), GAN with Gradient Penalty (GP) (Gulrajani et al., 2017), SN- GAN (Miyato et al., 2018), and BRE-GAN (Cao et al., 2018) with the code size of d E (2, 10, 100). In both the MNIST and CIFAR experiments, we observe that in general increasing the code size has the effect of extending the curve leftwards. This is expected, since the high-rate regime is effectively measuring reconstruction ability, and additional dimensions in z improves the reconstruction.
We can also observe from Fig. 3a that increasing the depth pushes the curves down and the left. In other words, the distortion in both high-rate and mid-rate regimes improves. In these regimes, increasing the depth increases the capacity of the network, which enables the network make better use of the information in the code space. In the low-rate regime, however, increasing the depth, similar to increasing the latent size, does not improve the distortion. Comparing GANS, VAES and
Rate Distortion Curves of VAEs. Fig. 4 com- pares VAES, AAES and GP-GANs (Gulrajani al., 2017) with the code size of E (2, 10, 100), and the same decoder architecture on the MNIST dataset. In general, we can see that in the mid-rate to high-rate regimes, VAEs achieve better distortions than GANs with the same architecture. This is expected as the VAE is trained with the ELBO objective, which en- courages good reconstructions (in the case of factor- ized Gaussian decoder). We can see from Fig. 4 that in VAEs, increasing the latent capacity pushes the rate distortion curve up and to the left. In other words, in contrast with GANs where increasing the latent capacity always improves the rate distortion curve, in VABs, there is trade-off whereby
CECAAG
ANU
ANN330
28
Figure 4: RD curves of VAES, GANs, AABs.
8

---

Under review as a conference paper at ICLR 2020
Mixture of Priors
VAES vs. Guassian Burred VAES
2
Oederiann Disaaro Droat
Dkdlation SSeuared Brors
(a) Mixture Prior
(b) Blurring Conditional Likelihood
Figure 5: (a) Effect of damaging the prior of VAE by using a mixture with a bad prior. (b) Effect damaging the conditional likelihood of VAE by adding a Gaussian kernel after the last decoder layer.
increasing the capacity reduces the distortion at the high-rate regime, at the expense of increas- ing the distortion in the low-rate regime (or equivalently, increasing the rate required to adequately approximate the data).
We believe the performance drop of VAES in the low-rate regime is symptomatic of the "holes problem" (Rezende & Viola, 2018: Makhzani et al., 2015) in the code space of VAEs with large code size: because these VAEs allocate large fraction of their latent spaces to garbage images, requires many bits to get close to the image manifold. Interestingly, this trade-off could also help explain the well-known problem of blurry samples from VAEs: order to avoid garbage samples (corresponding to large distortion in the low-rate regime), one needs to reduce the capacity, thereby increasing the distortion at the high-rate regime. By contrast, GANs do not suffer from this tradcoff, and one can train high-capacity GANs without sacrificing performance in the low-rate regime.
Rate Distortion Curves of AAEs. The AAE was introduced by Makhzani et al. (2015) to address the holes problem of VAEs, by directly matching the aggregated posterior the prior in addition to optimizing the reconstruction cost. Fig. 4 shows the RD curves of AAEs. In comparison to GANs, AAES can match the low-rate performane of GANs, but achieve a better high-rate performance. This is expected as AAES directly optimize the reconstruction cost as part of their objective. In comparison to VAEs, AAES perform slightly worse the high-rate regime, which is expected as the adversarial regularization of AAEs is stronger than the KL regularization of VAEs. But AAEs perform slightly better in the low-rate regime, as they can alleviate the holes problem some extent.
6.2 DISTINGUISHING DIFFERENT FAILURE MODES IN GENERATIVE MODELING
Since log- ligliikeisoods constitute only a scalar value, they are unable to distinguish different aspects of a generative model which could be good or bad, such as the prior or the observation model. Here, we show that two manipulations which damage a trained VAE in different ways result in very different behavior of the RD curves.
Our first manipulation, originally proposed by Theis et al. (2015), is to use a mixture of the VAE's density and another distribution concentrated away from the data distribution. As pointed out by Theis et al. (2015), this results in a model which achieves high log-likelihood while gen- erating poor samples. Specifically, after training the VAE10 on MNIST, we "damage" its prior p(z) = =0 I) by altering it to a mixture prior (1 - a))((z) + (z(2), where q(2) = (0,, 101) is a "poor" prior, which is chosen to be far away from the original prior p(z); and a is close to 1. This process would results in a "poor" generative model that generates garbage samples most of the time (more precisely with the probability of a). Suppose p(x) and q(x) are the likelihood of the good and the poor generative models. It is straightforward to see that log g(x) is at most 4.6 nats worse that log p(x), and thus likelihooo fails to tell these models apart:
log xxx) = log (0.01p(x) + 0.99 f ((z)p(x|z))dz) log(O.0lp(x)) v logp(x) - 4.6 (18)
Fig. 5a plots the rate distortion curves of this model for different values of a. We can see that the high-rate and likeliiodd performance of the good and poor generative models are almost identical, whereas in the low-rate regime, the RD curves show significant drop in the performance and successfully detect this failure mode of likeliodl.
9

---

Under review as a conference paper at ICLR 2020
Deep and Shallow GANs
Comparing GANS, VAES and AAES
CF-CAR2
OhOxep- SAI0E
VAC2
0hShalre Gh103
AKC2
2003 Distorrion (MSE on Deep Features)
1093
15:0
Disontion IMSE on Deep Festurest
(a) GANs
(b) VAE, GANs and AAEs
Figure 6: The RD curves of GANs, VABS and AABS with MSE distortion on the deep feature space. The behavior is qualitatively similar to the results for MSE in images (see Fig. 3 and Fig. 4), suggesting that the RD analysis is not particularly sensitive the particular choice of metric.
Our second manipulation is to damage the decoder by adding a Gaussian blur kernel after the output layer. Fig. 5b shows the rate distortion curves for different standard deviations of the Gaussian kernel. We can see that, in contrast to the mixture prior experiment, the high-rate performance of the VAE drops due to inability of the decoder to output sharp images. However, can also an improvement in the low-rate performance of the VAE. This is because (similarly to log-likelihoods with Gaussian observation models) the data distribution does not necessarily achieve the minimal distortion, and in fact, in the extremely low-rate regime, blurring appears to help by reducing the average Euclidean distance between low-rate reconstructions and the input images. Hence, our two manipulations result in very different effects to the RD curves, suggesting that these curves provide a much richer picture of the performance of generative models, compared to scalar loglliielsooods
6.3 BEYOND PIXELWISE MEAN SQUARED ERROR
The experiments discussed above all used pixelwise MSE as the distortion metric. However, for natural images, could use more perceptually valid distortion metries such SSIM (Wang al., 2004), MSSIM (Wang et al., 2003), or distances between deep features of a CNN (Johnson et al., 2016). Fig. 6 shows the RD curves of GANs, VAEs, and AAFS on the MNIST dataset, using the MSE on the deep features of CNN as distortion metric. In all cases, the qualitative behavior of the RD curves with this distortion metric closely matches the qualitative behaviors for pixelwise MSE. We can see from Fig. 6a that similar to the RD curves with MSE distortion, GANs with different depths and code sizes have the same low-rate performance, but as the model gets deeper and wider, the RD curves are pushed down and extended to the left. Similarly, we can see from Fig. 6b that compared to GANs and AAES, VAEs generally have better high-rate performance, but worse low- rate performance. The fact that the qualitative behaviors of RD curves with this metric closely match those of pixelwise MSE indicates that the results of our analysis are not overly sensitive to the particular choice of distortion metric.
CONCLUSION
In this work, we studied rate distortion approximations for evaluating different generative models such VAEs, GANs and AAEs. We showed that rate distortion curves provide more insights about the model than the log-likelihoog alone while requiring roughly the same computational cost. For instance, we observed that while VAEs with larger code size can generally achieve better lossless compression rates, their performances drop at lossy compression in the low-rate regime. Conversely, expanding the capacity of GANs appears to bring substantial reductions in distortion at the high-rate regime without any corresponding deterioration in quality in the low-rate regime. This may help explain the success of large GAN architectures (Brock al., 2019; Karras et al., 2018); We also discovered that increasing the capacity of GANs by increasing the code size (width) has very different effect than increasing the depth. The former extends the rate distortion curves leftwards, while the latter pushes the curves down. We also found that different GAN variants with the same code size has almost similar rate distortion curves, and that the code size dominates the algorithmic differences of GANs. Overall, lossy compression yields a richer and more complete picture of the distribution modeling performance of generative models. The ability to quantitatively measure performance tradeoffs should lead algorithmic insights which can improve these models.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alexander Alemi, Ben Poole. Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken ELBO. In International Conference on Machine Learning, pp. 159-168, 2018.
Alexander A Alemi and lan Fischer. GILBO: one metric to measure them all. In Advances Neural Information Processing Systems, pp. 7037-7046, 2018.
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv: 2016.
Suguru Arimoto. An algorithm for computing the capacity of arbitrary discrete memoryless chan- nels. IEEE Transactions on Information Theory, 18(1):14-20, 1972.
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein GAN.
arXiv preprint
arXiv:1701.07877 2017.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with scale hyperprior. arXiv preprint arXiv:/802.0144 18800.11133, 2018.
Shane Barratt and Rishi Sharma. note on the inception score. arXiv preprint arXiv:180101193. 2018.
Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The rate-distortion-percpppi tradeoff. arXiv preprint UrXiv:1901.0782) 2019.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Yanshuai Cao, Gavin Weiguang Ding, Kry Yik-Chau Lui, and Ruitong Huang. Improving GAN training via binarized representation entropy (bre) regularization. ICLR, 2018, accepted as poster.
Ivo Danihelka, Balaji Lakshminarayanan, Benigno Uria, Daan Wierstra, and Peter Dayan. Com- parison of maximum likelihood and GAN-based training of real NVPs. arXiv preprint 17Xiv: 2017.
Justin Domke and Daniel R Sheldon. Importance weighting and variational inference. In Advances in neural information processing systems, pp. 4470-4473 2018.
Brendan Frey and Geoffrey E Hinton. Free energy coding. Proceedings of Data Compression Conference-DCCC%% pp. 73-81. IEEE, 1996.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Roger Grosse, Zoubin Ghahramani, and Ryan P Adams. Sandwiching the marginal likelihood using bidirectional Monte Carlo. arXiv preprint arXiv: 1.1002543, 2015.
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-GAN: Combining maximum likelihood and adversarial learning in generative models. In Thirty-Second AAAI Conference Artificial Intelligence, 2018
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein GANs. In Advances Neural Information Processing Systems, pp. 5767-57777 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach. R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems 30, pp. 6626-6637. Curran Associates, Inc.,
2017.
11

---

Under review as a conference paper at ICLR 2020
Geoffrey Hinton and Drew Van Camp. Keeping neural networks simple by minimizing the descrip- tion length of the weights. In in Proc. of the 6th Ann. ACM Conf. on Computational Learning Theory. Citeseer, 1993.
Gao Huang, Yang Yuan, Qiantong Xu, Chuan Guo, Yu Sun, Felix Wu, and Kilian Weinberger. An empirical study on evaluation metrics of generative adversarial networks, 2018.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018a.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812.04944 2018b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Abss/1412.6900, 2014.
CoRR,
Diederik P Kingma and Max Welling, Auto-encoding variational Bayes, arXiv:1312.61 2013.
arXiv preprint
Friso H Kingma, Picter Abbeel, and Jonathan Ho. Bit-swap: Recursive bits-back coding for lossless compression with hierarchical latent variables. arXiv preprint arXiv: 1XXX:::900.008535 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Luis A Lastras. Information theoretic lower bounds negative log likelihood. arXiv preprint arXiv:1904.06:: 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-22222 2324,, 1998.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs created equal? large-scale study. In Advances in neural information processing systems, pp- 700-709, 2018.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.05644 2015.
L Metz, B Poole, D Pfau, and Sohl-Dickstein. Unrolled generative adversarial networks. arXiv preprint wwXiv:1611.021e6 2018
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint wwXiv:1802.059555 2018.
Radford M Neal. Annealed importance sampling. Statistics and computing, 1 11(2):12--13)) 139, 2001.
Radford Neal. Estimating ratios of normalizing constants using linked importance sampling. arXiv preprint math/051/216, 2005.
Radford M Neal al. MCMC using Hamiltonian dynamics. Handbook of markov chain Monte Carlo, 2(11):2, 2011.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint
2015.
Danilo Jimenez Rezende and Fabio Viola. Taming VAEs, arXiv preprint arXiv:1810.00597 2018
Mchdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assess- ing generative models via precision and recall. Advances Neural Information Processing Systems, pp. 5228-5237, 2018.
12

---

Under review as a conference paper at ICLR 2020
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pp. 1234-22422 2016.
Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving GANs using optimal transport. In International Conference on Learning Representations, 2018.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. arXiv preprint 1110v4451 2015.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. arXiv preprint arXiv: 2017
James Townsend, Tom Bird, and David Barber. Practical lossless compression with latent variables using bits back coding. arXiv preprint arXiv:1901..4866, 2019.
Chris S Wallace. Classification by minimum-messsage-lrn.. inference. In International Conference on Computing and Information, pp- 72-81. Springer, 1990.
Zhou Wang, Eero Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2. pp. 1398-1402. Ieee, 2003.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Yuhuai Wu, Yuri Burda, Ruslan Salakhutdinov, and Roger Grosse. On the quantitative analysis of decoder-based generative models. arXiv preprint 11.04161,. 2016.
13

---

Under review as a conference paper at ICLR 2020
APPENDIX A RELATED WORKS
Practical Compression Schemes. We have justified our use of compression terminology in terms of Shannon's fundamental result implying that there exist rate distortion code for any rate dis- tortion pair that is achievable according to the rate distortion function. Interestingly, for lossless compression with generative models, there is practical compression scheme which nearly achieves the theoretical rate (i.e. the negative BLBO): bits-back encoding. The basic scheme was proposed by Wallace (1990); Hinton & Van Camp (1993), and later implemented by Frey & Hinton (1996). Practical versions for modern deep generative models were developed by Townsend et al. (2019); Kingma et al. (2019). We do not currently know of analogous practical scheme for lossy com- pression with deep generative models. Other researchers have developed practical coding schemes achieving variational rate distortion bounds for particular latent variable models which exploited the factorial structure of the variational posterior (Ballé et al., 2018; 88; Theis et al., 2017). These meth- ods are not directly applicable in our setting, since we don't 't assume an explicit encoder network, and our variational posteriors lack a convenient factorized form. We don't know whether our varia- tional approximation will lead to a practical lossy compression scheme, but the successes for other variational methods give us hope.
Relationship with the Pate-Distortion-Pereettii Tradeoff. Our work is related to Blau & Michaeli (2019) which incorporates perceptual quality loss function in the rate-distortion frame- work and characterizes the triple tradeoff between rate distortion and perception. More specifically, Blau & Michaeli (2019) defines the perceptual loss using divergence between the marginal re- construction distribution and the data distribution. This perceptual loss is then incorporated as an additional constraint in the rate-distortion framework encourage the reconstruction distribution to perceptually look like the data distribution. It is shown that as the perceptual constraint becomes tighter. the rute-distortion function elevates more. In our rate-prior distortion framework, we are also enforcing perceptual constraint on the reconstruction distribution by incorporating the regu- larization term of (T.(g(z) [lp(z)) in the rate-distortion objective, which encourages matching the aggregated posterior to the prior (Makhzani et al., 2015). More precisely, let us define the recon- struction distribution r(x) as the the distribution obtained by passing the data distribution through the encoder and then the decoder:
r(x, 2) = (2))p(x(2) & ~~ r(x) = / r(xx, 2)dz.
(19)
can be shown that the regularization term (L(((z)IP((z) upper bounds (L((()))l(xx))
(L((())lp(z)) KL(r(x, (L((xx,)))lx(xx z)) 2 (L(r(x)
(20)
In other words, in the rate-prior distortion optimization, for given distortion constraint, we are not only interested in minimizing the rate I(x; z). but also at the same time, we are interested in preserving the perceptual quality of the reconstruction distribution by matching it to the model distribution. In the low-rate regime, when the model is allowed to have large distortions, the model obtains small rates and at the same time preserves the perceptual distribution of the reconstruction samples. As the distortion constraint becomes tighter, the model starts to trade off the rate I I(x; z) and the perceptual quality (L(((z) (zz). which results in an elevated rate distortion curve.
Relationship with the Precision- Tradeoff. One of the main drawbacks of the IS or FID is that they can only provide a single sealar value that cannot distinguish the mode dropping behav- ior from the mode inventing behavior (generating outlier garbage samples) in generative mod- els. In order to address this issue, Lucic et al. (2018), 8); Sajjadi et al. (2018) propose to study the precision-recall tradoff for evaluating generative models. In this context, high precision implies that the samples from the model distribution are close to the data distribution, and high recall implies the generative model can reconstruct (or generate a sample similar to) any sample from the data distribution. The precision-recit curves enable us to identify both the mode dropping and the mode inventing behavior of the generative model. More specifically, mode dropping drops the precision of the model at the high-recall regime, and mode inventing drops the precision of the model in the low-recall regime. Our rate-prior distortion framework can be thought as the information theoretic analogue of the precision-recall curves, which extends the scalar notion of log- ligliikll to rate distortion curves. More specifically, in our framework, mode dropping drops the distortion perfor- mance of the model at the high-rate regime, and mode inventing drops the distortion performance of the model at the low-rate regime. In Section 6, we will empirically study the effect of mode dropping and mode inventing on our rate-prior distortion eurves.
14

---

Under review as a conference paper at ICLR 2020
APPENDIX B AIS VALIDATION EXPERIMENTS
AIS Settings. All the AIS settings including the HMC parameters are provided in Appendix D.2.
AIS Validation. We conducted several experiments to validate the correctness of our implementation and the accuracy of the AIS estimates. Firstly, compared our AIS results with the analytical solution of rate-prior dis- tortion curve on a linear VAE (derived in Appendix D.3.1)
Ais Analyties
trained on MNIST. As shown in Fig. 7, the RD curve esti- mated by AIS agrees closely with the analytical solution. Secondly, for the main experiments of the paper, we eval- uated the tightness of the AIS estimate by computing the BDMC gap. The largest BDMC gap for VAEs and AAEs was 0.127 nats, and the largest BDMC gap for GANs was 1.649 nats, showing that our AIS upper bounds are tight. More details are provided Appendix D.3.2.
741 722 742 741 715
766 700
346
Seartnnhhuaabe Log Und
Figure Analytical vs. AIS rate distortion curves for linear VAF.
APPENDIX C PROOFS
C.I
PROOF OF PROP.
Proof of Prop. la. As D increases, (D) is minimized over a larger set, so Rp(D) is non- increasing function of D.
The distortion 1x [d(xx, f())) is a linear function of the channel conditional distribution g(2)x)-
The mutual information is a convex function of q())x). The (L(((z))lp(z)) is also convex function of q(z), which itself is linear funetion of q(z[x). Thus the rate-prior objective is
a convex function of ((][x). Suppose for the distortions D, and D2, (1(z)x) and ga (z)xx x) achieve the optimal rates in Eq. respectively. Suppose the conditional qx (x(z)x) is defined as 4x (z(x) = Ag1(2|x) + (1 - (2)x( The rate-prior objective that the conditional qx (z x) achieves is xz: (L (z) [[())) and the distortion D> that this conditional achieves
D> = AD1 + (1 22. Now we have
(Dx) < IA (z; x) + KL (z))((()))
(21)
$ AI x + AKL (q (z)llp())) + (1- A)[2(z + ())L())()))
= AR (i) + (1 - (D2)
which proves the convexity of (D).
Alternative Proof of Prop. la. We know the rate-prior term (L_(,,)L((()))))llp is convex
function of g(2 x), and Bxyx.2) [d(x, [())) is a linear and thus convex function of g(z(x). As the result, the following optimization problem is convex optimization problem.
min E px)) (L(((2)))lp(())
8.l. Bg(x,2) [d(x (z)) s 0.
(22)
((xxx)
The rate distortion function (D) is the perturbation function of the convex optimization problem of Eq. 22. The convexity of R2 (D) follows from the fact that the perturbation function of any convex optimization problem is convex function (Boyd & Vandenberghe, 2004).
Proof of Prop. 1b. We have
min (D) min min p(z)
I(x;2) (L( (p())
(23)
p(x) dltbs
min min I(x;)) (L(g(z) [p())
(24)
B[cx) p(x)
min 2(x;
(25)
R(D).
(26)
where in Eq. 24, we have used the fact that for any function fæ, y). we have
min min f(x, v) = min min fa 3) = min f(z, v),
(27)
2.8
15

---

Under review as a conference paper at ICLR 2020
and in Eq. 25, we have used the fact that (L(((z)[lp(z)) is minimized when p(z) q()).
Proof of Prop. Ic. In Prop. la, we showed that the rate-prior term convex function of g( x))
and that the distortion is a linear function of g(z|x). So the summation of them in Eq. will
convex function of q(z|x). The unique global optimum this convex optimization can be found by
rewriting Eq. 9 as
1 Kl(g(z)))lx(()) + (B)))) f())) = KL(g(z|x)] 2(x) exp( - log (x(x)
(28)
where Zz(x) p(z) exp(-Ad(x, fz))) The minimum of Eq. 28 is obtained when the KL di-
vergence is zero. Thus the optimal channel conditional is (z)xx) Zz(x) Pa(x) p(z) exp(- fz))).
C.2 PROOF OF PROP. 2.
Proof of Prop. 2a. R(D) & (D( was proved in Prop. lb. To prove the first incquality, note that the summation of rate and distortion is
(D( D I(z; + Ey Ex logp((z)) = Hy - (x (x]](((((x 2 Ha.
(29)
where (x(xx z) is the optimal joint channel conditional, and (z) and q² (x(2) are its marginal and conditional. The equality happens if there is joint distribution q(x, 2), whose conditional ((xlz) = p(x[z), and whose marginal over x is px(x). But note that such a joint distribution might not exist for an arbitrary p(x(z).
Proof of Prop. 2b.
The proof can be easily obtained by using d(x, f(z)) = - llgg((x in
Prop. lc.
Proof of Prop. 2c, Based on Prop. 2b, at B = 1, we have
Zz(x) / p(=))(x(2)dz==ppp p(xx)
(30)
C.3 PROOF OF PROP. 3.
The set of pairs of (x), (x)) are achievable rate-prior distortion pairs (achieved by AAl (z(x)). Thus, by the definition of (D). RDD)(D falls in the achievable region of RR D) %%
and, thus maintains upper bound it: RR (DD) (D).
C.4 PROOF OF PROP. 4.
AIS has the property that for any step & of the algorithm, the set of chains up to step k. and the partial computation of their weights, can be viewed as the result of a complete run of AIS with target distribution of (z|x). Hence, we assume without loss of generality that we are looking at complete run of AIS (but our analysis applies to the intermediate distributions as well).
Let (z|x) denote the distribution of final samples produced by AIS. More precisely, it is dis- tribution encoded by the following procedure:
1. For each data point x, we run M independent AIS chains, numbered i 1, I,.., M. Let x denotes the k-th state of the i-th chain. The joint distribution of the forward pass up to the k-th state is denoted by q (z'i......, z x). The un-normalized joint distribution of the backward pass is denoted by
(z""").... 2x(xx) = p(z") exp( Bed(x, f(x"))) "o(z"i....,,'"" (i.i x).
(31)
2. Compute the importance weights and normalized importance weights of each chain using w (x'i bx) and पं ΣM
3. Select chain index S with probability of oi
16

---

Under review as a conference paper at ICLR 2020
4. Assign the selected chain values to (z), ........ x
(z],...,Z{))
(z) 2 = (z',....,''') (.)).
(32)
5. Keep the unselected chain values and re-label them as (z 2M
Zk ))
(z)
- = (z'
(33)
where -S denotes the set of all indices except the selected index S.
6. Return z
More formally, the AIS distribution is
M > uz
AIS (z)x)
TT or(2") xxxx)
(34)
Using the AIS distribution (z)x) defined above, we define the AIS distortion D (x) and the
AIS rate-prior R (x) (T. (z|x) (p(()) as follows:
D AIS (x) Bi (ete) f())
(35)
{xx (x) = KL
AIS (z)x)llp()))
(36)
In order to estimate x (x) and (xx), we define
Dx Σ
w d(x f(z(t))
(37)
2 (x)
1
w
(38)
M
R
ÁAS (x) = log 2,5 (x) BxxDx (xx)).
(39)
We would like to prove that
E I oy (ort..... (DAiS (xx] DA
(40)
E IIM, ( RAIS (x) 2 (x).
(41)
The proof Eq. 40 is straightffrward: f()))]
D (x)
/
AIS (z)x)d(xx f()))
9h
M
/ ITT (r(x
r ö(z z dx)),((x, f(z))dz,
M
(42)
ūi / ō(z - z 2'x) f(z)
1=1
w Wi f(z
1=1
ETT (y ( 2 [D (x)).
Eq. 42 shows that DAIS (x) is an unbiased estimate A (x). We also know log (x)) obtained by Eq. 38 is the estimate of the log partition function, and by the Jenson's inequality lower bounds in expectation the true log partition function: Ellog A1 (x) S log Ze (x). After obtaining DAIS (x)
17

---

Under review as a conference paper at ICLR 2020
and log 2 (x15(x),
Eq. 39 to obtain R& (x). Now, it remains prove Eq. 41, which states
that R (x) upper bounds the AIS rate term R (x) expectation.
Let A (z):A1 ..... L. (x) denote the joint AIS distribution over all states of (z!-M.
2k 2:30 }.
defined in Eq. 32 and Eq. 33. It can shown that (see Domke & Sheldon (2018))
2k x) (6(z)..... z x) I 9s (z) x
ge (z) AM
(43)
7 (x)
p(z)) exp( (xx(-x f (z).......
12 x) I (r
z (x)
(x)
(44)
In order to simplify notation, suppose 2 is denoted by 2, and all the other variables {z] l:M ...., 2- k-1 2/ M) are denoted by V. Using this notation, we define pl p(V]z,x) x) and
I:M
ot (z. V(x) as follows:
AIS
M
p( (V|z,, x) (z),.
2/_ 2k
x)
II
as (5 (2),.... ....... 211x),
(45)
i=2
AIS z V(x) == lk (z] I:M
2/
1:M
(x)
(46)
Using the above notation, Eq. 44 can be re-written as
2 (x) p(z) exp( - BLd(x, f(z)) P( (V
(47)
lk (z, V(x)
Hence,
E[log 2A1 (x)] Ellog (zz) - log 22 (z. Vxx) + log. V (xE|d(
KLL (z. (1x)ll(2)p(([zz>>>)
B, f()))
(48)
S -KL( A [z)))]lp())) - 3x (z)),
where the inequality follows from the monotonicity of KL divergence. Rearranging terms, we bound the rate:
R (x) = (z|x) < Elloo 2 (x)] - f(x( = (E[AA((x))
(49)
Eq. 49 shows that RA (x) upper bounds the AIS rate- rate-prior RAI (x) in expectation. We also showed
DAIS (x) is an unbiased estimate of the AIS distortion DA (x). Hence, the estimated AIS rate-prior curve upper bounds the AIS rate-prior distortion curve in expectation: EE[RA'S (D) 2 R (D).
APPENDIX D EXPERIMENTAL DETAILS
The code for reproducing all the experiments of this paper will be open sourced publicly.
D.1 DATASETS AND MODELS
We used MNIST (LeCun et al., 1998) and CIFAR-10 (Krizhevsky & Hinton, 2009) datasets in our experiments.
Real-Valued MNIST. For the VAE experiments on the real-valued MNIST dataset (Fig. 4a), we used the "VAE-50" architecture described in (Wu et al., 2016), and only changed the code size in experiments. The decoder variance is a global parameter learned during the training. The network was trained for 1000 epochs with the learning rate of 0.0001 using the Adam optimizer (Kingma & Ba, 2014).
For the GAN experiments MNIST (Fig. 3a), used the "GAN-50" architecture described in (Wu et al., 2016). In order to stabilize the training dynamic, we used the gradient penalty (GP) (Salimans al., 2016). In our deep architectures, we used code sizes of d E {2, 5, 10, 100) and three hidden
18

---

Under review as a conference paper at ICLR 2020
VAE10 Preliminary and Loaded Runs
meimrarryocc
Isste 11:1
loaded,
Rato
10
Distortion (Squared Emror)
Figure 8: The rate-prior distortion curves obtained by adaptively tuning the HMC parameters in the preliminary run, and pre-loading the HMC parameters in the second formal run. "rs" in the legend indicates the random seed used in the second run.
layers each having 1024 hidden units to obtain the following GAN models: Deep-GAN2, Deep- GAN5, Deep-GAN10 and Deep-GAÑ100, The shallow GANs architectures are similar to the deep architectures but with one layer of hidden units.
CIFAR-10. For the CIFAR-10 experiments (Fig. 3b), we experimented with different GAN mod- els such as DCGAN (Radford al., 2015), DCGAN with Gradient Penalty (GP-GAN) (Gulrajani et al., 2017), Spectral Normalization (SN-GAN) (Miyato et al., 2018), and DCGAN with Binarized Representation Entropy Regularization (BRE-GAN) (Cao al., 2018). The numbers at the end of each GAN name in Fig. 3b indicate the code size.
D.2 AIS SETTINGS FOR RD CURVES
For each RD curve, there 1999 points computed with only AIS chain, for 999 3's spaced linearly from 9max to 1 and another 999 3's spaced linearly from 1 to 9min and plus = 1, thus 1999 points. Amin
for all models. 3mux 0.0003 3333 for 100 dimensional models such as GANI00, VAE100 AAE 100, and 3max 0,00002770224 36098 for the rest (2, and 10 dimensional models). The AIS temperature schedule (f for intermediate distributions) is generated by three steps:
1. Build sigmoid temperature schedule used in Wu al. (2016) for 9max with N intermediate distributions.
2. If there are fewer than 800 intermediate distributions before the first RD point in the temper- ature schedule initialized in Step 1, overwrite this part of the schedule with 800 intermediate distributions linearly spaced between 3o O the first RD point (31).
3. If there are fewer than 10 intermediate distributions between two RD points in the temper- ature schedule initialized in Step I, overwrite this part of the schedule with 10 intermediate distributions linearly spaced between two RD points.
For the 2, and 10 dimensional models, N 40000, and the above procedure will result in 60292 intemediate distributions in total. For 100 dimensional models, to ensure accuracy of our AIS es- timator with small BDMC gap. we used N 1600000 and the above procedure will result in 161 1463 intermediate distributions in total. We used 20 leap frog steps for HMC, 40 independent chains, on a single batch of 50 images. On MNIST, we also tested with a larger batch of 500 MNIST images, but did not observe significant difference compared with batch 50 images, thus did all of our experiments with a single batch 50 images. On a P100 GPU, for MNIST, it takes 4-7 hours to compute RD curve for 60292 intermediate distributions and takes around 7 days for 161 1463 intermediate distributions. For all of the CIFAR experiments, we used the schedule with 60292 intermediate distributions, and each experiment takes about days to complete.
Adaptive Tuning of HMC Parameters. While running the AIS chain, the parameters of the HMC kernel cannot be adaptively tuned, since it would violate the Markovian property of the chain. So in
19

---

Under review as a conference paper at ICLR 2020
order to be able to adaptively tune HMC parameters such the number of leapfrog steps and the step size, in all our experiments, we first do a preliminary run where the HMC parameters are adaptively tuned to yield an average acceptance probability of 65% as suggested in Neal (2001). Then in the second "formal" run, we pre-load and fix the HMC parameters found in the preliminary run, and start the chain with new random seed to obtain the difference in the RD curves obtained from the preliminary run and the formal runs with various different random seeds is very small, as shown in Fig. 8. This figure shows that the AIS with the HMC kernel is robust against different choices of random seeds for approximating the RD curve of VAE1O.
final results. Interestingly, we observed that
D.3 VALIDATION OF AIS EXPERIMENTS
We conducted several experiments validate the correctness of our implementation and the accu- racy of the AIS estimates.
D.3.1 ANALYTICAL SOLUTION OF THE RATE-PRIOR DISTORTION OPTIMIZATION ON THE LINEAR VAE
We compared our AIS results with the analytical solution of the rate-prior distortion optimization on a linear VAE trained on MNIST as shown in Fig. 7.
In order to derive the analytical solution, we first find the optimal distribution (z(2)x) x) from Prop. 2b. For simplicity, we assume a fixed identity covariance matrix I at the output of the conditional likeli- hood of the linear VAE decoder. In other words, the decoder of the VAE is simply: x Wz +- b++ where x is the observation, z is the latent code vector, W the decoder weight matrix and b the bias. The observation noise of the decoder is (O, I). It's easy to show that the conditional likelihood raised to a power is: p(x|z) = (x + b, 3 I). Then, (2(x)) = [Zz2/,,, Eg), where
143 = Bygl [z]) = WT(WWT B II) (x bb) b)
(50)
Σg = Cov [2]) =I WT(WWT + g 1-1 'w
For numerical stability, we can further simplify thw above by taking the SVD of W let W UDVT, and then apply the Woodbury Matrix Identity to the matrix inversion, we can get:
H3 VR, UT - b)
(51)
Σg VS VT
(52)
Where Rg is a diagonal matrix with the 7th diagonal entry being कमउ and S is a diagonal matrix with the 7th diagonal entry being [ where d; is the 7th diagonal entry of D Then, the analytical solution for optimum rate is:
(xz(g)(Z)x)II(z))
DKL(W(zlMs E2)11((().0])
(53)
1
2 (1x())) + (-M3) (-Ms) k In ((det -1)
(54)
2 (tr (Eg) + (m²) (M1) - -k 3--m in (det Ez))
(55)
Where k is the dimension of the latent code z. With negative log-likelihood as the distortion metric, the analytically form of distortion term is:
E o ())
[- log p( p(x|z)]
(56)
llg(((7))
k/2 -k/2 exp { -x- (Wzz+b)) (Wz + b3)
1
(57)
2
-(log((2T)) -k/2]
+
1 10)
{(x- (W2z +b)) - (Wz
(58)
-
1
1
log(2T) + (x-
- b)) (5bbb b) (WH3) b) Bgz(elx) ((W2)
(59)
2
2
20

---

Under review as a conference paper at ICLR 2020
where (x/(x [())]]WWWW] (Wz))(Wz)) can be obtained by change of variable: Let Wz, then:
(x) [x] WMg - U(1 - (x bb)
(60)
Covg- [y] = WE; = DDR
(61)
Bgg (x/x)) [(Zz) = (v) [y = E (v) [y]* Bgr(s) [y] + tr(Cov (- [x])
(62)
(63)
D.3.2 THE BDMC GAP
We evaluated the tightness of the AIS estimate by computing the BDMC gaps using the same AIS settings. Fig. 9, shows the BDMC gaps at diffrent compression rates for the VAE, GAN and AAE experiments the MNIST dataset. The largest BDMC gap for VAEs and AAEs is 0.127 nats, and the largest BDMC gap for GANs is 1.649 nats, showing that our AIS upper bounds are tight.
D.4 HIGH-RATE VS. LOW-RATE RECONSTRUCTIONS
this section, we visualize the high-rate (B 3500) and low-rate (3 = 0) reconstructions of the MNIST images for VAEs, GANs and AAEs with different hidden code sizes. The qualitative results are shown in Fig. 10 and Fig. 11, which in consistent with the quantitative results presented in experiment section of the paper.
21

---

Under review as a conference paper at ICLR 2020
BDMC Gaps for VAES
10a
0.053
VAEZ
VAE100
Rate
0.030
0:812
077
10"
00000
10
30
40
Distortion (Squared Error)
(a)
BDMC Gaps for GANs
GP-CAN2
10a
GP-CIA 100
10a
10a
0.12
0.070
000
40
50
60
Distortion (Squared Error]
(b)
BDMC Gaps for AAES
p.009
AAEZ
AAE100
0.127
Rate
..033
10a
4..09
0.071
0.000
0.019
10a
50
60
Distortion (Squared Error)
(c)
Figure 9: The BDMC gaps annotated estimated AIS Rate-Prior Distortion curves of (a) VAEs, (b) GANs, and (c) AAEs.
22

---

Under review as a conference paper at ICLR 2020
7
2
1
0
4
4
9
5
9
0
6
9
0
1
5 5
9
7
8
4
9
6
6
4
0
1074
4
0
3
3 4
2
1 2
7
4
3
5
4
7
2
0
4
4
9
5
9
0
6
0
(a) Original MNIST test images
8
1
8
9
9
8
5
9
0
0
8
7
3
3
0
8
0 4
3
8
6
6
8 7
0
6
9
5
5
9
5
9
5
5
2
4
8
2
4
9 2
0
7
3
3 6 6 0 4 9 4
4
5
6
0
9
4
6
9
9
6
2
6
0 6
5
6
4
2
6
6 8 8 0 3 4
2
8
5
3
0
9
3
0
6
9
0
3
0
3
0
7
1
0
9
6
4
7
5
3
6
8
8
1
0
2
5
8
7
6
3
2
5 t
8
1
5
5
5
4 9 1
7
3
8
8
4
1
4
3
2
2
2
2
2
9
7
7
6
0
8
5
3
9
4
9
3
7
2
4
7
6
9
2
2
7
2
2
4
O
3
2
7
6
g
(b) Low Rate VAE2
(c) Low Rate AAE2
(d) Low Rate GAN2
8
3
0
9
9
4
2
4 4 3
5
6
6
4 3 1 / 7 2
2
0 0
9
3
7
3
6
2
8
6
3
2
4
9 2 8
7
8 6
6
3
4
3
4
3 0
2
2
7
7
5
2
3
O
6
4
3
7
4 4 a
5 4
1
7
5
4 2
6
1
1
4
/ 7 7 4 0
6
8
8
R
4
0
4
0 7
3
9
8
8
4 5
0 6
0
8
9 6 7 1
9
1
2
6
4
0
4
5 8
5 9
7
7
6
1
2
S
5
4 3
8
7
5
5
6
6
/
7
9
3
/ 7
3 6
2
2
4
]
8
6
9
9
2
L
3
3
5
6
S
3
9
0
9
6
2
^
3
4
6
1
+
3
9
2
2
O
4
2
5
5
3
9
S
3
0
A
(c) Low Rate VAE10
(f) Low Rate AAEIO
(g)
Low Rate GAN10
B
1
$
7
4
9
3
4
0
1
4
1
O
9
7
5
2 6
1
8
9
7
3
8
3
5
3
8
4
8
8
0
3
8
5
D
8
5
0
0
5
4 0
4
2
3
3
4
4 7 4 C
9
3
K
3
9
7
J
7 4 1
2
O
4
4
0
4
6
7
9
3
2
6
7
6
7
4
3
&
4 6
5 2
3
0
7
8 / 8
8
9
2
9
0
9
5
7
4
3
5
3
5
8
6
0
3
3
7
4
2
7
3
2
9
3 4 0
2
5
7
6
7
0
6
9
7
2
8
3
6
3
2
7
2
2
8
9 0
2
5
4
0
0
7 9
7 2
3
9
4
4
6
7
3
7
2
8
4
5
3
4
(h) Low Rate VAE100
(i) Low Rate AAE100
(i) Low Rate GAN100
Figure 10: Low-rate reconstructions (3 0) of VAEs, GANs and AAEs on MNIST.
23

---

Under review as a conference paper at ICLR 2020
7
2
6
4
4
9
5
9
0
6
9
0
1
5
9
7
8
4
9
6
6
5
4
0
7
4
0
1
3
3
4
7
2
2
1
7
4
4
3
5
1 4
2
4
4
2
0
4
9
6
9
0
6
9
0
(a) Original MNIST test images.
7
2
0
4
4
9
7
2
0
9
4
9
7 2
0
4
/
9
9
4
9
0
6
9
0
5
9 9
0
0
9
0
1
5
5
0
6
9
0
1
5
9
7
5
4
9
6
6 3
5
4
3
4
9
6
6
5
9
7
6
4
7
6
6
5
4
0
4
0
1
9
0
7
4
0
3
1
4
0
7
4
0
1
3
3
0
7
2
7
3
1
3
4
7
2
7
1
3
1
3
6
7
2
7
1
3
1
1
7
4
2
5
1 4
1
7
4
3
5
1 9 9
4
7 4
4
6
8
4
4
7
2
0
/
9
9
7
0
/
4
7
5
1
0
9 9
4
9
5
9
O
6
9
0
4
9
5
9
O
6
0
4
6
4
0
0
0
High Rate VAE2
(c) High Rate AAE2
High Rate GAN2
7
2
1
0
4
4
9
7
2
0
4
4
9
2
0
4
1
4
A
5
990
0
6
9
0
1
5
5
9
0
6
9
0
1
5
9
9
0
6
9
0
1
5
9 7
779
3
4
9
6
6 3
5
9
7
3
4
9
6
6
5
9
7
3
4
9
6
6
5
4 1011
4
0
1
4 1012
4
0
1
3 2 1 4
4
0
4
0
1
3
3 4
34727
2 1 4
1
3
84727
4
1
3
4
7
2
7
1
2
1
7
4
4
3
2
7
4
4
3
5
2
7
4
4
3
3
2
4
4
4
2
0
/
4
4
7
2
0
/
4
7
2
0
4
4
9
S
9
0
6
9
0
4
9
6
9
0
6
9
0
5
9
0
6
9
0
(c) High Rate VAE10
(f) High Rate AAE10
(g) High Rate GAN10
7
2
1
0
4
1
4 1
9
7 2
0
4
1
4
9
7
2
1
0
4
/
4
9 5
5
9
0
6
9
0
5
5 9
0
6
9
0
1
5
5 9
0
6
9
0
1
9
7
8
9
6
6 3
5
97
8
4
9
6
6
5
9
7
8
4
9
6
6
5
4
447
0
1
4 401
4
0
1
3
4 10000
4
0
1
3
3 4
7
227
2
1
3 4
7
27
2
7
L
2
1
2 1 4 9
1
7 4
2
5
2
1
7
4
2
5
1 4
2
7 4
4 7
2
3
5
2
4
4
4
2
0
4 9
4
4
7
2
0
/
4
2
0
/
4
9
5
9
0
6
0
4
9
0
6
9
0
4
9
5
9
0
6
0
(h) High Rate VAE100
(i) High Rate AAE100
(i) High Rate GAN100
Figure 11: High-rate reconstructions (Amax) of VAEs, GANs and AAEs on MNIST. Smax 3333 for 100 dimensional models, and 3max = 36098 for the 2 and 10 dimensional models.
24


Reviews:
[{"review_id": "ryldo2qvYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a new way to evaluate generative models that don't have tractable likelihoods, such as VAEs or GANs. Such generative models are composed of a prior over latent variables and a decoder that maps latent variables to data. The idea is to evaluate a trained model in terms of the best (lossy) compression rate that can be achieved by encoding a datapoint (e.g. an image) into the latent space, as a function of a permitted distortion between the datapoint and its reconstruction after decoding. The paper describes a method that estimates an upper bound on this rate-distortion curve using annealed importance sampling. The method is applied in evaluating and comparing a few VAE, GAN and AAE architectures on images (MNIST and CIFAR-10).\n\nOverall evaluation:\n\nThis is a very good paper, and I'm happy to recommend it for acceptance.\n\nThe problem considered (evaluating generative models with intractable likelihoods) is an interesting and important one. In general, such models are hard to evaluate and compare with each other. The paper proposes a new method for evaluating them, which can also improve our understanding of these models and potentially diagnose them in practice.\n\nThe method is well-motivated and backed by theoretical results. One clever aspect of the method is the way annealed importance sampling is used to approximate the rate-distortion curve: instead of sampling separately the rate for each distortion level with a different AIS run, a single AIS run is used to approximate the whole curve. This is done by taking the various points on the curve to correspond to intermediate distributions in AIS, which is quite clever.\n\nThe paper is well written, precise, and contains sufficient theoretical background to motivate the method.\n\nThe experiments are done carefully, and the results are interesting. I found particularly interesting the fact that VAEs behave differently to GANs (in terms of their rate-distortion tradeoff) when the dimensionality of the latent space is increased.\n\nSome discussion and critical feedback:\n\nI found the paper too long (10 full pages). I appreciate the detail, precision and depth of explanation, but I think it would be good to reduce the amount of text if possible.\n\nI though that the introduction was too specific to VAEs/GANs and to image modelling, which may give the impression that these are the main models/tasks of interest. I understand that these are the models and tasks that the paper is interested in, but I think it would be better if the introduction acknowledged the existence of other types of generative models (e.g. likelihood-based models such as autoregressive models and normalizing flows) and were less specific to image applications.\n\nProposition 3 is true almost by definition, since R_p is defined to be the minimum of all rate-distortion curves. I wonder if something more informative can be shown here. For example, my understanding is that the reason R^{AIS}_p is not optimal is due to the bias of the importance-sampling estimator in eq. (12). Since this bias asymptotically goes to zero, I suspect that R^{AIS}_p may become equal to R_p for M -> infinity, and perhaps the bound improves monotonically as M increases.\n\nIf my understanding is correct, the reason for the inequality in proposition 4 is that log\\hat{Z} is a biased estimate of logZ (due to Jensen's inequality) despite \\hat{Z} being an unbiased estimate of Z. If that's all there is to it, the proof in the appendix, although precise, is a bit of an overkill. It also seems to me that the log\\hat{Z} bias also approaches zero as M -> infinity, so this inequality maybe also becomes an equality asymptotically.\n\nIn future work, it would be interesting to also evaluate flow-based models (such as Glow). Since these models give exact likelihoods, it may be good to observe how the evaluation based on rate-distortion curves compares with a likelihood-based evaluation.\n\nSection 6 attributes the performance drop of VAEs in the low-rate regime to the \"holes problem\". If that's true, then I would expect the situation to be improved with more flexible prior / posterior models. What prior / posterior models were used in the experiments? If only diagonal Gaussians were used, then it would be interesting to see whether more flexible priors / posteriors such as normalizing flows would change the results.\n\nMinor corrections and suggestions for improvement:\n\n\"For continuous inputs, the metric is often dominated by the fine-grained distribution over pixels rather than the high-level structure\"\nThis statement is specific to images, not continuous inputs in general.\n\nOn the quantitative analysis of deep belief networks, https://dl.acm.org/citation.cfm?id=1390266\nis an older example of AIS used to evaluate generative models that could be cited.\n\nI think it would be better to drop z_0 and T_0 in equations (1) and (2), and have z_1 sampled from p_0 directly, to make the equations consistent with the equations that follow afterwards.\n\n\"using a latent variable z with a fixed prior distribution\"\nIn VAEs the prior can also be learned, it doesn't have to be fixed (this may in fact help alleviate the holes problem).\n\nIt could be mentioned that the objective in Eq. (9) has the same form as a generalized VAE objective, such as the one used by beta-VAE, https://openreview.net/forum?id=Sy2fzU9gl\n\nAt the bottom of page 4 and page 5, R_p(D) uses a different font for R than the rest of the paper.\n\nFig. 1 is too small to read on paper, I had to zoom in using the pdf in order to see it properly.\n\nLast paragraph of section 4 specifically mentions images, even though it doesn't need to (datapoints don't have to be images).\n\nThe last two paragraphs of page 7 contain a few grammatical mistakes:\n- fixing the architecture of neural network --> fixing the architecture of the neural network\n- as the result --> as a result\n- there exist a rate distortion code for any rate distortion pairs  -->  there exists a rate distortion code for any rate distortion pair\n- While in our definition of rate distortion -->Whereas, in out definition of rate distortion\n\nTop of page 8, use \\citet instead of \\citep where appropriate.\n\nCapitalize names and acronyms in references, such as ELBO, GAN, MMD, VAE, Bayes, Monte Carlo, etc.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BygiiXj6KS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the rate-distortion tradeoffs of deep generative models such as variational autoencoders (VAEs) and generative adversarial networks (GANs).\nThe authors propose an annealed importance sampling (AIS) method to compute the rate-distortion curve efficiently. In experiments, the authors compare the rate-distortion curves for VAEs and GANs and discuss the properties of rate-distortion curves.\n\nThe method for computing the rate-distortion curves of deep generative models is interesting and the rate-prior distortion curve is promising as a performance measure. However, the main technical contribution of this work is the estimated AIS rate-prior distortion curve and it is based on a straight-forward application of AIS. \n\nIn fact, Sections 2 and 3 discuss already known result in literature although summarizing them in a paper is nice for readers.\n\nAlthough the findings in the experiments are interesting and insightful, they are still preliminary and further investigations are desirable.\n\nIn Section 5, the authors mention the consistency of their framework with Shannon’s rate distortion theorem. This seems to be a little overstatement because the authors discuss little about the optimization of the prior p(z).\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkeatl8PcB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for evaluating latent-variable generative models in terms of the rate-distortion curve that compares the number of bits needed to encode the representation with how well you can reconstruct an input under some distortion measure. To estimate this curve, the author’s use AIS and show how intermediate distributions in AIS can be used to bound and estimate rate and distortion. They apply their evaluation to GANs, VAEs, and AAEs trained on MNIST and CIFAR-10.\n\nI found this paper well written, with a number of interesting technical contributions, particularly how to leverage AIS to compute R-D curves for an individual model. However, the utility and interpretation of these R-D curves for single models remains confusing to me, and there is insufficient discussion and comparison to other joint diversity/sample quality metrics proposed in the GAN literature. The compute time required for evaluation may also limit the applicability: 4-7 hours for 50 images on MNIST, and 7 days for CIFAR-10. \n \nMajor comments:\n* How should we interpret rate-prior distortion for an individual model vs. rate-distortion where models are optimized for each rate? Past work in learned compression and generative models (Theis et al. 2016, Balle et al. 2016, Alemi et al., 2018) show that models must adjust their decoder (and prior) as a function of rate to be optimal in terms of distortion. For a fixed decoder, optimizing the prior may still be required to achieve low rate. Given that many of the models you compare are trained to do well at one point on the R-D curve, why does it make sense to evaluate them at other points? Additionally, you only evaluate models with deterministic decoders and many of the experimental conclusions are highly specific to this setting but not noted. \n* As you focus on general distortion metrics instead of NLL alone, it'd be interesting to compare curves under different distortion measures, e.g. MS-SSIM for images or L1 vs. L2. Right now there's not much experimental novelty vs. prior work that looked at rate-distortion curves with NLL distortion and Gaussian observation models.\n* It’d be useful to include experiments comparing Rate-Prior distortion curves and Rate-distortion curves where you a) optimize over the prior, b) optimize the decoder, fixing the prior, and c) optimize both the prior and decoder. \n* There’s no comparison of other approaches to generate the rate-prior distortion curve. For example, you could just use an amortized inference network like in VAE w/ a flexible variational family and anneal beta over time.\n* There are several related papers which should be discussed and contrasted, in particular https://arxiv.org/abs/1901.07821 which looks at rate-distortion-perception tradeoffs, and https://arxiv.org/abs/1806.00035 which presents precision-recall curves for diversity/quality metrics applied to implicit models. How do the insights gained from the rate-distortion curves relate to precision/recall and why should one be preferred over the other? https://arxiv.org/abs/1611.02163 also looked at distortion as a metric for GANs (equivalent to beta -> infinity in your framework).\n\nMinor comments:\n* Missing discussion of several related works: that presents a complexity measure for the latent space of GANs: https://arxiv.org/abs/1802.04874\n* “Wasserstein distance remains difficult to approximate…” - see https://openreview.net/forum?id=HkxKH2AcFm that advocates for evaluation with Wasserstein\n* Tightness of bound on simulated data (what BDMC provides) may not correspond to tightness of bound on real data (what you care about in practice). \n* The treatment of VAEs as implicit models only makes sense with location scale family p(x|z), thus the entire framework proposed here doesn’t make sense with e.g. autoregressive p(x|z), as used in PixelVAE and others.\n* Why focus on fixed prior p(z)? An alternative would be to optimize p(z), q(z|x) and fix p(x|z). How would this change the resulting rate-prior distortion curves?\n* “We can compute the R-D curve by sweeping over \\beta rather than by sweeping over D” - this is not the case when the R-D curve has linear segments, see e.g. Rezende & Viola 2018\n* Many of the properties and discussion around rate-prior distortion functions (especially w/NLL distortion) are also in Alemi et al. 2018 as their definition of “rate” is identical to your definition of “rate-prior”. Also many of these properties are specific to continuous latents which isn’t noted.\n* Should clarify that q_k(z|x) correspond to points along R_p(D)\n* The results in Eqn 14-17 showing you can tractably estimate distortion and get an upper bound on rate using the AIS-derived distributions are very cool!\n* “AIS variance is proportional to 1/MK” - this is for variance in the partition function? How does this translate to variance in estimates of rate/distortion?\n* “In the case of probabilistic decoders, …” -> need the caveat this is with NLL distortion\n* Validation on the linear VAE is great!It looks like some of the points for AIS at low distortion are below the analytic rate, but the proofs indicate the estimated rate should be greater than the analytic rate. Is this just noise?\n* Fig 4 and 5: hard to see difference between the dashed lines\n* VAE results would change drastically if you targeted them to different regimes (e.g. beta-VAE or constrained optimization like GECO)\n* Statements like “VAE is trained with the ELBO objective, which encourages good reconstructions” only make sense when the decoder is location-scale. VAEs w/rich autoregressive decoders typically do a horrible job reconstructing.\n* How robust are model differences across random initialization? It’d be great to add error bars to all these plots, especially given that GAN training can stochastically succeed.\n* Eqn 18/Fig6a: depending on the dataset, you could easily notice the difference of 4.6 nats and log-likelihood could still tell these two models apart. It’d be useful to add a line at beta=1 to show that the likelihood would be the same but the R-D curves are different.\n\n========================\nUpdate after rebuttal\n\nThank you to the authors for addressing a number of my concerns, adding additional text and experiments. However, my main concern remains: if rate-distortion in an individual model is a useful method for evaluating generative models, you should compare it empirically with other metrics that have been proposed for this purpose (e.g. precision-recall). Additionally, while the theoretical novelty of getting the full R-D curve from a single AIS run is very cool, I'm skeptical of the practical utility as a metric for generative models due to the computational costs of AIS (4-7 hours for 50 images on MNIST). The simple baseline I suggested of training an amortized inference network with beta annealed over time would not require training a separate encoder for each point in R-D, you could just start beta large, and anneal beta in steps to 0 over time, tracing out an R-D curve. Given the current experiments, it's not obvious if the win of AIS in terms of accurate posterior inference is worth the increased computational cost over a simple VI baseline.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a method to evaluate latent variable based generative models by estimating the compression in the latents (rate) and the distortion in the resulting reconstructions. While reviewers have clearly appreciated the theoretical novelty in using AIS to get an upper bound on the rate, there are concerns on missing empirical comparison with other related metrics (precision-recall) and limited practical applicability of the method due to large computational cost. Authors should consider comparing with PR metric and discuss some directions that can make the method practically as relevant as other related metrics. 
Decision: Reject