Evaluating Lossy Compression Rates of Deep Generative Models

Deep generative models have achieved remarkable progress in recent years . Despite this progress , quantitative evaluation and comparison of generative models remains as one of the important challenges . One of the most popular metrics for evaluating generative models is the log-likelihood . While the direct computation of log-likelihood can be intractable , it has been recently shown that the loglikelihood of some of the most interesting generative models such as variational autoencoders ( VAE ) or generative adversarial networks ( GAN ) can be efficiently estimated using annealed importance sampling ( AIS ) . In this work , we argue that the log-likelihood metric by itself can not represent all the different performance characteristics of generative models , and propose to use rate distortion curves to evaluate and compare deep generative models . We show that we can approximate the entire rate distortion curve using one single run of AIS for roughly the same computational cost as a single log-likelihood estimate . We evaluate lossy compression rates of different deep generative models such as VAEs , GANs ( and its variants ) and adversarial autoencoders ( AAE ) on MNIST and CIFAR10 , and arrive at a number of insights not obtainable from log-likelihoods alone . 1 INTRODUCTION . Generative models of images represent one of the most exciting areas of rapid progress of AI ( Brock et al. , 2019 ; Karras et al. , 2018b ; a ) . However , evaluating the performance of generative models remains a significant challenge . Many of the most successful models , most notably Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) , are implicit generative models for which computation of log-likelihoods is intractable or even undefined . Evaluation typically focuses on metrics such as the Inception score ( Salimans et al. , 2016 ) or the Fréchet Inception Distance ( FID ) score ( Heusel et al. , 2017 ) , which do not have nearly the same degree of theoretical underpinning as likelihood-based metrics . Log-likelihoods are one of the most important measures of generative models . Their utility is evidenced by the fact that likelihoods ( or equivalent metrics such as perplexity or bits-per-dimension ) are reported in nearly all cases where it ’ s convenient to compute them . Unfortunately , computation of log-likelihoods for implicit generative models remains a difficult problem . Furthermore , log-likelihoods have important conceptual limitations . For continuous inputs in the image domain , the metric is often dominated by the fine-grained distribution over pixels rather than the high-level structure . For models with low-dimensional support , one needs to assign an observation model , such as ( rather arbitrary ) isotropic Gaussian noise ( Wu et al. , 2016 ) . Lossless compression metrics for GANs often give absurdly large bits-per-dimension ( e.g . 1014 ) which fails to reflect the true performance of the model ( Grover et al. , 2018 ; Danihelka et al. , 2017 ) . See Theis et al . ( 2015 ) for more discussion of limitations of likelihood-based evaluation . Typically , one is not interested in describing the pixels of an image directly , and it would be sufficient to generate images close to the true data distribution in some metric such as Euclidean distance . For this reason , there has been much interest in Wasserstein distance as a criterion for generative models , since the measure exploits precisely this metric structure ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Salimans et al. , 2018 ) . However , Wasserstein distance remains difficult to approximate , and hence it is not routinely used to evaluate generative models . We aim to achieve the best of both worlds by measuring lossy compression rates of deep generative models . In particular , we aim to estimate the rate distortion function , which measures the number of bits required to match a distribution to within a given distortion . Like Wasserstein distance , it can exploit the metric structure of the observation space , but like log-likelihoods , it connects to the rich literature of probabilistic and information theoretic analysis of generative models . By focusing on different parts of the rate distortion curve , one can achieve different tradeoffs between the description length and the fidelity of reconstruction — thereby fixing the problem whereby lossless compression focuses on the details at the expense of high-level structure . It has the further advantage that the distortion metric need not have a probabilistic interpretation ; hence , one is free to use more perceptually valid distortion metrics such as structural similarity ( SSIM ) ( Wang et al. , 2004 ) or distances between hidden representations of a convolutional network ( Huang et al. , 2018 ) . Algorithmically , computing rate distortion functions raises similar challenges to estimating loglikelihoods . We show that the rate distortion curve can be computed by finding the normalizing constants of a family of unnormalized probability distributions over the noise variables z. Interestingly , when the distortion metric is squared error , these distributions correspond to the posterior distributions of z for Gaussian observation models with different variances ; hence , the rate distortion analysis generalizes the evaluation of log-likelihoods with Gaussian observation models . Annealed Importance Sampling ( AIS ) ( Neal , 2001 ) is currently the most effective general-purpose method for estimating log-likelihoods of implicit generative models , and was used by Wu et al . ( 2016 ) to compare log-likelihoods of a variety of implicit generative models . The algorithm is based on gradually interpolating between a tractable initial distribution and an intractable target distribution . We show that when AIS is used to estimate log-likelihoods under a Gaussian observation model , the sequence of intermediate distributions corresponds to precisely the distributions needed to compute the rate distortion curve . Since AIS maintains a stochastic lower bound on the normalizing constants of these distributions , it automatically produces an upper bound on the entire rate distortion curve . Furthermore , the tightness of the bound can be validated on simulated data using bidirectional Monte Carlo ( BDMC ) ( Grosse et al. , 2015 ; Wu et al. , 2016 ) . Hence , we can approximate the entire rate distortion curve for roughly the same computational cost as a single log-likelihood estimate . We use our rate distortion approximations to study a variety of variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) , GANs and adversarial autoencoders ( AAE ) ( Makhzani et al. , 2015 ) , and arrive at a number of insights not obtainable from log-likelihoods alone . For instance , we observe that VAEs and GANs have different rate distortion tradeoffs : While VAEs with larger code size can generally achieve better lossless compression rates , their performances drop at lossy compression in the low-rate regime . Conversely , expanding the capacity of GANs appears to bring substantial reductions in distortion at the high-rate regime without any corresponding deterioration in quality in the low-rate regime . We find that increasing the capacity of GANs by increasing the code size ( width ) has a qualitatively different effect on the rate distortion tradeoffs than increasing the depth . We also find that that different GAN variants with the same code size achieve nearly identical RD curves , and that the code size dominates the performance differences between GANs . 2 BACKGROUND . 2.1 ANNEALED IMPORTANCE SAMPLING . Annealed importance sampling ( AIS ) ( Neal , 2001 ) is a Monte Carlo algorithm based on constructing a sequence of n intermediate distributions pk ( z ) = p̃k ( z ) Zk , where k ∈ { 0 , . . . , n } , between a tractable initial distribution p0 ( z ) and the intractable target distribution pn ( z ) . At the the k-th state ( 0 ≤ k ≤ n ) , the forward distribution qf and the un-normalized backward distribution q̃b are qf ( z0 , . . . , zk ) = p0 ( z0 ) T0 ( z1|z0 ) . . . Tk−1 ( zk|zk−1 ) , ( 1 ) q̃b ( z0 , . . . , zk ) = p̃k ( zk ) T̃k−1 ( zk−1|zk ) . . . T̃0 ( z0|z1 ) , ( 2 ) where Tk is an MCMC kernel that leaves pk ( z ) invariant ; and T̃k is its reverse kernel . We run M independent AIS chains , numbered i = 1 , . . . , M . Let zik be the k-th state of the i-th chain . The importance weights and normalized importance weights are wik = q̃b ( z i 1 , . . . , z i k ) qf ( zi1 , . . . , z i k ) = p̃1 ( z i 1 ) p0 ( zi1 ) p̃2 ( z i 2 ) p̃1 ( zi2 ) . . . p̃k ( z i k ) p̃k−1 ( zik ) , w̃ik = wik∑M i=1 w i k . ( 3 ) At the k-th step , the unbiased partition function estimate of pk ( z ) is Ẑk = 1M ∑M i=1 w i k. At the k-th step , we define the AIS distribution qAISk ( z ) as the distribution obtained by first sampling z1k , . . . , z M k from the M parallel chains using the forward distribution qf ( z i 1 , . . . , z i M ) , and then re-sampling these samples based on w̃ik . More formally , the AIS distribution is defined as follows : qAISk ( z ) = E∏M i=1 qf ( z i 1 , ... , z i k ) [ M∑ i=1 w̃ikδ ( z− zik ) ] . ( 4 ) Bidirectional Monte Carlo . We know that the log partition function estimate log Ẑ is a stochastic lower bound on logZ ( Jensen ’ s inequality ) . As the result , using the forward AIS distribution as the proposal distribution results in a lower bound on the data log-likelihood . By running AIS in reverse , however , we obtain an upper bound on logZ . However , in order to run the AIS in reverse , we need exact samples from the true posterior , which is only possible on the simulated data . The combination of the AIS lower bound and upper bound on the log partition function is called bidirectional Monte Carlo ( BDMC ) , and the gap between these bounds is called the BDMC gap ( Grosse et al. , 2015 ) . We note that AIS combined with BDMC has been used to estimate log-likelihoods for implicit generative models ( Wu et al. , 2016 ) . In this work , we validate our AIS experiments by using the BDMC gap to measure the accuracy of our partition function estimators . 2.2 RATE DISTORTION THEORY . Let x be a random variable that comes from the data distribution pd ( x ) . Shannon ’ s fundamental compression theorem states that we can compress this random variable losslessly at the rate of H ( x ) . But if we allow lossy compression , we can compress x at the rate of R , where R ≤ H ( x ) , using the code z , and have a lossy reconstruction x̂ = f ( z ) with the distortion of D , given a distortion measure d ( x , x̂ ) = d ( x , f ( z ) ) . The rate distortion theory quantifies the trade-off between the lossy compression rate R and the distortion D. The rate distortion function R ( D ) is defined as the minimum number of bits per sample required to achieve lossy compression of the data such that the average distortion measured by the distortion function is less than D. Shannon ’ s rate distortion theorem states thatR ( D ) equals the minimum of the following optimization problem : min q ( z|x ) I ( z ; x ) s.t . Eq ( x , z ) [ d ( x , f ( z ) ) ] ≤ D. ( 5 ) where the optimization is over the channel conditional distribution q ( z|x ) . Suppose the datadistribution is pd ( x ) . The channel conditional q ( z|x ) induces the joint distribution q ( z , x ) = pd ( x ) q ( z|x ) , which defines the mutual information I ( z ; x ) . q ( z ) is the marginal distribution over z of the joint distribution q ( z , x ) , and is called the output marginal distribution . We can rewrite the optimization of Eq . 5 using the method of Lagrange multipliers as follows : min q ( z|x ) I ( z ; x ) + βEq ( z , x ) [ d ( x , f ( z ) ) ] . ( 6 )

Summary: The paper proposes a new way to evaluate generative models that don't have tractable likelihoods, such as VAEs or GANs. Such generative models are composed of a prior over latent variables and a decoder that maps latent variables to data. The idea is to evaluate a trained model in terms of the best (lossy) compression rate that can be achieved by encoding a datapoint (e.g. an image) into the latent space, as a function of a permitted distortion between the datapoint and its reconstruction after decoding. The paper describes a method that estimates an upper bound on this rate-distortion curve using annealed importance sampling. The method is applied in evaluating and comparing a few VAE, GAN and AAE architectures on images (MNIST and CIFAR-10).