RATE-DISTORTION OPTIMIZATION GUIDED AUTOENCODER FOR GENERATIVE APPROACH

1 INTRODUCTION . Capturing the inherent features of a dataset from high-dimensional and complex data is an essential issue in machine learning . Generative model approach learns the probability distribution of data , aiming at data generation by probabilistic sampling , unsupervised/weakly supervised learning , and acquiring meta-prior ( general assumptions about how data can be summarized naturally , such as disentangle , clustering , and hierarchical structure ( Bengio et al. , 2013 ; Tschannen et al. , 2019 ) ) . It is generally difficult to directly estimate a probability density function ( PDF ) Px ( x ) of real data x . Accordingly , one promising approach is to map to the latent space z with reduced dimension and capture PDF Pz ( z ) . In recent years , deep autoencoder based methods have made it possible to compress dimensions and derive latent variables . While there is remarkable progress in these areas ( van den Oord et al. , 2017 ; Kingma et al. , 2014 ; Jiang et al. , 2016 ) , the relation between x and z in the current deep generative models is still not clear . Variational autoencoder ( VAE ) ( P.Kingma & Welling , 2014 ) is one of the most successful generative models for capturing latent representation . In VAE , lower bound of log-likelihood of Px ( x ) is introduced as ELBO . Then latent variable is obtained by maximizing ELBO . In order to maximize ELBO , various methods ( Alemi et al. , 2018 ; Zhao et al. , 2019 ; Brekelmans et al. , 2019 ) have been proposed . However , many previous works did not care about the value of Jacobian between two spaces , despite the fact that the ratio between Pz ( z ) and Px ( x ) is equal to the Jacobian . Even in models that provide more flexible estimation ( Johnson et al. , 2016 ; Liao et al. , 2018 ; Zong et al. , 2018 ) , this point is overlooked . Here , when we turn our sight toward acquiring meta-prior , it is straightforward to evaluate the quantitative influence of each latent variable on the given metrics between data x1 and x2 . To do so , the scale of the latent variable should be appropriately controlled so that the changes in latent variables is proportional to the changes of given metrics in data space . In addition , this scaling should be adjusted according to the definition of the metrics . For instance , with respect to image quality metrics , different meta-prior should be derived from mean square error ( MSE ) and the structural similarity ( SSIM ) . Considering the mechanism of principal component analysis ( PCA ) would be one of the directions to solve it . In PCA , PCA components are derived by optimal orthonormal transformation , then the importance of each component can be identified quantitatively by their variance . It should be noted that orthogonality is not enough for quantitative analysis , but both orthogonality and normalized scaling , namely , orthonormality is required . This fact implies that , if Jacobi matrix of autoencoder has orthonormality , the characteristics of acquired latent space can be evaluated quantitatively . To deal with this , we propose RaDOGAGA ( Rate-Distortion Optimization Guided Autoencoder for Generative Approach ) , based on the rate-distortion optimization ( RDO ) which has been widely used in image compression with orthonormal transform coding ( Sullivan & Wiegand ( 1998 ) ) . In this paper , we show the effect of RaDOGAGA in the following steps . ( 1 ) We prove that RaDOGAGA has the following property theoretically and experimentally . • Jacobi matrix between real space and latent space leads to be constant-scaled orthonormal . So the response of the minute change of z to the real space data x is constant at any z . • Because of constant Jacobian ( or pseudo-Jacobian ) , Px ( x ) and Pz ( z ) are almost proportional . Therefore , Px ( x ) can be estimated , by directly maximizing log-likelihood of parametric PDF Pzψ ( z ) in reduced-dimensional space , without considering ELBO . • When univariate independent distribution is used to estimate Pz ( z ) parametrically , it behaves as ” continuous PCA ” where energy is concentrated on several principal components . ( 2 ) Thanks to this property , RaDOGAGA achieve the state-of-the-art performance in anomaly detection task with four public datasets , where probability density estimation is important . ( 3 ) We show that our approach can directly evaluate how the z impact on the given metrics in real space . This feature is promising to further interpretation of latent variables . 2 RELATED WORK . Flow based model : Flow based generative models generates astonishing quality of image ( Kingma & Dhariwal , 2018 ; Dinh et al. , 2014 ) . Flow mechanism explicitly takes Jacobian of x and z into account . The transformation function z = f ( x ) is learned , calculating and preserving Jacobian of x and z . Unlike ordinary autoencoder , which reverse z to x with function g ( · ) different from f ( · ) , inverse function transforms z as x = f−1 ( z ) . Since the model preserves Jacobian , Pz ( x ) can be estimated by maximizing log likelihood of Pz ( z ) without considering ELBO . Although , in this approach , f ( · ) need to be bijection . Because of this limitation , it is difficult to fully utilize the flexibility of neural networks . Interpretation of latent variables : While it is expected to acquire meta-prior by deep autoencoder , interpreting latent variables is still challenging . In recent years , research aiming to acquire disentangled latent variables , which encourages them to be independent , is flourishing ( Lopez et al. , 2018 ; Chen et al. , 2018 ; Kim & Mnih , 2018 ; Chen et al. , 2016 ) . With these methods , qualitative effects for disentanglement can be seen . For example , when a certain latent variable is displaced , image changes corresponding to specific attributes ( size , color , etc. ) . Some works also propose quantitative metrics for meta-prior . In beta-VAE ( Higgins et al. , 2017 ) , the metric evaluates the independence of latent variables by solving the classification task . Actually , the Jacobi matrix of VAE is orthogonal ( Rolı́nek et al. , 2019 ) , which enables to make latent variables disentangled implicitly . However , orthonormality is not supported and it is difficult to define the metric which evaluates the effect of latent variables to the metrics between data directly . Deep image compression with rate-distortion optimization : Rate-distortion ( RD ) theory ( Berger , 1971 ) is a part of Shannon ’ s information theory for lossy compression which formulates the optimal condition between information rate and distortion . Then RD theory for Gaussian source with memory has been further extended to transform coding ( Goyal , 2001 ) for image and audio lossy compression where orthonormal transforms such as Karhunen-Loève transform ( KLT ) ( Rao & Yip ( 2000 ) ) and discrete cosine transform ( DCT ) are used for decorrelation . Furthermore , rate-distortion optimization ( RDO ) has been widely used in image compression ( Sullivan & Wiegand , 1998 ) . In RDO , a cost L = R+λ ·D is minimized at given Lagrange parameter λ to realize the best trade-off between rate R and distortion D such that L = R + λ · D becomes a tangent line of true RateDistortion curve . Recently , deep learning based image compression ( Ballé et al. , 2018 ; Zhou et al. , 2019 ) has been proposed . In these works , instead of orthonormal transform with L2 norm metrics in the conventional lossy compression method , a deep autoencoder is trained with flexible metrics such as SSIM for RDO . To understand the success of these works , we prove theoretically that RDO guides a Jacobi matrix of deep autoencoder to orthonormal in the space depending on the metrics . In the next section , we explain the idea of RDO guided autoencoder and its relationship with VAE . 3 OVERVIEW OF RATE-DISTORTION OPTIMIZATION GUIDED APPROACH . 3.1 DERIVATION FROM RATE-DISTORTION OPTIMAZATION IN TRANSFORM CODING . Figure 1 shows the overview of our idea inspired by Rate-Distortion optimization of transform coding . In the transform coding , the optimal method to encode data with Gaussian distribution is as follows ( Goyal , 2001 ) . At first , the data are transformed deterministically to decorrelated data using the orthonormal transform such as KLT and DCT . Then these decorrelated data are quantized stochastically with uniform quantizer for all channels such that quantization noise for each channel is equal . At last optimal entropy encoding is applied to quantized data where the rate can be calculated by the logarithm of symbol ’ s estimated probability . From this fact , we have an intuition that the Jacobi matrix of autoencoder becomes orthonormal if the data were compressed based on RateDistortion optimization with uniform quantized noise and parametric distribution of latent variables . Inspired by this , we propose autoencoder which scales latent variables according to the definition of metrics of data , without limitation of the transformation function as in Flow models . Thanks to this feature , our scheme can estimate Pz ( z ) quantitatively , which is suitable for clustering and anomaly detection . Furthermore , in the case factorized distribution is used for Pz ( z ) , our model behaves as continuous PCA . This property is considered to promote the interpretation of latent variables . 3.2 RELATIONSHIP WITH VAE . There is a number of VAE studies that taking rate-distortion trade-off into account . In VAEs , instead of maximizing log-likelihood of P ( x ) directly , maximizing ELBO . In beta-VAE ( Higgins et al. , 2017 ) , Objective function LV AE is described as LV AE = Lrec − β · Lkl . Lkl is KL divergence between encoder output and prior distribution , usually Gaussian distribution . By changing β , the rate-distortion trade-off at desirable rate can be realized as discussed in ( Alemi et al. , 2018 ) . When the relation between beta-VAE and RDO in image compression is considered , both are analogous to each other . That is , β−1 is corresponding to λ , −Lkl is corresponding to a rate R , and Lrec is corresponding to a distortion D. However , the probability distribution models of latent variables are quite different . VAE uses a fixed prior distribution . This causes a non-linear scaling relationship between real data and latent variables . Figure 2 shows the conditions to achieve Rate-Distortion optimization in both VAE and RDO guided autoencoder . In VAE , precise control of noise distribution is required for each data and channel as suggested in Brekelmans et al . ( 2019 ) in order to compensate for the nonlinearities between the data space and the latent space . Jacobi matrix becomes orthogonal as proved in Rolı́nek et al . ( 2019 ) . However , Jacobian is not constant because of non-linear scaling . In RDO guided autoencoder , uniform noises are added to all channels . Instead , parametric probability distribution should be used in order to make the scaling linear between the data space and the latent space . As a result , the Jacobi matrix becomes orthonormal because both orthogonality and scaling normalization are simultaneously achieved . As discussed above , the precise noise control in VAE and distribution parameter optimization in RDO guided autoencoder are essentially the same . Accordingly , complexities in both methods are estimated to be at the same degree .

Summary: of the paper: The authors propose a latent variable model RaDOGAGA, a generative autoencoding model. The model is trained via a tradeoff between distortion (the reconstruction error) and the rate (the capacity of the latent space, measured by entropy). The paper provides an analysis of theoretical properties of their approach, and presents supporting experimental results.