Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding
Authors: Akira Nakagawa
Keizo Kato
Venue:  (2021)
Keywords: unsupervised representation learning
deep image compression

Abstract: Variational autoencoder (VAE)  estimates the posterior parameters (mean and variance) of latent variables corresponding to each input data. While it is used for many tasks, the transparency of the model is still an underlying issue. This paper provides a quantitative understanding of VAE property by interpreting VAE as a non-linearly scaled isometric embedding. According to the Rate-distortion theory, the optimal transform coding is achieved by using a PCA-like orthonormal transform where the transform space is isometric to the input. From this analogy, we show theoretically and experimentally that VAE can be mapped to an implicit isometric embedding with a scale factor derived from the posterior parameter. As a result, we can estimate the data probabilities in the input space from the prior, loss metrics, and corresponding posterior parameters. In addition, the quantitative importance of each latent variable can be evaluated like the eigenvalue of PCA.


Under review as a conference paper at ICLR 2021
QUANTITATIVE UNDERSTANDING OF VAE AS A NON- LINEARLY SCALED ISOMETRIC EMBEDDING
Anonymous authors Paper under double-blind review
ABSTRACT
Variational autoencoder (VAE) estimates the posterior parameters (mean and vari- ance) of latent variables corresponding to each input data. While it is used for many tasks, the transparency of the model is still underlying issue. This paper provides a quantitative understanding of VAE property by interpreting VAE as a non-linearly scaled isometric embedding. According to the Rate-distortion the- ory, the optimal transform coding is achieved by using a PCA-like orthonormal transform where the transform space isometric to the input. From this analogy, we show theoretically and experimentally that VAE can be mapped to an implicit isometric embedding with a scale factor derived from the posterior parameter. As result, we can estimate the data probabilities in the input space from the prior, loss metrics, and corresponding posterior parameters. In addition, the quantitative importance of each latent variable can be evaluated like the eigenvalue of PCA.
INTRODUCTION
Variational autoencoder (VAF) (Kingma& & Welling, 2014) is one of the most successful generative models, estimating posterior parameters of latent variables for each input data. In VAE, the latent representation is obtained by maximizing an evidence lower bound (ELBO). A number of studies (Higgins et al., 2017: Kim & Mnih, 2018; Lopez al., 2018: Chen et al., 2018: Locatello et al., 2019; Alemi et al., 2018; Rolínek et al., 2019) have tried to reveal the property of latent variables. However, quantitative behavior of VAE is still not well clarified. For example, there has not been theoretical formulation of the reconstruction loss and KI. divergence in ELBO after optimization. More specifically, although the conditional distribution pa (=(()) the reconstruction loss of ELBO is predetermined such as the Gaussian or Bernoulli distributions, it has not been discussed well whether the true conditional distribution after optimization matches the predetermined distribution.
Rate-distortion (RD) theory (Berger, 1971), which is an important part of Shannon information theory and successfully applied to image compression, quantitatively formulates the RD trade-off optimum in lossy compression. To realize quantitative data analysis, Rate-distortion (RD) theory based autoencoder, RaDOGAGA (Kato et al., 2020), has been proposed with isometric embedding (Han & Hong, 2006) where the distance between arbitrary two points of input space in given metrics is always the same as L2 distance in the isometric embedding space. In this paper, by mapping VAE latent space to an implicit isometric space like RaDOGAGA on brriabl-by-variabbe basis and analysing VAE quantitatively as a well-examined lossy compression, we thoroughly clarify the quantitative properties of VAE theoretically and experimentally as follows.
1) Implicit isometric embedding is derived in the loss metric defined space such that the entropy of data representation becomes minimum. scaling factor between the VAE latent space and implicit isometric space is formulated by the posterior for each input. In the case of 3-VAE, the posterior variance of each dimensional component in the implicit isometric embedding space is a constant 8/2, which is analogous to the rate-distortion optimal of transform coding in RD theory. As a result, the reconstruction loss and KL divergence in ELBO can be quantitatively formulated. 2) From these properties, VAE can provide practical quantitative analysis of input data. First, the data probabilities in the input space can be estimated from the prior, loss metric, and poste- rior parameters. In addition, the quantitative importance of each latent variable, analogous to the eigenvalue of PCA, can be evaluated from the posterior variance of VAE.
This work will lead the information theoretic generative models in the right direction.

---

Under review as a conference paper at ICLR 2021
2 RELATED WORKS
2.1 VARIATIONAL AUTOENCODER AND THEORETICAL ANALYSIS
In VAE, ELBO is maximized instead of maximizing the log-likelihood directly. Let E IRm be point in a dataset. The original VAE model consists of latent variable with fixed prior 2 ~ p(z) = (2; 0, In) E IR" parametric encoder Enc, æ + 2, and parametric decoder Decy 2 + a. In the encoder, (z(x)=) = (()), (ax) is provided by estimating parameters (()) and (z)) Let Le be a local cost data æ. Then, ELBO is described by
ELBO Em-p(a) [Ezrg Po =]] (=<](((\\
In Paoop(t)) ] the first term E (x xe) is called the reconstruction loss. The second term (- is Bellbackk-Libble (KL) divergence. Let (j(a)). (j()), and DKIj(z) be j-th dimensional values of ((w)), (w)) and KL divergence. Then DKr. (-) is derived as:
a
DRL(-) Σ DKIs(s); where DKIJ(0) = (1)(m) 2 + dz(m) --lo-llllooooo )(=) 2 j-l
Dxx, .)) denotes a metric such as sum square error (SSE) and binary cross-entropy (BCE) as log- likelihoods of Gaussian and Bernoulli distributions, respectively. In training VAE, the next objective used instead of Eq. 1, where
parameter control the trade-off (Higgins al., 2017).
Le = E=>gg<<>le) [D(x, +)] + (DkI.
(3)
However, it has not been fully discussed whether the true conditional distribution matches the pre- determined distribution, how the value of KL divergence is derived after training.
There have been several studies to analyse VAE theoretically. Alemi et al. (2018) introduced the RD trade-off based on the ixformation-theoreee framework analyse 3-VAE. However, they did not clarify the quantitative property after optimization. Dai et al. (2018) showed that VAE restricted as linear transform can be considered the analysis on each latent variable basis because of the linearity assumption. Rolínek et al. (2019) showed that the Jacobian matrix of VAE each latent variable is orthogonal, which makes latent variables disentangled implicitly. However, they do not uncover the orthonormality and quantitative properties because they simplify KL. divergence as a constant, Dai & Wipf (2019) also showed that the expected rate of VAE for the r-dimensional manifold is close to - (rr/2) log + O(1) at - 0 when pa (á)x) = N (w; æ, 7Im) holds. The remaining challenge is to clearly figure out what latent space is obtained at given dataset, loss metric, and / in the model.
robust PCA. However, their model has limitation for
2.2 RATE-DISTORTION THEORY, TRANSFORM CODING, AND ISOMETRIC EMBEDDING
RD theory (Berger, 1971) formulated the optimal transform coding (Goyal, 2001) for the Gaussian source with square error metric follows. Let E Rm be point in a dataset. First, the data transformed deterministically with the orthonormal transform (orthogonal and unit norm) such as Karhunen-Loëve transform (KLT) (Rao Yip. 2000). Let E 1T be a point transformed from æ. Then, is entropy-coded by allowing equivalent stochastic distortion (or posterior with constant variance) in each dimension.
lower bound of a rate R distortion is denoted by R(D). The derivation of R(D) is as follows. Let zj be the j-th dimensional component of z and σx) be the variance of 2j in dataset. It is noted that Øzj² 2 is the equivalent to cigenvalues of PCA for the dataset. Let d be a distortion equally allowed in each dimensional channel. At the optimal condition, the distortion Dopt and rate Ropt the curve R( R(D) is calculated as a function of d:
Ropt Σ max(log(a,,)/d))
Dopt min(d, 22
(4)
The simplest way to allow equivalent distortion is to use uniform quantization (Goyal, 2001). Let T' be quantization step, and round(.) be a round function. Quantized value Éÿ is derived as kT', where lo = round(2,/T)) (T). Then, d is approximated by T explained in Appendix H.1.
To practically achieve the best RD trade-off in image compression, rate-distortion optimization (RDO) has also been widely used (Sullivan & Wiegand, 1998). In RDO, the best trade-off is
2

---

Under review as a conference paper at ICLR 2021
Dataset
Input space Spp (c R")
Metrics (x, &) for points x, & Siger
Implicit isometric embedding
Implicit isometric space
Large
Prior Variance
Small
Prior
Posterior
variance p/2
Spo R"), E Sn
P(r)
P(s)
PUa
Pla
Nonlinear scaling: dz, dy, 4/12
Posterior
VAE anisometric ohthogll space SwnE (c R*), Z SWL
P(D)
P(aa)
Fixed Prior P(=.)
Iymore (m-n) variables Large
Small
Posterior Variance
Figure 1: Mapping of VAE to implicit isometric embedding.
achieved by finding a encoding parameter that minimizes cost D AR at given Lagrange parameter X. Recently, deep image compression (Ballé et al., 201 8) has been proposed. In these works, instead of an orthonormal transform with sum square error (SSE) metric in the conventional lossy compression, deep autoencoder is trained with flexible metrics, such as structural similarity (SSIM) (Wang et al., 2001) for RDO. Recently, an isometric autoencoder, RaDOGAGA (Kato et al., 2020) was proposed based Ballé al. (2018). They proved that the latent space be isometric to the input space if the model is trained by RDO using a parametric prior and posterior with constant variance. By contrast, VAE uses fixed prior with variable posterior. In section 3, explain that VAE can be quantitatively understood as the rate-distortion optimum as in Eq. 4 by mapping VAE latent space implicit isometric embedding on variable-to-variable basis in Fig. 1.
UNDERSTANDING OF VAE AS SCALED ISOMETRIC EMBEDDING
This section shows the quantitative understanding of VAE. First, we present the hypothesis of map- ping VAE latent space to implicit isometric embedding space. Second, reformulate the ob- jective of V-V for easy analysis. Third, we prove the hypothesis from the minimum condition of the objective. Then, we show that ELBO can be interpreted as optimized RDO cost of trans- form coding where the quantitative properties are well clarified, as well as discuss and correct some prior theoretical studies. Lastly, we explain the quantitative properties of VAE validate the theory including approximations and provide a practical data analysis.
3.1 HYPOTHESIS OF MAPPING VAE TO THE IMPLICIT ORTHONORMAL TRANSFORM
Figure shows the mapping of VAE to the implicit isometric embedding. Assume the data manifold
smooth and differentiable. Let Smput (C Rmm) be an input space of the dataset. D(x, .) denotes metric for points x, 1 E Simput. Using the second order Taylor expansion, D(x: + 8x) can be approximated by '8x Gæōæ where G, and dæ are an æ dependent positive definite Hermitian metric tensor and an arbitrary infinitesimal displacement in Sinput, respectively. The derivations of Ge for SSE, BCE, and SSIM are shown Appendix H.2. Next, an implicit isometric embedding space (c IR such that the entropy of data representation is minimum in the inner product space Gz. Let y and Uj be a point in Siso and its j-th component, respectively. Because of the isometricity, p(æ) = ply) will hold. We will also show the posterior variance of each dimensional component yj is a constant 3/2. In addition, the variance of yj will show the importance like PCA when the data manifold has a disentangled feature by nature in the metric space of G2 and the prior covariance is diagonal.
is introduced like the isometric latent space in RaDOGAGA (Kato et al., 2020),
Then, Stso is nonlinearly scaled to the VAE's anisometric orthogonal space SvAE(C R*) on basis. Let z be point in SVAE, and 2j denotes the j-th component of z. Let P(y,) and p(2;) be the probability distribution of the j-th variable Sis0 and SVAE- Each variable Yj is nonlincarly scaled to 2j. such that dz,/dy; = p(y,)/ //(22)) ) to fit the cumulative distribution. dzy/dy, is (/)) / 3/2, the ratio of posterior's standard deviations for 2j and 3/, such that KL divergences in both spaces are equivalent. In addition, dimensional components whose KL diver- gences are be discarded because such dimensions have no information.
3

---

Under review as a conference paper at ICLR 2021
3.2 REFORMULATION OF OBJECTIVE THE FORM USING dz 0z/0z, AND dz 0z/Oz
We reformulate the objective Le to the form using dx/00 02z and dx 0z/ Here, the dimensions of æ and z, i.e., and nt set as the same. The condition reduce shown in section 3.3.
Reformulation of (ar, ar, .:) loss: In accordance with Kato et al. (2020), the loss D(x. .) can be decomposed into (æx #) + D)x where 0 denotes ( The first term [ @) is distortion between the decoded values of ((x)). with and without noise (w)). We call this term as a coding loss. This term is expanded as follows. 5ø denotes approximated by "ō Gz Let Ø2 be Ox/ / 0zz at 2j = M)(a)) and ěz₃ (O, ()())) ) be added noise in 2j. Then, dæ is approximated by dæ ===1 dzy æs,, Because őzj and òzz for + k are uncorrelated, the average of D(t, 2)) over qo 2 (x) can be finally reformulated by
σ. Then, D( .) term can be
E (()g)((( [1 x)] = F (x(1)) da G 530 = Σ ))(0) 2 ''ZZ, G.
(5)
The second term (ar, db is loss between the input data and (ecg(( We call this term transform loss. We presume VAE is analogous to the Wiener filter (Wiener, 1964; Jin et al., 2003) where the coding loss is regarded as an added noise. From the Wiener filter theory, the ratio between the transform loss and coding loss is close to the ratio between the coding loss and the variance of the input data. The coding loss, approximately n.8/2 in Eq. 14, should smaller than the variance of the input data to capture meaningful information. Thus the transform loss, usually small, is not considered the following discussion. Appendix explains the detail in a simple 1-dimensional VAE. We show the exhaustive and quantitative evaluation of coding loss and transform loss in the toy dataset in appendix E.2 to validate this approximation.
Reformulation of KL divergence: When j(()) < م << - log )ه(ر)² is observed. For example, when )(x)²² 2 < 0.1, we have (0)((m) 2/ "/log (((m)²) < 0.05. In such dimensions, DKIj(z) can be approximated Eq. 6 by ignoring the (x)) 2 term and setting p(()(=)) (22;;0,1)
DKIS(=) = 2 (1)(a)) log ))(m) 1) log (o)(m) p(U)(=)) log 2nre
(6)
2
Eq. 6 can be considered as rate of entropy coding for a symbol with mean My(#) allowing quantiza- tion noise 0z(x)² 2 as shown in Appendix H.3. Thus, in the dimension with meaningful information, ))w=)* 2 is much smaller than the prior variance 1, and the approximation in Eq.6 is reasonable. Let p(H(m)) be IT (())(=)) p(() p(x) [det(Ox /0x) holds where det /dz) is Jacobian determinant at ((z)). Let Dia.. be constant / log 2Tc. Then, DKL (-) is reformulated by
D.I( log (p(p())) II ( CDRL = log log (((x) det
Oa: Oz
II
(0)))) - CD (7)
Final objective form: From Eqs. and 7, the objective I' to minimise is derived as:
Fx: 0z
n II 1=1
L'= = Σ j(()) Dx, Gr - 3 log (((x)
det
-DDI
(8)
1=1
3.3 PROOF OF THE HYPOTTHESIS
Mapping VAE to implicit isometric embedding: The minimum condition of L'g at x is examined.
Let ž., be the j-th column vector of cofactor matrix for Jacobian matrix dx/ /2z. Note that d log |det. (0x/ / = ..,/ /det (0z))/0))))))) holds as is also used in Kato et al. (2020). Using this equation, the derivative of I'2 by æz, is described by
dL'3
B
)())())) GG det (0x/Oz)
dæ
Note that 'xz, 3. det dānt 85k holds by the cofactor's property. Here, denotes the dot product, and 65z denotes the Kronecker delta. By setting Eq. 9 to zero and multiplying 'æz, from the left, the condition to minimize 1/g is derived by the next orthogonal form of æzyi
(()) 213) 'x,, G.D:. jjk-
(10)
4

---

Under review as a conference paper at ICLR 2021
Here, the diagonal posterior covariance is the key for orthogonality. Next, implicit latent variable and its j-th dimensional component yj are introduced. Set yj to zero at 2j = 0. The derivative between yj and 2j at (5())) is defined by
dy
-1
(1I)
dzj m
By denotes dz /0g By applying Zzz, d By; to Eq. 10, Fy, shows the isometric property (Han & Hong, 2006; Kato et al., 2020) in the inner product space with metric tensor G. as follows:
My Ga ddgk-
(12)
Minimum entropy of implicit isometric representation: Let Lqqin be minimum of I' æ. Dmin and Rmine denote a coding loss and KL divergence in m respectively. By applying Eqs. 10-11 and p(2z) = (dy;/da) p(y;) to Eqs. and 7, the following equations are derived:
n33
Ruin æ - log n nlog(3me) (13)
I' Diuin a + Raiin my where Dunn x 2
2
Here, Dmin æ is derived as Σj= (8/2) "z GT = 33/2, implying each dimensional poste- rior variance of the implicit isometric variable is constant 3/2. In addition, exp(-I a // p(y) exp(Const.) x ply) = p(x)) will hold in the inner product space of G2 from the isometricity.
By averaging I I' as over x ~ p(æ)) and approximating this average by the integration over y p(y). the global minimum L'g is derived as:
1/2 = DG + BRG, where DG n,
(-/ (-/pio)) P(y) logr(y)dy) p(y)
n (lgg(B
Rg
min
(14)
The term p(y) log ply)d in Rg is the entropy of y. Thus, the optimal implicit isometric space is derived such that the entropy of data representation is minimum in the inner product space of Ga-
When the data manifold has disentangled property in the given metric, each y, will capture disentangled feature with minimum entropy, as shown in Kato et al. (2020). This is analogous to PCA for Gaussian data, which gives the disentangled representation with minimum entropy in SSE. Considering the similarity to the PCA eigenvalues, the variance of yj will indicate the importance of each dimension. In the dimensions where the variance of yj is less than 3/2, 0)(=)) 1, M5(m)) 0, because this needs not to be balanced with D This similar to the case in the RD theory in Eq. 4 where az,² is less than d, meaning no information. As result, Eqs. 10-14 will not hold here. Thus, latent variables with variances from the largest to the n-th with DKL((w) > 0 are sufficient for the representation and the dimensions with DKLλ(s) O can be ignored, allowing the reduction of the dimension for z.
and DKI)(=) = 0 will hold. In addition, ()())* 2; Gz..s, will be close
Some approximations may be slightly violated, however, our analysis still helps to understand VAF.
3.4 DISCUSSION AND RELATIONSHIP WITH PRIOR THEORETICAL STUDIES
the rate-distortion optimum (Eq. 4) in RD theory when the uniform distortion d in Eq. 4 is set to 3/2 in the metric defined space. H (X) f p(æ:) p(z) dæ denotes differential entropy for a set x X; x: p(z). For the 1-dimensional Gaussian data ~ Mxx,, 0, 0² H(X)= 2 log( (2gTeeo)) holds. Thus, Ropt in Eq. is derived as a difference of the differential entropy between transformed data z ~ II, . [ O, 0,,),) and uniform distortion D ~ N(D; 0, d.m)). R& is also derived as a difference of the differential entropy between transformed data y ~ p(y) and uniform distortion D ((D: 0, (3/2) Furthermore, DG in Eq. 14 can be interpreted as Dopt in Eq. 4 by setting d = 3/22. As a result, the VAE optimal corresponds to the rate-distortion optimal of transform coding in RD theory, and 3/2 is regarded as a variance of the constant distortion equally added to each dimensional component. Because of the isometricity, the power of distortion (i.e., posterior variance) in the implicit isometric space is the same that in the metric defined input space. Thus the conditional distribution after optimization in the metric defined space is derived as Po [x)2) = Pa (z) () ~ A' N(x 2, (B/2)I). This is consistent with the fact that the quality of the reconstructed data becomes worse in larger 8.
First, show 3-VAE optimum in Eq. 14 can be interpreted
5

---

Under review as a conference paper at ICLR 2021
Next, estimate the reconstruction loss ( [lo p=(x= and KL divergence DKL(-) in B- VAE and also correct the analysis in Alemi et al. (2018). Let H _p((r) p(z) be differential entropy of input data. When 3 = 1, Alemi et al. (2018) suggest "the ELBO objective alone (and the marginal likelihood) cannot distinguish between models that make no use of the latent variable (autodecoders) versus models that make large use of the latent variable and learn useful representa- tions for reconstruction (autoencoders)), " because the reconstruction loss and KL divergence can be arbitrary value on the line - -E (= æ) [log (<) [)] + (-x,.-) = H. Correctly, the reconstruction loss and KL divergence after optimization are deterministically estimated at any / (including / 1) as:
(x]() [lo po =)] - (n1/2) log(Sre), (+I(-) = log - (n/2) log(BBTe.
(15)
The proof is explained in Appendix A.I. Thus ELBO can be estimated as:
ELBO (E=gg)_= [log po (a]l=) D+I.(-) Ep(w) log p(y)] = pp [ogg(x)]] (16)
As result, when the objective of 3-VAE is optimised, ELBO (Eq. 1) in the original form (Kingma & Welling, 2014) is approximately equal to the log-likelihood of æ; regardless 3 = 1 or not.
Finally, the predetermined conditional distribution P and the true conditional distribution after optimization pRe (x) (t) are examined using B in the input Euclidean space of x. Assume (2p( Φ) Nax; &t, I)... In this case, the metric D(x, .t) is derived as log
wllgggppeee PSp (x]()
(1/20") x =13 Const. From Eq. 13, the following equations are derived:
( [D(xr, x]] (((w))
20 x
Eg
Σ(s; (ar: 2:) n3/2, (17) 20
1
-5:)a=))))) (yg(( [1r] - =;) 30².
(18)
Because the variance of each dimension is estimated as _8 the true conditional distribution after optimization is approximated as (2o(2 ~ Nax è. B,02 I). If 3 1, i.c. the original VAE, ( æ)) and pRo (zl(²) are equivalent expected. If 3 o 1, however, PRp [z] (ē) and pro (z) (é) are different. Actually, what B- VAE does only to scale the variance of the pre-determined conditional distribution in the original VAE by factor of B, because 3-VAE objective can be rewritten as:
(log) ( [lo à o² I]] - 3DLL (-) 3 (E llog (logN((ss (x;; aē 30 3)] DKI(-)) + const. (19)
More detailed discussions about prior works (Higgins et al. (2017); Alemi et al. (2018); Dai et al. (2018); Dai & Wipf (2019); Tishby et al. (1999); Goyal (2001)) are explained Appendix A.
3.5 QUANTITATIVE PROPERTIES TO VALIDATE THE THEORY
This section shows three quantitative properties in VAE with prior (x;; 0, I,), to validate the the- ory in section 3.3. The second and third properties also provide practical data analysis approaches. The derivation of equations in the second and third properties are explained in appendix C.
Norm of By equal to 1: Let eli) be a vector (O,...,, I,.... ,0) where the j-th dimension is 1, and 5tt
others are O. Let (z) ) be (z)) Deca (z + e)(e)) where e denotes a minute value for the numerical differential. From Eq. 10, the squared norm of Fy, can be numerically evaluated as the first term of Eq. 20. This value will be equal to at any æ and dimension j except DKI,((2) = 0.
2 )d 2)(x) ~ 3 (o)(mm) 2 'I., Gats,) ~ 'xy, GT = 1.
(20)
If observed, the existence of an implicit isometric embedding can be shown because of unit norm
and orthogonality (Rolínek et al., 2019). Eq. 20 also show j({)) 2 "xz, Gz ~ 2. implying that a
noise (=))) added to each dimension of latent variable causes an equal noise 3/2 in the input space.
PCA-like feature: When the data manifold has a disentangled property in the given metric, the variance of the j-th implicit latent component y, can be roughly estimated as
f Un (p(u,)dy ~ 2 x<>p(=)
3
E
-2
(21)
The average Ela 2 on the right allows evaluating the quantitative importance of each dimension in practice, like the eigenvalue of PCA. Note that a dimension whose average is close to implies DKLj(z) 0. Such dimension has information and is an exceptions of the property in Eq. 20.

---

Under review as a conference paper at ICLR 2021
PDF
E[3 E,, 1-16
V2/4
p(s,): H[5 2is
V3/6
p(s.)
0, 1155)
51-55
3/33/3
-V2
V2
8/3/3
Figure 2: PDFs of three variables generate a toy dataset.
Estimation of the data probability distribution: First, assume the case m = n. Since the y space is isometric to the inner product space of G.z the PDFs in both spaces are the same. The Jacobian determinant between the input space and inner product space, giving the the ratio PDFs, is derived as |G We set (()) to the prior. Thus, the data probability in the input space can be estimated by G and either the prior/posterior La after training, as the following last two equations:
p(x)= = Gz Gz)rp(1) x |Gal (())
II ((a) x |Gzl exp - - j=1
(22)
In the case m > the derivation of the PDF ratio between the input space and the inner product space generally intractable, except for G = ag where ow is an z-dependent scalar factor. In this case, the PDF ratio is given by a. n/2, Thus, p(ar) can be estimated as follows:
p(x) x az ((P
II Tyta) j=1
x On
exp
Lo
(23)
Equations 22 and 23 enable probability-based quantitative data analysisssmpllig in practice.
4 EXPERIMENT
We show the experiments of the quantitative properties presented in Section 3.5. First, the results of the toy dataset are presented. Then, the results of CelebA are shown as a real data example.
4.1 EVALUATION OF QUANTITATIVE PROPERTIES IN THE TOY DATASET
The toy dataset is generated as follows. First, three dimensional variables $1, $2, and s3 are sampled in accordance with the three different shapes of distributions p(s1). p(22), and p(sa), as shown in Fig. 2. The variances of $1, $2, and Sy 1/6. 2/3, and 8/3, respectively, such that the ratio of the variances is 1:4:66. Second, three 16-dimensional uncorrelated vectors V1, U2 and Us with L2 norm 1, are provided. Finally, 50, 000 toy data with 16 dimensions are generated by x = Ei 8,0% The data generation probability p(x) is also set to p(s1)p((2) (p))))))) If our hypothesis correct, p(y,) will be close to p(,)). Then, (a(()) xc dzy/dy, = (((ys)/((s will also vary lot with these varicties of PDFs. Because the properties presented in Section 3.5 are calculated from (ze),, our theory can be easily validated by evaluating those properties.
Then, the VAE model is trained using Eq. 1. We two kinds of the reconstruction loss D(.,) )) to analyze the effect of the loss metrics. The first is the square error loss equivalent to sum square error (SSE). The second is the downward-convex loss which we design Eq. 24, such that the shape becomes similar to the BCE loss as in Appendix H.2:
((x, .) az x 12
where az (2/3 1|=11/211 and G, Gr I
(24)
Here, an is chosen such that the mean of an for the toy dataset is 1.0 since the variance of æ is 1/6+23++8/=7/2. The details the networks and training conditions are written in Appendix D.I.
Then the network is trained with two types of reconstruction losses. The ratio of transform loss to coding loss for the square error loss 0.023, and that for the downward-conves. loss is 0.024. As expected in section 3.2, the transform losses are negligibly small. Tables I and show the measurements
2 ) 2 (z) (shown as D D)(2), and ()()) described in Section 3.5. In these tables, 21, 22, and 23 show acquired latent variables. "Av." and "SD" are the average and standard deviation, respectively. To begin with, the norm of the implicit orthonormal basis
7

---

Under review as a conference paper at ICLR 2021
Table 1: Property measurements of the toy dataset trained with the square error loss.
Table 2: Property measurements the toy dataset trained with the downward-convex loss.
variable
21
2/
23
variable
21
2/
23
Dj
Av.
0.965
0.925
0.972
2
Av.
0.964
0.928
0.978
SD
0.054
0.164
0.098
SD
0.060
0.160
0.088
(x)
Av.
0.162
0.726
2.922
Av.
0.161
0.696
2.695
SD
0.040
0.466
1.738
SD
0.063
0.483
1.573
(((z)
Av.
3.33el
1.46e2
5.89e2
(a)
3.30e1
1.40e2
5.43e2
(Ratio) Av.
1.000
4.39
17.69
(Ratio) Av.
1.000
4.25
16.22
x100
=30-
Estimeted Prob.
RR 0.434
Pood Salmmna
R 0.771
Estimated Pron.
R 0.910
Estimatedd Prob.
R 0.905
6.00
0.25
0.50
0.75
1,80
800
0.25
0.50
0.75
1.00
8:0
0.25
0.50
0.75
1,00
800
0.25
0.50
6.75 1.00
Gonaratian ProD.
Generation Prob.
Generation Prob.
Genaration Prob.
(a) (())
(b) exp(-Lxx
(c)
3/2 p()()) II, f()) (d)
3/2
exp(-Lz//3)
Figure 3: Scattering plots of the data generation probability (x-axis) versus four estimated prob- abilitics (y-axes) for the downward-conwe loss. y-axes are (a) p(H())), (b) exp(-L=/3), (c) 3/2 ag (()) II, Jj(m): and (d) a
3/2
discussed. In both tables, the values of (m))) ( are close 1.0 in each dimension as described in Eq. 23. By contrast, the average of (z), which corresponds to 'x,, Gy is different in each dimension. Therefore, the derivative of with zj. the original latent variable of VAE, is not normalized.
Next, the PiA-like feature is examined. The average of ))(m) 2 in Eq.21 and its ratio are shown in Tables 1 and 2. Although the average of dz(=)) 22 is rough estimation of variance, the ratio is close to 1:4:16, i.e., the variance ratio of generation parameters 81, 82, and 83. When comparing both losses, the ratio of 82 and 83 for the downward-convex loss is somewhat smaller than that for the square eπrоr. This is explained as follows. In the downward-conver loss, [ tends to be 1/aa from Eq. 12, i.c. 'xy; (az Dy = dzk- Therefore, the region in the inner product space with larger norm is shrunk, and the estimated variances corresponding $2 and 83 become smaller.
Figure shows the scattering plots of the data generation probability p(æ) and estimated probabil- ities for the downward-conee loss. The plots for the square error loss are shown Appendix E. Figure 3a shows the plots of p(x) and the prior probabilities p(H(z)). This graph implies that it is difficult to estimate p(x) only from the prior. The correlation coefficient shown as "R" (0.434) is also low. Figure 3b shows the plots of p(x) and Lap (3), i.c., the lower bound of likelihood. The correlation coefficient (0.771) becomes better, but is still not high. Next, Figures 3c and 3d show the plots of 3/2 (()())) II (z())) and 3/2 exp(- (3) in Eq. 23. These graphs, showing high correlation coefficients around 0.91, support that the objective L. in Eq. 3 is optimized in also
the inner product space G, In the case of the square error loss, the plots with cxp(- shows a high correlation coefficient 0.904 because an is allowing the probability estimation from La in Eq. 3. The ablation study with different PDR losses, and 3 is shown in Appendix E.
4.2 EVALUATIONS IN CELEBA DATASET
This section evaluates the first and second quantitative properties of VAE trained with the CelebA dataset (Liu et al., 2015) as example of real data. This dataset is composed of 202,599 celebrity images. In use, the images are center-cropped to form 64 x 64 sized images.
http::/wwabbbiccu.....c.h//moomm cuhk
8

---

Under review as a conference paper at ICLR 2021
-1
Estimated Variance
Variance
Norm: Avenge
Nonn: Awenee
388.7
Deviation
Nomm: Deviation
Large
Average of G100 -
18-3
Mean: 1.83/ SD 0.13
1E43
Meam: 0.92/ SD: 0.04
214,4
2.0
(4) b wldd
( l (lldd
1E12
IE+2
176.1
Averuge of
10
Small Medium
F16 14.2
[F+1
IE+1
F17
IE-0
0.0
1E40
Numbher lstent varisbles
16
30
Number lasent variallss
10
20
#22
1.0
(deseending onder Estimated Variance)
(descendingoddrr of Estimated Variance)
Figure 4: Graph of ()(x)) average and VAE for CelebA dataset.
Figure 5: Graph of j(a))
Figure 6: Dependency of decoded image changes with
in
average and VAE for CelebA dataset with explicit decomposed loss.
in
2j = -2 to 2 on the average of )(()) -2
We use SSIM, which is popular in image compression, as reconstruction loss. The details of networks and training conditions are written in Appendix D.2.
Figure shows the averages of ((()) -2 in Eq.21 as the estimated variances, well the average and the standard deviation of (z) in Eq.20 as the estimated square norm of implicit transform. The latent variables 2/ are numbered in descending order by the estimated variance. In the dimensions greater than the 27th, the averages of (z(x) -2 are close to 1 and that of / 2 is close O, implying (-I.(-) O. Between the 1st and 26th dimensions, the mean and standard deviation of 3 (D) averages are 1.83 and 0.13, respectively. These values seem almost constant with small standard deviation; however, the mean is somewhat larger than the expected value This result implies that the implicit transform can be considered as almost orthonormal by dividing v1.83= 1.35. Thus, the average of j(x) -2 still can determine the quantitative importance of each latent variable. This also mean that the added noise to each yj is around 1.83(3/2). We also train VAE by the decomposed loss explicitly, where Tag is set to D(x. .) + D(# &t) + (+)(()) Figure 5 shows the result. Here, the mean and standard deviation of f Dz averages are 0.92 and 0.04, respectively, which suggests almost a unit norm. As result, the explicit use of decomposed loss matches the theory better, allowing better analysis. The slight violation of the norm in the conventional form needs more exact analysis as a futuree study.
Figure 6 shows decoder outputs where the selected latent variables are traversed from -2 to 2 while setting the rest to O. The average of (m) -2 is also shown there. The components are grouped by the average of (((()) -2 such that 21, #2, 23 to the large, 216, 217 to the medium, and 232 to the small, accordingly. In the large group, significant changes of background brightness, the direction of the face, and hair color are observed. In the medium group, we can see minor changes such as facial expressions. However, in the small group, there are almost changes. This result strongly supports that the average of 0j(x)) shows the importance of each latent variable. The traversed outputs for all the component and results with another conditions are shown in Appendix F.
CONCLUSION
This paper provides quantitative understanding of VAE by non-linear mapping to isometric em- bedding. According to the Rate-distortion theory, the optimal transform coding is achieved by using PCA/KLT orthonormal transform, where the transform space is isometric to the input. From this analogy, we show theoretically and experimentally that VAE can be mapped to an implicit isometric embedding with a scale factor derived from the posterior parameter. Based on this property, we also clarify that VAE can provide a practical quantitative analysis of input data such as the probability estimation in the input space and the PCA-like quantitative multivariate analysis. We believe the quantitative properties thoroughly uncovered in this paper will be milestone to further advance the information theory-based generative models such as VAE the right direction.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alexander Alemi, Ben Poole, lan Fischer, Joshua Dillon, Rif A. Saurous, and Kevin Murphy. Fix- ing broken ELBO. In Proceedings of the 35th International Conference on Machine Learn- ing(ICML), pp. 159-168. PMLR, July 2018.
Johannes Ballé, Laparra Valero, and Simoncelli Eero Density modeling of images using general- ized normalization transformation. In Proceedings of the 41 International Conference Learning Representations (ICLR), May 2016.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In Proceedings of the 6th International Conference on Learning Representations (ICLR), April 2018.
Toby Berger (ed.). Rate Distortion Theory: Mathematical Basis for Data Compression. Prentice Hall, 1971. ISBN 0137531036.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances Neural Information Processing Systems, pp- 2610-2620, 2018.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. In International Conference on Learning Representations (ICLR), May 2019.
Bin Dai, Yu Wang, John Aston, Gang Hua, and David Wipf. Hidden talents of the variational autoencoder. The Journal of Machine Learning Research, 19:1573-1614, January 2018.
Vivek K Goyal. Theoretical foundations of transform coding. IEEE Signal Processing Magazine, 18:9-21, September 2001.
Qing Han and Jia-Xing Hong. Isometric Embedding of Riemannian Manifolds in Euclidean Spaces. American Mathematical Society, 2006. ISBN 0821840711.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. DAB: Learning basic visual concepts with constrained variational framework. In Proceedings of the 5th International Conference on Learn- ing Representations (ICLR), April 2017.
Sicong Huang, Alireza Makhzani, Yanshuai Cao, and Roger Grosse. Evaluating lossy compression rates of deep generative models. In Proceedings of the 37th International Conference on Machine Learning (ICML), July 2020.
F. Jin, P. Fieguth, L.. Winger, and E. Jernigan. Adaptive wiener filtering of noisy images and image sequences. In IEEE International Conference Image Processing, Sept 2003.
Keizo Kato, Zhing Zhou, Tomotake Sasaki, and Akira Nakagawa. Rate-distortion optimization guided autoencoder for generative analysis. In Proceedings of the 37th International Conference on Machine Learning (ICML), July 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In Proceedings of the 35th Interna- Jul 2018.
tional Conference on Machine Learning (ICML), pp.
Diederik Kingma and Max Welling. Auto- encoddng variational bayes. In Proceedings of the 2nd International Conference Learning Representations (ICLR), Banff, Canada, April 2014.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning (ICMI.) volume 97, pp. 41 4124, PMLR, June 2019.
10

---

Under review as a conference paper at ICLR 2021
Romain Lopez, Jeffrey Regier, Michael Jordan, and Nir Yosef. Information constraints auto- encoding variational bayes. Advances Neural Information Processing Systems, pp. 6114 6125, 2018.
William A. Pearlman and Amir Said. Digital Signal Compression: Principles and Practice. Cam- bridge University Press, 2011. ISBN 0521899822
Kamisetty Ramamohan Rao and Pat Yip (eds.). The Transform and Data Compression Handbook. CRC Press, Inc., Boca Raton, FL, USA, 2000. ISBN 0849336929.
Michal Rolinck, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea diree- tions (by accident). In Proceedings of Computer Vision and Pattern Recognition (CVPR), June 2019.
Gary J. Sullivan and Thomas Wiegand. Rate-distortion optimization for video compression. 1EEE Signal Processing Magazine, 15:74-90, November 1998.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. In The 37th annual Allerton Conference Communication, Control, and Computing, September 1999.
Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Processing, 13: 600-612, April 2001.
Norbert Wiener. Extrapolation, Interpolation, and Smoothing of Stationary Time Series. The MIT Press, 1964. ISBN 978 978-0-262-73005-1.

---

Under review as a conference paper at ICLR 2021
A DETAILED RELATION TO PRIOR WORKS
Firstly, we clarify the the difference between ELBO in Eq. and the objective L in Eq. 3 in the right direction. Then, we discuss the relation to the prior works. We also point out the incorrectness of some works.
A.I DERIVATION OF ELBO WITH CLEAR AND QUANTITATIVE FORM
We derive the reconstruction loss and KL divergence terms in ELBO (without 9) at in Eq. 1 when the objective of 3-VAE L' in Eq. 13 optimised. The reconstruction loss can rewritten as
(2 [log pa [x]=]] = 4s ()) log pe (z|2))2 = go (3/((z) log (oll))d
(25)
Let My(a)) be implicit isometric variable corresponding ((a))- Because the posterior variance in each isometric latent variable is constant 3/2, go (1)la)) = = Ny: My(a), (3/2)I,) will hold. If B/2 is small, p()) p(æ) will hold. Then, the next equation will hold also using isometricity;
pa [1)= pa p p = p( = p(o x)) (o())=)
(26)
Thus the reconstruction loss is estimated as:
f M(y: (3/2) (3/2) Mu(a), (B/2)I.) dy
(x)))
= - (n/2) Log(BBr)).
(27)
From Eq. 13, KL divergence is derived as:
DKL(') Rminn = log p(y) (n/2) log(Bre))
(28)
By summing both terms, ELBO at a can be estimated as
ELBO = Ex-)(a)) (ele) (x|=)] DKL(-)]
~ pp log p(y)
Ex log p(x)).
(29)
As a result, ELBO (Eq. 1) in the original form (Kingma Welling, 2014) is close the log- likelihood of æ, regardless 3 1 or not, when the objective of B-VAE (Higgins et al., 2017) is optimised.
Some of the prior VAE works do not explicitly distinguish between the reconstruction loss Ea-p(ála) [x] ELBO and the distortion D(#, .) in the objective Le by mistake, which leads to some incorrect discussion.
In addition, there have also been incorrect discussions in some prior works. In ELBO derivation, they use Ea-p(á am) [p( ( [t]] reconstruction loss, without discussing what kinds of properties the distortion probability p( (x) should be. In training VAE with real dataset, by contrast, they use a predetermined distortion metric D(x, .) like BCE and SSE as a reconstruction loss instead of log-likelihood of the distortion probability, without discussing what distortion probability should be after optimization.
Correctly, the distortion probability p()(x) after training is determined by 8 and the metric as Po( (x) (â; æ, (3/2) in the metric defined space. Then, by applying po (x) (o(() to Eq. 1, the value of ELBO will become log p(y) log p(x) regardless
not.
If D(x, + 5x)) G 8x 0(( (3) is not SSE, by introducing variable á L x where Le satisfies LL La = Gz. the metric D(..)) can be replaced by SSE in the Euclidean space of d.
A.2 RELATION TO TISHBY ET AL. (1999)
The theory described in Tishby et al. (1999) is consistent with our analysis. Tishby et al. (1999) clar- ified the behaviour of the compressed representation when the rate-distortion trade-off is optimized.
12

---

Under review as a conference paper at ICLR 2021
X denotes the signal space with fixed probability p(z) and ø X denotes its compressed representation. Let D æ) be a loss metric. Then the rate- rate-distortion trade-off can be described as:
L = Ixx; + 3' E (D(x,
(30)
p(æ..)
By solving this condition, they derive the following equation:
p( (+) x exp(-B" D(r,,))
(31)
As shown in our discussion above, p(o|z) Mit; a, (3/2) will hold in the metric defined space from our VAE analysis. This result is equivalent to Eq. 31 in their work if Dix, æ) is SSE and 3 is set to 3-1, as follows:
(- || 1|11-=zl1 =113
p() & exp(-B'D(a, = exp cx
x N (m)
(32)
2(B/2)
If (( a) is not SSE, the use of the space transformation explained in appendix A.I will lead to the same result.
A.3 RELATION TO 3- VAE (HIGGINS ET AL.., 2017)
In the 3-VAE work by Higgins et al. (2017), it presumed that the objective Le was not mistakenly distinguished from ELBO. In their work, ELBO equation is modified as:
Ep(m)] (a= (x] (BDb) (-)
(33)
However, they use the predetermined probabilities of po (2)2) such as the Bernoulli and Gaussian distributions in training (described in table I in Higgins et al. (2017)). As shown in appendix H.2, the log-likelihoods of the Bernoulli and Gaussian distributions can be regarded as BCE and SSE metrics, respectively. As result, the actual objective for training in Higgins et al. (2017) Eq. 33, but the objective Le in Eq. 3 using BCE and SSE metrics with varying B. Thus ELBO as Eq. form will become log p(x) in the BCE/ / SSE metric defined space regardless 3 1 or not, as shown in appendix A.I.
Actually, the equation 33 dose not show the log ligllikel of æ. When (+ ~ - log p(x) (n/2) log(3 and ) [p(x [ = (n/22) 2) (Bg() are applied, the value of Eq. 33 is derived as 3 p(z) + (8 - (n/2) log(Bre), which is different from the log-likelihood of x 3+1. 1.
Correctly, what 3-VAE really does is only to scale the variance of the pre-determined conditional distribution the original VAE by a factor of B. In the case the pre-determined conditional dis- tribution is Gaussian (x: ãt, o² o²I), the objective of B can be can be rewritten as a linearly scaled original VAE objective with a Gaussian N(x; &, I)²
() log)) 0²I) (D) = Eas()
[
log 2TT
x 2--a0
- ())
2
202
1x: 10-2. 213
= 3 Bgg())
log 27, 2
DKL(.)
2
2,302
+ log 2n 30²
log 2To²
2
= 3 (Eg,() N(;; B0² 1]] - DKL(.)) + const.
(34)
A.4 RELATION TO ALEMI ET AL. (2018)
Alemi et al. (2018) discuss the rate-distortion trade-off by the theoretical entropy analysis. Their work is also presumed that the objective Le was not mistakenly distinguished from ELBO, which leads to the incorrect discussion. In their work, the differential entropy for the input H, distortion D, and rate R are derived carefully. They suggest that VAE with = 1 is sensitive (unstable) because D and R can be arbitrary value the line R = H -73 3D H D. Furthermore, they also suggest
13

---

Under review as a conference paper at ICLR 2021
that R 2 H, D 0 0 and R 0, D H at B oo will hold shown the figure 1 of their work.
In this appendix, will show that 3 determines the value of R and D specifically. We also show that ~ H D will hold regardless 3 or not.
In their work, these values of H, D, and D mathematically defined as:
H f dæ (x) (og)"(()),
(35)
/ dæ p" (x)) / dz log
D /dm
log d(z)=),
(36)
/ dx pxp"(x) dz e(z))) log m(z)
e(zla)
R
(37)
Here, p'(x:) is a true PDF of æ, c(2|1:) stochastic encoder, <(z|x)) is a decoder, and m(2) is marginal probability of z.
Our work allows rough estimation of Eqs. 35-37 with / by introducing the implicit isometric variable explained in our work.
Using isometric variable y and the relation dz e(zla=) = dy e(y az). Eq. can be rewritten as:
D = p==
p- (xx)) f dy e(grlar) (x)) log d(lly))
(38)
Let My be the implicit isometric latent variable corresponding to the mean of encoder output ((z)- As discussed in section 3.3, e(ylax)) = Nw My: (B/2) will hold. Because of isometricity, the value of d(xaly) will be also close to e e(ylaz) = N (y:: Myy (8/2) on e(z (2), this important point has been discussed well this work. By using the implicit isometric variable, we can connect both theoretically, Thus, D can be estimated as:
Though ([=]z) must depend
/ /dx p'(()) f dy No pgg (3/2)L,) log Nog:N py: (3)
D
/ dx p" (m() (" log())re)
n 2 log(BTT)
=
(39)
Second, R is examined. m(y) is marginal probability of y. Using the relation dæ e(z[æ) dy e(ylæ) ()) and e(z) = (e(yl=)(dy//<z) (mll)//) = be rewritten
Eq. 37 can
R ~ dxxp((a) / dy (()l=) log
c(y x
(40)
m(y)
Because of isometricity, e(y ()) ~ p(((z) ~ Naē: (m) will approximately hold where a denotes a decoder output. Thus mzy can approximated by:
m I dx p" (x))()))) fde p=() N( x (3/2)Im)
(41)
Here, if //2, i.e., added noise, is small enough compared to the variance of æ, a normal distribution function term in this equation will act like delta function. Thus m(y) can be approximated as:
m(y) ddd p" (+) 8( x) (x)
(42)
In the similar way, the following approximation will also hold.
{ dy e(y|z) log log ~ dy e(y)) log pe (x) ~ / dit 8( - x) log p" (+) ~ loggp(x) (43)
14

---

Under review as a conference paper at ICLR 2021
By using these approximation and applying Eqs. 38-39, R in Eq. 37 can be approximated as: <(ylar)
/ |dw dæ (lxpp"0 p" (a) / ddy e(y))z) log pr(x))
R =
- - - dx p² (xx) logg"(x) (-))))))))pp)
(-fdartion p()) f dy e(g)1z) log log ((y)))))))
H log(3 H-D
(44)
As discussed above, and D can be specifically derived from 3. In addition, Shannon lower bound discussed in Alemi et al. (2018) can be roughly verified in the optimized VAE with clearer notations using/ B.
From the discussion above, we presume Alemi et al. (2018) might wrongly treat D in their work. They suggest that VAE with 3 = 1 is sensitive (unstable) because D and R can be arbitrary value on the line R H BD H D; however, our work as well as Tishby et al. (1999) (appendix A.2) and Dai & Wipf (appeniapiiiix A.5) show that the differential entropy of the distortion and rate, i.e. D and R. are specifically determined by 33 after optimization, and R H D will hold for any 3 regardless 3 1 or not. Alemi et al. (2018) also suggest D should satisfy D 2 0 because D is a distortion; however, we suggest D should be treated as a differential entropy and can be less than O because æ is once handled as a continuous signal with stochastic process in Eqs. 35-37. Here, D = (n/22) (fg(e can be -oo if 3 0, as also shown in Dai Wipf (2019). Thus, upper bound of R at B - 0 is not H, but R = H (-oo) = oo, as shown in RD theory for continuous signal. Huang et al. (2020) show this property experimentally in their figures 4-8 such that A seems to diverge if MSE is close to O.
A.5 RELATION TO DAI ET AL. (2018) AND DAI & WIPF (2019)
Our work is consistent with Dai et al. (2018) and Dai & Wipf (2019).
Dai et al. (2018) analyses VAE by assuming a linear model. As result, the estimated posterior is constant. the distribution of the manifold is the Gaussian, our work and Dai et al. (2018) give a similar result with constant posterior variances. For non-Gaussian data, however, the quantitative analysis such as probability estimation is intractable using their linear model. Our work reveals that the posterior variance gives scaling factor between in VAE and y in the isometric space when VAE is ideally trained with rich parameters. This is validated by Figures 3c and 3d, where the estimation of the posterior variance at each data point is key.
Next, the relation to Dai & Wipf (2019) is discussed. They analyse a behavior of VAE when ideally trained. For example, the theorem 5 in their work shows that D + (d/2) log 7 + (1) and R > - -(f/2) +0((1) hold i - +0, where 7, d, and fy denote variance of d(æ) data dimension, and latent dimension, respectively. By setting y = 3/2 and d = á n, this theorem consistent with R and D derived in Eq. 39 and Eq. 44.
A.6 RELATION TO TRANSFORM CODING (GOYAL, 2001)
We show the optimum condition of VAE shown in Eq. 14 can be mapped to the optimum condition of transform coding (Goyal, 2001) as shown in Eq. 4. First, the derivation of Eq. 4 is explained by solving the optimal distortion assignment to each dimension. In the transform coding for m dimensional the Gaussian data, an input data æ is transformed to using an orthonormal transform such as KLT/DTT. Then each dimensional component 2j is encoded with allowing distortion dy- Let D be a target distortion satisfying D ΣØ= dy. o denotes a variance of each dimensional component 2j for the input dataset. Then, a rate R can be derived as Σド= 2 log(o?;/d,), By introducing a Lagrange parameter À and minimizing rate-distortion optimization cost L = D+AR, the optimum condition derived as:
(45) This result is consistent with Eq. 14 by setting 3 Aopt 2D/m. m.. This implies that L'g in Eq. 14 is a rate- diee-orttttiionnn optimization (RDO) cost of transform coding when æ is deterministically transformed to y in the implicit isometric space and stochastically encoded with distortion B/2..
Aope = 2D/m, dz= = DDmm= 22
15

---

Under review as a conference paper at ICLR 2021
B ESTIMATION OF THE CODING LOSS AND TRANSFORM LOSS IN 1-DIMENSIONAL LINEAR VAE
This appendix estimates the coding loss and transform loss in 1-dimensional linear 33-AE for the Gaussian data, and also shows that the result is consistent with the Wiener filter. Let æ be a one dimensional data with the normal distribution:
I E IR, x ~ N(z;0,,,,))
(46)
Let be one dimensional latent variable. Following two linear encoder and decoder are provided with constant parameters b, and σ₂ optimize:
2 ar + oge where E O,,
$ bz.
(47)
First, KL divergence at x, DKL is derived. Due to the above relationship, we have
p(z) (2; 0, (aoz))²).
(48)
Using Eq. 6, Kl-ddivergence at x can be evaluated as:
log 2TEe = - log ox + a²x²² 1
DKL= = - log(o=p(z)) 2
(49)
2 2
Second, the reconstruction loss x D₂ is evaluated as:
Dr
F. c~N((1001) [(x (b(ax a=c)))²] ((ab- - 1)x;) + 12o02
(50)
Then, the loss objective T.r = D2 + 3KKL is averaged over ~ Ni; 0, 202), and the objective minimize is derived as:
E N=;0,,
[Le] (ab - +bo + 2o:
(-
+ a²az² - --2
(51)
2
Here, (ab - bubb-)aap=² 2 and o²a=² coding loss Dc, respectively.
2 in the last equation are corresponding to the transform loss Dr and
By solving dL/da- O, dL/db = 0, and dL/do= 0, a. b. and a₂ are derived as follows:
1/O2-
ox 11 V 23/0z²
2
2//32
a= ox (1+ (1+ v - 23/0
(52)
From Eq. 52, Dr and De derived as:
2
Vv 2/3/0
0
2
Dr
oz
2
De 3/2.
(53)
As shown in section 3.3, the added noise, 3/2, should be reasonably smaller than the data variance a=² 2. o > 3, and a, Eq. 52 can be approximated as:
(8/2)² 3/2 Dc-
Dr 0
(54)
As shown in this equation, Dr/De is small in the VAE where the added noise is reasonably small, and DT be ignored.
Next, the relation to the Wiener filter discussed. We consider an simple 1-dimensional Gaussian process. Let x A(x;; 0, o2) be input data. Then, is scaled by 8, and Gaussian noise ~
16

---

Under review as a conference paper at ICLR 2021
(n; O, o is added. Thus, y + n is observed. From the Wiener filter theory, the estimated value with minimum distortion, & can be formulated as:
30²²
f 0²0²² 2 + on 2
(55)
In this case, the estimation error derived as:
On El( 21) (5 +0,
8 0x
2
0x 02 2 (o," 2/52)
(n
2 2/2.)
(56)
0 o 2)2
In the second equation, the first term is corresponding to the transform loss, and the second term is corresponding to the coding loss. Here the ratio of the transform loss and coding loss is derived as on 2 as 2) By appying = 1/0, and on = og to on /1820 x²²) and assuming of > 8/2, this
ratio can be described as:
On 2
8/2 o²
4
3/2 o
((
8/2 2
o2= 2
+0
(57)
0²
1
V 23/02²
This result is consistent with Eq. 54, implying that optimized VAE and the Wiener filter show similar behaviours.
C DERIVATION OF QUANTITATIVE PROPERTIES IN SECTION 3.5
C.I DERIVATION OF THE ESTIMATED VARIANCE
This appendix explains the derivation of Eq. 21 in Section 3.3. Here, we assume that 2ÿ is mapped to Yj such that yj is set to 0 at 2j = 0. We also assume that the prior distribution is (2; 0, In). The variance is derived by the subtraction of E'y;; the square of the mean, from E[y]], the square mean. Thus, the approximations of both Ely,] and Elyÿ] are needed.
First, the approximation of the mean E E'gy,] is explained. Because the cumulative distribution func- tions (CDFs) of yj are the same CDF of Zj. the following equations hold:
L p(uj)dy; = [ p(zy)dz, 0.5, / p(u,) Jdy, = So p(zz)dz, 0.5.
(58)
This equation means that the median of the y, distribution is O. Because the mean and median are close in most cases, the mean E[y,] can be approximated as O. As result, the variance of yy can be approximated by the square mean Fly]].
Second, the approximation of the square mean Elyÿ] is explained. The standard deviation of the posterior (()) assumed as a function of zj. regardless of æ. This function is denoted as oz(2)). For zj 2 0, yj is approximated as follows, using Eq. 11 and replacing the average of 1/a, (4,) over 4j [0, zj] by 1/0;(2;);
2 dy dá, = 1 /. o5(E,) dá = (2),(,,) V2 dí, V
B
2j
// dı,
(59)
oz,))²)
The same approximation applied 2, Then the square mean of yi approximated as follows, assuming that the correlation between o(2,) and 2,² is low:
f y pppp((;)))))) )dy; = 3
2j
p(zj)dz, ~ B a,(t,) ((z))ddzs)) (=((,))"p((,>>)) 2j =,2p(zz)) 2p(zz)dzz.
(60)
(,(25)
Finally, the square mean of yı is approximated as the following equation, using f =, (.(((z))dz and replacing 0, oz,2))² by ,0)))² 2, i.e., the posterior variance derived from the input data:
J v, p(y)))y)
8 / - (2(2)) ( p(z)) 8 E (o,())
E [o -21. (61)
xx>p(a,) =)>p(es)
a<p((a)
Although some rough approximations are used in the expansion, the estimated variance in the last equation seems still reasonable, because j(mر shows scale factor between yj and 2j while the vari- ance of 2, is always for the prior N22 0, 1). Considering the variance of the prior 22,pp(z,)) (22))dz in the expansion, this estimation method can be applied to any prior distribution.
17

---

Under review as a conference paper at ICLR 2021
Input space
Isometric space en (c R
Implicit orthonormal space
S Suyur (c R
Sartoo (c R*)
Ōwiey: m = Orthogonal Matrix
Ox/Cow :
1 a 1/2 Im
ay Mayy pp dyy
dy
dys
Scaling
dy
Rotation Reflection
dys
V "n(dVinpul) ax (dVViss)
Vn (TV
V ort
dyj
j=1
Figure 7: Projection of the volume element from the implicit orthonormal space to the isometric space and input space. (-) denotes n--dimensional volume.
C.2 DERIVATION OF THE DATA PROBABILITY ESTIMATION
This appendix shows the derivation of variables in Eqs. 22 and 23. First, the derivation of Le for the input æ is described. Then, the PDF ratio between the input space and inner product space is explained for the cases m and m > n.
Derivation of Lep for the input æ:
We approximate Ez-go mn) [-] 1 (Dx Decs (Mz σ²)) + Deco oz))), i.e. the average of two samples, instead of the average over z ~ (= x) DKL( ()) can be calculated from μ and o using Eq. 2.
As shown in in Eq. 1, Le is denoted as (=l=-
The PDF ratio in the
de By holds from Eq. 12. |dx/Oy" Gz = 1 also holds by calculating the determinant. Finally, Ox Op// is derived as |., 1/2 using 1yy 03x = 10x/Oyl
The PDF ratio for m = n is a Jacobian determinant between two spaces. First,
) G Oz Im
The PDF ratio in the Although the strict derivation needs the treatment of the Riemannian manifold, we provide simple explanation this appendix. Here, assumed that (+) > holds for all [1, ..r]. If
and G. = Gz I
(-) for some j, is replaced by the number of latent variables with DKLG) (-) 0.
For the implicit isometric space Siso ","", there exists matrix L. such that both y Lr and Ge = Lg Lg holds. w denotes a point in Si i.c., w E Siso Because G2 is assumed as aze Im in Section 3.5, L = 0g 1/2 Im holds. Then, the mapping function w h(x:) between Simput and Sio is defined, such that:
ah(w:) Ow La, and h(xx())) (0) for 3 az E Staput and (0) S Ōx
(62)
dz
Let dæ and dw are infinitesimal displacements around æ and w h(x),, such that w + dw 10( + da).. Then the next equation holds from Eq. 62:
8w Lz
(63)
Let (1) (2), (1) and (2) be two arbitrary infinitesimal displacements around æ and h(wx), such that (1) (1) and (2) (2) Then the following equation holds, where denotes the dot product.
(1)
"(Lzd 8x (L = (1) (2)
(64)
This equation shows the isometric mapping from the inner product space for x E Sippt with the metric tensor G to the Euclidean space for S
Note that all of the column vectors in the Jacobian matrix dx: 0x/0g ldy also have a unit norm and are orthogonal each other in the metric space for æ Supput with the metric tensor Gz. Therefore, the m x n Jacobian matrix 0w/0g /dy should have a property that all of the column vectors have a unit norm and orthogonal to each other in the Fuclidean space.
18

---

Under review as a conference paper at ICLR 2021
Then n-dimensional space which is composed of the meaningful dimensions from the implicit iso- metric space is named as the implicit orthonormal space Sortho- Figure 7 shows the projection of the volume element from the implicit orthonormal space to the isometric space and input space. Let ortho be an infinitesimal n-diiennsional volume element in Sorhhn- This volume element is a n-dimensional rectangular solid having each edge length dyz. Let VdVx) be the n-dimensional volume of a volume element dVx. Then, Vn (dVorth)) = II dy, holds. Next, ortho is pro- jected to n dimensional infinitesimal element dv in So by 0w/Oy. Because of the orthonor- mality, dVis is equivalent to the rotation reflection of vVootho and (v(vVis)) is the same as Vn (dVortho)) i.e., [Л dyj- Then, dViso is projected to n-dimensional element d'Vinput in Sinput by 0x/Ow = - = Ox 1 Im. Because each dimension is scaled equally by the scale factor az -1/2, Vn (dVumut = II aæ 1/2dyy, = ax nn/2 Vn (dVooho))) holds. Here, the ratio of the vol- ume element between Sinput and Sortho is Vn (dVippu)))) // (dVorth)))) aæ mn/2 Note that the PDF ratio is derived by the reciprocal of Vn VddVmpp)) )/ //V,, (d (oVothh)) As a result, the PDF ratio is derived
as n/2
D DETAILS OF THE NETWORKS AND TRAINING CONDITIONS IN THE EXPERIMENTS
This appendix explains the networks and training conditions in Section 4.
D.1 TOY DATA SET
This appendix explains the details of the networks and training conditions the experiment of the toy data set in Section 4.1.
Network configurations:
FC(i, o, f) denotes FC layer with input dimension i, output dimension o, and activate function f.
The encoder network is composed of FC(16, 128, tan2)-FC(128, 64, tahh)-FC(64, 3, linear) (for and o). The decoder network is composed of FC(3, 64, tanh)-FC(64, 128, tahh)-FC(128,, 16, linear).
Training conditions:
The reconstruction loss (.:.)) is derived such that the loss per input dimension calculated and all of the losses are averaged by the input dimension summation of (-)
16. The KI. divergence is derived
(-) as explained in Eq. 2.
In our code, use essentially the same, but constant factor scaled loss objective from the original 3-VAE form La = : + (DDL()) in Eq. 1, such as:
L = A D(..) + (.).G))).
(65)
Equation 65 is essentially equivalent to = = D(:-))+ BDLLn)() multiplying a constant À = 3-1 to the original form. The reason why we use this form is as follows. Let ELBO true be the true ELBO in the sense of log-likelihood, such as E'log (x)). As shown in Section 3.3, the minimum of the loss objective the original VAE form is likely to be a -EEBOO + Constant. If we use Eq. 65, the minimum of the loss objective will be ELBO + Constant, which seems more natural form of ELBO. Thus, Eq. 65 allows estimating a data probability from L, in Eqs. 22 and 23, without scaling La by 1/B.
Then the network is trained with A = 3-1 100 using 500 epochs with a batch size of 128. Here, Adam optimizeri is used with the learning rate of 1e-3. We use a PC with CPU Inter(R) Xcon(R) CPU E3-1280v5 @ 3. 7,00GHz, 32GB memory equipped with NVIDIA GeForee GTX 1080. The simulation time for each trial is about 20 minutes, including the statistics evaluation codes.
In our experiments, À or 3 -1 i.e., 100, seems somewhat large. This is caused by the use of the mean square error reconstruction loss. In contrast, KL divergence is the sum for the whole image, which can be thought of as a rate for the whole image. Considering the number of input dimensions, 3' (2/16) 16/X 0.16 is thought of B in the general form of VAE.
19

---

Under review as a conference paper at ICLR 2021
D.2 CELEBA DATA SET
This appendix explains the details of the networks and training conditions in the experiment of the toy data set in Section 4.2.
Network configurations:
CNN(W, h, S, c. denotes CNN layer with kernel size (w, h), stride size dimension c, and activate function f. GDN and IGDN are activation functions designed for image compression (Ballé et al., 2016). This activation function is effective and popular in deep image compression studies.
The encoder network is composed of CNN(9, 9, 2, 64, GDN) - CNN(5, 5, 2. 64, GDN) CNN(5, 5, 2, 64, GDN) CNN(5, 5, 64, GDN) FC(1024, 1024, softplus) FC(1024, 32, None) (for μ and σr) in encoder.
The decoder network composed FC(32, 1024, (ittplus)-FC(IOOO) 1024, softplus) CNN(5, 5, 2, 64, IGDN) CNN(5, 5, 2, 64, IGDN) CNN(5, 5, 2, 64, IGDNN)CNN(9, 9, 2, 3, IGDN).
Training conditions:
In this experiment, SSIM explained in Appendix H.2 is used as reconstruction loss. The recon- struction loss ((,,)) is derived as follows. Let SSTM be SSIM calculated from two input images. Then 1 - SSIM is set to D...)) The KI. divergence is derived explained in Eq.
summation of DKL() as
We also use the loss form in Equation 65 in our code. In the case of the decomposed loss, the loss function L set to A ((D(z, .) + (& a)) + DKL(-) ) our code. Then, the network trained with À 3-1 used with the learning rate of le-3.
000 using batch size of 64 for 300,000 iterations. Here, Adam optimizer is
We use PC with CPU Intel(R) Core(TM) 17-6850K CPU @ 3.60GHz, 12GB memory equipped with NVIDIA GeForce GTX 1080. The simulation time for each trial is about 180 minutes, includ- ing the statistics evaluation codes.
In our experiments, À = 3- = 000 seems large. This caused by the use of SSIM. As explained in Appendix H.2, SSIM is measured for whole image, and its range is between 0 and 1. The order of 1 SSIM is almost equivalent to the mean square error per pixel, as shown in Eq. 74. As explained in Appendix D. KL divergence is thought of as rate for the whole image. Considering the number of pixels in image, 3 = (A/(64 x 64)) = 4096/A X = 4.096 is comparable to B in the general form of VAE.
"Google provides code in the official Tensorflow library http:://gtthh..ccmmmesssaoossmmm
20

---

Under review as a conference paper at ICLR 2021
E ADDITIONAL RESULTS IN THE TOY DATASETS
E. SCATTERING PLOTS FOR THE SQUARE ERROR LOSS IN SECTION
4.1 Figure 8a shows the plots of p(az) and estimated probabilities for the square error coding loss in Section 4.1, where the scale factor az in Eq. 23 is 1. Thus, both exp(-L,/B) and (()) II )(r) show a high correlation, allowing easy estimation of the data probability in the input space. In contrast, (x)) still shows low correlation. These results are consistent with our theory.
Estimated Prob.
R 0.434
Estirnated Prob.
R 0.904
Estimated Prob.
3
0.911
2
8.00
0.25
0.50 0.75
1.00
8:00
0.25
0.50
0.75
1.00
8:00
0.25
0.50
0.75 1.00
Generation Prob.
xio
Generation Prob.
Generation Prob.
(a) (()
(b) exp( L,/8)
(c) p(H(() II, (a))
Figure 8: Plots of the data generation probability (x-axis) versus estimated probabilities (y-axes) for the square error loss. y-axes are (a) p(u(m))) (b) exp(-Lz/ and (c) p(H)m) II, f(())
E.2 ABLATION STUDY USING 3 TOY DATASETS, 3 CODING LOSSES, AND 10 33 PARAMETERS.
In this appendix, we explain the ablation study for the toy datasets. We introduce three toy datasets and three coding losses including those used in Section 4.1. We also change 3 X from 1 to 1, 0,000 in training. The details of the experimental conditions are shown as follows.
First, we call the toy dataset used in Section 4.1 the Mix dataset in order to distinguish three datasets. The second dataset is generated such that three dimensional variables 81, 82, and 83 sampled in accordance with the distributions p(s1), p(2)), and p(s3) in Figure 9. The variances of the variables are the same as those of the Mix dataset, i.e., 1/6, 2/3, and 8/3, respectively. We call this the Ramp dataset. Because the PDF shape of this dataset is quite different from the prior (2 0, Iz), the fitting will be the most difficult among the three. The third dataset is generated such that three dimensional variables 81, 82, and 83 are sampled in accordance with the normal distributions (s1 ; 0, 1 166., (s22; 0, 2/3). and ((sx; 0, 8/3), respectively. We call this the Norm dataset. The fitting will be the easiest, because both the prior and input have the normal distributions, and the posterior standard deviation, given by the PDF ratio at the same CDF, can be constant.
Datasets:
Coding losses: Two of the three coding losses is the square error loss and the downward-convex loss described in Section 4.1. The third coding loss is upward-convex loss which we design as Eq. 66 such that the scale factor aæ becomes the reciprocal of the scale factor in Eq. 24:
D(x, x) = az x - 212 where 0g = (2/3 + 2 || 221) and G Ggz I
(66)
Figure 10 shows the scale factors in Eqs. 24 and 66, where 81 in (,1,0,0) moves within +5.
Parameters: As explained in Appendix D.1, A = 1/8 is used as hyper parameter. Specifically, A 1, 5, 10, 20, 50, 100, 200, 500, and 1, 0,00 are used.
Figures 11 - 19 show the property measurements for all combinations of the datasets and coding losses, with changing X. In each Figure, the estimated norms of the implicit transform are shown in the figure (a), the ratios of the estimated variances are shown in the figure (b), and the correlation coefficients between p(x) and estimated data probabilities are shown in the figure (c), respectively.
First, the estimated norm of the implicit transform in the figures (a) is discussed. In all conditions, the norms are close to 1 as described in Eq. 20 the À range 50 to 1000. These results show consistency with our theoretical analysis. supporting the existence of the implicit orthonormal transform. The values in the Norm dataset are the closest to 1, and those in the Ramp dataset are the most different, which seems consistent with the difficulty of the fitting.
21

---

Under review as a conference paper at ICLR 2021
downward-conves loss un upward-conves loss
d
2V3/3
PDF
p(s)): E[s 1-1
v3/3
p(s)): E[s.]=0, B[3
v3/6
p(y): E[s.J H(s
5
4v333 -2v3/3 -3333
2v3/3
4w3/3
8v3/3
4
Figure 9: PDFs of three variables to generate a Ramp dataset.
Figure 10: Scale factor dg for the downward-convex loss and upward- convex loss.
Second, the ratio of the estimated variances is discussed. In the figures (b), Var(2;) denotes the estimated variance, given by the average of -2 j(e) Then, Var(z2)/ Var(z1) ((rr2))/ar((z) and Var(za)/ plotted. In all conditions, the ratios of Var(22)/Varr /ar((2) and (arrg)//Va(()))) Var(za)/Var(z)) /Va((=)) close the variance ratios of the input variables, i..., 4 and 16, in the À range to 500. Figure 20 shows the detailed comparison of the ratio for the three datasets and three coding losses = 100. In most cases, the estimated variances in the downward-convex loss are the smallest, and those in the upward-convex loss are the largest, which is more distinct for Var(za)/ (V)r) This can be explained as follows. When using the downward-convex loss, the space region with large norm thought of shrinking in the inner product space, as described in Section 4.1. This will make the variance smaller. In contrast, when using the upward-convex loss, the space region with large norm is thought of as expanding in the inner product space, making the variance larger. Here, the dependency of the losses on the ratio changes is less in the Norm dataset. The possible reason is that data the normal distribution concentrate around the center, having less effect on the loss scale factor in the downward-come
loss and upward-conves loss.
Third, the correlation coefficients between p(x) and the estimated data probabilities in the figures (c) are discussed. In the Mix dataset and Ramp dataset, the correlation coefficients are around 0.9 in the À range from 20 to 200 when the estimated probabilities az (((((( II;=1 ((m) and az n/2 exp(-(1/3)La) in Eq. 23 are used. When using (()())) I1 [(x) and exp(-(1/3) in the downward-conve loss and upward-convex loss, the correlation coefficients become worse. In addition, when using the prior probability p(p(a)),, the correlation coefficients always show the worst. In the Norm dataset, the correlation coefficients are close to 1.0 the wider range of X when using the estimated distribution in Eq. 23. When using (()) IT ()(x)) and exp( the downward-conee
in loss and upward-conver loss, the correlation coefficients also become worse. When using the prior probability p=)) however, the correlation coefficients are close to I in contrast to the other two datasets. This can be explained because both the input distribution and the prior distribution are the same normal distribution, allowing the posterior variances almost constant. These results also show consistency with our theoretical analysis.
shows the dependency of the coding loss on 33 for the Mix, Ramp, and Norm dataset using square the error loss. From DG in Eq. 14 and n = 3, the theoretical value coding loss is 33 also shown in the figure. Unlike Figs. 11-19, r-axis is 3 = A to evaluate the linearity. As expected in section 3.3, the coding losses are close the theoretical value where / < 0.1, i.e., À 10.
Figure 21
2 as
Figure/ 22 shows the dependency of the ratio of transform loss to coding loss on , for the Mix, Ramp, and Norm dataset using square the error loss. From Eq. 54, the estimated transform loss is Σ3 (B/(2/)) /Var(v)) = 32
213 16 as is also shown in the figure. r-axis is also 8 = - like Figure 21. Considering the correlation coefficient discussed above, the useful range of (3 seems between 0.005-0.05 (20-200 for A). In this range, the ratio is less than 0.1, implying the transform loss is almost negligible. As expected in section 3.2 and appendix B. the ratio close to the theoretical value where 3 0.01, i.c. X 100. For < 0.01, the conversion loss is still negligibly small, but the ratio is somewhat off the theoretical value. The reason is presumably that the transform loss is too small to fit the network.
63,6 Thus the theoretical value is 6339 )/(3))
32
As shown above, this ablation study strongly supports our theoretical analysis in sections 3.
22

---

Under review as a conference paper at ICLR 2021
PPP( p(o)
G/ s(a) esp(-aa
Vant22Wantal))
VartaVVarte))
Ratio of d -2 Average
20.0
1.0
1.0
15.0
Cogrelation
(2/08)903) (D)
10.0
5.0
0.0
0.0
0.0
10
100
1000
kmf "logarithmicc scale)
A-ff" (logarithmic scale)
XmB (logurithmic scale)
(a) Estimated norm 3 (z). (b) Ratio the estimated variances Var(za)/ Var(zs)/Var(zz Var(z1) and
(c) Correlation coefficient the estimated data probability
Var(za)/Var(z1)
Figure 11: Property measurements of the Mix dataset using the square error loss, À is changed from 1 to 1, 000. Var(zz) denotes the estimated variance, given by the average of -2 ((x)"
>21.Ar. SD
>2.x> x2SD
23.Av. x3
+ P(jl G-(0 00 Sespi-LL)
Grye expt-ls)
Van(z2VVarizl) VartzVVarttz))
Ratio of 3jo -2 Average
(2/0)) 50 (0)
1.0
Cogrelation
0.5
0.0
3
10
100
1000
10
100
1000
10
100
1000
X-p² (logarilumic scale)
X-p" (logarittanic scale)
2= (logarithmic scale)
Estimated norm (( 2 (=). (b) Ratio of the estimated variances Var(zs)/ Var(zs)/Var(z.) /Var(=)) and
(c) Correlation coefficient of the estimated data probability
Var(z)))Var((1)
Figure 12: Property measurements of the Mix dataset using the downward-convex loss, À is changed from to 1, .000. Var(z,) denotes the estimated variance, given by the average of o j(m)
+.22.As
+ 23Ar.. 2535
-+-((s]
+ p(i)
G/10 Sie p() esp(-L-s
20.0
VanieyVVaial)
Varie//Vrttal)
88 G*(0
-0000.55pppp Skap(-Lss)
dica) -2 Average
1.0
15.0
Cogrelation
(2/ FD)
10.0
0.5
5.0
Ratio
0.0
10
100
1000
10
100
1000
10
100
1000
A-D' (logarithuic scale)
A=p" (logarittini scale)
>=B (logurithmic scale)
(a) Estimated norm 3 (z). (b) Ratio of the estimated variances Var(za)/ Var(za)/Var(z:) Var(z:) ) and
(c) Correlation coefficient the estimated data probability
Var(zz)/Var(aa)
Figure 13: Property measurements of the Mix dataset using the upward-convex loss. À is changed -2
from to 000. Var(z;) denotes the estimated variance, given by the average of o ((x)
23

---

Under review as a conference paper at ICLR 2021
PPP( p(o)
G/ s(A) osp(-aa
Ratio dite -2 Average
20.0
+ Vanti2Vaatza))
Varta)Varte)
Sespp-a)
1.0
1.0
15.0
Cogrelation
(2/08)900) (D)
10.0
5.0
0.0
0.0
0.0
10
100
1
10
100
1000
kmf "logarithmic scale)
2- (logarithmic scale)
XmB (logurithmic scale)
(a) Estimated norm ) (z). (b) Ratio the estimated variances Var(za) and
(c) Correlation coefficient the estimated data probability
Var(za)/Var(z1)
Figure 14: Property measurements of the Ramp dataset using the square error loss. changed from 1 to 1, 000. Var(z;) denotes the estimated variance, given by the average of -2
j(x)
-0-1:AAY. SDD
-0-22:Av =#-225D
+--2AA ==235D
+
Grye exp(-la)
Vani2/Variz))
VartzWVartzi)
G-(0 0010.5)eppplaa Sespl-L))
of 300 -2 AAerage
(2/0))060) 5D2 (
1.0
Cogrelation
0.5
5.0
0.0
10
100
1000
10
100
1000
10
100
1000
X-p" (Jogarithmic scale)
X-8 (logarittmic scale)
2=131 (logarithmic scale)
Estimated norm faa (=). (b) Ratio of the estimated variances Var(zs)/ /Var(=1) and
(c) Correlation coefficient of the estimated data probability
(ar(z))/Var(zz) (ar(z2)/Var(a)
Figure 15: Property measurements of the Ramp dataset using the downward-come -onvexx loss, A is changed from 1 to 1, 000. Var(z;) denotes the estimated variance, given by the average of j(m)
+22.Av.
+-22.Ax.
# 2335
+ 23A..
VanieyVanial
Varta/V
-o-Pis]
-0-G7
p(u)
G/10.5 p(a)
expl-pp)
20.0
Ratio of diaa) -2 Average
(2/ FD) ④
1.0
15.0
Cogrelation
10.0
0.5
5.0
0.0
3
10
100
1000
10
100
1000
1
10
100
1000
X-D' (logrrime scale)
A=P" dlogarittins scale)
X-B" (logurithmic scale)
(a) Estimated norm ))(()) 2 (z). (b) Ratio of the estimated variances Var(za)/ Var(za)/Var(z:) Var(z:) ) and
(c) Correlation coefficient of the estimated data probability
Var(zz)/Var((a)
Figure 16: Property measurements of the Ramp dataset using the upward-convex loss. À is changed -2
from to 000.. Var(z;) denotes the estimated variance, given by the average of or ((x)
24

---

Under review as a conference paper at ICLR 2021
PPP(
G/18.5 s(a) osp(-as
p(u)
Ratio of d -2 Average
20.0
Vant22Wantal))
Varte)Vartee))
Seap(-a)
1.0
1.0
15.0
Cogrelation
(2/08)900) (D)
10.0
5.0
0.0
0.0
0.0
10
100
1000
kmf (logarithmic scale)
A-ff" (logarithmic scale)
(logurithmic scale)
(a) Estimated norm ) Dz).). (b) Ratio the estimated variances Var(za)/ Var(z))/Var((z) and
(c) Correlation coefficiint the estimated data probability
Var(za)/Var(z1)
Figure 17: Property measurements the Norm dataset using the square error loss. changed from 1 to 1, 000. Var(z;) denotes the estimated variance, given by the average of -2
j(z)
>21.Av. SD
~2Ax> x25D
23.Av. x3:DD
+
Grye expt-la)
Van(z2VVarizl) VartzVVarttz))
Gr10. 0%00. Seapp-Lxx
Ratio of 3j0 -2 Average
(2/1)) 50 (6)
1.0
Cogrelation
0.5
0.0
3
10
100
1000
10
100
1000
10
100
1000
X-p² (logarilunic scale)
X-p" (logarittanic scale)
2=131 (logarithmic scale)
Estimated norm a 2 (=). (b) Ratio of the estimated variances Var(zs)/ Var(za)/Var(z)) /Var(=)) and
(c) Correlation coefficient of the estimated data probability
Vaa(z2))Var((1)
Figure 18: Property measurements of the Norm dataset using the downward-come loss, A is changed from 1 to 1, 000. Var(z;) denotes the estimated variance, given by the average of -2 j(m)
+-22.Ax. 22225
2535D
23A..
20.0
Vani2yVaria)
Varte\V
-+P(js]
+ p(i)
= G*(0.
GG00. Stap(-Lss)
G/y0 Sie p() esp(-L.s)
Ratio of dica) -2 Average
(x) FD) ④
1.0
15.0
Cogrelation
Cotbs
(2/00)
10.0
0.5
5.0
0.0
3
10
100
1000
10
100
1000
10
100
1000
A=D 'logariiimeie scale)
A=p dlogarittans scale)
>=B (logurithmic scale)
(a) Estimated norm (0)()) (z). (b) Ratio of the estimated variances Var(za)/ Var(za)/Var(zz) Var(z:) ) and
(c) Correlation coefficient of the estimated data probability
Var(zz)/Var(a))
Figure 19: Property measurements of the Mix dataset using the upward-convex loss. À is changed -2
from to 000. Var(z;) denotes the estimated variance, given by the average of o [(x)
25

---

Under review as a conference paper at ICLR 2021
4.
-IX
MIX
18,5
+ Ramp
Var(zz)) Vartza)
4.4
Var(z.) / Var(z,)
17.5
16.3
Downward
Square
Upwand Convex
Downward
Square Froo
Upward
Comex
Convex
Convex
Reconstruction Loss
Reconstruction Loss
(a) Var(z))/Var(zi)
(b) (ar(za)/Var(z1))
Figure 20: Ratio of the estimated variances Var(z3)/Var(21) and Var(=2) /Var(()))))) for the three datasets and three coding losses at À = 100. Var(z,) denotes the estimated variance, given by the average of ((x)
Theoretical estimation
Transfom Loss /odding Loss
Theoretical estimation
Míx,
Mix,
Coding Loss
Norm, MSE
Norm, MSE
Ramp, MSE
Ramp, MSE
0.1
0.1
0.01
0.001
0.001
0.01
B-
0.1
0.001
0.01
B=X-1
0.1
Figure 21: for Mix, Norm, and Ramp dataset using square loss.
Dependency of Coding Loss
Figure 22: Dependency of Transform loss / Coding Loss Ratio on 3 for Mix, Norm, and Ramp dataset using square loss.
26

---

Under review as a conference paper at ICLR 2021
F ADDITIONAL RESULTS IN CELEBA DATASET
F.I TRAVERSED OUTPUTS FOR ALL THE COMPONENT IN THE EXPERIMENTAL SECTION 4.2
2
21
z1 388.7
21 579.9
22
214.4
z2
459.6
23 176.1
x3
389.2
24 118.7
211.7
R
25
106.2
142.1
26
92.8
z6
140.3
z7
46.9
27
76.8
28
43,9
28
68.2
29
35.0
67.2
32.9
z1O
63.7
32.0
zl1
57.9
z12
28.7
X12
57.5
z13 27.0
Z13
47.6
z14 24.2
z14
39.6
215 18.8
x15
36.3
14.2
Z16
31,9
z17 13.2
Z17
28.5
218
13.1
25.2
219
11.6
Z19
24.8
z20
11.4
Z20
24,4
221
10.7
Z21
23.9
z22
10.3
Z22
21.1
223
10.2
Z23
18.8
x24
8.6
x24
18.2
R
225
6.2
225
14.8
z26
5.8
226
14.7
x27
3.0
x27
13.7
1.0
Z28
12.7
z29
1.0
Z29
11.9
z30
1.0
230
10.3
231
1.0
Z31
10.3
z32
1.0
x32
9.1
(a) Trained using the conventional loss form. (b) Trained using the decomposed loss form.
Figure 23: Traversed outputs for all the component, changing 2j from -2 to 2. The latent variables Zj are numbered in descending order by the estimated variance o,² shown in Figures 4 and 5.
27

---

Under review as a conference paper at ICLR 2021
Figure 23 shows decoder outputs for all the components, where each latent variable is traversed from -2 to 2. The estimated variance of each μ/, i.c., o,² 2 also shown in these figures. The latent variables 2; are numbered in descending order by the estimated variances. Figure 23a is result using the conventional loss form, i.e., L = ( æ) + (-) The degrees of change seem descend in accordance with the estimated variances. In the range where j is 1 from 10, the degrees of changes are large. In the range j > 10, the degrees of changes becomes gradually smaller. Furthermore, almost change is observed in the range
27. As shown in Figure 4, D (-) is close to zero for j > 27, meaning no information. Thus, this result is clearly consistent with our theoretical analysis in Section
Figure 23b is a result using the decomposed loss form, i.e., La D(x, Ø) + D(ž, a) + 3DL The degrees of change also seem to descend in accordance with the estimated variances. When look- ing at the detail, there are still minor changes even = 32. As shown in Figure 5, KL divergences (-) for all the components are larger than zero. This implies all of the dimensional compo- nents have meaningful information. Therefore, we can see minor change even j = 32. Thus, this result also consistent with our theoretical analysis.
Another minor difference is sharpness. Although the quantitative comparison is difficult, the de- coded images in Figure 23b seems somewhat sharper than those Figure 23a. A possible reason for this minor difference is as follows. The transform loss D(x, .) serves to bring the decoded image (() closer to the input. In the conventional image coding, the orthonormal transform and its inverse transform used for encoding and decoding, respectively. Therefore, the input and the decoded output are equivalent when not using quantization. If not so, the quality of the decoded image will suffer from the degradation. Considering this analogy, the use of decomposed loss might improve the decoded images for ((x)), encouraging the improvement of the orthonormality of the encoder/decoder in VAE.
F.2 ADDITIONAL EXPERIMENTAL RESULT WITH OTHER CONDITION
In this Section, we provide the experimental results with other condition. We use essentially the same condition as described in Appendix D.2, except for the following conditions. The bottleneck size and À are set to 256 and 10000, respectively. The encoder network is composed of CNN(9, 9, 2, 64, GDN) CNN(5, 5, 2, 64, GDN) CNN(5, 5, 2, 64, GDN) CNN(5. 5, 2, 64, GDN) FC(1024, 2048, softplus)- - FC(2048, 256, 2 (for and σr) in encoder. The decoder network is composed of FC(256, 2048, softplus) FC(2048, 1024, softplus) CNN(5, 5, 2, 64, IGDN) CNN(5, 5, 2, 64, IGDN) CNN(5, 5, 2. 64, IGDN)-CNNO, 9, 2, 3, IGDN).
Figures 24a and 24b show the averages of j)(m)) as well as the average and the standard deviation 3 2)'(z) in the conventional loss form and the decomposed loss form, respectively. When using the conventional loss form, the mean of 2 )(m)) 2 D' is 1.25, which closer to than the
of 2
Variance
Variance
Nom:: Average
Norm; Average
Norm: Standard Deviation
Norm: Standurd Devistion
1E+3
Meun:
SD: /.25/SD: 0.12
IE-3
Mean: 0.95 SD: 0.10
Average of djoo
Average of djos
1.5
1B+2
1B-2
1.0
LE+1
IE+1
18+0
0.0
18+0
0.0
100
Number of ofllatent variables (descending order of Estimated Variance)
Number variables
(descending order of Estimated Variance)
(a) Conventional loss form
(b) Decomposed loss form
Figure 24: Graph of (a) average and 30 (z)(2) ) in CelebA dataset. The bottleneck size and À are set 256 and 10000, respectively.
28

---

Under review as a conference paper at ICLR 2021
mean 1.83 in Section 4.2. This suggests that the implicit transform is closer to the orthonormal. The possible reason that a bigger reconstruction error is likely to cause the interference to RD-trade off and slight violation of the theory. and it might be compensated with larger lambda. When using the decomposed loss form, the mean 2 These results also support that VAE provides the implicit orthonormal transform even if the lambda or bottleneck size is varied.
3 )0 2 (2) ) is 0.95, meaning almost unit norm.
G ADDITIONAL EXPERIMENTAL RESULT WITH MNIST DATASET
In this Appendix, we provide the experimental result of Section 4.2 with MNIST dataset' consists of binary hand-written digits with a dimension of 768(=28 x 28). We use standard training split which includes 50,000 data points. For the reconstruction loss, we use the binary cross entropy loss (BCE) for the Bernoulli distribution. We averaged BCE by the number of pixels.
The encoder network is composed of FC(768, 1024, relu) FC(1024, 1024, relu) FC(1024, bot- tleneck size) in encoder. The decoder network is composed of FC(bottleneck size, 1024, relu) FC(1024, 1024, relu)- FC(1024, 768, sigmoid). The batch size 256 and the training iteration number is 50,000. In this section, results with two parameters, (bottleneck size=32, x=2000) and (bottleneck size=64, x=10000) are provided. Note that since we averaged BCE loss by the number of pixels, 3 in the conventional / VAE is derived by 768/ X. Then, the model is optimized by Adam optimizer with the learning rate of le-3, using the conventional (not decomposed) loss form.
We use PC with CPU Intel(R) Cor)(TM) i7-6850K CPU @ 3.60GHz, 12GB memory equipped with NVIDIA GeForce GTX 1080. The simulation time for each trial is about 10 minutes, including the statistics evaluation codes.
Figure 25 shows the averages of (((x) 2 as well as the average and the standard deviation of (m)( (m) In both conditions, the means of 3 (0)) "D'(2) averages are also close to except in the dimensions where (zx) -2 is less than 10. These results suggest the theoretical property still holds when using the BCE loss. the dimensions where (()()) -2 is less than 10, the )()) (z) is somewhat lower than 1. The possible reason is that DKLO (-) in such dimension is 0 for some inputs and is larger than in other inputs. The understanding of the transition region needs further
study.
Variance
Variance
Nonm: Average
Average
Nonm: Standard Deviation
Nome: Standard Deviation
1843
21
Mcan: 0.93 SD: 0.05
IE:3
21 21-252 Mean: 0.95 SD: 0.07
Average of doo0
1.0
1 1 (9/
16+2
Average of
0.5
(2) (2)
IE+1
IE+l
2
1E+0
30
40
60
Number of latent variables
Number of latent variables
(descending order Variance)
(descending order of Estimated Variance)
(a) bottle neck=32, 2=2000
(b) bottle neck=64. A=10000
Figure 25: Graph of d)(w) 22 average and f0, 2 (2) in MNIST dataset.
hhtp://yam..eeuu.ccm/edddmmmi
29

---

Under review as a conference paper at ICLR 2021
H DERIVATIONIEXPLAAATIIO IN RDO-RELATED EQUATION EXPANSIONS
H. APPROXIMATION OF DISTORTION IN UNIFORM QUANTIZATION
Let T be quantization step. Quantized values Ej is derived l: T, where k: round(2;/T) Then d, the distortion per channel, is approximated by
1/2// p(2y)(z) - T)2 dzj
E
1/2yT
(k+1/2)/
E T P(k T)
7 k 17² dzj
(k-1/2)T
T
Σ T p)
72
(67)
12"
Here, Ex T' p( T) ~
I (zz))dz = 1 is used. The distortion for the given quantized value is T)2 dzz.
also estimated as T /1,, because this value is approximated by (
1(k=1/22y +
H.2 APPROXIMATION OF RECONSTRUCTION LOSS AS A QUADRATIC FORM.
In this appendix, the approximations of the reconstruction losses as a quadratic form '8x G. Ca are explained for the sum of square error (SSE), binary cross entropy (BCE) and Structural Similarity (SSIM). Here, we have borrowed the derivation of BCE and SSIM from Kato et al. (2020), and add some explanation and clarification to them for convenience. We also describe the log- logllikel of the Gaussian distribution.
Let a and 24 be decoded sample Deco(z) and its i-th dimensional component respectively. 5æ and óx; denote æ and x; e. respectively. It also assumed that dæ and 8x; are infinitesimal. The details of the approximations are described as follows.
Sum square error:
In the case of sum square error, G. is equal to Im. This can be derived as:
m
m
(a 2,)2
Σ dn 5n..
(68)
Binary cross entropy:
Binary cross entropy a log likelihood of the Bernoulli distribution. The Bernoulli distribution is described as:
II 2 (1 L,) (1-,)
(69)
1=1
Then, the binary cross-entropy (BCE) can be expanded as:
m
-log
log
II
(1 (1
x, log Lx (1 - =2, log (1 f.))
> log [ + dxx,) - (1 xz) log(11 - - 8x;))
Σ -x;; log 1 8z: (1 xz)logg (1- 6x;
1 23:
+ Σ (-xy log(az) - (1 - 24) log(1 - -x))).
(70)
30

---

Under review as a conference paper at ICLR 2021
(1/2X 1/(1=x))
6
4
0.25
0.5
0.75
Figure 26: Graph of 1 2 (= + 1-x in the BCE approximation.
Here, the second term of the last equation is constant C. depending Using log(1 x)
- /2 + (xx)), the first term of the last equation is further expanded as follows:
5x,
5x,
5x,
Σ
(1 2z)
O (dx:²)
2x;²
1 32
2(1 - xx22
1
+
óz, 2 + 0 (dx; :)
(71)
As result, metric tensor Ge can be approximated the following positive definite Hermitian matrix:
G
F2
F2
(72)
(1 + I-m1
Here, the loss function in each dimension in Figure 26.
is a downward-convex function as shown
2
Structural similarity (SSIM):
Structural similarity (SSIM) (Wang al., 2001) is widely used for picture quality metric, which is close to human subjective quality. Let SSIM be a SSIM value between two pictures. The range of the SSIM value is between and The higher the value, the better the quality. In this appendix, we also show that (1 SSIM) can be approximated to quadratic form such as '5æ Gz
SSIM ((...)) (x. y) denotes a SSIM value between N N windows in pictures and Y. where E RN2 and y N denote N x N pixels cropped from the top-left coordinate (h, v) in the images X and Y. respectively. Let Pæs Hy be the averages of all dimensional components in x, y, and dx By be the variances of all dimensional components in æ, y in the N x N windows, respectively. Then, SSIM ((h, (xx y) is derived as
(2) = 24aay Me 2 fly ox²
20
(73)
ooy
In order to calculate SSIM value for picture, the window is shifted in whole picture and all of SSIM values are averaged. Therefore, if (1 - SSTM x (h..))))) (x., v)) is expressed as a quadratic form 5ax (h dx. (1 SSIM) can be also expressed in quadratic form f8x Gzöz.
Let da be a minute displacement of æ. f48m and ar denote an average and variance of all dimen- sional components in dæ. respectively. Then, SSIM between and æ dæ can be approximated as:
Tix 2
SSTM N hx:)221 m1
+ +00
((1621//111)))
(74)
2p,² 2
20
31

---

Under review as a conference paper at ICLR 2021
Then Mew 2 and Jee can be expressed
11::
1
i
1 N2
'6x Möx, where M
(75)
and
Ozæ = 'dæ Vóx, where V M,
(76)
respectively. As result, (1 N(.. ōx:) be expressed in the following quadratic form as:
1
1 SSIMNxN(N
6x) ~ 'ox 6Z where G (h,n)m
M
V
2p122 20
(77)
It is noted that M is a positive definite Hermitian matrix and V is positive semidefinite Hermitian matrix. Therefore, G(h,r) a positive definite Hermitian matrix. As a result, (1 SSIM) can be also expressed in quadratic form 'ōx oz where Gz is positive definite Hermitian matrix.
bog-likelihood of Gaussian distribution:
Gaussian distribution is described as:
Pa (x]22) =
(2. 223/202
5n 3 /20"
(78)
2
2
1=1
where 2² is a variance as a hyper parameter. Then, the log-liielihood of the Gaussian distribution is denoted as:
log (arl2)) log II
5x
1
8a
log (og(2To²).
(79)
2 1=1
2x
The first term can be rewritten as (1/2o²) I òx. Thus, Ge = (1/20²) ) Im holds. Cz is derived as the second term the last equation in Eq.78
H.3 DETAILED EXPLANATION OF KL DIVERGENCE AS RATE OF ENTROPY CODING.
This appendix explains the detail how KL divergence can be interpreted as a rate in the transform coding. the transform coding, input data transformed by an orthonormal transform. Then, the transformed data is quantized, and an entropy code is assigned to the quantized symbol, such that the length the entropy code is equivalent to the logarithm the estimated symbol probability.
It is generally intractable to derive the rate and distortion of individual symbols in the ideal infor- mation coding. Thus, we first discuss the case of uniform quantization. Let Pzy and Ray be the probability and rate in the uniform quantization coding of 2j ~ (2z; 0, 1). Here, Mi(æ)) and )a())) regarded quantized value and coding noise after the uniform quantization, respectively. Let be a quantization step size. The coding noise after quantization is T /12 for the quantization step size T, as explained in Appendix H.I. Thus, T is derived as T = 2v/3a ((x) from d)(=) 2 = T /12. (
We also assume (j(=)* 2 < 1. As shown in Fig.27a, Psy is denoted by J P(zz)dz, where p(2;) (22; 0, 1). Using Simpson's numerical integration method and ex = + r + 0 expansion, Psg approximated as:
T
Pe.
6 (p(M5(0) - [T) 4p(()(()) + p( + +))
Tylus(0))
4+c 4nslee T2
+ 4n 72
~ Tp (Uj(²)) (1 (1- 12-12/211
= )()) e (Pstaa) 2/2 (1
Tj(m))
(80)
2
32

---

Under review as a conference paper at ICLR 2021
T Hiit) 1 zz
T = 2=233ojxx
Paj
11000 dzz)
Pej T .((H)())
T T=2130j(
((z) - NN0,1
p(zi) N(0.1)
/T 1100)) Mj(0) +
T
TT ((() 2 1(0)) 11(00 +
T Zi
M)(x)
(a) Probability Paj
(b) Approximation of P.
Figure 27: Probability for symbol with mean μ and noise o²
Using log(1 + 2) = x + 0(x2) expansion, Ryw is derived as:
log Ps, 2 MM((x) 2 + dj(x) - log )(() 2 - log = ))= = DRIj(a) (+) + log πе Rs, = log
6
(81)
π
2
6
When Rz, and (xI.j(a)))) (.) in Eq. 2 are compared, both equations are equivalent except small constant difference & log(Te/6) = 0.176 for each dimension. As result, KL divergence for j-th dimension is equivalent the rate for the uniform quantization coding, allowing a small constant difference.
To make theoretical analysis easier, we use the simpler upproximation as Ps, = T p(p)(t))
=
2 2v30 p(U)(=)) instead of 80, as shown in Fig. Then, Ray is derived as:
Ra; - 1og(2 ((a)) p(11(()))) Eq. 6 + log me
1
(82)
2
This equation also means that the approximation of KL divergence in Eq. 6 equivalent to the rate in the uniform quantization coding with P., 2v30 j(x) P(Hj(a)) approximation, allowing the same small constant difference as in Eq. 81. It is noted that the approximation Ps, 2v/30 (()) in Figure 27b can be applied to any kinds of prior PDFs because there is no explicit assumption for the prior PDF. This implies that the theoretical discussion after Eq. 6 in the main text will hold in arbitrary prior PDFs.
Finally, the meaning of the small constant difference log in Eqs. 81 and 82 is shown. Pearlman & Said (2011) explains that the difference of the rate between the ideal information coding and log 6 This caused by the entropy difference of the noise distributions. In the ideal case, the m distribution is known as a Gaussian. In the case the noise variance is o², the entropy of the Gaussian noise is 1 log(o"2ne). For the uniform quantization with uniform noise distribution, the entropy is Ioggo>22). As result, the difference is just log ze. Because 6 xe can be the rate estimation in this appendix uses a uniform quantization, the small offset & log B regarded as a difference between the ideal information coding and the uniform quantization. As result, KL divergence in Eq. 2 and Eq. 6 can be regarded as a rate in the ideal informaton coding for the symbol with the mean (z( and variance (())
uniform quantization
33


Reviews:
[{"review_id": "l7bPgpb_95t", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper proposes a methodology to understand quantitively the VAE model, based on the Rate-distortion theory.\n\nComments: I think that the writing of the paper is rather confusing. Unfortunately, I am not able to judge the proposed idea (which might be interesting) and provide a reasonable review, because I could not understand it after trying to read the paper couple of times. In general, I find the clarity and coherence of the paper lacking. In my opinion, the paper should be improved significantly along these lines, such that to be accessible from the reader. \n​\nSome examples of the problems I think that should be fixed:\n​\n- The text is super \"wordy\". There is too much compact information, but with low clarity. In general, I could not understand the main story of the paper.\n- Many terms are used without a definition e.g. what is the definition of isometric embedding? what is the definition of rate-distortion?\n- In general, the mathematical notation that is used is a bit unclear.\n- There are many places where the authors say: \"We (will) show later ...\". I believe that this causes a confusion. I think is better to avoid this tactic and directly show the result in place.\n- How the VAE model is perceived? In particular, I cannot understand what it means to \"map the VAE\"?\n- In my opinion the Figure 1 is hard to understand.\n- There is a lot of context and connections to related works, which the authors presume is already known from the reader. However, I think that this is not the case for a general reader.\n​\nOverall: Unfortunately, I am not able to provide a constructive feedback. The main idea of the paper might be interesting, but the writing style does not help. Therefore, I believe that the paper needs to be improved significantly, such that to be accessible.\n\n===== After rebuttal =====\n\nI appreciate the fact that the authors update their manuscript, taking into account some of our comments (+1 for this). However, I still feel that the content is quite hard to follow. In general, I believe that the motivation is still unclear and the clarity low. Even Fig. 1 (that potentially should help the understanding) is so complicated without any explanation in the caption. Maybe the technical content is interesting, but I think that the paper is not accessible.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kWhh1Rjs7XJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Based mainly on the rate-distortion theory, the authors propose to interpret VAE as a non-linearly scaled isometric embedding. That interpretation brings several advantages, for example, enabling the estimation of the data probabilities in the input space. \n\nOverall, the paper is well written. However, I find it challenging to fully understand the main contributions of the paper, even though I use VAE a lot.\n\nThe current manuscript might not be self-contained. It relies heavily on the prior work RaDOGAGA (Kato et al., 2020), with which I am not familiar. I would suggest the authors elaborate on that. \n\nThe main interpretation/discussions are proposed for the classical VAE with a (conditional) isometric Gaussian variational arm, whose model capacity may be limited in practice. So, is it possible that the proposed interpretation/discussions also generalize to other VAE variants (like one exploiting a flow on top of the variational arm)?\n\nIn Section 3.2, many approximations are employed; what’s the influence of those approximations? Discussions are necessary.\n\nIn Eq (8) as well as several other equations, implicit assumptions have been made, that is, the Jacobian determinant is always positive; however, those assumptions might not be true in practice. How to address related concerns in practice?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YRf2Q_BrG-T", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyses the theoretical properties of the VAE, showing that under some assumptions the VAE can be interpreted as an implicit isometric embedding of the data manifold. The authors show that with this analogy one can make inferences about the data distribution and the latent dimensions from the components of the VAE.\n\nIt seems to me on the whole that the theoretical analysis and contribution is correct and solid, and the work does give a valuable insight into the theoretical properties of the VAE as interpreted as a isometric embedding of the data space into the latent space. The empirical results given demonstrate that the quantitative measures the authors describe give a way to understand the importance of the latents and the properties of the learned approximate data distribution. These are things that I believe the community would find useful.\n\nMy main negative impression of this paper is that it is hard to read. The paper is written in a way which is hard to understand. It is information-dense, with a lack of intuitive explanation for many of the statements. For example, the introduction of the rate-distortion theory is not motivated at all. Clearly, the reason it has been introduced is to draw the parallel to lossy data compression via mapping to a low dimensional latent space. This parallel allows one to use knowledge from the compression literature to aid our understanding of these autoencoder-like models. However, this is not mentioned anywhere in the text, which makes it difficult for the reader to follow the narrative of the paper. Note that the rate-distortion is just one example. I will say that I am not very familiar with the literature on isometric embeddings, which has made it harder for me to understand.\n\nOne other negative point I would raise is that the paper is similar in nature to the RaDOGAGA work by Kato et al. Their method also establishes that a VAE-like architecture can isometrically embed the data into the latent space, which allows them to draw strong inferences about the data distribution, given the isometric properties. At a high level, this is very similar to the contribution of this paper. There are certainly differences, primarily in the theoretical analysis given to the VAE here, and the focus on examining the contributions of each latent dimension through the variances, in a PCA-like fashion. Still, I think perhaps the novelty is lacking.\n\nOverall, I think the paper should be rejected, but I do not feel strongly given that I am not an expert in the subject matter that much of the paper pertains to. If the differences between this work and RaDOGAGA were made, and the paper itself was made more clear then I would be willing to upgrade my score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NRYiF9N7JT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper builds on a branch of recent works that consider and analyse Variational autoencoders (VAE) from the view point of data compression. This started from Alemi et al. 2018, where the authors consider and analyse the mutual information between data and latent codes and culminates in Kato et al. 2020, where the authors consider models in which both the encoder and the decoder are assumed as deterministic (isometric) mappings. The submitted paper aims at reconciling this type of models with standard VAEs by claiming to prove that VAEs can be obtained from the former by an non-linear component-wise scaling of the latent space. The authors claim among others that this approach allows to estimate the data probability density from the learned VAE model.\n\nUnfortunately, the paper is not well written. This begins with sloppy wording and imprecise notations. Moreover the authors do not explain precisely enough the concepts used in the main building blocks of their proof and their final composition. Altogether, this makes it impossible (at least for me) to follow the steps of the construction and the related proof. In my opinion, the paper in its present state is not publishable at ICLR because it requires a major revision.\n\nI have tried to read the revised version submitted by the authors. Unfortunately, it is still very hard for me to fully grasp the proposed concept and to follow the derivation steps and the proofs. In my opinion, the paper is not publishable in its present state.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission analyses the VAE objective from the perspective of non-linearly scaled isometric embeddings, with the aim of improving our information-theoretic understanding of the variational objective. 

Reviewers are in consensus that this submission in its current form is very difficult to read, even after revisions by the authors. The metareviewer, who is highly familiar with information-theoretic and even information-geometric interpretations of VAEs, similarly struggled to understand this paper. Many concepts (e.g. KLT transforms) are not introduced in a self-contained manner, nor are related works like RaDOGAGA. Moreover the exposition introduces lots of notation (often somewhat implicitly) and requires more  high-level plain-English statements that signal the structure of the overall narrative to the reader. As a result, it is hard to understand the paper, even at the level of the contributions that are claimed by the authors. It appears that Section 3.4 should be read as culminating in a "correction" of the rate-distortion view of VAEs proposed by Alemi et al. Unfortunately the metareviewer is not able to understand from the writing what fault the authors find with the proposed interpretation, and how their view informs a better perspective.

It is difficult to provide the authors with concrete addressable suggestions at this stage of revision of their manuscript. The metareviewer's advice would be to attempt to focus on defining a narrative structure that clearly explains what insights this perspective of VAEs contributes, what misconception it corrects, and how it corrects it –– and then focus on streamlining notation in a manner that makes it possible to follow along with the exposition more easily. 
Decision: Reject