RATE-DISTORTION OPTIMIZATION GUIDED AUTOENCODER FOR GENERATIVE APPROACH
Authors: Keizo Kato
Jing Zhou
Akira Nakagawa
Venue:  (2020)
Keywords: Autoencoder
Rate-distortion optimization
Generative model
Unsupervised learning
Jacobian

Abstract: In the generative model approach of machine learning, it is essential to acquire an accurate probabilistic model and compress the dimension of data for easy treatment. However, in the conventional deep-autoencoder based generative model such as VAE, the probability of the real space cannot be obtained correctly from that of in the latent space, because the scaling between both spaces is not controlled. This has also been an obstacle to quantifying the impact of the variation of latent variables on data. In this paper, we propose a method to learn parametric probability distribution and autoencoder simultaneously based on Rate-Distortion Optimization to support scaling control. It is proved theoretically and experimentally that (i) the probability distribution of the latent space obtained by this model is proportional to the probability distribution of the real space because Jacobian between two spaces is constant: (ii) our model behaves as non-linear PCA, which enables to evaluate the influence of latent variables on data. Furthermore, to verify the usefulness on the practical application, we evaluate its performance in unsupervised anomaly detection and outperform current state-of-the-art methods.

Under review as a conference paper at ICLR 2020
RATE-DISTORTION OPTIMIZATION GUIDED AUTOEN- CODER FOR GENERATIVE APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
In the generative model approach of machine learning, it is essential to acquire an accurate probabilistic model and compress the dimension of data for easy treat- ment. However, in the conventional deep-autoencoder based generative model such as VAE, the probability of the real space cannot be obtained correctly from that of in the latent space, because the scaling between both spaces is not con- trolled. This has also been an obstacle to quantifying the impact of the variation of latent variables on data. In this paper, we propose Rate- DistortDon Optimization guided autoencoder, in which the Jacobi matrix from real space to latent space has orthonormality. It is proved theoretically and experimentally that (i) the probabil- ity distribution of the latent space obtained by this model is proportional to the probability distribution of the real space because Jacobian between two spaces is constant; (ii) our model behaves as non-linear PCA, where energy of acquired la- tent space is concentrated on several principal components and the influence of each component can be evaluated quantitatively. Furthermore, to verify the use- fulness on the practical application, we evaluate its performance in unsupervised anomaly detection and it outperforms current state-of-the-art methods.
INTRODUCTION
Capturing the inherent features of dataset from high- nigh-dimensional and complex data is an essential issue in machine learning. Generative model approach learns the probability distribution of data, aiming at data generation by probabilistic sampling, unsupervised/weakly supervised learning, and acquiring meta-prior (general assumptions about how data can be summarized naturally, such as disentangle, clustering, and hierarchical structure (Bengio et al., 2013; Tschannen et al., 2019)). It is generally difficult to directly estimate a probability density function(PDF) Px(x) of real data æ. Accordingly, one promising approach is to map to the latent space with reduced dimension and capture PDF Pz(z). In recent years, deep autoencoder based methods have made it possible to compress dimensions and derive latent variables. While there is remarkable progress in these areas (van den Oord et al., 2017; Kingma et al., 2014; Jiang et al., 2016), the relation between æ and z in the current deep generative models is still not clear.
Variational autoencoder (VAE) (P.Kingma & Welling, 2014) is one of the most successful generative models for capturing latent representation. In VAE, lower bound of log-likelihood of Px(x) is introduced as ELBO. Then latent variable is obtained by maximizing ELBO. In order to maximize ELBO, various methods (Alemi et al., 2018: Zhao al., 2019; Brekelmans et al., 2019) have been proposed. However, many previous works did not care about the value of Jacobian between two spaces, despite the fact that the ratio between Pz(z) and Pr(x) is equal to the Jacobian. Even in models that provide more flexible estimation (Johnson et al., 2016; Liao et al., 2018; Zong et al., 2018), this point overlooked.
Here, when we turn our sight toward acquiring meta-prior, it is straightforward to evaluate the quan- titative influence of each latent variable on the given metrics between data Ø and Ø2. To do so, the scale of the latent variable should be appropriately controlled so that the changes in latent variables is proportional to the changes of given metrics in data space. In addition, this scaling should be ad- justed according to the definition of the metrics. For instance, with respect to image quality metries, different meta meta-prior should be derived from mean square enror (MSE) and the structural similarity (SSIM). Considering the mechanism of principal component analysis (PCA) would be one of the

---

Under review as a conference paper at ICLR 2020
directions to solve it. In PCA, PCA components are derived by optimal orthonormal transforma- tion, then the importance of each component can be identified quantitatively by their variance. should be noted that orthogonality is not enough for quantitative analysis, but both orthogonality and normalized scaling, namely, orthonormality is required. This fact implies that, if Jacobi ma- trix of autoencoder has orthonormality, the characteristics of acquired latent space can be evaluated quantitatively. To deal with this, we propose RaDOGAGA Autoencoder for Generative Approach), based on the rate-distortion optimization (RDO) which has been widely used in image compression with orthonormal transform coding (Sullivan & Wiegand (1998)). In this paper, we show the effect of RaDOGAGA in the following steps.
Cate--Dstoortion Optimization Guided
(1) We prove that RaDOGAGA has the following property theoretically and experimentally.
Jacobi matrix between real space and latent space leads to be constant- scalled orthonormal.
So the response of the minute change of to the real space data a: is constant any z.
Because of constant Jacobian (or pseudo-Jacobian), Pæ(x)) and Pz(z) are almost pro- portional. Therefore, Pz(x)) can be estimated, by directly maximizing log-like l-likkiihoodd of parametric PDF (z)(z) in reduced-dimensional space, without considering ELBO.
When univariate independent distribution is used to estimate Pz(z) parametrically, it be- haves as "continuous PCA" where energy is concentrated on several principal components.
(2) Thanks to this property, RaDOGAGA achieve the state-of-the-arr performance in anomaly de- tection task with four public datasets, where probability density estimation is important.
(3) We show that our approach can directly evaluate how the z impact on the given metrics in real space. This feature is promising to further interpretation of latent variables.
2 RELATED WORK
Flow based model: Flow based generative models generates astonishing quality of image (Kingma& & Dhariwal, 2018; Dinh et al., 2014). Flow mechanism explicitly takes Jacobian of æ and z into account. The transformation function z fax) is learned, calculating and preserving Jacobian of æ and z. Unlike ordinary autoencoder, which reverse z to æ with function g(-) differ- ent from f()), inverse function transforms z as = (z). Since the model preserves Jacobian, Pz(x) can be estimated by maximizing log likelihood of Pz(z) without considering ELBO, Al- though, in this approach, f(.) need to be bijection. Because of this limitation, it is difficult to fully utilize the flexibility of neural networks.
Interpretation of latent variables: While it is expected to acquire meta-prior by deep autoencoder, interpreting latent variables is still challenging. In recent years, research aiming to acquire disentan- gled latent variables, which encourages them to be independent, is flourishing (Lopez et al., 2018; Chen et al., 2018; Kim & Mnih, 2018; Chen et al., 2016). With these methods, qualitative effects for disentanglement can be seen. For example, when a certain latent variable is displaced, image changes corresponding to specific attributes (size, color, etc.). Some works also propose quantitative metrics for meta-prior. In beta- VAE (Higgins et al., 2017), the metric evaluates the independence of latent variables by solving the classification task. Actually, the Jacobi matrix of VAE is orthogo- nal (Rolínek et al., 2019), which enables to make latent variables disentangled implicitly. However, orthonormality is not supported and it is difficult to define the metric which evaluates the effect of latent variables to the metrics between data directly.
Deep image compression with rate-distortion optimization: Rate-distortion (RD) theory (Berger, 1971) is part of Shannon's information theory for lossy compression which formulates the optimal condition between information rate and distortion. Then RD theory for Gaussian source with memory has been further extended to transform coding (Goyal, 2001) for image and audio lossy compression where orthonormal transforms such as Karhunen-Loevw transform (KLT) (Rao & Yip (2000)) and discrete cosine transform (DCT) are used for decorrelation. Furthermore, rate-distortion optimization (RDO) has been widely used in image compression (Sullivan & Wiegand, 1998). In RDO, cost L = R X D is minimized at given Lagrange parameter À to realize the best trade-off between rate R and distortion D such that L = R + X D becomes a tangent line of true Rate- Distortion curve. Recently, deep learning based image compression (Ballé et al., 2018; Zhou al., 2019) has been proposed. In these works, instead orthonormal transform with L2 norm metrics in the conventional lossy compression method, deep autoencoder is trained with flexible metrics
2

---

Under review as a conference paper at ICLR 2020
such SSIM for RDO. To understand the success of these works, prove theoretically that RDO guides a Jacobi matrix of deep autoencoder to orthonormal the space depending on the metrics. In the next section, explain the idea of RDO guided autoencoder and its relationship with VAE.
OVERVIEW OF RATE-DISTORTION OPTIMIZATION GUIDED APPROACH
3.1 DERIVATION FROM RATE-DISTOREE OPTIMAZATION IN TRANSFORM CODING
Figure shows the overview of our idea inspired by Rate-Distortion optimization of transform cod- ing. In the transform coding, the optimal method to encode data with Gaussian distribution as follows (Goyal, 2001). At first, the data are transformed deterministically to decorrelated data us- ing the orthonormal transform such as KLT and DCT. Then these decorrelated data are quantized stochastically with uniform quantizer for all channels such that quantization noise for each channel equal. At last optimal entropy encoding is applied to quantized data where the rate can be caleu- lated by the logarithm of symbol's estimated probability. From this fact, we have an intuition that the Jacobi matrix of autoencoder becomes orthonormal if the data were compressed based on Rate- Distortion optimization with uniform quantized noise and parametric distribution of latent variables. Inspired by this, we propose autoencoder which scales latent variables according to the definition of metries of data, without limitation of the transformation function as in Flow models. Thanks this feature, our scheme can estimate P=(z) quantitatively, which is suitable for clustering and anomaly detection. Furthermore, in the case factorized distribution is used for Pz(z), our model behaves as continuous PCA. This property is considered to promote the interpretation of latent variables.
3.2 RELATIONSHIP WITH VAE
There is a number of VAE studies that taking rate-distortion trade-off into account. In VAEs, instead of maximizing log-likelihood of P(x) directly, maximizing ELBO. In bet (Higgins al., 2017), Objective function LV is described as LV = Lree 3 Lxt- L is KL divergence between encoder output and prior distribution, usually Gaussian distribution. By changing 3, the rate-distortion trade-off at desirable rate can be realized as discussed in (Alemi et al., 2018).
When the relation between beta- VAE and RDO in image compression is considered, both analo- gous to each other. That is, 3-1 is corresponding to X, - -La corresponding to a rate R, and Lyee corresponding to distortion D. However, the probability distribution models of latent variables are quite different. VAE uses a fixed prior distribution. This causes a non-linear scaling relationship between real data and latent variables.
Figure 2 shows the conditions to achieve Rate-Distortion optimization in both VAE and RDO guided autoencoder. In VAE, precise control of noise distribution is required for each data and channel as suggested in Brekelmans et al. (2019) in order to compensate for the nonlincarities between the data space and the latent space. Jacobi matrix becomes orthogonal as proved Rolínek et al. (2019). However, Jacobian is not constant because of non-linear scaling. In RDO guided autoencoder, uni- form noises are added to all channels. Instead, parametric probability distribution should be used in order to make the scaling linear between the data space and the latent space. As a result, the Jacobi matrix becomes orthonormal because both orthogonality and scaling normalization are simultane- ously achieved. As discussed above, the precise noise control in VAE and distribution parameter optimization in RDO guided autoencoder are essentially the same. Accordingly, complexities in both methods estimated to be at the same degree.
METHOD AND THEORETICAL PROPERTY
4.1 METHOD
Our model is based on the Rate-Distortion optimization of the autoencoder for the hyperprior pro- posed in Ballé et al. (2018) for image compression with some modification. The cost function con- sists of (i) reconstruction error D between input data and decoder output with noise to latent variable and (ii) Rate R of latent variable. This is analogous to beta- VAE where À = 3-1
L= R A
3

---

Under review as a conference paper at ICLR 2020
Conditions to achieve Rate- Date-Distottion Optimization
Model of Transform coding based on DistDiittotion Theory
Method
PDF model
Noise
Jacobi Matrix
Image Owis
O+tonooma
Transform
data
Decomelated
Unffrm
Enropy coding
Quanizaton on
Das
Compreesed
VAE with fixed
Fixed as prfor
Precise control for each data and channels
(Variable
Orthogonel
Jacobiaan)
Source Coding
Channel Coding
RDO guided
Variable
Uniform for data and chennels
Orthonormal
Analogy belween transform coding and RDO guided aufoencoder
Autoencodor
parametric
(Constant
Method
Source Coding
Channel Coding
Ratte-Distortion
Jacobian)
(Detorministee
(lacchastic)
Relation
Transform
Condition: Orthonormal
Condition:
Result:
Relationship between Noise, Jacobi Matrix
coding
Uniform
Rate-Distortion
X
VAE
RDO guided autoencader
RDO guided Aufoencoder Autoencoder with
fransform
KLT PGA/
Expected Result:
quantization
Condition:
Entropy coding
Opfimization
achieved
Condition:
(Vanible
Oithagonal
Jacobian)
FO as peor
Oithoriommal (Constant Jacobian)
Paramatric PDF
Uniform noise
Sale--Dittrtion based Loss
orthonormal Jacobi matrix
for all channel/
Rate ectimetion
function
Precisely control led
Unfomm
Figure 1:
Overview of our idea based on RDO. Figure 2:
The condition to optimize rate- distortion in VAE and our models
The specific method is described as follows. First, let be M-dimensional domain data(æ: R and (z(x) the probability of æ. Then æ is converted to N N-dimensional latent space R by encoder. Let fo(z), (z), and (z)(zz) be parametric encoder, decoder, and probability distribution function of latent space with parameters 0, Φ, and th. It is noted that encoder and decoder are deterministic unlike VAE. Then latent variable z and decoded data & are generated as bellow:
= fa(
z= =0])
(2)
Let e E R~ be noise with each dimension being independent with an average of 0 and a equal variance of a² to emulate uniform quantization.
c= (11, <, .(c)), E [[G] = 0, E (fl..))= = 6iz o²
(3)
Then, given the sum of latent variable and noise the decoder output ø is obtained. This is analogous to the stochastic sampling and decoding procedure in VAE.
i (z E)
(4)
Here, the cost function is defined by Eq. (5) with some modifications from Eq. (1).
L - log( (2)) + \ h (D (x. 2)) + )2 D (át,
(5)
In this equation, the first term is corresponding to the estimated rate of the latent variables. In Ballé et al. (2018), a rate for quantized symbol is calculated by - __.55 u.s model, this is replaced by - log( (zz())) for simplicity. To model Pz (z). factorized probability model or Gaussian mixture model (GMM) can be used for instance. D(x1, in the second and the third term is a metrics function between Φ1 and 22. Actually, the second and the third term is decomposition of (xx, go (x)))+))) ~ Dax, go ( fo(x)))++ D(s+DD(gs( ((gs( go (oxr))))))) as shown in Rolínek et al. (2019). The second term purely calculate reconstruction loss and the third term only influences the scaling of latent space. Accordingly. a₁ controls the degree of reconstruction, and >2(~~ 8-1 of beta-VAB) controls a scaling between the data space and the latent space. By this decomposition, we can adjust balance between reconstruction loss and other losses flexibly and easily lead better performance. h(-) in the second term of Eq. (5) is monotonically increasing function. According to the choice of h((-), the degree of reconstruction accuracy can be controlled. Roughly speaking, when hdd) d is chosen, the training much focus fitting the distribution and entropy. On the other hand, h(d)- = log(d) encourages better reconstruction, since the curve of loss steep around O. The effect of hdd) is further discussed in Appendix D. Then, Eq. (5) is averaged according to distributions, x ~ Px(æ)) and ~ P(e). For metrics function D(-, -), a variety of
(g)(P=(z))dd= In our
function is applicable such as MSE, SSIM, and so on. As long as the function can be approximated
by the following quadratic form in the neighborhood of x, the property of the model is hold
D(x, x + 8x) ~ '8x A(x) òx = [[ 8x
(6)
Here, dæ is arbitrary miero variation of x, A(x) is M AM positive definite matrix depending on and Lxx) is Cholesky decomposition of A A(x). For instance, when D(,, .) is square of L2 norm as in Eq. (7), A(x) and (xx) are Identity matrices.
D(21, x2) ||a: 22212
4

---

Under review as a conference paper at ICLR 2020
In the case of SSIM (Wang et al., 2001) metric which is close to subjective image quality, cost (1 - SSIM) can be also approximated in a quadratic form with positive definite matrix. This is explained in Appendix C. By deriving parameters that minimize this value, the encoder, decoder, and probability distribution of the latent space are trained as Eq. (8).
0, Φ, v
arg min E Px(a)))) P L1)
(8)
0,0.V
(D(x,8)
Encoder fdlx)
Decodar g,(3)
Domain
D(2,))
Psylt Porametric PDF of (=llog(
Decoder
E-P(E)
(2(++e)
x
8.9. argmin(E Maa + 21 (L-hhD((2.)) XZ D(3,
)D)
Figure Architecture of RaDOGAGA
4.2 THEORETICAL PROPERTY OF THE MODEL
In this section, show the theoretical property of the model. First of all, we provide a rough sketch of the theory. Let J dæ dz| be Jacobian between the data space æ and the latent space. Here, R and AD Eq. (1) can be approximated by log Px(x) log(J) and C² respectively where C..A is a constant depending on a and As a result, the Jacobi matrix with J = 1/ =1///2C., aC,, minimizes Eq. (1). Because Jacobi matrix is orthogonal (Rolínek et al., 2019) and the Jacobian is constant, the Jacobi matrix becomes orthonormal. full proof is described Appendix A. this section, we mainly explain important points. Let be rescaled space of æ according to the metrics :(.,), where dæ and dæ are differentials and L(x) is a Jacobi matrix between two spaces.
dæ L(x))
When D(--)) is square of L2 norm, both spaces of æp and æ are equivalent. As described above, each column of the Jacobian matrix of latent space z and rescaled space zp are constant multiple of orthonormal basis regardless of the value of z and æ after training based on Eqs. (5) and (8). Here, d.j denotes Kronecker delta and matrix 'T denotes transpose T.
(
dx
dx 0z;
8xx az.
Oxp 0z,
[((z))
((x)
6ij
(10)
0z.
2A202
From the orthonormal property, model has several features. The first feature is that The distance derivatives both real and latent spaces has a linear relationship. Therefore, the metrics between æ and æ + dæ proportional to || 1| derived by Eqs. (6), (9) and (10) where dz is differential.
N
Exp
D( x + dx) [(x)dx 1 2|
dzg
||dz||
(11)
Oz:
2A20
D x + dæ)//ldz11 || D / /2/22
const.
(12)
This equation also implies that the L2 distance of two data in the latent space is proportional to the distance between corresponding data in the real space which defined by the integration the root of metrics derivative.
The second feature is constant Jacobian. In case of M N. the Jacobian matrix dæp/dz is square matrix, and each column is the same 1 1/(v2Xp a) times orthonormal basis. For this reason, the Jacobian is a constant regardless of the value of z as shown below:
dap dz
1
(N/2)
(13)
2 0²
In this case, the probability distribution of zD and is proportional because of the constant Jacobian. For the case of M > N, we assume the situation where most energy is efficiently and effectively mapped to N-dimensional latent space. Then the product of the singular values of SVD for Jacobi
5

---

Under review as a conference paper at ICLR 2020
matrix can be regarded pseudo Jacobian between the real space and the latent space. Since all of the N singular values are 1/(v/Ag T), the pseudo Jacobian is also a constant.
The third feature is proportional probability density between data space and latent space. Because Jacobian or pscudo Jacobian J is constant, the following equation holds. Pxp (x)) and Pz(x) are true PDFs for each space æp and respectively.
Pz(z)
(æD( & PæD(ED)
(14)
The forth is that the log-lik log-liielhood of PDF of the domain data can be maximized without considering
ELBO. This is revealed by the transformation of Equation (5). Let (xp( be estimated proba- bility of dp. Because the Jacobian is constant, (xp( can be approximated by J Pz (z). Accordingly, the average of the first term in Equation (5) by a Px(x) can be transformed as follows.
E lo (z))) = -aa=-PP(e))) /og Pap(aD) - log(J)
(15)
Consequently, minimization of Equation (15) is equivalent to the log-likelihood maximization of Pæp(xp)
The fifth "continuous PCA" feature when the following factorized probability model is used.
N
Pa (2) = II
Zig (4.)
(16)
i=1
As shown in FactorVAE (Kim Mnih, 2018), the use of factorized model minimizes mutual infor- mation among latent variables and encourages disentangling. Combining with the first character of our model described before, the variance values of each latent variable show the quantitative impact on data space like linear PCA. The detailed derivation is explained in Appendix B..
5 EXPERIMENT
5.1 PROBABILITY DENSITY ESTIMATION WITH TOY DATA
In this section, we describe our experiment using toy data to demonstrate whether the probability density of the input data Pr(æ)) and that of estimated in the latent space Pz(z) are proportional to each other as in theory. First, we sample data ($1, $2...31,.000 0000)) from dimeesiimniii Gaus- sian distribution consists of
with mixture weight π = (1/3, 1/3,1/3), mean p4 (Mk1: P22- (23), and covariance Σg diag( Ø22 [33). 1 is the index for mixture component. Then, we seatter with uniform random noise d and m are index for dimension of sampled data and scattered data. The Uas are uncorrelated with each other. We produce 16-dimensional input data æ with a sample number of 10,000 in the end.
R3x udm Ua 2. 2). where
3
Ud8d
(17)
The appearance probability of the input data (r( is equivalent to the generation probability of s.
5.1.1 CONFIGURATION
In the experiment, we estimate the (z) using GMM with parameter as in DAGMM (Zong et al., 2018). Instead of EM algorithm, GMM parameters are learned using Estimation Network (EN), which consists of multi-layer neural network. When the GMM has K mixture- components and L is the size of batch samples, EN outputs the mixture-components membership prediction as K dimeensioonl vector 3 as follows:
p EN v softmar(p)
(18)
K-th mixture weight SR. mean fax covariance Σx. and entropy R of z are further calculated by Eqs. (19) and (20).
L
R
Rike L, Th = Σ nuzz
Tik+
ΣS =
Tue (zi - (2)) - -Ta
E
7uk
(19)
1=1
1=1
1=1

---

Under review as a conference paper at ICLR 2020
1 Σ 1==1
1 2n exp -2 Mx) Ix (z Thmi))
R = log
Fx/
(20)
Overall network is trained by Eqs. (5) and (8). In this experiment, we set D( z2) as square error 21 22 112, and test two types h(-). h(d) = d and h(d) log(d). We denote models trained with these h(-) as RaDOGAGA(d) and RaDOGAGA(loo(d)) As a comparison method, DAGMM used. DAGMM also consists of encoder, decoder, and EN. In DAGMM, to avoid falling into the trivial solution that entropy is minimized when the diagonal component of the covariance matrix is the inverse of the diagonal component added to the cost function as Eq. (21):
L x 21|² A1 (- log (2)(
P(E)
Σ ki
(21)
The only differences between our model and DAGMM are (i) RDO mechanism introduced (ii) the third regularization term in Eq. (21) is eliminated, since our model adjust latent scale appropriately not to falling into trivial solution. Thus, model complexity such as the number parameter is the same. For both RaDOGAGA and DAGMM, the autoencoder part is constructed with fully connected (FC) layers with sizes of 64, 32, 16, 3, 16, 32, and 64. For all FC layers except for the last of the encoder and the decoder, we attach tanh as the activation function. The EN part also constructed with FC layer with a size of 10, 3. For the first layer, we attach the tanh as activation function and dropout (ratio=0.5). For the last one, softmax is attached. (A1, X2) is set as 10%) and (1 10 x 104) for DAGMM, RaDOGAGA(d) and
(1 x 10-1, 10 10-9), (1 x 106, RaDOGAGA((lo(d)) respectively. Optimization is done by Adam optimizer (Kingma & Ba, 2014) with learning rate 10- for all model. We set as 1/12.
PDF (High)
S3
1x 10
15
10
S1
z2
x2
(Low)
20
14
15
20
21
-22
20
16
(a) input source
(b) DAGMM
(c) RaDOGAGA(d)
(d) RaDOGAGA(log(d)
Figure 4: Plot of source of input data and latent variable z. Normalized PDF value corresponds to a color bar located left of (a). Even though both DAGMM and RaDOGAGA capture three mixture components, the tendeney of PDF distribution in DAGMM looks different from input data source. Points with high PDF are not concentrated on the center of the clustering especially in upper two.
(zy())(1 10a)
(2) 100+)
(z x 10
10
16
1
0.01
0.02
-C
0.01
0,01
0,22
Px(x)
Px(X)
Px(x)
(a) DAGMM
(b) RaDOGAGA(d)
(c) RaDOGAGA((og())
Figure 5: Plot of of Pr(x) (x-axis) and Pau (z) (y-axis). In RaDOGAGA, Pr(x) and (z)(2) are proportional while we can't observe that in DAGMM. Thanks to this trait, we can estimate Pr(x)) directly from Pzp (2) without ELBO
5.1.2 RESULT
Figure displays the distribution input data source 8 and latent variable z. Even though both methods can capture that 8 is generated from three mixture-component, there is difference in PDF. Since the data is generated from GMM. the sample getting closer to the center of clustering, the PDF getting higher. Although, in DAGMM, this tendency looks distorted. This difference fur- ther demonstrated in Fig which shows a plot of Px(x)(x-axx) against (x)xaxxx),, In our method, Pa((:)) and (z)(2) approximately proportional each other as in theory while
7

---

Under review as a conference paper at ICLR 2020
cannot observe such proportionality in DAGMM. This difference is quantitatively obvious well. That is, correlation coefficients between Pr(x) and (z)(z) are 0.691 (DAGMM), 0.997 (RaDO- It also turns out that the correlation coefficient of
GAGA(d)), and 0.999 (RaDOGAGA(log(d))) RaDoGAGAAlogld)) is higher than that of RaDOGAGA(d), Actually, when Pz (2) is sufficiently fitted, h(d) (dl) makes Px(œ)) and Pzæ(2 be proportional more rigidly. On the other hand, h(d) = d makes the scale of latent space slightly bent in order to minimize entropy function, allow- ing relaxed fitting of (z). More detail described in Appendix D.
5.2 ANOMALY DETECTION TASK USING REAL DATA
In this section, examine whether the clear relationship between Pr((x) and Pz(z) is useful in the anomaly detection task using real data. We use four public datasets', KDDCUP99, Thyroid, Ar- rhythmia, and KDDCUP-Rev. The (instance number, dimension, anomaly ratio(%)) of each dataset is (494021, 121. 20), (3772, 6, 2.5), (452, 274, 15), and (121597, 121, 20) respectively. Detail of datasets described in Appendix E.
5.2.1 EXPERIMENTAL SETUP
We follow the setting in Zong al. (2018). Randomly extracted 50% of the data is assigned to training and the rest to testing. During training, only normal data is used. During the test, the R for each sample is calculated as the anomaly score, and if the anomaly score is higher than a threshold, it is detected as anomaly. The threshold is determined by the ratio of anomaly data in each data set. For example, KDDCup99, data with in the top 20 is detected as an anomaly. As metrics, precision, recall, and F1 score are calculated. We run 20 times for each dataset split by 20 different random seeds.
5.2.2 BASELINE MODEL
Same as in the previous section, DAGMM is taken as the baseline method. We also compare with the scores reported in previous works in which same experiments were conducted (Zenati et al., 2018; Song & Ou, 2018; Liao et al., 2018).
5.2.3 CONFIGURATION
As in Zong et al. (2018), in addition to the output from the encoder, concatenated to z. It is sent to EN. Note that is sent to the decoder before concatenation. Other configuration except for hyper parameter is same as in the previous experiment. Hyper parameter for each dataset is described in Appendix E.
||==='1 12 and -= are |-||=
5.2.4 RESULTS
Table 1 reports the average scores and standard deviations (in brackets). Comparing DAGMM and RaDOGAGA, RaDOGAGA has better performance regardless of types of h(-). Note that, our model does not increase model complexity at all. Simply introducing the RDO mechanism into the autoencoder has a valid efficacy for anomaly detection. Moreover, our approach achieves state- of-the-art performance compared to other previous works in which same datasets is used. Clear relationship between Pr(x) and Pz(z) by our model is considered to be effective in the task of anomaly detection where the estimating probability distribution is important. In RaDOGAGA, when we compare result of RaDOGAGA(G) and RaDOGAGA((og(d)) either of one is not always superior. As described section 5.1 and Appendix D. h(-) can be an option depending on fitting flexibility of (z(z).
5.3 QUANTIFYING THE IMPACT OF LATENT VARIABLES ON METRICS AND BEHAVIOR AS PCA
In this section, we confirm that (i) the latent variables in our model behaves as PCA components and (ii) the impact of each latent variable on the metrics function can be quantified. When dz in Eq.
Dataset can be dowonload from https://kddiies.uci.co and http:://odds..c.stonyyroo.....
8

---

Under review as a conference paper at ICLR 2020
Table 1: Average and standard deviations(in brackets) of Precision, Recall and Fl
Dataset
Methods
Precision
Recall
F1
ALAD
0.9427(0.008))
0.9577(0.0018)
0.9501(0.0018)
INRF"
0.9452(0.0105)
0.9600(0.0113)
0.9525(0.0108)
VAE"
0.7805
0.7903
0.7854
KDDCup
GMVAE"
0.952
0,9141
0,9326
DAGMM
0.9297
0.9442
0.9369
DAGMM+
0.9427(0. (0.0022)
0.9575(0.0053)
0.9500
RaDOGAGA(d)
0.9550(0.000 (0.003))
0,9700(0.0033)
0,9624
RaDOGAGA(log(d))
0.9563(0.0042)
0.9714(0.0042)
0.9638(0.0042)
GMVAE
0.7105
0,5745
0.6353
VAE"
0.3395
0.3592
0.3491
Thyroid
DAGMM
0.4766
0,4834
0.4782
DAGMM+
0.4656(0.044) )
0,4755(0.0491 (.485((000502)
RaDOGAGA(d)
0.631
0.6587 0496)
0.6447 (.486)
RaDOGAGA(log())
0.6562(0.0572)
0.6848(0.0597)
0.6702(0.0585)
ALAD
0.5000(0.0208)
0.5152(0.0214) 0.53T3(0.022))
VAE
0.3328
0.3392
0,3360
GMVAE"
0.4375
0,4242
0,4308
Arrythmia
DAGMM
0.4909
0.5078
0.4983
DAGMM+
0,4985
0.5136(0.0401)
0. 0.5060(0.0395)
RaDOGAGA(d)
0.5353
0.5515(0.0475)
0.5433(0.0468)
RaDOGAGAA(lg((d)
0.5294(0.0405)
0.5455(0.0418)
0.5373(0.0411)
DAGMM
0.937
0.939
0.938
DAGMM+
0.9778(0.0018)
0.9779(0.0017)
0.9779(0.0018)
KDDCup-rev
RaDOGAGA(d)
0.9768(0.0033)
0.9827(0.0012)
0.9797(0.0015)
RaDOGAGA(log(d)
0.9864(0.0009)
0.9865(0.0009)
0.9865(0.0009)
"Score is cited from Zenati et al. (2018)ALAD), Song & Ou (2018)INRF), Liao et al. (2018)(VAE, GM- VAE) and Zong al. (2018) (DAGMM) respectively.
DDAMM++ is our implementation. Note that we also test same configuration as Zong et al. (2018) and achieve similar score as reported (shown in Appendix E).
(12) is approximated by small displacement 6, the ratio D(g(z), 9( | will be almost constant regardless of and 5. If the ratio of each component in the latent space is equivalent, latent space can be regarded as isometric. So, we evaluate this ratio in each component. Let ō; be vector '(0, ... ,,,,,,,,,,,,,000 where only é-th component has minute value Then ()(x) denotes D(g(2).g(z=+b))) for i-th component. In the trained model, we encode the image æ; and obtain z1. Then, D((=;) is calculated for each sample. Finally, the average across all samples is measured. independently. We also observe the distribution
This operation is conducted to each dimension of x and how the image looks different in response to the variation of component 24
To train model, we use CclebA dataset (Liu et al., 2015), which consists 202,599 celebrity images. The images center-cropped that the image size 64 64.
5.3.1 CONFIGURATION
In this experiment, factorized distributions (Ballé al., 2018) is used to estimate Pz (z). For comparison, we evaluate beta- VAE. Both models are constructed with same depth of CNN and FC layers, with 256-dimensional Detail of networks and hyper parameter is written in Appendix F. For RaDOGAGA, we set D(x1. 222 as 1 SSIM(21, x2) and h as h(d) d. For beta-VAE, reconstruction loss also SSI M 22). SSIM 222 is defined by Eq. (61).
5.3.2 RESULT
Both models are trained so that the SSIM between input and reconstructed image is around 0.93. Figure 6a and 6b show the variance of each component in the latent variables arranged in de- scending order. The red line is the cumulative relative ratio of the variance. In Fig. 6b, variance is concentrated in specific dimension. On the other hand, in Fig. 6a, beta-VAF is trained so that each

---

Under review as a conference paper at ICLR 2020
latent variable is fitted to a Gaussian distribution with mean 0 and variance 1, there is significant difference in the variance of each latent variable. Figures 6c and 6d respectively depicts the aver- age of (z) of each of the top nine dimensions with the largest variance of In beta-VAE, the (z) varies drastically depending on the dimension i which shows anisometric latent space, while in RaDOGAGA, it is approximately constant regardless of which shows isometric latent space. Figure 7 shows decoder outputs when each component 2₁ is traversed from -2a to 2o, fixing rest of z as mean. From the top, each row corresponds 20, 21, =2 ... and the center column mean. In Fig. 7b, the image changes visually in any dimension of 2, while in Fig. 7a, depending on the dimension i, there are cases where no significant changes can be seen (such as 23. 21. and so on).
This result means that, in RaDOGAGA, the variance of z directly corresponds to the visual impact and the metrics i 22), behaving as PCA. Besides, since (2) is constant, the latent space shows isometric feature and the variance is regarded as quantitative importance.
Variance
Cumulative
Variance
Cumulative
(1 20
D1 10-3)
1.1
1.0
3.8
1.0
8
0.5
1.5
0.5
0.04
0.0
0.0
0.00
20
Pg
2g 21 42 23 kg
(a) beta- VAE
(b) RaDOGAGA
(c) beta-
(d) RaDOGAGA
Figure 6: Variance of (two on the left) and (2 on the right). Compared with VAE, in our model, the variance is concentrated in a few dimensions. Furthermore, the influence each z to the real image is constant. These results demonstrate that the latent variable in our model behaves as PCA components and the importance for the domain data can be quantified.
20
+20
2z
μ
+ Zo
Zo
Zo
1
Z
Z2
23
Z3
Z
Z
Z5
Z5
Z6
Z6
27
27
Z8
Zg
(a) Reta-VAE
(b) RaDOGAGA
Figure 7: Latent space traversal of z with top-9 variance. In beta- VAE, the variance of z is uncorre- lated to difference of image, while they are directly related in our model.
CONCLUSION
In this paper, we propose RaDOGAGA that learns parametric probability distribution and autoen- coder simultaneously based on rate-distortion optimization. It was proven that the probability dis- tribution of the latent variables obtained by the proposed method is proportional to the probability distribution of the input data theoretically and experimentally. This property is validated in anomaly detection achieving state-of-the-art performance. Moreover, our model has the trait as PCA which likely promotes interpretation of latent variables. For the future work, we will conduct experiments with different types of metrics functions that derived from semantical task, such in categorical classification. Mcanwhile, as mentioned in Tschannen et al. (2019), considering the usefulness of latent variables in downstream task is another research direction to explore.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A. Saurous, and Kevin Murphy, Fixing broken elbo. In Proceedings of the 35th International Conference on Machine Learning, pp- 159-168. PMLR, 2018.
Johannes Ballé, Valero Laparra, and Eero Simoncelli. Density modeling of images using a gener- alized normalization transformation. arXiv preprint arXiv:1511.06220 2015.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with scale hyperprior. In In International Conference Learning Repre- sentations, 2018.
Yoshua Bengio, Courville Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35:798- 1828, 2013.
Toby Berger (ed.). Rate Distortion Theory: A Mathematical Basis for Data Compression. Prentice Hall, 1971. ISBN 0137531036.
Rob Brekelmans, Daniel Moyer, Daniel Moyer, and Greg Ver Steeg. Exact rate-distortion in autoen- coders via echo noise. arXiv preprint (tt:::/994.0009990 2019.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp- 2610-26200 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:1410.8516, 2014.
Vivek K Goyal. Theoretical foundations of transform coding. IEEE Signal Processing Magazine, pp. 9-21, sep 2001.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR 2017, 2017.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. arXiv preprint 1rX1v:1611.05143 2016.
Matthew Johnson, David Duvenaud, Alexander B. Wiltschko, Sandeep R. Datta, and Ryan P. Adams. Composing graphical models with neural networks for structured representations and fast inference. In Advances in Neural Information Processing Systems, 2016.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Proc. of the International Confer- ence on Machine Learning, pp. 2649-2658, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint arXiv: MEt:::1412.6900 2014.
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. Semi-supervised learning with deep generative models. In Proceedings of the 27th International Conference Neural Information Processing Systems Volume 2, NIPS' 14, pp. 3581-3589, Cambridge, MA, USA, 2014. MIT Press.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions, In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
11

---

Under review as a conference paper at ICLR 2020
Weixian Liao, Yifan Guo, Xuhui Chen, and Pan Li. A unified unsupervised gaussian mixture varia- tional autoencoder for high dimensional outlier detection. In 2018 IEEE International Conference on Big Data (Big Data), pp. 2208-1217. IEEE, 2018.
M. Lichman. Uci machine learning repository. http://archive..cc..ci.c.e// 2013.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Romain Lopez, Jeffrey Regier, Michael Jordan, and Nir Yosef. Information constraints on auto- encoding variational bayes. In Advances Neural Information Processing Systems, pp. 6114- 2018. 2018.
Diederik P.Kingma and Max Welling, Aubb-amoodingvaarttiinabaayyy In ICLR 2014, Banff, Canada, Apr 2014.
Kamisetty Ramamohan Rao and Pat Yip (eds.). The Transform and Data Compression Handbook. CRC Press, Inc., Boca Raton, FL, USA, 2000. ISBN 0849336929.
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea direc- tions (by accident). In Proceedings of Computer Vision and Pattern Recognition (CVPR), June 2019.
Yunfu Song and Zhijian Ou. Learning neural random fields with inclusive auxiliary generators. arXiv preprint 1806:1800.00277 00271, 2018
Gary J. Sullivan and Thomas Wiegand, Rate-distortion optimization for video compression. lEEE Signal Processing Magazine, pp. 74-90, nov 1998.
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-basee rep- resentation learning. In Third workshop on Bayesian Deep Learning (NeurIPS 2018), Montreal, Canada, Dec 2019.
Aaron den Oord, Oriol Vinyals, al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility structural similarity, volume 13. IEEE Trans. Image Processing, 2001.
Sihan Wen, Jing Zhou, Akira Nakagawa, Kimihiko Kazui, and Zhiming Tan. Variational autoen- coder based image compression with pyramidal features and context entropy model. In In WORK- SHOP AND CHALLENGE ON LEARNED IMAGE COMPRESSION, 2019.
Houssam Zenati, Manon Romain, Chuan-Sheng Foo, Bruno Lecouat, and Vijay Chandrasekhar. Adversarially learned anomaly detection. In 2018 IEEE International Conference on Data Mining (ICDM), pp. 727-736. IEEE, 2018.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Balancing learning and inference in variational autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pp. 5885-5892 2019.
Jing Zhou, Sihan Wen, Akira Nakagawa, Kimihiko Kazui, and Zhiming Tan. Multi-scale and context-adaptive entropy model for image compression. In In WORKSHOP AND CHALLENGE ON LEARNED IMAGE COMPRESSION, 2019.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In /n International Conference Learning Representations, 2018.
12

---

Under review as a conference paper at ICLR 2020
A HOW JACOBI MATRIX BECOME A CONSTANTLY SCALED ORTHONORMAL BASIS
In this appendix section, we prove that all of column vectors in decoder's Jacobi Matrix have the same norm and are orthogonal to each other. In other words, each column of Jacobi matrix is constantly scaled orthonormal basis.
Here assume that data space sample æ has M dimension, that is R then encoded to N dimensional latent space sample y E Rx We also assume that an fixed encoder function = f o and fixed decoder & ginstt,( are given such that ((D (x) becomes minimal.
Sinit, (
(22)
Φ ginit o (ÿ) = minimal
(23)
s.t h(D(x, ())
We further assume that fixed parametric PDF of latent variable y is also given.
Py (y)
(24)
Here, it is noted that this function is not needed to be optimal in sense of DEL (Py (3) P (y)) where Py (y) an actual PDF of y.
Under these conditions, introduce new latent variable and y is transformed from z using following scaling function azz) : RN + R.
y a(z) (a1 (z), (an))))))))a))))))))))))))))
(25)
Here, our goal is to prove Eq. (10) by examining the condition of this scaling function which minimize the average of Eq. (5) with regard to e P(e).
Because of the assumption of minimal Dax, .), we ignore the second term of Eq. (5). Next, the PDF of can be derived using Jacobian of a(z).
da(x))
Pzy.a (2)
(26)
dz
By applying these conditions and equations to Eq. (5), the cost of scaling function a(z) to minimize the average is expressed as follows.
( [d((z)
La = -log
(a)x)) + A2 D(ginit, + t)), ginst, (a(z))
dz
(27)
Next, the latent variable is fixed you and 2n is defined the next equation.
(y0)
(28)
Then the average of Eq. (27) with regard to e P(e) is expressed as follows.
[da(2)
Ee-P(() [balz=oo]= log
(
dz
x=
+ d2 EeN [D (a(z0 + E)), gimit (((zr)
(29)
Then the condition a(zp) in the neighborhood of yo is examined when Eq. (29) is minimized. Here, it is noted that the first term of the right side doesn't depend
13

---

Under review as a conference paper at ICLR 2020
Before examining Eq. (29), some preparation of equations needed. At first, Jacobi matrix of (=)) at Zo is defined as B using notations of partial differentials bij
de.
0zj
b
b12
DIN D2N
Qy B
da(z)
b21
b22
Oz
dz
:
===0
bvi
ba2
bNN
ao.
where bij =
(30)
azj
The vector b, is also defined as follows.
bii b2i
dy
b;
(31)
0zj
:
==
bNs
It is clear by definition that b, and B have the following relation.
B (b1, b2, .b))
(32)
A.j is defined cofactor of matrix B with regard the element bij, and b; is also defined by the following equation.
Ai
A2i
6,
(33)
ANS
The following equations hold by the definition of cofactor.
N
bb 6,
=
bys
Abi
[
(34)
B db.j
B
Aij
b,
(35)
db;
In case of * j. inner product of b, and b, becomes zero because this value determinant of singular matrix with two equivalent column vectors b,.
N
bb by
bks
(b ... b,, b, ... ) =0 Aaj
(36)
Gi is defined as a M x N Jacobi matrix of ginit, ( y yo as follows.
dginsee (3) dy
G'mni
(37)
y=
Using these equations, we proceed to expand of Eq. (29). The first term of (29) in the right side can be expressed as follows, where the second term Eq. (38) in the right side is constant by definition.
da(z) dz
- llog
PPyinit (a(z))) -log( log({B] - log( Dog(Pyina Pyinst (yn))
(38)
14

---

Under review as a conference paper at ICLR 2020
Next step is an expansion of the second term in Eq. (29). First, the following approximation holds.
dgimit,p())
da(z)
J (a(zu + E)) o (((())
dy
dz
y=
==20
Gnnt B
Gmmit (E1 bj + 62 b.... ... +EN bN)
Σ G G G'ise b;
(39)
Then, the second term of Eq. (29) can be transformed to the next equation by using Eqs. (39), (3), and (6).
Ple) [D(genir E)), o (a((z)))
BenP(e)
L(x) Σ E Ginit
Σ Ele,") [ G'nst bill2
N
N
+2
Σ Ele . e,l
j=t+1
(L(x) Ginst (L)()) Gini bj)
N
a².
Σ
[xx) Gi bill?
(40)
As result, Eq. (29) can be rewritten as Eq. (41).
Ee-P(e) [Zulz=zo]
-log( Bl) - -log( Pyimit, (yo))
+d2 0².
Σ (()) G bill?
(41)
By examining the minimization process of Eq. (41), the conditions of optimal scaling function a(z) in the neighborhood of yo is clarified. Here, the condition of Jacobi matrix B is examined instead of a(z). Eq. (41) differentiated by vector b,, and the result set to be zero. Then the following equation (42) is derived.
2 a². ('(L(x) (L(r)) G'nit)) b
b,
(42)
B
Afterwards, Eq. (42) is multiplied by 'b, from the left, and divided by 2 o², As a result, Eq. (43) is derived by using Eqs. (34) and (36).
*(((x) G'nit bj) (L(x)) G'nitt b.) 2 o² 8:j
(43)
Here, define the following function gortho, which is composite function of ginit and a().
t = So (z) = Gimit, v (a(z))
(44)
Then the next equation holds by definition.
dgortho (2)
dá 3z,
G' b;
(45)
Oz,
==
2 ==
15

---

Under review as a conference paper at ICLR 2020
It is noted that this equation holds at any value of yo 2u- As result, the following equation, that is Eq. (10), can be derived.
(I(x)
dá dz: oee)
[(x))
2A20² 8ig
(46)
dzj
If encoder and decoder are trained well and æ holds, we can introduce new rescaled data space zp determined by metrics function such as dæp L(x) dx, and the next equation holds.
DxD Ozi
OzD Ōzg
81z
(47)
2A20²
In conclusion, all column vectors of Jacobi matrix between and up has the same L2 norm 1/ 2222a and all pairs of column vectors are orthogonal. In other words, when column vectors of Jacobi matrix are multiplied by the constant v2A20, the resulting vectors are orthonormal.
B EXPLANATION OF "CONTINUOUS PCA" FEATURE
In this section, we explain RaDOGAGA has continuous PCA feature when factorized probability density model as below is used.
N Pz (z) = II Pzrp
(48)
Here, our definition of "continuous PCA" feature means the following. 1) Mutual information be- tween latent variables are minimum and likely to be uncorrelated to each other. 2) Energy of latent space concentrated to several principal components, and the importance of each component can be determined.
Next we explain reason why these feature acquired. As explained in appendix A, all column vectors of Jacobi matrix of decoder from latent space to data space have the same norm and all combinations of pairwise vectors are orthogonal. the resulting vectors are orthonormal. Because encoder is inverse function of decoder ideally, each row vector of encoder's Jacobi matrix should be the same as column vector decoder under the ideal condition. Here, fortha, ( and g ((0)) defined as encoder and decoder with these feature. Because the latent variables depend on encoder parameter 0, latent variable is described as Ze fortho, (2x), and its PDF is defined Pza(za). PDFs of latent space and data space have the following relation where J is a Jacobian or pseudo-Jacobian between two spaces with constant value explained in appendix A.
other words, when constant value is multiplied,
Pzo(ze) ~ (xp(xp) Pxp(øp)
(49)
As described before, (z) is parametric PDF of the latent space to be optimized with parameter vb.
By applying the result of Eqs. (41) and (43), Eq. (5) can be transformed as Eq. (50) where =
J (f (x))).
Lortho = - log (Pzu(zo)) + br (D(x, 2)) + N/2
(goorhoo. (z0) )
(Dlortho.0 (zo)
s.l.
2X ōij
(50)
Izo:
Oza;
Here, the third term of the right side is constant, this term can be removed from the cost function as follows.
T log (2))(20)) + X (D( .))
(51)
Then the parameters of network and probability can be obtained by the next.
8, o. 10
arg min (annEE--Pa(())) (ottha)))
(52)
8.0.0
16

---

Under review as a conference paper at ICLR 2020
Eg Pa(n) (m) orthol in Eq. (52) can be transformed as the next.
Pan) = / Px(x)) (- (Pzp(zo)) + X1 (D(x, a))) da
/ PPooz00)
dæ
)) (- log (Pzp(zo)))
dæ
dæg
dzo +i1 / PxD(aD) [Lxz]] (D(x, +)) (<
dzg
| Pzo(ze) (- log (Pzw(ze))) dzæ
+hi /PED(ED) h (D(,,, 2)) dxp
(53)
At first, the first term of the third formula in Eq.(53) is examined. Let dzøza be differential of (N - 1) dimensional latent variables where i-th axis 20i is removed from the latent variable zø. Then marginal distribution of zpz can be derived from the next equation.
Proc(zos)= = / Pzo(zo)dzo/1
(54)
By using (48) and (54), the first term of the third formula Eq. (53) can be expanded as follows.
f Pzo(20) (- log (2)((2))) dzo
/ Pzo(zo)
(-log
(-log
II
Pzip (20s)
dzo
Pzod (zos)
+ I
Pro(ze)-
(-log
Proe (zes) \==1
1I
dæg
(-
(-1og
Pziw(ze))
) dzøs Pzor(20s)
Σ
Pzo (zo(zo)dza)1)
Pro(zo)dzzg
(- log (Pzai(Zni)) dz0s
D (Pzoi(zoi)) (zi)) +
H(z0))
(55)
(X) is an entropy of variable X. The first term of the third formula is KL-divergence between marginal probability P2ox(20;) and factorized parametric probability P2tp(zo) The second term of the third formula can be further transformed using mutual information between latent variables (22)) and equation (49).
N Σ
H(zo)) = Hz0)) + I(z)) ~ H(xD) - log(J) + I(ze)
(56)
The first term of the third formula is the entropy of input data with constant value. The second is also constant. As a result, order to minimize (55), mutual information / ((2g) must be minimized.
At second, the second term of the third formula in Eq. (53) is examined. a and åp denote mapped values to rescaled data space defined by the metrics function D(x1, x2) as (9). Because ap and æp are close, the following equations holds.
tD
TD + L(x) (# x))
(57)
((xr
[[x)) (fc m²)2 [ =pp
(58)
17

---

Under review as a conference paper at ICLR 2020
By using these expansions, Eq.(53) can be expressed follows.
N
Pa(e) Poot
DEL (Pzz
1=1
+1(z)) + Enp
[ &D ; Const.
(59)
Here, the rescaled real space ap divided into a plurality of small subspace partitionings px p2 Let fiz1, f2zy, be corresponding subspace partitionings in latent space.
as the division space of the latent space 2 E R corresponding to flzp. Then Eq. (59) can be
rewritten as follows.
Eg Pan( orttol
DRL DKL
Σ (T(zo E azokk EapeRaps
|ED #Dl)) + Const. (60)
For each subspace partitioning, the transformation from Qæe DA to ΩZz can be regarded as constantly scaled orthonormal transformation where orthonormal basis is Jacobi matrix with scale factor J
According to Karhunen-Loces Theory (Rao & Yip (2000)), the orthonormal basis which minimize both mutual information and reconstruction error leads to Karhunen- Loève transform(KLT)) is noted that the basis of KLT is equivalent PCA orthonormal basis.
As a result, when Eq. (60) is minimized, Jacobi matrix from ΩE to UZza for each subspace partitioning should be KLT/PCA. Accordingly, the same feature PCA will be realized such the determination of principal components etc.
From these consideration, conclude that RaDOGAGA has "continuous PCA" feature.
Rescaled domain space X₂ E RM with small subspace partitioning 2xxx
Latent Space Z RN with small subspace partitioning 92z
PCA
R22
For all small subspace partitioning fKxx of réscaled domain space, mapping from 2xxx 22, can be regarded PCA
Figure 8: Continuous KLT(PCA) Mapping from input domain to latent space
C EXPANSION OF SSIM TO A QUADRATIC FORM
Structural similarity (SSIM) (Wang et al., 2001) is widely used for picture quality metric which is close to human subjective quality. In this appendix, show (1 SSIM) be approximated to a quadratic form such as (6)
Eq. (61) is SSIM value for N N window between picture and y. In order to calculate SSIM index for picture, this window is shifted in a whole picture and all SSIM values are averaged.
2prky
2
SST SSTMNxN
(61)
us. +
Myy
2
0 2 + o
18

---

Under review as a conference paper at ICLR 2020
If (1 S (æ. y)) is expressed in quadratic form, the average for picture (1 SSIMpicture] can be also expressed in quadratic form.
Let dæ be minute displacement of æ. Then SSIM between and + dæ can be approximated as follows
S 6z))
M48 2
0n 20 2
0 ((1dxl/ /z1)))
(62)
Then Paw 2 and dee 2 can expressed as follows.
Môx = dox M δx
where M N2
(63)
'8x
N
N
where V
(64)
N2
N-1
noted that matrix M is positive definite and matrix positive semidefinite. As result, (1 - SSTMNxN (æ, v)) can be expressed in the following quadratic form with positive definite
matrix.
SSIMNxN(E,, x +6x) '8xx ( 2p
M +
20 v) 8x
(65)
D EFFECT OF h(x)
In this appendix section, the effects of two kinds of cost scaling function h(d) = d and h(d) log(d) are discussed. We evaluated the behaviors of encoder and decoder in one dimensional model using simple parametric linear encoder and decoder.
(|x 212)
x XER x~N(0,,,²)
Encoder z=a'x
Decoder 8=h.2
18 x|2
Probability z P(z)~ (O, (ao (ao,)²) (Assume Pu(()=P(z))
Decoder --b-(zte)
c~N(O.1)
a, arg min (Ex,e [- log(P(z)) + A1 -11hh(x---- - 2l2) + 12 x[2])
Figure 9: Simple encoder/decoder model to evaluate (dd))
Lex x be dimensional data with the normal distribution.
x E
N(O. 2,
19

---

Under review as a conference paper at ICLR 2020
Lex 2 be dimensional latent variable. Following two linear encoder and decoder are provided with parameter a and
4 b
Here, square error is used metrics function (r,, y). The distribution of noise added to latent variable Z is set to N (0,1) Then ö is derived by decoding 2 te.
D(r v) = | y
1)0,
= b. (2(+c)
For simplicity, we assume parametric PDF Pıb(z) is equal to the real PDF P(z). Because the distribution latent variable follows N(, (aax) )), the entropy of can be expressed as follows.
P(z) ~ No, (a
H(z) / P(2) log P()))d=
(a) log 2Te)
Using these notations, Eqs. (5) and (8) can be expressed as follows.
Loss
Ez-N(0. N(011 [- log P(z) d h(1 - 213) X2 | [1 log(a) + log(a, 2Tre) Å1 E2~N(0,.,.) [M(1 =]})] + A2 .²²
(66)
At first, the case of h(d) d is examined. By applying h(d) d, Eq. (66) can be expanded as follows.
Loss log(a) + log(o,, V2Te) + X (a b 1)² ox + A2 b²
(67)
By solving aLoss 0 and Lose ob
0, and b are derived as follows.
a.b 21o= + A1 0o, 2d10 2
2)10²
21o 2
V X1 20 1- 22102²
a = V2. A2
2)10²
b 1/V2->0 A2
1102² > 1, these equations are approximated as next.
b = (1- 2X102?
a = V2. 12.22: 2210,² 2
b 1/v2.>2
Here, a b is not equal to 1. That is, decoder is not a inverse function of encoder. In this case, the scale of latent space becomes slightly bent in order to minimize entropy function. As result, good fitting of parametric PDF P(z) ~ (2(2) could be realized while proportional relationship P(z) ((z) is relaxed.
Next, the case of h(d) = (d) examined. By applying h(d) = log(d) and introducing a minute variable d, Eq. (66) be expanded as follows.
Loss = log(a) + (o 2me)) + A1 log (a b 1)2 + + d2 .b²
(68)
20

---

Under review as a conference paper at ICLR 2020
By solving 8a
Losss
0 and
@Loss
0 and setting 5 0, and b are derived as follows.
b = a V X2 b 1/V2.
(69)
Here, a- is equal to and decoder becomes inverse function of encoder regardless of the variance a²², this case, good proportional relation P(z) x P(x) could realized regardless of the fitting (e)(2) to P(z).
Considering from these result, there could be a guideline to choose h(d). the parametric PDF (0((2) has enough ability to fit the real distribution P(z), h(d) log(d) could be better. If not, h(d)= = d could be better.
E DETAIL OF THE EXPERIMENT IN SECTION 4.2
In this section, we provide further detail of experiment in section 5.2. First, we describe the detail of following four public datasets:
KDDCUP99 (Lichman (2013)) The KDDCUP99 10 percent dataset from the UCI repository is dataset for cyber-attack detection. This dataset consists of 494,021 instances and contains 34 continuous features and categorical ones. We use one hot representation to encode the categorical features, and eventually obtain dataset with features of 121 dimensions. Since the dataset contains only 20% of instances labeled -normal- and the rest labeled as -attacks-, -normal- instances are used as anomalies, since they are in a minority group.
Thyroid (Lichman (2013)) This dataset contains 3,772 data sample with 6-dimensional feature from patients and can be divided in three classes: normal (not hypothyroid), hyperfunction, and subnormal functioning, We treat the hyperfunction class (2.5%) as an anomaly and rest two classes as normal.
Arrhythmia (Lichman (2013)) This is dataset to detect cardiac arrhythmia containing 452 data sample with 274-dimensional feature. We treat minor classes (3, 4, 5, 7,8, 8, 9, 14, and 15, accounting for 15% of the total) as anomalies, and the others are treated as normal.
KDDCUP-Rev (Lichman (2013)) To treat "normal" instances as majority in the KDDCUP dataset, we keep all norrmal" instances and randomly pick "ttack instances so that they compose 20% the dataset. the end, the number instance is 121,597.
Next, hyper parameter for RaDOGAGA is described in table 2. First and second column is number of neuron. For DAGMM, we set same number of neuron table 2 and (A₁, A₂) as (0.1, 0.005). Optimization is done by Adam optimizer with learning rate x 10 for all dataset.
Table 2: Hyper parameter for RaDOGAGA
Dataset
Autoencoder
EN
(1
A2(d)
((lo((())
(2)gg()))
KDDCup99
60, 30, 30, 60
10, 4
100
1000
10
100
Thyroid
30, 24, 24, 30
10, 2
100
10000
100
1000
Arrhythmia
4, 10
10, 2
1000
100
1000
100
KDDCup-rev
60, 30, 30, 60
10, 2
1000
100
100
100
In addition experiment in main page, also conducted experiment with network size in (Zong et al. (2018) with parameters in table 3
Now, provide results of setting in table 3. In table 4, RaDOGAGA- and DAGMM- are results of them and DAGMM is result cited from (Zong et al. (2018)). Even with this network size, our method has boost from baseline in all dataset.
21

---

Under review as a conference paper at ICLR 2020
Table 3: Hyper parameter for RaDOGAGA((eferring (Zong al. (2018)))
Dataset
Autoencoder
EN
(1
A2(d)
((lo((d)
A2(log(d)
KDDCup99
60, 30, 1, 30, 60
10, 4
100
100
100
1000
Thyroid
12, 4, 1, 12
10, 2
1000
10000
100
10000
Arrhythmia
10, 2,
10, 2
1000
100
1000
100
KDDCup-rev
60, 30, 1, 30, 60
10, 2
100
100
100
1000
Table 4: Average and standard deviations(in brackets) of Precision, Recall and F1
Dataset
Methods
Precision
Recall
F1
DAGMM
0.9297
0.9442
0.9369
DAGMM-
0.9338(0.005))
0.9484(0.0003
0.9410(0.0051)
KDDCup
RaDOGAGA-(22)
0.9455(0.0016)
0.9608(0.0018)
0.953((0.0017)
RaDOGAGA-(log)
0.9370(0.0024)
0.9517(0.0025)
0,9443(0.0024)
DAGMM
0.4766
0.4834
0.4782
DAGMM-
(4635(0.1054)
0,4837(0. 1100)
0,4734(0. 1076)
Thyroid
RLDOGAGA-(22)
0.5729(0.044) 0449)
0.5978(0.046))
0.585 (0.0459)
RaDOGAGA-(lgg)
0.5729(0.0398)
0.5978(0.0415)
(..85)(0.0406)
DAGMM
0.4909
0,5078
0,4983
DAGMM-
0.4721(0.045))
0.4864 0464))
0.4791(0.0457)
Arrythmia
RaDOGAGA-(22)
0.4897(0.0477 04.777
0.5045(0.049))
0,4970(0.048)
RaDOGAGA-(lgg)
0.5044(0.0364)
0.5197(0.0375)
0.5119(0.0369)
DAGMM
0.937
0.939
0.938
DAGMM-
0.9491(0.0163)
0.9498(0.0158)
0.9494(0.0160)
KDDCup-rev
RaDOGAGA-(22)
(.9761(0.0057)
0.9761(0.0056)
0.9761(0.0057)
RaDOGAGA-(log)
0.9791(0.0036)
0.9799(0.0035)
0.9795(0.0036)
F DETAIL OF THE EXPERIMENT 4.3
In this section, provide further detail of experiment in section 5.3. For both RaDOGAGA and beta VAE, we first extract feature with following Convolution Neural (etN)
CNN(9, 9, 2, 64, GDN)-CNN(5, 5, 2, 64, GDN)-CNN(5, 5, 2, 64, GDN)-CNN(55) 5, 2, 64, GDN).
Here, CNN(W, h, s, c, f) is CNN layer with kernel size (w, h), stride size s, dimension c, and activate function f. GDN(Ballé al. (2015)) is often used in image compression. Then, we reshape feature map and send to autoencoder as follows.
FC(1024, 8192, FCC1)) 256, None)-FC((56, 8192, softplus)-FC(256, 1024, softplus)
FC(i, o, f) is FC layer with input dimension i, output dimension o and activate function None means activate function. Note that, for beta-VAE, since it produces mean and variance, the bottom of the encoder has branches.
(A1, A2) is as (1.0, 0.1) is for RaDOGAGA and is set 10-1 for beta-VAE.
Optimization is done by Adam optimizer with learning rate x 10-4
G PDF MATCHING WITH BETA-VAL VAE
For the reader with interest, we provide the result of experiment in section 5.1 with beta- VAE. Network consists of FC layers of which have the same neuron numbers the DAGMM and RaDO- GAGA. We set 3 as 0.001. Figure 10a and 10b depict results. Since VAE dose not support Jacobian controlling, Par( can not be mapped into (z) tidily.
22

---

Under review as a conference paper at ICLR 2020
PZ
14
2
0
z2
-2
4
-2
Z1
0
0.01
0.02
3 44
z0
Px(x)
(a) Plot of z. Even though captures three com- ponents of input data source s, PDF is quite dif- ferent from that of s.
(b) Plot of of Pr(x:) (x-axis) and (z)(2) (y- axis). No clear correlation can been seen.
Figure 10: Result of PDF estimation with toy data (beta-VAB VAE)
23


Reviews:
[{"review_id": "r1xu3tgSYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose a noisy autoencoder that considers the jacobian between data and latent spaces to match the corresponding densities. This idea has already been proposed elsewhere, and here it is applied to autoencoders. Overall I had hard time understanding the paper, the motivation, the main contribution or the claim, the model definition and the jacobian method. The paper is poorly written, with lots of issues in math notation and poor motivation and explication of what the sections are introducing, and what parts of the presentation is novel and what is already known. Lots of the math machinery is too vague to follow.\n\nThe distribution p(z) is unclear, and whether z is random variable or not. It seems that “z\" is a non-random variable, and then adding noise \\eps makes it stochastic. However, then p(z) without \\eps does not make any sense since z is not random. It seems that p(z) is maybe a prior distribution instead (or maybe the variational posterior?), but then adding \\eps noise to an already stochastic variable is strange. Overall I have hard time understanding the motivation of the two discrepancies in eq 4, what is the point of adding more noise to “z”? This seems some kind of noisy or perhaps robust AE variant, but the paper does not explicate this. I have hard time following the eqs 8-15. I am not convinced of the orthogonality argument, and I fail to see what this section tries to show or demonstrate. It seems that eq 14 is the final result, but its difficult to follow due to most terms in eq 14 being undefined. Optimizing eq 14 seems trivial since we can always match pz and pz_\\varphi easily with neural networks, or similarly the two x-distributions. \n\nIn the experiment 4.1. the proposed method seems to achieve matching densities, although the distributions are wrongly normalized. How does the density matching improve? All three methods seem to have equally good scatters. The benchmarks on table 1 show clear improvement with the method. The face experiment is unconvincing since the VAE spreads variance across all latent dimensions while RADO seems to compress them to just first 20 or so. If one would visualise the z_100 there would be no variance in RADO and possibly some variance in VAE. The paper also should compare their model to simple MNIST/VAE to highlight what problems are there in standard approaches (such as VAE), and how does the proposed method alleviate them.\n\nOverall the paper is poorly presented and difficult to follow. Despite this the method does seem to work remarkably well, and the Jacobian idea is clearly very promising. Nevertheless in its current form the paper is badly premature for ICLR, and needs a lot more work and polish to be made understandable for wider ML audience.\n\nMinor comments\no Px(x), x1, x2 are probably missing subscripts\no The point of eq 5 is unclear, it seems unnecessary. It also does not contain h(), which is claimed after eq6\no The log pz(z) in eq 4 is not entropy\no eq 8 is unclear, is the dx a derivative, distance or change?\no the $^t$ prefix notation is confusing, what does it mean?\no what is the \\sim and line notation in eq 5?\no what are the products in eq9, are these inner products?\no in eq 13 pz, pxd or hat(pxd) have not been introduced or defined\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SylJiIjpqr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper: The authors propose a latent variable model RaDOGAGA, a generative autoencoding model. The model is trained via a tradeoff between distortion (the reconstruction error) and the rate (the capacity of the latent space, measured by entropy). The paper provides an analysis of theoretical properties of their approach, and presents supporting experimental results.\n\nReview tl;dr: weak reject, for three main reasons:\n(i) While the existing literature around VAEs, beta-VAEs,  and Rate-Distortion theory is mentioned in the related work, the connections are not nearly discussed sufficiently. \n(ii) On top of (i), the derivation of their loss function and architecture is not sufficiently motivated. This is in astonishing contrast to 1.5 pages of main text and 8 pages of (much appreciated!) analysis of properties.\n(iii) Given the paper is clearly related to existing approaches in the literature, the experiments would require a much more careful comparison to existing models. It remains unclear why an interested user should favor your model over conceptually simpler generative models with fewer hyperparameters.\n\nDetailed review:\n\nNota bene: This review is a late reassignment. While I reviewed the paper to the best of my ability, time constraints did not allow me to review parts of the paper in depth.  I am open to reassess my review during the second stage.\n\nConnection to prior art: As a probabilistic, neural autoencoding model, the connections to the family of VAE models are obvious. The loss function (eq. (4)) still looks very much like the ELBO, where the typical conditional log-likelihood was split into two distortion terms. How is this different from e.g. a beta-VAE? Particularly, what is the connection between the rate-distortion analysis of beta-VAE by Alemi et al. and yours? These things need to be discussed explicitly, with more than a sentence or two in the related work section.\nA lesser, but still important omission in your discussion of prior work: The Jacobian of the generator has also been studied, even for the VAE, cf. e.g. [1]. I believe this deserves more attention in your assessment of prior art.\n\nMotivation: You use two distortion terms: actual sample vs. undistorted reconstruction. Why is that? What is the interpretation of the multipliers? How do I choose them? Why is a large part of your architecture (the pipeline from x to \\hat(x)) actually deterministic? Why are you using the entropy of the prior over the latents, rather than the KL divergence between encoder and a prior? I think an interested reader could learn much more from your paper if you discussed your model embedded in th related work rather than in isolation.\n\nTheory: Due to aforementioned time constraints, I was not able to review the extensive theoretical analysis in depth. Still, I would strongly recommend structuring the respective sections more clearly. Separate model and architecture description from the theoretical analysis; precisely formulate your claims. In particular, state your assumptions clearly. For instance, you assume \"that each function's parameter is rich enough to fit ideally\" (and similar e.g. in Appendix A). Does this only mean that the true distributions are part of the parametric family? What if this is not the case? Do your parameters need to be in the optimum for your analysis to hold true? \n\nGiven that the full 20-page manuscript spends 10 pages on theory, I think this contribution is not given appropriate space in the main text.\n\nExperiments: There are three experiments: a simple 3D proof of concept; anomaly detection; analysis of the latent state in CelebA. As mentioned in my review of the methods section, I believe the approach to be very similar to established models. None of the experiments provides convincing evidence why I should prefer the new, arguably more complex model.\nFor instance, I would have much preferred that you investigate properties of your model against alternatives over the anomaly detection experiments, which did not further my understanding of the proposed model. \n\nSummary: The paper tackles an important problem, namely the lack of control over the latent embedding in autoencoding generative models. I believe the author's contribution can be valuable, and I particularly appreciate the effort to investigate theoretical properties. As is, the case is not sufficiently convincing to be accepted, but I encourage the authors to improve the paper.\n\nMinor comments:\n1. While I appreciate a pun, I would recommend to rename the model along with the acronym to a more concise name.\n2. Please revise your notation and typsetting. Examples: x1 instead of x_1, f of f(\\cdot) instead of f(), \\log instead of log.\n3. Introduce acronyms before using them (e.g. VAE, MSE, SSIM), even when they seem obvious to you.\n4. Please carefully check the manuscript for typos, missing articles, missing spaces etc.\n5. Your citations are inconsistent, in that they sometimes use first names, sometimes first name initials, and sometimes no first names.\n6. To my knowledge, the term scale function does not have an obvious definition. I think you are simply referring to monotonically increasing functions. Please clarify!\n7. Your figures should be understandable without too much context, they need more detailed captions.\n\n[1] http://proceedings.mlr.press/v84/chen18e.html", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJewScvRYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to obtain latent representation of data such that probability density for the real space can be calculated correctly from that in the latent space. The authors optimize a loss function that has components related to parametric probabilistic distribution and auto encoder simultaneously. While this might be an important problem (I am not sure), the paper is not written and organized well which makes a through evaluation very difficult. I provide below some of the problems with this the paper:\n\nWhy the introduced method is better than VAE as a generative model for capturing the latest representation is not explained well. It is not also used as a baseline in most of the experiments.\n\nThe motivation for having the third term in Equation (4) needs to be explained. Also what is h() in the second term. The authors only describe briefly both terms together after they used it here but failed to describe what each term is.  Why there is an h for the second term but not for the third term. h() becomes more clear much later in the paper but when it is used the first time, it not defined. \n\nI believe A in Equation (5) should be also positive-definite. \n\nWhat is L(x) in Equation (8).  It needs to be defined.\n\n\nExperiments:\n1-\tIt is useful to also plot the original data in space s to see how the results in Figure 2 make sense. \n2-\tFigure 3 is not clear.\n3-\tIn the Anomaly detection experiments, the authors make two assumptions that usually do not exist in real-worlds: (1) they assume that they have access to training set that only contains normal cases. (2) They assume that they know the correct rate of anomaly. I think both these assumptions are very restrictive and unreal. While these assumptions are used for all the comparing methods, it is not obvious how different algorithms behave in real scenario. \n4-\tFigure 4 and what it represents is not clear. \n\nWriting Problems:\n1-\tIn the text of paragraph before Figure 1, Eq. (5) in “in the second term of Eq. (5)” is a typo and should be Eq. (4). \n2-\tIn the paragraph before Figure 1, the following sentence is not complete: “Then, averaging Eq. (4) according to distribution, x~P_x(x) and epsilon~ P(epsilon).”\n3-\tSection 4.2.1: “there is a difference is PDF → “there is a difference in PDF”", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Agreement by the reviewers: although the idea is good, the paper is very hard to read and not accurately enough formulated to merit publication.  

This can be repaired, and the authors should try again after a thorough revision and rewrite.
Decision: Reject