RATE-DISTORTION OPTIMIZATION GUIDED AUTOEN-
Year: 2019

Abstract:
In the generative model approach of machine learning, it is essential to acquire an accurate probabilistic model and compress the dimension of data for easy treatment. However, in the conventional deep-autoencoder based generative model such as VAE, the probability of the real space cannot be obtained correctly from that of in the latent space, because the scaling between both spaces is not controlled. This has also been an obstacle to quantifying the impact of the variation of latent variables on data. In this paper, we propose Rate-Distortion Optimization guided autoencoder, in which the Jacobi matrix from real space to latent space has orthonormality. It is proved theoretically and experimentally that (i) the probability distribution of the latent space obtained by this model is proportional to the probability distribution of the real space because Jacobian between two spaces is constant; (ii) our model behaves as non-linear PCA, where energy of acquired latent space is concentrated on several principal components and the influence of each component can be evaluated quantitatively. Furthermore, to verify the usefulness on the practical application, we evaluate its performance in unsupervised anomaly detection and it outperforms current state-of-the-art methods.

heading: 1 INTRODUCTION
text: Capturing the inherent features of a dataset from high-dimensional and complex data is an essential issue in machine learning. Generative model approach learns the probability distribution of data, aiming at data generation by probabilistic sampling, unsupervised/weakly supervised learning, and acquiring meta-prior (general assumptions about how data can be summarized naturally, such as disentangle, clustering, and hierarchical structure (Bengio et al., 2013; Tschannen et al., 2019)). It is generally difficult to directly estimate a probability density function(PDF) Px(x) of real data x. Accordingly, one promising approach is to map to the latent space z with reduced dimension and capture PDF Pz(z). In recent years, deep autoencoder based methods have made it possible to compress dimensions and derive latent variables. While there is remarkable progress in these areas (van den Oord et al., 2017; Kingma et al., 2014; Jiang et al., 2016), the relation between x and z in the current deep generative models is still not clear.
Variational autoencoder (VAE) (P.Kingma & Welling, 2014) is one of the most successful generative models for capturing latent representation. In VAE, lower bound of log-likelihood of Px(x) is introduced as ELBO. Then latent variable is obtained by maximizing ELBO. In order to maximize ELBO, various methods (Alemi et al., 2018; Zhao et al., 2019; Brekelmans et al., 2019) have been proposed. However, many previous works did not care about the value of Jacobian between two spaces, despite the fact that the ratio between Pz(z) and Px(x) is equal to the Jacobian. Even in models that provide more flexible estimation (Johnson et al., 2016; Liao et al., 2018; Zong et al., 2018), this point is overlooked.
Here, when we turn our sight toward acquiring meta-prior, it is straightforward to evaluate the quantitative influence of each latent variable on the given metrics between data x1 and x2. To do so, the scale of the latent variable should be appropriately controlled so that the changes in latent variables is proportional to the changes of given metrics in data space. In addition, this scaling should be adjusted according to the definition of the metrics. For instance, with respect to image quality metrics, different meta-prior should be derived from mean square error (MSE) and the structural similarity (SSIM). Considering the mechanism of principal component analysis (PCA) would be one of the
directions to solve it. In PCA, PCA components are derived by optimal orthonormal transformation, then the importance of each component can be identified quantitatively by their variance. It should be noted that orthogonality is not enough for quantitative analysis, but both orthogonality and normalized scaling, namely, orthonormality is required. This fact implies that, if Jacobi matrix of autoencoder has orthonormality, the characteristics of acquired latent space can be evaluated quantitatively. To deal with this, we propose RaDOGAGA (Rate-Distortion Optimization Guided Autoencoder for Generative Approach), based on the rate-distortion optimization (RDO) which has been widely used in image compression with orthonormal transform coding (Sullivan & Wiegand (1998)). In this paper, we show the effect of RaDOGAGA in the following steps.
(1) We prove that RaDOGAGA has the following property theoretically and experimentally.
• Jacobi matrix between real space and latent space leads to be constant-scaled orthonormal. So the response of the minute change of z to the real space data x is constant at any z.
• Because of constant Jacobian (or pseudo-Jacobian), Px(x) and Pz(z) are almost proportional. Therefore, Px(x) can be estimated, by directly maximizing log-likelihood of parametric PDF Pzψ(z) in reduced-dimensional space, without considering ELBO. • When univariate independent distribution is used to estimate Pz(z) parametrically, it behaves as ”continuous PCA” where energy is concentrated on several principal components.
(2) Thanks to this property, RaDOGAGA achieve the state-of-the-art performance in anomaly detection task with four public datasets, where probability density estimation is important.
(3) We show that our approach can directly evaluate how the z impact on the given metrics in real space. This feature is promising to further interpretation of latent variables.

heading: 2 RELATED WORK
text: Flow based model: Flow based generative models generates astonishing quality of image (Kingma & Dhariwal, 2018; Dinh et al., 2014). Flow mechanism explicitly takes Jacobian of x and z into account. The transformation function z = f(x) is learned, calculating and preserving Jacobian of x and z. Unlike ordinary autoencoder, which reverse z to x with function g(·) different from f(·), inverse function transforms z as x = f−1(z). Since the model preserves Jacobian, Pz(x) can be estimated by maximizing log likelihood of Pz(z) without considering ELBO. Although, in this approach, f(·) need to be bijection. Because of this limitation, it is difficult to fully utilize the flexibility of neural networks.
Interpretation of latent variables: While it is expected to acquire meta-prior by deep autoencoder, interpreting latent variables is still challenging. In recent years, research aiming to acquire disentangled latent variables, which encourages them to be independent, is flourishing (Lopez et al., 2018; Chen et al., 2018; Kim & Mnih, 2018; Chen et al., 2016). With these methods, qualitative effects for disentanglement can be seen. For example, when a certain latent variable is displaced, image changes corresponding to specific attributes (size, color, etc.). Some works also propose quantitative metrics for meta-prior. In beta-VAE (Higgins et al., 2017), the metric evaluates the independence of latent variables by solving the classification task. Actually, the Jacobi matrix of VAE is orthogonal (Rolı́nek et al., 2019), which enables to make latent variables disentangled implicitly. However, orthonormality is not supported and it is difficult to define the metric which evaluates the effect of latent variables to the metrics between data directly.
Deep image compression with rate-distortion optimization: Rate-distortion (RD) theory (Berger, 1971) is a part of Shannon’s information theory for lossy compression which formulates the optimal condition between information rate and distortion. Then RD theory for Gaussian source with memory has been further extended to transform coding (Goyal, 2001) for image and audio lossy compression where orthonormal transforms such as Karhunen-Loève transform (KLT) (Rao & Yip (2000)) and discrete cosine transform (DCT) are used for decorrelation. Furthermore, rate-distortion optimization (RDO) has been widely used in image compression (Sullivan & Wiegand, 1998). In RDO, a cost L = R+λ ·D is minimized at given Lagrange parameter λ to realize the best trade-off between rate R and distortion D such that L = R + λ · D becomes a tangent line of true RateDistortion curve. Recently, deep learning based image compression (Ballé et al., 2018; Zhou et al., 2019) has been proposed. In these works, instead of orthonormal transform with L2 norm metrics in the conventional lossy compression method, a deep autoencoder is trained with flexible metrics
such as SSIM for RDO. To understand the success of these works, we prove theoretically that RDO guides a Jacobi matrix of deep autoencoder to orthonormal in the space depending on the metrics. In the next section, we explain the idea of RDO guided autoencoder and its relationship with VAE.

heading: 3 OVERVIEW OF RATE-DISTORTION OPTIMIZATION GUIDED APPROACH

heading: 3.1 DERIVATION FROM RATE-DISTORTION OPTIMAZATION IN TRANSFORM CODING
text: Figure 1 shows the overview of our idea inspired by Rate-Distortion optimization of transform coding. In the transform coding, the optimal method to encode data with Gaussian distribution is as follows (Goyal, 2001). At first, the data are transformed deterministically to decorrelated data using the orthonormal transform such as KLT and DCT. Then these decorrelated data are quantized stochastically with uniform quantizer for all channels such that quantization noise for each channel is equal. At last optimal entropy encoding is applied to quantized data where the rate can be calculated by the logarithm of symbol’s estimated probability. From this fact, we have an intuition that the Jacobi matrix of autoencoder becomes orthonormal if the data were compressed based on RateDistortion optimization with uniform quantized noise and parametric distribution of latent variables. Inspired by this, we propose autoencoder which scales latent variables according to the definition of metrics of data, without limitation of the transformation function as in Flow models. Thanks to this feature, our scheme can estimate Pz(z) quantitatively, which is suitable for clustering and anomaly detection. Furthermore, in the case factorized distribution is used for Pz(z), our model behaves as continuous PCA. This property is considered to promote the interpretation of latent variables.

heading: 3.2 RELATIONSHIP WITH VAE
text: There is a number of VAE studies that taking rate-distortion trade-off into account. In VAEs, instead of maximizing log-likelihood of P (x) directly, maximizing ELBO. In beta-VAE (Higgins et al., 2017), Objective function LV AE is described as LV AE = Lrec − β · Lkl. Lkl is KL divergence between encoder output and prior distribution, usually Gaussian distribution. By changing β, the rate-distortion trade-off at desirable rate can be realized as discussed in (Alemi et al., 2018).
When the relation between beta-VAE and RDO in image compression is considered, both are analogous to each other. That is, β−1 is corresponding to λ, −Lkl is corresponding to a rate R, and Lrec is corresponding to a distortion D. However, the probability distribution models of latent variables are quite different. VAE uses a fixed prior distribution. This causes a non-linear scaling relationship between real data and latent variables.
Figure 2 shows the conditions to achieve Rate-Distortion optimization in both VAE and RDO guided autoencoder. In VAE, precise control of noise distribution is required for each data and channel as suggested in Brekelmans et al. (2019) in order to compensate for the nonlinearities between the data space and the latent space. Jacobi matrix becomes orthogonal as proved in Rolı́nek et al. (2019). However, Jacobian is not constant because of non-linear scaling. In RDO guided autoencoder, uniform noises are added to all channels. Instead, parametric probability distribution should be used in order to make the scaling linear between the data space and the latent space. As a result, the Jacobi matrix becomes orthonormal because both orthogonality and scaling normalization are simultaneously achieved. As discussed above, the precise noise control in VAE and distribution parameter optimization in RDO guided autoencoder are essentially the same. Accordingly, complexities in both methods are estimated to be at the same degree.

heading: 4 METHOD AND THEORETICAL PROPERTY

heading: 4.1 METHOD
text: Our model is based on the Rate-Distortion optimization of the autoencoder for the hyperprior proposed in Ballé et al. (2018) for image compression with some modification. The cost function consists of (i) reconstruction errorD between input data and decoder output with noise to latent variable and (ii) Rate R of latent variable. This is analogous to beta-VAE where λ = β−1.
L = R+ λ ·D (1)
The specific method is described as follows. First, let x be M -dimensional domain data(x ∈ RM ) and Px(x) the probability of x. Then x is converted to N -dimensional latent space z ∈ RN by encoder. Let fθ(x), gϕ(z), and Pzψ(z) be parametric encoder, decoder, and probability distribution function of latent space with parameters θ, ϕ, and ψ. It is noted that encoder and decoder are deterministic unlike VAE. Then latent variable z and decoded data x̂ are generated as bellow:
z = fθ(x) x̂ = gϕ(z) (2)
Let ϵ ∈ RN be noise with each dimension being independent with an average of 0 and a equal variance of σ2 to emulate uniform quantization.
ϵ = (ϵ1, ϵ2, ..ϵN ), E [ϵi] = 0, E [ϵi · ϵj ] = δij · σ2 (3) Then, given the sum of latent variable z and noise ϵ, the decoder output x̆ is obtained. This is analogous to the stochastic sampling and decoding procedure in VAE.
x̆ = gϕ(z + ϵ) (4)
Here, the cost function is defined by Eq. (5) with some modifications from Eq. (1).
L = − log(Pzψ(z)) + λ1 · h (D (x, x̂)) + λ2 ·D (x̂, x̆) (5) In this equation, the first term is corresponding to the estimated rate of the latent variables. In Ballé et al. (2018), a rate for quantized symbol is calculated by − ∫ 0.5 −0.5 log(Pzψ(z))dz. In our model, this is replaced by − log(Pzψ(z)) for simplicity. To model Pzψ(z), factorized probability model or Gaussian mixture model (GMM) can be used for instance. D(x1,x2) in the second and the third term is a metrics function between x1 and x2. Actually, the second and the third term is decomposition ofD(x, gϕ(fθ(x)+ϵ)) ∼ D(x, gϕ(fθ(x)))+D(gϕ(fθ(x)), gϕ(fθ(x)+ϵ)) as shown in Rolı́nek et al. (2019). The second term purely calculate reconstruction loss and the third term only influences the scaling of latent space. Accordingly, λ1 controls the degree of reconstruction, and λ2(∼ β−1 of beta-VAE) controls a scaling between the data space and the latent space. By this decomposition, we can adjust balance between reconstruction loss and other losses flexibly and easily lead better performance. h(·) in the second term of Eq. (5) is a monotonically increasing function. According to the choice of h(·), the degree of reconstruction accuracy can be controlled. Roughly speaking, when h(d) = d is chosen, the training much focus on fitting the distribution and entropy. On the other hand, h(d) = log(d) encourages better reconstruction, since the curve of loss is steep around 0. The effect of h(d) is further discussed in Appendix D. Then, Eq. (5) is averaged according to distributions, x ∼ Px(x) and ϵ ∼ P (ϵ). For metrics function D(·, ·), a variety of function is applicable such as MSE, SSIM, and so on. As long as the function can be approximated by the following quadratic form in the neighborhood of x, the property of the model is hold
D(x,x+ δx) ∼ tδx A(x) δx = ∥L(x) δx∥2 (6) Here, δx is arbitrary micro variation of x, A(x) is M ×M positive definite matrix depending on x , and L(x) is Cholesky decomposition of A(x). For instance, when D(·, ·) is square of L2 norm as in Eq. (7), A(x) and L(x) are Identity matrices.
D(x1,x2) = ∥x1 − x2∥2 (7)
In the case of SSIM (Wang et al., 2001) metric which is close to subjective image quality, a cost (1 − SSIM) can be also approximated in a quadratic form with positive definite matrix. This is explained in Appendix C. By deriving parameters that minimize this value, the encoder, decoder, and probability distribution of the latent space are trained as Eq. (8).
θ, ϕ, ψ = arg min θ,ϕ,ψ (Ex∼Px(x), ϵ∼P (ϵ)[ L ]) (8)

heading: 4.2 THEORETICAL PROPERTY OF THE MODEL
text: In this section, we show the theoretical property of the model. First of all, we provide a rough sketch of the theory. Let J = |dx/dz| be a Jacobian between the data space x and the latent space. Here, R and λD in Eq. (1) can be approximated by − logPx(x)− log(J ) and Cσ,λJ2 respectively where Cσ,λ is a constant depending on σ and λ. As a result, the Jacobi matrix with J = 1/ √ 2Cσ,λ minimizes Eq. (1). Because Jacobi matrix is orthogonal (Rolı́nek et al., 2019) and the Jacobian is constant, the Jacobi matrix becomes orthonormal. A full proof is described in Appendix A. In this section, we mainly explain important points. Let xD be rescaled space of x according to the metrics D(·, ·), where dx and dxD are differentials and L(x) is a Jacobi matrix between two spaces.
dxD = L(x)dx (9)
When D(·, ·) is square of L2 norm, both spaces of xD and x are equivalent. As described above, each column of the Jacobian matrix of latent space z and rescaled space xD are constant multiple of orthonormal basis regardless of the value of z and x after training based on Eqs. (5) and (8). Here, δij denotes Kronecker delta and matrix tT denotes a transpose of T .
t ( L(x) ∂x
∂zi
)( L(x) ∂x
∂zj ) = t ( ∂xD ∂zi )( ∂xD ∂zj ) = 1 2λ2σ2 · δij (10)
From the orthonormal property, our model has several features. The first feature is that The distance derivatives of both real and latent spaces has a linear relationship. Therefore, the metrics between x and x+dx is proportional to ∥dz∥2 as derived by Eqs. (6), (9) and (10) where dz is a differential.
D(x,x+ dx) = ∥L(x)dx∥2 = ∥dxD∥2 = ∥∥∥∥∥ N∑ i=1 ∂xD ∂zi · dzi ∥∥∥∥∥ 2 = 1 2λ2σ2 · ∥dz∥2 (11)
D(x,x+ dx)/∥dz∥2 = ∥dxD∥2/∥dz∥2 = const. (12) This equation also implies that the L2 distance of two data in the latent space is proportional to the distance between corresponding data in the real space which is defined by the integration of the root of metrics derivative.
The second feature is constant Jacobian. In case of M = N , the Jacobian matrix dxD/dz is a square matrix, and each column is the same as 1/( √ 2λ2 · σ) times orthonormal basis. For this reason, the Jacobian is a constant regardless of the value of z as shown below:∣∣∣∣dxDdz ∣∣∣∣ = ( 12λ2 · σ2 )(N/2) (13)
In this case, the probability distribution of xD and z is proportional because of the constant Jacobian. For the case of M > N , we assume the situation where most energy is efficiently and effectively mapped to N-dimensional latent space. Then the product of the singular values of SVD for a Jacobi
matrix can be regarded as a pseudo Jacobian between the real space and the latent space. Since all of the N singular values are 1/( √ 2λ2 · σ), the pseudo Jacobian is also a constant.
The third feature is proportional probability density between data space and latent space. Because Jacobian or pseudo Jacobian J is constant, the following equation holds. PxD(xD) and Pz(z) are are true PDFs for each space xD and z respectively.
Pz(z) ∼ J · PxD(xD) ∝ PxD(xD) (14)
The forth is that the log-likelihood of PDF of the domain data can be maximized without considering ELBO. This is revealed by the transformation of Equation (5). Let P̂xD(xD) be estimated probability of xD. Because the Jacobian J is constant, P̂xD(xD) can be approximated by J−1 ·Pzψ(z). Accordingly, the average of the first term in Equation (5) by x ∼ Px(x) can be transformed as follows.
Ez∼Pz(z))[− log(Pzψ(z))] ≃ −ExD∼Px(xD))[log(P̂xD(xD)]− log(J) (15) Consequently, minimization of Equation (15) is equivalent to the log-likelihood maximization of P̂xD(xD).
The fifth is ”continuous PCA” feature when the following factorized probability model is used.
Pzψ(z) = N∏ i=1 Pziψ(Zi) (16)
As shown in FactorVAE (Kim & Mnih, 2018), the use of factorized model minimizes mutual information among latent variables and encourages disentangling. Combining with the first character of our model described before, the variance values of each latent variable show the quantitative impact on data space like linear PCA. The detailed derivation is explained in Appendix B. .

heading: 5 EXPERIMENT

heading: 5.1 PROBABILITY DENSITY ESTIMATION WITH TOY DATA
text: In this section, we describe our experiment using toy data to demonstrate whether the probability density of the input data Px(x) and that of estimated in the latent space Pz(z) are proportional to each other as in theory. First, we sample data s = (s1, s2...s10,000) from three-dimensional Gaussian distribution consists of three-mixture-components with mixture weight π = (1/3, 1/3, 1/3), mean µk = (µk1, µk2, µk3), and covariance Σk = diag(σk1, σk2, σk3). k is the index for mixture component. Then, we scatter s with uniform random noise u ∈ R3×16, udm ∼ Ud ( − 12 , 1 2 ) , where d and m are index for dimension of sampled data and scattered data. The Uds are uncorrelated with each other. We produce 16-dimensional input data x with a sample number of 10,000 in the end.
x = 3∑ d=1 udsd (17)
The appearance probability of the input data Px(x) is equivalent to the generation probability of s.

heading: 5.1.1 CONFIGURATION
text: In the experiment, we estimate the Pzψ(z) using GMM with parameter ψ as in DAGMM (Zong et al., 2018). Instead of EM algorithm, GMM parameters are learned using Estimation Network (EN), which consists of multi-layer neural network. When the GMM has K mixturecomponents and L is the size of batch samples, EN outputs the mixture-components membership prediction as K-dimensional vector γ̂ as follows:
p = EN(z;ψ), γ̂ = softmax(p) (18)
K-th mixture weight ϕ̂k, mean µ̂k, covariance Σ̂k, and entropy R of z are further calculated by Eqs. (19) and (20).
π̂k = L∑ l=1 γ̂lk/L, µ̂k = L∑ l=1 γ̂lkzl/ L∑ l=1 γ̂lk, Σ̂k = L∑ l=1 γ̂lk(zl − µ̂k)(zl − µ̂k)T / L∑ l=1 γ̂lk (19)
R = − log ( L∑ k=1 π̂k/ √∣∣∣2πΣ̂k∣∣∣ · exp(−1 2 (z − µ̂k)T Σ̂−1k (z − µ̂k) )) (20)
Overall network is trained by Eqs. (5) and (8). In this experiment, we set D(x1, x2) as square error ∥ x1 − x2 ∥2, and test two types of h(·), h(d) = d and h(d) = log(d). We denote models trained with these h(·) as RaDOGAGA(d) and RaDOGAGA(log(d)). As a comparison method, DAGMM is used. DAGMM also consists of encoder, decoder, and EN. In DAGMM, to avoid falling into the trivial solution that entropy is minimized when the diagonal component of the covariance matrix is 0, the inverse of the diagonal component is added to the cost function as Eq. (21):
L = ∥x− x̂∥2 + λ1 · (− log(Pzψ(z))) + λ2P (Σ̂), P (Σ̂) = K∑ k=1 N∑ i=1 Σ̂−1kii (21)
The only differences between our model and DAGMM are (i) RDO mechanism is introduced (ii) the third regularization term in Eq. (21) is eliminated, since our model adjust latent scale appropriately so as not to falling into trivial solution. Thus, model complexity such as the number or parameter is the same. For both RaDOGAGA and DAGMM, the autoencoder part is constructed with fully connected (FC) layers with sizes of 64, 32, 16, 3, 16, 32, and 64. For all FC layers except for the last of the encoder and the decoder, we attach tanh as the activation function. The EN part is also constructed with FC layer with a size of 10, 3. For the first layer, we attach the tanh as activation function and dropout (ratio=0.5). For the last one, softmax is attached. (λ1, λ2) is set as (1× 10−4, 1× 10−9), (1× 106, 1× 106) and (1× 103, 1× 104) for DAGMM, RaDOGAGA(d) and RaDOGAGA(log(d)) respectively. Optimization is done by Adam optimizer (Kingma & Ba, 2014) with learning rate 1× 10−4 for all model. We set σ2 as 1/12.

heading: 5.1.2 RESULT
text: Figure 4 displays the distribution of input data source s and latent variable z. Even though both methods can capture that s is generated from three mixture-component, there is a difference in PDF. Since the data is generated from GMM, the sample getting closer to the center of clustering, the PDF getting higher. Although, in DAGMM, this tendency looks distorted. This difference is further demonstrated in Fig 5 which shows a plot of Px(x)(x-axis) against Pzψ(z)(y-axis). In our method, Px(x) and Pzψ(z) are approximately proportional to each other as in theory while we
cannot observe such proportionality in DAGMM. This difference is quantitatively obvious as well. That is, correlation coefficients between Px(x) and Pzψ(z) are 0.691 (DAGMM), 0.997 (RaDOGAGA(d)), and 0.999 (RaDOGAGA(log(d))). It also turns out that the correlation coefficient of RaDOGAGA(log(d)) is higher than that of RaDOGAGA(d). Actually, when Pzψ(z) is sufficiently fitted, h(d) = log(d) makes Px(x) and Pzψ(z) be proportional more rigidly. On the other hand, h(d) = d makes the scale of latent space slightly bent in order to minimize entropy function, allowing relaxed fitting of Pzψ(z). More detail is described in Appendix D.

heading: 5.2 ANOMALY DETECTION TASK USING REAL DATA
text: In this section, we examine whether the clear relationship between Px(x) and Pz(z) is useful in the anomaly detection task using real data. We use four public datasets1, KDDCUP99, Thyroid, Arrhythmia, and KDDCUP-Rev. The (instance number, dimension, anomaly ratio(%)) of each dataset is (494021, 121, 20), (3772, 6, 2.5), (452, 274, 15), and (121597, 121, 20) respectively. Detail of datasets is described in Appendix E.

heading: 5.2.1 EXPERIMENTAL SETUP
text: We follow the setting in Zong et al. (2018). Randomly extracted 50% of the data is assigned to training and the rest to testing. During training, only normal data is used. During the test, the R for each sample is calculated as the anomaly score, and if the anomaly score is higher than a threshold, it is detected as an anomaly. The threshold is determined by the ratio of anomaly data in each data set. For example, in KDDCup99, data with R in the top 20 % is detected as an anomaly. As metrics, precision, recall, and F1 score are calculated. We run 20 times for each dataset split by 20 different random seeds.

heading: 5.2.2 BASELINE MODEL
text: Same as in the previous section, DAGMM is taken as the baseline method. We also compare with the scores reported in previous works in which same experiments were conducted (Zenati et al., 2018; Song & Ou, 2018; Liao et al., 2018).

heading: 5.2.3 CONFIGURATION
text: As in Zong et al. (2018), in addition to the output from the encoder, ∥x−x ′∥2 ∥x∥2 and x·x′
∥x∥2∥x′∥2 are concatenated to z. It is sent to EN. Note that z is sent to the decoder before concatenation. Other configuration except for hyper parameter is same as in the previous experiment. Hyper parameter for each dataset is described in Appendix E.

heading: 5.2.4 RESULTS
text: Table 1 reports the average scores and standard deviations (in brackets). Comparing DAGMM and RaDOGAGA, RaDOGAGA has a better performance regardless of types of h(·). Note that, our model does not increase model complexity at all. Simply introducing the RDO mechanism into the autoencoder has a valid efficacy for anomaly detection. Moreover, our approach achieves stateof-the-art performance compared to other previous works in which same datasets is used. Clear relationship between Px(x) and Pz(z) by our model is considered to be effective in the task of anomaly detection where the estimating probability distribution is important. In RaDOGAGA, when we compare result of RaDOGAGA(d) and RaDOGAGA(log(d)), either of one is not always superior. As described in section 5.1 and Appendix D, h(·) can be an option depending on fitting flexibility of Pz(z).

heading: 5.3 QUANTIFYING THE IMPACT OF LATENT VARIABLES ON METRICS AND BEHAVIOR AS PCA
text: In this section, we confirm that (i) the latent variables in our model behaves as PCA components and (ii) the impact of each latent variable on the metrics function can be quantified. When dz in Eq.
1Dataset can be dowonload from (https://kdd.ics.uci.edu/) and (http://odds.cs.stonybrook.edu)
(12) is approximated by a small displacement δ, the ratio D(g(z), g(z + δ))/∥δ∥2 will be almost constant regardless of z and δ. If the ratio of each component zi in the latent space is equivalent, the latent space can be regarded as isometric. So, we evaluate this ratio in each component. Let δi be a vector t(0, · · · , ϵ, ·, 0) where only i-th component has a minute value ϵ. Then D′i(z) denotes D(g(z),g(z+δi)) ϵ2 for i-th component. In the trained model, we encode the image xl and obtain zl. Then, D′i(zl) is calculated for each sample. Finally, the average across all samples is measured. This operation is conducted to each dimension of z independently. We also observe the distribution of z and how the image looks different in response to the variation of component zi. To train model, we use CelebA dataset (Liu et al., 2015), which consists of 202,599 celebrity images. The images are center-cropped so that the image size is 64 x 64.

heading: 5.3.1 CONFIGURATION
text: In this experiment, factorized distributions (Ballé et al., 2018) is used to estimate Pzψ(z). For comparison, we evaluate beta-VAE. Both models are constructed with same depth of CNN and FC layers, with 256-dimensional z. Detail of networks and hyper parameter is written in Appendix F. For RaDOGAGA, we set D(x1,x2) as 1 − SSIM(x1,x2) and h as h(d) = d. For beta-VAE, reconstruction loss is also 1− SSIM(x1,x2). SSIM(x1,x2) is defined by Eq. (61).

heading: 5.3.2 RESULT
text: Both models are trained so that the SSIM between input and reconstructed image is around 0.93. Figure 6a and 6b show the variance of each component in the latent variables z arranged in descending order. The red line is the cumulative relative ratio of the variance. In Fig. 6b, variance is concentrated in a specific dimension. On the other hand, in Fig. 6a, beta-VAE is trained so that each
latent variable is fitted to a Gaussian distribution with mean 0 and variance 1, there is no significant difference in the variance of each latent variable. Figures 6c and 6d respectively depicts the average of D′i(z) of each of the top nine dimensions with the largest variance of z. In beta-VAE, the D′i(z) varies drastically depending on the dimension i which shows anisometric latent space, while in RaDOGAGA, it is approximately constant regardless of i which shows isometric latent space. Figure 7 shows decoder outputs when each component zi is traversed from -2σ to 2σ, fixing rest of z as mean. From the top, each row corresponds to z0, z1, z2 ..., and the center column is mean. In Fig. 7b, the image changes visually in any dimension of z, while in Fig. 7a, depending on the dimension i, there are cases where no significant changes can be seen (such as z3, z4, and so on). This result means that, in RaDOGAGA, the variance of z directly corresponds to the visual impact and the metrics D(x1,x2), behaving as PCA. Besides, since D′i(z) is constant, the latent space shows isometric feature and the variance is regarded as a quantitative importance.

heading: 6 CONCLUSION
text: In this paper, we propose RaDOGAGA that learns parametric probability distribution and autoencoder simultaneously based on rate-distortion optimization. It was proven that the probability distribution of the latent variables obtained by the proposed method is proportional to the probability distribution of the input data theoretically and experimentally. This property is validated in anomaly detection achieving state-of-the-art performance. Moreover, our model has the trait as PCA which likely promotes interpretation of latent variables. For the future work, we will conduct experiments with different types of metrics functions that derived from semantical task, such as in categorical classification. Meanwhile, as mentioned in Tschannen et al. (2019), considering the usefulness of latent variables in downstream task is another research direction to explore.

heading: A HOW JACOBI MATRIX BECOME A CONSTANTLY SCALED ORTHONORMAL BASIS
text: In this appendix section, we prove that all of column vectors in decoder’s Jacobi Matrix have the same norm and are orthogonal to each other. In other words, each column of Jacobi matrix is a constantly scaled orthonormal basis.
Here we assume that data space sample x has M dimension, that is x ∈ RM , then encoded to N dimensional latent space sample y ∈ RN . We also assume that an fixed encoder function y = finit,θ(x) and a fixed decoder x̂ = ginit,ϕ(y) are given such that h(D(x, x̂)) becomes minimal.
y = finit,θ(x) (22) x̂ = ginit,ϕ(y) (23)
s.t h(D(x, x̂)) =⇒ minimal
We further assume that fixed parametric PDF of latent variable y is also given.
Pyinit,ψ(y) (24)
Here, it is noted that this function is not needed to be optimal in a sense of DKL (Py (y) ∥Pyinit,ψ (y)) where Py (y) is an actual PDF of y. Under these conditions, we introduce new latent variable z, and y is transformed from z using a following scaling function a(z) : RN →RN .
y = a(z) = (a1(z), a2(z), · · · , aN (z)) (25)
Here, our goal is to prove Eq. (10) by examining the condition of this scaling function which minimize the average of Eq. (5) with regard to ϵ ∼ P (ϵ). Because of the assumption of minimal D(x, x̂), we ignore the second term of Eq. (5). Next, the PDF of z can be derived using a Jacobian of a(z).
Pzψ,a(z) = ∣∣∣∣da(z)dz ∣∣∣∣ · Pyinit,ψ(a(z)) (26)
By applying these conditions and equations to Eq. (5), the cost of scaling function a(z) to minimize the average is expressed as follows.
La = − log (∣∣∣∣da(z)dz ∣∣∣∣ · Pyinit,ψ(a(z)))+ λ2 ·D(ginit,ϕ(a(z + ϵ)), ginit,ϕ(a(z))) (27)
Next, the latent variable is fixed as y0, and z0 is defined in the next equation.
z0 = a −1(y0) (28)
Then the average of Eq. (27) with regard to ϵ ∼ P (ϵ) is expressed as follows.
Eϵ∼P (ϵ) [La|z=z0 ] = − log (∣∣∣∣da(z)dz ∣∣∣∣ z=z0 · Pyinit,ψ(a(z0)) ) + λ2 · Eϵ∼P (ϵ) [D(ginit,ϕ(a(z0 + ϵ)), ginit,ϕ(a(z0)))]
(29)
Then the condition of a(z0) in the neighborhood of y0 is examined when Eq. (29) is minimized. Here, it is noted that the first term of the right side doesn’t depend on ϵ.
Before examining Eq. (29), some preparation of equations are needed. At first, Jacobi matrix of a(z) at z = z0 is defined as B using notations of partial differentials bij = ∂ai∂zj ∣∣∣ z=z0 .
B = ∂y
∂z ∣∣∣∣∣ z=z0 = da(z) dz ∣∣∣∣∣ z=z0 =  b11 b12 . . . b1N b21 b22 . . . b2N ... ... . . . ...
bN1 bN2 . . . bNN  where bij =
∂ai ∂zj ∣∣∣∣∣ z=z0
(30)
The vector bi is also defined as follows.
bi = ∂y
∂zj ∣∣∣∣∣ z=z0 =  b1i b2i ... bNi  (31) It is clear by definition that bi and B have the following relation.
B = (b1, b2, · · · , bN ) (32)
∆ij is defined as a cofactor of matrix B with regard to the element bij , and b̃i is also defined by the following equation.
b̃i =  ∆1i ∆2i
... ∆Ni  (33) The following equations hold by the definition of cofactor.
tbi b̃i = N∑ k=1 bki ·∆ki = |B| (34)
d|B| dbij = ∆ij , d|B| dbi = b̃i (35)
In case of i ̸= j, inner product of bi and b̃j becomes zero because this value is a determinant of a singular matrix with two equivalent column vectors bi.
tbi b̃j = N∑ k=1 bki ·∆kj = |(b1, · · · , bi, · · · , bi, · · · )| = 0 (36)
G′init is defined as a M ×N Jacobi matrix of ginit,ϕ(y) at y = y0 as follows.
G′init = dginit,ϕ(y)
dy
∣∣∣∣∣ y=y0
(37)
Using these equations, we proceed to expand of Eq. (29). The first term of (29) in the right side can be expressed as follows, where the second term Eq. (38) in the right side is constant by definition.
− log (∣∣∣∣da(z)dz ∣∣∣∣ z=z0 · Pyinit,ψ(a(z0)) ) = − log(|B|)− log(Pyinit,ψ(y0)) (38)
Next step is an expansion of the second term in Eq. (29). First, the following approximation holds.
ginit,ϕ(a(z0 + ϵ))− ginit,ϕ(a(z0)) ∼ dginit,ϕ(y) dy ∣∣∣∣∣ y=y0 (da(z) dz ∣∣∣∣∣ z=z0 ) ϵ
= G′init B ϵ = G′init (ϵ1 · b1 + ϵ2 · b2 · · ·+ ϵN · bN )
= N∑ i=1 ϵi ·G′initbi (39)
Then, the second term of Eq. (29) can be transformed to the next equation by using Eqs. (39), (3), and (6).
Eϵ∼P (ϵ) [D(ginit,ϕ(a(z0 + ϵ)), ginit,ϕ(a(z0))] ∼ Eϵ∼P (ϵ) ∥∥∥∥∥L(x) N∑ i=1 ϵi ·G′init bi ∥∥∥∥∥ 2 
= N∑ i=1 E[ϵi 2] · ∥L(x)G′init bi∥2
+2 · N∑ i=1 N∑ j=i+1 E[ϵi · ϵj ] ·
t(L(x)G′init bi)(L(x)G ′ init bj)
= σ2 · ( N∑ i=1 ∥L(x)G′init bi∥2 )
(40)
As a result, Eq. (29) can be rewritten as Eq. (41).
Eϵ∼P (ϵ) [La|z=z0 ] ∼ − log(|B|)− log(Pyinit,ψ(y0))
+λ2 · σ2 · ( N∑ i=1 ∥L(x)G′init bi∥2 )
(41)
By examining the minimization process of Eq. (41), the conditions of optimal scaling function y = a(z) in the neighborhood of y0 is clarified. Here, the condition of Jacobi matrix B is examined instead of a(z). Eq. (41) is differentiated by vector bi, and the result is set to be zero. Then the following equation Eq.(42) is derived.
2λ2 · σ2 · ( t(L(x)G′init)(L(x)G ′ init) ) bi = 1
|B| · b̃i (42)
Afterwards, Eq. (42) is multiplied by tbj from the left, and divided by 2λ2 ·σ2. As a result, Eq. (43) is derived by using Eqs. (34) and (36).
t(L(x)G′init bj) (L(x)G ′ init bi) =
1
2λ2 · σ2 · δij (43)
Here, we define the following function gortho,ψ(z) which is a composite function of ginit,ψ() and a().
x̂ = gortho,ψ(z) = ginit,ψ(a(z)) (44)
Then the next equation holds by definition.
∂gortho,ψ(z)
∂zi
∣∣∣∣∣ z=z0 = ∂x̂ ∂zi ∣∣∣∣∣ z=z0 = G′init bi (45)
It is noted that this equation holds at any value of y0 or z0. As a result, the following equation, that is Eq. (10), can be derived.
t ( L(x) ∂x̂
∂zi
)( L(x) ∂x̂
∂zj
) = 1
2λ2σ2 · δij (46)
If encoder and decoder are trained well and x ≃ x̂ holds, we can introduce new rescaled data space xD determined by metrics function such as dxD = L(x) · dx, and the next equation holds.
t ( ∂xD ∂zi )( ∂xD ∂zj ) = 1 2λ2σ2 · δij (47)
In conclusion, all column vectors of Jacobi matrix between z and xD has the same L2 norm 1/ √ 2λ2σ and all pairs of column vectors are orthogonal. In other words, when column vectors
of Jacobi matrix are multiplied by the constant √ 2λ2σ, the resulting vectors are orthonormal.

heading: B EXPLANATION OF ”CONTINUOUS PCA” FEATURE
text: In this section, we explain RaDOGAGA has a continuous PCA feature when factorized probability density model as below is used.
Pzψ(z) = N∏ i=1 Pziψ(zi) (48)
Here, our definition of ”continuous PCA” feature means the following. 1) Mutual information between latent variables are minimum and likely to be uncorrelated to each other. 2) Energy of latent space is concentrated to several principal components, and the importance of each component can be determined.
Next we explain the reason why these feature is acquired. As explained in appendix A, all column vectors of Jacobi matrix of decoder from latent space to data space have the same norm and all combinations of pairwise vectors are orthogonal. In other words, when constant value is multiplied, the resulting vectors are orthonormal. Because encoder is a inverse function of decoder ideally, each row vector of encoder’s Jacobi matrix should be the same as column vector of decoder under the ideal condition. Here, fortho,θ(x) and gortho,ϕ(zθ) are defined as encoder and decoder with these feature. Because the latent variables depend on encoder parameter θ, latent variable is described as zθ = fortho,θ(x), and its PDF is defined as Pzθ(zθ). PDFs of latent space and data space have the following relation where J is a Jacobian or pseudo-Jacobian between two spaces with constant value as explained in appendix A.
Pzθ(zθ) ≃ J · PxD(xD) ∝ PxD(xD) (49)
As described before, Pzψ(z) is a parametric PDF of the latent space to be optimized with parameter ψ.
By applying the result of Eqs. (41) and (43), Eq. (5) can be transformed as Eq. (50) where x̂ = gortho,ϕ(fortho,θ(x)).
Lortho = − log (Pzψ(zθ)) + λ1 · h (D(x, x̂)) +N/2 s.t. t ( ∂gortho,ϕ(zθ)
∂zθi
)( ∂gortho,ϕ(zθ)
∂zθj
) = 1
2λσ2 · δij (50)
Here, the third term of the right side is constant, this term can be removed from the cost function as follows.
L′ortho = − log (Pzψ(zθ)) + λ1 · h (D(x, x̂)) (51)
Then the parameters of network and probability can be obtained by the next.
θ, ϕ, ψ = arg min θ,ϕ,ψ
(Ex∼Px(x)[L ′ ortho]) (52)
Ex∼Px(x)[L ′ ortho] in Eq.（52）can be transformed as the next.
Ex∼Px(x)[L ′ ortho] = ∫ Px(x) · (− log (Pzψ(zθ)) + λ1 · h (D(x, x̂))) dx
= ∫ ( Pzθ(zθ) · ∣∣∣∣ dxdzθ ∣∣∣∣−1 ) · (− log (Pzψ(zθ))) · ∣∣∣∣ dxdzθ ∣∣∣∣dzθ +λ1 · ∫ PxD(xD) · |L(x)|−1 · h (D(x, x̂)) · |L(x)|dxD
= ∫ Pzθ(zθ) · (− log (Pzψ(zθ))) dzθ
+λ1 · ∫ PxD(xD) · h (D(x, x̂)) dxD (53)
At first, the first term of the third formula in Eq.(53) is examined. Let dzθ/i be a differential of (N − 1) dimensional latent variables where i-th axis zθi is removed from the latent variable zθ. Then a marginal distribution of zθi can be derived from the next equation.
Pzθi(zθi) = ∫ Pzθ(zθ)dzθ/i (54)
By using Eqs.(48) and (54), the first term of the third formula in Eq. (53) can be expanded as follows.∫ Pzθ(zθ) · (− log (Pzψ(zθ))) dzθ = ∫ Pzθ(zθ) · ( − log (∏N i=1 Pziψ(zθi)∏N i=1 Pzθi(zθi) )) dzθ
+ ∫ Pzθ(zθ) · ( − log ( N∏ i=1 Pzθi(zθi) )) dzθ
= N∑ i=1 ∫ (∫ Pzθ(zθ)dzθ/i ) · ( − log ( Pziψ(zθi) Pzθi(zθi) )) dzθi
+ N∑ i=1 ∫ (∫ Pzθ(zθ)dzθ/i ) · (− log (Pzθi(zθi))) dzθi
= N∑ i=1 DKL(Pzθi(zθi)∥Pziψ(zθi)) + N∑ i=1 H(zθi) (55)
H(X) is an entropy of variable X . The first term of the third formula is KL-divergence between marginal probability Pzθi(zθi) and factorized parametric probability Pziψ(zθi). The second term of the third formula can be further transformed using mutual information between latent variables I(zθ) and equation (49).
N∑ i=1 H(zθi) = H(zθ) + I(zθ) ≃ H(xD)− log(J) + I(zθ) (56)
The first term of the third formula is the entropy of input data with constant value. The second is also constant. As a result, in order to minimize (55), mutual information I(zθ) must be minimized.
At second, the second term of the third formula in Eq. (53) is examined. xD and x̂D denote mapped values to rescaled data space defined by the metrics function D(x1,x2) as Eq.(9). Because xD and x̂D are close, the following equations holds.
x̂D ∼ xD +L(x) (x̂− x) (57) D(x, x̂) ∼ |L(x) (x̂− x)|2 ∼ |xD − x̂D|2 (58)
By using these expansions, Eq.(53) can be expressed as follows.
Ex∼Px(x)[L ′ ortho] ≃ N∑ i=1 DKL(Pzθi(zθi)∥Pziψ(zθi))
+I(zθ) + ExD [ |xD − x̂D|2 ] +Const. (59)
Here, the rescaled real space xD is divided into a plurality of small subspace partitionings ΩxD1, ΩxD2, · · · . Let Ωz1, Ωz2, · · · be corresponding subspace partitionings in latent space. as the division space of the latent space z ∈ RN corresponding to ΩxD. Then Eq. (59) can be rewritten as follows.
Ex∼Px(x)[L ′ ortho] ≃ N∑ i=1 DKL(Pzθi(zθi)∥Pziψ(zθi))
+ ∑ k ( I(zθ ∈ Ωzθk) + ExD∈ΩxDk [ |xD − x̂D|2 ]) +Const. (60)
For each subspace partitioning, the transformation from ΩxDk to Ωzθk can be regarded as constantly scaled orthonormal transformation where orthonormal basis is Jacobi matrix with scale factor J−1.
According to Karhunen-Loève Theory (Rao & Yip (2000)), the orthonormal basis which minimize both mutual information and reconstruction error leads to be Karhunen-Loève transform(KLT). It is noted that the basis of KLT is equivalent to PCA orthonormal basis.
As a result, when Eq. (60) is minimized, Jacobi matrix from ΩxDk to Ωzθk for each subspace partitioning should be KLT/PCA. Accordingly, the same feature as PCA will be realized such as the determination of principal components etc.
From these consideration, we conclude that RaDOGAGA has a ”continuous PCA” feature.

heading: C EXPANSION OF SSIM TO A QUADRATIC FORM
text: Structural similarity (SSIM) (Wang et al., 2001) is widely used for picture quality metric which is close to human subjective quality. In this appendix, we show (1− SSIM) can be approximated to a quadratic form such as Eq.(6).
Eq. (61) is a SSIM value for a N ×N window between picture x and y. In order to calculate SSIM index for a picture, this window is shifted in a whole picture and all of SSIM values are averaged.
SSIMN×N (x,y) = 2µxµy µx2 + µy2 · 2σxy σx2 + σy2
(61)
If (1− SSIMN×N (x,y)) is expressed in quadratic form, the average for a picture (1 − SSIMpicture) can be also expressed in quadratic form.
Let δx be a minute displacement of x. Then SSIM between x and x+ δx can be approximated as follows
SSIMN×N (x,x+ δx) = 1− µδx
2
2µx2 − σδx
2
2σx2 +O
( (|δx|/|x|)3 ) (62)
Then µδx2 and σδx2 can be expressed as follows.
µδx 2 = tδx ·M · δx
whereM = 1
N2 ·  1 1 . . . 1 1 1 . . . 1 ... ... . . .
... 1 1 . . . 1
 (63)
σδx 2 = tδx · V · δx
where V = 1
N2 ·
 N − 1 −1 . . . −1 −1 N − 1 . . . −1
... ...
. . . ...
−1 −1 . . . N − 1  (64) It is noted that matrix M is positive definite and matrix V is positive semidefinite. As a result, (1− SSIMN×N (x,y)) can be expressed in the following quadratic form with positive definite matrix.
1− SSIMN×N (x,x+ δx) ≃ tδx · ( 1
2µx2 ·M + 1 2σx2 · V ) · δx (65)
D EFFECT OF h(x)
In this appendix section, the effects of two kinds of cost scaling function h(d) = d and h(d) = log(d) are discussed. We evaluated the behaviors of encoder and decoder in a one dimensional model using simple parametric linear encoder and decoder.
Lex x be a one dimensional data with the normal distribution.
x ∈ R x ∼ N(0, σx2)
Lex z be a one dimensional latent variable. Following two linear encoder and decoder are provided with parameter a and b.
z = a · x x̂ = b · z
Here, square error is used as metrics function D(x, y). The distribution of noise ϵ added to latent variable z is set to N(0, 1). Then x̆ is derived by decoding z + ϵ.
D(x, y) = |x− y|2
ϵ ∼ N(0, 1) x̆ = b · (z + ϵ)
For simplicity, we assume parametric PDF Pψ(z) is equal to the real PDF P (z). Because the distribution of latent variable z follows N(0, (aσx)2), the entropy of z can be expressed as follows.
P (z) ∼ N(0, (aσx)2)
H(z) = ∫ −P (z) · log(P (z))dz
= log(a) + log(σx √ 2πe)
Using these notations, Eqs. (5) and (8) can be expressed as follows. Loss = Ex∼N(0,σx2), ϵ∼N(0,1) [ − logP (z) + λ1 · h(|x− x̂|2) + λ2 · |x̂, x̆|2 ] = log(a) + log(σx √ 2πe) + λ1 · Ex∼N(0,σx2) [ h(|x− x̂|2) ] + λ2 · b2 (66)
At first, the case of h(d) = d is examined. By applying h(d) = d, Eq. (66) can be expanded as follows.
Loss = log(a) + log(σx √ 2πe) + λ1 · (a · b− 1)2 · σx2 + λ2 · b2 (67)
By solving ∂Loss∂a = 0 and ∂Loss ∂b = 0, a and b are derived as follows.
a · b = λ1σx 2 +
√ λ1
2σx4 − 2λ1σx2 2λ1σ2
a = √ 2 · λ2 ·
( λ1σx 2 + √ λ1
2σx4 − 2λ1σx2 2λ1σ2 ) b = 1/ √ 2 · λ2
If λ1σx2 ≫ 1, these equations are approximated as next. a · b ≃ ( 1− 1
2λ1σx2 ) a = √ 2 · λ2 · ( 1− 1
2λ1σx2 ) b = 1/ √ 2 · λ2
Here, a · b is not equal to 1. That is, decoder is not a inverse function of encoder. In this case, the scale of latent space becomes slightly bent in order to minimize entropy function. As a result, good fitting of parametric PDF P (z) ∼ Pψ(z) could be realized while proportional relationship P (z) ∝ P (x) is relaxed. Next, the case of h(d) = log(d) is examined. By applying h(d) = log(d) and introducing a minute variable δ, Eq. (66) can be expanded as follows.
Loss = log(a) + log(σx √ 2πe) + λ1 · log ( (a · b− 1)2 + δ ) + λ2 · b2 (68)
By solving ∂Loss∂a = 0 and ∂Loss ∂b = 0 and setting δ → 0, a and b are derived as follows.
a · b = 1 a = √ 2 · λ2 (69)
b = 1/ √ 2 · λ2
Here, a ·b is equal to 1 and decoder becomes a inverse function of encoder regardless of the variance σx
2. In this case, good proportional relation P (z) ∝ P (x) could be realized regardless of the fitting Pψ(z) to P (z).
Considering from these result, there could be a guideline to choose h(d). If the parametric PDF Pψ(z) has enough ability to fit the real distribution P (z), h(d) = log(d) could be better. If not, h(d) = d could be better.

heading: E DETAIL OF THE EXPERIMENT IN SECTION 4.2
text: In this section, we provide further detail of experiment in section 5.2. First, we describe the detail of following four public datasets:
KDDCUP99 (Lichman (2013)) The KDDCUP99 10 percent dataset from the UCI repository is a dataset for cyber-attack detection. This dataset consists of 494,021 instances and contains 34 continuous features and 7 categorical ones. We use one hot representation to encode the categorical features, and eventually obtain a dataset with features of 121 dimensions. Since the dataset contains only 20% of instances labeled -normal- and the rest labeled as -attacks-, -normal- instances are used as anomalies, since they are in a minority group.
Thyroid (Lichman (2013)) This dataset contains 3,772 data sample with 6-dimensional feature from patients and can be divided in three classes: normal (not hypothyroid), hyperfunction, and subnormal functioning. We treat the hyperfunction class (2.5%) as an anomaly and rest two classes as normal.
Arrhythmia (Lichman (2013)) This is dataset to detect cardiac arrhythmia containing 452 data sample with 274-dimensional feature. We treat minor classes (3, 4, 5, 7, 8, 9, 14, and 15, accounting for 15% of the total) as anomalies, and the others are treated as normal.
KDDCUP-Rev (Lichman (2013)) To treat“normal”instances as majority in the KDDCUP dataset, we keep all“ normal”instances and randomly pick up“ attack”instances so that they compose 20% of the dataset. In the end, the number of instance is 121,597.
Next, hyper parameter for RaDOGAGA is described in table 2. First and second column is number of neuron. For DAGMM, we set same number of neuron in table 2 and (λ1, λ2) as (0.1, 0.005). Optimization is done by Adam optimizer with learning rate 1× 10−4 for all dataset.
In addition to experiment in main page, we also conducted experiment with same network size as in (Zong et al. (2018)) with parameters in table 3
Now, we provide results of setting in table 3. In table 4, RaDOGAGA- and DAGMM- are results of them and DAGMM is result cited from (Zong et al. (2018)). Even with this network size, our method has boost from baseline in all dataset.

heading: F DETAIL OF THE EXPERIMENT 4.3
text: In this section, we provide further detail of experiment in section 5.3. For both RaDOGAGA and beta-VAE, we first extract feature with following Convolution Neural Network(CNN).
CNN(9, 9, 2, 64, GDN)-CNN(5, 5, 2, 64, GDN)-CNN(5, 5, 2, 64, GDN)-CNN(5, 5, 2, 64, GDN).
Here, CNN(w, h, s, c, f) is a CNN layer with kernel size (w, h), stride size s, dimension c, and activate function f. GDN(Ballé et al. (2015)) is often used in image compression. Then, we reshape feature map and send to autoencoder as follows.
FC(1024, 8192, softplus)-FC(8192, 256, None)-FC(256, 8192, softplus)-FC(256, 1024, softplus)
FC(i, o, f) is FC layer with input dimension i, output dimension o and activate function f. None means no activate function. Note that, for beta-VAE, since it produces mean and variance, the bottom of the encoder has 2 branches.
(λ1, λ2) is as (1.0, 0.1) is set for RaDOGAGA and β is set as 1× 10−4 for beta-VAE. Optimization is done by Adam optimizer with learning rate 1× 10−4.

heading: G PDF MATCHING WITH BETA-VAE
text: For the reader with interest, we provide the result of experiment in section 5.1 with beta-VAE. Network consists of FC layers of which have the same neuron numbers as the DAGMM and RaDOGAGA. We set β as 0.001. Figure 10a and 10b depict results. Since VAE dose not support Jacobian controlling, Px(x) can not be mapped into Pzψ(z) tidily.

Review summaries:
of the paper: The authors propose a latent variable model RaDOGAGA, a generative autoencoding model. The model is trained via a tradeoff between distortion (the reconstruction error) and the rate (the capacity of the latent space, measured by entropy). The paper provides an analysis of theoretical properties of their approach, and presents supporting experimental results.