Title: RATE-DISTORTION OPTIMIZATION GUIDED AUTOEN-CODER FOR GENERATIVE APPROACH

Abstract: In the generative model approach of machine learning, it is essential to acquire an accurate probabilistic model and compress the dimension of data for easy treatment. However, in the conventional deep-autoencoder based generative model such as VAE, the probability of the real space cannot be obtained correctly from that of in the latent space, because the scaling between both spaces is not controlled. This has also been an obstacle to quantifying the impact of the variation of latent variables on data. In this paper, we propose Rate-Distortion Optimization guided autoencoder, in which the Jacobi matrix from real space to latent space has orthonormality. It is proved theoretically and experimentally that (i) the probability distribution of the latent space obtained by this model is proportional to the probability distribution of the real space because Jacobian between two spaces is constant; (ii) our model behaves as non-linear PCA, where energy of acquired latent space is concentrated on several principal components and the influence of each component can be evaluated quantitatively. Furthermore, to verify the usefulness on the practical application, we evaluate its performance in unsupervised anomaly detection and it outperforms current state-of-the-art methods.

1. INTRODUCTION:

Capturing the inherent features of a dataset from high-dimensional and complex data is an essential issue in machine learning. Generative model approach learns the probability distribution of data, aiming at data generation by probabilistic sampling, unsupervised/weakly supervised learning, and acquiring meta-prior (general assumptions about how data can be summarized naturally, such as disentangle, clustering, and hierarchical structure (Bengio et al., 2013; Tschannen et al., 2019) ). It is generally difficult to directly estimate a probability density function(PDF) P x(x) of real data x. Accordingly, one promising approach is to map to the latent space z with reduced dimension and capture PDF P z(z). In recent years, deep autoencoder based methods have made it possible to compress dimensions and derive latent variables. While there is remarkable progress in these areas (van den Oord et al., 2017; Kingma et al., 2014; Jiang et al., 2016) , the relation between x and z in the current deep generative models is still not clear.

Variational autoencoder (VAE) (P. Kingma & Welling, 2014) is one of the most successful generative models for capturing latent representation. In VAE, lower bound of log-likelihood of P x(x) is introduced as ELBO. Then latent variable is obtained by maximizing ELBO. In order to maximize ELBO, various methods (Alemi et al., 2018; Zhao et al., 2019; Brekelmans et al., 2019) have been proposed. However, many previous works did not care about the value of Jacobian between two spaces, despite the fact that the ratio between P z(z) and P x(x) is equal to the Jacobian. Even in models that provide more flexible estimation (Johnson et al., 2016; Liao et al., 2018; Zong et al., 2018) , this point is overlooked.

Here, when we turn our sight toward acquiring meta-prior, it is straightforward to evaluate the quantitative influence of each latent variable on the given metrics between data x 1 and x 2 . To do so, the scale of the latent variable should be appropriately controlled so that the changes in latent variables is proportional to the changes of given metrics in data space. In addition, this scaling should be adjusted according to the definition of the metrics. For instance, with respect to image quality metrics, different meta-prior should be derived from mean square error (MSE) and the structural similarity (SSIM). Considering the mechanism of principal component analysis (PCA) would be one of the directions to solve it. In PCA, PCA components are derived by optimal orthonormal transformation, then the importance of each component can be identified quantitatively by their variance. It should be noted that orthogonality is not enough for quantitative analysis, but both orthogonality and normalized scaling, namely, orthonormality is required. This fact implies that, if Jacobi matrix of autoencoder has orthonormality, the characteristics of acquired latent space can be evaluated quantitatively. To deal with this, we propose RaDOGAGA (Rate-Distortion Optimization Guided Autoencoder for Generative Approach), based on the rate-distortion optimization (RDO) which has been widely used in image compression with orthonormal transform coding (Sullivan & Wiegand (1998) ). In this paper, we show the effect of RaDOGAGA in the following steps.

(1) We prove that RaDOGAGA has the following property theoretically and experimentally.

• Jacobi matrix between real space and latent space leads to be constant-scaled orthonormal.

So the response of the minute change of z to the real space data x is constant at any z. • Because of constant Jacobian (or pseudo-Jacobian), P x(x) and P z(z) are almost proportional. Therefore, P x(x) can be estimated, by directly maximizing log-likelihood of parametric PDF P z ψ (z) in reduced-dimensional space, without considering ELBO. • When univariate independent distribution is used to estimate P z(z) parametrically, it behaves as "continuous PCA" where energy is concentrated on several principal components.

(2) Thanks to this property, RaDOGAGA achieve the state-of-the-art performance in anomaly detection task with four public datasets, where probability density estimation is important.

(3) We show that our approach can directly evaluate how the z impact on the given metrics in real space. This feature is promising to further interpretation of latent variables.

2. RELATED WORK:

Flow based model: Flow based generative models generates astonishing quality of image (Kingma & Dhariwal, 2018; Dinh et al., 2014) . Flow mechanism explicitly takes Jacobian of x and z into account. The transformation function z = f (x) is learned, calculating and preserving Jacobian of x and z. Unlike ordinary autoencoder, which reverse z to x with function g(•) different from f (•), inverse function transforms z as x = f -1 (z). Since the model preserves Jacobian, P z(x) can be estimated by maximizing log likelihood of P z(z) without considering ELBO. Although, in this approach, f (•) need to be bijection. Because of this limitation, it is difficult to fully utilize the flexibility of neural networks.

Interpretation of latent variables: While it is expected to acquire meta-prior by deep autoencoder, interpreting latent variables is still challenging. In recent years, research aiming to acquire disentangled latent variables, which encourages them to be independent, is flourishing (Lopez et al., 2018; Chen et al., 2018; Kim & Mnih, 2018; Chen et al., 2016) . With these methods, qualitative effects for disentanglement can be seen. For example, when a certain latent variable is displaced, image changes corresponding to specific attributes (size, color, etc.) . Some works also propose quantitative metrics for meta-prior. In beta-VAE (Higgins et al., 2017) , the metric evaluates the independence of latent variables by solving the classification task. Actually, the Jacobi matrix of VAE is orthogonal (Rolínek et al., 2019) , which enables to make latent variables disentangled implicitly. However, orthonormality is not supported and it is difficult to define the metric which evaluates the effect of latent variables to the metrics between data directly.

Deep image compression with rate-distortion optimization::

Rate-distortion (RD) theory (Berger, 1971 ) is a part of Shannon's information theory for lossy compression which formulates the optimal condition between information rate and distortion. Then RD theory for Gaussian source with memory has been further extended to transform coding (Goyal, 2001) for image and audio lossy compression where orthonormal transforms such as Karhunen-Loève transform (KLT) (Rao & Yip (2000) ) and discrete cosine transform (DCT) are used for decorrelation. Furthermore, rate-distortion optimization (RDO) has been widely used in image compression (Sullivan & Wiegand, 1998) . In RDO, a cost L = R + λ • D is minimized at given Lagrange parameter λ to realize the best trade-off between rate R and distortion D such that L = R + λ • D becomes a tangent line of true Rate-Distortion curve. Recently, deep learning based image compression (Ballé et al., 2018; Zhou et al., 2019) has been proposed. In these works, instead of orthonormal transform with L2 norm metrics in the conventional lossy compression method, a deep autoencoder is trained with flexible metrics such as SSIM for RDO. To understand the success of these works, we prove theoretically that RDO guides a Jacobi matrix of deep autoencoder to orthonormal in the space depending on the metrics. In the next section, we explain the idea of RDO guided autoencoder and its relationship with VAE.

3. OVERVIEW OF RATE-DISTORTION OPTIMIZATION GUIDED APPROACH:



3.1. DERIVATION FROM RATE-DISTORTION OPTIMAZATION IN TRANSFORM CODING:

Figure 1 shows the overview of our idea inspired by Rate-Distortion optimization of transform coding. In the transform coding, the optimal method to encode data with Gaussian distribution is as follows (Goyal, 2001) . At first, the data are transformed deterministically to decorrelated data using the orthonormal transform such as KLT and DCT. Then these decorrelated data are quantized stochastically with uniform quantizer for all channels such that quantization noise for each channel is equal. At last optimal entropy encoding is applied to quantized data where the rate can be calculated by the logarithm of symbol's estimated probability. From this fact, we have an intuition that the Jacobi matrix of autoencoder becomes orthonormal if the data were compressed based on Rate-Distortion optimization with uniform quantized noise and parametric distribution of latent variables. Inspired by this, we propose autoencoder which scales latent variables according to the definition of metrics of data, without limitation of the transformation function as in Flow models. Thanks to this feature, our scheme can estimate P z(z) quantitatively, which is suitable for clustering and anomaly detection. Furthermore, in the case factorized distribution is used for P z(z), our model behaves as continuous PCA. This property is considered to promote the interpretation of latent variables.

3.2. RELATIONSHIP WITH VAE:

There is a number of VAE studies that taking rate-distortion trade-off into account. In VAEs, instead of maximizing log-likelihood of P (x) directly, maximizing ELBO. In beta-VAE (Higgins et al., 2017) , Objective function

L V AE is described as L V AE = L rec -β • L kl .

L kl is KL divergence between encoder output and prior distribution, usually Gaussian distribution. By changing β, the rate-distortion trade-off at desirable rate can be realized as discussed in (Alemi et al., 2018) .

When the relation between beta-VAE and RDO in image compression is considered, both are analogous to each other. That is, β -1 is corresponding to λ, -L kl is corresponding to a rate R, and L rec is corresponding to a distortion D. However, the probability distribution models of latent variables are quite different. VAE uses a fixed prior distribution. This causes a non-linear scaling relationship between real data and latent variables.

Figure 2 shows the conditions to achieve Rate-Distortion optimization in both VAE and RDO guided autoencoder. In VAE, precise control of noise distribution is required for each data and channel as suggested in Brekelmans et al. (2019) in order to compensate for the nonlinearities between the data space and the latent space. Jacobi matrix becomes orthogonal as proved in Rolínek et al. (2019) . However, Jacobian is not constant because of non-linear scaling. In RDO guided autoencoder, uniform noises are added to all channels. Instead, parametric probability distribution should be used in order to make the scaling linear between the data space and the latent space. As a result, the Jacobi matrix becomes orthonormal because both orthogonality and scaling normalization are simultaneously achieved. As discussed above, the precise noise control in VAE and distribution parameter optimization in RDO guided autoencoder are essentially the same. Accordingly, complexities in both methods are estimated to be at the same degree.

4. METHOD AND THEORETICAL PROPERTY:



4.1. METHOD:

Our model is based on the Rate-Distortion optimization of the autoencoder for the hyperprior proposed in Ballé et al. (2018) for image compression with some modification. The cost function consists of (i) reconstruction error D between input data and decoder output with noise to latent variable and (ii) Rate R of latent variable. This is analogous to beta-VAE where λ = β -1 . The specific method is described as follows. First, let x be M -dimensional domain data(x ∈ R M ) and P x(x) the probability of x. Then x is converted to N -dimensional latent space z ∈ R N by encoder. Let f θ (x), g ϕ (z), and P z ψ (z) be parametric encoder, decoder, and probability distribution function of latent space with parameters θ, ϕ, and ψ. It is noted that encoder and decoder are deterministic unlike VAE. Then latent variable z and decoded data x are generated as bellow:

L = R + λ • D (1)

EQUATION

Let ϵ ∈ R N be noise with each dimension being independent with an average of 0 and a equal variance of σ 2 to emulate uniform quantization.

EQUATION

Then, given the sum of latent variable z and noise ϵ, the decoder output x is obtained. This is analogous to the stochastic sampling and decoding procedure in VAE.

x = g ϕ (z + ϵ) (4)

Here, the cost function is defined by Eq. ( 5) with some modifications from Eq. (1).

EQUATION

In this equation, the first term is corresponding to the estimated rate of the latent variables. In Ballé et al. (2018) , a rate for quantized symbol is calculated by -0.5 -0.5 log(P z ψ (z))dz. In our model, this is replaced by -log(P z ψ (z)) for simplicity. To model P z ψ (z), factorized probability model or Gaussian mixture model (GMM) can be used for instance. D(x 1 , x 2 ) in the second and the third term is a metrics function between x 1 and x 2 . Actually, the second and the third term is decomposition of Rolínek et al. (2019) . The second term purely calculate reconstruction loss and the third term only influences the scaling of latent space. Accordingly, λ 1 controls the degree of reconstruction, and λ 2 (∼ β -1 of beta-VAE) controls a scaling between the data space and the latent space. By this decomposition, we can adjust balance between reconstruction loss and other losses flexibly and easily lead better performance. h(•) in the second term of Eq. ( 5) is a monotonically increasing function. According to the choice of h(•), the degree of reconstruction accuracy can be controlled. Roughly speaking, when h(d) = d is chosen, the training much focus on fitting the distribution and entropy. On the other hand, h(d) = log(d) encourages better reconstruction, since the curve of loss is steep around 0. The effect of h(d) is further discussed in Appendix D. Then, Eq. ( 5) is averaged according to distributions, x ∼ P x(x) and ϵ ∼ P (ϵ). For metrics function D(•, •), a variety of function is applicable such as MSE, SSIM, and so on. As long as the function can be approximated by the following quadratic form in the neighborhood of x, the property of the model is hold

D(x, g ϕ (f θ (x)+ϵ)) ∼ D(x, g ϕ (f θ (x)))+D(g ϕ (f θ (x)), g ϕ (f θ (x)+ϵ)) as shown in

D(x, x + δx) ∼ t δx A(x) δx = ∥L(x) δx∥ 2 (6)

Here, δx is arbitrary micro variation of x, A(x) is M × M positive definite matrix depending on x , and L(x) is Cholesky decomposition of A(x). For instance, when D(•, •) is square of L2 norm as in Eq. ( 7), A(x) and L(x) are Identity matrices.

D(x 1 , x 2 ) = ∥x 1 -x 2 ∥ 2 (7)

In the case of SSIM (Wang et al., 2001) metric which is close to subjective image quality, a cost (1 -SSIM ) can be also approximated in a quadratic form with positive definite matrix. This is explained in Appendix C. By deriving parameters that minimize this value, the encoder, decoder, and probability distribution of the latent space are trained as Eq. ( 8).

θ, ϕ, ψ = arg min θ,ϕ,ψ

EQUATION

Encoder

f θ (x) Decoder g φ (z) Pzψ(z) : Parametric PDF of z R=-log(Pzψ(z)) Domain D 𝑥𝑥 𝑧𝑧 � 𝑥𝑥 h (D(𝑥𝑥, � 𝑥𝑥)) 𝜃𝜃, 𝜑𝜑, 𝜓𝜓 = 𝒂𝒂𝒂𝒂𝒂𝒂𝒂𝒂𝒂𝒂𝒂𝒂(𝑬𝑬 𝒙𝒙~𝑷𝑷𝒙𝒙 𝒙𝒙 ,𝝐𝝐~𝑵𝑵(𝟎𝟎,𝝈𝝈) 𝑴𝑴 𝑹𝑹 + 𝝀𝝀1･𝒉𝒉 D( 𝑥𝑥, � 𝑥𝑥 ) + λ2･D( � 𝑥𝑥, � 𝑥𝑥 ) ) Decoder g φ (z+ε) � 𝑥𝑥 D( � 𝑥𝑥, � 𝑥𝑥) 𝝐𝝐~𝑷𝑷(𝝐𝝐) + Figure 3: Architecture of RaDOGAGA

4.2. THEORETICAL PROPERTY OF THE MODEL:

In this section, we show the theoretical property of the model. First of all, we provide a rough sketch of the theory. Let J = |dx/dz| be a Jacobian between the data space x and the latent space.

Here, R and λD in Eq. ( 1) can be approximated by -log P x(x) -log(J ) and C σ,λ J 2 respectively where C σ,λ is a constant depending on σ and λ. As a result, the Jacobi matrix with J = 1/ 2C σ,λ minimizes Eq. ( 1). Because Jacobi matrix is orthogonal (Rolínek et al., 2019) and the Jacobian is constant, the Jacobi matrix becomes orthonormal. A full proof is described in Appendix A. In this section, we mainly explain important points. Let x D be rescaled space of x according to the metrics D(•, •), where dx and dx D are differentials and L(x) is a Jacobi matrix between two spaces.

dx D = L(x)dx (9)

When D(•, •) is square of L2 norm, both spaces of x D and x are equivalent. As described above, each column of the Jacobian matrix of latent space z and rescaled space x D are constant multiple of orthonormal basis regardless of the value of z and x after training based on Eqs. ( 5) and ( 8). Here, δ ij denotes Kronecker delta and matrix t T denotes a transpose of T .

t L(x) ∂x ∂z i L(x) ∂x ∂z j = t ∂x D ∂z i ∂x D ∂z j = 1 2λ 2 σ 2 • δ ij (10)

From the orthonormal property, our model has several features. The first feature is that The distance derivatives of both real and latent spaces has a linear relationship. Therefore, the metrics between x and x + dx is proportional to ∥dz∥ 2 as derived by Eqs. ( 6), ( 9) and ( 10) where dz is a differential.

D(x, x + dx) = ∥L(x)dx∥ 2 = ∥dx D ∥ 2 = N i=1 ∂x D ∂z i • dz i 2 = 1 2λ 2 σ 2 • ∥dz∥ 2 (11) D(x, x + dx)/∥dz∥ 2 = ∥dx D ∥ 2 /∥dz∥ 2 = const.

(12) This equation also implies that the L2 distance of two data in the latent space is proportional to the distance between corresponding data in the real space which is defined by the integration of the root of metrics derivative.

The second feature is constant Jacobian. In case of M = N , the Jacobian matrix dx D /dz is a square matrix, and each column is the same as 1/( √ 2λ 2 • σ) times orthonormal basis. For this reason, the Jacobian is a constant regardless of the value of z as shown below:

dx D dz = 1 2λ 2 • σ 2 (N/2) (13)

In this case, the probability distribution of x D and z is proportional because of the constant Jacobian. For the case of M > N , we assume the situation where most energy is efficiently and effectively mapped to N-dimensional latent space. Then the product of the singular values of SVD for a Jacobi matrix can be regarded as a pseudo Jacobian between the real space and the latent space. Since all of the N singular values are 1/( √ 2λ 2 • σ), the pseudo Jacobian is also a constant.

The third feature is proportional probability density between data space and latent space. Because Jacobian or pseudo Jacobian J is constant, the following equation holds. P x D (x D ) and P z(z) are are true PDFs for each space x D and z respectively.

EQUATION

)

The forth is that the log-likelihood of PDF of the domain data can be maximized without considering ELBO. This is revealed by the transformation of Equation ( 5). Let P x D (x D ) be estimated probability of x D . Because the Jacobian J is constant, P x D (x D ) can be approximated by

J -1 • P z ψ (z).

Accordingly, the average of the first term in Equation ( 5) by x ∼ P x(x) can be transformed as follows.

EQUATION

Consequently, minimization of Equation ( 15) is equivalent to the log-likelihood maximization of

P x D (x D ).

The fifth is "continuous PCA" feature when the following factorized probability model is used.

EQUATION

As shown in FactorVAE (Kim & Mnih, 2018) , the use of factorized model minimizes mutual information among latent variables and encourages disentangling. Combining with the first character of our model described before, the variance values of each latent variable show the quantitative impact on data space like linear PCA. The detailed derivation is explained in Appendix B. .

5. EXPERIMENT:



5.1. PROBABILITY DENSITY ESTIMATION WITH TOY DATA:

In this section, we describe our experiment using toy data to demonstrate whether the probability density of the input data P x(x) and that of estimated in the latent space P z(z) are proportional to each other as in theory. First, we sample data s = (s 1 , s 2 ...s 10,000 ) from three-dimensional Gaussian distribution consists of three-mixture-components with mixture weight π = (1/3, 1/3, 1/3), mean µ k = (µ k1 , µ k2 , µ k3 ), and covariance Σ k = diag(σ k1 , σ k2 , σ k3 ). k is the index for mixture component. Then, we scatter s with uniform random noise u ∈ R 3×16 , u dm ∼ U d -1 2 , 1 2 , where d and m are index for dimension of sampled data and scattered data. The U d s are uncorrelated with each other. We produce 16-dimensional input data x with a sample number of 10,000 in the end.

x = 3 d=1 u d s d (17)

The appearance probability of the input data P x(x) is equivalent to the generation probability of s.

5.1.1. CONFIGURATION:

In the experiment, we estimate the P z ψ (z) using GMM with parameter ψ as in DAGMM (Zong et al., 2018) . Instead of EM algorithm, GMM parameters are learned using Estimation Network (EN), which consists of multi-layer neural network. When the GMM has K mixturecomponents and L is the size of batch samples, EN outputs the mixture-components membership prediction as K-dimensional vector γ as follows:

EQUATION

K-th mixture weight ϕ k , mean µ k , covariance Σ k , and entropy R of z are further calculated by Eqs. ( 19) and (20).

π k = L l=1 γ lk /L, µ k = L l=1 γ lk z l / L l=1 γ lk , Σ k = L l=1 γ lk (z l -µ k )(z l -µ k ) T / L l=1 γ lk (19) R = -log L k=1 π k / 2π Σ k • exp - 1 2 (z -µ k ) T Σ -1 k (z -µ k ) (20)

Overall network is trained by Eqs. ( 5) and ( 8). In this experiment, we set D(x 1 , x 2 ) as square error

∥ x 1 -x 2 ∥ 2 ,

and test two types of h(•), h(d) = d and h(d) = log(d).:

We denote models trained with these h(•) as RaDOGAGA(d) and RaDOGAGA(log(d)). As a comparison method, DAGMM is used. DAGMM also consists of encoder, decoder, and EN. In DAGMM, to avoid falling into the trivial solution that entropy is minimized when the diagonal component of the covariance matrix is 0, the inverse of the diagonal component is added to the cost function as Eq. ( 21):

L = ∥x -x∥ 2 + λ 1 • (-log(P z ψ (z))) + λ 2 P ( Σ), P ( Σ) = K k=1 N i=1 Σ -1 kii (21)

The only differences between our model and DAGMM are (i) RDO mechanism is introduced (ii) the third regularization term in Eq. ( 21) is eliminated, since our model adjust latent scale appropriately so as not to falling into trivial solution. Thus, model complexity such as the number or parameter is the same. For both RaDOGAGA and DAGMM, the autoencoder part is constructed with fully connected (FC) layers with sizes of 64, 32, 16, 3, 16, 32, and 64. For all FC layers except for the last of the encoder and the decoder, we attach tanh as the activation function. The EN part is also constructed with FC layer with a size of 10, 3. For the first layer, we attach the tanh as activation function and dropout (ratio=0.5). For the last one, softmax is attached. (λ 1 , λ 2 ) is set as (1 × 10 -4 , 1 × 10 -9 ), (1 × 10 6 , 1 × 10 6 ) and (1 × 10 3 , 1 × 10 4 ) for DAGMM, RaDOGAGA(d) and RaDOGAGA(log(d)) respectively. Optimization is done by Adam optimizer (Kingma & Ba, 2014) with learning rate 1 × 10 -4 for all model. We set σ 2 as 1/12. Since the data is generated from GMM, the sample getting closer to the center of clustering, the PDF getting higher. Although, in DAGMM, this tendency looks distorted. This difference is further demonstrated in Fig 5 which shows a plot of P x(x)(x-axis) against P z ψ (z)(y-axis). In our method, P x(x) and P z ψ (z) are approximately proportional to each other as in theory while we cannot observe such proportionality in DAGMM. This difference is quantitatively obvious as well.

That is, correlation coefficients between P x(x) and P z ψ (z) are 0.691 (DAGMM), 0.997 (RaDO-GAGA(d)), and 0.999 (RaDOGAGA(log(d))). It also turns out that the correlation coefficient of RaDOGAGA(log(d)) is higher than that of RaDOGAGA(d). Actually, when P z ψ (z) is sufficiently fitted, h(d) = log(d) makes P x(x) and P z ψ (z) be proportional more rigidly. On the other hand, h(d) = d makes the scale of latent space slightly bent in order to minimize entropy function, allowing relaxed fitting of P z ψ (z). More detail is described in Appendix D.

5.2. ANOMALY DETECTION TASK USING REAL DATA:

In this section, we examine whether the clear relationship between P x(x) and P z(z) is useful in the anomaly detection task using real data. We use four public datasets1 , KDDCUP99, Thyroid, Arrhythmia, and KDDCUP-Rev. The (instance number, dimension, anomaly ratio(%)) of each dataset is (494021, 121, 20) , (3772, 6, 2.5), (452, 274, 15), and (121597, 121, 20) respectively. Detail of datasets is described in Appendix E.

5.2.1. EXPERIMENTAL SETUP:

We follow the setting in Zong et al. (2018) . Randomly extracted 50% of the data is assigned to training and the rest to testing. During training, only normal data is used. During the test, the R for each sample is calculated as the anomaly score, and if the anomaly score is higher than a threshold, it is detected as an anomaly. The threshold is determined by the ratio of anomaly data in each data set. For example, in KDDCup99, data with R in the top 20 % is detected as an anomaly. As metrics, precision, recall, and F1 score are calculated. We run 20 times for each dataset split by 20 different random seeds.

5.2.2. BASELINE MODEL:

Same as in the previous section, DAGMM is taken as the baseline method. We also compare with the scores reported in previous works in which same experiments were conducted (Zenati et al., 2018; Song & Ou, 2018; Liao et al., 2018) .

As in Zong et al. (2018) , in addition to the output from the encoder, ∥x-x ′ ∥2 ∥x∥2 and

x•x ′ ∥x∥2∥x ′ ∥2 are concatenated to z. It is sent to EN. Note that z is sent to the decoder before concatenation. Other configuration except for hyper parameter is same as in the previous experiment. Hyper parameter for each dataset is described in Appendix E.

5.2.4. RESULTS:

Table 1 reports the average scores and standard deviations (in brackets). Comparing DAGMM and RaDOGAGA, RaDOGAGA has a better performance regardless of types of h(•). Note that, our model does not increase model complexity at all. Simply introducing the RDO mechanism into the autoencoder has a valid efficacy for anomaly detection. Moreover, our approach achieves stateof-the-art performance compared to other previous works in which same datasets is used. Clear relationship between P x(x) and P z(z) by our model is considered to be effective in the task of anomaly detection where the estimating probability distribution is important. In RaDOGAGA, when we compare result of RaDOGAGA(d) and RaDOGAGA(log(d)), either of one is not always superior. As described in section 5.1 and Appendix D, h(•) can be an option depending on fitting flexibility of P z(z).

5.3. QUANTIFYING THE IMPACT OF LATENT VARIABLES ON METRICS AND BEHAVIOR AS PCA:

In this section, we confirm that (i) the latent variables in our model behaves as PCA components and (ii) the impact of each latent variable on the metrics function can be quantified. When dz in Eq. † DAGMM+ is our implementation. Note that we also test same configuration as in Zong et al. (2018) and achieve similar score as reported (shown in Appendix E). ( 12) is approximated by a small displacement δ, the ratio D(g(z), g(z + δ))/∥δ∥ 2 will be almost constant regardless of z and δ. If the ratio of each component z i in the latent space is equivalent, the latent space can be regarded as isometric. So, we evaluate this ratio in each component. Let δ i be a vector t (0, • • • , ϵ, •, 0) where only i-th component has a minute value ϵ. Then D ′ i (z) denotes

D(g(z),g(z+δi)) ϵ 2

for i-th component. In the trained model, we encode the image x l and obtain z l . Then, D ′ i (z l ) is calculated for each sample. Finally, the average across all samples is measured. This operation is conducted to each dimension of z independently. We also observe the distribution of z and how the image looks different in response to the variation of component z i .

To train model, we use CelebA dataset (Liu et al., 2015) , which consists of 202,599 celebrity images. The images are center-cropped so that the image size is 64 x 64.

In this experiment, factorized distributions (Ballé et al., 2018) is used to estimate P z ψ (z). For comparison, we evaluate beta-VAE. Both models are constructed with same depth of CNN and FC layers, with 256-dimensional z. Detail of networks and hyper parameter is written in Appendix F. For RaDOGAGA, we set D(x 1 , x 2 ) as 1 -SSIM (x 1 , x 2 ) and h as h(d) = d. For beta-VAE, reconstruction loss is also 1 -SSIM (x 1 , x 2 ). SSIM (x 1 , x 2 ) is defined by Eq. (61).

5.3.2. RESULT:

Both models are trained so that the SSIM between input and reconstructed image is around 0.93. Figure 6a and 6b show the variance of each component in the latent variables z arranged in descending order. The red line is the cumulative relative ratio of the variance. In Fig. 6b , variance is concentrated in a specific dimension. On the other hand, in Fig. 6a , beta-VAE is trained so that each latent variable is fitted to a Gaussian distribution with mean 0 and variance 1, there is no significant difference in the variance of each latent variable. Figures 6c and 6d respectively depicts the average of D ′ i (z) of each of the top nine dimensions with the largest variance of z. In beta-VAE, the D ′ i (z) varies drastically depending on the dimension i which shows anisometric latent space, while in RaDOGAGA, it is approximately constant regardless of i which shows isometric latent space. Figure 7 shows decoder outputs when each component z i is traversed from -2σ to 2σ, fixing rest of z as mean. From the top, each row corresponds to z 0 , z 1 , z 2 ..., and the center column is mean. In Fig. 7b , the image changes visually in any dimension of z, while in Fig. 7a , depending on the dimension i, there are cases where no significant changes can be seen (such as z 3 , z 4 , and so on). This result means that, in RaDOGAGA, the variance of z directly corresponds to the visual impact and the metrics D(x 1 , x 2 ), behaving as PCA. Besides, since D ′ i (z) is constant, the latent space shows isometric feature and the variance is regarded as a quantitative importance. 

6. CONCLUSION:

In this paper, we propose RaDOGAGA that learns parametric probability distribution and autoencoder simultaneously based on rate-distortion optimization. It was proven that the probability distribution of the latent variables obtained by the proposed method is proportional to the probability distribution of the input data theoretically and experimentally. This property is validated in anomaly detection achieving state-of-the-art performance. Moreover, our model has the trait as PCA which likely promotes interpretation of latent variables. For the future work, we will conduct experiments with different types of metrics functions that derived from semantical task, such as in categorical classification. Meanwhile, as mentioned in Tschannen et al. (2019) , considering the usefulness of latent variables in downstream task is another research direction to explore.

A HOW JACOBI MATRIX BECOME A CONSTANTLY SCALED ORTHONORMAL BASIS:

In this appendix section, we prove that all of column vectors in decoder's Jacobi Matrix have the same norm and are orthogonal to each other. In other words, each column of Jacobi matrix is a constantly scaled orthonormal basis.

Here we assume that data space sample x has M dimension, that is x ∈ R M , then encoded to N dimensional latent space sample y ∈ R N . We also assume that an fixed encoder function y = f init,θ (x) and a fixed decoder x = g init,ϕ (y) are given such that h(D(x, x)) becomes minimal.

y = f init,θ (x) (22) x = g init,ϕ (y) (23) s.t h(D(x, x)) =⇒ minimal

We further assume that fixed parametric PDF of latent variable y is also given.

P y init,ψ (y) (24)

Here, it is noted that this function is not needed to be optimal in a sense of D KL (P y (y) ∥P y init,ψ (y)) where P y (y) is an actual PDF of y.

Under these conditions, we introduce new latent variable z, and y is transformed from z using a following scaling function a(z) :

EQUATION

Here, our goal is to prove Eq. ( 10) by examining the condition of this scaling function which minimize the average of Eq. ( 5) with regard to ϵ ∼ P (ϵ).

Because of the assumption of minimal D(x, x), we ignore the second term of Eq. ( 5). Next, the PDF of z can be derived using a Jacobian of a(z).

EQUATION

By applying these conditions and equations to Eq. ( 5), the cost of scaling function a(z) to minimize the average is expressed as follows.

EQUATION

Next, the latent variable is fixed as y 0 , and z 0 is defined in the next equation.

EQUATION

Then the average of Eq. ( 27) with regard to ϵ ∼ P (ϵ) is expressed as follows.

EQUATION

Then the condition of a(z 0 ) in the neighborhood of y 0 is examined when Eq. ( 29) is minimized.

Here, it is noted that the first term of the right side doesn't depend on ϵ.

Under review as a conference paper at ICLR 2020 Before examining Eq. ( 29), some preparation of equations are needed. At first, Jacobi matrix of a(z) at z = z 0 is defined as B using notations of partial differentials b ij = ∂ai ∂zj z=z0

.

B = ∂y ∂z z=z0 = da(z) dz z=z0 =     b 11 b 12 . . . b 1N b 21 b 22 . . . b 2N . . . . . . . . . . . . b N 1 b N 2 . . . b N N     where b ij = ∂a i ∂z j z=z0 (30)

The vector b i is also defined as follows.

EQUATION

It is clear by definition that b i and B have the following relation.

EQUATION

∆ ij is defined as a cofactor of matrix B with regard to the element b ij , and bi is also defined by the following equation.

bi:

EQUATION

The following equations hold by the definition of cofactor.

EQUATION

In case of i ̸ = j, inner product of b i and bj becomes zero because this value is a determinant of a singular matrix with two equivalent column vectors b i .

EQUATION

G ′ init is defined as a M × N Jacobi matrix of g init,ϕ (y) at y = y 0 as follows.

EQUATION

)

Using these equations, we proceed to expand of Eq. ( 29). The first term of (29) in the right side can be expressed as follows, where the second term Eq. ( 38) in the right side is constant by definition.

-log da(z)

EQUATION

Next step is an expansion of the second term in Eq. ( 29). First, the following approximation holds.

g init,ϕ (a(z 0 + ϵ)) -g init,ϕ (a(z 0 )) ∼   dg init,ϕ (y) dy y=y0   da(z) dz z=z0 ϵ = G ′ init B ϵ = G ′ init (ϵ 1 • b 1 + ϵ 2 • b 2 • • • + ϵ N • b N ) = N i=1 ϵ i • G ′ init b i (39)

Then, the second term of Eq. ( 29) can be transformed to the next equation by using Eqs. ( 39), (3), and ( 6).

EQUATION

As a result, Eq. ( 29) can be rewritten as Eq. ( 41).

EQUATION

By examining the minimization process of Eq. ( 41), the conditions of optimal scaling function y = a(z) in the neighborhood of y 0 is clarified. Here, the condition of Jacobi matrix B is examined instead of a(z). Eq. ( 41) is differentiated by vector b i , and the result is set to be zero. Then the following equation Eq.( 42) is derived.

EQUATION

Afterwards, Eq. ( 42) is multiplied by t b j from the left, and divided by 2λ 2 • σ 2 . As a result, Eq. ( 43) is derived by using Eqs. ( 34) and (36).

t (L(x) G ′ init b j ) (L(x) G ′ init b i ) = 1 2λ 2 • σ 2 • δ ij (43)

Here, we define the following function g ortho,ψ (z) which is a composite function of g init,ψ () and a().

EQUATION

Then the next equation holds by definition.

∂g ortho,ψ (z) ∂z i z=z0 = ∂ x ∂z i z=z0 = G ′ init b i (45)

It is noted that this equation holds at any value of y 0 or z 0 . As a result, the following equation, that is Eq. ( 10), can be derived.

t L(x) ∂ x ∂z i L(x) ∂ x ∂z j = 1 2λ 2 σ 2 • δ ij (46)

If encoder and decoder are trained well and x ≃ x holds, we can introduce new rescaled data space x D determined by metrics function such as dx D = L(x) • dx, and the next equation holds.

t ∂x D ∂z i ∂x D ∂z j = 1 2λ 2 σ 2 • δ ij (47)

In conclusion, all column vectors of Jacobi matrix between z and x D has the same L2 norm 1/ √ 2λ 2 σ and all pairs of column vectors are orthogonal. In other words, when column vectors of Jacobi matrix are multiplied by the constant √ 2λ 2 σ, the resulting vectors are orthonormal.

B EXPLANATION OF "CONTINUOUS PCA" FEATURE:

In this section, we explain RaDOGAGA has a continuous PCA feature when factorized probability density model as below is used.

EQUATION

Here, our definition of "continuous PCA" feature means the following. 1) Mutual information between latent variables are minimum and likely to be uncorrelated to each other. 2) Energy of latent space is concentrated to several principal components, and the importance of each component can be determined.

Next we explain the reason why these feature is acquired. As explained in appendix A, all column vectors of Jacobi matrix of decoder from latent space to data space have the same norm and all combinations of pairwise vectors are orthogonal. In other words, when constant value is multiplied, the resulting vectors are orthonormal. Because encoder is a inverse function of decoder ideally, each row vector of encoder's Jacobi matrix should be the same as column vector of decoder under the ideal condition. Here, f ortho,θ (x) and g ortho,ϕ (z θ ) are defined as encoder and decoder with these feature. Because the latent variables depend on encoder parameter θ, latent variable is described as z θ = f ortho,θ (x), and its PDF is defined as P z θ (z θ ). PDFs of latent space and data space have the following relation where J is a Jacobian or pseudo-Jacobian between two spaces with constant value as explained in appendix A.

EQUATION

As described before, P z ψ (z) is a parametric PDF of the latent space to be optimized with parameter ψ.

By applying the result of Eqs. ( 41) and ( 43), Eq. ( 5) can be transformed as Eq. ( 50) where x = g ortho,ϕ (f ortho,θ (x)).

L ortho = -log (P z ψ (z θ )) + λ 1 • h (D(x, x)) + N/2 s.t. t ∂g ortho,ϕ (z θ ) ∂z θi ∂g ortho,ϕ (z θ ) ∂z θj = 1 2λσ 2 • δ ij (50)

Here, the third term of the right side is constant, this term can be removed from the cost function as follows.

EQUATION

Then the parameters of network and probability can be obtained by the next.

θ, ϕ, ψ = arg min θ,ϕ,ψ

EQUATION

E x∼P x(x) [L ′ ortho ] in Eq.( 52)can be transformed as the next.

E x∼P x(x) [L ′ ortho ] = P x(x) • (-log (P z ψ (z θ )) + λ 1 • h (D(x, x))) dx = P z θ (z θ ) • dx dz θ -1 • (-log (P z ψ (z θ ))) • dx dz θ dz θ +λ 1 • P x D (x D ) • |L(x)| -1 • h (D(x, x)) • |L(x)|dx D = P z θ (z θ ) • (-log (P z ψ (z θ ))) dz θ +λ 1 • P x D (x D ) • h (D(x, x)) dx D (53)

At first, the first term of the third formula in Eq.( 53) is examined. Let dz θ/i be a differential of (N -1) dimensional latent variables where i-th axis z θi is removed from the latent variable z θ . Then a marginal distribution of z θi can be derived from the next equation.

P z θi (z θi ) = P z θ (z θ )dz θ/i (54)

By using Eqs.( 48) and ( 54), the first term of the third formula in Eq. ( 53) can be expanded as follows.

EQUATION

H(X) is an entropy of variable X. The first term of the third formula is KL-divergence between marginal probability P z θi (z θi ) and factorized parametric probability P z iψ (z θi ). The second term of the third formula can be further transformed using mutual information between latent variables I(z θ ) and equation ( 49).

EQUATION

The first term of the third formula is the entropy of input data with constant value. The second is also constant. As a result, in order to minimize (55), mutual information I(z θ ) must be minimized.

At second, the second term of the third formula in Eq. ( 53) is examined. x D and xD denote mapped values to rescaled data space defined by the metrics function D(x 1 , x 2 ) as Eq.( 9). Because x D and xD are close, the following equations holds.

xD ∼ x D + L(x) ( x -x) (57) D(x, x) ∼ |L(x) ( x -x)| 2 ∼ |x D -xD | 2 (58)

By using these expansions, Eq.( 53) can be expressed as follows.

EQUATION

Here, the rescaled real space x D is divided into a plurality of small subspace partitionings

Ωx D1 , Ωx D2 , • • • . Let Ωz 1 , Ωz 2 , • • • be corresponding subspace partitionings in latent space.

as the division space of the latent space z ∈ R N corresponding to Ωx D . Then Eq. ( 59) can be rewritten as follows.

EQUATION

)

For each subspace partitioning, the transformation from Ωx Dk to Ωz θk can be regarded as constantly scaled orthonormal transformation where orthonormal basis is Jacobi matrix with scale factor J -1 .

According to Karhunen-Loève Theory (Rao & Yip ( 2000)), the orthonormal basis which minimize both mutual information and reconstruction error leads to be Karhunen-Loève transform(KLT). It is noted that the basis of KLT is equivalent to PCA orthonormal basis.

As a result, when Eq. ( 60) is minimized, Jacobi matrix from Ωx Dk to Ωz θk for each subspace partitioning should be KLT/PCA. Accordingly, the same feature as PCA will be realized such as the determination of principal components etc.

From these consideration, we conclude that RaDOGAGA has a "continuous PCA" feature. (Wang et al., 2001) is widely used for picture quality metric which is close to human subjective quality. In this appendix, we show (1 -SSIM ) can be approximated to a quadratic form such as Eq.( 6).

Eq. ( 61) is a SSIM value for a N × N window between picture x and y. In order to calculate SSIM index for a picture, this window is shifted in a whole picture and all of SSIM values are averaged.

EQUATION

If (1 -SSIM N ×N (x, y)) is expressed in quadratic form, the average for a picture (1 -SSIM picture ) can be also expressed in quadratic form.

Let δx be a minute displacement of x. Then SSIM between x and x + δx can be approximated as follows

EQUATION

Then µ δx 2 and σ δx 2 can be expressed as follows.

EQUATION

It is noted that matrix M is positive definite and matrix V is positive semidefinite. As a result, (1 -SSIM N ×N (x, y)) can be expressed in following quadratic form with positive definite matrix.

1 -SSIM N ×N (x, x + δx) ≃ t δx • 1 2µ x 2 • M + 1 2σ x 2 • V • δx (65)

D EFFECT OF h(x):

In this appendix section, the effects of two kinds of cost scaling function h(d) = d and h(d) = log(d) are discussed. We evaluated the behaviors of encoder and decoder in a one dimensional model using simple parametric linear encoder and decoder. Lex x be a one dimensional data with the normal distribution.

P(z)～ 𝑁𝑁 0, 𝑎𝑎𝜎𝜎 𝑥𝑥 2 (Assume Pψ(z)=P(z)) 𝑥𝑥 ∈ 𝑅𝑅 𝑥𝑥~𝑁𝑁(0, 𝜎𝜎 𝑥𝑥 2 ) 𝑥𝑥 𝑧𝑧 � 𝑥𝑥 h (|𝑥𝑥 -� 𝑥𝑥| 2 ) 𝑎𝑎, 𝑏𝑏 = arg 𝑚𝑚𝑚𝑚𝑚𝑚(𝐸𝐸 𝑥𝑥,𝜖𝜖 -log 𝑃𝑃 𝑧𝑧 + 𝜆𝜆1･ℎ |𝑥𝑥 -� 𝑥𝑥| 2 + 𝜆𝜆𝜆･| � 𝑥𝑥 -� 𝑥𝑥| 2 ) Decoder =b･(z+ε) � 𝑥𝑥 | � 𝑥𝑥 -� 𝑥𝑥| 2 𝜖𝜖~𝑁𝑁(0,

x

∈ R x ∼ N (0, σ x 2 )

Lex z be a one dimensional latent variable. Following two linear encoder and decoder are provided with parameter a and b.

z = a • x x = b • z

Here, square error is used as metrics function D(x, y). The distribution of noise ϵ added to latent variable z is set to N (0, 1). Then x is derived by decoding z + ϵ.

D(x, y) = |x -y| 2 ϵ ∼ N (0, 1) x = b • (z + ϵ)

For simplicity, we assume parametric PDF P ψ(z) is equal to the real PDF P (z). Because the distribution of latent variable z follows N (0, (aσ x ) 2 ), the entropy of z can be expressed as follows.

P (z) ∼ N (0, (aσ x ) 2 ) H(z) = -P (z) • log(P (z))dz = log(a) + log(σ x √ 2πe)

Using these notations, Eqs. ( 5) and ( 8) can be expressed as follows.

Loss = E x∼N (0,σx 2 ), ϵ∼N (0,1) -log

EQUATION

At first, the case of h(d) = d is examined. By applying h(d) = d, Eq. ( 66) can be expanded as follows. a

Loss:

• b = λ 1 σ x 2 + λ 1 2 σ x 4 -2λ 1 σ x 2 2λ 1 σ 2 a = 2 • λ 2 • λ 1 σ x 2 + λ 1 2 σ x 4 -2λ 1 σ x 2 2λ 1 σ 2 b = 1/ 2 • λ 2

If λ 1 σ x 2 ≫ 1, these equations are approximated as next.

a • b ≃ 1 - 1 2λ 1 σ x 2 a = 2 • λ 2 • 1 - 1 2λ 1 σ x 2 b = 1/ 2 • λ 2

Here, a • b is not equal to 1. That is, decoder is not a inverse function of encoder. In this case, the scale of latent space becomes slightly bent in order to minimize entropy function. As a result, good fitting of parametric PDF P (z) ∼ P ψ (z) could be realized while proportional relationship P (z) ∝ P (x) is relaxed.

Next, the case of h(d) = log(d) is examined. By applying h(d) = log(d) and introducing a minute variable δ, Eq. ( 66) can be expanded as follows. Here, a • b is equal to 1 and decoder becomes a inverse function of encoder regardless of the variance σ x 2 . In this case, good proportional relation P (z) ∝ P (x) could be realized regardless of the fitting P ψ(z) to P (z).

Considering from these result, there could be a guideline to choose h(d). If the parametric PDF P ψ(z) has enough ability to fit the real distribution P (z), h(d) = log(d) could be better. If not, h(d) = d could be better.

E DETAIL OF THE EXPERIMENT IN SECTION 4.2:

In this section, we provide further detail of experiment in section 5.2. First, we describe the detail of following four public datasets: KDDCUP99 (Lichman ( 2013)) The KDDCUP99 10 percent dataset from the UCI repository is a dataset for cyber-attack detection. This dataset consists of 494,021 instances and contains 34 continuous features and 7 categorical ones. We use one hot representation to encode the categorical features, and eventually obtain a dataset with features of 121 dimensions. Since the dataset contains only 20% of instances labeled -normal-and the rest labeled as -attacks-, -normal-instances are used as anomalies, since they are in a minority group.

Thyroid (Lichman (2013) ) This dataset contains 3,772 data sample with 6-dimensional feature from patients and can be divided in three classes: normal (not hypothyroid), hyperfunction, and subnormal functioning. We treat the hyperfunction class (2.5%) as an anomaly and rest two classes as normal.

Arrhythmia (Lichman (2013) ) This is dataset to detect cardiac arrhythmia containing 452 data sample with 274-dimensional feature. We treat minor classes (3, 4, 5, 7, 8, 9, 14, and 15 , accounting for 15% of the total) as anomalies, and the others are treated as normal.

KDDCUP-Rev (Lichman (2013) ) To treat normal instances as majority in the KDDCUP dataset, we keep all normal instances and randomly pick up attack instances so that they compose 20% of the dataset. In the end, the number of instance is 121,597.

Next, hyper parameter for RaDOGAGA is described in table 2. First and second column is number of neuron. For DAGMM, we set same number of neuron in table 2 and (λ 1 , λ 2 ) as (0.1, 0.005). Optimization is done by Adam optimizer with learning rate 1 × 10 -4 for all dataset. In addition to experiment in main page, we also conducted experiment with same network size as in (Zong et al. (2018) ) with parameters in table 3 Now, we provide results of setting in table 3. In table 4, RaDOGAGA-and DAGMM-are results of them and DAGMM is result cited from (Zong et al. (2018) ). Even with this network size, our method has boost from baseline in all dataset. 

:

Dataset can be dowonload from (https://kdd.ics.uci.edu/) and (http://odds.cs.stonybrook.edu)

Figures and Tables:

Figure 2: The condition to optimize ratedistortion in VAE and our models

Figure 4: Plot of source of input data s and latent variable z. Normalized PDF value corresponds to a color bar located left of (a). Even though both DAGMM and RaDOGAGA capture three mixture components, the tendency of PDF distribution in DAGMM looks different from input data source. Points with high PDF are not concentrated on the center of the clustering especially in upper two.

Figure 6: Variance of z (two on the left) and D ′ i (z)(two on the right). Compared with beta-VAE, in our model, the variance is concentrated in a few dimensions. Furthermore, the influence each z to the real image is constant. These results demonstrate that the latent variable in our model behaves as PCA components and the importance for the domain data can be quantified. 𝝁𝝁 -𝟐𝟐𝟐𝟐 +𝟐𝟐𝟐𝟐 𝒛𝒛 𝟎𝟎 𝒛𝒛 𝟏𝟏 𝒛𝒛 𝟐𝟐 𝒛𝒛 𝟑𝟑 𝒛𝒛 𝟒𝟒 𝒛𝒛 𝟓𝟓 𝒛𝒛 𝟔𝟔 𝒛𝒛 𝟕𝟕 𝒛𝒛 𝟖𝟖

Figure 7: Latent space traversal of z with top-9 variance. In beta-VAE, the variance of z is uncorrelated to difference of image, while they are directly related in our model.

Rescaled domain space x D ∈ R M with small subspace partitioning Ωx Dk Latent Space z ∈ R Ｎ with small subspace partitioning Ωz k For all small subspace partitioning Ωx DK of rescaled domain space, mapping from Ωx DK to Ωz k can be regarded as PCA Subspace wise PCA

Figure 8: Continuous KLT(PCA) Mapping from input domain to latent space

Figure 9: Simple encoder/decoder model to evaluate h(d)

= log(a) + log(σ x √ 2πe) + λ 1 • (a • b -1) 2 • σ x 2 + λ 2 • b 2 (67)By solving ∂Loss ∂a = 0 and ∂Loss ∂b = 0, a and b are derived as follows.

Loss = log(a) + log(σ x √ 2πe) + λ 1 • log (a • b -1) 2 + δ + λ 2 • b 2 (68)By solving ∂Loss ∂a = 0 and ∂Loss ∂b = 0 and setting δ → 0, a and b are derived as follows.

Plot of z. Even though it captures three components of input data source s, PDF is quite different from that of s. Plot of of P x(x) (x-axis) and P z ψ (z) (yaxis). No clear correlation can been seen.

Figure 10: Result of PDF estimation with toy data (beta-VAE)



Table content: <table><tr><td>Image Data</td><td colspan="2">Source Coding Orthonormal Transform</td><td colspan="2">Decorrelated data</td><td>Channel Coding Uniform Quantization / Entropy coding</td><td>Compressed Data</td></tr><tr><td>Method</td><td/><td colspan="2">Source Coding</td><td colspan="2">Channel Coding</td><td>Rate-Distortion</td></tr><tr><td/><td/><td colspan="2">(Deterministic)</td><td colspan="2">(Stochastic)</td><td>Relation</td></tr><tr><td colspan="2">Transform</td><td colspan="2">Condition:</td><td colspan="2">Condition:</td><td>Result:</td></tr><tr><td>coding</td><td/><td colspan="2">Orthonormal</td><td colspan="2">Uniform</td><td>Rate-Distortion</td></tr><tr><td/><td/><td colspan="2">KLT / PCA / DCT</td><td colspan="2">quantization /</td><td>Optimization is</td></tr><tr><td/><td/><td>transform</td><td/><td colspan="2">Entropy coding</td><td>achieved</td></tr><tr><td/><td/><td/><td/><td/><td>Condition:</td></tr><tr><td/><td/><td/><td/><td colspan="2">Uniform noise</td><td>Rate-Distortion</td></tr><tr><td/><td/><td/><td/><td colspan="2">for all channel /</td><td>based Loss</td></tr><tr><td/><td/><td/><td/><td colspan="2">Rate estimation</td><td>function</td></tr></table>

1: Overview of our idea based on RDO.

Table content: <table><tr><td>Method</td><td>PDF model</td><td>Noise</td><td>Jacobi Matrix</td></tr><tr><td>VAE with fixed</td><td>Fixed as prior</td><td>Precise control for</td><td>Orthogonal</td></tr><tr><td>prior</td><td/><td>each data and</td><td>(Variable</td></tr><tr><td/><td/><td>channels</td><td>Jacobian)</td></tr><tr><td>RDO guided</td><td>Variable</td><td>Uniform for all</td><td>Orthonormal</td></tr><tr><td>Autoencoder</td><td>parametric PDF</td><td>data and</td><td>(Constant</td></tr><tr><td/><td/><td>channels</td><td>Jacobian)</td></tr></table>



Table content: <table><tr><td>x</td><td/><td>VAE</td><td/></tr><tr><td colspan="2">Orthogonal</td><td/><td>Orthonormal</td></tr><tr><td colspan="2">(Variable Jacobian)</td><td>Fixed PDF as prior</td><td>(Constant Jacobian)</td><td>Parametric PDF</td></tr><tr><td>z</td><td colspan="2">Precisely controlled noise</td><td/></tr></table>

Average and standard deviations(in brackets) of Precision, Recall and F1

Table content: <table><tr><td>Dataset</td><td>Methods</td><td>Precision</td><td>Recall</td><td>F1</td></tr><tr><td/><td>ALAD  *</td><td colspan="3">0.9427(0.0018) 0.9577(0.0018) 0.9501(0.0018)</td></tr><tr><td/><td>INRF  *</td><td colspan="3">0.9452(0.0105) 0.9600(0.0113) 0.9525(0.0108)</td></tr><tr><td/><td>VAE  *</td><td>0.7805</td><td>0.7903</td><td>0.7854</td></tr><tr><td>KDDCup</td><td>GMVAE  *</td><td>0.952</td><td>0.9141</td><td>0.9326</td></tr><tr><td/><td>DAGMM  *</td><td>0.9297</td><td>0.9442</td><td>0.9369</td></tr><tr><td/><td>DAGMM+  †</td><td colspan="3">0.9427(0.0052) 0.9575(0.0053) 0.9500(0.0052)</td></tr><tr><td/><td>RaDOGAGA(d)</td><td colspan="3">0.9550(0.0037) 0.9700(0.0038) 0.9624(0.0038)</td></tr><tr><td/><td colspan="4">RaDOGAGA(log(d)) 0.9563(0.0042) 0.9714(0.0042) 0.9638(0.0042)</td></tr><tr><td/><td>GMVAE  *</td><td>0.7105</td><td>0.5745</td><td>0.6353</td></tr><tr><td/><td>VAE  *</td><td>0.3395</td><td>0.3592</td><td>0.3491</td></tr><tr><td>Thyroid</td><td>DAGMM  *</td><td>0.4766</td><td>0.4834</td><td>0.4782</td></tr><tr><td/><td>DAGMM+  †</td><td colspan="3">0.4656(0.0481) 0.4859(0.0502) 0.4755(0.0491)</td></tr><tr><td/><td>RaDOGAGA(d)</td><td colspan="3">0.6313(0.0476) 0.6587(0.0496) 0.6447(0.0486)</td></tr><tr><td/><td colspan="4">RaDOGAGA(log(d)) 0.6562(0.0572) 0.6848(0.0597) 0.6702(0.0585)</td></tr><tr><td/><td>ALAD  *</td><td colspan="3">0.5000(0.0208) 0.5313(0.0221) 0.5152(0.0214)</td></tr><tr><td/><td>VAE  *</td><td>0.3328</td><td>0.3392</td><td>0.3360</td></tr><tr><td>Arrythmia</td><td>GMVAE  *  DAGMM  *</td><td>0.4375 0.4909</td><td>0.4242 0.5078</td><td>0.4308 0.4983</td></tr><tr><td/><td>DAGMM+  †</td><td colspan="3">0.4985(0.0389) 0.5136(0.0401) 0.5060(0.0395)</td></tr><tr><td/><td>RaDOGAGA(d)</td><td colspan="3">0.5353(0.0461) 0.5515(0.0475) 0.5433(0.0468)</td></tr><tr><td/><td colspan="4">RaDOGAGA(log(d)) 0.5294(0.0405) 0.5455(0.0418) 0.5373(0.0411)</td></tr><tr><td/><td>DAGMM  *</td><td>0.937</td><td>0.939</td><td>0.938</td></tr><tr><td>KDDCup-rev</td><td>DAGMM+  † RaDOGAGA(d)</td><td colspan="3">0.9778(0.0018) 0.9779(0.0017) 0.9779(0.0018) 0.9768(0.0033) 0.9827(0.0012) 0.9797(0.0015)</td></tr><tr><td/><td colspan="4">RaDOGAGA(log(d)) 0.9864(0.0009) 0.9865(0.0009) 0.9865(0.0009)</td></tr><tr><td colspan="5">*  Score is cited from Zenati et al. (2018)(ALAD), Song &amp; Ou (2018)(INRF), Liao et al. (2018)(VAE, GM-</td></tr><tr><td colspan="3">VAE) and Zong et al. (2018) (DAGMM) respectively.</td><td/><td/></tr><tr><td/><td/><td/><td/><td/></tr></table>

Hyper parameter for RaDOGAGA

Table content: <table><tr><td>Dataset</td><td>Autoencoder</td><td>EN</td><td colspan="4">λ 1 (d) λ 2 (d) λ 1 ((log(d)) λ 2 (log(d))</td></tr><tr><td>KDDCup99</td><td colspan="3">60, 30, 8, 30, 60 10, 4 100</td><td>1000</td><td>10</td><td>100</td></tr><tr><td>Thyroid</td><td colspan="3">30, 24, 6, 24, 30 10, 2 100</td><td colspan="2">10000 100</td><td>1000</td></tr><tr><td>Arrhythmia</td><td>10, 4, 10</td><td colspan="2">10, 2 1000</td><td>100</td><td>1000</td><td>100</td></tr><tr><td colspan="4">KDDCup-rev 60, 30, 8, 30, 60 10, 2 1000</td><td>100</td><td>100</td><td>100</td></tr></table>

Peer review (iclr2020):
The paper propose a noisy autoencoder that considers the jacobian between data and latent spaces to match the corresponding densities. This idea has already been proposed elsewhere, and here it is applied to autoencoders. Overall I had hard time understanding the paper, the motivation, the main contribution or the claim, the model definition and the jacobian method. The paper is poorly written, with lots of issues in math notation and poor motivation and explication of what the sections are introducing, and what parts of the presentation is novel and what is already known. Lots of the math machinery is too vague to follow. The distribution p(z) is unclear, and whether z is random variable or not. It seems that “z* is a non-random variable, and then adding noise eps makes it stochastic. However, then p(z) without eps does not make any sense since z is not random. It seems that p(z) is maybe a prior distribution instead (or maybe the variational posterior?), but then adding eps noise to an already stochastic variable is strange. Overall I have hard time understanding the motivation of the two discrepancies in eq 4, what is the point of adding more noise to “z”? This seems some kind of noisy or perhaps robust AE variant, but the paper does not explicate this. I have hard time following the eqs 8-15. I am not convinced of the orthogonality argument, and I fail to see what this section tries to show or demonstrate. It seems that eq 14 is the final result, but its difficult to follow due to most terms in eq 14 being undefined. Optimizing eq 14 seems trivial since we can always match pz and pz_varphi easily with neural networks, or similarly the two x-distributions. In the experiment 4.1. the proposed method seems to achieve matching densities, although the distributions are wrongly normalized. How does the density matching improve? All three methods seem to have equally good scatters. The benchmarks on table 1 show clear improvement with the method. The face experiment is unconvincing since the VAE spreads variance across all latent dimensions while RADO seems to compress them to just first 20 or so. If one would visualise the z_100 there would be no variance in RADO and possibly some variance in VAE. The paper also should compare their model to simple MNIST/VAE to highlight what problems are there in standard approaches (such as VAE), and how does the proposed method alleviate them. Overall the paper is poorly presented and difficult to follow. Despite this the method does seem to work remarkably well, and the Jacobian idea is clearly very promising. Nevertheless in its current form the paper is badly premature for ICLR, and needs a lot more work and polish to be made understandable for wider ML audience. Minor comments o Px(x), x1, x2 are probably missing subscripts o The point of eq 5 is unclear, it seems unnecessary. It also does not contain h(), which is claimed after eq6 o The log pz(z) in eq 4 is not entropy o eq 8 is unclear, is the dx a derivative, distance or change? o the prefix notation is confusing, what does it mean? o what is the sim and line notation in eq 5? o what are the products in eq9, are these inner products? o in eq 13 pz, pxd or hat(pxd) have not been introduced or defined