SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Seunghyuk Cho
POSTECH GSAI
shhj1998@postech.ac.kr
Juyong Lee
agi.is@kaist.ac.kr
Dongwoo Kim
POSTECH GSAI & CSED
dongwoo.kim@postech.ac.kr
We propose a Gaussian manifold variational auto-encoder (GM-VAE) whose latent space consists of a set of Gaussian distributions. It is known that the set of the univariate Gaussian distributions with the Fisher information metric form a hyperbolic space, which we call a Gaussian manifold. To learn the VAE endowed with the Gaussian manifolds, we propose a pseudo-Gaussian manifold normal distribution based on the Kullback-Leibler divergence, a local approximation of the squared Fisher-Rao distance, to define a density over the latent space. We demonstrate the efficacy of GM-VAE on two different tasks: density estimation of image datasets and state representation learning for model-based reinforcement learning. GM-VAE outperforms the other variants of hyperbolic- and Euclidean-VAEs on density estimation tasks and shows competitive performance in model-based reinforcement learning. We observe that our model provides strong numerical stability, addressing a common limitation reported in previous hyperbolic-VAEs. The implementation is available at https://github.com/ml-postech/GM-VAE.
The geometry of latent space in generative models, such as variational auto-encoders (VAE) (Kingma & Welling, 2013), reflects the structure of the data representations. Mathieu et al. (2019); Nagano et al. (2019); Cho et al. (2022) show that employing hyperbolic space as the latent space improves the preservation of the hierarchical structure within the data. The theoretical background for adopting hyperbolic space lies in the analysis of Sarkar (2011); the tree-structured data can be embedded with arbitrary low distortion in hyperbolic space, while Euclidean space requires extensive dimensions.
Previously proposed hyperbolic VAEs rely on Poincaré normal distribution (Mathieu et al., 2019) or hyperbolic wrapped normal distribution Nagano et al. (2019) for the prior and variational distributions. Unlike the Gaussian distribution in Euclidean space, however, these distributions suffer from several shortcomings, including the absence of closed-form Kullback-Leibler (KL) divergence, numerical instability (Mathieu et al., 2019; Skopek et al., 2019), and high computational cost in sampling (Mathieu et al., 2019).
Meanwhile, we can form a Riemannian manifold from the set of univariate Gaussian distributions by equipping the Fisher information metric (FIM). It is known that the FIM of univariate Gaussian distributions is akin to that of the metric tensor of the Poincaré half-plane model (Costa et al., 2015), providing a perspective of viewing the points in hyperbolic space as univariate Gaussian distributions. In other words, a Gaussian distribution can be mapped to a single point in the open half-plane manifold as shown in Figure 1, where the FIM forms the shortest geodesic distance between two Gaussian distributions. Noting that the numerical issue of Poincaré normal arises from the geodesic distance of hyperbolic space, we question whether this perspective can lead us to define a new distribution with better analytic properties.
In this work, inspired by the fact that KL divergence itself is a statistical distance that locally approximates the geodesic distance (Tifrea et al., 2018), we propose a hyperbolic distribution by
(a) Gaussian manifold
(b) Univariate Gaussians
Figure 1: (a) The visualization of the Gaussian manifold consisting of a set of Gaussian distributions. Each point of the Gaussian manifold is a pair of two parameters of a univariate Gaussian distribution:  $(\mu, \sigma) \in \mathbb{R} \times \mathbb{R}_{>0}$ . The dashed lines are the geodesics, which are either the ellipses with eccentricity  $1/\sqrt{2}$  with the origin placed on the  $\mu$ -axis or straight lines parallel to the  $\sigma$ -axis. (b) Three univariate Gaussian distributions correspond to three points in the Gaussian manifold in (a).
substituting the geodesic distance of Poincaré normal with the KL divergence between the univariate Gaussian distributions. We then verify that this simple yet powerful alteration results in several practical analytic properties; the proposed distribution reduces into the product of two well-known distributions, i.e., the Gaussian and gamma distributions, which are easy to sample, with a closed-form KL divergence between the proposed distributions. By adopting the proposed hyperbolic distribution, we introduce a new variant of hyperbolic VAE, named Gaussian manifold VAE (GM-VAE), whose latent space is a set of Gaussian distributions.
During the experiments, we observe that the proposed distribution is robust in terms of sampling and KL divergence computation compared to the commonly-used hyperbolic distributions; we briefly explain the reason why others are numerically unstable. Experimental results on the density estimation task with image datasets show that GM-VAE can achieve outperforming generalization performances to unseen data against baselines of Euclidean and hyperbolic VAEs. Application of GM-VAE on model-based reinforcement learning (RL) verifies the feasibility of using hyperbolic space on another domain of task.
- We introduce a variant of VAE whose latent space is defined on a statistical manifold formed by univariate Gaussian distributions, namely Gaussian manifold.
- We propose a new distribution, called a pseudo Gaussian manifold normal distribution, which is easy to sample and has closed-form KL divergence, to train VAE on the Gaussian manifold.
- We empirically verify that the newly proposed VAE performs stable training without numerical issues on the density estimation task with several image datasets. The proposed model outperforms the baseline Euclidean VAE and other hyperbolic variants.
- We show that our method can be used for model-based RL. Specifically, we replace the latent space of the world model with hyperbolic space for learning environments, showing competitive results with a state-of-the-art baseline.
In this section, we first review the fundamental concepts of hyperbolic space and commonly used hyperbolic models. We then explain the Riemannian geometry between statistical objects, showing the connection between the statistical manifold and hyperbolic space.
2.1 Review on hyperbolic space
Riemannian manifold. A  $n$ -dimensional Riemannian manifold consists of a manifold  $\mathcal{M}$  and a metric tensor  $g: \mathcal{M} \to \mathbb{R}^{n \times n}$ , which is a smooth map from each point  $\mathbf{x} \in \mathcal{M}$  to a symmetric
positive definite matrix. The metric tensor  $g(\mathbf{x})$  defines the inner product of two tangent vectors for each point of the manifold  $\langle \cdot, \cdot \rangle_{\mathbf{x}}: \mathcal{T}_{\mathbf{x}}\mathcal{M} \times \mathcal{T}_{\mathbf{x}}\mathcal{M} \to \mathbb{R}$ , where  $\mathcal{T}_{\mathbf{x}}\mathcal{M}$  is the tangent space of  $\mathbf{x}$ .
The metric tensor induces basic Riemannian operations, such as a geodesic, exponential map, log map, and parallel transport. Given two points  $\mathbf{x},\mathbf{y}\in \mathcal{M}$ , geodesic  $\gamma_{\mathbf{x}}:[0,1]\to \mathcal{M}$  is a unit speed curve on  $\mathcal{M}$  being the shortest path between  $\gamma (0) = \mathbf{x}$  and  $\gamma (1) = \mathbf{y}$ . This curve can be interpreted as a generalized path of a straight line in Euclidean space. The exponential map  $\exp_{\mathbf{x}}:\mathcal{T}_{\mathbf{x}}\mathcal{M}\rightarrow \mathcal{M}$  is defined as  $\exp_{\mathbf{x}}(\mathbf{v}) = \gamma (1) = \mathbf{y}$  given  $\gamma$  is a geodesic starting from  $\gamma (0) = \mathbf{x}$  and  $\gamma^{\prime}(0) = \mathbf{v}\in \mathcal{T}_{\mathbf{x}}\mathcal{M}$ . The log map  $\log_{\mathbf{x}}:\mathcal{M}\to \mathcal{T}_{\mathbf{x}}\mathcal{M}$  is the inverse of the exponential map, i.e.,  $\log_{\mathbf{x}}(\exp_{\mathbf{x}}(\mathbf{v})) = \mathbf{v}$ . The parallel transport  $\mathrm{PT}_{\mathbf{x}\to \mathbf{y}}:\mathcal{T}_{\mathbf{x}}\mathcal{M}\to \mathcal{T}_{\mathbf{y}}\mathcal{M}$  moves the tangent vector  $\mathbf{v}$  along the geodesic between  $\mathbf{x}$  and  $\mathbf{y}$ . The geodesic distance  $d_{\mathcal{M}}(\mathbf{x},\mathbf{y})$  can be induced by the metric tensor as follows:
$$
d _ {\mathcal {M}} (\mathbf {x}, \mathbf {y}) = \int_ {0} ^ {1} \sqrt {\langle \dot {\gamma} (t) , \dot {\gamma} (t) \rangle_ {\gamma (t)}} d t.
$$
Hyperbolic space. One method of classifying Riemannian manifolds, a basic question of differential geometry, is based on curvature. Among different types of curvatures, one popular curvature is the sectional curvature  $\kappa_{g}$ , which is a generalization of the Gaussian curvature in classical surface geometry. Given two linearly independent vector fields  $X,Y\in \mathfrak{X}(\mathcal{M})$ , the sectional curvature  $\kappa_{g}$  can be computed with Riemannian curvature tensor  $R\colon \mathfrak{X}(\mathcal{M})\times \mathfrak{X}(\mathcal{M})\times \mathfrak{X}(\mathcal{M})\to \mathfrak{X}(\mathcal{M})$  as below:
$$
\kappa_ {g} = \frac {\langle R (X , Y) Y , X \rangle}{\langle X , X \rangle \langle Y , Y \rangle - \langle X , Y \rangle^ {2}},
$$
where  $R(X,Y)Y$  returns a tensor field assigning a tensor to each point of the Riemannian manifold  $\mathcal{M}$ . The hyperbolic space is a Riemannian manifold that has the sectional curvature value of constant negative (Nickel & Kiela, 2018). The hyperbolic space is known to be able to embed tree-structured data with arbitrarily low distortion (Sarkar, 2011).
Hyperbolic models. We utilize three famous models of hyperbolic space: the Poincaré disk model, the Lorentz model, and the Poincaré half-plane model.
The Poincaré disk model is a hyperbolic space with an open disk manifold. Earlier hyperbolic machine learning work uses the Poincaré disk model because it has a simple closed-form of the operations, such as exponential and log maps (Mathieu et al., 2019). However, the Poincaré disk model suffers from numerical stability issues when the points exist near the boundary of the manifold.
The Lorentz model is often used as an alteration of the Poincaré disk model (Nickel & Kiela, 2018; Nagano et al., 2019; Bose et al., 2020; Cho et al., 2022). The Lorentz model uses a half hyperboloid manifold, where the closed form of the Riemannian operations exists, so the numerical stability issue of employing the Poincaré disk model is relieved.
The Poincaré half-plane model is another well-known model of hyperbolic space with an open half-plane manifold. The metric tensor of a point of the two-dimensional Poincaré half-plane model  $(x,y)$  is  $y^{-2}\mathrm{diag}(1,1)$ .
Numerical stability issues of the hyperbolic models. Hyperbolic space suffers from numerical stability when applied to machine learning algorithms (Yu & Sa, 2021; Skopek et al., 2019; Mathieu et al., 2019). The numerical stability mainly occurs for two reasons: machine precision error and unstable Riemannian operations.
First, due to the machine precision error, the hyperbolic points represented with floating point differ from the real value (Yu & Sa, 2021, 2019). In contrast to Euclidean space, the points of hyperbolic space need to satisfy manifold constraints, e.g., the Poincaré disk model allows points whose Euclidean norm is less than one. A point can be placed near the boundary during the optimization or inference processes. Although the point does not violate the manifold constraint in theory, it can be located on or out of the boundary when represented with a floating point due to machine precision error. We empirically observe that the manifold constraint violation occurs frequently when we need to embed many data points in hyperbolic space. Figure 3 demonstrates the machine precision error of each hyperbolic model.
Second, the Riemannian operations of hyperbolic space can result in a not-a-number (NaN) value when the input value is not in the manifold. For example, the geodesic distance from the Poincaré
disk model and the log mapping of the Lorentz model are unstable Riemannian operations, which are written as:
$$
d _ {\mathcal {P}} (\mathbf {x}, \mathbf {y}) = \cosh^ {- 1} \left(1 + 2 \frac {\| \mathbf {x} - \mathbf {y} \| ^ {2}}{(1 - \| \mathbf {x} \| ^ {2}) (1 - \| \mathbf {y} \| ^ {2})}\right), \log_ {\mathbf {u}} (\mathbf {v}) = \frac {\cosh^ {- 1} (\alpha)}{\sqrt {\alpha^ {2} - 1}} (\mathbf {v} - \alpha \mathbf {u}),
$$
where  $\alpha = \mathbf{u}_0\mathbf{v}_0 - \sum_{i=1}^n \mathbf{u}_i\mathbf{v}_i$  is the Lorentzian inner product of  $\mathbf{u}$ ,  $\mathbf{v}$ . For the Poincaré disk model, when the points  $\mathbf{x}, \mathbf{y}$  are near the boundary of the unit disk, the floating point representation of the norm values  $\|\mathbf{x}\|^2$ ,  $\|\mathbf{y}\|^2$  becomes one. The denominator of the geodesic distance then becomes zero. For the Lorentz model, if  $\mathbf{u} = \mathbf{v}$  and  $\mathbf{u}$  contains large values in the coordinates,  $\alpha$  becomes less than one which results in NaN because the domain of  $\cosh^{-1}(x)$  is  $x \geq 1$ .
2.2 Statistical manifold of univariate Gaussians
A particular case of the Riemannian manifold is a statistical manifold, where each point in the manifold corresponds to a probability distribution. Specifically, the parameter manifold  $\mathcal{M}$  of the probability distributions  $p_{\theta}:\mathcal{X}\to \mathbb{R}$ , where  $\theta \in \mathcal{M}$ , equipped with the Fisher information metric (FIM) forms a Riemannian manifold (Rao, 1992). The FIM is defined as:
$$
g _ {i j} (\boldsymbol {\theta}) = \int_ {\mathcal {X}} \frac {\partial \log p _ {\theta} (x)}{\partial \theta_ {j}} \frac {\partial \log p _ {\theta} (x)}{\partial \theta_ {j}} p _ {\theta} (x) d x.
$$
In the parameter space of univariate Gaussian distributions  $\{(\mu, \sigma) \mid \mu \in \mathbb{R}, \sigma \in \mathbb{R}_{>0}\}$ , the FIM can be simplified as two-dimensional diagonal matrix  $\sigma^{-2} \mathrm{diag}(1, 2)$  (Costa et al., 2015).
Connection to the Poincaré half-plane model. The diagonal form of the FIM implies that the Riemannian manifold with  $\{(\mu, \sigma) \mid \mu \in \mathbb{R}, \sigma \in \mathbb{R}_{>0}\}$  has the same set of points as the manifold of the Poincaré half-plane, but with different curvature value of  $-0.5$ .
The parameter space of the  $n$ -dimensional diagonal Gaussian distributions becomes the product of  $n$  manifolds of the parameter space of univariate Gaussian distributions. In turn, the statistical manifold of  $n$ -dimensional diagonal Gaussian distributions can be viewed as the product of  $n$  hyperbolic spaces. The operations on the product of the Riemannian manifolds  $\bigotimes_{i=1}^{n}\mathcal{M}_{i}$  are defined manifold-wise. For example, an exponential map applied on a point  $(p_i)_{i=1}^n \in \bigotimes_{i=1}^n\mathcal{M}_i$ , with tangent vector  $v_i \in \mathcal{T}_{p_i}\mathcal{M}_i$  for each  $i \in \{1, \dots, n\}$ , can be represented as  $(\exp_{p_i}(v_i))_{i=1}^n$ .
Distance in the statistical manifold. In the statistical manifold, distance functions measure the difference between two distributions on the statistical manifold. One example is the geodesic distance derived from the FIM, which is called the Fisher-Rao distance. The Fisher-Rao distance of the statistical manifold of univariate Gaussian distributions is the same as the geodesic distance of the Poincaré half-plane model with constant negative curvature  $-0.5$ .
KL divergence is another widely-used statistical distance for distributions, defined as  $D_{\mathrm{KL}}(p \parallel q) := \int_x p(x) \log \frac{p(x)}{q(x)} dx$  for two distributions  $p, q$  in the same statistical manifold. One notable property of KL divergence is that it can locally approximate the squared Fisher-Rao distance (Tifrea et al., 2018):
$$
D _ {\mathrm {K L}} \left(p (\cdot ; \boldsymbol {\theta} + d \boldsymbol {\theta}) \| p (\cdot ; \boldsymbol {\theta})\right) = \frac {1}{2} \sum_ {i j} g _ {i j} (\boldsymbol {\theta}) d \theta_ {i} d \theta_ {j} + \mathcal {O} \left(\| d \boldsymbol {\theta} \| ^ {3}\right).
$$
In this section, we first present the concept of the Gaussian manifold, which can have an arbitrary curvature by reparameterizing univariate Gaussian distribution. We then propose a pseudo Gaussian manifold normal distribution. Finally, we suggest a new variant of the VAE defined over the Gaussian manifold with PGM normal as prior. We denote the density function of the Gaussian distribution as  $\mathcal{N}(x;\mu ,\sigma^2) = 1 / (\sqrt{2\pi\sigma^2})\exp \left(-(\mu -x)^2 /(2\sigma^2)\right)$ .
3.1 Gaussian manifold with arbitrary curvature
Previous studies on hyperbolic space emphasize the importance of having an arbitrary curvature (Skopek et al., 2019; Mathieu et al., 2019). These works empirically show that the generalization performances of hyperbolic VAEs can be improved with varying curvatures. However, as shown in Section 2.2, the univariate Gaussian distributions form a manifold with curvature value of  $-0.5$ , limiting the flexibility of the manifold.
We show that the statistical manifold of univariate Gaussian distributions can have an arbitrary curvature by reparameterizing the univariate Gaussian distribution properly. Let  $\mathcal{N}(\sqrt{2c}\mu, \sigma^2)$  be the reparameterized univariate Gaussian distribution with additional parameter  $c > 0$ . The reparameterization leads to the FIM of  $\sigma^{-2}\mathrm{diag}(1,1/c)$  showing that the curvature of the statistical manifold is  $-c$ . The computation of the sectional curvature of the extended FIM is described in Appendix B.1.
We call the statistical manifold with the reparameterized univariate Gaussian distributions and the extended FIM as the Gaussian manifold and denote it as  $\mathcal{G}_c$ , where  $-c$  is the curvature of the Gaussian manifold.
We then verify that the KL divergence between the distributions of the Gaussian manifold approximates the geodesic distance, even in the presence of arbitrary curvature in the Gaussian manifold. Let  $(\mu, \sigma) \in \mathcal{G}_c$  be an arbitrary point of the Gaussian manifold. The KL divergence between  $(\mu, \sigma)$  and its neighbor  $(\mu + d\mu, \sigma + d\sigma)$  can be computed as:
$$
\frac {D _ {\mathrm {K L}} \left(\mathcal {N} \left(\sqrt {2 c} (\mu + d \mu) , (\sigma + d \sigma) ^ {2}\right) \mid \mid \mathcal {N} \left(\sqrt {2 c} \mu , \sigma^ {2}\right)\right)}{2 c} = \frac {1}{2} \binom {d \mu} {d \sigma} ^ {\top} \left( \begin{array}{c c} \frac {1}{\sigma^ {2}} & 0 \\ 0 & \frac {1}{c \sigma^ {2}} \end{array} \right) \binom {d \mu} {d \sigma} + \mathcal {O} ((d \sigma) ^ {3}), \tag {1}
$$
where the first term is the squared Riemannian norm of the tangent vector  $(d\mu, d\sigma)$  approximating the squared Fisher-Rao distance. The detailed derivation of the KL divergence of the Gaussian manifold is described in Appendix B.2.
3.2 Pseudo Gaussian manifold normal distribution
We propose a pseudo Gaussian manifold (PGM) normal distribution defined over the Gaussian manifold. Let  $(\mu, \sigma) \in \mathcal{G}_c$  be a point in the Gaussian manifold. Inspired by the Riemannian normal distribution (Pennec, 2006), we define the probability density function of PGM normal with the KL divergence as:
$$
\mathcal {K} _ {c} (\mu , \sigma ; \alpha , \beta , \gamma) = \frac {\sigma^ {3}}{Z (c , \beta , \gamma)} \times \exp \left(- \frac {D _ {\mathrm {K L}} \left(\mathcal {N} \left(\sqrt {2 c} \cdot \mu , \sigma^ {2}\right) \| \mathcal {N} \left(\sqrt {2 c} \cdot \alpha , \beta^ {2}\right)\right)}{2 c \cdot \gamma^ {2}}\right), \tag {2}
$$
where  $(\alpha, \beta) \in \mathcal{G}_c$ , and  $\gamma \in \mathbb{R}_{>0}$  are the parameters of the distribution. The distribution is centered at  $(\alpha, \beta)$  with additional scale parameter  $\gamma$ . We verify the convergence of the PGM normal and compute the normalizing constant  $Z(c, \beta, \gamma)$  over the probability measure of the Gaussian manifold at Appendix C.1. As shown in Equation 1, the KL divergence of the Gaussian manifold approximates the Fisher-Rao distance between  $\mathcal{N}(\sqrt{2c} \cdot \alpha, \beta^2)$  and  $\mathcal{N}(\sqrt{2c} \cdot \mu, \sigma^2)$ . Therefore, the PGM normal accounts for the geometric structure of the univariate Gaussian distributions.
The factorization of the probability density function in Equation 2 multiplied with the square root of the determinant of the FIM shows the advantages of the PGM normal, which can be written as:
$$
\mathcal {K} _ {c} (\mu , \sigma ; \alpha , \beta , \gamma) \cdot \sqrt {\det  (g)} = \mathcal {N} (\mu ; \alpha , \beta^ {2} \gamma^ {2}) \cdot 2 \sigma \operatorname {G a m m a} \left(\sigma^ {2}; \frac {1}{4 c \gamma^ {2}} + 1, \frac {1}{4 c \beta^ {2} \gamma^ {2}}\right), \tag {3}
$$
where  $\mathrm{Gamma}(z;a,b) = \frac{b^a}{\Gamma(a)} z^{a - 1}\exp (-bz)$  and  $g$  is the FIM of the Gaussian manifold. The  $\sqrt{\operatorname*{det}g}$  term enables us to sample and compute the KL divergence in an Euclidean manner. Thanks to the properties of Gaussian and gamma distributions, the PGM normal is easy to sample and has a closed-form KL divergence. The detailed derivation is available in Appendix C.2 and Appendix C.3. The factorization has the same form as the well-known conjugate prior to the Gaussian distribution. In that sense, the PGM normal explicitly incorporates the geometric structure between Gaussians into the known prior distribution.
We note that the PGM normal can be easily extended for the diagonal Gaussian manifold, a manifold formed by diagonal Gaussian distributions since the diagonal Gaussian manifold is the product of the Gaussian manifolds.
3.3 Gaussian manifold VAE
We introduce a Gaussian manifold VAE (GM-VAE) whose latent space is defined over the diagonal Gaussian manifold with the help of the PGM normal. We use the PGM normal for variational and prior distributions. To be specific, with the PGM normal, the evidence lower bound (ELBO) of the GM-VAE can be formalized with the diagonal Gaussian manifold  $\{(\pmb {\mu},\Sigma)\mid \pmb {\mu}\in \mathbb{R}^n,\Sigma \in \mathbb{R}_{>0}^n\}$  as:
$$
\mathbb {E} _ {q _ {\phi} (\pmb {\mu}, \Sigma | \mathbf {x}) \cdot \sqrt {\det (g)}} \left[ \log p _ {\theta} (\mathbf {x} \mid \pmb {\mu}, \Sigma) \right] - D _ {\mathrm {K L}} \left(q _ {\phi} (\pmb {\mu}, \Sigma \mid \mathbf {x}) \cdot \sqrt {\det (g)} \parallel p (\pmb {\mu}, \Sigma) \cdot \sqrt {\det (g)}\right),
$$
where  $p_{\theta}(\mathbf{x}\mid \boldsymbol {\mu},\Sigma)$  is the decoder network,  $q_{\phi}(\boldsymbol {\mu},\Sigma \mid \mathbf{x})$  is the encoder network and  $p(\boldsymbol {\mu},\Sigma)$  is the prior. The variational distribution is set to  $q_{\phi}(\boldsymbol {\mu},\Sigma \mid \mathbf{x}) = \mathcal{K}_{c}(\alpha_{\phi}(\mathbf{x}),\beta_{\phi}(\mathbf{x}),\gamma_{\phi}(\mathbf{x}))$  , where  $\alpha_{\theta}(\mathbf{x})\in$ $\mathbb{R}^n$  and  $\beta_{\phi}(\mathbf{x}),\gamma_{\phi}(\mathbf{x})\in \mathbb{R}_{>0}^{n}$  , and the prior is set to  $p(\boldsymbol {\mu},\Sigma) = \mathcal{K}_c(\mathbf{0},I,I)$  in our experiments given curvature  $-c$  . The pseudo-algorithm for the decoder of GM-VAE is present at Algorithm 1.
Information geometry on VAE. Focusing on the bridge between probability theory and differential geometry, the adaptation of information geometry to the deep learning framework has been investigated in various aspects (Karakida et al., 2019; Bay & Sengupta, 2017; Gomes et al., 2022). Having said that, Han et al. (2020) show that the training process of VAE can be seen as minimizing the distance between the two statistical manifolds: manifolds with the parameters of the decoder and the encoder. Not only can the parameters but the outputs from the VAE decoder be modeled as probability distributions. Arvanitidis et al. (2021) suggest a method of using the pull-back metric defined with arbitrary decoders on the latent space. Our work focuses more on the statistical manifolds lying on the outputs of the encoder with the benefits from the information geometry.