SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Huh $^{1,*}$ , Changwook Jeong $^{2,\dagger}$ , Jae Myung Choe $^{1}$ , Young-Gu Kim $^{1}$ , Dae Sin Kim $^{1}$ $^{1}$ CSE Team, Innovation Center, Samsung Electronics  
 $^{2}$ Graduate School of Semiconductor Materials and Devices Engineering, UNIST  
*in.huh@samsung.com †changwook.jeong@unist.ac.kr
We study structure-preserving low-dimensional representation of a data manifold embedded in a high-dimensional observation space based on variational auto-encoders (VAEs). We approach this by decomposing the data manifold  $\mathcal{M}$  as  $\mathcal{M} = \mathcal{M} / G \times G$ , where  $G$  and  $\mathcal{M} / G$  are a group of symmetry transformations and a quotient space of  $\mathcal{M}$  up to  $G$ , respectively. From this perspective, we define the structure-preserving representation of such a manifold as a latent space  $\mathcal{Z}$  which is isometrically isomorphic (i.e., distance-preserving) to the quotient space  $\mathcal{M} / G$  rather  $\mathcal{M}$  (i.e., symmetry-preserving). To this end, we propose a novel auto-encoding framework, named isometric quotient VAEs (IQVAEs), that can extract the quotient space from observations and learn the Riemannian isometry of the extracted quotient in an unsupervised manner. Empirical proof-of-concept experiments reveal that the proposed method can find a meaningful representation of the learned data and outperform other competitors for downstream tasks.
There has been a common consensus that natural image datasets form a low-dimensional manifold  $\mathcal{M}$  embedded in a high-dimensional observation space  $\mathbb{R}^d$ , i.e.,  $\dim(\mathcal{M}) \ll d$  [6]. From this perspective, a good neural network is a mapping function that can recognize the underlying structure of the data manifold well [25]. A question that arises is what structures should be represented from the data.
For unsupervised learning task, especially in generative modeling, if we suppose the data manifold to be a Riemannian one, then preserving geometric structures in the representation space is a key consideration [1, 8, 25]. To make things more explicit, if  $\mathcal{Z}$  is a latent representation of  $\mathcal{M}$ , then  $\mathcal{Z}$  should be isometrically isomorphic to  $\mathcal{M}$  in the sense of the Riemannian isometry. This means that an infinitesimal distance on the data manifold should be preserved in the latent space as well, promoting geodesic distances in the latent space that accurately reflect those in the data space. Recently, several papers [8, 40, 25] have suggested using the concept of Riemannian isometry for deep generative models, such as variational auto-encoders (VAEs) [22] and generative adversarial networks (GANs) [14], to obtain more meaningful and relevant representations. These concepts have been applied to improve latent space interpolations [30] or clustering [45] of data manifolds.
However, the Riemannian isometry does not tell the whole story of the data structure, particularly in the case of vision datasets. One common property of natural images is that there are symmetry transformations that preserve the semantic meaning of the given image. For example, if one applies a rotational transformation to a certain image, then its semantic meaning, e.g., its label, does not change. In this situation, what one really wants to do is represent the inherent geometry of the manifold, i.e., the geometry up to such symmetry transformations [29].
The concept of the inherent geometry can be formalized by using the notion of the principal bundle  $\mathcal{M} = \mathcal{M} / G\times G$ , a fiber bundle that consists of the group  $G$  of symmetry transformations and a quotient space  $\mathcal{M} / G$  as the fiber and base spaces, respectively [16, 26]. In this case, the inherent geometry of  $\mathcal{M}$  up to  $G$  is indeed determined by its quotient  $\mathcal{M} / G$  solely; the quotient formulates an invariant structure of the data, thus a measure on  $\mathcal{M} / G$  gives a measure on  $\mathcal{M}$  that is invariant to the actions of  $G$ . Therefore, one should find the Riemannian isometry of  $\mathcal{M} / G$  rather than  $\mathcal{M}$  for a geometricmeaningful representation of the dataset (see Figure 1). Nevertheless, to the best of our knowledge, there has been a lack of studies, at least in the field of generative models, that integrate both the concepts of the quotient space and its Riemannian geometry.
In this paper, we propose a novel auto-encoding framework, named isometric quotient VAEs (IQVAEs), that can extract the quotient space from observations and learn its Riemannian isometry in an unsupervised manner, allowing for a cor
rect representation of the data manifold and accurate measurement of distances between samples from the learned representation. The proposed IQVAE model can be applied to a variety of practical tasks such as downstream clustering and out-of-distribution (OoD) detection. We evaluate the model's performance using three datasets: rotated MNIST [44], mixed-type wafer defect maps (MixedWM38) [42] and cervical cancer cell images (SIPaKMeD) [32]. In summary, our contribution is threefold:
- We propose a novel approach for structure-preserving representation learning by viewing a data manifold as a principal bundle, and formulate it as an unsupervised learning task that finds a Riemannian isometry for the quotient of the data manifold.
- We introduce a practical method, called the IQVAE, which can learn such a structure-preserving representation in an unsupervised manner by using the auto-encoding framework.
- We demonstrate through experimental evaluations on various datasets that our proposed method can find useful representations and outperform other competing methods for downstream tasks such as classification, clustering, and OoD detection.
2.1  $G$ -Invariance and Quotient Space Learning
Quotient space learning is highly related to the notion of the  $G$ -invariance. Data augmentation is the most common approach to deal with the  $G$ -invariance of label distributions [18]. To achieve such a symmetry more explicitly, various  $G$ -invariant and  $G$ -equivariant architectures have been proposed; [10] has proposed  $G$ -convolution beyond the conventional translational equivariance of convolutional neural networks. [12] has generalized it further for Lie groups.  $G$ -orbit pooling, proposed by [23], can be applied to both arbitrary discrete groups and continuous groups by discretizing them. Some research has attempted to discover the underlying invariances within data [4, 46, 31, 35]. These approaches can be integrated as a major component of auto-encoders to represent the quotient symmetry of the data manifold, as we will discuss later in Section 3.3.
More relevant to our work, [37] has proposed using the consistency regularization for VAEs to encourage consistency in the latent variables of two identical images with different poses. It can be considered as the unsupervised learning version of the data augmentation and does not guarantee the explicit  $G$ -invariance of the regularized latent space. [5] has disentangled image content from pose variables, e.g., rotations or translations under the VAE framework. However, it has some limitations,
Figure 1: We denote  $\mathcal{X} \subset \mathbb{R}^d$  as an extrinsic view of the data manifold  $\mathcal{M}$ . It consists as  $\mathcal{X} = \mathcal{X} / G \times G$  where  $\mathcal{X} / G$  is a quotient space of  $\mathcal{X}$  and  $G$  is a group of symmetry transformations that naturally act on  $\mathcal{X}$ , respectively. From this perspective, the structure-preserving representation  $\mathcal{Z}$  should preserve all geodesic distances of  $\mathcal{X} / G$  ( $d_{\mathcal{X} / G}(\cdot, \cdot)$ ) rather than them of  $\mathcal{X}(d_{\mathcal{X}}(\cdot, \cdot))$ , i.e.,  $d_{\mathcal{Z}}(\mathbf{z}_1, \mathbf{z}_2) = c \cdot d_{\mathcal{X} / G}(\hat{\mathbf{x}}_1, \hat{\mathbf{x}}_2)$ ,  $\forall \hat{\mathbf{x}}_1, \hat{\mathbf{x}}_2 \in \mathcal{X} / G, \mathbf{z}_1, \mathbf{z}_2 \in \mathcal{Z}$ , where  $d_{\mathcal{Z}}(\cdot, \cdot)$  is a distance on  $\mathcal{Z}$ . Such a space  $\mathcal{Z}$  preserves all the essential geometry and symmetry of the data.
such as the requirement of using a specific decoder, the spatial transformer network [20]. It also has a problem that it cannot handle arbitrary groups beyond the rotation and translation. [11] has dealt with more complicated SO(3) group under the VAE framework. [29] has proposed the quotient auto-encoder (QAE) that can learn the quotient space of observations with respect to a given arbitrary group, extending early works in quotient images [34, 41]. However, it is a deterministic model and therefore cannot generate new samples easily. In addition, all the above-mentioned studies have not considered the Riemannian geometry of the learned latent representation.
2.2 Rimeannian Geometry of Generative Models
Recent studies have proposed to view the latent space of generative models as a Riemannian manifold [1, 9, 7]. In this perspective, the metric tensor of the latent space is given by a pullback metric induced from the observation space through the generator function. In line with this theoretical perspective, the flat manifold VAE (FMVAE) [8] and its variant [25, 15] have been proposed to regularize the generator function to be a Riemannian isometry by matching the pullback metric with the Euclidean one. The regularized latent space preserves geodesic distances of the observation space, making it a more geometrically meaningful representation that can be useful. However, the FMVAE and its variants do not take into account the underlying group symmetry, which can lead to inaccurate estimation of distances between data points. For example, they could estimate a non-zero distance between two identical images with different rotational angles, which is unsuitable for clustering tasks.
Suppose  $\mathbf{x} \in \mathcal{X} \subset \mathbb{R}^d$  is an observation where  $\mathcal{X}$  is an extrinsic view of  $\mathcal{M}$ , i.e., the data manifold is realized via a mapping  $\pi: \mathcal{M} \to \mathbb{R}^d$  such that  $\mathcal{X} = \pi(\mathcal{M})$ , and  $\mathbf{z} \in \mathcal{Z} \subset \mathbb{R}^n$  is its latent representation where  $n \ll d$ . In addition, suppose  $G$  to be a group (of symmetry transformations) that its group elements  $g \in G$  naturally act on  $\mathcal{X}$  by a left group action  $\alpha: G \times \mathcal{X} \to \mathcal{X}$ . We will denote  $\alpha(g, \mathbf{x})$  as  $g * \mathbf{x}$ . Typical examples of  $G$  include the special orthogonal group SO(2), a group of 2-dimensional rotations that can also naturally act on  $d$ -dimensional images by rotating them.
3.1 Auto-Encoders
The auto-encoder framework consists of two parameterized neural networks, an encoder  $\mu_{\theta}:\mathbb{R}^{d}\to$ $\mathbb{R}^n$  and a decoder  $\mu_{\phi}:\mathbb{R}^n\to \mathbb{R}^d$ . Vanilla auto-encoders find a low-dimensional compression of observations by minimizing the following reconstruction loss:
$$
\mathbb {E} _ {\mathbf {x} \sim p _ {\mathcal {X}} (\mathbf {x})} \left[ \mathcal {L} _ {\mathrm {A E}} (\theta , \phi ; \mathbf {x}) \right] = \mathbb {E} _ {\mathbf {x} \sim p _ {\mathcal {X}} (\mathbf {x})} \left[ \| \mathbf {x} - \mu_ {\phi} \circ \mu_ {\theta} (\mathbf {x}) \| _ {2} ^ {2} \right], \tag {1}
$$
where  $p_{\mathcal{X}}(\mathbf{x})$  is the data distribution. The expectation over  $p_{\mathcal{X}}(\mathbf{x})$  can be computed via Monte Carlo (MC) estimation with finite samples  $\mathrm{X} = \{\mathbf{x}_i\}_{i=1}^N$  as  $\mathbb{E}_{p_{\mathcal{X}}(\mathbf{x})}\left[\mathcal{L}_{\mathrm{AE}}(\theta, \phi; \mathbf{x})\right] \approx (1/N)\sum_{i=1}^N\|\mathbf{x}_i - \mu_{\phi} \circ \mu_{\theta}(\mathbf{x}_i)\|_2^2$ . Unless otherwise mentioned, we will omit the expectation over  $p_{\mathcal{X}}(\mathbf{x})$  for brevity.
Although sufficiently deep auto-encoders can provide effective latent compression  $\mu_{\theta}(\mathbf{X}) = \mathbf{Z}\subset \mathbb{R}^n$  of high-dimensional data, they tend to overfit and the learned latent manifolds are often inaccurate [25], i.e., they cannot consider the underlying geometry of data. This can lead to problems such as incorrect latent interpolation or poor performance in downstream tasks, in addition to the absence of the ability to generate new samples.
3.2 Variational Auto-Encoders
The VAE [22] is a stochastic auto-encoding architecture belonging to the families of deep generative models. Contrary to vanilla auto-encoders, the VAE framework consists of two parameterized distributions  $q_{\theta}(\mathbf{z}|\mathbf{x})$  and  $p_{\phi}(\mathbf{x}|\mathbf{z})$  where the former and the latter are variational posterior and likelihood, respectively. VAEs try to maximize the marginal log-likelihood of observations by optimizing the following evidence lower bound:
$$
\log p _ {\phi} (\mathbf {x}) \geq \mathbb {E} _ {\mathbf {z} \sim q _ {\theta} (\mathbf {z} | \mathbf {x})} \left[ \log p _ {\phi} (\mathbf {x} | \mathbf {z}) \right] - D _ {\mathrm {K L}} \left(q _ {\theta} (\mathbf {z} | \mathbf {x}) \| p _ {\mathcal {Z}} (\mathbf {z})\right) \triangleq - \mathcal {L} _ {\mathrm {V A E}} (\theta , \phi ; \mathbf {x}), \tag {2}
$$
where  $p_{\mathcal{Z}}(\mathbf{z})$  is the latent prior and  $D_{\mathrm{KL}}(\cdot ,\cdot)$  is the Kullback-Leibler (KL) divergence.  $p_{\mathcal{Z}}(\mathbf{z})$  is often given by a normal distribution  $\mathcal{N}(\mathbf{z}|\mathbf{0},\mathbf{I}_n)$  where  $\mathbf{0}$  is a zero vector and  $\mathbf{I}_n$  is  $n\times n$  identity matrix.
For vanilla VAEs,  $q_{\theta}(\mathbf{z}|\mathbf{x})$  is chosen as a multivariate Gaussian distribution with a diagonal covariance matrix  $\mathcal{N}(\mathbf{z}|\mu_{\theta}(\mathbf{x}),\mathrm{diag}[\sigma_{\theta}^{2}(\mathbf{x})])$  and represented by a neural network encoder  $(\mu_{\theta},\sigma_{\theta}):\mathbb{R}^d\to \mathbb{R}^n\times \mathbb{R}_+^n$ . A latent variable is sampled by using the reparameterization trick  $\mathbf{z} = \mu_{\theta}(\mathbf{x}) + \sigma_{\theta}(\mathbf{x})\odot \epsilon$  where  $\epsilon \sim \mathcal{N}(\epsilon |0,\mathbf{I}_n)$ . Although  $p_{\phi}(\mathbf{x}|\mathbf{z})$  is chosen depending on the modeling of the data, it is often taken as a simple distribution such as a Gaussian with fixed variance,  $\mathcal{N}(\mathbf{x}|\mu_{\phi}(\mathbf{z}),\beta \mathbf{I}_{d})$ , represented by a neural network decoder  $\mu_{\phi}:\mathbb{R}^n\to \mathbb{R}^d$ . In this case, the VAE objective (2) can be seen as a regularized version of (1) given by a sum of the stochastic auto-encoding reconstruction term  $\mathcal{L}_{\mathrm{AE}}(\theta ,\phi ;\mathbf{x},\epsilon) = \| \mathbf{x} - \mu_{\phi}(\mu_{\theta}(\mathbf{x}) + \sigma_{\theta}(\mathbf{x})\odot \epsilon)\| _2^2$  and  $\beta$ -weighted KL divergence  $\mathcal{L}_{\mathrm{KL}}(\theta ;\mathbf{x},\beta) = \beta D_{\mathrm{KL}}(\mathcal{N}(\mathbf{z}|\mu_{\theta}(\mathbf{x}),\mathrm{diag}[\sigma_{\theta}^{2}(\mathbf{x})]))\| \mathcal{N}(\mathbf{z}|\mathbf{0},\mathbf{I}_n))$  as follows:
$$
\mathcal {L} _ {\mathrm {V A E}} (\theta , \phi ; \mathbf {x}, \beta) = \mathbb {E} _ {\epsilon \sim \mathcal {N} (\epsilon)} \left[ \mathcal {L} _ {\mathrm {A E}} (\theta , \phi ; \mathbf {x}, \epsilon) \right] + \mathcal {L} _ {\mathrm {K L}} (\theta ; \mathbf {x}, \beta). \tag {3}
$$
Practically, the expectation over  $\mathcal{N}(\epsilon)$  term is estimated via a single-point MC when the mini-batch size is sufficiently large [22]. When  $\beta \rightarrow 0$  and  $\sigma_{\theta}^{2}\rightarrow 0$ , VAEs reduce to vanilla auto-encoders.
By minimizing (3), VAEs can learn the probabilistic process on the smooth latent space [13] that can easily generate plausible new samples by sampling a latent variable and decoding it, i.e.,  $\mu_{\phi}(\mathcal{Z}) \approx \mathcal{X}$ . However, VAEs, like vanilla auto-encoders, do not ensure that the learned latent representation preserves the crucial information about the symmetry and geometry of the data manifold.
3.3 Quotient Auto-Encoders
QAE [29] is a modification of the deterministic auto-encoding framework that can find the quotient of the observation set  $\mathrm{X} / G$  by replacing (1) as the following quotient reconstruction loss:
$$
\mathcal {L} _ {\mathrm {Q A E}} (\theta , \phi ; \mathbf {x}, G) = \inf  _ {g \in G} \| g * \mathbf {x} - \mu_ {\phi} \circ \mu_ {\theta} (\mathbf {x}) \| _ {2} ^ {2}. \tag {4}
$$
As shown in (4), QAEs aim to minimize the set distance between the auto-encoded sample  $\mu_{\phi} \circ \mu_{\theta}(\mathbf{x})$  and  $G$ -orbit  $G * \mathbf{x} = \{g * \mathbf{x} | g \in G\}$  of the given  $\mathbf{x}$  rather than  $\mathbf{x}$  itself. It is noteworthy that  $G$ -orbit loses the information of  $G$  for a given  $\mathbf{x}$ , which is the case of the natural quotient for  $\mathbf{x}$  up to  $G$ . Thus, QAE can learn the quotient space of observations directly: for the ideal case of QAEs, i.e., when  $\mathcal{L}_{\mathrm{QAE}}(\theta, \phi; \mathbf{x}, G) \to 0$ , the following property holds:
$$
\mu_ {\phi} \circ \mu_ {\theta} (\mathbf {x}) = \mu_ {\phi} \circ \mu_ {\theta} (\mathbf {x} = g * \mathbf {x}) \triangleq \hat {\mathbf {x}}, \quad \forall g \in G,
$$
which means that the image  $\mu_{\phi} \circ \mu_{\theta}(\mathbf{X}) \triangleq \hat{\mathbf{X}}$  does not contain any information on  $g$ . In addition, QAEs typically adopt  $G$ -invariant architectures such as  $G$ -orbit pooling [23] (see Section B of Supplementary Material (SM) for details) for the encoder part to explicitly guarantee the following  $G$ -invariant latent representation:
$$
\mu_ {\theta} ^ {G} (\mathbf {x}) = \mu_ {\theta} ^ {G} (g * \mathbf {x}) \triangleq \hat {\mathbf {z}}, \quad \forall g \in G,
$$
where  $\mu_{\theta}^{G}:\mathbb{R}^{d}\to \mathbb{R}^{n}$  is the  $G$  -invariant encoder. It is worth mentioning that these encoder architectures are unable to learn a meaningful representation of the data when using a vanilla autoencoding framework, as the reconstruction loss (1) cannot be reduced enough; on the other hand, QAEs can use the explicitly  $G$  -invariant encoding architectures easily by taking advantage of the infimum in (4). This allows for both  $\hat{\mathrm{X}}$  and  $\mu_{\theta}^{G}(\mathrm{X})\triangleq \hat{\mathrm{Z}}$  to possess the invariant property up to  $G$
3.4 Isometric Quotient Variational Auto-Encoders
Although QAEs with  $G$ -invariant encoders can extract the underlying symmetry of observations, they still have the following drawbacks. First, QAEs are deterministic and may overfit to limited observations, similar to vanilla auto-encoders. Second, the Riemannian geometry of the extracted quotient manifold may not be preserved in the latent space. To address these issues, we present a stochastic version of QAEs called QVAEs, and further improve them with a novel framework for isometric learning called IQVAEs.
QVAEs. Inspired by the connection among (1), (3), and (4), we propose the stochastic version of QAEs (QVAEs) which minimizes the following stochastic quotient objective:
$$
\mathcal {L} _ {\mathrm {Q V A E}} (\theta , \phi ; \mathbf {x}, \beta , G) \triangleq \mathbb {E} _ {\epsilon \sim \mathcal {N} (\epsilon)} \left[ \mathcal {L} _ {\mathrm {Q A E}} (\theta , \phi ; \mathbf {x}, \epsilon , G) \right] + \mathcal {L} _ {\mathrm {K L}} (\theta ; \mathbf {x}, \beta , G), \tag {5}
$$
where the first term is defined as  $\mathcal{L}_{\mathrm{QAE}}(\theta, \phi; \mathbf{x}, \epsilon, G) \triangleq \inf_{g \in G} \| g * \mathbf{x} - \mu_{\phi}(\mu_{\theta}^{G}(\mathbf{x}) + \sigma_{\theta}^{G}(\mathbf{x}) \odot \epsilon) \|_2^2$  and the second term is  $\mathcal{L}_{\mathrm{KL}}(\theta; \mathbf{x}, \beta, G) \triangleq \beta D_{\mathrm{KL}}\left(\mathcal{N}(\hat{\mathbf{z}} | \mu_{\theta}^{G}(\mathbf{x}), \mathrm{diag}[\sigma_{\theta}^{G}(\mathbf{x})]^2)\right) \| p_{\mathcal{Z}}(\hat{\mathbf{z}})$ . We denote  $(\mu_{\theta}^{G}, \sigma_{\theta}^{G}): \mathbb{R}^{d} \to \mathbb{R}^{n} \times \mathbb{R}_{+}^{n}$  for the  $G$ -invariant multivariate Gaussian encoder.
QVAEs are able to generate new samples from the quotient space, unlike QAEs. It means that QVAEs can approximate the quotient space of the data manifold  $\mathcal{X} / G \triangleq \hat{\mathcal{X}}$  as a form of the immersed manifold  $\mu_{\phi}(\hat{\mathcal{Z}}) \approx \hat{\mathcal{X}}$  where  $\hat{\mathcal{Z}}$  is a  $G$ -invariant latent representation. This leads us to introduce the concept of Riemannian isometry as a geometric regularization for QVAEs.
Isometric regularization to quotient spaces. Before introducing the isometric regularization of the proposed QVAE, we review some basic concepts of the Riemannian geometry [2, 25].
Definition 3.1. (Riemannian manifolds) A  $n$ -dimensional Riemannian manifold  $\mathcal{X}$  is a  $n$ -dimensional differentiable manifold that is equipped with a symmetric positive-definite metric tensor  $\mathbf{H}_{\mathcal{X}}(\mathbf{x}) \in \mathbb{R}^{n \times n}$  ( $n \times n$  matrix). It defines a local smooth inner product on the tangent space  $\mathcal{T}_{\mathbf{x}}\mathcal{X}$  as  $\langle \mathbf{u}_{\mathbf{x}}, \mathbf{v}_{\mathbf{x}} \rangle_{\mathbf{H}_{\mathcal{X}}} = \mathbf{u}_{\mathbf{x}}^{\mathrm{T}}\mathbf{H}_{\mathcal{X}}(\mathbf{x})\mathbf{v}_{\mathbf{x}}$  with tangent vectors  $\mathbf{u}_{\mathbf{x}}, \mathbf{v}_{\mathbf{x}} \in \mathcal{T}_{\mathbf{x}}\mathcal{X}$  at each  $\mathbf{x} \in \mathcal{X}$ , allowing us to measure the distance and angles between tangent vectors at each point.
Definition 3.2. (Riemannian distances) Suppose  $\mathcal{X}$  is a connected Riemannian manifold equipped with a Riemannian metric  $\mathbf{H}_{\mathcal{X}}(\mathbf{x})$  at each point  $\mathbf{x} \in \mathcal{X}$ . A length of a curve on  $\mathcal{X}$  between two points  $\mathbf{x}_0$  and  $\mathbf{x}_1$  is defined as:
$$
L [ \gamma_ {\mathcal {X}} ] = \int_ {0} ^ {1} d t \sqrt {\langle \dot {\gamma} _ {\mathcal {X}} (t) , \dot {\gamma} _ {\mathcal {X}} (t) \rangle_ {\mathbf {H} _ {\mathcal {X}}}} = \int_ {0} ^ {1} d t \sqrt {\dot {\gamma} _ {\mathcal {X}} ^ {\mathrm {T}} (t) \mathbf {H} _ {\mathcal {X}} (\gamma_ {\mathcal {X}} (t)) \dot {\gamma} _ {\mathcal {X}} (t)},
$$
where  $\gamma_{\mathcal{X}}:[0,1]\to \mathcal{X}$  is a curve on  $\mathcal{X}$  that travels from  $\mathbf{x}_0 = \gamma_{\mathcal{X}}(0)$  to  $\mathbf{x}_1 = \gamma_{\mathcal{X}}(1)$  and  $\dot{\gamma}_{\mathcal{X}}(t) = \partial_t\gamma_{\mathcal{X}}(t)\in \mathcal{T}_{\gamma_{\mathcal{X}}(t)}\mathcal{X}$  is the velocity of the curve. The curve minimizing  $L$  is called a geodesic. The Riemannian distance is given by the length of the geodesic, i.e.,  $d_{\mathcal{X}}(\mathbf{x}_0,\mathbf{x}_1) = \inf_{\gamma_{\mathcal{X}}}L[\gamma_{\mathcal{X}}]$ .
Definition 3.3. (Immersions) Let  $\mathcal{Z}$  be an open set in  $\mathbb{R}^n$ . A smooth map  $\mu : \mathcal{Z} \to \mathbb{R}^d$  ( $n < d$ ) is called an immersion if its pushforward  $d\mu_{\mathbf{z}} : \mathcal{T}_{\mathbf{z}}\mathcal{Z} \to \mathcal{T}_{\mu(\mathbf{z})}\mu(\mathcal{Z})$  is injective for all  $\mathbf{z} \in \mathcal{Z}$ . Then,  $\mu(\mathcal{Z})$  is called an immersed submanifold.
Definition 3.4. (Geometry of immersed submanifolds) Suppose  $\mu : \mathcal{Z} \to \mathbb{R}^d$  is an immersion onto a submanifold  $\mathcal{X} \subset \mathbb{R}^d$ . The Euclidean inner product in  $\mathbb{R}^d$  is a proper inner product of tangent vectors of  $\mathcal{X}$ . Thus,  $\mathcal{X}$  is a Riemannian manifold equipped with a Riemannian metric  $\mathbf{H}_{\mathcal{X}}(\mathbf{x}) = \mathbf{I}_d$ , i.e.,  $\langle \mathbf{u_x}, \mathbf{v_x} \rangle_{\mathbf{H}_{\mathcal{X}}} = \mathbf{u_x^T}\mathbf{I}_d\mathbf{v_x} = \mathbf{u_x^T}\mathbf{v_x}$  for tangent vectors  $\mathbf{u_x}, \mathbf{v_x} \in \mathcal{T}_{\mathbf{x}}\mathcal{X}$  at each  $\mathbf{x} \in \mathcal{X}$ .