SlowGuess/NAACL_2021_OCR huggingface dataset:
Note that the term  $\mathbb{E}_{\boldsymbol{x} \sim \boldsymbol{X}} \mathbb{E}_{\boldsymbol{z} \sim q_{\phi}(\boldsymbol{z} | \boldsymbol{x})} \nu_{\psi}(\boldsymbol{x}, \boldsymbol{z})$  is maximized in (10) while it is minimized in (11), i.e., adversarial learning. In other words, one can consider the dual function as a discriminative network that distinguishes between the prior  $\boldsymbol{z} \sim p(\boldsymbol{z})$  and the variational posterior  $\boldsymbol{z} \sim q_{\phi}(\boldsymbol{z} | \boldsymbol{x})$ , both of which are paired with the input data  $\boldsymbol{x} \sim \boldsymbol{X}$ .
3.4 Data-driven Prior
While the use of a standard normal prior is a simple choice in Euclidean space, we argue that it induces bias in the hyperbolic setup. This is because natural sentences have specific meaning, and it is unrealistic to have the bulk of mass concentrated in the center (this is for low dimension; for high dimensions, it will concentrate near the surface of a sphere, which may partly explain why cosine similarity works favorably compared with Euclidean distance for NLP applications).
To reduce the induced bias from a pre-fixed prior, we adopt the VampPrior framework (Tomczak and Welling, 2018), which is a mixture of variational posteriors conditioned on learnable pseudo-data points. Specifically, we consider the prior as a
learnable distribution given by
$$
p _ {\delta} (z) = \frac {1}{K} \sum_ {k = 1} ^ {K} q _ {\phi} (z | s _ {k}), \tag {12}
$$
where  $q_{\phi}$  is the learned approximate posterior, and we call the parameter  $\delta \coloneqq \{s_k\}_{k = 1}^K$  pseudo inputs. Intuitively,  $p_{\delta}(z)$  seeks to match the aggregated posterior (Makhzani et al., 2015):  $q(z) = \frac{1}{N}\sum_{i = 1}^{N}q_{\phi}(z|\boldsymbol {x}_i)$  in a cost-efficient manner via parameterizing the pseudo inputs. By replacing the prior distribution  $p(z)$  in (10) with  $p_{\delta}(z)$ , we complete the final objective of the proposed APo-VAE. The detailed training procedure is summarized in Algorithm 1.
VAE for Text Generation. Many VAE models have been proposed for text generation, most of which focus on solving the issue of posterior collapse. The most popular strategy is to alter the training dynamics, keeping the encoder away from bad local optima. For example, variants of KL annealing (Bowman et al., 2016; Zhao et al., 2018; Fu et al., 2019) dynamically adjust the weight on the KL penalty term as training progresses. Lagging VAE (He et al., 2019) aggressively optimizes the encoder before each decoder update, to overcome the imbalanced training issue between the encoder and decoder. Alternative strategies have also been proposed based on competing theories or heuristics.  $\delta$ -VAE (Razavi et al., 2019) tackles this issue by enforcing a minimum KL divergence between the posterior and the prior. Yang et al. (2017) blames mode-collapse on the auto-regressive design of the decoder and advocates alternative architectures. A semi-amortized inference network is considered by Kim et al. (2018) to bridge the amortization gap between log-likelihood and the ELBO.
Recent work has also shown that posterior collapse can be ameliorated by using more expressive priors and variational posteriors other than Gaussian. Flow-based VAE is considered in Ziegler and Rush (2019) to enhance the flexibility of prior distributions. A topic-guided prior is proposed in Wang et al. (2019a) to achieve more controllable text generation. Fang et al. (2019) explores implicit sample-based representations, without requiring an explicit density form for the approximate posterior. Xu and Durrett (2018) considers replacing the Gaussian with the spherical von Mises-Fisher (vMF) distribution. Compared to these prior arts,
our model features structured representation in hyperbolic space, which not only captures latent hierarchies but also combats posterior collapse.
Hyperbolic Space Representation Learning. There has been a recent surge of interest in representation learning in hyperbolic space, largely due to its exceptional effectiveness modeling data with underlying graphical structure (Chamberlain et al., 2017), such as relation nets (Nickel and Kiela, 2017). In the context of NLP, hyperbolic geometry has been considered for word embeddings (Tifrea et al., 2018). A popular vehicle for hyperbolic representation learning is the autoencoder (AE) framework (Grattarola et al., 2019; Ovinnikov, 2019), where the decoders are built to efficiently exploit the hyperbolic geometry (Ganea et al., 2018). Closest to our APo-VAE are the works of hyperbolic VAEs (Mathieu et al., 2019; Nagano et al., 2019), where wrapped normal distributions have been used. Drawing power from the dual form of the KL, the proposed APo-VAE highlights an implicit posterior and data-driven prior, showing improved training stability.
We evaluate the proposed model on three tasks: (i) language modeling, (ii) unaligned style transfer, and (iii) dialog-response generation, with quantitative results, human evaluation and qualitative analysis.
Datasets. We use three datasets for language modeling: Penn Treebank (PTB) (Marcus et al., 1993), Yahoo and Yelp corpora (Yang et al., 2017). PTB contains one million words of 1989 Wall Street Journal material annotated in Treebank II style, with 42k sentences of varying lengths. Yahoo and Yelp are much larger datasets, each containing 100k sentences with greater average length.
For unaligned style transfer, we use the Yelp restaurant reviews dataset (Shen et al., 2017), which is obtained by pre-processing the Yelp dataset, i.e., sentences are shortened for more feasible sentence level sentiment analysis. Overall, the dataset includes  $350\mathrm{k}$  positive and  $250\mathrm{k}$  negative reviews (based on user rating).
Following Gu et al. (2019), we use the Switchboard (Godfrey and Holliman, 1997) dataset for dialogue-response generation. The former contains 2.4k two-sided telephone conversations, manually
transcribed and aligned. We split the data into training, validation and test sets following the protocol described in Zhao et al. (2017b).
Evaluation Metrics. To benchmark language modeling performance, we report the ELBO and Perplexity (PPL) of APo-VAE and baselines. In order to verify our proposed Apo-VAE is more resistant to posterior collapse, we also report the KL-divergence  $\mathbb{D}_{\mathrm{KL}}(q_{\phi}(z|\boldsymbol{x})||p(z))$  and mutual information (MI) between  $z$  and  $\boldsymbol{x}$  (He et al., 2019). The number of active units (AU) of the latent code is also reported, where the activity of a latent dimension  $z$  is measured as  $A_z = \operatorname{Cov}_x \mathbb{E}_{z \sim q_{\phi}(z|\boldsymbol{x})}[z]$ , and defined as active if  $A_z > 0.01$ .
To evaluate our model on unaligned style transfer, we consider the transfer accuracy from one sentiment to another, the BLEU scores between original and transferred sentences, the reconstruction perplexity of original sentences, and the reverse perplexity (RPPL) based on a language model from the transferred sentences.
For dialogue-response generation, we adopt the evaluation metrics used in previous studies (Zhao et al., 2017b; Gu et al., 2019), including BLEU (Papineni et al., 2002), BOW (Liu et al., 2016), and intra/inter-dist values (Gu et al., 2019). The first two metrics are used to assess the relevance of the generated response, and the third is for diversity evaluation.
Model Implementation. For language modeling, we adopt the LSTM (Hochreiter and Schmidhuber, 1997) for both the encoder and decoder, with dimension of the latent code set to 32. Following Mathieu et al. (2019), the hyper-parameter  $c$  is set to 0.7. For unaligned style transfer, we extend our model in the same fashion as Fang et al. (2019). For dialogue-response generation, we modify APo-VAE following the conditional VAE framework (Zhao et al., 2017b). Specifically, an extra input of context embedding  $s$  is supplied to the model (i.e.,  $p_{\theta}(\boldsymbol{x}, \boldsymbol{z}|\boldsymbol{s})$ ,  $q_{\phi}(\boldsymbol{z}|\boldsymbol{x}, \boldsymbol{s})$ ). The prior  $p(\boldsymbol{z}|\boldsymbol{s})$  is a wrapped normal conditioned on context embedding, learned together with the posterior.
Language Modeling. Table 1 shows results on language modeling. We compare APo-VAE with other VAE-based solutions, including  $\beta$ -VAE (Higgins et al., 2017), SA-VAE (Kim et al., 2018), lagging VAE (LAG-VAE) (He et al., 2019), vMF-VAE (Xu and Durrett, 2018), Poincaré VAE ( $\mathcal{P}$ -
Model	-ELBO	PPL	KL	MI	AU
PTB
VAE	102.6	108.26	1.1	0.8	2
β-VAE	104.5	117.92	7.5	3.1	5
SA-VAE	102.6	107.71	1.2	0.7	2
vMF-VAE	95.8	93.70	2.9	3.2	21
P-VAE	91.4	76.13	4.5	2.9	23
iVAE	87.2	53.44	12.5	12.2	32
APo-VAE	87.2	53.32	8.4	4.8	32
APo-VAE+VP	87.0	53.02	8.9	4.5	32
Yahoo
VAE	328.6	61.21	0.0	0.0	0
β-VAE	328.7	61.29	6.3	2.8	8
SA-VAE	327.2	60.15	5.2	2.9	10
LAG-VAE	326.7	59.77	5.7	2.9	15
vMF-VAE	318.5	53.92	6.3	3.7	23
P-VAE	313.4	50.57	7.2	3.3	27
iVAE	309.1	47.93	11.4	10.7	32
APo-VAE	286.2	47.00	6.9	4.1	32
APo-VAE+VP	285.6	46.61	8.1	4.9	32
Yelp
VAE	357.9	40.56	0.0	0.0	0
β-VAE	358.2	40.69	4.2	2.0	4
SA-VAE	357.8	40.51	2.8	1.7	8
LAG-VAE	355.9	39.73	3.8	2.4	11
vMF-VAE	356.2	51.03	4.1	3.9	13
P-VAE	355.4	50.64	4.3	4.8	19
iVAE	348.7	36.88	11.6	11.0	32
APo-VAE	319.7	34.10	12.1	7.5	32
APo-VAE+VP	316.4	32.91	12.7	6.2	32
Table 1: Results on PTB, Yahoo, and Yelp datasets. A better language model achieves lower negative ELBO and PPL. Higher KL and MI indicate a better utilization of the latent space.
VAE) (Mathieu et al., 2019) and iVAE $^3$  (Fang et al., 2019). On all three datasets, the proposed model achieves lower negative ELBO and PPL than other models, demonstrating its strong ability to better model sequential text data. Meanwhile, the larger KL term and higher mutual information (between  $z$  and  $x$ ) of APo-VAE model indicate its robustness in handling posterior collapse. In addition, the introduction of a data-driven prior (denoted as APo-VAE+VP) further boosts the performance, especially on negative ELBO and PPL.
Visualization. To verify our hypothesis that the proposed model is capable of learning latent tree structure in text data, we visualize the two-dimensional projection of the learned latent code in Figure 3. For visualization, we randomly draw  $5\mathrm{k}$  samples from PTB-test, and encode them to the latent space using the APo-VAE encoder. We color-code each sentence based on its length (i.e., blue for long sentences and red for short sentences). Note that only a small portion of data have a length longer than  $32(< 10\%)$ , and human inspection
3We report iVAEMi results in all our experiments.
Model	BLEU	BOW	Intra-dist	Inter-dist
R	P	F1	A	E	G	dist-1	dist-2	dist-1	dist-2
CVAE	0.295	0.258	0.275	0.836	0.572	0.846	0.803	0.415	0.112	0.102
CVAE-BOW	0.298	0.272	0.284	0.828	0.555	0.840	0.819	0.493	0.107	0.099
CVAE-CO	0.299	0.269	0.283	0.839	0.557	0.855	0.863	0.581	0.111	0.110
DialogWAE	0.394	0.254	0.309	0.897	0.627	0.887	0.713	0.651	0.245	0.413
iVAE	0.427	0.254	0.319	0.930	0.670	0.900	0.828	0.692	0.391	0.668
APO-VAE	0.438	0.261	0.328	0.937	0.683	0.904	0.861	0.792	0.445	0.717
Figure 3: Visualization of the hyperbolic latent space of 5,000 randomly sampled sentences from the test set of PTB. The lengths of the samples are color-coded (red for short ones and blue for longer ones). The six listed sentences are created by modifying data samples.
the national cancer institute ban smoking
the national cancer institute warns citizens to avoid smoking cigarette
the national cancer institute claims that smoking cigarette too often would increase the chance of getting lung cancer
the national cancer institute study the effect of chemical in cigarette on different group of workers.
the national cancer institute also projected that overall u.s. mortality rates from lung cancer should begin to drop in several years if cigarette smoking continues to decrease
the national cancer institute report a form of asbestos once used to make cigarette filters has caused a high percentage of cancer deaths among a group of workers exposed to it
Table 2: Results on SwitchBoard (P: precision, R: recall, A: average, E: extreme, G: greedy). Higher BLEU and BOW Embedding indicate better quality of generated responses. Higher intra/inter-dist means better generation diversity.
Model	ACC	BLEU	PPL	RPPL
ARAE	95	32.5	6.8	395
iVAE	92	36.7	6.2	285
APO-VAE	95	37.8	6.1	273
verified that most of them contain multiple subsentences. We exclude these samples from our analysis.
As shown in Figure 3, longer sentences (dark blue) tend to occupy the outer rim of the Poincaré ball, while the shorter ones (dark red) are concen
Table 3: Unaligned style transfer on the Yelp restaurant reviews dataset.
vs iVAE	vs DialogWAE
win	loss	tie	win	loss	tie
Informativeness	52.8	27.9	19.3	63.7	27.1	19.2
Coherence	41.7	35.5	22.8	41.2	34.4	24.4
Diversity	51.2	26.4	22.4	62.1	25.1	12.8
Table 4: Human evaluation results. Win/loss/tie indicates the percentage of responses generated by APOVAE being better/worse/equal to the compared model.
trated in the inner area. We also select some long sample sentences (dark blue), and manually shorten them to create several variants of different lengths (ranging from 6 to 27), which are related in a hierarchical manner based on human judgement. We visualize their latent codes projected by the trained APo-VAE. The resulting plot is consistent with a hierarchical structure for APo-VAE: as the sentence becomes more specific, the embedding moves outward. We also decode from the neighbours of these latent codes, the outputs (see the Appendix) of which demonstrate a similar hierarchical structure.
Unaligned Style Transfer. Table 3 shows the results on the Yelp restaurant reviews dataset. APOVAE achieves over 1 point increased BLEU scores than iVAE, capturing a more informative and structured feature space. Comparable performance is achieved for the other evaluation metrics.
Dialogue Response Generation. Results on SwitchBoard are summarized in Table 2. Our proposed model generates comparable or better responses than the baseline models in terms of both relevance (BLEU and BOW) and diversity (intra/inter-dist). A Po-VAE improves the average recall from 0.427 (by iVAE) to 0.438, while significantly enhancing generation diversity (e.g., from 0.692 to 0.792 for intra-dist-2).
Human Evaluation. We further perform human evaluation via Amazon Mechanical Turk. We asked the turkers to compare generated responses from two models, and assess each model's informative-
ness, relevance to the dialog context (coherence), and diversity. We use 500 randomly sampled contexts from the test set, each assessed by three judges. In order to evaluate diversity, 5 responses are generated for each dialog context. For quality control, only workers with a lifetime task approval rating greater than  $98\%$  were allowed to participate in our study. Table 4 summarizes the human evaluation results. The responses generated by our model are clearly preferred by the judges compared with other competing methods.
We present APo-VAE, a novel model for text generation in hyperbolic space. Our model can learn latent hierarchies in natural language via the use of wrapped normals for the prior. A primal-dual view of KL divergence is adopted for robust model training. Extensive experiments on language modeling, text style transfer, and dialog response generation demonstrate the superiority of the model. For future work, we plan to combine APo-VAE with the currently prevailing large-scale pre-trained language models.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In CoNLL.
Benjamin Paul Chamberlain, James Clough, and Marc Peter Deisenroth. 2017. Neural embeddings of graphs in hyperbolic space. arXiv preprint arXiv:1705.10359.
Liquun Chen, Shuyang Dai, Chenyang Tao, Dinghan Shen, Zhe Gan, Haichao Zhang, Yizhe Zhang, and Lawrence Carin. 2018a. Adversarial text generation via feature-mover's distance. arXiv preprint arXiv:1809.06297.
Liqun Chen, Chenyang Tao, Ruiyi Zhang, Ricardo Henao, and Lawrence Carin Duke. 2018b. Variational inference and model selection with generalized evidence bounds. In International conference on machine learning, pages 893-902. PMLR.
Bo Dai, Hanjun Dai, Niao He, Weiyang Liu, Zhen Liu, Jianshu Chen, Lin Xiao, and Le Song. 2018. Coupled variational bayes via optimization embedding. In NeurIPS.
Christopher De Sa, Albert Gu, Christopher Ré, and Frederic Sala. 2018. Representation tradeoffs for hyperbolic embeddings. Proceedings of machine learning research.
Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. 2019. Implicit deep latent variable models for text generation. In EMNLP.
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. 2019. Cyclic annealing schedule: A simple approach to mitigating kl vanishing. In NAACL.
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. 2018. Hyperbolic neural networks. In NeurIPS.
John J Godfrey and Edward Holliman. 1997. Switchboard-1 release 2. Linguistic Data Consortium, Philadelphia, 926:927.
Daniele Grattarola, Lorenzo Livi, and Cesare Alippi. 2019. Adversarial autoencoders with constant-curvature latent manifolds. Applied Soft Computing, 81:105511.
Xiaodong Gu, Kyunghyun Cho, Jung-Woo Ha, and Sunghun Kim. 2019. Dialogwae: Multimodal response generation with conditional Wasserstein autoencoder. In ICLR.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Lagging inference networks and posterior collapse in variational autoencoders. In ICLR.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. *ICLR*, 2(5):6.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation.
Yoon Kim, Sam Wiseman, Andrew C Miller, David Sontag, and Alexander M Rush. 2018. Semi-amortized variational autoencoders. In ICML.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114.
Nathan Linial, Eran London, and Yuri Rabinovich. 1995. The geometry of graphs and some of its algorithmic applications. Combinatorica, 15(2):215-245.
Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. 2015. Adversarial autoencoders. arXiv preprint arXiv:1511.05644.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics.
Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Teh. 2019. Continuous hierarchical representations with poincaré variational auto-encoders. In NeurIPS.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. 2017. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In ICML.
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. 2019. A wrapped normal distribution on hyperbolic space for gradient-based learning. In ICML.
Maximillian Nickel and Douwe Kiela. 2017. Poincaré embeddings for learning hierarchical representations. In NeurIPS.
Ivan Ovinnikov. 2019. Poincar\`e wasserstein autoencoder. arXiv preprint arXiv:1901.01427.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL.
Ali Razavi, Aaron van den Oord, Ben Poole, and Oriol Vinyals. 2019. Preventing posterior collapse with delta-vaes. In ICLR.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In ICML.
R Tyrrell Rockafellar et al. 1966. Extension of fenchel'duality theorem for convex functions. Duke mathematical journal.
Rik Sarkar. 2011. Low distortion delaunay embedding of trees in hyperbolic plane. In International Symposium on Graph Drawing, pages 355-366. Springer.
Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarek Abdelzaher. 2020. Controlvae: Controllable variational autoencoder. In ICML.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In NeurIPS.
Wenxian Shi, Hao Zhou, Ning Miao, Shenjian Zhao, and Lei Li. 2019. Fixing gaussian mixture vaes for interpretable text generation. arXiv preprint arXiv:1906.06719.
Chenyang Tao, Liquun Chen, Shuyang Dai, Junya Chen, Ke Bai, Dong Wang, Jianfeng Feng, Wenlian Lu, Georgiy V Bobashev, and Lawrence Carin. 2019. On fenchel mini-max learning. In NeurIPS, pages 10427-10439.
Alexandru Tifrea, Gary Bécigneul, and Octavian-Eugen Ganea. 2018. Poincar\`e glove: Hyperbolic word embeddings. arXiv preprint arXiv:1810.06546.
Jakub M Tomczak and Max Welling. 2018. Vae with a vampprior. In AISTATS.
Abraham Albert Ungar. 2008. A gyrovector space approach to hyperbolic geometry. Synthesis Lectures on Mathematics and Statistics, 1(1):1-194.
Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019a. Topic-guided variational autoencoders for text generation. arXiv preprint arXiv:1903.07137.
Wenlin Wang, Chenyang Tao, Zhe Gan, Guoyin Wang, Liquun Chen, Xinyuan Zhang, Ruiyi Zhang, Qian Yang, Ricardo Henao, and Lawrence Carin. 2019b. Improving textual network learning with variational homophilic embeddings. arXiv preprint arXiv:1909.13456.
Jiacheng Xu and Greg Durrett. 2018. Spherical latent spaces for stable variational autoencoders. In EMNLP.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. In ICML.
Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M Rush, and Yann LeCun. 2017a. Adversarily regularized autoencoders. arXiv preprint arXiv:1706.04223.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. 2018. Infovae: Information maximizing variational autoencoders. In AAAI.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017b. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In ACL.
Zachary M Ziegler and Alexander M Rush. 2019. Latent normalizing flows for discrete sequences. In ICML.
Appendix for APo-VAE: Text Generation in Hyperbolic Space
A1 Basics of Riemannian Geometry (Extended)
This section provides additional coverage of the basic mathematical concepts used in APo-VAE. For more detailed mathematics expositions, readers are referred to Ganea et al. (2018).
A real, smooth  $n$ -dimensional manifold  $\mathcal{M}$  is a set of points locally similar to a linear space  $\mathbb{R}^n$ . At each point  $\pmb{x}$  of the manifold  $\mathcal{M}$  is defined a real vector space of the space of the same dimensionality as  $\mathcal{M}$ , called the tangent space in  $\pmb{x}$ :  $\mathcal{T}_{\pmb{x}}\mathcal{M}$ . Intuitively it contains all the possible directions in which one can tangentially pass through  $\pmb{x}$ . For each point  $\pmb{x}$  there also defines a metric tensor  $g_{\pmb{x}}(\cdot ,\cdot):\mathcal{T}_{\pmb{x}}\mathcal{M}\times \mathcal{T}_{\pmb{x}}\mathcal{M}\to \mathbb{R}$ , which defined an inner product on the associated tangent space  $\mathcal{T}_{\pmb{x}}\mathcal{M}$ . More specifically, given a coordinate system, the inner product is given in the quadratic form  $g_{\pmb{x}}(\pmb {u},\pmb {v}) = \langle \pmb {u},\pmb {v}\rangle_{g_{\pmb{x}}} = \pmb{u}^{T}\pmb{G}_{\pmb{x}}\pmb {v}$ , where by slight abuse of notation  $\pmb {u},\pmb {v}\in \mathbb{R}^n$  are vector representations of the tangent vectors wrt the coordinate system and  $\pmb {G}_{\pmb{x}}\in \mathbb{R}^{n\times n}$  is a positive-definite matrix. Collectively,  $(\mathcal{M},g)$  defines a Riemannian manifold.
(a) Parallel transport a tangent vector (arrow) from one location (black) to another (orange).