SlowGuess/NAACL_2021_OCR huggingface dataset:
(b) Map the transported tangent vector (orange) to a point (green) in the hyperbolic space by using the exponential map.
Figure A1: Visualization of different mathematical operations in a hyperbolic space, that are used to define the wrapped distribution.
The metric tensor is used to generalize the notations such as distance and volume in Euclidean space to the Riemannian manifold. Given a curve  $\gamma(t): [0,1] \to \mathcal{M}$ , its length is given by
$$
L (\gamma) = \int_ {0} ^ {1} \| \gamma^ {\prime} (t) \| _ {\gamma (t)} \mathrm {d} t. \tag {A1}
$$
Figure A2: Mapping a Gaussian distribution (red) and a implicit distribution (blue) to the hyperbolic space.
The concept of straight lines can then be generalized to geodesics, which is the shortest path between pairs of points  $\pmb{x}, \pmb{y}$  on the manifold  $\gamma^{*}(\pmb{x}, \pmb{y}) = \arg \min_{\gamma} L(\gamma)$  such that  $\gamma(0) = \pmb{x}$  and  $\gamma(1) = \pmb{y}$  with  $\gamma$  traveling at constant speed (i.e.,  $\| \gamma'(t) \|_{\gamma(t)} = c$ , where  $c$  is the distance). The concept of moving along a straight line with constant speed defines the exponential map, where for  $\pmb{v} \in \mathcal{T}_{\pmb{x}}\mathcal{M}$  there is a unique unit speed geodesic  $\gamma$  satisfying  $\gamma(0) = \pmb{x}$  with initial tangent vector  $\pmb{v}$ , and the corresponding exponential map is defined by  $\exp_{\pmb{x}}(\pmb{v}) = \gamma(1)$ . We call the inverse of exponential map the logarithm map  $\log_{\pmb{x}} = \exp_{\pmb{x}}^{-1} : \mathcal{M} \to \mathcal{T}_{\pmb{x}}\mathcal{M}$ , mapping from the manifold to the tangent space. For the Poincaré ball model, it is geodesically complete in the sense that  $\exp_{\pmb{x}}$  is well-defined on the full tangent space  $\mathcal{T}_{\pmb{x}}\mathcal{M}$ .
A2 Additional Related Work
VAE with Adversarial Learning. One of the first to apply adversarial learning to VAE is Adversarial Variational Bayes (AVB) (Mescheder et al., 2017; Pu et al., 2017). Motivated by Generative Adversarial Network (GAN) (Goodfellow et al., 2014), AVB introduces an auxiliary discriminator that transforms the maximum-likelihood-problem into a two-player game. Similarly, Adversarial Autoencoder (AAE) (Makhzani et al., 2015) uses adversarial learning to match aggregated posterior with the prior. Based on this, Coupled Variational Bayes (CVB) (Dai et al., 2018) connects the primal-dual view of ELBO with adversarial learning, where the discriminator takes both data sample and latent code as input. This approach is also adopted in implicit VAE (Fang et al., 2019) for text generation. However, the prior used in implicit
the national cancer institute ban smoking
too often would increase the chance of getting lung cancer
VAE is still standard Gaussian, while our proposed model uses hyperbolic geometry.
A3 Additional Details for Experiments
A3.1 Metrics (Dialogue Response Generation)
BLEU (Papineni et al., 2002) is used to measure the amount of  $n$ -gram overlap between a generated response with the reference. Specifically, BLEU scores of  $n < 4$  are computed; the average and the maximum scores are considered as  $n$ -gram precision and  $n$ -gram recall, respectively. In addition, the BOW embedding metric (Liu et al., 2016) is used to measure cosine similarity between bag-of-word embeddings of response and reference. Three metrics are considered for cosine distance: (i) computed by greedily matching words in two utterances; (ii) between the averaged embeddings in two utterances; and (iii) between the largest extreme values among the embeddings in two utterances. We also follow Gu et al. and use the distinct metric to measure the diversity of generated text. Dist -  $n$  is the ratio of unique  $n$ -grams over all  $n$ -grams in the generated sentences. Intra-dist and inter-dist are the average distinct values within each generated sample and among all generated samples, respectively.
(a) Hyperbolic latent space for Yahoo.
the national cancer institute claims that smoking cigarette
Figure A3: Visualization of the Euclidean space (VAE) of 5,000 randomly sampled sentences from the test set of PTB. The four listed sentences are created by modifying data samples.
(b) Hyperbolic latent space for Yelp.
Figure A4: Visualization of the hyperbolic latent space of 5,000 randomly sampled sentences from different datasets.
the national cancer institute warns citizens to avoid smoking cigarette
the national cancer institute also projected that overall u.s. mortality rates from lung cancer should begin to drop in several years if cigarette smoking continues to decrease
A3.2 Additional Implementation Details
For language modeling, we adopt the LSTM (Hochreiter and Schmidhuber, 1997) for both the encoder and decoder, which have 256 hidden units for PTB, and 1024 hidden units for Yahoo and Yelp. The dimension of the latent code is set to 32. Following Mathieu et al. (2019), the hyper-parameter  $c$  is set to 0.7. We set the vocabulary size to 10,000 for PTB, and 20,000 for both Yahoo and Yelp. The word embedding size is 256 for PTB, and 512 for Yahoo and Yelp. For dialogue response generation, we follow Gu et al. (2019), and use the GRU (Cho et al., 2014) with 300 hidden units in each direction for both the response encoder and context encoder, and 300 hidden units for decoder. The latent code  $z$  has a dimension of 200. The size of the vocabulary is set to 10,000, and the word-embedding size is 200,
Input	Sample
the national cancer institute ban smoking	the <unk> and drug administration were talking
the national cancer institute warns citizens to avoid smoking cigarette	the <unk> and drug administration officials also are used to <unk>
the national cancer institute claims that smoking cigarette too often would increase the chance of getting lung cancer	the u.s. and drug administration officials say they are n’t <unk> to be used by the government
the national cancer institute also projected that overall u.s. mortality rates from lung cancer should begin to drop in several years if cigarette smoking continues to decrease	the u.s. and drug administration officials <unk> they are looking for ways to <unk> their own accounts for some of their assets to be sold by some companies
Table A1: Generate a sample based on the latent code of a specific input (PTB).
Model	BLEU	BOW	Intra-dist	Inter-dist
R	P	F1	A	E	G	dist-1	dist-2	dist-1	dist-2
CVAE	0.265	0.222	0.242	0.923	0.543	0.811	0.938	0.973	0.177	0.222
CVAE-BOW	0.256	0.224	0.239	0.923	0.540	0.812	0.947	0.976	0.165	0.206
CVAE-CO	0.259	0.244	0.251	0.914	0.530	0.818	0.821	0.911	0.106	0.126
DialogWAE	0.341	0.278	0.306	0.948	0.578	0.846	0.830	0.940	0.327	0.583
iVAE	0.355	0.239	0.285	0.951	0.609	0.872	0.897	0.975	0.501	0.868
APO-VAE	0.359	0.265	0.305	0.954	0.616	0.873	0.919	0.989	0.511	0.869
initialized by GloVe (Pennington et al., 2014).
A3.3 Additional Results
For language modeling, we plot the hyperbolic latent space for Yahoo and Yelp as shown in Figure A4. To demonstrate the hierarchical structure in the generated sentences (i.e., the decoder), we choose 4 sentences (from short to long) with some hierarchy, listed on the left hand side of Table A1. These sentences are encoded to hyperbolic space by using a well trained APo-VAE. Then, we decode by randomly select a neighbor of each of the 4 latent codes. The output sentences are shown on the right hand side of Table A1, demonstrating similar hierarchy as the input sentences. Moreover, we directly measure the generation quality by using PPL and reverse PPL, shown in Table A3. Our APo-VAE achieves consistently better performance.
For dialog response generation, we include additional results on the DailyDialog dataset (Li et al., 2017), which contains 13k daily conversations for an English learner in daily life. We also provide examples of generated responses along with their corresponding dialog context in Table A5. Samples generated by APo-VAE are more relevant to the contexts than the baseline models. In addition,
Table A2: Results on DailyDialog (P: precision, R: recall, A: average, E: extreme, G: greedy). Higher BLEU and BOW Embedding indicate better quality of generated responses. Higher intra/inter-dist means better generation diversity.
Model	Forward	Reverse
VAE	18494	10149
Cyc-VAE	3390	5587
β-VAE	625	1897
SA-VAE	341	10651
vMF-VAE	274	2364
iVAE	116	1520
APO-VAE	109	1385
Table A3: Results on forward and reverse PPL on the PTB dataset.
vs iVAE	vs DialogWAE
win	loss	tie	win	loss	tie
Informativeness	45.4	26.9	17.7	46.1	26.5	27.4
Coherence	40.1	25.9	24.0	40.7	24.2	25.1
Diversity	43.9	30.8	25.3	47.5	31.4	21.1
Table A4: Human evaluation results on DailyDialog. Win/loss/tie indicates the percentage of responses generated by APo-VAE being better/worse/equal to the compared model.
APo-VAE is capable of providing both positive and negative responses, demonstrating better generation diversity. The human evaluation results for DailyDialog can be found in Table A4.
A3.4 Human Evaluation
We provide the instruction of human evaluation on the dialog response generation task in Figure A5.
Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnnc encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
Bo Dai, Hanjun Dai, Niao He, Weiyang Liu, Zhen Liu, Jianshu Chen, Lin Xiao, and Le Song. 2018. Coupled variational bayes via optimization embedding. In NeurIPS.
Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. 2019. Implicit deep latent variable models for text generation. In EMNLP.
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. 2018. Hyperbolic neural networks. In NeurIPS.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In NeurIPS.
Xiaodong Gu, Kyunghyun Cho, Jung-Woo Ha, and Sunghun Kim. 2019. Dialogwae: Multimodal response generation with conditional Wasserstein autoencoder. In ICLR.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957.
Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. 2015. Adversarial autoencoders. arXiv preprint arXiv:1511.05644.
Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Teh. 2019. Continuous hierarchical representations with poincaré variational auto-encoders. In NeurIPS.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. 2017. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In ICML.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Yuchen Pu, Weiyao Wang, Ricardo Henao, Liquin Chen, Zhe Gan, Chunyuan Li, and Lawrence Carin. 2017. Adversarial symmetric variational autoencoder. In NIPS.
Below is a short exchange in which two possible responses are given. Compare the two candidate RESPONSES. Using the radio buttons, determine which is more relevant/appropriate and informative/interesting in light of the preceding context. Then decide which of the two responses seems to be more human-like, again in light of the previous turn.
There is room for subjectivity here, but you should primarily consider whether it would be reasonable to think that a person who was engaged in the conversation might find the response interesting and informative, and contributing to the conversation. A generic or bland response like "I don't know" may be relevant, but not especially informative or interesting. It might not contribute much to the conversation.
Please ignore punctuation, spacing, and other minor formatting issues. It is expected that you will need to spend at least 10 seconds working on this page.
Context: This looks like a cover to a Goosebumps book.
PERSON B:
RESPONSE #1: It does look a bit like a book
RESPONSE #2: Looks like a light Iron Maiden album cover
RELEVANCE/APPROPRIATENESS: Which response is more relevant and appropriate to the immediately preceding turn?
$\bigcirc$  RESPONSE #1
$\bigcirc$  Neither or Equally
$\bigcirc$  RESPONSE #2
INTEREST/INFORMATIVENESS: Which response is more interesting and informative (has more content)?
$\bigcirc$  RESPONSE #1
Neither or Equally
$\bigcirc$  RESPONSE #2
Figure A5: AMT interface for user study. Human judges are required to compare responses given context text.
Context	DialogWAE	iVAE	APo-VAE
i would wondering if you like to stay here for a few days?	Eg.1: ok.	Eg.1: I'm not going to take a look.	Eg.1: good idea. how about shop- ping tomorrow morning?
Eg.2: great, thanks for asking.	Eg.2: is it a prob- lem?	Eg.2: of course. would you like to have lunch with me?
Eg.3: next week is a good idea.	Eg.3: it is a day va- cation for you here in the us.	Eg.3: i'm afraid i can't make it to the next few days.
i have to check out today. i'd like my bill ready by 10.	Eg.1: that's a long time.	Eg.1: that's the problem, sir?	Eg.1: i'm sorry to hear that. are you looking for another one?
Eg.2: yes.	Eg.2: i was the only child in the room.	Eg.2: no problem. may i have it now?
Eg.3: i'm afraid you can't do that.	Eg.3: you can get me a little time, and the red will take you get off.	Eg.3: i see, sir. what is the room number?
the weather has been terrible this week.	Eg.1: i'm sure we can do it again.	Eg.1: i've got a few things i'll get you there is a real one.	Eg.1: yes. i have been looking for- ward to better days.
Eg.2: oh, this is the correct.	Eg.2: what is the rest of the days.	Eg.2: i don't think it is the best time for us to visit.
Eg.3: what kind of job do you like to see?	Eg.3: oh, no.	Eg.3: yes, we did.
i wonder whether i could possibly bor- row your new bicy-cle now.	Eg.1: that's a great idea. what did you do?	Eg.1: a problem. what do you think of this one?	Eg.1: of course. let's see.
Eg.2: i'd like to tell you a fees, but i'm not sure i can do that.	Eg.2: yes sure, thanks. you are very tough.	Eg.2: i sorry to hear that. anything else? do you have any other account?
Eg.3: oh, what is your problem?	Eg.3: what is the problem? what time? oh yes, it's been a long time. let me see.	Eg.3: i would love to do that. there have been more 1 for you.
Table A5: Examples of context-response pairs for different models. 'Eg.i' means the  $i$ -th sample.