Venue: NAACL 2021

# APo-VAE: Text Generation in Hyperbolic Space

Shuyang Dai\({}^{1}\) Zhe Gan\({}^{2}\) Yu Cheng\({}^{2}\) Chenyang Tao\({}^{1}\) Lawrence Carin\({}^{1}\) Jingjing Liu\({}^{2}\)

\({}^{1}\)Duke University \({}^{2}\)Microsoft Corporation

{shuyang.dai, chenyang.tao, lcarin}@duke.edu

{zhe.gan, yu.cheng, jingjl}@microsoft.com

Work was done when the author interned at Microsoft.

###### Abstract

Natural language often exhibits inherent hierarchical structure infrained with complex syntax and semantics. However, most state-of-the-art deep generative models learn embeddings only in Euclidean vector space, without accounting for this structural property of language. We investigate text generation in a hyperbolic latent space to learn continuous hierarchical representations. An Adversarial Poincare Variational Autoencoder (APo-VAE) is presented, where both the prior and variational posterior of latent variables are defined over a Poincare ball via wrapped normal distributions. By adopting the primal-dual formulation of Kullback-Leibler divergence, an adversarial learning procedure is introduced to empower robust model training. Extensive experiments in language modeling, unaligned style transfer, and dialog-response generation demonstrate the effectiveness of the proposed APo-VAE model over VAEs in Euclidean latent space, thanks to its superb capabilities in capturing latent language hierarchies in hyperbolic space.

## 1 Introduction

The Variational Autoencoder (VAE) [35, 4] is a generative model widely applied to language-generation tasks, which propagates latent codes drawn from a simple prior to manifest data samples through a decoder. The generative model is augmented by an inference network, which feeds observed data samples through an encoder to yield a distribution on the corresponding latent codes. Since natural language often manifests a latent hierarchical structure, it is desirable for the latent code in a VAE to reflect such inherent language structure, so that the generated text can be more natural and expressive. An example of language structure is illustrated in Figure 1, where sentences are organized into a tree structure. The root node corresponds to simple sentences (_e.g._, "_Yes_"), while nodes on outer leaves represent sentences with more complex syntactic structure and richer, more specific semantic meaning (_e.g._, "_The food in the restaurant is awesome_")1.

Footnote 1: Another possible way to organize sentences is a hierarchy of topics, _e.g._, a parent node can be a sentence on “_sports_”, while its children are sentences on “_basketball_” and “_skiking_”.

In existing VAE-based generative models, such structures are not _explicitly_ considered. The latent code often employs a simple Gaussian prior, and the posterior is approximated as a Gaussian with diagonal covariance matrix. Such embeddings assume Euclidean structure, which is inadequate in capturing geometric structure illustrated in Figure 1. While some variants have been proposed to enrich the prior distributions [60, 78, 9, 10], there is no evidence that structural information in language can be recovered effectively by the model.

Hyperbolic geometry has recently emerged as an effective method for representation learning from data with hierarchical structure [19]. In contrast, the latent space is a hierarchical structure, but it does not necessarily capture the latent space. This is because the latent space is a manifold, and the latent space is a manifold.

Figure 1: Illustration of the latent hierarchy in natural language. Each tree node is a latent code of its corresponding sentence.



2019; Nickel and Kiela, 2017). Informally, hyperbolic space can be considered as a continuous map of trees. For example, a Poincare disk (a hyperbolic space with two dimensions) can represent any tree with arbitrary low distortion (De Sa et al., 2018; Sarkar, 2011). In Euclidean space, however, it is difficult to learn such structural representation even with infinite dimensions (Linial et al., 1995).

Motivated by these observations, we propose Adversarial Poincare Variational Autoencoder (APOVAE), a text embedding and generation model based on hyperbolic representations, where the latent code is encouraged to capture the underlying tree-like structure in language. Such latent structure provides more control of the generated sentences, \(i.e.\), an increase of sentence complexity and diversity can be achieved along some trajectory from a root to its children. In practice, we define both the prior and the variational posterior of the latent code over a Poincare ball, via the use of a wrapped normal distribution (Nagano et al., 2019). To obtain more stable model training and learn more flexible representation of the latent code, we exploit the primal-dual formulation of Kullback-Leibler (KL) divergence (Dai et al., 2018) based on the Fenchel duality (Rockafellar et al., 1966), to adversarially optimize the variational bound. Unlike the primal form that relies on Monte Carlo approximation (Mathieu et al., 2019), our dual formulation bypasses the need for tractable posterior likelihoods via the introduction of an auxiliary dual function.

We apply the proposed approach to language modeling, unaligned style transfer and dialogue-response generation. For language modeling, in order to enhance the distribution complexity of the prior, we use an additional "variational mixture of posteriors" prior (VampPrior) design (Tomczak and Welling, 2018) for the wrapped normal distribution. Specifically, VampPrior uses a mixture distribution with components from variational posteriors, coupling the parameters of the prior and variational posterior. For unaligned style transfer, we add a sentiment classifier to our model, and disentangle content and sentiment information by using adversarial training (Zhao et al., 2017). For dialog-response generation, a conditional model variant of APo-VAE is designed to take into account the dialog context.

Experiments also show that the proposed model addresses _posterior collapse_(Bowman et al., 2016), a major obstacle preventing efficient learning of a VAE on text data. In posterior collapse, the encoder learns an approximate posterior similar to the prior, and the decoder tends to ignore the latent code for generation. Experiments show that our proposed model can effectively avoid posterior collapse. We hypothesize that this is due to the use of a more informative prior in hyperbolic space that enhances the complexity of the latent representation, which aligns well with previous work (Tomczak and Welling, 2018; Wang et al., 2019) that advocates a better prior design.

Our main contributions are summarized as follows. (\(i\)) We present Adversarial Poincare Variational Autoencoder (APo-VAE), a novel approach to text embedding and generation based on hyperbolic latent representations. (\(ii\)) In addition to the use of a wrapped normal distribution, an adversarial learning procedure and a VampPrior design are incorporated for robust model training. (\(iii\)) Experiments on language modeling, unaligned style transfer, and dialog-response generation benchmarks demonstrate the superiority of the proposed approach compared to Euclidean VAEs, as it benefits from capturing informative latent hierarchies in natural language.

## 2 Preliminaries

### Variational Autoencoder

Let \({\mathbf{X}}=\{{\mathbf{x}}_{i}\}_{i=1}^{N}\) be a dataset of sentences, where each \({\mathbf{x}}_{i}=[x_{i,1},...,x_{i,T_{i}}]\) is a sequence of tokens of length \(T_{i}\). Our goal is to learn \(p_{\mathbf{\theta}}({\mathbf{x}})\) that best models the observed sentences so that the expected log-likelihood is maximized, _i.e._, \(\mathcal{L}({\mathbf{\theta}})=\frac{1}{N}\sum_{i}\log p_{\mathbf{\theta}}({\mathbf{x}}_{ i})\).

The variational autoencoder (VAE) (Kingma and Welling, 2013; Chen et al., 2018) considers a latent-variable model \(p_{\mathbf{\theta}}({\mathbf{x}},{\mathbf{z}})\) to represent sentences, with an auxilary encoder that draws samples of latent code \({\mathbf{z}}\) from the conditional density \(q_{\mathbf{\phi}}({\mathbf{z}}|{\mathbf{x}})\), known as the approximate posterior. Given a latent code \({\mathbf{z}}\), the decoder samples a sentence from the conditional density \(p_{\mathbf{\theta}}({\mathbf{x}}|{\mathbf{z}})=\prod_{t}p({\mathbf{x}}_{t}|{\mathbf{x}}_{<t},{\mathbf{ z}})\), where the "decoding" pass takes an auto-regressive form. Together with prior \(p({\mathbf{z}})\), the model is given by the joint \(p_{\mathbf{\theta}}({\mathbf{x}},{\mathbf{z}})=p_{\mathbf{\theta}}({\mathbf{x}}|{\mathbf{z}})p({\mathbf{z}})\). The VAE leverages the approximate posterior to derive an _evidence lower bound_ (ELBO) to the (intractable) marginal log-likelihood 

\[\log p_{\mathbf{\theta}}(\mathbf{x})=\log\int p_{\mathbf{\theta}}(\mathbf{x},\mathbf{z}) \operatorname*{d}\!\mathbf{z}:\\]

\[\mathcal{L}(\mathbf{x};\mathbf{\theta},\mathbf{\phi})=\mathbb{E}_{\mathbf{z}\sim q_{\mathbf{\phi}}( \mathbf{z}|\mathbf{x})}\left[\log\frac{p_{\mathbf{\theta}}(\mathbf{x},\mathbf{z})}{q_{\mathbf{\phi}}( \mathbf{z}|\mathbf{x})}\right]\,, \tag{1}\]

where \((\mathbf{\theta},\mathbf{\phi})\) are jointly optimized during training, and the gap is given by the decomposition

\[\log p_{\mathbf{\theta}}(\mathbf{x})=\mathcal{L}(\mathbf{x};\mathbf{\theta},\mathbf{\phi})+ \mathbb{D}_{\text{KL}}(p_{\mathbf{\theta}}(\mathbf{z}|\mathbf{x})\parallel q_{\mathbf{\phi}}( \mathbf{z}|\mathbf{x}))\,, \tag{2}\]

where \(\mathbb{D}_{\text{KL}}\) denotes Kullback-Leibler divergence. Alternatively, the ELBO can be written as:

\[\mathcal{L}(\mathbf{x};\mathbf{\theta},\mathbf{\phi}) =\mathbb{E}_{\mathbf{z}\sim q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x})}\left[\log p _{\mathbf{\theta}}(\mathbf{x}|\mathbf{z})\right]\] \[~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} -\mathbb{D}_{\text{KL}}(q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x})\parallel p(\mathbf{z}))\,, \tag{3}\]

where the first conditional likelihood and second KL terms respectively characterize reconstruction and generalization capabilities. Intuitively, a good model is expected to strike a balance between good reconstruction and generalization. In most cases, both the prior and variational posterior are assumed to be Gaussian for computational convenience. However, such over-simplified assumptions may not be ideal for capturing the intrinsic characteristics of data that have unique geometrical structure, such as natural language.

### Hyperbolic Space

Riemannian manifolds can provide a more powerful and meaningful embedding space for complex data with highly non-Euclidean structure, that cannot be effectively captured in a vectorial form (_e.g._, social networks, biology and computer graphics). Of particular interest is the hyperbolic space (Ganea et al., 2018), where (\(i\)) the relatively simple geometry allows tractable computations, and (\(ii\)) the exponential growth of distance in finite dimensions naturally embeds rich hierarchical structure in a compact form.

Riemannian Geometry.An \(n\)-dimensional Riemannian manifold \(\mathcal{M}^{n}\) is a set of points locally similar to a linear space \(\mathbb{R}^{n}\). At each point \(\mathbf{x}\) of the manifold \(\mathcal{M}^{n}\), we can define a real vector space \(\mathcal{T}_{\mathbf{x}}\mathcal{M}^{n}\) that is tangent to \(\mathbf{x}\), along with an associated _metric tensor_\(g_{\mathbf{x}}(\cdot,\cdot):\mathcal{T}_{\mathbf{x}}\mathcal{M}^{n}\times\mathcal{ T}_{\mathbf{x}}\mathcal{M}^{n}\rightarrow\mathbb{R}\) which is an inner product on \(\mathcal{T}_{\mathbf{x}}\mathcal{M}^{n}\). Intuitively, a Riemannian manifold behaves like a vector space only in its infinitesimal neighborhood, allowing the generalization of common notation like angle, straight line and distance to a smooth manifold. For each tangent space \(\mathcal{T}_{\mathbf{x}}\mathcal{M}^{n}\), there exists a specific one-to-one map \(\exp_{\mathbf{x}}(\mathbf{v}):\mathcal{T}_{\mathbf{x}}\mathcal{M}^{n}\rightarrow\mathcal{ M}^{n}\) from an \(\epsilon\)-ball at the origin of \(\mathcal{T}_{\mathbf{x}}\mathcal{M}^{n}\) to a neighborhood of \(\mathbf{x}\) on \(\mathcal{M}^{n}\), called the _exponential map_. We refer to the inverse of an exponential map as the _logarithm map_, denoted \(\log_{\mathbf{x}}(\mathbf{y}):\mathcal{M}^{n}\rightarrow\mathcal{T}_{\mathbf{x}}\mathcal{ M}^{n}\). In addition, a parallel transport \(P_{\mathbf{x}\rightarrow\mathbf{x}^{\prime}}:\mathcal{T}_{\mathbf{x}}\mathcal{M}^{n} \rightarrow\mathcal{T}_{\mathbf{x}^{\prime}}\mathcal{M}^{n}\) intuitively transports tangent vectors along a "straight" line between \(\mathbf{x}\) and \(\mathbf{x}^{\prime}\), so that they remain "parallel." This is the basic machinery that allows us to generalize distributions and computations in the hyperbolic space, as detailed in later sections.

Poincare Ball Model.Hyperbolic geometry is one type of non-Euclidean geometry with a constant negative curvature. As a classical example of hyperbolic space, an \(n\)-dimensional Poincare ball, with curvature parameter \(c\geq 0\) (_i.e._, radius \(\frac{1}{\sqrt{c}}\)), can be denoted as \(\mathbb{B}_{c}^{n}:=\big{\{}\mathbf{z}\in\mathbb{R}^{n}\,|\,c\|\mathbf{z}\|^{2}<1\big{\}}\) with its metric tensor given by \(g_{\mathbf{z}}^{c}=\lambda_{\mathbf{z}}^{2}g^{E}\), where \(\lambda_{\mathbf{z}}=\frac{2}{1-c\|\mathbf{z}\|^{2}}\) and \(g^{E}\) denotes the regular Euclidean metric tensor. Intuitively, as \(\mathbf{z}\) moves closer to the boundary \(\frac{1}{\sqrt{c}}\), the hyperbolic distance between \(\mathbf{z}\) and a nearby \(\mathbf{z}^{\prime}\) diverges at a rate of \(\frac{1}{1-c\|\mathbf{z}\|^{2}}\rightarrow\infty\). This implies significant representation capacity, as very dissimilar objects can be encoded on a compact domain. Note that as \(c\to 0\), the model recovers the Euclidean space \(\mathbb{R}^{n}\), \(i.e.\), the lack of hierarchy. In comparison, a larger \(c\) implies a stronger hierarchical organization.2

Footnote 2: The fact that APo-VAE outperforms standard VAE evidences the existence of the hierarchical organization in NLP data.

Mathematical Operations.We review the closed-form mathematical operations that enable differentiable training for hyperbolic space models, namely the hyperbolic algebra (vector addition) and tangent space computations (exponential/logarithm map and parallel transport). The hyperbolic algebra is formulated under the framework of _gyrovector spaces_(Ungar, 2008), with the addition of two points \(\mathbf{z},\mathbf{z}^{\prime}\in\mathbb{B}_{c}^{n}\) given by the _Mobius addition_:

\[\mathbf{z}\oplus_{c}\mathbf{z}^{\prime}:= \tag{4}\] \[\quad\frac{(1+2c\langle\mathbf{z},\mathbf{z}^{\prime}\rangle+c\|\mathbf{z}^{ \prime}\|^{2})\mathbf{z}+(1-c\|\mathbf{z}\|^{2})\mathbf{z}^{\prime}}{1+2c\langle\mathbf{z},\bm {z}^{\prime}\rangle+c^{2}\|\mathbf{z}\|^{2}\|\mathbf{z}^{\prime}\|^{2}}.\]

For any point \(\mathbf{\mu}\in\mathbb{B}_{c}^{n}\), the exponential map and the logarithmic map are given for \(\mathbf{u}\neq\mathbf{0}\) and \(\mathbf{y}\neq\mathbf{\mu}\) 

by

\[\exp^{c}_{\mathbf{\mu}}(\mathbf{u}):=\mathbf{\mu}\oplus_{c}(\tanh(\sqrt{c} -\frac{\lambda^{c}_{\mathbf{\mu}}\|\mathbf{u}\|}{2})\frac{\mathbf{u}}{\sqrt{c}\|\mathbf{u}\|}), \tag{5}\] \[\log^{c}_{\mathbf{\mu}}(\mathbf{y}):=\frac{2}{\sqrt{c}\lambda^{c}_{\mathbf{\mu }}}\tanh^{-1}(\sqrt{c}\|\mathbf{\kappa}_{\mathbf{\mu},\mathbf{y}}\|)\frac{\mathbf{\kappa}_{\bm {\mu},\mathbf{y}}}{\|\mathbf{\kappa}_{\mathbf{\mu},\mathbf{y}}\|},\]

where \(\mathbf{\kappa}_{\mathbf{\mu},\mathbf{y}}:=(-\mathbf{\mu})\oplus_{c}\mathbf{y}.\) Note that the Poincare ball model is _geodesically complete_ in the sense that \(\exp^{c}_{\mathbf{\mu}}\) is well-defined on the full tangent space \(\mathcal{T}_{\mathbf{\mu}}\mathbb{B}^{n}_{c}\). The parallel transport map from a vector \(\mathbf{v}\in\mathcal{T}_{0}\mathbbm{B}^{n}_{c}\) to another tangent space \(\mathcal{T}_{\mathbf{\mu}}\mathbb{B}^{n}_{c}\) is given by

\[P^{c}_{\mathbf{0}\rightarrow\mathbf{\mu}}(\mathbf{v})=\log^{c}_{\mathbf{\mu}}(\mathbf{\mu} \oplus_{c}\exp^{c}_{\mathbf{0}}(\mathbf{v}))=\frac{\lambda^{c}_{\mathbf{0}}}{ \lambda^{c}_{\mathbf{\mu}}}\mathbf{v}. \tag{6}\]

## 3 Adversarial Poincare VAE

We first introduce our hyperbolic encoder and decoder, and how to apply reparametrization. We then provide detailed descriptions on model implementation, explaining how the primal-dual form of KL divergence can help stabilize training. Finally, we describe how to adopt VampPrior Tomczak and Welling (2018) to enhance performance. A summary of our model scheme is provided in Figure 2.

### Flexible Wrapped Distribution Encoder

We begin by generalizing the standard normal distribution to a Poincare ball Ganea et al. (2018). While there are a few competing definitions of the hyperbolic normal, we choose the wrapped normal as our prior and variational posterior, largely due to its flexibility for more expressive generalization. A wrapped normal distribution \(\mathcal{N}_{\mathbb{B}^{n}_{c}}(\mathbf{\mu},\mathbf{\Sigma})\) is defined as follows: (\(i\)) sample vector \(\mathbf{v}\) from \(\mathcal{N}(\mathbf{0},\mathbf{\Sigma})\), (\(ii\)) parallel transport \(\mathbf{v}\) to \(\mathbf{u}:=P^{c}_{\mathbf{0}\rightarrow\mathbf{\mu}}(\mathbf{v})\), and (\(iii\)) using exponential map to project \(\mathbf{u}\) back to \(\mathbf{z}:=\exp^{c}_{\mathbf{\mu}}(\mathbf{u})\). Putting these together, a latent sample has the following reparametrizable form:

\[\mathbf{z}=\exp^{c}_{\mathbf{\mu}}\left(\frac{\lambda^{c}_{\mathbf{0}}}{\lambda^{c}_ {\mathbf{\mu}}}\mathbf{v}\right),\mathbf{v}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma}). \tag{7}\]

For approximate posteriors, \((\mathbf{\mu},\mathbf{\Sigma})\) depends on \(\mathbf{x}\). We further generalize the (restrictive) hyperbolic wrapped normal by acknowledging that under the implicit VAE Fang et al. (2019) framework, one does not need the approximate posterior \(q_{\phi}(\mathbf{z}|\mathbf{x})\) to be analytically tractable. This allows us to replace the tangent space sampling step \(\mathbf{v}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma})\) in (7) with a more flexible implicit distribution from which we draw samples as \(\mathbf{v}:=G(\mathbf{x},\mathbf{\xi};\mathbf{\phi}_{1})\) for \(\mathbf{\xi}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\). Note that now \(\mathbf{\mu}:=F(\mathbf{x};\mathbf{\phi}_{2})\) can be regarded as a deterministic displacement vector that anchors embeddings to the correct semantic neighborhood, allowing the stochastic \(\mathbf{v}\) to only focus on modeling the local uncertainty of the semantic embedding. The synergy between the deterministic and stochastic parts enables efficient representation learning relative to existing alternatives. For simplicity, we denote the encoder neural network as EncNet\({}_{\mathbf{\phi}}\), which contains \(G\) and \(F\), with parameters \(\mathbf{\phi}=\{\mathbf{\phi}_{1},\mathbf{\phi}_{2}\}\).

### Poincare Decoder

To build a geometry-aware decoder for a hyperbolic latent code, we follow Ganea et al. (2018), and use a generalized linear function analogously defined in the hyperbolic space. A Euclidean linear function takes the form \(f(\mathbf{z})=\langle\mathbf{a},\mathbf{z}-\mathbf{b}\rangle=\text{sign}(\langle\mathbf{a},\mathbf{z} -\mathbf{b}\rangle)\lVert\mathbf{a}\rVert d^{E}(\mathbf{z},H_{\mathbf{a},\mathbf{b}})\), where \(\mathbf{a}\) is the coefficient, \(\mathbf{b}\) is the intercept, \(H_{\mathbf{a},\mathbf{b}}\) is a hyperplane passing through \(\mathbf{b}\) with \(\mathbf{a}\) as the normal direction, and \(d^{E}(\mathbf{z},H)\) is the distance between \(\mathbf{z}\) and hyperplane \(H\). The 

counterpart in Poincare ball analogously writes

\[f^{c}_{\mathbf{a,b}}(\mathbf{z})=\text{sign}(\langle\mathbf{a},\log^{c}_{\mathbf{b} }(\mathbf{z})\rangle_{\mathbf{b}})\|\mathbf{a}\|_{\mathbf{b}}d^{\mathrm{B}}_{c}(\mathbf{z},H^{c}_{ \mathbf{a},\mathbf{b}}), \tag{8}\]

where \(H^{c}_{\mathbf{a},\mathbf{b}}=\left\{\mathbf{z}\in\mathbb{E}_{c}^{n}|\langle\mathbf{a},\log^{ c}_{\mathbf{b}}(\mathbf{z})\rangle_{\mathbf{b}}=0\right\}\), and \(d^{\mathrm{B}}_{c}(\mathbf{z},H^{c}_{\mathbf{a},\mathbf{b}})=\frac{1}{\sqrt{c}}\sinh^{-1} \left(\frac{2\sqrt{|\langle\mathbf{\kappa}_{\mathbf{b}},\mathbf{z}\rangle}\cdot\mathbf{a}|}{(1 -c\epsilon\mathbf{\kappa}_{\mathbf{b},\mathbf{z}})^{2}\|\mathbf{a}\|}\right)\) are the the gyrophane and the distance between \(\mathbf{z}\) and the gyroplane, respectively. Specifically, we use the hyperbolic linear function in (8) to extract features from the Poincare embedding \(\mathbf{z}\). The feature \(f^{c}_{\mathbf{a},\mathbf{b}}(\mathbf{z})\) will be the input to the RNN decoder. We denote the combined network of \(f^{c}_{\mathbf{a},\mathbf{b}}\) and the RNN decoder as DecNet\({}_{\mathbf{\theta}}\), where parameters \(\mathbf{\theta}\) contain \(\mathbf{a}\) and \(\mathbf{b}\).

### Implementing APo-VAE

While it is straightforward to compute the ELBO (3) via Monte Carlo estimates using the explicit wrapped normal density Mathieu et al. (2019), we empirically observe that: (\(i\)) the normal assumption restricts the expressiveness of the model, and (\(ii\)) the wrapped normal likelihood makes the training unstable. Therefore, we appeal to a primal-dual view of VAE training to overcome such difficulties Rockafellar et al. (1966); Dai et al. (2018); Tao et al. (2019); Fang et al. (2019). Specifically, the KL term in (3) can be reformulated as:

\[\mathbb{D}_{\textrm{KL}}(q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x})\parallel p( \mathbf{z}))=\max_{\mathbf{\psi}} \tag{9}\] \[\left\{\mathbb{E}_{\mathbf{z}\sim q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x})}\nu_{ \mathbf{\psi}}(\mathbf{x},\mathbf{z})-\mathbb{E}_{\mathbf{z}\sim p(\mathbf{z})}\exp\nu_{\mathbf{\psi} }(\mathbf{x},\mathbf{z})\right\},\]

where \(\nu_{\mathbf{\psi}}(\mathbf{x},\mathbf{z})\) is the (auxiliary) dual function (_i.e._, a neural network) with parameters \(\mathbf{\psi}\). The primal-dual view of the KL term enhances the approximation ability, while also being tractable computationally. Meanwhile, since the density function in the original KL term in (3) is replaced by the dual function \(\nu_{\mathbf{\psi}}(\mathbf{x},\mathbf{z})\), we can avoid direct computation with respect to the probability density function of the wrapped normal distribution.

To train our proposed APo-VAE with the primal-dual form of the VAE objective, we follow the training schemes of coupled variational Bayes (CVB) Dai et al. (2018) and implicit VAE Fang et al. (2019), which optimize the objective adversarially. Specifically, we update \(\mathbf{\psi}\) in the dual function \(\nu_{\mathbf{\psi}}(\mathbf{x},\mathbf{z})\) to maximize:

\[\mathcal{L}_{1}=\mathbb{E}_{\mathbf{x}\sim\mathbf{X}}[ \,\mathbb{E}_{\mathbf{z}\sim q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x})}\nu_{\mathbf{ \psi}}(\mathbf{x},\mathbf{z})\] \[-\mathbb{E}_{\mathbf{z}\sim p(\mathbf{z})}\exp\nu_{\mathbf{\psi}}(\mathbf{x},\mathbf{ z})\,]\,, \tag{10}\]

where \(\mathbb{E}_{\mathbf{x}\sim\mathbf{X}}[\cdot]\) denotes the expectation over empirical distribution on observations. Accordingly, parameters \(\mathbf{\theta}\) and \(\mathbf{\phi}\) are updated to maximize:

\[\mathcal{L}_{2}=\mathbb{E}_{\mathbf{x}\sim\mathbf{X}}\mathbb{E}_{\mathbf{z} \sim q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x})}[ \,\log p_{\mathbf{\theta}}(\mathbf{x}|\mathbf{z})\] \[-\nu_{\mathbf{\psi}}(\mathbf{x},\mathbf{z})\,]. \tag{11}\]

Note that the term \(\mathbb{E}_{\mathbf{x}\sim\mathbf{X}}\mathbb{E}_{\mathbf{z}\sim q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{ x})}\nu_{\mathbf{\psi}}(\mathbf{x},\mathbf{z})\) is maximized in (10) while it is minimized in (11), _i.e._, adversarial learning. In other words, one can consider the dual function as a discriminative network that distinguishes between the prior \(\mathbf{z}\sim p(\mathbf{z})\) and the variational posterior \(\mathbf{z}\sim q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x})\), both of which are paired with the input data \(\mathbf{x}\sim\mathbf{X}\).

```
1:Input: Data samples \(\mathbf{X}=\{\mathbf{x}_{i}\}_{i=1}^{N}\), Poincaré curvature \(c\), and number of pseudo-input \(K\).
2:Initialize \(\mathbf{\theta}\), \(\mathbf{\phi}\), \(\mathbf{\psi}\), and \(\mathbf{\delta}\).
3:for\(iter\) from 1 to \(max\_iter\)do
4: Sample a mini-batch \(\{\mathbf{x}_{m}\}_{m=1}^{M}\) from \(\mathbf{X}\) of size \(M\).
5:# Sampling in the Hyperbolic Space.
6: Obtain \(\mathbf{\mu}_{m}\) and \(\mathbf{v}_{m}\) from EncNet\({}_{\mathbf{\phi}}(\mathbf{x}_{m})\).
7: Move \(\mathbf{v}_{m}\) to \(\mathbf{u}_{m}=P^{C}_{\mathbf{0}\to\mathbf{\mu}_{m}}(\mathbf{v}_{m})\) by (6).
8: Map \(\mathbf{u}_{m}\) to \(\mathbf{z}_{m}=\exp^{c}_{\mathbf{\mu}_{m}}(\mathbf{u}_{m})\) by (5).
9:# Update the dual function and the pseudo-input.
10: Sample \(\tilde{\mathbf{z}}_{m}\) by (12).
11: Update \(\mathbf{\psi}\) and \(\mathbf{\delta}\) by gradient ascent on (10)
12:# Update the encoder and decoder networks.
13: Update \(\mathbf{\theta}\) and \(\mathbf{\phi}\) by gradient ascent on (11).
14:endfor
```

**Algorithm 1** Training procedure of Apo-VAE.

 

learnable distribution given by

\[p_{\mathbf{\delta}}(\mathbf{z})=\frac{1}{K}{\sum_{k=1}^{K}}q_{\mathbf{\phi}}( \mathbf{z}|\mathbf{s}_{k}), \tag{12}\]

where \(q_{\mathbf{\phi}}\) is the learned approximate posterior, and we call the parameter \(\mathbf{\delta}:=\{\mathbf{s}_{k}\}_{k=1}^{K}\) pseudo inputs. Intuitively, \(p_{\mathbf{\delta}}(\mathbf{z})\) seeks to match the aggregated posterior Makhzani et al. (2015): \(q(\mathbf{z})=\frac{1}{N}\sum_{i=1}^{N}q_{\mathbf{\phi}}(\mathbf{z}|\mathbf{x}_{i})\) in a cost-efficient manner via parameterizing the pseudo inputs. By replacing the prior distribution \(p(\mathbf{z})\) in (10) with \(p_{\mathbf{\delta}}(\mathbf{z})\), we complete the final objective of the proposed APo-VAE. The detailed training procedure is summarized in Algorithm 1.

## 4 Related Work

VAE for Text Generation.Many VAE models have been proposed for text generation, most of which focus on solving the issue of posterior collapse. The most popular strategy is to alter the training dynamics, keeping the encoder away from bad local optima. For example, variants of KL annealing Bowman et al. (2016); Zhao et al. (2018); Fu et al. (2019) dynamically adjust the weight on the KL penalty term as training progresses. Lagging VAE He et al. (2019) aggressively optimizes the encoder before each decoder update, to overcome the imbalanced training issue between the encoder and decoder. Alternative strategies have also been proposed based on competing theories or heuristics. \(\delta\)-VAE Razavi et al. (2019) tackles this issue by enforcing a minimum KL divergence between the posterior and the prior. Yang et al. (2017) blames mode-collapse on the auto-regressive design of the decoder and advocates alternative architectures. A semi-amortized inference network is considered by Kim et al. (2018) to bridge the amortization gap between log-likelihood and the ELBO.

Recent work has also shown that posterior collapse can be ameliorated by using more expressive priors and variational posteriors other than Gaussian. Flow-based VAE is considered in Ziegler and Rush (2019) to enhance the flexibility of prior distributions. A topic-guided prior is proposed in Wang et al. (2019a) to achieve more controllable text generation. Fang et al. (2019) explores implicit sample-based representations, without requiring an explicit density form for the approximate posterior. Xu and Durrett (2018) considers replacing the Gaussian with the spherical von Mises-Fisher (vMF) distribution. Compared to these prior arts, our model features structured representation in hyperbolic space, which not only captures latent hierarchies but also combats posterior collapse.

Hyperbolic Space Representation Learning.There has been a recent surge of interest in representation learning in hyperbolic space, largely due to its exceptional effectiveness modeling data with underlying graphical structure Chamberlain et al. (2017), such as relation nets Nickel and Kiela (2017). In the context of NLP, hyperbolic geometry has been considered for word embeddings Tifrea et al. (2018). A popular vehicle for hyperbolic representation learning is the autoencoder AE framework Grattarola et al. (2019); Ovinnikov (2019), where the decoders are built to efficiently exploit the hyperbolic geometry Ganea et al. (2018). Closest to our APo-VAE are the works of hyperbolic VAEs Mathieu et al. (2019); Nagano et al. (2019), where wrapped normal distributions have been used. Drawing power from the dual form of the KL, the proposed APo-VAE highlights an implicit posterior and data-driven prior, showing improved training stability.

## 5 Experiments

We evaluate the proposed model on three tasks: (\(i\)) language modeling, (\(ii\)) unaligned style transfer, and (\(iii\)) dialog-response generation, with quantitative results, human evaluation and qualitative analysis.

### Experimental Setup

Datasets.We use three datasets for language modeling: _Penn Treebank_ (PTB) Marcus et al. (1993), _Yahoo_ and _Yelp_ corpora Yang et al. (2017). PTB contains one million words of 1989 Wall Street Journal material annotated in Treebank II style, with 42k sentences of varying lengths. Yahoo and Yelp are much larger datasets, each containing 100k sentences with greater average length.

For unaligned style transfer, we use the Yelp restaurant reviews dataset Shen et al. (2017), which is obtained by pre-processing the Yelp dataset, \(i.e.\), sentences are shortened for more feasible sentence level sentiment analysis. Overall, the dataset includes 350k positive and 250k negative reviews (based on user rating).

Following Gu et al. (2019), we use the Switchboard Godfrey and Holliman (1997) dataset for dialogue-response generation. The former contains 2.4k two-sided telephone conversations, manually 

transcribed and aligned. We split the data into training, validation and test sets following the protocol described in Zhao et al. (2017).

Evaluation Metrics.To benchmark language modeling performance, we report the ELBO and _Perplexity_ (PPL) of APo-VAE and baselines. In order to verify our proposed Apo-VAE is more resistant to posterior collapse, we also report the KL-divergence \({\mathbb{D}}_{\text{KL}}(q_{\phi}(\mathbf{z}|\mathbf{x})||p(\mathbf{z}))\) and _mutual information_ (MI) between \(\mathbf{z}\) and \(\mathbf{x}\)He et al. (2019). The number of active units (AU) of the latent code is also reported, where the activity of a latent dimension \(z\) is measured as \(A_{\mathbf{z}}=\text{Cov}_{\mathbf{z}}{\mathbb{E}}_{\mathbf{z}\sim q_{\phi}(\mathbf{z}|\mathbf{x}) }[z]\), and defined as active if \(A_{\mathbf{z}}>0.01\).

To evaluate our model on unaligned style transfer, we consider the transfer accuracy from one sentiment to another, the BLEU scores between original and transferred sentences, the reconstruction perplexity of original sentences, and the reverse perplexity (RPPL) based on a language model from the transferred sentences.

For dialogue-response generation, we adopt the evaluation metrics used in previous studies Zhao et al. (2017); Gu et al. (2019), including BLEU Papineni et al. (2002), BOW Liu et al. (2016), and _intra/inter-dist_ values Gu et al. (2019). The first two metrics are used to assess the relevance of the generated response, and the third is for diversity evaluation.

Model Implementation.For language modeling, we adopt the LSTM Hochreiter and Schmidhuber (1997) for both the encoder and decoder, with dimension of the latent code set to \(32\). Following Mathieu et al. (2019), the hyper-parameter \(c\) is set to \(0.7\). For unaligned style transfer, we extend our model in the same fashion as Fang et al. (2019). For dialogue-response generation, we modify APO-VAE following the conditional VAE framework Zhao et al. (2017). Specifically, an extra input of context embedding \(\mathbf{s}\) is supplied to the model (_i.e._, \(p_{\theta}(\mathbf{x},\mathbf{z}|\mathbf{s}),q_{\phi}(\mathbf{z}|\mathbf{x},\mathbf{s})\)). The prior \(p(\mathbf{z}|\mathbf{s})\) is a wrapped normal conditioned on context embedding, learned together with the posterior.

### Experimental Results

Language Modeling.Table 1 shows results on language modeling. We compare APo-VAE with other VAE-based solutions, including \(\beta\)-VAE Higgins et al. (2017), SA-VAE Kim et al. (2018), lagging VAE (LAG-VAE) He et al. (2019), vMF-VAE Xu and Durrett (2018), Poincare VAE (\(\mathcal{P}\)-VAE) Mathieu et al. (2019) and iVAE3Fang et al. (2019). On all three datasets, the proposed model achieves lower negative ELBO and PPL than other models, demonstrating its strong ability to better model sequential text data. Meanwhile, the larger KL term and higher mutual information (between \(\mathbf{z}\) and \(\mathbf{x}\)) of APo-VAE model indicate its robustness in handling posterior collapse. In addition, the introduction of a data-driven prior (denoted as APo-VAE+VP) further boosts the performance, especially on negative ELBO and PPL.

Footnote 3: We report iVAE\({}_{\text{MH}}\) results in all our experiments.

Visualization.To verify our hypothesis that the proposed model is capable of learning latent tree structure in text data, we visualize the two-dimensional projection of the learned latent code in Figure 3. For visualization, we randomly draw 5k samples from PTB-test, and encode them to the latent space using the APo-VAE encoder. We color-code each sentence based on its length (_i.e._, blue for long sentences and red for short sentences). Note that only a small portion of data have a length longer than \(32\) (\(<10\%\)), and human inspection

\begin{table}
\begin{tabular}{l|r r r r r} \hline Model & -ELBO & PPL & KL & MI & AU \\ \cline{2-4}  & \multicolumn{4}{c}{PTB} \\ \hline VAE & 102.6 & 108.26 & 1.1 & 0.8 & 2 \\ \(\beta\)-VAE & 104.5 & 117.92 & 7.5 & 3.1 & 5 \\ SA-VAE & 102.6 & 107.71 & 1.2 & 0.7 & 2 \\ vMF-VAE & 95.8 & 93.70 & 2.9 & 3.2 & 21 \\ p-VAE & 91.4 & 76.13 & 4.5 & 2.9 & 23 \\ iVAE & 87.2 & 53.44 & **12.5** & **12.2** & **32** \\ APo-VAE & 87.2 & 53.32 & 8.4 & 4.8 & **32** \\ APo-VAE+VP & **87.0** & **53.02** & 8.9 & 4.5 & **32** \\ \hline \end{tabular} \begin{tabular}{l l l l l l} \hline VAE & 328.6 & 61.21 & 0.0 & 0.0 & 0 \\ \(\beta\)-VAE & 328.7 & 61.29 & 6.3 & 2.8 & 8 \\ SA-VAE & 327.2 & 60.15 & 5.2 & 2.9 & 10 \\ LAG-VAE & 326.7 & 59.77 & 5.7 & 2.9 & 15 \\ VMF-VAE & 318.5 & 53.92 & 6.3 & 3.7 & 23 \\ \(p\)-VAE & 313.4 & 50.57 & 7.2 & 3.3 & 27 \\ iVAE & 309.1 & 47.93 & **11.4** & **10.7** & **32** \\ APo-VAE & 286.2 & 47.00 & 6.9 & 4.1 & **32** \\ APo-VAE+VP & **285.6** & **46.61** & 8.1 & 4.9 & **32** \\ \hline \end{tabular} 
\begin{tabular}{l l l l l l} \hline VAE & 357.9 & 40.56 & 0.0 & 0.0 & 0 \\ \(\beta\)-VAE & 358.2 & 40.69 & 4.2 & 2.0 & 4 \\ SA-VAE & 357.8 & 40.51 & 2.8 & 1.7 & 8 \\ LAG-VAE & 355.9 & 39.73 & 3.8 & 2.4 & 11 \\ vMF-VAE & 356.2 & 51.03 & 4.1 & 3.9 & 13 \\ \(p\)-VAE & 355.4 & 50.64 & 4.3 & 4.8 & 19 \\ iVAE & 348.7 & 36.88 & 11.6 & **11.0** & **32** \\ APo-VAE & 319.7 & 34.10 & 12.1 & 7.5 & **32** \\ APo-VAE+VP & **316.4** & **32.91** & **12.7** & 6.2 & **32** \\ \hline \end{tabular}
\end{table}
Table 1: Results on PTB, Yahoo, and Yelp datasets. A better language model achieves lower negative ELBO and PPL. Higher KL and MI indicate a better utilization of the latent space.

 

verified that most of them contain multiple sub-sentences. We exclude these samples from our analysis.

As shown in Figure 3, longer sentences (dark blue) tend to occupy the outer rim of the Poincare ball, while the shorter ones (dark red) are concentrated in the inner area. We also select some long sample sentences (dark blue), and manually shorten them to create several variants of different lengths (ranging from 6 to 27), which are related in a hierarchical manner based on human judgement. We visualize their latent codes projected by the trained APo-VAE. The resulting plot is consistent with a hierarchical structure for APo-VAE: as the sentence becomes more specific, the embedding moves outward. We also decode from the neighbours of these latent codes, the outputs (see the Appendix) of which demonstrate a similar hierarchical structure.

Unaligned Style Transfer.Table 3 shows the results on the Yelp restaurant reviews dataset. APo-VAE achieves over 1 point increased BLEU scores than iVAE, capturing a more informative and structured feature space. Comparable performance is achieved for the other evaluation metrics.

Dialogue Response Generation.Results on SwitchBoard are summarized in Table 2. Our proposed model generates comparable or better responses than the baseline models in terms of both relevance (BLEU and BOW) and diversity (intra/inter-dist). APo-VAE improves the average recall from \(0.427\) (by iVAE) to \(0.438\), while significantly enhancing generation diversity (_e.g._, from \(0.692\) to \(0.792\) for intra-dist-2).

Human Evaluation.We further perform human evaluation via Amazon Mechanical Turk. We asked the turkers to compare generated responses from two models, and assess each model's informative

Figure 3: Visualization of the hyperbolic latent space of 5,000 randomly sampled sentences from the test set of PTB. The lengths of the samples are color-coded (red for short ones and blue for longer ones). The six listed sentences are created by modifying data samples.

\begin{table}
\begin{tabular}{l|c c c|c c c} \hline  & \multicolumn{2}{c|}{vs iVAE} & \multicolumn{2}{c}{vs DialogWAE} \\ \cline{2-7}  & win & loss & tie & win & loss & tie \\ \hline Informativeness & 52.8 & 27.9 & 19.3 & 63.7 & 27.1 & 19.2 \\ Coherence & 41.7 & 35.5 & 22.8 & 41.2 & 34.4 & 24.4 \\ Diversity & 51.2 & 26.4 & 22.4 & 62.1 & 25.1 & 12.8 \\ \hline \end{tabular}
\end{table}
Table 4: Human evaluation results. Win/loss/tie indicates the percentage of responses generated by APo-VAE being better/worse/equal to the compared model.

 

ness, relevance to the dialog context (coherence), and diversity. We use \(500\) randomly sampled contexts from the test set, each assessed by three judges. In order to evaluate diversity, 5 responses are generated for each dialog context. For quality control, only workers with a lifetime task approval rating greater than 98% were allowed to participate in our study. Table 4 summarizes the human evaluation results. The responses generated by our model are clearly preferred by the judges compared with other competing methods.

## 6 Conclusions

We present APo-VAE, a novel model for text generation in hyperbolic space. Our model can learn latent hierarchies in natural language via the use of wrapped normals for the prior. A primal-dual view of KL divergence is adopted for robust model training. Extensive experiments on language modeling, text style transfer, and dialog response generation demonstrate the superiority of the model. For future work, we plan to combine APo-VAE with the currently prevailing large-scale pre-trained language models.

## References

* B Bowman et al. (2016) Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In _CoNLL_.
* Chamberlain et al. (2017) Benjamin Paul Chamberlain, James Clough, and Marc Peter Deisenroth. 2017. Neural embeddings of graphs in hyperbolic space. _arXiv preprint arXiv:1705.10359_.
* Chen et al. (2018a) Liqun Chen, Shuyang Dai, Chenyang Tao, Dinghan Shen, Zhe Gan, Haichao Zhang, Yizhe Zhang, and Lawrence Carin. 2018a. Adversarial text generation via feature-mover's distance. _arXiv preprint arXiv:1809.06297_.
* Chen et al. (2018b) Liqun Chen, Chenyang Tao, Ruiyi Zhang, Ricardo Henao, and Lawrence Carin Duke. 2018b. Variational inference and model selection with generalized evidence bounds. In _International conference on machine learning_, pages 893-902. PMLR.
* Dai et al. (2018) Bo Dai, Hanjun Dai, Niao He, Weiyang Liu, Zhen Liu, Jianshu Chen, Lin Xiao, and Le Song. 2018. Coupled variational bayes via optimization embedding. In _NeurIPS_.
* De Sa et al. (2018) Christopher De Sa, Albert Gu, Christopher Re, and Frederic Sala. 2018. Representation tradeoffs for hyperbolic embeddings. _Proceedings of machine learning research_.
* Fang et al. (2019) Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. 2019. Implicit deep latent variable models for text generation. In _EMNLP_.
* Fu et al. (2019) Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. 2019. Cyclical annealing schedule: A simple approach to mitigating kl vanishing. In _NAACL_.
* Ganea et al. (2018) Octavian Ganea, Gary Becigneul, and Thomas Hofmann. 2018. Hyperbolic neural networks. In _NeurIPS_.
* Godfrey and Holliman (1997) John J Godfrey and Edward Holliman. 1997. Switchboard-1 release 2. _Linguistic Data Consortium, Philadelphia_, 926:927.
* Grattarola et al. (2019) Daniele Grattarola, Lorenzo Livi, and Cesare Alippi. 2019. Adversarial autoencoders with constant-curvature latent manifolds. _Applied Soft Computing_, 81:105511.
* Gu et al. (2019) Xiaodong Gu, Kyunghyun Cho, Jung-Woo Ha, and Sunghun Kim. 2019. Dialogwae: Multimodal response generation with conditional wasserstein autoencoder. In _ICLR_.
* He et al. (2019) Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Lagging inference networks and posterior collapse in variational autoencoders. In _ICLR_.
* Higgins et al. (2017) Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. _ICALR_, 2(5):6.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_.
* Kim et al. (2018) Yoon Kim, Sam Wiseman, Andrew C Miller, David Sontag, and Alexander M Rush. 2018. Semi-amortized variational autoencoders. In _ICML_.
* Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. _arXiv preprint arXiv:1312.6114_.
* Linial et al. (1995) Nathan Linial, Eran London, and Yuri Rabinovich. 1995. The geometry of graphs and some of its algorithmic applications. _Combinatorica_, 15(2):215-245.
* Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. _arXiv preprint arXiv:1603.08023_.
* Makhzani et al. (2015) Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. 2015. Adversarial autoencoders. _arXiv preprint arXiv:1511.05644_ 

Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. _Computational Linguistics_.
* Mathieu et al. (2019) Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Teh. 2019. Continuous hierarchical representations with poincare variational auto-encoders. In _NeurIPS_.
* Mescheder et al. (2017) Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. 2017. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In _ICML_.
* Nagano et al. (2019)Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. 2019. A wrapped normal distribution on hyperbolic space for gradient-based learning. In _ICML_.
* Nickel and Kiela (2017) Maximillian Nickel and Douwe Kiela. 2017. Poincare embeddings for learning hierarchical representations. In _NeurIPS_.
* Ovinnikov (2019) Ivan Ovinnikov. 2019. Poincar\'e wasserstein autoencoder. _arXiv preprint arXiv:1901.01427_.
* Papineni et al. (2002)Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _ACL_.
* Razavi et al. (2019) Ali Razavi, Aaron van den Oord, Ben Poole, and Oriol Vinyals. 2019. Preventing posterior collapse with delta-vaes. In _ICLR_.
* Rezende et al. (2014) Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In _ICML_.
* Rockafellar et al. (1966)R Tyrrell Rockafellar et al. 1966. Extension of fenchel'duality theorem for convex functions. _Duke mathematical journal_.
* Sarkar (2011) Rik Sarkar. 2011. Low distortion delaunay embedding of trees in hyperbolic plane. In _International Symposium on Graph Drawing_, pages 355-366. Springer.
* Shao et al. (2020) Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarek Abdelzaher. 2020. Controlvae: Controllable variational autoencoder. In _ICML_.
* Shen et al. (2017) Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In _NeurIPS_.
* Shi et al. (2019) Wenxian Shi, Hao Zhou, Ning Miao, Shenjian Zhao, and Lei Li. 2019. Fixing gaussian mixture vaes for interpretable text generation. _arXiv preprint arXiv:1906.06719_.
* Tao et al. (2019) Chenyang Tao, Liquen Chen, Shuyang Dai, Junya Chen, Ke Bai, Dong Wang, Jianfeng Feng, Wenlian Lu, Georgiy V Bobashev, and Lawrence Carin. 2019. On fenchel mini-max learning. In _NeurIPS_, pages 10427-10439.
* Tfifrea et al. (2018) Alexandru Tfifrea, Gary Becigneul, and Octavian-Eugen Ganea. 2018. Poincar\'e glove: Hyperbolic word embeddings. _arXiv preprint arXiv:1810.06546_.
* Tomczak and Welling (2018) Jakub M Tomczak and Max Welling. 2018. Vae with a vamprior. In _AISTATS_.
* Uagar (2008) Abraham Albert Uagar. 2008. A gyrovector space approach to hyperbolic geometry. _Synthesis Lectures on Mathematics and Statistics_, 1(1):1-194.
* Wang et al. (2019a) Wenlin Wang, Zhe Gan, Hongteng Xu, Ruyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019a. Topic-guided variational autoencoders for text generation. _arXiv preprint arXiv:1903.07137_.
* Wang et al. (2019b) Wenlin Wang, Chenyang Tao, Zhe Gan, Guoyin Wang, Liqun Chen, Xinyuan Zhang, Ruyi Zhang, Qian Yang, Ricardo Henao, and Lawrence Carin. 2019b. Improving textual network learning with variational homophilic embeddings. _arXiv preprint arXiv:1909.13456_.
* Xu and Durrett (2018) Jiacheng Xu and Greg Durrett. 2018. Spherical latent spaces for stable variational autoencoders. In _EMNLP_.
* Yang et al. (2017) Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. In _ICML_.
* Zhao et al. (2017a) Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M Rush, and Yann LeCun. 2017a. Adversarially regularized autoencoders. _arXiv preprint arXiv:1706.04223_.
* Zhao et al. (2018) Shengjia Zhao, Jiaming Song, and Stefano Ermon. 2018. Infovae: Information maximizing variational autoencoders. In _AAAI_.
* Zhao et al. (2017b) Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017b. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In _ACL_.
* Ziegler and Rush (2019) Zachary M Ziegler and Alexander M Rush. 2019. Latent normalizing flows for discrete sequences. In _ICML_.

 



**Appendix for APo-VAE: Text Generation in Hyperbolic Space**

### Basics of Riemannian Geometry (Extended)

This section provides additional coverage of the basic mathematical concepts used in APo-VAE. For more detailed mathematics expositions, readers are referred to Ganea et al. (2018).

A real, smooth \(n\)-dimensional manifold \({\cal M}\) is a set of points locally similar to a linear space \(\mathbb{R}^{n}\). At each point \(x\) of the manifold \({\cal M}\) is defined a real vector space of the space of the same dimensionality as \({\cal M}\), called the tangent space in \(x\): \({\mathcal{T}_{\mathbf{x}}}{\cal M}\). Intuitively it contains all the possible directions in which one can tangentially pass through \(x\). For each point \(x\) there also defines a _metric tensor_\(g_{\mathbf{x}}(\cdot,\cdot):{\mathcal{T}_{\mathbf{x}}}{\cal M}\times{\mathcal{T}_{\mathbf{ x}}}{\cal M}\to\mathbb{R}\), which defined an inner product on the associated tangent space \({\mathcal{T}_{\mathbf{x}}}{\cal M}\). More specifically, given a coordinate system, the inner product is given in the quadratic form \(g_{\mathbf{x}}(\mathbf{u},\mathbf{v})=\langle\mathbf{u},\mathbf{v}\rangle_{g_{\mathbf{x}}}=\mathbf{u}^{T}G _{\mathbf{x}}\mathbf{v}\), where by slight abuse of notation \(\mathbf{u},\mathbf{v}\in\mathbb{R}^{n}\) are vector representations of the tangent vectors wrt the coordinate system and \(\mathbf{G}_{\mathbf{x}}\in\mathbb{R}^{n\times n}\) is a positive-definite matrix. Collectively, \(({\cal M},g)\) defines a _Riemannian manifold_.

The metric tensor is used to generalize the notations such as distance and volume in Euclidean space to the Riemannian manifold. Given a curve \(\gamma(t):[0,1]\to{\cal M}\), its length is given by

\[L(\gamma)=\int_{0}^{1}\|\gamma^{\prime}(t)\|_{\gamma(t)}\,\mathrm{d}t\,.\] (A1)

The concept of straight lines can then be generalized to _geodesics_, which is the shortest path between pairs of points \(\mathbf{x},\mathbf{y}\) on the manifold \(\gamma^{*}(\mathbf{x},\mathbf{y})=\operatorname*{arg\,min}_{\gamma}L(\gamma)\) such that \(\gamma(0)=\mathbf{x}\) and \(\gamma(1)=\mathbf{y}\) with \(\gamma\) traveling at constant speed (_i.e._, \(\|\gamma^{\prime}(t)\|_{\gamma(t)}=c\), where \(c\) is the distance). The concept of moving along a straight line with constant speed defines the _exponential map_, where for \(\mathbf{v}\in{\mathcal{T}_{\mathbf{x}}}{\cal M}\) there is a unique unit speed geodesic \(\gamma\) satisfying \(\gamma(0)=\mathbf{x}\) with initial tangent vector \(\mathbf{v}\), and the corresponding exponential map is defined by \(\exp_{\mathbf{x}}(\mathbf{v})=\gamma(1)\). We call the inverse of exponential map the _logarithm map_\(\log_{\mathbf{x}}=\exp_{\mathbf{x}}^{-1}:{\cal M}\to{\mathcal{T}_{\mathbf{x}}}{\cal M}\), mapping from the manifold to the tangent space. For the Poincare ball model, it is _geodesically complete_ in the sense that \(\exp_{\mathbf{x}}\) is well-defined on the full tangent space \({\mathcal{T}_{\mathbf{x}}}{\cal M}\).

### Additional Related Work

VAE with Adversarial Learning.One of the first to apply adversarial learning to VAE is Adversarial Variational Bayes (AVB) Mescheder et al. (2017); Pu et al. (2017). Motivated by Generative Adversarial Network (GAN) Goodfellow et al. (2014), AVB introduces an auxiliary discriminator that transforms the maximum-likelihood-problem into a two-player game. Similarly, Adversarial Autoencoder (AAE) Makhzani et al. (2015) uses adversarial learning to match aggregated posterior with the prior. Based on this, Coupled Variational Bayes (CVB) Dai et al. (2018) connects the primal-dual view of ELBO with adversarial learning, where the discriminator takes both data sample and latent code as input. This approach is also adopted in implicit VAE Fang et al. (2019) for text generation. However, the prior used in implicit

VAE is still standard Gaussian, while our proposed model uses hyperbolic geometry.

## Appendix A3 Additional Details for Experiments

### Metrics (Dialogue Response Generation)

BLEU Papineni et al. (2002) is used to measure the amount of \(n\)-gram overlap between a generated response with the reference. Specifically, BLEU scores of \(n<4\) are computed; the average and the maximum scores are considered as \(n\)-gram precision and \(n\)-gram recall, respectively. In addition, the BOW embedding metric Liu et al. (2016) is used to measure cosine similarity between bag-of-word embeddings of response and reference. Three metrics are considered for cosine distance: (\(i\)) computed by **greedily** matching words in two utterances; (\(ii\)) between the **averaged** embeddings in two utterances; and (\(iii\)) between the largest **extreme** values among the embeddings in two utterances. We also follow Gu et al. and use the _distinct_ metric to measure the diversity of generated text. \(Dist-n\) is the ratio of unique \(n\)-grams over all \(n\)-grams in the generated sentences. _Intra-dist_ and _inter-dist_ are the average _distinct_ values within each generated sample and among all generated samples, respectively.

### Additional Implementation Details

For language modeling, we adopt the LSTM Hochreiter and Schmidhuber (1997) for both the encoder and decoder, which have 256 hidden units for PTB, and 1024 hidden units for Yahoo and Yelp. The dimension of the latent code is set to 32. Following Mathieu et al. (2019), the hyper-parameter \(c\) is set to \(0.7\). We set the vocabulary size to 10,000 for PTB, and 20,000 for both Yahoo and Yelp. The word embedding size is 256 for PTB, and 512 for Yahoo and Yelp. For dialogue response generation, we follow Gu et al. (2019), and use the GRU Cho et al. (2014) with 300 hidden units in each direction for both the response encoder and context encoder, and 300 hidden units for decoder. The latent code \(z\) has a dimension of 200. The size of the vocabulary is set to 10,000, and the word-embedding size is 200, 

initialized by GloVe (Pennington et al., 2014).

### Additional Results

For language modeling, we plot the hyperbolic latent space for Yahoo and Yelp as shown in Figure A4. To demonstrate the hierarchical structure in the generated sentences (_i.e._, the decoder), we choose 4 sentences (from short to long) with some hierarchy, listed on the left hand side of Table A1. These sentences are encoded to hyperbolic space by using a well trained APo-VAE. Then, we decode by randomly select a neighbor of each of the 4 latent codes. The output sentences are shown on the right hand side of Table A1, demonstrating similar hierarchy as the input sentences. Moreover, we directly measure the generation quality by using PPL and reverse PPL, shown in Table A3. Our APo-VAE achieves consistently better performance.

For dialog response generation, we include additional results on the DailyDialog dataset Li et al. (2017), which contains 13k daily conversations for an English learner in daily life. We also provide examples of generated responses along with their corresponding dialog context in Table A5. Samples generated by APO-VAE are more relevant to the contexts than the baseline models. In addition, APo-VAE is capable of providing both positive and negative responses, demonstrating better generation diversity. The human evalutation results for DailyDialog can be found in Table A4.

 



### Human Evaluation

We provide the instruction of human evaluation on the dialog response generation task in Figure A5.

## References

* Cho et al. (2014) Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. _arXiv preprint arXiv:1406.1078_.
* Dai et al. (2018) Bo Dai, Hanjun Dai, Niao He, Weiyang Liu, Zhen Liu, Jianshu Chen, Lin Xiao, and Le Song. 2018. Coupled variational bayes via optimization embedding. In _NeurIPS_.
* Fang et al. (2019) Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. 2019. Implicit deep latent variable models for text generation. In _EMNLP_.
* Ganea et al. (2018) Octavian Ganea, Gary Becigneul, and Thomas Hofmann. 2018. Hyperbolic neural networks. In _NeurIPS_.
* Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In _NeurlPS_.
* Gu et al. (2019) Xiaodong Gu, Kyunghyun Cho, Jung-Woo Ha, and Sunghun Kim. 2019. Dialogwae: Multimodal response generation with conditional wasserstein autoencoder. In _ICLR_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_.
* Li et al. (2017) Yanran Li, Hui Su, Xiaoyu Shen, Wernje Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. _arXiv preprint arXiv:1710.03957_.
* Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. _arXiv preprint arXiv:1603.08023_.
* Makhzani et al. (2015) Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. 2015. Adversarial autoencoders. _arXiv preprint arXiv:1511.05644_.
* Mathieu et al. (2019) Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Teh. 2019. Continuous hierarchical representations with poincare variational auto-encoders. In _NeurIPS_.
* Mescheder et al. (2017) Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. 2017. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In _ICML_.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Weijing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _ACL_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_, pages 1532-1543.
* Pu et al. (2017) Yuchen Pu, Weiyao Wang, Ricardo Henao, Liqun Chen, Zhe Gan, Chunyuan Li, and Lawrence Carin. 2017. Adversarial symmetric variational autoencoder. In _NIPS_.





## References

* [1] A. B. Cook, and R. J. Brox. The _Theoretical Computer Science_. Cambridge University Press, Cambridge, MA, USA, 2003.
* [2] M. C. Davis, D. P. Dowling, and S. F. King. _A Practical Guide to the Philosophy of Philosophy of Philosophy_. Addison-Wesley Professional Publishing Company, New York, NY, 2004.

[MISSING_PAGE_POST]





\begin{table}
\begin{tabular}{c|p{142.3pt}|p{85.4pt}|p{76.9pt}} \hline Context & DialogWAE & iVAE & APo-VAE \\ \hline i would wondering if you like to stay here for a few days? & Eg.1: ok. & Eg.1: I’m not going to take a look. & Eg.1: good idea. how about shopping tomorrow morning? \\ \cline{2-4}  & Eg.2: great, thanks for asking. & Eg.2: is it a problem? & Eg.2: of course. would you like to have lunch with me? \\ \cline{2-4}  & Eg.3: next week is a good idea. & Eg.3: it is a day vacation for you here in the us. & Eg.3: i’ m afraid i can’t make it to the next few days. \\ \hline i have to check out today. i’d like my bill ready by 10. & Eg.1: that’s a long time. & Eg.1: that’s the problem, sir? & Eg.1: i’m sorry to hear that. are you looking for another one? \\ \cline{2-4}  & Eg.2: yes. & Eg.2: i was the only child in the room. & Eg.2: no problem. may i have it now? \\ \cline{2-4}  & Eg.3: i’m afraid you can’t do that. & Eg.3: you can get me a little time, and the red will take you get off. & Eg.3: i see, sir. what is the room number? \\ \hline the weather has been terrible this week. & Eg.1: i’m sure we can do it again. & Eg.1: i’ve got a few things i’ll get you there is a real one. & Eg.1: yes. i have been looking forward to better days. \\ \cline{2-4}  & Eg.2: oh, this is the correct. & Eg.2: what is the rest of the days. & Eg.2: i don’t think it is the best time for us to visit. \\ \cline{2-4}  & Eg.3: what kind of job do you like to see? & Eg.3: oh, no. & Eg.3: yes, we did. \\ \hline i wonder whether i could possibly borrow your new bicycle now. & Eg.1: that’s a great idea. what did you do? & Eg.1: a problem. what do you think of this one? & Eg.1: of course. let’s see. \\ \cline{2-4}  & Eg.2: i’d like to tell you a fees, but i’m not sure i can do that. & Eg.2: yes sure, thanks. you are very tough. & Eg.2: i sorry to hear that. anything else? do you have any other account? \\ \cline{2-4}  & Eg.3: oh, what is your problem? & Eg.3: what is the problem? what time? oh yes, it’s been a long time. & Eg.3: i would love to do that. there have been more 1 for you. \\ \hline \end{tabular}
\end{table}
Table A5: Examples of context-response pairs for different models. ‘Eg.i’ means the \(i\)-th sample.