SlowGuess/ACL_2020_OCR huggingface dataset:
Ines Chami $^{1*}$ , Adva Wolf $^{1}$ , Da-Cheng Juan $^{2}$ , Frederic Sala $^{1}$ , Sujith Ravi $^{3\dagger}$  and Christopher Ré $^{1}$
3Amazon Alexa
{chami,advaw,fredsala,chrismre}@cs.stanford.edu
dacheng@google.com
Knowledge graph (KG) embeddings learn low-dimensional representations of entities and relations to predict missing facts. KGs often exhibit hierarchical and logical patterns which must be preserved in the embedding space. For hierarchical data, hyperbolic embedding methods have shown promise for high-fidelity and parsimonious representations. However, existing hyperbolic embedding methods do not account for the rich logical patterns in KGs. In this work, we introduce a class of hyperbolic KG embedding models that simultaneously capture hierarchical and logical patterns. Our approach combines hyperbolic reflections and rotations with attention to model complex relational patterns. Experimental results on standard KG benchmarks show that our method improves over previous Euclidean- and hyperbolic-based efforts by up to  $6.1\%$  in mean reciprocal rank (MRR) in low dimensions. Furthermore, we observe that different geometric transformations capture different types of relations while attention-based transformations generalize to multiple relations. In high dimensions, our approach yields new state-of-the-art MRRs of  $49.6\%$  on WN18RR and  $57.7\%$  on YAGO3-10.
Knowledge graphs (KGs), consisting of (head entity, relationship, tail entity) triples, are popular data structures for representing factual knowledge to be queried and used in downstream applications such as word sense disambiguation, question answering, and information extraction. Real-world KGs such as Yago (Suchanek et al., 2007) or Wordnet (Miller, 1995) are usually incomplete, so a common approach to predicting missing links in KGs is via embedding into vector spaces. Embedding
Figure 1: A toy example showing how KGs can simultaneously exhibit hierarchies and logical patterns.
methods learn representations of entities and relationships that preserve the information found in the graph, and have achieved promising results for many tasks.
Relations found in KGs have differing properties: for example, (Michelle Obama, married to, Barack Obama) is symmetric, whereas hypernym relations like (cat, specific type of, feline), are not (Figure 1). These distinctions present a challenge to embedding methods: preserving each type of behavior requires producing a different geometric pattern in the embedding space. One popular approach is to use extremely high-dimensional embeddings, which offer more flexibility for such patterns. However, given the large number of entities found in KGs, doing so yields very high memory costs.
For hierarchical data, hyperbolic geometry offers an exciting approach to learn low-dimensional embeddings while preserving latent hierarchies. Hyperbolic space can embed trees with arbitrarily low distortion in just two dimensions. Recent research has proposed embedding hierarchical graphs into these spaces instead of conventional Euclidean space (Nickel and Kiela, 2017; Sala et al., 2018). However, these works focus on embedding simpler graphs (e.g., weighted trees) and cannot express the diverse and complex relationships in KGs.
We propose a new hyperbolic embedding ap
*Work partially done during an internship at Google.
$\dagger$  Work done while at Google AI.
6901
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6901-6914
proach that captures such patterns to achieve the best of both worlds. Our proposed approach produces the parsimonious representations offered by hyperbolic space, especially suitable for hierarchical relations, and is effective even with low-dimensional embeddings. It also uses rich transformations to encode logical patterns in KGs, previously only defined in Euclidean space. To accomplish this, we (1) train hyperbolic embeddings with relation-specific curvatures to preserve multiple hierarchies in KGs; (2) parameterize hyperbolic isometries (distance-preserving operations) and leverage their geometric properties to capture relations' logical patterns, such as symmetry or anti-symmetry; (3) and use a notion of hyperbolic attention to combine geometric operators and capture multiple logical patterns.
We evaluate the performance of our approach, ATTH, on the KG link prediction task using the standard WN18RR (Dettmers et al., 2018; Bordes et al., 2013), FB15k-237 (Toutanova and Chen, 2015) and YAGO3-10 (Mahdisoltani et al., 2013) benchmarks. (1) In low (32) dimensions, we improve over Euclidean-based models by up to  $6.1\%$  in the mean reciprocal rank (MRR) metric. In particular, we find that hierarchical relationships, such as WordNet's hypernym and member meronym, significantly benefit from hyperbolic space; we observe a  $16\%$  to  $24\%$  relative improvement versus Euclidean baselines. (2) We find that geometric properties of hyperbolic isometries directly map to logical properties of relationships. We study symmetric and anti-symmetric patterns and find that reflections capture symmetric relations while rotations capture anti-symmetry. (3) We show that attention based-transformations have the ability to generalize to multiple logical patterns. For instance, we observe that ATTH recovers reflections for symmetric relations and rotations for the anti-symmetric ones.
In high (500) dimensions, we find that both hyperbolic and Euclidean embeddings achieve similar performance, and our approach achieves new state-of-the-art results (SotA), obtaining  $49.6\%$  MRR on WN18RR and  $57.7\%$  YAGO3-10. Our experiments show that trainable curvature is critical to generalize hyperbolic embedding methods to high-dimensions. Finally, we visualize embeddings learned in hyperbolic spaces and show that hyperbolic geometry effectively preserves hierarchies in KGs.
Previous methods for KG embeddings also rely on geometric properties. Improvements have been obtained by exploiting either more sophisticated spaces (e.g., going from Euclidean to complex or hyperbolic space) or more sophisticated operations (e.g., from translations to isometries, or to learning graph neural networks). In contrast, our approach takes a step forward in both directions.
Euclidean embeddings In the past decade, there has been a rich literature on Euclidean embeddings for KG representation learning. These include translation approaches (Bordes et al., 2013; Ji et al., 2015; Wang et al., 2014; Lin et al., 2015) or tensor factorization methods such as RESCAL (Nickel et al., 2011) or DistMult (Yang et al., 2015). While these methods are fairly simple and have few parameters, they fail to encode important logical properties (e.g., translations can't encode symmetry).
Complex embeddings Recently, there has been interest in learning embeddings in complex space, as in the ComplEx (Trouillon et al., 2016) and RotatE (Sun et al., 2019) models. RotatE learns rotations in complex space, which are very effective in capturing logical properties such as symmetry, anti-symmetry, composition or inversion. The recent QuatE model (Zhang et al., 2019) learns KG embeddings using quaternions. However, a downside is that these embeddings require very high-dimensional spaces, leading to high memory costs.
Deep neural networks Another family of methods uses neural networks to produce KG embeddings. For instance, R-GCN (Schlichtkrull et al., 2018) extends graph neural networks to the multi-relational setting by adding a relation-specific aggregation step. ConvE and ConvKB (Dettmers et al., 2018; Nguyen et al., 2018) leverage the expressiveness of convolutional neural networks to learn entity embeddings and relation embeddings. More recently, the KBGAT (Nathani et al., 2019) and A2N (Bansal et al., 2019) models use graph attention networks for knowledge graph embeddings. A downside of these methods is that they are computationally expensive as they usually require pre-trained KG embeddings as input for the neural network.
Hyperbolic embeddings To the best of our knowledge, MuRP (Balažević et al., 2019) is the
6902
only method that learns KG embeddings in hyperbolic space in order to target hierarchical data. MuRP minimizes hyperbolic distances between a re-scaled version of the head entity embedding and a translation of the tail entity embedding. It achieves promising results using hyperbolic embeddings with fewer dimensions than its Euclidean analogues. However, MuRP is a translation model and fails to encode some logical properties of relationships. Furthermore, embeddings are learned in a hyperbolic space with fixed curvature, potentially leading to insufficient precision, and training relies on cumbersome Riemannian optimization. Instead, our proposed method leverages expressive hyperbolic isometries to simultaneously capture logical patterns and hierarchies. Furthermore, embeddings are learned using tangent space (i.e., Euclidean) optimization methods and trainable hyperbolic curvatures per relationship, avoiding precision errors that might arise when using a fixed curvature, and providing flexibility to encode multiple hierarchies.
3 Problem Formulation and Background
We describe the KG embedding problem setting and give some necessary background on hyperbolic geometry.
3.1 Knowledge graph embeddings
In the KG embedding problem, we are given a set of triples  $(h,r,t)\in \mathcal{E}\subseteq \mathcal{V}\times \mathcal{R}\times \mathcal{V}$ , where  $\mathcal{V}$  and  $\mathcal{R}$  are entity and relationship sets, respectively. The goal is to map entities  $v\in \mathcal{V}$  to embeddings  $\mathbf{e}_v\in$ $\mathcal{U}^{d_{\mathcal{V}}}$  and relationships  $r\in \mathcal{R}$  to embeddings  $\mathbf{r}_r\in$ $\mathcal{U}^{d_{\mathcal{R}}}$ , for some choice of space  $\mathcal{U}$  (traditionally  $\mathbb{R}$ ), such that the KG structure is preserved.
Concretely, the data is split into  $\mathcal{E}_{Train}$  and  $\mathcal{E}_{Test}$  triples. Embeddings are learned by optimizing a scoring function  $s: \mathcal{V} \times \mathcal{R} \times \mathcal{V} \to \mathbb{R}$ , which measures triples' likelihoods.  $s(\cdot, \cdot, \cdot)$  is trained using triples in  $\mathcal{E}_{Train}$  and the learned embeddings are then used to predict scores for triples in  $\mathcal{E}_{Test}$ . The goal is to learn embeddings such that the scores of triples in  $\mathcal{E}_{Test}$  are high compared to triples that are not present in  $\mathcal{E}$ .
3.2 Hyperbolic geometry
We briefly review key notions from hyperbolic geometry; a more in-depth treatment is available in standard texts (Robbin and Salamon). Hyperbolic geometry is a non-Euclidean geometry with constant negative curvature. In this work, we use the  $d$ -
Figure 2: An illustration of the exponential map  $\exp_{\mathbf{x}}(\mathbf{v})$ , which maps the tangent space  $\mathcal{T}_{\mathbf{x}}M$  at the point  $\mathbf{x}$  to the hyperbolic manifold  $M$ .
dimensional Poincaré ball model with negative curvature  $-c$  ( $c > 0$ ):  $\mathbb{B}^{d,c} = \{\mathbf{x} \in \mathbb{R}^d : ||\mathbf{x}||^2 < \frac{1}{c}\}$ , where  $||\cdot||$  denotes the  $L_2$  norm. For each point  $\mathbf{x} \in \mathbb{B}^{d,c}$ , the tangent space  $\mathcal{T}_{\mathbf{x}}^c$  is a  $d$ -dimensional vector space containing all possible directions of paths in  $\mathbb{B}^{d,c}$  leaving from  $\mathbf{x}$ .
The tangent space  $\mathcal{T}_{\mathbf{x}}^{c}$  maps to  $\mathbb{B}^{d,c}$  via the exponential map (Figure 2), and conversely, the logarithmic map maps  $\mathbb{B}^{d,c}$  to  $\mathcal{T}_{\mathbf{x}}^{c}$ . In particular, we have closed-form expressions for these maps at the origin:
$$
\exp_ {\mathbf {0}} ^ {c} (\mathbf {v}) = \tanh  (\sqrt {c} | | \mathbf {v} | |) \frac {\mathbf {v}}{\sqrt {c} | | \mathbf {v} | |}, \tag {1}
$$
$$
\log_ {0} ^ {c} (\mathbf {y}) = \operatorname {a r c t a n h} (\sqrt {c} | | \mathbf {y} | |) \frac {\mathbf {y}}{\sqrt {c} | | \mathbf {y} | |}. \tag {2}
$$
Vector addition is not well-defined in the hyperbolic space (adding two points in the Poincaré ball might result in a point outside the ball). Instead, Möbius addition  $\oplus^c$  (Ganea et al., 2018) provides an analogue to Euclidean addition for hyperbolic space. We give its closed-form expression in Appendix A.1. Finally, the hyperbolic distance on  $\mathbb{B}^{d,c}$  has the explicit formula:
$$
d ^ {c} (\mathbf {x}, \mathbf {y}) = \frac {2}{\sqrt {c}} \operatorname {a r c t a n h} \left(\sqrt {c} | | - \mathbf {x} \oplus^ {c} \mathbf {y} |\right). \tag {3}
$$
The goal of this work is to learn parsimonious hyperbolic embeddings that can encode complex logical patterns such as symmetry, anti-symmetry, or inversion while preserving latent hierarchies. Our model, ATTH, (1) learns KG embeddings in hyperbolic space in order to preserve hierarchies (Section 4.1), (2) uses a class of hyperbolic isometries parameterized by compositions of Givens transformations to encode logical patterns (Section 4.2), (3) combines these isometries with hyperbolic attention (Section 4.3). We describe the full model in Section 4.4.
6903
4.1 Hierarchies in hyperbolic space
As described, hyperbolic embeddings enable us to represent hierarchies even when we limit ourselves to low-dimensional spaces. In fact, two-dimensional hyperbolic space can represent any tree with arbitrarily small error (Sala et al., 2018).
It is important to set the curvature of the hyperbolic space correctly. This parameter provides flexibility to the model, as it determines whether to embed relations into a more curved hyperbolic space (more "tree-like"), or into a flatter, more "Euclidean-like" geometry. For each relation, we learn a relation-specific absolute curvature  $c_{r}$ , enabling us to represent a variety of hierarchies. As we show in Section 5.5, fixing, rather than learning curvatures can lead to significant performance degradation.
4.2 Hyperbolic isometries
Relationships often satisfy particular properties, such as symmetry: e.g., if (Michelle Obama, married to, Barack Obama) holds, then (Barack Obama, married to, Michelle Obama) does as well. These rules are not universal. For instance, (Barack Obama, born in, Hawaii) is not symmetric.
Creating and curating a set of deterministic rules is infeasible for large-scale KGs; instead, embedding methods represent relations as parameterized geometric operations that directly map to logical properties. We use two such operations in hyperbolic space: rotations, which effectively capture compositions or anti-symmetric patterns, and reflections, which naturally encode symmetric patterns.
Rotations Rotations have been successfully used to encode compositions in complex space with the RotatE model (Sun et al., 2019); we lift these to hyperbolic space. Compared to translations or tensor factorization approaches which can only infer some logical patterns, rotations can simultaneously model and infer inversion, composition, symmetric or anti-symmetric patterns.
Reflections These isometries reflect along a fixed subspace. While some rotations can represent symmetric relations (more specifically  $\pi$ -rotations), any reflection can naturally represent symmetric relations, since their second power is the identity. They provide a way to fill-in missing entries in symmetric triples, by applying the same operation to both the tail and the head entity. For instance, by modelling sibling of with a reflection, we can
(a) Rotations
(b) Reflections
Figure 3: Euclidean (left) and hyperbolic (right) isometries. In hyperbolic space, the distance between start and end points after applying rotations or reflections is much larger than the Euclidean distance; it approaches the sum of the distances between the points and the origin, giving more "room" to separate embeddings. This is similar to trees, where the shortest path between two points goes through their nearest common ancestor.
directly infer (Bob, sibling of, Alice) from (Alice, sibling of, Bob) and vice versa.
Parameterization Unlike RotatE which models rotations via unitary complex numbers, we learn relationship-specific isometries using Givens transformations,  $2 \times 2$  matrices commonly used in numerical linear algebra. Let  $\Theta_r := (\theta_{r,i})_{i \in \{1, \dots, \frac{d}{2}\}}$  and  $\Phi_r := (\phi_{r,i})_{i \in \{1, \dots, \frac{d}{2}\}}$  denote relation-specific parameters. Using an even number of dimensions  $d$ , our model parameterizes rotations and reflections with block-diagonal matrices of the form:
$$
\operatorname {R o t} \left(\Theta_ {r}\right) = \operatorname {d i a g} \left(G ^ {+} \left(\theta_ {r, 1}\right), \dots , G ^ {+} \left(\theta_ {r, \frac {d}{2}}\right)\right), \tag {4}
$$
$$
\operatorname {R e f} \left(\Phi_ {r}\right) = \operatorname {d i a g} \left(G ^ {-} \left(\phi_ {r, 1}\right), \dots , G ^ {-} \left(\phi_ {r, \frac {n}{2}}\right)\right), \tag {5}
$$
$$
\text {w h e r e} G ^ {\pm} (\theta) := \left[ \begin{array}{l l} \cos (\theta) & \mp \sin (\theta) \\ \sin (\theta) & \pm \cos (\theta) \end{array} \right]. \tag {6}
$$
Rotations and reflections of this form are hyperbolic isometries (distance-preserving). We can therefore directly apply them to hyperbolic embeddings while preserving the underlying geometry. Additionally, these transformations are computationally efficient and can be computed in linear time in the dimension. We illustrate two-dimensional isometries in both Euclidean and hyperbolic spaces in Figure 3.
4.3 Hyperbolic attention
Of our two classes of hyperbolic isometries, one or the other may better represent a particular relation. To handle this, we use an attention mechanism to learn the right isometry. Thus we can represent symmetric, anti-symmetric or mixed-behaviour relations (i.e. neither symmetric nor anti-symmetric) as a combination of rotations and reflections.
Let  $\mathbf{x}^H$  and  $\mathbf{y}^H$  be hyperbolic points (e.g., reflection and rotation embeddings), and  $\mathbf{a}$  be an
6904
attention vector. Our approach maps hyperbolic representations to tangent space representations,  $\mathbf{x}^{E} = \log_{0}^{c}(\mathbf{x}^{H})$  and  $\mathbf{y}^{E} = \log_{0}^{c}(\mathbf{y}^{H})$ , and computes attention scores:
$$
(\alpha_ {\mathbf {x}}, \alpha_ {\mathbf {y}}) = \mathrm {S o f t m a x} (\mathbf {a} ^ {T} \mathbf {x} ^ {E}, \mathbf {a} ^ {T} \mathbf {y} ^ {E}).
$$
We then compute a weighted average using the recently proposed tangent space average (Chami et al., 2019; Liu et al., 2019):
$$
\operatorname {A t t} \left(\mathbf {x} ^ {H}, \mathbf {y} ^ {H}; \mathbf {a}\right) := \exp_ {\mathbf {0}} ^ {c} \left(\alpha_ {\mathbf {x}} \mathbf {x} ^ {E} + \alpha_ {\mathbf {y}} \mathbf {y} ^ {E}\right). \tag {7}
$$
4.4 The ATTH model
We have all of the building blocks for  $\mathrm{ATTH}$ , and can now describe the model architecture. Let  $(\mathbf{e}_v^H)_{v\in \mathcal{V}}$  and  $(\mathbf{r}_r^H)_{r\in \mathcal{R}}$  denote entity and relationship hyperbolic embeddings respectively. For a triple  $(h,r,t)\in \mathcal{V}\times \mathcal{R}\times \mathcal{V}$ ,  $\mathrm{ATTH}$  applies relation-specific rotations (Equation 4) and reflections (Equation 5) to the head embedding:
$$
\mathbf {q} _ {\mathrm {R o t}} ^ {H} = \mathrm {R o t} (\boldsymbol {\Theta} _ {r}) \mathbf {e} _ {h} ^ {H}, \mathbf {q} _ {\mathrm {r e f}} ^ {H} = \mathrm {R e f} (\boldsymbol {\Phi} _ {r}) \mathbf {e} _ {h} ^ {H}. (8)
$$
ATTH then combines the two representations using hyperbolic attention (Equation 7) and applies a hyperbolic translation:
$$
\mathrm {Q} (h, r) = \operatorname {A t t} \left(\mathbf {q} _ {\text {R o t}} ^ {H}, \mathbf {q} _ {\text {R e f}} ^ {H}; \mathbf {a} _ {r}\right) \oplus^ {c _ {r}} \mathbf {r} _ {r} ^ {H}. \tag {9}
$$
Intuitively, rotations and reflections encode logical patterns while translations capture tree-like structures by moving between levels of the hierarchy. Finally, query embeddings are compared to target tail embeddings via the hyperbolic distance (Equation 3). The resulting scoring function is:
$$
s (h, r, t) = - d ^ {c _ {r}} \left(Q (h, r), \mathbf {e} _ {t} ^ {H}\right) ^ {2} + b _ {h} + b _ {t}, \tag {10}
$$
where  $(b_v)_{v\in \mathcal{V}}$  are entity biases which act as margins in the scoring function (Tifrea et al., 2019; Balažević et al., 2019).
The model parameters are then  $\{(\Theta_r,\Phi_r,\mathbf{r}_r^H,\mathbf{a}_r,c_r)_{r\in \mathcal{R}},(\mathbf{e}_v^H,b_v)_{v\in \mathcal{V}}\}$ . Note that the total number of parameters in ATTH is  $\mathcal{O}(|\mathcal{V}|d)$ , similar to traditional models that do not use attention or geometric operations. The extra cost is proportional to the number of relations, which is usually much smaller than the number of entities.
Dataset	#entities	#relations	#triples	ξG
WN18RR	41k	11	93k	-2.54
FB15k-237	15k	237	310k	-0.65
YAGO3-10	123k	37	1M	-0.54
Table 1: Datasets statistics. The lower the metric  $\xi_G$  is, the more tree-like the knowledge graph is.
In low dimensions, we hypothesize (1) that hyperbolic embedding methods obtain better representations and allow for improved downstream performance for hierarchical data (Section 5.2). (2) We expect the performance of relation-specific geometric operations to vary based on the relation's logical patterns (Section 5.3). (3) In cases where the relations are neither purely symmetric nor antisymmetric, we anticipate that hyperbolic attention outperforms the models which are based on solely reflections or rotations (Section 5.4). Finally, in high dimensions, we expect hyperbolic models with trainable curvature to learn the best geometry, and perform similarly to their Euclidean analogues (Section 5.5).
Datasets We evaluate our approach on the link prediction task using three standard competition benchmarks, namely WN18RR (Bordes et al., 2013; Dettmers et al., 2018), FB15k-237 (Bordes et al., 2013; Toutanova and Chen, 2015) and YAGO3-10 (Mahdisoltani et al., 2013). WN18RR is a subset of WordNet containing 11 lexical relationships between 40,943 word senses, and has a natural hierarchical structure, e.g., (car, hypernym of, sedan). FB15k-237 is a subset of Freebase, a collaborative KB of general world knowledge. FB15k-237 has 14,541 entities and 237 relationships, some of which are non-hierarchical, such as born-in or nationality, while others have natural hierarchies, such as part-of (for organizations). YAGO3-10 is a subset of YAGO3, containing 123,182 entities and 37 relations, where most relations provide descriptions of people. Some relationships have a hierarchical structure such as playsFor or actedIn, while others induce logical patterns, like isMarriedTo.
For each KG, we follow the standard data augmentation protocol by adding inverse relations (Lacroix et al., 2018) to the datasets. Additionally, we estimate the global graph curvature  $\xi_{G}$  (Gu et al., 2019) (see Appendix A.2 for more details),
6905
U	Model	WN18RR	FB15k-237	YAGO3-10
MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10
Rd	RotatE	.387	.330	.417	.491	.290	.208	.316	.458	-	-	-	-
MuRE	.458	.421	.471	.525	.313	.226	.340	.489	.283	.187	.317	.478
Cd	ComplEx-N3	.420	.390	.420	.460	.294	.211	.322	.463	.336	.259	.367	.484
Bd,1	MuRP	.465	.420	.484	.544	.323	.235	.353	.501	.230	.150	.247	.392
Rd	REFE	.455	.419	.470	.521	.302	.216	.330	.474	.370	.289	.403	.527
ROTE	.463	.426	.477	.529	.307	.220	.337	.482	.381	.295	.417	.548
ATTE	.456	.419	.471	.526	.311	.223	.339	.488	.374	.290	.410	.537
Bd,c	REFH	.447	.408	.464	.518	.312	.224	.342	.489	.381	.302	.415	.530
ROTH	.472	.428	.490	.553	.314	.223	.346	.497	.393	.307	.435	559
ATTH	.466	.419	.484	.551	.324	.236	.354	.501	.397	.310	.437	.566
Table 2: Link prediction results for low-dimensional embeddings ( $d = 32$ ) in the filtered setting. Best score in **bold** and best published **underlined**. Hyperbolic isometries significantly outperform Euclidean baselines on WN18RR and YAGO3-10, both of which exhibit hierarchical structures.
Figure 4: WN18RR MRR dimension for  $d \in \{10, 16, 20, 32, 50, 200, 500\}$ . Average and standard deviation computed over 10 runs for Roth.
which is a distance-based measure of how close a given graph is to being a tree. We summarize the datasets' statistics in Table 1.