SlowGuess/ACL_2020_OCR huggingface dataset:
Baselines We compare our method to SotA models, including MurP (Balazevic et al., 2019), MurE (which is the Euclidean analogue or MurP), RotatE (Sun et al., 2019), ComplEx-N3 (Lacroix et al., 2018) and TuckER (Balazevic et al., 2019). Baseline numbers in high dimensions (Table 5) are taken from the original papers, while baseline numbers in the low-dimensional setting (Table 2) are computed using open-source implementations of each model. In particular, we run hyper-parameter searches over the same parameters as the ones in the original papers to compute baseline numbers in the low-dimensional setting.
Ablations To analyze the benefits of hyperbolic geometry, we evaluate the performance of ATTE, which is equivalent to ATTH with curvatures set to zero. Additionally, to better understand the role of attention, we report scores for variants of ATTE/H using only rotations (ROTE/H) or reflections (REFE/H).
Evaluation metrics At test time, we use the scoring function in Equation 10 to rank the correct tail or head entity against all possible entities, and use in use inverse relations for head prediction (Lacroix et al., 2018). Similar to previous work, we compute two ranking-based metrics: (1) mean reciprocal rank (MRR), which measures the mean of inverse ranks assigned to correct entities, and (2) hits at  $K$  ( $\mathrm{H}@\mathcal{K}, K \in \{1,3,10\}$ ), which measures the proportion of correct triples among the top  $K$  predicted triples. We follow the standard evaluation protocol in the filtered setting (Bordes et al., 2013): all true triples in the KG are filtered out during evaluation, since predicting a low rank for these triples should not be penalized.
Training procedure and implementation We train ATH by minimizing the full cross-entropy loss with uniform negative sampling, where negative examples for a triple  $(h,r,t)$  are sampled uniformly from all possible triples obtained by perturbing the tail entity:
$$
\begin{array}{l} \mathcal {L} = \sum_ {t ^ {\prime} \sim \mathcal {U} (\mathcal {V})} \log \left(1 + \exp \left(y _ {t ^ {\prime}} s \left(h, r, t ^ {\prime}\right)\right)\right), \tag {11} \\ \text {w h e r e} y _ {t ^ {\prime}} = \left\{ \begin{array}{l l} - 1 & \text {i f} t ^ {\prime} = t \\ 1 & \text {o t h e r w i s e .} \end{array} \right. \\ \end{array}
$$
Since optimization in hyperbolic space is practically challenging, we instead define all parameters in the tangent space at the origin, optimize embeddings using standard Euclidean techniques, and use the exponential map to recover the hyperbolic parameters (Chami et al., 2019). We provide more details on tangent space optimization in Appendix A.4. We conducted a grid search to select the learning rate, optimizer, negative sample size, and batch size, using the validation set to select the best hy
6906
Relation	KhsG	ξG	ROTE	ROTH	Improvement
member meronym	1.00	-2.90	.320	.399	24.7%
hypernym	1.00	-2.46	.237	.276	16.5%
has part	1.00	-1.43	.291	.346	18.9%
instance hypernym	1.00	-0.82	.488	.520	6.56%
member of domain region	1.00	-0.78	.385	.365	-5.19%
member of domain usage	1.00	-0.74	.458	.438	-4.37%
synset domain topic of	0.99	-0.69	.425	.447	5.17%
also see	0.36	-2.09	.634	.705	11.2%
derivationally related form	0.07	-3.84	.960	.968	0.83%
similar to	0.07	-1.00	1.00	1.00	0.00%
verb group	0.07	-0.50	.974	.974	0.00%
perparameters. Our best model hyperparameters are detailed in Appendix A.3. We conducted all our experiments on NVIDIA Tesla P100 GPUs and make our implementation publicly available*.
5.2 Results in low dimensions
We first evaluate our approach in the low-dimensional setting for  $d = 32$ , which is approximately one order of magnitude smaller than SotA Euclidean methods. Table 2 compares the performance of ATTH to that of other baselines, including the recent hyperbolic (but not rotation-based) MuRP model. In low dimensions, hyperbolic embeddings offer much better representations for hierarchical relations, confirming our hypothesis. ATTH improves over previous Euclidean and hyperbolic methods by  $0.7\%$  and  $6.1\%$  points in MRR on WN18RR and YAGO3-10 respectively. Both datasets have multiple hierarchical relationships, suggesting that the hierarchical structure imposed by hyperbolic geometry leads to better embeddings. On FB15k-237, ATTH and MurP achieve similar performance, both improving over Euclidean baselines. We conjecture that translations are sufficient to model relational patterns in FB15k-237.
To understand the role of dimensionality, we also conduct experiments on WN18RR against SotA methods under varied low-dimensional settings (Figure 4). We include error bars for our method with average MRR and standard deviation computed over 10 runs. Our approach consistently outperforms all baselines, suggesting that hyperbolic embeddings still attain high-accuracy across a broad range of dimensions.
Additionally, we measure performance per relation on WN18RR in Table 3 to understand the benefits of hyperbolic geometric on hierarchical relations. We report the Krackhardt hierarchy score
Table 3: Comparison of H@10 for WN18RR relations. Higher  $\mathrm{Khs}_G$  and lower  $\xi_{G}$  means more hierarchical.
Relation	Anti-symmetric	Symmetric	ROTH	REFH	ATTH
hasNeighbor	✗	✓	.750	1.00	1.00
isMarriedTo	✗	✓	.941	.941	1.00
actedIn	✓	✗	.145	.110	.150
hasMusicalRole	✓	✗	.431	.375	.458
directed	✓	✗	.500	.450	.567
graduatedFrom	✓	✗	.262	.167	.274
playsFor	✓	✗	.671	.642	.664
wroteMusicFor	✓	✗	.281	.188	.266
hasCapital	✓	✗	.692	.731	.731
dealsWith	✗	✗	.286	.286	.429
isLocatedIn	✗	✗	.404	.399	.420
Table 4: Comparison of geometric transformations on a subset of YAGO3-10 relations.
$(\mathrm{Khs}_G)$  (Balažević et al., 2019) and estimated curvature per relation (see Appendix A.2 for more details). We consider a relation to be hierarchical when its corresponding graph is close to tree-like (low curvature, high  $\mathrm{Khs}_G$ ). We observe that hyperbolic embeddings offer much better performance on hierarchical relations such as hypernym or has part, while Euclidean and hyperbolic embeddings have similar performance on non-hierarchical relations such as verb group. We also plot the learned curvature per relation versus the embedding dimension in Figure 5b. We note that the learned curvature in low dimensions directly correlates with the estimated graph curvature  $\xi_G$  in Table 3, suggesting that the model with learned curvatures learns more "curved" embedding spaces for tree-like relations.
Finally, we observe that MurP achieves lower performance than MurE on YAGO3-10, while ATTH improves over ATTE by  $2.3\%$  in MRR. This suggests that trainable curvature is critical to learn embeddings with the right amount of curvature, while fixed curvature might degrade performance. We elaborate further on this point in Section 5.5.
5.3 Hyperbolic rotations and reflections
In our experiments, we find that rotations work well on WN18RR, which contains multiple hierarchical and anti-symmetric relations, while reflections work better for YAGO3-10 (Table 5). To better understand the mechanisms behind these observations, we analyze two specific patterns: relation symmetry and anti-symmetry. We report performance per-relation on a subset of YAGO3-10 relations in Table 4. We categorize relations into symmetric, anti-symmetric, or neither symmetric nor anti-symmetric categories using data statistics. More concretely, we consider a relation to satisfy a logical pattern when the logical condition is satisfied by most of the triplets (e.g., a relation  $r$  is symmetric if for most KG triples  $(h,r,t)$ ,  $(t,r,h)$  is also in the KG). We observe that reflections encode
*Code available at https://github.com/tensorflow/neural-structured-learning/tree/master/research/kg_hyp_emb
6907
(a) MRR for fixed and trainable curvatures on WN18RR.
(b) Curvatures learned by with ROTH on WN18RR.
Figure 5: (a): Roth offers improved performance in low dimensions; in high dimensions, fixed curvature degrades performance, while trainable curvature approximately recovers Euclidean space. (b): As the dimension increases, the learned curvature of hierarchical relationships tends to zero.
symmetric relations particularly well, while rotations are well suited for anti-symmetric relations. This confirms our intuition—and the motivation for our approach—that particular geometric properties capture different kinds of logical properties.
5.4 Attention-based transformations
One advantage of using relation-specific transformations is that each relation can learn the right geometric operators based on the logical properties it has to satisfy. In particular, we observe that in both low- and high-dimensional settings, attention-based models can recover the performance of the best transformation on all datasets (Tables 2 and 5). Additionally, per-relationship results on YAGO3-10 in Table 4 suggest that ATTH indeed recovers the best geometric operation.
Furthermore, for relations that are neither symmetric nor anti-symmetric, we find that ATTH can outperform rotations and reflections, suggesting that combining multiple operators with attention can learn more expressive operators to model mixed logical patterns. In other words, attention-based transformations alleviate the need to conduct experiments with multiple geometric transformations by simply allowing the model to choose which one is best for a given relation.
5.5 Results in high dimensions
In high dimensions (Table 5), we compare against a variety of other models and achieve new SotA results on WN18RR and YAGO3-10, and third-best results on FB15k-237. As we expected, when the embedding dimension is large, Euclidean and hyperbolic embedding methods perform similarly across all datasets. We explain this behavior by noting that when the dimension is sufficiently large,
both Euclidean and hyperbolic spaces have enough capacity to represent complex hierarchies in KGs. This is further supported by Figure 5b, which shows the learned absolute curvature versus the dimension. We observe that curvatures are close to zero in high dimensions, confirming our expectation that Roth with trainable curvatures learns a roughly Euclidean geometry in this setting.
In contrast, fixed curvature degrades performance in high dimensions (Figure 5a), confirming the importance of trainable curvatures and its impact on precision and capacity (previously studied by (Sala et al., 2018)). Additionally, we show the embeddings' norms distribution in the Appendix (Figure 7). Fixed curvature results in embeddings being clustered near the boundary of the ball while trainable curvatures adjusts the embedding space to better distribute points throughout the ball. Precision issues that might arise with fixed curvature could also explain MurP's low performance in high dimensions. Trainable curvatures allow ROTH to perform as well or better than previous methods in both low and high dimensions.
5.6 Visualizations
In Figure 6, we visualize the embeddings learned by ROTE versus Roth for a sub-tree of the organism entity in WN18RR. To better visualize the hierarchy, we apply  $k$  inverse rotations for all nodes at level  $k$  in the tree.
By contrast to ROTE, ROTH preserves the tree structure in the embedding space. Furthermore, we note that ROTE cannot simultaneously preserve the tree structure and make non- neighboring nodes far from each other. For instance, virus should be far from male, but preserving the tree structure (by going one level down in the tree) while making
6908
U	Model	WN18RR	FB15k-237	YAGO3-10
MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10
Rd	DistMult	.430	.390	.440	.490	.241	.155	.263	.419	.340	.240	.380	.540
ConvE	.430	.400	.440	.520	.325	.237	.356	.501	.440	.350	.490	.620
TuckER	.470	.443	.482	.526	.358	.266	.394	.544	-	-	-	-
MurE	.475	.436	.487	.554	.336	.245	.370	.521	.532	.444	.584	.694
Cd	ComplEx-N3	.480	.435	.495	.572	.357	.264	.392	.547	.569	.498	.609	.701
RotatE	.476	.428	.492	.571	.338	.241	.375	.533	.495	.402	.550	.670
Hd	Quaternion	.488	.438	.508	.582	.348	.248	.382	.550	-	-	-	-
Bd,1	MurP	.481	.440	.495	.566	.335	.243	.367	.518	.354	.249	.400	.567
Rd	REFE	.473	.430	.485	.561	.351	.256	.390	.541	.577	.503	.621	.712
ROTE	.494	.446	.512	.585	.346	.251	.381	.538	.574	.498	.621	.711
ATTE	.490	.443	.508	.581	.351	.255	.386	.543	.575	.500	.621	.709
Bd,c	REFH	.461	.404	.485	.568	.346	.252	.383	.536	.576	.502	.619	.711
ROTH	.496	.449	.514	.586	.344	.246	.380	.535	.570	.495	.612	.706
ATTH	.486	.443	.499	.573	.348	.252	.384	.540	.568	.493	.612	.702
Table 5: Link prediction results for high-dimensional embeddings (best for  $d \in \{200,400,500\}$ ) in the filtered setting. DistMult, ConvE and ComplEx results are taken from (Dettmers et al., 2018). Best score in bold and best published underlined. ATTE and ATTH have similar performance in the high-dimensional setting, performing competitively with or better than state-of-the-art methods on WN18RR, FB15k-237 and YAGO3-10.
(a) ROTE embeddings.
(b) Roth embeddings.
Figure 6: Visualizations of the embeddings learned by ROTE and ROTH on a sub-tree of WN18RR for the hypernym relation. In contrast to ROTE, ROTH preserves hierarchies by learning tree-like embeddings.
these two nodes far from each other is difficult in Euclidean space. In hyperbolic space, however, we observe that going one level down in the tree is achieved by translating embeddings towards the left. This pattern essentially illustrates the translation component in RotH, allowing the model to simultaneously preserve hierarchies while making non-neighbouring nodes far from each other.
We introduce ATTH, a hyperbolic KG embedding model that leverages the expressiveness of hyperbolic space and attention-based geometric transformations to learn improved KG representations in low-dimensions. ATTH learns embeddings with trainable hyperbolic curvatures, allowing it to learn the right geometry for each relationship and generalize across multiple embedding dimensions. ATTH achieves new SotA on WN18RR and YAGO3-10, real-world KGs which exhibit hierar
chical structures. Future directions for this work include exploring other tasks that might benefit from hyperbolic geometry, such as hypernym detection. The proposed attention-based transformations can also be extended to other geometric operations.
We thank Avner May for their helpful feedback and discussions. We gratefully acknowledge the support of DARPA under Nos. FA86501827865 (SDH) and FA86501827882 (ASED); NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); ONR under No. N000141712266 (Unifying Weak Supervision); the Moore Foundation, NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, the Okawa Foundation, American Family Insurance, Google Cloud, Swiss Re, the HAI-AWS Cloud Credits for Research program, TOTAL, and members of the Stanford DAWN project: Teradata, Facebook, Google, Ant Financial, NEC, VMWare, andInfosys. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of DARPA, NIH, ONR, or the U.S. Government.
6909
Ivana Balačević, Carl Allen, and Timothy Hospedales. 2019. Multi-relational poincaré graph embeddings. In Advances in Neural Information Processing Systems, pages 4465-4475.
Ivana Balazevic, Carl Allen, and Timothy Hospedales. 2019. Tucker: Tensor factorization for knowledge graph completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5188-5197.
Trapit Bansal, Da-Cheng Juan, Sujith Ravi, and Andrew McCallum. 2019. A2n: Attending to neighbors for knowledge graph inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4387-4392.
Silvere Bonnabel. 2013. Stochastic gradient descent on Riemannian manifolds. IEEE Transactions on Automatic Control, 58(9):2217-2229.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems, pages 2787-2795.
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. 2019. Hyperbolic graph convolutional neural networks. In Advances in Neural Information Processing Systems, pages 4869-4880.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2D knowledge graph embeddings. In Thirty-Second AAAI Conference on Artificial Intelligence.
John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121-2159.
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. 2018. Hyperbolic neural networks. In Advances in Neural Information Processing Systems.
Albert Gu, Fred Sala, Belize Gunel, and Christopher Ré. 2019. Learning mixed-curvature representations in product spaces. In International Conference on Learning Representations.
Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge graph embedding via dynamic mapping matrix. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 687-696.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference for Learning Representations.
David Krackhardt. 1994. Graph theoretical dimensions of informal organizations. In Computational organization theory, pages 107-130. Psychology Press.
Timothée Lacroix, Nicolas Usunier, and Guillaume Obozinski. 2018. Canonical tensor decomposition for knowledge base completion. International Conference on Machine Learning.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Twenty-ninth AAAI Conference on Artificial Intelligence.
Qi Liu, Maximilian Nickel, and Douwe Kiela. 2019. Hyperbolic graph neural networks. In Advances in Neural Information Processing Systems, pages 8228-8239.
Farzaneh Mahdisoltani, Joanna Biega, and Fabian M Suchanek. 2013. Yago3: A knowledge base from multilingual wikipediaias.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41.
Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. 2019. Learning attention-based embeddings for relation prediction in knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. 2018. A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 327-333.
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way model for collective learning on multi-relational data. In International Conference on Machine Learning, pages 809-816. Omnipress.
Maximillian Nickel and Douwe Kiela. 2017. Poincaré embeddings for learning hierarchical representations. In Advances in Neural Information Processing Systems, pages 6338-6347.
Joel W Robbin and Dietmar A Salamon. Introduction to differential geometry.
Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré. 2018. Representation tradeoffs for hyperbolic embeddings. In International Conference on Machine Learning, pages 4457-4466.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling.
6910
2018. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pages 593-607. Springer.
Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web, pages 697-706. ACM.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations.
Alexandru Tifrea, Gary Bécigneul, and Octavian-Eugen Ganea. 2019. Poincaré GloVe: Hyperbolic word embeddings. In International Conference on Learning Representations.
Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pages 57-66.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In International Conference on Machine Learning, pages 2071-2080.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Twenty-Eighth AAAI Conference on Artificial Intelligence.
Canran Xu and Ruijiang Li. 2019. Relation embedding with dihedral group in knowledge graph. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In International Conference on Learning Representations.
Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. 2019. Quaternion knowledge graph embeddings. In Advances in Neural Information Processing Systems, pages 2731-2741.
6911
Below, we provide additional details. We start by providing the formula for the hyperbolic analogue of addition that we use, along with additional hyperbolic geometry background. Next, we provide more information about the metrics that are used to determine how hierarchical a dataset is. Afterwards, we give additional experimental details, including the table of hyperparameters and further details on tangent space optimization. Lastly, we include an additional comparison against the Dihedral model (Xu and Li, 2019).
A.1 Möbius addition
The Möbius addition operation (Ganea et al., 2018) has the closed-form expression:
$$
\mathbf {x} \oplus^ {c} \mathbf {y} = \frac {\alpha_ {\mathbf {x y}} \mathbf {x} + \beta_ {\mathbf {x y}} \mathbf {y}}{1 + 2 c \mathbf {x} ^ {T} \mathbf {y} + c ^ {2} | | \mathbf {x} | | ^ {2} | | \mathbf {y} | | ^ {2}},
$$
where  $\alpha_{\mathbf{xy}} = 1 + 2c\mathbf{x}^T\mathbf{y} + c||\mathbf{y}||^2$
and  $\beta_{\mathbf{xy}} = 1 - c||\mathbf{x}||^2$
In contrast to Euclidean addition, it is neither commutative nor associative. However, it provides an analogue through the lens of parallel transport: given two points  $\mathbf{x},\mathbf{y}$  and a vector  $\mathbf{v}$  in  $\mathcal{T}_{\mathbf{x}}^{c}$ , there is a unique vector in  $\mathcal{T}_{\mathbf{y}}^{c}$  which creates the same angle as  $\mathbf{v}$  with the direction of the geodesic (shortest path) connecting  $\mathbf{x}$  to  $\mathbf{y}$ . This map is the parallel transport  $P_{\mathbf{x}\rightarrow \mathbf{y}}^{c}(\cdot)$ ; Euclidean parallel transport is the standard Euclidean addition. Analogously, the Möbius addition satisfies (Ganea et al., 2018):  $\mathbf{x}\oplus^{c}\mathbf{y} = \exp_{\mathbf{x}}^{c}(P_{\mathbf{0}\to \mathbf{x}}^{c}(\log_{\mathbf{0}}^{c}(\mathbf{y}))).$
A.2 Hierarchy estimates
We use two metrics to estimate how hierarchical a relation is: the curvature estimate  $\xi_G$  and the Krackhardt hierarchy score  $\mathrm{Khs}_G$ . While the curvature estimate captures global hierarchical behaviours (how much the graph is tree-like when zooming-out), the Krackhardt score captures a more local behaviour (how many small loops the graph has). See Figure 8 for examples.