Enhancing Hyperbolic Knowledge Graph Embeddings via Lorentz Transformations (2024)

Abstract: Knowledge Graph Embedding (KGE) is a powerful technique for predicting missing links in Knowledge Graphs (KGs) by learning the entities and relations. Hyperbolic space has emerged as a promising embedding space for KGs due to its ability to represent hierarchical data. Nevertheless, most existing hyperbolic KGE methods rely on tangent approximation and are not fully hyperbolic, resulting in distortions and inaccuracies. To overcome this limitation, we propose LorentzKG, a fully hyperbolic KGE method that represents entities as points in the Lorentz model and represents relations as the intrinsic transformation—the Lorentz transformations between entities. We demonstrate that the Lorentz transformation, which can be decomposed into Lorentz rotation/reflection and Lorentz boost, captures various types of relations including hierarchical structures. Experimental results show that our LorentzKG achieves state-of-the-art performance1.

heading: 1 Introduction
text: Knowledge graphs (KGs) (Dong et al., 2014), which comprise factual triples of the form (head entity, relation, tail entity), are essential in questionanswering (Huang et al., 2019), information extraction (Xiong et al., 2017), and recommendations (Wang et al., 2014; Chen et al., 2022a). Recent advancements related to KGs centered around knowledge graph embedding (KGE), which involves mapping entities and relations to some dedicated representation spaces, providing an effective tool for interpreting semantic meaning within KGs.
While many existing models utilize Euclidean space for modeling relation patterns in KGs (Bordes et al., 2013; Chen and Li, 2019; Yang et al., 2015; Sun et al., 2019), they face challenges in effectively modeling semantic hierarchies due to the limited expressive power of low-dimensional
1https://github.com/LorentzKG/LorentzKG
Euclidean space. Increasing the dimensions incurs high computational costs, especially given the large number of entities in KGs. Recent work has explored hyperbolic space, a non-Euclidean space with constant negative curvature, as a more effective and efficient embedding space for data with hierarchical structures (Nickel and Kiela, 2017; Chen and Li, 2020b; Sala et al., 2018; Wang et al., 2023; Zhao et al., 2024). Several hyperbolic KGE models have been proposed to represent KGs (Balazevic et al., 2019; Chami et al., 2020; Kolyvakis et al., 2020; Pan and Wang, 2021; Bai et al., 2021), demonstrating the advantages of hyperbolic space, particularly in the low-dimensional settings.
Most existing hyperbolic KGE models (Balazevic et al., 2019; Chami et al., 2020; Pan and Wang, 2021; Bai et al., 2021) rely on tangent space approximations to perform vector space operations, such as matrix-vector multiplication and vector addition. Tangent space is a vector space that approximates the manifold at a given base point. However, this approach is not fully hyperbolic since it involves lifting the data from hyperbolic space to tangent space for feature transformation and subsequently projecting it back to hyperbolic space. This projection introduces additional complexities and numerical instability, as it requires the calculation of a series of hyperbolic functions. Moreover, the tangent approximation of data that is located away from the base point is distorted and inaccurate, compromising the expressibility of hyperbolic space (Chen et al., 2022b; Fan et al., 2023).
To address above issues, it is essential to explore fully hyperbolic approaches that bypass the reliance on tangent space approximations and instead work directly within hyperbolic space. In this paper, we address this challenge by utilizing the Lorentz model (Ratcliffe, 2006) of hyperbolic space, and its isometry group, the positive Lorentz group (Gallier and Quaintance, 2012), to develop a fully hyperbolic KGE model, called LorentzKG.
4575
The property of the positive Lorentz group acting transitively on the Lorentz model enables the definition of matrix multiplication on the hyperbolic space as a linear operation—multiplying a group element of the Lorentz group, the Lorentz transformation, to a point on the Lorentz model. This linear approach avoids tangent approximation, and furthermore, benefits from the numerical stability exhibited by the Lorentz model (Nickel and Kiela, 2018; Fan et al., 2022), ensuring reliable and accurate calculations.
Our LorentzKG model follows a translationbased KGE methodology, where relation-specific transformations using Lorentz transformation are applied to the head and tail entities embedded in the Lorentz model. The plausibility of a given fact is measured by the hyperbolic distance between the transformed entities. The highlights of our LorentzKG are: (i) The transitive nature of Lorentz transformation on the Lorentz model ensures that the transformation is an intrinsic linear operation and guarantees that the transformed entity embedding remains a point in the Lorentz model; (ii) The nature of the Lorentz transformation, allowing for decomposition into various types of transformations, enables us to effectively capture diverse relational patterns observed in KGs, including symmetry/anti-symmetry, inversion, composition, and hierarchy (Sec 3.3); (iii) Different relational transformations on head and tail entities enable modeling of relation cardinalities (such as 1- to-1, 1-to-N, N-to-1, and N-to-N (Bordes et al., 2013; Sun et al., 2019)) (Sec 3.3); (iv) Experimental results demonstrate the superiority of our LorentzKG compared to several Euclidean and hyperbolic KGE models. In addition, we point out connections with prior translation-based Euclidean KGE models (Sec 3.4) and contrasts our LorentzKG with prevalent hyperbolic KGE models (Sec 3.2), underscoring our work’s novelty.
Notations and Problem Setup. A KG is defined as a collection of factual triples G = {(h, r, t)}, where head and tail entities h, t ∈ E and relation r ∈ R are taken from an entity set E and a relation set R, respectively. The primary relation patterns in KGs include symmetry/antisymmetry, inversion, composition and hierarchy. The relationships can be categorized based on the cardinalities (of their heads or tails) as 1-to-1, 1-to-N, N-to-1, and N-to-N (Bordes et al., 2013). In KGE models, the objective is to map entities h, t ∈ E to low-
dimensional embeddings h, t ∈ UdE and relations r ∈ R to their embeddings r ∈ UdR , where U is a chosen representation space, e.g, Euclidean space R, and dE and dR are the dimensionalities of entity and relation embeddings, respectively. The learning of embeddings in KGE involves optimizing a score function fr(h, t) : UdE × UdR × UdE → R, which evaluates the validity of each triple. The embeddings of entities and relations are trained by optimizing the score function, with the objective of maximizing the scores of valid triples.

heading: 2 Preliminaries
text: Hyperbolic space is a Riemannian manifold with constant negative curvature. There are five isometric models of hyperbolic space (Cannon et al., 1997; Ratcliffe, 2006), including the Lorentz model L and the Poincaré ball model B (see Appendix B). We select the Lorentz model as the representation space in LorentzKG due to its nature as a homogeneous space and its intrinsic characteristic of the positive Lorentz group being its isometry group. In this section, we provide a brief overview of these concepts and important geometric notions in hyperbolic space that are relevant to our work.

heading: 2.1 Lorentz Model and Lorentz Transformation
text: Lorentzian Space. The (n + 1)-dimensional Lorentzian space R1,n is the Euclidean space Rn+1 equipped with a non-positive-definite bilinear form:
⟨x,y⟩L = −x0y0 + x1y1 + · · ·+ xnyn, (1)
where x = [x0, x1, . . . , xn]T ,y = [y0, y1, . . . , yn]
T ∈ Rn+1. The bilinear form ⟨·, ·⟩L is called Lorentzian inner product.
Lorentz Model. The n-dimensional Lorentz model Ln is a submanifold in R1,n defined as:
Ln := {x ∈ Rn+1 : ⟨x,x⟩L = −1, x0 > 0}. (2) The Lorentz model is the upper sheet of the twosheeted n-dimensional hyperboloid in R1,n.
Squared Lorentzian Distance. The squared Lorentzian distance is a modified distance metric used in the hyperbolic space (Ratcliffe, 2006). For x,y ∈ Ln, the squared Lorentzian distance is:
d2L = ⟨x− y,x− y⟩L = −2− 2⟨x,y⟩L. (3)
Lorentz Transformation. Linear isometries that preserve the Lorentzian inner product between every pair of points in the Lorentzian space are called Lorentz transformations. Specifically, a map ϕ : R1,n → R1,n is a Lorentz transformation if ⟨ϕ(x), ϕ(y)⟩L = ⟨x,y⟩L, ∀x,y ∈ Ln.
Lorentz Group. All Lorentz transformations form a group under composition. This group is called the Lorentz group, denoted by O(1, n). Let Jn = diag(−1,1n) where 1n is n-dimensional vector with all entries being 1 and diag(·) denotes a diagonal matrix, the Lorentz group is defined as:
O(1, n) := {A ∈GL(n+ 1,R) : AJnA T = ATJnA = Jn}, (4)
where GL(n+ 1,R) is the general linear group of (n+ 1)× (n+ 1)-invertible matrices over R.
Positive Lorentz Group. The positive Lorentz group is a subgroup of O(1, n) defined as O+(1, n) := {A ∈ O(1, n) : a11 > 0}. The group acts transitively on Ln through the group action x 7→ Ax, where x ∈ Ln and A ∈ O+(1, n). Specifically, Ax ∈ Ln, ∀x ∈ Ln and ∀A ∈ O+(1, n). As a result, O+(1, n) is the isometry group of the Lorentz model Ln.
Lorentz Rotation/Reflection and Lorentz Boost. A Lorentz transformation A ∈ O+(1, n) can be decomposed using a polar decomposition (Moretti, 2002) and expressed as:
A = [ 1 0 0 R ] [ c vT v √ In + vvT ] , (5)
where In is the identity matrix, R ∈ O(n), the orthogonal group (i.e., RTR = In), v ∈ Rn, and c = √ ∥v∥2 + 1. The first component is
called a Lorentz rotation Fig. 1(a) if det(R) = 1, or a Lorentz reflection Fig. 1(b) if det(R) = −1. Lorentz rotation/reflection is space rotation/reflection as it does not change the time axis. The second component is a Lorentz boost Fig. 1(c), where v is the velocity vector that describes the magnitude and direction of the boost.

heading: 3 Methodology
text: Here we present our proposed LorentzKG which uses the Lorentz model for entity embeddings and Lorentz transformation for relation embeddings in KGs. The score function measures triple plausibility using the squared Lorentzian distance between transformed entity embeddings, along with a bias term. We also discuss LorentzKG’s capability to model KG relations, highlighting its potential in capturing diverse relational patterns.

heading: 3.1 LorentzKG
text: In our approach, entities are represented as points in the Lorentz model, i.e., h, t ∈ Lk, while relations are represented as geometric transformations applied to entity embeddings. Specifically, relations are represented as Lorentz transformations embedded in the positive Lorentz group O+(1, k). Our approach brings forth the following significant benefits: (i) We can ensure that the transformed entities remain within hyperbolic space since Lorentz transformations are linear isometries of the Lorentz model. (ii) The linear operation is simple and stable to compute. (iii) The use of the Lorentz transformations allows for multiple transformations, including rotation, reflection, inversion, and translation as it is the isometry group of the Lorentz model, which will be discussed later. Next, we present our simple yet effective score function of LorentzKG.
Score Function. The score function in hyperbolic KGE models is typically defined as the squared hyperbolic distance between transformed entities in the hyperbolic space, followed by scalar biases for entities and a margin (Balažević et al., 2019; Chami et al., 2019; Chen et al., 2022b). Building upon this framework, we design the score function in our LorentzKG as follows:
fr(h, t) = −d2L(Λr,1h,Λr,2t)+bh+bt+δ, (6)
where h, t ∈ Lk are entity embeddings in Lorentz model, Λr,1,Λr,2 ∈ O+(1, k) are relation matrices, bh, bt ∈ R are scalar biases of head entity h and tail entity t, respectively, δ ∈ R is a margin hyperparameter and d2L is the squared Lorentz distance function given in Eq. 3. The terms Λr,1h and Λr,2t are referred to as the transformed (by relation) head and tail entities, respectively.
The geometric interpretation of the score function given in Eq. 6 is straightforward. The head entity h and the tail entity t are embedded in the Lorentz model, and a Lorentz transformation, which is the intrinsic operation on the Lorentz model, is used to model the relation r. This transformation determines the projected position of the entity embeddings adjusted by the relation. In a valid fact (h, r, t), the transformed h and t (by relation) should be brought closer to each other, as measured by the hyperbolic distance. Additionally, the entity-specific biases bh and bt can be interpreted as the sphere of influence of each entity, as suggested by (Balazevic et al., 2019). In practice, we enforce the constraints on the bias term bh, bt ∈ (−1, 1) following (Lin et al., 2015).

heading: 3.2 Comparison with Existing Hyperbolic KGE Models
text: In the context of entity transformation, a comparison between LorentzKG and other hyperbolic KGE models reveals distinct approaches. For a given head entity h, we call the relation-specified head entity transformed-h and analyze its distinctions across various hyperbolic KGE models.
In LorentzKG, the process involves applying a Lorentz transformation directly to entities represented in the Lorentz model (h ∈ Lk):
transformed-h = Λrh (7)
where Λr ∈ O+(1, k) a Lorentz transformation. Consequently, transformed-h naturally exists as a point in the hyperbolic space Lk.
In contrast, HyboNet (Chen et al., 2022b) proposed a general hyperbolic linear layer for entiry transformation (h ∈ Lk):
transformed-h = [ √
∥ϕ(W rh)∥+ 1 ϕ(W rh)
]
k+1×k+1 (8)
where W r ∈ Rk×k is an arbitrary matrix and ϕ(·) is an operation function, which includes a composition of operations such as dropout, normalization, and non-linear activation. To ensure the maintenance of transformed-h within the hyperbolic space, an additional scalar element is computed according to Eq. 2 and appended to the output ϕ(W rh). While Chen et al. (2022b) purports to present a fully hyperbolic method, it is noteworthy that the general hyperbolic linear layer employed in their implementation does not conform to the hyperbolic nature (Chen et al., 2022b, §3.1). Their implementation involves moving embeddings out of the manifold and then projecting them back, making it non-intrinsic. In contrast, we introduce simple and scalable entity transformations that consistently keep the learning trajectory on the manifold.
Moreover, AttH (Chami et al., 2020) employs a block-wise rotation/reflection matrix to transform entities in the Poincaré ball model (h ∈ Bk):
transformed-h =   G1r 0 0 0 . . . 0
0 0 G k/2 r   k×k h
(9)
where Gir = [ cos θi ∓ sin θi sin θi ± cos θi ] represents a 2 × 2 rotation/reflection matrix. Our LorentzKG distinguishes itself by its capacity to execute a k×k rotation/reflection within the k-dimensional hyperbolic space. This capability contrasts AttH’s limitation to a diagonal 2 × 2 rotation/reflection, thereby constraining its expressive power.

heading: 3.3 Capability in Modeling Relation Patterns
text: The Lorentz transformations offer a powerful approach for modeling various relational patterns. Our model captures symmetric/anti-symmetric relations through Lorentz rotations/reflections, as these transformations correspond to Euclidean rotations/reflections. Also, our model captures composition and inversion since all Lorentz transformations form the Lorentz group. Please refer to Appendix D for proofs.
Moreover, our method enables the modeling of hierarchical relations using Lorentz boosts. The idea in modeling hierarchical relations in KGs is to embed the entities at the same hierarchical level with similar radii (in the Poincaré ball model). In contrast to existing hierarchy-aware KGE models such as HAKE (Zhang et al., 2020) (Euclidean) and HBE (Pan and Wang, 2021) (hyperbolic), where the score function is a weighted average of modulus distance and phase distance in polar coordinates, our method provides a more streamlined solution that avoids the need to introduce separate distance measures for radii and angles. By applying a Lorentz boost, the transformed entity embedding experiences a change in radii (in the Poincaré ball model), thereby effectively capturing hierarchy.
Furthermore, LorentzKG effectively models relation cardinalities by applying different Lorentz transformations to both the head and tail entities. This implementation allows LorentzKG to capture and represent relation cardinalities, facilitating a comprehensive understanding of the underlying relationships between entities.

heading: 3.4 Connection to Euclidean Translation-based KGE Models
text: Our LorentzKG can be regarded as a natural generalization of Euclidean translation-based models to hyperbolic space. For example, the score function in STransE (Nguyen et al., 2016) is fr(h, t) = ∥M r,1h + r −M r,2t∥2 where h, t, r ∈ Rk and M r,i ∈ Rk×k. If we reformulate the score function by writing entity embeddings in homogeneous coordinates, the score is:
fr(h, t) = ∥M̂ r,1ĥ−M̂ r,2t̂∥2,
M̂ r,1 = [ M r,1 0 0 r ] , ĥ = [ h 1 ] ,
M̂ r,2 = [ M r,1 0 0 0 ] , t̂ = [ t 1 ] .
(10)
The score function in Eq. 6 of LorentzKG has the same form as that in STransE, making our method a generalization of STransE to hyperbolic space. Note that in STransE, the matrix embeddings of relations are arbitrary, implying that the representation space for relations forms the affine group. This arbitrariness frequently results in suboptimal performance due to over-parametrization, a limitation also observed in RESCAL (Nickel et al., 2011). In contrast, our approach in LorentzKG involves employing the Lorentz group to curtail the degree
of freedom in embedding relations while ensuring that the transformed entities remain points within the hyperbolic space.

heading: 3.5 Training and Optimization
text: Loss function. We follow the setup in (Balazevic et al., 2019; Chen et al., 2022b) to train our proposed LorentzKG model. We construct K negative samples for each triple (h, r, t) by corrupting either the tail entity (h, r, t′) or the head entity (h′, r, t) with a randomly chosen new entity from the entity set E . Let y ∈ {0, 1} denote the true label indicating the validity of a triplet. We apply the sigmoid function σ(·) to the scores fr(h, t) to calculate probabilities for triples and minimize the binary cross entropy loss:
L(y, p) = − 1 N
N∑
i=1
[log(pi) + K∑
k=1
log(1− p′(i,k))]
(11) where pi and p′(i,k) are the probabilities for correct and corrupted triples, respectively. N is the number of training samples and K is the number of negative samples for a positive sample.
Optimization. Instead of directly updating the relation embeddings Λr,i, a Lorentz transformation is decomposed using Eq. 5 into a composition of a Lorentz rotation/reflection and a Lorentz boost, which are then sequentially updated during optimization. Specifically, the Lorentz rotation/reflection is learned by learning a rotation (orthogonal) matrix R, while the Lorentz boost is learned by learning a real vector v. We implemented R as the Householder matrix (Uhlig, 2001) to enforce its orthogonality. We use Riemannian Adam (rAdam) (Becigneul and Ganea, 2019) to optimize the proposed model.

heading: 4 Experiments
text: Dataset. We evaluate our proposed model on two widely used knowledge graph datasets - WN18RR (Dettmers et al., 2018) and FB15k-237 (Toutanova et al., 2015). Some statistics of these two knowledge graph datasets are summarized in Table. 1. ξG is the global graph curvature that measures the level of hierarchy in graphs. The lower ξG indicates the more tree-like the KG is (Chami et al., 2020). More experiments are included in Sec G.
Evaluations. We evaluate the performance of link prediction in the filtered setting by ranking
test triples against all other candidate triples that do not appear in the training, validation, or test sets. To generate these candidates, we corrupt either the subject or object of the original triple to form (h′, r, t) or (h, r, t′). Evaluation is conducted using standard measures for these datasets, including Mean Reciprocal Rank (MRR) and Hits at N (H@N ), where N = 1, 3 or 10.
Baselines. We compare our LorentzKG with several baseline methods. For Euclidean KGE models, we include TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), ComplEx (Trouillon et al., 2016) ConvE (Dettmers et al., 2018), RotatE (Sun et al., 2019), QuatE (Zhang et al., 2019), TuckER (Balažević et al., 2019) and HAKE (Zhang et al., 2020). Regarding hyperbolic KGE models , we consider GIE (Cao et al., 2022), M2GNN (Wang et al., 2021), UltraE (Xiong et al., 2022), HyperKG (Kolyvakis et al., 2020), MuRP (Balazevic et al., 2019), RotH and AttH (Chami et al., 2020), ConE (Bai et al., 2021) and HyboNet (Chen et al., 2022b). For the hyperbolic KGE model, we mainly present
the results in the lower dimensions (dim = 32).
Implementation Details. The implementation of our LorentzKG is based on PyTorch, with the corresponding PyTorch-based pseudocode presented in Section E. All experiments are conducted on a NVIDIA Tesla Nvidia GPU A100 40GB machine. We employ random initialization on a manifold using the Python package Geoopt (Kochurov et al., 2020). To determine the optimal hyperparameters for our method, we conduct a grid search for learning rate, negative sample size, batch size, and margin. The best hyperparameters for each dataset are reported as follows: {WN18RR: 0.05, 200, 512, 1.08}, {FB15k-237:0.05,100,512,1.15}.
Overall Performance. We evaluate our approach in the low-dimensional setting for d = 32. The experimental results presented in Table 2 clearly illustrate the superior performance of our proposed LorentzKG model compared to existing methods in both Euclidean and hyperbolic spaces. On WN18RR, HAKE shows comparable results but with a 1000-dimensional representation space. The
reason may lie in the fact that hierarchy is one of the main relation patterns in WN18RR, as also evidenced by the lower ξG . Furthermore, considering the low-dimensional embedding space, the overall performance of hyperbolic KGE models is comparable, aligning with the understanding that hyperbolic space is efficient in representing hierarchical structures. Thus, our standout performance establishes the superiority of our approach in modeling various relation patterns, including hierarchy. FB15k-237 exhibits a larger number of relations but fewer entities compared to WN18RR, resulting in a more complex graph structure. The utilization of different geometric structures in GIE and M2GNN enhances their capability to effectively handle such intricate structures, leading to improved performance. However, LorentzKG outperforms M2GNN (the second-best model across most metrics) with a (relative improvement) 6.1% higher MRR, a 2.5% higher H@10, a 6.0% higher H@3, and a 4.4% higher H@1, respectively. This improvement is mainly attributed to the expressivity offered by the various types of Lorentz transformations utilized in our approach. As a result, our approach demonstrates wide applicability in handling complex knowledge graph structures.
Model Capacity. We compare the number of parameters of several KGE models: RotatE, MuRP, and GIE (values are sourced from (Cao et al., 2022)), and our proposed LorentzKG. As illustrated in Table. 4, it is evident that our LorentzKG achieves superior performance with a significantly
reduced parameter count, attributed to our exploration of low-dimensional hyperbolic space. Please refer to Sec H for more analysis on model capacity.
Ablation Studies. We conducted two types of ablation studies on our model. First, to assess the effects of applying the transformation exclusively to either the head entity (LorentzKG-h) or to the tail entity (LorentzKG-t). As demonstrated in Table 3, both variants of LorentzKG exhibited comparable results. However, through the synergistic utilization of both transformations, LorentzKG achieved superior performance by affording enhanced flexibility in capturing relational patterns. Second, to examine the impact of different types of Lorentz transformations (Lorentz rotation/reflection and Lorentz boost) in relational modeling, we compared our LorentzKG with its two variants: LorentzKG-r, which applies only Lorentz rotation/reflection to transform entities, and LorentzKG-b, which applies only Lorentz boost. The experimental results, presented in Table. 3, indicate that LorentzKG-r performs comparably to LorentzKG, as the majority of relation information is encoded by Lorentz rotation/reflection, which possesses a higher number of parameters compared to Lorentz boost. However, by leveraging the benefits of both types of transformations, LorentzKG achieves a more comprehensive representation of relational patterns and captures a wider range of semantic relationships, as evidenced by its superior performance in the table.
Embedding Dimension. To investigate the influence of embedding dimension, we perform experiments on WN18RR and FB15k-237 datasets using various settings {32, 64, 100, 200}, as shown in Table. 5. Our observations reveal improved performance (in most metrics) with increasing embedding dimensions, demonstrating that LorentzKG excels in both low and high dimensions.

heading: 5 Related Work
text: We review some KGE models, highlighting their differences in two aspects: the selection of the embedding space and the score function (Table. A2).

heading: 5.1 Euclidean Embeddings
text: Bilinear models, such as RESCAL (Nickel et al., 2011), represent relations as linear transformations acting on entity vectors. DistMult (Yang et al., 2015) is a special case of RESCAL that reduces the number of parameters per relation, while ComplEx (Trouillon et al., 2016) extends DistMult into the complex domain to model antisymmetry.
Translation-based models including TransE (Bordes et al., 2013) and its variants, treat relations as translations from head to tail entity embeddings, e.g., h + r ≈ t. However, they cannot capture specific logical properties such as symmetry and cannot model relations with higher cardinalities if the translation is only applied to the head entity, e.g., TransE. Rotation-based models define relations as rotations in complex space, RotatE (Sun et al., 2019), or hypercomplex space, QuatE (Zhang et al., 2019) to enable expressive semantic matching between head and tail entities. While these models can capture important relation patterns in KGs, they are limited in modeling hierarchy and usually suffer from high complexity (Bordes et al., 2013; Chen and Li, 2020a; Sun et al., 2019; Lai et al., 2023; Jin et al., 2024; Moosaei et al., 2022).

heading: 5.2 Hyperbolic Embeddings
text: As HAKE (Zhang et al., 2020) demonstrates, relations in KGs often exhibit semantic hierarchies. For instance, within the triplets (Alice, _is_friend_of, Bob) and (Bob, _is_supervisor_of, Chris) (Cai et al., 2018), entities Alice and Bob share the same hierarchical level, whereas Chris is considered at a different hierarchical level. HAKE employes the polar coordinate system in Euclidean space to model these hierarchies, by assuming that entities at different hierarchical levels have distinct mag-
nitudes in polar coordinates. Recently, hyperbolic space has been increasingly used for KGs due to its effectiveness in capturing hierarchical structures.
In hyperbolic KGE models, the likelihood of a fact is assessed by measuring the hyperbolic distance between the translated embeddings of its entities in the hyperbolic space. MuRP (Balazevic et al., 2019) and HyperKG (Kolyvakis et al., 2020) are two early attempts in the development of translation-based hyperbolic KGE methods on the Poincaré ball model. MuRP employs Möbius matrix-vector multiplication and Möbius addition (Ungar, 2005), which are generalizations of Euclidean operations in tangent space, to translate entity embeddings. HyperKG utilizes Euclidean addition and circular permutation. However, these operations do not completely reflect hyperbolic geometry. AttH (Chami et al., 2019) uses a combination of block-wise hyperbolic reflections/rotations, Eq. 9, and attention modules to model symmetric/antisymmetric relational patterns. UltraE (Xiong et al., 2022) adopts the same approach but uses ultrahyperbolic manifold as embedding space. HBE (Pan and Wang, 2021) adopts the idea of using a polar coordinate system to capture hierarchical structures to the extended Poincaré ball model. ConE (Bai et al., 2021) uses hyperbolic cones to represent entities and models relations as transformations between the cones. However, Möbius operations are applied in both HBE and ConE. HyboNet (Chen et al., 2022b) operates on the Lorentz model, where the head entity is transformed using a Lorentz linear transformation layer as described in Eq. 8. However, none of the existing hyperbolic KGE models apply transformations to both the head and tail entities, i.e., the hyperbolic distance is measured for the transformed head entity and the tail entity, which prevents these models from capture relation cardinalities, similar to the case of STransE (Nguyen et al., 2016).
Several recent studies have investigated combining other topological structures (e.g., hypersphere) with hyperbolic space in KGE. Notable examples
include M2GNN (Wang et al., 2021) and GIE (Cao et al., 2022) where tangent space operations are applied in these models.

heading: 6 Conclusions
text: Here we introduce LorentzKG, a fully translationbased hyperbolic KGE model that utilizes Lorentz transformations. LorentzKG enables linear translation of entity embeddings and facilitates the modeling of various relation patterns in KGs. By applying different transformations to the head and tail entities, LorentzKG is able to capture relation cardinalities. Experimental results show the superiority of LorentzKG over existing models, highlighting its enhanced expressivity and modeling capabilities, particularly in low-dimensional settings.

heading: 7 Limitations
text: This study primarily concentrates on leveraging hyperbolic geometry to improve hyperbolic knowledge graph embeddings, with a particular emphasis on low-dimensional embeddings. Our comparative analysis was confined to translation-based KGE models and hyperbolic models. This particular focus might potentially narrow the breadth of our findings. Future research endeavors should consider diversifying the scope to include more diverse settings and larger-scale data to further validate and extend the applicability of our proposed methodologies.

heading: A Notations
text: Notation Description G A knowledge graph E A set of entities R A set of relations h Head entity t Tail entity r Relation (h, r, t) Fact in KG (h, r, t) Embedding of a fact M · Mapping matrix fr(h, t) Score function d·(·, ·) Distance metric in specific space Rk k-dimensional Euclidean space Ck k-dimensional complex space Bk k-dimensional Poincaré ball model Lk k-dimensional Lorentz model ◦ Element-wise product ⊗ Möbius matrix-vector multiplication ⊕ Möbius addition
O(n) Orthogonal group O(1, n) Lorentz group O+(1, n) Positive Lorentz group
Λ Lorentz transformations
Table A1: Notations
B More on Hyperbolic Geometry
(a) (b)
Figure A1: (a) Isometric transformation between the Poincaré ball model B2 (grey disk) and the Lorentz model L2 (blue surface); (b) Points on B2.
The Poincaré ball model. The k-dimensional Poincaré ball model of hperbolic space Bk is the interior of a unit ball in Rk, i.e.,
Bk = {x = [x1, . . . , xk] ∈ Rk : ∥x∥ < 1} (A1)
where ∥·∥ is the Euclidean norm.
The correspondence between the Poincaré ball model Bk and the Lorentz model Lk is given by [x0, x1, . . . , xk] ∈ Lk ⇔ [
x1 1 + x0 , . . . xk 1 + x0
] ∈ Bk,
(A2) the stereographic projection of hyperbolic space (Cannon et al., 1997), as illustrated in Fig. A1 for the 2-dimensional case.
Geodesic distance. Recall that the geodesic distance in hyperbolic space is given by:
dL(x,y) = cosh −1(−⟨x,y⟩L) ≜ θ (A3)
where x,y ∈ Lk and ⟨·, ·⟩L is the Lorentzian inner product.
Tangent space. Given x ∈ Lk, the tangent space centered at x is
TxLk := {v ∈ Rk+1 : ⟨v,x⟩L = 0}. (A4)
TxLk is a local approximation of the Lorentz model at x and a Euclidean subspace of Rk+1. v ∈ TxLk is known as the tangent vector.
Exponential and logarithmic maps. The mapping between tangent space and hyperbolic space is conducted by exponential and logarithmic maps. For any x ∈ Ln and v ∈ TxLn, the exponential map at x is given by
Expx(v) = cosh(∥v∥L)x+ sinh(∥v∥L)v/∥v∥L. (A5) the inverse of the exponential map, the logarithmic map, is given by
Logx(y) = θ
sinh(θ) (y − cosh(θ)x) (A6)
where x,y ∈ Ln and θ is the geodesic distance between x and y.
C Definitions
The definitions of relation patterns are presented in this section. With a slight abuse of notation, we denote a fact in the knowledge graph as r(x, y), where r ∈ R is relation and x, y ∈ E are entities. Definition 1. A relation r is symmetric if ∀x, y
r(x, y) ⇒ r(y, x) (A7)
A clause with such form is a symmetric pattern. Definition 2. A relation r is antisymmetric if ∀x, y
r(x, y) ⇒ ¬r(y, x) (A8) A clause with such form is a antisymmetric pattern.
Model Score Function fr(h, t) Embedding Spaces SE (Bordes et al., 2011) −∥Mr,1h−Mr,2t∥21 h, t ∈ Rk,Mr,i ∈ Rk×k TransE (Bordes et al., 2013) −∥h+ r − t∥1/2 h, r, t ∈ Rk StransE (Nguyen et al., 2016) −∥Mr,1h+ r −Mr,2t∥1/2 h, r, t ∈ Rk,Mr,i ∈ Rk×k RotatE (Sun et al., 2019) −∥h ◦ r − t∥2 h, r, t ∈ Ck, |ri| = 1 RESCAL (Nickel et al., 2011) ⟨hTM rt⟩ h, t ∈ Rk,M r ∈ Rk×k DistMult (Yang et al., 2015) ⟨hT diag(r)t⟩ h, r, t ∈ Rk ComplEx (Trouillon et al., 2016) Re(⟨hT diag(r)t̄⟩) h, r, t ∈ Ck HAKE (Zhang et al., 2020) −∥hm ◦ rm − tm∥2 − λ∥sin(hp + rp − tp)/2∥1 hm, tm ∈ Rk, rm ∈ Rk+ hp, rp, tp ∈ [0, 2π)k, λ ∈ R MuRP (Balazevic et al., 2019) −dB(Mr ⊗ h, t⊕ r))2 + bh + bt h, r, t ∈ Bk,Mr ∈ Rk×k AttH (Chami et al., 2019) −dB(Q(h, r), t)2 + bh + bt h, r, t ∈ Bk HyboNet (Chen et al., 2022b) −d2L(gr(h), t) + bh + bt + δ h, t ∈ Lk LorentzKG −d2L(Λr,1h,Λr,2t) + bh + bt + δ h, t ∈ Lk,Λr,i ∈ O+(1, n)
Table A2: The score functions fr(h, t) and embedding spaces of several knowledge graph embedding models, where ◦ denotes the Hadamard product, ⟨·⟩ denotes the dot product, ⊕ and ⊗ denotes the Möbius addition and the Möbius matrix multiplication, respectively. Also, dM(·, ·) is the distance metric in manifold M and bh, bt ∈ R are scalar biases. δ ∈ R is a margin hyper-parameter. Q(h, r) in AttH is a hyperbolic attention module, and gr(h) in HyboNet is Euclidean linear transformation followed by normalization.
Definition 3. Relation r1 is inverse to relation r2 if ∀x, y r2(x, y) ⇒ r1(y, x) (A9) A clause with such form is a inversion pattern. Definition 4. Relation r1 is composed of relation r2 and relation r3 if ∀x, y, z
r2(x, y) ∧ r3(y, z) ⇒ r1(x, z) (A10)
A clause with such form is a composition pattern.

heading: D Proofs of Capability of LorentzKG in Modeling Relation Patterns
text: If a triple (h, r, t) is a valid fact, we would expect that in our embedding Λr,1h = Λr,2t, or equivalently Λ−1r,2Λr,1h = t. Note that Λ −1 r,2Λr,1 ∈ O+(1, n) since O+(1, n) is a group. In the following proofs of relation patterns, we denote Λ−1r,tΛr,1 as Λr and check if Λrh = t in a valid fact (h, r, t).
Proof. (symmetry pattern) If r(x, y) and r(y, x) hold, we have
(Λrx = y) ∧ (Λry = x) ⇒ ΛrΛr = I (A11)
Proof. (antisymmetry pattern) If r(x, y) and ¬r(y, x) hold, we have
(Λrx = y) ∧ (Λry ̸= x) ⇒ ΛrΛr ̸= I (A12)
Proof. (inversion pattern) If r1(x, y) and r2(y, x) hold, we have
(Λr1x = y) ∧ (Λr2y = x) ⇒ Λr1 = Λ−1r2 (A13)
Proof. (compositon pattern) If r2(x, y), r3(y, z) and r1(x, z) hold, we have
(Λr2x = y)∧(Λr3y = z) ∧ (Λr1x = z) ⇒Λr3Λr2 = Λr1
(A14)

heading: E Pseudocode for LorentzKG
text: We illustrate the implementation of our proposed LorentzKG with pseudocode in Algorithm 1.

heading: F Analysis on Hierarchy Pattern
text: In order to assess the hierarchy patterns captured by our LorentzKG model, we gather the embeddings of head and tail entities for some relations in WN18RR. Subsequently, we compute the hyperbolic distance from these embeddings to the origin point o = [1, 0, . . . , 0] ∈ Lk, and plot the resulting distribution histograms, as illustrated in Fig. A2. We expect that entities at the same hierarchical level would exhibit similar distances from
Algorithm 1: Pytorch-style Pseudocode for LorentzKG # LorentzRotation: Lorentz rotation/reflection neural network module # LorentzBoost: Lorentz boost neural network module # LorentzDistance: squared Lorentzian distance function # # Train LorentzKG for N epochs for epoch in range(N):
for (h,r,t) in training data and negative samples: trans_h = LorentzRotation(h, r) trans_h = LorentzBoost(trans_h, r) trans_t = LorentzRotation(t, r) trans_t = LorentzBoost(trans_t, r) d = LorentzDistance(trans_h, trans_t) + bias_h + bias_t + margin loss = nn.BCEWithLogitsLoss(d, y) # y = ± 1 loss.backward() optimizer_radam.step()
(a) _hypernym (b) _instance_hypernym
(c) _derivationally_related_form (d) _similar_to
Figure A2: Distribution histograms of entity embeddings in some relations in WN18RR. For a specific relation, the distribution of distances from the head entity embeddings to the origin point o = [1, 0, . . . , 0] ∈ Lk is shown in orange. And the distribution of distances from the tail entity embeddings to the origin point is shown in blue.
the origin point, while entities with higher-level hierarchies would be closer to the origin point.
In the WN18RR dataset, the relations _hypernym and _instance_hypernym are hierarchical relations where tail entities are at higher levels than head
entities of the hierarchy. As depicted in Fig. 2(a) and Fig. 2(b), we observe that the tail embeddings tend to be situated at higher levels compared to the head embeddings, aligning with our expectations.
On the other hand, the relations _derivation-
CoDEx-s CoDEx-m CoDEx-l
Model MRR Hit@1 MRR Hit@1 MRR Hit@1
TransE 35.4 21.9 30.3 22.3 18.7 11.6 ComplEx 46.5 37.2 33.7 26.2 29.4 23.7 ConvE 44.4 34.3 31.8 23.9 30.3 24.0 TuckE 44.4 33.9 32.8 23.9 30.9 24.4
AttH 40.2 28.6 31.5 23.7 - - MuRP 42.0 31.1 30.6 22.6 - - CoPE 44.6 35.0 32.6 25.1 - -
LorentzKG 52.1 37.9 36.8 24.5 33.4 22.6
Table A3: Performances of various KGE models on CoDEx datasets.
ally_related_form and _similar_to are symmetric relations. As shown in Fig. 2(c) and Fig. 2(d), the tail embeddings exhibit similar levels to the head embeddings, as anticipated.

heading: G Additional Experimental Results
text: We validate our results on a new dataset, CoDEx (Safavi and Koutra, 2020). CoDEx, which comprises three diverse knowledge graphs of varying sizes (CoDEx-s, CoDEx-m, CoDEx-l), is a recently developed benchmark for KG completion tasks. The performance of our model, along with several competing models, is presented in the Table. A3 Most results are sourced from the original work(Safavi and Koutra, 2020), while the results from some hyperbolic KGE018 models (AttH and MuRP) are taken from a recent work, CoPE (Zeb et al., 2024). As evidenced in the table, our LorentzKG demonstrates a competitive to superior performance on the new benchmark.

heading: H More on Model Capacity
text: Note that the number of parameters taken from the original GIE (Cao et al., 2022) paper for MuRP (Balazevic et al., 2019) is for an embedding dimension other than 32. For d=32, the number of parameters in MuRP is 5.41M (WN18RR) and 1.94M (FB15k-237), which is more than that in LorentzKG (1.39M and 1.42M). In LorentzKG, the number of parameters increases linearly with the number of entities and quadratically with the number of relations — the same complexity as in HyboNet (Chen et al., 2022b). However, since transformations are applied to both head and tail, whereas only the head is transformed in HyboNet, we have a few more parameters compared to HyboNet.

heading: I Ethics Statement
text: This study adhered to rigorous ethical guidelines. All data used were publicly available and contained no personally identifiable information. No human participants or animals were involved, hence no ethical approval was required. The research was conducted responsibly, with any potential conflicts of interest declared. The findings are the result of unbiased scientific investigation.

heading: J Use of AI Assistants
text: AI assistants are used in this paper for spelling and grammar check.

heading: Future Work
text: Future research endeavors should consider diversifying the scope to include more diverse settings and larger-scale data to further validate and extend the applicability of our proposed methodologies