Title: RIEMANNIAN TRANSE: MULTI-RELATIONAL GRAPH EMBEDDING IN NON-EUCLIDEAN SPACE

Abstract: Multi-relational graph embedding which aims at achieving effective representations with reduced low-dimensional parameters, has been widely used in knowledge base completion. Although knowledge base data usually contains tree-like or cyclic structure, none of existing approaches can embed these data into a compatible space that in line with the structure. To overcome this problem, a novel framework, called Riemannian TransE, is proposed in this paper to embed the entities in a Riemannian manifold. Riemannian TransE models each relation as a move to a point and defines specific novel distance dissimilarity for each relation, so that all the relations are naturally embedded in correspondence to the structure of data. Experiments on several knowledge base completion tasks have shown that, based on an appropriate choice of manifold, Riemannian TransE achieves good performance even with a significantly reduced parameters.

1. INTRODUCTION:

1.1 BACKGROUND Multi-relational graphs, such as social networks and knowledge bases, have a variety of applications, and embedding methods for these graphs are particularly important for these applications. For instance, multi-relational graph embedding has been applied to social network analysis (Krohn-Grimberghe et al., 2012) and knowledge base completion (Bordes et al., 2013) . A multi-relational graph consists of entities V, a set R of relation types, and a collection of real data triples, where each triple (h, r, t) ∈ V × R × V represents some relation r ∈ R between a head entity h ∈ V and a tail entity t ∈ V. Embedding a multi-relational graph refers to a map from the entity and the relation set to some space. Mathematical operations in this space enable many tasks, including clustering of entities and completion, prediction, or denoising of triples. Indeed, completion tasks for knowledge bases attract considerable attention, because knowledge bases are known to be far from complete, as discussed in (West et al., 2014) (Krompaß et al., 2015) . Multi-relational graph embedding can help its completion and improve the performance of applications that use the graph. This is the reason why much work focuses on multi-relational graph embedding. Figure 1 shows an example of a multi-relational graph and a completion task.

In multi-relational graph embedding, reducing the number of parameters is an important problem in the era of big data. Many parameters are needed with tensor-factorization-based methods, such as Bayesian clustered tensor factorization (BCTF) (Sutskever et al., 2009) , RESCAL (Nickel et al., 2011) , and a neural tensor network (NTN) (Socher et al., 2013) , where each relation has a dense matrix or tensors (O D 2 or more parameters, where D is dimensionality of the space). Thus, TransE (Bordes et al., 2013) was proposed to reduce the number of parameters, to overcome this problem. In TransE, each entity is mapped to a point in Euclidean space and each relation is no more than a vector addition (O (D) parameters), rather than a matrix operation. The successors to TransE, TransH (Wang et al., 2014) and TransD (Ji et al., 2016) , also use only a small number of parameters. Some methods succeeded in reducing parameters using diagonal matrices instead of dense matrices: e.g. DISTMULT (Yang et al., 2015) , ComplEx (Trouillon et al., 2016) , HolE (through the Fourier transform) (Nickel et al., 2016) , and ANALOGY (Liu et al., 2017) . In these methods, all relations share one space for embedding, but each relation uses its own dissimilarity criterion. The success of these methods implies that one common space underlies whole data, and each relation can be regarded as a dissimilarity criterion in the space.

Whereas these methods use distances or inner products in Euclidean space as dissimilarity criteria, recent work has shown that using non-Euclidean space can further reduce the number of parameters. One typical example of this is Poincaré Embedding (Nickel & Kiela, 2017) for hierarchical data, where a hyperbolic space is used as a space for embedding. Here, the tree structure of hierarchical data has good compatibility with the exponential growth of hyperbolic space. Recall the circumference with radius R is given by 2π sinh R(≈ 2π exp R) in a hyperbolic plane. As a result, Poincaré embedding achieved good graph completion accuracy, even in low dimensionality such as 5 or 10. On the other hand, spheres (circumference: 2π sin R) are compatible with cyclic structures. Since Poincaré embedding, several methods have been proposed for single-relational graph embedding in non-Euclidean space (e.g. (Ganea et al., 2018b) , (Nickel & Kiela, 2018) ) and shown good results. The success of these methods suggests that the appropriate choice of a manifold (i.e., space) can retain low dimensionality, although these methods are limited to single-relational graph embedding.

According to the success of the TransE and its derivation and Poincaré embedding, it is reasonable in multi-relational graph embedding to assume the existence of a single structure compatible with a non-Euclidean manifold. For example, we can consider a single tree-like structure, which contains multiple hierarchical structures, where root selection gives multiple hierarchical structures from a single tree, which is compatible with hyperbolic spaces (See Figure 2 ). Therefore, embedding in a single shared non-Euclidean manifold with multiple dissimilarity criteria used in TransE is promising. Taking Poincaré embedding's success with low dimensionality into consideration, this method should work well (e.g., in graph completion tasks) with small number of parameters. This is the main idea of this paper.

Figure 1 : Multi-relational graph and its completion. There are five entities and two kinds of relation (hypernym and synonym). Graph completion refers to answering questions such as "is mammal a hypernym of cannis?" Figure 2 : Multiple hierarchical relations in a single tree. As this example shows, it is possible that multiple relations are given by multiple dissimilarity criteria in a single structure.

1.2. CONTRIBUTIONS:

We propose a novel method, called Riemannian TransE, for multi-relation graph embedding using a non-Euclidean manifold. In Riemannian TransE, the relations share one non-Euclidean space and the entities are mapped to the space, whereas each relation has its own dissimilarity criterion based on the distance in the space. Specifically, the dissimilarity criteria in Riemannian TransE are similar to those in TransE (Bordes et al., 2013) based on vector addition, which is known to be effective. Unfortunately, we cannot straightforwardly use TransE's dissimilarity criteria. This is due to non-existence of a parallel vector field (See Figure 4 ), which is implicitly but essentially used in "vector addition." However, the parallel condition is not essential in TransE's idea. For example, hierarchical bottom to top relations should be regarded as attraction to the top in the hierarchy, which is not parallel but has an attractive point. Moreover, parallel vector fields can be regarded as a vector field attracted to a point at infinity. Therefore, we replace parallel vector fields in TransE by vector fields with an attractive point that are well-defined in Riemannian manifolds, and as a result, we obtain Riemannian TransE. Advantages of non-Euclidean spaces enable our Riemannian TransE to achieve good performance (e.g. in graph completion) with low-dimensional parameters. Riemannian TransE further exploits the advantages of TransE: that is, the method needs only O (D) parameters for each relation. Numerical experiments on graph completion tasks show that with an appropriate choice of manifold, our method can improve the performance of multi-relational graph embedding with few parameters.

2. RELATED WORK:



2.1. MULTI-RELATIONAL GRAPH EMBEDDING:

Let V and R denote the entities and relations in a multi-relational graph, and let T ⊂ V × R × V denote the triples in the graph. Multi-relational graph embedding refers to a pair of maps from V and R into M e and M r , respectively. Particularly, learning multi-relational graph embedding refers to obtaining an appropriate pair of maps v → p v (v ∈ V, p v ∈ M e ) and r → w r (r ∈ R, w r ∈ M r ) from the triples T . In this paper, we call p v the planet of entity v, w r the launcher of relation r, and M e and M r the planet manifold and launcher manifold, respectively. The quality of embedding is measured through a score function f : (M e × M e ) × M r → R, which is designed by each method. Embedding is learned such that the value score function f (p h , p t ; w r ) will be low when p h , p t ; w r ∈ T and high when p h , p t ; w r / ∈ T . For specific loss functions designed from the score function, see Subsection 2.3. We interpret the score function of multi-relational graph embedding as dissimilarity in a manifold, which we call a satellite manifold M s . We rewrite the score function f in multi-relational graph embedding using two maps H , T : M e × M r → M s and the dissimilarity measure function D : M s × M s → R as follows:

EQUATION

We call H and T the head and tail launch map, respectively, and call s H v;r and s T v;r the head and tail satellite of entity v (or of planet p v ) with respect to relation r. The idea of this formulation is embedding in one shared space with multiple dissimilarity criteria. Specifically, each entity has only one planet and their satellite pairs give multiple dissimilarity criteria, each of which corresponds to a relation. In other words, all of the relations shares one space and the planets in it, and the differences among the relations are reduced to the difference of their launcher maps and the satellites given by them. We regard the planets as the embeddings of the entities, whereas dissimilarity between entities with respect to a relation is evaluated through their satellites which correspond to the relation.

A simple example of this is TransE (Bordes et al., 2013) , where all of the planets, satellites, and launchers share the same Euclidean space, i.e. M e = M s = M r = R D , the launch maps are given by vector addition as H (p; w) = p + w and T (p; w) = p, and the distance in a norm spacei.e. the norm of the difference-is used as a dissimilarity criterion i.e. D s H , s T = s Ts H (the L1 or L2 norm is often used in practice). See Figure 5 (left). As Nguyen (2017) suggested, one can associate the idea of representing relations as vector additions with the fact that we can find a relation through a substraction operator in Word2Vec Mikolov et al. (2013) . That is, we can find relations such as p France -p Paris ≈ p Italy -p Rome in Word2Vec. As explained above, TransE is based on the distance between satellites, and each satellite is given by simple vector addition. Regardless of this simplicity, the performance of TransE has been exemplified in review papers (Nickel et al., 2016 ) (Nguyen, 2017) . Indeed, the addition operation in a linear space is essential in the launcher map, and hence TransE can easily be extended to a Lie group, which is a manifold equipped with an addition operator, as suggested in Ebisu & Ichise (2017) . Some methods, such as TransH (Wang et al., 2014) , TransR (Lin et al., 2015) , and TransD (Ji et al., 2016) , also use a norm in linear space as a dissimilarity measure, integrating a linear map into a latent space.

Another simple example is RESCAL (Nickel et al., 2011) , which uses the negative inner product as a dissimilarity measure. In RESCAL, the launcher of relation r is a matrix W ∈ M r = R D×D , the launch maps are given by a linear map, i.e. H (p; (W , w)) = W p and T (p; (W , w)) = p, and the dissimilarity measure is the negative inner product D s H , s T = -s H s T . Other methods are also based on the (negative) inner product dissimilarity: e.g., DISTMULT (Yang et al., 2015) , ComplEx (Trouillon et al., 2016) , HolE (through the Fourier transform) (Nickel et al., 2016) , and ANALOGY (Liu et al., 2017) . Table 1 shows score functions of these methods.

Whereas some methods are based on a neural network (e.g., the neural tensor network (Socher et al., 2013) and ConvE (Dettmers et al., 2017) ), their score function consists of linear operations and element-wise nonlinear functions.

2.2. GRAPH EMBEDDING IN NON-EUCLIDEAN SPACE:

Graph embedding using non-Euclidean space has attracted considerable attention, recently. Specifically, embedding methods using hyperbolic space have achieved outstanding results (Nickel & Kiela, Table 1 : Score Functions. The launcher w r of r determines the dissimilarity criterion of r through satellites. In this table, the dimensionality is set so that the (real) dimensionality of the planets is D.

† denotes conjugate transpose. F denotes the discrete Fourier Transform. The interpretation here of HolE is given by Liu et al. (2017) and Hayashi & Shimbo (2017) . 

= ( r , pr) ∈ R × M m [-r ] + ,pr (pt) ∈ M (See (6)) D |V| + (D + 1) |R| TransE p v ∈ R D p h + wr ∈ R D s T t;r -s H h;r Bordes et al. (2013) wr ∈ R D p t ∈ R D D |V| + D |R| TransH p v ∈ R D I -w pr r w pr r p h + wr ∈ R D s T t;r -s H h;r Wang et al. (2014) (wr, w pr r ) ∈ R D × R D I -w pr r w pr r p t ∈ R D D |V| + 2D |R| TransR p v ∈ R D W r p h + wr ∈ R D s T t;r -s H h;r Lin et al. (2015) (W r , wr) ∈ R D× D × R D W r p t ∈ R D D |V| + D D + D |R| TransD (p v , p pr v ) ∈ R D/2 × R D/2 I + w pr r p pr h p h + wr ∈ R D s T t;r -s H h;r Ji et al. (2016) (wr, w pr r ) ∈ R D × R D I + w pr r p pr t p t ∈ R D D |V| + 2 D |R| RESCAL p v ∈ R D W r p h ∈ R D -s H h;r s T t;r Nickel et al. (2011) W ∈ R D×D p t ∈ R D D |V| + D 2 |R| DISTMULT p v ∈ R D diag {wr} p h ∈ R D -s H h;r s T t;r Yang et al. (2015) wr ∈ R D p t ∈ R D D |V| + D |R| ComplEx p v ∈ C D/2 diag {wr} p h ∈ C D/2 -Re s H h;r † s T t;r Trouillon et al. (2016) wr ∈ C D/2 p t ∈ C D/2 D |V| + D |R| HolE p v ∈ R D F (p h ) ∈ C D -Re s H h;r † s T t;r Nickel et al. (2016) wr ∈ R D diag {F (wr)} F (p t ) ∈ C D D |V| + D |R| ANALOGY p C v , p R v ∈ C D/4 × R D/2 diag {wr} p h ∈ C 3 4 D -Re s H h;r † s T t;r Liu et al. (2017) w C r , w R r ∈ C D/4 × R D/2 p t ∈ C 3 4 D D |V| + D |R|

2017) (Ganea et al., 2018b ) (Nickel & Kiela, 2018) . With these methods, each node in the graph is mapped to a point in hyperbolic space and the dissimilarity is measured by a distance function in the space. Although these methods exploit the advantages of non-Euclidean space, specifically those of a negative curvature space, they focus on single-rather than multi-relational graph embedding.

By contrast, TransE has been extended to an embedding method in a Lie group-that is, a manifold with the structure of a group (Ebisu & Ichise, 2017) . As such, the regularization problem in TransE is avoided by using torus, which can be regarded as a Lie group. Although this extension to TransE deals with multi-relational embedding, it cannot be applied to all manifolds. This is because not all manifolds have the structure of a Lie group. Indeed, we cannot regard a hyperbolic space (if D = 1) or a sphere (if D = 1, 3) as a Lie group.

2.3. LOSS FUNCTION:

We can simply design a loss function on the basis of the negative log likelihood of a Bernoulli model as follows:

L {p v } v∈V , {w r } r∈R := - (h,r,t)∈T log (σ (f (p h , p t ; w r ))) - (h ,r ,t )∈T c log (1 -σ (f (p h , p t ; w r ))) ,

(2) where T c := (V × R × V)\T and σ : R → [0, 1] is a sigmoid function. However, this loss function needs evaluation of the score function for all negative triplets (V × R × V) \ T . To avoid this, most methods (e.g., TransE) use the following margin-based loss function:

EQUATION

)

where Q is the set of the triples with its corrupted head and tail. That is,

EQUATION

where δ ∈ R ≥0 is the margin hyperparameter, and [•] + denotes the negative value clipping-i.e. for all x ∈ R, [x] + := max(x, 0). We use this loss function throughout this paper.

3. RIEMANNIAN TRANSE:

In this section, we formulate Riemannian TransE exploiting the advantages of TransE in non-Euclidean manifolds. Firstly, we give a brief introduction of Riemannian geometry. Secondly, we explain the difficulty in application of TransE in non-Euclidean manifolds. Lastly, we formulate Riemannian TransE.

3.1. RIEMANNIAN MANIFOLDS AND OPERATIONS:

Let (M, g) be a Riemannian manifold with metric g. We denote the tangent and cotangent space of M on p by T p M and T * p M, respectively, and we denote the collection of all smooth vector fields on M by X (M).

Let ∇ : X (M) × X (M) (X, Y ) → ∇ X Y ∈ X (M)

denote the Levi-Civita connection, the unique metric-preserving torsion-free affine connection. A smooth curve γ : (-, ) → M is a geodesic when ∇ γ γ = 0 on curve γ, where γ is the differential of curve γ. Geodesics are generalizations of straight lines, in the sense that they are constant speed curves that are locally distance-minimizing. We define the exponential map Exp p , which moves point p ∈ M towards a vector by the magnitude of the vector. In this sense, the exponential map is regarded as an extension of vector addition in a Riemannian manifold. Figure 3 shows an intuitive example of an exponential map on a sphere. Let γ v (v ∈ T p M) denote the geodesic that satisfies γv (0) = v. The exponential map Exp p : T p M → M is given by Exp p (v) := γ v (1). We define the logarithmic map Log p : M → T p M as the inverse of the exponential map. Note that the exponential map is not always bijective, and we have to limit the domain of the exponential and logarithmic map appropriately, while some manifolds, such as Euclidean and hyperbolic space, do not suffer from this problem.

3.2. DIFFICULTIES IN RIEMANNIAN MANIFOLDS:

In TransE, a single vector w r determines the head and tail launch maps H , T as a transform: R D → R D . In fact, these launch maps are given by vector addition. Note that this constitution of the launcher maps implicitly but essentially uses the fact that a vector is identified with a parallel vector field in Euclidean space. Specifically, a vector w determines a parallel vector field, denoted by W r here, which gives a tangent vector [W r ] p ∈ T p R D on every point p ∈ R D , and each tangent vector determines the exponential map Exp p ([W r ] p ) at p, which is used as a launch map in TransE. However, because there is no parallel vector field in non-zero curvature spaces, we cannot apply TransE straightforwardly in non-zero curvature spaces. Thus, extention of TransE in non-Euclidean space non-trivial. This is the difficulty in Riemannian Manifolds.

3.3. FORMULATION OF RIEMANNIAN TRANSE:

As we have explained in Introduction, our idea is replacing parallel vector fields in TransE by vector fields attracted to a point. Specifically, we obtain the Riemannian TransE as an extension of TransE, replacing the launchers w r ∈ R D in TransE by pairs w r = ( r , p r ) ∈ R × M of a scalar value and point, indicating the length and destination of the satellites' move, respectively. We call p r the attraction point of relation r. In other words, we replace parallel vector field W r = w r in TransE by r Log q (p) Log q (p) q . Note that, we use a fixed manifold M e = M for entity embedding and use direct product manifold M r = R × M for relation embedding.

However, the extension still has arbitrariness. For instance, we could launch the tail satellite instead of the head satellite in TransE; in other words, the following launching map also gives us a score function equivalent to that of the original TransE: H (p; w) = p and T (p; w) = pw (Figure 5 center). On the other hand, the score function depends on whether we move the head or tail satellites in our case, where the attraction points are not at infinity. With hierarchical data, an entity at a higher hierarchy has many related entities in a lower hierarchy. Therefore, it is best to always launch the satellites of "children," the entities in a lower hierarchy, toward their parent. Hence, we move the head satellites when r > 0 and fix the tail satellites, and vice versa when r < 0; specifically, we move the head satellites by length λ = [ r ] + and move the tail satellites by length λ = [r ] + . Thus, bottom-to-top relation cases correspond to r > 0 (Figure 6 , left), and top-to-bottom relation cases correspond to r < 0 (Figure 6 , right). Another problem pertains to launching the satellites near the attraction point. If λ > ∆ (p r , p v ), the naive rule causes overrun. In this case, we simply clip the move and set the satellite in the place of p r .

We turn now to the score function of Riemannian TransE. The score function f : (M × M) × (R, M) → R in Riemannian TransE is given as follows:

EQUATION

where transform m λ,p

: M → M denotes a move, defined as follows:

EQUATION

Here, note that m ,p (q) is on the geodesic that passes through p and q. Although the exponential map and logarithmic map in closed form are required to implement Riemannian TransE, we can obtain them when the manifold M is a sphere S D (positive curvature), Euclidean space R D (zero curvature), and hyperbolic space H D (negative curvature), or a direct product of them. These are practically sufficient. Also note that the computation costs of these maps are O(D), which is small enough.

3.4. OPTIMIZATION:

In typical cases, the number of entities is very large. Therefore, stochastic gradient methods are effective for optimization. Although we can directly apply stochastic gradient methods of Euclidean space or the natural gradient method (Amari, 1998) , Riemannian gradient methods (e.g. (Zhang & Sra, 2016 ) (Zhang et al., 2016) ) work better for non-Euclidean embedding (Enokida et al., 2018) .

In this paper, we use stocastic Riemannian sub gradient methods Zhang & Sra (2016) with norm clipping (See Appendix). Note that in spheres or hyperbolic spaces, the computation costs of the gradient is O(D), which is as small as TransE. 

4. EXPERIMENTS:

Evaluation Tasks We evaluated the performance of our method for a triple classification task (Socher et al., 2013) on real knowledge base datasets. The triple classification task involved predict-ing whether a triple in the test data is correct. We label a triple positive when f (p h , p t ; ( r , p r )) > θ r , and vice versa. Here, θ r ∈ R ≥0 denotes the threshold for each relation r, which is determined by the accuracy of the validation set. We evaluated the accuracy of classification with the FB13 and WN11 datasets (Socher et al., 2013) . Although we do not report the results of link prediction tasks (Bordes et al., 2013) here because there are many evaluation criteria for the task, which makes it difficult to interpret the results, we report the results in Appendix. Baselines and Implementation We compared our method with the following baselines: RESCAL (Nickel et al., 2011) , TransE (Bordes et al., 2013) , TransH (Wang et al., 2014) , TransR (Lin et al., 2015) , TransD (Ji et al., 2016) , TorusE Ebisu & Ichise (2017) , RESCAL (Nickel et al., 2011) , DISTMULT (Yang et al., 2015) , HolE (Nickel et al., 2016) , ComplEx (Trouillon et al., 2016) and Analogy (Liu et al., 2017) . We used implementations of these methods on the basis of OpenKE http://openke.thunlp.org/static/index.html, and we used the evaluation scripts there. Note that we compensated for some missing constraints (for example, in TransR and TransD) and regularizers (for example, in DISTMULT and Analogy) in OpenKE. We also found that omitting the constraint of the entity planets onto the sphere in TransE gave much better results in our setting, so we also provide these unconstrained results (UnconstraintTransE). We determined the hyperparameters by following each paper. For details, see the Appendix.

Manifolds in Riemannian TransE:



Results:

Table 2 shows the results for the triple classification task in each dimensionality. In WN11, the sphere-based Riemannian TransEs achieved good accuracy. The accuracy did not degrade dramatically even with low dimensionality. On the other hand, in FB13, the hyperbolic-space-based Riemannian TransEs was more accurate than other methods. Moreover for each dimensionality, these results with the proposed Riemannian TransE were at least comparable to those of the baselines. The accuracy of Euclidean-space-based methods (e.g. the original TransE, and Euclidean TransE) are between that of the sphere-based Riemannian TransEs and that of the hyperbolic-spacebased Riemannian TransEs in most cases. Note that these results are compatible with the curvature of each space (i.e. Sphere: positive, Euclidean space: 0, a hyperbolic space: negative). Note that Euclidean methods are sometimes better than non-Euclidean methods. In Appendix, we also report the triple classification task results in FB15k, where Euclidean TransE as well as baseline methods outperformed Riemannian TransE did not always outperform the baseline methods. In summary, positive curvature spaces were good in WN11 and negative curvature spaces were good in FB13, and zero curvature spaces were good in FB15k. These results show that Riemannian TransE can attain good accuracy with small dimensionality provided that an appropriate manifold is selected. What determines the appropriate manifold? Spheres are compatible with cyclic structure and hyperbolic spaces are compatible with tree-like structure. One possible explanation is that WN11 has cyclic structure and FB13 has tree-like structure and the structure of FB15k is between them. However, further discussion remains future work.

5. CONCLUSION AND FUTURE WORK:

We proposed Riemannian TransE, a novel framework for multi-relational graph embedding, by extending TransE to a Riemannian TransE. Numerical experiments showed that Riemannian TransE outperforms baseline methods in low dimensionality, although its performance depends significantly on the choice of manifold. Hence, future research shall clarify which manifolds work well with particular kinds of data, and develop a methodology for choosing the appropriate manifold. This is important work not only for graph completion tasks but also for furthering our understanding of the global characteristics of a graph. In other words, observing which manifold is effective can help us to understand the global "behavior" of a graph. Other important work involves using "subspaces" in non-Euclidean space. Although the notion of a subspace in a non-Euclidean manifold is nontrivial, it may be that our method offers advantages over TransH and TransD, which exploit linear subspaces.

Hongyi Zhang, Sashank J. Reddi, and Suvrit Sra. Riemannian svrg: Fast stochastic optimization on riemannian manifolds. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4592-4600. Curran Associates, Inc., 2016. URL http://papers.nips.cc/paper/ 6515-riemannian-svrg-fast-stochastic-optimization-on-riemannian-manifolds. pdf.

A OPTIMIZATION:

In this paper, we use the following simple (projected) stochastic (Riemannian) (sub-) gradient methods Zhang & Sra ( 2016)

EQUATION

where

θ (τ ) ∈ M |V| × (R × M)

|R| denotes the parameter in the τ -th step, η ∈ R ≥0 is the learning rate, and

∇(τ) ∈ T θ (τ ) M |V| × (R × M) |R| is a stochastic gradient that satisfies E ∇(τ) = gradL θ (τ ) = dL θ (τ )

. Recall that denotes index raising. Specifically, we use the following stochastic loss function based on the mini-batch method:

EQUATION

where the stochastic quintet set

Q (τ ) ⊂ Q is a set of uniform-distributed random variables on Q. ∆ s H r (p h ) , s T r (p t )

. We obtain a stochastic gradient as follows:

EQUATION

where θ is a local coordinate representation of θ. We obtain ∇ (τ ) easily using an automatic differentiation framework. Algorithm 1 shows the learning algorithm for Riemannian TransE. In the experiments, we applied norm clipping such that the norm of a stochastic gradient is smaller than 1.

Algorithm 1 Learning Riemannian TransE:

for τ = 1, 2, . . . do Sample Q (τ ) from uniform distribution on Q. ∇(τ) ← ∂ ∂θ (h,h ,r,t ,t)∈Q [δ + f (p h , pt; wr) -f (p h , p t ; wr)] + θ (τ +1) ← Exp θ (τ ) -η (τ ) ∇(τ)

end for return θ (τ )

B PARALLEL VECTOR FIELDS AND PARALLEL TRANSFORM IN RIEMANNIAN MANIFOLDS:

We give additional explanations of the reason why we cannot define a parallel vector field on a non-Euclidean manifold. Specifically we describe the relationship between parallel vector fields and parallel transform. We can define a parallel transform along a geodesic. This parallel transform maps a tangent vector in a tangent space to one in another. At one glance, it seems that we can define a parallel vector field using the parallel transform. However, a parallel transform is not determined only by the origin and destination but depends on the path i.e. the geodesic. Figure 7 shows an example on a sphere, where two ways to map a vector from a tangent space to another are shown and these two give different maps. As this figure shows we cannot obtain a well-defined vector on more than two points.

Figure 7 : Parallel transforms in a sphere S 2 . This figure shows two ways to transform vector v ∈ T p S 2 to T r S 2 . We denote the parallel transform from along segment pq by Π q p : T p S 2 → T q S 2 . The red vector on T r S 2 denotes the vector obtained by the direct transform along segment pr. The blue vector T r S 2 denotes the vector obtained by the transform via q. As this figure shows we cannot obtain a well-defined vector on more than two points.

C EXAMPLES OF RIEMANNIAN MANIFOLDS:

We introduces some Riemannian manifolds useful in applications, and the formula of the exponential map and logarithmic map in these manifolds. The closed form of exponential map and logarithmic map enables implementation of Riemannian TransE in these manifolds. In the following, we omit symbols ∂ ∂x and d x of the basis in a tangent and cotangent space, respectively, for notation simplicity. Moreover, we give the composition of the exponential map and index raising and that of the index lowering and logarithmic map instead of the exponential map and logarithmic map themselves. This is because we use a cotangent vector rather than a tangent vector in a practical implementation and map from/to cotangent space is more useful (Recall that ∂ ∂θ L is not the coordinate of a tangent but the coordinate of a cotangent vector).

C.1 EUCLIDEAN SPACE:

In a D-dimensional Euclidean Space, the exponential map (with the index raising) Exp p • :

EQUATION

Apparently, the logarithmic map (with the index lowering) • Log p : R D → T * p R D is given by

EQUATION

C.2 SPHERE:

A D-dimensional (unit) sphere is given by point set S D := p ∈ R (D+1) p p = 1 , and the cotangent space T * p S D on p ∈ S D is identified with δ ∈ R (D+1) p δ = 0 . The distance ∆ (p, q) between two points p ∈ S D and q ∈ S D is given as follows:

EQUATION

)

where arccos : [-1, 1] → [0, π] denote arc-cosine function. The exponential map (with the index raising) Exp p • : T * p S D → S D is given by

EQUATION

)

where sinc denotes the cardinal sine function defined as follows:

EQUATION

)

The logarithmic map (with the index lowering) • Log p : S D → T * p S D is given by

EQUATION

)

Note that in optimization, we need the projection of the differential δ = ∂ ∂θ L (θ)| θ=p of the loss function L to cotangent vector δ given by: δ = δp δ p.

(16)

C.3 HYPERBOLIC SPACE:

In this subsection, we introduces models of a hyperbolic space, which are mathematically equivalent to each other, but have practically different aspects. There are many models of a hyperbolic space.

We introduce two of them: the hyperboloid model and Poincaré disk model.

C.3.1 HYPERBOLOID MODEL:

Some formulae here are also given and used in Nickel & Kiela (2018) . Let G M denote diagonal matrix

EQUATION

)

In the hyperboloid model, a (canonical) hyperbolic space is given by point set H D := p ∈ R D+1 p, p M = -1, p 0 > 0 . The tangent space T p H D on p ∈ H D is identified with δ ∈ R D+1 p, δ = 0 , and the metric g p : T p H D × T p H D → R in the tangent space is given by g p (u, v) = u, v M . Hence, the cotangent space T * p H D on p ∈ H D is identified with δ ∈ R D+1 p δ = 0 , and the metric g * p :

T * p H D × T * p H D → R in the cotangent space is given by g * p (γ, δ) = γ, δ M . Note that δ ∈ T * p H D is identified with δ = G -1 M δ ∈ T p H D .

The distance ∆ (p, q) between two points p ∈ H D and q ∈ H D is given as follows:

EQUATION

where, arcosh : [1, ∞) → [0, ∞) denotes the area hyperbolic cosine function, i.e. the inverse fucntion of the hyperbolic cosine function. The exponential map (with the index raising) Exp p • :

EQUATION

)

where sinhc denotes the hyperbolic sine cardinal function defined as follows:

EQUATION

The logarithmic map (with the index lowering) • Log p : H D → T * p H D is given by

EQUATION

Under review as a conference paper at ICLR 2019 Note that in optimization, we need the projection of the differential δ = ∂ ∂θ L (θ)| θ=p of the loss function L to cotangent vector δ given by: 

EQUATION

EQUATION

where

µ p := 1-p p 2 .

The metric is given by g

p (u, v) = 2 1-p p 2 u v.

The distance ∆ (p, q) between two points p ∈ H D and q ∈ H D is given as follows:

EQUATION

where

EQUATION

The exponential map (with index raising) Exp p • :

EQUATION

where

EQUATION

The logarithmic map (with index lowering)

EQUATION

These formulae can be obtained by the coordinate transformation and can be interpreted as a modification of existing formulae such as ones in Ganea et al. (2018a) . In addition, these formulae are useful in an automatic differentiation system, because sech √ x, tanhc √ x, and arcosh(1+x) √

x , and their derivatives do not diverge when x → 0.

D DETAILS OF EXPERIMENTS D.1 EVALUATION TASKS:

We evaluated the performance of our method in the link prediction Bordes et al. (2013) and task and the triple classification task Socher et al. (2013) on real knowledge base data sets.

D.1.1 LINK PREDICTION TASK:

In the link prediction task, we predict the head or the tail entity given the relation type and the other entity. We evaluate the ranking of each correct test triple (h, r, t) in the corrupted triples. We corrupt each triple as follows. In our setting, either its head or tail is replaced by one of the possible head or entity, respectively. In addition, we applied "filtered" setting proposed by Bordes et al. (2013) , where the correct triples, that is, the triples T in the original multi-relational graph are excluded. Thus, the corrupted triples are given by (h , r, t

) h ∈ V h ∧ (h , r, t) / ∈ T (head corruption) or {(h, r, t ) | t ∈ V t ∧ (h, r, t ) /

∈ T } (tail corruption). where V h r and V t r denote the possible heads and tails in relation r, given as follows:

EQUATION

As evaluation metrics, we use the following:

Mean rank (MR) the mean rank of the correct test triples. The value of this metric is always equal to or greater than 1, and the lower, the better. Hits @ n (@n) the propotion of correct triples ranked in the top n predictions (n = 1, 3, 10). The value ranges from 0 to 1, and the higher, the better. Mean reciprocal rank (MRR) the mean of the reciprocal rank of the correct test triples. The value ranges from 0 to 1, and the higher, the better.

D.1.2 TRIPLE CLASSIFICATION TASK:

In triple classification tasks, we predict whether a triple in the test data is correct or not. The classification is simply based on the score function i.e. we label a triple positive when f (p h , p t ; ( r , p r )) > θ r , and the other way around. Here, θ r ∈ R ≥0 denotes the threshold for each relation r, which is determined by the accuracy in the validation set.

D.2 DATASETS:

In link prediction tasks, we used WN18 and FB15k Bordes et al. (2013) datasets, and WN11 and FB13 datasets Socher et al. (2013) . In triple classification tasks, we used WN11 and FB13 datasets, as well as FB15k. Note that WN18 and FB15k are originally used for link prediction tasks, whereas WN11 and FB13 are originally used for triple classification tasks. Also note that WN18 cannot be used for the triple classification task because WN18 does not have test negative data. Table 3 shows the number of the entities, relations, and triples in each dataset. 

D.3 BASELINES AND IMPLEMENTATION:

We compared our method with baselines. As baselines, we used RESCAL Nickel et al. (2011 ), TransE Bordes et al. (2013 ), TransH Wang et al. (2014 ), TransR Lin et al. (2015 ), TransD Ji et al. (2016) , DISTMULT Yang et al. (2015) , HolE Nickel et al. (2016) and ComplEx Trouillon et al. (2016) . We used implementations of the baselines in OpenKE http://openke.thunlp. org/static/index.html, a Python library of knowledge base embedding based on Tensorflow Abadi et al. (2015) , and moreover, we implemented some lacked constraints (for example, in TransR, TransD) and regularizers (for example, in DistMult, Analogy) in OpenKE. We also found that omitting the constraint of the entity planets onto sphere in TransE gives much better results in our setting, and this is why we also show the result without the constraint (UnconstraintTransE). We also implemented Riemannian TransEs as derivations of the base class of OpenKE.

We set the dimensionality of the entity manifold as D = 8, 16, 32, 64, 128. Although we also have to determine the dimensionality of the projected space in TransR and TransD, we let them be equal to D. Due to limitation of the computational costs, we fixed the batch size in baselines and Riemannian TransEs such that that the training data are split to 100 batches. We also fixed the number of epochs to 1000. Note that in the first 100 epochs in Riemannian TransEs, we fixed the launchers. Also note that we applied norm clipping such that the norm of a stochastic gradient in the tangent space is smaller than 1. We did not use "bern" setting introduced in Wang et al. (2014) , where the ratio between head and tail corruption is not fixed to one to one; in other words, we replaced head and tail with equal probability.

Other than the dimensionality and batch sizes, we used hyperparameters such as learning rate η and margin paremeter δ of baselines used in each paper. Note that some methods only reports link prediction tasks, and reports hyperparameters for WN18 and FB15k and do not reports ones for WN11 and FB13. Some methods do not mention settings of hyperparameters, and in these cases, we used the default parameters in OpenKE. In these cases, we used hyperparameters of WN18 and FB15k also for WN11 and FB13, respectively. Note that the parameters of TorusE is supposed to be used with very high dimensionality, and the hyperparameters are designed for high dimensionality settings. In Riemannian TransEs, we simply followed the hyperparameters in TransE.

We used the Xavier initializer Glorot & Bengio (2010) as an initializer. When we have to use the points on a sphere (in the original TransE and Spherical TransEs), we projected the points generated by the initialization onto the sphere. We found that choice of an initializer has significant effect on embedding performance, and the Xavier initializer achieves very good performance.

We selected optimizers in baselines following each paper. Note that while using ADADELTA (Zeiler, 2012) is also proposed in TransD, we used SGD in TransD. In Riemannian TransEs, we used we simply followed the hyperparameters in TransE. Table 4 shows the hyperparameters and optimization method for each method. Differences between criteria are also interesting phenomena. Note that MRR and hit@10 is generous for heavy mistakes. It is possible that inner-product-based methods earn good scores in trivial relations, but further intensive investigation is needed. 

:



Figures and Tables:

Figure 3: Tangent space and exponential map. The exponential map moves the point p ∈ M along a geodesic (the white line) that tangent to v ∈ T p M.

Figure4: Parallel vector field in a sphere. The left figure shows a parallel vector field in a plane. In a sphere, there is no parallel vector field. Even if a vector field seems parallel from one view (center), it turns out to be not parallel(right)

Figure 6: Relation of the sign for . If is positive (e.g. the orange relation), the relation runs from low (e.g. Entity 2 and 3) to high hierarchy (e.g. Entity 1), and vice versa (e.g. the red relation).

Figure 5 (right) shows the Riemannian TransE model. If M = R D and the attraction points are at infinity, the score function is equivalent to that of TransE (without the sphere constraint).

To evaluate the dependency of performance for Riemannian TransE, we compared Riemannian TransE using the following five kinds of manifolds: Euclidean space R D (Euclidean TransE), hyperbolic space H D (Hyperbolic TransE), a sphere S D (Spherical TransE), the direct product H 4 × H 4 × • • • × H 4 of hyperbolic space (PHyperbolic TransE), and the direct product S 4 × S 4 × • • • × S 4 of a sphere (PSpherical TransE).

.2 POINCAR É DISK MODEL In the Poincaré disk model, the D-dimensional hyperbolic space is given by the unit open hyper-ball D D := p ∈ R D p p < 1 . The Poincaré disk model and the hyperboloid model are derived from each other by the following map:

Triple classification performance. Bold: Top 1, Italic: Top 3.

Table content: <table><tr><td>Dataset</td><td/><td/><td>WN11</td><td/><td/><td/><td/><td>FB13</td><td/><td/></tr><tr><td>Dim.</td><td>8</td><td>16</td><td>32</td><td>64</td><td>128</td><td>8</td><td>16</td><td>32</td><td>64</td><td>128</td></tr><tr><td>Hyperbolic TransE</td><td>64.74</td><td>66.51</td><td>67.78</td><td>67.92</td><td>67.87</td><td>80.05</td><td>78.06</td><td>77.53</td><td>84.65</td><td>84.67</td></tr><tr><td>PHyperbolic TransE</td><td>68.51</td><td>72.88</td><td>74.70</td><td>75.83</td><td>77.03</td><td>78.42</td><td>77.06</td><td>77.39</td><td>77.74</td><td>78.53</td></tr><tr><td>Spherical TransE</td><td>82.07</td><td>83.11</td><td>82.99</td><td>83.13</td><td>83.30</td><td>64.45</td><td>63.38</td><td>64.69</td><td>70.07</td><td>69.74</td></tr><tr><td>PSpherical TransE</td><td>80.73</td><td>81.37</td><td>77.12</td><td>69.05</td><td>63.42</td><td>71.26</td><td>71.34</td><td>71.23</td><td>73.03</td><td>74.83</td></tr><tr><td>Euclidean TransE</td><td>72.66</td><td>73.99</td><td>75.27</td><td>76.69</td><td>77.04</td><td>81.84</td><td>80.03</td><td>75.44</td><td>76.99</td><td>77.52</td></tr><tr><td>TransE</td><td>60.94</td><td>64.63</td><td>63.20</td><td>61.92</td><td>58.46</td><td>67.60</td><td>68.29</td><td>68.86</td><td>75.68</td><td>74.92</td></tr><tr><td>TransE (unconstraint)</td><td>67.55</td><td>66.18</td><td>64.07</td><td>63.23</td><td>61.51</td><td>76.44</td><td>80.22</td><td>77.24</td><td>76.01</td><td>75.59</td></tr><tr><td>TorusE</td><td>62.34</td><td>62.78</td><td>63.33</td><td>63.19</td><td>63.45</td><td>61.51</td><td>58.04</td><td>63.06</td><td>60.31</td><td>58.14</td></tr><tr><td>TransH</td><td>77.55</td><td>75.44</td><td>70.03</td><td>65.46</td><td>63.75</td><td>71.07</td><td>75.25</td><td>76.89</td><td>78.32</td><td>80.32</td></tr><tr><td>TransR</td><td>52.58</td><td>53.13</td><td>55.30</td><td>53.10</td><td>55.80</td><td>57.43</td><td>52.41</td><td>51.65</td><td>51.87</td><td>52.38</td></tr><tr><td>TransD</td><td>53.43</td><td>54.61</td><td>55.76</td><td>63.32</td><td>61.59</td><td>55.02</td><td>56.68</td><td>53.69</td><td>56.28</td><td>56.02</td></tr><tr><td>RESCAL</td><td>60.36</td><td>57.65</td><td>56.85</td><td>56.62</td><td>57.62</td><td>74.28</td><td>70.17</td><td>67.88</td><td>65.90</td><td>63.20</td></tr><tr><td>DistMult</td><td>61.05</td><td>61.01</td><td>58.97</td><td>57.11</td><td>55.85</td><td>64.54</td><td>65.04</td><td>63.32</td><td>59.77</td><td>54.76</td></tr><tr><td>ComplEx</td><td>62.63</td><td>62.47</td><td>57.91</td><td>56.02</td><td>53.47</td><td>70.07</td><td>72.36</td><td>71.11</td><td>67.36</td><td>64.49</td></tr><tr><td>HolE</td><td>53.01</td><td>53.03</td><td>51.19</td><td>52.62</td><td>53.09</td><td>58.12</td><td>62.13</td><td>61.35</td><td>60.74</td><td>54.61</td></tr><tr><td>Analogy</td><td>63.60</td><td>59.24</td><td>58.55</td><td>57.51</td><td>57.00</td><td>66.38</td><td>66.18</td><td>64.54</td><td>59.48</td><td>55.26</td></tr></table>

Statistics of the experimental datasets Manifolds in Riemannian TransE To evaluate the dependency of performance of Riemannian TransE, we compared Riemannian TransE using the following five kinds of manifolds: Euclidean space R D (Euclidean TransE), hyperbolic spaces H D (HyperbolicTransE), spheres S D (Spherical-TransE), the direct product H 4 × H 4 × • • • × H 4 of hyperbolic spaces (PHyperbolicTransE), and the direct product S 4 × S 4 × • • • × S 4 of spheres (PSphericalTransE).

Table content: <table><tr><td>Dataset</td><td>|V|</td><td>|R|</td><td>train</td><td># triples valid</td><td>test</td></tr><tr><td>WN18</td><td>40943</td><td colspan="2">18 141442</td><td>5000</td><td>5000</td></tr><tr><td>FB15k</td><td colspan="5">14951 1345 483142 50000 59071</td></tr><tr><td>WN11</td><td>38696</td><td colspan="2">11 112581</td><td colspan="2">2609 10544</td></tr><tr><td>FB13</td><td>70543</td><td colspan="2">13 316232</td><td colspan="2">5908 23733</td></tr></table>

Hyperparameters and optimizers: SGD denotes the stochastic gradient descent method (in a Euclidean space). SRGD denotes the stochastic Riemannian gradient descent methodZhang & Sra (2016) with gradient clipping. Adagrad is proposed byDuchi et al. (2011).

Table content: <table><tr><td colspan="2">Method Optimizer</td><td/><td colspan="2">Learning rate η</td><td/><td/><td>Margin δ</td><td/><td/></tr><tr><td/><td/><td colspan="8">WN18 FB15k WN11 FB13 WN18 FB15k WN11 FB13</td></tr><tr><td>RiemannianTransEs</td><td>SRGD</td><td>0.01</td><td>0.01</td><td>0.01</td><td>0.01</td><td>2.0</td><td>1.0</td><td>2.0</td><td>1.0</td></tr><tr><td>TransE</td><td>SGD</td><td>0.01</td><td>0.01</td><td>0.01</td><td>0.01</td><td>2.0</td><td>1.0</td><td>2.0</td><td>1.0</td></tr><tr><td>TransH</td><td>SGD</td><td>0.01</td><td>0.005</td><td colspan="2">0.001 0.005</td><td>1.0</td><td>0.5</td><td>2.0</td><td>0.25</td></tr><tr><td>TransR</td><td>SGD</td><td>0.01</td><td>0.005</td><td/><td/><td/><td/><td/><td/></tr></table>

Table 7, and Table8shows hit@10, mean rank, and mean reciprocal rank score of link prediction tasks, respectively. As in triple classification tasks, the sphere-based Riemannian TransEs achieved good accuracy in WN11, whereas the hyperbolic-space-based Riemannian TransEs was more accurate than other methods in FB13. The Riemannian TransEs did not outperform the baselines in WN18 and FB15k. This tendency is apparent in MR score. The distance-based methods such as TransE, TransH and Riemannian TransEs tend to attain good scores in MR and the inner-product-based methods such as DistMult, ComplEx and Analogy tend to attain good scores in MRR and hit@10.D.5 ADDITIONAL DISCUSSIONWhy do these baselines attain good results in WN18 and FB15k but bad results in WN11 and FB13? One reason may simply be that WN18 and FB15k datasets have good compatibility with zero curvature spaces i.e. Euclidean space. This is supported by the results of Euclidean TransE. A possible second reason is the redundancy of FB15k. Whereas some "easy" relations are excluded from FB15kBordes et al. (2013), it still contain many reversible triples, as noted byToutanova & Chen (2015). By contrast, these are removed in WN11 and FB13. Recall that projection-based methods such as TransH, TransR and TransD, and inner-product-based methods such as ComplEx and DISTMULT can exploit a linear subspace. When a dataset has apparent clusters inside which one relation is easily recovered from the others, we can allocate each cluster to a subspace and separate subspaces from one another. This separation is easily realized by setting some elements in the launchers to zero in these methods. Indeed, the TransE without the sphere constraint attains good accuracies in WN11 and FB13.

Table content: <table/>

Triple classification performance. Bold: Top 1, Italic: Top 3.

Table content: <table><tr><td>Dataset</td><td/><td/><td>FB15K</td><td/><td/></tr><tr><td>Dim.</td><td>8</td><td>16</td><td>32</td><td>64</td><td>128</td></tr><tr><td>Hyperbolic TransE</td><td>75.46</td><td>76.86</td><td>77.34</td><td>77.73</td><td>77.87</td></tr><tr><td>PHyperbolic TransE</td><td>76.78</td><td>81.40</td><td>85.89</td><td>89.33</td><td>91.13</td></tr><tr><td>Spherical TransE</td><td>68.43</td><td>68.36</td><td>70.12</td><td>68.51</td><td>70.28</td></tr><tr><td>PSpherical TransE</td><td>74.38</td><td>79.73</td><td>84.31</td><td>88.39</td><td>90.31</td></tr><tr><td>Euclidean TransE</td><td>79.46</td><td>83.31</td><td>87.22</td><td>90.11</td><td>91.52</td></tr><tr><td>TransE</td><td>74.02</td><td>78.72</td><td>81.05</td><td>80.50</td><td>76.67</td></tr><tr><td>TransE (unconstraint)</td><td>78.05</td><td>81.72</td><td>84.42</td><td>85.45</td><td>84.74</td></tr><tr><td>TorusE</td><td>56.17</td><td>56.09</td><td>56.15</td><td>56.10</td><td>56.22</td></tr><tr><td>TransH</td><td>78.10</td><td>82.74</td><td>85.83</td><td>87.33</td><td>87.82</td></tr><tr><td>TransR</td><td>69.85</td><td>75.09</td><td>77.65</td><td>78.01</td><td>75.53</td></tr><tr><td>TransD</td><td>56.44</td><td>60.11</td><td>63.12</td><td>66.17</td><td>71.87</td></tr><tr><td>RESCAL</td><td>77.66</td><td>81.36</td><td>84.08</td><td>83.71</td><td>81.10</td></tr><tr><td>DistMult</td><td>77.13</td><td>82.89</td><td>88.19</td><td>89.64</td><td>89.90</td></tr><tr><td>ComplEx</td><td>78.72</td><td>85.66</td><td>89.22</td><td>90.37</td><td>89.75</td></tr><tr><td>HolE</td><td>68.87</td><td>73.61</td><td>78.37</td><td>83.80</td><td>86.12</td></tr><tr><td>Analogy</td><td>76.41</td><td>83.87</td><td>88.23</td><td>89.75</td><td>90.13</td></tr></table>

hit@10 in link prediction task. Bold: Top 1, Italic: Top 3.

Table content: <table><tr><td>WN18 (hit@10) / dim</td><td>8</td><td>16</td><td>32</td><td>64</td><td>128</td></tr><tr><td>Hyperbolic TransE</td><td>21.73</td><td>31.88</td><td>38.09</td><td>41.55</td><td>40.93</td></tr><tr><td>PHyperbolic TransE</td><td>29.18</td><td>75.03</td><td>85.15</td><td>86.52</td><td>87.56</td></tr><tr><td>Spherical TransE</td><td>32.16</td><td>31.38</td><td>33.63</td><td>34.80</td><td>36.18</td></tr><tr><td>PSpherical TransE</td><td>30.13</td><td>55.56</td><td>85.82</td><td>92.83</td><td>93.22</td></tr><tr><td>Euclidean TransE</td><td>38.37</td><td>75.05</td><td>84.84</td><td>86.50</td><td>87.17</td></tr><tr><td>TransE</td><td>12.35</td><td>17.80</td><td>19.56</td><td>17.00</td><td>11.19</td></tr><tr><td>TransE (unconstraint)</td><td>37.36</td><td>66.23</td><td>86.34</td><td>88.82</td><td>86.94</td></tr><tr><td>TorusE</td><td>19.45</td><td>30.04</td><td>31.62</td><td>36.52</td><td>36.85</td></tr><tr><td>TransH</td><td>42.11</td><td>76.36</td><td>88.34</td><td>92.41</td><td>90.72</td></tr><tr><td>TransR</td><td>00.89</td><td>01.32</td><td>04.95</td><td>18.57</td><td>43.19</td></tr><tr><td>TransD</td><td>03.23</td><td>10.64</td><td>23.65</td><td>66.14</td><td>92.14</td></tr><tr><td>RESCAL</td><td>14.91</td><td>39.31</td><td>74.80</td><td>81.98</td><td>77.26</td></tr><tr><td>DistMult</td><td>15.69</td><td>63.18</td><td>93.95</td><td>94.09</td><td>94.08</td></tr><tr><td>ComplEx</td><td>19.29</td><td>73.79</td><td>93.99</td><td>94.20</td><td>93.88</td></tr><tr><td>HolE</td><td>10.54</td><td>08.64</td><td>15.00</td><td>28.59</td><td>81.44</td></tr><tr><td>Analogy</td><td>12.77</td><td>70.67</td><td>94.20</td><td>94.22</td><td>94.33</td></tr><tr><td>FB15K (hit@10) / dim</td><td>8</td><td>16</td><td>32</td><td>64</td><td>128</td></tr><tr><td>Hyperbolic TransE</td><td>44.45</td><td>48.70</td><td>50.85</td><td>51.94</td><td>52.37</td></tr><tr><td>PHyperbolic TransE</td><td>43.17</td><td>51.67</td><td>61.38</td><td>71.86</td><td>79.35</td></tr><tr><td>Spherical TransE</td><td>34.92</td><td>34.98</td><td>37.60</td><td>38.47</td><td>39.98</td></tr><tr><td>PSpherical TransE</td><td>40.17</td><td>49.38</td><td>59.45</td><td>70.48</td><td>77.88</td></tr><tr><td>Euclidean TransE</td><td>45.52</td><td>53.82</td><td>64.15</td><td>74.91</td><td>81.11</td></tr><tr><td>TransE</td><td>39.10</td><td>47.01</td><td>53.82</td><td>55.85</td><td>51.86</td></tr><tr><td>TransE (unconstraint)</td><td>44.18</td><td>51.86</td><td>60.42</td><td>67.09</td><td>69.80</td></tr><tr><td>TorusE</td><td>18.85</td><td>19.64</td><td>20.29</td><td>19.71</td><td>19.51</td></tr><tr><td>TransH</td><td>42.99</td><td>52.06</td><td>61.21</td><td>70.48</td><td>75.29</td></tr><tr><td>TransR</td><td>31.49</td><td>40.40</td><td>47.44</td><td>50.41</td><td>49.34</td></tr><tr><td>TransD</td><td>20.79</td><td>25.35</td><td>31.13</td><td>37.71</td><td>44.01</td></tr><tr><td>RESCAL</td><td>44.50</td><td>51.16</td><td>56.21</td><td>58.22</td><td>53.92</td></tr><tr><td>DistMult</td><td>38.58</td><td>45.69</td><td>58.86</td><td>74.09</td><td>83.37</td></tr><tr><td>ComplEx</td><td>39.05</td><td>49.87</td><td>62.92</td><td>79.96</td><td>81.59</td></tr><tr><td>HolE</td><td>35.50</td><td>39.75</td><td>45.13</td><td>55.86</td><td>63.45</td></tr><tr><td>Analogy</td><td>38.16</td><td>47.01</td><td>58.85</td><td>74.50</td><td>83.49</td></tr><tr><td>WN11 (hit@10) / dim</td><td>8</td><td>16</td><td>32</td><td>64</td><td>128</td></tr><tr><td>Hyperbolic TransE</td><td>08.97</td><td>13.41</td><td>15.23</td><td>16.43</td><td>15.69</td></tr><tr><td>PHyperbolic TransE</td><td>08.74</td><td>14.48</td><td>17.93</td><td>19.42</td><td>20.62</td></tr><tr><td>Spherical TransE</td><td>11.29</td><td>11.65</td><td>12.07</td><td>11.07</td><td>13.41</td></tr><tr><td>PSpherical TransE</td><td>11.86</td><td>17.61</td><td>17.69</td><td>12.18</td><td>07.98</td></tr><tr><td>Euclidean TransE</td><td>10.47</td><td>14.65</td><td>17.80</td><td>19.87</td><td>20.57</td></tr><tr><td>TransE</td><td>03.28</td><td>05.71</td><td>06.45</td><td>05.14</td><td>04.34</td></tr><tr><td>TransE (unconstraint)</td><td>10.32</td><td>09.75</td><td>09.80</td><td>09.62</td><td>08.38</td></tr><tr><td>TorusE</td><td>07.24</td><td>09.28</td><td>10.49</td><td>10.85</td><td>10.05</td></tr><tr><td>TransH</td><td>16.80</td><td>18.25</td><td>13.17</td><td>08.80</td><td>07.38</td></tr><tr><td>TransR</td><td>00.77</td><td>01.17</td><td>01.72</td><td>01.10</td><td>02.07</td></tr><tr><td>TransD</td><td>00.83</td><td>01.42</td><td>02.60</td><td>05.31</td><td>04.81</td></tr><tr><td>RESCAL</td><td>03.61</td><td>02.94</td><td>03.12</td><td>03.19</td><td>03.21</td></tr><tr><td>DistMult</td><td>02.32</td><td>03.62</td><td>03.60</td><td>02.85</td><td>02.46</td></tr><tr><td>ComplEx</td><td>02.78</td><td>04.21</td><td>03.03</td><td>02.35</td><td>01.61</td></tr><tr><td>HolE</td><td>04.02</td><td>03.12</td><td>30.55</td><td>01.27</td><td>01.30</td></tr><tr><td>Analogy</td><td>03.92</td><td>03.04</td><td>03.22</td><td>03.12</td><td>02.50</td></tr><tr><td>FB13 (hit@10) / dim</td><td>8</td><td>16</td><td>32</td><td>64</td><td>128</td></tr><tr><td>Hyperbolic TransE</td><td>31.11</td><td>35.18</td><td>37.20</td><td>38.38</td><td>39.33</td></tr><tr><td>PHyperbolic TransE</td><td>34.05</td><td>37.44</td><td>39.15</td><td>39.90</td><td>40.79</td></tr><tr><td>Spherical TransE</td><td>25.12</td><td>28.69</td><td>29.73</td><td>42.87</td><td>45.47</td></tr><tr><td>PSpherical TransE</td><td>32.60</td><td>33.81</td><td>33.00</td><td>33.29</td><td>33.55</td></tr><tr><td>Euclidean TransE</td><td>31.85</td><td>33.91</td><td>36.36</td><td>38.43</td><td>39.49</td></tr><tr><td>TransE</td><td>21.57</td><td>27.39</td><td>23.37</td><td>29.01</td><td>31.40</td></tr><tr><td>TransE (unconstraint)</td><td>32.64</td><td>34.39</td><td>32.21</td><td>31.86</td><td>32.04</td></tr><tr><td>TorusE</td><td>16.66</td><td>17.22</td><td>16.63</td><td>17.49</td><td>17.63</td></tr><tr><td>TransH</td><td>18.28</td><td>22.50</td><td>26.82</td><td>28.94</td><td>29.84</td></tr><tr><td>TransR</td><td>14.29</td><td>12.80</td><td>13.21</td><td>13.65</td><td>13.78</td></tr><tr><td>TransD</td><td>15.31</td><td>13.31</td><td>15.39</td><td>17.51</td><td>18.23</td></tr><tr><td>RESCAL</td><td>33.58</td><td>32.18</td><td>28.88</td><td>26.46</td><td>23.48</td></tr><tr><td>DistMult</td><td>23.54</td><td>22.36</td><td>21.71</td><td>19.84</td><td>17.88</td></tr><tr><td>ComplEx</td><td>26.26</td><td>27.89</td><td>28.11</td><td>27.13</td><td>23.99</td></tr><tr><td>HolE</td><td>27.64</td><td>30.74</td><td>30.24</td><td>26.03</td><td>18.05</td></tr><tr><td>Analogy</td><td>23.27</td><td>23.29</td><td>22.58</td><td>19.46</td><td>17.77</td></tr></table>

MRR in link prediction task. Bold: Top 1, Italic: Top 3.

Table content: <table><tr><td>WN18 (MRR) / dim</td><td>8</td><td>16</td><td>32</td><td>64</td><td>128</td></tr><tr><td>Hyperbolic TransE</td><td>11.07</td><td>16.55</td><td>19.59</td><td>21.00</td><td>20.49</td></tr><tr><td>PHyperbolic TransE</td><td>13.77</td><td>36.26</td><td>42.57</td><td>44.18</td><td>44.94</td></tr><tr><td>Spherical TransE</td><td>16.42</td><td>16.22</td><td>19.12</td><td>17.81</td><td>19.04</td></tr><tr><td>PSpherical TransE</td><td>15.49</td><td>29.82</td><td>49.00</td><td>56.07</td><td>55.92</td></tr><tr><td>Euclidean TransE</td><td>17.87</td><td>35.64</td><td>42.01</td><td>44.15</td><td>44.65</td></tr><tr><td>TransE</td><td>06.82</td><td>09.70</td><td>11.05</td><td>09.35</td><td>06.45</td></tr><tr><td>TransE (unconstraint)</td><td>18.73</td><td>38.72</td><td>54.88</td><td>55.44</td><td>53.00</td></tr><tr><td>TorusE</td><td>11.72</td><td>21.54</td><td>23.25</td><td>28.56</td><td>29.22</td></tr><tr><td>TransH</td><td>22.45</td><td>39.07</td><td>49.83</td><td>57.85</td><td>58.10</td></tr><tr><td>TransR</td><td>00.58</td><td>00.65</td><td>02.31</td><td>08.19</td><td>18.27</td></tr><tr><td>TransD</td><td>01.52</td><td>05.38</td><td>13.30</td><td>39.37</td><td>55.03</td></tr><tr><td>RESCAL</td><td>08.58</td><td>20.78</td><td>46.32</td><td>61.98</td><td>60.63</td></tr><tr><td>DistMult</td><td>08.14</td><td>34.52</td><td>76.38</td><td>83.72</td><td>83.67</td></tr><tr><td>ComplEx</td><td>09.72</td><td>43.12</td><td>80.42</td><td>92.25</td><td>92.90</td></tr><tr><td>HolE</td><td>07.27</td><td>06.02</td><td>08.20</td><td>17.00</td><td>58.33</td></tr><tr><td>Analogy</td><td>06.83</td><td>41.48</td><td>81.11</td><td>93.22</td><td>93.99</td></tr><tr><td>FB15K (MRR) / dim</td><td>8</td><td>16</td><td>32</td><td>64</td><td>128</td></tr><tr><td>Hyperbolic TransE</td><td>26.60</td><td>29.92</td><td>31.79</td><td>32.74</td><td>32.94</td></tr><tr><td>PHyperbolic TransE</td><td>25.81</td><td>31.14</td><td>38.23</td><td>47.36</td><td>55.43</td></tr><tr><td>Spherical TransE</td><td>20.05</td><td>21.17</td><td>22.82</td><td>24.09</td><td>24.83</td></tr><tr><td>PSpherical TransE</td><td>23.55</td><td>29.33</td><td>36.31</td><td>46.20</td><td>54.63</td></tr><tr><td>Euclidean TransE</td><td>26.65</td><td>32.36</td><td>40.24</td><td>49.91</td><td>57.24</td></tr><tr><td>TransE</td><td>22.48</td><td>27.91</td><td>32.56</td><td>33.98</td><td>30.79</td></tr><tr><td>TransE (unconstraint)</td><td>26.53</td><td>31.76</td><td>37.28</td><td>42.26</td><td>45.18</td></tr><tr><td>TorusE</td><td>10.66</td><td>11.16</td><td>11.40</td><td>11.04</td><td>11.43</td></tr><tr><td>TransH</td><td>24.68</td><td>30.25</td><td>36.62</td><td>44.21</td><td>50.79</td></tr><tr><td>TransR</td><td>18.27</td><td>23.60</td><td>27.59</td><td>29.71</td><td>28.73</td></tr><tr><td>TransD</td><td>12.00</td><td>14.89</td><td>19.07</td><td>23.65</td><td>27.36</td></tr><tr><td>RESCAL</td><td>27.84</td><td>32.19</td><td>35.06</td><td>36.19</td><td>33.14</td></tr><tr><td>DistMult</td><td>23.30</td><td>27.45</td><td>36.07</td><td>47.29</td><td>60.70</td></tr><tr><td>ComplEx</td><td>22.93</td><td>29.24</td><td>37.77</td><td>51.84</td><td>60.47</td></tr><tr><td>HolE</td><td>26.14</td><td>28.44</td><td>33.31</td><td>41.68</td><td>46.73</td></tr><tr><td>Analogy</td><td>23.29</td><td>28.06</td><td>35.81</td><td>47.22</td><td>61.53</td></tr><tr><td>WN11 (MRR) / dim</td><td>8</td><td>16</td><td>32</td><td>64</td><td>128</td></tr><tr><td>Hyperbolic TransE</td><td>04.57</td><td>06.86</td><td>07.71</td><td>08.18</td><td>07.87</td></tr><tr><td>PHyperbolic TransE</td><td>04.42</td><td>07.05</td><td>08.72</td><td>09.48</td><td>09.65</td></tr><tr><td>Spherical TransE</td><td>05.67</td><td>06.03</td><td>06.38</td><td>05.95</td><td>07.02</td></tr><tr><td>PSpherical TransE</td><td>06.16</td><td>08.86</td><td>08.56</td><td>06.37</td><td>04.37</td></tr><tr><td>Euclidean TransE</td><td>05.35</td><td>07.22</td><td>08.56</td><td>09.36</td><td>09.50</td></tr><tr><td>TransE</td><td>01.66</td><td>02.96</td><td>03.31</td><td>02.67</td><td>02.31</td></tr><tr><td>TransE (unconstraint)</td><td>05.55</td><td>05.31</td><td>05.44</td><td>05.22</td><td>04.42</td></tr><tr><td>TorusE</td><td>03.66</td><td>04.84</td><td>05.98</td><td>06.30</td><td>05.62</td></tr><tr><td>TransH</td><td>10.22</td><td>09.88</td><td>06.61</td><td>04.51</td><td>03.82</td></tr><tr><td>TransR</td><td>00.49</td><td>00.73</td><td>01.03</td><td>00.64</td><td>01.05</td></tr><tr><td>TransD</td><td>00.55</td><td>00.85</td><td>01.50</td><td>02.82</td><td>02.52</td></tr><tr><td>RESCAL</td><td>01.93</td><td>01.56</td><td>01.81</td><td>01.88</td><td>01.91</td></tr><tr><td>DistMult</td><td>01.16</td><td>01.92</td><td>01.93</td><td>01.58</td><td>01.36</td></tr><tr><td>ComplEx</td><td>01.33</td><td>02.22</td><td>01.65</td><td>01.28</td><td>00.84</td></tr><tr><td>HolE</td><td>03.79</td><td>02.92</td><td>30.57</td><td>00.83</td><td>00.71</td></tr><tr><td>Analogy</td><td>01.91</td><td>01.58</td><td>01.77</td><td>01.67</td><td>01.32</td></tr><tr><td>FB13 (MRR) / dim</td><td>8</td><td>16</td><td>32</td><td>64</td><td>128</td></tr><tr><td>Hyperbolic TransE</td><td>20.60</td><td>22.23</td><td>24.51</td><td>27.36</td><td>28.61</td></tr><tr><td>PHyperbolic TransE</td><td>20.83</td><td>21.53</td><td>23.52</td><td>24.29</td><td>25.29</td></tr><tr><td>Spherical TransE</td><td>14.35</td><td>16.24</td><td>16.94</td><td>30.67</td><td>33.05</td></tr><tr><td>PSpherical TransE</td><td>18.77</td><td>18.60</td><td>18.97</td><td>19.64</td><td>21.81</td></tr><tr><td>Euclidean TransE</td><td>22.84</td><td>24.08</td><td>22.54</td><td>23.48</td><td>24.05</td></tr><tr><td>TransE</td><td>13.71</td><td>16.62</td><td>16.48</td><td>21.66</td><td>23.07</td></tr><tr><td>TransE (unconstraint)</td><td>22.12</td><td>24.58</td><td>22.85</td><td>22.86</td><td>22.90</td></tr><tr><td>TorusE</td><td>11.37</td><td>09.35</td><td>12.98</td><td>11.61</td><td>09.58</td></tr><tr><td>TransH</td><td>12.73</td><td>14.54</td><td>17.22</td><td>19.76</td><td>21.05</td></tr><tr><td>TransR</td><td>12.48</td><td>07.58</td><td>08.13</td><td>09.06</td><td>09.42</td></tr><tr><td>TransD</td><td>12.44</td><td>10.24</td><td>08.81</td><td>10.55</td><td>11.36</td></tr><tr><td>RESCAL</td><td>23.91</td><td>21.69</td><td>17.44</td><td>15.68</td><td>13.86</td></tr><tr><td>DistMult</td><td>13.89</td><td>14.61</td><td>14.39</td><td>13.16</td><td>11.56</td></tr><tr><td>ComplEx</td><td>14.65</td><td>17.61</td><td>17.78</td><td>16.41</td><td>14.09</td></tr><tr><td>HolE</td><td>23.12</td><td>27.43</td><td>24.33</td><td>21.11</td><td>12.46</td></tr><tr><td>Analogy</td><td>14.99</td><td>15.51</td><td>14.93</td><td>13.02</td><td>11.67</td></tr></table>

Peer review (iclr2019):
 The paper proposes a new approach to compute embeddings of multi-relational data such as knowledge graphs. For this purpose, the paper introduces a variant of TransE that operates on Riemannian manifolds, in particular, Euclidean, Spherical, and Hyperbolic space. This approach is motivated by the results of Nickel & Kiela (2017), who showed tha Hyperbolic space can provide important advantages for embedding graphs with hierarchical structure. Hyperbolic and Riemannian embeddings are a promising research area that fits well into ICLR. Extending hyperbolic, and more general, Riemannian embeddings to multi-relational data is an important aspect in this context, as it allows to extend such methods to new applications such as Knowledge Graph Completion. Overall, the paper is written well and mostly good to understand. However, I am concerned about multiple aspects in the current version: - What is the motivation for using this particular form of translation? In Riemannian manifolds, the analogue of vector addition and subtraction is typically taken as the exponential or logarithmic map (as expm and logm in Euclidean space are exactly vector addition and subtraction). For the spherical and hyperbolic manifold both maps have closed form expressions and are differentiable. It is therefore not clear to me what the advantage of the proposed approach is compared to these standard methods. In any case, it would be important to include them in the experimental results. - It is also not clear to me whether the benefits of hyperbolic embeddings translate into the setting that is proposed here. The advantage of hyperbolic embeddings is that they impose a hierarchical structure in the latent space. The method proposed in this paper uses then a single (hyperbolic) embedding of entities across all relation types. This implies that there should be a single consistent hierarchy that explains all the links in all relations. This seems unlikely and might explain some of the hyperbolic results. A more detailed discussion and motivation would be important here. - Regarding the experimental results: Why are the results for HolE and ComplEx so different? As [1,2] showed, both models are identical, and for that reason should get close to identical results. The large differences seem inconsistent with these results. Furthermore it seems that the results reported in this paper do not match previously reported results. What is the reason for these discrepancies? [1] Hayashi, K., and Shimbo, M. "On the equivalence of holographic and complex embeddings for link prediction", 2017. [2] Trouillon, T, and Nickel, M. "Complex and Holographic Embeddings of Knowledge Graphs: A Comparison", 2017.