Knowledge Graph Embeddings using Neural Itô Process: From Multiple Walks to Stochastic Trajectories
Year: 2023

Abstract:
Knowledge graphs mostly exhibit a mixture of branching relations, e.g., hasFriend, and complex structures, e.g., hierarchy and loop. Most knowledge graph embeddings have problems expressing them, because they model a specific relation r from a head h to tails by starting at the node embedding of h and transitioning deterministically to exactly one other point in the embedding space. We overcome this issue in our novel framework ItôE by modeling relations between nodes by relation-specific, stochastic transitions. Our framework is based on stochastic Itô processes, which operate on low-dimensional manifolds. ItôE is highly expressive and generic subsuming various stateof-the-art models operating on different, also non-Euclidean, manifolds. Experimental results show the superiority of ItôE over other deterministic embedding models with regard to the KG completion task.

heading: 1 Introduction
text: Knowledge graphs (KGs) play a central role in many AI-related tasks (Nickel et al., 2016) such as recommendation systems (Lukovnikov et al., 2017) and question answering (Zhang et al., 2016). KGs represent real-world knowledge as a set of facts in the form of triples (entity, relation, entity), e.g., (Alice, FriendOf, Bob). Entities are nodes of a graph and relations are the directed edges. KGs are highly incomplete which adversely affects the outcome of various KG-centered tasks. To tackle this problem, various link prediction approaches have been proposed to leverage the existing links for the prediction of new ones (a.k.a., knowledge graph completion). Among existing approaches (Wang et al., 2017; Ji et al., 2021), KG embedding (KGE) succeeded and became a popular line of work.
KGE models map entities and relations in KGs into a low dimensional geometric space and measure the plausibility of each triple (entity, relation,
entity) by a scoring function f that uses the embedded vectors of this triple (entity, relation, entity) to gauge its plausibility by a real, positive value f(entity, relation, entity). Prominent examples of KGEs include translational/rotational families (Wang et al., 2014; Lin et al., 2015), bilinear families (Nickel et al., 2011; Yang et al., 2014) and neural network families (Dettmers et al., 2018; Nguyen et al., 2018). However, these models are all defined in Euclidean space that suffers from modeling complex structures such as hierarchies (Chami et al., 2020) in low-dimensional vector space. Therefore, KG embeddings based on non-Euclidean geometric spaces such as hyperbolic space (Chami et al., 2020; Balazevic et al., 2019) have been proposed. Both lines of work often formulate the relationspecific transitions between subsequent nodes in a deterministic way and represent them, e.g., by translations or rotations, implying that with regard to a particular relation each node is connected to a single other node.
In a broader view, this implies that starting from an embedded node with a given single relation type, there will be only one walk with the length m,
7165
which is m subsequent relation-specific transitions from the starting node, and this walk will lead to another specific embedded node. However, starting at any specific node and considering a branching relation such as hasFriend, there may be several relation-specific walks of length m. These walks define a group of nodes in distance m. Figure 1 illustrates the limits of traditional embedding models based on translation or rotation with deterministic transitions (a) and contrasts it with the tree-like structure that unfolds in a non-deterministic model (b) able to represent a branching relationship.
In this paper, we formalize probabilistic walks of length m − 1 between two groups of nodes (Ωt1 ,Ωtm) in order to represent branching relationships. Each group of nodes Ωti from start Ωt1 to end Ωtm is represented by a random variable Xti , which indicates a probabilistic distribution on the manifold Xti : Ωti → M, i = 1, . . . ,m. The sequence of random variables S = {Xt1 , Xt2 , . . . , Xtm} constitutes a stochastic process. A sample, also called realization, of the stochastic process S is a walk with m−1 transitions containing m entity representations {et1 , . . . , etm} on the manifold, eti ∈ M.
We model probabilistic transitions between groups of nodes using the Itô process, which defines transitions between nodes through an integral containing drift and diffusion. Drift models the deterministic nature of a transition, while diffusion captures stochastic variation. Drift and diffusion operations are learned separately via a relationspecific neural network.
Owing to the probabilistic perspective, we develop a KGE model, called ItôE, that uses the Itô process for stochastic transitions as well as various, also non-Euclidean, manifolds (i.e., sphere, Poincaré ball, hyperboloid) to support the modeling of heterogeneous graph structures on manifolds. We also provide an extensive theoretical investigation and prove that ItôE: a) is fully expressive; b) is capable of encoding various structures as stochastic processes; c) subsumes various state-of-the-art KGE models, including ComplEx and QuatE, d) models various relational patterns, e.g., symmetric patterns. Experimental results show the superiority of our model, especially in low-dimensional space.

heading: 2 Related Work
text: Euclidean KGE Models The earliest KGE models employ Euclidean geometry, i.e., a d dimen-
sional real space Rd, with its corresponding Euclidean distance function as well as inner product. TransE represents each relation as a translation from a head to a tail embedding. Several variants of TransE have been developed to model one-to-many relations, as well as symmetric and reflexive patterns (Wang et al., 2014; Lin et al., 2015). DistMult (Yang et al., 2014) proposes a tensor factorization with a diagonal relation matrix. The model captures symmetric relations, but not anti-symmetry. ComplEx(Trouillon et al., 2016) and QuatE (Zhang et al., 2019) models have been proposed as extensions of DistMult from the real space to the (hyper)complex space. RotatE (Sun et al., 2019) models each relation as rotation in the complex space using the Euler formula, which captures symmetric, anti-symmetric, inverse, and composition patterns. Apart from the mentioned shallow embedding models, there is a thread of neural network KGE models such as Neural Tensor Network (Socher et al., 2013), ConvE (Dettmers et al., 2018), and ConvKB (Nguyen et al., 2018). In summary, these models cannot capture various graph structures, especially in low dimensional space due to the underlying Euclidean space. A series of graph embedding models have been proposed based on random walks in the graph space.
Node2vec (Grover and Leskovec, 2016; Ristoski and Paulheim, 2016; Portisch and Paulheim, 2022; Huang et al., 2021; Perozzi et al., 2014) are among the models which perform a biased random walk in graph space and compute the low dimensional representation of nodes in such a way to maximize the likelihood of preserving network neighborhoods of nodes. The random walk-based models are not among state-of-the-art KGE models in the link prediction task. In addition, in most cases, the random walk is performed in the graph space to obtain the sequences of nodes for embedding, but the notion of walk in the embedding space is neglected as there is no transition function in the embedding space to model this.
Non-Euclidean KGE Models Euclidean-based KGE models are not capable of preserving complex graph structures in a low-dimensional space. However, embedding KGs on non-Euclidean manifolds has shown promising performance in the preservation of a few structures, especially in lowdimensional spaces. (Nickel and Kiela, 2017; Chami et al., 2020; Balazevic et al., 2019; Weber and Nickel, 2018) showed the advantage of
Poincaré Ball and other geometries for embedding graphs with various structures including hierarchical structures. 5∗E (Nayyeri et al., 2021a) utilized the projective geometry with the five main transitions, namely translation, rotation, inversion, reflection, and homothety, to capture heterogeneous structures such as loop-path subgraphs. (Nayyeri et al., 2021b) embedded KGs on the vector field. UltraE (Xiong et al., 2022) considers a mixture manifold–pseudo-Riemannian space that generalizes hyperbolic and spherical spaces. Other manifold-based KGEs can be found in (Suzuki et al., 2019). Overall, the mentioned models suffer from the same problem mentioned for the Euclidean counterpart, i.e., deterministic transitions between embedded nodes.

heading: 3 Preliminaries
text: Stochastic Process Let T be an arbitrary index set. A stochastic process is a collection of random variables S = {Xt : t ∈ T} defined on a probability space P with the index set T of size |T |. All random variables Xt : Ωt −→ Rd, et 7−→ Xt(et), t ∈ T belong to the same probability space. A random experiment (realization) is a selection of an outcome Xt(et) ∈ M at random considering the probability measure P. A sample path of a stochastic process S = {Xt : t ∈ T} is a function from t to Xt(et) using an ordered index set T , giving us a random walk. Figure 2b illustrates a random walk in the vector space with 50000 steps.
Brownian Motion A stochastic process B = {B(t), t ≥ 0} defined on a probability space P = (Ω,F ,P) (F is a sigma algebra (events)) is a Brownian motion if a) B(0) = 0; b) B has independent and stationary increment, i.e., B(t1), B(t2)−B(t1), . . . , B(tn)−B(tn−1) are independent random variables for 0 < t1 < . . . < tn; c) B has Gaussian increments, i.e., B(tn + α) − B(t) ∼ N(0, α), ∀t ≥ 0, α > 0; and d) B has continuous sample paths, i.e., B is continuous in t. Figure 2a shows the evolution of two tree structures via the Brownian motion of several particles.
Brownian motion is suitable to model probabilistic branching in random walks as path diffusion when traversing the embedded KG in a vector space. In our setting, we will write interchangeably B(et) and B(et) for B(t).
Itô Process The stochastic process S = {Xt, t ≥ 0} that solves the following integral is called an Itô
process,
Xt = X0 +
∫ t
0
a(Xs, s) ds +
∫ t
0
b(Xs, s) dBs, (1)
where t ≥ 0 and X0 is a scalar starting point. {a(Xt, t) : t ≥ 0}, {b(Xt, t) : t ≥ 0} are stochastic processes which are called drift and diffusion, respectively. B is Brownian motion, dB is normally distributed with zero mean and variance dt.
The above formulation of Itô process is approximated by the Euler-Maruyama approximation as follows
X(tn+1) = X(tn) + a(X(tn), tn)∆t+ b(X(tn), tn)∆B(tn), tn = n∆t. (2)
Non-Euclidean Geometry In this part, we introduce three popular non-Euclidean manifolds namely spherical S = {x ∈ Rd+1|⟨x,x⟩ = 1}, Hyperboloid H = {x ∈ Rd+1|⟨x,x⟩ = 1K }, and Poincaré ball B = {x ∈ Rd | ∥x∥ < − 1K }. K > 0 is curvature. The tangent space T Kx at a point x on manifold is a d-dimensional vector space. This space covers all possible directions of paths on a manifold starting from x. Each point on the tangent space is mapped to the manifold via an exponential map. Given v as a tangent vector at point x on the manifold, the exponential map for sphere, Hyperboloid and Poincaré ball are
expx(v) = cos(∥v∥)x+ sin(∥v∥) v∥v∥ ,
cosh( √ |K|∥v∥)x+ v sinh( √
|K|∥v∥)√ |K|∥v∥ , (3)
andx⊕K (tanh( √ |K|λxK∥v∥ 2 ) v√ |K|∥v∥ ),
respectively. The exponential map projects the tangent vector at a point x on a manifold to another point laying on the geodesic curve, i.e, a curve with
the shortest distance on the manifold. K is curvature and ⊕K is the Möbius addition (Balazevic et al., 2019). Note that the tangent vector is defined as v = dxdt , which is orthogonal to the manifold at the point x. Later in the paper, we show that by using the Itô process, as stochastic differential equations, we can derive a stochastic process evolving on the manifold.

heading: 4 ItôE: Neural Itô Process Embedding
text: We introduce ItôE, a novel KGE model that utilizes the stochastic processes on manifolds for KG embedding. ItôE is capable of preserving various graph structures and capturing branching relations by modeling the evolution of graph structures in the embedding space as a stochastic process.
KGE models have four essential components: entity and relation representation, score function, and loss function. In the following, these four components of ItôE are explained.
Entity Representation Let’s suppose that E represents the collection of all entities present in the knowledge graph (KG). Each symbolic entity e ∈ E in a KG is embedded on a d-dimensional manifold M, i.e., e ∈ M. Therefore, entity embeddings are points on the manifold. In the proposed model, we use Poincaré Ball, Hyperboloid, Euclidean, and Sphere manifolds.
Relation Representation Each fact in a KG is represented by a triple (et, r, et+1). Because the entities in a triple are subsequent but with arbitrary indexes, we use the notation (etn , r, etn+1) to represent the triple, where n is between 1 and |E| (number of entities in the KG). Most KGE methods model the transition from etn to etn+1 via a relation-specific transition (e.g., translation, or rotation). Therefore, each relation is modeled as a point-wise deterministic transition.
However, in a broader view, the relational dependencies between nodes are stochastic processes, i.e., a transition to a tail given a head node and a relation happens with a probability so that the sequence of such probabilistic transitions constitutes a stochastic process. Each random variable in the stochastic process includes all the entities E , a few of which has a non-zero likelihood since the transition from a given entity is only possible to its neighbors. The group of neighbors at each step together with their likelihood in a probability space is considered a random variable. As a result, there
is a relational mapping between two random variables Xtn , Xtn+1 , associated with the two groups of nodes Ωtn ,Ωtn+1 at each step. In the following, we model a relation r (defining transitions between nodes) as a stochastic process Sr.
To explain this, let us view the notion of walks formed by a relation r from a stochastic processes angle. A sequence of symbolic entities, connected by a relation r forms a walk with a particular length. A walk of length n − 1 includes n entities and is shown by Pr = {et1 , et2 , . . . , etn}.
Assume that there are multiple walks from a set of given entities Ωt1 associated to Xt1 , to a set of entities in Ωtn associated to Xtn . The transition between the nodes in a walk is done randomly according to a distribution.
During traversing the graph from a starting node taken from Xt1 to a target node, taken from Xtn , and at each step ti, there are ni possible options for selecting the next node. Therefore, traversing a graph with walks of length n − 1 (with relation r) leads to a stochastic process Sr = {Xt1 , Xt2 , . . . , Xtn}. Consequently, each relation in the KG is represented by a stochastic process.
In this paper, among various stochastic processes, we employ the Itô process for modeling each relation r due to the simplicity of implementation and controlling drift and diffusion. The Itô process is defined as
Xt = X0 +
∫ t
0
ar(Xs, s) ds +
∫ t
0 br(Xs, s) dBs, t ≥ 0, (4)
where ar(., .) and br(., .) are relation-specific drift and diffusion in the Itô integral, respectively. The drift part captures the deterministic transitions, while the diffusion part captures the stochastic tran-
sitions. In this equation, we consider the random variables corresponding to the group of entities that can be seen at each step as a continuous representation. For implementation, we provide the Euler-Maruyama approximation (Jahnke, 2016) as a well-known approximation for discrete space:
X(tn+1) = X(tn) + ar(X(tn), tn)∆t+
br(X(tn), tn)∆B(tn), tn = n∆t, (5)
where X(ti) is the random variable at step i. For simplicity, we set ∆t = 1 from now onward. Note that this approximation is used for the Euclidean manifold. The drift ar(X(tn), tn) and diffusion br(X(tn), tn) parts are parameterized by two separate multi-layer neural networks. In this way, both drift and diffusion at each step are learned by the model. To enable the Itô process acting on a manifold, it is essential to obtain the tangent vector v. The tangent vector determines the direction of movement on manifold M. The exponential map (Equation 3) is used to map the two subsequent random variables on the manifold while considering drift and diffusion as follows
Xtn+1 =expXtn (ar(X(tn), tn)∆t+
br(X(tn), tn)∆B(tn)), tn = n∆t. (6)
Therefore, each two subsequent sampled entities lie on a geodesic curve (shortest path). Because the above equation is a stochastic process on a manifold, at each iteration of batch learning, a set of entities in batch triples are observed randomly to hold this equation (realization) as follows:
Xtn+1(etn+1) = expXtn (etn )(ar(Xtn(etn , tn))
∆t+ br(Xtn(etn , tn))∆B(tn)), tn = n∆t,
where the equation is held for each sample walk. Note that Xti(eti) = eti ∈ M, i = 1, . . . , n is entity embeddings.
Scoring Function For a given triple (etn , r, etn+1), the score is as follows
f(etn , r, etn+1) = −∥etn+1 − expetn (ar(etn , tn) ∆t+ br(etn , tn)∆B(tn))∥, tn = n∆t.
For positive (negative) triples, f(etn , r, etn+1) is a high (low) value. That is, the two sampled entities etn , etn+1 lie on a geodesic curve on a manifold.
Loss Function For training the model, we use the following loss function (Chami et al., 2020)
L = ∑
e′∈E log(1 + exp(ye′(f(etn , r, e
′) + δetn + δe′))),
where ye′ = 1 if e′ = etn+1 , and ye′ = −1 if e′ ̸= etn+1 , and δetn and δe′ are trainable entity biases.
In the next section, we present important insights about our formulation, followed by theoretical justification for the core formulation of our model.

heading: 5 Insights and Theoretical Analysis
text: Memory Complexity In ItôE, the number of relation parameters grows linearly with the relation’s dimensionality. Hence, ItôE’s space complexity is O(Ne × de +Nr × dr), where Ne and Nr are the numbers of entities and relations, de and dr are the embedding dimensionality of entities and relations, respectively. The additional parameters come from the neural network that approximates the ItôE’ process, which is in our case, shared across all entities.
Relational Sub-structures Here we show the capability of the stochastic process in equation (5) for modeling various graph sub-structures. To this end, let us have a sample walk from the stochastic process Sr, which is {Xt1(et1), . . . , Xtn(etn)} = {et1 , . . . , etn}. A stochastic process Sr inherently covers various graph structures such as hierarchy, loop, and path. This is due to the fact that various sample walks over a stochastic process generate different parts of a subgraph. Figure 3 shows an example of a stochastic process for the relation ConnectedTo which forms several relational and structural patterns such as loop and path by sampling from the process in the airport example (airports are connected with various shapes). Another example is a tree-like structure. For tree-like structures (see Figure 1), Xt1 contains only a root node, and Xtn contains all the leaves. Xti , i = 2, . . . , n − 1 generates the intermediate nodes from the root to the leaves. Therefore, any walk from the root node to each of the leaves is a sample taken from the stochastic process. The relation-specific stochastic process generates various sample walks at random covering various parts of a sub-graph.
Subsumption of Other KGE Models ItôE provides a general framework that covers various baselines and state-of-the-art KGE models. We prove,
in this part, that our model subsumes TransE, RotatE, QuatE, 5*E, ComplEx, and DistMult. That is, given any set of triples with arbitrary true/false labeling, any score value represented by each of the mentioned models for the triples in the set is also represented by the score function of ItôE. In this regard, the following theorem holds:
Theorem 1. ItôE subsumes TransE, RotatE, DistMult, ComplEx, QuatE, and 5*E.
As a consequence of Theorem 1, ItôE is fully expressive and capable of capturing various graph structures and relational patterns that each of the mentioned models is capable of. Therefore, the following corollaries hold:
Corollary 1. ItôE is fully expressive, i.e., for every ground truth over an arbitrary KG, there are assignments to the entities and relations embedding to capture the ground truth.
Corollary 2. ItôE models symmetric, antisymmetric, composition, inversion, transitive, and reflection patterns.
Corollary 3. ItôE models one-to-many relation.
Corollary 4. a) Let Lnr1 be a loop structure with a single relation r1 and n nodes. ItôE models the loop structure Lnr1 .
b) Let Pnr2 be a path structure with relation r2 and n nodes. In addition, each nodes of Lnr1 is connected to one node in Pnr1 . The combined structure is denoted by LPnr1r2r3 . ItôE models LPnr1r2r3 .
c) ItôE models loop-path structure with single relation r1, i.e., LPnr1r1r1 .

heading: 6 Experiments and Results
text: Experimental Setup In this section, we evaluate the performance of ItôE against various state-ofthe-art KGE models in the link prediction task. Our evaluation in this section includes link prediction in low-dimensional space, analysis on capturing complex structures, analysis of capturing hierarchical structures, and time and memory complexity. Further evaluations including results per manifold, variance of results of our model, influence of embedding dimension, and discussion on the effect of loss function on deterministic models can be found in appendix.
Evaluation Metrics We use four standard metrics for link prediction namely Mean Reciprocal Rank (MRR), Hits@k (k=1,3,10). To compute each of the metrics, we use the procedure in (Bordes
et al., 2013; Lacroix et al., 2018). For each test triple (etn , r, etn+1), we first replace the head entity etn by each of the entities in the dictionary, i.e., e′ ∈ E − {etn}. This results in ne corrupted triples {(e′, r, etn+1)}, where ne is the number of entities in the KG. We filtered this set by removing all triples that are already appeared the dataset as well as self loop. We then compute the scores of the original test triple (etn , r, etn+1) and the corrupted triples {(e′, r, etn+1)}, sort them based on scores and rank them.
The resulted rank of the original triple is the left rank rl. The same procedure is performed to compute the right rank by the corruption of the tail entities. rr denotes the right rank. The average of the left and the right ranks is denoted by ra. The mean rank of all testing triples is MR. The percentage of the test triples ranked lower than k = 1, 3, 10 denotes Hits@k. MRR is the average reciprocal of rank for all testing triples.
Environment and Hyperparameters We implemented1 our model using Python and PyTorch library. We added reciprocal relations to the training samples as a standard technique used in (Kazemi and Poole, 2018; Lacroix et al., 2018). We added N3 regularization for training the models (Lacroix et al., 2018). Because one of the main goal of this paper is modeling graphs in low dimensional space, we follow the common practice of existing works in low dimensional embedding (Chami et al., 2020) and trained the models in a low dimension (d = 32). The other dimensions have been done as further analysis in the appendix. We split data into several batches and used the Adagrad/Adam as an optimizer. An early stopping technique based on validation MRR has been used to terminate the running and perform testing. Batch size b, learning rate lr, N3 regularization coefficient α are among the hyperparameters used in this paper. In addition, we set the number of hidden layers for drift and diffusion neural networks to two. l denotes the number of neurons in the hidden layer of each neural networks. The used distribution for Brownian motion ∆B is a normal distribution with zero mean and σ variance. For simplicity, we set σ = 1. Due to randomness, for our model, we perform experiments 10 times and report the average results in Table 1. Because the variances were low, we did not report them in the main table. The manifold M is selected from Poincaré ball B, Hyperboloid
1https://github.com/ColdMist/ItoE
model H, Euclidean real manifold R and spherical manifold S. The optimal hyperparameters per each dataset are reported in a separate table in appendix.
Dataset We used the two standard benchmark datasets namely FB15k-237 (Toutanova and Chen, 2015) and WN18RR (Dettmers et al., 2018) for evaluating ItôE on static KGs. Both datasets contain structural and relational patterns including symmetric/anti-symmetric and composition patterns. In addition, WN18RR contains hierarchical structures associated with hypernym and part-of. Furthermore, both datasets include various types of relationships, including one-to-one relationships, one-to-many relationships (where a subject can have multiple objects but each object has only one
subject), many-to-one relationships, and many-tomany relationships (where a subject can have multiple objects and an object can have multiple subjects).
KGE Models Two classes of baselines and stateof-the-art KGE models have been selected as competitors: a) baselines KGEs: TransE, DistMult, and ComplEx, b) state-of-the-art KGEs: RotatE, QuatE, 5∗E, MurP/MurE, and RotH/RefH/AttH. We trained these models using entity bias, and cross-entropy loss, regularization, and also we added reverse triples as in (Chami et al., 2020). Such techniques improves the performance of the models including TransE comparing to their original results.
MurP, RotH/RefH/AttH employ Poincaré ball as a non-Euclidean manifold to preserve hierarchical structures. RotH additionally takes the advantage of rotation in hyperbolic space to model various relational patterns such as symmetry, anti-symmetry, inversion, and composition. To train ComplEx, QuatE, RotH/RefH/AttH, MurP, DistMult, and our model, we enriched the data with reverse triples, as the standard technique employed in (Kazemi and Poole, 2018), and used N3 regularization (Lacroix et al., 2018).
Results Table 1 shows the results of ItôE (Poincare (P), Euclidean (R), Sphere (S)) and other models in low-dimensional embedding d = 32 on FB15K-237 and WN18RR. Note that d = 32 is a common practice of KGE literature for evaluation of the models in low dimensional embedding (Chami et al., 2020). According to our experiments, ItôE with the Poincaré ball outperforms all models on WN18RR dataset which contains mainly hierarchical relations such as hypernym and part-of. This dataset also contains relations forming loop structures such as similar-to. MuRP, REFH, ROTH, and ATTH utilize Poincaré ball with deterministic transitions (e.g., rotation, reflection, translation, and Affine mapping). ItôE with the stochastic transition on Poincaré ball outperforms all of these models on WN18RR across all metrics. This is especially visible by looking at Hits@3 and Hits@10. The results show the superiority of stochastic transitions over deterministic transitions to model various structures such as Hierarchical and loop structures in a low-dimensional space. Using FB15k-237, ItôE with Spherical manifold outperforms other competitors including the Hyperbolic models. Figure 8 presents performance per dimension. As shown in the figure, our model outperforms other KGE models in low dimensions. In high dimension, our model get competitive performance to other models on WN18RR dataset.
Capturing hierarchical structure In this part, we generate tree structure in the embedding space by using the transition functions of the Itô process, translation, and rotation. Starting from a point on
the embedding space (i.e., root shown in red in Figure 4), a model generates n child for each node and traverses the vector space to generate the whole tree structure. As shown in Figure 4b, the Itô process generates the tree-like structure by generating stochastic trajectories in the vector space. However, translation ( Figure 4c) and rotation (Figure 4d) generate a single trajectory which consequently cannot traverse the vector space to generate a tree structure. Figure 4a illustrates the evolution of tree structure in a sphere. The root is located on the center and the leaves are distributed on the border of the sphere. All the intermediate nodes on the lth level of the tree are distributed on a sphere with radius of rl, inside the main sphere where rl1 < rl2, l1 < l2. A trajectory (dashed line) is shown from root to leaf.
Capturing Complex structures Here we examine ItôE and several other Euclidean and nonEuclidean models to preserve various complex structures. We train the models on the substructures and then present the ranking results given by the models in a heatmap. For the graph in Figure 7, the heatmap is presented in Figure 6. As shown in the figure, ItôE gets a very low rank (mostly 1 which is ideal) to the graph edges. This shows that the model learns the structure. For other models including manifold-based models, e.g., AttH and RotH, the ranking for the edges are high, i.e., these models do not learn the structure. Figure 5 shows the ranking results of modeling heterogeneous structure on the example of a loop connected to a path.
Memory and Time Complexity Table 2 shows the training time (per epoch) and the number of model parameters for ItôE, TransE, MurE, RefH, RotH and AttH. According to the table, ItôE has a close number of parameters to other state-of-the-art models. Among the models, TransE is the most efficient model in terms of the number of parameters. In addition, our model is competitive with other models in terms of training time.

heading: 7 Conclusion
text: This paper presented ItôE, a knowledge graph embedding model that considers the stochastic transitions between nodes of a knowledge graph on a manifold. For doing so, ItôE modeled the relations in a KG as stochastic processes so that the transitions between two nodes could only happen with an associated likelihood. Such stochastic transitions allowed ItôE to present multiple stochastic trajectories between any two embedded nodes and to capture more sophisticated structures in KGs, including loops connected to paths and is mathematically proved to be a generalization of several state-of-the-art models. Experiments on the syn-
thesized datasets showed that the proposed model can capture heterogeneous complex structures and patterns.
Limitations
In this section, we discuss the limitation of the proposed model. Currently, the hidden layer of the two neural networks for drift and diffusion are shared between all entities and relations. This might cause over-fitting on relations that show simple structures if the neural networks are set to be very deep. On the other hand, if the neural networks are set to be shallow it might negatively influence modeling complex relations as the direction of trajectories will be limited. One possible solution is to cluster relations based on complexities and use separate neural networks for each cluster depending on the complexity of the corresponding relation. This requires, however, prior knowledge about the structure of different relations which we leave as future work.
Ethics Statement
The authors declare that we have no conflicts of interest. This article does not contain any studies involving business data and personal information.

heading: Acknowledgements
text: The authors thank the International Max Planck Research School for Intelligent Systems (IMPRSIS) for supporting Bo Xiong. Bo Xiong is funded by the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No: 860801. Mojtaba Nayyeri is funded by the German Federal Ministry for Economic Affairs and Climate Action under Grant Agreement Number 01MK20008F (Service-Meister). This research was partially funded by the Ministry of Science, Research, and the Arts (MWK) Baden-Württemberg, Germany, within the Artificial Intelligence Software Academy (AISA) and the German Research Foundation (DFG) via grant agreement number STA 572/18-1 (Open Argument Mining). We acknowledge the support of the Stuttgart Center for Simulation Science (SimTech). The authors would like to thank the reviewers for their constructive comments and suggestions.

heading: A Appendix
text: We organize the appendix as follows: We first present the proof of Theorem 1, and corollaries 1-4. We then discuss the effect of loss functions on deterministic models, followed by the results per manifold and hyperparameters specification.
Proof of Theorem 1
Proof. Here we prove that ItôE subsumes 5∗E (Nayyeri et al., 2021a). We start with the formulation of 5∗E. For simplicity, we remove the index of relation from the relation matrix in 5∗E and consider one-dimensional complex projective line. The following equation is modeling triple in the vector space by 5∗E
eptn+1 ≈ ηe p tn ,
η = ( a b c d ) ,
eptn = ( etn 1 ) ,
a, b, c, d, z ∈ C.
(7)
Let η = ( ar + aIi br + bIi cr + cIi dr + dIi ) = ( ar br cr dr ) +
( aI bI cI dI ) i = ηr + ηIi, and e p tn = ( ertn 1 ) +
( eItn 0 ) = eprtn + e pI tn . We have
eptn+1 = ηe p tn = (ηre pr tn−ηIe pI tn )+(ηre pI tn−ηIe pr tn)i.
(8) We merge the matrices and rewrite the formula-
tion as follows:
eptn+1 = ( ηr ηI )( eprtn −epItn ) + ( ηr ηI )(epItn eprtn ) i.
(9)
Let ηrI = ( ηr ηI ) , erItn = ( eprtn −epItn ) , eIrtn =
( epItn eprtn ) .
Therefore, we have eptn+1 = ηrIe Ir tn + ηrIe Ir tn i.
The above formulation can be rewritten in the vectored form as follows
eptn+1 = ηrI ( erItn eIrtn ) = ηrIe IrI tn . The formulation of ItôE is etn+1 ≈ etn + ar(etn , tn)∆t +
br(etn , tn)∆B(tn).
If ∆t = 1, br(etn , tn) = 0, there is a neural network ar(etn , tn) that approximates the multivariate function (ηrI − I)eIrItn with the error as close as zero due to universal approximation ability of the NNs. Therefore, ItôE can approximate the score of 5∗E with an arbitrary small error. Consequently, ItôE subsumes 5∗E. Because 5∗E subsumes TransE, RotatE, ComplEx, ItôE subsumes these models as well.
We now prove that ItôE subsumes QuatE. Considering the formulation of ItoE, etn+1 = etn + ar(etn , tn)∆t + br(etn , tn)∆B(tn), and setting ∆t = 1, br(etn , tn) = 0, ar(etn , tn) = (R − I)ev, where ev is the vector representation of a Quaternion number, the assumption of QuatE is fulfilled by ItôE
evtn+1 ≈ Revtn . (10) Therefore, ItôE subsumes QuatE (Zhang et al.,
2019).
Proof of Corollaries
Proof. Here we present the proof of Corollaries 1- 4. Because ItôE subsumes 5∗E, ComplEx, QuatE, RotatE, and TransE, it can encode all relational and structural patterns (symmetric, anti-symmetric, reflexive, transitive, inverse, and combination of loop and path) modeled by these models. Moreover, the ItôE model is fully expressive because it subsumes ComplEx which is fully expressive.
Effect of Loss Function on Deterministic Models As mentioned in the paper, the models based on deterministic transitions such as TransE with etn + r = etn+1 provide a single trajectory between any two nodes. However, by using a loss function forcing an upper-bound for the score of positive samples we have etn + r = etn+1 + ϵ. This allows the models based on deterministic transitions to mitigate the problem of a single trajectory. However, the problem is not fully solved because the nodes after each transition are embedded very closely. In this way, there is a single trajectory that is retrievable by transition function in which the other embedded nodes are not reachable by using the transition function in the embedding space. In contrast, using the stochastic transitions, the model can learn at each embedded node etn via diffusion NN, br(etn , tn), the degree of branching, i.e., degree of diffusion. Therefore, different trajectories
are learned between any two nodes which can be either very close or far.
Variance in ItôE Performance Table 4 shows the variance of ItôE with dimension 32 on WN18RR. As shown in the table, the variance of 10 times running on the model is very low. Therefore, the model obtains stable performance on different runs.
Results On WN18RR Per Manifolds In this part, we analyze the performance of ItôE using various manifolds (Poincare ball, Hyperboloid, Euclidean, and Sphere) on WN18RR. Figure 9 illustrates the performance comparison according to different metrics namely MRR, Hits@1, Hits@3, Hits@10. The experiments have been done in a very low dimension of 32. According to the figure, ItôE with Poincare ball and Hyperboloid outperformed ItôE with Sphere and Euclidean manifold. This is consistent with the nature of the used KG where most relations are hierarchical in WN18RR.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitation section
3 A2. Did you discuss any potential risks of your work? Ethic statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 6 and Appendix
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 6 and Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 6 and Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: This requires, however, prior knowledge about the structure of different relations which we leave as future work