SlowGuess/ACL_2020_OCR huggingface dataset:
Yun Tang, Jing Huang, Guangtao Wang, Xiaodong He, Bowen Zhou JD AI Research
{yun.tang, jing.huang, guangtao.wang, xiaodong.he, bowen.zhou}@jd.com
Distance-based knowledge graph embeddings have shown substantial improvement on the knowledge graph link prediction task, from TransE to the latest state-of-the-art RotatE. However, complex relations such as N-to-1, 1-to-N and N-to-N still remain challenging to predict. In this work, we propose a novel distance-based approach for knowledge graph link prediction. First we extend the RotatE from 2D complex domain to high dimensional space with orthogonal transforms to model relations. The orthogonal transform embedding for relations keeps the capability for modeling symmetric/anti-symmetric, inverse and compositional relations while achieves better modeling capacity. Second, the graph context is integrated into distance scoring functions directly. Specifically, graph context is explicitly modeled via two directed context representations. Each node embedding in knowledge graph is augmented with two context representations, which are computed from the neighboring outgoing and incoming nodes/edges respectively. The proposed approach improves prediction accuracy on the difficult N-to-1, 1-to-N and N-to-N cases. Our experimental results show that it achieves state-of-the-art results on two common benchmarks FB15k-237 and WNRR-18, especially on FB15k-237 which has many high in-degree nodes. Code available at https://github.com/JD-AI-Research-Silicon-Valley/KGEmbedding-OTE.
Knowledge graph is a multi-relational graph whose nodes represent entities and edges denote relationships between entities. Knowledge graphs store facts about people, places and world from various sources. Those facts are kept as triples (head entity, relation, tail entity) and denoted as  $(h,r,t)$ . A large number of knowledge graphs, such as
Freebase (Bollacker et al., 2008), DBpedia (Auer et al., 2007), NELL (Carlson et al., 2010) and YAGO3 (Mahdisoltani et al., 2013), have been built over the years and successfully applied to many domains such as recommendation and question answering (Bordes et al., 2014; Zhang et al., 2016). However, these knowledge graphs need to be updated with new facts periodically. Therefore many knowledge graph embedding methods have been proposed for link prediction that is used for knowledge graph completion.
Knowledge graph embedding represents entities and relations in continuous vector spaces. Started from a simple and effective approach called TransE (Bordes et al., 2013), many knowledge graph embedding methods have been proposed, such as TransH (Wang et al., 2014), DistMult (Yang et al., 2014), ConvE (Dettmers et al., 2018) to the latest RotatE (Sun et al., 2019) and QuatE (Zhang et al., 2019).
Though much progress has been made, 1-to-N, N-to-1, and N-to-N relation predictions (Bordes et al., 2013; Wang et al., 2014) still remain challenging. In Figure 1, relation "profession" demonstrates an N-to-N example and the corresponding edges are highlighted as green. Assuming the triple (SergeiRachmaninoff, Profession, Pianist) is unknown. The link prediction model takes "SergeiRachmaninoff" and relation "Profession" and rank all entities in the knowledge graph to predict "Pianist". Entity "SergeiRachmaninoff" connected to multiple entities as head entity via relation "profession", while "Pianist" as a tail entity also reaches to multiple entities through relation "profession". It makes the N-to-N prediction hard because the mapping from certain entity-relation pair could lead to multiple different entities. Same issue happens with the case of 1-to-N and N-to-1 predictions.
The recently proposed RotatE (Sun et al., 2019)
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2713-2722 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Snapshot of knowledge graph in FB15k-237. Entities are represented as golden blocks.
models each relation as a 2-D rotation from the source entity to the target entity. The desired properties for relations include symmetry/antisymmetry, inversion and composition which have been demonstrated to be useful for link prediction in knowledge graph. Many existing methods model one or a few of these relation patterns, while RotatE naturally handles all these relation patterns. In addition, the entity and relation embeddings are divided into multiple groups (for example, 1000 2-D rotations are used in (Sun et al., 2019)). Each group is modeled and scored independently. The final score is computed as the summation of all these scores, which can be viewed as an ensemble of different models and further boost the performance of link prediction. However, RotatE is limited to 2-D rotations and thus has limited modeling capacity. In addition, RotatE does not consider graph context, which is helpful in handling 1-to-N, N-to-1, and N-to-N relation prediction.
In this work, a novel distance-based knowledge graph embedding called orthogonal transform embedding (OTE) with graph context is proposed to alleviate the 1-to-N, N-to-1 and N-to-N issues, while keeps the desired relation patterns as RotatE. First, we employ orthogonal transforms to represent relations in high dimensional space for better modeling capability. The Orthogonal transform embedding also models the symmetry/antisymmetry, inversion and compositional relation patterns just as RotatE does. RotatE can be viewed as an orthogonal transform in 2D complex space.
Second, we integrate graph context directly into the distance scoring, which is helpful to predict 1-to-N, N-to-1 and N-to-N relations. For example, from the incomplete knowledge graph, people find useful context information, such as (SergeiRachmaninoff, role, Piano) and (SergeiRachmaninoff, Profession,Composer) in Figure 1. In this work, each node embedding in knowledge graph is aug
mented with two graph context representations, computed from the neighboring outgoing and incoming nodes respectively. Each context representation is computed based on the embeddings of the neighbouring nodes and the corresponding relations connecting to these neighbouring nodes. These context representations are used as part of the distance scoring function to measure the plausibility of the triples during training and inference. We show that  $OTE$  together with graph context modeling performs consistently better than  $RotatE$  on the standard benchmark FB15k-237 and WN18RR datasets.
In summary, our main contributions include:
- A new orthogonal transform embedding  $OTE$ , is proposed to extend RotatE from 2D space to high dimensional space, which also models symmetry/antisymmetry, inversion and compositional relation patterns;
- A directed graph context modeling method is proposed to integrate knowledge graph context (including both neighboring entity nodes and relation edges) into the distance scoring function;
- Experimental results of  $OTE$  on standard benchmark FB15k-237 and WN18RR datasets show consistent improvements over RotatE, the state of art distance-based embedding model, especially on FB15k-237 with many high in-degree nodes. On WN18RR our results achieve the new state-of-the-art performance.
2.1 Knowledge Graph Embedding
Knowledge graph embedding could be roughly categorized into two classes (Wang et al., 2017): distance-based models and semantic matching models. Distance-based model is also known as additive models, since it projects head and tail enti
ties into the same embedding space and the distance scoring between two entity embeddings is used to measure the plausibility of the given triple. TransE (Bordes et al., 2013) is the first and most representative translational distance model. A series of work is conducted along this line such as TransH (Wang et al., 2014), TransR (Lin et al., 2015) and TransD (Ji et al., 2015) etc. RotatE (Sun et al., 2019) further extends the computation into complex domain and is currently the state-of-art in this category. On the other hand, Semantic matching models usually take multiplicative score functions to compute the plausibility of the given triple, such as DistMult (Yang et al., 2014), ComplEx (Trouillon et al., 2016), ConvE (Dettmers et al., 2018), TuckER (Balazevic et al., 2019) and QuatE (Zhang et al., 2019). ConvKB (Nguyen et al., 2017) and CapsE (Nguyen et al., 2019) further took the triple as a whole, and fed head, relation and tail embeddings into convolutional models or capsule networks.
The above knowledge graph embedding methods focused on modeling individual triples. However, they ignored knowledge graph structure and did not take advantage of context from neighbouring nodes and edges. This issue inspired the usage of graph neural networks (Kipf and Welling, 2016; Velicković et al., 2017) for graph context modeling. Encoder-decoder framework was adopted in (Schlichtkrull et al., 2017; Shang et al., 2019; Bansal et al., 2019). The knowledge graph structure is first encoded via graph neural networks and the output with rich structure information is passed to the following graph embedding model for prediction. The graph model and the scoring model could be end-to-end trained together, or the graph encoder output was only used to initialize the entity embedding (Nathani et al., 2019). We take another approach in this paper: we integrate the graph context directly into the distance scoring function.
2.2 Orthogonal Transform
Orthogonal transform is considered to be more stable and efficient for neural networks (Saxe et al., 2013; Vorontsov et al., 2017). However, to optimize a linear transform with orthogonal property reserved is not straightforward. Soft constraints could be enforced during optimization to encourage the learnt linear transform close to be orthogonal. Bansal et al. (2018) extensively compared different orthogonal regularizations and find regularizations
make the training faster and more stable in different tasks. On the other hand, some work has been done to achieve strict orthogonal during optimization by applying special gradient update scheme. Harandi and Fernando (2016) proposed a Stiefel layer to guarantee fully connected layers to be orthogonal by using Reimannian gradients. Huang et al. (2017) consider the estimation of orthogonal matrix as an optimization over multiple dependent stiefel manifolds problem and solve it via eigenvalue decomposition on a proxy parameter matrix. Vorontsov et al. (2017) applied hard constraint on orthogonal transform update via Cayley transform. In this work, we construct the orthogonal matrix via Gram Schmidt process and the gradient is calculated automatically through autograd mechanism in PyTorch (Paszke et al., 2017).
3 Our Proposed Method
We consider knowledge graph as a collection of triples  $\mathcal{D} = \{(h,r,t)\}$  with  $V$  as the graph node set, and  $R$  as the graph edge set. Each triple has a head entity  $h$  and tail entity  $t$ , where  $h,t\in V$ . Relation  $r\in R$  connects two entities with direction from head to tail. As discussed in the introduction section, 1-to-N, N-to-1 and N-to-N relation prediction (Bordes et al., 2013; Wang et al., 2014) are difficult to deal with. They are addressed in our proposed approach by: 1) orthogonal relation transforms that operate on groups of embedding space. Each group is modeled and scored independently, and the final score is the sum of all group scores. Hence, each group could address different aspects of entity-relation pair and alleviate the 1-to-N and N-to-N relation mapping issues; and 2) directed graph context to integrate knowledge graph structure information to reduce the ambiguity.
Next, we first briefly review RotatE that motivates our orthogonal transform embedding (OTE), and then describe the proposed method in details.
3.1 RotatE
OTE is inspired by RotatE (Sun et al., 2019). In RotatE, the distance scoring is done via Hadamard production (element-wise) defined on the complex domain. Given a triple  $(h,r,t)$ , the corresponding embedding are  $e_h$ ,  $\theta_r$ ,  $e_t$ , where  $e_h$  and  $e_t \in \mathcal{R}^{2d}$ ,  $\theta_r \in \mathcal{R}^d$ , and  $d$  is the embedding dimension. For each dimension  $i$ ,  $e[2i]$  and  $e[2i + 1]$  are corresponding real and imaginary components. The projection  $\tilde{e}_t$  of  $t$  from corresponding relation and head
entities is conducted as an orthogonal transform as below:
$$
\begin{array}{l} \left[ \begin{array}{c} \tilde {e} _ {t} [ 2 i ] \\ \tilde {e} _ {t} [ 2 i + 1 ] \end{array} \right] = M _ {r} (i) \left[ \begin{array}{c} e _ {h} [ 2 i ] \\ e _ {h} [ 2 i + 1 ] \end{array} \right] \\ = \left[ \begin{array}{c c} \cos \theta_ {r} (i) & - \sin \theta_ {r} (i) \\ \sin \theta_ {r} (i) & \cos \theta_ {r} (i) \end{array} \right] \left[ \begin{array}{c} e _ {h} [ 2 i ] \\ e _ {h} [ 2 i + 1 ] \end{array} \right] \\ \end{array}
$$
where  $M_r(i)$  is a 2D orthogonal matrix derived from  $\theta_r$ .
Though RotatE is simple and effective for knowledge graph link prediction, it is defined in  $2D$  complex domain and thus has limited modeling capability. A natural extension is to apply similar operation on a higher dimensional space.
3.2 Orthogonal Transform Embedding (OTE)
We use  $e_h, M_r, e_t$  to represent embeddings of head, relation and tail entity, where  $e_h, e_t \in \mathcal{R}^d$ , and  $d$  is the dimension of the entity embedding. The entity embedding  $e_x$ , where  $x = \{h, t\}$ , is further divided into  $K$  sub-embeddings, e.g.,  $e_x = [e_x(1); \dots; e_x(K)]$ , where  $e_x(i) \in \mathcal{R}^{d_s}$  and  $d = K \cdot d_s$ .  $M_r$  is a collection of  $K$  linear transform matrix  $M_r = \{M_r(1), \dots, M_r(K)\}$ , and  $M_r(i) \in \mathcal{R}^{d_s \times d_s}$ .
For each sub-embedding  $e_t(i)$  of tail  $t$ , we define the projection from  $h$  and  $r$  to  $t$  as below:
$$
\tilde {e} _ {t} (i) = f _ {i} (h, r) = \phi \left(M _ {r} (i)\right) e _ {h} (i) \tag {1}
$$
where  $\phi$  is the Gram Schmidt process (see details in Section 3.3) applied to square matrix  $M_r(i)$ . The output transform  $\phi(M_r(i))$  is an orthogonal matrix derived from  $M_r(i)$ .  $\tilde{e}_t$  is the concatenation of all sub-vector  $\tilde{e}_t(i)$  from Eq. 1, e.g.,  $\tilde{e}_t = f(h,r) = [\tilde{e}_t(1);\dots;\tilde{e}_t(K)]$ . The  $L_2$  norm of  $e_h(i)$  is preserved after the orthogonal transform. We further use a scalar tensor  $s_r(i) \in \mathcal{R}^{d_s}$  to scale the  $L_2$  norm of each group of embedding separately. Eq. 1 is re-written as
$$
\tilde {e} _ {t} (i) = \operatorname {d i a g} \left(\exp \left(s _ {r} (i)\right)\right) \phi \left(M _ {r} (i)\right) e _ {h} (i) \tag {2}
$$
Then, the corresponding distance scoring function is defined as
$$
d ((h, r), t) = \sum_ {i = 1} ^ {K} \left(\left| \left| \tilde {e} _ {t} (i) - e _ {t} (i) \right| \right|\right) \tag {3}
$$
For each sub-embedding  $e_h(i)$  of head  $h$ , we define the projection from  $r$  and  $t$  to  $h$  as below:
$$
\tilde {e} _ {h} (i) = \operatorname {d i a g} \left(\exp \left(- s _ {r} (i)\right)\right) \phi \left(M _ {r} (i)\right) ^ {T} e _ {t} (i) \tag {4}
$$
where the reverse project from tail to head is simply transposing the  $\phi(M_r(i))$  and reversing the sign of  $s_r$ . Then, the corresponding distance scoring function is defined as
$$
d (h, (r, t)) = \sum_ {i = 1} ^ {K} \left(\left| \left| \tilde {e} _ {h} (i) - e _ {h} (i) \right| \right|\right). \tag {5}
$$
3.3 Gram Schmidt Process
We employ Gram-Schmidt process to orthogonalize a linear transform into an orthogonal transform (i.e.,  $\phi(M_r(i))$  in Section 3.2). The Gram-Schmidt process takes a set of tensor  $S = \{v_1, \dots, v_k\}$  for  $k \leq d_s$  and generates an orthogonal set  $S' = \{u_1, \dots, u_k\}$  that spans the same  $k$ -dimensional subspace of  $\mathcal{R}^{d_s}$  as  $S$ .
$$
t _ {i} = v _ {k} - \sum_ {j = 1} ^ {k - 1} \frac {\left\langle v _ {k} , t _ {j} \right\rangle}{\left\langle t _ {j} , t _ {j} \right\rangle} t _ {j} \tag {6}
$$
$$
u _ {i} = \frac {t _ {i}}{| | t _ {i} | |} \tag {7}
$$
where  $t_1 = v_1,||t||$  is the  $L_{2}$  norm of vector  $t$  and  $\langle v,t\rangle$  denotes the inner product of  $v$  and  $t$ .
Orthogonal transform has many desired properties, for example, the inverse matrix is obtained by simply transposing itself. It also preserves the  $L_{2}$  norm of a vector after the transform. For our work, we are just interested in its property to obtain inverse matrix by simple transposing. This saves the number of model parameters (see Table 3).
It can be easily proved that  $OTE$  has the ability to model and infer all three types of relation patterns: symmetry/antisymmetry, inversion, and composition as  $RotatE$  does. The proof is listed in Appendix A.
It should be noted that,  $M_r(i)$  is calculated every time in the neural networks forward computation to get orthogonal matrix  $\phi(M_r(i))$ , while the corresponding gradient is calculated and propagated back to  $M_r(i)$  via autograd computation within PyTorch during the backward computation. It eliminates the need of special gradient update schemes employed in previous hard constraint based orthogonal transform estimations (Harandi and Fernando, 2016; Vorontsov et al., 2017). In our experiments, we initialize  $M_r(i)$  to make sure they are with full rank<sup>1</sup>. During training, we also keep checking the determinant of  $M_r(i)$ . We find the update is fairly
<sup>1</sup>A real random matrix has full rank with probability 1 (Slinko, 2000). We use different random seeds to make sure the generated matrix is full rank.
stable that we don't observe any issues with subembedding dimensions varied from 5 to 100.
3.4 Directed Graph Context
The knowledge graph is a directed graph: valid triple  $(h,r,t)$  does not mean  $(t,r,h)$  is also valid. Therefore, for a given entity in knowledge graph, there are two kinds of context information: nodes that come into it and nodes that go out of it. Specially, in our paper, for each entity  $e$ , we consider the following two context settings:
1. If  $e$  is a tail, all the (head, relation) pairs in the training triples whose tail is  $e$  are defined as Head Relation Pair Context.
2. If  $e$  is a head, all the (relation, tail) pairs in the training triples whose head is  $e$  are defined as Relation Tail Pair Context.
Figure 1 demonstrates the computation of graph context for a testing triple (SergeiRachmaninoff, profession, Pianist). Edges for relation "profession" are colored as green. Entities marked with  $\circ$  are head entities to entity "Pianist", and these entities and corresponding relations to connect "Pianist" form the head relation pair context of "Pianist". While entities with  $\star$  are tail entities for entity "SergeiRachmaninoff". Those entities and corresponding relations are the relation tail graph context of entity "SergeiRachmaninoff".
3.4.1 Head Relation Pair Context
For a given tail  $t$ , all head-relation pairs  $(h', r')$  of the triples with tail as  $t$  are considered as its graph context and denoted as  $Ng(t)$ .
First, we compute the head-relation context representation  $\tilde{e}_t^c$  as the average from all these pairs in  $Ng(t)$  as below:
$$
\tilde {e} _ {t} ^ {c} = \frac {\sum_ {\left(h ^ {\prime} , r ^ {\prime}\right) \in N g (t)} f \left(h ^ {\prime} , r ^ {\prime}\right) + e _ {t}}{| N g (t) | + 1} \tag {8}
$$
where  $e_t$  is the embedding of the tail  $t$ ,  $f(h', r')$  is the representation of  $(h', r')$  induced from Eq. 2. We use  $e_t$  in Eq. 8 to make the computation of context representation possible when  $Ng(t)$  is empty. This can be viewed as a kind of additive smoothing for context representation computation.
Then, we compute the distance of the head-relation context of  $t$  and the corresponding orthogonal transform based representation of a triple
$(h,r,t)$  as follow.
$$
d _ {c} ((h, r), t) = \sum_ {i = 1} ^ {K} \left(\left| \left| \tilde {e} _ {t} (i) - \tilde {e} _ {t} ^ {c} (i) \right| \right|\right) \tag {9}
$$
where  $\tilde{e}_t(i)$  is computed from Eq. 2.
There is no new parameter introduced for the graph context modeling, since the message passing is done via OTE entity-relation project  $f(h', r')$ . The graph context can be easily applied to other translational embedding algorithms, such as RotatE and TransE etc, by replacing OTE.
3.4.2 Relation Tail Pair Context
For a given head  $h$ , all relation-tail pairs  $(r', t')$  of the triples with head as  $h$  are considered as its graph context and denoted as  $Ng(h)$ .
First, we compute the relation-tail context representation  $\tilde{e}_h^c$  as the average from all these pairs in  $Ng(h)$  as below:
$$
\tilde {e} _ {h} ^ {c} = \frac {\sum_ {\left(r ^ {\prime} , t ^ {\prime}\right) \in N g (h)} f \left(r ^ {\prime} , t ^ {\prime}\right) + e _ {h}}{\left| N g (h) \right| + 1} \tag {10}
$$
where  $f(r', t')$  is computed from Eq. 4.
Then, we compute the distance of the relation-tail context of  $h$  and the corresponding orthogonal transform based representation of a triple  $(h,r,t)$  as follow.
$$
d _ {c} (h, (r, t)) = \sum_ {i = 1} ^ {K} \left(\left| \left| \tilde {e} _ {h} (i) - \tilde {e} _ {h} ^ {c} (i) \right| \right|\right) \tag {11}
$$
where  $\tilde{e}_h(i)$  is computed from Eq. 4.
3.5 Scoring Function
We further combine all four distance scores (Eq. 3, Eq. 5, Eq. 9 and Eq. 11) discussed above as the final distance score of the graph contextual orthogonal transform embedding (GC-OTE) for training and inference
$$
\begin{array}{l} d _ {a l l} (h, r, t) = d ((h, r), t) + d _ {c} (h, (r, t)) \\ + d (h, (r, t)) + d _ {c} ((h, r), t). \tag {12} \\ \end{array}
$$
Therefore the full GC-OTE model can be seen as an ensemble of  $K$  local GC-OTE models. This view provides an intuitive explanation for the success of GC-OTE.
Optimization Self-adversarial negative sampling loss (Sun et al., 2019) is used to optimize the embedding in this work,
$$
\begin{array}{l} L = - \sum p (h ^ {\prime}, r, t ^ {\prime}) \log \sigma (d _ {a l l} (h ^ {\prime}, r, t ^ {\prime}) - \gamma) \\ - \log \sigma (\gamma - d _ {a l l} (h, r, t)) \tag {13} \\ \end{array}
$$
where  $\gamma$  is a fixed margin,  $\sigma$  is sigmoid function,  $(h^{\prime},r,t^{\prime})$  is negative triple, and  $p(h^{\prime},r,t^{\prime})$  is the negative sampling weight defined in (Sun et al., 2019).
Two commonly used benchmark datasets (FB15k-237 and WN18RR) are employed in this study to evaluate the performance of link prediction.
FB15k-237 (Toutanova and Chen, 2015) dataset contains knowledge base relation triples and textual mentions of Freebase entity pairs. The knowledge base triples are a subset of the FB15K (Bordes et al., 2013), originally derived from Freebase. The inverse relations are removed in FB15k-237.
WN18RR (Dettmers et al., 2018) is derived from WN18 (Bordes et al., 2013), which is a subset of WordNet. WN18 consists of 18 relations and 40,943 entities. However, many text triples obtained by inverting triples from the training set. Thus WN18RR (Dettmers et al., 2018) is created to ensure that the evaluation dataset does not have test leakage due to redundant inverse relation.
Dataset	FB15k-237	WN18RR
Entities	14,541	40,943
Relations	237	11
Train Edges	272,115	86,835
Val. Edges	17,535	3,034
Test Edges	20,466	3,134
Table 1: Statistics of datasets.
Each dataset is split into three sets for: training, validation and testing, which is same with the setting of (Sun et al., 2019). The statistics of two data sets are summarized at Table 1. Only triples in the training set are used to compute graph context.
Following the evaluation protocol in (Dettmers et al., 2018; Sun et al., 2019), each test triple  $(h,r,t)$  is measured under two scenarios: head focused  $(?,r,t)$  and tail focused  $(h,r,?)$ . For each case, the test triple is ranked among all triples with masked entity replaced by entities in knowledge graph. Those true triples observed in either train/validation/test set except the test triple will be excluded during evaluation. Top 1, 3, 10 (Hits@1, Hits@3 and Hits@10), and the Mean Reciprocal Rank (MRR) are reported in the experiments.