SlowGuess/ACL_2020_OCR huggingface dataset:
Hyper-parameter settings The hyper-parameters of our model are tuned by grid search during training process, including learning rate, embedding dimension  $d$  and sub-embedding dimension  $d_{s}$ . In our setting, the embedding dimension is defined as the number of parameters in each entity embedding. Each entity embedding consists of  $K$  sub-embeddings with dimension  $d_{s}$ , i.e.,  $d = K \times d_{s}$ . There are two steps in our model training: 1) the model is trained with OTE or RotatE models, and 2) graph context based models are fine tuned on these pre-trained models. The parameter settings are selected by the highest MRR with early stopping on the validation set. We use the adaptive moment (Adam) algorithm (Kingma and Ba, 2014) to train the models.
Specially, for FB15k-237, we set embedding dimension  $d = 400$ , sub-embedding dimension  $d_{s} = 20$ , and the learning rates to  $2e - 3$  and  $2e - 4$  for pre-training and fine-tuning stages respectively; for WN18RR dataset, we set  $d = 400$ ,  $d_{s} = 4$ , and the learning rates to  $1e - 4$  and  $3e - 5$  for pre-training and fine-tuning stages.
Implementation Our models are implemented by PyTorch and run on NVIDIA Tesla P40 Graphics Processing Units. The pre-training OTE takes 5 hours with 240,000 steps and fine-tuning GC-OTE takes 23 hours with 60,000 steps. Though, it takes more computation for graph context based model training, the inference could be efficient if both head and tail context representations are precomputed and saved for each entity in the knowledge graph.
In this section, we first present the results of link prediction, followed by the ablation study and error analysis of our models.
4.4.1 Results of Link Prediction
Table 2 compares the proposed models (OTE and graph context based GC-OTE) to several state-of-the-art models: including translational distance based TransE (Bordes et al., 2013), RotatE (Sun et al., 2019); semantic matching based DistMult (Yang et al., 2014), ComplEx (Trouillon et al., 2016), ConvE (Dettmers et al., 2018), TuckER (Balazevic et al., 2019) and QuatE (Zhang et al., 2019), and graph context information based R-GCN+ (Schlichtkrull et al., 2017), SACN (Shang et al., 2019) and A2N (Bansal et al., 2019). These
Model	FB15k-237	WN18RR
MRR	H1	H3	H10	MRR	H1	H3	H10
TransE	.294	-	-	.465	.226	-	-	.501
RotatE	.338	.241	.375	.533	.476	.428	.492	.571
DistMult	.241	.155	.263	.419	.43	.39	.44	.49
ComplEx	.247	.158	.275	.428	.44	.41	.46	.51
ConvE	.325	.237	.356	.501	.43	.40	.44	.52
QuatE	.348	.248	.382	.550	.488	.438	.508	.582
TurkER	.358	.266	.392	.544	.470	.443	.482	.526
R-GCN+	.249	.151	.264	.417	-	-	-	-
SACN	.352	.261	.385	.536	.47	.43	.48	.54
A2N	.317	.232	.348	.486	.45	.42	.46	.51
OTE	.351	.258	.388	.537	.485	.437	.502	.587
GC-OTE	.361	.267	.396	.550	.491	.442	.511	.583
baseline numbers are quoted directly from published papers.
From Table 2, we observe that: 1) on FB15k-237, OTE outperforms RotatE, and GC-OTE outperforms all other models on all metrics. Specifically MRR is improved from 0.338 in RotatE, to 0.361, about  $7\%$  relative performance improvement. OTE which increases sub-embedding dimension from 2 to 20, and graph context each contributes about half the improvement; 2) on WN18RR, OTE outperforms RotatE and GC-OTE achieves the new state-of-the-art results (as far as we know from published papers). These results show the effectiveness of the proposed OTE and graph context for the task of predicting missing links in knowledge graph.
Moreover, GC-OTE improves more on FB15k-237 than on WN18RR. This is because FB15k-237 has richer graph structure context compared to WN18RR: an average of 19 edges per node v.s. 2 edges per node in WN18RR. These results indicate that the proposed method GC-OTE is more effective on data set with rich context structure information.
Table 2: Link prediction for FB15k-237 and WN18RR on test sets.
Model	ds	MRR	@10	#param
RotatE-S	-	.330	.515	5.9
RotatE-L	-	.340	.530	29.3
OTE	2	.327	.511	6.1
OTE	20	.355	.540	7.8
OTE - scalar	20	.352	.535	7.7
LNE	20	.354	.538	9.6
GC-RotatE-L	-	.354	.546	29.3
GC-OTE	20	.367	.555	7.8
Table 3: Ablation study on FB15k-237 validation set.
4.4.2 Ablation Study
Table 3 shows the results of ablation study of the proposed models and compares the number of model parameters with RotatE on FB15k-237 validation set. We perform the ablation study with
embedding dimension of 400. The entity embedding dimension for RotatE-S and RotatE-L are 400 and 2000, respectively.
First we notice that increasing embedding size from 400 to 2000 makes RotatE model size more than quadrupled while the performance gain is very limited (Row 1 and 2 in Table 3); increasing group embedding size from 2 to 20 does not increase the model size of OTE much, but with nice performance gain (Row 3 and 4 in Table 3). The model size of OTE is less than one-third of the size of RotatE-L but with better performance. This shows the effectiveness of the OTE.
We examine the proposed model in terms of the following aspects:
Impact of sub-embedding dimension: we fix the embedding dimension as 400, and increase the sub-embedding dimension  $d_{s}$  from 2 to 20, the MRR of OTE is improved from 0.327 to 0.355 (See Row 3 and Row 4). For RotatE, the entity is embedded in complex vector space, this is similar to our setting with sub-embedding dimension  $= 2$ . Our results show that increasing the sub-dimension with OTE is beneficial to link prediction.
Impact of orthogonal transform: we replace the orthogonal transform operation in OTE with two different settings, 1) removing the diagonal scalar tensor as Eq. 1 (See OTE-scalar) and 2) using normal linear transform rather than orthogonal transform (See LNE). Both settings lead to MRR degradation. This indicates the proposed orthogonal transform is effective in modeling the relation patterns which are helpful for link prediction.
Impact of graph context: we add the graph context based model to both OTE (See GC-OTE) and RotatE-L (See GC-RotatE-L). We observe that MRRs are improved for both RotatE-L and OTE. This shows the importance of modeling context information for the task of link prediction.
Figure 2: FB15k-237 for OTE with different subembedding dimension.
Sub-embedding dimension size: in Table 3 we show that increasing sub-embedding dimension brings a nice improvement on MRR. Is the larger size always better? Figure 2 shows the impact of  $d_{s}$  on the OTE performance with the changing of sub-embedding size. We fix the entity embedding dimension as 400, and vary the sub-embedding size from 2, 5, 10, 20, 50, all the way to 100. The blue line and green bar represent MRR and  $H@10$  value, respectively.
From Figure 2 we observe that, both MRR and Hit@10 are improved and slowly saturated around  $d_{s} = 20$  The similar experiments are also conducted on WN18RR data set and we find the best sub-embedding dimension is 4 on WN18RR.
Type	Num.	RotatE-L	GC-OTE
H	T	A	H	T	A
1-to-N	2255	.710	.169	.440	.718	.204	.461
N-to-1	5460	.156	.850	.503	.209	.863	.536
N-to-N	9763	.490	.631	.561	.508	.651	.579
Table 4: H@10 from FB15-237 validation set by categories (1-to-N, N-to-1 and N-to-N).
4.4.3 Error Analysis
We present error analysis of the proposed model on 1-to-N, N-to-1 and N-to-N relation predictions on FB15k-237. Table 4 shows results in terms of Hit@10, where “Num.” is the number of triples in the validation set belonging to the corresponding category, “H”/“T” represents the experiment to predict head entity /tail entity, and “A” denotes average result for both “H” and “T”.
Assume  $c(h, r)$  and  $c(r, t)$  are the number of  $(h, r)$  and  $(r, t)$  pairs appeared in triples from the training set respectively. A triple  $(h, r, t)$  from the validation set is considered as one of the categories in the following:
$$
(h, r, t) = \left\{ \begin{array}{l} \mathrm {N - t o - 1 , i f} c (h, r) > 1 \text {a n d} c (r, t) \leq 1 \\ 1 \text {- t o - N , i f} c (h, r) \leq 1 \text {a n d} c (r, t) > 1 \\ \mathrm {N - t o - N , i f} c (h, r) > 1 \text {a n d} c (r, t) > 1 \\ \text {o t h e r .} \end{array} \right.
$$
From Table 4 we observe that, comparing to  $RotatE$  large model, the proposed model gets better
Hit@10 on all cases, especially for the difficult cases when we attempt to predicting the head entity for 1-to-N/N-to-N relation type, and tail entity in N-to-1/N-to-N relation type. The reason is because that in the proposed model, the groupings of sub-embedding relation pairs in OTE and graph context modeling both are helpful to distinguish N different tails/heads when they share the same (head, rel)/(rel, tail).
In this paper we propose a new distance-based knowledge graph embedding for link prediction. It includes two-folds. First, OTE extends the modeling of RotatE from 2D complex domain to high dimensional space with orthogonal relation transforms. Second, graph context is proposed to integrate graph structure information into the distance scoring function to measure the plausibility of the triples during training and inference.
The proposed approach effectively improves prediction accuracy on the difficult N-to-1, 1-to-N and N-to-N link predictions. Experimental results on standard benchmark FB15k-237 and WN18RR show that OTE improves consistently over RotatE, the state-of-the-art distance-based embedding model, especially on FB15k-237 with many high in-degree nodes. On WN18RR our model achieves the new state-of-the-art results. This work is partially supported by Beijing Academy of Artificial Intelligence (BAAI).
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In The semantic web.
Ivana Balazevic, Carl Allen, and Timothy Hospedales. 2019. TuckER: Tensor factorization for knowledge graph completion. In EMNLP.
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. 2018. Can we gain more from orthogonality regularizations in training deep networks? In NeurIPS.
Trapit Bansal, Da-Cheng Juan, Shrividya Ravi, and Andrew McCallum. 2019. A2N: Attending to neighbors for knowledge graph inference. In ACL.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD.
Antoine Bordes, Sumit Chopra, and Jason Weston. 2014. Question answering with subgraph embeddings. In EMNLP.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NeurIPS.
Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R Hruschka Jr, and Tom M Mitchell. 2010. Toward an architecture for never-ending language learning. In AAAI.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2D knowledge graph embeddings. In AAAI.
Mehrtash Harandi and Basura Fernando. 2016. Generalized backpropagation, étude de cas: Orthogonality. ArXiv.
Lei Huang, Xianglong Liu, Bo Lang, Adams Wei Yu, Yongliang Wang, and Bao Qin Li. 2017. Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks. In AAAI.
Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge graph embedding via dynamic mapping matrix. In ACL.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In *ICLR*.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. In ICLR.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In AAAI.
Farzaneh Mahdisoltani, Joanna Biega, and Fabian M Suchanek. 2013. Yago3: A knowledge base from multilingual wikipedias. In CIDR.
Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. 2019. Learning attention-based embeddings for relation prediction in knowledge graphs. In ACL.
Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. 2017. A novel embedding model for knowledge base completion based on convolutional neural network. arXiv preprint arXiv:1712.02121.
Dai Quoc Nguyen, Thanh Vu, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. 2019. A Capsule Network-based Embedding Model for Knowledge Graph Completion and Search Personalization. In NAACL.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. 2013. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In ICLR.
Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2017. Modeling relational data with graph convolutional networks. In ESWC.
Chao Shang, Yun Tang, Jing Huang, Jinbo Bi, Xiaodong He, and Bowen Zhou. 2019. End-to-end structure-aware convolutional networks for knowledge base completion. In AAAI.
Arkadii Slinko. 2000. A generalization of Komlós theorem on random matrices. Univ., Department of Mathematics, School of Mathematical and Information.
Zhiqing Sun, Zhi-Hong Deng, Jing Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. In ICLR.
Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In ICML, pages 2071-2080.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. In ICLR.
Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Christopher Joseph Pal. 2017. On orthogonality and learning recurrent networks with long term dependencies. In ICML.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. TKDE, 29:2724-2743.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In AAAI.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding entities and relations for learning and inference in knowledge bases. In ICLR.
Fuzheng Zhang, Nicholas Jing Yuan, Defu Lian, Xing Xie, and Wei-Ying Ma. 2016. Collaborative knowledge base embedding for recommender systems. In KDD.
Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. 2019. _Quaternion knowledge graph embeddings_. In _NeurIPS_.
A Discussion on the Ability of Pattern Modeling and Inference
It can be proved that OTE can infer all three types of relation patterns, e.g., symmetry/antisymmetry, inversion and composition patterns.
A.1 Symmetry/antisymmetry
If  $e_t = f(r, h)$  and  $e_h = f(r, t)$  hold, we have
$$
\begin{array}{l} e _ {t} = \operatorname {d i a g} \left(\exp \left(s _ {r}\right)\right) \phi \left(M _ {r}\right) \\ \operatorname {d i a g} \left(\exp \left(s _ {r}\right)\right) \phi \left(M _ {r}\right) e _ {t} \\ \Rightarrow \quad \phi (M _ {r}) \phi (M _ {r}) = I \\ s _ {r} = 0 \\ \end{array}
$$
In other words, if  $\phi(M_r)$  is a symmetry matrix and no scale is applied, the relation is symmetry relation.
If the relation is antisymmetry, e.g.,  $e_t = f(r,h)$  and  $e_h \neq f(r,t)$ , we just need to one of the  $\phi(M_r(i))$  is not symmetry matrix or  $s_r(i) \neq 0$ .
A.2 Inversion
If  $e_2 = f(r_1, e_1)$  and  $e_1 = f(r_2, e_2)$  hold, we have
$$
\begin{array}{l} e _ {2} = \operatorname {d i a g} \left(\exp \left(s _ {r _ {1}}\right)\right) \phi \left(M _ {r _ {1}}\right) \\ \operatorname {d i a g} \left(\exp \left(s _ {r _ {2}}\right)\right) \phi \left(M _ {r _ {2}}\right) e _ {2} \\ \end{array}
$$
In other words, if  $\text{diag}(\exp(s_{r_1}))\phi(Mr_1) = \phi(M_{r_2})^T \text{diag}(\exp(-s_{r_2}))$ , the relation  $r_2$  is inverse relation of  $r_1$ .
A.3 Composition
If  $e_2 = f(r_1, e_1)$ ,  $e_3 = f(r_2, e_2)$  and  $e_3 = f(r_3, e_1)$  hold, we have
$$
\begin{array}{l} \operatorname {d i a g} \left(\exp \left(s _ {r _ {3}}\right)\right) \phi \left(M _ {3}\right) e _ {1} = \\ d i a g \left(\exp \left(s _ {r _ {2}}\right)\right) \phi \left(M _ {2}\right) \\ \operatorname {d i a g} \left(\exp \left(s _ {r _ {1}}\right)\right) \phi \left(M _ {1}\right) e _ {1} \\ \end{array}
$$
It means if  $\text{diag}(\exp(s_{r_3}))\phi(M_3)$  is equal to  $\text{diag}(\exp(s_{r_2}))\phi(M_2)\text{diag}(\exp(s_{r_1}))\phi(M_1)$  then relation  $r_3$  is composition of relation  $r_1$  and  $r_2$ .