SlowGuess/ACL_2020_OCR huggingface dataset:
In this section, we present thorough empirical studies to evaluate and analyze our proposed SEEK framework. We first introduce the experimental setting. Then we evaluate our SEEK framework on the task of link prediction. Then, we study the influence of the number of segments  $k$  to the SEEK framework, and present the case studies to demonstrate why our SEEK framework has high effectiveness.
Datasets In our experiments, we firstly use a de facto benchmark dataset: FB15K. FB15K is a subset of the Freebase dataset (Bollacker et al., 2008), and we used the same training, validation and test set provided by (Bordes et al., 2013). We also use another two new datasets proposed in recent years: DB100K (Ding et al., 2018) and YAGO37 (Guo et al., 2018). DB100K was built from the mapping-based objects of core DBpedia (Bizer et al., 2009); YAGO37 was extracted from the core facts of YAGO3 (Mahdisoltani et al., 2013). Table 2 lists the statistics of the three datasets.
Dataset	#Ent	#Rel	#Train	#Valid	#Test
FB15K	14,951	1,345	483,142	50,000	59,071
DB100K	99,604	470	597,572	50,000	50,000
YAGO37	123,189	37	989,132	50,000	50,000
Table 2: Statistics of datasets.
Compared Methods There are many knowledge graph embedding methods developed in recent years. We categorize the compared methods as the following groups:
- Some simple knowledge graph embedding methods that have low time and space complexity, like TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), HolE (Nickel et al., 2016), ComplEx (Trouillon et al., 2016), and Analogy (Liu et al., 2017). Specifically, TransE is a translation based method, and others are the multi-linear dot product-based framework.
- Some methods that achieve state-of-the-art performance on DB100K and YAGO37, which include RUGE (Guo et al., 2018) and ComplExNNE+AER (Ding et al., 2018).
- Some latest methods that achieve current state-of-the-art performance on FB15K, including Single DistMult (Kadlec et al., 2017), ConvE (Dettmers et al., 2018), SimplE (Kazemi and Poole, 2018), RotatE (Sun et al., 2019), and DihEdral (Xu and Li, 2019).
- We evaluate the scoring function  $f_{2}$  to apply an ablation study for our approach. Then we can observe the respective effect of facilitating sufficient feature interactions and preserving the relation properties. Since the scoring function  $f_{2}$  can only preserve the symmetric property, we refer to it as Sym-SEEK.
Since our framework does not use additional information like text (Toutanova and Chen, 2015), relational path (Ebisu and Ichise, 2019), or external memory (Shen et al., 2017), we do not compare the methods with additional information. Moreover, we only compare our method with single models, and the Ensemble DistMult (Kadlec et al., 2017) is a simple ensemble of multiple different methods, so we do not compare with it.
Experimental Details We use the asynchronous stochastic gradient descent (SGD) with the learning rate adapted by AdaGrad (Duchi et al., 2011)
Methods	DB100K	YAGO37
MRR	Hits@N	MRR	Hits@N
1	3	10	1	3	10
TransE (Bordes et al., 2013)	0.111	1.6	16.4	27.0	0.303	21.8	33.6	47.5
DistMult (Yang et al., 2015)	0.233	11.5	30.1	44.8	0.365	26.2	41.1	57.5
HolE (Nickel et al., 2016)	0.260	18.2	30.9	41.1	0.380	28.8	42.0	55.1
ComplEx (Trouillon et al., 2016)	0.242	12.6	31.2	44.0	0.417	32.0	47.1	60.3
Analogy (Liu et al., 2017)	0.252	14.2	32.3	42.7	0.387	30.2	42.6	55.6
RUGE (Guo et al., 2018)	0.246	12.9	32.5	43.3	0.431	34.0	48.2	60.3
ComplEx-NNE+AER (Ding et al., 2018)	0.306	24.4	33.4	41.8	-	-	-	-
Sym-SEEK*	0.306	22.5	34.3	46.2	0.452	36.7	49.8	60.6
SEEK*	0.338	26.8	37.0	46.7	0.454	37.0	49.8	62.2
* Statistically significant improvements by independent  $t$ -test with  $p = 0.01$ .
to optimize our framework. The loss function of our SEEK framework is given by Equation 5. We conducted a grid search to find hyperparameters which maximize the results on validation set, by tuning number of segments  $k \in \{1,2,4,8,16,20\}$ , the dimension of embeddings  $D \in \{100,200,300,400\}$ ,  $L_{2}$  regularization parameter  $\lambda \in \{0.1,0.01,0.001,0.0001\}$  and the number of negative samples per true triple  $\eta \in \{10,50,100,500,1000\}$ . The optimal combinations of hyperparameters are  $k = 8$ ,  $D = 400$ ,  $\lambda = 0.001$ ,  $\eta = 1000$  on FB15K;  $k = 4$ ,  $D = 400$ ,  $\lambda = 0.01$ ,  $\eta = 100$  on DB100K; and  $k = 4$ ,  $D = 400$ ,  $\lambda = 0.001$ ,  $\eta = 200$  on YAGO37. We set the initial learning rate  $lr$  to 0.1 and the number of epochs to 100 for all datasets.
4.2 Link Prediction
We study the performance of our method on the task of link prediction, which is a prevalent task to evaluate the performance of knowledge graph embeddings. We used the same data preparation process as (Bordes et al., 2013). Specifically, we replace the head/tail entity of a true triple in the test set with other entities in the dataset and name these derived triples as corrupted triples. The goal of the link prediction task is to score the original true triples higher than the corrupted ones. We rank the triples by the results of the scoring function.
We use the MRR and Hit@N metrics to evaluate the ranking results: a) MRR: the mean reciprocal rank of original triples; b) Hits@N: the percentage rate of original triples ranked at the top  $n$  in prediction. For both metrics, we remove some of the corrupted triples that exist in datasets from the ranking results, which is also called filtered setting
Table 3: Results of link prediction on DB100K and YAGO37.
Methods	FB15K
MRR	Hits@N
1	3	10
TransE	0.380	23.1	47.2	47.1
DistMult	0.654	54.6	73.3	72.8
HolE	0.524	40.2	61.3	73.9
ComplEx	0.692	59.9	75.9	84.0
Analogy	0.725	64.6	78.5	85.4
RUGE	0.768	70.3	81.5	86.5
ComplEx-NNE+AER	0.803	76.1	83.1	87.4
Single DistMult	0.798	-	-	89.3
ConvE	0.745	67.0	80.1	87.3
Simple	0.727	66.0	77.3	83.8
RotatE	0.797	74.6	83.0	88.4
DihEdral	0.733	64.1	80.3	87.7
Sym-SEEK*	0.796	74.7	82.9	88.2
SEEK*	0.825	79.2	84.1	88.6
* Statistically significant improvements by independent  $t$ -test with  $p = 0.01$ .
Table 4: Results of link prediction on FB15K.
in (Bordes et al., 2013). We use Hits@1, Hits@3, and Hits@10 for the metrics of Hits@N.
Table 3 summarizes the results of link prediction on DB100K and YAGO37, and Table 4 shows the results on FB15K. Note, the results of compared methods on DB100K and YAGO37 are taken from (Ding et al., 2018; Guo et al., 2018); the results on FB15K are taken from (Kadlec et al., 2017; Ding et al., 2018; Kazemi and Poole, 2018; Sun et al., 2019; Xu and Li, 2019).
On the DB100K, SEEK outperforms the compared methods in all metrics, and the Sym-SEEK also can achieve a good performance. On the YAGO37, the SEEK and Sym-SEEK have a similar result and outperform other previous methods. The results on YAGO37 show that exploiting more
Figure 3: The influence of the number of segments  $k$  to the MRR and the running time of link prediction on FB15K.
feature interactions can significantly improve the performance of the embeddings on YAGO37 while preserving the semantic properties have a slight improvement. On FB15K, SEEK achieves the best performance on MRR, Hit@1 and Hit@3. Although SEEK is worse than the Single DistMult on the metrics of Hit@10, the Single DistMult is just a higher dimensional version of DistMult. The Single DistMult uses 512-dimensional embeddings, which is larger than the 400-dimensional embeddings of the SEEK framework. On the whole, our method's improvements on these datasets demonstrate that our method has a higher expressiveness.
4.3 Influence of the Number of Segments  $k$
In the SEEK framework, a larger number of segments  $k$  implies more feature interactions and higher computational cost. To empirically study the influence of the number of segments  $k$  to the performance and computation time of SEEK, we let  $k$  vary in  $\{1, 4, 8, 16, 20\}$  and fix all the other hyperparameters, then we observe the MRR and time costs for the link prediction task on the test set of FB15K.
Figure 3 shows the MRR and time costs of different segment counts  $k$  on FB15K. As we can see, changing  $k$  affects the performance of knowledge graph embeddings significantly. When  $k$  varies from 1 to 8, the performance is increased steadily. However, when  $k$  becomes even larger, no consistent and dramatic improvements observed on the FB15K dataset. This phenomenon suggests that excessive feature interactions cannot further improve performance. Therefore,  $k$  is a sensitive hyperparameter that needs to be tuned for the best performance given a dataset. Figure 3 also illustrates the influence of the number of segments on the performance of different segment counts  $k$  on FB15K.
a) Symmetric relations
Figure 4: The correct probabilities of four triples in DB100K and their reverse triples. The probabilities  $P_{1}$ ,  $P_{2}$  and  $P_{4}$  are corresponding to the scoring functions  $f_{1}$ ,  $f_{2}$  and  $f_{4}$ , respectively.
b) Antisymmetric relations
trates that the running time of SEEK is linear in  $k$ , and it verifies that the time complexity of SEEK is  $O(kd)$ .
4.4 Case Studies
We employ case studies to explain why our framework has a high expressiveness. Specifically, we utilize the scoring functions  $f_{1}$ ,  $f_{2}$  and  $f_{4}$  to train the embeddings of DB100K, respectively. Then we use the corresponding scoring functions to score the triples in the test set and their reverse triples, and we feed the scores to the sigmoid function to get the correct probabilities  $P_{1}$ ,  $P_{2}$  and  $P_{4}$  of each triple. Figure 4 shows the correct probabilities of some triples. In these triples, two triples have symmetric relations, and the other two have antisymmetric relations. On the triples with symmetric relations, the original triples in the test set and their reverse triples are true triples, and the scoring functions  $f_{1}, f_{2}, f_{4}$  can result in high probabilities on original and reverse triples. On the triples with antisymmetric relations, the reverse triples are false. Since the values of  $f_{1}(h,r,t)$  or  $f_{2}(h,r,t)$  are equal to  $f_{1}(t,r,h)$  or  $f_{2}(t,r,h)$ , the scoring functions  $f_{1}$  and  $f_{2}$  result in high probabilities on the reverse triples. But the scoring function  $f_{4}$ , which can model both symmetric and antisymmetric relations, results in low probabilities on the reverse triples. Meanwhile, we can also find that function  $f_{2}$  have higher probabilities than function  $f_{1}$  on the true triples. This phenomenon further explains that facilitating sufficient feature interactions can improve the expressiveness of embeddings.
In this paper, we propose a lightweight KGE framework (SEEK) that can improve the expressiveness of embeddings without increasing the model complexity. To this end, our framework focuses on designing scoring functions and highlights two critical characteristics: 1) facilitating sufficient feature interactions and 2) preserving various relation properties. Besides, as a general framework, SEEK can incorporate many existing models, such as DistMult, ComplEx, and HolE, as special cases. Our extensive experiments on widely used public benchmarks demonstrate the efficiency, the effectiveness, and the robustness of SEEK. In the future, we plan to extend the key insights of segmenting features and facilitating interactions to other representation learning problems.
This work is supported by the National Natural Science Foundation of China (U1711262, U1611264, U1711261, U1811261, U1811264, U1911203), National Key R&D Program of China (2018YFB1004404), Guangdong Basic and Applied Basic Research Foundation (2019B1515130001), Key R&D Program of Guangdong Province (2018B010107005).
Christian Bizer, Jens Lehmann, Georgi Kobilarov, Soren Auer, Christian Becker, Richard Cyganiak, and Sebastian Hellmann. 2009. Dbpedia-a crystallization point for the web of data. Journal of Web Semantics, 7(3):154-165.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of SIGMOD, pages 1247-1250.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Proceedings of NIPS, pages 2787-2795.
Tim Dettmers, Minervini Pasquale, Stenetorp Pontus, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In Proceedings of AAAI, pages 1811-1818.
Boyang Ding, Quan Wang, Bin Wang, and Li Guo. 2018. Improving knowledge graph embedding using simple constraints. In Proceedings of ACL, pages 110-121.
John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121-2159.
Takuma Ebisu and Ryutaro Ichise. 2019. Graph pattern entity ranking model for knowledge graph completion. arXiv preprint arXiv:1904.02856.
Shu Guo, Quan Wang, Lihong Wang, Bin Wang, and Li Guo. 2018. Knowledge graph embedding with iterative guidance from soft rules. In Proceedings of AAAI, pages 4816-4823.
Katsuhiko Hayashi and Masashi Shimbo. 2017. On the equivalence of holographic and complex embeddings for link prediction. arXiv preprint arXiv:1702.05563.
Xiao Huang, Jingyuan Zhang, Dingcheng Li, and Ping Li. 2019. Knowledge graph embedding based question answering. In Proceedings of WSDM, pages 105-113.
Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge graph embedding via dynamic mapping matrix. In Proceedings of ACLIJCNLP, pages 687-696.
Rudolf Kadlec, Ondrej Bajgar, and Jan Kleindienst. 2017. Knowledge base completion: Baselines strike back. arXiv preprint arXiv:1705.10744.
Seyed Mehran Kazemi and David Poole. 2018. Simple embedding for link prediction in knowledge graphs. In Proceedings of NIPS, pages 4289-4300.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of AAAI, pages 2181-2187.
Hanxiao Liu, Yuexin Wu, and Yiming Yang. 2017. Analogical inference for multi-relational embeddings. In Proceedings of ICML, pages 2168-2178.
Farzaneh Mahdisoltani, Joanna Biega, and Fabian M Suchanek. 2013. Yago3: A knowledge base from multilingual wikipedias. In Proceedings of CIDR.
Maximilian Nickel, Lorenzo Rosasco, Tomaso A Poggio, et al. 2016. Holographic embeddings of knowledge graphs. In Proceedings of AAAI, pages 1955-1961.
Yelong Shen, Po-Sen Huang, Ming-Wei Chang, and Jianfeng Gao. 2017. Modeling large-scale structured relationships with shared memory for knowledge base completion. In Proceedings of the 2nd Workshop on RepL4NLP, pages 57-68.
Baoxu Shi and Tim Weninger. 2017. Proje: Embedding projection for knowledge graph completion. In Proceedings of AAAI, pages 1236-1242.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. In Proceedings of ICLR.
Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on CVSC, pages 57-66.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Proceedings of ICML, pages 2071-2080.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, Nilesh Agrawal, and Partha Talukdar. 2019. Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions. arXiv preprint arXiv:1911.00219.
Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018. Dkn: Deep knowledge-aware network for news recommendation. In Proceedings of WWW, pages 1835-1844.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of AAAI, pages 1112-1119.
Qizhe Xie, Xuezhe Ma, Zihang Dai, and Eduard Hovy. 2017. An interpretable knowledge transfer model for knowledge base completion. In Proceedings of ACL, pages 950-962.
Canran Xu and Ruijiang Li. 2019. Relation embedding with dihedral group in knowledge graph. In Proceedings of ACL, pages 263-272.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In Proceedings of ICLR.
Fuzheng Zhang, Nicholas Jing Yuan, Defu Lian, Xing Xie, and Wei-Ying Ma. 2016. Collaborative knowledge base embedding for recommender systems. In Proceedings of SIGKDD, pages 353-362.