SlowGuess/ACL_2020_OCR huggingface dataset:
We conduct the experiments on two real-world datasets  $(\mathcal{D})$  widely used in KG embedding literature, i.e. FB15k (Bordes et al., 2013) and YAGO43k (Moon et al., 2017), which are subsets of Freebase (Bollacker et al., 2008) and YAGO (Suchanek et al., 2007) respectively. They consist of triples, each of which is formed as (left entity, relationship, right entity). We utilize two entity type data  $(\mathcal{H},$  each of it is formed as (entity, entity type)) built in (Moon et al., 2017), called FB15kET and YAGO43kET, in which the entity types are mapped to entities from FB15k and YAGO43k respectively.
Moreover, we build new type triple datasets  $(\mathcal{Z},$  each one in it is formed as (head type, relationship, tail type)), to train our model. They are built based on  $\mathcal{D}$  and  $\mathcal{H}$ . First, for each triple  $(e,r,\tilde{e})\in \mathcal{D}$ , we replace the head and the tail with their types
3We also call it ConnectE-(E2T+TRT), and use ConnectE $(\mathrm{E2T} + 0)$  to denote E2T for uniformity in the experiments.
6423
according to  $\mathcal{H}$ . The generated datasets are called FB15kTRT(full) and YAGO43kTRT(full). Second, considering about the scalability of the proposed approach for full KGs, we further modify the generation method of type triples, which is the major training bottleneck. We discard newly generated ones with low-frequency (i.e. #frequency = 1). After that the size of both FB15kTRT(full) and YAGO43kTRT(full) decreased by about  $90\%$ , called FB15kTRT disc.) and YAGO43kTRT disc.) respectively. The statistics of the datasets are showed in Table 2. For saving space, we put more data processing details (include cleaning  $\mathcal{H}$ , building  $\mathcal{Z}$ , etc.) on our github website.
Table 2: Statistics of  $\mathcal{D},\mathcal{H},\mathcal{Z}$  .
Dataset	#Ent	#Rel	#Train	#Valid	#Test
FB15k	14,951	1,345	483,142	50,000	59,071
YAGO43k	42,335	37	331,687	29,599	29,593
Dataset	#Ent	#Type	#Train	#Valid	#Test
FB15kET	14,951	3,851	136,618	15,749	15,780
YAGO43kET	41,723	45,182	375,853	42,739	42,750
Dataset	#Type	#Rel	#Train	Valid	Test
FB15kTRT(full)	3,851	1,345	2,015,338	-	-
FB15kTRT disc.)	2,060	614	231,315	-	-
YAGO43kTRT(full)	45,128	37	1,727,708	-	-
YAGO43kTRT disc.)	17,910	32	189,781	-	-
4.2 Entity Type Prediction
This task concentrates to complete a pair (entity, entity type) when its type is missing, which aims to verify the capability of our model for inferring missing entity type instances.
Evaluation Protocol. We focus on entity type prediction determined by Formula (3) and (4). We use ranking criteria for evaluation. Firstly for each test pair, we remove the type and replace it by each of the types in  $\mathcal{T}$  in turn. The function value of the negative pairs would be computed by the related models and then sorted by ascending order. We can obtain the exact rank of the correct type in the candidates. Finally, we use two metrics for comparison: (1) the mean reciprocal rank (MRR), and (2) the proportion of correct entities ranked in the top 1/3/10 (HITS@1/3/10)(%). Since the evaluation setting of "Raw" is not as accurate as "Filter" (Bordes et al., 2013), we only report the experimental results with latter setting in this paper.
$$
M R R = \frac {1}{| C |} \sum_ {i = 1} ^ {| C |} \frac {1}{r a n k _ {i}} ,
$$
where  $C$  is a set of test pairs, and  $\text{rank}_i$  is the rank position of the true entity type for the  $i$ -th pair.
Implementation. The results of entity type prediction are shown in Table 3, where the results for the baselines are directly taken from original literature (Moon et al., 2017). We do not choose LM and PEM (Neelakantan et al., 2015) as baselines since they do not utilize triple knowledge, thus it is not fair to compare with them. For training our model, we select the learning rate  $\alpha \in \{0.1, 0.05, 0.001\}$ , the margins  $\gamma_1, \gamma_2, \gamma_3 \in \{0.5, 1, 2, 5, 10\}$ , the embedding dimension pairs  $(\kappa, \ell) \in \{(100, 50), (150, 75), (200, 100), (250, 125)\}$ , and the weight  $\lambda \in \{0.5, 0.65, 0.85, 0.95\}$ . We use negative sampling, and gradient descent with AdaGrad as our optimization approach to improve convergence performance. During the initialization process, each embedding vector of the entities, entity types and relationships is initialized with a random number following a uniform distribution  $-\sqrt{6} / (m + n)$ , where  $n \in \{\#Ent, \#Type, \#Rel\}$  and  $m \in \{\kappa, \ell\}$ . During the whole training process, we normalize the entity embeddings after each epoch.
We select the parameters based on MRR in valid dataset. The optimal configurations are:  $\{\alpha = 0.1, \gamma_1 = \gamma_2 = \gamma_3 = 2, \kappa = 200, \ell = 100, \lambda = 0.85\}$  on FB15k/ET/TRT;  $\{\alpha = 0.1, \gamma_1 = \gamma_2 = \gamma_3 = 1, \kappa = 250, \ell = 125, \lambda = 0.85\}$  on YAGO43k/ET/TRT. We run 800 epochs on both datasets, and the batch size is 4096.
Experimental Results. We can see from Table 3 that our ConnectEs outperform all baselines for entity type prediction in terms of all metrics on FB15kET and YAGO43kET. It confirms the capability of ConnectEs in modeling with local typing and global triple knowledge and inferring missing entity type instances in KGs. The model ConnectE(E2T+TRT)(full) achieves the highest scores.
Analysis. (1) In E2T, we utilize a mapping matrix M which compresses entity embeddings into type embedding space, considering that entity type embedding represents common information of all the entities which belong to this type. The type embedding should be in a sharing subspace of entity embeddings. The experimental results of E2T compared with the baselines demonstrate that this assumption would be quite reasonable. (2) In  $\mathrm{E2T + TRT}$ , we build new type-relation-type data, and then connect them with entity type instances. This approach provides more direct useful information to (weakly) supervise entity type prediction. For example, given a fact that head entity Barack Obama belongs to type /people/person
6424
Table 3: Entity type prediction results. Evaluation of different models on FB15kET and YAGO43kET.
DATASET	FB15kET	YAGO43kET
METRICS	MRR	HITS@1	HITS@3	HITS@10	MRR	HITS@1	HITS@3	HITS@10
RESCAL (Nickel et al., 2011)	0.19	9.71	19.58	37.58	0.08	4.24	8.31	15.31
RES-ET (Moon et al., 2017)	0.24	12.17	27.92	50.72	0.09	4.32	9.62	19.40
HOLE (Nickel et al., 2016)	0.22	13.29	23.35	38.16	0.16	9.02	17.28	29.25
HOLE-ET (Moon et al., 2017)	0.42	29.40	48.04	66.73	0.18	10.28	20.13	34.90
TransE (Bordes et al., 2013)	0.45	31.51	51.45	73.93	0.21	12.63	23.24	38.93
TransE-ET (Moon et al., 2017)	0.46	33.56	52.96	71.16	0.18	9.19	19.41	35.58
ETE (Moon et al., 2017)	0.50	38.51	55.33	71.93	0.23	13.73	26.28	42.18
ConnectE-(E2T+0)	0.57 + .00	45.54 +/- .28	62.31 +/- .29	78.12 +/- .12	0.24 + .01	13.54 +/- .12	26.20 +/- .18	44.51 + .09
ConnectE-(E2T+TRT)(disc.)	0.59 + .01	48.54 +/- .71	63.66 +/- .39	78.27 +/- .16	0.27 + .01	15.1 + .15	29.14 +/- .13	47.08 + .09
ConnectE-(E2T+TRT)(full)	0.59 +/- .00	49.55 +/- .62	64.32 +/- .37	79.92 +/- .14	0.28 + .01	16.01 +/- .12	30.85 +/- .13	47.92 +/- .07
and the relationship born_in, we could make the best guess of the type of tail entity Honolulu as /location/location. Hence, the addition of type triples in ConnectE-(E2T+TRT) provides superior performance than ConnectE-(E2T+0). (3) Concerning about the scalability of our approach for big KGs, we utilize FB15kTRT disc.) and YAGO43kTRT disc.) for prediction, the training time of which reduced by  $90\%$  as the training data size decreased by  $90\%$ . Moreover, the results of ConnectE-(E2T+TRT)(disc.) show that it's comparable with the best ConnectE-(E2T+TRT)(full).
4.3 Entity Type Classification
This task aims to judge whether each entity type instance in testing data holds or not, which could be viewed as a binary classification problem.
Evaluation Protocol. Since there are no explicit negative entity type instances in existing KGs, in order to create datasets for classification, we build negative facts by randomly switching type from entity type pairs in validation and testing set with equal number of positive and negative examples. Inspired by the evaluation metric of triple classification in (Socher et al., 2013), we calculate the scores of all entity type instances based on model energy function, and rank all instances in testing set with these scores. Those instances with lower scores are considered to be true. We use precision/recall curves to show the performances of all models. Moreover, we also compare the accuracy among different models. We first use validate set to find best threshold  $\eta$ . For instance, if the model score  $\mathbf{S}_{e2t + trt}(e,t_e) \leq \eta$  in classification, the entity type instance will be classified to be positive, otherwise to be negative. The final accuracy is based on how many facts are classified correctly.
Implementation. We utilize the source codes and parameter settings of several baselines provided by (Moon et al., 2017) for this task. The optimal parameter settings for our proposed models are:
$\{\alpha = 0.1, \gamma_1 = \gamma_2 = \gamma_3 = 2, \kappa = 200, \ell = 100, \lambda = 0.85\}$  on FB15kET;  $\{\alpha = 0.1, \gamma_1 = \gamma_2 = \gamma_3 = 1, \kappa = 250, \ell = 125, \lambda = 0.85\}$  on YAGO43kET. In both datasets, we learn all the training data for 800 epochs and the batch size is 4096. After training, we firstly draw PR-curves with dynamic thresholds. We select the best threshold based on the accuracy in valid dataset, which is used to calculate the accuracy in test dataset.
Experimental Results. We draw the PR-curves for type classification task on both datasets in Fig.3. Note that we only report the results of ConnectE-(E2T+TRT)(disc.) not ConnectE-(E2T+TRT)(full), since the learning speed of the former is much more faster than the latter and its results are close to the best results of the latter. We can see from Fig.3 that when the recall rate is between  $0.88 \sim 0.97$ , ConnectE-(E2T+TRT)(disc.) model could achieve the highest precision rate on FB15kET. In other ranges, our ConnectE-(E2T+TRT)(disc.) model also shows comparable performance. The result is consistent on YAGO43kET. Specifically, ConnectE-(E2T+TRT)(disc.) achieves the best F1 score of  $94.66\%$  when recall  $= 94.27\%$  and precision  $= 95.05\%$  on FB15kET. Also, ConnectE-(E2T+TRT)(disc.) surpasses other models and gets F1 score of  $92.13\%$  when precision  $= 93.18\%$  and recall  $= 91.11\%$  on YAGO43kET. It confirms the capability of our model, for they could not only infer missing types in KGs, but also perform well in KG entity type classification.
Table 4 demonstrates the evaluation accuracy results of entity type classification, from which we can observe that: (1) On FB15kET, ConnectE(E2T+TRT)(disc.) achieves the best accuracy score  $(94.49\%)$ . Compared to the mostly related model ETE, our model shows  $0.48\%$  absolute performance improvement. On YAGO43kET, ConnectE(E2T+TRT)(disc.) model outperforms other models as well. The improvement of our model com
Figure 3: Entity type classification results (Precision/Recall Curve). Evaluate on FB15kET, YAGO43kET.
pared to ETE is almost  $1.51\%$ . (2) Comparing to the improvement on YAGO43kET, the advantage ConnectE-(E2T+TRT)(disc.) has over ConnectE(E2T+0) in this task on FB15kET seems to be insignificant, which indicates that the type triples in FB15kTRT have fewer contribution on entity type classification than ones in YAGO43kTRT. It may be partially caused by the fact that the number of relations in YAGO43k (#Rel=37) is far less than that in FB15k (#Rel=1,345), which could considerably influence the effectiveness of the type-relationship type training set. Due to the rareness of relationships in YAGO43k, each entity usually connects with a large number of other entities through one single relationships, which means that the magnitude of  $|P|$  and  $|Q|$  in the composite model scoring function are large. After averaging in ConnectE(E2T+TRT)(disc.), it could achieve more stable and significant results on YAGO43kET.
Table 4: Entity type classification results (accuracy).
Dataset	| FB15kET	| YAGO43kET
RESCAL-ET	| 90.02%	| 82.28%
HOLE-ET	| 93.23%	| 90.14%
TransE-ET	| 93.88%	| 90.76%
ETE	| 94.01%	| 90.82%
ConnectE (E2T+0)	| 94.45%	| 91.78%
ConnectE (E2T+TRT)(disc.)	| 94.49%	| 92.33%
Table 5 shows the examples of entity type prediction by our model from  $FB15k/ET/TRT$ , which demonstrate our motivation of Mech. 2 that head type and tail type really maintain the relationship between head entity and tail entity. Given entity Peter Berg, TRT can find HITS@1 type pre
diction /people/person for it via the existing entity type assertion (New Youk, /location/location) and the relationship (/loc./loc./people_born_here) between them, i.e. PeterBerg - NewYork + /location/location= /people/person.
Table 5: Entity type prediction examples. Extraction from  ${FB15k}/{ET}/{TRT}$  .
Type prediction: HIT@1	Rel	Tail type
1	type=? /people/person	/location/location/ people_born_here	/location/location
head entity	Peter Berg	New York	tail entity
Gus Van Sant	Louisville
2	type=? /americancomedy/movie	/film/film/ directed_by	/film/director
head entity	Very Bad Things	Peter Berg	tail entity
Rush Hour	Brett Ratner
3	type=? /medicine/disease	people/cause_of _death/people	/people/person
head entity	Myocardial infarction	Dick Clark	tail entity
Pancreatic cancer	John Hurt
In this paper, we described a framework for leveraging global triple knowledge to improve KG entity typing by training not only on (entity, entity type) assertions but also using newly generated (head type, relationship, tail type) type triples. Specifically, we propose two novel embedding-based models to encode entity type instances and entity type triples respectively. The connection of both models is utilized to infer missing entity type instances. The empirical experiments demonstrate the effectiveness of our proposed model. Our modeling method is general and should apply to other type-oriented tasks. Next, we are considering to use this framework to conduct KG entity type noise detection.
The authors would like to thank all anonymous reviewers for their insightful comments. We also want to thank Zhiyuan Liu (Tsinghua University) and Linmei Hu (BUPT) for their useful suggestions and comments on early drafts. This work was supported by the National Natural Science Foundation of China under Grant No.61922085, 61906159, the Sichuan Science and Technology Program under Grant No.2018JY0607, the Fundamental Research Funds for the Central Universities under Grant No.JBK2003008, Fintech Innovation Center, and Financial Intelligence and Financial Engineering Key Laboratory of Sichuan Province.
Abhishek Abhishek, Ashish Anand, and Amit Awekar. 2017. Fine-grained entity type classification by jointly learning representations and label embeddings. In Proceedings of EACL, page 797807.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of EMNLP, pages 1533-1544.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of SIGMOD, pages 1247-1250.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Proceedings of NeurIPS, pages 2787-2795.
Eunsol Choi, Omer Levy, Yejin Choi, and Luke S. Zettlemoyer. 2018. Ultra-fine entity typing. In Proceedings of ACL.
Tim Dettmers, Minervini Pasquale, Stenetorp Pontus, and Sebastian Riedel. 2017. Convolutional 2d knowledge graph embeddings. In Proceedings of AAAI, pages 1811-1818.
Boyang Ding, Quan Wang, Bin Wang, and Li Guo. 2018. Improving knowledge graph embedding using simple constraints. In Proceedings of ACL, pages 110-121, Melbourne, Australia.
Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In Proceedings of SIGKDD, pages 601-610.
Hady Elsahar, Christophe Gravier, and Frederique Laforest. 2018. Zero-shot question generation from knowledge graphs for unseen predicates and entity types. In Proceedings of NAACL, pages 218-228.
Nitish Gupta, Sameer Singh, and Dan Roth. 2017. Entity linking via joint encoding of types, descriptions, and context. In Proceedings of EMNLP, pages 2681-2690.
Hannaneh Hajishirzi, Leila Zilles, Daniel S. Weld, and Luke Zettlemoyer. 2013. Joint coreference resolution and named-entity linking with multi-pass sieves. In Proceedings of EMNLP, pages 289-299.
Junheng Hao, Muhao Chen, Wenchao Yu, Yizhou Sun, and Wei Wang. 2019. Universal representation learning of knowledge bases by jointly embedding instances and ontological concepts. In Proceedings of KDD 2019.
Prachi Jain, Pankaj Kumar, and Soumen Chakrabarti. 2018. Type-sensitive knowledge base inference without explicit type supervision. In Proceedings of ACL.
Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. 2016. Knowledge graph completion with adaptive sparse transfer matrix. In Proceedings of AAAI.
Hailong Jin, Lei Hou, Juanzi Li, and Tiansi Dong. 2018. Attributed and predictive entity embedding for fine-grained entity typing in knowledge bases. In Proceedings of COLING 2018, pages 282-292.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In arXiv preprint arXiv:1412.6980.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of AAAI, pages 2181-2187.
Xin Lv, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2018. Differentiating concepts and instances for knowledge graph embedding. In Proceedings of EMNLP 2018, page 19711979.
Chang Moon, Paul Jones, and Nagiza F. Samatova. 2017. Learning entity type embeddings for knowledge graph completion. In Proceedings of CIKM, pages 2215-2218.
Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. 2019. Learning attention-based embeddings for relation prediction in knowledge graphs. In Proceedings of ACL.
Arvind Neelakantan, Ming-Wei Chang, and . 2015. Inferring missing entity type instances for knowledge base completion: New dataset and methods. In Proceedings of NAACL 2012, page 515525.
6427
Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2016. Holographic embeddings of knowledge graphs. In Proceedings of AAAI, pages 1955-1961.
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way model for collective learning on multi-relational data. In Proceedings of ICML, pages 809-816.
Patrick Pantel, Thomas Lin, and Michael Gamon. 2012. Mining entity types from query logs via user intent modeling. In Proceedings of ACL, pages 563-571.
Richard Socher, Danqi Chen, Christopher D. Manning, and Andrew Y. Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In Proceedings of NeurIPS, pages 926-934.
Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In Proceedings of WWW, pages 697-706.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. TKDE, 29(12):2724-2743.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of AAAI, pages 1112-1119.
Ruobing Xie, Zhiyuan Liu, and Maosong Sun. 2016. Representation learning of knowledge graphs with hierarchical types. In Proceedings of IJCAI, pages 2965-2971.
Peng Xu and Denilson Barbosa. 2018. Neural fine grained entity type classification with hierarchy aware loss. In Proceedings of NAACL.
Yadollah Yaghoobzadeh, Heike Adel, and Hinrich Schtze. 2018. Corpus-level fine-grained entity typing. Journal of Artificial Intelligence Research, 61:835-862.
Limin Yao, Sebastian Riedel, , and Andrew McCallum. 2013. Universal schema for entity type prediction. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, pages 79-84.
Dani Yogatama, Daniel Gillick, and Nevena Lazic. 2015. Embedding methods for fine grained entity type classification. In Proceedings of ACL, page 291296.
Zheng Yuan and Doug Downey. 2018. Otyper: A neural architecture for open named entity typing. In Proceedings of AAAI.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING, pages 23-29.
Richong Zhang, Fanshuang Kong, Chenyue Wang, and Yongyi Mao. 2018. Embedding of hierarchically typed knowledge bases. In Proceedings of AAAI.
Ben Zhou, Daniel Khashabi, Chen-Tse Tsai, and Dan Roth. 2018. Zero-shot open entity typing as type-compatible grounding. In Proceedings of EMNLP, pages 2065-2076.
6428