SlowGuess/ACL_2020_OCR huggingface dataset:
As for metrics, we adopt the standard Mean Re
Dataset	|R|	|E|	#Train	#Valid	#Test
JF17K	501	28,645	76,379	-	24,568
WikiPeople	193	47,765	305,725	38,223	38,281
Table 1: The statistics of the datasets.
ciprocal Rank (MRR) and Hits@  $N$ . For each n-ary test fact, one of its elements is removed and replaced by all the elements in  $E / R$ . These corrupted n-ary facts are fed into NeuInfer to obtain the final scores. Based on these scores, the n-ary facts are sorted in descending order, and the rank of the n-ary test fact is stored. Note that, except the n-ary test fact, other corrupted n-ary facts existing in the training/validation/test set, are discarded before sorting. This process is repeated for all other elements of the n-ary test fact. Then, MRR is the average of these reciprocal ranks, and Hits@  $N$  is the proportion of the ranks less than or equal to  $N$ .
Knowledge inference includes entity inference and relation inference. As presented in Table 1, the number of relations and attributes in each dataset is far less than that of entities and attribute values (on JF17K,  $|R| = 501$ , while  $|E| = 28,645$ ; on WikiPeople,  $|R| = 193$ , while  $|E| = 47,765$ ). That is, inferring a relation/attribute is much simpler than inferring an entity/attribute value. Therefore, we adopt MRR and Hits@{1,3,10} on entity inference, while pouring attention to more fine-grained metrics, i.e., MRR and Hits@1 on relation inference.
The hyper-parameters of NeuInfer are tuned via grid search in the following ranges: The embedding dimension  $k \in \{50,100\}$ , the batch size  $\beta \in \{128,256\}$ , the learning rate  $\lambda \in \{5e^{-6}, 1e^{-5}, 5e^{-5}, 1e^{-4}, 5e^{-4}, 1e^{-3}\}$ , the numbers  $n_1$  and  $n_2$  of the neural network layers of "hrt-FCNs" and "hrtav-FCNs" in  $\{1,2\}$ , the dimension  $d$  of the interaction vector  $\mathbf{o}_{\mathrm{hrta_i v_i}}$  in  $\{50,100,200,400,500,800,1000,1200\}$ , the weight factor  $w$  of the scores in  $\{0.1,0.2,\dots,0.9\}$ . The adopted optimal settings are:  $k = 100$ ,  $\beta = 128$ ,  $\lambda = 5e^{-5}$ ,  $n_1 = 2$ ,  $n_2 = 1$ ,  $d = 1200$ , and  $w = 0.1$  for JF17K;  $k = 100$ ,  $\beta = 128$ ,  $\lambda = 1e^{-4}$ ,  $n_1 = 1$ ,  $n_2 = 1$ ,  $d = 1000$ , and  $w = 0.3$  for WikiPeople.
5.3 Simple Knowledge Inference
Simple knowledge inference includes simple entity inference and simple relation inference. For an n-ary fact, they infer one of the entities/the relation in
6146
Method	JF17K	WikiPeople
MRR	Hits@1	Hits@3	Hits@10	MRR	Hits@1	Hits@3	Hits@10
RAE	0.310	0.219	0.334	0.504	0.172	0.102	0.182	0.320
NaLP	0.366	0.290	0.391	0.516	0.338	0.272	0.364	0.466
NeuInfer	0.517	0.436	0.553	0.675	0.350	0.282	0.381	0.467
the primary triple or the attribute value/attribute in an auxiliary description, given its other information.
5.3.1 Baselines
Knowledge inference methods on n-ary facts are scarce. The representative methods are m-TransH (Wen et al., 2016) and its modified version RAE (Zhang et al., 2018), and the state-of-the-art one is NaLP (Guan et al., 2019). As m-TransH is worse than RAE, following NaLP, we do not adopt it as a baseline.
5.3.2 Simple Entity Inference
The experimental results of simple entity inference are reported in Table 2. From the results, it can be observed that NeuInfer performs much better than the best baseline NaLP, which verifies the superiority of NeuInfer. Specifically, on JF17K, the performance gap between NeuInfer and NaLP is significant. In essence, 0.151 on MRR,  $14.6\%$  on Hits@1,  $16.2\%$  on Hits@3, and  $15.9\%$  on Hits@10. On WikiPeople, NeuInfer also outperforms NaLP. It testifies the strength of NeuInfer treating the information in the same n-ary fact discriminatingly. By differentiating the primary triple from other auxiliary description(s), NeuInfer considers the validity of the primary triple and the compatibility between the primary triple and its auxiliary description(s) to model each n-ary fact more appropriately and reasonably. Thus, it is not surprising that NeuInfer beats the baselines. And on simpler JF17K (see Section 5.1), NeuInfer gains more significant performance improvement than on WikiPeople.
5.3.3 Simple Relation Inference
Since RAE is deliberately developed only for simple entity inference, we compare NeuInfer only with NaLP on simple relation inference. Table 3 demonstrates the experimental results of simple relation inference. From the table, we can observe that NeuInfer outperforms NaLP consistently. Detailedly, on JF17K, the performance improvement of NeuInfer on MRR and Hits@1 is 0.036 and  $7.0\%$ , respectively; on WikiPeople, they are 0.030
and  $9.1\%$ , respectively. It is ascribed to the reasonable modeling of n-ary facts, which not only improves the performance of simple entity inference but also is beneficial to pick the exact right relations/attributes out.
Table 2: Experimental results of simple entity inference.
Method	JF17K	WikiPeople
MRR	Hits@1	MRR	Hits@1
NaLP	0.825	0.762	0.735	0.595
NeuInfer	0.861	0.832	0.765	0.686
Table 3: Experimental results of simple relation inference.
We perform an ablation study to look deep into the framework of NeuInfer. If we remove the compatibility evaluation component, NeuInfer is reduced to a method for binary but not n-ary facts. Since we handle knowledge inference on n-ary facts, it is inappropriate to remove this component. Thus, as an ablation, we only deactivate the validity evaluation component, denoted as  $\mathrm{NeuInfer}^{-}$ . The experimental comparison between NeuInfer and  $\mathrm{NeuInfer}^{-}$ is illustrated in Figure 2. It can be observed from the figure that NeuInfer outperforms  $\mathrm{NeuInfer}^{-}$ significantly. It suggests that the validity evaluation component plays a pivotal role in our method. Thus, each component of our method is necessary.
5.5 Flexible Knowledge Inference
The newly proposed flexible knowledge inference focuses on n-ary facts of arities greater than 2. It includes flexible entity inference and flexible relation inference. For an n-ary fact, they infer one of the entities/the relation in the primary triple given any number of its auxiliary description(s) or infer the attribute value/attribute in an auxiliary description given the primary triple and any number of other auxiliary description(s). In existing knowledge inference methods on n-ary facts, each n-ary fact is represented as a group of peer attributes and attribute values. These methods have not poured attention to the above flexible knowledge inference. Thus, we conduct this new type of task only on
6147
Figure 2: The experimental comparison between NeuInfer and NeuInfer-.
Dataset	Flexible entity inference	Flexible relation inference
MRR	Hits@1	Hits@3	Hits@10	MRR	Hits@1
JF17K	0.398	0.348	0.422	0.494	0.616	0.599
WikiPeople	0.200	0.161	0.208	0.276	0.477	0.416
Table 4: Experimental results of flexible knowledge inference.
NeuInfer. Before elaborating on the experimental results, let us look into the new test set used in this section first.
5.5.1 The New Test Set
We generate the new test set as follows:
- Collect the n-ary facts of arities greater than 2 from the test set.
- For each collected n-ary fact, compute all the subsets of the auxiliary description(s). The primary triple and each subset form a new n-ary fact, which is added to the candidate set.
- Remove the n-ary facts that also exist in the training/validation set from the candidate set and then remove the duplicate n-ary facts. The remaining n-ary facts form the new test set.
The size of the resulting new test set on JF17K is 34,784, and that on WikiPeople is 13,833.
5.5.2 Flexible Entity and Relation Inference
The experimental results of flexible entity and relation inference on these new test sets are presented in Table 4. It can be observed that NeuInfer well tackles flexible entity and relation inference on partial facts, and achieves excellent performance. We also attribute this to the reasonable modeling of n-ary facts. For each n-ary fact, NeuInfer distinguishes the primary triple from other auxiliary
description(s) and models them properly. Thus, NeuInfer well handles various types of entity and relation inference concerning the primary triple coupled with any number of its auxiliary description(s).
5.6 Performance under Different Scenarios
To further analyze the effectiveness of the proposed NeuInfer method, we look into the breakdown of its performance on different arities, as well as on primary triples and auxiliary descriptions. Without loss of generality, here we report only the experimental results on simple entity inference.
The test sets are grouped into binary and n-ary  $(n > 2)$  categories according to the arities of the facts. Table 5 presents the experimental results of simple entity inference on these two categories of JF17K and WikiPeople. From the tables, we can observe that NeuInfer consistently outperforms the baselines on both categories on simpler JF17K. On more difficult WikiPeople, NeuInfer is comparable to the best baseline NaLP on the binary category and gains much better performance on the n-ary category in terms of the fine-grained MRR and Hits@1. In general, NeuInfer performs much better on JF17K than on WikiPeople. We attribute this to the simplicity of JF17K.
Where does the above performance improvement come from? Is it from inferring the head/tail
6148
Dataset	Method	MRR	Hits@1	Hits@3	Hits@10
Binary	N-ary	Binary	N-ary	Binary	N-ary	Binary	N-ary
JF17K	RAE	0.115	0.397	0.050	0.294	0.108	0.434	0.247	0.618
NaLP	0.118	0.477	0.058	0.394	0.121	0.512	0.246	0.637
NeuInfer	0.267	0.628	0.173	0.554	0.300	0.666	0.462	0.770
WikiPeople	RAE	0.169	0.187	0.096	0.126	0.178	0.198	0.323	0.306
NaLP	0.351	0.283	0.291	0.187	0.374	0.322	0.465	0.471
NeuInfer	0.350	0.349	0.278	0.303	0.385	0.364	0.473	0.439
Table 5: Experimental results of simple entity inference on binary and n-ary categories of JF17K and WikiPeople.
Dataset	Method	MRR	Hits@1	Hits@3	Hits@10
Binary	N-ary	Overall	Binary	N-ary	Overall	Binary	N-ary	Overall	Binary	N-ary	Overall
JF17K	NaLP	0.118	0.456	0.313	0.058	0.369	0.237	0.121	0.491	0.334	0.246	0.625	0.464
NeuInfer	0.267	0.551	0.431	0.173	0.467	0.342	0.300	0.588	0.466	0.462	0.720	0.611
WikiPeople	NaLP	0.351	0.237	0.337	0.291	0.161	0.276	0.374	0.262	0.361	0.465	0.384	0.455
NeuInfer	0.350	0.280	0.342	0.278	0.225	0.272	0.385	0.299	0.382	0.473	0.375	0.463
Table 6: Detailed experimental results on inferring head/tail entities.
Method	JF17K	WikiPeople
MRR	Hits@1	Hits@3	Hits@10	MRR	Hits@1	Hits@3	Hits@10
NaLP	0.510	0.432	0.545	0.655	0.345	0.223	0.402	0.589
NeuInfer	0.746	0.687	0.787	0.848	0.443	0.408	0.453	0.516
Table 7: Experimental results on inferring attribute values.
entities in primary triples or the attribute values in auxiliary descriptions? To go deep into it, we study the performance of NeuInfer on inferring the head/tail entities and the attribute values and compare it with the best baseline NaLP. The detailed experimental results are demonstrated in Tables 6 and 7. It can be observed that NeuInfer brings more performance gain on inferring attribute values. It indicates that combining the validity of the primary triple and the compatibility between the primary triple and its auxiliary description(s) to model each n-ary fact is more effective than only considering the relatedness of attribute-value pairs in NaLP, especially for inferring attribute values.
In this paper, we distinguished the information in the same n-ary fact and represented each n-ary fact as a primary triple coupled with a set of its auxiliary description(s). We then proposed a neural network model, NeuInfer, for knowledge inference on n-ary facts. NeuInfer combines the validity evaluation of the primary triple and the compatibility evaluation of the n-ary fact to obtain the validity score of the n-ary fact. In this way, NeuInfer has the ability of well handling simple knowledge inference, which copes with the inference on whole
facts. Furthermore, NeuInfer is capable of dealing with the newly proposed flexible knowledge inference, which tackles the inference on partial facts consisting of a primary triple coupled with any number of its auxiliary descriptive attribute-value pair(s). Experimental results manifest the merits and superiority of NeuInfer. Particularly, on simple entity inference, NeuInfer outperforms the state-of-the-art method significantly in terms of all the metrics. NeuInfer improves the performance of Hits@3 even by  $16.2\%$  on JF17K.
In this paper, we use only n-ary facts in the datasets to conduct knowledge inference. For future works, to further improve the method, we will explore the introduction of additional information, such as rules and external texts.
The work is supported by the National Key Research and Development Program of China under grant 2016YFB1000902, the National Natural Science Foundation of China under grants U1911401, 61772501, U1836206, 91646120, and 61722211, the GFKJ Innovation Program, Beijing Academy of Artificial Intelligence (BAAI) under grant BAAI2019ZD0306, and the Lenovo-CAS Joint Lab Youth Scientist Project.
6149
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, pages 1247-1250.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Proceedings of the 26th International Conference on Neural Information Processing Systems, pages 2787-2795.
Mingyang Chen, Wen Zhang, Wei Zhang, Qiang Chen, and Huajun Chen. 2019. Meta relational learning for few-shot link prediction in knowledge graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4208-4217, Hong Kong, China.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2D knowledge graph embeddings. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pages 1811-1818.
Boyang Ding, Quan Wang, Bin Wang, and Li Guo. 2018. Improving knowledge graph embedding using simple constraints. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 110-121.
Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2015. Question answering over Freebase with multicolumn convolutional neural networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 260-269.
Takuma Ebisu and Ryutaro Ichise. 2018. TorusE: Knowledge graph embedding on a Lie group. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pages 1819-1826.
Saiping Guan, Xiaolong Jin, Yuzhhuo Wang, and Xueqi Cheng. 2018. Shared embedding based neural networks for knowledge graph completion. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 247-256.
Saiping Guan, Xiaolong Jin, Yuzhuo Wang, and Xueqi Cheng. 2019. Link prediction on n-ary relational data. In Proceedings of the 28th International Conference on World Wide Web, pages 583-593.
Shu Guo, Quan Wang, Bin Wang, Lihong Wang, and Li Guo. 2015. Semantically smooth knowledge graph embedding. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 84-94.
Prachi Jain, Pankaj Kumar, Soumen Chakrabarti, et al. 2018. Type-sensitive knowledge base inference without explicit type supervision. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 75-80.
Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge graph embedding via dynamic mapping matrix. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 687-696.
Yantao Jia, Yuanzhuo Wang, Hailun Lin, Xiaolong Jin, and Xueqi Cheng. 2016. Locally adaptive translation for knowledge graph embedding. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, pages 992-998.
Xiaotian Jiang, Quan Wang, and Bin Wang. 2019. Adaptive convolution for multi-relational learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 978-987, Minneapolis, Minnesota.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference for Learning Representations.
Yankai Lin, Zhiyuan Liu, Huanbo Luan, Maosong Sun, Siwei Rao, and Song Liu. 2015a. Modeling relation paths for representation learning of knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 705-714.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015b. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the 29th AAAI Conference on Artificial Intelligence, pages 2181-2187.
Denis Lukovnikov, Asja Fischer, Jens Lehmann, and Soren Auer. 2017. Neural network-based question answering over knowledge graphs on word and character level. In Proceedings of the 26th International Conference on World Wide Web, pages 1211-1220.
Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. 2019. Learning attention-based embeddings for relation prediction in knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4710-4723, Florence, Italy.
Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. 2018. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 327-333.
6150
Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2016. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11-33.
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on Machine Learning, pages 809-816.
Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: A core of semantic knowledge. In Proceedings of the 16th International Conference on World Wide Web, pages 697-706.
Yi Tay, Anh Tuan Luu, and Siu Cheung Hui. 2017. Non-parametric estimation of multiple embeddings for link prediction on dynamic knowledge graphs. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 1243-1249.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Eric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Proceedings of the 33rd International Conference on Machine Learning, pages 2071-2080.
Denny Vrandecic and Markus Krötzsch. 2014. Wiki-data: A free collaborative knowledgebase. Communications of the ACM, 57(10):78-85.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12):2724-2743.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the 28th AAAI Conference on Artificial Intelligence, pages 1112-1119.
Jianfeng Wen, Jianxin Li, Yongyi Mao, Shini Chen, and Richong Zhang. 2016. On the representation and embedding of knowledge bases beyond binary relations. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, pages 1300-1307.
Han Xiao, Minlie Huang, and Xiaoyan Zhu. 2016. TransG: A generative model for knowledge graph embedding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 2316-2325.
Richong Zhang, Junpeng Li, Jiajie Mei, and Yongyi Mao. 2018. Scalable instance reconstruction in knowledge bases via relatedness affiliated embedding. In Proceedings of the 27th International Conference on World Wide Web, pages 1185-1194.
6151