SlowGuess/ACL_2020_OCR huggingface dataset:
Saiping Guan, Xiaolong Jin, Jiafeng Guo, Yuanzhuo Wang, and Xueqi Cheng
School of Computer Science and Technology, University of Chinese Academy of Sciences;
CAS Key Laboratory of Network Data Science and Technology, Institute of
Computing Technology, Chinese Academy of Sciences
{guansaiping, jinxiaolong, guojiafeng, wangyuanzhuo, cxq}@ict.ac.cn
Knowledge inference on knowledge graph has attracted extensive attention, which aims to find out connotative valid facts in knowledge graph and is very helpful for improving the performance of many downstream applications. However, researchers have mainly poured attention to knowledge inference on binary facts. The studies on n-ary facts are relatively scarcer, although they are also ubiquitous in the real world. Therefore, this paper addresses knowledge inference on n-ary facts. We represent each n-ary fact as a primary triple coupled with a set of its auxiliary descriptive attribute-value pair(s). We further propose a neural network model, NeuInfer, for knowledge inference on n-ary facts. Besides handling the common task to infer an unknown element in a whole fact, NeuInfer can cope with a new type of task, flexible knowledge inference. It aims to infer an unknown element in a partial fact consisting of the primary triple coupled with any number of its auxiliary description(s). Experimental results demonstrate the remarkable superiority of NeuInfer.
With the introduction of connotative valid facts, knowledge inference on knowledge graph improves the performance of many downstream applications, such as vertical search and question answering (Dong et al., 2015; Lukovnikov et al., 2017). Existing studies (Nickel et al., 2016; Wang et al., 2017) mainly focus on knowledge inference on binary facts with two entities connected with a certain binary relation, represented as triples, (head entity, relation, tail entity). They attempt to infer the unknown head/tail entity or the unknown relation of a given binary fact. However, n-ary facts involving more than two entities are also ubiquitous. For example, in Freebase, more than  $1/3$  entities participate in n-ary facts (Wen et al., 2016). The
fact that John Bardeen received Nobel Prize in Physics in 1956 together with Walter Houser Brattain and William Shockley<sup>1</sup> is a typical 5-ary fact. So far, only a few studies (Wen et al., 2016; Zhang et al., 2018; Guan et al., 2019) have tried to address knowledge inference on n-ary facts.
In existing studies for knowledge inference on n-ary facts, each n-ary fact is represented as a group of peer attributes and attribute values. In practice, for each n-ary fact, there is usually a primary triple (the main focus of the n-ary fact), and other attributes along with the corresponding attribute values are its auxiliary descriptions. Take the above 5-ary fact for example, the primary triple is (John Bardeen, award-received, Nobel Prize in Physics), and other attribute-value pairs including point-in-time : 1956, together-with : Walter Houser Brattain and together-with : William Shockley are its auxiliary descriptions. Actually, in YAGO (Suchanek et al., 2007) and Wikidata (Vrandecic and Krötzsch, 2014), a primary triple is identified for each n-ary fact.
The above 5-ary fact is a relatively complete example. In the real-world scenario, many n-ary facts appear as only partial ones, each consisting of a primary triple and a subset of its auxiliary description(s), due to incomplete knowledge acquisition. For example, (John Bardeen, awardreceived, Nobel Prize in Physics) with point-in-time : 1956 and it with {together-with : Walter Houser Brattain, together-with : William Shockley} are two typical partial facts corresponding to the above 5-ary fact. For differentiation, we call those relatively complete facts as whole ones. We noticed that existing studies on n-ary facts infer an unknown element in a well-defined whole fact and have not paid attention to knowledge inference on partial facts. Later on, we
<sup>1</sup>https://www.wikidata.org/wiki/Q949
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6141-6151 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
refer the former as simple knowledge inference, while the latter as flexible knowledge inference.
With these considerations in mind, in this paper, by discriminating the information in the same n-ary fact, we propose a neural network model, called NeuInfer, to conduct both simple and flexible knowledge inference on n-ary facts. Our specific contributions are summarized as:
- We treat the information in the same n-ary fact discriminatingly and represent each n-ary fact as a primary triple coupled with a set of its auxiliary descriptive attribute-value pair(s).
- We propose a neural network model, NeuInfer, for knowledge inference on n-ary facts. NeuInfer can particularly handle the new type of task, flexible knowledge inference, which infers an unknown element in a partial fact consisting of a primary triple and any number of its auxiliary description(s).
- Experimental results validate the significant effectiveness and superiority of NeuInfer.
2.1 Knowledge Inference on Binary Facts
They can be divided into tensor/matrix based methods, translation based methods, and neural network based ones.
The quintessential one of tensor/matrix based methods is RESCAL (Nickel et al., 2011). It relates a knowledge graph to a three-way tensor of head entities, relations, and tail entities. The learned embeddings of entities and relations via minimizing the reconstruction error of the tensor are used to reconstruct the tensor. And binary facts corresponding to entries of large values are treated as valid. Similarly, ComplEx (Trouillon et al., 2016) relates each relation to a matrix of head and tail entities, which is decomposed and learned like RESCAL. To improve the embeddings and thus the performance of inference, researchers further introduce the constraints of entities and relations (Ding et al., 2018; Jain et al., 2018).
Translation based methods date back to TransE (Bordes et al., 2013). It views each valid binary fact as the translation from the head entity to the tail entity via their relation. Thus, the score function indicating the validity of the fact is defined based on the similarity between the translation result and the tail entity. Then, a flurry of methods
spring up (Wang et al., 2014; Lin et al., 2015b; Ji et al., 2015; Guo et al., 2015; Lin et al., 2015a; Xiao et al., 2016; Jia et al., 2016; Tay et al., 2017; Ebisu and Ichise, 2018; Chen et al., 2019). They modify the above translation assumption or introduce additional information and constraints. Among them, TransH (Wang et al., 2014) translates on relation-specific hyperplanes. Entities are projected into the hyperplanes of relations before translating.
Neural network based methods model the validity of binary facts or the inference processes. For example, ConvKB (Nguyen et al., 2018) treats each binary fact as a three-column matrix. This matrix is fed into a convolution layer, followed by a concatenation layer and a fully-connected layer to generate a validity score. Nathani et al. (2019) further proposes a generalized graph attention model as the encoder to capture neighborhood features and applies ConvKB as the decoder. ConvE (Dettmers et al., 2018) models entity inference process via 2D convolution over the reshaped then concatenated embedding of the known entity and relation. ConvR (Jiang et al., 2019) further adaptively constructs convolution filters from relation embedding and applies these filters across entity embedding to generate convolutional features. SENN (Guan et al., 2018) models the inference processes of head entities, tail entities, and relations via fully-connected neural networks, and integrates them into a unified framework.
2.2 Knowledge Inference on N-ary Facts
As aforesaid, only a few studies handle this type of knowledge inference. The m-TransH method (Wen et al., 2016) defines n-ary relations as the mappings from the attribute sequences to the attribute values. Each n-ary fact is an instance of the corresponding n-ary relation. Then, m-TransH generalizes TransH (Wang et al., 2014) on binary facts to n-ary facts via attaching each n-ary relation with a hyperplane. RAE (Zhang et al., 2018) further introduces the likelihood that two attribute values co-participate in a common n-ary fact, and adds the corresponding relatedness loss multiplied by a weight factor to the embedding loss of m-TransH. Specifically, RAE applies a fully-connected neural network to model the above likelihood. Differently, NaLP (Guan et al., 2019) represents each n-ary fact as a set of attribute-value pairs directly. Then, convolution is adopted to get the embeddings of the attribute-value pairs, and a fully-connected neural
6142
network is applied to evaluate their relatedness and finally to obtain the validity score of the input n-ary fact.
In these methods, the information in the same n-ary fact is equal-status. Actually, in each n-ary fact, a primary triple can usually be identified with other information as its auxiliary description(s), as exemplified in Section 1. Moreover, these methods are deliberately designed only for the inference on whole facts. They have not tackled any distinct inference task. In practice, the newly proposed flexible knowledge inference is also prevalent.
3.1 The Representation of N-ary Facts
Different from the studies that define n-ary relations first and then represent n-ary facts (Wen et al., 2016; Zhang et al., 2018), we represent each n-ary fact as a primary triple (head entity, relation, tail entity) coupled with a set of its auxiliary description(s) directly. Formally, given an n-ary fact  $Fct$  with the primary triple  $(h,r,t)$ ,  $m$  attributes and attribute values, its representation is:
$$
\begin{array}{l} \left(\left(h, r, t\right), \left\{ \begin{array}{l} \end{array} \right. \right. \\ \vert - a _ {1}: v _ {1}, \\ \vert - a _ {2}: v _ {2}, \\ \vdash \dots , \\ \left. \left| - a _ {m}: v _ {m} \right\rbrace\right), \\ \end{array}
$$
where each  $a_i: v_i (i = 1,2,\ldots,m)$  is an attribute-value pair, also called an auxiliary description to the primary triple. An element of  $Fct$  refers to  $h/r/t/a_i/v_i$ ;  $A_{Fct} = \{a_1, a_2, \ldots, a_m\}$  is  $Fct$ 's attribute set and  $a_i$  may be the same to  $a_j (i,j = 1,2,\ldots,m, i \neq j)$ ;  $V_{Fct} = \{v_1, v_2, \ldots, v_m\}$  is  $Fct$ 's attribute value set.
For example, the representation of the 5-ary fact, mentioned in Section 1, is:
((John Bardeen, award-received, Nobel Prize in Physics), {
point-in-time:1956,
together-with: Walter Houser Brattain,
$\left|\text{together-with:William Shockley}\right)$ .
Note that, in the real world, there is a type of complicated cases, say, where more than two entities participate in the same n-ary fact with the same primary attribute. We follow Wikidata (Vrandecic and Krötzsch, 2014) to view the cases from different aspects of different entities. Take the case that John Bardeen, Walter Houser Brattain,
and William Shockley received Nobel Prize in Physics in 1956 for example, besides the above 5-ary fact from the view of John Bardeen, we get other two 5-ary facts from the views of Walter Houser Brattain $^2$  and William Shockley $^3$ , respectively:
((Walter Houser Brattain, award-received, Nobel Prize in Physics), {
-point-in-time:1956,
together-with:John Bardeen,
$\left|\text{together-with:William Shockley}\right)$
((William Shockley, award-received, Nobel Prize in Physics), {
point-in-time:1956,
together-with: Walter Houser Brattain,
$\left|\text{together-with:John Bardeen}\right)$
3.2 Task Statement
In this paper, we handle both the common simple knowledge inference and the newly proposed flexible knowledge inference. Before giving their definitions under our representation form of n-ary facts, let us define whole fact and partial fact first.
Definition 1 (Whole fact and partial fact). For the fact  $Fct$ , assume its set of auxiliary description(s) as  $S_{d} = \{a_{i}:v_{i}|i = 1,2,\ldots ,m\}$ . Then a partial fact of  $Fct$  is:  $Fct^{\prime} = ((h,r,t),S_{d}^{\prime})$ , where  $S_d^\prime \subset S_d$ , i.e.,  $S_d^\prime$  is a subset of  $S_{d}$ . And we call  $Fct$  the whole fact to differentiate it from  $Fct^{\prime}$ .
Notably, whole fact and partial fact are relative concepts, and a whole fact is a relatively complete fact compared to its partial fact. In this paper, partial facts are introduced to imitate a typical open-world setting where different facts of the same type may have different numbers of attribute-value pair(s).
Definition 2 (Simple knowledge inference). It aims to infer an unknown element in a whole fact.
Definition 3 (Flexible knowledge inference). It aims to infer an unknown element in a partial fact.
4 The NeuInfer Method
4.1 The Framework of NeuInfer
To conduct knowledge inference on n-ary facts, NeuInfer first models the validity of the n-ary facts and then casts inference as a classification task.
$^{2}$ https://www.wikidata.org/wiki/Q184577
<sup>3</sup>https://www.wikidata.org/wiki/Q163415
6143
Figure 1: The framework of the proposed NeuInfer method.
4.1.1 The Motivation of NeuInfer
How to estimate whether an n-ary fact is valid or not? Let us look into two typical examples of invalid n-ary facts:
((John Bardeen, award-received, Turing Award), {
-point-in-time:1956,
together-with: Walter Houser Brattain,
$\left|\text{together-with:William Shockley}\right\rangle$
((John Bardeen, award-received, Nobel Prize in Physics), {
-point-in-time:1956,
together-with: Walter Houser Brattain,
— place-of-marriage : New York City\}.
In the above first n-ary fact, the primary triple is invalid. In the second one, some auxiliary description is incompatible with the primary triple.
Therefore, we believe that a valid n-ary fact has two prerequisites. On the one hand, its primary triple should be valid. If the primary triple is invalid, attaching any number of attribute-value pairs to it does not make the resulting n-ary fact valid; on the other hand, since each auxiliary description presents a qualifier to the primary triple, it should be compatible with the primary triple. Even if the primary triple is basically valid, any incompatible attribute-value pair makes the n-ary fact invalid. Therefore, NeuInfer is designed to characterize these two aspects and thus consists of two components corresponding to the validity evaluation of the primary triple and the compatibility evaluation of the n-ary fact, respectively.
4.1.2 The Framework of NeuInfer
The framework of NeuInfer is illustrated in Figure 1, with the 5-ary fact presented in Section 1 as an example.
For an n-ary fact  $Fct$ , we look up the embeddings of its relation  $r$  and the attributes in  $A_{Fct}$  from the embedding matrix  $\mathbf{M}_{\mathbf{R}} \in \mathbb{R}^{|R| \times k}$  of relations and attributes, where  $R$  is the set of all the relations and attributes, and  $k$  is the dimension of the latent vector space. The embeddings of  $h, t,$  and the attribute values in  $V_{Fct}$  are looked up from the embedding matrix  $\mathbf{M}_{\mathbf{E}} \in \mathbb{R}^{|E| \times k}$  of entities and attribute values, where  $E$  is the set of all the entities and attribute values. In what follows, the embeddings are denoted with the same letters but in boldface by convention. As presented in Figure 1, these embeddings are fed into the validity evaluation component (the upper part of Figure 1) and the compatibility evaluation component (the bottom part of Figure 1) to compute the validity score of  $(h, r, t)$  and the compatibility score of  $Fct$ , respectively. These two scores are used to generate the final score of  $Fct$  by weighted sum  $\oplus$  and further compute the loss. Note that, following RAE (Zhang et al., 2018) and NaLP (Guan et al., 2019), we only apply fully-connected neural networks in NeuInfer.
4.2 Validity Evaluation
This component estimates the validity of  $(h,r,t)$ , including the acquisition of its interaction vector and the assessment of its validity, corresponding to "hrt-FCNs" and "FCN $_1$  " in Figure 1, respectively.
Detailedly, the embeddings of  $h$ ,  $r$ , and  $t$  are
6144
concatenated and fed into a fully-connected neural network. After layer-by-layer learning, the last layer outputs the interaction vector  $\mathbf{o}_{\mathbf{h}\mathbf{r}\mathbf{t}}$  of  $(h,r,t)$ :
$$
\begin{array}{l} \mathbf {o} _ {\mathbf {h r t}} = f (f (\dots f (f ([ \mathbf {h}; \mathbf {r}; \mathbf {t} ] \mathbf {W} _ {1, 1} + \mathbf {b} _ {1, 1}) \cdot \\ \left. \left. \mathbf {W} _ {\mathbf {1}, 2} + \mathbf {b} _ {\mathbf {1}, 2}\right) \dots\right) \left. \left. \mathbf {W} _ {\mathbf {1}, n _ {1}} + \mathbf {b} _ {\mathbf {1}, n _ {1}}\right), \right. \\ \end{array}
$$
where  $f(\cdot)$  is the ReLU function;  $n_1$  is the number of the neural network layers;  $\{\mathbf{W}_{1,1},\mathbf{W}_{1,2},\ldots ,$ $\mathbf{W}_{1,\mathbf{n}_1}\}$  and  $\{\mathbf{b}_{1,1},\mathbf{b}_{1,2},\dots ,\mathbf{b}_{1,\mathbf{n}_1}\}$  are their weight matrices and bias vectors, respectively.
With  $\mathbf{o}_{\mathbf{h}\mathbf{r}\mathbf{t}}$  as the input, the validity score  $val_{hrt}$  of  $(h,r,t)$  is computed via a fully-connected layer and then the sigmoid operation:
$$
v a l _ {h r t} = \sigma \left(\mathbf {o} _ {\mathbf {h r t}} \mathbf {W} _ {\mathbf {v a l}} + b _ {v a l}\right), \tag {2}
$$
where  $\mathbf{W}_{\mathrm{val}}$  and  $b_{val}$  are the weight matrix and bias variable, respectively;  $\sigma (x) = \frac{1}{1 + e^{-x}}$  is the sigmoid function, which constrains  $val_{hrt} \in (0,1)$ .
For simplicity, the number of hidden nodes in each fully-connected layer of "hrt-FCNs" and "FCN1" gradually reduces with the same difference between layers.
4.3 Compatibility Evaluation
This component estimates the compatibility of  $Fct$ . It contains three sub-processes, i.e., the capture of the interaction vector between  $(h,r,t)$  and each auxiliary description  $a_{i}:v_{i}$ $(i = 1,2,\dots ,m)$ , the acquisition of the overall interaction vector, and the assessment of the compatibility of  $Fct$ , corresponding to "hrtav-FCNs", "min" and "FCN $_2$  " in Figure 1, respectively.
Similar to "hrt-FCNs", we obtain the interaction vector  $\mathbf{o}_{\mathbf{hrt}\mathbf{a}_{i}\mathbf{v}_{i}}$  of  $(h,r,t)$  and  $a_{i}:v_{i}$ :
$$
\begin{array}{l} \mathbf {o} _ {\mathrm {h r t a} _ {\mathrm {i}} \mathbf {v} _ {\mathrm {i}}} = f (f (\dots f (f ([ \mathbf {h}; \mathbf {r}; \mathbf {t}; \mathbf {a} _ {\mathrm {i}}; \mathbf {v} _ {\mathrm {i}} ] \mathbf {W} _ {2, 1} + \mathbf {b} _ {2, 1}) \cdot \\ \left. \left. \mathbf {W} _ {2, 2} + \mathbf {b} _ {2, 2}\right) \dots\right) \left. \mathbf {W} _ {2, n _ {2}} + \mathbf {b} _ {2, n _ {2}}\right), \tag {3} \\ \end{array}
$$
where  $n_2$  is the number of the neural network layers;  $\{\mathbf{W}_{2,1},\mathbf{W}_{2,2},\dots ,\mathbf{W}_{2,\mathbf{n}_2}\}$  and  $\{\mathbf{b}_{2,1}$ $\mathbf{b}_{2,2},\ldots ,\mathbf{b}_{2,\mathbf{n}_2}\}$  are their weight matrices and bias vectors, respectively. The number of hidden nodes in each fully-connected layer also gradually reduces with the same difference between layers. And the dimension of the resulting  $\mathbf{o_{hrtai}v_i}$  is  $d$  All the auxiliary descriptions share the same parameters in this sub-process.
The overall interaction vector  $\mathbf{o}_{\mathrm{hrtav}}$  of  $Fct$  is generated based on  $\mathbf{o}_{\mathrm{hrtat_i v_i}}$ . Before introducing this sub-process, let us see the principle behind first.
Straightforwardly, if  $Fct$  is valid,  $(h, r, t)$  should be compatible with any of its auxiliary description. Then, the values of their interaction vector, measuring the compatibility in many different views, are all encouraged to be large. Therefore, for each dimension, the minimum over it of all the interaction vectors is not allowed to be too small.
Thus, the overall interaction vector  $\mathbf{o}_{\mathrm{hrtav}}$  of  $(h,r,t)$  and its auxiliary description(s) is:
$$
\mathbf {o} _ {\mathbf {h r t a v}} = \min  _ {i = 1} ^ {m} \left(\mathbf {o} _ {\mathbf {h r t a v} _ {i} \mathbf {v} _ {i}}\right), \tag {4}
$$
where  $\min (\cdot)$  is the element-wise minimizing function.
Then, similar to "FCN $_1$  ", we obtain the compatibility score  $comp_{Fct}$  of  $Fct$ :
$$
c o m p _ {F c t} = \sigma \left(\mathbf {o} _ {\mathbf {h r t a v}} \mathbf {W} _ {\mathbf {c o m p}} + b _ {c o m p}\right), \tag {5}
$$
where  $\mathbf{W}_{\mathrm{comp}}$  of dimension  $d\times 1$  and  $b_{comp}$  are the weight matrix and bias variable, respectively.
4.4 Final Score and Loss Function
The final score  $s_{Fct}$  of  $Fct$  is the weighted sum  $\oplus$  of the above validity score and compatibility score:
$$
\begin{array}{l} s _ {F c t} = \operatorname {v a l} _ {h r t} \oplus \operatorname {c o m p} _ {F c t} \tag {6} \\ = w \cdot v a l _ {h r t} + (1 - w) \cdot c o m p _ {F c t}, \\ \end{array}
$$
where  $w\in (0,1)$  is the weight factor.
If the arity of  $Fct$  is 2, the final score is equal to the validity score of the primary triple  $(h,r,t)$ . Then, Equation (6) is reduced to:
$$
s _ {F c t} = \operatorname {v a l} _ {h r t}. \tag {7}
$$
Currently, we obtain the final score  $s_{Fct}$  of  $Fct$ . In addition,  $Fct$  has its target score  $l_{Fct}$ . By comparing  $s_{Fct}$  with  $l_{Fct}$ , we get the binary cross-entropy loss:
$$
\mathcal {L} _ {F c t} = - l _ {F c t} \log s _ {F c t} - (1 - l _ {F c t}) \log (1 - s _ {F c t}), \tag {8}
$$
where  $l_{Fct} = 1$ , if  $Fct \in T$ , otherwise  $Fct \in T^{-}$ ,  $l_{Fct} = 0$ . Here,  $T$  is the training set and  $T^{-}$  is the set of negative samples constructed by corrupting the n-ary facts in  $T$ . Specifically, for each n-ary fact in  $T$ , we randomly replace one of its elements with a random element in  $E / R$  to generate one negative sample not contained in  $T$ .
We then optimize NeuInfer via backpropagation, and Adam (Kingma and Ba, 2015) with learning rate  $\lambda$  is used as the optimizer.
We conduct experiments on two n-ary datasets. The first one is JF17K (Wen et al., 2016; Zhang et al., 2018), derived from Freebase (Bollacker et al., 2008). In JF17K, an n-ary relation of a certain type is defined by a fixed number of ordered attributes. Then, any n-ary fact of this relation is denoted as an ordered sequence of attribute values corresponding to the attributes. For example, for all n-ary facts of the n-ary relation olympics.olympic-medal_honor, they all have four attribute values (e.g., 2008 Summer Olympics, United States, Natalie Coughlin, and Swimming at the 2008 Summer Olympics - Women's  $4 \times 100$  metre freestyle relay), corresponding to the four ordered attributes of this n-ary relation. The second one is WikiPeople (Guan et al., 2019), derived from Wikidata (Vrandecic and Krötzsch, 2014). Its n-ary facts are more diverse than JF17K's. For example, for all n-ary facts that narrate award-received, some have the attribute together-with, while some others do not. Thus, WikiPeople is more difficult.
To run NeuInfer on JF17K and WikiPeople, we transform the representation of their n-ary facts. For JF17K, we need to convert each attribute value sequence of a specific n-ary relation to a primary triple coupled with a set of its auxiliary description(s). The core of this process is to determine the primary triple, formed by merging the two primary attributes of the n-ary relation and the corresponding attribute values. The two primary attributes are selected based on RAE (Zhang et al., 2018). For each attribute of the n-ary relation, we count the number of its distinct attribute values from all the n-ary facts of this relation. The two attributes that correspond to the largest and second-largest numbers are chosen as the two primary attributes. For WikiPeople, since there is a primary triple for each n-ary fact in Wikidata, with its help, we simply reorganize a set of attribute-value pairs in WikiPeople to a primary triple coupled with a set of its auxiliary description(s).
The statistics of the datasets after conversion or reorganization are outlined in Table 1, where #Train, #Valid, and #Test are the sizes of the training set, validation set, and test set, respectively.