SlowGuess/ACL_2020_OCR huggingface dataset:
Zhiwen Xie $^{1}$ , Guangyou Zhou $^{2*}$ , Jin Liu $^{1*}$ , Jimmy Xiangji Huang $^{3}$
$^{1}$ School of Computer Science, Wuhan University
$^{2}$ School of Computer Science, Central China Normal University
$^{3}$ School of Information Technology, York University
xiezhiwen@whu.edu.cn, gyzhou@mail.ccnu.edu.cn
jinliu@whu.edu.cn, jhuang@yorku.ca
The goal of Knowledge graph embedding (KGE) is to learn how to represent the low-dimensional vectors for entities and relations based on the observed triples. The conventional shallow models are limited to their expressiveness. ConvE (Dettmers et al., 2018) takes advantage of CNN and improves the expressive power with parameter efficient operators by increasing the interactions between head and relation embeddings. However, there is no structural information in the embedding space of ConvE, and the performance is still limited by the number of interactions. The recent KBGAT (Nathani et al., 2019) provides another way to learn embeddings by adaptively utilizing structural information. In this paper, we take the benefits of ConvE and KBGAT together and propose a Relation-aware Inception network with joint local-global structural information for knowledge graph Embedding (ReInceptionE). Specifically, we first explore the Inception network to learn query embedding, which aims to further increase the interactions between head and relation embeddings. Then, we propose to use a relation-aware attention mechanism to enrich the query embedding with the local neighborhood and global entity information. Experimental results on both WN18RR and FB15k-237 datasets demonstrate that ReInceptionE achieves competitive performance compared with state-of-the-art methods.
Knowledge graphs (KGs) are at the core of most state-of-the-art natural language processing solutions and have been spotlighted in many real-world applications, including question answering (Hao et al., 2017), dialogue generation (He et al., 2017; Madotto et al., 2018) and machine reading comprehension (Yang and Mitchell, 2017). Typically, KGs
are directed graphs whose nodes denote the entities and edges represent the different relations between entities. The structured knowledge in KGs is organized in the form of triples  $(h, r, t)$ , where  $h$  and  $t$  stand for the head and tail entities respectively, and  $r$  represents the relation from  $h$  to  $t$ . Although large-scale KGs (e.g., Freebase (Bollacker et al., 2008), DBpedia (Lehmann et al., 2015)) have already contained millions or even billions of triples, they are still far from complete since the emerging new knowledge appears. Knowledge graph embedding (KGE) is an effective solution to solve the incompleton problem.
KGE aims to learn the low-dimensional vectors (embeddings) for entities and relations based on the observed triples in KGs. Conventional models including TransE (Bordes et al., 2013) and its numerous extensions (e.g., TransD (Ji et al., 2015), TransR (Lin et al., 2015), DistMul (Yang et al., 2015), ComplEx (Trouillon et al., 2016), etc.) have been proposed. These shallow models are limited to their expressiveness (Dettmers et al., 2018). Recently, CNN-based methods have been proposed to capture the expressive features with parameter efficient operators. ConvE (Dettmers et al., 2018) takes advantage of CNN and uses convolution filters on 2D reshapings of the head entity and relation embeddings. Through this, ConvE can increase the interactions between head and relation embeddings. Empirical results have proved that increasing the number of interactions is beneficial to the KGE task, but ConvE is still limited by the number of interactions (Jiang et al., 2019; Vashishth et al., 2020).
Furthermore, ConvE does not consider the structural information. In contrast, graph-based methods are effective to aggregate neighborhood information to enrich the entity/relation representation (Schlichtkrull et al., 2018; Bansal et al., 2019; Nathani et al., 2019). Among them, KB-
5929
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5929-5939 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: An example of relation-aware local and global information (left) and the general framework of our proposed ReInceptionE (right).
GAT (Nathani et al., 2019) achieves state-of-the-art performance on various benchmark datasets via using graph attention networks (GAT) (Velickovic et al., 2018). KBGAT learns embeddings for every entity by taking all possible relations into account, which requires multiple hops of reasoning. In contrast, it can be beneficial to learn embeddings from a query-relevant subgraph of the local neighborhood and global entities. As an example shown in Figure 1, given a query (Jack London, nationality, ?) for Jack London, we can gather the relation-aware local neighbor (place_lived, Okaland). The local neighbor allows us to project Jack London into the Okaland region of the embedding space, which can lead to a high score for predicting the target America, as Okaland and America are close in embedding space. Besides, we also note that a specific relation can be acted as the "bridge" to link the related entities. Considering the relation nationality, the related head entities  $\{Kaneto Shiozawa, Shammi Kapoor, Will Smith, \dots\}$  and tail entities  $\{America, Canada, Japan, \dots\}$  tend to be a set of person names and countries. These related entities act as a strong signal to judge whether a triple is valid or not.
Based on the above observations, we take the benefits of ConvE and KBGAT together and propose a Relation-aware Inception network with joint local-global structural information for knowledge graph Embedding, and we name it ReIncep
tionE. In ReInceptionE, we first adapt Inception network (Szegedy et al., 2015, 2016) – a high performing convolutional neural network with carefully designed filters, to increase the interactions using multiple convolution filters with different scales, while at the same time to keep parameter efficient. Then, we construct a local neighborhood graph and a global entity graph by sharing the head and relation respectively for a given query. With the constructed graphs, we apply a relation-aware attention mechanism to aggregate the local neighborhood features and gather the global entity information to enrich the head/relation representation. Finally, we aggregate the joint local-global structural information using a fully connected layer to predict the missing links.
In summary, we make the following three contributions: (1) It is the first to explore Inception network to learn query embedding which aims to further increase the interactions between head and relation embeddings; (2) We propose to use a relation-aware attention mechanism to enrich the query embedding with the local neighborhood and global entity information; (3) We conduct a series of experiments to evaluate the performance of the proposed method. Experimental results demonstrate that our method obtains competitive performance in comparison to these state-of-the-art models on both WN18RR and FB15k-237.
The rest of this paper is structured as follows.
5930
Section 2 describes our proposed method for KGE. In Section 3, the experimental results are presented. We make a conclusion in Section 4.
In this section, we first describe the background and definition in Subsection 2.1, and Inception-based query encoder in Subsection 2.2. Then, we introduce the relation-aware local attention and global attention in Subsection 2.3 and 2.4, respectively. Finally, we describe the joint using of them in Subsection 2.5.
2.1 Background and Definition
Definition 3.1 Knowledge Graph  $\mathcal{G}$ : A knowledge graph  $\mathcal{G} = \{(h,r,t) | (h,r,t) \in \mathcal{E} \times \mathcal{R} \times \mathcal{E}\}$  denotes a collection of triples, where  $\mathcal{E}$  and  $\mathcal{R}$  indicate entities and relations, respectively,  $h, t \in \mathcal{E}$  represent the head entity and tail entity, and  $r \in \mathcal{R}$  denotes the specific relation linking from the head entity  $h$  to tail entity  $t$ .
Definition 3.2 Knowledge Graph Embedding: Knowledge graph embedding aims to learn embeddings of entities and relations with the valid triples in  $\mathcal{G}$ , and then predict the missing head entity  $h$  given query  $(?,r,t)$  or tail entity  $t$  given query  $(h,r,?)$  with the learned entity and relation embeddings.
The framework of the proposed ReInceptionE is shown in Figure 1 (right). ReInceptionE consists of four modules: (1) Inception-based query encoder (InceptionE), which is used to transform the input query  $q = (h, r, ?)$  into a  $k$ -dimensional vector  $\mathbf{v}_q$ ; (2) relation-aware local attention and (3) relation-aware global attention are used to capture the local neighborhood information and the global entity information; and (4) joint relation-aware attention is used to aggregate the different structural information using a fully connected layer. Finally, we compute the score for the given triple  $(h, r, t)$  based on the query embedding and the tail entity embedding.
2.2 Inception-Based Query Encoder
ConvE (Dettmers et al., 2018) is the first model to apply CNN for KGE, which uses 2D convolution operation to model the head and relation in a query. However, ConvE is limited by the number of interactions between the head and relation embeddings (Jiang et al., 2019; Vashisth et al., 2020). In this paper, we propose to employ the Inception network
Figure 2: The structures of ConvE (left) and the proposed Inception-based query encoder (right). The red squares denote the slide windows of convolution filters.
(Szegedy et al., 2015, 2016), a high performing convolutional neural network with carefully designed filters, to increase the interactions by taking the head and relation as two channels of the input. Figure 2 shows the differences between InceptionE (right) and ConvE (left). Obviously, ConvE cannot capture full interactions between the head and relation embeddings since the convolution operations in ConvE only slides on the entity or relation 2D matrices independently. On the contrary, InceptionE can increase the interactions between the head and relation embeddings using multiple convolution filters with different scales, while at the same time keep parameter efficient.
As shown in Figure 2, given a query  $q = (h,r,?)$ , we first reshape the head and relation embeddings as 2D matrices denoted as  $\mathbf{v}_h$  and  $\mathbf{v}_r$ . Then, the 2D embeddings are viewed as two channels of the input for the Inception network. Thus, the entries at the same dimension of  $\mathbf{v}_h$  and  $\mathbf{v}_r$  are aligned over the channel dimension, which enables the convolution operations to increase the interactions between the head and relation embeddings. Specifically, We first use  $1 \times 1$  convolutions to capture the direct interactions at the same dimension, which can be formulated as:
$$
\mathbf {v} _ {1 \times 1} = \operatorname {R e l u} \left(\left[ \mathbf {v} _ {h} \right| \left| \mathbf {v} _ {r} \right] * \omega_ {1 \times 1}\right) \tag {1}
$$
where Relu (Glorot et al., 2011) is a non-linear activation function,  $||$  denotes the concatenation operation,  $*$  denotes the convolutional operation and
5931
$\omega_{1\times 1}$  is the parameter of convolution filters with  $1\times 1$  size,  $\mathbf{v}_{1\times 1}$  denotes the interaction features of the first  $1\times 1$  convolutional layer.
Then, filters with different sizes, such as  $2 \times 2$  and  $3 \times 3$ , are applied to capture high-level interaction features in various scales. Thus, we can get interaction features of the  $2 \times 2$  and  $3 \times 3$  convolutional layers, denoted by  $\mathbf{v}_{2 \times 2}$  and  $\mathbf{v}_{3 \times 3}$ , respectively.
As suggested in (Szegedy et al., 2016), we use two  $3 \times 3$  convolutions instead of a  $5 \times 5$  convolution to capture interaction features in larger spatial filters, which is able to reduce the number of parameters. The two  $3 \times 3$  convolutions are denoted as:
$$
\mathbf {v} _ {2 (3 \times 3)} = \mathrm {R e l u} (\mathrm {R e l u} (\mathbf {v} _ {1 \times 1} ^ {2 (3 \times 3)} * \omega_ {3 \times 3} ^ {1}) * \omega_ {3 \times 3} ^ {2}) (2)
$$
where  $\mathbf{v}_{1\times 1}^{2(3\times 3)}$  is the input interaction features,  $\omega_{3\times 3}^1$  and  $\omega_{3\times 3}^2$  are parameters of the two  $3\times 3$  convolution layers.
Finally, the output interaction features with different scales and levels are concatenated and a fully connected layer is applied to obtain the embedding of the given query. Formally, we define the Inception-based query encoder model as:
$$
\begin{array}{l} \mathbf {v} _ {q} = \text {I n c e p t i o n} (\mathbf {v} _ {h}, \mathbf {v} _ {r}) \\ = \operatorname {R e l u} (\operatorname {v e c} ([ \mathbf {v} _ {1 \times 1} | | \mathbf {v} _ {2 \times 2} | | \mathbf {v} _ {3 \times 3} | | \mathbf {v} _ {2 (3 \times 3)} ]) \mathbf {W}) \tag {3} \\ \end{array}
$$
where  $\mathbf{W}$  is the parameter of the fully connected layer.
2.3 Relation-Aware Local Attention
KBGAT learns embedding for every entity by taking all possible relations into account, and the embedding learning is impaired by the irrelevant neighbors. In contrast, it can be beneficial to learn embedding from a query-relevant neighborhood graph. In this subsection, we first construct a relation-aware neighborhood graph and then apply an attention mechanism to aggregate local graph structure information.
For the query  $q = (h,r,?)$ , we denote its neighbors as  $\mathcal{N}_q = \{n_i = (e_i,r_i)|(e_i,r_i,h)\in \mathcal{G}\}$ . Note that, for each triple  $(h,r,t)$ , we create an inverse triple  $(t,r^{-1},h)$ , which has also been used in (Lacroix et al., 2018; Dettmers et al., 2018). Thus, query  $(?,r,t)$  can be converted to  $(t,r^{-1},?)$ . And the neighbors  $\{(r_j,e_j)|(h,r_j,e_j)\in \mathcal{G}\}$  for head entity  $h$  can be converted to a format of  $\{(e_j,r_j^{-1})|(h,r_j,e_j)\in \mathcal{G}\}$ . Thus,  $\mathcal{N}_q$  contains
both the outgoing and incoming neighbors for a query  $q = (h,r,?)$ .
Each neighbor  $n_i = (e_i, r_i) \in \mathcal{N}_q$  is also a query with a head entity  $e_i$  and a relation  $r_i$ . Thus, each entity and relation in neighbor  $n_i = (e_i, r_i)$  can be encoded using the Inception-based query encoder:
$$
\mathbf {v} _ {n _ {i}} = I n c e p t i o n (\mathbf {v} _ {e _ {i}}, \mathbf {v} _ {r _ {i}}) \qquad (4)
$$
where  $\mathbf{v}_{e_i}$  and  $\mathbf{v}_{r_i}$  are the 2D embedding vectors of entity  $e_i$  and relation  $r_i$ .
In practice, different neighbors may have different impacts for a given query. It is useful to determine the importance of each neighbor for a specific query. As an example in Figure 1, for the query (Jack London, nationality, ?), it is reasonable to focus on the neighbors related to the relation nationality, such as (Jack London, place_lived, Oakland). To this end, we use relation-aware attention mechanism to assign different importance for each neighbor and compute the relevant score for each neighbor using a non-linear activation layer:
$$
s _ {i} = \text {L e a k y R e l u} \left( \right.\mathbf {W} _ {1} \left[ \right. \mathbf {W} _ {2} \mathbf {v} _ {q} \left. \right\| \mathbf {W} _ {3} \mathbf {v} _ {n _ {i}} \left. \right]\left. \right) \tag {5}
$$
where  $\mathbf{W}_1$ ,  $\mathbf{W}_2$  and  $\mathbf{W}_3$  are parameters to be trained and LeakyRelu (Maas et al., 2013) is the activation function.
We then normalize the relevant scores for different neighbors using a softmax function to make it comparable across the neighbors, which is denoted as:
$$
\alpha_ {i} = \frac {\exp \left(s _ {i}\right)}{\sum_ {n _ {j} \in \mathcal {N} _ {q}} \exp \left(s _ {j}\right)} \tag {6}
$$
Finally, we aggregate the neighborhood information according to their attention scores and apply a non-linear function to obtain the neighborhood vector. To keep more information of the original query embedding, we also apply a residual operation:
$$
\mathbf {v} _ {n} = \operatorname {R e l u} \left(\sum_ {n _ {i} \in \mathcal {N} _ {q}} \alpha_ {i} \mathbf {W} _ {3} \mathbf {v} _ {n _ {i}}\right) + \mathbf {W} _ {2} \mathbf {v} _ {q} \quad (7)
$$
For simplification, we denote the above relation-aware attention operations as:
$$
\mathbf {v} _ {n} = \operatorname {R e A t t} \left(\mathbf {V} _ {n}, \mathbf {v} _ {q}\right) \tag {8}
$$
where  $\mathbf{V}_n = \{\mathbf{v}_{n_i}|n_i\in \mathcal{N}_q\}$  is a set of local neighborhood vectors.
5932
2.4 Relation-Aware Global Attention
The number of relation-aware local neighbors for each node (entity) varies from one to another, making the neighbor graph very sparse. The sparse nature would affect the accuracy of the embedding. In fact, a specific relation can be acted as the "bridge" to link the related entities. In this subsection, we construct a relation-aware head graph and tail graph by gathering all entities for relation  $r$  in the given query  $q = (h, r, ?)$ . Intuitively, all head entities for relation  $r$  share some common type information. And the tail entities for relation  $r$  contain some implicit information about the type of the target entity  $t$ . For example in Figure 1, given the relation nationality, all heads { Kaneto Shiozawa, Shammi Kapoor, Will Smith, ...,} and tails { America, Canada, Japan, ...,} are the names of a person and a country, sharing the similar entity types. These relation-aware global heads and tails can provide some useful information for the KGE task. Thus, we construct relation-aware global head and tail graphs according to the head and tail entities of the relation.
Let  $\mathcal{H}_r = \{e_i|(e_i,r,e_j)\in \mathcal{G}\}$  and  $\mathcal{T}_r = \{e_j|(e_i,r,e_j)\in \mathcal{G}\}$  denote a set of head and tail entities for relation  $r$ , respectively. For each head entity  $h_{ri}\in \mathcal{H}_r$ , we first represent it as an embedding vector  $\mathbf{v}_{h_{ri}}$ . Then, we use relation-aware attention mechanism to capture the relevant information from all the relation-aware head entities, which is denoted as:
$$
\mathbf {v} _ {r h} = R e A t t \left(\mathbf {V} _ {r h}, \mathbf {v} _ {q}\right) \tag {9}
$$
where  $\mathbf{V}_{rh} = \{\mathbf{v}_{h_{ri}}|h_{ri}\in \mathcal{H}_r\}$  is a set of entity vectors for relation-aware global entities.
Similarly, we use relation-aware attention mechanism to capture global tail informations, which is computed as:
$$
\mathbf {v} _ {r t} = \operatorname {R e A t t} \left(\mathbf {V} _ {r t}, \mathbf {v} _ {q}\right) \tag {10}
$$
where  $\mathbf{V}_{rt} = \{\mathbf{v}_{t_{ri}}|t_{ri}\in \mathcal{T}_r\}$  is a set of entity embeddings for relation-aware global tails.
2.5 Joint Relation-Aware Attention
Once obtained the relation-aware local neighborhood information  $\mathbf{v}_n$  and global head and tail vectors  $\mathbf{v}_{ht}$  and  $\mathbf{v}_{rt}$ , we concatenate these vectors and merge them by using a linear feed-forward layer:
$$
\mathbf {v} _ {q} ^ {\prime} = \mathbf {W} _ {4} \left[ \mathbf {v} _ {n} \| \mathbf {v} _ {r h} \| \mathbf {v} _ {r t} \right] + \mathbf {b} \tag {11}
$$
Dataset	WN18RR	FB15k-237
#Entities	40,943	14,541
#Relations	11	237
#Training	86,835	141,442
#Validation	3,034	17,535
#Test	3,134	20,466
where  $\mathbf{W}_4$  and  $\mathbf{b}$  are the parameters of the feedforward layer.
Finally, we compute the score for each triple  $(h,r,t)$  by applying a dot product of the query embedding  $\mathbf{v}_q^{\prime}$  and the tail embedding  $\mathbf{v}_t$ :
$$
f (h, r, t) = \mathbf {v} _ {q} ^ {\prime T} \mathbf {v} _ {t} \tag {12}
$$
To optimize the parameters in our model, we compute the probability of the tail  $t$  using a softmax function:
$$
p (t | h, r) = \frac {\exp (\lambda f (h , r , t))}{\sum_ {(h , r , t ^ {\prime}) \in \mathcal {G} ^ {\prime} \cup \{(h , r , t) \}} \exp (\lambda f (h , r , t ^ {\prime}))} \tag {13}
$$
where  $\lambda$  is a smoothing parameter, and  $\mathcal{G}'$  is a set of invalid triples created by randomly replacing the tail  $t$  with an invalid entity  $t'$ .
We train the model by minimizing the following loss function:
$$
\mathcal {L} = - \frac {1}{| \mathcal {E} |} \sum_ {i = 0} ^ {| \mathcal {E} |} \log p \left(t _ {i} \mid h _ {i}, r _ {i}\right) \tag {14}
$$
where  $(h_i, r_i, t_i) \in \mathcal{G}$  is a valid triple, and  $|\mathcal{E}|$  is the number of valid triples in  $\mathcal{G}$ .
Datasets: We conduct experiments for KGE on two widely used public benchmark datasets: WN18RR (Dettmers et al., 2018) and FB15k-237 (Toutanova et al., 2015). WN18RR is a subset of WN18 (Bordes et al., 2013) while FB15k-237 is a subset of FB15k (Bordes et al., 2013). Since WN18 and FB15k contain a large number of inverse relations, making the triples in the test set can be obtained simply by inverting triples in the training set. To address the above problem, both WN18RR (Dettmers et al., 2018) and FB15k-237 (Toutanova et al., 2015) are generated by removing the inverse relations from WN18 and FB15k. In recent two
5933
Models	WN18RR	FB15k-237
MR	MRR	Hits@10	MR	MRR	Hits@10
TransE (Bordes et al., 2013)*	2300	0.243	0.532	323	0.279	0.441
DistMult (Yang et al., 2015)*	5110	0.430	0.490	512	0.281	0.446
ComplEx (Trouillon et al., 2016)*	5261	0.440	0.510	546	0.278	0.450
R-GCN+ (Schlichtkrull et al., 2018)	-	-	-	-	0.249	0.417
CACL (Oh et al., 2018)	3154	0.472	0.543	235	0.349	0.487
ConvE (Dettmers et al., 2018)	4187	0.430	0.520	244	0.325	0.501
NKGE (Wang et al., 2018)	4170	0.450	0.526	237	0.330	0.510
TransMS (Yang et al., 2019)	6523	-	0.460	249	-	0.445
AnyBURL (Meilicke et al., 2019)	-	0.470	0.552	-	0.310	0.486
SACN (Shang et al., 2019)	-	0.470	0.540	-	0.350	0.540
A2N (Bansal et al., 2019)	-	0.450	0.510	-	0.317	0.486
GRank (Ebisu and Ichise, 2019)	-	0.470	0.539	-	0.322	0.489
ConvR (Jiang et al., 2019)	-	0.475	0.537	-	0.350	0.528
MuRE (Balazevic et al., 2019b)	-	0.475	0.554	-	0.336	0.521
RotatE (Sun et al., 2019)	3340	0.476	0.571	177	0.338	0.533
QuatE (Zhang et al., 2019)	3472	0.481	0.564	176	0.311	0.495
InteractE (Vashishth et al., 2020)	5202	0.463	0.528	172	0.354	0.535
ConvKB (Nguyen et al., 2018)b	3433	0.249	0.524	309	0.243	0.421
CapsE (Nguyen et al., 2019)b	718	0.415	0.559	403	0.150	0.356
KBGAT (Nathani et al., 2019)b	1921	0.412	0.554	270	0.157	0.331
ReInceptionE (ours)	1894	0.483	0.582	173	0.349	0.528
ConvKB (Nguyen et al., 2018)a	2554	0.248	0.525	257	0.396	0.517
CapsE (Nguyen et al., 2019)a	719	0.415	0.560	303	0.523	0.593
KBGAT (Nathani et al., 2019)a	1940	0.440	0.581	210	0.518	0.626
Table 2: Link prediction results on WN18RR and FB15k-237 test sets. * denotes that the results are taken from (Dettmers et al., 2018), the superscript  $a$  represents the results reported in the original papers while  $b$  represents the results are taken from (Sun et al., 2020), other results are directly taken from the corresponding papers. Both MRR and Hits@1 have a strong correlation, thus we do not report the results of Hits@1 since it does not give any new insight (Nguyen et al., 2019). The best results are in bold and the second best results are in underline.
years, WN18RR and FB15k-237 have become the most popular datasets for the KGE task. Table 1 shows the summary statistics of the datasets.
Implementations: For a test triple  $(h,r,t)$ , the purpose of KGE task is to predict missing links, e.g. predict tail entity  $t$  given head entity  $h$  and relation  $r$  or predict head entity  $h$  given tail entity  $t$  and relation  $r$ . To evaluate our method, three metrics are used, including Mean Rank (MR), Mean Reciprocal Rank (MRR), and Hit@10 (e.g. the accuracy in top 10 predictions). Please note that lower MR, higher MRR and Hits@10 indicate better performance. We follow the "Filtered" setting protocol (Bordes et al., 2013) to evaluate our model, i.e., ranking all the entities excluding the set of other true entities that appeared in training, validation and test sets. We initialize the embedding of entity and relation in our ReInceptionE model using the