SlowGuess/ACL_2020_OCR huggingface dataset:
Jingyuan Zhang, Mingming Sun, Yue Feng, Ping Li
No. 10 Xibeiwang East Road, Beijing 100085, China
{zhangjingyuan03,sunmingming01,v_fengyue,liping11}@baidu.com
Concept graphs are created as universal taxonomies for text understanding in the open domain knowledge. The nodes in concept graphs include both entities and concepts. The edges are from entities to concepts, showing that an entity is an instance of a concept. In this paper, we propose the task of learning interpretable relationships from open domain facts to enrich and refine concept graphs. The Bayesian network structures are learned from open domain facts as the interpretable relationships between relations of facts and concepts of entities. We conduct extensive experiments on public English and Chinese datasets. Compared to the state-of-the-art methods, the learned network structures help improving the identification of concepts for entities based on the relations of entities on both English and Chinese datasets.
Concept graphs are created as universal taxonomies for text understanding and reasoning in the open domain knowledge (Dagan et al., 2010; Bowman et al., 2015; Zamir et al., 2018; Huang et al., 2019; Hao et al., 2019; Jiang et al., 2019). The nodes in concept graphs include both entities and concepts. The edges are from entities to concepts, showing that an entity is an instance of a concept. The task of extracting and building concept graphs from user-generated texts has attracted a lot of research attentions for a couple of decades (Fellbaum, 1998; Wu et al., 2012; Shwartz et al., 2016; Chang et al., 2018; Le et al., 2019; Lewis, 2019). Most of these methods rely on high quality syntactic patterns to determine whether an entity belongs to a concept. For example, given the pattern "  $X$  is a  $Y$  " or "  $Y$ , including  $X$  " appearing in sentences, we can infer that the entity  $X$  is an instance of the concept  $Y$ . These pattern-based methods require that an entity and concept pair
co-occurs in sentences. However, due to the different expressions of a certain concept, an entity and a concept may rarely appear in sentences together. We conduct a data analysis of millions of sentences extracted from Wikipedia and discover that only  $10.61\%$  of entity-concept pairs co-occur in sentences out of more than six million of pairs from the public Microsoft concept graph (https://concept.research.microsoft.com). We also analyze Baidu Baike (http://baike.baidu.com) and its corresponding concept graph. A similar phenomenon is observed that only  $8.56\%$  entity-concept pairs co-occur in sentences. Table 1 shows the statistics for Wikipedia and Baidu Baike. With such limitations, the existing approaches have difficulties in helping build a complete concept graph from open domain texts.
Dataset	# Pairs	# Sentences	# Co-occurrence	Percentage
Wikipedia	6,347,294	7,871,825	673,542	10.61%
Baike	3,229,301	9,523,183	276,485	8.56%
Table 1: Entity-concept pairs that co-occur in sentences from Wikipedia (English) and Baidu Baike (Chinese).
Nowadays, the task of open domain information extraction (OIE) has become more and more important (Christensen et al., 2011; Wu and Weld, 2010; Etzioni et al., 2011; Mausam et al., 2012; Sun et al., 2018b,a; Di et al., 2019; Rashed et al., 2019; Liu et al., 2020a,b). OIE aims to generate entity and relation level intermediate structures to express facts from open domain sentences. These open domain facts usually express natural languages as triples in the form of (subject, predicate, object). For example, given the sentence "Anderson, who hosted Whose Line, is a winner of a British Comedy Award in 1991.", two facts will be extracted. They are ("Anderson", "host", "Whose Line") and ("Anderson", "winner of a British Comedy Award", "1991"). The subject and object in a fact are both
8045
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8045-8056 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: The workflow of learning interpretable relationships from open domain facts for concept discovery.  $f_{i} = (s_{i},r_{i},o_{i})$  represents a fact, where  $s_i$  and  $o_i$  are both entities, and  $r_i$  is a relation. We use  $e_i$  to denote an entity and  $c_{i}$  to represent a concept.
entities. The open domain facts contain rich information about entities by representing the subject or object entities via different types of relations (i.e., groups of predicates).
It would be helpful for concept graph completion if we can take advantage of the relations in open domain facts. We again take the above two facts of "Anderson" as an instance. If we have explored the connections between relations of facts and concepts, and learned that "host" and "winner of a British Comedy Award" are associated with an "English presenter" subject with a higher probability than a "Japanese presenter" subject, we can infer that "Anderson" belongs to the "English presenter" concept regardless of whether these two co-appear in a sentence or not. In real-world open domain corpus, however, the connections between relations and concepts are not available to us.
In this paper, we propose the task of learning interpretable relationships between entities, relations and concepts from open domain facts to help enriching and refining concept graphs. Learning Bayesian networks (BNs) from data has been studied extensively (Heckerman et al., 1995; Koivisto and Sood, 2004; Scanagatta et al., 2015; Niinimaki et al., 2016) in the last few decades. The BNs formally encode probabilistic connections in a certain domain, yielding a human-oriented qualitative structure that facilitates communication between a user and a system incorporating the probabilistic model. Specifically, we apply the Bayesian network structure learning (BNSL) (Chow and Liu, 1968; Yuan et al., 2011; Yuan and Malone, 2013) to discover meaningful relationships between entities, relations and concepts from open domain facts. The learned network encodes the dependen
cies from the relations of entities in facts to the concepts of entities, leading to the identification of more entity-concept pairs from open domain facts for the completion of concept graphs. Figure 1 illustrates the proposed workflow of learning interpretable relationships from open domain facts. We summarize our contributions as follows:
- We propose the task of learning interpretable relationships between entities, relations and concepts from open domain facts, which is important for enriching and refining concept graphs.
- We build the BNSL model to discover meaningful network structures that express the connections from relations of entities in open domain facts to concepts of entities in concept graphs.
- Experimental results on both English and Chinese datasets reveal that the learned interpretable relationships help identify concepts for entities based on the relations of entities, resulting in a more complete concept graph.
Concept Graph Construction. Concept graph construction has been extensively studied in the literature (Fellbaum, 1998; Ponzetto and Strube, 2007; Banko et al., 2007; Suchanek et al., 2007; Wu et al., 2012; Shwartz et al., 2016; Chang et al., 2018; Le et al., 2019; Lewis, 2019). Notable works toward creating open domain concept graphs from scratch include YAGO (Suchanek et al., 2007) and Probase (Wu et al., 2012). In addition, a wide variety of methods (Nakashole et al., 2012; Weeds et al., 2014; Roller et al., 2014; Shwartz et al., 2016; Roller et al., 2018; Chang et al., 2018; Le et al., 2019; Lewis, 2019) are developed to detect the
8046
hypernymy between entities and concepts for a more complete concept graph. Distributional representations of entities and concepts are learned for good hypernymy detection results (Weeds et al., 2014; Roller et al., 2014; Chang et al., 2018; Lewis, 2019). In contrast to distributional methods, path-based algorithms (Nakashole et al., 2012; Shwartz et al., 2016; Roller et al., 2018; Le et al., 2019) are proposed to take advantage of the lexico-syntactic paths connecting the joint occurrences of an entity and a concept in a corpus. Most of these methods require the co-occurrence of entity and concept pairs in sentences for the graph completion task. However, due to the different expressions of a certain concept, an entity and a concept may rarely appear in one sentence together. With such limitations, the existing methods in the literature cannot deal with those non co-occurring entity concept pairs, leading to an incomplete concept graph.
Open Domain Information Extraction. Open domain information extraction (OIE) has attracted a lot of attention in recent years (Wu and Weld, 2010; Christensen et al., 2011; Etzioni et al., 2011; Mausam et al., 2012; Pal and Mausam, 2016; Yahya et al., 2014; Sun et al., 2018b,a; Roy et al., 2019; Liu et al., 2020a,b). It extracts facts from open domain documents and expresses facts as triples of (subject, predicate, object). Recently, a neural-based OIE system Logician (Sun et al., 2018b,a; Liu et al., 2020a,b) is proposed. It introduces a unified knowledge expression format SAOKE (symbol aided open knowledge expression) and expresses the most majority information in natural language sentences into four types of facts (i.e., relation, attribute, description and concept). Logician is trained on a human labeled SAOKE dataset using a neural sequence to sequence model. It achieves a much better performance than traditional OIE systems in Chinese language and provides a set of open domain facts with much higher quality to support upper-level algorithms. Since the subject and object in a fact are both entities, the open domain facts contain rich information about entities by representing the subjects or objects via different types of relations (i.e., groups of predicates). It can help the task of concept graph completion by making full use of the relations in open domain facts. In this paper, we leverage the high-quality facts of Logician as one dataset in the experiment.
Bayesian Network Structure Learning. Learning a Bayesian network structure from real-
world data is a well-motivated but computationally hard task (Heckerman et al., 1995; Koivisto and Sood, 2004; de Campos et al., 2009; Malone et al., 2011; Scanagatta et al., 2015; Niinimaki et al., 2016). A Bayesian network specifies a joint probability distribution of a set of random variables in a structured fashion. A key component in this model is the network structure, a directed acyclic graph on the variables, encoding a set of conditional independence assertions. Several exact and approximate algorithms are developed to learn optimal Bayesian networks (Chow and Liu, 1968; Koivisto and Sood, 2004; Singh and Moore, 2005; Silander and Myllymäki, 2006; Yuan et al., 2011; Yuan and Malone, 2013). Some exact algorithms (Koivisto and Sood, 2004; Singh and Moore, 2005; Silander and Myllymäki, 2006) are based on dynamic programming to find the best Bayesian network. In 2011, an  $\mathbf{A}^{\star}$  search algorithm is introduced (Yuan et al., 2011) to formulate the learning process as a shortest path finding problem. However, these exact algorithms are inefficient due to the full evaluation of an exponential solution space. In this paper, we consider the Chow-Liu tree building algorithm (Chow and Liu, 1968) to approximate the underlying relationships between entities, relations and concepts as a dependency tree. This method is very efficient when there are large numbers of variables.
3 Finding Interpretable Relationships
We formulate the relationships between entities, relations, and concepts as follows:
- Entities are associated with a set of relations that represent the behaviors and attributes of entities;
- A concept is defined by a set of relations. The instances of a concept are those entities that associate with the corresponding set of relations.
In concept graphs, a concept is associated with a set of entities which share some common behaviors or attributes. However, the essence of a concept is a set of relations, and entities which associate with these relations automatically become the instance of the concept. So our formulation of the relationships between entities, relations and concepts can be illustrated by Figure 2.
In the closed domain, a knowledge base has a predefined ontology and the relationships in Figure 2 are already known. For example, DBPedia (Auer et al., 2007) builds a knowledge graph
8047
Figure 2: Relationships of entities, relations and concepts.
from Wikipedia to encode the relationships between entities and relations in the forms of facts. The relationships between relations and concepts are represented in the ontology structure of DBPedia, where each concept is associated with a group of relations.
However, in the open domain, a predefined ontology does not exist, and hence the components in Figure 2 may not be associated with each other. For instance, given an open domain concept graph, we can discover the relationships between entities and concepts. Given the open domain corpus/facts, we can find the relationships between entities and relations. But the relationships between open domain concepts and relations are not available, to our knowledge. In this paper, we aim to find the connection between open domain relations and concepts, so that we can provide interpretations to the question "why the entity is associated with those concepts in open domain".
Suppose we have a set of entities  $E = \{e_1, \dots, e_m\}$ , a set of relations  $R = \{r_1, \dots, r_p\}$ , a set of concepts  $C = \{c_1, \dots, c_q\}$ , and a set of observed triplets  $O = \{(e, r, c)\}$ . Here  $E$  and  $C$  are from a concept graph  $G$ .  $R$  is from a set of facts  $F = \{f_1, \dots, f_n\}$  extracted from a text corpus  $D$ . A triplet  $(e, r, c)$  is observed means that the entity  $e$  with relation  $r$  and concept of  $c$  is found in above data sources. Given a set of observations  $O$  with  $N$  samples, the Bayesian network can be learned by maximizing the joint probability  $p(O)$ :
$$
\begin{array}{l} p (O) = \prod_ {(e, r, c) \in O} p ((e, r, c)) \\ = \prod_ {(e, r, c) \in O} p (c | (e, r)) \cdot p (r | e) \cdot p (e) \\ = \prod_ {(e, r, c) \in O} p (c | r) \cdot p (r | e) \cdot p (e) \\ \end{array}
$$
where  $p(c|(e,r)) = p(c|r)$  is due to our Bayesian network assumption (see Figure 2). By learning with the observed triplets with above model, we can infer the missing triplets, especially give interpretable relationship between entities and concepts.
Since  $p(r|e)$  can be approximated by the information from OIE corpus, the core of the above problem becomes to learn the part of the network of  $p(c|e)$ . The difficulty of learning  $p(c|e)$  is the unknown structure of the Bayesian network. Due to sparsity of real-world knowledge base, the target network would be sparse. But the sparse structure must be known beforehand for probability learning.
In this paper, we employ the Bayesian Network Structure Learning (BNSL) technique to explore the connections between relations and concepts. Due to the large number of variables (i.e., entities, relations and concepts) in open domain facts and concept graphs, we develop an approximate algorithm to learn the network structure.
3.2 The Proposed Approximate Algorithm
Due to the sparsity of the relationships between relations and concepts, we decompose the problem into several sub-problems, with each sub-problem containing only one concept variable. Then for each concept variable, we identify possible related relations and apply a BNSL algorithm to discover the network structure between them. Finally, we use the learned network for concept discovery. The procedure is shown in Algorithm 1. We will state the key steps in detail in the next sub-sections.
3.2.1 Sub-problem Construction
Given a concept  $c \in C$ , we first collect all its entities  $E_{c} \subset E$  from the concept graph. Then we can obtain a set of facts  $F_{c}$  that contain these entities. Since an entity can appear in a fact as a subject or an object, we split the facts  $F_{c}$  into subject-view facts  $F_{c,s}$  and object-view facts  $F_{c,o}$ . If we make use of all the relations under the subject or object view, it would be inefficient or event impossible to learn the sparse network structure with a large number of relation variables. Hence, based on the facts, we select possible related relations to the concept  $c$  to reduce the complexity of the problem.
3.2.2 Relation Selection
There are various strategies which can be applied for the relation selection. We can assume that a relation is highly related to the concept if it appears many times in the fact set  $F_{c}$ . In this way, we can
8048
count the frequencies of relations for each view and select the top  $K$  as the most relevant ones with a concept. We call it TF selection since we measure the relevance of a relation according to its frequency. We can also select relations according to the TFIDF measurement (Wu et al., 2008). For each view, we select the most relevant  $K$  relations for the concept  $c$ . We denote them as  $R_{c,s} \subset R$  for the subject-view facts and  $R_{c,o} \subset R$  for the object-view facts. In summary, for each concept, we construct two sub-problems for the BNSL task. One is from the subject view and the other is from the object view. Under each view, the sub-problem contains one concept and at most  $K$  relations. The goal is to learn a network structure from the concept and corresponding relations.
3.2.3 Data Observations
Given a sub-problem for a concept  $c$ , we first obtain the corresponding data observations and then feed them as the input of BNSL for interpretable relationship discoveries. For each concept, we can learn a Bayesian network structure from its top subject-view or object view relations. The data observations  $X_{c,s}$  with TF relation selection for the subject-view of the concept  $c$  are generated as
follows: for each entity  $e \in E_c$ , we use 1 to be the concept observation, meaning that the entity  $e$  is an instance of concept  $c$ . We use the times of the subject  $e$  and a top relation  $r \in R_{c,s}$  appearing together in facts  $F_{c,s}$  as a relation observation for  $e$  and  $r$ . The  $K$  relation observations and the concept observation together become the positive data observations for  $c$ . In order to learn meaningful network structures, we generate an equal number of negative data observations for  $c$ . We first randomly sample the same number of entities from  $E_{c'} = \{e_i : e_i \in E \setminus E_c\}$  as negative entities of  $c$ . We use 0 as the concept observation for negative entities. Then for each negative entity  $e'$ , we count the times of the subject  $e'$  and a relation  $r \in R_{c,s}$  appearing in all the collected facts as a relation observation for  $e'$  and  $r$ . The  $K$  relation observations and the concept observation together become the negative data observations for  $c$ .  $X_{c,s}$  consists of both the positive and negative data observations. Similarly, we can obtain the data observations  $X_{c,o}$  for the object view.
3.2.4 Network Structure Learning
In this paper, we employ the widely-used Chow-Liu tree building algorithm (Chow and Liu, 1968) as the BNSL method. This algorithm approximates the underlying distributions of variables as a dependency tree, which is a graph where each node only has one parent and cycles are not allowed. It will first calculate the mutual information between each pair of nodes (i.e., variables), and then take the maximum spanning tree of that matrix as the approximation. While this will only provide a rough approximation of the underlying data, it provides good results for many applications (Suzuki, 2010; Tavassolipour et al., 2014; Hassan-Moghaddam and Jovanovic, 2018; Ding et al., 2019), especially when you need to know the most important influencer on each variable. In addition, this algorithm becomes extremely efficient when it deals with to a large number of variables.
Since both the subject and object views reflect some properties of entities, we can concatenate the subject-view relations and object-view relations together for a more complete representation of entities. The concatenated data can be forwarded into BNSL for a more comprehensive result of interpretable relationship discovery. Given  $q$  concept variables and  $K$  relevant relations for each concept, the number of parameters in BNSL is at most  $q \times K$ .
8049
3.2.5 Prediction
After we learn a network structure for each concept, we can learn the concept of a new entity  $e$  easily. We first identify the open domain facts with  $e$  as its subject or object, and then feed the observation of relations for a concept  $c$  into the network to calculate the probability of  $p(c|e)$ . We still use the open domain entity "Anderson" and its two facts introduced in Section 1 as an example to show how BNSL works. Assume we have two open domain concepts, "English presenter" and "Japanese presenter". Given the entity "Anderson" and its open domain relations "host" and "winner of a British Comedy Award" as input of BNSL, the output is the probabilities that "Anderson" belongs to each concept. BNSL will predict a higher probability for "Anderson" having the concept "English presenter" than having "Japanese presenter".
With the learned relationship between relations and concepts from BNSL, we indirectly associate entities with their concepts and give interpretations to the question "why the entity is associated with those concepts in open domain". The hypernymy detection task aims to identify concepts for entities in open domain. It is helpful for us to evaluate the quality of the learned relationships from BNSL. In this section, we conduct extensive experiments to evaluate the performance of BNSL.
We test the performance of our proposed method on two public datasets, one is in English and the other is in Chinese. For the English dataset, we use 15 million high-precision OIE facts<sup>1</sup>, the Microsoft concept graph<sup>2</sup> and 7.87 million Wikipedia sentences<sup>3</sup> for our experiments. Since there are more than 5 million concepts in the English dataset and most of them have few entities, we focus on those concepts with more than 50 entities in the experiments. For the Chinese dataset, we use sentences and the corresponding facts<sup>4</sup> in (Sun et al., 2018b). The concept graph is also built by Baidu Baike. Table 2 shows the statistics of the concept
Concept Graphs	Dataset	# entities	# concepts	# overlaps	% overlaps
English	12,501,527	5,376,526	613,454	27.10%
Chinese	9,230,727	3,245	475,507	48.14%
Facts	Dataset	# facts	# subjects	# objects	# predicates
English	14,728,268	1,396,793	1,698,028	664,746
Chinese	37,309,458	624,632	550,404	10,145
Table 2: Statistics of concept graphs and facts.
graphs and open domain facts.
In open domain facts, each mention of a subject or object is considered as an open domain entity. So we naturally map an entity in open domain facts and concept graphs by the same mention. In Table 2, the column “# of overlap” is about the number of fact entities appearing in the concept graph and the last column is the percentage of fact entities in the concept graph. With the predicates as relations for the open domain facts, we build the Bayesian network structure learning method to bridge the gap between relations in open domain facts and concepts in the concept graph.
4.2 Experimental Setting
In the experiment, we compare with the state-of-the-art model HypeNet (Shwartz et al., 2016) for hypernymy detection. HypeNet improves the detection of entity-concept pairs with an integrated path-based and distributional method. An entity and a concept must appear together in a sentence so that HypeNet can extract lexico-syntactic dependency paths for training and prediction. However, only less than  $11\%$  of entity-concept pairs co-occur in Wikipedia sentences in reality (Table 1). Therefore, we compare BNSL with HypeNet only on the entity-concept pairs that co-appear in sentences.