Venue: EMNLP 2017

# KGEval: Accuracy Estimation of Automatically Constructed Knowledge Graphs

Prakhar Ojha

Indian Institute of Science

prakhar@iisc.ac.in &Partha Talukdar

Indian Institute of Science

ppt@iisc.ac.in

###### Abstract

Automatic construction of large knowledge graphs (KG) by mining web-scale text datasets has received considerable attention recently. Estimating accuracy of such automatically constructed KGs is a challenging problem due to their size and diversity and has largely been ignored in prior research. In this work, we try to fill this gap by proposing KGEval. KGEval uses _coupling constraints_ to bind facts and crowdsource those few that can _infer_ large parts of the graph. We demonstrate that the objective optimized by KGEval is submodular and NP-hard, allowing guarantees for our approximation algorithm. Through experiments on real-world datasets, we demonstrate that KGEval best estimates KG accuracy compared to other baselines, while requiring significantly lesser number of human evaluations.

## 1 Introduction

Automatic construction of Knowledge Graphs (KGs) from Web documents has received significant interest over the last few years, resulting in the development of several large KGs consisting of hundreds of predicates (e.g., _isCity_, _stadiumLocatedInCity(Stadium, City)_) and millions of their instances called beliefs (e.g., _(Joe Luis Arena, stadiumLocatedInCity, Detroit)_). Examples of such KGs include NELL Mitchell et al. (2015), Knowledge-Vault Dong et al. (2014) etc.

Due to imperfections in the automatic KG construction process, many incorrect beliefs are also found in these KGs. Knowing accuracy for each predicate in the KG can provide targeted feedback for improvement and highlight its strengths from weaknesses, whereas overall accuracy of a KG can quantify the effectiveness of its construction-process. Knowing accuracy at predicate-level granularity is immensely helpful for Question-Answering (QA) systems that integrate opinions from multiple KGs Samadi et al. (2015). For such systems, being aware that a particular KG is more accurate than others in a certain domain, say sports, helps in restricting the search over relevant and accurate subsets of KGs, thereby improving QA-precision and response time. In comparison to the large body of recent work focused on construction of KGs, the important problem of accuracy estimation of such large KGs is unexplored - we address this gap in this paper.

True accuracy of a predicate may be estimated by aggregating human judgments on correctness of each and every belief in the predicate1. Even though crowdsourcing marketplaces such as Amazon Mechanical Turk (AMT) provide a convenient way to collect human judgments, accumulating such judgments at the scale of larges KGs is prohibitively expensive. We shall refer to the task of manually classifying a single belief as true or false as a Belief Evaluation Task (BET). Thus, the crucial problem is: _How can we select a subset of beliefs to evaluate which will best estimate the true (but unknown) accuracy of KG and its predicates?_

Footnote 1: Note that belief evaluation can not be completely automated and will require human-judgent. If an algorithm could accurately predict correctness of a belief, then it may as well be used during KG construction rather than during evaluation.

A naive and popular approach is to evaluate randomly sampled subset of beliefs from the KG. Since random sampling ignores relational-couplings present among the beliefs, it usually results in oversampling and poor accuracy estimates. Let us motivate this through an example.





**Motivating example**: We motivate efficient accuracy estimation through the KG fragment shown in Figure 1. Here, each belief is an edge-triple in the graph, for example (_RedWings, isA, SportsTeam_). There are six correct and two incorrect beliefs (the two incident on _Taj Mahal_), resulting in an overall accuracy of \(75\%(=6/8)\) which we would like to estimate. Additionally, we would also like to estimate accuracies of the predicates: _homeStadiumOf_, _homeCity_, _stadiumLocateDInCity_, _cityInState_ and _isA_.

We now demonstrate how evaluation labels of beliefs are constrained by each other. _Type consistency_ is one such coupling constraint. For instance, we know from KG ontology that the _homeStadiumOf_ predicate connects an entity from _Statium_ category to another entity in _Sports Team_ category. Now, if (_Joe Louis Arena, homeStadiumOf, Red Wingss_) is evaluated to be correct, then from these _type constraints_ we can infer that (_Joe Louis Arena, isA, Stadium_) and (_Red Wings, isA, Sports Team_) are also correct. Similarly, by evaluating (_Taj Mahal, isA, State_) as false, we can infer that (_Detroit, cityInState, TajMahal_) is incorrect.

Additionally, we have _Horn-clause coupling constraints_Mitchell et al. (2015); Lao et al. (2011), such as _homeStadiumOf(x, y) \(\wedge\) homeCity(y, z) \(\rightarrow\) stadiumLocatedInCity(x, z)_. By evaluating _(Red Wings, homeCity, Detroit)_ and applying this horn-clause to the already evaluated facts mentioned above, we infer that (_Joe Louis Arena, stadiumLocatedInCity, Detroit)_ is also correct. We explore generalized forms of these constraints in Section 3.1.

Thus, evaluating only three beliefs, and exploiting constraints among them, we exactly estimate the _overall_ true accuracy as \(75\%\) and also cover all predicates. In contrast, the empirical accuracy by randomly evaluating three beliefs, averaged over 5 trials, is \(66.7\%\).

**Our contributions** in this paper are the following: (1). Systematic study into the important problem of evaluation of automatically constructed Knowledge Graphs. (2). A novel crowdsourcing-based system KGEval to estimate accuracy of large knowledge graphs (KGs) by exploiting dependencies among beliefs for more accurate and faster KG accuracy estimation. (3). Extensive experiments on real-world KGs to demonstrate KGEval's effectiveness and also evaluate its robustness and scalability.

All the data and code used in the paper are available at [http://talukdar.net/mall-lab/KGEval](http://talukdar.net/mall-lab/KGEval)

## 2 Overview and Problem Statement

### KGEval: Overview

We try to estimate correctness of as many beliefs as possible while evaluating only a subset of them through crowdsourcing. KGEval achieves this goal using an iterative algorithm which alternates between the following two stages:

* **Control Mechanism** (Section 3.4): In this step, KGEval selects the belief which is to be evaluated next using crowdsourcing.
* **Inference Mechanism** (Section 3.3): Coupling constraints are applied over evaluated beliefs to automatically estimate correctness of additional beliefs.

This iterative process is repeated until there are no more beliefs to be evaluated. Single iteration of KGEval over the KG fragment from Figure 1 is shown in Figure 2 where, belief _(John Louis Arena, homeStadiumOf, Red Wings)_ is selected and evaluated by crowdsourcing. Subsequently, the inference mechanism uses type coupling constraints to infer (_JL Arena, isA, Stadium)_ and _(R. Wings, isA, Team)_ also as true. Next, we formalize the notations used in this paper.

### Notations and Problem Statement

We are given a KG with \(n\) beliefs. Evaluating a single belief as true or false forms a Belief Evaluation Task (BET). Coupling constraints are derived by determining relationships among BETs, which we further discuss in Section 3.1. Notations are also summarized in Table 1.

Figure 1: Sample knowledge-graph (KG) fragment that is consistent but has erroneous beliefs.



_Inference algorithm_ helps us work out evaluation labels of other BETs using constraints \({\cal C}\). For a set of already evaluated BETs \({\cal Q}\subseteq{\cal H}\), we define _inferable set_\({\cal I}\big{(}G,{\cal Q}\big{)}\subseteq{\cal H}\) as BETs whose evaluation labels can be deduced by the inference algorithm. We calculate the average true accuracy of a given set of evaluated BETs \({\cal Q}\subseteq{\cal I}\big{(}G,{\cal Q}\big{)}\subseteq{\cal H}\) by \(\Phi({\cal Q})=\frac{1}{|{\cal Q}|}\sum_{h\in{\cal Q}}t(h)\).

KGEval aims to sample and crowdsource a BET set \({\cal Q}\) with the largest inferable set, and solves the optimization problem:

\[\arg\max_{{\cal Q}\subseteq{\cal H}}~{}\left|{\cal I}\big{(}G,{\cal Q}\big{)}\right| \tag{1}\]

## 3 KGEval: Method Details

In this section, we describe various components of KGEval.

### Coupling Constraints

The evaluation labels of beliefs are often dependent on each other due to rich relational structure of KGs. In this work, we derive coupling constraints \({\cal C}\) from the KG ontology and link-prediction algorithms, such as PRA (Lao et al., 2011) over NELL and AMIE (Galarraga et al., 2013) over Yago. These rules are jointly learned over entire KG with millions of facts and are assumed true.

We use conjunction-form first-order-logic rules and refer to them as _Horn clauses_. Examples of a few such coupling constraints are shown below.

\({\cal C}_{2}\)_: (x, homeStadiumOf, y) \(\rightarrow\) (y, isA, sportsTeam) \({\cal C}_{5}\)_: (x, homeS StadiumOf, y) \(\wedge\) (y, homeCity, z) \(\rightarrow\) (x, stadiumLocatedInCity, z)_

Each coupling constraint \({\cal C}_{i}\) operates over \({\cal H}_{i}\subseteq{\cal H}\) to the left of its arrow and infers label of the BET on the right of its arrow. \({\cal C}_{2}\) enforces _type consistency_ and \({\cal C}_{5}\) is an instance of PRA path. These constraints have also been successfully employed earlier during knowledge extraction (Mitchell et al., 2015) and integration (Pujara et al., 2013). Note that the constraints are _directional_ and inference propagates in forward direction.

### Evaluation Coupling Graph (ECG)

To combine all beliefs and constraints at a common place, for all \({\cal H}\) and \({\cal C}\), we construct a graph with two types of nodes: (1) a node for each BET \(h\in{\cal H}\), and (2) a node for each constraint \({\cal C}_{i}\in{\cal C}\). Each \({\cal C}_{i}\) node is connected to all \(h\) nodes that participate in it. We call this graph the _Evaluation Coupling Graph_ (_ECG_), represented as \(G=({\cal H}\cup{\cal C},{\cal E})\) with set of edges \({\cal E}=\{({\cal C}_{i},h)~{}|~{}h\in{\rm Dom}({\cal C}_{i})~{}\forall{\cal C }_{i}\in{\cal C}\}\). Note that ECG is a bipartite _factor graph_(Kschischang et al., 2001) with \(h\) as variable-nodes and \({\cal C}_{i}\) as factor-nodes.

\begin{table}
\begin{tabular}{|l|p{148.9pt}|} \hline Symbol & Description \\ \hline \({\cal H}=\{h_{1},\ldots,h_{n}\}\) & Set of all \(n\) Belief Evaluation Tasks (BETs) \\ \(c(h)\in\mathbb{R}_{+}\) & Cost of labeling \(h\) from crowd Coupling constraints \({\cal C}_{i}\) with weights \(\theta_{i}\in\mathbb{R}_{+}\) \\ \(t(h)\in\{0,1\}\) & True label of \(h\) \\ \(I(h)\in\{0,1\}\) & Estimated label of \(h\) \\ \({\cal H}_{i}={\rm Dom}({\cal C}_{i})\) & \({\cal H}_{i}\subseteq{\cal H}\) which participate in \({\cal C}_{i}\) \\ \(G=({\cal H}\cup{\cal C},{\cal E})\) & Evaluation Coupling Graph, \(e\in{\cal F}\) between \({\cal H}_{j}\) and \({\cal C}_{j}\) denotes \({\cal H}_{j}\in{\rm Dom}({\cal C}_{j})\) . \\ \({\cal Q}\subseteq{\cal H}\) & BETS evaluated using crowd Inferable set for evidence \({\cal Q}\): \\ \({\cal I}(G,{\cal Q})\subseteq{\cal H}\) & True accuracy of evaluated Inferable set for evidence \({\cal Q}\): \\ \(\Phi({\cal Q})=\) & True accuracy of evaluated Inferable set \({\cal Q}\) \\ \cline{2-3}  & \multicolumn{1}{p{148.9pt}}{} \\ \end{tabular}
\end{table}
Table 1: Summary of notations used (Section 2.2).

Figure 2: Demonstration of one iteration of KGEval. Control mechanism selects a belief whose correctness is evaluated from crowd. In the above example, _(J.L. Arena, homeStadiumOf, Red Wings)_ is crowd-evaluated to be true (indicated by tick with dotted square). (Section 2.1 and Section 3).

 

Figure 3 shows ECG constructed out of the motivating example in Figure 1 with \(|{\cal C}|=8\) and separate nodes for each of the edges (beliefs or BETs) in KG. We pose the KG evaluation problem as classification of BET nodes in the ECG by allotting them a label of \(1\) or \(0\) to represent true or false respectively.

### Inference Mechanism

Inference mechanism helps propagate true/false labels of evaluated beliefs to other non-evaluated beliefs using available coupling constraints [2]. We use Probabilistic Soft Logic (PSL), [5] as our inference engine.Below we briefly describe the internal workings of our inference engine for accuracy estimation problem.

Potential function \(\psi_{j}\) is defined for each \({\cal C}_{j}\) using Lukasiewicz t-norm and it depicts how satisfactorily constraint \({\cal C}_{j}\) is satisfied. For example, \({\cal C}_{5}\) mentioned earlier is transformed from first-order logical form to a real valued number by

\[\psi_{j}({\cal C}_{5})=\big{(}\max\{0,h_{x}+h_{y}-1-h_{w}\}\big{)}^{2} \tag{2}\]

where \({\cal C}_{5}=h_{x}\wedge h_{y}\to h_{w}\), where \(h_{x}\) denotes the evaluation score \(\in[0,1]\) associated with the BETs.

The probability distribution over label assignment is so structured such that labels which satisfy more coupling constraints are more probable. Probability of any label assignment \(\Omega\big{(}{\cal H}\big{)}\in\{0,1\}^{|{\cal H}|}\) over BETs in \(G\) is given by

\[\mathbb{P}\big{(}\Omega\big{(}{\cal H}\big{)}\big{)}=\frac{1}{Z}\exp\Big{[}- \sum_{j=1}^{|{\cal C}|}\theta_{j}\psi_{j}\big{(}{\cal C}_{j}\big{)}\Big{]} \tag{3}\]

where \(Z\) is the normalizing constant and \(\psi_{j}\) corresponds to potential function acting over BETs \(h\in\mathrm{Dom}({\cal C}_{j})\). Final assignment of \(\Omega({\cal H})_{PSL}\in\{1,0\}^{|{\cal H}|}\) labels is obtained by solving the _maximum a-posteriori (MAP)_ optimization problem

We denote by \(M_{PSL}(h,\gamma)\in[0,1]\) the PSL-estimated score for label \(\gamma\in\{1,0\}\) on BET \(h\) in the optimization above.

**Inferable Set using PSL**: We define estimated label for each BET \(h\) as shown below.

\[l(h)=\begin{cases}1&\text{if $M_{PSL}(h,1)\geq\tau$}\\ 0&\text{if $M_{PSL}(h,0)\geq\tau$}\\ \varnothing&\text{otherwise}\end{cases}\]

where threshold \(\tau\) is system hyper-parameter. Inferable set is composed of BETs for which inference algorithm (PSL) confidently propagates labels.

\[{\cal I}(G,{\cal Q})=\{h\in{\cal H}\mid l(h)\neq\varnothing\}\]

Note that two BET nodes from ECG can interact with varying strengths through different constraint nodes; this multi-relational structure requires soft probabilistic propagation.

### Control Mechanism

Control mechanism selects the BET to be crowd-evaluated at every iteration. We first present the following two theorems involving KGEval's optimization in Equation (1). Please refer Appendix for proofs of both theorems.

**Theorem 1**.: **[Submodularity]** _The function optimized by KGEval (Equation (1)) using the PSL-based inference mechanism is submodular [4]._

The proof follows from the fact that all pairs of BETs satisfy the regularity condition [7, 6], further used by a proven conjecture [9].

**Theorem 2**.: **[NP-Hardness]** _The problem of selecting optimal solution in KGEval's optimization (Equation (1)) is NP-Hard._

Proof follows by reducing NP-complete Set-cover Problem (SCP) to selecting \({\cal Q}\) which covers \({\cal I}(G,{\cal Q})\).

**Justification for Greedy Strategy**: From Theorem 1 and 2, we observe that the function optimized by KGEval is NP-hard and submodular.

Figure 3: Evaluation Coupling Graph (ECG) constructed for example in Figure 1. (Section 3.2)



Results from (Nemhauser et al., 1978) prove that greedy hill-climbing algorithms solve such maximization problem within an approximation factor of \((1-1/e)\approx\) 63% of the optimal solution. Hence we iteratively select the next BET which gives the greatest increase in size of inferable set.

We acknowledge the importance of crowdsourcing aspects such as label-aggregation, worker's quality estimation etc. Appendix A.1 presents a mechanism to handle noisy crowd workers under limited budget.

### Bringing it All Together

Algorithm 1 presents KGEval. In Lines 1-3, we build the Evaluation Coupling Graph \(G=({\cal H}\cup{\cal C},{\cal E})\) and use the labels of seed set \({\cal S}\) to initialize \(G\). In lines 4-16, we repetitively run our inference mechanism, until either the accuracy estimates have converged, or all the BETs are covered. In each iteration, the BET with the largest inferable set is identified and evaluated using crowdsourcing (Lines 5-6). The new inferable set \({\cal Q}_{t}\) is estimated. These automatically annotated nodes are added to \({\cal Q}\) (Lines 7-10).

**Convergence:** In this paper, we define convergence whenever the variance of sequence of accuracy estimates [ \(\mathrm{Acc}_{\rm t-k}\), \(\ldots\), \(\mathrm{Acc}_{\rm t-1}\), \(\mathrm{Acc}_{\rm t}\)] is less than \(\alpha\). We set \(k=9\) and \(\alpha=0.002\) for our experiments.

## 4 Experiments

To assess the effectiveness of KGEval, we ask the following questions: (1).How effective is KGEval in estimating KG accuracy, both at predicate-level and at overall KG-level? (Section 4.3). (2). What is the importance of coupling constraints on its performance? (Section 4.4). (3). And lastly, how robust is KGEval to estimating accuracy of KGs with varying quality? (Section 4.5).

### Model Description

#### 4.1.1 Setup

**Datasets**: For experiments, we consider two KGs: NELL and Yago2. From NELL (NELL), we choose a sub-graph of sports related beliefs **NELL-sports**, mostly pertaining to athletes, coaches, teams, leagues, stadiums etc. We construct coupling constraints set \({\cal C}_{\mathbf{N}}\) using top-ranked PRA inference rules for available predicate-relations (Lao et al., 2011). The confidence score returned by PRA are used as weights \(\theta_{i}\). We use NELL-ontology's predicate-signatures to get information for _type_ constraints. Please note that PSL is capable of handling weighted constraints and also learn their weights (relative importance). So, it is not critical to provide absolutely correct constraints. We also select **YAGO2-sample** (YAGO) , which unlike NELL-sports, is not domain specific. We use AMIE horn clauses (Galarraga et al., 2013) to construct multi-relational coupling constraints \({\cal C}_{\mathbf{Y}}\). For each \({\cal C}_{i}\), the score returned by AMIE is used as rule weight \(\theta_{i}\). Table 2 reports the statistics of datasets used, their true accuracy and number of coupling constraints. Obtaining gold-labels for millions of facts is non-trivial and expensive as crowdsourcing over full KG incurs significant cost.

**Size of evaluation set**: NELL-sort consists of \(23,422\) beliefs with \(13,290\) unique entities and \(53\) unique predicates. Whereas YAGO-sample has \(31,720\) beliefs, with unique \(32,103\) entities and \(17\) predicates. In order to calculate accuracy, we

\begin{table}
\begin{tabular}{|c|c|c|} \hline Evaluation set & \({\cal H}_{N}\) & \({\cal H}_{Y}\) \\ \hline \#BETs & \(1860\) & \(1386\) \\ \#Constraints & \(|{\cal C}_{N}|=130\) & \(|{\cal C}_{Y}|=28\) \\ \#Predicates & \(18\) & \(16\) \\ Gold Acc. & \(91.34\%\) & \(99.20\%\) \\ \hline \end{tabular}
\end{table}
Table 2: Details of BET subsets used for accuracy evaluation. (Section 4.1.2).

 

require gold evaluation of all beliefs in the evaluation set. Since obtaining gold evaluation of the entire (or large subsets of) NELL and Yago2 KGs will be prohibitively expensive, we take subset of these KGs for evaluation. (KGEval) consists of datasets used, their crowdsourced labels, coupling constraints and code for inference/control.

**Initialization:** Algorithm 1 requires initial seed set \({\cal S}\) which we generate by randomly evaluating \(|{\cal S}|=50\) beliefs from \({\cal H}\). To maintain fairness, all baselines start from \({\cal S}\). For asserting true (or false) value for beliefs, we set a high soft label confidence threshold at \(\tau=0.8\) (see Section 3.3).

#### 4.1.2 Crowdsourcing of BETs

To compare KGEval predictions against human evaluations, we evaluate all BETs \(\{{\cal H}_{N}\cup{\cal H}_{Y}\}\) on AMT. For the ease of workers, we translate each _entity-relation-entity_ belief into human readable format before posting to crowd.

We published BETs on AMT under 'classification project' category. We hired AMT recognized master workers for high quality labels and paid $0.01 per BET. To compare between 'master' and 'noisy' workers, we correlated their labels individually to expert labels on random subset and observed that master workers were better correlated (\(93\%\)) as compared to three non-masters (\(89\%\)). Consequently we consider votes of master workers for \(\{{\cal H}_{N}\cup{\cal H}_{Y}\}\) as gold labels, which we would like our inference algorithm to be able to predict. As the average turnaround time for AMT tasks runs into a few minutes Dupuis et al. (2013), KGEval is effectively real-time within such turnaround time range.

#### 4.1.3 Performance Evaluation Metrics

Performance of various methods are evaluated using the following two metrics. To capture accuracy at the predicate level, we define \(\Delta_{predicate}\) as the average of difference between gold and estimated accuracy of each of the \(R\) predicates in KG.

\[\Delta_{predicate} = \frac{1}{|R|}\Bigg{(}\sum_{r\in R}\left|\Phi({\cal H}_{r})-\frac {1}{|{\cal H}_{r}|}\sum_{\forall h\in{\cal H}_{r}}l(h)\right|\Bigg{)}\]

We define \(\Delta_{overall}\) as the difference between gold and estimated accuracy over the entire evaluation set.

\[\Delta_{overall} = \bigg{|}\Phi({\cal H})-\frac{1}{|{\cal H}|}\sum_{\forall h\in{ \cal H}}l(h)\bigg{|}\]

Above, \(\Phi({\cal H})\) is the overall gold accuracy, \(\Phi({\cal H}_{r})\) is the gold accuracy of predicate \(r\) and \(l(h)\) is the label assigned by the currently evaluated method. \(\Delta_{overall}\) treats entire KG as a single bag of BETs whereas \(\Delta_{predicate}\) segregates beliefs based on their type of predicate-relation. For both metrics, lower is better.

### Baseline Methods

Since accuracy estimation of large multi-relational KGs is a relatively unexplored problem, there are no well established baselines for this task (apart from random sampling). We present below the baselines which we compared against KGEval.

**Random:** Randomly sample a BET \(h\in{\cal H}\) without replacement and crowdsource for its correctness. Selection of every subsequent BET is independent of previous selections.

**Max-Degree:** Sort the BETs in decreasing order of their degrees in ECG and select them from top for evaluation; this method favors selection of more centrally connected BETs first.

**Independent Cascade:** This method is based on contagion transmission model where nodes only infect their immediate neighbors Kempe et al. (2003). At every time iteration \(t\), we choose a BET which is not evaluated yet, crowdsource for its label and let it propagate its evaluation label in ECG.

**KGEval:** Method proposed in Algorithm 1.

### Effectiveness of KGEval

Experimental results of all methods comparing \(\Delta_{overall}\) and \(\Delta_{predicate}\) at convergence, are presented in Table 3. We observe that KGEval is able to achieve the best estimate across both datasets and metrics. Due to the significant positive bias in \({\cal H}_{Y}\) (see Table 2), all methods do fairly well as per \(\Delta_{overall}\) on this dataset, even though KGEval still outperforms others. Also, KGEval is able to estimate KG accuracy most closely while utilizing least number of crowd-evaluated queries. This clearly demonstrates KGEval's effectiveness.

Nodes in coupling graph with higher degrees are those which participate in large number of constraints. In real KGs, such facts tend to be correct as they interact with several other facts. Hence, MaxDegree overestimates the accuracy by propagating True label. In contrast, Random samples True and False labels in unbiased fashion.

**Predicate-level Analysis**: Here, we consider the top two baselines from Table 3, viz., Random and 

KGEval, and compare performance on the \({\cal H}_{N}\) dataset. We use \((1-[\Delta_{overall}]_{predicate})\) as the metric, which means \(\Delta_{overall}\) computed over individual predicates. Here, we are interested in evaluating how well the two methods have estimated per-predicate accuracy when KGEval's \(\Delta_{overall}\) has converged. Comparison of per-predicate performances of the two methods is shown in Figure 4. Observe that KGEval significantly outperforms Random baseline. Its advantage lies in exploiting the coupling constraints among beliefs, where evaluating a belief from certain predicate helps infer beliefs from other predicates.

### Importance of Coupling Constraints

This paper is largely motivated by the thesis - _exploiting richer relational couplings among BETs may result in faster and more accurate evaluations_. To evaluate this thesis, we successively ablated Horn clause coupling constraints of body-length 2 and 3 from \({\cal C}_{N}\).

We observe that with the full (non-ablated) constraint set \({\cal C}_{N}\), KGEval takes least number of crowd evaluations of BETs to convergence, while providing best accuracy estimate. Whereas with ablated constraint sets, KGEval takes up to \(2.4\)x more crowd-evaluation queries for convergence; thus validating our thesis.

### Additional Experiments

* \(1.2\)x and \(1.35\)x more, respectively. This shows effectiveness of greedy mechanism.

**Rate of Coverage:** In case of large KGs with scarce budget, it is imperative to have a mechanism which covers greater parts of KG with lesser number of crowdsource queries. Figure 5 shows the fraction of total beliefs whose evaluations were automatically inferred by different methods as a function of number of crowd-evaluated beliefs. We observe that KGEval infers evaluation for the largest number of BETs at each supervision level.

**Robustness to Noise:** In order to test robustness of the methods in estimating accuracies

Figure 4: Comparing \((1-[\Delta_{overall}]_{precicate})\) of individual predicates (higher is better) in \({\cal H}_{N}\) between KGEval and Random, the two top performing systems in Table 3. (see Section 4.3)



of KGs with different gold accuracies, we artificially added noise to \({\cal H}_{N}\) by flipping a fixed fraction of edges, otherwise following the same evaluation procedure as in Section 3.5. We analyze \(\Delta_{overall}\) (and not \(\Delta_{predicate}\)) because flipping edges in KG distorts predicate-relations dependencies and present in Table 5. We evaluated all the methods and observed that while performance of other methods degraded significantly with diminishing KG quality (more noise), KGEval was significantly robust to noise.

**Scalability comparisons with MLN:** Markov Logic Networks Richardson and Domingos (2006) can serve as a candidate for Inference Mechanism. We compared the runtime performance of KGEval with PSL and MLN as inference engines. While PSL took 320 seconds to complete one iteration, the MLN implementation (PyMLN) could not finish grounding the rules even after 7 hours. This justifies our choice of PSL as the inference engine for KGEval.

## 5 Related Work

Even though Knowledge Graph (KG) construction is an active area of research, we are not aware of any previous research which systematically studies the important problem of estimating accuracy of such automatically constructed KGs. Random sampling has traditionally been the most preferred way for large-scale KG accuracy estimation Mitchell et al. (2015).

Traditional crowdsourcing research has typically considered atomic allocation of tasks where the requester posts them _independently_. KGEval operates in a rather novel crowdsourcing setting as it exploits dependencies among its tasks (BETs or belief evaluations). Our notion of interdependence (coupling constraints) among tasks is more general and different than related ideas explored in the crowdsourcing literature before Kolobov et al. (2013); Bragg et al. (2013); Sun et al. (2015). Even though coupling constraints have been used for KG construction Nakashole et al. (2011); Galarraga et al. (2013); Mitchell et al. (2015), they have so far not been exploited for KG evaluation. We address this gap in this paper.

The task of knowledge corroboration Kasneci et al. (2010) proposes probabilistic model to utilize a fixed set of basic first-order logic rules for label propagation and is closely aligned with our motivations. However, unlike KGEval, it does not try to reduce the number of queries to crowdsource or maximize coverage.

## 6 Conclusion

In this paper, we have initiated a systematic study into the important problem of evaluation of automatically constructed Knowledge Graphs. In order to address this challenge, we have proposed KGEval, an instance of a novel crowdsourcing paradigm where dependencies among tasks presented to humans (BETs) are exploited. To the best of our knowledge, this is the first method of its kind. We demonstrated that the objective optimized by KGEval is in fact NP-Hard and submodular, and hence allows for the application of simple greedy algorithms with guarantees. Through extensive experiments on real datasets, we demonstrated effectiveness of KGEval. We hope to extend KGEval to incorporate varying evaluation cost, and also explore more sophisticated evaluation aggregation.

## Acknowledgments

This research has been supported by MHRD (Govt. of India) and in part by a gift from Google.

Figure 5: Fraction of total beliefs whose evaluation where automatically inferred by different methods for varying number of crowd-evaluated queries (x-axis) in \({\cal H}_{N}\).

\begin{table}
\begin{tabular}{|l|c|c|} \hline \multicolumn{4}{|c|}{NELL sports + 5\% noise (\({\cal H}_{NS}\))} \\ \hline Method & \(\Delta_{overall}\) (\%) & \# Queries \\ \hline Random & \(1.8\) & 563 \\ \hline Max-Degree & \(4.2\) & 1249 \\ \hline Ind-Cascade & \(1.2\) & 370 \\ \hline KGEval & **0.8** & **106** \\ \hline \end{tabular} 
\begin{tabular}{|l|c|c|} \hline \multicolumn{4}{|c|}{NELL sorts + 10\% noise (\({\cal H}_{NN10}\))} \\ \hline Random & \(1.8\) & 728 \\ \hline Max-Degree & \(6.2\) & 1501 \\ \hline Ind-Cascade & \(1.2\) & 406 \\ \hline KGEval & **0.2** & **115** \\ \hline \end{tabular}
\end{table}
Table 5: Accuracy estimate (higher is better) over entire KG by various baselines in the presence of noise.

 

Appendix

**Submodularity:** A real valued function \(f\), which acts over subsets of any finite set \({\cal H}\), is said to be _submodular_ if \(\forall R,S\subset{\cal H}\) it fulfills

\[f(R)+f(S)\geq f(R\cup S)+f(R\cap S).\]

We call potential function \(\psi\) as pairwise _regular_ if for all pairs of BETs \(\{p,q\}\in{\cal H}\) it satisfies

Proof.: **(for Theorem 1)** The additional utility, in terms of label inference, obtained by adding a BET to larger set is lesser than adding it to any smaller subset. By construction, any two BETs which share a common factor node \(C_{j}\) are encouraged to have similar labels in \(G\).

Potential functions \(\psi_{j}\) of Equation (3) satisfy pairwise regularity property i.e., for all BETs \(\{p,q\}\in{\cal H}\)

\[\psi(0,1)+\psi(1,0)\geq\psi(0,0)+\psi(1,1) \tag{4}\]

where \(\{1,0\}\) represent true/false. Equivalence of _ submodular_ and _regular_ properties are established (Kolmogorov and Zabih, 2004; Jegelka and Bilmes, 2011). Using non-negative summation property (Lovasz, 1983), \(\sum_{j\in{\cal C}}\theta_{j}\psi_{j}\) is submodular for positive weights \(\theta_{j}\geq 0\).

We consider a BET to be confidently inferred when the soft score of its label assignment in \({\cal I}(G,{\cal Q})\) is greater than threshold \(\tau_{h}\in[0,1]\). From above we know that \({\mathbb{P}}(l(h)|{\cal Q})\) is submodular with respect to fixed initial Set \({\cal Q}\). Although max or min of submodular functions are not submodular in general, but (Kempe et al., 2003) conjectured that global function of Equation (1) is submodular if local threshold function \({\mathbb{P}}(h|{\cal Q})\geq\tau_{h}\) respected submodularity, which holds good in our case of Equation (3). This conjecture was further proved in (Mossel and Roch, 2007) and thus making our global optimization function of Equation (1) submodular. 

Proof.: **(for Theorem 2)** We reduce KGEval to NP-complete Set-cover Problem (SCP) so as to select \({\cal Q}\) which covers \({\cal I}(G,{\cal Q})\). For the proof to remain consistent with earlier notations, we define SCP by collection of subsets \({\cal I}_{1},{\cal I}_{2},\ldots,{\cal I}_{m}\) from set \({\cal H}=\{h_{1},h_{2},\ldots,h_{n}\}\) and we want to determine if there exist \(k\) subsets whose union equals \({\cal H}\). We define a bipartite graph with \(m+n\) nodes corresponding to \({\cal I}_{i}\)'s and \(h_{j}\)'s respectively and construct edge \(({\cal I}_{i},h_{j})\) if \(h_{j}\in{\cal I}_{i}\). We need to find a set \({\cal Q}\), with cardinality k, such that \(|{\cal I}(G,{\cal Q})|\geq n+k\).

Choosing our BET-set \({\cal Q}\) from SCP solution and further inferring evaluations of other remaining BETs using PSL will solve the problem in hand. 

### Noisy Crowd Workers and Budget

Here, we provide a scheme to allow crowd workers so as to remain within specified budget and upper bound total error on accuracy estimate. We have not integrated this mechanism with Algorithm 1 to maintain its simplicity.

We resort to majority voting in our analysis and assume that crowd workers are not adversarial. So expectation over responses \(r_{h}(u)\) for a task \(h\) with respect to multiple workers \(u\) is close to its true label \(t(h)\)(Tran-Thanh et al., 2013), i.e.,

\[\left|\operatorname*{E}_{w\sim{\mathbb{D}}(u,h)}[r_{h}(u)]-t(h)\right|\leq \frac{1}{2} \tag{5}\]

where \({\mathbb{D}}\) is joint probability distribution of workers \(u\) and tasks \(h\).

Our key idea is that we want to be more confident about BETs \(h\) with larger inferable set (as they impact larger parts of KG) and hence allocate them more budget to post to more workers. We determine the number of workers \(\{w_{h_{1}},\ldots,w_{h_{n}}\}\) for each task such that \(h_{t}\) with larger inference set have higher \(w_{h_{t}}\). For total budget \(B\), we allocate

\[w_{h_{t}}=\lfloor\tfrac{B~{}i_{t}~{}(1-\gamma)}{c~{}i_{max}}\rfloor\]

where \(i_{t}\) denotes the cardinality of inferable set \({\cal I}(G,{\mathcal{Q}}\cup h_{t})\), \(c\) the cost of querying crowd worker, \(i_{max}\) the size of largest inferable set and \(\gamma\in[0,1]\) constant.

This allocation mechanism easily integrates with Algorithm 1; in (Line 8) we determine size of inferable set \(i_{t}=|{\cal Q}_{t}|\) for task \(h\) and allocate \(w_{h}\) crowd workers. Budget depletion (Line 9) is modified to \(B_{F}=P_{r}-w_{h}c(h)\). The following theorem bounds the error with such allocation scheme.

**Theorem 3**.: **[Error bounds]** _The allocation scheme of redundantly posing \(h_{t}\) to \(w_{h_{t}}\) workers does not exceed the total budget \(B\) and its expected estimation error is upper bounded by \(e^{-O(t_{t})}\), keeping other parameters fixed. The expected estimation error over all tasks is upper bounded by \(e^{-O(B)}\)._

Proof.: Let \(\gamma\in[0,1]\) control the reduction in size of inferable set by \(~{}i_{t+1}=\gamma~{}i_{t}\). By allocating \(w_{h_{t}}\) redundant workers for task \(h_{t},\forall t\in\{1\cdots n\}\) with size of inferable set \(i_{t}\), we incur total cost of

\[\sum_{t=1}^{n}c~{}w_{h_{t}} = \sum_{t=1}^{n}\dfrac{B~{}i_{t}~{}(1-\gamma)}{c~{}i_{max}}\cdot c\] \[= \left(\sum_{t=1}^{n}i_{t}\right)\cdot\left(\dfrac{B~{}(1-\gamma)} {i_{max}}\right)\] \[= \left(\dfrac{i_{max}~{}(1-\gamma^{T})}{(1-\gamma)}\right)\cdot \left(\dfrac{B~{}(1-\gamma)}{i_{max}}\right)\] \[\leq B\]

Note that the above geometric approximation helps in estimating summation \(\sum_{t=1}^{n}i_{t}\) at iteration \(t\leq n\).

Error Bounds:Here we show that the expected error of estimating of \(h_{t}\) for any time \(t\) decreases exponentially in the size of inferable set \(i_{t}\). We use majority voting to aggregate \(w_{h_{t}}\) worker responses for \(h_{t}\), denoted by \(\hat{r}_{h_{t}}\in\{0,1\}\)

\[\hat{r}_{h_{t}}=\left\lfloor\frac{1}{w_{h_{t}}}\sum_{k=1}^{w_{h_{t}}}r_{h_{t}}( u_{k})-\frac{1}{2}\right\rfloor+1 \tag{6}\]

where \(r_{h_{t}}(u_{k})\) is the response by \(k^{th}\) worker for \(h_{t}\). The error from aggregated response can be given by \(\Delta(h_{t})=|\hat{r}_{h_{t}}-t(h_{t})|\), where \(t(h_{t})\) is its true label. From Equation (5) and Hoeffding-Azuma bounds over \(w_{h_{t}}\) i.i.d responses and error margin \(\varepsilon_{t}\), we have

\[\Delta(h_{t}) = {\mathbb{P}}\Bigg{\{}\bigg{|}\frac{1}{w_{h_{t}}}\sum_{k=1}^{w_{h_{ t}}}r_{h_{t}}(u_{k})-{\mathbb{E}}(r_{h}(u))\bigg{|}\geq\varepsilon_{t}\Bigg{\}}\] \[= 2\exp\left(-2~{}\dfrac{B~{}i_{t}~{}(1-\gamma)}{c~{}i_{max}}~{} \varepsilon_{t}^{2}\right)\]

For fixed budget \(B\) and given error margin \(\varepsilon_{t}\), we have \(\Delta(h_{t})=e^{-O(t_{t})}\). Summing up over all tasks \(t\), by union bounds we get the total expected error from absolute truth as \(\Delta(B)=\sum_{t=1}^{n}\Delta(h_{t})\).

\[\Delta(B) \leq \sum_{t=1}^{n}2\exp\left(-2~{}\dfrac{B~{}i_{t}~{}(1-\gamma)}{c~{}i_ {max}}~{}\varepsilon_{t}^{2}\right)\] \[\leq n\cdot 2\exp\left(-2~{}\dfrac{B~{}i_{min}~{}(1-\gamma)}{c~{}i_{max}}~{} \varepsilon_{min}^{2}\right)\]

The accuracy estimation error will decay exponentially with increase in total budget for fixed parameters 



## References

* AMT ([https://www.mturk.com](https://www.mturk.com)).
* Bragg et al. (2013) Jonathan Bragg, Daniel S Weld, et al. 2013. Crowdsourcing multi-label classification for taxonomy creation. In _HCOMP_.
* Broecheler et al. (2010) Matthias Broecheler, Lilyana Mihalkova, and Lise Getoor. 2010. Probabilistic similarity logic. In _UAI_.
* Dong et al. (2014) Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In _SIGKDD_, pages 601-610.
* Dupuis et al. (2013) Marc Dupuis, Barbara Endicott-Popovsky, and Robert Crossler. 2013. An analysis of the use of amazons mechanical turk for survey research in the cloud. In _ICCSM'2013-Proceedings of the International Conference on Cloud Security Management: ICCSM 2013_, page 10. Academic Conferences Limited.
* Galarraga et al. (2013) Luis Antonio Galarraga, Christina Teflioudi, Katja Hose, and Fabian Suchanek. 2013. Amie: association rule mining under incomplete evidence in ontological knowledge bases. In _WWW_, pages 413-422.
* Jegelka and Bilmes (2011) Stefanie Jegelka and Jeff Bilmes. 2011. Submodularity beyond submodular energies: coupling edges in graph cuts. In _CVPR_, pages 1897-1904.
* Kasneci et al. (2010) Gjergji Kasneci, Jurgen Van Gael, Ralf Herbrich, and Thore Graepel. 2010. Bayesian knowledge corroboration with logical rules and user feedback. In _Machine Learning and Knowledge Discovery in Databases_, pages 1-18.
* Kempe et al. (2003) David Kempe, Jon Kleinberg, and Eva Tardos. 2003. Maximizing the spread of influence through a social network. In _SIGKDD_.
* KGEval ([http://taukdar.net/mall-lab/KGEval](http://taukdar.net/mall-lab/KGEval)).
* Kolmogorov and Zabih (2004) Vladimir Kolmogorov and Ramin Zabih. 2004. What energy functions can be minimized via graph cuts? _Pattern Analysis and Machine Intelligence, IEEE Transactions on_, 26(2):147-159.
* Kolobov et al. (2013) Andrey Kolobov, Daniel S Weld, et al. 2013. Joint crowdsourcing of multiple tasks. In _HCOMP_.
* Keschischang et al. (2001) Frank R Keschischang, Brendan J Frey, and H-A Loeliger. 2001. Factor graphs and the sum-product algorithm. _Information Theory, IEEE Transactions on_, 47(2):498-519.
* Lao et al. (2011) Ni Lao, Tom Mitchell, and William W Cohen. 2011. Random walk inference and learning in a large scale knowledge base. In _EMNLP_, pages 529-539.
* Lovasz (1983) Laszlo Lovasz. 1983. Submodular functions and convexity. In _Mathematical Programming The State of the Art_, pages 235-257. Springer.
* Mitchell et al. (2015) T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, J. Betteridge, A. Carlson, B. Dalvi, M. Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Platanios, A. Ritter, M. Samadi, B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen, A. Saparov, M. Greaves, and J. Welling. 2015. Never-ending learning. In _Proceedings of AAAI_.
* Mossel and Roch (2007) Elchanan Mossel and Sebastien Roch. 2007. On the submodularity of influence in social networks. In _ACM symposium on Theory of computing_, pages 128-134.
* Nakashole et al. (2011) Ndapandula Nakashole, Martin Theobald, and Gerhard Weikum. 2011. Scalable knowledge harvesting with high precision and high recall. In _Proceedings of WSDM_.
* NELL ([http://rtw.ml.cmu.edu/rtw/resources](http://rtw.ml.cmu.edu/rtw/resources)).
* Nemhauser et al. (1978) George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. 1978. An analysis of approximations for maximizing submodular set functionsi. _Mathematical Programming_, 14(1):265-294.
* PSL ([http://www.psl.umiacs.umd.edu](http://www.psl.umiacs.umd.edu)).
* Pujara et al. (2013) Jay Pujara, Hui Miao, Lise Getoor, and William Cohen. 2013. Knowledge graph identification. In _The Semantic Web-ISWC 2013_, pages 542-557. Springer.
* PyMLN ([http://ias.cs.tum.edu/people/jan/mlns](http://ias.cs.tum.edu/people/jan/mlns)).
* Richardson and Domingos (2006) Matthew Richardson and Pedro Domingos. 2006. Markov logic networks. _Machine learning_, 62(1-2):107-136.
* Samadi et al. (2015) Mehdi Samadi, Partha Talukdar, Manuela Veloso, and Tom Mitchell. 2015. Askworld: budget-sensitive query evaluation for knowledge-on-demand. In _Proceedings of the 24th International Conference on Artificial Intelligence_, pages 837-843. AAAI Press.
* Sun et al. (2015) Yuyin Sun, Adish Singla, Dieter Fox, and Andreas Krause. 2015. Building hierarchies of concepts via crowdsourcing. _arXiv preprint arXiv:1504.07302_.
* Tran-Thanh et al. (2013) Long Tran-Thanh, Matteo Venazzi, Alex Rogers, and Nicholas R Jennings. 2013. Efficient budget allocation with accuracy guarantees for crowdsourcing classification tasks. In _AAMAS_.
* YAGO ([https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-nasq](https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-nasq)).