Venue: NAACL 2018

# Joint Bootstrapping Machines for High Confidence Relation Extraction

Pankaj Gupta1,2, Benjamin Roth2, Hinrich Schutze2

1Corporate Technology, Machine-Intelligence (MIC-DE), Siemens AG Munich, Germany

2CIS, University of Munich (LMU) Munich, Germany

pankaj.gupta@siemens.com | pankaj.gupta@campus.lmu.de

{beroth, inquiries}@cis.lmu.de

###### Abstract

Semi-supervised bootstrapping techniques for relationship extraction from text iteratively expand a set of initial seed instances. Due to the lack of labeled data, a key challenge in bootstrapping is semantic drift: if a false positive instance is added during an iteration, then all following iterations are contaminated. We introduce BREX, a new bootstrapping method that protects against such contamination by highly effective confidence assessment. This is achieved by using entity and template seeds jointly (as opposed to just one as in previous work), by expanding entities and templates in parallel and in a mutually constraining fashion in each iteration and by introducing higher-quality similarity measures for templates. Experimental results show that BREX achieves an \(F_{1}\) that is 0.13 (0.87 vs. 0.74) better than the state of the art for four relationships.

## 1 Introduction

Traditional semi-supervised bootstrapping relation extractors (REs) such as BREDS [6], SnowBall [5] and DIPRE [9] require an initial set of seed _entity pairs_ for the target binary relation. They find occurrences of positive seed entity pairs in the corpus, which are converted into extraction patterns, i.e., _extractors_, where we define an extractor as a cluster of instances generated from the corpus. The initial seed entity pair set is expanded with the relationship entity pairs newly extracted by the extractors from the text iteratively. The augmented set is then used to extract new relationships until a stopping criterion is met.

Due to lack of sufficient labeled data, rule-based systems dominate commercial use [11]. Rules are typically defined by creating patterns around the entities (entity extraction) or entity pairs (relation extraction). Recently, supervised machine learning, especially deep learning techniques [14; 15; 16; 17; 18], have shown promising results in entity and relation extraction; however, they need sufficient hand-labeled data to train models, which can be costly and time consuming for web-scale extractions. Bootstrapping machine-learned rules can make extractions easier on large corpora. Thus, open information extraction systems [19; 20; 18; 19; 21] have recently been popular for domain specific or independent pattern learning.

Hearst (1992) used hand written rules to generate more rules to extract hypernym-hyponym pairs, without distributional similarity. For entity extraction, Riloff (1996) used seed entities to generate extractors with heuristic rules and scored them by counting positive extractions. Prior work [13; 14] investigated different extractor scoring measures. Gupta and Manning (2014) improved scores by introducing expected number of negative entities.

Brin (1998) developed the bootstrapping relation extraction system DIPRE that generates extractors by clustering contexts based on string matching. SnowBall [5] is inspired by DIPRE but computes a TF-IDF representation of each context. BREDS [6] uses word embeddings [12] to bootstrap relationships.

Related work investigated adapting extractor scoring measures in bootstrapping entity extraction with either entities or _templates_ (Table 1) as seeds (Table 2). The state-of-the-art relation extractors bootstrap with only seed entity pairs and suffer due to a surplus of unknown extractions and the lack of labeled data, leading to low confidence extractors. This in turn leads to to low confidence in the system output. Prior RE sys

tems do not focus on improving the extractors' scores. In addition, SnowBall and BREDS used a weighting scheme to incorporate the importance of contexts around entities and compute a similarity score that introduces additional parameters and does not generalize well.

**Contributions. (1)** We propose a _Joint Bootstrapping Machine_1 (JBM), an alternative to the entity-pair-centered bootstrapping for relation extraction that can take advantage of both entity-pair and template-centered methods to jointly learn extractors consisting of instances due to the occurrences of both entity pair and template seeds. It scales up the number of positive extractions for _non-noisy_ extractors and boosts their confidence scores. We focus on improving the scores for _non-noisy-low-confidence_ extractors, resulting in higher _recall_. The relation extractors bootstrapped with entity pair, template and joint seeds are named as _BREE_, _BBET_ and _BREJ_ (Table 1), respectively.

Footnote 1: github.com/pgcool/Joint-Bootstrapping-Machines

* "bought"). In reality, there is _large variation in the syntax_ of how relations are expressed, e.g., "MSFT to acquire NOK for $8B" vs. "MSFT earnings hurt by NOK acquisition". We introduce cross-context similarities that compare all parts of the context (e.g., "to acquire" and "acquisition") and show that these perform better (in terms of recall) than measures assuming consistent syntactic expression of relations.

**(3)** Experimental results demonstrate a 13% gain in \(F1\) score on average for four relationships and suggest eliminating four parameters, compared to the state-of-the-art method.

The _motivation_ and _benefits_ of the proposed JBM for relation extraction is discussed in depth in section 2.3. The method is applicable for both entity and relation extraction tasks. However, in _context of relation extraction_, we call it _BREJ_.

## 2 Method

### Notation and definitions

We first introduce the notation and terms (Table 1).

Given a relationship like "\(x\) acquires \(y\)", the task is to extract pairs of entities from a corpus for which the relationship is true. We assume that the arguments of the relationship are typed, e.g., \(x\) and \(y\) are organizations. We run a named entity tagger in preprocessing, so that the types of all candidate entities are given. The objects the bootstrapping algorithm generally handles are therefore _typed entities_ (an entity associated with a type).

For a particular sentence in a corpus that states that the relationship (e.g., "acquires") holds between \(x\) and \(y\), a _template_ consists of three vectors that represent the context of \(x\) and \(y\). \(\vec{v}_{-1}\) represents the context before \(x\), \(\vec{v}_{0}\) the context between \(x\) and \(y\) and \(\vec{v}_{1}\) the context after \(y\). These vectors are simply sums of the embeddings of the corresponding words. A template is "typed", i.e., in addition to the three vectors it specifies the types of the two entities. An _instance_ joins an entity pair and a template. The types of entity pair and template must be the same.

The first step of bootstrapping is to extract a set of instances from the input corpus. We refer to this set as \(\gamma\). We will use \(i\) and \(j\) to refer to instances. \(x(i)\) is the entity pair of instance \(i\) and \(\mathtt{r}(\mathtt{i})\) is the template of instance \(i\).

A required input to our algorithm are sets of positive and negative seeds for either entity pairs (\(G_{p}\) and \(G_{n}\)) or templates (\(\mathfrak{G}_{p}\) and \(\mathfrak{G}_{n}\)) or both. We define \(\mathcal{G}\) to be a tuple of all four seed sets.

We run our bootstrapping algorithm for \(k_{\text{it}}\) iterations where \(k_{\text{it}}\) is a parameter.

\begin{table}
\begin{tabular}{l l} BREE & Bootstrapping Relation Extractor with _Entity pair_ \\ BRET & Bootstrapping Relation extractor with _Template_ \\ BEJJ & Bootstrapping Relation Extractor in Joint learning \\ type & a named entity preo, e.g., person \\ typed entity & a typed entity, e.g., “CObama”,person> \\ entity pair & a pair of two typed entities \\ template & a triple of vectors (\(\vec{v}_{-1}\), \(\vec{v}_{0}\), \(\vec{v}_{1}\)) and an entity pair \\ instance & entity pair and template (types must be the same) \\ \(\gamma\) & instance set extracted from corpus \\ \(i\) & a member of \(\gamma\); i.e., an instance \\ \(x(i)\) & the entity pair of instance \(i\) \\ \(\mathtt{f}(i)\) & the template of instance \(i\) \\ \(G_{p}\) & a set of positive seed entity pairs \\ \(G_{n}\) & a set of negative seed entity pairs \\ \(\mathfrak{G}_{p}\) & a set of positive seed templates \\ \(\mathfrak{G}_{n}\) & a set of negative seed templates \\ \(\mathfrak{G}_{q}\) & \(<G_{p},G_{n},\mathfrak{G}_{p},\mathfrak{G}_{n}>\) \\ \(k_{\text{it}}\) & number of iterations \\ \(\lambda_{out}\) & cluster of instances (_extractor_) \\ \(cat\) & category of _extractor_\(\lambda\) \\ \(\lambda_{NNIC}\) & Non-Noisy-High Confidence extractor (True Positive) \\ \(\lambda_{NNIC}\) & Non-Noisy-Low-Confidence extractor (True Negative) \\ \(\lambda_{NNIC}\) & Noisy-High-Confidence extractor (False Positive) \\ \(\lambda_{NLC}\) & Noisy-Low-Confidence extractor (False Negative) \\ \hline \end{tabular}
\end{table}
Table 1: Notation and definition of key terms 

A key notion is the similarity between two instances. We will experiment with different similarity measures. The baseline is (Batista et al., 2015)'s measure given in Figure 4, first line: the similarity of two instances is given as a weighted sum of the dot products of their before contexts (\(\vec{v}_{-1}\)), their between contexts (\(\vec{v}_{0}\)) and their after contexts (\(\vec{v}_{1}\)) where the weights \(w_{p}\) are parameters. We give this definition for instances, but it also applies to templates since only the context vectors of an instance are used, not the entities.

The similarity between an instance \(i\) and a cluster \(\lambda\) of instances is defined as the maximum similarity of \(i\) with any member of the cluster; see Figure 2, right, Eq. 5. Again, there is a straightforward extension to a cluster of templates: see Figure 2, right, Eq. 6.

The extractors \(\Lambda\) can be categorized as follows:

\[\Lambda_{NNHC}=\{\lambda\in\Lambda|\underbrace{\lambda\mapsto\mathfrak{R}}_{ non\text{-}noisy}\wedge\operatorname*{cnf}(\lambda,\mathcal{G})\geqslant\tau_{ enf}\} \tag{1}\]

\[\Lambda_{NNLC}=\{\lambda\in\Lambda|\lambda\mapsto\mathfrak{R}\wedge \operatorname*{cnf}(\lambda,\mathcal{G})<\tau_{enf}\} \tag{2}\]

\[\Lambda_{NHC}=\{\lambda\in\Lambda|\underbrace{\lambda\mapsto\mathfrak{R}}_{ noisy}\wedge\operatorname*{cnf}(\lambda,\mathcal{G})\geqslant\tau_{cnf}\} \tag{3}\]

\[\Lambda_{NLC}=\{\lambda\in\Lambda|\lambda\mapsto\mathfrak{R}\wedge \operatorname*{cnf}(\lambda,\mathcal{G})<\tau_{enf}\} \tag{4}\]

where \(\mathfrak{R}\) is the relation to be bootstrapped. The \(\lambda_{cat}\) is a member of \(\Lambda_{cat}\). For instance, a \(\lambda_{NNLC}\) is called as a _non-noisy-low-confidence_ extractor if it represents the target relation (i.e., \(\lambda\mapsto\mathfrak{R}\)), however with the confidence below a certain threshold (\(\tau_{cnf}\)). Extractors of types \(\Lambda_{NNHC}\) and \(\Lambda_{NLC}\) are desirable, those of types \(\Lambda_{NHC}\) and \(\Lambda_{NNLC}\) undesirable within bootstrapping.

### The Bootstraping Machines: BREX

To describe Brex (Figure 1) in its most general form, we use the term _item_ to refer to an entity pair, a template or both.

The input to Brex (Figure 2, left, line 01) is a set \(\gamma\) of instances extracted from a corpus and \(\mathcal{G}_{\text{seed}}\), a structure consisting of one set of positive and one set of negative seed items. \(\mathcal{G}_{\text{yield}}\) (line 02) collects the items that BREX extracts in several iterations. In each of \(k_{\text{it}}\) iterations (line 03), BREX first initializes the cache \(\mathcal{G}_{\text{cache}}\) (line 04); this cache collects the items that are extracted in this iteration. The design of the algorithm balances elements that ensure high recall with elements that ensure high precision.

High recall is achieved by starting with the seeds and making three "hops" that consecutively consider order-1, order-2 and order-3 neighbors of the seeds. On line 05, we make the first hop: all instances that are similar to a seed are collected where "similarity" is defined differently for different BREX configurations (see below). The collected instances are then clustered, similar to work on bootstrapping by Agichtein and Gravano (2000) and Batista et al. (2015). On line 06, we make the second hop: all instances that are within \(\tau_{\text{sim}}\) of a hop-1 instance are added; each such instance is only added to one cluster, the closest one; see definition of \(\mu\): Figure 2, Eq. 8. On line 07, we make the third hop: we include all instances that are within \(\tau_{\text{sim}}\) of a hop-2 instance; see definition of \(\psi\): Figure 2, Eq. 7. In summary, every instance that can be reached by three hops from a seed is being considered at this point. A cluster of hop-2 instances is named as _extractor_.

High precision is achieved by imposing, on line 08, a stringent check on each instance before its information is added to the cache. The core function of this check is given in Figure 2, Eq. 9. This definition is a soft version of the following hard max, which is easier to explain:

\[\operatorname*{cnf}(i,\Lambda,\mathcal{G})\approx\max_{\{\lambda\in\mathbf{A} \mid\sharp\psi(\lambda)\}}\operatorname*{cnf}(i,\lambda,\mathcal{G})\]

We are looking for a cluster \(\lambda\) in \(\Lambda\) that licenses the extraction of \(i\) with high confidence. \(\operatorname*{cnf}(i,\lambda,\mathcal{G})\) (Figure 2, Eq. 10), the _confidence_ of a single cluster (i.e., extractor) \(\lambda\) for an instance, is defined as the product of the overall reliability of \(\lambda\) (which is independent of \(i\)) and the similarity of \(i\) to \(\lambda\), the second factor in Eq. 10, i.e., \(\operatorname*{sm}(i,\lambda)\). This factor \(\operatorname*{sim}(i,\lambda)\) prevents an extraction by a cluster whose members are all distant from the instance - even if the cluster itself is highly reliable.

Figure 1: Joint Bootstrapping Machine. The red and blue filled circles/rings are the instances generated due to seed entity pairs and templates, respectively. Each dashed rectangular box represents a cluster of instances. Numbers indicate the flow. Follow the notations from Table 1 and Figure 2.

 

The first factor in Eq. 10, i.e., \({\rm cnf}(\lambda,\mathcal{G})\), assesses the reliability of a cluster \(\lambda\): we compute the ratio \(\frac{N_{+}(\lambda,\mathcal{G}_{n})}{N_{+}(\lambda,\mathcal{G}_{p})}\), i.e., the ratio between the number of instances in \(\lambda\) that match a negative and positive gold seed, respectively; see Figure 3, line (i). If this ratio is close to zero, then likely false positive extractions are few compared to likely true positive extractions. For the simple version of the algorithm (for which we set \(w_{n}=1\), \(w_{u}=0\)), this results in \({\rm cnf}(\lambda,\mathcal{G})\) being close to 1 and the reliability measure it not discounted. On the other hand, if \(\frac{N_{+}(\lambda,\mathcal{G}_{n})}{N_{+}(\lambda,\mathcal{G}_{p})}\) is larger, meaning that the relative number of likely false positive extractions is high, then \({\rm cnf}(\lambda,\mathcal{G})\) shrinks towards 0, resulting in progressive discounting of \({\rm cnf}(\lambda,\mathcal{G})\) and leading to _non-noisy-low-confidence_ extractor, particularly for a reliable \(\lambda\). Due to lack of labeled data, the scoring mechanism cannot distinguish between noisy and non-noisy extractors. Therefore, an extractor is judged by its ability to extract more positive and less negative extractions. Note that we carefully designed this precision component to give good assessments while at the same time making maximum use of the available seeds. The reliability statistics are computed on \(\lambda\), i.e., on hop-2 instances (not on hop-3 instances). The ratio \(\frac{N_{+}(\lambda,\mathcal{G}_{n})}{N_{+}(\lambda,\mathcal{G}_{p})}\) is computed on instances that directly match a gold seed - this is the most reliable information we have available.

After all instances have been checked (line 8) and (if they passed muster) added to the cache (line 9), the inner loop ends and the cache is merged into the yield (line 10). Then a new loop (lines 03-10) of hop-1, hop-2 and hop-3 extensions and cluster reliability tests starts.

Thus, the algorithm consists of \(k_{\text{it}}\) iterations. There is a tradeoff here between \(\tau_{\text{sim}}\) and \(k_{\text{it}}\). We will give two extreme examples, assuming that we want to extract a fixed number of \(m\) instances where \(m\) is given. We can achieve this goal either by setting \(k_{\text{it}}\)=1 and choosing a small \(\tau_{\text{sim}}\), which will result in very large hops. Or we can achieve this goal by setting \(\tau_{\text{sim}}\) to a large value and running the algorithm for a larger number of \(k_{\text{it}}\). The flexibility that the two hyperparameters \(k_{\text{it}}\) and \(\tau_{\text{sim}}\) afford is important for good performance.

Figure 3: BREX configurations





### BREE, BRET and Brej

The main contribution of this paper is that we propose, as an alternative to entity-pair-centered BREE (Batista et al., 2015), template-centered BRET as well as BREJ (Figure 1), an instantiation of BREX that can take advantage of both entity pairs and templates. The differences and advantages of BREJ over BREE and BRET are:

**(1) Disjunctive Matching of Instances:** The first difference is realized in how the three algorithms match instances with seeds (line 05 in Figure 3). BREE checks whether the entity pair of an instance is one of the entity pair seeds, BRET checks whether the template of an instance is one of the template seeds and BREJ checks whether the disjunction of the two is true. The disjunction facilitates a higher hit rate in matching instances with seeds. The introduction of a few handcrafted templates along with seed entity pairs allows BREJ to leverage discriminative patterns and learn similar ones via distributional semantics. In Figure 1, the joint approach results in _hybrid_ extractors \(\Lambda\) that contain instances due to seed occurrences \(\Theta\) of both entity pairs and templates.

**(2) Hybrid Augmentation of Seeds:** On line 09 in Figure 3, we see that the bootstrapping step is defined in a straightforward fashion: the entity pair of an instance is added for BREE, the template for BRET and both for BREJ. Figure 1 demonstrates the hybrid augmentation of seeds via _red_ and _blue_ rings of _output instances_.

**(3) Scaling Up Positives in Extractors:** As discussed in section 2.2, a good measure of the quality of an extractor is crucial and \(N_{+}\), the number of instances in an extractor \(\lambda\) that match a seed, is an important component of that. For BREE and BRET, the definition follows directly from the fact that these are entity-pair and template-centered instantiations of BREX, respectively. However, the disjunctive matching of instances for an extractor with entity pair and template seeds in BREJ (Figure 3 line "(i)" ) boosts the likelihood of finding positive instances. In Figure 5, we demonstrate computing the count of positive instances

Figure 4: Similarity measures. These definitions for instances equally apply to templates since the definitions only depend on the “template part” of an instance, i.e., its vectors. (value is 0 if types are different)

Figure 6: An illustration of scaling positive extractions and computing confidence for a non-noisy extractor generated for _acquired_ relation. The dashed rectangular box represents an extractor \(\lambda\), where \(\lambda\) (BREJ) is _hybrid_ with 6 instances. Text segments matched with seed template are shown in italics. Unknowns (bold in black) are considered as negatives. \(\mathcal{G}_{cache}\) is a set of output instances where \(\tau_{cnf}=0.70\).



\(N_{+}(\lambda,\mathcal{G})\) for an extractor \(\lambda\) within the three systems. Observe that an instance \(i\) in \(\lambda\) can scale its \(N_{+}(\lambda,\mathcal{G})\) by a factor of maximum 2 in BREJ if \(i\) is matched in both entity pair and template seeds. The reliability \(\text{cnf}(\lambda,\mathcal{G})\) (Eq. 11) of an extractor \(\lambda\) is based on the ratio \(\frac{N_{+}(\lambda,\mathcal{G}_{n})}{N_{+}(\lambda,\mathcal{G}_{p})}\), therefore suggesting that the scaling boosts its confidence.

In Figure 6, we demonstrate with an example how the joint bootstrapping scales up the positive instances for a _non-noisy_ extractor \(\lambda\), resulting in \(\lambda_{NNHC}\) for BREJ compared to \(\lambda_{NNLC}\) in BREE.

Due to unlabeled data, the instances not matching in seeds are considered either to be ignored/unknown \(N_{0}\) or negatives in the confidence measure (Eq. 11). The former leads to high confidences for noisy extractors by assigning high scores, the latter to low confidences for non-noisy extractors by penalizing them. For a simple version of the algorithm in the illustration, we consider them as negatives and set \(w_{n}=1\). Figure 6 shows the three extractors (\(\lambda\)) generated and their confidence scores in BREE, BRET and BREJ. Observe that the scaling up of positives in BREJ due to BRET extractions (without \(w_{n}\)) discounts \(\text{cnf}(\lambda,\mathcal{G})\) relatively lower than BREE. The discounting results in \(\lambda_{NNHC}\) in BreJJ and \(\lambda_{NNLC}\) in BREE. The discounting in BREJ is adapted for _non-noisy_ extractors facilitated by BRET in generating mostly non-noisy extractors due to stringent checks (Figure 3, line "(i)" and 05). Intuitively, the intermixing of non-noisy extractors (i.e., _hybrid_) promotes the scaling and boosts recall.

### Similarity Measures

The before (\(\vec{v}_{-1}\)) and after (\(\vec{v}_{1}\)) contexts around the entities are highly sparse due to large variation in the syntax of how relations are expressed. SnowBall, DIPRE and BREE assumed that the between (\(\vec{v}_{0}\)) context mostly defines the syntactic expression for a relation and used weighted mechanism on the three contextual similarities in pairs, sim\({}_{match}\) (Figure 4). They assigned higher weights to the similarity in between (\(p=0\)) contexts, that resulted in lower recall. We introduce attentive (max) similarity across all contexts (for example, \(\vec{v}_{-1}(i)\vec{v}_{0}(j)\)) to automatically capture the large variation in the syntax of how relations are expressed, without using any weights. We investigate asymmetric (Eq 13) and symmetric (Eq 14 and 15) similarity measures, and name them as _cross-context attentive_ (sim\({}_{cc}\)) similarity.

## 3 Evaluation

### Dataset and Experimental Setup

We re-run BREE Batista et al. (2015) for **baseline** with a set of 5.5 million news articles from AFP and APW Parker et al. (2011). We use processed dataset of 1.2 million sentences (released by BREE) containing at least two entities linked to FreebaseEasy Bast et al. (2014). We extract four relationships: _acquired_ (ORG-ORG), _founderof_ (ORG-PER), _headquartered_ (ORG-LOC) and _affiliation_ (ORG-PER) for Organization (ORG), Person (PER) and Location (LOC) entity types. We bootstrap relations in BREE, BRET and BREJ, each with 4 similarity measures using seed entity

\begin{table}
\begin{tabular}{c|c|c} _Parameter_ & _Description/Search_ & _Optimal_ \\ \hline \(\mathrm{F}_{u-1}\) & maximum number of tokens in before context & 2 \\ \(\mathrm{F}_{0}[]\) & maximum number of tokens in between context & 6 \\ \(\mathrm{F}_{1}[]\) & maximum number of tokens in after context & 2 \\ \(r_{min}\) & similarity threshold [0.6, 0.7, 0.8] & 0.7 \\ \(r_{ref}\) & instance confidence thresholds [0.6, 0.7, 0.8] & 0.7 \\ \(w_{n}\) & weights to negativeFunctions [0.0, 0.5, 1.0, 2.0] & 0.5 \\ \(w_{x}\) & weights to unknown extractions [0.0001, 0.90001] & 0.0001 \\ \(k_{t}\) & number of bootstrapping epochs & 3 \\ \(dimmab\) & dimension of embedding vector, \(V\) & 300 \\ \(PMI\) & PMH Hershold in evaluation & 0.5 \\ _Entity Pairs_ & Ordered Pairs (\(OP\)) or Bisets (\(BS\)) & \(OP\) \\ \end{tabular}
\end{table}
Table 4: Hyperparameters in BREE, BRET and BREJ

\begin{table}
\begin{tabular}{l|l l} \multicolumn{2}{l}{_Relationship_} & \multicolumn{2}{c}{_Seed Entity Pairs_} \\ \hline acquired & \{Adias;Rebeok\},(Google:DoubleClick), & \{[X] acquire Y\},(|X| acquisition Y\},(|X|\) buy Y\},\\  & \{Windes;Warrington\},(|Heuker:Packard:Compaq) & \{[X] takeover Y\},(|X|\) merger with Y\} \\ \hline founder-of & \{CNN;Ted Turner\},(Facebook:Mark Zuckerberg), & \{[X] founded by YY\},(|X|\) co-founder \{Y\},(|X|\) started by \{Y\}, \\  & Microsoft;Paul Allen\},(Amazon:Zeff Bezos), & \{[X] founder of \{Y\},(|X|\) owner of \{Y\}\} \\ \hline headquartered & \{Nkaia;Expoo\},(Pfizer:New York), & \{[X] based in YY\},(|X|\) headquarters in YY\},(|X|\) head office in YY\}, \\  & \{United Nations;New York\},(NATO:Brussels), & \{[X] main office building in YY\},(|X|\) campus branch in YY\} \\ \hline affiliation & \{Google;Marissa Mayer\},\{Score;Ursula Burns\}, & \{[X] CEO Y\},(|X|\) resign from YY\},(|X|\) founded by YY\}, \\  & \{Microsoft;Steve Ballmer\},\{Microsoft;Bill Gates\}, & \{[X] worked for \{Y\},(|\)X chairman director [YY] \\ \end{tabular}
\end{table}
Table 2: Seed Entity Pairs and Templates for each relation. [X] and [Y] are slots for entity type tags.

\begin{table}
\begin{tabular}{c|c c|c} _Permaneter_ & _Description/Search_ & _Open_ \\ \hline \(\mathrm{F}_{u-1}\) & maximum number of tokens in before context & 2 \\ \(\mathrm{F}_{0}[]\) & maximum number of tokens in between context & 6 \\ \(\mathrm{F}_{1}[]\) & maximum number of tokens in after context & 2 \\ \(r_{min}\) & similarity threshold [0.6, 0.7, 0.8] & 0.7 \\ \(r_{ref}\) & instance confidence thresholds [0.6, 0.7, 0.8] & 0.7 \\ \(w_{n}\) & weights to negativeFunctions [0.0, 0.5, 1.0, 2.0] & 0.5 \\ \(w_{x}\) & weights to unknown extractions [0.0001, 0.90001] & 0.0001 \\ \(k_{t}\) & number of bootstrapping epochs & 3 \\ \(dimmab\) & dimension of embedding vector, \(V\) & 300 \\ \(PMI\) & PMH Hershold in evaluation & 0.5 \\ _Entity Pairs_ & Ordered Pairs (\(OP\)) or Bisets (\(BS\)) & \(OP\) \\ \end{tabular}
\end{table}
Table 3: Count of entity-type pairs in corpus 

pairs and templates (Table 2). See Tables 3, 4 and 5 for the count of candidates, hyperparameters and different configurations, respectively.

Our evaluation is based on Bronzi et al. (2012)'s framework to estimate precision and recall of large-scale RE systems using FreebaseEasy (Bast et al., 2014). Also following Bronzi et al. (2012), we use Pointwise Mutual Information (PMI) (Turney, 2001) to evaluate our system automatically, in addition to relying on an external knowledge base. We consider only extracted relationship instances with confidence scores \({\text{cnf}}(i,\Lambda,\mathcal{G})\) equal or above 0.5. We follow the same approach as BREE (Batista et al., 2015) to detect the correct order of entities in a relational triple, where we try to identify the presence of passive voice using part-of-speech (POS) tags and considering any form of the verb to be, followed by a verb in the past tense or past participle, and ending in the word 'by'. We use GloVe (Pennington et al., 2014) embeddings.

### Results and Comparison with baseline

Table 5 shows the experimental results in the three systems for the different relationships with _ordered_ entity pairs and similarity measures (sim\({}_{match}\), sim\({}_{cc}\)). Observe that BRET (configs) is _precision-oriented_ while BREJ (config\({}_{9}\)) _recall-oriented_ when compared to BREE (baseline). We see the number of output instances \(\#out\) are also higher in BREJ, therefore the higher recall. The BREJ system in the different similarity configurations outperforms the baseline BREE and BRET in terms of \(F1\) score. On an average for the four relations, BREJ in configurations config\({}_{9}\) and config\({}_{10}\) results in \(F1\) that is \(0.11\) (0.85 vs 0.74) and \(0.13\) (0.87 vs 0.74) better than the baseline BREE.

We discover that sim\({}_{cc}\) improves \(\#out\) and _recall_ over sim\({}_{\tt match}\) correspondingly in all three systems. Observe that sim\({}_{cc}\) performs better with BRET than BREE due to _non-noisy_ extractors in BRET. The results suggest an alternative to the weighting scheme in sim\({}_{\tt match}\) and therefore, the state-of-the-art (sim\({}_{cc}\)) performance with the 3 parameters (\(w_{-1}\), \(w_{0}\) and \(w_{1}\)) ignored in bootstrap

\begin{table}
\begin{tabular}{c c|r r r|r r|r r r}  & \multicolumn{1}{l|}{\(\tau\)} & \multicolumn{1}{l|}{\(\#out\)} & \multicolumn{1}{l}{\(P\)} & \multicolumn{1}{l}{\(R\)} & \multicolumn{1}{l|}{\(F1\)} \\ \cline{3-11} \multirow{6}{*}{**chair**} & \(\tau\) & \(\#out\) & \(P\) & \(R\) & \(F1\) & \(\tau\) & \(\#out\) & \(P\) & \(R\) & \(F1\) \\ \cline{2-11}  & .60 & 1785 & .91 & .39 & .55 & .70 & 1222 & .94 & .31 & .47 \\  & .80 & 868 & .95 & .25 & .39 & .90 & 626 & .96 & .19 & .32 \\ \hline \multirow{6}{*}{**chair**} & .60 & 2995 & .89 & .51 & .65 & .70 & 1859 & .90 & .40 & .55 \\  & .80 & 1312 & .91 & .32 & .47 & .90 & .75 & .94 & .22 & .35 \\ \cline{1-1} \cline{2-11}  & .60 & 18271 & .81 & .85 & .83 & .70 & 14900 & .84 & .83 & .83 \\ \cline{1-1}  & .80 & 8896 & .88 & .75 & .81 & .90 & .5158 & .93 & .65 & .77 \\ \end{tabular}
\end{table}
Table 6: Iterations (\(k_{it}\)) Vs Scores with thresholds (\(\tau\)) for relation _acquired_ in BREJ. \(\tau\) refers to \(\tau_{sim}\) and \(\tau_{cnf}\)

\begin{table}
\begin{tabular}{c l|r r r r|r r r r|r r r r r}  & \multicolumn{1}{l|}{_Relinships_} & \multicolumn{1}{l|}{\(\#out\)} & \multicolumn{1}{l}{\(P\)} & \multicolumn{1}{l}{\(R\)} & \multicolumn{1}{l}{\(F1\)} & \multicolumn{1}{l}{\(\#out\)} & \multicolumn{1}{l}{\(P\)} & \multicolumn{1}{l}{\(R\)} & \multicolumn{1}{l}{\(F1\)} & \multicolumn{1}{l}{\(\#out\)} & \multicolumn{1}{l}{\(P\)} & \multicolumn{1}{l}{\(R\)} & \multicolumn{1}{l}{\(F1\)} \\ \hline \multirow{6}{*}{**chair**} & \multirow{2}{*}{**baseline**} & \multicolumn{1}{l}{**baseline**} & \multicolumn{1}{l}{BBEE+sim\({}_{match}\)} & \multicolumn{1}{l}{**config\({}_{9}\)**} & \multicolumn{1}{l}{BBEE+sim\({}^{agn}_{cc}\)} & \multicolumn{1}{l}{**config\({}_{9}\)**} & \multicolumn{1}{l}{BBEE+sim\({}^{syn}_{min}\)} & \multicolumn{1}{l}{**config\({}_{9}\)**} & \multicolumn{1}{l}{BBEE+sim\({}^{syn}_{max}\)} & \multicolumn{1}{l}{**config\({}_{9}\)**} & \multicolumn{1}{l}{BBEE+sim\({}^{syn}_{max}\)} \\ \cline{2-11}  & acquired & 2687 & 0.88 & 0.48 & 0.62 & 5771 & 0.88 & 0.66 & 0.76 & 3471 & 0.88 & 0.55 & 0.68 & 3279 & 0.88 & 0.53 & 0.66 \\  & founder-of & 628 & 0.98 & 0.70 & 0.82 & 9553 & 0.86 & 0.95 & 0.89 & 1532 & 0.94 & 0.84 & 0.89 & 1182 & 0.95 & 0.81 & 0.87 \\  & headquartered & 16786 & 0.62 & 0.80 & 0.69 & 2129 & 0.66 & 0.85 & 0.74 & 17301 & 0.70 & 0.83 & 0.76 & 9842 & 0.72 & 0.74 & 0.73 \\  & affiliation & 20948 & 0.99 & 0.73 & 0.84 & 27424 & 0.97 & 0.78 & 0.87 & 36797 & 0.95 & 0.82 & 0.88 & 28416 & 0.97 & 0.78 & 0.87 \\ \cline{2-11}  & **avg** & 10262 & 0.86 & 0.68 & 0.74 & 16011 & 0.84 & 0.81 & 0.82 & 14475 & 0.87 & 0.76 & 0.80 & 10680 & 0.88 & 0.72 & 0.78 \\ \hline \multirow{6}{*}{**chair**} & \multirow{2}{*}{**aug**} & \multicolumn{1}{l}{BBEE+sim\({}^{syn}_{match}\)} & \multicolumn{1}{l}{**config\({}_{9}\)**} & \multicolumn{1}{l}{BBET+sim\({}^{syn}_{rec}\)} & \multicolumn{1}{l}{**config\({}_{9}\)**} & \multicolumn{1}{l}{BBET+sim\({}^{syn}_{max}\)} & \multicolumn{1}{l}{**config\({}_{9}\)**} & \multicolumn{1}{l}{BBET+sim\({}^{syn}_{max}\)} & \multicolumn{1}{l}{**config\({}_{9}\)**} & \multicolumn{1}{l}{BBET+sim\({}^{syn}_{max}\)} \\ \cline{2-11}  & acquired & 4206 & 0.99 & 0.62 & 0.76 & 1566 & 0.90 & 0.85 & 0.87 & 18273 & 0.87 & 0.86 & 0.87 & 14319 & 0.92 & 0.84 & 0.87 \\  & founder-of & 920 & 0.97 & 0.77 & 0.86 & 43554 & 0.81 & 0.98 & 0.89 & 41978 & 0.81 & 0.99 & 0.89 & 46453 & 0.81 & 0.99 & 0.89 \\  & headquartered & 3065 & 0.98 & 0.55 & 0.72 & 39267 & 0.68 & 0.92 & 0.78 & 36374 & 0.71 & 0.91 & 0.80 & 56815 & 0.69 & 0.94 & 0.

 

ping. Observe that \({\text{sim}}_{cc}^{asym}\) gives higher recall than the two symmetric similarity measures.

Table 6 shows the performance of BREJ in different iterations trained with different similarity \(\tau_{sim}\) and confidence \(\tau_{cnf}\) thresholds. Table 7 shows a comparative analysis of the three systems, where we consider and evaluate the extracted relationship instances at different confidence scores.

### Disjunctive Seed Matching of Instances

As discussed in section 2.3, BREJ facilitates disjunctive matching of instances (line 05 Figure 3) with seed entity pairs and templates. Table 8 shows \(\#hit\) in the three systems, where the higher values of \(\#hit\) in BREJ conform to the desired property. Observe that some instances in BREJ are found to be matched in both the seed types.

### Deep Dive into Attributes of Extractors

We analyze the extractors \(\Lambda\) generated in BREE, BRET and BREJ for the 4 relations to demonstrate the impact of joint bootstrapping. Table 9 shows the attributes of \(\Lambda\). We manually annotate the extractors as _noisy_ and _non-noisy_. We compute \(ANNLC\) and the lower values in BREJ compared to BREE suggest fewer non-noisy extractors with lower confidence in BREJ due to the scaled confidence scores. _ANNE_ (higher), _ANNLC_ (lower), _AP_ (higher) and _AN_ (lower) collectively indicate that BRET mostly generates _NNHC_ extractors. _AP_ and _AN_ indicate an average of \(N_{+}(\lambda,\mathcal{G}_{l})\) (line " (i)" Figure 3) for positive and negative seeds, respectively for \(\lambda\in\Lambda\) in the three systems. Observe the impact of scaling positive extractions (_AP_) in BREJ that shrink \(\frac{N_{+}(\lambda,\mathcal{G}_{n})}{N_{+}(\lambda,\mathcal{G}_{p})}\) i.e., _ANP_. It facilitates \(\lambda_{NNLC}\) to boost its confidence, i.e., \(\lambda_{NNHC}\) in BREJ suggested by _AES_ that results in higher \(\#out\) and recall (Table 5, BREJ).

### Weighting Negatives Vs Scaling Positives

As discussed, Table 5 shows the performance of BREE, BRET and BREJ with the parameter \(w_{n}=0.5\) in computing extractors' confidence \(\text{cnf}(\lambda,\mathcal{G})\)(Eq. 11). In other words, config\({}_{9}\) (Table 5) is combination of both weighted negative and scaled positive extractions. However, we also investigate ignoring \(w_{n}(=1.0)\) in order to demonstrate the capability of BREJ with only scaling positives and without weighting negatives. In Table 10, observe that BREJ outperformed both BREE and BRET for all the relationships due to higher \(\#out\) and recall. In addition, BREJ scores are comparable to config\({}_{9}\) (Table 5) suggesting that the scaling in BREJ is capable enough to remove the parameter \(w_{n}\). However, the combination of both weighting negatives and scaling positives results in the state-of-the-art performance.

### Qualitative Inspection of Extractors

Table 11 lists some of the non-noisy extractors (simplified) learned in different configurations to illustrate boosting extractor confidence \(\text{cnf}(\lambda,\mathcal{G})\). Since, an extractor \(\lambda\) is a cluster of instances, therefore to simplify, we show one instance of the \(\text{cnf}(\lambda,\mathcal{G})\)(Eq. 11). The first instance of BREJ is shown in Fig. 10. As can be seen from the figure, the \(\text{cnf}(\lambda,\mathcal{G})\)(Eq. 11) has no effect on the classification accuracy. This means that the classifier performs better than the other methods. For example, the \(\text{cnf}(\lambda,\mathcal{G})\)(Eq. 11) is not able to distinguish between the two classes of BREJ. Therefore, it is important to note that the classifier performs better than the other methods.

## 3 Results

In this section, we present our results on the dataset using the proposed method. Our experiments show that the proposed approach outperforms the existing 

stance (mostly populated) from every \(\lambda\). Each cell in Table 11 represents either a simplified representation of \(\lambda\) or its confidence. We demonstrate how the confidence score of a non-noisy extractor in BREE (config\({}_{1}\)) is increased in BREJ (config\({}_{9}\) and config\({}_{10}\)). For instance, for the relation _acquired_, an extractor {[_X__acquiring__Y_]} is generated by BREE, BEET and BREJ; however, its confidence is boosted from \(0.75\) in BREE (config\({}_{1}\)) to \(0.95\) in BREJ (config\({}_{9}\)). Observe that BRET generates high confidence extractors. We also show extractors (marked by \(\dagger\)) learned by BREJ with sim\({}_{cc}\) (config\({}_{10}\)) but not by config\({}_{1}\), config\({}_{8}\) and config\({}_{9}\).

### Entity Pairs: Ordered Vs Bi-Set

In Table 5, we use ordered pairs of typed entities. Additionally, we also investigate using entity sets and observe improved recall due to higher \(\#out\) in both BREE and BREJ, comparing correspondingly Table 12 and 5 (_baseline_ and config\({}_{9}\)).

## 4 Conclusion

We have proposed a Joint Bootstrapping Machine for relation extraction (BREJ) that takes advantage of both entity-pair-centered and template-centered approaches. We have demonstrated that the joint approach scales up positive instances that boosts the confidence of NNLC extractors and improves recall. The experiments showed that the cross-context similarity measures improved recall and suggest removing in total four parameters.

## Acknowledgments

We thank our colleagues Bernt Andrassy, Mark Buckley, Stefan Langer, Ulli Waltinger and Usama Yaseen, and anonymous reviewers for their review comments. This research was supported by Bundeswirtschaftsministerium (bmwi.de), grant 01MD15010A (Smart Data Web) at Siemens AGCT Machine Intelligence, Munich Germany.

\begin{table}
\begin{tabular}{r|c c c|c c c c} \multicolumn{3}{l}{**Relationships**} & \multicolumn{3}{c|}{**BREE + sim\({}_{match}\)**} & \multicolumn{3}{c}{**BREJ + sim\({}_{match}\)**} \\ \cline{2-6}  & \(\#out\) & \(P\) & \(R\) & \(F1\) & \(\#out\) & \(P\) & \(R\) & \(F1\) \\ \hline acquired & 2786 & .90 & .50 & .64 & 21733 & .80 & .87 & .83 \\ founder-of & 543 & 1.0 & .67 & .80 & 31890 & .80 & .99 & .89 \\ headquarterred & 16832 & .62 & .81 & .70 & 52286 & .64 & .94 & .76 \\ affiliation & 21812 & .99 & .74 & .85 & 42601 & .96 & .85 & .90 \\ \hline \end{tabular}
\end{table}
Table 12: BREX+sim\({}_{match}\):Scores with entity _bisets_ 



## References

* Agichtein and Gravano (2000) Eugene Agichtein and Luis Gravano. 2000. Snowball: Extracting relations from large plain-text collections. In _Proceedings of the 15th ACM conference on Digital libraries_. Association for Computing Machinery, Washington, DC USA, pages 85-94.
* Angeli et al. (2015) Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D Manning. 2015. Leveraging linguistic structure for open domain information extraction. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_. Association for Computational Linguistics, Beijing, China, volume 1, pages 344-354.
* Bast et al. (2014) Hannah Bast, Florian Baurle, Bjorn Buchhold, and Elmar Haussmann. 2014. Easy access to the freebase dataset. In _Proceedings of the 23rd International Conference on World Wide Web_. Association for Computing Machinery, Seoul, Republic of Korea, pages 95-98.
* Batista et al. (2015) David S. Batista, Bruno Martins, and Mario J. Silva. 2015. Semi-supervised bootstrapping of relationship extractors with distributional semantics. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Lisbon, Portugal, pages 499-504.
* Brin (1998) Sergey Brin. 1998. Extracting patterns and relations from the world wide web. In _International Workshop on The World Wide Web and Databases_. Springer, Valencia, Spain, pages 172-183.
* Bronzi et al. (2012) Mirko Bronzi, Zhaochen Guo, Filipe Mesquita, Denislon Barbosa, and Paolo Merialdo. 2012. Automatic evaluation of relation extraction systems on large-scale. In _Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX)_. Association for Computational Linguistics, Montreal, Canada, pages 19-24.
* Carlson et al. (2010) Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R. Hruschka Jr., and Tom M. Mitchell. 2010. Toward an architecture for never-ending language learning. In _Proceedings of the 24th National Conference on Artificial Intelligence (AAAI)_. Atlanta, Georgia USA, volume 5, page 3.
* Chiticariu et al. (2013) Laura Chiticariu, Yunyao Li, and Frederick R. Reiss. 2013. Rule-based information extraction is dead! long live rule-based information extraction systems! In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Seattle, Washington USA, pages 827-832.
* Fader et al. (2011) Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Edinburgh, Scotland UK, pages 1535-1545.
* Gupta et al. (2015) Pankaj Gupta, Thomas Runkler, Heike Adel, Bernt Andrassy, Hans-Georg Zimmermann, and Hinrich Schutze. 2015. Deep learning methods for the extraction of relations in natural language text. Technical report, Technical University of Munich, Germany.
* Gupta et al. (2016) Pankaj Gupta, Hinrich Schutze, and Bernt Andrassy. 2016. Table filling multi-task recurrent neural network for joint entity and relation extraction. In _Proceedings of the 26th International Conference on Computational Linguistics: Technical Papers_. Osaka, Japan, pages 2537-2547.
* Gupta et al. (2014) Sonal Gupta, Diana L. MacLean, Jeffrey Heer, and Christopher D. Manning. 2014. Induced lexicov syntactic patterns improve information extraction from online medical forums. _Journal of the American Medical Informatics Association_ 21(5):902-909.
* Gupta and Manning (2014) Sonal Gupta and Christopher Manning. 2014. Improved pattern learning for bootstrapped entity extraction. In _Proceedings of the 18th Conference on Computational Natural Language Learning (CoNLL)_. Association for Computational Linguistics, Baltimore, Maryland USA, pages 98-108.
* Hearst (1992) Marti A Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In _Proceedings of the 15th International Conference on Computational Linguistics_. Nantes, France, pages 539-545.
* Lin et al. (2003) Winston Lin, Roman Yangarber, and Ralph Grishman. 2003. Bootstrapped learning of semantic classes from positive and negative examples. In _Proceedings of ICML 2003 Workshop on The Continuum from Labeled to Unlabeled Data in Machine Learning and Data Mining_. Washington, DC USA, page 21.
* Mansam et al. (2012) Musam, Michael Schmitz, Robert Bart, Stephen Soderland, and Oren Etzioni. 2012. Open language learning for information extraction. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_. Association for Computational Linguistics, Jeju Island, Korea, pages 523-534.
* Mesquita et al. (2013) Filipe Mesquita, Jordan Schmidek, and Denilson Barbosa. 2013. Effectiveness and efficiency of open relation extraction. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Seattle, Washington USA, pages 447-457.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In _Proceedings of the Workshop at the International Conference on Learning Representations_. ICLR, Scottsdale, Arizona USA

Thien Huu Nguyen and Ralph Grishman. 2015. Relation extraction: Perspective from convolutional neural networks. In _Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing_. Association for Computational Linguistics, Denver, Colorado USA, pages 39-48.
* Parker et al. (2011) Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English gigaword. _Linguistic Data Consortium_ .
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Doha, Qatar, pages 1532-1543.
* Riloff (1996) Ellen Riloff. 1996. Automatically generating extraction patterns from untagged text. In _Proceedings of the 13th National Conference on Artificial Intelligence (AAAI)_. Portland, Oregon USA, pages 1044-1049.
* Turney (2001) Peter D. Turney. 2001. Mining the web for synonyms: Pmi-ir versus lsa on toefl. In _Proceedings of the 12th European Conference on Machine Learning_. Springer, Freiburg, Germany, pages 491-502.
* Vu et al. (2016a) Ngoc Thang Vu, Heike Adel, Pankaj Gupta, and Hinrich Schutze. 2016a. Combining recurrent and convolutional neural networks for relation classification. In _Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_. Association for Computational Linguistics, San Diego, California USA, pages 534-539.
* Vu et al. (2016b) Ngoc Thang Vu, Pankaj Gupta, Heike Adel, and Hinrich Schutze. 2016b. Bi-directional recurrent neural network with ranking loss for spoken language understanding. In _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, Shanghai, China, pages 6060-6064.