Relation extraction systems populate knowledge bases with facts from an unstructured text corpus. When the type of facts (relations) are predefined, one can use crowdsourcing BIBREF0 or distant supervision BIBREF1 to collect examples and train an extraction model for each relation type. However, these approaches are incapable of extracting relations that were not specified in advance and observed during training. In this paper, we propose an alternative approach for relation extraction, which can potentially extract facts of new types that were neither specified nor observed a priori.We show that it is possible to reduce relation extraction to the problem of answering simple reading comprehension questions. We map each relation type $R(x,y)$ to at least one parametrized natural-language question $q_x$ whose answer is $y$ . For example, the relation $educated\_at(x,y)$ can be mapped to “Where did $x$ study?” and “Which university did $x$ graduate from?”. Given a particular entity $x$ (“Turing”) and a text that mentions $x$ (“Turing obtained his PhD from Princeton”), a non-null answer to any of these questions (“Princeton”) asserts the fact and also fills the slot $y$ . Figure 1 illustrates a few more examples.This reduction enables new ways of framing the learning problem. In particular, it allows us to perform zero-shot learning: define new relations “on the fly”, after the model has already been trained. More specifically, the zero-shot scenario assumes access to labeled data for $N$ relation types. This data is used to train a reading comprehension model through our reduction. However, at test time, we are asked about a previously unseen relation type $R_{N+1}$ . Rather than providing labeled data for the new relation, we simply list questions that define the relation's slot values. Assuming we learned a good reading comprehension model, the correct values should be extracted.Our zero-shot setup includes innovations both in data and models. We use distant supervision for a relatively large number of relations (120) from Wikidata BIBREF2 , which are easily gathered in practice via the WikiReading dataset BIBREF3 . We also introduce a crowdsourcing approach for gathering and verifying the questions for each relation. This process produced about 10 questions per relation on average, yielding a dataset of over 30,000,000 question-sentence-answer examples in total. Because questions are paired with relation types, not instances, this overall procedure has very modest costs.The key modeling challenge is that most existing reading-comprehension problem formulations assume the answer to the question is always present in the given text. However, for relation extraction, this premise does not hold, and the model needs to reliably determine when a question is not answerable. We show that a recent state-of-the-art neural approach for reading comprehension BIBREF4 can be directly extended to model answerability and trained on our new dataset. This modeling approach is another advantage of our reduction: as machine reading models improve with time, so should our ability to extract relations.Experiments demonstrate that our approach generalizes to new paraphrases of questions from the training set, while incurring only a minor loss in performance (4% relative F1 reduction). Furthermore, translating relation extraction to the realm of reading comprehension allows us to extract a significant portion of previously unseen relations, from virtually zero to an F1 of 41%. Our analysis suggests that our model is able to generalize to these cases by learning typing information that occurs across many relations (e.g. the answer to “Where” is a location), as well as detecting relation paraphrases to a certain extent. We also find that there are many feasible cases that our model does not quite master, providing an interesting challenge for future work.We are interested in a particularly harsh zero-shot learning scenario: given labeled examples for $N$ relation types during training, extract relations of a new type $R_{N+1}$ at test time. The only information we have about $R_{N+1}$ are parametrized questions.This setting differs from prior art in relation extraction. Bronstein2015 explore a similar zero-shot setting for event-trigger identification, in which $R_{N+1}$ is specified by a set of trigger words at test time. They generalize by measuring the similarity between potential triggers and the given seed set using unsupervised methods. We focus instead on slot filling, where questions are more suitable descriptions than trigger words.Open information extraction (open IE) BIBREF5 is a schemaless approach for extracting facts from text. While open IE systems need no relation-specific training data, they often treat different phrasings as different relations. In this work, we hope to extract a canonical slot value independent of how the original text is phrased.Universal schema BIBREF6 represents open IE extractions and knowledge-base facts in a single matrix, whose rows are entity pairs and columns are relations. The redundant schema (each knowledge-base relation may overlap with multiple natural-language relations) enables knowledge-base population via matrix completion techniques. Verga2017 predict facts for entity pairs that were not observed in the original matrix; this is equivalent to extracting seen relation types with unseen entities (see Section "Unseen Entities" ). Rocktaschel2015 and Demeester2016 use inference rules to predict hidden knowledge-base relations from observed natural-language relations. This setting is akin to generalizing across different manifestations of the same relation (see Section "Unseen Question Templates" ) since a natural-language description of each target relation appears in the training data. Moreover, the information about the unseen relations is a set of explicit inference rules, as opposed to implicit natural-language questions.Our zero-shot scenario, in which no manifestation of the test relation is observed during training, is substantially more challenging (see Section "Unseen Relations" ). In universal-schema terminology, we add a new empty column (the target knowledge-base relation), plus a few new columns with a single entry each (reflecting the textual relations in the sentence). These columns share no entities with existing columns, making the rest of the matrix irrelevant. To fill the empty column from the others, we match their descriptions. Toutanova2015 proposed a similar approach that decomposes natural-language relations and computes their similarity in a universal schema setting; however, they did not extend their method to knowledge-base relations, nor did they attempt to recover out-of-schema relations as we do.We consider the slot-filling challenge in relation extraction, in which we are given a knowledge-base relation $R$ , an entity $e$ , and a sentence $s$ . For example, consider the relation $occupation$ , the entity “Steve Jobs”, and the sentence “Steve Jobs was an American businessman, inventor, and industrial designer”. Our goal is to find a set of text spans $A$ in $s$ for which $R(e,a)$ holds for each $a \in A$ . In our example, $A=\lbrace \textnormal {businessman},\textnormal {inventor}, \textnormal {industrial designer}\rbrace $ . The empty set is also a valid answer ( $A = \emptyset $ ) when $e$0 does not contain any phrase that satisfies $e$1 . We observe that given a natural-language question $e$2 that expresses $e$3 (e.g. “What did Steve Jobs do for a living?”), solving the reading comprehension problem of answering $e$4 from $e$5 is equivalent to solving the slot-filling challenge.The challenge now becomes one of querification: translating $R(e,?)$ into $q$ . Rather than querify $R(e,?)$ for every entity $e$ , we propose a method of querifying the relation $R$ . We treat $e$ as a variable $x$ , querify the parametrized query $R(x,?)$ (e.g. $occupation(x,?)$ ) as a question template $q_x$ (“What did $q$0 do for a living?”), and then instantiate this template with the relevant entities, creating a tailored natural-language question for each entity $q$1 (“What did Steve Jobs do for a living?”). This process, schema querification, is by an order of magnitude more efficient than querifying individual instances because annotating a relation type automatically annotates all of its instances.Applying schema querification to $N$ relations from a pre-existing relation-extraction dataset converts it into a reading-comprehension dataset. We then use this dataset to train a reading-comprehension model, which given a sentence $s$ and a question $q$ returns a set of text spans $A$ within $s$ that answer $q$ (to the best of its ability).In the zero-shot scenario, we are given a new relation $R_{N+1}(x,y)$ at test-time, which was neither specified nor observed beforehand. For example, the $deciphered(x,y)$ relation, as in “Turing and colleagues came up with a method for efficiently deciphering the Enigma”, is too domain-specific to exist in common knowledge-bases. We then querify $R_{N+1}(x,y)$ into $q_x$ (“Which code did $x$ break?”) or $q_y$ (“Who cracked $y$ ?”), and run our reading-comprehension model for each sentence in the document(s) of interest, while instantiating the question template with different entities that might participate in this relation. Each time the model returns a non-null answer $deciphered(x,y)$1 for a given question $deciphered(x,y)$2 , it extracts the relation $deciphered(x,y)$3 .Ultimately, all we need to do for a new relation is define our information need in the form of a question. Our approach provides a natural-language API for application developers who are interested in incorporating a relation-extraction component in their programs; no linguistic knowledge or pre-defined schema is needed. To implement our approach, we require two components: training data and a reading-comprehension model. In Section "Dataset" , we construct a large relation-extraction dataset and querify it using an efficient crowdsourcing procedure. We then adapt an existing state-of-the-art reading-comprehension model to suit our problem formulation (Section "Model" ).To collect reading-comprehension examples as in Figure 2 , we first gather labeled examples for the task of relation-slot filling. Slot-filling examples are similar to reading-comprehension examples, but contain a knowledge-base query $R(e,?)$ instead of a natural-language question; e.g. $spouse(\textnormal {Angela Merkel}, ?)$ instead of “Who is Angela Merkel married to?”. We collect many slot-filling examples via distant supervision, and then convert their queries into natural language.Given a sentence $s$ and a question $q$ , our algorithm either returns an answer span $a$ within $s$ , or indicates that there is no answer.The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset BIBREF8 , BIBREF12 , BIBREF13 , BIBREF14 . In SQuAD, every question is answerable from the text, which is why these models assume that there exists a correct answer span. Therefore, we modify an existing model in a way that allows it to decide whether an answer exists. We first give a high-level description of the original model, and then describe our modification.We start from the BiDAF model BIBREF4 , whose input is two sequences of words: a sentence $s$ and a question $q$ . The model predicts the start and end positions ${\bf y}^{start}, {\bf y}^{end}$ of the answer span in $s$ . BiDAF uses recurrent neural networks to encode contextual information within $s$ and $q$ alongside an attention mechanism to align parts of $q$ with $s$ and vice-versa.The outputs of the BiDAF model are the confidence scores of ${\bf y}^{start}$ and ${\bf y}^{end}$ , for each potential start and end. We denote these scores as ${\bf z}^{start}, {\bf z}^{end} \in \mathbb {R}^N$ , where $N$ is the number of words in the sentence $s$ . In other words, ${\bf z}^{start}_i$ indicates how likely the answer is to start at position $i$ of the sentence (the higher the more likely); similarly, ${\bf z}^{end}_i$ indicates how likely the answer is to end at that index. Assuming the answer exists, we can transform these confidence scores into pseudo-probability distributions ${\bf p}^{start}, {\bf p}^{end}$ via softmax. The probability of each $i$ -to- ${\bf y}^{end}$0 -span of the context can therefore be defined by: $$P(a = s_{i...j}) = {\bf p}^{start}_i {\bf p}^{end}_j$$   (Eq. 13) where ${\bf p}_i$ indicates the $i$ -th element of the vector ${\bf p}_i$ , i.e. the probability of the answer starting at $i$ . Seo:16 obtain the span with the highest probability during post-processing. To allow the model to signal that there is no answer, we concatenate a trainable bias $b$ to the end of both confidences score vectors ${\bf z}^{start}, {\bf z}^{end}$ . The new score vectors ${\tilde{\bf z}}^{start}, {\tilde{\bf z}}^{end} \in \mathbb {R}^{N+1}$ are defined as ${\tilde{\bf z}}^{start} = [{\bf z}^{start}; b]$ and similarly for ${\tilde{\bf z}}^{end}$ , where $[;]$ indicates row-wise concatenation. Hence, the last elements of $i$0 and $i$1 indicate the model's confidence that the answer has no start or end, respectively. We apply softmax to these augmented vectors to obtain pseudo-probability distributions, $i$2 . This means that the probability the model assigns to a null answer is: $$P(a = \emptyset ) = {\tilde{\bf p}}^{start}_{N+1} {\tilde{\bf p}}^{end}_{N+1}.$$   (Eq. 14) If $P(a = \emptyset )$ is higher than the probability of the best span, $\arg \max _{i,j \le N} P(a = s_{i...j})$ , then the model deems that the question cannot be answered from the sentence. Conceptually, adding the bias enables the model to be sensitive to the absolute values of the raw confidence scores ${\bf z}^{start}, {\bf z}^{end}$ . We are essentially setting and learning a threshold $b$ that decides whether the model is sufficiently confident of the best candidate answer span.While this threshold provides us with a dynamic per-example decision of whether the instance is answerable, we can also set a global confidence threshold $p_{min}$ ; if the best answer's confidence is below that threshold, we infer that there is no answer. In Section "Unseen Relations" we use this global threshold to get a broader picture of the model's performance.To understand how well our method can generalize to unseen data, we design experiments for unseen entities (Section "Unseen Entities" ), unseen question templates (Section "Unseen Question Templates" ), and unseen relations (Section "Unseen Relations" ).We show that our reading-comprehension approach works well in a typical relation-extraction setting by testing it on unseen entities and texts.We partitioned our dataset along entities in the question, and randomly clustered each entity into one of three groups: train, dev, or test. For instance, Alan Turing examples appear only in training, while Steve Jobs examples are exclusive to test. We then sampled 1,000,000 examples for train, 1,000 for dev, and 10,000 for test. This partition also ensures that the sentences at test time are different from those in train, since the sentences are gathered from each entity's Wikipedia article.Table 1 shows that our model generalizes well to new entities and texts, with little variance in performance between KB Relation, NL Relation, Multiple Templates, and Question Ensemble. Single Template performs significantly worse than these variants; we conjecture that simpler relation descriptions (KB Relation & NL Relation) allow for easier parameter tying across different examples, whereas learning from multiple questions allows the model to acquire important paraphrases. All variants of our model outperform off-the-shelf relation extraction systems (RNN Labeler and Miwa & Bansal) in this setting, demonstrating that reducing relation extraction to reading comprehension is indeed a viable approach for our Wikipedia slot-filling task. An analysis of 50 examples that Multiple Templates mispredicted shows that 36% of errors can be attributed to annotation errors (chiefly missing entries in Wikidata), and an additional 42% result from inaccurate span selection (e.g. “8 February 1985” instead of “1985”), for which our model is fully penalized. In total, only 18% of our sample were pure system errors, suggesting that our model is very close to the performance ceiling of this setting (slightly above 90% F1).We test our method's ability to generalize to new descriptions of the same relation, by holding out a question template for each relation during training.We created 10 folds of train/dev/test samples of the data, in which one question template for each relation was held out for the test set, and another for the development set. For instance, “What did $x$ do for a living?” may appear only in the training set, while “What is $x$ 's job?” is exclusive to the test set. Each split was stratified by sampling $N$ examples per question template ( $N=1000,10,50$ for train, dev, test, respectively). This process created 10 training sets of 966,000 examples with matching development and test sets of 940 and 4,700 examples each.We trained and tested Multiple Templates on each one of the folds, yielding performance on unseen templates. We then replicated the existing test sets and replaced the unseen question templates with templates from the training set, yielding performance on seen templates. Revisiting our example, we convert test-set occurrences of “What is $x$ 's job?” to “What did $x$ do for a living?”.Table 2 shows that our approach is able to generalize to unseen question templates. Our system's performance on unseen questions is nearly as strong as for previously observed templates (losing roughly 3.5 points in F1).We examine a pure zero-shot setting, where test-time relations are unobserved during training.We created 10 folds of train/dev/test samples, partitioned along relations: 84 relations for train, 12 dev, and 24 test. For example, when $educated\_at$ is allocated to test, no $educated\_at$ examples appear in train. Using stratified sampling of relations, we created 10 training sets of 840,000 examples each with matching dev and test sets of 600 and 12,000 examples per fold.Table 3 shows each system's performance; Figure 4 extends these results for variants of our model by applying a global threshold on the answers' confidence scores to generate precision/recall curves (see Section "Model" ). As expected, representing knowledge-base relations as indicators (KB Relation and Miwa & Bansal) is insufficient in a zero-shot setting; they must be interpreted as natural-language expressions to allow for some generalization. The difference between using a single question template (Single Template) and the relation's name (NL Relation) appears to be minor. However, training on a variety of question templates (Multiple Templates) substantially increases performance. We conjecture that multiple phrasings of the same relation allows our model to learn answer-type paraphrases that occur across many relations (see Section "Analysis" ). There is also some advantage to having multiple questions at test time (Question Ensemble).To understand how our method extracts unseen relations, we analyzed 100 random examples, of which 60 had answers in the sentence and 40 did not (negative examples).For negative examples, we checked whether a distractor – an incorrect answer of the correct answer type – appears in the sentence. For example, the question “Who is John McCain married to?” does not have an answer in “John McCain chose Sarah Palin as his running mate”, but “Sarah Palin” is of the correct answer type. We noticed that 14 negative examples (35%) contain distractors. When pairing these examples with the results from the unseen relations experiment in Section "Unseen Relations" , we found that our method answered 2/14 of the distractor examples incorrectly, compared to only 1/26 of the easier examples. It appears that while most of the negative examples are easy, a significant portion of them are not trivial.For positive examples, we observed that some instances can be solved by matching the relation in the sentence to that in the question, while others rely more on the answer's type. Moreover, we notice that each cue can be further categorized according to the type of information needed to detect it: (1) when part of the question appears verbatim in the text, (2) when the phrasing in the text deviates from the question in a way that is typical of other relations as well (e.g. syntactic variability), (3) when the phrasing in the text deviates from the question in a way that is unique to this relation (e.g. lexical variability). We name these categories verbatim, global, and specific, respectively. Figure 5 illustrates all the different types of cues we discuss in our analysis.We selected the most important cue for solving each instance. If there were two important cues, each one was counted as half. Table 4 shows their distribution. Type cues appear to be somewhat more dominant than relation cues (58% vs. 42%). Half of the cues are relation-specific, whereas global cues account for one third of the cases and verbatim cues for one sixth. This is an encouraging result, because we can potentially learn to accurately recognize verbatim and global cues from other relations. However, our method was only able to exploit these cues partially.We paired these examples with the results from the unseen relations experiment in Section "Unseen Relations" to see how well our method performs in each category. Table 5 shows the results for the Multiple Templates setting. On one hand, the model appears agnostic to whether the relation cue is verbatim, global, or specific, and is able to correctly answer these instances with similar accuracy (there is no clear trend due to the small sample size). For examples that rely on typing information, the trend is much clearer; our model is much better at detecting global type cues than specific ones.Based on these observations, we think that the primary sources of our model's ability to generalize to new relations are: global type detection, which is acquired from training on many different relations, and relation paraphrase detection (of all types), which probably relies on its pre-trained word embeddings.We showed that relation extraction can be reduced to a reading comprehension problem, allowing us to generalize to unseen relations that are defined on-the-fly in natural language. However, the problem of zero-shot relation extraction is far from solved, and poses an interesting challenge to both the information extraction and machine reading communities. As research into machine reading progresses, we may find that more tasks can benefit from a similar approach. To support future work in this avenue, we make our code and data publicly available.The research was supported in part by DARPA under the DEFT program (FA8750-13-2-0019), the ARO (W911NF-16-1-0121), the NSF (IIS-1252835, IIS-1562364), gifts from Google, Tencent, and Nvidia, and an Allen Distinguished Investigator Award. We also thank Mandar Joshi, Victoria Lin, and the UW NLP group for helpful conversations and comments on the work.

Q: How is the input triple translated to a slot-filling task?
A: The relation R(x,y) is mapped onto a question q whose answer is y