Introduction
Semantic applications typically work on the basis of intermediate structures derived from sentences. Traditional word-level intermediate structures, such as POS-tags, dependency trees and semantic role labels, have been widely applied. Recently, entity and relation level intermediate structures attract increasingly more attentions.
In general, knowledge based applications require entity and relation level information. For instance, in BIBREF0 , the lexicalized dependency path between two entity mentions was taken as the surface pattern facts. In distant supervision BIBREF1 , the word sequence and dependency path between two entity mentions were taken as evidence of certain relation. In Probase BIBREF2 , candidates of taxonomies were extracted by Hearst patterns BIBREF3 . The surface patterns of relations extracted by Open Information Extraction (OIE) systems BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 worked as the source of question answering systems BIBREF9 , BIBREF10 . In addition, entity and relation level intermediate structures have been proven effective in many other tasks such as text summarization BIBREF11 , BIBREF12 , BIBREF13 , text comprehension, word similarity, word analogy BIBREF14 , and more.
The task of entity/relation level mediate structure extraction studies how facts about entities and relations are expressed by natural language in sentences, and then expresses these facts in an intermediate (and convenient) format. Although entity/relation level intermediate structures have been utilized in many applications, the study of learning these structures is still in an early stage.
Firstly, the problem of extracting different types of entity/relation level intermediate structures has not been considered in a unified fashion. Applications generally need to construct their own handcrafted heuristics to extract required entity/relation level intermediate structures, rather than consulting a commonly available NLP component, as they do for word level intermediate structures. Open IE-v4 system (http://knowitall.github.io/openie/) attempted to build such components by developing two sub-systems, with each extracting one type of intermediate structures, i.e., SRLIE BIBREF15 for verb based relations, and ReNoun BIBREF16 , BIBREF17 for nominal attributes. However, important information about descriptive tags for entities and concept-instance relations between entities were not considered.
Secondly, existing solutions to the task either used pattern matching technique BIBREF2 , BIBREF4 , BIBREF6 , BIBREF7 , or were trained in a self-supervised manner on the data set automatically generated by heuristic patterns or info-box matching BIBREF7 , BIBREF4 , BIBREF8 . It is well-understood that pattern matching typically does not generalize well and the automatically generated samples may contain lots of noises.
This paper aims at tackling some of the well-known challenging problems in OIE systems, in a supervised end-to-end deep learning paradigm. Our contribution can be summarized as three major components: SAOKE format, SAOKE data set, and Logician.
Symbol Aided Open Knowledge Expression (SAOKE) is a knowledge expression form with several desirable properties: (i) SAOKE is literally honest and open-domain. Following the philosophy of OIE systems, SAOKE uses words in the original sentence to express knowledge. (ii) SAOKE provides a unified view over four common types of knowledge: relation, attribute, description and concept. (iii) SAOKE is an accurate expression. With the aid of symbolic system, SAOKE is able to accurately express facts with separated relation phrases, missing information, hidden information, etc.
SAOKE Data Set is a human annotated data set containing 48,248 Chinese sentences and corresponding facts in the SAOKE form. We publish the data set for research purpose. To the best of our knowledge, this is the largest publicly available human annotated data set for open-domain information extraction tasks.
Logician is a supervised end-to-end neural learning algorithm which transforms natural language sentences into facts in the SAOKE form. Logician is trained under the attention-based sequence-to-sequence paradigm, with three mechanisms: restricted copy mechanism to ensure literally honestness, coverage mechanism to alleviate the under extraction and over extraction problem, and gated dependency attention mechanism to incorporate dependency information. Experimental results on four types of open information extraction tasks reveal the superiority of the Logician algorithm.
Our work will demonstrate that SAOKE format is suitable for expressing various types of knowledge and is friendly to end-to-end learning algorithms. Particularly, we will focus on showing that the supervised end-to-end learning is promising for OIE tasks, to extract entity and relation level intermediate structures.
The rest of this paper is organized as follows. Section "SAOKE Format:  Symbol Aided Open Knowledge Expression" presents the details of SAOKE. Section "SAOKE Data Set" describes the human labeled SAOKE data set. Section "Logician" describes the Logician algorithm and Section "Empirical Evaluation" evaluates the Logician algorithm and compares its performance with the state-of-the-art algorithms on four OIE tasks. Section "Related Works" discusses the related work and Section "Conclusion" concludes the paper.
SAOKE Format:  Symbol Aided Open Knowledge Expression
When reading a sentence in natural language, humans are able to recognize the facts involved in the sentence and accurately express them. In this paper, Symbolic Aided Open Knowledge Expression (SAOKE) is proposed as the form for honestly recording these facts. SAOKE expresses the primary information of sentences in n-ary tuples $(subject,predicate,object_{1},\cdots ,object_{N})$ , and (in this paper) neglects some auxiliary information. In the design of SAOKE, we take four requirements into consideration: completeness, accurateness, atomicity and compactness.
Completeness
After having analyzed a large number of sentences, we observe that the majority of facts can be classified into the following classes:
Relation: Verb/preposition based n-ary relations between entity mentions BIBREF15 , BIBREF6 ;
Attribute:Nominal attributes for entity mentions BIBREF16 , BIBREF17 ;
Description: Descriptive phrases of entity mentions BIBREF18 ;
Concept: Hyponymy and synonym relations among concepts and instances BIBREF19 .
SAOKE is designed to express all these four types of facts. Table 1 presents an example sentence and the involved facts of these four classes in the SAOKE form. We should mention that the sentences and facts in English are directly translated from the corresponding Chinese sentences and facts, and the facts in English may not be the desired outputs of OIE algorithms for those English sentences due to the differences between Chinese and English languages.
Accurateness
SAOKE adopts the ideology of “literally honest”. That is, as much as possible, it uses the words in the original sentences to express the facts. SAOKE follows the philosophy of OIE systems to express various relations without relying on any predefined schema system. There are, however, exceptional situations which are beyond the expression ability of this format. Extra symbols will be introduced to handle these situations, which are explained as follows.
Separated relation phrase: In some languages such as Chinese, relation phrases may be divided into several parts residing in discontinued locations of the sentences. To accurately express these relation phrases, we add placeholders ( $X$ , $Y$ , $Z$ , etc) to build continuous and complete expressions. UTF8gbsn “深受X影响” (“deeply influenced by X” in English) in the example of Table 1 is an instance of relation phrase after such processing.
Abbreviated expression: We explicitly express the information in abbreviated expressions by introducing symbolic predicates. For example, the expression of “Person (birth date - death date)” is transformed into facts: (Person, BIRTH, birth date) (Person, DEATH, death date), and the synonym fact involved in “NBA (National Basketball Association)” is expressed in the form of (NBA, = , National Basketball Association) .
Hidden information: Description of an entity and hyponymy relation between entities are in general expressed implicitly in sentences, and are expressed by symbolic predicates “DESC” and “ISA” respectively, as in Table 1 . Another source of hidden information is the address expression. For example, UTF8gbsn “法国巴黎” (“Paris, France” in English) implies the fact UTF8gbsn (巴黎, LOC, 法国) ((Paris, LOC, France) in English), where the symbol “LOC” means “location”.
Missing information: A sentence may not tell us the exact relation between two entities, or the exact subject/objects of a relation, which are required to be inferred from the context. We use placeholders like “ $X,Y,Z$ ” to denote the missing subjects/objects, and “ $P$ ” to denote the missing predicates.
Atomicity
Atomicity is introduced to eliminate the ambiguity of knowledge expressions. In SAOKE format, each fact is required to be atomic, which means that: (i) it is self-contained for an accurate expression; (ii) it cannot be decomposed into multiple valid facts. We provide examples in Table 2 to help understand these two criteria.
Note that the second criterion implies that any logical connections (including nested expressions) between facts are neglected (e.g., the third case in Table 2 ). This problem of expression relations between facts will be considered in the future version of SAOKE.
Compactness
Natural language may express several facts in a compact form. For example, in a sentence UTF8gbsn “李白爱饮酒作诗” (“Li Bai loved to drink and write poetry” in English ), according to atomicity, two facts should be extracted: UTF8gbsn (李白, 爱, 饮酒)(李白, 爱, 作诗) ( (Li Bai, loved to, drink)(Li Bai, loved to, write poetry) in English ). In this situation, SAOKE adopts a compact expression to merge these two facts into one expression: UTF8gbsn (李白, 爱, [饮酒|作诗]) ( (Li Bai, loved to, [drink| write poetry]) in English ).
The compactness of expressions is introduced to fulfill, but not to violate the rule of “literally honest”. SAOKE does not allow merging facts if facts are not expressed compactly in original sentences. By this means, the differences between the sentences and the corresponding knowledge expressions are reduced, which may help reduce the complexity of learning from data in SAOKE form.
With the above designs, SAOKE is able to express various kinds of facts, with each historically considered by different open information extraction algorithms, for example, verb based relations in SRLIE BIBREF15 and nominal attributes in ReNoun BIBREF16 , BIBREF17 , descriptive phrases for entities in EntityTagger BIBREF18 , and hypernyms in HypeNet BIBREF19 . SAOKE introduces the atomicity to eliminate the ambiguity of knowledge expressions, and achieves better accuracy and compactness with the aid of the symbolic expressions.
SAOKE Data Set
We randomly collect sentences from Baidu Baike (http://baike.baidu.com), and send those sentences to a crowd sourcing company to label the involved facts. The workers are trained with labeling examples and tested with exams. Then the workers with high exam scores are asked to read and understand the facts in the sentences, and express the facts in the SAOKE format. During the procedure, one sentence is only labeled by one worker. Finally, more than forty thousand sentences with about one hundred thousand facts are returned to us. The manual evaluation results on 100 randomly selected sentences show that the fact level precision and recall is 89.5% and 92.2% respectively. Table 3 shows the proportions of four types of facts (described in Section "SAOKE Data Set" ) contained in the data set. Note that the facts with missing predicates represented by “P” are classified into “Unknown”. We publicize the data set at https://ai.baidu.com/broad/subordinate?dataset=saoke.
Prior to the SAOKE data set, an annotated data set for OIE tasks with 3,200 sentences in 2 domains was released in BIBREF20 to evaluate OIE algorithms, in which the data set was said BIBREF20 “13 times larger than the previous largest annotated Open IE corpus”. The SAOKE data set is 16 times larger than the data set in BIBREF20 . To the best of our knowledge, SAOKE data set is the largest publicly available human labeled data set for OIE tasks. Furthermore, the data set released in BIBREF20 was generated from a QA-SRL data set BIBREF21 , which indicates that the data set only contains facts that can be discovered by SRL (Semantic Role Labeling) algorithms, and thus is biased, whereas the SAOKE data set is not biased to an algorithm. Finally, the SAOKE data set contains sentences and facts from a large number of domains.
Logician
Given a sentence $S$ and a set of expected facts (with all the possible types of facts) $\mathbb {F}=\lbrace F_{1},\cdots ,F_{n}\rbrace $ in SAOKE form, we join all the facts in the order that annotators wrote them into a char sequence $F$ as the expected output. We build Logician under the attention-based sequence-to-sequence learning paradigm, to transform $S$ into $F$ , together with the restricted copy mechanism, the coverage mechanism and the gated dependency mechanism.
Attention based Sequence-to-sequence Learning 
The attention-based sequence-to-sequence learning BIBREF22 have been successfully applied to the task of generating text and patterns. Given an input sentence $S=[w_{1}^{S},\cdots ,w_{N_{S}}^{S}]$ , the target sequence $F=[w_{1}^{F},\cdots ,w_{N_{F}}^{F}]$ and a vocabulary $V$ (including the symbols introduced in Section "SAOKE Format:  Symbol Aided Open Knowledge Expression" and the OOV (out of vocabulary) tag ) with size $N_{v}$ , the words $w_{i}^{S}$ and $w_{j}^{F}$ can be represented as one-hot vectors $v_{i}^{S}$ and $v_{j}^{F}$ with dimension $N_{v}$ , and transformed into $N_{e}$ -dimensional distributed representation vectors by an embedding transform $F=[w_{1}^{F},\cdots ,w_{N_{F}}^{F}]$0 and $F=[w_{1}^{F},\cdots ,w_{N_{F}}^{F}]$1 respectively, where $F=[w_{1}^{F},\cdots ,w_{N_{F}}^{F}]$2 . Then the sequence of $F=[w_{1}^{F},\cdots ,w_{N_{F}}^{F}]$3 is transformed into a sequence of $F=[w_{1}^{F},\cdots ,w_{N_{F}}^{F}]$4 -dimensional hidden states $F=[w_{1}^{F},\cdots ,w_{N_{F}}^{F}]$5 using bi-directional GRU (Gated Recurrent Units) network BIBREF23 , and the sequence of $F=[w_{1}^{F},\cdots ,w_{N_{F}}^{F}]$6 is transformed into a sequence of $F=[w_{1}^{F},\cdots ,w_{N_{F}}^{F}]$7 -dimensional hidden states $F=[w_{1}^{F},\cdots ,w_{N_{F}}^{F}]$8 using GRU network.
For each position $t$ in the target sequence, the decoder learns a dynamic context vector $c_{t}$ to focus attention on specific location $l$ in the input hidden states $H^{S}$ , then computes the probability of generated words by $p(w_{t}^{F}|\lbrace w_{1}^{F},\cdots ,w_{t-1}^{F}\rbrace ,c_{t})=g(h_{t-1}^{F},s_{t},c_{t})$ , where $s_{t}$ is the hidden state of the GRU decoder, $g$ is the word selection model (details could be found in BIBREF22 ), and $c_{t}$ is computed as $c_{t}=\sum _{j=1}^{N_{S}}\alpha _{tj}h_{j},$ where $\alpha _{tj}=\frac{\exp (e_{tj})}{\sum _{k=1}^{N_{S}}\exp (e_{tk})}$ and $c_{t}$0 is the alignment model to measure the strength of focus on the $c_{t}$1 -th location. $c_{t}$2 , $c_{t}$3 , and $c_{t}$4 are weight matrices.
Restricted Copy Mechanism
The word selection model employed in BIBREF22 selects words from the whole vocabulary $V$ , which evidently violates the “literal honest” requirement of SAOKE. We propose a restricted version of copy mechanism BIBREF24 as the word selection model for Logician:
We collect the symbols introduced in Section "SAOKE Format:  Symbol Aided Open Knowledge Expression" into a keyword set $K=\lbrace $ “ $ISA$ ”, “ $DESC$ ”, “ $LOC$ ”, “ $BIRTH$ ”, “ $DEATH$ ”, “ $=$ ”, “ $($ ”, “)”, “ $\$$ ”,“ $[$ ”, “ $ISA$0 ”, “ $ISA$1 ”, “ $ISA$2 ”, “ $ISA$3 ”, “ $ISA$4 ”, “ $ISA$5 ” $ISA$6 where “ $ISA$7 ” is the separator of elements of fact tuples. “ $ISA$8 ”, “ $ISA$9 ”, “ $DESC$0 ”, “ $DESC$1 ” are placeholders . When the decoder is considering generating a word $DESC$2 , it can choose $DESC$3 from either $DESC$4 or $DESC$5 .
$$p(w_{t}^{F}|w_{t-1}^{F},s_{t},c_{t})=p_{X}(w_{t}^{F}|w_{t-1}^{F},s_{t},c_{t})+p_{K}(w_{t}^{F}|w_{t-1}^{F},s_{t},c_{t}),$$   (Eq. 15)
where $p_{X}$ is the probability of copying from $S$ and $p_{K}$ is the probability of selecting from $K$ . Since $S\cap K=\phi $ and there are no unknown words in this problem setting, we compute $p_{X}$ and $p_{K}$ in a simpler way than that in BIBREF24 , as follows: $ p_{X}(w_{t}^{F}=w_{j}^{S}) & = & \frac{1}{Z}\exp (\sigma ((h_{j}^{S})^{T}W_{c})s_{t}),\\ p_{K}(w_{t}^{F}=k_{i}) & = & \frac{1}{Z}\exp (v_{i}^{T}W_{o}s_{t}), $
where the (generic) $Z$ is the normalization term, $k_{i}$ is one of keywords, $v_{i}$ is the one-hot indicator vector for $k_{i}$ , $W_{o}\in \mathbb {R}^{(|K|\times N_{h})}$ , $W_{c}\in \mathbb {R}^{(N_{h}\times N_{h})}$ , and $\sigma $ is a nonlinear activation function.
Coverage Mechanism
In practice, Logician may forget to extract some facts (under-extraction) or extract the same fact many times (over-extraction). We incorporate the coverage mechanism BIBREF25 into Logician to alleviate these problems. Formally, when the decoder considers generating a word $w_{t}^{F}$ , a coverage vector $m_{j}^{t}$ is introduced for each word $w_{j}^{S}$ , and updated as follows: $ m_{j}^{t} & = & \mu (m_{j}^{t-1},\alpha _{tj},h_{j}^{S},s_{t-1})=(1-z_{i})\circ m_{j}^{t-1}+z_{j}\circ \tilde{m}_{j}^{t},\\ \tilde{m}_{j}^{t} & = & \tanh (W_{h}h_{j}^{S}+u_{\alpha }\alpha _{tj}+W_{s}s_{t-1}+U_{m}[r_{i}\circ m_{j}^{t-1}]), $
where $\circ $ is the element-wise multiplication operator. The update gate $z_{j}$ and the reset gate $r_{j}$ are defined as, respectively, $ z_{j} & = & \sigma (W_{h}^{z}h_{j}^{S}+u_{\alpha }^{z}\alpha _{tj}+W_{s}^{z}s_{t-1}+U_{m}^{z}m_{j}^{t-1}),\\ r_{j} & = & \sigma (W_{h}^{r}h_{j}^{S}+u_{\alpha }^{r}\alpha _{tj}+W_{s}^{r}s_{t-1}+U_{m}^{r}m_{j}^{t-1}), $
where $\sigma $ is a logistic sigmoid function. The coverage vector $m_{j}^{t}$ contains the information about the historical attention focused on $w_{j}^{S}$ , and is helpful for deciding whether $w_{j}^{S}$ should be extracted or not. The alignment model is updated as follows BIBREF25 : $ e_{tj}=a(s_{t-1},h_{j}^{S},m_{j}^{t-1})=v_{a}^{T}\tanh (W_{a}s_{t-1}+U_{a}h_{j}^{S}+V_{a}m_{j}^{t-1}), $
where $V_{a}\in \mathbb {R}^{(N_{h}\times N_{h})}$ .
Gated Dependency Attention
The semantic relationship between candidate words and the previously decoded word is valuable to guide the decoder to select the correct word. We introduce the gated dependency attention mechanism to utilize such guidance.
For a sentence $S$ , we extract the dependency tree using NLP tools such as CoreNLP BIBREF26 for English and LTP BIBREF27 for Chinese, and convert the tree into a graph by adding reversed edges with a revised labels (for example, adding $w_{j}^{S}\xrightarrow{}w_{i}^{S}$ for edge $w_{i}^{S}\xrightarrow{}w_{j}^{S}$ in the dependency tree). Then for each pair of words $(w_{i}^{S},w_{j}^{S})$ , the shortest path with labels $L=[w_{1}^{L},\cdots ,w_{N_{L}}^{L}]$ in the graph is computed and mapped into a sequence of $N_{e}$ -dimensional distributed representation vectors $[l_{1},\cdots ,l_{N_{L}}]$ by the embedding operation. One can employ RNN network to convert this sequence of vectors into a feature vector, but RNN operation is time-consuming. We simply concatenate vectors in short paths ( $N_{L}\le $ 3) into a $3N_{e}$ dimensional vector and feed the vector into a two-layer feed forward neural network to generate an $N_{h}$ -dimensional feature vector $w_{j}^{S}\xrightarrow{}w_{i}^{S}$0 . For long paths with $w_{j}^{S}\xrightarrow{}w_{i}^{S}$1 , $w_{j}^{S}\xrightarrow{}w_{i}^{S}$2 is set to a zero vector. We define dependency attention vector $w_{j}^{S}\xrightarrow{}w_{i}^{S}$3 , where $w_{j}^{S}\xrightarrow{}w_{i}^{S}$4 is the sharpened probability $w_{j}^{S}\xrightarrow{}w_{i}^{S}$5 defined in Equation ( 15 ). If $w_{j}^{S}\xrightarrow{}w_{i}^{S}$6 , $w_{j}^{S}\xrightarrow{}w_{i}^{S}$7 represents the semantic relationship between $w_{j}^{S}\xrightarrow{}w_{i}^{S}$8 and $w_{j}^{S}\xrightarrow{}w_{i}^{S}$9 . If $w_{i}^{S}\xrightarrow{}w_{j}^{S}$0 , then $w_{i}^{S}\xrightarrow{}w_{j}^{S}$1 is close to zero. To correctly guide the decoder, we need to gate $w_{i}^{S}\xrightarrow{}w_{j}^{S}$2 to remember the previous attention vector sometimes (for example, when $w_{i}^{S}\xrightarrow{}w_{j}^{S}$3 is selected), and to forget it sometimes (for example, when a new fact is started). Finally, we define $w_{i}^{S}\xrightarrow{}w_{j}^{S}$4 $w_{i}^{S}\xrightarrow{}w_{j}^{S}$5 ) as the gated dependency attention vector, where $w_{i}^{S}\xrightarrow{}w_{j}^{S}$6 is the GRU gated function, and update the alignment model as follows: $w_{i}^{S}\xrightarrow{}w_{j}^{S}$7
where $D_{a}\in \mathbb {R}^{(N_{h}\times N_{h})}$ .
Post processing
For each sequence generated by Logician, we parse it into a set of facts, remove tuples with illegal format or duplicated tuples. The resultant set is taken as the output of the Logician.
Experimental Design 
We first measure the utility of various components in Logician to select the optimal model, and then compare this model to the state-of-the-art methods in four types of information extraction tasks: verb/preposition-based relation, nominal attribute, descriptive phrase and hyponymy relation. The SAOKE data set is split into training set, validating set and testing set with ratios of 80%, 10%, 10%, respectively. For all algorithms involved in the experiments, the training set can be used to train the model, the validating set can be used to select an optimal model, and the testing set is used to evaluate the performance.
For each instance pair $(S,F)$ in the test set, where $S$ is the input sentence and $F$ is the formatted string of ground truth of facts, we parse $F$ into a set of tuples $\mathbb {F}=\lbrace F_{i}\rbrace _{j=1}^{M}$ . Given an open information extraction algorithm, it reads $S$ and produces a set of tuples $\mathbb {G}=\lbrace G_{i}\rbrace _{j=1}^{N}$ . To evaluate how well the $\mathbb {G}$ approximates $\mathbb {F}$ , we need to match each $G_{i}$ to a ground truth fact $S$0 and check whether $S$1 tells the same fact as $S$2 . To conduct the match, we compute the similarity between each predicted fact in $S$3 and each ground truth fact in $S$4 , then find the optimal matching to maximize the sum of matched similarities by solving a linear assignment problem BIBREF28 . In the procedure, the similarity between two facts is defined as $S$5
where $G_{i}(l)$ and $F_{j}(l)$ denote the $l$ -th element of tuple $G_{i}$ and $F_{j}$ respectively, $\mathbf {g}(\cdot ,\cdot )$ denotes the gestalt pattern matching BIBREF29 measure for two strings and $\mathbf {n}(\text{$\cdot $)}$ returns the length of the tuple.
Given a matched pair of $G_{i}$ and $F_{j}$ , we propose an automatic approach to judge whether they tell the same fact. They are judged as telling the same fact if one of the following two conditions is satisfied:
$\mathbf {n}(G_{i})=\mathbf {n}(F_{j})$ , and $\mathbf {g}(G_{i}(l),F_{j}(l))\ge 0.85,l=1,\cdots ,\mathbf {n}(G_{i})$ ;
$\mathbf {n}(G_{i})=\mathbf {n}(F_{j})$ , and $\mathbf {g}(\mathcal {S}(G_{i}),\mathcal {S}(F_{j})\ge 0.85$ ;
where $\mathcal {S}$ is a function formatting a fact into a string by filling the arguments into the placeholders of the predicate.
With the automatic judgment, the precision ( $P$ ), recall ( $R$ ) and $F_{1}$ -score over a test set can be computed. By defining a confidence measure and ordering the facts by their confidences, a precision-recall curve can be drawn to illustrate the overall performance of the algorithm. For Logician, the confidence of a fact is computed as the average of log probabilities over all words in that fact.
Beyond the automatic judgment, human evaluation is also employed. Given an algorithm and the corresponding fact confidence measure, we find a threshold that produces approximately 10% recall (measured by automatic judgment) on the validation set of SAOKE data set. A certain number of sentences (200 for verb/preposition based relation extraction task, and 1000 for other three tasks) are randomly chosen from the testing set of SAOKE data set, and the facts extracted from these sentences are filtered with that threshold. Then we invite three volunteers to manually refine the labeled set of facts for each sentence and vote to decide whether each filtered fact is correctly involved in the sentence. The standard precision, recall and $F_{1}$ -score are reported as the human evaluation results.
For each instance pair $(S,F)$ in the training set of SAOKE data set, we split $S$ and $F$ into words using LTP toolset BIBREF27 , and words appearing in more than 2 sentences are added to the vocabulary. By adding the OOV (out of vocabulary) tag, we finally obtain a vocabulary $V$ with size $N_{V}=65,293$ . The dimension of all embedding vectors is set to $N_{e}=200$ , and the dimension of hidden states is set to $N_{h}=256$ . We use a three-layer bi-directional GRU with dimension 128 to encode $\lbrace x_{i}\rbrace _{i=1}^{N_{S}}$ into hidden states $\lbrace h_{i}^{S}\rbrace _{i=1}^{N_{S}}$ , and a two-layer GRU with hidden-dimension 256 to encode the sequence of $\lbrace y_{j}\rbrace _{j=1}^{N_{F}}$ into hidden states $S$0 . Finally, the Logician network is constructed as stated in Section "Logician" . The Logician is then trained using stochastic gradient descent (SGD) with RMSPROP BIBREF30 strategy for 20 epochs with batch size 10 on the training set of SAOKE data set. The model with best $S$1 -score by automatic judgment on the validation set is selected as the trained model. When the model is trained, given a sentence, we employ the greedy search procedure to produce the fact sequences.
Evaluating Components' Utilities
In this section, we analyze the effects of components involved in Logician: restricted copy, coverage, and gated dependency. Since the restricted copy mechanism is the essential requirement of Logician in order to achieve the goal of literally honest, we take the Logician with only copy mechanism (denoted by $Copy$ ) as the baseline, and analyze the effeteness of coverage mechanism (denoted by $Copy+Coverage$ ), gated dependency mechanism (denoted by $Copy+GatedDep$ ) and both (denoted by $All$ ). Furthermore, there is another option of whether or not to involve shallow semantic information such as POS-tag and NER-tag into the model. For models involving such information, the POS-tag and NER-tag of each word in sentence $S$ are annotated using LTP. For each word in $F$ that is not any keyword in $K$ , the POS-tag and NER-tag are copied from the corresponding original word in $S$ . For each keyword in $K$ , a unique POS-tag and a unique NER-tag are assigned to it. Finally, for each word in $S$ or $Copy+Coverage$0 , the POS-tag and NER-tag are mapped into $Copy+Coverage$1 -dimensional distributed representation vectors and are concatenated into $Copy+Coverage$2 or $Copy+Coverage$3 to attend the training.
All models are trained using the same settings described in above section, and the default output facts (without any confidence filtering) are evaluated by the automatic judgment. The results are reported in Table 4 . From the results, we can see that the model involving all the components and shallow tag information archives the best performance. We use that model to attend the comparisons with existing approaches.
Comparison with Existing Approaches
In the task of extracting verb/preposition based facts, we compare our Logician with the following state-of-the-art Chinese OIE algorithms:
SRLIE: our implementation of SRLIE BIBREF15 for the Chinese language, which first uses LTP tool set to extract the semantic role labels, and converts the results into fact tuples using heuristic rules. The confidence of each fact is computed as the ratio of the number of words in the fact to the number of words in the shortest fragment of source sentence that contains all words in the fact.
ZORE : the Chinese Open Relation Extraction system BIBREF31 , which builds a set of patterns by bootstrapping based on dependency parsing results, and uses the patterns to extract relations. We used the program provided by the author of ZORE system BIBREF31 to generate the extraction results in XML format, and developed an algorithm to transform the facts into n-ary tuples, where auxiliary information extracted by ZORE is removed. The confidence measure for ZORE is the same as that for SRLIE.
SRL $_{\text{SAOKE}}$ : our implementation of the states-of-the-art SRL algorithm proposed in BIBREF32 with modifications to fit OIE tasks. $\text{SRL}_{\text{SAOKE}}$ extracts facts in two steps: (i) Predicate head word detection: detects head word for predicate of each possible fact, where head word of a predicate is the last word in the predicate depending on words outside the predicate in the dependency tree. (ii) Element phrase detection: For each detected head word, detects the subject phrase, predicate phrase and object phrases by tagging the sentence with an extended BIOE tagging scheme, which tags the word neighboring the separation point of the phrase by “M” to cope with the separated phrase. We modify the code provided by the author of BIBREF32 to implement above strategy, and then train a model with the same parameter setting in BIBREF32 on the training set of SAOKE data set. The confidence measure for $\text{SRL}_{\text{SAOKE}}$ is computed as the average of log probabilities over all tags of words in facts. Note that $\text{SRL}_{\text{SAOKE}}$ can extract both verb/preposition based relation and nominal attributes, but in this section, we only evaluate the results of the former type of facts.
The precision-recall curves of Logician and above three comparison algorithms are shown in Figure 1 , and the human evaluation results are shown in the first section of Table 5 .
The state-of-the-art
nominal attribute extraction method is ReNoun BIBREF16 , BIBREF17 . However, it relies on a pre-constructed English attribute schema system BIBREF33 which is not available for Chinese, so it is not an available baseline for Chinese. Since $\text{SRL}_{\text{SAOKE}}$ can extract nominal attributes, we compare Logician with $\text{SRL}_{\text{SAOKE}}$ on this task. The precision-recall curves of Logician and $\text{SRL}_{\text{SAOKE}}$ on the nominal attribute extraction task are shown in Figure 1 , and the human evaluation results are shown in the second section of Table 5 .
Descriptive phrase extraction has been considered in BIBREF18 , in which domain names are required to develop patterns to extract candidates for descriptive phrases, so this method is not applicable to open domain tasks. We develop a baseline algorithm (called Semantic Dependency Description Extractor, SDDE) to extract descriptive phrase. It extracts semantic dependency relation between words using LTP toolset, and for each noun $w_n$ which is the parent of some semantic “Desc” relations, identifies a noun phrase $N$ with $w_n$ as its heading word, assembles a descriptive phrase $D$ containing all words with “Desc” relation to $w_n$ , and finally outputs the fact “( $N$ , $DESC$ , $D$ )”. The confidence of fact in SDDE is computed as the ratio of the number of adverbs and adjectives in $D$ to the number of words in $D$ . The precision-recall curves of Logician and SDDE on the descriptive phrase extraction task are shown in Figure 1 , and the human evaluation results are shown in the third section of Table 5 .
HypeNet BIBREF19 is the state-of-the-art algorithm recommended for hyponymy extraction BIBREF34 , which judges whether hyponymy relation exists between two given words. To make it capable of judging hyponymy relation between two phrases, we replace the word embedding vector component in HypeNet by an LSTM network. Two modified HypeNet models are built using different training data sets: (i) $\text{HypeNet}_{\text{Phrase}}$ : using the pairs of phrases with ISA relation in the training set of SAOKE data set (9,407 pairs after the compact expression expansion); (ii) $\text{HypeNet}_{\text{Phrase}}^{\text{Extra}}$ : besides the training set for $\text{HypeNet}_{\text{Phrase}}$ , adding two Chinese hyponymy data sets (1.4 million pair of words in total in hyponymy relation): Tongyici Cilin (Extended) (CilinE for short) BIBREF27 and cleaned Wikipedia Category data BIBREF35 . In both cases, the sentences from both Chinese Wikipedia pages and training set of SAOKE data set are taken as the background corpus for the HypeNet algorithm. In the testing phase, the trained models are used to predict whether the hyponymy relation exists for each pair of noun phrases/words in sentences of the testing set of SAOKE data set. The confidence of a judgment is the predicted probability of the existence of hyponymy relation. The precision-recall curves of Logician, $\text{HypeNet}_{\text{Phrase}}$ and $\text{HypeNet}_{\text{Phrase}}^{\text{Extra}}$ are shown in Figure 1 , and the human evaluation results in the fourth section of Table 5 .
Results Analysis
The experimental results reveal that, Logician outperforms the comparison methods with large margin in first three tasks. For hyponymy detection tasks, Logician overwhelms the $\text{HypeNet}_{\text{Phrase}}$ using the same training data, and produces comparable results to $\text{HypeNet}_{\text{Phrase}}^{\text{Extra}}$ with much less training data. Table 6 exhibits several example sentences and the facts extracted by these algorithms.
The poor performance of pattern-based methods is plausibly due to the noise in SAOKE data set. The sentences in SAOKE data set are randomly selected from a web encyclopedia, with free and casual writing style, are thus more noisy than the training data of NLP toolset used by these methods. In this situation, the NLP toolset may produce poor results, so do the pattern-based methods.
Models learned from the SAOKE data set archive much better performance. Nevertheless, $\text{SRL}_{\text{SAOKE}}$ extracts each fact without knowing whether a candidate word has been used in other facts, which results in the misleading overlap of the word UTF8gbsn“学” (“Learn” in English) between two facts in the first case of Table 6 . Similarly, $\text{HypeNet}_{\text{Phrase}}$ and $\text{HypeNet}_{\text{Phrase}}^{\text{Extra}}$ focus on the semantic vectors of pairs of phrases and their dependency paths in the background corpus. They extract each fact independently from other facts and hence do not know whether there have been any other relations extracted about these two phrases. In other words, for those comparison methods, an important source of information is neglected and a global optimization for all facts involved in sentences is absent.
On the contrary, Logician performs global optimization over the facts involved in each sentence by the sequence-to-sequence learning paradigm with the help of the coverage mechanism, in which facts compete each other to attract the attention of words, but also cooperate to share words. Valuable information is shared between these multiple tasks, which makes Logician consistently superior to other algorithms in these tasks.
Furthermore, $\text{SRL}_{\text{SAOKE}}$ and $\text{HypeNet}$ methods suffer from the OOV problem, such as unfamiliar words/phrases like the person name and school name in the last case of Table 6 . In this situation they may fail to produce a reasonable result. Logician is able to cope with unfamiliar words/phrases by exploiting the context information using deep RNN network with the help of copy mechanism.
Extraction Error Analysis of Logician
We do a preliminary analysis for the results produced by the Logician model. The most notable problem is that it is unable to recall some facts for long or complex sentences. The last case in Table 6 exhibits such situation, where the fact UTF8gbsn(蔡竞,ISA,经济学博士)((Cai Jing, ISA, Ph. D. in economics) in English) is not recalled. This phenomenon indicates that the coverage mechanism may lose effectiveness in this situation. The second class of error is incomplete extraction, as exhibited in the third case in Table 6 . Due to the incomplete extraction, the left parts may interfere the generation of other facts, and result in nonsense results, which is the third class of error. We believe it is helpful to introduce extra rewards into the learning procedure of Logician to overcome these problems. For example, the reward could be the amount of remaining information left after the fact extraction, or the completeness of extracted facts. Developing such rewards and reinforcement learning algorithms using those rewards to refine Logician belongs to our future works.
Knowledge Expressions
Tuple is the most common knowledge expression format for OIE systems to express n-ary relation between subject and objects. Beyond such information, ClausIE BIBREF36 extracts extra information in the tuples: a complement, and one or more adverbials, and OLLIE BIBREF6 extracts additional context information. SAOKE is able to express n-ary relations, and can be easily extended to support the knowledge extracted by ClausIE, but needs to be redesigned to support context information, which belongs to our future work.
However, there is a fundamental difference between SAOKE and tuples in traditional OIE systems. In traditional OIE systems, knowledge expression is generally not directly related to the extraction algorithm. It is a tool to reorganize the extracted knowledge into a form for further easy reading/storing/computing. However, SAOKE is proposed to act as the direct learning target of the end-to-end Logician model. In such end-to-end framework, knowledge representation is the core of the system, which decides what information would be extracted and how complex the learning algorithm would be. To our knowledge, SAOKE is the first attempt to design a knowledge expression friendly to the end-to-end learning algorithm for OIE tasks. Efforts are still needed to make SAOKE more powerful in order to express more complex knowledge such as events.
Relation Extraction
Relation extraction is the task to identify semantic connections between entities. Major existing relation extraction algorithms can be classified into two classes: closed-domain and open-domain. Closed-domain algorithms are learnt to identify a fixed and finite set of relations, using supervised methods BIBREF37 , BIBREF38 , BIBREF39 , BIBREF40 or weakly supervised methods BIBREF1 , BIBREF41 , while the open-domain algorithms, represented by aforementioned OIE systems, discover open-domain relations without predefined schema. Beyond these two classes, methods like universal schema BIBREF42 are able to learn from both data with fixed and finite set of relations, such as relations in Freebase, and data with open-domain surface relations produced by heuristic patterns or OIE systems.
Logician can be used as an OIE system to extract open-domain relation between entities, and act as sub-systems for knowledge base construction/completion with the help of schema mapping BIBREF43 . Compared with existing OIE systems, which are pattern-based or self-supervised by labeling samples using patterns BIBREF13 , to our knowledge Logician is the first model trained in a supervised end-to-end approach for OIE task, which has exhibited powerful ability in our experiments. There are some neural based end-to-end systems BIBREF39 , BIBREF40 , BIBREF41 proposed for relation extraction, but they all aim to solve the close-domain problem.
However, Logician is not limited to relation extraction task. First, Logician extracts more information beyond relations. Second, Logician focuses on examining how natural languages express facts BIBREF5 , and producing helpful intermediate structures for high level tasks.
Language to Logic 
Efforts had been made to map natural language sentences into logical form. Some approaches such as BIBREF44 , BIBREF45 , BIBREF46 , BIBREF47 learn the mapping under the supervision of manually labeled logical forms, while others BIBREF48 , BIBREF49 are indirectly supervised by distant information, system rewards, etc. However, all previous works rely on a pre-defined, domain specific logical system, which limits their ability to learn facts out of the pre-defined logical system.
Logician can be viewed as a system that maps language to natural logic, in which the majority of information is expressed by natural phrase. Other than systems mentioned above which aim at execution using the logical form, Logician focuses on understanding how the fact and logic are expressed by natural language. Further mapping to domain-specific logical system or even executor can be built on the basis of Logician's output, and we believe that, with the help of Logician, the work would be easier and the overall performance of the system may be improved.
Facts to Language
The problem of generating sentences from a set of facts has attracted a lot of attentions BIBREF50 , BIBREF51 , BIBREF52 , BIBREF53 . These models focus on facts with a predefined schema from a specific problem domain, such as people biographies and basketball game records, but could not work on open domain. The SAOKE data set provides an opportunity to extend the ability of these models into open domain.
Duality between Knowledge and Language
As mentioned in above sections, the SAOKE data set provides examples of dual mapping between facts and sentences. Duality has been verified to be useful to promote the performance of agents in many NLP tasks, such as back-and-forth translation BIBREF54 , and question-answering BIBREF55 . It is a promising approach to use the duality between knowledge and language to improve the performance of Logician.
Conclusion
In this paper, we consider the open information extraction (OIE) problem for a variety of types of facts in a unified view. Our solution consists of three components: SAOKE format, SAOKE data set, and Logician. SAOKE form is designed to express different types of facts in a unified manner. We publicly release the largest manually labeled data set for OIE tasks in SAOKE form. Using the labeled SAOKE data set, we train an end-to-end neural sequence-to-sequence model, called Logician, to transform sentences in natural language into facts. The experiments reveal the superiority of Logician in various open-domain information extraction tasks to the state-of-the-art algorithms.
Regarding future work, there are at least three promising directions. Firstly, one can investigate knowledge expression methods to extend SAOKE to express more complex knowledge, for tasks such as event extraction. Secondly, one can develop novel learning strategies to improve the performance of Logician and adapt the algorithm to the extended future version of SAOKE. Thirdly, one can extend SAOKE format and Logician algorithm in other languages.

Question: What open relation extraction tasks did they experiment on?
Answer: verb/preposition-based relation, nominal attribute, descriptive phrase and hyponymy relation.

Question: How is Logician different from traditional seq2seq models?
Answer: restricted copy mechanism to ensure literally honestness, coverage mechanism to alleviate the under extraction and over extraction problem, and gated dependency attention mechanism to incorporate dependency information

Question: What's the size of the previous largest OpenIE dataset?
Answer: 3,200 sentences