Model Architecture and Decoding	Our training method in sec:ours could potentially be used with any probabilistic open IE model, since we make no assumptions about the model and only the likelihood of the extraction is required for iterative rank-aware learning. As a concrete instantiation in our experiments, we use RnnOIE  ,  , a stacked BiLSTM with highway connections  ,  and recurrent dropout  . Input of the model is the concatenation of word embedding and another embedding indicating whether this word is predicate: $
\mathbf {x}_t = [\mathbf {W}_{\text{emb}}(w_t), \mathbf {W}_{\text{mask}}(w_t = v)].
$  The probability of the label at each position is calculated independently using a softmax function: $
P(y_t|\mathbf {s}, v) \propto \text{exp}(\mathbf {W}_{\text{label}}\mathbf {h}_t + \mathbf {b}_{\text{label}}),
$  where $\mathbf {h}_t$ is the hidden state of the last layer.

Problem Formulation	Given sentence $\mathbf {s}=(w_1, w_2, ..., w_n)$ , the goal of open IE is to extract assertions in the form of tuples $\mathbf {r}=(\mathbf {p}, \mathbf {a}_1, \mathbf {a}_2, ..., \mathbf {a}_m)$ , composed of a single predicate and $m$ arguments. Generally, these components in $\mathbf {r}$ need not to be contiguous, but to simplify the problem we assume they are contiguous spans of words from $\mathbf {s}$ and there is no overlap between them. Methods to solve this problem have recently been formulated as sequence-to-sequence generation  ,  ,  or sequence labeling  ,  . We adopt the second formulation because it is simple and can take advantage of the fact that assertions only consist of words from the sentence.

Introduction	Open information extraction (IE, sekine2006demand, Banko:2007:OIE) aims to extract open-domain assertions represented in the form of $n$ -tuples (e.g., was born in; Barack Obama; Hawaii) from natural language sentences (e.g., Barack Obama was born in Hawaii). Open IE started from rule-based  and syntax-driven systems  ,  , and recently has used neural networks for supervised learning  ,  ,  ,  ,  . A key step in open IE is confidence modeling, which ranks a list of candidate extractions based on their estimated quality. This is important for downstream tasks, which rely on trade-offs between the precision and recall of extracted assertions. For instance, an open IE-powered medical question answering (QA) system may require its assertions in higher precision (and consequently lower recall) than QA systems for other domains. For supervised open IE systems, the confidence score of an assertion is typically computed based on its extraction likelihood given by the model  ,  . However, we observe that this often yields sub-optimal ranking results, with incorrect extractions of one sentence having higher likelihood than correct extractions of another sentence.

Iterative Learning	Compared to using external models for confidence modeling, an advantage of the proposed method is that the base model does not change: the binary classification loss just provides additional supervision. Ideally, the resulting model after one-round of training becomes better not only at confidence modeling, but also at assertion generation, suggesting that extractions of higher quality can be added as training samples to continue this training process iteratively. The resulting iterative learning procedure (alg:iter) incrementally includes extractions generated by the current model as training samples to optimize the binary classification loss to obtain a better model, and this procedure is continued until convergence. [t] training data $\mathcal {D}$ , initial model $\theta ^{(0)}$ model after convergence $\theta $ $t \leftarrow 0$ # iteration  $\mathcal {E} \leftarrow \emptyset $ # generated extractions not converge $\mathcal {E} \leftarrow \mathcal {E} \cup \lbrace (\mathbf {s}, v, \hat{\mathbf {y}})|v,\hat{\mathbf {y}} \in g_{\theta ^{(t)}}(\mathbf {s}),

Neural Models for Open IE	We briefly revisit the formulation of open IE and the neural network model used in our paper.

We hypothesize this is due to the issue of a disconnect between training and test-time objectives. Specifically, the system is trained solely to raise likelihood of gold-standard extractions, and during training the model is not aware of its test-time behavior of ranking a set of system-generated assertions across sentences that potentially include incorrect extractions. To calibrate open IE confidences and make them more globally comparable across different sentences, we propose an iterative rank-aware learning approach, as outlined in fig:arch. Given extractions generated by the model as training samples, we use a binary classification loss to explicitly increase the confidences of correct extractions and decrease those of incorrect ones. Without adding additional model components, this training paradigm naturally leads to a better open IE model, whose extractions can be further included as training samples. We further propose an iterative learning procedure that gradually improves the model by incrementally adding extractions to the training data. Experiments on the OIE2016 dataset  indicate that our method significantly outperforms both neural and non-neural models.

Evaluation Results	tab:expmain lists the evaluation results. Our base model (RnnOIE, sec:oie) performs better than non-neural systems, confirming the advantage of supervised training under the sequence labeling setting. To test if the binary classification loss (E.q. 9 , sec:ours) could yield better-calibrated confidence, we perform one round of fine-tuning of the base model with the hinge loss ( $+$ Binary loss in tab:expmain). We show both the results of using the confidence (E.q. 7 ) of the fine-tuned model to rerank the extractions of the base model (Rerank Only), and the end-to-end performance of the fine-tuned model in assertion generation (Generate). We found both settings lead to improved performance compared to the base model, which demonstrates that calibrating confidence using binary classification loss can improve the performance of both reranking and assertion generation. Finally, our proposed iterative learning approach (alg:iter, sec:ours) significantly outperforms non-iterative settings. We also investigate the performance of our iterative learning algorithm with respect to the number of iterations in fig:iter.

Improving Open Information Extraction via Iterative Rank-Aware Learning	Open information extraction (IE) is the task of extracting open-domain assertions from natural language sentences. A key step in open IE is confidence modeling, ranking the extractions based on their estimated quality to adjust precision and recall of extracted assertions. We found that the extraction likelihood, a confidence measure used by current supervised open IE systems, is not well calibrated when comparing the quality of assertions extracted from different sentences. We propose an additional binary classification loss to calibrate the likelihood to make it more globally comparable, and an iterative learning process, where extractions generated by the open IE model are incrementally included as training samples to help the model learn from trial and error. Experiments on OIE2016 demonstrate the effectiveness of our method. Code and data are available at https://github.com/jzbjyb/oie_rank.

Within this framework, an assertion $\mathbf {r}$ can be mapped to a unique BIO  label sequence $\mathbf {y}$ by assigning $O$ to the words not contained in $\mathbf {r}$ , $B_{p}$ / $I_{p}$ to the words in $\mathbf {p}$ , and $B_{a_i}$ / $I_{a_i}$ to the words in $\mathbf {a}_i$ respectively, depending on whether the word is at the beginning or inside of the span. The label prediction $\hat{\mathbf {y}}$ is made by the model given a sentence associated with a predicate of interest $(\mathbf {s}, v)$ . At test time, we first identify verbs in the sentence as candidate predicates. Each sentence/predicate pair is fed to the model and extractions are generated from the label sequence.

The model obtained at each iteration is used to both rerank the extractions generated by the previous model and generate new extractions. We also report results of using only positive samples for optimization. We observe the AUC and F1 of both reranking and generation increases simultaneously for the first 6 iterations and converges after that, which demonstrates the effectiveness of iterative training. The best performing iteration achieves AUC of 0.125 and F1 of 0.315, outperforming all the baselines by a large margin. Meanwhile, using both positive and negative samples consistently outperforms only using positive samples, which indicates the necessity of exposure to the errors made by the system. tab:casererank compares extractions from RnnOIE before and after reranking. We can see the order is consistent with the annotation after reranking, showing the additional loss function's efficacy in calibrating the confidences; this is particularly common in extractions with long arguments. tab:casegen shows a positive extraction discovered after iterative training (first example), and a wrong extraction that disappears (second example), which shows that the model also becomes better at assertion generation. Why is the performance still relatively low?

Conclusion	We propose a binary classification loss function to calibrate confidences in open IE. Iteratively optimizing the loss function enables the model to incrementally learn from trial and error, yielding substantial improvement. An error analysis is performed to shed light on possible future directions.

Binary Classification Loss	To alleviate the problem of incomparable confidences across sentences, we propose a simple binary classification loss to calibrate confidences to be globally comparable. Given a model $\theta ^\prime $ trained with MLE, beam search is performed to generate assertions with the highest probabilities for each predicate. Assertions are annotated as either positive or negative with respect to the gold standard, and are used as training samples to minimize the hinge loss:  $$\hspace{-2.84526pt}\hat{\theta } = \underset{\theta }{\operatornamewithlimits{arg\,min}}\hspace{-8.53581pt}\underset{\begin{array}{c}\mathbf {s} \in \mathcal {D}\\ v, \hat{\mathbf {y}} \in g_{\theta ^\prime }(\mathbf {s})\end{array}}{\operatorname{\mathbb {E}}}\hspace{-11.38109pt}\max {(0,1-t \cdot c_{\theta }(\mathbf {s}, v, \hat{\mathbf {y}}))},$$   (Eq.

We randomly sample 50 extractions generated at the best performing iteration and conduct an error analysis to answer this question. To count as a correct extraction, the number and order of the arguments should be exactly the same as the ground truth and syntactic heads must be included, which is challenging considering that the OIE2016 dataset has complex syntactic structures and multiple arguments per predicate. We classify the errors into three categories and summarize their proportions in tab:err. “Overgenerated predicate” is where predicates not included in ground truth are overgenerated, because all the verbs are used as candidate predicates. An effective mechanism should be designed to reject useless candidates. “Wrong argument” is where extracted arguments do not coincide with ground truth, which is mainly caused by merging multiple arguments in ground truth into one. “Missing argument” is where the model fails to recognize arguments. These two errors usually happen when the structure of the sentence is complicated and coreference is involved. More linguistic information should be introduced to solve these problems.

Experimental Settings	We use the OIE2016 dataset  to evaluate our method, which only contains verbal predicates. OIE2016 is automatically generated from the QA-SRL dataset  , and to remove noise, we remove extractions without predicates, with less than two arguments, and with multiple instances of an argument. The statistics of the resulting dataset are summarized in tab:data. We follow the evaluation metrics described by Stanovsky:2016:OIE2016: area under the precision-recall curve (AUC) and F1 score. An extraction is judged as correct if the predicate and arguments include the syntactic head of the gold standard counterparts. We compare our method with both competitive neural and non-neural models, including RnnOIE  , OpenIE4, ClausIE  , and PropS  . Our implementation is based on AllenNLP  by adding binary classification loss function on the implementation of RnnOIE. The network consists of 4 BiLSTM layers (2 forward and 2 backward) with 64-dimensional hidden units. ELMo  is used to map words into contextualized embeddings, which are concatenated with a 100-dimensional predicate indicator embedding.

Acknowledgements	This work was supported in part by gifts from Bosch Research, and the Carnegie Bosch Institute.

Iterative Rank-Aware Learning	In this section, we describe our proposed binary classification loss and iterative learning procedure.

9)  where $\mathcal {D}$ is the training sentence collection, $g_{\theta ^\prime }$ represents the candidate generation process, and $t \in \lbrace 1,-1\rbrace $ is the binary annotation. $c_{\theta }(\mathbf {s}, v, \hat{\mathbf {y}})$ is the confidence score calculated by average log probability of the label sequence. The binary classification loss distinguishes positive extractions from negative ones generated across different sentences, potentially leading to a more reliable confidence measure and better ranking performance.

\forall \mathbf {s} \in \mathcal {D}\rbrace $   $\theta ^{(t+1)} \leftarrow \underset{\theta }{\operatornamewithlimits{arg\,min}}\hspace{-8.53581pt}\underset{(\mathbf {s}, v, \hat{\mathbf {y}})\in \mathcal {E}}{\operatorname{\mathbb {E}}}\hspace{-8.53581pt}\max {(0,1-t \cdot c_{\theta }(\mathbf {s}, v, \hat{\mathbf {y}}))}$   $t \leftarrow t+1$ Iterative learning.

The recurrent dropout probability is set to 0.1. Adadelta  with $\epsilon =10^{-6}$ and $\rho =0.95$ and mini-batches of size 80 are used to optimize the parameters. Beam search size is 5.

At decoding time, we use the Viterbi algorithm to reject invalid label transitions  , such as $B_{a_2}$ followed by $I_{a_1}$ . We use average log probability of the label sequence  as its confidence:  $$c(\mathbf {s}, v, \hat{\mathbf {y}}) = \frac{\sum _{t=1}^{|\mathbf {s}|}{\log {P(\hat{y_t}|\mathbf {s}, v)}}}{|\mathbf {s}|}.$$   (Eq. 7)  The probability is trained with maximum likelihood estimation (MLE) of the gold extractions. This formulation lacks an explicit concept of cross-sentence comparison, and thus incorrect extractions of one sentence could have higher confidence than correct extractions of another sentence.

Question: What's the input representation of OpenIE tuples into the model?
Answer: word embeddings