SlowGuess/ACL_2020_OCR huggingface dataset:
IMOJIE: Iterative Memory-Based Joint Open Information Extraction
Keshav Kolluru<sup>1</sup>, Samarth Aggarwal<sup>1</sup>, Vipul Rathore<sup>1</sup>, Mausam<sup>1</sup>, and Soumen Chakrabarti<sup>2</sup>
$^{1}$  Indian Institute of Technology Delhi
keshav.kolluru@gmail.com, samarth.aggarwal.2510@gmail.com  
rathorevipul28@gmail.com, mausam@cse.iitd.ac.in
$^{2}$  Indian Institute of Technology Bombay
soumen@cse.iitb.ac.in
While traditional systems for Open Information Extraction were statistical and rule-based, recently neural models have been introduced for the task. Our work builds upon CopyAttention, a sequence generation OpenIE model (Cui et al., 2018). Our analysis reveals that CopyAttention produces a constant number of extractions per sentence, and its extracted tuples often express redundant information.
We present IMOJIE, an extension to CopyAttention, which produces the next extraction conditioned on all previously extracted tuples. This approach overcomes both shortcomings of CopyAttention, resulting in a variable number of diverse extractions per sentence. We train IMOJIE on training data bootstrapped from extractions of several non-neural systems, which have been automatically filtered to reduce redundancy and noise. IMOJIE outperforms CopyAttention by about 18 F1 pts, and a BERT-based strong baseline by 2 F1 pts, establishing a new state of the art for the task.
Extracting structured information from unstructured text has been a key research area within NLP. The paradigm of Open Information Extraction (OpenIE) (Banko et al., 2007) uses an open vocabulary to convert natural text to semi-structured representations, by extracting a set of (subject, relation, object) tuples. OpenIE has found wide use in many downstream NLP tasks (Mausam, 2016) like multi-document question answering and summarization (Fan et al., 2019), event schema induction (Balasubramanian et al., 2013) and word embedding generation (Stanovsky et al., 2015).
Traditional OpenIE systems are statistical or rule-based. They are largely unsupervised in nature, or bootstrapped from extractions made by earlier systems. They often consist of several components
like POS tagging, and syntactic parsing. To bypass error accumulation in such pipelines, end-to-end neural systems have been proposed recently.
Recent neural OpenIE methods belong to two categories: sequence labeling, e.g., RnnOIE (Stanovsky et al., 2018) and sequence generation, e.g., CopyAttention (Cui et al., 2018). In principle, generation is more powerful because it can introduce auxiliary words or change word order. However, our analysis of CopyAttention reveals that it suffers from two drawbacks. First, it does not naturally adapt the number of extractions to the length or complexity of the input sentence. Second, it is susceptible to stuttering: extraction of multiple triples bearing redundant information.
These limitations arise because its decoder has no explicit mechanism to remember what parts of the sentence have already been 'consumed' or what triples have already been generated. Its decoder uses a fixed-size beam for inference. However, beam search can only ensure that the extractions are not exact duplicates.
In response, we design the first neural OpenIE system that uses sequential decoding of tuples conditioned on previous tuples. We achieve this by adding every generated extraction so far to the encoder. This iterative process stops when the EndOfExtractions tag is generated by the decoder, allowing it to produce a variable number of extractions. We name our system Iterative MemOry Joint Open Information Extraction (IMoJIE).
CopyAttention uses a bootstrapping strategy, where the extractions from OpenIE-4 (Christensen et al., 2011; Pal and Mausam, 2016) are used as training data. However, we believe that training on extractions of multiple systems is preferable. For example, OpenIE-4 benefits from high precision compared to ClausIE (Del Corro and Gemulla, 2013), which offers high recall. By aggregating extractions from both, IMOJIE could potentially
5871
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5871-5886 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Sentence	He was appointed Commander of the Order of the British Empire in the 1948 Queen’s Birthday Honours and was knighted in the 1953 Coronation Honours.
CopyAttention	(He;was appointed;Commander ...Birthday Honours)
(He;was appointed;Commander ...Birthday Honours and was knighted ...Honours)
(Queen’s Birthday Honours;was knighted ;in the 1953 Coronation Honours)
(He;was appointed;Commander of the Order of the British Empire in the 1948 )
(the 1948;was knighted ;in the 1953 Coronation Honours)
IMOJIE	(He;was appointed;Commander of the Order ...Birthday Honours)
(He;was knighted ;in the 1953 Coronation Honours)
Table 1: IMOJIE vs. CopyAttention. CopyAttention suffers from stuttering, which IMOJIE does not.
Sentence	Greek and Roman pagans , who saw their relations with the gods in political and social terms , scorned the man who constantly trembled with fear at the thought of the gods , as a slave might fear a cruel and capricious master .
OpenIE-4	( the man ; constantly trembled ; )
IMOJIE	( a slave ; might fear ; a cruel and capricious master )
( Greek and Roman pagans ; scorned ; the man who … capricious master )
( the man ; constantly trembled ; with fear at the thought of the gods )
( Greek and Roman pagans ; saw ; their relations with the gods in political and social terms )
Table 2: IMOJIE vs. OpenIE-4. Pipeline nature of OpenIE-4 can get confused by long convoluted sentences, but IMOJIE responds gracefully.
obtain a better precision-recall balance.
However, simply concatenating extractions from multiple systems does not work well, as it leads to redundancy as well as exaggerated noise in the dataset. We devise an unsupervised Score-and-Filter mechanism to automatically select a subset of these extractions that are non-redundant and expected to be of high quality. Our approach scores all extractions with a scoring model, followed by filtering to reduce redundancy.
We compare IMOJIE against several neural and non-neural systems, including our extension of CopyAttention that uses BERT (Devlin et al., 2019) instead of an LSTM at encoding time, which forms a very strong baseline. On the recently proposed CaRB metric, which penalizes redundant extractions (Bhardwaj et al., 2019), IMOJIE outperforms CopyAttention by about 18 pts in F1 and our strong BERT baseline by 2 pts, establishing a new state of the art for OpenIE. We release IMOJIE & all related resources for further research<sup>1</sup>. In summary, our contributions are:
- We propose IMOJIE, a neural OpenIE system that generates the next extraction, fully conditioned on the extractions produced so far. IMOJIE produce a variable number of diverse extractions for a sentence,
- We present an unsupervised aggregation scheme to bootstrap training data by combining extractions from multiple OpenIE systems.
- IMOJIE trained on this data establishes a new
SoTA in OpenIE, beating previous systems and also our strong BERT-baseline.
Open Information Extraction (OpenIE) involves extracting (arg1 phrase, relation phrase, arg2 phrase) assertions from a sentence. Traditional open extractors are rule-based or statistical, e.g., Textrunner (Banko et al., 2007), ReVerb (Fader et al., 2011; Etzioni et al., 2011), OLLIE (Mausam et al., 2012), Stanford-IE (Angeli et al., 2015), ClausIE (Del Corro and Gemulla, 2013), OpenIE-4 (Christensen et al., 2011; Pal and Mausam, 2016), OpenIE-5 (Saha et al., 2017, 2018), PropS (Stanovsky et al., 2016), and MinIE (Gashteovski et al., 2017). These use syntactic or semantic parsers combined with rules to extract tuples from sentences.
Recently, to reduce error accumulation in these pipeline systems, neural OpenIE models have been proposed. They belong to one of two paradigms: sequence labeling or sequence generation.
Sequence Labeling involves tagging each word in the input sentence as belonging to the subject, predicate, object or other. The final extraction is obtained by collecting labeled spans into different fields and constructing a tuple. RnnOIE (Stanovsky et al., 2018) is a labeling system that first identifies the relation words and then uses sequence labelling to get their arguments. It is trained on OIE2016 dataset, which postprocesses SRL data for OpenIE (Stanovsky and Dagan, 2016).
https://github.com/dair-iitd/imojie
5872
Figure 1: One step of the sequential decoding process, for generating the  $i^{\text{th}}$  extraction, which takes the original sentence and all extractions numbered  $1, \ldots, i - 1$ , previously generated, as input.
SenseOIE (Roy et al., 2019), improves upon Rn-nOIE by using the extractions of multiple OpenIE systems as features in a sequence labeling setting. However, their training requires manually annotated gold extractions, which is not scalable for the task. This restricts SenseOIE to train on a dataset of 3,000 sentences. In contrast, our proposed Score-and-Filter mechanism is unsupervised and can scale unboundedly. Jiang et al. (2019) is another labeling system that better calibrates extractions across sentences.
SpanOIE (Zhan and Zhao, 2020) uses a span selection model, a variant of the sequence labelling paradigm. Firstly, the predicate module finds the predicate spans in a sentence. Subsequently, the argument module outputs the arguments for this predicate. However, SpanOIE cannot extract nominal relations. Moreover, it bootstraps its training data over a single OpenIE system only. In contrast, IMoJIE overcomes both of these limitations.
Sequence Generation uses a Seq2Seq model to generate output extractions one word at a time. The generated sequence contains field demarcators, which are used to convert the generated flat sequence to a tuple. CopyAttention (Cui et al., 2018) is a neural generator trained over bootstrapped data generated from OpenIE-4 extractions on a large corpus. During inference, it uses beam search to get the predicted extractions. It uses a fixed-size beam, limiting it to output a constant number of extractions per sentence. Moreover, our analysis shows that CopyAttention extractions severely lack in diversity, as illustrated in Table 1.
Sun et al. (2018) propose the Logician model, a restricted sequence generation model for extracting tuples from Chinese text. Logician relies on coverage attention and gated-dependency attention, a language-specific heuristic for Chinese. Using coverage attention, the model also tackles generation of multiple extractions while being globally-aware.
We compare against Logician's coverage attention as one of the approaches for increasing diversity.
Sequence-labeling based models lack the ability to change the sentence structure or introduce new auxiliary words while uttering predictions. For example, they cannot extract (Trump, is the President of, US) from "US President Trump", since 'is', 'of' are not in the original sentence. On the other hand, sequence-generation models are more general and, in principle, need not suffer from these limitations.
Evaluation: All neural models have shown improvements over the traditional systems using the OIE2016 benchmark. However, recent work shows that the OIE2016 dataset is quite noisy, and that its evaluation does not penalize highly redundant extractions (Léchelle et al., 2018). In our work, we use the latest CaRB benchmark, which crowdsources a new evaluation dataset, and also provides a modified evaluation framework to downscore near-redundant extractions (Bhardwaj et al., 2019).
3 Sequential Decoding
We now describe IMOJIE, our generative approach that can output a variable number of diverse extractions per sentence. The architecture of our model is illustrated in Figure 1. At a high level, the next extraction from a sentence is best determined in context of all other tuples extracted from it so far. Hence, IMOJIE uses a decoding strategy that generates extractions in a sequential fashion, one after another, each one being aware of all the ones generated prior to it.
This kind of sequential decoding is made possible by the use of an iterative memory. Each of the generated extractions are added to the memory so that the next iteration of decoding has access to all of the previous extractions. We simulate this iterative memory with the help of BERT encoder, whose input includes the [CLS] token and original
5873
Figure 2: Ranking-Filtering subsystem for combining extractions from multiple open IE systems in an unsupervised fashion. ('Exts'=extractions.)
sentence appended with the decoded extractions so far, punctuated by the separator token [SEP] before each extraction.
IMOJIE uses an LSTM decoder, which is initialized with the embedding of [CLS] token. The contextualized-embeddings of all the word tokens are used for the Copy (Gu et al., 2016) and Attention (Bahdanau et al., 2015) modules. The decoder generates the tuple one word at a time, producing  $\langle rel\rangle$  and  $\langle obj\rangle$  tokens to indicate the start of relation and object respectively. The iterative process continues until the EndOfExtractions token is generated.
The overall process can be summarized as:
1. Pass the sentence through the Seq2Seq architecture to generate the first extraction.
2. Concatenate the generated extraction with the existing input and pass it again through the Seq2Seq architecture to generate the next extraction.
3. Repeat Step 2 until the EndOfExtractions token is generated.
IMOJIE is trained using a cross-entropy loss between the generated output and the gold output.
4 Aggregating Bootstrapped Data
4.1 Single Bootstrapping System
To train generative neural models for the task of OpenIE, we need a set of sentence-extraction pairs. It is ideal to curate such a training dataset via human annotation, but that is impractical, considering the scale of training data required for a neural model. We follow Cui et al. (2018), and use bootstrapping — using extractions from a pre-existing OpenIE system as 'silver'-labeled (as distinct from 'gold'-labeled) instances to train the neural model. We first order all extractions in the decreasing order of confidence output by the original system. We then construct training data in IMOJIE's input-output format, assuming that this is the order in which it should produce its extractions.
4.2 Multiple Bootstrapping Systems
Different OpenIE systems have diverse quality characteristics. For example, the human-estimated (precision, recall) of OpenIE-4 is (61, 43) while that of ClausIE is (40, 50). Thus, by using their combined extractions as the bootstrapping dataset, we might potentially benefit from the high precision of OpenIE-4 and high recall of ClausIE.
However, simply pooling all extractions would not work, because of the following serious hurdles.
No calibration: Confidence scores assigned by different systems are not calibrated to a comparable scale.
Redundant extractions: Beyond exact duplicates, multiple systems produce similar extractions with low marginal utility.
Wrong extractions: Pooling inevitably pollutes the silver data and can amplify incorrect instances, forcing the downstream open IE system to learn poor-quality extractions.
We solve these problems using a Score-and-Filter framework, shown in Figure 2.
Scoring: All systems are applied on a given sentence, and the pooled set of extractions are scored such that good (correct, informative) extractions generally achieve higher values compared to bad (incorrect) and redundant ones. In principle, this score may be estimated by the generation score from IMOJIE, trained on a single system. In practice, such a system is likely to consider extractions similar to its bootstrapping training data as good, while disregarding extractions of other systems, even though those extractions may also be of high quality. To mitigate this bias, we use an IMOJIE model, pre-trained on a random bootstrapping dataset. The random bootstrapping dataset is generated by picking extractions for each sentence randomly from any one of the bootstrapping systems being aggregated. We assign a score to each extraction in the pool based on the confidence value given to it by this IMOJIE (Random) model.
Filtering: We now filter this set of extractions for
5874
redundancy. Given the set of ranked extractions in the pool, we wish to select that subset of extractions that have the best confidence scores (assigned by the random-boostrap model), while having minimum similarity to the other selected extractions.
We model this goal as the selection of an optimal subgraph from a suitably designed complete weighted graph. Each node in the graph corresponds to one extraction in the pool. Every pair of nodes  $(u,v)$  are connected by an edge. Every edge has an associated weight  $R(u,v)$  signifying the similarity between the two corresponding extractions. Each node  $u$  is assigned a score  $f(u)$  equal to the confidence given by the random-bootstrap model.
Given this graph  $G = (V, E)$  of all pooled extractions of a sentence, we aim at selecting a subgraph  $G' = (V', E')$  with  $V' \subseteq V$ , such that the most significant ones are selected, whereas the extractions redundant with respect to already-selected ones are discarded. Our objective is
$$
\max  _ {G ^ {\prime} \subseteq G} \sum_ {i = 1} ^ {| V ^ {\prime} |} f (u _ {i}) - \sum_ {j = 1} ^ {| V ^ {\prime} | - 1} \sum_ {k = j + 1} ^ {| V ^ {\prime} |} R \left(u _ {j}, u _ {k}\right), \tag {1}
$$
where  $u_{i}$  represents node  $i \in V'$ . We compute  $R(u,v)$  as the ROUGE2 score between the serialized triples represented by nodes  $u$  and  $v$ . We can intuitively understand the first term as the aggregated sum of significance of all selected triples and second term as the redundancy among these triples.
If  $G$  has  $n$  nodes, we can pose the above objective as:
$$
\max  _ {\boldsymbol {x} \in \{0, 1 \} ^ {n}} \boldsymbol {x} ^ {\top} \boldsymbol {f} - \boldsymbol {x} ^ {\top} \boldsymbol {R} \boldsymbol {x}, \tag {2}
$$
where  $\pmb{f} \in \mathbb{R}^n$  representing the node scores, i.e.,  $f[i] = f(u_i)$ , and  $\pmb{R} \in \mathbb{R}^{n \times n}$  is a symmetric matrix with entries  $R_{j,k} = \mathrm{ROUGE2}(u_j, u_k)$ .  $\pmb{x}$  is the decision vector, with  $x[i]$  indicating whether a particular node  $u_i \in V'$  or not. This is an instance of Quadratic Boolean Programming and is NP-hard, but in our application  $n$  is modest enough that this is not a concern. We use the QPBO (Quadratic Pseudo Boolean Optimizer) solver $^2$  (Rother et al., 2007) to find the optimal  $\pmb{x}^*$  and recover  $V'$ .
5.1 Training Data Construction
We obtain our training sentences by scraping Wikipedia, because Wikipedia is a comprehensive source of informative text from diverse domains,
rich in entities and relations. Using sentences from Wikipedia ensures that our model is not biased towards data from any single domain.
We run OpenIE-4 $^3$ , ClausIE $^4$  and RnnOIE $^5$  on these sentences to generate a set of OpenIE tuples for every sentence, which are then ranked and filtered using our Score-and-Filter technique. These tuples are further processed to generate training instances in IMOJIE's input-output format.
Each sentence contributes to multiple (input, output) pairs for the IMOJIE model. The first training instance contains the sentence itself as input and the first tuple as output. For example, ("I ate an apple and an orange.", "I; ate; an apple"). The next training instance, contains the sentence concatenated with previous tuple as input and the next tuple as output ("I ate an apple and an orange. [SEP] I; ate; an apple", "I; ate; an orange"). The final training instance generated from this sentence includes all the extractions appended to the sentence as input and EndOfExtractions token as the output. Every sentence gives the seq2seq learner one training instance more than the number of tuples.
While forming these training instances, the tuples are considered in decreasing order of their confidence scores. If some OpenIE system does not provide confidence scores for extracted tuples, then the output order of the tuples may be used.
5.2 Dataset and Evaluation Metrics
We use the CaRB data and evaluation framework (Bhardwaj et al., 2019) to evaluate the systems at different confidence thresholds, yielding a precision-recall curve. We identify three important summary metrics from the P-R curve.
Optimal F1: We find the point in the P-R curve corresponding to the largest F1 value and report that. This is the operating point for getting extractions with the best precision-recall trade-off.
AUC: This is the area under the P-R curve. This metric is useful when the downstream application can use the confidence value of the extraction.
Last F1: This is the F1 score computed at the point of zero confidence. This is of importance when we cannot compute the optimal threshold, due to lack of any gold-extractions for the domain.
<sup>3</sup>https://github.com/knowitall/openie
<sup>4</sup>https://www.mpi-inf.mpg.de/clausie
<sup>5</sup>https://github.com/gabrielStanovsky/supervised-oie
Our reported CaRB scores for OpenIE-4 and OpenIE-5 are slightly different from those reported by Bhardwaj et al. (2019). The authors of CaRB have verified our values.
$^{2}$ https://pypi.org/project/thinqpbo/
5875
System	Metric
Opt. F1	AUC	Last F1
Stanford-IE	23	13.4	22.9
OllIE	41.1	22.5	40.9
PropS	31.9	12.6	31.8
MinIE	41.9	-*	41.9
OpenIE-4	51.6	29.5	51.5
OpenIE-5	48.5	25.7	48.5
ClausIE	45.1	22.4	45.1
CopyAttention	35.4	20.4	32.8
RNN-OIE	49.2	26.5	49.2
Sense-OIE	17.2	-*	17.2
Span-OIE	47.9	-*	47.9
CopyAttention + BERT	51.6	32.8	49.6
IMoJIE	53.5	33.3	53.3
Many downstream applications of OpenIE, such as text comprehension (Stanovsky et al., 2015) and sentence similarity estimation (Christensen et al., 2014), use all the extractions output by the OpenIE system. Last F1 is an important measure for such applications.
5.3 Comparison Systems
We compare IMoJIE against several nonneural baselines, including Stanford-IE, OpenIE-4, OpenIE-5, ClausIE, PropS, MinIE, and OLLIE. We also compare against the sequence labeling baselines of RnnOIE, SenseOIE, and the span selection baseline of SpanOIE. Probably the most closely related baseline to us is the neural generation baseline of CopyAttention. To increase CopyAttention's diversity, we compare against an English version of Logician, which adds coverage attention to a single-decoder model that emits all extractions one after another. We also compare against CopyAttention augmented with diverse beam search (Vijayakumar et al., 2018) — it adds a diversity term to the loss function so that new beams have smaller redundancy with respect to all previous beams.
Finally, because our model is based on BERT, we reimplement CopyAttention with a BERT encoder — this forms a very strong baseline for our task.
5.4 Implementation
We implement IMOJIE in the AllenNLP framework $^{7}$  (Gardner et al., 2018) using Pytorch 1.2. We use “BERT-small” model for faster training. Other
Table 3: Comparison of various OpenIE systems - nonneural, neural and proposed models. (*) Cannot compute AUC as Sense-OIE, MinIE do not emit confidence values for extractions and released code for Span-OIE does not provision calculation of confidence values. In these cases, we report the Last F1 as the Opt. F1
System	Metric
Opt. F1	AUC	Last F1
CopyAttention	35.4	20.4	32.8
CoverageAttention	41.8	22.1	41.8
CoverageAttention+BERT	47.9	27.9	47.9
Diverse Beam Search	46.1	26.1	39.6
IMOJIE (w/o BERT)	37.9	19.1	36.6
IMOJIE	53.2	33.1	52.4
Table 4: Models to solve the redundancy issue prevalent in Generative Neural OpenIE systems. All systems are bootstrapped on OpenIE-4.
Bootstrapping Systems	Metric
Opt. F1	AUC	Last F1
ClausIE	49.2	31.4	45.5
RnnOIE	51.3	31.1	50.8
OpenIE-4	53.2	33.1	52.4
OpenIE-4+ClausIE	51.5	32.5	47.1
OpenIE-4+RnnOIE	53.1	32.1	53.0
ClausIE+RnnOIE	50.9	32.2	49.8
All	53.5	33.3	53.3
Table 5: IMOJIE trained with different combinations of bootstrapping data from 3 systems - OpenIE-4, ClausIE, RNNOIE. Graph filtering is not used over single datasets.