SlowGuess/NAACL_2022_OCR huggingface dataset:
On the more coarse-grained CaRB dataset, almost all OpenIE systems achieve comparable performance in terms of overall F1 using the CaRB scoring function. The neural systems still outperform non-neural systems in terms of F1, which is in line with previous studies. However, neural OpenIE systems are tuned based on the CaRB scoring function and thus tend to produce extractions that are biased towards this scoring method. Previous works (Kolluru et al., 2020a) also report issues with the scoring function not being able to handle conjunctions properly. Table 7 shows the limitations of the CaRB benchmark and scoring function through an example. As illustrated, the set of extractions produced by COMPACTIE is more exhaustive than IMoJIE and ground truth extractions. However, the CaRB scoring function assigns an F1 score of 62.0 to IMoJIE extractions, and 39.7 to COMPACTIE extractions. To resolve incompleteness of the CaRB benchmark and potential bias towards its scoring function, we undertake a fact-centered evaluation, detailed in Section 6.2, and a manual evaluation, described in Section 6.3.
6.2 Fact-centric Evaluation
(Gashteovski et al., 2021) claims that CaRB and Wire57 benchmarks and scoring functions overestimate a system's ability to extract correct facts. They propose an alternative benchmark and evaluation framework, BenchIE, that exhaustively lists all fact-equivalent extractions and clusters them into fact synsets. The scoring function considers an extraction as correct, if and only if it exactly matches any of the gold extractions from any of the fact synsets. They report Precision, Recall, and F1 based on exact triple matching.
Table 5 shows the performance of different OpenIE systems on BenchIE. As shown, COMPACTIE outperforms all other systems except MinIE. We found that MinIE aims to exhaustively produce different representations of the same fact. In contrast, COMPACTIE follows the setup of neural OpenIE systems and encourages at most one repre
<sup>4</sup>https://github.com/dair-iitd/CaRB
<sup>5</sup>https://github.com/rali-udem/WiRe57
System	Wire57	CaRB
P	R	F1	ACL	NCC	RPA	P	R	F1	ACL	NCC	RPA
NestIE	35.0	15.0	21.0	4.65	0.07	1.16	53.4	32.8	40.6	4.29	0.08	1.21
MinIE	31.3	30.7	31.0	4.93	0.2	1.6	35.3	50.5	41.6	4.97	0.4	1.57
IMoJIE	41.2	20.1	27.0	6.23	0.26	1.07	48.5	44.6	46.5	6.43	0.39	1.08
OpenIE6	27.7	19.4	22.8	5.98	0.66	1.14	44.3	44.5	44.4	6.26	0.56	1.29
Multi2OIE	33.4	18.9	24.1	5.54	0.42	1.05	48.2	44.5	46.3	6.06	0.42	1.08
COMPACTIE	41.4	25.8	31.8	5.23	0.05	1.37	51.3	39.9	45.0	5.08	0.07	1.32
Table 3: Performance of OpenIE systems on Wire57 and CaRB datasets. The three analytic metrics (ACL, NCC, RPA) are discussed in Section 7.
System	Precision	Compactness
NestIE	49.1 (84/171)	98.8 (83/84)
MinIE	58.0 (217/374)	78.8 (171/217)
IMoJIE	90.0 (156/173)	53.2 (83/156)
OpenIE6	78.0 (210/269)	65.2 (137/210)
Multi2OIE	78.6 (151/192)	59.6 (90/151)
COMPACTIE	75.8 (175/231)	94.9 (166/175)
sensation per fact. As a result, MinIE produces  $1.36 \times$  more extractions than COMPACTIE, achieving much higher recall than its neural counterparts.
6.3 Manual Evaluation
Limitations in the aforementioned benchmarks and evaluation frameworks encouraged us to perform human evaluation on triples generated by various systems. To this end, we randomly select 100 sentences from the CaRB validation set and feed them to all systems to investigate the generated triples. Next, we ask two graduate CS students, blind to the OpenIE systems, to mark each triple for correctness (0 or 1) based on whether it is asserted in the text and correctly captures the semantic information. They also label extractions for compactness (0 or 1). We consider an extraction compact if none of its constituents is longer than 10 words, includes conjunction or can be an independent extraction. We found an inter-annotator agreement of 0.68 on correctness and 0.83 on compactness using the Cohens Kappa metric. We report the results of the manual evaluation in Table 4. Neural systems target informativeness, which results in high precision at the cost of compactness. On the other hand, non-neural systems that aim for compact triples suffer from low precision. COMPACTIE offers a better trade
Table 4: Manual evaluation of OpenIE systems on CaRB validation set. Precision indicates the percentage of correct extractions. Compactness indicates the percentage of compact extractions amongst the correct ones.
System	BenchIE
P	R	F1
NestIE	37.1	10.2	16.0
MinIE	42.9	27.8	33.7
IMoJIE	34.3	12.8	18.6
OpenIE6	31.1	21.4	25.3
Multi2OIE	39.2	16.1	22.8
COMPACTIE	40.3	19.0	26.2
Table 5: Performances of OpenIE systems on the BenchIE dataset.
off between precision and compactness. It achieves comparable precision to neural models  $(-6\%)$  while providing substantially more compact extractions  $(+36\%)$ . Compared to the MinIE, COMPACTIE produces triples with significantly higher precision  $(+22\%)$  while producing a comparable number of compact triples. NestIE achieves comparable compactness rate to COMPACTIE but suffers from low precision and total number of extractions.
7.1 Compact and Overlapping Constituents
To understand the performance of COMPACTIE in generating compact triples that share constituents, we introduce the following metrics:
- Average Constituent Length (ACL): average length of constituents across all system-generated triples. This is a "syntactic" measure of compactness. The lower the ACL score, the higher the compactness of triples.
- Number of Constituent Clauses (NCC): average number of clauses per constituent that could be extracted as independent triples. The lower the NCC score, the better the compactness of triples.
- Repetitions Per Argument (RPA): number of total arguments divided by the number of unique arguments. The higher the RPA score, the higher fraction of total constituents produced per sen
tence are shared.
Table 3 summarizes the performance on these metrics over CaRB and Wire57 benchmarks. We do not conduct a separate analysis over BenchIE since it uses a subset of CaRB sentences. As shown, the ACL scores of COMPACTIE are significantly lower than its neural counterparts and closely follows MinIE. The average constituent length (ACL) of NestIE triples is the lowest since it breaks sentences into small triples with verb, noun, preposition, and adjective mediated relations. For instance, the sentence: "2 million people died of AIDS." is broken down into T1: (2 million people; died), and T2: (T1; of; AIDS). However, its fine-grained strategy greatly sacrifices F1 for compactness. COMPACTIE achieves the lowest NCC score which indicates that the constituents in triples contain the fewest verbal clauses. As a result, these triples are more suitable for downstream applications such as text summarization and knowledge-base construction than other counterparts.
Finally, high RPA scores of COMPACTIE demonstrate the effectiveness of our approach as it enables the system to reuse the same constituent to generate multiple triples. MinIE achieves a slightly higher RPA score than COMPACTIE since it extracts multiple triples to represent the same fact leading to a higher repetition of unique constituents.
7.2 Effectiveness of Design Choices
Pipelined Approach vs. Unified Table Filling. To compare our pipelined approach with a unified extraction model, we follow UniRE (Wang et al., 2021), which decodes a single table to identify entities and relations jointly. We follow their 3-step decoding algorithm to obtain the constituents and links between them from the same table (with the schema shown in Figure 2). We refer to this model as COMPACTIE<sub>uni</sub>. We report the performances in Table 6 and show that performance drops by jointly training the constituent and linking model. This aligns with the observations in recent entity-relation extraction work that pipelined approaches are more effective than joint models.
Effectiveness of Schema Design. Our table schema for constituent extraction includes both labels for constituents as well as labels to link them. We argued that this design captures the contextual dependency information between the constituents that boosts extraction performance. We compare the effectiveness of this schema design to
Method	Wire57	CaRB
COMPACTIE	31.8	45.0
COMPACTIEuni	17.6	35.8
COMPACTIEconst table	26.0	40.1
Table 6: Comparing F1 scores of CompactIE against joint extraction systems.
another schema that uses only constituent labels  $Y_{c} : \{ \text{Argument}, \text{Predicate} \} \cup \text{None}$ . Note that we use the same constituent linking model to obtain triples from the extracted constituents. We refer to this setting as COMPACTIE_const table. Table 6 illustrates the performance of this system on both CaRB and Wire57 datasets. We find that COMPACTIE achieves significantly higher F1 compared to COMPACTIE_const table and conclude that incorporating additional context in the table schema improves the performance of the constituent extraction model.
7.3 Error Analysis
We examine COMPACTIE triples produced for 50 randomly selected sentences of the CaRB validation dataset and 20 randomly selected sentences of the Wire57 dataset. Upon close analysis, we identify five major sources of error:
Constituent Not Found:  $(49.29\%)$  We find that the constituent extraction model can fail to correctly label the constituents in the table. We found that the model gets biased towards producing None labels due to the imbalanced distribution of labels.
Wrong Relation Type: (28.17%) These involve errors where the constituent linking model fails to correctly predict the link between the constituents. The current model encodes one sentence per predicate to find its arguments. Alternatively, we can encode one sentence per predicate-argument pair to focus more on each relation. Relation labels in the constituent extraction model can also assist the linking model in predicting the correct relations. We reserve this issue for future work.
Boundary Detection Error: (11.26%) These include errors where the decoder in constituent extraction fails to correctly identify the boundaries of the constituents. Boundary detection in constituent extraction model is highly dependant on the choice of distance threshold  $(\alpha)$ , as explained in A.2, which limits its robustness.
Inexpressive Table Error: (7.04%) These include errors where constituents have overlapping spans that participate in two roles within the same extrac
System	Subject	Predicate	Object	F1
Gold	Applications	use this service to record	activity for a system
other OSIDs	use the service to record	data	-
other OSIDs	use the service to record data	during analysis
IMojIE	Applications	use this service to record	activity for a system
other OSIDs	use	the service to record data during ... analysis	62.0
COMPACTIE	Applications	use	this service to record activity for a system
other OSIDs	use	service to record data during development
other OSIDs	use	the service	39.7
the service	record	data during debugging
the service	record	data during analysis
Table 7: Gold, IMoJIE and COMPACTIE extractions for the sentence: "Applications use this service to record activity for a system while other OSIDs use the service to record data during development, debugging, or analysis." and their CaRB F1 score that evaluates extractors triples against gold triples.
tion or two different extractions.
Less than  $4.22\%$  of the errors were because of incorrect constituent type predictions. This indicates the effectiveness of our table filling method on constituent type detection.
OpenIE has been studied extensively for over a decade with a history of statistical and rule-based systems (Banko et al., 2007; Fader et al., 2011; Corro and Gemulla, 2013; Mausam et al., 2012; Angeli et al., 2015) that extract triples from sentences without using any training data. Recently, neural models have been developed that are trained end-to-end on extractions bootstrapped from previous OpenIE systems. These can broadly be classified into labeling-based and generation-based systems.
Labeling-based systems (Stanovsky et al., 2018; Kolluru et al., 2020a; Ro et al., 2020) tag each word in the sentence and construct triples in an auto-regressive manner or by using a unique predicate for each triple. Generation-based systems (Kolluru et al., 2020b; Bhutani et al., 2019) use a sequence-to-sequence model to generate triples one word at a time. Labeling-based systems can handle redundancy in extracted triples and are faster than generation-based systems (Kolluru et al., 2020a).
Compactness in OpenIE: There has been prior work (Bhutani et al., 2016; Gashteovski et al., 2017; Stanovsky and Dagan, 2016; Angeli et al., 2015) that focuses on finding compact triples and shows that concise triples are useful in several semantic tasks. However, recent studies (Léchelle et al., 2018; Gashteovski et al., 2020) indicate that neural OpenIE systems produce more specific triples with additional information than conventional OpenIE systems and are harder to align with existing knowledge bases. Therefore, we focus on designing a neural OpenIE system that extracts compact triples.
Grid Labeling: Also known as table filling, grid labeling has been recently applied to entity relation extraction (Gupta et al., 2016; Wang et al., 2021) and open information extraction tasks (Kolluru et al., 2020b). However, these models map entities (constituents) and relations (subject, object) in a unified label space to capture the inter-dependency between them. (Zhong and Chen, 2020) shows that a pipelined approach for entity and relation extraction outperforms prior joint models that use the same encoder for the two sub-tasks. In this work, we validate this claim for the OpenIE task. Furthermore, we design a grid labeling schema that identifies constituents and their types, akin to entities in the entity relation extraction task.
In this work we extract compact triples from single sentences using an end-to-end pipelined approach, first extracting triple constituents using a novel table filling model and then determining relations between them with a classifier. Our method achieves excellent performance in producing exhaustive compact triples with high precision. We hope that COMPACTIE serves as a strong baseline and makes us re-think the value of all-at-once information extraction systems.
The research described herein was sponsored by the U.S. Army Research Institute for the Behavioral and Social Sciences, Department of the Army (Contract No. W911NF-20-C-0028). The views expressed in this presentation are those of the author and do not reflect the official policy or position of the Department of the Army, DOD, or the U.S. Government.
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proc. ACL-IJCNLP '15, pages 344-354.
Michele Banko, Michael J. Cafarella, Stephen Soderland, Matt Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Proc. IJCAI '07, page 2670-2676, San Francisco, CA, USA.
Nikita Bhutani, H. V. Jagadish, and Dragomir Radev. 2016. Nested propositions in open information extraction. In Proc. EMNLP '16, pages 55-64, Austin, Texas. Association for Computational Linguistics.
Nikita Bhutani, Yoshihiko Suhara, Wang-Chiew Tan, Alon Halevy, and HV Jagadish. 2019. Open information extraction from question-answer pairs. In Proc. NAACL-HLT '19, pages 2294-2305.
Luciano Corro and Rainer Gemulla. 2013. Clausie: Clause-based open information extraction. In Proc. WWW '13, pages 355-366.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information extraction. In Proc. EMNLP '11, pages 1535-1545.
Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. 2019. Using local knowledge graph construction to scale seq2seq models to multi-document inputs. CoRR, abs/1910.08435.
Kiril Gashteovski, Rainer Gemulla, and Luciano del Corro. 2017. MinIE: Minimizing facts in open information extraction. In Proc. EMNLP '17, pages 2630-2640, Copenhagen, Denmark.
Kiril Gashteovski, Rainer Gemulla, Bhushan Kotnis, Sven Hertling, and Christian Meilicke. 2020. On aligning openie extractions with knowledge bases: A case study. In Proc. EMNLP '20 Workshop on Evaluation and Comparison of NLP Systems, pages 143-154.
Kiril Gashteovski, Mingying Yu, Bhushan Kotnis, Carolin Lawrence, Goran Glavas, and Mathias Niepert. 2021. Benchie: Open information extraction evaluation based on facts, not tokens. CoRR, abs/2109.06850.
Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy. 2016. Table filling multi-task recurrent neural network for joint entity and relation extraction. In Proc. COLING '16, pages 2537-2547.
Zengguang Hao, Binxia Xu, Shiyuan Zheng, and Yang Gao. 2018. Structured text summarization via open domain information extraction. In Proc. CSCWD '18, pages 701-706. IEEE.
Heng Ji, Benoit Favre, Wen-Pin Lin, Dan Gillick, Dilek Hakkani-Tur, and Ralph Grishman. 2013. Open-domain multi-document summarization via information extraction: Challenges and prospects. In *Multi-source, multilingual information extraction and summarization*, pages 177-201. Springer.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2017. Answering complex questions using open information extraction. CoRR, abs/1704.05572.
Keshav Kolluru, Vaibhav Adlakha, Samarth Aggarwal, Musam, and Soumen Chakrabarti. 2020a. Openie6: Iterative grid labeling and coordination analysis for open information extraction.
Keshav Kolluru, Samarth Aggarwal, Vipul Rathore, Musam, and Soumen Chakrabarti. 2020b. Imojie: Iterative memory-based joint open information extraction.
William Léchelle, Fabrizio Gotti, and Philippe Langlais. 2018. Wire57: A fine-grained benchmark for open information extraction. arXiv preprint arXiv:1809.08962.
Mausam, Michael Schmitz, Stephen Soderland, Robert Bart, and Oren Etzioni. 2012. Open language learning for information extraction. In Proc. EMNLP '12, pages 523-534, Jeju Island, Korea. Association for Computational Linguistics.
Mausam Mausam. 2016. Open information extraction systems and downstream applications. In Proc. IJ-CAI '16, page 4074-4077.
Youngbin Ro, Yukyung Lee, and Pilsung Kang. 2020. Multi2oie: Multilingual open information extraction based on multi-head attention with bert. arXiv preprint arXiv:2009.08128.
Gabriel Stanovsky and Ido Dagan. 2016. Creating a large benchmark for open information extraction. In Proc. EMNLP '16, pages 2300-2305, Austin, Texas.
Gabriel Stanovsky, Ido Dagan, et al. 2015. Open ie as an intermediate structure for semantic tasks. In Proc. ACL-IJCNLP '15, pages 303-308.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018. Supervised open information extraction. In Proc. NAACL '18, pages 885-895.
Yijun Wang, Changzhi Sun, Yuanbin Wu, Hao Zhou, Lei Li, and Junchi Yan. 2021. Unire: A unified label space for entity relation extraction.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. Ernie: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129.
Zexuan Zhong and Danqi Chen. 2020. A frustratingly easy approach for joint entity and relation extraction. CoRR, abs/2010.12812.
A.1 Benchmark Creation
The Algorithm 2 gives a high-level overview of our benchmark creation mechanism while a lot of details and difficulties have been omitted. The Benchmark Creation Algorithm extracts triples for each sentence using the Algorithm 1. The OpenIE system used to produce triples out of simple clauses is IMoJIE (Kolluru et al., 2020b).
The following example illustrates the benchmark creation algorithm. Given the sentence: "The group reach a small shop, where the crocodile breaks through a wall", the algorithm first builds the sentence tree as shown in Figure 4. Then, starting from the root, ExtractTriple function traverses the tree until it reaches a child ("breaks") with no further clausal children. At this point, a clause for the subtree rooted at "breaks" is generated and fed into the IMoJIE system. IMoJIE extracts triple: (the crocodile; breaks; through a wall) out of this clause. Then, since both children of the root ("reach") are processed, the IMoJIE triple of the root's corresponding clause is extracted as (The rest of the group; reach; a small shop).
Figure 4: Sentence Tree for input sentence: "The group reach a small shop, where the crocodile breaks through a wall".
A.2 Table Decoding
Following the (Wang et al., 2021) work, in the testing phase, we rely on the label probability tensor  $\mathcal{P} \in \mathbb{R}^{|s| \times |s| \times |Y|}$  of the sentence  $s$ , to first extract constituent spans, and then predict the constituent type. Next, we describe the decoding procedure.
A.2.1 Constituent Span Detection
One important observation of the ground truth table is that a constituent's corresponding rows and columns are identical (e.g., row 2 and row 3 of Figure 2 are identical). Therefore, given the tensor  $\mathcal{P}$ , we compute the distance of adjacent rows (and columns). If the distance is larger than a predefined threshold  $\alpha$  (which is set to 1.2), a split position is detected. This means that the two adjacent rows (columns) belong to different constituents or one belongs to a constituent while the other is not. Following the (Wang et al., 2021) work, we flatten the  $\mathcal{P}$  tensor from both row and column perspectives and calculate the euclidean distance of adjacent rows and adjacent columns. Finally, we average these two distances as the final distance and compare the final distance to  $\alpha$  to find the span of different constituents.
A.2.2 Constituent Type Detection
Given a constituent's span  $(i,j)$ , we decode the constituent type  $t^* \in Y$ , where  $Y = Y_c \cup Y_r \cup$  None, according to its corresponding square symmetric about the diagonal:
$$
t ^ {*} = \arg \max _ {t \in \{Y _ {c} \cup N o n e \}} A v g (P _ {i: j, i: j, t})
$$
Spans with predicted type  $t^* \in Y_c$  are regarded as constituents and passed to the constituent linking model.