Abstractive Open Information Extraction
Authors: Kevin Song Pei
Ishan Jindal
Kevin Chang
Venue: EMNLP 2023 Main (2023)
Keywords: Open Information Extraction
Relation Extraction
Natural Language Generation
TL;DR: We introduce abstractive Open Information Extraction to extract relations without needing to use the tokens from the original sentence, in contrast to preexisting extractive Open Information models.

Abstract: Open Information Extraction (OpenIE) is a traditional NLP task that extracts structured information from unstructured text to be used for other downstream applications. Traditionally, OpenIE focuses on extracting the surface forms of relations as they appear in the raw text, which we term extractive OpenIE. One of the main drawbacks of this approach is that implicit semantic relations (inferred relations) can not be extracted, compromising the performance of downstream applications. In this paper, we broaden the scope of OpenIE relations from merely the surface form of relations to include inferred relations, which we term abstractive OpenIE. This new task calls for the development of a new abstractive OpenIE training dataset and a baseline neural model that can extract those inferred relations. We also demonstrate the necessity for a new semantics-based metric for evaluating abstractive OpenIE extractions. Via a case study on Complex QA, we demonstrate the effectiveness of abstractive OpenIE.

Abstractive Open Information Extraction
Kevin Peit, Ishan Jindal', Kevin Chen-Chual Chang! University of Illinois at Urbana- Chanmaagn,,,,,,, 'IBM Research "kspeii,, kcchang)@ill edu, ?ishan. jishan.jindall jindal@ibm. com
Abstract
Tokyo, officially Tokyo Metropolis, is the capital
Sample Sentence
city of Japan and one its 47 prefectures.
Open Information Extraction (OpenIE) is a tra- ditional NLP task that extracts structured infor- mation from unstructured text to be used for other downstream applications. application..Taadiionnlllll Traditionally, OpenIE focuses on extracting the surface forms relations as they appear the raw text, which we term extractive OpenIE. One of the main drawbacks of this approach is that implicit semantic relations (inferred relations) can not be extracted, compromising the performance downstream applicationsIn this paper, we broaden the scope of OpenIE relations from merely the surface form of relations to include inferred relations, which we term abstractive OpenIE. This new task calls for the devel- opment of new abstractive OpenlE training dataset and a baseline neural model that can extract those inferred relations. We also demon- strate the necessity for a new semantics- metric for evaluating abstractive OpenIE ex- tractions. Via case study Complex QA, we demonstrate the effectiveness abstractive OpenIE.
Extractive
(Tokyo; is; the capital city of Japan]
OpenIE Extractions
(Tokyo; is; one its prefoctures)
(Tkkya; is; the capital city Japan]
Abstractive
Tokyo; officially: Tokyo Metropolis)
OpenlE Extractions
Tokyo: is; prefecture)
(Tokyo; is; one of Japan's 47 prefectures!
Table Examples of relations that extractive OpenIE models can not extract. In this sentence, the apposition "officially Tokyo Metropolis" has no predicate but still a relation with the noun "Tokyo". In the last abstrac- tive relation, "one of its 47 prefectures" is meaningless without the context of the rest of the sentence. It would be more useful to replace or "one of Japan's 's 47 prefectures", neither of which appear in the sentence. Preexisting OpenIE models can extract these abstractive relations.
object with prefecture"
Relation" to be a relation where the predicate con- tains words that are not in the original sentence. For example, given the sentence "Albert Einstein (14 March 1879 18 April 1955) was German- born theoretical physicist", the relation (Albert Ein- stein, died on, 18 April 1955) can be inferred even though "died on" is not in the original sentence. Ex- tracting inferred relations increases recall, which is explicitly desired by various downstream tasks including question- answering, slot filling, event schema induction, summarization, and knowledge base population (Pei et al., 2022). Based on the number of inferred relations in the manually an- notated dataset WiRe57, extracting inferred rela- tions could increase the total number of relations extracted by 50% (Léchelle et al., 2018). Exist- ing neural OpenIE models struggle to extract these inferred relations, with only one previous model, Open1E6, including hand-written rules to extract only some cases of inferred relations (Kolluru al., 2020a). Table has example of an inferred rela- tion.
1 Introduction
Open Information Extraction (OpenIE) is the task of extracting relation tuples from unstructured text (Etzioni et al., 2008; Mausam et al., 2012; Angeli al., 2015). Unlike traditional information extrac- tion, OpenIE open domain, intended be easy to deploy different domains without fine-tuning. These relations can then be used in downstream ap- plications like summarization (Zhang et al., 2021), question-ans -anweerng (Lu et al., 2019), and knowl- edge base population (Kroll et al., 2021). In or- der to support these applications, OpenlE needs extract many different types of relations as possible One particular relation type of interest is "Inferred Relations". We define an "Inferred
Another problem is that the extraction is very de- pendent on the sentence's syntax. For downstream
'Code and models are availahle at ttttps://githhh com/ kevinpei/Abor heaiinpe//bsttactiveeeeee

---

applications using OpenIli, is important to be able to extract either different surface forms of relation or its canonical form. The surface form refers to how it appears within the text, while the canonical form refers to the semantic meaning. In question answering (QA), several methods repeat- edly paraphrase the questions so that the surface forms of extracted relations match at least one of the question paraphrases, indicating that extracting more surface forms of relation would answer more questions (Fader et al., 2013, 2014; Yin et al., 2015). addition, the more complex a sentence's syntax is, such as having more clauses, the more difficult it is to extract high-quality relations. An illustrative example of how being limited to extracting surface forms can be found in Table 1.
OpenlLó in the sense that abstractive OpenlE mod- els trained on abstractive OpenIE training datasets generate this inferred relation on the fly and do not require defining a list of vocablavyya to- kens. Therefore, in this paper, we derive abstractive OpenIE training datasets from existing information extraction datasets and train a baseline machine- learning model that extracts abstractive relations.
Further, we also develop an abstractive OpenIE evaluation metric to evaluate the quality of abstrac- tive OpenlE' models. Our problem warrants a new evaluation metric because all the existing OpenIE evaluation metrics are lexical and evaluated based on the token overlap between the predicted rela- tions and the gold standard relation. These lexical metrics undesirable for the proposed task the relations extracted using the abstractive Ope- nlE model do not have use the tokens present in the input sentence. Therefore, we propose a remantics-based metric for evaluating abstractive OpenIE models.
By design, all existing neural OpenIE models are unable to extract these abstractive relations, which could be utilized by the downstream application. Therefore, in this work, we propose an abstractive Open Information Extraction (abstractive OpenlE) task. The purpose of this task is to extract relation tuples that are far beyond the reach of any existing OpenIE tasks. We define abstractive OpenIE task that given an input sentence generates ordered tuples the form of (subject, predicate, object) for all possible relations (inferred or non-inferred) within the sentence.
In summary, our contributions are as follows:
We propose an abstractive OpenlE task to expand the scope of OpenlE extractions com- pared to prior extractive OpenlE models.
We derive an abstractive OpenlE training dataset and develop initial abstractive Ope- nIE model baseline.
Although not explicitly defined as such, exist- ing neural models often treat OpenIE as labeling problem, where tokens are labeled as being part of the subject, predicate, or object of relation (Kolluru et al., 2020a; Vasilkovsky al., 2022). Even in cases where OpenIE is defined as a genera- tive problem, the generated relations don't contain words outside the vocabulary of the original sen- tence (Kolluru et al., 2020b) (Han and Wang, 2021). Due to the labeling problem definition, prior neural OpenIE models struggle to extract relations with predicates that don't I appear in the original sentence. We refer to all preexisting neural OpenIE models as extractive OpenIE methods, because they can only generate relations by extracting tokens from the original sentence.
We propose general-purpose semantics- based evaluation metric for evaluating any OpenlEi model.
We perform a comprehensive comparison be- tween abstractive and extractive OpenIE mod- els.
Related Work
Given how data-hungry deep learning models are and how costly is to manu- ally label OpenIE datasets, most OpenIE training sets are weakly labeled using high-confidence ex- tractions from prior OpenIE models to get "silver- standard" labels. For example, the Copy Attention (Cui et al., 2018), 8), SpanOIE (Zhan and Zhao, 2020), and O1E4 (Kolluru et al., 2020b) training sets are created from high-confidence OpenlE44 extractions from Wikipedia. L.OOE (Solawetz and Larson, 2021) is instcad created from examples from the QA-SRL 2.0 dataset. Because traditional OpenIE is
OpenIE Datasets:
One such attempt to go beyond extractive Ope- nIE is the Open1E6 model Kolluru et al. (2020a). It explicitly concatenates manually defined out-of- vocabulary tokens at the end of each sentence to allow for the extraction of specific inferred rela- tions. However, obtaining such a list non-trivial and can not scale to every domain. We differ from

---

Number Sentences
Number Relations
Number Relations with Inferred Predicates
Number of Relations with
Dataset
Inferred Predicates or Arguments
O1E4
90K
160K
0
0
Training
O1E4 Backtranslated
44K
61K
19K
48K
Scts
OIE4 with SuRE
90K
16K
Relations
OIL4 Backtranslated
44K
69K
26K
56K
with SuRE Relations
WiRe57
57
343
116
120
Test Sets
CaRB
2715
736
ReO1E2016
683
1508
155
156
LSOIE
2402
5371
0
0
Table 2: Comparison of the attributes of different datasets. SuRE is the relation extraction model we to obtain additional inferred relations for training (Lu et al., 2022).
Sample Sentence
The purse contains the seal of Order of the Garter.
Back Translated Sentence
In the handbag is the seal of the Order of the Garter.
Relations
(The purse; contains; the seal of Order of the Garter)
Table 3: An example of paraphrasing via back translation. The sentence is from the OIB4 training set.
extractive, there are no inferred relations in OpenlE training sets, with only hand-labeled benchmarks containing inferred relations. As result, these training sets are not well-suited for training ab- stractive OpenlE model.
relations to the end of cach sentence (Kolluru et al., 2020b). This new sentence is then given as input so the model can identify what relations have pre- viously been extracted at the cost of significantly reduced extraction speed. Although it uses a gen- erative neural model, IMoJIE relies on its copy mechanism to extract relations, so its vocabulary is limited so that it only generates tokens that are within the original sentence. In addition, the fo- cus on reducing redundancy means it is also con- strained to extracting only single surface form of each relation in each sentence.
In contrast, there are several benchmarks with inferred relations. WiRe57 (Léchelle et al., 2018) is 57 manually annotated sentences. CaRB (Bhard- waj et al., 2019) uses crowdsoureing to re-annotate the sentences in the 01E2016 benchmark, the first commonly used OpenIE benchmark (Stanovsky and Dagan, 2016). ReO1E2016 (Zhan and Zhao, 2020) is a different manual re-annotation of O1E2016 to attempt to resolve problems arising from incorrect extractions. LSOIE also has own benchmark created using the same method as its training set. WiRe57, CaRB, and RcO1E2016 all contain inferred relations, making them useful for evaluating abstractive OpenIE.
Gen2OlE is an OpenlE model that fine-tunes a seq2seq model to generate relations (Kolluru al., 2022). It follows a two-stage approach, where pred- icates are first extracted, then arguments are ex- tracted for each predicate. Unlike previous OpenIE models, Gen2OIE can generate relations using to- kens that do not appear in the original sentence.
OpenIE Models: Open1E6 is neural OpenIE model that performs BIOES tagging for the subject, predicate, and object of each relation (Kolluru et al., 2020a). At the end of each sentence, it appends the tokens "be", "of", and "from" so that they can also be tagged as part of the predicate. However, this method limits inferred relation extraction to only those containing the tokens they manually specify and doesn't 't help with the issue of extracting only the surface form of the relation.
Closed Information Extraction (CIE) is related task where relations within an existing KB are ex- tracted from unstructured text. GenlE proposes a generative model to perform this task (Josifoski et al., 2021). However, CIE is an inherently more limited task than OpenIE due to its dependence on preexisting domain. CIE models are unable to ex- tract relations from new and emerging domains and require human effort to transfer new domains. OpenIE Evaluation Metrics: Existing OpenTE metrics are lexical. This means that extracted re- lations are evaluated based on the token overlap
IMoJIE an OpenIE model that tries to reduce the redundancy of relations by appending extracted

---

Sample Sentence
In 569, unopposed, Alboin took northern Italy's main city. Milan.
Extractive Relations
(Alhoin; look; In 569 nothem Italy's 's main city, Milan)
SuRE-Extracted Relations
(northern Inly's main city; is also known as: Milan)
Table 4: An example of data augmentation via relation extraction. The method used for relation extraction is SuRE (Lu et al., 2022). The sentence is from the OIE4 training set.
between the predicted relations and the gold stan- dard relations. In particular, OIE2016 is based on tuple-level matching, treating relations extraction as a binary classification problem where gold standard relation is extracted a predicted relation contains a majority of tokens in the gold standard relation (Stanovsky and Dagan, 2016). WiRE57 and CaRB use token-level matching, where pre- dicted relations are evaluated based on the token overlap between the best matches between the pre- dicted and gold standard relations (Léchelle et al., 2018) (Bhardwaj et al., 2019). Because the abstrac- tive relations extracted using abstractive OpenIE do not have to use the original sentence's tokens, evaluating them using lexical metrics is undesir- able.
3.1 Training Sets
Although there are existing OpenIE training sets, they do not fit our goals because they are purely extractive. The training set needs to contain in- ferred relations that trained models extract inferred relations. To address this problem, we use two methods derive abstractive OpenIE training sets from OIE4, a preexisting OpenIE training set: Paraphrasing Via Back Translation
Back translation is the translation of a text into different language, then translation back into the original language (Edunov et al., 2018). The result- ing text should retain the same semantic meaning, but may differ in the specific words or syntax used. To generate abstractive OpenTE training data, we generate back translations of the sentences but re- tain the gold standard relations. Because the back translated sentences use different words and syntax. the gold standard relations may longer consist of only words from the original sentence, thus be- coming inferred relations. We provide an example in Table
There has been previous interest in semanties- based metrics for evaluating abstractive summa- rization and machine translation. BERTScore is a popular metric that calculates the cosine simi- larity between the BERT contextual embeddings of each token in the document and each token in the summary. The highest total similarity score possible from the mapping of tokens in the docu- ment to tokens in the summary then chosen as the BERTScore (Zhang al., 2019). In theory, this metric would take into account the context of each word, which would capture the semantics of each word. However, it has been found that BERTScore may still be insufficient in cases where individ- ual tokens like negations significantly change the meaning of the sentence, even if it is marginally better than lexical methods like BLEU, ROUGE, and METEOR (Saadany and Orasan, 2021).
When generating paraphrases, we need to make sure that the paraphrased sentence has the same semantic meaning as the original sentence and con- tains the same relations. Thus, we perform val- idation step where we use entailment measure the quality of the paraphrase. During this step, we use three measures to ensure the quality of the paraphrase. We measure whether the original sentence entails the paraphrase to ensure the para- phrase doesn't It contain extraneous information not in the original sentence. We measure whether the paraphrase entails the original sentence to ensure the paraphrase contains all information present in the original sentence. Finally, we measure whether the paraphrased sentence entails all of the gold stan- dard relations to ensure that the relations are the same for the original sentence and the paraphrase. If any of these hypotheses does not have an entail- ment confidence above a certain threshold, then we do not use the paraphrase in the training data. Data Augmentation Via Relation Extraction Although paraphrasing can create inferred rela-
3 Abstractive OpenlE
Abstractive OpenIE is defined as a task that gener- ates ordered tuples in the form of (subject, predi- cate, object) for all possible relations (inferred or non-inferred) within a given sentence. this sec- tion, we will describe all the pieces required to accomplish this task.

---

Sample Sentence
Sharon had been a coma since suffering stroke in January 2006.
Sample Relations
(Sharon; had been; a coma)
(Sharon; suffering; stroke in January 20066
Sample Predicate Prediction Input
predicates: Sharon had been in coma since suffering stroke in January 2006. [pred] had been [pred] suffering
args: Sharon had been in coma since suffering stroke Jamuary
Sumple Argument Prediction Inputs
2006. [pred] had been [arg!] Sharon [arg2] in coma
args: Sharon had been in coma since suffering stroke January
2006. [pred] had been [pred] suffering [urg]] Sharon [urg2] stroke in
January 2006
Table 5: Illustrative training example. For each sentence, there is one predicate prediction example and a number of argument prediction examples equal to the number gold standard relations. The model first extracts all predicates, then for each predicate extracts the arguments.
tions that the words used may not match the sen- tence exactly, the relations remain fundamentally the same. The inferred relations that the bench- marks such as WiRe57 contain are not derived from paraphrases of the sentence, so creating para- phrases as training data for them is not appropriate. Instead, we augment the data with additional in- ferred relations derived using relation extraction (RE). We provide an example in Table 4.
the other benchmarks, LSOIE contains no inferred relations at all, meaning in theory extractive Ope- nIE methods should be able to extract all relations. Thus, we can use performance on LSOIE to di- rectly compare abstractive OpenlE and extractive OpenIE models on the extractive OpenIE task.
Statistics for the derived training sets and bench- marks available in Table 2.
3.3 Abstractive Tuple Generator
RE also aims to extract relations from unstruc- tured text, but instead of being completely open domain, RE is limited to extracting a specific set of relations that must be defined beforehand (Bach and Badaskar, 2007). However, those relations may take variety of surface forms. For instance, the relation "country_of "oattryyoo_brrhhh""" birth" could take the form "Einstein was born in Ulm", "Einstein (born 14 March 1879 in Ulm)", other forms. We thus use RE models to extract additional inferred relations for abstractive OpenlE training. To ensure quality and prevent redundancy, we only keep extracted relations above certain level of confidence and which are not entailed by or entail preexisting Ope- nIE gold standard relations.
Prior OpenIE models are not suited for the pro- posed task because all existing models are extrac- tive models. As a result, we use generative mod- els to generate relations for a given sentence. We choose to fine-tune T5, text-to-text transformer model, generate relations from sentence (Raffel et al., 2020).
Inspired by MultiPOIE, we perform relation gen- cration in two stages, a predicate and an argument stage (Ro et al., 2020). In the predicate stage, all predicates are extracted from the sentence at once. The input for this stage is the sentence, while the gold standard is the predicates of all gold stan- dard relations separated by the special "[pred]" to- ken. Although the order of relations in our output doesn't matter, we need enforce a strict order for the model to learn. Thus, we order the predicates by their position within the sentence.
3.2 Benchmarks
contrast to existing OpenlE training dataset, there are several OpenlE benchmarks which con- tain inferred relations because they were manu- ally annotated or used crowdsourcing for anno- tation. For evaluation, we use WiRe57, CaRB, Rc-O1E2016, and LSOIE test sets. Each of these benchmarks contains different proportion of in- ferred relations, in Table 2. In particular, the man- ual annotation of WiRe57 makes prior extractive OpenIE methods perform poorly compared to their performance on other OpenTE benchmarks. Unlike
For the argument prediction stage, for each pred- icate the model predicts the arguments for the rela- tion with that predicate. Because multiple relations may have the same predicate, specify the pred- icate by including all predicates before it in the sentence. For each relation, we assume there are two arguments, which the model extracts simultane- ously. The input for this stage is the sentence with the predicate concatenated to the end, separated by

---

"[pred]" special token, while the gold standard is the arguments for the gold relation corresponding to that predicate, in Table 5.
by (Dušek and Kasner, 2020). For this metric, we use the gold standard relations to evaluate the ex- tracted tuples. The combined tuple this case refers to the combination of all gold standard re- lations. For recall, we combine all the predicted relations together and see if the combined relations entail the combined gold relations. If the combined predictions do not entail the combined gold, that means the gold relations contains information not in any prediction and thus the extracted relations as a whole have poor recall. For precision, we take the average of the entailment score obtained when seeing if the combined gold entails an individual relation for all extracted relations. If the prediction is not entailed, that means it contains information not in any gold relation and thus has poor precision. Compared to the sentence- tuple entailment metric, this one excludes any extraneous information in the sentence not the gold standard relations from evaluation.
3.4 Semantic-based Evaluation Metrics
CaRB is popular metric for evaluating OpenIE models, but it requires the predicates of the predic- tion and gold standard to match to score given prediction. Although it serves as a good proxy for semantic metric in extractive OpenlE, it is signifi- cantly less useful for abstractive OpenTE where the space of all possible predicates much larger than just the tokens in the sentence.
To evaluate abstractive OpenIE, we require semantics-based metric rather than lexical met- ric based on token matching. Although pre- vious semantics-bare
evaluation metrics like BERTScore exist, we do not find them to be appro- priate for our use case. Previous semantics-based evaluation metrics do not work well for cases where a single token can dramatically change the seman- ties of statement, such negations like "not" (Saadany and Orasan, 2021). Thus, we introduce set of 3 evaluation metrics based on entailment for more accurate semantic evaluation. Each of these metries measures semantic coherence at different granularities, and which granularity is most impor- tant will depend on the application and properties of the datasets. We demonstrate this necessity with an example in Table 6.
Tuple-tuple entailment The third metric we pro- pose is tuple-tuple entailment. This metric is based on the OpenIE metric CaRB (Bhardwaj et al., 2019). For recall, for each gold standard relation we calculate the entailment for each extracted re- lation if the gold standard entails that prediction. Then, for each gold standard relation its recall is equal to the highest entailment score achieved by any of the predictions. The recall for the sentence is the average of the recall of its relations. Note that the highest recall for multiple gold standard relations can be achieved by the same predicted relation if the predicted relation contains all of those gold standard relations. For precision, for each gold standard relation we calculate the entail- ment for each extracted relation if the prediction entails that gold standard relation. Then, we find the optimal matching of gold standard relations to extracted relations that results in the highest av- erage precision. Unlike recall, when calculating precision predicted relation can only entail a sin- gle gold standard relation. This is because we want the number of predictions to match the number of gold relations.
When calculating the entailment score for a re- lation, we remove special characters so that it re- sembles a sentence. For instance, for the relation triple (Sharon; had been; in coma), we form the statement "Sharon had been in a coma."
Sentence-tuple entailment The first metric we propose is sentence-tuple entailment. For recall, we combine all the relations together and see the combined relations entail the sentence. If the combined relations do not entail the sentence, that means the sentence contains information not in any relation and thus the extracted relations whole have poor recall. For precision, we take the average of the entailment score obtained when seeing if the sentence entails an individual relation for all extracted relations. the relation is not entailed, that means it contains information not in the sentence and thus has poor precision.
Experimental Setup
Datasets and Metrics
We evaluate the trained abstractive OpenlE model on four benchmarks: WiRe57, CaRB, Re-O1E2016. and LSOIE-wike with respectively decreasing pro- portion of inferred relations.
Combined tuple-tuple entailment The second metric we propose is combined tuple-tuple entail- ment. This metric is inspired by a metric proposed

---

Sentence
Rival political factions were unable to resolve disagreements.
Gold Standard
(Rival political factions unable to; resolve; disagreements]
Prediction
(Rival political factions to; resolve; disagreements)
CaRB
ROUGE-1
BERTScore Tuple-Tuple Entailment
Evaluation
Fl Score
Score
Fl Score
Fl Score
Metric
0.923
0.923
0.976
0.005
Table 6: Comparison of different evaluation metrics on example from the training set. CaRB is popular lexical metric used to evaluate OpenIE (Bhardwaj et al., 2019). ROUGE-1 is popular lexical metric to evaluate summarization (Lin, 2004). BERTScore previous semantics-based metric used evaluate summarization (Zhang et al., 2019). Tuple-Tuple Entailment is a new semantics-baun metric we propose.
Sentence- Tuple Entallment F1
1.000
Abstractive OIE4 Back Translated and SuRe Augmented
0.750
Abstractive OIE4 Back Translated
Abstractive OIE4 SuRE Augmented
0.500
0.250
Abstractive OIE4
MultFOIE
0.000
0 l--kk))
0. 0.103 (Re- OIE2016)
0.271 (CaRB) 0.338 (WiRe57)
Proportion of Relations with Inferred Predicates
Figure 1: Comparison of Sentence-Tuun Tuple Entailment F1 Score of different OpenIE models on all relations in the benchmarks. models are trained on OIF4.
Since OIE4 trained OpenTE models showed su- perior F1 performance on all these benchmarks compared to other OpenIE training sets de- rive abstractive training data from this dataset. We generate four different versions of OIE4 using the methods we describe Section 3.1. The first ver- sion is the original extractive dataset, the second version uses backtranslation for paraphrasing, the third version is augmented by relation extraction, and the fourth uses both backtranslation and rela- tion extraction for augmentation. For backtransla- tion Facebook-FAIR'' WMT' 19 German- English and English-German models (Ng al., 2019) and retain only those back translated sen- tences whose entailment confidence above 80%. For relation extraction, we use pretrained SuRE model, a state-of-the-art relation extraction model (Lu et al., 2022) without any additional fine-tuning and keep all relations with confidence above 80%. These confidence thresholds are hyperparameters that may be adjusted.
combined tuple-tuple entailment, and sentence- tuple entailment. The entailment model we use for datasets and evaluation metrics is BERT- based encoder model trained on MNLI, SNLI, and the Hans dataset (Gao al., 2021). Models and Hyperparameters We fine-tune the T5-base model for our experi- ments. We fine-tuned T5 for epochs with an initial learning rate of 2c-5 and batch size of 12. We validate T5 on a subset of the OIE4 training set using the tuplc-tuple entailment metric. We also compare our model with MultiPOIE, state-of- the-art neural extractive OpenlE model (Ro et al.. 2020). We train Multi'OIE the original OIE4 dataset with no paraphrasing. We use the default hyperparameters of Multi"OIE.
Results and Analysis
For this section, focus the sentence-tuple mantic score because offers a holistic comparison of the extracted relations and the sentence and does not rely upon potentially incomplete or faulty gold relations. Full tables with our empirical results in-
We compare performance using the preexist- ing CaRB metric, as well as our own introduced semanticx-bare based metrics of tuple-tuple entailment,

---

1.000
MultiPOlE + Abstractive OIE4
Sentence- TeceeeTTpleee Entailment F1
Back Translated and SuRe
Augmented
0.750
MultiPOlE + Abstractive OlE4
Back Translated
0.500
MultiPOIE + Abstractive OlE4
SuRE Augmented
MultiPOIE + Abstractive OlE4
0.250
MultiPOlE
0.000
0.103 (Re-O1E2016)
0.271 (CaRB)
0.338 (WiRe57)
Proportion of Relations with Inferred Predicates
Figure 2: Comparison of Sentence- Tuple Entailment Recall different combinations of OpenIE models on all relations the benchmarks. All models are trained OIE4.
cluding other metrics can be found in Appendix A.
OpenIE Model
Documents
MRR
P@ 1
Hit@5
MultiPOIE
Top
0.193
0.127
0.267
We first compare performance on all relations in Figure 1. In general, abstractive OpenIE leads to better performance the higher the proportion of inferred relations in the test set. This is expected because MultiPOIE can not extract inferred rela- tions all. When considering the full benchmarks, of the data augmentation methods we use, SuRE augmentation works the best. Training on back- translated OIE4 degrades the performance com- pared to the base extractive OIE4 data. This may be because back translation reduces the amount of training data. Additionally, back translation of- ten just replaces the gold standard predicate with synonym instead of changing the syntax of the sentence, which does help in the extraction of inferred relations.
Abstractive OIE4
Top 10
0.154
0.080
0.240
Abstractive Back
Top
0.167
0.100
0.227
Translated OIE4
Abstractive SuRE
Top 10
0.157
0.093
0.220
Augmented OIE4
Abstractive SuRE Augmented Back Top
0.181
0.093
0.287
Translated O1E4
Table 7: Performance of QUEST on the CQ-W dataset using the Top Google documents (Lu al., 2019).
on only the inferred relations within each bench- mark. To do this, remove non-inferred relations from the gold standards. We can only measure the resulting recall of the models because the mod- els trained to generate both inferred and non- inferred relations and the metries we use penalize the precision when there are too many predicted relations for a given sentence, which would the case for any sentence that had non-inferred rela- tions. Figure 3 shows the results of these experi- ments. As before, the more inferred relations in the benchmark, the better suited abstractive OpenIE model is for the task.
To demonstrate the complementary nature of ab- stractive OpenIE extractive OpenIE, combine their extractions. When combining their extrac- tions, we remove redundant relations by removing relations that are entailed by any other relations. two relations entail each other, then we keep the longer one. A comparison of combined models can be found in Figure 2. When combining model pre- dictions, observe that back translation actually helps more than SuRE augmentation. This suggests that SuRE augmentation helps extractive OpenlE relations, while back translation is more useful for increasing the recall to inferred relations that could not be extracted by MuIti'OIE. The more inferred relations in the benchmark, the more beneficial merging extractions are.
Upon manual examination of the generated relations of each model, we observe that fine-tuning T5 SuRE-augmentes data results in generated relations replacing some of its predicates with the predicates from SuRE. Table demonstrates one example of a model generating a predicate that does not exist within the sentence but is a common predicate among the augmeened relations.
We also evaluate our abstractive OpenTE models

---

1.000
T5 Back Translated SuRe Augmented
Sentence- -upll Entailment R
0.750
TS Back Translated
T5 SuRE Augmented
0.500
TS
Multi20IE
0.250
0.000
ReOIE2016
CaRB
WiRE57
Figure 3: Comparison of Sentence-Tuple Entailment Recall of different combinations of OpenIE models on only the inferred relations in the benchmarks, All models trained OIE4.
Formerly known as Edo, it has heen the de facto seat of government since 1603 when Shogun Tokugawa leyasu made the city his headquarters.
training sets and the state-of-the-arr extractive Ope- nIE model using preexisting OpenIE benchmarks. Overall, we find that our models achieve higher performance inferred relations, which extractive OpenIE models have previously struggled with. We believe abstractive OpenIE has potential as a task that will greatly benefit downstream applications that use OpenIE in their pipeline.
Sentence
T5 Fine-Tuned on OIE4
(it: has been; the facto seat government since 1603)
T5 Fine-TTuned SuRE-
(it: also known as; facto seat of government since 1603)
Augmented OIE4
Table 8: A demonstration that T5 fine-tuned on OIE4 augmented with SuRE extractions generates predicates from the SuRE extractions rather than the sentence. This sentence is from the WiRE57 test set.
7 Limitations
In this work, we used a relatively smaller T5-base model. A model with more parameters may have led to improved performance. Further, the corpora chose all limited to English. As result, our results are not generalizable to any downstream task that relies on different languages.
Case Study
To further test the applicability of abstractive Ope- nIE, evaluate its performance QUEST, downstream Complex QA task that uses OpenlE in its pipeline (Lu et al., 2019). QUEST specifically desires higher recall from its OpenIE model, which can achieved by extracting inferred relations. We show the results in Table 7. The results show that augmenting the training data improves down- stream performance, indicating that including more inferred relations in the training data is helpful for this task.
Ethics Statement
We did not create any of the models, datasets, or applications covered in this paper. Any ethical issues with the preexisting OpenlE datasets we use in this paper will reflect this work.
Acknowledgements
This material is based upon work supported by the National Science Foundation IIS 16- 16-19302 and IIS 16-33755, Zhejiang University ZJU Research 083650, Futurewei Technologies and 094013, IBM-IIlinois Center for Cognitive Computing Systems Research (C3SR) and IBM- Illinois Discovery Accelerator Institute (IIDAI), grants from eBay and Microsoft Azure, UIUC OVCR CCIL Planning Grant 434S34, UIUC CSBS Small Grant 434C8U, and UIUC New Frontiers Initiative. Any opinions, findings, conclusions, or recommendations expressed in this publication are
6 Conclusion
In this paper, we introduce abstractive OpenlE, an alternative to what we call extractive OpenIE, the paradigm all current OpenIE models currently fol- low, in order to address the problems of inferred relations and surface form extraction. We find that existing OpenlE datasets and metrics are ill-suited for this task. As a result, we introduce abstractive training set, model, and metrics. We then com- pare our models trained on different abstractive

---

those of the author(s) and do not necessarily reflect the views of the funding agencies.
Keshav Kolluru, Vaibhav Adlakha, Samarth Aggarwal. Soumen Chakrabarti, et al. 2020a. Openie6: Iterative grid labeling and coordination analysis for open infor- mation extraction. arXiv preprint
References
Keshav Kolluru, Samarth Aggarwal, Vipul Rathore, Soumen Chakrabarti, et al. 2020b. Imojie: Itera- tive memory- joint open information extraction. arXiv preprint 2rXiv2205.08877
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meering of the As- sociation for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 344-354,
Keshav Kolluru, Muqeeth Mohammed. Shubham Mit- tal, Soumen Chakrabarti, et al. 2022. Alignment- augmented consistent translation for multilingual open information extraction. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Papers), pages 2502-2517..
Nguyen Bach and Sameer Badaskar. 2007. A review of relation extraction. Literature review for Language and Statistics 11, 2:1-15.
Sangnie Bhardwaj. Samarth Aggarwal, and Mausam Mausam. 2019, Carb: A crowdsourced benchmark for open ie. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNL.--JCNLP)), pages 2262-6267.
Hermann Kroll, Jan PirkIbauer, and Wolf-Tilo Balke. 2021. A toolbox for the nearly-unsupervisee con- struction of digital library knowledge graphs. In Proceedings of the ACM/IEEE Joint Conference Digital Libraries in.
William Léchelle, Fabrizio Gotti, and Philippe Langlais. 2018. Wire57: fine-grained benchmark for open information extraction. 1oXiv:1809.0892.
Lei Cui, Furu Wei, and Ming Zhou. 2018. Neu- ral open information extraction. arXiv preprint arXiv:1805.04270
arXiv preprint
Ondiej Dušck and Zdeněk Kasner. 2020. Evaluat- ing semantic accuracy of data-to-text generation with natural language inference. arXiv preprint
Chin- Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branckes out, pages 74 81.
arXiv:2011.10819.
Keming Lu, Hsu, Wenxuan Zhou, Mingyu Derek Ma.
Sergey Edunov, Myle Ott, Michael Auli, and David at
Muhao Chen, et al. 2022. Summarization as indirect supervision for relation extraction. arXiv preprint trXiv:2205.09833
Grangier. 2018. Understanding scale. arXiv preprint 1808:
Oren Etzioni. Michele Banko, Stephen Soderland, and Daniel S Weld. 2008. Open information extrac- tion from the web. Communications of the ACM, 51(12):68-75 74.
Xiaolu Lu, Soumajit Pramanik, Rishiraj Saha Roy, Abdalghani Abujabal, Yafang Wang, and Gerhard Weikum. 2019. Answering complex questions by joining multi-document evidence with quasi knowl- edge graphs In Proceedings of the 42nd Interna- tional ACM SIGIR Conference Research and De- velopment in Information Retrieval, pages 105-114.
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-die driven learning for open question answering. In Proceedings of the 51st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1608-1618.
Mausam, Michael Schmitz, Stephen Soderland, Robert Bart, and Oren Etzioni. 2012. Open language learn- ing for information extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Nat- ural Language Processing and Computational Natu- ral Language Learning, pages 523-532-3330 133-534, Jeju Island, Korea. Association for Computational Linguistics.
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and ex- tracted knowledge bases. In Proceedings of the 20th ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 1156-1161 1165.
Yang Gao, Nicolo Colombo, and Wei Wang. 2021. Adapting by pruning: A case study on bert. arXiv preprint arXiv: 201.0305.344
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott. Michael Auli, and Sergey Edunov. 2019. Facebook fair's wmt19 news translation task submission. arXiv preprint arXiv: 1907.06616.
Jiabao Han and Hongzhi Wang. 2021. Generative ad- versarial networks for open information extraction. Advances in Computational Intelligence, 1(4):1-11.
Kevin Pei, Ishan Jindal, Kevin Chen-Chuan Chang. Chengxiang Zhai, and Yunyao Li. 2022. When to use what: An in-depth comparative empirical analysis of openie systems for downstream applications. arXiv preprint 2xXx:22211.02222
Martin Josifoski, Nicola De Cao, Maxime Peyrard, and Robert West. 2021. Genie: generative information extraction, arXiv preprint arXiv:21 12.

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-5511
Youngbin Ro, Yukyung Lee, and Pilsung Kang. 2020. Multi oie: Multilingual open information extrac- tion based on multi-head attention with bert. arXiv preprint 2009.0200.08//2
Hadeel Saadany and Constantin Orasan. 2021. Bleu, meteor, bertscore: Evaluation of metries performance in assessing critical translation errors in sentiment- oriented text. arXiv preprint arXiv: 109. 14250.
Jacob Solawetz and Stefan Larson. 2021. Lsoie: A large-scale dataset supervised open information extraction. arXiv preprint arXiv:2101.11177
Gabriel Stanovsky and Ido Dagan. 2016, Creating large benchmark for open information extraction. Proceedings of the 2016 Conference on Empir- ical Methods in Natural Language Processing, pages 2300-2305.
Michael Vasilkovsky, Anton Alekseev, Valentin Ma- lykh, Ilya Shenbin, Elena Tutubalina, Dmitriy Sa- likhov, Mikhail Stepnov, Andrey Chertok, and Sergey Nikolenko. 2022. Detie: Multilingual open informa- tion extraction inspired by object detection. Pro- ceedings of the 36th AAAI Conference on Artificial Intelligence.
Pengcheng Yin, Nan Duan, Ben Kao, Junwei Bao, and Ming Zhou. 2015. Answering questions with com- plex semantic constraints open knowledge bases. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Manage- ment, pages 1301-1310.
Junlang Zhan and Hai Zhao. 2020. Span model for open information extraction accurate corpus. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9523-9530.
Mengli Zhang, Gang Zhou, Wanting Yu, and Wenfen Liu. 2021. Far-ass: Fact-aware reinforced abstractive sentence summarization. Information Processing Management, 58(3): 102478.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.0965.
A Empirical Results
We present our empirical results tables 9, 10, and 11.

---

Sentence Tuple
Combined Tuple- Tuple Entailment
Tuple
CaRB Score
Model
Training Set
Benchmark
Entaihment
Entailment
F1
P
F1
P
R
Fl
P
R
F1
MultiOOE
OIE4
LSOIE-wiki
0,396
0.318
0.353
0.953
0.381
0.545
0.595
0.488
0.536
0.591
0.467
0.522
Abstractive TS
O1E4
LSOIE-wiki
0.496
0.369
0.423
0.964
0,432
0.596
0.614
0.525
0.566
0.608
0.499
0.548
Abstractive T5
OIFA Back Translated
LSOIE-wiki
0.5
0.483
0.491
0.961
0.439
0.003
0.627
0.546
0.584
0.640
0.510
0.568
OIE4 with SuRE
Abstractive 15
LSOIE-wiki
0.518
0.49
0.504
0.963
0,436
0.601
0.632
0.565
0.597
0.645
0.511
0.570
Relations
OIE4 Back Translated
Abstractive TS
wiki
0.538
0.527
0.532
0.974
0.571
0.720
0.645
0.670
0.657
0.660
0.611
0.634
with SuRE Relations
OuIiiOIE
OIE4
ReOIE2016
0.565
0.373
0.449
0.939
0.351
0.511
0.835
0.504
0.629
0.763
0.477
0.587
Abstractive 13
O1E4
RcOIE2016
0.733
0.449
0.557
0.953
0,425
0.588
0.861
0.580
0.693
0.779
0.543
0.640
Abstractive T3
OIB4 Back Translated
Re01E2016
0.706
0.565
0.628
0.948
0,418
0,580
0.855
0.582
0.693
0.806
0.531
0.640
QIF4 with SaRE
Abstractive T5
RcOIE2016
0.757
0.572
0.652
0.953
0.424
0.587
0.871
0.602
0.712
0.814
0.531
0.643
Relations
OIE4 Back Translated with SuRE Relations
Abstractive T3
Rc01E2016
0.813
0.647
0.72
0.976
0.574
0.723
0.894
0.736
0.808
0.823
0.684
0.747
MuliPOOE
OlE4
CaRB
0.525
0.309
0.389
0.935
0.357
0.517
0.856
0.538
0.661
0.682
0.487
0.568
Abstractive TS
OIE4
CaRB
0.619
0.336
0.949
0.431
0.593
0.882
0.592
0,709
0.694
0.526
0.599
Abstractive T5
OIE4 Back Translated
CaRB
0.592
0.394
0.473
0.945
0.422
0.583
0.843
0.578
0.686
0.682
0.491
0.571
O1E4 with SuRE
Abstractive TS
CaRB
0.619
0.389
0.478
0.951
0.428
0.591
0.862
0.584
0.697
0.701
0.495
0.580
Relations
OIF4 Back Translated
Abstractive T5
CaRB
0.647
0.442
0.525
0.975
0.572
0.721
0.884
0.707
0.786
0.702
0.619
0.658
with SuRF Relations
MultPOIE
OIE4
WiRe57
0.45
0.343
0.389
0.960
0.362
0.526
0.668
0.572
0.617
0.378
0.574
0.456
Abstractive T3
O1E4
WiRc57
0,519
0.357
0.423
0,988
0.355
0.523
0.665
0.613
0.638
0.361
0,586
0.447
Abstractive T3
OIE4 Back Translated
WiRe57
0.502
0.399
0.445
0.946
0.475
0.632
0.642
0.675
0.658
0.290
0.661
0.403
O1E4 with SuRE
Abstractive T3
WiRe57
0.506
0.391
0.981
0.469
0.635
0.633
0.670
0.651
0.284
0.678
0.401
Relations
O1E4 Back Translated
Abstractive TS
WiRe57
0.537
0.37
0.439
0.990
0.371
0.539
0.665
0.611
0.637
0.377
0.556
0.449
with SuRE Relations
Table 9: Empiriaal results of different models on different benchmarks. Differences in the number of inferred relations each of the benchmarks influences the relative performance of each model. The benchmarks are listed from lowest to highest proportion of relations with inferred predicates.
Sentence: Tuple Fntailment
Comhined Tuple- Tuple Fintillment
Tuple-Tuple Entailment
CaRB Score
Model
Training Set
Benchmark
P
R
F1
MultiPOBE
O1B4
Re-OIE2016
0,813
0.647
0.72
0.976
0.574
0.723
0.894
0.736
0,808
0.823
0.684
0.747
MultPOIE +
O1B4
Re-OIE2016
0.601
0.561
0,581
0.972
0.563
0.713
0.869
0.737
0.798
0.675
0.729
Abstractive TS
MuluPOLE+
OIB4 Back Translated
Re-O1E2016
0.64
0.554
0.594
0.964
0.678
0.796
0.864
0.789
0.255
0,757
0.722
0.739
Abstractive T5
Mulu'OlE+ +
OIE4 with SuRE Relations
Re OIE2016
0.584
0.569
0.577
0.962
0.591
0.732
0.860
0.756
0.805
0,788
0.689
0.735
Abstractive TS
MultPOIE
OIF4 Back Translated
Re-OIE2016
0.599
0.542
0.569
0.965
0.683
0.800
0.846
0.768
0.805
0.764
0.701
0.731
Abstractive T5
with SuRE Relations
MultPOIE
OIB4
CaRB
0.647
0.442
0,525
0.975
0.572
0.721
0.884
0.707
0.786
0,702
0.619
0.658
MulrPOIE +
OIB4
CaRB
0.525
0.404
0.457
0.970
0.568
0.717
0.859
0.699
0,770
0.678
0.621
0.648
Abstractive T5
MulufOLE+ +
OIE4 Back Translated
CaRB
0.544
0.402
0.463
0.958
0.684
0.798
0.874
0.767
0.817
0.651
0.670
0.661
Absiractive T5
MultPOIF
OIF4 with SuRE
Abstractive TS
Relations
CaRB
0.518
0.413
0.46
0.961
0.545
0.735
0.851
0.719
0.780
0.670
0.630
0.649
MulifOlE +
OIE4 Back Translated
CaRB
0.531
0.409
0.462
0.957
0.690
0.802
0.859
0.757
0,805
0.634
0.663
0.648
Abstractive T5
with SuRE Relations
MuluPorE
OIB4
WiRe57
0.537
0.37
0.439
0.990
0.371
0.539
0.665
0.611
0.637
0.377
0.556
0.449
MulufolE+ +
OIF4
WiRe57
0.481
0.376
0.422
0.992
0.621
0.764
0.625
0.755
0.684
0.264
0.732
0.388
Absiractive T5
MultPOIF
Abstractive TS
OIE4 Back Translated
WiRe57
0.476
0.39
0.429
0.988
0.564
0.718
0.639
0.730
0.681
0.312
0.691
0.430
MuliPOIE+ +
OIF4 with SuRF
WiRe57
0.482
0.39
0.431
0.958
0.600
0.738
0.651
0.728
0.688
0,283
0.722
0.407
Abstractive TS
Relations
MuluPOLE+ +
OIB4 Back Translated
WiRe57
0.458
0.398
0.426
0.948
0.656
0.775
0.633
0.227
0.677
0.294
0.723
0.418
Abstractive 15
with SuRE Relations
Table 10: Empirical results where the relations extracted by MultiPOlE and abstractive OpenIE are combined. Redundant relations removed after the combination of extractions. Redundant relations are relations that are entailed by at least one other relation in the same sentence. If two relations entail each other, the shorter one is removed.

---

Sentence- Tuple Entailment
Combined Tuple- Tuple Entailment
Tuple.Tophe Entailment
CaRB
Model
Training Set
Benchmark
RcO1E2016 Inferred
MultiPOlE
OIE4
0.231
0.304
0.452
0.411
Predicates Args
ReO1E2016 Inferred
Abstractive T5
OIE4
0.231
0.219
0.394
0.343
Predicates or Args
RcO1E2016 Inferred
Abstractive
OIE4 Back Translated
0.152
0.236
0.424
0,380
Predicates or Args
OIE4 with SuRE Relations
Re01E2016 Inferred
Abstractive T5
0.223
0.236
0.386
0.408
Predicutes Args
OIE4 Back Translated
ReO1E2016 Inferred
Abstraetive T5
0.087
0.189
0.481
0.475
with SuRE Relations
Predicates or Args
CaRB Inferred
MultiPOIE
OIE4
0.116
0.520
0.641
0.605
Predicutes Angs
CaRB Inferred
Abstractive T5
OIE4
0.109
0.362
0.522
0.482
Predicates or Args
CaRB Inferred
Abstractive
OIE4 Back Translated
0.082
0.346
0.509
0.495
Predicates Args
OIF4 with SuRF Relations
CaRB Inferred
Abstractive T5
0.128
0.360
0.511
0,490
Predicates Args
OIE4 Back Translated
CaRB Inferred
Abstractive T5
0.099
0.289
0.532
0.498
with SuRF Relations
Predicates or Args
WiRe57 Inferred
MultiPOIE
OIE4
0.051
0.346
0.546
0.536
Predicates Args
WiRe57 Inferred
Abstractive T5
OIE4
0.059
0.427
0.544
0.599
Predicutes or Angs
WiRe57 Inferred
Abstractive
OIE4 Back Translated
0.057
0.501
0.523
Predicates Args
OIE4 with SuRE Relations
WiRe57 Inferred
Abstractive T5
0.067
0.433
0.574
0.605
Predicates or Args
OIE4 Back Translated
WiRe57 Inferred
Abstractive T5
0.043
0.341
0.468
0.509
with SuRE Relations
Predicutes or Args
Table 11: Empirical results of models where the gold standard consists only of relations with inferred predicates or arguments. We only measure recall in this case hecause relations are extracted per-sentence, so relations that do not have inferred predicates will also be extracted, which will lower the precision.


Reviews:
[{"review_id": "YL0b7FQ2H4", "reviewer": "Reviewer_8EaF", "summary": "", "questions": "• Line 292-294 is not clear, as the gold standard already indicates that there is an entailment relationship with the original sentence. Please clarify why you add this extra computation.\n• It would have been nicer to conduct more experiments in downstream tasks to showcase the trained model on the created abstractive OIE dataset.\n• In the experiments in Figure 1, the T5 trained model with \"backtranslated and SuRe augmented\" underperforms even the model trained with the original data. If this happens, please explain what the benefit is of creating datasets using both approaches for this task?\n• It would be nice to evaluate the trained model on a proportion of the datasets that only contain inferred relations to better understand the benefits of different approaches for augmentation (before combining the models, more analysis on inferred relations is required).\n• Why does Multi2OIE outperform the trained models on 0.271(CaRB)? Isn't it expected that on this dataset, the T5 trained model on abstractive data gets closer to or better than Multi2OIE? Qualitative analysis can help understand the model performances.\n• It would be nice to add another benchmark test dataset, which is subsampled from the augmented abstractive OIE4 dataset, and evaluate the trained models on the same dataset given they are from the same distribution.\n• Adding qualitative analysis is useful to understand the limitations of the model.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank your review and feedback on our paper. We feel encouraged that you find our contributions significant to the NLP community. All concerns raised will be fully addressed in the camera-ready version.\n\nQuestion: Line 292-294 is not clear, as the gold standard already indicates that there is an entailment relationship with the original sentence. Please clarify why you add this extra computation.\n\nResponse 1: Thank you for catching this typo. It is meant to be whether the paraphrased sentence entails the gold standard relations. This is intended to ensure the backtranslated sentence contains the same relations as the original sentence. We will fix this in the camera-ready version.\n\nQuestion: It would have been nicer to conduct more experiments in downstream tasks to showcase the trained model on the created abstractive OIE dataset.\n\nResponse 2: We agree it would be nicer to add more experiments on downstream tasks. We wanted to use an OpenIE application that was recent and had a quantitative metric for evaluation. Unfortunately, when we reached out to different authors for their code for several applications that use OpenIE we were only able to get code for QUEST. When we gain access to further applications we will analyze the impact of the created Abstractive OpenIE dataset.\n\nQuestion:  In the experiments in Figure 1, the T5 trained model with \"backtranslated and SuRe augmented\" underperforms even the model trained with the original data. If this happens, please explain what the benefit is of creating datasets using both approaches for this task?\n\nResponse 3: This is a very good point. Figure 1 is more like an ablation study to understand whether backtranslation, SuRE-augmentation or both together results in a high quality Abstractive OpenIE training dataset. All these derived datasets are useful, representing different levels of abstraction. We anticipate that a better abstractive model (better than T5) can benefit from any/all of these derived datasets.\n\nQuestion: It would be nice to evaluate the trained model on a proportion of the datasets that only contain inferred relations to better understand the benefits of different approaches for augmentation (before combining the models, more analysis on inferred relations is required).\n\nResponse 4: Thank you for this suggestion. We have thought of these experiments, but the task definition makes performing these experiments tricky. Because OpenIE works on a per-sentence level, we have previously investigated performance on sentences with at least one inferred relation. Our observations were that the performance on sentences with at least one inferred relation was highly correlated with the performance on all sentences, so we did not include those results.  For instance, Multi2OIE outperformed all Abstractive OpenIE models in sentence-tuple entailment F1 score for each benchmark except WiRE57, where T5 trained on SuRE-augmented OIE4 had 0.617 and Multi2OIE had 0.524.\n\nWe also performed an experiment where we computed recall for only the inferred relations in the benchmarks, and found that T5 trained on SuRE-augmented OIE4 had the highest sentence-tuple entailment recall for WiRE57, with 0.433 compared to 0.346 for Multi2OIE. On CaRB, T5 trained on SuRE-augmented OIE4 had 0.350 recall and Multi2OIE had 0.519, and on ReOIE2016 T5 trained on SuRE-augmented OIE4 had 0.238 recall and Multi2OIE had 0.306. With the addition of extra space in the camera-ready paper, we will add these experiments.\n\nQuestion: Why does Multi2OIE outperform the trained models on 0.271(CaRB)? Isn't it expected that on this dataset, the T5 trained model on abstractive data gets closer to or better than Multi2OIE? Qualitative analysis can help understand the model performances. \n\nResponse 5: The performance of Multi2OIE is better than Abstractive OpenIE on every benchmark except 0.338 (WiRE57), which contains the highest number of inferred relations among test sets. Upon manual inspection, we find that the annotation of relations among different test sets is not consistent, which is one of the reasons why the performance of all OpenIE models on WiRE57 is lower than on other benchmarks. We will include this qualitative analysis in the camera-ready version.\n\nOur focus is on inferred relations in the datasets\nWe want models to be good on datasets with no inferred relations, but for that we might need better higher-parameter models.\nThis is a first step towards abstractive OpenIE\nExpect better models in the future\n\nQuestion: It would be nice to add another benchmark test dataset, which is subsampled from the augmented abstractive OIE4 dataset, and evaluate the trained models on the same dataset given they are from the same distribution.\n\nResponse 6: Thank you for this suggestion. The benchmarks we use are all manually annotated or annotated using crowd sourcing, while the training data are all weakly labeled. As a result, we believe the gold standards for the benchmarks are more likely to be correct than the gold standards for the training data, which do sometimes contain labeling errors such as typos or ignoring negations like not or never. As a result, we do not want to introduce a new gold standard based on potentially erroneous training data and instead focus on evaluating on existing benchmarks because they already contain some inferred relations and we can be more certain of their correctness.\n\nQuestion: Adding qualitative analysis is useful to understand the limitations of the model.\n\nResponse 7: Thank you for this suggestion. We did perform some and found that the abstractive model tends to copy the training data, such as using relations from SuRE when fine-tuned on SuRE augmented data. We will include qualitative analysis in the camera-ready version. \n\nWe also thank the reviewer for pointing out style issues and typos. We will fix these in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "• New methods for evaluating models in the abstractive OpenIE task are explored, which is encouraging for future research in this area.\n• The approach for augmenting data and the proposed evaluation metrics are significant contributions and could be useful in any area within the community.\n• The dataset is evaluated on a downstream question-answering task, which demonstrates the different applications of the dataset.\n• The quality of the augmented data is comprehensively evaluated and checked.\n• In general, the authors present clear contributions, and the paper is well-motivated. It is nicely structured, and the problem and proposed solution in each section are clear and easy to understand.\n", "reasons_to_reject": "The limited scope of the experiments is not sufficient to assess if the method would work equally well in other scenarios (models and downstream tasks). Additionally, there is no qualitative analysis to help understand the limitations and strengths of the annotation task, dataset, and the trained model.", "questions_to_authors": "• Line 292-294 is not clear, as the gold standard already indicates that there is an entailment relationship with the original sentence. Please clarify why you add this extra computation.\n• It would have been nicer to conduct more experiments in downstream tasks to showcase the trained model on the created abstractive OIE dataset.\n• In the experiments in Figure 1, the T5 trained model with \"backtranslated and SuRe augmented\" underperforms even the model trained with the original data. If this happens, please explain what the benefit is of creating datasets using both approaches for this task?\n• It would be nice to evaluate the trained model on a proportion of the datasets that only contain inferred relations to better understand the benefits of different approaches for augmentation (before combining the models, more analysis on inferred relations is required).\n• Why does Multi2OIE outperform the trained models on 0.271(CaRB)? Isn't it expected that on this dataset, the T5 trained model on abstractive data gets closer to or better than Multi2OIE? Qualitative analysis can help understand the model performances.\n• It would be nice to add another benchmark test dataset, which is subsampled from the augmented abstractive OIE4 dataset, and evaluate the trained models on the same dataset given they are from the same distribution.\n• Adding qualitative analysis is useful to understand the limitations of the model.\n", "paper_topic_and_main_contributions": "The paper has two main contributions. First, they introduce an abstractive Open Information Extraction (OpenIE) task as opposed to the extractive open information task, which is a comparatively easier problem. Second, they extend the existing dataset to create a new abstractive OpenIE dataset and a semantic-based evaluation metric to train and evaluate models on them for extracting abstractive relations.\n• The paper is well-motivated, and the problem statement is clear.\n• They describe the dataset creation task in detail.\n• Experiments support the authors' claims. They show that combining the results of the trained model with the augmented abstractive dataset and previous extractive models improves the final prediction on inferred relations. They also study the impact of each augmentation approach separately on benchmark test datasets.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Section 3.1, you have not mentioned the dataset (OIE4) you used to create your training data, although you mentioned it in the tables."}, "extra_text": {}}, {"review_id": "FyqAlEVSN0", "reviewer": "Reviewer_GwEy", "summary": "", "questions": "Results showed that the proposed abstractive OpenIE models achieved higher performance on inferred relations, which previous extractive OpenIE struggled with.\nThe paper is well-written and explains clearly the related works, motivation, as well as the proposed approach.\nA thorough analysis is needed. It should show that the proposed model does not harm the performance of explicit, or un-inferred relations.\nThe approach is interesting and will probably be useful for the community. However, the proposed method relies on quite many other tasks, which may lead to propagated errors and not adaptable to other domains and languages.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank the reviewer for your constructive criticism and feedback. We feel encouraged that you find our contributions will be useful to the NLP community. All concerns raised will be fully addressed in the camera-ready version.\n\nQ: A thorough analysis is needed. It should show that the proposed model does not harm the performance of explicit, or un-inferred relations. \n\nThank you for this suggestion. We agree that looking at the performance on uninferred and inferred relations only would be an interesting topic to explore further. We have thought of these experiments, but the task definition makes performing these experiments tricky. Because OpenIE works on a per-sentence level, we have previously investigated performance on sentences with at least one inferred relation. Our observations were that the performance on sentences with at least one inferred relation was highly correlated with the performance on all sentences, so we did not include those results.  For instance, Multi2OIE outperformed all Abstractive OpenIE models in sentence-tuple entailment F1 score for each benchmark except WiRE57, where T5 trained on SuRE-augmented OIE4 had 0.617 and Multi2OIE had 0.524.\n\nWe also performed an experiment where we computed recall for only the inferred relations in the benchmarks, and found that T5 trained on SuRE-augmented OIE4 had the highest sentence-tuple entailment recall for WiRE57, with 0.433 compared to 0.346 for Multi2OIE. On CaRB, T5 trained on SuRE-augmented OIE4 had 0.350 recall and Multi2OIE had 0.519, and on ReOIE2016 T5 trained on SuRE-augmented OIE4 had 0.238 recall and Multi2OIE had 0.306. With the addition of extra space in the camera-ready paper, we will add these experiments.\n\nAs for non-inferred relations, we have not investigated whether performance on non-inferred relations is degraded by Abstractive OpenIE. With the addition of extra space in the camera-ready paper, we will add these experiments.\n\n\n\nQ: The proposed method relies on quite many other tasks, which may lead to propagated errors and not adaptable to other domains and languages.\n\nWe would like to draw attention to the fact that all OpenIE training sets are generated using weak labeling. We agree that adding more components to the pipeline will lead to more errors. However, manually labeling from scratch is a very tedious task and we show that though noisy there is a way to construct Abstractive OpenIE training dataset at scale. We also attempted to minimize noise by testing the entailment of the back translation and SuRE augmentation we used when constructing an abstractive training dataset. For the evaluation metric, we also rely on entailment. However, evaluating LLM generations is still an active area of research. Once a better evaluation metric is available, we will gladly use it.\n\nAs mentioned in the limitation section, this approach may not be suitable for other languages, but future works may still take inspiration from this one.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Results showed that the proposed abstractive OpenIE models achieved higher performance on inferred relations, which previous extractive OpenIE struggled with.\nThe paper is well-written and explains clearly the related works, motivation, as well as the proposed approach.", "reasons_to_reject": "A thorough analysis is needed. It should show that the proposed model does not harm the performance of explicit, or un-inferred relations.\nThe approach is interesting and will probably be useful for the community. However, the proposed method relies on quite many other tasks, which may lead to propagated errors and not adaptable to other domains and languages.", "questions_to_authors": "Results showed that the proposed abstractive OpenIE models achieved higher performance on inferred relations, which previous extractive OpenIE struggled with.\nThe paper is well-written and explains clearly the related works, motivation, as well as the proposed approach.\nA thorough analysis is needed. It should show that the proposed model does not harm the performance of explicit, or un-inferred relations.\nThe approach is interesting and will probably be useful for the community. However, the proposed method relies on quite many other tasks, which may lead to propagated errors and not adaptable to other domains and languages.", "paper_topic_and_main_contributions": "In this paper, the authors proposed and developed an abstractive OpenIE (Open Information Extraction) model, in contrast to traditional ones, which are extractive. First, a dataset specifically for abstractive IE is built via two methods, back translation and data augmentation via relation extraction (RE). They make use of generative models, e.g. T5, to generate relations from a sentences. First, predicates are extracted from the sentence. Then, for each predicate the model predicts the arguments for the relation with that predicate. Results showed that the proposed abstractive OpenIE models achieved higher performance on inferred relations, which previous extractive OpenIE struggled with.\n\nThe paper is well-written and explains clearly the related works, motivation, as well as the proposed approach.\n\nA thorough analysis is needed. It should show that the proposed model does not harm the performance of explicit, or un-inferred relations.\nThe approach is interesting and will probably be useful for the community. However, the proposed method relies on quite many other tasks, which may lead to propagated errors and not adaptable to other domains and languages.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AGm7haXBnZ", "reviewer": "Reviewer_TQSQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to thank the reviewer for your constructive criticism and feedback. However, we respectfully disagree with the statements that the \"Motivation is flawed\" or \"Justification of the work is flawed\", \"evaluation metrics are ill-suited for the task\", and \"authors do not take into consideration this limitation [English]\". \n\nWe believe there are some misunderstandings with regards to the data generation process we describe in this paper. We are not using back translation to generate new triples, but rather to generate new sentences that maintain the same relations as the sentences they are derived from. We will rewrite this section to be clearer in the camera ready version.\n\nWe do not inflate the number of relations in the benchmarks. Instead, we are attempting to rectify deficiencies in existing OpenIE models in extracting certain relations by introducing inferred relations and transforming existing extractive relations in training sets in a scalable way.\n\nRegarding your first point, we agree that there is previous work with inferred relations. However, of those approaches, ClausIE, LSOE, and PragmaticOIE are rule-based methods which have since been replaced in favor of supervised OpenIE models. We consider them similar to OpenIE6, a supervised OpenIE model which has baked-in components to deal with specific inferred relations containing [is], [of], and [from]. Our aim is to introduce a general method for extracting inferred relations that does not require manually pre-specifying exceptions so that it is readily usable to different domains. We will cite these as relevant previous works in our camera-ready paper. We agree that Gen2OIE is a very relevant model that is able to extract inferred relations, and we will cite it and include it in our experiments in our camera-ready paper.\n\nRegarding your second point, the creation of relation tuples is strictly for the training sets, not the benchmarks we are evaluating on. All OpenIE training sets have been weakly labeled. As a result, there is no guarantee that the gold standards for a given sentence in an OpenIE training set constitute all relations within that sentence. This can be observed by the fact that SuRE, a relation extraction model that can only extract a predefined set of relations, is still able to extract new relations from sentences in the training set that are not entailed by any existing gold standard relations in that sentence. Table 2 shows that this data augmentation increases the number of relations in the training set by roughly 10\\%, which is not a trivial amount.\n\nRegarding generation of the dataset, the purpose of back translation is not to generate new relations, but rather a way to generate a new sentence where the old relation tuples could become inferred. The sentence retains the same meaning and the relation tuples stay the same so that we can be reasonably confident that even if the relation tuples contain tokens no longer in the back translated sentence, they still can be extracted from that sentence. The additional validation steps are intended to protect against noise resulting from the back translation process. It is by design that the meaning of the relation tuples and the sentence are kept the same. We use an off-the-shelf SuRE model and state in lines 488-489 that we chose it because it is a state-of-the-art relation extraction model. We will clarify that it is off-the-shelf within the camera-ready version.\n\nRegarding why we chose back translation, we required that the paraphrasing method would retain proper nouns while changing the syntax and word choice of the sentence. We previously experimented with different sentence simplification methods like MUSS, but found that they either did not retain the proper nouns or did not substantially change the syntax of the sentence. Back translation is another paraphrasing method that has been previously studied which we decided to use because it suited our particular needs.\n\nRegarding the evaluation metrics, we agree that the parts of the tuple are best evaluated part-to-part rather than combining them before measuring entailment. However, entailment models behave very unpredictably when the hypothesis and premise are very short. For example, using our entailment model \"is\" does not entail \"is\" very much, giving around a 0.5 entailment score. Similarly, \"Nepal\" and \"Nepal\" gives around a 0.55 entailment score. Given that many predicates in all datasets are single tokens and some arguments are single tokens, this makes it very difficult to obtain accurate scores unless you combine the parts of each tuple to form phrases. Our purpose is to introduce a new task, and the creation of a new metric that can perform this part-to-part comparison would be an interesting direction for future research.\n\nRegarding the limitation to English, we agree that our work is not generalizable to other languages, as we mention in the Limitations section, lines 585-588. \n\nWe also thank the reviewer for pointing out style issues and typos. We will fix these in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think the focus of extracting synthetic relations is interesting and few work have investigated the topic, especially in the neural approaches, but it is not clear that the proposed data augmentation is actually helping with this since it removes implicit and non-lexicalised relations in the entailment-based denoising. The improvement in performance on CaRB, for example, may be due to more examples allowing the model to overcome ordering problem of relations for the model\nRegarding the evaluation metrics, I completely agree with the authors on the necessity of semantic-based metrics for the area. I suggest the authors to think how to combine their idea with structure preservation of the extractions to guarantee semantic coherence between the extraction and the gold standard\n\nTL;DR\nFocus on the extraction of synthetic relations in OIE, which is often neglected in the literature;\nProposes a semantics-based evaluation metric.", "reasons_to_reject": "First, it is important to point out that I find the main argument of the authors regarding the limitation of the task and the available models to stem mostly from unknowing the relevant literature. However, I agree with them that there is a heavy focus on \"extractive\" OIE, especially recently due to OIE as a labelling approach that became popular with available datasets for OIE. Examples of work on OIE that deal with synthetic relations are present in the literature since earlier approaches, e.g. [1,2], or newer approaches, e.g. [3,4], both for the English language and others. Second, while the ability to extract implicit or non-lexicalised relations is an important feature for OIE extractors and datasets, the argument that current manually curated datasets have too few such relations and the proposal to artificially inflate their number doesn't seem natural or even reasonable. The authors' proposal for the generation of new triples through back-translation paraphrasing relies on the fact that no new information is included in the dataset. Such a resource only makes sense in an \"extractive\" paradigm, as the main problem for \"abstractive\" models would thus be how to evaluate their extractions, in my point of view - a problem the authors accurately identify. Notice that a small number of extraction per text/sentence is *not* a problem faced by OIE. As such, there is little incentive to artificially create more extractions. As such, I find the justification of the work flawed.\n\nRegarding the generation of the dataset, it is not clear why the authors employ translation models, which are more prone to errors than simply paraphrase technology. Also, by relying on extractions on traditional relation extraction technology in a context of a high diversity of relations with similar semantics, there seems to be a high introduction of noise in the dataset. Further, the authors fail to discuss the training and testing procedures of the SuRE model or report on its performance. \n\nRegarding the evaluation metrics, the proposed metrics have similar shortcomings to previous ones, such as that of Stanovsky and Dagan or of CaRB. OIE is a structural task, meaning that the results are structured. By concatenating the parts of the tuple, i.e. conflating the structure, in order to evaluate entailment, we lose the structure. As such, extractions with different semantic content such as (Salleri, is, a city in Nepal) -this informs on the category of things or concept that the individual Salleri participates-  and (Salleri, is a city in, Nepal) -this informs a located_in relationship between the entities Salleri and Nepal- are treated as the same.\n\nIn Section 5, pg 7 col 2, the authors seem to manipulate the vertical alignment of the text block as well as avoid using subsections in order to fit their context within the 8-page limit, making the text in the first line of each paragraph clash with the preceding paragraph.\n\nThe authors only consider the English language in their work. English is relatively poor in terms of syntax, and for a task that is so dependent on syntax, such as OIE, the performance of the method may stem from the language characteristics. Also, since the authors are limited to the English language, in my opinion, they should very explicitly state that, as their results may simply not generalise to OIE as a whole but to OIE in English.\n\nTL;DR\nThe motivation is flawed, and the authors do not seem to have a good grasp of the related literature;\nThe creation of the dataset is not clearly discussed, and the noise control performed by the authors seems to make their data augmentation processing innocuous as any implicit and non-lexicalised relation, i.e. the ones they are trying to capture, would be cancelled, thus only introducing paraphrases of existing relations in the dataset;\nThe evaluation metrics are, in my opinion, ill-suited for the task;\nThe authors only consider the English language, which is a considerable limitation for such a syntax-reliant task as OIE and the authors do not take into consideration this limitation in their work.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The present paper proposes a dataset, a model, and evaluation metrics for Abstractive Open Information Extraction, or as more common in the literature OIE with synthetic relations. The authors create a dataset for OIE by data augmentation off the OIE dataset through back-translation of sentences and extraction of new tuples employing a system for traditional relation extraction trained on the original corpus. Further, the authors propose and implement a model for abstractive OIE by fine-tunning the T5 model on the proposed corpus in a simialr fashion to what is done by Kolluru et al in GEN2OIE. additionally the authors propose two semantic-based evaluation metrics employing BERT-based textual entailment model and compare their proposal with Multi2OIE on different datasets using both CaRB lexical metric and their proposed three metrics.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Del Corro, L., & Gemulla, R. (2013, May). Clausie: clause-based open information extraction. In Proceedings of the 22nd international conference on World Wide Web (pp. 355-366).\n[2]  Xavier, C. C., Strube de Lima, V. L., & Souza, M. (2015). Open information extraction based on lexical semantics. Journal of the Brazilian Computer Society, 21, 1-14.\n[3] Sena, C. F. L., & Claro, D. B. (2020). PragmaticOIE: a pragmatic open information extraction for Portuguese language. Knowledge and Information Systems, 62(9), 3811-3836.\nquestions\n[4] Kolluru, K., Mohammed, M., Mittal, S., & Chakrabarti, S. (2022, May). Alignment-augmented consistent translation for multilingual open information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 2502-2517).", "Typos_Grammar_Style_and_Presentation_Improvements": "- references to tables and figures in the text should be capitalised.\n-pg 3 col 2 line 204: Closed Relation Extraction -> this is not a common term to the area. Either use Traditional Relation Extraction or simply Relation Extraction\npg 7 line 497: T he -> The\n- section 5: get rid of the negative \\vspaces\nHow are the prompts for the segmented narratives constructed? in which sense do they differ from the nin-segmented narratives?\nWhat does Figure 3 means? What is measured on X-axis in the graphs? What do the authors mean by partial observability and full observability?"}}]

Author rebuttal:

Meta-review: This paper focuses on abstractive OIE where the relation may not explicitly appear in the text. The research is meaningful and interesting. Most of reviewers express their concerns about the experiments. The experiments are not sufficient and need further improvements according to the reviewers' comments.
Decision: Accept-Main