Efficient Data Learning for Open Information Extraction with Pre-trained Language Models
Authors: Zhiyuan Fan
Shizhu He
Venue: EMNLP 2023 Findings (2023)
Keywords: information extraction
efficient
low-resource

Abstract: Open Information Extraction (OpenIE) is a fundamental yet challenging task in Natural Language Processing, which involves extracting all triples (subject, predicate, object) from a given sentence. While labelling-based methods have their merits, generation-based techniques offer unique advantages, such as the ability to generate tokens not present in the original sentence. However, these generation-based methods often require a significant amount of training data to learn the task form of OpenIE and substantial training time to overcome slow model convergence due to the order penalty. In this paper, we introduce a novel framework, OK-IE, that ingeniously transforms the task form of OpenIE into the pre-training task form of the T5 model, thereby reducing the need for extensive training data. Furthermore, we introduce an innovative concept of 'anchors' to control the sequence of model outputs, effectively eliminating the impact of order penalty on model convergence and significantly reducing training time. Experimental results indicate that, compared to previous SOTA methods, OK-IE requires only 1/100 of the training data (900 instances) and 1/120 of the training time (3 minutes) to achieve comparable results.

Efficient Data Learning for Open Information Extraction with Pre-trained Language Models
Zhiyuan Fan and Shizhu He
'The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Aulomation, Chinese Academy of Sciences, Beijing, China TTinjjn University, Tianjin, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China 2 fan20028gmail. com, shizhu. he@nlpr. shizhu.he@nlpr. ia..aaa............... ac. cn
Abstract
Recently, neural-based Openlli methods can be broadly divided into two categories: labeling-based methods (Kolluru ct al., 2020a; Vasilkovsky et al., 2022; Kotnis et al., 2022) and generation- based methods (Kolluru et al., 2020b, 2022). Labeling- based methods are known for their fast inference speed but have a limitation in that they cannot label tokens that not present in the input sentence. On the other hand, generation-bast based methods pos- sess the capacity produce supplementary tokens in order to fulfill syntactic and comprehensive se- mantic requirements, albeit necessitating extensive training data and time investment to learn the task format of OpenIE and converge towards desirable outcomes.
Open Information Extraction (OpenIE) is fun- damental yet challenging task in Natural Lan- guage Processing, which involves extracting all triples (subject. predicate, object) from given sentence. While laheling-based methods have their merits, generation-basse -based techniques offer unique advantages. such as the ability to gener- ate tokens not present in the original sentence. However, these generation- ten require significant amount of training data to learn the task form of OpenlE and substantial training time to overcome slow model conver- gence due to the order penalty. In this paper, we introduce a novel framework, OK-IE, that ingeniously transforms the task form of Ope- nIE into the pre-training task form of the TS model, thereby reducing the need for extensive training data. Furthermore, innovative concept of Anchor to control the sequence of model outputs, effectively climi- nating the impact of order penalty model convergence and significantly reducing train- ing time. Experimental results indicate that, compared to previous SOTA methods, OK-IE requires only 1/100 of the training data (900 instances) and 1/120 of the training time (3 minutes) to achieve comparable results.
methods of-
GEN2OIE (Kolluru et al., 2022) has set robust baseline for generation-base methods by model- ing the OpenIE task seq2seq problem using two T5 (Raffel et al., 2022) models. The first T5 model is employed to extract all predicates from the sentence, while the second T5 model takes a sequence concatenated from the sentence and an individual predicate as input, outputting the corre- sponding unique triple. While the GENO2IE sets strong baseline, there are still areas for improve- ment. We address these with our proposed OK-I IIE framework, an Order Agnostic Knowledge Embed- ded Information Extraction framework. Consid- ering the simplicity of predicate extraction from sentences, we retain the first stage of GEN2OIE and focus on optimizing the second stage, which is responsible for generating the complete triple.
introduce
1 Introduction
Open information extraction (Banko al., 2007) is the task of transforming unstructured text into semi-structured text. Given an input sentence, an open information extraction system can output a set of corresponding sentence with elements in a pre-defined structure (Etzioni et al., 2011). These output can be used as an additional source of in- formation to augment other tasks, such as building semi-automated knowledge graph construction sys- tems (Mausam, 2016), Question Answering (Khot al., 2017), Trigger Detection (Dukić al., 2023) and so on.
Specifically, unlike GEN2OIE, OK-IE eleverly transforms OpenIE into T5's pre-training task, span corruption. This allows the model to learn the task with less data, eliminating the need for additional training adapt the language knowledge from span corruption pre-training to the seq2seq. addition. the seq2seq presumes that only one sequence order is correct. This order information is not explic- itly provided to the model, which must learn it on
"Conresponding author.

---

OpenIE System
F1
F1%
Parforranass GOTA
IMOJIE
53.5
99.07
.50
Open1E6
52.7
97.6
Multi
52.5
97.2
IGL-OIE
52.4
97.03
CIGL-OIE
54.0
100
DetLE
52.1
96.4
GEN2OIE
51.1
94.6
OK-IE (Ours)
53.2
98.5
Pergentage of Training Data
Table I: Comparison of OpenIE systems with full data (100%)
Figure 1: The performance of both methods improves as the volume of data increases. Compared to our OK-IE, IMOJIE needs vs 0.9%) times more data to achieve approximately the same results.
about 83(50% 0.6%) to 89(80%
important, will be briefly overviewed for the sake of context.
labeling-based methods aim to tag each token in sentence with one of four labels: subject, predicate. object, or none. Representative work includes Open1E6 (Kolluru et al., 2020a), which initially marks the first triple in sentence, then merges this marked triple with the original sen- tence to form new input for identifying the sec- ond triple, itcrating in this manner until termination. DetlE (Vasilkovsky et al., 2022), on the other hand, borrows ideas from object detection in computer vision to identify all triples in sentence in one go. Nonetheless, these techniques encounter shared challenge: they lack the ability to label tokens ab- sent in the original sentence. This shortcoming hampers their ability invariably yield triples that satisfy grammatical coherency and semantic com- pleteness.
its own This leads to additional order penalty, causing confusion for the model and requiring long time to converge. OK-IE addresses this is- sue by introducing an Anehor method, explicitly instructing the model on the sequence order for gen- eration. The comprehensive workflow of OK-IE are elaborated in the Appendix A.
The effectiveness of our proposed OK-IE frame- work is exemplified particularly in low-resource scenarios the CaRB (Bhardwaj et al., 2019) benchmark, illustrated in Figure Summariz- ing, our key contributions are:
1) We introduce the OK-IE framework that har- nesses the potential of PLMs for efficient OpenIE. It uses minimal data and computational resources, an important attribute in eesource-limited scenar- ios, while achieving results comparable to full-data training.
Generation-based approaches present an alter- native perspective in OpenIE, treating both the in- put sentences and output triples as sequences. For example, IMOJIE (Kolluru et al., 2020b), lever- aging an LSTM model, sequentially generates triples. It takes an input sentence and produces the first triple, then concatenates this generated triple with the original sentence form new se- quence. The process continues iteratively until the system reaches an end token. GEN2OIE (Kolluru et al., 2022), a powerful baseline in our study, em- ploys dual T5 model setup. The first T5 model is responsible for generating all predicates from a given sentence. Subsequently, the second T5 model accepts the concatenation of the sentence and individual predicate as input, producing the corre- sponding triple for that predicate. However, these techniques generally require extensive training data and time to yield satisfying results.
2) We address the order challenge within the OK- IE framework, allowing for faster training conver- gence by eliminating the additional order penalty. 3) Our OK-IE framework not only achieves SOTA performance in a low-resource scenario but also presents a new perspective for OpenlE re- search, potentially inspiring more efficient and ef- fective approaches in this area.
2 Related Work
Predominant solutions to the OpenIE task fall into two categories: labeling- basedd and generation- based methods. This paper aims address the excessive requirements of training data and time inherent in generation-basee methods, and thus we will primarily focus on discussing generation- based techniques. Labeling- babsee methods, while

---

Despite GEN2OIE's utilization of the pre- trained language model T5 as its backbone, con- siderable discrepancy exists between the seq2seq task and T5's vre-training task, span corruption. The model still necessitates an ample amount of data to learn this new task format. Concurrently, in the context of triples, both (subject, predicate, object) and (predicate, object, subject) orders are valid. However, due to the auto-regressive charac- teristic inherent in sequence generation, training presumes only one correct generation order. More- over, this assumed order is implicit and not directly conveyed to the model. Consequently, the model might generate a correctly composed triple in an order inconsistent with the provided ground truth, resulting in an additional order penalty. This dis- crepancy lead to model confusion, slow con- vergence, and thus, requires substantial training time.
predicate founded, <id 0> <id 1> <id 2>. With predicate is the CEO of, <id 3> <id 4> <id 5>
and '<id 0> Elon Musk <id 2> founded <id 2> SpaceX <id_3> Elon Musk <id <d_4> is the CEO <id_5> Tesla', respectively. The efficacy of this seemingly simple yet innovative method is demonstrated in subsequent experiments.
3.2 Controlling Generation Order
As evidenced by the aforementioned examples, it is clear that sentinels such as '<id_i>' only denote the locations of corrupted spans within sentences. Due to the reusable nature of sentinels, it's difficult assign specific meanings to each sentinel. Hence, in the example given, we still implicitly assume particular order. To mitigate the impact of or- der penalty, we introduce the concept of Anchors which temporarily assign meanings to each sentinel to help explicitly control the generation order.
3 Method
An anchor refers to tokens surrounding the sen- tinel, intended to express the sentinel's temporary meaning. More specifically, if we want the sentinel '<id O>> to generate only subjects and nothing else, we can add anchors representing the need to gen- erate a subject on either side of the sentinel. Each anchor consists of two parts: an embedding from word table representing clear meaning, and tunable embedding(Liu et al., 2021); the afore- mentioned embeddings are concatenated to form cohesive semantic unit. For simplicity, we denote the anchors for subjects, predicates, and objects S, P. and O, respectively. It's worth noting that the same anchor is used for each subject in all triples of a sentence, ensuring that the tunable embeddings can learn more general semantic representations.
addressing the aforementioned issues, our pro- posed framework, OK-IE, pivots around two core aspects: the transformation of task format and the control over generation order. To better control the generation order, we introduce Anchors and merge them with sentinels.
3.1 Task Format Transformation
Seq2Seq are designed model relationships between two sequences. However, the T5 model is trained with a span-mask denoising objective. For instance, when the input 'Thank you <id_0> 0> me to your party <id_1> 1> week,' the expected output is '<id 0> for inviting <id_I> last.' Here, <id_i> i> are referred to as sentinels, which serve to locate the positions corrupted in the original sentence. Notably, discerned an opportunity ingeniously transform the OpenIE task to match the span corruption objective. More specifically, the input sentence and output triples are concatenated into one sentence, treating each subject, predicate, and object in the triple as a corrupted span. For instance, under the Seq2Seq task format, the input and output are: 'Elon Musk, who is the CEO of Tesla, also founded SpaceX', and '(Elon Musk; is the CEO of; Tesla) (Elon Musk; founded: SpaceX), respectively. After the task format transformation, the input and output become: 'Elon Musk, who the CEO of Tesla, also founded SpaceX. With
By integrating anchors and sentinels, we can explicitly control the model's generation der. For instance, in the original example, <id 0> <id 1> <id 2> implies an order of sub- ject, predicate, object. we want to make this order explicit, we can use bxid__>00 If we prefer the model to generate in the order of predicate, object, subject, it would be P<id Sdp O<id 1>0 S<id _=d' Note that, in practical applications,
P<i
can select appropriate order through practice, or we can control the model to generate all orders and then use majority vote strategy to determine the final output. In Figure 2, we illustrate how anchors are employed to control token generation at each sentinel position.

---

4 Experiments
Approach
F1#
F1
Baseline
29.9
31.2
4.1
Experimental Setup
Convert Form
45.3(+15.4)
49.8(+18.6)
Anchor
46.0(+0.7)
50.5(+0.7)
OpenIE System
F1
F1%
Order Control
52.1(+6.1)
52.9(+2.4)
IMOJIE
Open1E6 OK-IE
42.9
79.4
Table
Results ablation experiments on various components of framework OK-IE.
52.9
97.9
Table 2: Comparison of OpenIE systems with 0.9% data
0.9%, 5%, and 20%. We selected the most rep- resentative models from both labeling- -baiin and generation- methods, namely OpenlE6 and IMOJIE, for comparison. From Tables 2, 3, and 4, is evident that our OK-IE markedly outperforms the other two methods across all three data sizes. It's worth noting that due the minimal data avail- ability at 0.9%, IMOJIE is incapable of generating effective triples, hence its result is denoted by -.
OpenIE System
F1
F1%
IMOJIE
36.9
68.5
Open1E6
43.1
79.8
OK-IE
53.1
98.3
Table 3: Comparison of OpenlE systems with 5% data
OpenIE System
F1
F1%
4.3 Ablation Studies on System Components
IMOJIE
39.5
73.2
Open1E6
45.3
83.8
In order to analyze the impact of individual com- ponents of OK-IE on the results with finer gran- ularity, we carried out ablation study based the baseline GEN2OIE in scenario with only 900 data items. Simultaneously, to validate the influ- ence of order control on model convergence speed, we defined a new metric, F1*, which refers to the F1 score obtained after just one epoch of training. As can be seen from the Table 5, noticeable en- hancement in the optimal F1 is achieved following the transformation of the task form, which indeed confirms decrease in training data requirement. Upon introducing order control, there is swift increase in F1* and the result after only one epoch of training is very close to the optimal score. This confirms that model convergence speed has indeed been accelerated, thus reducing the training time. For details on the analysis of training data and training time, please refer to the Appendix B.2.
OK-IE
53.3
98.7
Table 4: Comparison of OpenIB systems with 20% data
To establish objective and fair comparison with other models, we opted for the benchmark dataset, CaRB (Bhardwaj al., 2019), used in pre- vious studies, applying the same evaluation metric, Fl score. For consistency, we utilized the previ- ously established OpenIE system training dataset, IMOJIE (Kolluru et al., 2020b). Owing to space limitations, further experimental details are pro- vided in Appendix B.I.
4.2 Results and Discussion
To visually demonstrate the performance of OK-IE, we compared it with several recent OpenlE systems under full training data scenarios, shown in Ta- ble ??. To facilitate a straightforward comparison between different methods, we introduce the F1% metric. This represents the Fl score of a given method within a specific data scenario, expressed as a percentage of the Fl score of the SOTA method when trained with full data. This metric simplifies performance comparisons in low-resource scenar- ios. As can be seen, the Fl scores of all methods are comparable. A significant advantage of OK- IE, however, its ability to substantially reduce the required training resources. To investigate the performance of OK-IE in low-resource situations, we set three data sizes relative to the full data set:
Conclusion
In this paper, we have presented OK-IE, an efficient framework for Open Information Extraction. By effectively leveraging pre-trained language models and addressing the challenges of generation order, OK-IE overeomes the computational inefficiencies of previous systems. Our empirical evaluations demonstrate that OK-IE only achieves compa- rable results to existing systems, but does so with significantly fewer resources. This underlines its capability in resource-constraine environments.

---

Our research stands as testament to the possi- bility of accomplishing efficient information ex- traction with reduced data and minimal training time, thereby presenting new pathways for future research within the OpenIE landscape.
and practicality for wide range of users and appli- cations.
However, it should noted that there are cir- cumstances under which OK-IE may output erro- neous triples, leading to serious inaccuracies in the knowledge databases or incorrect triple inputs to applications like ChatGPT. These errors could, in turn, result in significant misconceptions, as flawed knowledge produced upstream could greatly dam- age downstream applications, potentially leading to misdirection human activities.
Limitations
Following an examination of the CARB test set and the output generated by existing OpenlE systems, we found that these systems struggle to effectively manage cases that involve extensive sentences with multiple triples.
Therefore, we strongly urge users of our sys- tem to implement rigorous human oversight and checks on extracted knowledge, while fully eval- uating the potential harm that could be caused by inaccuracies. Our primary ethical responsibility lies in minimizing any potential for misuse of the system or misinterpretation of the output, ensure that serves the purpose of advancing knowledge rather than misleading users.
While OK-IE has the ability to enhance com- putational efficiency and the optimization of data resources, it does not sufficiently address the afore- mentioned issues. The generation-based strategy is capable of creating long, meticulously refined triple sequences, but there is still the possibility that some triples may be omitted. Similarly, the conventional labeling-basee based approach adopted in Open1166 sets an upper threshold on the number of triples that require laheling, hence capping its performance. Accurately generating all triples in extended sentences represents a promising avenue for future research.
A Acknowledgements
This work was supported by the National Key R&D Program of China (20227D0160033) and the National Natural Science Foundation of China (No.62376270, No.61831022), This work was supported by the Strategic Priority Re- search Program of Chinese Academy of Sciences (No. No.XAA2722000,, Youth Innovation Promotion Association CAS and OPPO Research Fund.
A potential pitfall for OK-IE, given its modest data requirements, is the potential for bias in sce- narios characterized by substantial differences in data characteristics. However, this issue can be tackled by the framework itself, as OK-IE exhibits the ability to rapidly adapt to new scenarios and ameliorate these biases with minimal amount of data.
References
Michele Banko, Michael Cafarella, Stephen Soder- land, Matt Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Proceedings of the 20th International Joint Conference on Artif- ical Intelligence, IJCAI'07, page 2670-2676, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
should be underscored that our current method has been specifically evaluated within the confines of English OpenIE. This presents a limitation, and we intend to explore and validate the efficacy of our approach across other languages in future work.
Sangnie Bhardwaj. Samarth Aggarwal, and Mausam Mausam. 2019. CaRB: A crowdsourced benchmark for open IE. In Proceedings of the 2019 Confer- ence Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6262-6267, Hong Kong, China. Association for Computational Linguistics.
Ethics Statement
Our proposed OK-IE system has the potential to contribute to the construction of comprehensive knowledge databases. Furthermore, given the strin- gent context window size constraints associated with upplications like ChatGPT, the use of triples extracted by OK-IE could effectively alleviate text length in input. Significantly, one of the notable advantages of our OK-IE system its ability to achieve commendable performance with minimal resource requirements, enhancing its accessibility
David Dukić, Kiril Gashteovski, Goran Glavaš, and Jan Snajder. 2023, Leveraging open information extrac- tion for improving few-shot trigger detection domain transfer.
Oren Etzioni, Anthony Fader, Janara Christensen. Stephen Soderland, and Mausam Mausam. 201 I.

---

Open information extraction: The second generation. In Proceedings of the Twenty-Second International Joint Conference Artificial Intelligence Volume Volume One, IJAAT 11, page 3-10. AAAI Press.
Antiga, Alban Desmaison, Andreas Köpf. Edward Yang. Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Jun- jie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning li- brary.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2017. Answering complex questions using open informa- tion extraction. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics Vooluee 2: Short Papers), pages 311-31 111-316, Vancouver, Canada. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2022. Exploring the limits of transfer learning with a unified text-to-tex, trans- former. J. Mach. Learn. Res., 21(1).
Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization.
Michael Vasilkovsky, Anton Alekseev, Valentin Ma- lykh, llya Shenbin, Elena Tutubalina, Dmitriy Sa- likhov, Mikhail Stepnov, Andrey Chertok, and Sergey Nikolenko. 2022. Detie: Multilingual open informa- tion extraction inspired by object detection. Proceed- ings of the AAAI Conference on Artificial Intelligence, 36(10); (:00::1111-11444 14400.
Keshav Kolluru, Vaibhav Adlakha, Samarth Aggarwal, Mausam, and Soumen Chakrabarti. 2020a. Open1E6: Iterative Grid Labeling and Coordination Analysis for Open Information Extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3748-3761, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-a natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguistics.
Keshav Kolluru, Samarth Aggarwal, Vipul Rathore, Mausam, and Soumen Chakrabarti. 2020b. IMo- JIE: Iterative memory-based joint open information extraction, In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 5871-5886, Online. Association for Computa- tional Linguistics.
Keshav Kolluru, Muqeeth Mohammed, Shubham Mit- tal, Soumen Chakrabarti, and Mausam 2022. Alignment-augmeente consistent translation for mul- tilingual open information extraction, In Proceedings of the 60th Annual Meeting of the Association for Compustatioall Linguisties (Volume I: Long Papers), pages 25172-217, Dublin, Ireland. Association for Computational Linguistics.
A
Overview of OK-IE
A.1 Workflow
The extraction process employed by OK-IE is both intuitive and flexible. Initially, an input sentence is introduced, and T5 model is employed to extract all predicates within the sentence. Subsequently, as illustrated in Section 3, the sentence and its extracted predicates are combined, with expected triple positions replaced by sentinel <id_ <اا place- holders. This combination is then fed into second T5 model, which generates the appropriate span for each sentinel- position, filling in the respec- tive subject, predicate, or object. It is important to note that the order in which OK-IE extracts the final results is entirely dependent on the training set.
Bhushan Kotnis, Kiril Gashteovski, Daniel Rubio, Am- mar Shaker, Vanesa Remrrr, Makoto Takamoto, Mathias Niepert, and Carolin Lawrence. 2022. MILIE: Modular & iterative multilingual open information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6939- 6950, Dublin, Ireland. Association for Computational Linguistics.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing.
Mausam Mausam. 2016. Open information extraction systems and downstream applications. In Proceed- ings of the Twenty-Fifth International Joint Con- ference on Artificial Intelligence, IJCAI' page 4074-4077. AAAI Press.
To manage the generation order, we rely on an- chors. For example, if the extraction is intended to proceed in predicate, object, subject order, the combination of anchor and sentinel would result in
Peid 0>P Okid 1>0 S<id 2>S.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca
Two key aspects should be noted. Firstly, the op- timal extraction order can be determined through

---

(retired to, Mothra, Infant Island After battle)
(Battra, rested in, the Arctic Ocean After the battle)
(Mothra, two Casmos to Infant Island After the battle, was accompanied by)
PLM
} Set
Structure
Prefix Embedding
After the battle, Battra rested in the Arctic Ocean, whereas Mothra retired to Infant Island, accompanied by the two Cosmos.
[TO] [S]E [P]E [OJE
OpeniE
Sentence Embedding
[T1] [PJE [SJE [OJE
Task
In In-context Embedding
Prefix
[T2] [SJE [OJE [P]E
Embedding Table
figure, 'E' denotes the sentinel, while [S), (P), and [O] represent anchors for subject, predicate, and object respectively. Blue, yellow, and purple lines illustrate three distinet generation sequences: SPO, PSO, and SOP, By integrating anchors and sentinels, control over token generation at each position is achieved, thereby governing the generation order individual triples.
Figure 2: Workflow of OK-IE.
trial and errоr in practical scenarios, or by generat- ing results under all extraction orders and utilizing majority vote strategy to determine the final ex- traction results. Secondly, for given sentence, the order of the triples to be extracted can vary. wish to extract the triple for the first predicate in subject, predicate, object order, and the triple for the second predicate in a predicate, object, subject order, it can conveniently be incorporated in the same generation process:... Skid 0>S P<id_I I>P O<id 100l22200000 ... P<id 3>P Okid 4>0 S<id_5>S. This flexible generation method opens up myriad of possibilities.
corresponding to predicate each time, whereas OK-IE can extract either one or all triples corre- sponding to the predicates, depending how the anchor and sentinel are set. This contributes to the differences in model computation efficiency.
Lastly, to extract triples using generation-ban based method, GEN2OIE introduces additional delim- iters to denote the boundaries of the subject, predi- cate, and object. In contrast, OK-IE utilizes native sentinels, effectively sidestepping this unnecessary issue.
B Experiments
B.1 Experimental Setup Details
A.2 Difference
In this study, we employed the PyTorch 1.11 (Paszke et al., 2019) framework to conduct our ex- periments. Specifically, we utilized the base series of the T5 model from the Hugging Face's 's trans- formers (Wolf et al., 2020) library, without incorpo- rating any additional parameters, except for word embeddings. The cross-entropy loss function and Adam (Kingma and Ba, 2017) optimizer were uti- lized for training. All models were trained for a consistent duration of 7 epochs.
Several distinct differences exist between OK-IE and GEN2OIE:
Firstly, GEN2OIE approaches the OpenIE prob- lem using seq2seq task format, while OK-IE em- ploys span corruption task format. This difference in task format leads to substantial variation in the amount of training data required during the training process.
Secondly, the order of the triples extracted by GEN2OIE is determined by the training data and can only follow one specific order, i.e., the order of the triples in the training data. Conversely, OK- IE, through the introduction of the Anchor, can flexibly determine the extraction order of the triples in accordance with the actual scenario, allowing for the coexistence of multiple extraction orders. Next, GEN2OIE can only extract a single triple
With regards to hyperparameter selection, we investigated batch sizes of (2, 4, 8, 16, 32) and ulti- matcly chose batch size of 4 to optimize memory usage while having no significant impact the sults. For the learning rate, we experimented with values of (5e-5, 2e-5, le-5, 5e-6) and selected 5e-5 as the optimal value. All other parameters were kept at their default values and we did not use any

---

form of hyperparameter search algorithm. In order mitigate the influence of random variance, conducted three separate random samplings under each limited training data scenario and reported the mean values as final results.
Model
Task Format
F1
flan-t5-base
seq2seq
0.0021
flan-t5-large
seq2seq
0.244
flan-t5-xl
seq2seq
0.2852
ts-base
seq2seq
As previously mentioned, OK-IE can specify any order for generating extraction results by designing anchors. During the training process, we utilized all possible orders of triples to train the model. As a side note, if GEN2OIE were to adopt this strat- egy, it would result in significant model confusion. In the evaluation phase, to avoid the influence of additional factors, did not the strategy of generating extraction results in all possible orders and then using majority vote obtain the final output. Instead, we explicitly set a unique extrac- tion order, i.e., S<id_ssii 0>S P<id_ I>P Okid 2>0.
15-large
seq2seq
t5-xl
seq2seq
0.0016
t5-base
span corruption 0.0007
t5-large
span corruption
0.0019
15-xl
span corruption 0.0023
Table Zero Shot Performance
tively reduced the required training time for the model.
C Zero Shot Performance
We employed the GEN2OIE model to generate predicates the first phase, following which we conducted zero-shot inference on i5 models across three size attributes: base, large, and XL, as well as on the flan-t5 model. The outcomes of these experiments are reported in Table 6. Due to the incapability of flan-t5 to generate in the form of span corruption, only report the results under the seq2seq task paradigm.
B.2 Training Data and Time Analysis
Evidently, conducting a direct analysis of the quan- tity of training data and training time required by two models to achieve identical performance lev- els is a complex undertaking. On one hand, the correlation between training data and time is non- negligible; in most cases, reduction in data vol- ume correlates with decreased training time. On the other hand, achieving parity in evaluation re- sults berween the two models can be challenging. Thus, the previously mentioned ratios of 1/100 for training data and 1/120 for training time were derived directly from comparing the resources re- quired by OK-IE to achieve comparable results with GEN2OIE on the full dataset, but with only 900 instances. While could theoretically start from 900 and gradually reduce the amount of train- ing data to see how performance fluctuates, the potential findings from such an approach would likely be marginal.
In order accurately quantify the model's de- mands for training data and time, we set up various scenarios with limited training data and selected some representative models to compare their stand- out results within these contexts. Regarding train- ing time, in order to impartially offset the impact of training data volume and focus on the influence of methodological improvements, we kept the train- ing data fixed. We gauged the model's convergence speed by evaluating the Fl score (IF1*) after sin- gle epoch of training. In other words, larger F1 suggests quicker model convergence, indicating that the methodological improvements have effec-


Reviews:
[{"review_id": "TfDUO1tbAM", "reviewer": "Reviewer_WhNg", "summary": "", "questions": "- Which size of T5 is used in the experiments? Would the proposed methods have different improvement levels for models of different sizes?\n- It would be helpful to report the zero-shot performance and compare OK-IE with removing the proposed methods. Especially given that span corruption is part of the pre-training task for T5, by simply transforming the task formulation, would the zero-shot performance show more difference compared to using traditional formulation?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely appreciate the time and effort you dedicated to reviewing our work. Below, we respond to each of your comments and questions.**\n\n---\n\n### Question 1:\nThe authors claim that the proposed methods improve open IE, but only show its performance on the predicate extraction task. The claim does not match the evidence provided in the paper.\n\n### Answer 1: \n\nTo begin with, it is imperative to clarify a critical misconception. In our study, what we presented is the performance of Open Information Extraction, not merely predicate extraction.\n\nConsistent with the methods we compared against and the datasets we employed; we conducted the open information extraction task, which extracts all triples, encompassing subject, predicate, and object, from given sentences. Notably, it's not confined to merely extracting the predicate. To illustrate, consider the example sentence provided in our paper `lines L196-L198`: “Elon Musk, who is the CEO of Tesla, also founded SpaceX”. Our proposed method can extract multiple triples such as '(Elon Musk; is the CEO of; Tesla)' and '(Elon Musk; founded; SpaceX)', rather than solely the predicates ‘is the CEO of’ and ‘founded’. In this instance, the common subject is 'Elon Musk', with paired predicates and objects being '(is the CEO of; Tesla)' and '(founded; SpaceX)', respectively. We've showcased the efficacy of our method in the open information extraction task within **Section 4 “Experiments”**.\n\n\nAs mentioned in `lines L068-L072` of our paper: \n>\"Considering the simplicity of predicate extraction from sentences, we retain the first stage of GEN2OIE and focus on optimizing the second stage, which is responsible for generating the complete triple \".\n\n---\n\n### Question 2:\nWould the proposed methods have different improvement levels for models of different sizes?\n\n### Answer 2:\n\nYes, the proposed methods exhibit varied improvement levels across different model sizes (base, large, xl, xxl). However, the differences in these improvements are minimal.\n\nNevertheless, it's necessary to underline that the core objective of this paper centers on enhancing model training efficiency within low-resource settings. Taking into account that increasing the model's parameter scale leads to a marked surge in memory consumption, ranging from 990MB to 45GB, this deviates from our fundamental goals of **\"low resource\"** and **\"efficiency\"**. Due to this rationale, such results were not showcased in the main body of the paper. We intend to incorporate this data in the appendix of the revised version.\n\n---\n\n### Question 3:\nEspecially given that span corruption is part of the pre-training task for T5, by simply transforming the task formulation, would the zero-shot performance show more difference compared to using traditional formulation?\n\n### Answer 3:\n\nYes, in the zero-shot setting, our method that incorporates the task format transformation demonstrates a notable performance difference when compared to the traditional formulation. \n\nIn our previous experiments, specifically with T5-xl and T5-xxl models, we found that without this transformation, the models had difficulty in generating well-structured triples. However, when applying our proposed **'task format transformation'**, they successfully generated the desired triples. In contrast, models with fewer parameters did not show significant improvement, whether the transformation was applied or not.\n\nConsidering the main focus of our paper is on enhancing model **training** efficiency in low-resource settings, we did not include these zero-shot experiment results in the main content. However, we plan to provide this data in the appendix of our revised version.\n\n---\n\n### Question 4: \nAs the proposed methods are developed based on GEN2OIE, it would be nice to see the comparison with GEN2OIE directly for low-resource settings besides the comparison with IMOJIE.\n\n### Answer 4: \nYes, we compare with GEN2OIE directly in low-resource setting. In **Section 4.3, \"Ablation Studies on System Components,\"** and as detailed in Table 5, we provide an in-depth comparison between OK-IE and GEN2OIE in low-resource settings. \nWhen compared to GEN2OIE, OK-IE boosted the F1 score in low-resource settings from 31.2 to 52.9 and accelerated the rate of convergence.\n\n---\n\n### Question 5:\nIt would be helpful to report the zero-shot performance and compare OK-IE with removing the proposed methods. \n\n### Answer 5:\nThanks for your suggestion. Based on Answer 3, we plan to supplement our appendix with additional experimental data showcasing zero-shot performance across the four parameter sizes—base, large, xl, and xxl. Specifically, this will detail the impact of utilizing versus not utilizing the “task format transformation” on their performance. We will provide detailed description in the revised version.\n\n---\n\n### Question 6:\nThe proposed techniques are based on a particular existing model.\n\n### Answer 6:\n\nIn the OpenIE task, leveraging an existing pretrained model has proven to enhance performance and reduce the requisite training data. Therefore, both OK-IE (ours) and GEN2OIE (the baseline for this paper) are based on the T5 model.\n\n---\n\n### Question 7: \nWhich size of T5 is used in the experiments? \n\n### Answer 7:\nWe utilized the T5 \"base\" configuration, comprising 220 million parameters. This is elaborated in Appendix B.1 \"Experimental Setup Details\".\n\n---\n\n### Question 8:\nThe proposed techniques keep the entire first stage of GEN2OIE.\n\n### Answer 8:\n\nThe crux of Open Information Extraction lies in extracting the full, cohesive triple, while predicate extraction is more straightforward. \n\nAs stated in `lines L068-L072` of our paper: \"Considering the simplicity of predicate extraction from sentences, we retain the first stage of GEN2OIE and focus on optimizing the second stage, which is responsible for generating the complete triple.\"\n\nMoreover, in **Appendix A2** of our paper, we have meticulously delineated the distinctions between OK-IE and GEN2OIE. It becomes evident that in aspects such as data requirements for training, generation sequence, generation efficiency, and format, OK-IE manifests clear advantages and differentiation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Significant empirical results show the effectiveness of the two proposed techniques, especially transforming task format to span corruption.\n- The task format transformation technique is well motivated", "reasons_to_reject": "- The authors claim that the proposed methods improve open IE, but the proposed techniques are based on a particular existing model (and keep the entire first stage of GEN2OIE), and only show its performance on the predicate extraction task. The claim does not match the evidence provided in the paper\n- As the proposed methods are developed based on GEN2OIE, it would be nice to see the comparison with GEN2OIE directly for low-resource settings besides the comparison with IMOJIE", "questions_to_authors": "- Which size of T5 is used in the experiments? Would the proposed methods have different improvement levels for models of different sizes?\n- It would be helpful to report the zero-shot performance and compare OK-IE with removing the proposed methods. Especially given that span corruption is part of the pre-training task for T5, by simply transforming the task formulation, would the zero-shot performance show more difference compared to using traditional formulation?", "paper_topic_and_main_contributions": "This work introduces two improvements for an existing predicate extraction model GEN2OIE. The first technique is transforming the IE task into T5’s pre-training task span corruption to improve the data efficiency of the model. The second technique is the anchor method that instructs the model on the sequence order for generation. The experimental results show the effectiveness of the two proposed techniques.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "U26tEqqsu3", "reviewer": "Reviewer_iPLc", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 2, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely appreciate the time and effort you dedicated to reviewing our work. Below, we respond to each of your comments and questions.**\n\n### Question 1:\nThere is a single dataset. \n\n### Answer 1:\nImportantly, it should be emphasized that representative works in the Open Information Extraction domain, such as **OpenIE6 (presented at EMNLP)**, **IMOJIE (at ACL)**, and **GEN2OIE (at ACL)**, have all solely relied on the CaRB dataset for their evaluations. This singular choice was underscored in the OpenIE6 paper, **Section 6, \"Experimental Setup\"**, where it's mentioned, \n>\"We evaluate all systems against CaRB’s reference extractions, as they have higher coverage and quality compared to other datasets.\"\n\nIndeed, as detailed in Section 4.1 \"Experimental Setup\" of our paper,`lines L251-L255`, we stated, \n>\"To establish an objective and fair comparison with other models, we opted for the benchmark dataset,CaRB (Bhardwaj et al., 2019), used in previous studies, applying the same evaluation metric, F1 score.\"\n\n---\n\n### Question 2:\nThe experimental setup is also weak. \n\n### Answer 2:\nAlthough the space is limited and we have not shown too many experiments, we believe that our experimental setup is comprehensive and thorough. Specifically, our study incorporates four pivotal experiments:\n\n1.\tA direct comparison of our method with other SOTA techniques was carried out using the full training data. The results showcased that our proposed approach is comparable to the best-performing techniques under these settings.\n2.\tIn the context of three low-resource settings, we benchmarked our approach against two types of techniques: labeling-based and generation-based. The evidence demonstrated that our method achieved SOTA performance in these low-resource conditions.\n3.\tA detailed contrast was drawn between our approach and the baseline, GEN2OIE. We meticulously reported and analyzed the performance metrics of every component of OK-IE. Empirical results indicate that, in comparison to the baseline, our approach has increased the F1 score from 31.2 to 52.9 and has notably expedited the convergence rate.\n4.\tWe also tracked the performance trends of both generation-based methods and our proposed technique as training data scaled from limited quantities to the full dataset. This trend analysis vividly underscores that our proposed approach demands less data and converges at a swifter pace throughout the entire spectrum of training data.\n\n---\n\n### Question 3:\nThe ablation experiment is also unclear for the reader.\n\n### Answer 3:\nIn the ablation study detailed in **Section 4.3** and depicted in **Table 5**, we transparently illustrate the impacts of OK-IE's individual components - task format transformation, anchor, and controlling order generation - on performance and convergence speed under low-resource settings. Moreover, we underscore this enhancement by investigating how the model converges to a result after just one single training epoch. We also provide a more detailed analysis of the ablation study in the **Appendix B.2** of our paper.\n\n---\n\n### Question 4:\n\nThe tables could be just integrated in Figure 1. \n\n### Answer 4:\n\nIt's worth noting that in Figure 1, we track the performance of both generation-based methods and our proposed approach as training data increases from a small subset to the full amount. This curve provides an intuitive visualization of the changing trends.\n\nHowever:\n\n1. Table 1 presents our method in contrast to other techniques under traditional settings. This table aims for an immediate comparison of the performance of various methods.\n2. Tables 2 through 4 detail the performance of our proposed method against generation-based and labelling-based strategies, furnishing readers with a clear view of the performance and distinctive advantages of each category.\n3. Table 5 showcases the incremental enhancements introduced by different components compared to the baseline. As part of the ablation study, it's imperative to distinctly lay out these nuances for a granular examination of our proposed methodology.\n\t\nIn summary, given the disparate dependent variables across these tables, it's challenging, if not counterproductive, to integrate them into a single figure. Such an attempt could compromise clarity, obscuring the immediate insights that our current paper offers to readers.\n\n---\n\n### Question 5:\nThe paper is too short and not well explained. I cannot understand the key component of the proposed idea, neither the working flow.\n\n### Answer 5:\n\nDue to space constraints, our description might have lacked sufficient clarity. We will revise and refine the paper based on your suggestions in the revised version. To address your concerns immediately, we will provide a clearer explanation below.\n\n\n1. ***The Problem***: In our paper, we address two main challenges in open information extraction. The first is the mismatch in task format between upstream and downstream tasks: While we can model the open information extraction task as a seq2seq problem, this is inconsistent with T5's pre-training task, namely span corruption. This inconsistency suggests that a significant amount of data is required to train our model to adapt to new task format. The second challenge is the 'order penalty'. Due to our reliance on teacher forcing during training, the model tends to rigidly perceive only one sequence as correct, unjustly penalizing other viable sequences. This requires more training time to converge to a stable state.\n\n    We've explained this in the beginning parts of our paper, specifically in the **Abstract** `L009-L013` and **Introduction** `L073-L084`. And if you're wondering why existed methods faced these issues, we've covered that too in the section **“Related work”** `from L154 to L172`.\n\n2. ***Our Solution***: We've come up with two ways to handle these issues. The first involves a  transformation of the task format (as detailed in **section 3.1**). This approach effectively bridges the mismatch between the types of tasks in the upstream and downstream processes.  The second method （as detailed in **section 3.2**）introduces an 'Anchor' to explicitly specify which of the “subject, predicate, object\" the model should generate at the sentinel position. This technique helps eliminate the 'order penalty'.\n\n3. ***Proof***:  Not only did we propose these solutions, but we also validated them through comprehensive experiments. In the **Experiments** section, we show how our methods work well. We've also pinpointed exactly how our two techniques, the \"task format transformation\" and \"controlling generation order\", speed up training. We used less data and less time and still managed top-notch results, especially when resources were limited.\n\n4. ***Working Flow Details***: If you're looking for a step-by-step guide for working flow, we've got you covered in **Appendix A.1**. And throughout our paper, we have provided real-world examples to give you a clearer picture of each part.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "If the results are correct, it could be very useful.", "reasons_to_reject": "The paper is too short and not well explained. I cannot understand the key component of the proposed idea, neither the working flow.\n\nThe experimental setup is also weak. There is a single dataset. The tables could be just integrated in Figure 1. The ablation experiment is also unclear for the reader.", "questions_to_authors": "", "paper_topic_and_main_contributions": "They present a method called OK-IE that is supposed to transform Open Information Extraction task into a pretraining task form for T5.", "excitement": 1, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JNsCujyNbl", "reviewer": "Reviewer_8h8N", "summary": "", "questions": "1. Can the paradigm shift work in a more general form (i.e. extract the triplets end2end without the step1 of GENO2IE) ？", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely appreciate the time and effort you dedicated to reviewing our work. Below, we respond to each of your comments and questions.**\n\n---\n\n### Question 1:\nCan the paradigm shift work in a more general form (i.e. extract the triplets end2end without the step1 of GENO2IE) ?\n\n### Answer 1:\nCertainly, your question is astute and insightful. Indeed, the paradigm can work in a more general form, enabling end2end extraction of triples without the preliminary step of GENO2IE.\n\nHowever, from our prior experiments, we observed that direct end2end extraction utilizing the paradigm shift didn't yield as significant improvements as when the shift was employed in the pipeline approach. To provide a specific instance, when leveraging the paradigm shift within the pipeline under a low-resource setting, we achieved an F1 score of 52.9. In contrast, the end2end approach with the paradigm shift resulted in an F1 of 39.7. It's crucial to note that this increment is still considerable, especially given that without the paradigm shift, the end2end approach performs rather poorly under low resource conditions.\n\nSuch findings are undeniably intriguing. One reason we didn't discuss this aspect in the current paper, along with the associated data, is due to the evident performance disparity. In the pipeline extraction method—where we first extract the predicate and then the entire triple—the predicate extraction phase is relatively simpler. Therefore, when viewed holistically, we opted to forgo the convenience of the end2end process in favor of more significant performance gains in the pipeline approach. It's a conscious tradeoff we decided to make based on empirical evidence.\n\n---\n\n\n### Question 2:\nAlthough the idea of anchors can be roughly understand through careful reading. Some implementation details about the anchor guidance are missing. The description between L226 and L229 are not enough for presenting the anchor to all readers, especially the p-tuning process of anchors is missing.\n\n### Answer 2:\n\nTo elucidate the concepts of the Anchor and the p-tuning process more thoroughly, we offer the following elaboration:\n\n- **Anchor**: \n  As delineated in lines `L226-L229` of the paper, \n  >\"Each anchor consists of two parts: an embedding from a word table representing a clear meaning, and a tunable embedding (Liu et al., 2021).\" \n\n  In this context, the former refers to a fixed vector, obtained from the word embedding layer of a language model, which represents a distinct semantic meaning. This vector is derived from token ids that are acquired through tokenization (specifically within the transformers library as `input_ids`). The latter, on the other hand, refers to a soft prompt embedding. This is an initially set vector that remains tunable, encapsulating a latent semantic meaning.\n- **P-tuning process**: \n  Within the implementation, the aforementioned embeddings are concatenated to form a cohesive semantic unit. This unit explicitly defines the type (subject, predicate, object) to be generated at each sentinel's position. Specifically, if we intend to produce a 'subject' at the `<extra_id_0>` (the sentinel symbol in the transformers library for T5) position , the anchor comprises the vector derived from the word embedding layer for the token “subject”, combined with a tunable vector. This anchor is then placed on both sides of the sentinel's embedding to constrain the contextual surroundings of this position to 'subject'. The result is an `input_embeds` that is fed into the model.\n\nWe believe that this detailed explanation, complemented by specifics from the implementation, will make the concepts under discussion more comprehensible. We have augmented this section in the appendix of the revised version.\n\n---\n\n\n### Question 3:\n\nIf the idea of anchors can be presented in the form of a small figure (even in the Appendix), it would be better.\n\n### Answer 3:\n\nThank you for your valuable input. In response to your suggestion, we have completed a figure that elucidates the concept of anchors. This illustration, accompanied by the details mentioned in A2, will be included in the Appendix for enhanced clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a paradigm shift that is technically sound and demonstrates good performance in low resource scenarios.\n2. The experimental results provide evidence that the proposed anchor guidance and order control techniques are effective in enhancing performance and alleviating the order problem.\n\nOverall, the contributions of this paper are suitable enough to present as a short paper.", "reasons_to_reject": "1. Although the idea of anchors can be roughly understand through careful reading. Some implementation details about the anchor guidance are missing. The description between L226 and L229 are not enough for presenting the anchor to all readers, especially the p-tuning process of anchors is missing.", "questions_to_authors": "1. Can the paradigm shift work in a more general form (i.e. extract the triplets end2end without the step1 of GENO2IE) ？", "paper_topic_and_main_contributions": "This paper presents a study on low resource open information extraction and introduces a novel approach to tackle the issue of data inefficiency. The authors propose a shift in the task paradigm of the second stage of GENO2IE, opting for a span corruption paradigm instead. This paradigm change helps overcome the data inefficiency caused by the pretraining and fine-tuning gap. Additionally, the authors suggest the use of anchors to guide the generation order, addressing a critical challenge in Open IE.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "If the idea of anchors can be presented in the form of a small figure (even in the Appendix), it would be better. "}, "extra_text": {}}, {"review_id": "pk6kK5RcuU", "reviewer": "Reviewer_DMKQ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely appreciate the time and effort you dedicated to reviewing our work. Below, we respond to each of your comments and questions.**\n\n---\n\n### Question 1:\nThe major limitation, in my opinion, is its reliance on the English language. English is relatively poor in terms of syntax, and for a task that is so dependent on syntax, such as OIE, the performance of the method may stem from the language characteristics. Notice that Kolluru et al. have investigated GEN2OIE in a multilingual setting, which gives us a better understanding of the performance and limitations of the work. Also, since the authors are limited to the English language, in my opinion, they should very explicitly state that, as their results may simply not generalise to OIE as a whole but to OIE in English.\n\n### Answer 1:\nThank you for your discerning observation. In the limitation section of our paper, we primarily discuss the challenge of extracting from complex sentences containing multiple triples. We concur with your perspective and will amend our limitation section to explicitly state that our method has been specifically evaluated in the context of English OpenIE. Explorations and adaptations of our proposed approach for other languages are earmarked as a crucial direction for our future endeavors.\n\nFurthermore, in the revised version, post line L348 in the limitation section, we have added: \"It should be underscored that our current method has been specifically evaluated within the confines of English OpenIE. This presents a limitation, and we intend to explore and validate the efficacy of our approach across other languages in future work.\"\n\n---\n\n### Question 2:\npg 4 col 2 line 295: table 5 -> Table 5\n\n### Answer 2:\nThank you for your meticulous attention to detail. You are right, and we acknowledge the oversight, it was a typographical error regarding capitalization during typing. We have corrected this error in the revised version.\n\nBesides, we will check the entire paper for any potential typo errors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think the work is fairly clear, and the results are interesting. It is not groundbreaking in its approach, but it is not trying to be, and the authors are very clear on the objectives of their work.\nThe work relies on a simple but effective idea of encoding triple generation as span corruption and leveraging T5 to learn to predict accurately for the English language. Their empirical validation is consistent and their analysis are insightful on how each part contributes to the performance\n", "reasons_to_reject": " The major limitation, in my opinion, is its reliance on the English language. English is relatively poor in terms of syntax, and for a task that is so dependent on syntax, such as OIE, the performance of the method may stem from the language characteristics. Notice that Kolluru et al. have investigated GEN2OIE in a multilingual setting, which gives us a better understanding of the performance and limitations of the work. Also, since the authors are limited to the English language, in my opinion, they should very explicitly state that, as their results may simply not generalise to OIE as a whole but to OIE in English.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The present paper proposes an encoding of OIE as a data corruption task and employs a T5 model to perform the triple generation process of the GEN2OIE model by Kolluru et al. (2022) in a few-shot scenario. While the idea of generation-based OIE is not innovative, and their architecture is based on Kolluru et al.'s model, the paper relies on a very ingenious encoding of the task in a way that reduces the computational complexity of training the T5 triple generation model by requiring only a fraction of the training examples. The authors conduct an empirical validation of their approach and ablation studies to show how each aspect contributes to the performance, Particularly, they evaluate the impact of the encoding of the task as span prediction by investigating how the model converges to a result after one single training epoch.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "pg 4 col 2 line 295: table 5 -> Table 5 "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This research paper explores open information extraction and introduces an approach aimed at addressing the problem of data inefficiency. The study proposes two enhancements for the GEN2OIE predicate extraction model. The first enhancement involves reconfiguring the information extraction task to align with T5's pre-training task known as "span corruption," which significantly enhances the model's data efficiency. The second improvement is the introduction of an anchor method that guides the model in determining the sequence order for generation. In summary, this paper builds upon a previous method and achieves significant advancements within a particular module.
Decision: Accept-Findings