Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction
Authors: Ji Qi
Chuchun Zhang
Xiaozhi Wang
Kaisheng Zeng
Jifan Yu
Jinxin Liu
Lei Hou
Juanzi Li
Xu Bin
Venue: EMNLP 2023 Main (2023)
Keywords: open information extraction
robustness evaluation
high-quality paraphrase

Abstract: The robustness to distribution changes ensures that NLP models can be successfully applied in the realistic world, especially for information extraction tasks. However, most prior evaluation benchmarks have been devoted to validating pairwise matching correctness, ignoring the crucial validation of robustness. In this paper, we present the first benchmark that simulates the evaluation of open information extraction models in the real world, where the syntactic and expressive distributions under the same knowledge meaning may drift variously. We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique that consists of sentences with structured knowledge of the same meaning but with different syntactic and expressive forms. By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques. We perform experiments on typical models published in the last decade as well as a representative large language model, and the results show that the existing successful models exhibit a frustrating degradation, with a maximum drop of $23.43$ $F_1$ score. Our resources and code will be publicly available.

Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction
Ji Qi', Chuchun Zhang', Xiaozhi Wang', Kaisheng Zeng', Jifan Yu', Jinxin Liu', Lei Hou', Juanzi Li', Bin Xu
Department of Computer Science and Technology, BNRist, Tsinghua University 2 University of International Business and Economics qj qj20@mails. tsinghua. edu. cn, 202025003@uibe, edu.
Abstract
inpot Sentencee
CaRg
Dench ROBUST
The robustness to distribution changes ensures that NLP models can be successfully applied in the realistic world, especially for information extraction tasks. However, most prior evalua- tion benchmarks have been devoted to validat- ing pairwise matching correctness, ignoring crucial validation of robustness. In this paper. we present the first benchmark that simulates the evaluation of open information extraction models in the real world, where the syntac- tic and expressive distributions under the same knowledge meaning may drift variously. We design and annotate large-scale testbed in which each example is mnowledge-invariann clique that consists of sentences with structured knowledge of the same meaning but with differ- ent syntactic and expressive forms. By further elaborating the robustness metric, a model is judged to be robust if its performance is consis- tently accurate on the overall cliques. We per- form experiments on typical models published the last decade as well as a representative large language model, and the results show that the existing successful models exhibit a frustrat- ing degradation, with maximum drop of 23.43 Fi score. Our resources and code are available at https://g/t thub. com qi c/ROBUST.
Watsona has served aag Minority Laaderag since
46.4
32.3
elected by his caurus November 1998
Since his el electin by his caucus in Noventer 1998 Wonson has been the Minontt Loadet
13.5
Watson, was elected by caucus in November 1963 has served on Minortled mndel since then.
Rea
World
Figure I: Extraction results of Open116 for three seman- tically equivalent sentences from CaRB and ROBUST. The proposed benchmark ROBUST computes the bustness score on a clique of sentences.
based on the lexical matching of syntactic heads between elements. To tackle the overly lenient met- ric, subsequent approaches (Lechelle al., 2019; Bhardwaj et al., 2019; Gashteovski et al., 2022) propose use of exact matching between tokens for delicate evaluation. Among these benchmarks, CaRB (Bhardwaj et al., 2019) adopts the all-pair matching table to compute the tuple match scores between extractions, which has been considered the de facto standard for evaluation. Research in- cluding these efforts has been devoted evaluating the pairwise matching correctness between model extractions and golden facts on sentence.
1 Introduction
However, the conventional evaluation bench- marks do not measure the robustness of models in the realistic open-wort world scenario, where the syntac- tic and expressive forms may vary under the same knowledge meaning (Qi et al., 2023). As shown in Figure 1, while the three sentences 81, 82, 83 con- tain the same structured knowledge (a1, P. 42 0z), the of-te-of-the-ar model Open1E6 successfully ex- tracts facts (in green color) sentence $1, but fails to predict arguments (in red color) on the other sen- tences due the syntactic and expressive drifts. In this example, the sentence 81 comes from CaRB which has a similar syntactic distribution to the training set, and existing benchmarks can only eval-
Open Information Extraction (OpenIE) aims to ex- tract n-ary knowledge tuples {(11, a2, .... an)} consisting of n arguments and one predicate from the natural text in commiininddependet manner, which has been served as the backbone to benefit NLP applications for many years (Liu al., 2021; Pei al., 2022: Chen et al., 2021).
Due to its structural flexibility, the evaluation of OpenIE a nontrivial problem, which in turn drives the advancement of the task. Early stud- ies (Stanovsky and Dagan, 2016; Zhan and Zhao, 2020) measure the performance of extractions
"Corresponding author: xubin tiiiini@nnnghu.cc..c

---

uate models this limited target attributing it the commendable scores (46.4/33.3), rather than on the other world samples. For accurate and faithful evaluation, we should measure the performance of models on sentences with various syntactic and ex- pressive distributions under the same knowledge meaning (Zhong et al., 2022).
benchmarks calculated at one sentence magnitude. To explore the robustness of existing models, we implement typical OpenIE systems published the past decade. The experimental results show dramatic degradation in model performance on ROBUST, with average drop of 18 percentage points in Fi scores, indicating that the robustness of existing successful models is far from satisfactory. We then further analyze the correlation between the variances of the model performance and the di- vergences of the syntactic distances on the cliques. The results find that the variance grows as the syn- tactic distance increases, and models behaved with similar variance on most of the cliques also demon- strate the inner consistency of our benchmark. In addition, we also evaluate the representative large language model ChatGPT' for OpenIE. Experimen- tal results demonstrate that ChatGPT achieves re- markable performance that compatible with the state-of of-ee-of-the-rr model on CaRB (Fi score of 0.516 under the 10-shot setting), yet it still exhibits the robustness issue on ROBUST (F1 score of 0.275 under the 10-shot setting).
Nevertheless. is not trivial construct bench- mark that satisfies the aforementioned conditions of encompassing both knowledge invariance and distributional shift. First, manual annotation of par- allel texts to maintain the same knowledge meaning with different syntactic and expressive forms may result in either too trivial or artificial. Second, it is difficult build a metric that measures the robust- ness as well as be compatible with existing bench- marks (c.g., (Bhardwaj al., 2019; Gashtcovski al., 2022)) to ensure comparability.
On the other hand, natural language paraphras- ing is defined as producing sentences with different surface forms (syntactic and lexical) by conveying the same semantic meaning (Zhou and Bhat, 2021). Going beyond the pairwise correctness comparison, can we evaluate the robustness of models based on reliable paraphrases equipped with syntactic and expressive transformations?
The ROBUST Benchmark
In this section, we describe the details of the benchmark construction. 'he benchmark con- sists of cliques based on syntactically diverse para- phrase generation and human annotation to unsure the knowledge invariance and distributional shift, where both the syntactic transformations sampled from real world and the human experience guaran- tee the naturalness. We also provide details of anno- tations and strategies in the Appendix A.1 and A.2.
In this paper, we introduce ROBUST, a Robust OpenIE Benchmark with Ubiquitous Syntactic Transformations, aiming to evaluate the robust- ness of OpenlE models. ROBUST is large-scale human-annotated benchmark consisting of 272 robustness testing cliques, where each clique con- tains sentences with different syntactic and expres- sive variations while conveying the same underly- ing knowledge meaning, for a total of 4,971 sen- tences and 191 knowledge extractions. To ob- tain each clique, we first adopt a syntactically con- trollable paraphraser with diversified syntactic sam- pling and expressive filtering strategies to generate paraphrases for each sentence in CaRB. We then design a two-stage annotation pipeline to perform sentence correction and knowledge extraction for each individual paraphrase in cliques based on hu- man experts. This data paradigm enables evalua- tion to go beyond pairwise matching to clique- clique-wise comparisons. Upon the testbed structure, calcu- late the robustness scores with respect to the worst performance within a clique and further analyze the performance variances on all cliques. This metric fairly reflects the robustness of models to distribu- tional drifts and is also compatible with existing
2.1 Data Preparation
Paraphrase Generation. Considering the com- patibility with previous benchmarks, we build our benchmark based on CaRB (Bhardwaj et al., 2019), which contains 1,272 sentences? of general domain originated from OIE2016 (Stanovsky and Dagan, 2016) with high-quality n-tuples annotations. To build sufficient paraphrases, we adopt AESOP (Sun et al., 2021), syntactically controllable paraphras- ing model generating paraphrases by specifying pruned target syntactic trees that can be sampled diversely. The model used in our work trained on parallel annotated data with two-level target
"We remove 10 sentences that do not have extractions from the original data.

---

Syntax
Knowledge
Extractions
Watson has served as Minority Leader since elected by his November 1988.
<Wistone, Manunity November
in November 1998
Since election by his taucus in November 1998, Watson has been the Minority Leader.
has served
<Wattum, beum, Misority Leader, since his ellectio by his cancus in November 1998-
PP
caque
Wastson
<Wastion, was elected by, his caucus November 1998s
caucus elected has
served as Minority Loader
Minority Leuder
then.
VP
SOAR
<Wastoon, has Minurity there-
Figure 2: example of robustness clique consisting of three sentences from ROBUST, where the sentences exhibit syntactic and expressive variants while preserving the same structured knowledge meaning. In contrast to conventional metrics, ROBUST measures the robustness score on a clique of all nodes.
syntactic trees. During generation, we first col-
2.2 Annotation
lect of constituency parse pairs ((T?. TP) }
For each paraphrase within a clique, we further design a two-stage annotation pipeline based on human experts to perform sentence correction and structured knowledge extraction. All annotators un- dergo training with tutorials to pass final examina- tion, and our batch-wise sampling validation ensure an annotation accuracy of over 90%. Detailed anno- tation including annotators, platforms, and quality checking can be found in Appendix A.I.
pruned height from the ParaNMT-50M (Wi-
eting and Gimpel, 2018). And then for each sen-
tence s with its constituency parse tree T, we ob-
tain most similar parses {(T?P TT)) by cal- culating weighted ROUGE scores between parse strings and select 5 top-ranked parses from {T?P) for each T.P by sampling with the distribution of (#T,, IP)
TP- Σn #(T? T
We thus generate 10 syntac- tically varying paraphrases each sentence.
Paraphrase Annotation. While automatically gen- erated paraphrases present syntactic and expressive variants, the correctness of the sentences cannot be fully guaranteed. To ensure the quality of the sentences, we perform a thorough paraphrase anno- tation with three types of corrections:
Diversified Expressive Filtering. Though differ- ent syntactic trees are specified the paraphrase generation, we find that there are still similar ex- pressions in the generated sentences. Therefore, we further filter the paraphrases with a heuristic search strategy to maintain the most diverse ones. For each clique composed of multiple sentence nodes, including original sentence and multiple para- phrases, first caleulate the BLEU scores (Pap- ineni et al., 2002) between all pairs of nodes. We then repeat the following simple strategy on para- phrase nodes until reaching the maximum accept- able number to eliminate homogeneity: (1) find the pair of nodes with the largest score in the cur- rent clique; (2) remove a node if its length is less than 2/3 of the original sentence, otherwise remove the node with the highest sum of scores with all other nodes. As depicted in Figure the remain- ing sentences 82 and $3 exhibit distinct syntactic structures and expressive forms compared to the original sentence $1. The detailed process with an example is shown in Appendix A.2.2.
Grammar Correcting: Correct grammatical mistakes in sentences to ensure the fluency.
Phrase Replacing: Replace the incorrect
phrases in sentences to ensure the correctness.
Sentence Rewriting: Rewrite the entire sen- tence if it has a semantic difference from the original sentence.
All operations are required to preserve both the distinctiveness of the annotation from the original sentence and their semantic equivalence. Based on this paradigm, all paraphrases are guaranteed to dif- fer from the original sentence in expression, while retaining the same semantic meaning. As shown in Figure 2, the three sentences in the 1st column exhibit different syntactic and expressive forms. A detailed process is available in Appendix A.1.1.

---

Benchmark
#Sent.
#Extr
#Cliques
Source
0.8-
0.75
OIE2016
3,200
10,359
Wil.
g6700
0.5
0.00
Re-O1E2016
600
OIE2016
035
035
CaRB
1,272
5,262
O1E2016
BenchlE
300
136,357
CaRB
ROBUST
4,971
16,191
1,272
CalB
Table 1: Quantitative statistics of ROBUST. (Sent.: sen- tence, Extr: extraction)
Figure 3: The average syntactic distances/similarity in each clique is calculated using HWS distance and Convolutional Tree Kernels, where the x-axis refers to the hierarchical discounting weights for two algorithms.
Knowledge Annotation. In the second stage, leverage human experts to annotate N-ary knowl- edge tuples on the paraphrases finished in the first stage. We design a guideline involving an iterative process to instruct annotators in extracting all pos- sible facts from a sentence. By referring to the an- notation of CaRB, in each iteration, we also divide the task of annotating into three steps: (1) recogniz- ing the predicate, (2) finding the arguments for that predicate, and (3) optionally obtaining the time and location arguments for the tuple possible.
based on the annotation paradigm, ROBUST raises a new data structure, the clique, which establishes the interconnection of sentences with underlying knowledge. The average number of sentences per clique is 3.877.
In addition, we find that previous benchmarks completely originate from O1E2016 based on Wiki and newswires, potentially leading to distribution bias to similar training corpus, especially for pre- trained language models (e.g. BERT (Devlin et al., 2019)) trained the general corpora. ROBUST mitigates this bias by extending syntactic and ex- pressive distributions to realistic scenarios. We further compute the vocabulary sizes for CaRB and ROBUST, resulting in 7648 and 7981, respectively, demonstrating that our natural annotations do not introduce many rare words.
In particular, we distribute the complete clique to individual annotators to obtain extractions with the same structured knowledge meaning. This anno- tation process ensures the characteristics in CaRB (i.e. Completeness, Assertedness, Informativeness, and Atomicity) while maintaining consistency with the underlying knowledge. As illustrated in the fourth column of Figure 2, the extractions from different sentences correspond the same under- lying knowledge. Detailed annotation process is available in Appendix A.1.2.
3.2 Syntactic Analysis
3 Data Analysis
The proposed benchmark measures the robustness of models on the drifts of linguistic observations. Therefore, the syntactic divergence in the clique is the key to ensuring robustness evaluation. We provide a thorough syntactic analysis of cliques to investigate the divergence.
To understand the general characteristics of RO- BUST, we provide quantitative statistics differ- ent granularities in comparison previous bench- marks. contrast to the traditional analysis on words and sentences, we further investigate the syntactic phenomena on cliques to explain the ro- bustness evaluation.
Metries of Syntactic Correlation. In order to an- alyze the syntactic divergence in the cliques, we need a metric to measure the syntactic correlation between two sentences. fast and effective algo- rithm is the HWS distance proposed in (Qi et al., 2023), which calculates the syntactic tree distance between two sentences based hicrarchically weighted matching strategy, where smaller weights imply a greater focus on the comparison of skele- tons. The value domain of this [o, 1]. where indicates the farthest distance. However, we find that their method may lead to the overcounting
3.1 Data Statisties
Table shows the quantitative statistics of RO- BUST and representative OpenTE benchmarks, in- cluding 0IE2016 (Stanovsky and Dagan, 2016), Re-O1E2016 (Zhan and Zhao, 2020), CaRB (Bhard- waj et al., 2019) and BenchIE (Gashteovski al., 2022). In comparison with the conventional dataset, ROBUST provides the largest number of human- annotated high-quality sentences. Meanwhile,

---

yond the individual clique, we further explore the syntactic divergence over all cliques. As shown in Figure 4, we average the mean of clique-wise syntactic distance/similarity on all cliques, based on the linearly increased discounting weights. We find that the average similarity of syntactic trees on ROBUST decreases rapidly as the discounted weight of the algorithm increases. Considering that increasing the weights implies reduced focus the low-level tree fragments, this result suggests that ROBUST involves prominent variability in the high-level skeleton of syntactic trees.
0.8
0.3
0.
011
0.5
0.6 0//
0.8
0.9
1.0
level-wise discounting weight
Figure 4: The average syntactic distance/simillariyy over all cliques with the hicrarchical discounting weights. Cliques containing only one point will be a line with value of 0 or 1.
Experiments
In this section, we explore the robustness of ex- isting successful OpenlE systems and further an- alyze the impact of different model architectures on robustness. We first introduce the proposed ROBUST metric, which calculates the robustness performance on clique, and then extensively eval- uate six typical models from three major categories and a large language model ChatGPT. Furthermore. based the clique structure, analyze the corre- lation between the variances of the model perfor- mance and the syntactic divergences in cliques.
problem for repeated consecutive spans We re- vise the original algorithm to solve the problem while maintaining efficiency. The details of the revised algorithm are shown in Appendix A.2.1 1 for ease of use.
We additionally implement the algorithm of Con- volutional Tree Kernel (CTK) similarity proposed in (Collins and Duffy, 2001) to fairly illustrate the syntactic phenomenon. In contrast to distance, measures the similarity between a pair of tree struc- tures by counting the number of tree fragments in common. The value domain of this algorithm is also [0, 1]. where 1 means the maximum similarity. Intra-clique Syntactically Analysis. To exhaus- tively investigate the syntactic divergence on the cliques, we calculate the average syntactic dis- tance/similarity in each individual clique based on the algorithms described above. The result is shown in Figure 3, where the horizontal axis and vertical axis the output and the discounting weights of two algorithms, respectively.
4.1 Evaluation Metries
The existing widely used CaRB scorer computes pairwise matching scores based on extractions on a sentence. Though accurate, it has rather limitations. We extend this scorer on cliques to calculate the robustness scores.
The CaRB Metric. To evaluate the correctness of system tuples, CaRB first creates an all-pair match- ing table, with each column as a system tuple and each row as a gold tuple, and computes precision and recall scores each cell. Then, it calculates the overall recall R by averaging the maximum values of all rows and the overall precision P by averaging the one-to-one precisions between sys- tem tuples and gold tuples in the order of the best match score the worst. Finally, the overall F1 is computed with R and P.
Overall, we observe that the values of syntactic distance and syntactic similarity are mainly scat- tered between (0.6, 0.9) and [0.0, 0.7], respectively, indicating that most of the cliques exhibit signifi- cant syntactic discrepancies. Another notable ob- servation is that the distribution of the HWS scatter representing the distance is closer to the dis- count weight decreases, suggesting that the differ- ences in syntactic skeletons are more significant in ROBUST.
The ROBUST Metric. An OpenIE system con- sidered robust if it behaves consistently on sen- tences with the same underlying knowledge mean- ing but differing syntactic and expressive variations. indicating the preservation of knowledge invari- ance. Therefore, we naturally calculate the robust- ness scores of model on each elique. Given clique including sentences C
Inter-cliques Syntactically Analysis. Going be-
'For two strings s 818884 and 8 8281 with consecutive span in common (c.g, SVPNP and SVPNPYP), the resulting distance may increase with the repetition of span 81.

---

Open1E4
ClauselE
OpenIE5
RnnOIE
SpanOIE
Open1E6
CaRB ROBUST
61.0
41.66
57.39
49.26
31.31
60.86
P
38.63
23.97
33.95
26.15
18.98
37.46
A
J22.37
U17.69
J23.43
123.11
U12.3
123.4
CaRB
48.25
50.16
47.48
49.46
42.21
50.54
R
ROBUST
30.14 14
38.58
30.88
32.52
27.94
34.25
A
418.11
111.58
116.60
J16.94
114.27
L16.29
CaRB
53.88
45.51
51.97
49.36
35.95
55.22
F ROBUST
33.86
29.57
32.34
28.99
22.60
35.78
A
J20.02
115.94
419.63
220.37
L13.35
119.44
Table The performance typical OpenIE systems on CaRB ROBUST benchmarks. The row represents the the difference between CaRB score and ROBUST score (1 means the degradation from CaRB). Bold numbers refers to the highest score per metric or highest difference per row (i.e. highest A for P. R and Fj).
{s....., } in ROBUST, first calculate the P/R/Fi scores of the model on each sentence, and then select the scores from the sentence with the worst F1 as the ultimate robustness scores probust Probust /Fobust. As mentioned above, we can compute the pair-wise P/R/Fi scores based on the CaRB scorer.
3.5-turbo version for all experiments. We perform few-shot experiments with manually constructed prompts and sampled demonstrations for CaRB and ROBUST benchmarks. The prompt template is available Appendix A.3.
4.3 Major Results
It is noteworthy that the ROBUST evaluation metric compatible with existing benchmarks be- cause we calculate on the order of magnitude of one sentence, and we can directly compare our robustness scores with CaRB and others.
4.3.1 Results on Typical OIE Models
We run the source code of all baselines on both CaRB and ROBUST and compute the average across all samples. All results are shown in Table Note that although the ROBUST scores are calculated different environment than CaRB, still offers fair comparison due to the calcu- lation manner. Based on the result, we can see that current successive OpenlE systems experience considerable performance decline on ROBUST across the board. Compared with CaRB, the av- erage degradation for precision, recall, and the Fi score is 20%, 15%, and 18%, respectively. This ob- servation suggests that research on the robustness of existing OpenlE models still needs to be com- pleted, as overly idealized evaluations encourage models to match fixed expressions strietly.
4.2 Evaluation Models
To exhaustively evaluate the robustness of exist- ing paradigms, we select six typical OpenIE ap- proaches from 3 categories. (1) Rule-based mod- els, which adopt linguistic patterns to identify knowledge facts, including Open1E4 (Christensen et al., 2011), ClauselE (Del Corro and Gemulla, 2013), and OpenlE5 (Saha et al., 2017, 2018). (2) Independent NN-based models, that train neu- ral networks from scratch with designed architec- ture, including RnnOlE (Stanovsky et al., 2018) and SpanOIE (Zhan and Zhao, 2020). (3) PLM- based models, that rely on pre-trained language model usually trained on large-scale text corpus, including 0pen1E6 (Kolluru al., 2020a) which in- troduces novel iterative grid labeling architecture, which treats OpenIE 2-D grid labeling task to produce extractions gradually based on BERT.
With the concrete comparison of model archi- tectures, find that the SpanOIE model demon- strates relatively small decrease in all three scores compared other models, indicating its robustness to syntactic transformations. This result suggests that the extraction strategy enumerating geomet- ric spans is, to some extent, independent of syn- tactic drift, making it less susceptible to sentence form transformations.
We also evaluate the OpenIE performance of ChatGPT, We use the python API interface gpt-
'the experimental period is 0023.04.0112023.0

---

6.55
S01001
1230M
15.0)
(111 pritww pessees ae peltors pety
(a) Heliques with variances
(b) variances with HWS distances
(c) variances with CTK similarites
Figure 5: (a) The distribution of the number of cliques with the variance of Fi scores in each clique. (b) The variance of Fi scores with the values HWS distance. (c) The variance of Fi scores with the values of Convolutional Tree Kernel similarity. The both correlation values are divided into several intervals avoid abnormal values.
tence in clique C (shown in green). The results in Figure 6b show that the correctness and robust- ness of ChatGPT can be improved by giving more diversified demonstrations.
U.3
0.0
4.4 Detailed Analysis
(a) sampling sentences
(b) sampling cliques-pairs
In this section, we investigate the coherence among cliques ROBUST, as well as the variations model performance across different cliques.
Figure 6: ChatGPT performance on CaRB/ROBUST. (a) The averaged Fi/Frobust scores on CaRB/ROBUST by randomly sampling demonstrations in CaRB. (b) The averaged Fi/ F Probust scores on ROBUST by randomly sampling 100 clique-pairs and specifying demonstra- tions and questions from different cliques in each pair.
Is the evaluation of model performance consis- tent across cliques? It is necessary investigate whether our evaluation of the model consistent across the majority of cliques in order to explore the internal consistency of our data samples. Based on the main results, we calculate the F1 score variance in each clique for three representative models, Rn- nOIE, SpanOIE, and Open1E6. The distribution of the number of cliques based on variance is depicted in Figure 5a. We find that the majority of cliques exhibit relatively slight variances, indicating high degree of consistency among robustness cliques. addition, we sample 11 subsets of interval 100 ROBUST and calculate the Person's Correlation Coefficient between the average probust of Ope- nIE6 each subset and the number of cliques of each subset. This result is 00.1480, indicating weak correlation between these factors.
4.3.2 Results on ChatGPT
We evaluate ChatGPT's OpenIE capability on CaRB and ROBUST. We randomly select 1/3/5/10 demonstrations from CaRB, and prompt ChatGPT to extract knowledge tuples by incorporating these demonstrations. We exclude sentences that belong to the same clique as the demonstrations during extraction. The result shows that ChatGPT exhibits impressive capability on CaRB, attaining 51.6 Fi score 10-shot setting, comparable the super- vised state-of-the-aar model Open/E6. However, still faces the robustness problem, as evidenced by a decline in the Frobust score to 27.5 on ROBUST in the same setting.
How does the syntactic divergence affeet the performance of models? Benefiting from the data structure of ROBUST, we can further investigate the effect of syntactic divergence on the perfor- mance of models. Concretely, for each clique, we calculate the average HWS/CTK values between all pairs of sentences and the variance of Fi across all sentences. The result shown in Figure 53, The results indicate a general trend where the variance of model performance decreases with increasing
We also investigate the impact of ChatGPT's performance on the diversity of demonstrations. We first randomly select 100 pairs of cliques [(C,, C,)(C, = (s] s?..)) For each sentence in clique C, prompt Chat- GPT by specifying 1/2/3/4 demonstrations from clique Cy. We then calculate the CaRB F1 score for each sentence (shown in blue), the average CaRB Fi score for all sentence (s) s7,, (shown in or- ange), and the ROBUST Frobust score on all sen-
from ROBUST.
*We divide all samples into five intervals and calculate the average variance to avoid abnormal values.

---

syntactic divergence. Based on the main experi- ment results, which indicate low performance of models on the overall benchmark, the observed degradation implies a consistent trend of poorer model performance in more open scenarios.
Zhao, 2020) re-annotated the corpus to improve the dataset's quality for more accurate evalua- tion. Wire57 (Lechelle et al., 2019) provided high-quality expert annotations, but the size is too small to serve as comprehensive test dataset with only 57 sentences. DocOIE (Dong et al., 2021) argued that in reality a sentence usually ex- ists part of document rather than standalone; the contextual information can help models under- stand better and annotate a document-leve Ope- nIE dataset. LSOIE (Solawetz and Larson, 2021) was built by converting the QA-SRL BANK 2.0 dataset (FitzGerald et al., 2018) to OpenIE which had a significant improvement over previous work in terms of data quantity. BenchIE (Gashteovski et al., 2022) created fact-based benchmark and framework for multi-facete comprehensive evalua- tion of OpenIE models in the multi-lingual setting.
5 Related Work
OpenIE Approaches. The OpenIE task was first proposed by (Banko et al., 2007) and is a fun- damental NLP task. Earlier models focused on statistical or rule-based methods to handle this task (Christensen et al., 2011; Schmitz et al., 2012; Del Corro and Gemulla, 2013; Angeli et al., 2015; Pal et al., 2016; Saha et al., 2017, 2018). Recently, with the rapid development of deep representa- tion learning, many supervised neural models have been proposed for OpenTE. These approaches could be roughly classified into two lines: 1 1.Sequence Labeling- gabeling-bassd models. RnnOIE (Stanovsky et al., 2018) applies a BiLSTM transducer, extending deep Semantic Role Labeling models to extract tuples. SenseOIE (Roy et al., 2019) leverages an ensemble of multiple unsupervised OpenlEi sys- tems' outputs and the lexical and syntactic infor- mation to improve performance. SpanRel (Jiang et al., 2020) represents the OpenIE task in a single format consisting of spans and relations between spans. SpanOIE: (Zhan and Zhao, 2020) predicts the candidate relation spans and classifies all pos- sible spans of the sentence as subject or object for each span. Multi (Ro et al., 2020) first predicts all relational arguments by BERT and then predicts the subject and object arguments associated with each relation using nuldi-headed attention. Ope- nIE6 (Kolluru et al., 2020a) provides an iterative grid labeling architecture, which treats OpenIE as a 2-D grid labeling task. 2.Sequence Generative models. Neural Open IE (Cui et al., 2018) and Logician (Sun et al., 2018) generate OpenlE ex- tractions by seq2seq paradigm. IMoJIE (Kolluru et al., 2020b) leverages BERT-based encoder and generates the next extraction which is fully condi- tioned on the extractions produced so far. Several benchmark datasets have been proposed to evaluate existing OpenIE approaches. 01E2016 (Stanovsky and Dagan, 2016) developed a method to create large-scale OpenlE dataset using QA-SRL annota- tions (He et al., 2015) which was found be noisy with missing extractions. After that, CaRB (Bhard- waj et al., 2019) and Re-O1E2016 (Zhan and
Despite the widespread interest in these bench- marks and the related OpenIE' approaches provides promising results. However, the traditional pecr-to- peer matching- evaluation can measure the robustness of those approaches, where the syn- tax and expression may be various with underly- ing mcaning (Qi et al., 2023). This work signifi- cantly fills the gap between traditional metrics and missed robustness evaluation for OpenIE and calls for more efforts in this research area.
Conclusion and Future Work
In this work, we propose ROBUST, a large-scale OpenIE benchmark consisting of 1272 robustness testing cliques, where each clique contains sentences with different syntactic and ex- pressive variations while conveying the same un- derlying knowledge meaning. We introduce our methodology for constructing the benchmark, in- cluding a syntactically and expressively diverse paraphrase generation, and a two-stage manual an- notation. A comprehensive analysis is then per- formed to demonstrate the consistency of the pro- posed data with the real world. We finally perform extensive experiments on existing successive mod- cls as well as representative large language model, and the results show that the robustness of existing methods is far from satisfied. The further detailed analysis demonstrates the substantial internal coher- ence of our benchmark, providing inspiration for the future development of robustness benchmarks.
OpenIE Benchmarks.

---

Acknowledgement
International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 344-
We sincerely appreciate the three reviewers from ACL2023, who provided thorough reviews and sug- gestions during the initial submission of this work. This work is supported by the National Natural Science Foundation of China (No. 62277033). also got partial support from National Engineer- ing Laboratory for Cyberlearning and Intelligent Technology, and Beijing Key Lab of Networked Multimedia. This work is also supported by grant from the Institute for Guo Qiang, Tsinghua Univer- sity (2019GQB0003) and the NSFC Youth Project (62006136)
Michele Banko, Michael Cafarella, Stephen Soder- land, Matt Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In IN IJCAI.
Sangnie Bhardwaj, Samarth Aggarwal, and Mausam Mausam. 2019. Carb: A crowdsourced benchmark for open ie. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6262-620 6267.
Huajun Chen, Ning Hu, Guilin Qi, Haofen Wang, Zhen Bi, Jie Li, and Fan Yang. 2021. Openkg chain: A blockchain infrastructure for open knowledge graphs Data Intelligence. 3(2):205-227.
7 Limitations
Janara Christensen, Stephen Soderland, and Oren Et- zioni. 201 I. An analysis of open information extrac- tion based on semantic role labeling. In Proceedings of the sixth international conference on Knowledge capture, pages 113-120.
We have presented a dataset with metrics to eval- uate the robustness of OpenlE models this pa- per. However, there are still several limitations that need to be improved in further study. First, there are a few studies exploring the pre-trained language models to perform zero-shot information extraction with advantages. To the lack of open source code, we have not explored the robustmess performance of these zero-shot models. Second, we think the robustness problem generally exists the NLP community, we remain the extensive study of robustness examination for more domains and models in future works.
Michael Collins and Nigel Duffy. 2001. Convolution kernels for natural language. Advances in neural information processing systems, 14.
Lei Cui, Furu Wei, and Ming Zhou. 2018. Neural open information extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 407- 413.
Luciano Del Corro and Rainer Gemulla. 2013. Clausie:
clause-based open information extraction. In Pro- ceedings of the 22nd international conference on World Wide Web, pages 355-366.
8 Ethic Consideration
There are two major considerations for conducting the evaluation of our proposed new benchmark. First, the source sentences are selected as same as CaRB, the original dev and test splits of O1E2016 in the open domain source of Wall Street Journal text and Wikipedia. All these data files are leveraged for the research purpose, and the result will be publicly available. Second, the annotators in this research are paid a salary higher than the market average and further allowed to choose flexible working time for human rights. For data utilization, we will make all annotation results publicly available under the CC BY-SA 4.0 license (free for research use).
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers).
Kuicai Dong, 7hao Yilin, Aixin Sun, Jung-jae Kim, and Xiaoli Li. 2021. Docoie: A document-level context- aware dataset for openie. In Findings of the Associ- ation for Computational Linguistics: ACL-IJCNPP 2021, pages 2389.
Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale QA-SRL pars- ing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 2051-2060. Melbourne, Australia. Association for Computational Linguistics.
References
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53dd Annual Meeting of the As- sociation for Computational Linguistics and the 7th
Kiril Gashteovski, Mingying Yu, Bhushan Kotnis, Car- olin Lawrence, Mathias Niepert, and Goran Glavaš. 2022. BenchlE: A framework for multi-faceted fact- based open information extraction evaluation. In

---

Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( \Volume 1:
Youngbin Ro. Yukyung Lee, and Pilsung Kang. 2020. 20IE: Multilingual open information extrac- tion based on multi-head attention with BERT. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 1107-1117, Online. Association for Computational Linguisties.
Long Papers), pages 4472-4490, Dublin, Ireland. As- sociation for Computational Linguistics.
Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-anae -nnwer driven semantic role labeling: Us- ing natural language to annotate natural language. In Proceedings of the 2015 conference on empiri- cal methods in natural language processing, pages 643-653.
Arpita Roy, Youngja Park, Taesung Lee, and Shimei Pan. 2019. Supervising unsupervised open information ex- traction models. In Proceedings of the 2019 Confer- ence Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 728-737.
Zhengbao Jiang, Wei Xu, Jun Araki, and Graham Neu- big. 2020. Generalizing natural language analysis Proceed-
through span-relation representations. ings of the 58th Annual Meeting of the Association for Compulational Linguistics, pages 2120-2133.
Swarnadeep Saha, Harinder Pal, et al. 2017. Bootstrap- ping for numerical open ie. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), pages 317-323.
Keshav Kolluru, Vaibhav Adlakha. Samarth Aggarwal, Soumen Chakrabarti, et al. 2020a. Openie6: Itera- tive grid labeling and coordination analysis for open information extraction. In P Proceedings of the 2020 Conference on Empirical Methods Natural Lan- guage Processing (EMNLP), pages 3748-3761
Swarnadeep Saha et al. 2018. Open information extrac- tion from conjunctive sentences. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2288-2299.
Keshav Kolluru, Samarth Aggarwal, Vipul Rathore, Soumen Chakrabarti, al. 2020b. Imojie: Itera- tive memory-baa -based joint open information extraction. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 5871- 5886.
Michael Schmitz, Stephen Soderland, Robert Bart, Oren Etzioni, et al. 2012. Open language learning for in- formation extraction. In Proceedings of the 2012 joint conference on empirical methods natural language processing and computational natural lan- guage learning, pages 523-534.
William Lechelle, Fabrizio Gotti, and Phillippe Langlais. 2019. WiRe57 : A fine-grained benchmark for open information extraction. In Proceedings of the 13th Linguistic Annotation Workshop, pages 6-15, Flo- rence, Italy, Association for Computational Linguis- tics.
Jacob Solawetz and Stefan Larson. 2021. Lsoie: A large-scale dataset for supervised open information extraction. In Proceedings of the 16th Conference of the European Chapter of the Association for Com- putational Linguisties: Main Volume, pages 2595- 2600.
Ruibo Liu, Guoqing Zheng. Shashank Gupta, Radhika Gaonkar, Chongyang Gao, Soroush Vosoughi, Mi- lad Shokouhi, and Ahmed Hassan Awadallah. 2021. Knowledge infused decoding. In International Con- ference Learning Representations.
Gabriel Stanovsky and Ido Dagan. 2016. Creating large benchmark for open information extraction. In Proceedings of the 2016 Conference on Empir- ical Methods in Natural Language Processing, pages 2300-2305.
Harinder Pal et al. 2016. Demonyms and compound relational nouns in nominal open ic, In Proceedings of the 5th Workshop on Automated Knowledge Base Construction, pages 35-39.
Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer. and Ido Dagan. 2018. Supervised open information extraction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 885- 895.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 31 1-318.
Jiao Sun, Xuezhe Ma, and Nanyun Peng. 2021. Aesop: Paraphrase generation with adaptive syntactic control. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Kevin Pei, Ishan Jindal, Kevin Chen-Chuan Chang, Chengxiang Zhai, and Yunyao Li. 2022. When to use what: An in-depth comparative empirical analysis of openie systems for downstream applications. arXiy preprint 2211:222208822
Mingming Sun, Xu Li, Xin Wang, Miao Fan, Yue Feng. and Ping Li. 2018. Logician: unified end-to-end neural approach for open-domain information extrac- tion. In Proceedings of the Eleventh ACM Interna- tional Conference on Web Search and Data Mining. pages 556-564.
Ji Qi, Yuxiang Chen, Lei Hou, Juanzi Li, and Bin Xu. 2023. Syntactically robust training partially- observed data for open information extraction. arXiv preprint arXiv: 2301.06841.

---

John Wieting and Kevin Gimpel. 2018. ParaNMT-5OMe Pushing the limits of paraphrastic sentence embed- dings with millions of machine translations. In Pro- ceedings of the 56th Annual Meeting of the Associa- lion for Compulational Linguistics (Volume 1: Long Papers).
Junlang Zhan and Hai Zhao. 2020. Span model for open information extraction on accurate corpus. In Proceedings of the AAAI Conference on Artificial Intelligence.
Victor Zhong, Weijia Shi, Wen-tau Yih, and Luke Zettle- moyer. 2022. Romqa: benchmark for robust, multi- evidence, multi-answer question answering. arXiv preprint 2:X.::2210.13353
Jianing Zhou and Suma Bhat. 2021. Paraphrase genera- tion: A survey of state of the art. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

---

A Appendix
Algorithm 1 HWS Distance
Input: Constituency parses T1, 72 of sentences
$1,82, pruning height h, discount factor
A.1 Annotation Details
Output: Syntactic distance between $1, 82
Get trees T T pruned height h, and their
We have the following detailed annotation informa- tion. Who: For Task1 and Task2, we employed two separate annotation teams consisting of 6 and 9 students respectively, who are all majoring in CS at universities. We ensured their professional- ism through the tutorials and final examination. Where: As both tasks are easy read and write for annotators, we distributed the data directly without using a special annotation platform. Quality: We adopted a batched iterative annotation and evalua- tion process to ensure that the sampling accuracy is above 90%. License: We will release all annota- tion results under the CC BY-SA 4.0 license (free for research use).
level-order traversal sequences 91, 92
2: Initialize total length and count I = 0; m 0
3: A[[[[] = if q [ == 92 [1], i = ..... Jeleon
4: A[]][]] 1 if q1 [1] - 92 j = -, 1,.., I2den
5: J[1] = g = 22 [1], i = 1, ..... 2enen
I[1 = 1 = q2bil.j j- 1, 92
for i = gleen do
for - 92.len do
if [1(] = 02 5 then
10:
if Ali - 1|6 1 & Ali
1][ - 1] [ then
11:
A = Ali 1|1 +1
12:
[[1] = Ali - 11 1 +1
13:
J[1] Ali 116 - +1
14:
end if
A.1.1 Paraphrase Annotation Process
15:
else
16:
A[i][p]] 0
The goal of paraphrase annotation is to correct the automatically generated sentences from the models based on human intelligence. Overall, we adopt
17:
if Ali 1|[; - 1] > then
18:
[ (= I - 1[1 - 1] x a
19:
m ++
an iterative step of combining human annotation
20:
end if'
paired with expert evaluation to ensure accuracy
21:
end if
and efficiency. In each iteration, at least three hu- man workers who are fluent in English reading and writing annotate batch of samples, and then two domain experts will check the annotation results on a random sample of 40% of the batch. The batch annotations will be accepted until the validation accuracy is greater than 90%. For the annotation of each paraphrase, the annotators are asked to correct the sentence with syntactic, phrasal, semantic- different mistakes against the original sentence.
22: end for
23: end for
24: if A
2 1 then 1= l A[i][] aM
25:
26: end if
27: Return - l/min(gq frmmn(qq.ten. Ieen I2.len)
A.2
Algorithms Details
A.2.1
Hierarchically Weighted Syntactic Distance Algorithm (Revised)
A.1.2 N-tuples Annotation Process
The revised Hierarchically Weighted Syntactic Dis- tance Algorithm (HWS distance) is shown in al- gorithm We fix the over-counting problem for repeated consccutive spans while preserving the efficiency with the same time complexity in the original work (Qi et al., 2023).
We leverage the same iterative annotation strategy with the paraphrase annotation for OpenIE N-tuples annotation. In particular, we design an annotation flowchart for the workers according to the similar process in CaRB, by dividing the task into 4 steps: (1) identifying the relation, (2) identifying the ar- guments for that relation, and (3) optionally identi- fying the location and time attributes for the tuple. The same validation meaner with the paraphrase annotation is adopted to reach each acceptable an- notation batch.
A.2.2 Diversified Filtering Process
We perform diversified filtering based on BLEU scores between all pairs of sentences in each set of generated paraphrases to maintain the most di- verse paraphrases. For example, given the gener- ated paraphrases following:

---

In 1840, he was appointed command his regi- ment, post he held for nearly fourteen years. 1840, the regiment's commander, which he held for nearly 14 years.
prompt = "Open information extraction requires extraction of all relations the sentence, i.e., predicates, the subjects and objects corresponding to these relations, and the possible time and place thesis elements, For example, in the sentence: Watson, who was elected by his caucus in November 1998. has served as Minority Leader since then.
P1
P2
In 1840 he took command of the regiment and held it for nearly 14 years.
P5
When he was 14 years old, he became a member of the regiment...
From sentence, following tuple can be extracted: (was elected by, Wastson, his caucus in November 1998); (has served as. Wastson, Minority Leader, since then)
P4
1840, the command of the regiment, which he held for nearly 14 years.
ps
The regiment, then, in 1840, the rank of captain, which he held for nearly 14 years.
In these tuples, we always put the predicate first, the second is the subject corresponding to the predicate, the third the object corresponding to the predicate (if there none, it is not labeled). and the last two are time and place in that order. which can be omitted if there is none.
Table 3: The original sentence ori with 5 paraphrases
Pi ~ Ps-
Please follow the example above and extract all the relational tuples in the following sentence: <sentence>
As shown in Figure 7, we first calculate the BLEU scores between all pairs of paraphrases (shown on the edges). We then find the two sen- tences pl, P4 with the maximum BLEU score. Be- cause the lengths of these two sentences are larger than 2/3 of the original sentence, we then calcu- late the summation of scores from each of them to all other sentences which results sum(P1, P//1)) 136.9 and sum P/1) 158.7, and remove the sentence P4 that has larger summation score. We repeat the strategy above remove the sentence P1 and obtain expressively diverse paraphrases.
Please show the results in line strictly the form the results above"
Figure 8: The 1-shot prompt to ChatGPT for the OpenIE task, where the <sentence> corresponds to the query sentence.
in ROBUST. Then for each pair, select all ex- amples in clique Cj as demonstrations and select all sentences in Cj as questions to calculate the Probust-scome For syntactie correlations, we first calculate the averaged value as between question i and all sentences in C₁ and further calculate the average on (01, .2....) as the final correlation on current clique-pairs. We divide the scores into sev- eral intervals and compute the average value in each corresponding interval to avoid abnormal val- ues. The results based on both implementations of HWS distance and Tree Kernel similarity as the syntactic correlation are shown in Figure 9.
P4
remove P4
remove
Figure 7: By performing the diversified filtering, 3 para- phrases p2, Pa, Ps is maintained.
In the left figure of the result, we can see that the probust -score of the model gradually increases the average syntactic similarity of the two cliques increases. The same observation is also shown in the right figure with the averaged syntactic dis- tance between two cliques. These results suggest that ChaGGT is sensitive to the syntactic distri- bution between questions and demonstrations and that giving demonstrations with similar syntactic distribution enhances the effectiveness of ChatGPT.
A.3 Prompts and Analysis for ChatGPT
A.3.1 Prompt Design
We create a prompt template for the task of OpenIE to query the ChatGPT. An example of a --ohot prompt is shown in Figure 8, where the highlighted demonstration and the variable <sentence> can be replaced with specified examples.
A.3.2 Performance with Syntactic Correlations
In this section, we further investigate the correla- tion between the model performance and syntactic distance of demonstrations and questions for the ChatGPT model. We first randomly sample a set of 100 pairs of cliques (Cj.C)) = 1,... 100)
10 MEAD S0C (R127
31477
.005 didinnc
250
1002
Figure 9: The Frobust scores of Open1166 model with syntactic correlatioss between clique-pairs.

---

A.4 Error Analysis for OIE Systems
We conduct error analysis for three typical OpenIE models OpenIE4, SpanOIE, and Open1E6 on ro- bustness clique. The model predictions with the CaRB and ROBUST scores are shown in Table 4. First, we can see that the sentences in the clique exhibit significant syntactic and expressive diver- gence. It implies that the constructed data source satisfies the expectation. Second, we find all sen- tences in the clique have more than one extraction, while the OpenIE4 and OpenIE6 models predict the extractions insufficiently. which causes lower recall. On the other hand, the SpanOIE model outputs predictions by enumerating all possible geometric spans, which build sufficient outputs gardless of syntactic features. This architecture offers SpanOIE consistent performance.

---

Sentences & Extractions
CaRB
ROBUST
They can relieved only by changing that system, not by pouring Western money into
"xcan be relseved, They)
changing. They, that system)
Only change
system, (can reliove, Ouly change the system, them)
injection
money
Clique
>>
injection Westem money,
relieve,
1
> (can be changed, the system)
Instead of pouring money from the West, changing the system the only way to relieve them.
Changing the system. the only way relieve them)
(is not, Pouring money from the West, the only way to relieve them)
(can relieve. Only change the system, them)
What can relieve them only to change that system, not put money from the West into
reliove, only change system,
(canent relieve, put maney from the West into them)
They)
OpenIEI4
+ (can relieve, Only change in the system, not, them)
0.378
0.119
changing the the only way relieve them)
0.553
can relieve them, only change system, pat money relieved only by. They. chunging system),
0.119
(cun be relieved not by pouring). They, Western money into that system))
0.597
(is
the West)
SpanOte
& (can relieve. Only change
them.).
(cannot relieve, An injection Western money, themn.)
0.414
0.157
(can be changed, the system
Changing system,
reliove
(is not. Poaring money from the West, the only way to relieve them.)
0.359
(can relieve, only clange that system, them).
0.157
(cannot relieve. put money from
them)
(They, can be relieved, only by changing that system
0.48
(They, only changing, that system)
Open1E6
A (Only chango the system, not, can relieve, them)
0.421
0.334
(changing the
the only way relieve them)
0.533
(Whot them, only change system, yot money West
0.334
Table 4: An analysis for model predictions with the F. /probust scores of two benchmarks.


Reviews:
[{"review_id": "S0gmkdRoZ4", "reviewer": "Reviewer_59er", "summary": "", "questions": "- Question A: The syntactic-aware paraphrasing process is very valuable as a common data augmentation method for other tasks. When performing error corrections in the paraphrase annotation, do you have statistics on the percentage of each type of error?\n- Question B: In line 276-277, I agree that wiki- or newswire-based texts may bring distribution bias. But I do not think the syntactic paraphrasing could mitigate the domain bias in a more realistic scenario. After all, the contents' semantics are not changed, they are just be paraphrased to another form.\n- Question C: As illustrated in Figure 1, the OIE model may generalize pooly on sentences in different types of syntactic structures. To substantiate the problem, have you analyzed a specific dataset (e.g. CaRB) to see if there are obvious syntactic patterns or characteristics?\n- Question D: What if we use a pre-trained paraphrasing model (or ChatGPT) to directly generate paraphrased pairs without explicit syntactic information guidance? Would the syntactic distance/similarity be larger or smaller? Is the syntactic-guided paraphrasing better than vanilla free-form paraphrasing?\n- Question E: Comparing with the performance drop (~20% F1 scores) on ROBUST, the variance (<5%) seems to be trivial. So why do you choose the worst F1 score as the ROBUST metric instead of the averaged or the median scores? Could you provide more explanations on the metric score design?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the effort you have invested in reviewing our work and the helpful suggestions. We are honored to address the questions you have raised:\n\n\n> The syntax-guided paraphrasing is intuitive, but it could be better if free-form paraphrasing is applied for comparison.\n\nA1: During the data preparation, we have indeed experimented with both two types of models: (1) the free-form paraphrasing models, including SOW-REAP [1] and UPSA [2], and (2) the syntactically-controllable paraphrasing models, including SGCP [3] and AESOP. By applying the models to our data, we find that the former ones tend to generate paraphrases with minor changes compared to the original sentences (low syntactic distance and high BLEU score), usually exhibiting the replaces of words or phrases of different surface forms that do not satisfy our realistic assumption. So we choose the best performant model AESOP based on our experience.\n\n> The motivation is clear, but the evidence to the syntax-disturbance problem may be not sufficient.\n\nA2: The Figure 1 from RobustOIE [4] demonstrated that the OIE model performance exhibits a significant degradation as the syntactic similarity between the training set and testing sentences decreases, supporting the syntax-disturbance problem. We mentioned this conclusion in line 58  without explaining it in details, and we will revise the statement to clarify the disturbance phenomenon.\n\n> Question A: The syntactic-aware paraphrasing process is very valuable as a common data augmentation method for other tasks. When performing error corrections in the paraphrase annotation, do you have statistics on the percentage of each type of error?\n\nAnswer to question A: We have defined the three types of mistakes in the guideline to enable annotators to efficiently and logically correct the sentence. However, as our goal is robustness benchmark of OIE, we have not labeled these error types. We appreciate your valuable suggestion and will perform sampling on the generated sentences to observe the error cases of the paraphrase generation model in future.\n\n> Question B: In line 276-277, I agree that wiki- or newswire-based texts may bring distribution bias. But I do not think the syntactic paraphrasing could mitigate the domain bias in a more realistic scenario. After all, the contents' semantics are not changed, they are just be paraphrased to another form.\n\nAnswer to question B: We agree with your perspective that the content's semantics of the dataset remain unchanged after paraphrasing. As the main experiment from RobustOIE [5] shows that syntactically diverse data-augmentation will partially improve the OIE performance on the specific test-set,  we suggest that the more diverse training corpora incorporating domain-specific priors (e.g, syntactic and semantic knowledge) may be a good solution to mitigate the domain bias for the adaption of models.\n\n> Question C: As illustrated in Figure 1, the OIE model may generalize poorly on sentences in different types of syntactic structures. To substantiate the problem, have you analyzed a specific dataset (e.g. CaRB) to see if there are obvious syntactic patterns or characteristics?\n\nAnswer to question C: To analyze the syntactic patterns for a specific dataset (e.g., CaRB), one of the most efficient ways is to perform the K-means clustering based on the proposed syntactic metric of HWS-distance. We did this on CaRB and found that the cluster with the largest number of sentences has the smallest average syntactic distance to the OIE2016 training set, and the samples in the cluster tend to have more formal and declarative sentence structures (e.g., one of the cluster centers is (ROOT(S(NP(NNP)(NNP))(VP(VBZ)(ADJP))(. )))).\n\n> Question D: What if we use a pre-trained paraphrasing model (or ChatGPT) to directly generate paraphrased pairs without explicit syntactic information guidance? Would the syntactic distance/similarity be larger or smaller? Is the syntactic-guided paraphrasing better than vanilla free-form paraphrasing?\n\nAnswer to question D: Kindly refer to the explanation in A1.\n\n> Question E: Comparing with the performance drop (~20% F1 scores) on ROBUST, the variance (<5%) seems to be trivial. So why do you choose the worst F1 score as the ROBUST metric instead of the averaged or the median scores? Could you provide more explanations on the metric score design?\n\nAnswer to question E: The worst performance indicates the lower bound of the model when applying to realistic scenarios, which is more reliable for estimating the robustness of the model in practical applications and is inline with some existing literatures [5]. However, we acknowledge and appreciate your concern, so our scoring code is developed to support the reporting of variances of model performance for more flexible requirements.\n\n\nReference:  \n[1] Goyal, Tanya, and Greg Durrett. \"Neural Syntactic Preordering for Controlled Paraphrase Generation.\" In ACL 2020.  \n[2] Liu, Xianggen, et al. \"Unsupervised Paraphrasing by Simulated Annealing.\" In ACL 2020.  \n[3] Kumar, Ashutosh, et al. \"Syntax-guided controlled generation of paraphrases.\" In TACL 2020.  \n[4] Qi, Ji, et al. \"Syntactically Robust Training on Partially-Observed Data for Open Information Extraction.\" In EMNLP 2022.  \n[5] Zhong, Victor, et al. \"Romqa: A benchmark for robust, multi-evidence, multi-answer question answering.\" arXiv preprint arXiv:2210.14353 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The syntax-guided paraphrasing process is a valuable data augmentation method that could be utilized in many tasks.\n- The new dataset could provide a quantitive robustness analysis for current OIE models, and the metric is compatible with other benchmarks.", "reasons_to_reject": "- The syntax-guided paraphrasing is intuitive, but it could be better if free-form paraphrasing is applied for comparison.\n- The motivation is clear, but the evidence to the syntax-disturbance problem may be not sufficient.", "questions_to_authors": "- Question A: The syntactic-aware paraphrasing process is very valuable as a common data augmentation method for other tasks. When performing error corrections in the paraphrase annotation, do you have statistics on the percentage of each type of error?\n- Question B: In line 276-277, I agree that wiki- or newswire-based texts may bring distribution bias. But I do not think the syntactic paraphrasing could mitigate the domain bias in a more realistic scenario. After all, the contents' semantics are not changed, they are just be paraphrased to another form.\n- Question C: As illustrated in Figure 1, the OIE model may generalize pooly on sentences in different types of syntactic structures. To substantiate the problem, have you analyzed a specific dataset (e.g. CaRB) to see if there are obvious syntactic patterns or characteristics?\n- Question D: What if we use a pre-trained paraphrasing model (or ChatGPT) to directly generate paraphrased pairs without explicit syntactic information guidance? Would the syntactic distance/similarity be larger or smaller? Is the syntactic-guided paraphrasing better than vanilla free-form paraphrasing?\n- Question E: Comparing with the performance drop (~20% F1 scores) on ROBUST, the variance (<5%) seems to be trivial. So why do you choose the worst F1 score as the ROBUST metric instead of the averaged or the median scores? Could you provide more explanations on the metric score design?", "paper_topic_and_main_contributions": "To evaluate the robustness of open information extraction (OIE) models on sentences with different syntactic structures, this paper introduces a new dataset.\nThe dataset is based on cliques, where sentences with the same semantic meaning but in different syntactic structures are in the same clique.\nSyntactic-guided paraphrasing is applied to build the dataset with manual annotation as post-correction.\nThe experimental results demonstrate that current popular models suffer from syntactic shifts, and this dataset is able to be a benchmark to evaluate the robustness.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- It would be better if the numbers in Table 1 could be right-aligned."}, "extra_text": {}}, {"review_id": "4bCcX6HtkP", "reviewer": "Reviewer_snS1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We are grateful for your recognition and encouragement, which motivates us to further explore and address the challenges facing the OpenIE and and NLP community, and make more meaningful contributions in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work portrays the robustness issue in evaluating OpenIE models, which could be significant when using such models for inferencing in a zero/few-shot setting using real-world data. This problem has been well introduced and adequately motivated.\n\n2. The subsequent sections detailing the creation of the ROBUST benchmark and the shortcomings of current state-of-the-art models regarding robustness are explained well and are easy to follow. The analysis sections provide detailed insights into the benchmark properties; the variation of a model's performance across different cliques could be used for a detailed error analysis regarding the model's generalization capabilities.\n\n3. Release of the ROBUST benchmark, which can serve as a test-bed suite for the robustness evaluation of the OpenIE models.", "reasons_to_reject": "None", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this work, the authors introduce ROBUST, a benchmark for the OpenIE task, along with a specialized metric to evaluate the robustness (to paraphrasing) of the OpenIE models. The authors argue that conventional evaluation benchmarks for the OpenIE task do not measure the robustness of the models in a real-world scenario. As an example, the authors provide a sentence s from the CaRB benchmark and show that if this sentence is paraphrased, i.e., the same semantic meaning but a different distribution of tokens, then the state-of-the-art models such as OpenIE6 fail to identify correctly the knowledge tuples from the paraphrased sentence. Therefore, instead of evaluating the pair-wise exact match between the model-extracted knowledge tuple and the ground-truth knowledge tuple, the ROBUST benchmark proposes to assess at a clique level; Given a query sentence q, a clique of semantically equivalent yet paraphrased sentences is constructed via a carefully designed paraphraser model (with diversified syntactic sampling and filtering strategies). Evaluating at a clique level, the authors claim that one can assess the performance variations (and, in turn, model robustness) due to distributional drifts within the paraphrased sentences. This analysis provides a better understanding of the model and is also compatible with existing benchmarks for the OpenIE task.\n\nFor generating the paraphrases, the authors start with the existing CarB benchmark and use the AESOP paraphrasing model (trained on a parallel annotated dataset) and the ParaNMT-50M corpus to identify ten syntactically varying paraphrases for each sentence within the CarB benchmark. A heuristic search strategy is applied to ensure that the syntactic structures and the expressive forms of the paraphrased sentences are diverse from the original sentence. The resulting paraphrases are then subject to a two-stage annotation pipeline done manually for sentence correction and structured knowledge extraction. For the sentence correction step, the annotators look at correcting grammar, replacing phrases (if required), or rewriting the entire sentence if it's semantically different from the original. For the Knowledge annotation step, human annotators are asked to annotate N-ary knowledge tuples on the paraphrases generated previously.\n\nSubsequently, the authors provide a comprehensive analysis of the ROBUST benchmark. To be exact, they quantify the syntactic similarity/distance of the sentences in a given clique via the HWS distance/Convolutional Tree Kernel similarity functions and demonstrate that the sentences in a clique are syntactically divergent. For experimental evaluations, the authors introduce the ROBUST metric.\nThe ROBUST metric is calculated on each clique of sentences instead of a single sentence. It uses the CarB scorer to evaluate the model output, i.e., P/R/F1 scores on each sentence. It selects the scores from the sentence with the worst F-measure as the ultimate robustness P/R/F score. This makes the ROBUST metric comparable to the CaRB.\n\nFor empirical evaluations, the authors evaluate six typical OpenIE models from three categories: Rule-based, Independent neural network-based, and Pretrained language model-based. They observe that all six models experience a considerable decline in the ROBUST benchmark, thereby enhancing the need to build OpenIE systems that are robust to paraphrasing.\n\nAdditionally, the authors evaluate the OpenIE capability of ChatGPT by providing 1/3/5/10 examples from the CarB benchmark as prompts. While ChatGPT under the ten-shot setting performs comparably to the supervised state-of-the-art OpenIE6 model, it too registers a decline in the performance on the ROBUST benchmark. Moreover, the authors also provide detailed analyses investigating the clique coherence in the ROBUST benchmark and the variations in model performance across different cliques.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hWahdfcVTe", "reviewer": "Reviewer_2rx1", "summary": "", "questions": "Question A: In Figure 5 and Section 4.4, is there a relation between variance in F1 within cliques, and the general robustness of OpenIE systems? If so, does the detailed analysis imply that the cliques with more diverse, out-of-distribution paraphrases will actually lead to higher consistency (and hence higher robustness)? \n\nQuestion B: Are the new paraphrases in ROBUST dataset as a whole different from CaRB, in terms of syntax and expressions? Section 3.2 only compares this within ROBUST.\n\nQuestion C: Do you have insights on the root cause for the accuracy degradation? Was that because of overfitting, or the intrinsic hardness of certain paraphrases?\n\nQuestion D: Line 190: What’s the motivation behind the removal?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the careful review and helpful suggestions. In response to the questions you raised, we have made the following explanations and revisions:\n\n\n> The proposed metric (lowest F1 score among paraphrases) does not seem particularly appropriate for evaluating “robustness”. The authors defined robustness as “behaves consistently on (paraphrases)”, but the min can not reflect variance / inconsistency. A counterexample: a really bad OpenIE system that consistently get a very low F1 score on any sentence will have a very low ROBUST score, but it’s performance is actually consistent.\n\nA1:  The worst performance indicates the lower bound of the model when applying to the realistic scenarios, which is more reliable for estimating the robustness of models in practical applications and is inline with some existing literatures [1]. We highly acknowledge the counterexample you have provided. In fact, as mentioned in line 20, we believe that a model needs to perform accurately and consistently on a clique in order to be considered robust. Thank you for your suggestion. We found that there was ambiguity in describing the robustness metric in line 379, and we will revise the description to “behave consistently accurate ..”. to make the statement clear.\n\n> Some findings in “detailed analysis” are conflicting with the main conclusion and no explanations were provided. Figure 5 suggests that the more diverse the paraphrases are, the more consistency there is. In contrast, the main argument of the paper is that different form leads to degraded accuracy, which means more variance because some F1 scores are lower.\n\nA2: We also observed and analyzed the phenomenon shown in Figure 5(a/b). By examine the cliques with high syntactic variance and the models’ performance on these cliques, we found that (1) the paraphrases tend to deviate more from the original sentence in syntactic structures and expressive forms, and (2) the models show consistent low performance (i.e., low variance) across all paraphrases (an average of 3 paraphrases in each clique). Therefore, we hypothesize that this accounts for the slight trend reflected in the Figure 5(a/b).\n\n> Question A: In Figure 5 and Section 4.4, is there a relation between variance in F1 within cliques, and the general robustness of OpenIE systems? If so, does the detailed analysis imply that the cliques with more diverse, out-of-distribution paraphrases will actually lead to higher consistency (and hence higher robustness)?\n\nAnswer to question A: Although we have generated as comprehensive syntactic and expressive variations as possible during preparation and annotation, the diversity within cliques still has limitations compared to the real-world. Therefore, the variances in model performance within cliques cannot rigorously reflect the robustness of models, but rather reflects the inter-consistencies of the cliques. This is in hopes of promoting the development of an effective data structure for other robustness evaluations. In fact, this is also one of the reasons that we consider the lowest performance as the final robustness score in the definition.\n\n> Question B: Are the new paraphrases in ROBUST dataset as a whole different from CaRB, in terms of syntax and expressions? Section 3.2 only compares this within ROBUST.\n\nAnswer to question B: Based on the syntactically controlled generation and diversified filtering, we devoted ourselves to ensure the differences of generated sentences from CaRB. We manually checked the generation results based on randomly sampling on both two stages, and we found that most of the paraphrases exhibit the obvious syntactic differences (e.g, transformation from active to passive voice or the inclusion of clauses) in stage 1, and the same paraphrases as the original sentence are further rewritten into different forms in syntax and expressions at stage 2.\n\n> Question C: Do you have insights on the root cause for the accuracy degradation? Was that because of overfitting, or the intrinsic hardness of certain paraphrases?\n\nAnswer to question C: The conclusion from Figure 1 in RobustOIE [2] shows that the OpenIE model performance exhibits a significant degradation as the syntactic similarity between the training set OIE2016 and testing set CaRB decreases. We hypothesis that the existing models trained on the OIE2016 which originated from Wiki and WSJ newswires with fixed semantic-syntactic pairs may suffer from the generalization issue to syntactic/expressive variants, and the existing testset with the design is difficult to validate this generalization performance. Based on our analysis, we suggest that the more diverse training corpora incorporating domain-specific priors (e.g, syntactic and semantic knowledge) may be a good solution to mitigate the domain bias for the adaption of models.\n\n> Question D: Line 190: What’s the motivation behind the removal?\n\nAnswer to question D: Based on the observation on the generated paraphrases, we find that most of the low-quality cases tend to be a paraphrase of partial content of the original sentence. By statistics on randomly sampled generations, we remove the generated paraphrases that the lengths are less than 2/3 of the original sentence.\n\n\nReference:  \n[1] Zhong, Victor, et al. \"Romqa: A benchmark for robust, multi-evidence, multi-answer question answering.\" arXiv preprint arXiv:2210.14353 (2022).  \n[2] Qi, Ji, et al. \"Syntactically Robust Training on Partially-Observed Data for Open Information Extraction.\" In EMNLP 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and easy to follow. The process for creating the dataset is systematic and rigorous.\n\nThe paper confirms an important, unsolved, and ubiquitous issue in existing OpenIE systems, and also provides a reasonable benchmark to evaluate the issue. These could bring about future research ideas and improved OpenIE systems.\n\nIf the ROBUST dataset is released, the community will have a high-quality resource for evaluating robustness in OpenIE. The automated method and manual annotation process for paraphrasing also have the potential to benefit research related to paraphrasing and language education.", "reasons_to_reject": "The proposed metric (lowest F1 score among paraphrases) does not seem particularly appropriate for evaluating “robustness”. The authors defined robustness as “behaves consistently on (paraphrases)”, but the min can not reflect variance / inconsistency. A counterexample: a really bad OpenIE system that **consistently** get a very low F1 score on any sentence will have a very low ROBUST score, but it’s performance is actually consistent.\n\nSome findings in “detailed analysis” are conflicting with the main conclusion and no explanations were provided. Figure 5 suggests that the more diverse the paraphrases are, the more consistency there is. In contrast, the main argument of the paper is that different form leads to degraded accuracy, which means more variance because some F1 scores are lower.", "questions_to_authors": "Question A: In Figure 5 and Section 4.4, is there a relation between variance in F1 within cliques, and the general robustness of OpenIE systems? If so, does the detailed analysis imply that the cliques with more diverse, out-of-distribution paraphrases will actually lead to higher consistency (and hence higher robustness)? \n\nQuestion B: Are the new paraphrases in ROBUST dataset as a whole different from CaRB, in terms of syntax and expressions? Section 3.2 only compares this within ROBUST.\n\nQuestion C: Do you have insights on the root cause for the accuracy degradation? Was that because of overfitting, or the intrinsic hardness of certain paraphrases?\n\nQuestion D: Line 190: What’s the motivation behind the removal?", "paper_topic_and_main_contributions": "This paper introduces a new benchmark (a dataset and a metric) named ROBUST for evaluating robustness in OpenIE systems. It attempts to answer the question: will the performance of existing OpenIE systems drop when provided paraphrases of test sentences that express the same relation differently?\n\nA systematic method to generate paraphrases that are diverse in both syntax and expressions was used to derive the new dataset from CaRB. Human annotations were involved to ensure the quality of the dataset.\n\nEvaluation was performed on a wide range of existing methods.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Studies related to robustness, in the sense of consistency across paraphrases, were not mentioned as related work. Some seminal papers:\n\nDi Jin, Zhijing Jin, Joey Tianyi Zhou, Peter Szolovits:\nIs BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment. AAAI 2020: 8018-8025\n\nYanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard H. Hovy, Hinrich Schütze, Yoav Goldberg:\nMeasuring and Improving Consistency in Pretrained Language Models. Trans. Assoc. Comput. Linguistics 9: 1012-1031 (2021)", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 297, 517 and multiple other occurrences: from ACLPUB paper formatting guidelines:\n> Refrain from using full citations as sentence constituents. Instead of “(Gusfield, 1997) showed that … In (Gusfield, 1997), …” write “Gusfield (1997) showed that … In Gusfield (1997), …”\n\nLine 323 and Figure 3: I suppose “x-axis” in the caption means the horizontal axis? It is a bit ambiguous. Perhaps having axis labels is better.\n\nLine 417: python -> Python\n\nLine 418 and footnote 4: OpenAI provides official versioning for the models behind their API, for example “gpt-3.5-turbo-0613”"}}]

Author rebuttal:

Meta-review: This paper introduces a new benchmark for evaluating the robustness of OIE systems to distributional drifts surfaced as syntactic structure variations of semantically similar statements. 

The reviewers unanimously acknowledge the importance of the proposed resource, the quality of the dataset construction and the in-depth analysis of the experiments. Reviewers 59er and 2rx1 raised some issues and posed some interesting questions that were successfully addressed by the authors and the clarifications will further enhance the manuscript.
Decision: Accept-Main