RexUIE: A Recursive Method with Explicit Schema Instructor for Universal Information Extraction
Authors: Chengyuan Liu
Fubang Zhao
Yangyang Kang
Jingyuan Zhang
Xiang Zhou
Changlong Sun
Kun Kuang
Fei Wu
Venue: EMNLP 2023 Findings (2023)
Keywords: Universal Information Extraction
Few-Shot Learning

Abstract: Universal Information Extraction (UIE) is an area of interest due to the challenges posed by varying targets, heterogeneous structures, and demand-specific schemas. 
Previous works have achieved success by unifying a few tasks, such as Named Entity Recognition (NER) and Relation Extraction (RE), while they fall short of being true UIE models particularly when extracting other general schemas such as quadruples and quintuples.
Additionally, these models used an implicit structural schema instructor, which could lead to incorrect links between types, hindering the model's generalization and performance in low-resource scenarios. 
In this paper, we redefine the true UIE with a formal formulation that covers almost all extraction schemas. To the best of our knowledge, we are the first to introduce UIE for any kind of schemas. 
In addition, we propose RexUIE, which is a Recursive Method with Explicit Schema Instructor for UIE. To avoid interference between different types, we reset the position ids and attention mask matrices. RexUIE shows strong performance under both full-shot and few-shot settings and achieves state-of-the-art results on the tasks of extracting complex schemas.

RexUIE: A Recursive Method with Explicit Schema Instructor
for Universal Information Extraction
Chengyuan Liu' liucy @2ju.edu.cc
Fubang Zhao 2 fubang.zfb warg alibubu inc
Yangyang Kang²t yungyang kangyy @ aalibaba inc
Jingyuan Zhang zhangiing uan 1994 gmaii.ccm
Xiang Zhou
Changlong Sun changlong sclo ttthhnn.comm
Kun Kuanglt kunkuang 6 wjj..cc..n
Fei Wu wafei @rzj..ddo.on
0020358rjjn.coo rju.edu.cm
College Computer Science Technology, Zhejiang Univenstyy. Damo Academy, Group Guanghua Law School, Zhejiong University. "Stanghai Institute for Advanced Study of Zhejjang University
Abstract
Edacated ALc Universlt I: Acsiame Degres,
Schema
Universal Information Extraction (UIE) is am area of interest due the challenges posed by varying targets, heterogeneous structures, and demand-spenn -peecific schemas. Previous works have achieved success by unifying few tasks, such as Named Entity Recognition (NER) and Re- lation Extraction (RE), while they fall short of being true UIE models particularly when ex- tracting other general schemas such as quadru- ples and quintuples. Additionally, these models used an implicit structural schema instructor, which could lead to incorrect links between types, hindering the model's generalization and performance in low-resource scenarios. In this paper, we redefine the true UIE with formal formulation that covers almost all extraction schemas. To the best of our knowledge, we the first introduce UIE for any kind of schemas. In addition, we propose RexUIE, which is a Recursive Method with Explicit Schema Instructor for UIE To avoid interfer- ence between different types, we reset the posi- tion ids and attention mask matrices. RexUIE shows strong performance under both full-shot and few-shot settings and achieves state-of-the- art results on the tasks of extracting complex schemas.
(2) Previous UIE
r: Educated At
Person
Leonard Parker (4) received (53) from Harvard University (sg)
(gt Academic
(g University
Person
Degree
(Educated At)
(b) RexUIE
Comparison of RexUIE with previous UIE, (a) The previous UIE models the information extraction task by defining the text spans and the relation between span pairs, but is limited to extracting only two spans. (b) Our proposed RexUIE recursively extracts text spans for each type based on given schema, and feeds the extracted information to the following extraction.
Figure 1:
et al. (2021) modeled the cross-task dependency by Graph Neural Networks. Another successful attempt is Universal Information Extraction (UIE). Lu et al. (2022) designed novel Structural Schema Instructor (SSI) as inputs and Structured Extraction Language (SEL) outputs, and proposed unified text-to-structure generation framework based on T5-Large. While Lou al. (2023) introduced three unified token linking operations and uniformly ex- tracted substructures in parallel, which achieves new SoTAs on IE tasks.
1 Introduction
As fundamental task of natural language under- standing, Information Extraction (IE) has been widely studied, such as Named Entity Recogni- tion (NER), Relation Extraction (RE), Event Ex- traction (EE), Aspect-Based Sentiment Analysis (ABSA), etc. However, the task-specific model structures hinder the sharing of knowledge and structure within the IE community.
However, they have only achieved limited suc- cess by unifying few tasks, such Named Entity Recognition (NER) and Relation Extraction (RE), while ignoring extraction with more than 2 spans, such as quadruples and quintuples, thus fall short of being true UIE models. As illustrated in Fig- ure (a), previous UIE can only extract pair of spans along with the relation between them, while ignoring other qualifying spans (such as location. time, etc.) that contain information related the two entities and their relation.
Some recent studies attempt model NER, RE, and EE together to take advantage of the dependen- cies between subtasks. Lin et al. (2020); Nguyen
"Equal contribution. 'Corresponding author.
Morcover, previous UIE models are short explicitly utilizing extraction schema to restrict

---

outcomes. The relation work for provides a case wherein the subject and object are the person and organization entities, respectively. Omitting an explicit schema can lead to spurious results, hinder- ing the model's generalization and performance in resource-liminn scenarios.
rather than only extracting spans and pairs.
2. We introduce RexUIE, which recursively runs queries for all schema types and utilizes three unified token-linking operations to compute the outcomes of each query. It employs ex- plicit schema instructions to augment label semantic information and enhance the perfor- mance in low-resource scenarios.
In this paper, redefine Universal Informa- tion Extraction (UTE) via a comprehensive for- mal framework that covers almost all extraction schemas. To the best of our knowledge, we are the first to introduce UIE for any kind of schema. Additionally, we introduce RexUIE, which a Recursive Method with Explicit Schema Instruc- tor for UIE. RexUIE recursively runs queries for all schema types and utilizes three unified token- linking operations to compute the results of cach query. We construct an Explicit Schema Instructor (EST), providing rich label semantic information to RexUIE, and assuring the extraction results meet the constraints the schema. ESI and the text are concatenated to form the query.
3. We pre-train RexUIE to enhance low-resource performance. Extensive experiments demon- strate its remarkable effectiveness, as RexUTE surpasses not only previous UIE models and task-specific SoTAs in extracting entities, re- lations, quadruples and quintuples, but also outperforms large language models (such as ChatGPT) under zero-shot setting.
Related Work
Task-specific models for IE have been exten- sively studied, including Named Entity Recogni- tion (Lample et al., 2016; Yan al., 2021a; Wang et al., 2021), Relation Extraction (Li al., 2022; Zhong and Chen, 2021; Zheng et al.. 2021), Event Extraction (Li et al., 2021), and Aspect-Based Sen- timent Analysis (Zhang et al., 2021; Xu et al., 2021).
Take Figure/ (b) traction schema, RexUIE firstly extracts "Leonard Parker" classified as "Person", then extracts "Har- vard University" classified "University" coupled with the relation "Educated At" according to the schema. Thirdly, based on the extracted tuples ( "Leonard Parker", "Person") and ( "Harvard Uni- versity", "Educated At (University)") ), RexUIE derives the span "PhD" classified as an "Academic Degree". RexUIE extracts spans recursively based on the schema, allowing extracting more than two spans such quadruples and quintuples, rather than exclusively limited to pairs of spans and their relation.
example, given the ex-
Some recent works attempted to jointly extract the entities, relations and events (Nguyen et al., 2022; Paolini et al., 2021). OnelE (Lin et al., 2020) firstly extracted the globally optimal IE result a graph from an input sentence, and incorporated global fcatures to capture the cross-subtask and cross-instance interactions. FourlE (Nguyen et al., 2021) introduced an interaction graph between in- stances of the four tasks. Wei et al. (2020) proposed using consistent tagging schemes to model the ex- traction of entities and relationships. Wang et al. (2020) extended the idea unified matrix repre- sentation. TPLinker formulates joint extraction as a token pair linking problem and introduces a novel handshaking tagging scheme that aligns the bound- ary tokens of entity pairs under each relation type. Another approach that has been used to address joint information extraction with great success is the text-to-text language generation model. Lu et al. (2021a) generated the linearized sequence of trig- ger words and argument in a text-to-text manner. Kan al. (2022) purposed to jointly extract in- formation by adding some general or task specific prompts in front of the text.
We pre-trained RexUIE on a combination of su- pervised NER and RE datasets, Machine Reading Comprehension (MRC) datasets, as well as 3 mil- lion Joint Entity and Relation Extraction (JERE) instances constructed via Distant Supervision. Ex- tensive experiments demonstrate that RexUIE passes the state-of-the-ee performance in various tasks and outperforms previous UIE models in few- shot experiments. Additionally, RexUIE exhibits remarkable superiority in extracting quadruples and quintuples.
The contributions of this paper can be summa- rized as follows:
1. We redefine true Universal Information Ex- traction (UIE) through a formal framework that covers almost all extraction schemas,

---

Qi
Output
Q111
[P]
[CLS]
P.
(P]
[CLS]
Prompts
M
Ps
ESI.
Pt+1
554,
Isolation
[T]
[Tex]]
[Text]
Zi
4
20
m
FFNN
Decoding Strategy
Encoder
Ø1
FFNN
Rotary Embedding
The overall framework of RexUIE. We illustrate the computation process of the i-th query and the construction of the i + 1-th query. M, denotes the attention mask matrix, and Z, denotes the score matrix obtained decoding. Yú denotes the output of the i-th query, with all outputs ultimately combined form the overall extraction result.
Figure 2:
Lu et al. (2022) introduced the unified structure generation for UIE. They proposed a framework based on T5 architecture to generate SEL contain- ing specified types and spans. However, the auto- regressive method suffers from low GPU utilization. Lou et al. (2023) proposed an end-to-end frame- work for UIE, called USM, by designing three uni- fied token linking operations. Empirical evaluation on 4 IE tasks showed that USM has strong general- ization ability in zero/few-shot transfer settings.
ity in Equation
II p((s,t))C", (s.t)CA
II (a.))
Il p (s,, x",x)
II (A)))
((ss,)) C" x)
where C" denotes the hierarchical schema (a tree structure) with depth n, A is the set of all sequences of annotated information. = [+1, f...... ... tx] is one of the type sequences (paths in the schema tree), and x is the text. s [s1, 12......... snl denotes the corresponding sequence of spans to t. We use (s, (); to denote the pair of 8; and 4, Similarly, (s, t) denotes (st, 82.--.,
3 Redefine Universal Information Extraction
While Lu et al. (2022) and Lou et al. (2023) pro- posed Universal Information Extraction methods of addressing NER, RE, EE, and ABSA with sin- gle unified model, their approaches were limited to only a few tasks and ignored schemas that con- tain more than two spans, such as quadruples and quintuples. Hence, we redefine UIE to cover the extraction of more general schemas.
and [t, t2, f....... ... 1] Ail (s, t) is the of the i-th items of all sequences led by (s, t) in A. To more clearly clarify the symbols, we present some examples in Appendix H.
In our view, genuine UIE extracts collection structured information from the text, with each item consisting of n spans s = [91, 82, .,, n corresponding types t [ , t.). The spans are extracted from the text, while the types are defined by given schema. Each pair of (st, 4)) is the target be extracted.
RexUIE
In this Section, we introduce RexUIE: A Recur- sive Method with Explicit Schema Instructor for Universal Information Extraction.
8n] Sn] and
RexUIE models the learning objective Equation as a series of recursive queries, with three unified token token-linking operations employed to compute the outcomes of each query. The condition (s, t)
Formally, we propose to maximize the probabil-

---

[T] peronn [T]
org [Test] to 1997
Steve Jobs
reter ned
Apple CEO
[P] pesion
Steve Jobs
work for
org
Apple
CBO
19]
[
person
[T]
0rg
[1xs]
1999
Str00
X05s
retumed
[Teak
Apple
Apple
CBO
Figure 3: Queries and score matrices NER RE. left sub-figure shows how to extract entities "Steve Jobs" and "Apple". The right sub figuurrre shows how to extract the relation given the entity "Steve Jobs" coupled with type "person". The schema is organized as ""person": work for (organization)". nullj, "organization" : mill The score matrix is separated into three valid parts: token head-tail, type-token tail and token head-type. The cells scored as I are darken, the others are scored as O.
in Equation I is represented by the prefix in the i-th query, and (s, (); is calculated by the linking operations.
Hadamard product. R(Pk- -!) E Rxddd denotes the rotary position embeddings (RoPE), which a relative position encoding method with promising theoretical properties.
4.1 Framework of RexUIE
Finally, we decode the score matrix Zi to obtain the output Y, and utilize it to create the subsequent query Q++1- All ultimate outputs merged into the result ) (Yi, Y2.... We utilize Circle Loss (Sun al., 2020; Su al., 2022) as the loss function of RexUIE, which very effective in calculating the loss of sparse matrices
Figure 2 shows the overall framework. RexUTE recursively runs queries for all schema types. Given the i-th query Qi, we adopt pre--reaaee pre-trained language model as the encoder to map the tokens to hidden representations h2 E Raxdd where n is the length the query, and is the dimension of the hidden states,
Σ ( log(1 + Σ e et)
C; = loggl11 +
h4 = Encoder((ii.. Pi, M:)
(2)
2/=0
24
L L Σ C.
where P. and Mj denote the position ids and atten- tion mask matrix of Q; respectively..
(4)
Next, the hidden states are fed into two feed- forward neural networks FFNN, FFNN,
where Z is a flattened version of Z,, and 2, de- notes the flattened ground truth, containing only and O.
Then we apply rotary embeddings following Su et al. (2021, 2022) to calculate the score matrix Z,
4.2
Explicit Schema Instructor
Zidh (FFNN, (K)) R(P) P!) FFNN, @ Mpkk (3)
The i-th query Qi consists of an Explicit Schema Instructor (ESI) and the text ESI is concatena- tion of a prefix p: and types = [t] ........ The prefix p; models (s, t)<i in Equation 1, which is constructed based on the sequence of previously extracted types and the corresponding sequence spans. t, specifies what types can be potentially
where M.*k and t denote the mask value and score from token j to respectively. p/ and pk denote the position ids of token j and k. @ is the

---

identified from x given p;-
Type Ids
Token
9
We insert a special token [P] before each prefix and a [T] before each type. Additionally, we insert a token [Text] before the text x. Then, the input Qi can be represented as
Postion lds
Atlenion Mask
101
[P]
Q: [CLS][P]>.CTTI ... [Text]0041
(5)
[V]
The biggest difference between ESI and implicit schema instructor that the sub-types that each type can undertake are explicitly specified. Given the parent type, the semantic meaning of each sub- type is richer, thus the UIE has a better under- standing to the labels.
[1]
[V]
Some detailed examples of ESI are listed in Ap- pendix I.
[Tdd
4.3
Token Linking Operations
Figure 4:
Token type ids, position ids and Attention mask for RexUIE. p and denote the prefix and types of the first group of previously extracted results. q and u denote the prefix and types for the second group.
Given the calculated score matrix Z, we obtain Z from Z by predefined threshold 8 following
z = {1 if Z' 2 8 0 otherwise
(6)
Type-Token Tail Linking Type-token tail link- ing refers to the connection established between the type of span and its tail. Similar to token head-type linking, the type token to represent the type. As highlighted in the blue section of Figure 3, a link exists from the [T] token preceding "person" to "Jobs" due to the prediction that "Steve Jobs" is a "person" span. During inference, for pair of token {i, if ZJJ > 5, and there exists a [T] k that satisfies Zíkk 2 8 and Z 2 8, we extract the span Q with the type after k.
Token linking is performed on 2, which takes bi- nary values of either 1 or 0 (Wang et al., 2020; Lou et al., 2023). A token linking is established from the i-th token to the j-th token only if 2 1; otherwise, no link exists. To illustrate this process, consider the example depicted in Figure We expound upon how entities and relations can be extracted based on the score matrix.
utilize the [T] token before
Token Head-Tail Linking Token head-tail link- ing serves the purpose of span detection. if i < j and Z = 1, the span QF should be extracted. The orange section in Figure 3 performs token head- tail linking, wherein both "Steve Jobs" and "Apple" are recognized entities. Consequently, a connec- tion exists from "Steve" to "Jobs" and another from "Apple" itself.
4.4 Prompts Isolation
RexUIE can receives queries with multiple pre- fixes. To save the cost of time, we put different prefix groups in the same query. For instance, con- sider the text "Kennedy was fatally shot by Lee Harvey Oswald on November 22, 1963", which contains two "person" entities. We concatenate the two entity spans, along with their correspond- ing types in the schema respectively to obtain ESI: [CLS][P|pprson: Kennedy [T] kill (person) (T] live in (location)... person: Lee Harvey Os- wald [T] kill(person) [T] live in (location).... However, the hidden representations of type kill (person) should not be interfered by type live (location). Similarly, the hidden representations of prefix person: Kennedy should not be interfered by
Token Head-Type Linking Token head-type linking refers to the linking established between the head of a span and its type. To signify the type, we utilize the special token [T], which is positioned just before the type token. As highlighted in the green section of Figure 3, "Steve Jobs" qualifies as a "person" type span, so link points from "Steve" to the [T] token that precedes "person". Similarly, a link exists from "Apple" to the [T] token preced- ing "org".

---

other prefixes (such as person: Lee Harvey Oswald) either.
also incorporates relative position information via disentangled attention, similar to our rotary mod- ule. We set the maximum token length to 512, and the maximum length of ESI to 256. We split a query into sub-queries when the length of the ESI is beyond the limit. Detailed hyper- parameters are available in Appendix B. Due to space limitation, we have included the implementation details of some experiments in Appendix C.
Inspired by Yang et al. (2022), we present Prompts Isolation, an approach that mitigates in- terferences among tokens of diverse types and pre- fixes. By modifying token type ids, position ids, and attention masks, the direct flow of information between these tokens is effectively blocked, en- abling clear differentiation among distinct sections in ESI. We illustrate Prompts Isolation in Figure For the attention masks, each prefix token can only interact with the prefix itself, its sub-type tokens, and the text tokens. Each type token can only in- teract with the type itself, its corresponding prefix tokens, and the text tokens.
5.1 Dataset
We mainly follow the data setting of Lu al. (2022); Lou et al. (2023), including ACE04 (Mitchell et al., 2005), ACE05 (Walker et al.. 2006), CoNLL03 (Tjong Kim Sang and De Meul- der. 2003), CoNLL04 (Roth and Yih, 2004), NYT (Riedel et al.. 2013), SciERC (Luan et al., 2018), CASIE (Satyapanich et al., 2020), SemEval-14 (Pontiki al., 2014), SemEval-15 (Pontiki et al., 2015) and SemEval-16 (Pontiki al., 2016). We add more tasks evaluate the ability of extract- ing schemas with more than two spans: 1) Quadru- ple Extraction. We use HyperRED (Chia et al.. 2022), which is a dataset for hyper-relational ex- traction to extract more specific and complete facts from the text. Each quadruple of HyperRED con- sists of standard relation triple and an additional qualifier field that covers various attributes such time, quantity, and location. 2) Comparative Opin- ion Quintuple Extraction (Liu al., 2021). COQE aims to extract all the comparative quintuples from review sentences. There are at most attributes for each instance to extract: subject, object, aspect, opinion, and the polarity of the opinion(e.g. better. worse, or equal). We only use the English subset Camera-COQE.
Then the position ids P and attention mask M in Equation can be updated. In this way, potentially confusing information flow is blocked. Addition- ally, the model would not be interfered by the order of prefixes and types either.
4.5 Pre-training
To enhance the zero-shot and few-shot performance of RexUIE, we pre-trained RexUIE on the follow- ing three distinct datasets:
Distant Supervision data Ddistont We gathered the corpus and labels from WikiPedia' and utilized Distant Supervision to align the texts with their respective labels.
Supervised NER and RE data D Dauperv Com- pared with Distant supervised data exhibits higher quality due to its absence of abstract or over- specialized classes, and there is no high false nega- tive rate caused by incomplete knowledge base.
MRC data Dmre The empirical results of previ- ous works (Lou et al., 2023) show that incorporat- ing machine reading comprehension (MRC) data into pre-training enhances the model's capacity to utilize semantic information in prompt. Accord- ingly we add MRC supervised instances to the pre- training data.
The detailed datasets and evaluation metrics are listed in Appedix A.
5.2 Main Results
We first conduct experiments with full-shot training data. Table presents a comprehensive compari- son of RexUIE against T5-UTE (Lu et al., 2022), USM (Lou et al., 2023), and previous task-specific models, both in pre- pre-training and non non-pre-trane prepr-rraiinng scenarios.
Details of the datasets for pre-training can be found in Appedix G.
5 Experiments
We can observe that: 1) RexUIE surpasses the task-specific state-of-the-art models on more than half of the IE tasks even without pre-training. RexUIE exhibits a higher F1 score than both USM and T5-UIE across all the ABSA datasets. Fur- thermore, RexUIE's performance in the task of
We conduct extensive experiments in this Section under both supervised settings and few-shot set- tings. For implementation, we adopt DeBERTaV3- Large (He et al., 2021) as our text encoder, which "htns://wwwwwiiipcci..gog

---

Without Pre-training
With Pre-training
Dataset
Task- Task-Specific SOTA Methods
T5-UIE
USM
RexUIE
TS-UIE
USM
RexUIE
ACE04
Lou et al. (2022)
87.90
86.52
87.79
88.02
86.89
87.62
87.25
ACE05-Ent
Lou al. (2022)
86.91
85.52
86.98
86.87
85.78
87.14
87.23
CoNILL03
Wang al. (2021)
93.21
92.17
92.79
93.31
92.99
93.16
93.67
ACE05-Rel
Yan et al. (2021b)
66.80
64.68
66.54
63.44
66.06
67.88
64.87
Huguet Cahot and Nav-
CoNLL04
75.40
73.07
75.86
76.79
75.00
78.84
78.39
igli (2021)
Huguet Cabot and Nav-
NYT
93.40
93.54
93.96
94.35
93.54
94.07
94.55
igli (2021)
SciERC
Yan et al. (2021b)
38.40
33.36
37.05
38.16
36.53
37.36
38.37
ACB05-EvE 1-E05-Evt-Tle Evt- Trg
Wang et al. (2022)
73.60
72.63
71.68
73.25
73.36
72.41
75.17
ACBo5-Evt-Arg
Wang et al. (2022)
55.10
54.67
55.37
57.27
54.79
55.83
59,15
CASIE-Trg
Lu et al. (2021b)
68.98
68.98
70.77
72.03
68.33
71.73
73.01
CASIE-Arg
Lu et al. (2021b)
60.37
60.37
63.05
62.15
61.30
63.26
63.87
14-res
Zhang et al. (2021)
72.16
73.78
76.35
76,36
74.52
77.26
77.46
14-lap
Zhang et al. (2021)
60.78
63.15
65.46
66,92
63.88
65.51
66.41
15-res
Xu et al. (2021)
63.27
66.10
68 80
70.48
67.15
69.86
70.84
16-res
Xu ct al. (2021)
70.26
73.87
76.73
78.13
75.07
78.25
77.20
HyperRED
Chia et al. (2022)
66.75
73.25
75.20
Camera-COQE
Liu (2021)
13.36
32.02
32.80
Table 1: Fl result for UIE models with pre-training. *-Trg means evaluating models with Event Trigger F1, +-Arg means evaluating models with Event Argument F1, while detailed metries listed in Appendix B. T5-UIE and USM are the previous SoTA UIE models proposed by Lu et al. (2022) and Lou et al. (2023), respectively.
Model
1-Shot
5-Shot
10-Shot
AVE-S
corresponding types. It is worth noting that the schema of trigger words and arguments ACE05- Evt is complex, and the model heavily relies on the semantic information of labels. 3) The bottom two rows describe the results of extracting quadruples and quintuples, and they are compared with the SoTA methods. Our model demonstrates signifi- cantly superior performance both HyperRED and Camera-COQE, which shows the effectiveness of extracting complex schemas.
Entity
T5-UIE
75.32
79.12
USM
71.11
83.25
84.58
79.65
CoNLLO3
RexUIE
$6.57
89.63
90.82
89.07
T5-UlB
51.64
58.98
CoNLLO4
USM
36.17
53.2
60.99
50.12
RexUIR
43.80
54.90
61.68
$3.46
Bvent
USE
53.07
54.35
Trigger
USM
40.86
55.61
$8.79
51.75
ACR05-Evt
RexUIE
56.95
64.12
65.41
62.16
Event
TS-UIE
31,20
35.19
26.98
Argument
USM
19.01
36.69
48
32.73
ACE05 Evt
RexUIE
30.43
41.04
45.14
38.87
Sentiment
T5-UIB
42.67
16-res
USM
30.81
$2.06
58.29
47.05
RexUIE
37.70
49.84
60.56
49.37
Table 2: Few-Shot experimental results. AVE-S denotes the average performance over I-Shot, 5-Shot and 10- Shot.
5.3 Few-Shot Information Extraction
We conducted few-shot experiments on one dataset for each task, following Lu al. (2022) and Lou al. (2023). The results are shown in Table 2.
Event Extraction is remarkably superior to that the baseline models. 2) Pre-training brings in slight performance improvements. By comparing the outcomes in the last three columns, we can ob- serve that RexUIE with pre-training is ahead of T5-UIE and USM the majority of datasets. Af- ter pre-training. ACE05-Evt showed significant improvement with an approximately 2% increase in Fl score. This implies that Rex UIF effectively utilizes the semantic information in prompt texts and establishes links between text spans and their
In general, RexUIE exhibits superior perfor- mance compared to T5-UIE and USM in low- resource setting. Specifically, RexUIE relatively outperforms T5-UIE by 56.62% and USM by 32.93% average in 1-shot scenarios. The suc- cess of RexUIB in low-resource settings can be tributed to its ability extract information learned during pre-training, and to the efficacy of our proposed query, which facilitates explicit schema learning by RexUIE.

---

Task
Model
Precision
Recall
F1
approximately 8% lower than RexUIE on Fl.
GPT-3 ChatGPT
18.10
5.6
Absense of Explicit Schema
25.16
19.16
21.76
Relation
DEEFSTRUCT
25.80
CoNLL04
USM
25.95
RexUIE*
44.95
23.22
30.62
incekonnoccaiion
0.005
0.29
Iostoonvonn
0.0066
0.56
Entity
ChatGPT
62.30
55.00
58.40
CoNLLpp
RexUIE"
91.57
66.09
76.77
locallon-poople
021
organio popicotonnooaaionn
0.7
004
Table 3: Zero-Shot performance on RE and NER. indicates that the experiment is conducted by ourselves.
*
organization none
0.075
0.04
0.22
orga organization
orgonizaiionpegg
offor-locaaion
0.011
people location
0.01
0.68
0
0:48
0.038
Model
Precision
Recall
F1
seop
T5-UIE RexUIE
20.92
21.56
21.23
Without Pre-train
31.43
32.62
32.02
peopple
0.025
organication
T5-UIE RexUIE
24.41
25.46
24.92
With Pre-train
34.07
31.63
32.80
localion boad
0.013
0.55
locekcn-rorm
0:43
0
Table 4: Extraction results COQE.
02808
5.4 Zero-Shot Information Extraction
people loca
0:91
0.018
0
-04
We conducted zero-shot experiments on RE and NER comparing RexUIE with other pre-trained models, including ChatGPT². We adopt the pipeline proposed by Wei et al. (2023) for Chat- GPT. We used CoNLL04 and CoNLLpp (Wang et al.. 2019) for RE and NER respectively, We report Precision, Recall and Fl in Table 3.
02
peoplopasple
0.82
600
onganizaaion on in live
Iscated
work
(6) Shot
Figure 5: The distribution of relation type versus subject type-object type predicted by T5-UIE. We circle the correct cases in orange.
RexUIE achieves the highest zero-shot extrac- tion performance on the two datasets. Furthermore, we analyzed bad cases of ChatGPT. 1) ChatGPT generated words that did not exist the original text. For example, ChatGPT output a span "Coats Michael", while the original text was "Michael Coats". 2) Errors caused by inappropriate granu- larity, such as "city in Italy" and "Italy". 3) Illegal extraction against the schema. ChatGPT outputs (Leningrad, located in, Kirov Baller), while "Kirov Ballet" an organization rather than location.
We analyse the distribution relation type ver- sus subject type-object type predicted by T5-UTE as illustrated Figure 5.
We observe that illegal extractions, such as per- workfor location, are not rare in 1-Shot, and considerable number of subjects objects not properly extracted during the NER stage. Although this issue is alleviated the 5-Shot scenario, we believe that the implicit schema instructor still neg- atively affects the model's performance.
5.5 Complex Schema Extraction
To illustrate the significance of the ability extract complex schemas, we designed forced approach extract quintuples for T5-UTE, which extracts three tuples to form one quintuple. Details are available in Appendix
6 Conclusion
In this paper, we introduce RexUIE, UIE model using multiple prompts to recursively link types and spans based on an extraction schema. We redefine UIE with the ability to extract schemas with any number of spans and types. Through ex- tensive experiments under both full-shot and few- shot settings, we demonstrate that RexUIE outper- forms state-of-the-ard methods on a wide range of datasets, including quadruples and quintuples ex- traction. Our empirical evaluation highlights the
Table 4 shows the results comparing RexUIE with T5-UIE. In general, RexUIE's approach of directly extracting quintuples exhibits superior per- formance. Although T5-UIE shows slight per- formance improvement after pre-training, still
Shaps///pppaai.co//bbaagahaa

---

significance of explicit schemas and emphasizes that the ability to extract complex schemas cannot be substituted.
4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang. Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD: A few-shot named entity recognition dataset. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1:
Limitations
Despite demonstrating impressive zero-shot en- tity recognition and relationship extraction per- formance, RexUTE currently lacks zero-shot ca- pabilities in events and sentiment extraction due the limitation of pre-training data. Furthermore, Rex not yet capable of covering all NLU tasks, such as Text Entailment.
Long Papers), pages 3198-3213, Online. Association for Computational Linguistics.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
Ethics Statement
Lluis Huguet Cabot and Roberto Navigli. 2021. REBEL: Relation extraction by end-to-end language generation. In Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 2370- 2381, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Our method is used to unify all of information ex- traction tasks within one framework. Therefore, ethical considerations of information extraction models generally apply to our method. We ob- tained and used the datasets in a legal manner. We encourage researchers to evaluate the bias when using RexUIE.
Zhigang Kan, Linhui Feng, Zhangyue Yin, Linho Qiao, Xipeng Qiu, and Dongsheng Li. 2022. A Unified Generative Framework based on Prompt Learning for Various Information Extraction Tasks. arXiv e- prints, page arXivv2209.111 11570....117......
Acknowledgements
This work was supported in part by National Key Research and Development Program of China (2022FF33340900) National Natural Sci- ence Foundation of China (62376243, 62037001, U20A20387), the StarryNigh Science Fund of Zhejiang University Shanghai Institute for Ad- Alibaba
Guillaume Lample, Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. CoRR, fbs/1603.01366
Qian Li, Shu Guo, Jia Wu, Jianxin Li, Jiawei Sheng, Lihong Wang. Xiaohan Dong, and Hao Peng. 2021, Event extraction by associating event types and argu- ment roles. CoRR, abs/2108. 10038.
vanced Study (SN-ZJU-SIAS-0010) Group through Alibaba Research Intern Program, Project by Shanghai AI Laboratory (P22KS001 1), Program of Zhejiang Province Science and Tech- nology (2022C01044), the Fundamental Research Funds for the Central Universities (226-2022- 00142, 206222200001.
Zhe Li, Luoyi Fu, Xinbing Wang. Haisong Zhang, and Chenghu Zhou. 2022. RFBEN: A relation-first blank filling network for joint relational triple extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 10-20. Association for Computa- tional Linguistics.
References
Ying Lin, Heng Ji. Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7999-8009, Online, Association for Computational Linguistics.
Yew Ken Chia, Lidong Bing, Sharifah Mahani Alju- nied, Luo Si, and Soujanya Poria. 2022. A dataset for hyper-relational extraction and a cube-filling ap- proach. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 101 14-10133, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Jingjing Liu, Panupong Pasupat, Scott Cyphers, and James Glass. 2013. Asgard: A portable architecture for multilingual dialogue systems. pages 8386-8390.
Jacob Devlin, Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 16622.

---

Zihan Liu, Yan Xu, Tiezheng Yu. Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pas- cale Fung. 2020. Crossner: Evaluating cross-domain named entity recognition.
Alexis Mitchell, Stephanie Strassel, Shudong Huang. and Ramez Zakhary. 2005. Ace 2004 multilingual training corpus.
Minh Van Nguyen, Viet Dac Lai, and Thien Huu Nguyen. 2021. Cross-task instance representation interactions and label dependencies for joint infor- mation extraction with graph convolutional networks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, pages 27-38, Online, Association for Computational Linguistics.
Ziheng Liu, Rui Xia, and Jianfei Yu. 2021. Comparative opinion quintuple extraction from product reviews. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 3955-3965, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. CoRR,
abs/1711.05101.
Minh Van Nguyen, Bonan Min, Franck Dernoncourt, and Thien Nguyen. 2022. Joint extraction of entities, relations, and events via modeling inter-instance and inter-label dependencies. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4363-4377 Seattle, United States. Association for Computational Lin- guistics.
Chao Lou, Songlin Yang, and Kewei Tu. 2022. Nested named entity recognition as latent lexicalized con- stituency parsing. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 6183-6198. Dublin, Ireland. Association for Computational Lin- guistics.
Jie Lou, Yaojie Lu, Dai Dai, Wei Jia, Hongyu Lin, Xi-
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma. Alessandro Achille, Rishita Anubhai, Cícero Nogueira dos Santos, Bing Xiang, and Ste- fano Soatto. 2021. Structured prediction as transla- tion between augmented natural languages. CoRR, 0e//2101.05779
ampei Han, Le Sun, and Hua Wu. 2023. Universal information extraction as unified semantic matching. arXiv preprint arXiv:2301.0333 23.002322.
Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021a. Text2Event: Controllable sequence-to- structure generation for end-to-end event extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Lan- guage Processing (Volume I: Long Papers), pages 2795-2806, Online. Association for Computational Linguisties.
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou. Ion Androutsopoulos, Suresh Manandhar, Moham- mad AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clereq. Véronique Hoste, Marianna Apidianaki, Xavier Tannier, Na- talia Loukachevitch, Evgeniy Kotelnikov, Nuria Bel, Salud María Jiménez- -afra, and Gülşen Eryigit. 2016. SemEval-2016 task 5: Aspect based sentiment analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-201 pages 19-30, San Diego, California. Association for Computational Linguistics.
Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021b. Text2Event: Controllable sequence-to- structure generation for end-to-end event extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 2795-2806, Online. Association for Computational Linguistics.
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and lon Androutsopoulos. 2015. HamEval-201s. task 12: Aspect based sentiment anal- ysis. In Proceedings of the 9th International Work- shop on Semantic Evaluation (SemEval 2015), pages 486-495, Denver, Colorado. Association for Compu- tational Linguistics.
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Uni- fied structure generation for universal information extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5755-5772, 5772, Dublin, Ireland. Association for Computational I Linguisties.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Har- ris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment amalysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (Se- mEval 2014), pages 27-35, Dublin, Ireland. Associa- tion for Computational Linguistics.
Yi Luan, Luheng He, Mari Ostendorf, and Hannanch Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Con- ference Empirical Methods in Natural Language Processing, pages 3219-3232, Brussels, Belgium. Association for Computational Linguistics.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards ro- bust linguistic analysis using OntoNotes. In Proceed- ings of the Seventeenth Conference on Computational

---

Natural Language Learning, pages 143-152, Sofia, Bulgaria. Association for Computational Linguistics.
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang. Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021. Improving named entity recognition by external con- text retrieving and cooperative learning. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), ), pages 1810- Online. Association for Computational Linguisties.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, page 10.55500
Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Pro- ceedings of the 2013 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 74 84, Atlanta, Georgia. Association for Computa- tional Linguistics.
Yucheng Wang. Bowen Yu, Yueyang Zhang, Tingwen Liu, Hongsong Zhu, and Limin Sun. 2020. Tplinker: Single-stage joint extraction of entities and relations through token pair linking.
Zihan Wang. Jingho Shang, Liyuan Liu, Lihao Lu, Ji- acheng Liu, and Jiawei Han. 2019. Crossweigh: Training named entity tagger from imperfect annota- tions, arxiv preprint arXivv1999.011
Dan Roth and Wen-tau Yih. 2004. A linear program- ming formulation for global inference in natural lan- guage tasks. Proceedings of the Fighth Confer- ence on Computational Natural Language Learn- ing (CoNLL-2004) at HLT-NAACL 2004, pages 1-8, Boston, Massachusetts, USA. Association for Com- putational Linguistics.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang. Xin Zhang, Shen Huang. Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023. Zero- shot information extraction via chatting with chatgpt. arXiv preprint 2tt::2302.10000
Tanceya Satyapanich, Francis Ferraro, and Tim Finin. 2020. Casie: Extracting cybersecurity event informa- tion from text. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8749 8757.
Zhepei Wei, Jianlin Su. Yue Wang, Yuan Tian, and Yi Chang. 2020. A novel cascade binary tagging framework for relational triple extraction. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1476- 1488.
Jianlin Su. Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2021. Roformer: En- hanced transformer with rotary position embedding. arXiv preprint arXiv:2104.0960 09864.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison. Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational Linguistics.
Jianlin Su, Ahmed Murtadha, Shengfeng Pan, Jing Hou, Jun Sun, Wanwei Huang, Bo Wen, and Yunfeng Liu. 2022. Global pointer: Novel efficient span-basee based ap- proach for named entity recognition. arXiv preprint arXiv:2208.0300
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. 2020. Circle loss: A unified perspective of pair simi- larity optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages -407.
Lu Xu, Yew Ken Chia, and Lidong Bing. 2021. Learn- ing span-level interactions for aspect sentiment triplet extraction. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I 11th International Joins Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 4755-4766, Online, Association for Computa- tional Linguistics.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-inder independent named entity recognition. In Proceedings of the Seventh Conference Natural Language Learning at HLT-NAACL 2003, pages 142- 147.
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021a. A unified generative framework for various NER subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5808-5822, Online. Association for Computational Linguistics.
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus.
Sijia Wang. Mo Yu, Shiyu Chang, Lichao Sun, and Lifu Huang. 2022. Query and extract: Refining event extraction as type-oriented binary decoding. In Find- ings of the Association for Computational Linguistics: ACL 2022, pages 169-182, Dublin, Ireland. Associa- tion for Computational Linguistics.
Zhiheng Yan, Chong Zhang, Jinlan Fu, Qi Zhang, and Zhongyu Wei. 1b. A partition filter network for

---

joint entity and relation extraction. In Proceedings of the 2021 Conference on Empirieal Methods Nat- ural Language Processing, pages 185-197, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Relation Striet F1 A relation is correct if its rela- tion type is correct and the offsets and entity types of the related entity mentions are correct.
Relation Triplet F1 A relation is correct if its relation type is correct and the string of the related entity mentions are correct.
Ping Yang, Junjie Wang, Ruyi Gan, Xinyu Zhu, Lin 7hang, Ziwei Wu, Xinyu Gao, Jiaxing Zhang, and Tetsuya Sakai. 2022. Zero-shot learners for natural language understanding via unified multiple choice perspective. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing. pages 7042-7055, Abu Dhabi, United Arah Emi- rates. Association for Computational Linguistics.
Event Trigger F1 An event trigger is correct if its offsets and event type matches a reference trigger.
Event Argument F1 An event argument is cor- rect if its offsets, role type, and event type match a reference argument mention.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Sentiment Striet F1 For triples, a sentiment is correct if the offsets of its target, opinion and the sentiment polarity match with the ground truth. For quintuples, sentiment is correct if the offsets of its subject, object, aspect, opinion and the sentiment polarity match with the ground truth.
Dongxu Zhang and Dong Wang. 2015. Relation classification via recurrent neural network. CoRR,
Quadruple Striet F1 A relation quadruple cor- rect the relation type and the type and offsets of its subject, object, qualifier match with the ground truth.
Wenxuan Zhang. Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2021. Towards generative aspect-based sentiment analysis. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 1Ith International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pages 504-510, Online. Association for Computational Linguistics.
Task
Metric
Dataset
Entity Striet FI
ACE04-Ent
Entity
Entity Striet FI Entity Striet F1
ACE0S-Em Ent
CoNLLO03
Hengyi Zheng, Rui Wen, Xi Chen, Yifan Yang, Yun- yan Zhang, Ziheng Zhang, Ningyu Zhang, Bin Qin, Xu Ming, and Yefeng Zheng. 2021. PRGC: Potential relation and global correspondence based joint rela- tional triple extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 6225-6235, Online. Association for Computational Linguistics.
Relation Striet
ACE0S-Ra
Relation Striet Fl
CoNLL04
Relation
Relation Triplet FI
NYT
Relation Striet
SciERC
Trigger Fl
ACE0S-Evt
Argument F1
ACE05-Evt
Event
Trigger F1
CASIE
Argument FI
CASIE
Sentiment Striet F1
14-res
Sentiment Striet F1
14-lap
Sentiment
Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 50-61, Online. Association for Computational Linguistics.
Sentiment Striet F1
15-res
Sentiment Striet F]
16-res
Quadruple
Quadruple Strict F1
HyperRED
COQE
Sentiment Striet F1
Momera-COQE
Table 5: Detailed supervised datasets and evaluation metrics for each task.
A Detailed Supervised Datasets for Downstream Tasks
B Implementation Details
The detailed datasets and evaluation metrics are listed in Table We explain the evaluation metrics as follows.
We download the supervised data for pre-training from HuggingFace' For all the downstream datasets, we follow the procedure by Lu et al. (2022); Lou et al. (2023) and then convert them to Phtros//huggnngfacccovaasss
Entity Striet F1 An entity mention is correct its offsets and type match reference entity.

---

Learning Rate
Batch
propose model the quintuple extraction ex- tracting three triples: (subject, "subject-object', ob- ject), (object, "object-aspect", aspect), and (aspect, sentiment, opinion).
Epoch
Size
Pre-training
Se-5
128
5
Low-resource
1e-5, 3e-5
16
50, 100
Entity
3e-5
3c-5
64
100, 200
Relation
lc-5, 3e-5 3e-5
64, 128
50, 100, 200
D Ablation Study
Event
3c-5
96
100
Sentiment
3e-5
100, 200
Quadruple
3e-5
24
10
COQE
3e-5
100,
Method
Precision
Recall
F1
RexUTE
92.95
93.66
93.31
Table 6: Detailed Hyper-parameters.
Entity
w/o Pl
93.00
93.79
93.39
CoNLL03
w/o Rt
92.16
93.84
92.99
w/o PI+Rt
92.37
93.52
92.94
the imput format of RIxUIE. We implement the pre- training model and trainer based on Transformers (Wolf et al., 2020). We adopt DeBERTaV3-Large (He et al., 2021) as our text encoder. We set the maximum token length to 512, and the maximum length of prompt to 256 We split a query into sub- queries containing prompt text segments when the length of the prompt text is beyond the limit. Our model is optimized by AdamW (Loshchilov and Hutter, 2017), with weight decay as 0.01, threshold 8 as O. We the clip gradient norm as 2, warmup ratio as 0.1. The syper-parameters for grid search are listed in Table
RexUIE
78.80
74.88
76.79
Relation CoNLL04
w/o PI
75.31
72.27
73.76
w/o Rt
75.06
72.75
73.89
w/o Pi+Rt
73.56
72.51
73.03
RexUIE
71.36
75.24
73.25
Event Trigger
w/o Pl
73.80
72.41
73.10
w/o Rt
72.06
76.65
74.29
ACE05-Evt
w/o Pl+Rt
73.51
72.64
73.07
RexUIE
56.69
57.86
57.27
Event
Argument
w/o
57.74
56.97
57.36
w/o Ri
57.93
59.64
58.77
ACB05-Evt
w/o PI+Rt
55.84
55.34
55.59
RexUIE
75.18
81.32
78.13
Sentiment 16-res
78.45
78.60
78.52
69.27
81.13
74.73
w/o PI+Rt
70.88
78.60
74.54
C Details of Experiments Settings
RexUIE
75.75
w/o
75.23
AVG
Few-Shot IE We conducted few-shot experi- ments on one dataset for each task, following Lu et al. (2022) and Lou et al. (2023). Specifically, we sample 1/5/10 sentences for each type of en- tity/relation/eeee//seetiii
74.93
Pi+Rt
73.83
Table 7: Ablation Study. PI denotes Prompts Isolation, Rt denotes rotary rmbedding. AVG is calculated over the five tasks.
from the training set. To avoid the influence of sampling noise, we re- peated each experiment 10 times with different samples.
We conducted an ablation experiment on RexUIE to explore the influence of Prompts Iso- lation and rotary embedding on the model, where RexUIE is not pre-trained. The results are listed in Table 7.
Zero-Shot IE We used CoNLL04 for RE due to its unique subject and object entity types for each relation. For NER, we employed CoNLLpp (Wang et al., 2019), which is a corrected version of the CoNLL2003 NER dataset. In order to prevent the performance of ChatGPT from being affected by randomly selected instructions, adopted the SoTA zero-shot information extraction framework with ChatGPT proposed by Wei al. (2023).
The experimental results demonstrate that moving Prompts Isolation Icads decrease in the performance of RE, while the exclusion of ro- tary embedding results detrimental effects on both relation and sentiment extraction. Overall, the complete RexUIE exhibits superior performance. Removing Prompts Isolation or rotary embedding results in slight decline in performance, with the most significant drop observed when both are deleted.
Complex Schema Extraction T5-UIE was ini- tially limited to extracting only triples. We de- signed forced approach to extract quintuples for T5-UIE, which extracts three tuples form one quintuple.
E Influence of Encoder
The quintuple in COQE can represented as (subject, object, aspect, opinion, sentiment). We
DeBERTa (He et al., 2021) improved BERT (De- vlin et al., 2019) and RoBERTa (Liu et al., 2019)

---

Dataset
USM
RE
RexUIE
row. Therefore, believe that the perfor-
mance improvement is negatively correlated
CoNLL03
92.76
92.98
93.31
CoNLL04
75.86
77.15
76.79
with training data size. We note the training data size as S.
ACBOS (Trigger)
71.68
72.15
73.25
ACR08.Fet (Argument)
55.37
57.77
57.27
16-res
76.73
77.13
78.13
To investigate the pattern, we introduce a media variable log(10000) x g). After removing certain outliers and event-trigger datasets, we find a posi- tive correlation between log(10000 } and the relative improvement in Table 9, which supports our hypothesis.
Average 74.48
75.44
75.75
Table Replacing DeBERTa with RoBERTa. We com- pare BeXURER. with USM as they share the same encoder structure.
models using the disentangled attention mecha- nism and enhanced mask decoder. Corrected sen- tence: To ensure that our work was not entirely dependent better text encoder, replaced DeBERTaV3-Large with RoBERTa-Large, denoted ReXUREROBERTa. thus maintaining the same text encoder settings USM. We present the compari- son in Table 8.
G Pre-training Data
D Diistant We remove abstract and over- specialized entity types and relation types (such "structural class of chemical compound") and remove categories that occur less than 10000 times. We also remove the examples that do not contain any relations. Finally, we collect 3M samples containing entities and relations as D Ddistant-
RexUIFRoBERTa shows a performance level be- tween USM and RexUIE, demonstrating that our proposed approach does indeed improve perfor- mance, and incorporating DeBERTaV3 to RexUIE further enhances this improvement.
D Daupero We collect some high-quality data for named entity recognition and relationship extrac- tion from publicly available open domain super- vised datasets. Specifically, we employ OntoNotes (Pradhan et al., 2013), NYT (Riedel et al., 2013), CrossNER (Liu et al., 2020), Few-NERD (Ding et al., 2021). kbp37 (Zhang and Wang, 2015), Mit Restaurant and Movie corpus (Liu et al., 2013) to- gether as D supere.
F Insights to the Schema Complexity and Training Data Size
Under the full-shot setting, the improvement of RexUIE compared to the previous UIE models is not significant (only 1% across 4 tasks and 14 met- rics). At the same time, have also found that for different tasks or datasets, the improvement of RexUIE seems exhibit some randomness, which may related several factors such as schema complexity, training data size, task type, and the extent of task exploration. Among these factors, we believe that schema complexity and training data size are more important, so we conducted statistical analysis to better summarize the patterns. (We only consider the case of full-shot without pre-training avoid the influence of pre-training.)
D mrc Specifically, we collect SQuAD (Rajpurkar et al., 2016) and HellaSwag (Zellers et al., 2019) together as Dyrrc- The MRC data is constructed with pairs of questions and answers. For imple- mentations, we use the question as the type, and consider the answer as the span to extract.
H Example of Schema
Schema examples for some datasets are listed Table 10.
Schema complexity: Due to ESI and recursive strategies, intuitively believe that Rex UIE has certain advantages in handling complex schemas. We use the number of leaf nodes in the schema to represent the complexity of the schema, noted as C.
I Query Example
Some query examples are listed in Table 11 and Table 12.
Training data size: We know that as the train- ing data size increases, the differences be- tween the performance of models will be nar-

---

Dataset
Schema Complexity C
Training Data
log( 10000 )
USM
RexUIE
Relative Improvement
Size S
CoNLL03
14041
0.6977
92.79
93.31
0.56%
ACE04
6202
0.8095
87.79
88.02
0.26%
ACEOS-Ent
7299
0.9818
86.98
86.87
-0.13%
NYT
55
56196
0,9907
94.07
94.55
0.51%
14-res
3
1266
1.3747
76.35
76.36
0.01% %
14-lap
906
1.5200
65.46
66.92
2.23%
16-res
857
1.5441
76.73
78.13
1.82%
ACE0S-EE ACE0S-Evt-Arg
79
19216
1.6100
55.37
57.27
3,43%
15-res
605
1.6954
68.8
70.48
2.44%
CoNLL04
922
1.8134
75.86
76.79
1.23%
SciERC
115
1861
2.7910
37.05
38.16
3.00%
Table 9: The correlation between schema complexity, training data size and relative improvement.

---

Dataset
Schema C
Example of
Entity CoNLLO3
("person": null, "location": null, "miscellaneous": null, "organi- zation": null)
["person")
("organization":
("organization in (location J": null), "other":
Relation CoNLL04
("located in location )": null], "people": ("iive in ( location )"; null, "work for organization )": null. "kill people )": null)]
null. "location":
("organization", "'rganization in (loca- tion ]"]
I"attack": ("attacker": null, "place": null, "target": null, "instru- ment": null), "end position": : ("person": null, "place": null, "en- tity:: null], "meet": ("place": null, "entity": null), "transport": ("artifact": null, "origin": : null, "destination": null, "agent": null, "vchicle": null), "dic": f"victim": null, "instrument", null, "place": null. "agent": null), "transfer money": ("giver": null, "beneficiary": null, "recipient": : null]. "trial hearing": ("adjudi- cator": null. "defendant": null, "place": null). "charge indict": ("defendant": null, "place": null, "adjudicator": null]. "trans- fer ownership": ("bencticiary": null, "artifact": null, "seller": null. "place": null, "buyer": null]. "sentence": ("defendant": null, "place": null, "adjudicator": null), "extradite": ("per- son": null, "destination". null, "agent": null), "start position": ("place": null, "entity": null, "person": null). "start organiza- tion": ("organization": null, "agent": null, "place"; null], "sue": (" ""defendatt": null, "plaintiff": null). "divorce": ["person": null. "place": null), "marry": ("person": null, "place": null), "phone write": ("place": : null, "entity"; : null], "injure": ("victim": null). "end organization": ("organization" : null]. "uppeal":: ("adjudi- cator": null, "plaintiff"; null). "convict": ("defendant": null, "place": mull, "adjudicator": nullj, "fine": ("entity": null, "ad- judicator": null), "declare bankruptey": ("organization": null], "demonstratc": ("place": null, "entity": mull), "elect": ("per- son": null, "place": null, "entity": null], "nominate") ("person": null). "acquit": ("defendant": null. "adjudicator"; null), "exe- cute": ""agent": null, "person": null, "place": null). "release parole": ["person": null]. "arrest jail": ("person": null, "place": null, "agent": null). "born": l perrson" null, "place": null)]
Event ACE05-Evt
I"attack", "attacker"]
Sentiment
["aspect": ("positive oopinion y": null, "neutral ooiinoon y":
("aspect", "positive (opiiion )"]
16-res
null, "negative opinion y": mull), "opinion": null]
("subject": ("object": ("aspect": ("worse ( oppooiion )": null. "equal opinion )": null, "better opinionn )": null, "different ( opinion y": null], "worse opionion y": null, "equal opinion 5" null, "better (opinion y": null, "different opinion y": null). "aspect": ["worse opionion )": null, "equal opinion y": null, "better opinion y": null. "different opinioo y": null). "worse ( opionion Y": null. "equal opinion y": null, "better opinion )": null. "different (opinion )": null). "object": ["aspeet": ["worse ( opionion y": null, "equal opinion )": null, "better opinion
COQE Camera
["subject", "object", "aspect", "better (
opinion )"]
)": null, "different opinion )": null]. "worse opionion )": null. "equal opinion )": null, "better opinion )": null, "different ( opinion y": null], "aspect": ("worse opionion y": null, "equal ( opinion )": null, "better ( opinion )"; null, "different opinion y": null), "worse opionion y"; null, "equal opinion y": null, "better opinion )": null, "different opinion y": null)]
Table 10: Schema examples.

---

Dataset
Sample Id Query
[CLS][P][T] location[T]] miscellaneous(T] organization[T] per- son[Text] EU rejects German call to boycott British lamb [SSP]
CoNLL03
0
[CLS][P][T] location[T]] organization[T] other[T] people[Text] The self-propelled rig Aveo 5 was headed to shore with 14 people aboard early Monday when it capsized about 20 miles off the Louisiana coast, near Morgan City Lifa said.[SEP]
CoNLL04
0
[CLS][P] location: Morgan City[T] located in ( (location [PP]
location: [ouisiana[T] located in (location )[P] people: Lifa[T] kill people }[]] live in location )TT] work for ( (organization
[[exxt] The self-propelled rig Avco was headed to shore with 14
people aboard early Monday when it capsized about 20 miles off the Louisiana coast, near Morgan City Lifa said.[SEP]
[CLS][P][T] acquit[T] appeal[T] arrest jail attack[T] born charge indict[T] convict[T] declare bankruptcy[T] demonstrate die(T) divorce(T] elect[T) end organization(TT end position[]] [T] execute[T] extradite[T] fine[T] injure[T] marry[T] meet[T] merge [T] organization|TT nominate[T] pardon[T] phone write|T] release pa- role(T] sentence[T] start organization[T] start position[[] sue(T] transfer money[T] transfer ownership[T] transport[T] trial hear- ing[Text The electricity that Enron produced was so exorbitant that the government decided it was cheaper not to buy electricity and pay Enron the mandatory fixed charges specified in the con- tract [SEP]
ACE05-Evt
[CLS[[P] transfer money: pay[T] beneficiary(T] giver[T] place[T] recipien([Tex() The electricity that Enron produced was so orbitant that the government decided it was cheaper not to buy electricity and pay Enron the mandatory fixed charges specified in the contract [SEP]
[CLS][P][T] acquit[T] appeal[T] arrest jail(T] attack[T] born [T] charge indict[T] convict[T] declare bankruptey[T] demonstrate [T] die[T] divorce[T] elect[T] end organization[T] end position[T]] cxecute[T] extradite[T] finelT] injure[T] marry[T] meet[T] merge organization[[T] nominate[T] pardon[T] phone write[T] release pa- role|T] sentence/T] start organization|[]] start position[T] suelT] transfer money(T] transfer ownership[T] transport[T] trial hear- ing[Text] and he has made the point repeatedly in interview after interview that he has never claimed to speak for god, nor has he claimed that this is god á war "[SEP]
[CLS][P] attack: war[T] attacker[T] instrument[T] place[T] tar- get[TT] victim[Text] and he has made the point repeatedly in inter- view after interview that he has never claimed to speak for god, nor has he claimed that this is god á war "[SEP]
Table 11: Query examples for CoNLLO3, CoNLL04 and ACE05-Evt.

---

Dataset
Sample Id
Query
[CLS[[PI[T] aspect[T] apinion[Texl Judging from previous posts this used to be good place, but not any longer .SSP]]
0
16-res
[CLS][P] aspect: place[T] negative (opinion )[T] neutral opinion
[[] positive (opinion )[Text] Judging from previous posts this used to be good place, but not any longer [[SEP]
[CIS][[][T] aspect[T] apinion[Texxt The food was lousy too sweet or too salty and the portions tiny .[SEP]
[CLS][P] aspect: portions[T] negative ( (opinion ){) neutral opinion ][T positive ( (oiinion )[P] aspect: food[T] negative (
opinion [T] neutral opinion )[T) positive ( opinioo ][Text] The
food was lousy too sweet or too salty and the portions tiny [SEP]
[CLS|[PI[T] aspect[T] better (piiioon I[T] different (opinion ) equal (opinion )T] object[T] subject[T] worse opionioon [Texx] Also both the Nikon D50 and D70S will provide sharper pictures with better color saturation and contrast right out of the camera
COQE-Camera
[SEP]
[CLS][P] subject: Nikon D50[T] aspect[T] better (opinion [[]
different ( opinion )[T] equal (opinion [ object[T] worse (
opionion [PP] subject: D70S(T) aspect(T] better opinion )[T]
different ( opinion )[T] equal (opinion )[]] object[T] worse (
opionion ][exxt] Also both the Nikon D50 and D70S will provide
sharper pictures with better color saturation and contrast right out of the camera [SEP]
[CLS][P] subject: jappccs- pictures[T]] [ better ( (opinion [T]
different opinion )T] equal (opinion )[T] worse (opionion [P]
subject: aspect color saturation[T] better (piiioon [[]]
different opinion )[T] equal (opinion
)[T] worse (opionion [P]
subject: Nikon D50,aseect: pictures[T] better opinion ][]] dif- ferent opinion )T] equal (opinion )[]] worse (
(piooioon [P]]
subject: Nikon D50,aspect: contrast(T] better opinion )[)] dif- ferent opinionn [T] equal (opinion [T] worse opiooioon [PP] subject: Nikon D50,aspect: color saturation[T] better ( (opinion ][T] different opinion [[] equal (opinion )[T] worse opionion
[P] subject: D70S,aspeed contrast(T] better (opinion [T]] dif- ferent ( opinion )[T] equal (opinion )/T] worse opiooioon [exxt] Also both the Nikon D50 and D70S will provide sharper pictures with better color saturation and contrast right out of the camera [SEP]
Table 12: Query examples for 16-res and COQE-Camera.


Reviews:
[{"review_id": "0SjkPrcM8E", "reviewer": "Reviewer_zUQ9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of the novelty and experimental effectiveness of RexUIE. Indeed, UIE is a field that requires extensive experiments across various settings. Thus, we conducted as comprehensive experiments as possible within our scope of considerations. We are delighted that you recognize our work.\n\nWe also appreciate your valuable suggestions, and we will provide explanations point by point to address your concerns. We hope that we can alleviate any worries you may have.\n\n> 1. The performance improvement over baselines is hard to interpret since the proposed model uses a different base model (DeBERTa) apart from the architecture change.\n\nDue to space limitations, **we have presented the performance comparison with the same base model in Table 8 (Appendix E)**. As shown in Table 8, based on RoBERTa-Large, our model outperforms USM by nearly 1 point, and falls slightly behind RexUIE based on DeBERTa-Large by 0.31 points under the full-shot setting. Therefore, **we believe that the architecture of RexUIE plays a more substantial role in enhancing performance**.\n\nWe genuinely appreciate your valuable suggestions, and we will provide a clearer explanation of these comparative experiments in the main body of our paper.\n\n> 2. The proposed method includes a flexible architecture to unify a couple of IE tasks but it cannot do real zero-shot extraction with unseen schemas.\n\n**In Section 5.4, we conducted zero-shot experiments for the NER and RE tasks.** As illustrated in Table 3, RexUIE exhibited a significant improvement over both USM and ChatIE. To ensure the fairness of the experiments, we **maintained a consistent zero-shot setting with USM** and deliberately excluded the data from Table 3 during the pre-training process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a new model for Universal Information Extraction (UIE), which shows strong empirical performance.\n2. The authors conduct extensive experiments in various settings to demonstrate the effectiveness of the proposed model.", "reasons_to_reject": "1. The performance improvement over baselines is hard to interpret since the proposed model uses a different base model (DeBERTa) apart from the architecture change.\n2. The proposed method includes a flexible architecture to unify a couple of IE tasks but it cannot do real zero-shot extraction with unseen schemas.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new model for Universal Information Extraction (UIE). Compared to previous UIE models, this model includes a novel component called \"Explicit Schema Instructor\" (ESI), which is the concatenation of extraction results in the previous iteration and the extraction schema. This component allows the model to perform IE under almost all extraction schema, and the model shows great performance on 17 different datasets across 6 different tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "REU8Ag8lQM", "reviewer": "Reviewer_zhWZ", "summary": "", "questions": "Question A: l.72: \"almost all extraction schemas\": What are the schemas not covered?\n\nQuestion B: What schemas or examples still challenge RexUIE?  \n\nQuestion C: You mentioned the gains over prior UIE models were relatively small in some cases. Do you have insights into why the improvements were marginal for certain tasks/datasets?\n\nQuestion D: Have you explored using external knowledge bases or ontologies during pre-training to further enhance the model's semantic understanding? This could help boost low-resource performance.\n\nQuestion E: The zero-shot evaluation was limited to NER and RE. What challenges do you foresee in extending zero-shot capabilities to events, sentiments or other tasks?\n\nQuestion F: Has any analysis been done on how the size and richness of the schema affects RexUIE's performance? E.g. very large or nested schemas.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your diligent efforts and valuable suggestions. We will address your questions and concerns in a thorough manner.\n\n# Concerns\n\n## The approach is complex with many components (encoder, ESI, token operations, pre-training etc.) making ablation analysis difficult\n\nThis is a particularly good question, and when conducting the RexUIE work, we also devoted significant thought to devising effective ablation experiments.\n\n1. Encoder: In order to remove the influence of schema order, we have adopted the Prompt Isolation strategy, and we believe that relative positional encoding would be more advantageous for prompt isolation. Therefore, we have utilized DeBERTa as our encoder. **To eliminate the impact of encoder variation, we have conducted a performance comparison with the same base model, which is presented in Table 8 (Appendix E).** Our model based on RoBERTa-Large outperforms USM by nearly 1 point and only lags behind RexUIE based on DeBERTa-Large by 0.31 points in the full-shot scenario. Thus, we contend that the architecture has played a more significant role in enhancing performance.\n2. ESI & Token Operation: As you mentioned, the Explicit Schema Instructor is an intuitive extension aimed at enhancing UIE performance under low-resource settings. In Section 5.6, we examined the influence of this module by analyzing the distribution of \"relation type\" versus \"subject type-object type\" predicted by T5-UIE, which utilizes the Implicit Schema Instructor. We observed that under low-resource setting, T5-UIE often generates outputs that do not adhere to the schema constraints. However, as you correctly pointed out, quantifying the impact of the Explicit Schema Instructor and our recursive strategy, including Token Operation, **presents a significant challenge due to their inseparability**.\n3. Pre-training: Since that T5-UIE and USM did not release their pre-training datasets, we can only strive to keep our pre-training data as consistent as possible with theirs.\n\n## The gains over prior UIE models, while consistent, are relatively small in some cases\n\nUnder the full-shot setting, the improvement of RexUIE compared to previous UIE models is not particularly significant (1% across 4 tasks and 14 metrics). We consider that the main reason is that the datasets we used in the experiment are **suffiently explored**. \n\nAdditionally, by checking the experimental results in Table 1 and Table 2, we report the average improvement of RexUIE over USM in the following Table, from which we can observe that **as the training data increases, the relative improvement gradually decreases**.\n\n|Datasets of Table 2|1-shot|5-shot|10-shot|Full-shot|\n|-|-|-|-|-|\n|AVG Improvement RexUIE/USM|32.9|6.7|6.0|1.3|\n\nA more detailed analysis is provided in the answer to Question C.\n\n## Zero-shot capabilities only demonstrated for NER and RE but not events or sentiment\n\nThis is also an insightful question. There are two reasons why we did not report the zero-shot performance of RexUIE for EE or ABSA:\n1. We attempted to compare our work with previous studies in terms of EE and ABSA datasets. However, since **USM did not report these experiments and they did not provide code**, it was challenging for us to find a suitable UIE baseline. \n2. As mentioned in Section \"Limitations\", we intuitively believe that the performance of RexUIE in EE and ABSA would not be as ideal as NER and RE due to **the limitation of pre-training data**.\n\nWe will explore the zero-shot capabilities for more IE tasks if we find a way to solve the problems above.\n\n# Questions\n## Question A: l.72: \"almost all extraction schemas\": What are the schemas not covered?\n\nOur current approach is not yet capable of solving discontiguous entity extraction and open information extraction.\n\n## Question B: What schemas or examples still challenge RexUIE?\n\n- Challenging Schemas: discontiguous entity extraction and open information extraction.\n- Challenging Examples: examples with very long context or challenging schemas.\n\n## Question C: You mentioned the gains over prior UIE models were relatively small in some cases. Do you have insights into why the improvements were marginal for certain tasks/datasets?\n\nVery insightful question. Under the full-shot setting, the improvement of RexUIE compared to the previous UIE models is not significant (only 1% across 4 tasks and 14 metrics). At the same time, we have also found that for different tasks or datasets, the improvement of RexUIE seems to exhibit some randomness, which may be related to several factors such as schema complexity, training data size, task type, and the extent of task exploration. Among these factors, we believe that **schema complexity and training data size** are more important, so we conducted a statistical analysis to better summarize the patterns. (We only consider the case of full-shot without pre-training to avoid the influence of pre-training.)\n1. Schema complexity: Due to ESI and recursive strategies, we intuitively believe that RexUIE has certain advantages in handling complex schemas. We use the number of leaf nodes in the schema to represent the complexity of the schema, noted as $C$.\n2. Training data size: We know that as the training data size increases, the differences between the performance of models will be narrow. Therefore, we believe that the performance improvement is negatively correlated with training data size. We note the training data size as $S$.\n\nTo illustrate this pattern, we have created a line plot that visually demonstrates the relationship between the improvement $R$ and $\\log(10000\\times C / S)$ (unfortunately, images are not allowed to appear during rebuttal). After removing certain outliers and event-trigger datasets, we find a clear **positive correlation between $R$ and $\\log(10000\\times C / S)$**，which supports our hypothesis. The results are consistent with our analysis and will be included in the final version. \n\nThe values of $R$ and $\\log(10000\\times C / S)$ across various datasets are shown in the Table below.\n\n||Schema Complexity $C$|Training Data Size $S$|$\\log(10000\\times C / S)$|USM|RexUIE|Relative Improvement $R$|\n|-|-|-|-|-|-|-|\n|CoNLL03|7|14041|0.6977 |92.79|93.31|0.56%|\n|ACE04|4|6202|0.8095|87.79|88.02|0.26%|\n|ACE05-Ent|7|7299|0.9818|86.98|86.87|-0.13%|\n|NYT|55|56196|0.9907|94.07|94.55|0.51%|\n|14-res|3|1266|1.3747|76.35|76.36|0.01%|\n|14-lap|3|906|1.5200|65.46|66.92|2.23%|\n|16-res|3|857|1.5441|76.73|78.13|1.82%|\n|ACE05-Evt-Arg|79|19216|1.6140|55.37|57.27|3.43%|\n|15-res|3|605|1.6954|68.8|70.48|2.44%|\n|CoNLL04|6|922|1.8134|75.86|76.79|1.23%|\n|SciERC|115|1861|2.7910|37.05|38.16|3.00%|\n\n## Question D: Have you explored using external knowledge bases or ontologies during pre-training to further enhance the model's semantic understanding? This could help boost low-resource performance.\n\nWe have not explored using external knowledge bases or ontologies, but it is really a promising suggestion. We will explore it in the future work.\n\n## Question E: The zero-shot evaluation was limited to NER and RE. What challenges do you foresee in extending zero-shot capabilities to events, sentiments or other tasks?\n\nAs our response to concern 3, we have not extended zero-shot capabilities to events and sentiments due to the lack of baselines and pre-training data.\n\n## Question F: Has any analysis been done on how the size and richness of the schema affects RexUIE's performance? E.g. very large or nested schemas.\n\nWe conducted a statistical analysis of the relative and absolute performance changes of RexUIE with the increasing complexity of schema (Unfortunately, images are not allowed to appear during rebuttal. It will be added to the final version). The following observations can be made:\n1. **As the Schema Complexity increases, the relative performance of RexUIE compared to USM improves**, which also demonstrates the advantage of RexUIE over previous work in solving complex schemas.\n2. **When the Schema Complexity raises up to a certain level, the absolute performance of RexUIE decreases**, which we consider to be a reasonable result since higher complexity increases the difficulty of the task.\n\nWe appreciate your careful review and notice. We will correct the typos and grammar style, and presentation issues in the final version.\n\n**Overall, we would like to express our gratitude for your recognition of our work and for providing such a detailed evaluation and insightful questions. We hope that our responses can alleviate any doubts you may have.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The recursive method and explicit schema instructions are intuitive extensions for improving universal extraction. I liked the idea of using a schema when possible.\n- Extensive experiments validate RexUIE's effectiveness on diverse datasets, especially for complex schemas.\n- RexUIE improves over prior UIE models like T5-UIE and USM in both full-shot and few-shot settings\n- The pre-training strategy is sensible for further boosting low-resource performance\n- Well-written", "reasons_to_reject": "- The approach is complex with many components (encoder, ESI, token operations, pre-training etc.) making ablation analysis difficult\n- The gains over prior UIE models, while consistent, are relatively small in some cases \n- Zero-shot capabilities only demonstrated for NER and RE but not events or sentiment", "questions_to_authors": "Question A: l.72: \"almost all extraction schemas\": What are the schemas not covered?\n\nQuestion B: What schemas or examples still challenge RexUIE?  \n\nQuestion C: You mentioned the gains over prior UIE models were relatively small in some cases. Do you have insights into why the improvements were marginal for certain tasks/datasets?\n\nQuestion D: Have you explored using external knowledge bases or ontologies during pre-training to further enhance the model's semantic understanding? This could help boost low-resource performance.\n\nQuestion E: The zero-shot evaluation was limited to NER and RE. What challenges do you foresee in extending zero-shot capabilities to events, sentiments or other tasks?\n\nQuestion F: Has any analysis been done on how the size and richness of the schema affects RexUIE's performance? E.g. very large or nested schemas.", "paper_topic_and_main_contributions": "This paper proposes RexUIE, a novel approach for universal information extraction (UIE) that can extract complex schemas beyond just entity pairs. The key contributions are:\n\n- Redefining UIE to cover extraction of any schema with multiple spans/types\n- Introducing a recursive method to query all schema types and compute results via unified token-linking operations  \n- Using explicit schema instructions (ESI) to provide richer label semantics and enhance low-resource performance.\n- Demonstrating state-of-the-art performance on simultaneous extraction of entities, relations, events, sentiment analysis, and complex schemas like quadruples and quintuples", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- When citing work, add the missing space before opening parenthesis (everywhere in the paper, eg, lines 138 or 959)\n- l.474: according to [Wikipedia](https://en.wikipedia.org/wiki/Soviet_(council)), \"Soviet\" could be considered as an organisation"}}, {"review_id": "ZyhdpawD2C", "reviewer": "Reviewer_SsUQ", "summary": "", "questions": "Question A: Does the premise for judging how to perform recursive extraction, the Explicit Schema Instructor, implies prior knowledge of all possible schemas?\n\nQuestion B: Is the explicit schema information also provide to compared methods?  Could the author provide a more clearly explanation about how much improvement is solely due to a more complete schema, and how much is attributed to the advancement in recursive extraction techniques?\n\nQuestion C: Are there any instance of incomplete schemas in practice? For example, the schema requires the extraction result to have four aspects, but only three of them appear in the utterance. Will the proposed method work correctly in such cases?\n\nQuestion D: CasRel and TPlinker (please refer to missing references) proposed a matrix-based tagging scheme partly to handle the phenomenon of overlapping relations in the extraction task. (For instance, in the sentence \"A and B boarded the plane together,\" two facts can be extracted: \"A boarded the plane\" and \"B boarded the plane,\" resulting in the plane being used twice). This work strictly follows the structure of an explicit schema for extraction. Does it also have the capability to handle overlapping relations?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your comments. And we appreciate you for referring to the questions in your concerns. This allows us to address them more effectively.**\n\nBefore answering your questions, we would like to clarify the difference between Explicit Schema Instructor (ESI) and Implicit Schema Instructor (ISI) more clearly. Taking relation extraction as an example, in previous UIE works (T5-UIE, USM), the Implicit Schema Instructor simply concatenats all entity types and relation types without explicitly representing the constraints between entity types and relation types. This means that the model can only implicitly learn the constraints between entity types and relation types through training. However, our method, RexUIE, **explicitly prevents the model from generating illegal matches between entity types and relation types through the behavior-level constraints** introduced by the recursive extraction strategy. This is the fundamental difference between ESI and ISI.\n\nAs you mentioned in Question A, ESI determines how to perform recursive extraction. Therefore, **ESI and the recursive inference strategy are tightly integrated**. We cannot separate ESI from the recursive extraction strategy to evaluate its contribution to the performance of RexUIE.\n\n## Question A: Does the premise for judging how to perform recursive extraction, the Explicit Schema Instructor, implies prior knowledge of all possible schemas?\n\nYes. Currently, the academic definition of the IE task is to extract information tuples that satisfy the schema constraints from a given text. We believe that the utilization of ESI and the requirement of prior knowledge of all possible schemas are unrelated. The definition of the IE tasks determines that we need to know all possible schemas in advance, and both T5-UIE and USM (using ISI) require prior knowledge of all possible schemas as well.\n\n## Question B: Is the explicit schema information also provide to compared methods? Could the author provide a more clearly explanation about how much improvement is solely due to a more complete schema, and how much is attributed to the advancement in recursive extraction techniques?\n\nThis is a very insightful question. We also considered this question when conducting ablation experiments. As you have mentioned in Question A, ESI determines how recursive extraction is performed, indicating that **ESI and recursive reasoning are closely intertwined**. However, since T5-UIE and USM do not utilize recursive reasoning, we were unable to incorporate ESI into their models. Additionally, the inseparability between ESI and recursion makes it difficult to quantify the individual contribution of each to RexUIE. \n\nNonetheless, in order to demonstrate the effectiveness of ESI as much as possible, we analyzed the distribution of \"relation types\" versus \"subject type-object type\" predicted by T5-UIE, which uses Implicit Schema Instructor, in Section 5.6. We discovered that T5-UIE often generates outputs that do not adhere to the schema constraints in low-resource settings. \n\n## Question C: Are there any instance of incomplete schemas in practice? For example, the schema requires the extraction result to have four aspects, but only three of them appear in the utterance. Will the proposed method work correctly in such cases?\n\nThis is a very insightful question as well. First of all, please allow us to reiterate the definition of IE. Currently, the academic definition of the IE is to extract information tuples from a given text that satisfy the schema constraints. Therefore, it is invalid if a model extracts tuples of information out of the schema.\n\nRegrading to your question, **RexUIE can extract instances of incomplete schemas**. In fact, the case you mentioned exists in COQE. Sometimes, not all elements of (subject, object, aspect, opinion, polarity) present in the sentence, and the task allows for the omission of arbitrary spans. When conducting experiments on this dataset, we found that RexUIE can effectively handle these cases. There is a showcase below, where the object (another camera) is omitted in the context:\n> By the way you can use the same battery the S50 comes with .\n>\n> subject: \"S50\"; aspect: \"battery\"; equal ( opinion ): \"same\"\n\nIn Appendix G (Table 9), we demonstrate how to write the schema to solve the problem. The results of COQE in Table 1 also demonstrate that RexUIE achieves good performance in the cases you mentioned.\n\n## Question D: CasRel and TPlinker (please refer to missing references) proposed a matrix-based tagging scheme partly to handle the phenomenon of overlapping relations in the extraction task. (For instance, in the sentence \"A and B boarded the plane together,\" two facts can be extracted: \"A boarded the plane\" and \"B boarded the plane,\" resulting in the plane being used twice). This work strictly follows the structure of an explicit schema for extraction. Does it also have the capability to handle overlapping relations?\n\nYes, RexUIE has the ability to handle overlapping spans, including SEO and EPO [1]. In the case you mentioned:\n1. In the first step, we input the Person type and extract (Person: A) and (Person: B).\n2. In the second step, we input (Person: A, board) and (Person: B, board) and extract (Person: A, board: plane) and (Person: B, board: plane).\n\n[1] Extracting Relational Facts by an End-to-End Neural Model with Copy Mechanism.\n\n## Overall\n\nBased on the answers to Question A, C and D, we firmly believe that compared with previous works, the practicality of RexUIE not only will not be limited by **explicit schema instructor**, but also enhanced due to the ability to support complex schemas, uncomplete schemas, et al.\n\n## Missing References\nWe will add the missing references to the final version. Thanks for reminding us.\n\n## Typos Grammar Style And Presentation Improvements\nWe will correct the typos and grammar style, and presentation issues in the final version. Thanks again!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The improvements are significaint.\n2. The writing is fluent, making the paper easy to read.\n3. The idea of recursive extraction is novel in joint extraction methods with matrix-based tagging schemes.", "reasons_to_reject": "1. The reasons behind the results improvement might require further discussion. (Please refer to Question B)\n2. The requirement for explicit schema may limit the practicality of this method. (Please refer to Question C and D)", "questions_to_authors": "Question A: Does the premise for judging how to perform recursive extraction, the Explicit Schema Instructor, implies prior knowledge of all possible schemas?\n\nQuestion B: Is the explicit schema information also provide to compared methods?  Could the author provide a more clearly explanation about how much improvement is solely due to a more complete schema, and how much is attributed to the advancement in recursive extraction techniques?\n\nQuestion C: Are there any instance of incomplete schemas in practice? For example, the schema requires the extraction result to have four aspects, but only three of them appear in the utterance. Will the proposed method work correctly in such cases?\n\nQuestion D: CasRel and TPlinker (please refer to missing references) proposed a matrix-based tagging scheme partly to handle the phenomenon of overlapping relations in the extraction task. (For instance, in the sentence \"A and B boarded the plane together,\" two facts can be extracted: \"A boarded the plane\" and \"B boarded the plane,\" resulting in the plane being used twice). This work strictly follows the structure of an explicit schema for extraction. Does it also have the capability to handle overlapping relations?", "paper_topic_and_main_contributions": "This paper proposes an improved version of UIE, the Unified Structure Generation approach. It replaces the structural schema instructor in UIE with an explicit schema instructor where not only the schema terms but also their structures are provided. This paper proposes the idea of recursively extracting aspects instead of doing so separately. The results are significantly improved on most of the evaluated benchmarks.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "CasRel[1] proposed using consistent tagging schemes to model the extraction of entities and relationships, and TPlinker[2] extended the idea to a unified matrix representation. CasRel was not cited in this paper, and while TPlinker was mentioned at line 269, no corresponding description was provided. Although they did not mention the extraction of multi-granular structures like events, the idea of transforming multi-granular structures into combinations of binary relations has been discussed in NLP research for a long time. Therefore, I suggest introducing CasRel and TPlinker as related works in joint extraction in Section 2.\n\n[1] Zhepei Wei, Jianlin Su, Yue Wang, Yuan Tian, and Yi Chang. 2020. A Novel Cascade Binary Tagging Framework for Relational Triple Extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1476–1488, Online. Association for Computational Linguistics.\n\n[2] Yucheng Wang, Bowen Yu, Yueyang Zhang, Tingwen Liu, Hongsong Zhu, and Limin Sun. 2020. Tplinker: Single-stage joint extraction of entities and relations through token pair linking.", "Typos_Grammar_Style_and_Presentation_Improvements": "1. I personally think that the discontinuous vertical lines in Table 1 are not proper formatting. (You can find relevant discussions in the documentation of booktab.) If the author indeed wants to use booktab, please remove the vertical lines or hack it to make the lines continuous."}}]

Author rebuttal:

Meta-review: The paper proposes a new model for Universal Information Extraction (UIE) named RexUIE, which concatenates extraction results from the previous iteration with the extraction schema. The proposed methods are evaluated across 17 different datasets and 6 different tasks, such as extracting entities, relations, events, sentiment, and quadruples and quintuples.

Overall, the proposed model demonstrates a minor performance gain over the baseline in some cases when the entire training dataset is used (full-shot training), but achieves higher performance gain under few-shot settings. The paper is well-written and easy to follow, and the proposed idea is novel. There are a few missing references, and a table does not follow the EMNLP template, but these issues will be fixed in the final version.
Decision: Accept-Findings