Mirror: A Universal Framework for Various Information Extraction Tasks
Authors: Tong Zhu
Junfei Ren
Zijian Yu
Mengsong Wu
Guoliang Zhang
Xiaoye Qu
Wenliang Chen
Zhefeng Wang
Baoxing Huai
Min Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Information Extraction
Non-Autoregressive Decoding
Multi-task
TL;DR: A universal Information Extraction model that supports not only complex multi-span and n-ary extraction but also machine reading comprehension and classification tasks.

Abstract: Sharing knowledge between information extraction tasks has always been a challenge due to the diverse data formats and task variations.
Meanwhile, this divergence leads to information waste and increases difficulties in building complex applications in real scenarios.
Recent studies often formulate IE tasks as a triplet extraction problem.
However, such a paradigm does not support multi-span and n-ary extraction, leading to weak versatility.
To this end, we reorganize IE problems into unified multi-slot tuples and propose a universal framework for various IE tasks, namely Mirror.
Specifically, we recast existing IE tasks as a multi-span cyclic graph extraction problem and devise a non-autoregressive graph decoding algorithm to extract all spans in a single step.
It is worth noting that this graph structure is incredibly versatile, and it supports not only complex IE tasks, but also machine reading comprehension and classification tasks.
We manually construct a corpus containing 57 datasets for model pretraining, and conduct experiments on 30 datasets across 8 downstream tasks.
The experimental results demonstrate that our model has decent compatibility and outperforms or reaches competitive performance with SOTA systems under few-shot and zero-shot settings.
The code, model weights, and pretraining corpus are available at https://github.com/Spico197/Mirror .

Mirror: A Universal Framework for Various Information Extraction Tasks
Tong Zhu 8 Junfei Ren v Zijian Yu², Mengsong Wu Guoliang Zhang", 9, Xiaoye Qu Wenliang Chen of ZZefeng Wang*, Baoxing Huai Min Zhang"
Institute of Artificial Intelligence, School of Computer Science and Technology, Soochow University, China
Huawei Cloud, China
jfahuu,,jreefffren mswumsw, glzhang)@stu. suda, edu. cn
{quxiaoye, wangzhefeng hawpreniggnaaiibbiiininnnneneuen mengghefeerrng.aaiibbooooooonnn Jehuawei. com {wlchen minnhannndanaaaaeeeuuada edu.
Abstract
Discontinuous NER
Sharing knowledge between information ex- traction tasks has always been challenge due to the diverse data formats and task vari- ations. Meanwhile, this divergence leads to information waste and increases difficulties in building complex applications in real scenar- ios. Recent studies often formulate IE tasks as a triplet extraction problem. However, such paradigm does not support multi-span and n- ary extraction, leading to weak versatility. To this end, we reorganize IE problems into uni- fied multi-slot tuples and propose a universal framework for various IE tasks, namely Mir- ror. Specifically, we recast existing IE tasks as multi-span cyelic graph extraction prob- lem and devise aonaautorerressvve graph de- coding algorithm to extract all spans in sin- gle step.
pain
ADR
muscle
fatigue
[ADR,, muscle pain). (ADR, muscle fatigue)]
Relation Extraction
CEO of
Tim
Cook
Apple
[(Tim Cook, CEO of, Apple)]
Figure I: Multi-span cyclic graph for discontinuous NER and RE tasks (best viewed in color). The spans are connected by three types of edges, including consecu- tive connections, dotted jump connections and tail-to- head connections. ADR in discontinuous NER refers to the entity label of Adverse Drug Reaction.
worth noting that this graph structure is incredibly versatile, and it sup- ports not only complex IE tasks, but also ma- chine reading comprehension and classification tasks. We manually construct a corpus con- taining 57 datasets for model pretraining, and conduct experiments
30 datasets across 8 downstream tasks. The experimental results demonstrate that our model has decent compat- ibility and outperforms reaches competitive performance with SOTA systems under few- shot and zero-shot settings. The code, model weights, and pretraining corpus are available at /ttps://githhe com/ comn/Spio11// 1o0oo199//MM 1M/Mirrr..
difficult to share knowledge across tasks (Lu et al., 2022; Josifoski et al., 2022).
In order to unify the data formats and take advan- tage of common features between different tasks, there are two main routes in recent studies. The first one is to utilize generative pretrained language models (PLMs) to generate the structured informa- tion directly. Lu al. (2022) and Paolini et al. (2021) structure the IE tasks as a sequence-to- sequence generation problem and use generative models to predict the structured information autore- gressively. However, such methods cannot provide the exact positions of the structured information, which is essential to the NER task and fair evalu- ations (Hao et al., 2023). Besides, the generation- based methods are usually slow and consume huge resources to train on large-scale datasets (Wang et al., 2022). The second way is to apply the extrac- tive PLMs, which are faster to train and inference.
1 Introduction
Information Extraction (IE) is a fundamental field in Natural Language Processing (NLP), which aims to extract structured information from unstruc- tured text (Grishman, 2019), such as Named Entity Recognition (NER) (Qu et al., 2023b; Gu et al., 2022: Qu et al., 2023a), Relation Extraction (RE) (Cheng et al., 1021), Event Extraction (EE). How- ever, each IE task usually isolated from specific data structures and delicate models, which makes Corresponding author

---

Model
TANL
UIE
DeepStruct
InstructULE
USM
Mirror
PLM
T5-base
T5-large
GLM
FlanT5
RoBERTa
DeBERTa-v3
#Params
220M
770M
10B
11B
large 372M
large 434M
Decoding
AR
AR
AR
AR
NAR
NAR
Indexing
Partly
X
X
X
Triplet
V
/
/
Single-span NER
/
V
/
/
/
Multi-span
X
X
N-ary tuple
X
X
X
X
Classification
X
X X
X
MRC
X
Table 1: Comparisons among systems. Circle o: the model may support the task theoretically, but the current implementation is not available. AR the auto-regressive decoding while NAR is non-autoregressive. Indexing: whether the model could provide exact position information. TANL partly supports indexing because the generated tail entity in relation extraction is text-based without position information. Please refer to Appendix A for more detailed comparisons. Triplet: "(head, relation, tail)" triplet extraction. Single-span NER: flat and nested NER tasks with consecutive spans. Multi-span: multi-span extraction, c.g., the discontinuous NER. N-ary tuple: the ability of n-ary tuple extraction, e.g., quadruple extraction. Classification: the classification tasks. MRC: extractive machine reading comprehension tasks. is worth noting that generative models (TANL, UIE, DeepStruct, and InstructUIF)) may be capable of all the tasks if their current paradigms patterns changed. However, since the original papers do not contain relevant experiments, we mark them as X or here.
USM (Lou et al., 2023) regards the IE tasks triplet prediction problem via semantic matching. However, this method is limited to a small range triplet-based tasks, and unable to address multi-span and n-ary extraction problems.
ually collect 57 datasets across 5 tasks into whole corpus for model pretraining. The experimental re- sults demonstrate that Mirror achieves competitive results under few-shot and zero-shot settings. Our contributions are summarized as follows:
To overcome the above challenges, we propose Mirror, novel framework that can handle com- plex multi-span extraction, n-ary extraction, ma- chine reading comprehension (MRC), and even classification tasks, which are not supported by the previous universal IE systems, As exemplified in Figure 1, we formulate IE tasks as a unified multi- slot tuple extraction problem and transform those tuples into multi-span cyclic graphs. This graph structure is rather flexible and scalable. It can be applied to not only complex IE tasks but also MRC and classification tasks. Mirror takes schemas as part of the model inputs, and this benefits few-shot and zero-shot tasks naturally.
We propose a unified schema-guided multi- slot extraction paradigm, which is capable of complex information extraction, machine reading comprehension, and even classifica- tion tasks.
We propose Mirror, a universal non- autoregressive framework that transforms multiple tasks into a multi-span cyclic graph.
We conduct extensive experiments on 30 datasets from tasks, and the results show that our model achieves competitive results under sew-shot and zero-shot settings.
Compared with other models in Table 1, Mir- ror supports efficient non-autoregressive decoding with position indexing and shows good compatibil- ity across different tasks and datasets. We conduct extensive experiments on 30 datasets from 8 tasks, including NER, RE, EE, Aspect-basee based Sentiment Analysis (ABSA), multi-span discontinuous NER, n-ary hyper RE, MRC, and classification. To en- hance the fhw-shot and zero-shot abilities, we man-
Related Work
2.1 Multi-task Information Extraction
Multi-task IE has been popular research topic in recent years. The main idea is to use sin- gle model perform multiple IE tasks. IE tasks could be formulated as different graph structures. Li et al. (2022) formulate flat, nested, and discontin- uous NER tasks as a graph with next-neig

---

Entity Extraction
[LMMonon Jerry Smith), (LMporson. Tom)]
identity possible entities
person
location
LM
organization
TL
Jerry Smith hammer
Relation Extraction
[(L Iriad or Jerry Smith, Tom))
I
Please extract relations with head and tail entities..
LR
friend of LR
CEO of
TL
Jerry Smith is friend of Tom
Event Extraction
->
[LLMattaci hit), (LRattackor. hit, Jerry Smith), (LRvidimm hit, Tom), (LRystrument hit, hammer))
atlack
atlacker
instrument
Jerry Smith with hammer
MRC & QA
(Jerry Smith)]
I
Who is Toms' friend?
TP
Jerry Smith friend of Tom
Classification
Find the relation two sentences.
entailmenn
contradict
neutral
Premise: Hypothesis:
Instruction
Schema Labels
Text
Figure 2: Unified data interface. We design list of tokens separate different parts: [T]: instruction. [LM]: mentions. [LR]: relations. [LC]: classifications. [TL]: text that connects with schema labels. [TP]: extractive MRC and QA texts without schema labels. [B]: the background text the classification task.
and tail-to-head connections. Maximal cliques also have been used to flat & discontinuous NER tasks (Wang et al., 2021) and trigger-available & trigger-free event extractions (Zhu al., 2022). DyGlE++ takes NER, RE, and EE tasks as span graphs and applies iterative propagation to enhance spans' contextual representations (Wadden et al., 2019). OnelE uses a similar graph structure with global constraint features (Lin et al., 2020).
the text representations. InstructUIE (Wang et al., 2023) formulates schemas into instructions and FlanT5-11B (Chung et al., 2022) to perform multi-task instruction tuning.
While the above methods utilize generative lan- guage models, they cannot predict exact positions, which brings ambiguity when evaluating (Hao et al., 2023). Besides, large generative language models are usually slow to train & infer and re- quire tons of computing resources. USM (Lou et al., 2023) applies BERT-family models to extract triplets con-autoregrrsssivly. USM regards IE a unified schema matching task and uses a label- text matching model to extract triplets. However, these methods cannot extend to complex IE tasks, such as multi-span discontinuous NER and n-ary information extractions.
In addition to explicit graph-based multi-task IE systems, generative language models are widely used. Yan et al. (2021b) and Yan al. (202 la) add special index tokens into BART (Lewis al., 2020) vocabulary to help perform various NER and ABSA tasks and obtain explicit span positions. TANL (Paolini et al., 2021) apply T5 (Raffel et al., 2020) to generate texts with special enclosures as the predicted information. GenlE (Josifoski et al., 2022) and DeepStruct (Wang et al., 2022) share similar idea to generate subject-relation-oojjer triplets, and DeepStruct extends the model size to 1OB with GLM (Du et al., 2022).
3 Mirror Framework
In this section, introduce the Mirror framework. We first address the unified data input format to the model, then introduce the unified task formulation and the model structure.
2.2 Schema-guided Information Extraction
In schema-guide IE systems, schemas are input as a guidance signal to help the model extract target information. UIE (Lu et al., 2022) categorize IE tasks into span spotting and associating elemen- tary tasks and devise linearized query language. Fei al. (2022) introduces the hyper relation ex- traction task to represent complex IE tasks like EE, and utilize external parsing tools to enhance
3.1 Unified Data Interface
To enable the model to handle different IE tasks, we propose unified data interface for the model input. As shown in Figure 2, there are three parts: in- struction, schema labels, and text. The instruction is composed of a leading token [I] and a natural language sentence. The [I] token indicates the

---

Graph Prediction
Graph Decoding
LM, LMbeaok
LR
attneter
LR Sealn
LR Li
Jerry
Smith
Tom
with
hammer
label of mentonns
a
(event types)
Instrument
hit
hammer
LR
label of relations (roles)
jump connection
LMattack
conseculivv connection
victim
hit
bri--p-head connection
Lattacier
LRvictim
Instrument
attacker
Jerry
Smith
attack
hit
Jerry
Smith
hit
trigger the attack event
hit
Tom
with
hammer
Jerry Smith
Tom
hammer
Extractive Prelrained Language Model
Please extract events
attack
ettacker
victim
Instrument
Jerry Smith Tom with hammer
Instruction
Schema Labels
Text
Figure 3: Model framework (best viewed in color). Mirror first constructs inputs each task, then utilizes a pretrained language model to predict the adjacency matrix via the biaffine attention. After that, final results are decoded from the adjacency matrix accordingly.
instruction part while the sentence tells the model what it should do. For example, the instruction of NER could be "Please identify possible entities". MRC and Question Answering (QA) tasks, the instruction is the question to answer.
unified data interface, and the model can be trained in a unified way even if the model is not based on generative language models. For the robust model training, we manually collect 57 datasets from 5 tasks to make corpus for model pretraining. The data statistics for each IE task are listed in Table 2. To balance the number of examples in each task, set different maximum number of samples Nmax for each task dataset. If the number of instances in dataset is less than Nmax. we keep the original dataset unchanged and do not perform oversampling. For NER, RE, and EE tasks, we manually design set of instructions and randomly pick one them for each sample. MRC datasets some classification datasets have inborn questions, so the numbers of instruction are much higher than the others. For detailed statistics each dataset. please refer to Appendix C.
The schema labels are task ontologies for schema-guided extraction. This part consists of special token labels ([LM], [LR], and [LC]) and corresponding label texts. Among the special to- kens, [LM] denotes the label of mentions (or event types), [LR] denotes the label of relations (or argu- ment roles), and [LC] denotes the label of classes,
The text part is the input text that the model should extract information from. It is composed of a leading token ([TL], [TP] or [B]) and natural language sentence. If the leading token is [TL], the model should link labels from schema labels to spans in the text. While the [TP] token indicates the target spans are only in the text, and the model should extract information from the text without schema labels. In classification tasks, the model should not extract anything from the text part. So we use a special leading token [B] (background) to distinguish it from the extractive text.
3.2 Multi-slot Tuple and Multi-span Cyclic Graph
We formulate IE tasks as a unified multi-slot tuple extraction problem. As exemplified in Figure 2. in the RE task, the model is expected to extract a three-slot tuple: (relation, head entity, tail entity). Here, the tuple is (LReriend Jerry
With the above three parts, we can formulate extractive MRC, classification, and IE tasks into a

---

Task
#Dutaset
Mma
#Instruction
#Instance
Algorithm 1 MULTI-SPAN CYCLIC GRAPH DE-
NER
15
20,000
42
171,609
CODING
Cls
27
5,000
54,070
134,758
Input: Adjacency matrix A
RE
20,000
123,876
Output: A set of multi-slot tuples T
MRC
5
30,000
75,200
85,658
1: < 0
EE
All
40
2,898
2: A + A4AAi > mmegge consecutive and jump
Total
57
518,799
connections
Find forward chains C. from A
Table 2: Pretraining dataset statistics. Classifica- tion tasks contain multi-choice MRC datasets. ? MRC stands for both extractive QA and extractive MRC datasets.
for e e do
find legal paths with
tail-to-head connections if c meets the need in A' then
split into a tuple via A 7: T
Smith, Tom). The length of tuple slots could vary across tasks, so Mirror able to solve n-ary extrac- tion problems.
8:
end if
9: end for
10: return T
As shown in Figure and the top right of Fig- ure 3, we formulate multi-slot tuples into a unified multi-span eyclic graph, and regard labels as the leading tokens in schema labels. There are three types of connections in the graph: the consecutive connection, the jump connection, and the tail-to- head connection. The consecutive connection is adopted spans in the same entity. For an entity with multiple tokens, the consecutive connection connects from the first to the last. As shown Fig- ure 3, "Jerry" connects to "Smith". there is only one token in an entity, the consecutive connection is not used. The connection connects different slots in tuple. Schema labels and spans from texts are in different slots, so they connected in jump connections. For instance, the head and tail entities of relation triplet are in different slots, so they are connected in jump connections. The tail-to-head connection helps locate the graph boundaries. connects from the last token of the last slot to the
ence, we first find the forward chain (9,16.17,22) and then verify the chain with the tail-to-head con- nection (22 >9). After that, the multi-slot tuple is obtained with jump connections(9:16) and (17:22).
3.3 Model Structure
With the unified data interface and the multi-span cyclic graph, we propose a unified model struc- ture for IE tasks. For each token I; from the inputs, Mirror transforms it into a vector hy E R4h via BERT-style extractive pretrained lan- guage model (PLM). We use biaffine attention (Dozat and Manning, 2017) obtain the adja- cency matrix A of the multi-span eyclic graph. Mirror calculates the linking probability ply k [consecutive, jump, tail-to-head) between x; and 2 as Equation shows. The final A is obtained via thresholding (At = if p/J > 0.5 else 0).
first token of the first slot in tuple.
h, = FFNN, (h:), hj FFNN, (hz) $ P.j = sigmoid (h4
In practice, we convert the answer of each slot into text positions. For schema labels, we use the position of leading tags instead of literal strings. For text spans, the position is one-digit number if there is only
(1)
T Uhg/ Vdng/Van).
where hi hi E IRds UE Rddx3xxd is the train- able parameter, and 3 denotes consecutive, jump, and tail-to-head connections. FFNN is the feed- forward neural network with rotary positional em- bedding as introduced in Su et al. (2021). The FFNN comprises a linear transformation, GELU activation function (Hendrycks and Gimpel, 2023), and dropout (Srivastava et al., 2014).
character, otherwise the start and end positions are listed. For example, the 3-slot relation tuple (LRfriend of, Jerry Smi Smith, Tom)
will be converted into (9 16 17 22), where: denotes the jump connection, stands for the con- secutive connection, 9 the position of LRrriend of- 16 and 17 express Jerry Smith, and 22 is the posi- tion of Tom. There is also a tail-to-head connection from 22 to 9. The corresponding graph decoding algorithm is shown in Algorithm 1. During infer-
During training, we adopt the imbalance-class multi-labe categorical cross entropy (Su et al., 2022) as the loss function:

---

Mirror on CoNLL04 in the RE task, while Mirror reaches very close results or outperforms Deep- Struet on the other datasets. InstructUIE (1IB) demonstrates similar performance on NER datasets, while achieving high scores in RE (SciERC) and EE (ACE05-Tgg & Arg), surpassing other models by significant margin. Apart from these datasets, InstructUIE performs perforrs about the same UIE, USM, and Mirror.
[(i.j) log
E+Σen Ane
1+
+log
1+1 Σ
Apu
(2)
where Qney stands for negative samples (A
0),
and Opss denotes positive samples (A = 1).
4 Experiments
We provide ablation studies on Mirror with dif- ferent pretraining and fine-tuning strategies. Perfor- mance degrades if either pretraining or instruction fine-tuning is not performed. Mirror benefits from pretraining when utilizing instructions (w/ Inst.) and increases 0.66% scores on average. However, when instructions are discarded (w/o Inst.), pre- training (w/ PT) does not bring performance gain. Pretraining has been confirmed on UIE and USM to enhance model performances, and it is crucial to enable the zero-shot inference ability. However, based on the results from Table 3, we find that if Mirror is applied in one specific task with sufficient training resources, it may not need to perform the pretraining step (e.g., NYT dataset).
4.1 Experiment Setup
We utilize DeBERTaa-3-Large (He et al., 2021) as
the PLM. The biaffine size dg is 512 with a dropout rate of 0.3. The epoch number of pretraining is 3 with a learning rate of 2e-5. Please refer to Ap- pendix B for detailed hyper-param settings.
Datasets are processed following Lu et al. (2022) (13 IE datasets in Table 3, and 4 datasets in Table 5),
Li al. (2022) (CADEC in Table 4), Chia al.
(2022) (HyperRED Table 4), Lou et al. (2023) (7 zero-shot NER datasets in Table 6), Rajpurkar
et al. (2018) (SQuAD v2.0 in Table 7), and Wang et al. (2019) (7 GLUE datasets in Table 7). Data statistics and metrics are listed in Appendix C.
Besides the traditional TE tasks in Table 3, Mirror also supports multi-span discontinuous NER and n-ary hyper relation extraction as shown in Table 4. We provide Mirror (w/ PT, w/ Inst) and Mirror (w/o PT, w/o Inst.) results on CADEC according to their good performances on the IE tasks in Table 3. However, Mirror is less powerful than task-specific SOTA models. On the n-ary hyper relation ex- traction task, Mirror outperforms the task-specific model CubeRE and achieves new SOTA results. Ta- ble 4 indicates Mirror's compatibility with complex multi-span and n-ary extraction problems.
4.2 Baselines
We compare Mirror with inneration-based TANL (Paolini et al., 2021), DeepStruct (Wang al., 2022), UIE (Lu al., 2022), Instruc- tUE (Wang al., 2023), and extraction-barn based USM (Lou et al., 2023) in triplet-hased IE tasks. In the multi-span discontinuous NER task, we com- pare Mirror with task-specific BART-NER (Yan et al., 2021b) and W2NER (Li et al., 2022). The baseline system in hyper RE is CubeRE (Chia et al., 2022). As to MRC tasks, the baseline models are BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and DeBERTa-v3 (He et al., 2021).
The above facts indicate that Mirror has good compatibility across different IE problems, and we extend the universal IE system to complex multi- span and n-ary extraction tasks, which are not sup- ported by previous universal IE systems.
4.3 Main Results
Mirror performances on 13 IE benchmarks are pre- sented in Table 3. Compared with other baseline models, Mirror surpasses baseline models on some datasets in NER (ACE04), RE (ACE05, NYT),, and EE CEASIE-Trigger tasks. When compared to Iiracciionbbaedd USM, Mirror achieves compet- itive results most of tasks, while lagging in NER (ACE05), RE (CoNLL04), and EE (CASIE- Arg). Compared generation-based methods, Mir- ror outperforms TANL. across all datasets and sur- passes UIE in most datasets. When the model pa- rameter comes to 1OB, DeepStruct outperforms
4.4 Few-shot Results
Followed by Lu et al. (2022) and Lou et al. (2023), we analyze Mirror's few-shot ability on NER, RF, EE, and ABSA tasks. As shown in Table 5, Mirror (w/ PT, w/ Inst.) outperforms USM and achieves SOTA results on CoNLLO3, ACE05, and 16-res datasets. the RE task on CoNLLO4, the best model is USM, achieving an average score of 12, while Mirror is less effective with only 43.16 av-

---

Mirror w/ Inst.
Minror w/
Mirror wo PT
Mirror
Task
Datasets
TANL
UIE
DeepStruct
InstructUIE
USM
PT
w/ Inst.
w/ Inst.
w/o Inst.
ACE04
86.89
87.62
87.16
86.39
87.66
87.26
NER
ACE05
84.90
85.78
86.90
86.66
87.14
85,34
85.70
86.72
86.45
CoNLLO3
91.70
92.99
93.00
92.94
93.16
92.73
91.93
92.11
92.97
ACE05
63.70
66.06
66.80
67.88
67.86
67.86
64.88
69.02
CoNLL04
71.40
75,00
78.30
78.48
78.84
75.22
72.96
71.19
73.58
NYT
93.54
93.30
90.47
94.07
93.85
94.25
93,95
93.31
SciERC
36.53
45.15
37.36
36.89
37.12
36.66
40.50
ACH05-TTg
68.40
73.36
69.80
77.13
72.41
74.44
73.05
72.66
73.38
ACH05-Arg
47.60
54.79
56.20
72.94
55.83
55.88
54.73
56.51
57.87
EE
CASIE-Tgg
69.33
67.80
71.73
71.81
71.60
73.09
71.40
CASIE-Arg
61,30
63.53
63.26
61.27
61.04
60.44
58.87
14-res
74.52
77.26
75.06
74.24
76.05
75.89
14-lap
63.88
65.51
64.08
62.48
59.56
60.42
ABSA
15-res
67.15
69.86
66.40
63.61
60.26
67.41
16-res
75.07
78.25
74.24
75.40
73.13
77.46
Avg.
71.75
73.35
72.15
71.49
70.99
72.39
Table 3: Results on 13 IE benchmarks (ACE-Tgg and ACE-Arg are in the same dataset with different evaluation metrics). is abbreviation pretraining, and Inst. denotes task instruction.
P
R
F1
Task
Model
1-shot
5-shot
10-shot
Avg.
UIE
57.53
75.32
79.12
70.66
Discontinuous NER: CADEC
NER CoNLL03
USM
71.11
83,25
84.58
79.65
BART-NER
70.08
71.21
70.64
Mirror
76.49
82.45
84.69
81.21
W2NER
74.09
72.35
73.21
UIE
34.88
51.64
58.98
48.50
Mirror PT & Inst.
74.83
65.45
69.83
RE CoNLL04
USM
36.17
53,20
60.99
50.12
Mirror PT & Inst.
68.80
68.38
68.59
Mirror
26.29
47.42
55.77
43.16
UIE
42.37
53.07
54.35
49.93
N-ary Tuples: HyperRED
Event Trigger ACF05
USM
40.86
55.61
58.79
51.75
CubeRE
66.39
67.12
66.75
Mirror
47.77
57.90
59.16
54.94
Mirrorw/ PT & Inst.
71.29
62.46
66.58
CIE
14.56
31.20
35.19
Mirrorwio PT & Inst.
75.41
61.14
67.53
Event Arg ACF05
USM
19.01
36.69
42.48
32.73
Miror
23.18
37.74
39.20
33.38
Table 4: Results multi-span and n-ary information extraction tasks. Tgg and Arg in Event Extraction refer to Trigger (Event Detection) and Argument (Event Ar- gument Extraction), respectively.
UIE
23.04
42.67
53.28
39.66
ABSA 16-res
USM
30.81
52.06
58.29
47.05
Mirror
36.21
51.65
58.59
48.82
Table Few-shot results on tasks. These datasets not included in the pretraining phase of Mirror.
erage scores. Among all the four tasks, NER may be relatively easier for the model deal with. The 10-shot NER score of Mirror is 84.69, while the fine-tuned Mirror on the full dataset gets an Fl score of 92.73. The gaps on other datasets between 10-shot and fully fine-tuned results are larger, indi- cating the task difficulties.
make predictions directly. The results show that Mirror outperforms USM by large margin (an av- erage Fl score of 9.44), and it is very competitive with InstructUIE, (FlanT5- 1B). Considering the model scale, Mirror is surprisingly good at zero- shot NER tasks. However, ChatGPT is very pow- erful in the zero-shot NER task and achieves abso- lute SOTA performance. Except for simple model scaling, we may need to collect a more diverse pretraining corpus for better results.
4.5 Zero-shot Results
Table shows the zero-shot performances on NER datasets. These datasets are not included in pretraining, and we use the pretrained Mirror to

---

Model
Movic
Restaurant
A1
Literature
Music
Politics
Science
Avg.
Davinci
0.84
2.94
2.97
9.87
13.83
18.42
10.04
8.42
ChatGPT
41.00
37.76
54.40
54.07
61.24
59.12
63.00
52.94
USM
37.73
14.73
28.18
56.00
44.93
36.10
44.09
37.39
InstructUIE
63.00
20.99
49.00
47.21
53.61
48.15
49.30
47.32
MirroTdirect 39.20
16.32
45.23
46.32
58.61
67.30
54.84
46.83
Table 6: Zero-shot results on 7 NER datasets. Results of Davinci and ChatGPT are derived from Wang et al. (2023).
Mirr is pretrained Mirror Inst. while these datasets are not included the pretraining phase.
SQuAD 2.0 (EM/F1)
CoLA (Mec)
QQP (Acc)
MNLI
SST-2
QNLI (Aec)
RTE (Acc)
MRPC (Acc)
Model
(Acc)
(Ace)
BERT-large
79.0 / 81.8
60.6
91.3
93.2
92.3
70.4
84.1
RoBERTa-large
86.5 / 89,4
68.0
92.2
90.2
96.4
93.9
86.6
88.8
DeBERTa v3-large
89.0/
75.3
93.0
91.9
96.9
96.0
92.7
92.2
Mirrordicet
40.4 67.4
63.9
84.8
85.9
93.6
91.6
85.9
89.2
Table 7: Results MRC and classification tasks. We list Mirror performance on SQuAD 2.0 development and GLUE development sets. Baseline results are derived from He et al. (2021). Because SQuAD v2 and GLUE datasets are included Mirror pretraining for 3 epochs, we direct make inferences with the pretrained model (noted Mirrorarect the same model used in zero-shot NER), and do not perform further fine-tuning, while other baselines are fine-tuned with full dataset on every single task.
4.6 Results MRC and Classification
w/ Inst.
Label Type
P
R
F1
To show the model compatibility on extractive MRC and classification tasks, we conduct experi- ments SQuAD v2 and GLUE language under- standing benchmarks. The experimental results are demonstrated in Table 7. Comparing the re- sults in He et al. (2021), we do not report per- formance on the STS-B dataset since Mirror's s ex- traction paradigm does not support the regression task. Although Mirrordirset does not perform full fine-tuning like the other systems, it still produces competitive results. It outperforms BERT-large CoLA and SST-2 and better than RoBERTa-large MRPC The results indicate that Mirror is capa- ble of various tasks besides information extraction. We leave full fine-tuning for future work to improve Mirror performances.
Tag
92.12
92.10
92.11
Content
91.89
92.71
92.30
Tag
92.79
93.15
92.97
x
Content
91.91
92.58
92.25
Table Results on different label span types. This experiment is conducted on the CoNLLO3 dataset w/o pretraining.
The results are demonstrated in Table We find that the label type does not bring too many differ- ences. In Mirror w/ Inst., the literal content string is slightly better than bare tags with only 0.19 F1 score advantage. While in Mirror w/o Inst., the tag- based method surpasses the content-based method by 0.72 Fl scores. Similar Baldini Soares et al. (2019), these results show that although the label tag is a simple token without pretraining, it does not affect the model's ability to incorporate features from global and local contexts.
4.7 Analysis on Label Span Types
Mirror adopts the leading token in schema labels ([LC), [LM] and [LR]) as the label span that con- nects to target text spans. To analyze the effect of different label span types, conduct experiments change the leading token into literal content string. In other words, NER task that extract person entities, we compare the effect of [LM] to- ken and person string as the label span.
4.8 Analysis on Pretraining Datasets
Traditionally, the classification task is different from the extraction task as they optimize differ- ent objectives. Since Mirror unifies the two tasks

---

NER (FI)
RE (FI)
MRC (FI)
Cls (Acc)
Pretrain Data
CoNLL03
NYT
SQuAD v2
MRPC
Average
All
66.91
69.67
67.39
89.22
73.30
w/o Cls
66.82
57.03
67.14
0.00
47.75
w/o IE
0.00
0.00
68.77
89.22
39.50
w/o Span
66.76
54.76
0.00
87.50
52.26
Table 9: Ablation study on the pretraining data. We evaluate the pretrained Mirroraneet without further fine-tuning.
into one framework, it is interesting to find how they affect each other the pretraining phase. We provide an ablation study on different types of pre- training data in Table 9. It is surprising that pre- training on classification datasets help improve the extraction tasks, and relation extraction is the most affected one. This may be due to the similarity between relation labels and semantic class labels. It is also interesting that span-based datasets (e.g. MRC datasets) are beneficial the classification task (87.50 - 89.22). Overall, all kinds of the pre- training datasets bring greater mutual benefits and improve the model performance.
tasks. We manually collect 57 datasets for pre- training and conduct experiments on 30 datasets across 8 tasks. The experimental results show good compatibility, and Mirror achieves competitive per- formances with waae-offtheeaae systems.
Limitations
Content input length: Due to the backbone De- BERTa model constraint, the maximal sequence length is 512 and can hardly extend to longer texts. This limits the exploration of tasks with many schema labels and document-level IE.
Multi-turn result modification: Mirror predicts the multi-span cyclic graph in paralleled non- autoregressive style. Although it efficient training and inference, it may lack global history knowledge from previous answers.
4.9 Analysis on Inference Speed
We conduct speed tests on the CoNLLO3's valida- tion set with one NVIDIA V100 GPU under the same environment. The results are presented in Table 10. Compared the popular generative T5- large UIE model (Lu al., 2022), our model is up to 32.61 times faster when inference, and the advantage grows when increasing the batch size from to
Data format unification: There are many IE tasks, and the formats may vary lot. Although the cur- rent unified data interface supports most common tasks, it may not be practical for some tasks.
Lack of large-scale event datasets for pretraining: There are many NER and RE datasets. However, there are few large-scale event extraction corpus with high diversity in domains and schemas, which may limit the model performance on event-relevant information extraction tasks.
batch size
UIE
Mirror
Speed-Up
1
0.21
5.68
27.24
2
0.32
10.56
32.61
Table 10: Inference speed (instances per second) test on CoNLLO3 validation set.
cknowledgmentss
This work supported by the National Natural Sci- ence Foundation of China (Grant No. 61936010) and Provincial Key Laboratory for Computer Infor- mation Processing Technology, Soochow Univer- sity. This work is also supported by Collaborative Innovation Center of Novel Software Technology and Industrialization, the Priority Academic Pro- gram Development of Jiangsu Higher Education In- stitutions, and the joint research project of Huawei Cloud and Soochow University. We would also like to thank the anonymous reviewers for their insightful and valuable comments.
5 Conclusion
We propose Mirror, schema-guided framework for universal information extraction. Mirror trans- forms IE tasks into a unified multi-slot tuple extrac- tion problem and introduces the multi-span cyclic graph to represent such structures. Due to the flex- ible design, Mirror is capable of multi-span and n-ary extraction tasks. Compared to previous sys- tems, Mirror supports not only complex informa- tion extraction but also MRC and classification

---

References
General language model pretraining with autoregres- sive blank infilling. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 320-335, Dublin, Ireland. Association for Computational Lin- guistics.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Pro- ceedings of the 57th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 2895- 2905, Florence, Italy, Association for Computational Linguistics.
Hao Fei. Shengqiong Wu, Jingye Li, Bobo Li, Fei Li. Libo Qin, Meishan Zhang, Min Zhang, and Tat-Seng Chua. 2022. Lasuie: Unifying information extrac- tion with latent adaptive structure- generative language model. In NeurIPS.
Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. Hacred: A large- scale relation extraction dataset toward hard cases in practical applications. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 28 19-2831.
Ralph Grishman. 2019.
Twenty-five years of infor- mation extraction. Natural Language Engineering, 25(6))677-692.
Yingjie Gu, Xiaoye Qu, Zhefeng Wang, Yi Zheng, Baox- ing Huai, and Nicholas Jing Yuan. 2022. Delving deep into regularity: a simple but effective method for chinese named entity recognition. arXiv preprint 2.Xiv:2204.0555
Yew Ken Chia, Lidong Bing, Sharifah Mahani Alju- nied, Luo Si, and Soujanya Poria, 2022. A dataset for hyper-relational extraction and cube-filling ap- proach. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10133, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Peng Hao, Wang Xiaozhi. Yao Feng, Zeng Kaisheng, Hou Lei, Li Juanzi, Liu Zhiyuan, and Shen Weixing. 2023. The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation. ArX1v:2006.06918 [cs].
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang. Andrew M. Dai, Hongkun Yu, Slav Petrov, Fd H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts. Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction finetuuee language models. CoRR, 0to//210.114116
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with gradient-disentangler embedding shar- ing. CORR, abs/2111.099 11.0944..
Dan Hendrycks and Kevin Gimpel. 2023. Gaussian Error Linear Units (GELUs). ArXiv:1606.08415 [cs].
Martin Josifoski, Nicola De Cao, Maxime Peyrard, Fabio Petroni, and Robert West. 2022. GenlE: Gen- erative information extraction. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies. pages Seattle, United States. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedingy of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Sarvnaz Karimi, Alejandro Metke-Jimenez, Madonna Kemp, and Chen Wang. 2015. Cadec: A corpus of ad- verse drug event annotations. J. Biomed. Informatics. 55:73-81. 81.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, IWP@ IJCNLP 2005, Jeju Island, Korea, October 2005, 2005. Asian Federation of Nat- ural Language Processing.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequeen to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics, pages 787 1-8800 Online, Association for Computa- tional I. Linguistics.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency pars- ing. In 5th International Conference on Learning Representations, ICIR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. Open- Review.net.
Jingye Li, Hao Fei, Jiang Liu, Shengqiong Wu, Meishan Zhang, Chong Teng, Donghong Ji, and Fei Li. 2022. Unified named entity recognition as word-word word rela- tion classification. In Thirty-Sixth AAAI Conference
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLM:

---

on Artificial Intelligence, AAAI 2022, Thirty-Fourt Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances Artificial Intelligence, EAAI 2022 Virtual Event, February 22- March 1, 2022, pages 10965-10973. AAAT Press.
Cícero Nogueira dos Santos, Bing Xiang, and Ste- fano Soatto. 2021. Structured prediction as transla- tion between augmented natural languages. In 9th International Conference on Learning Representa- tions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Ying Lin, Heng Ji, Fei Huang. and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7999-8009, Online, Association for Computational Linguistics.
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Moham- mad AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clereq. Véronique Hoste, Mariaana Apidianaki, Xavier Tannier, Na- talia Loukachevitch, Evgeniy Kotelnikov, Nuria Bel, Salud María Jiménez- Linénez-Zafra, and Gülşen Eryigit. 2016. SemFval-2016 task 5: Aspect based sentiment analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2010) pages 19-30, San Diego, California. Association for Computational Linguistics.
Jingjin Liu, Panupong Pasupat, Scott Cyphers, and James R. Glass, 2013. Asgard: A portable archi- tecture for multilingual dialogue systems. In IEFF International Conference on Acoustics, Speech and Signal Processing, ICASSP 2013, Vancouver, BC, Canada, May 26-31, 2013, pages 8386-8390. IEEE
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and lon Androutsopoulos. 2015, SemEval-2015. task 12: Aspect based sentiment anal- ysis. In Proceedings of the 9th International Work- shop on Semantic Evaluation (SemEval 2015), pages 486 495, Denver, Colorado. Association for Compu- tational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CORR, abs/1907.11692.
Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pas- cale Fung. 2021. Crossner: Evaluating cross-domain named entity recognition. In Fiirty-Fifth AAAI Con- ference on Artificial Intelligence, AAAI 2021, Thirty- Third Conference on Innovative Applications of Arti- ficial Intelligence, IAAI 2021, The Eleventh Sympo- sium on Educational Advances in Artificial Inteli- gence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 11342-13460 AAAI Press.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos. Har- ris Papageorgion, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (Se- mEval 2014), pages 27-35, Dublin, Ireland. Associa- tion for Computational Linguistics.
Xiaoye Qu Yingjie Gu, Qingrong Xia, Zechang Li. Zhefeng Wang, and Baoxing Huai. 2023a. A sur- vey on arabic named entity recognition: Past, re- cent advances, and future trends. arXiv preprint arXiv:2302.03512
Jie Lou, Yaojie Lu, Dai Dai, Wei Jia, Hongyu Lin, Xi- anpei Han, Le Sun, and Hua Wu. 2023. Universal information extraction as unified semantic matching. CoRR, abs/2301.00. .03282.
Xiaoye Qu Jun Zeng, Daizong Liu, Zhefeng Wang. Baoxing Huai, and Pan Zhou. 2023b. Distantly- supervised named entity recognition with adaptive teacher learning and fine reee-graindd student ensemble. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13501-135009
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun. and Hua Wu. 2022. Uni- fied structure generation for universal information extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 5755-5772 Dublin, Ireland. Association for Computational Linguisties.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-1404:
Yi Luan, Luheng He, Mari Ostendorf, and Hannanch Hajishirzi. 2018. Multi-task identification entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 5219-3232. Brussels. Belgium. Association for Computational Linguisties.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't 't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Anmual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Alexis Mitchell, Stephanie Strassel, Shudong Huang, and Ramez Zakhary. 2005. Ace 2004 multilingual training corpus.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai,
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without

---

labeled text. In Machine Learning and Knowledge Discovery in Databases, pages 148-163, Berlin, Hei- delberg. Springer Berlin Heidelberg.
Christopher Walker, Stephanie Strassel, Julie Medero. and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus.
Dan Roth and Wen-tau Yih. 2004. A linear program- ming formulation for global inference in natural lan- guage tasks. In Proceedings of the Fighth Confer- ence on Computational Natural Language Learn- ing (CoNLL-2004) at HLT-NAACL 2004, pages 1-8, Boston, Massachusetts, USA. Association for Com- putational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman, 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In 7th In- ternational Conference on Learning Repreventations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Taneeya Satyapanich, Francis Ferraro, and Tim Finin. 2020. Casie: Extracting cybersecurity event infor- mation from text. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 34, pages
Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong. Jie Tang, and Dawn Song. 2022. DeepStruct: Pre- training of language models for structure prediction. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 803 823,, Dublin, Ireland. Association for Computational Linguisties.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep mod- els for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. EMNLP 2013, 18-21 October 2013, Grand Hyaft Seattle, Seatlee Washington, USA, A meeting of SIG- DAT, a Special Interest Group of the ACL, pages 1631-1642. ACL.
Xiao Wang. Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang. Tao Gui. Jihua Kang, Jingsheng Yang, Siyuan Li, and Chunsai Du 2023. InstructUlE: Multi-task Instruction Tuning for Unified Informa- tion Extraction. ArXiv:2304.0808 [es].
Yucheng Wang, Bowen Yu, Hongsong Zhu, Tingwen Liu, Nan Yu, and Limin Sun. 2021. Discontinuous named entity recognition as maximal clique discov- ery. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 764-774, Online. Association for Computational Lin- guistics.
Nitish Srivastava, Geoffrey F. Hinton, Alex Krizhevsky, Πyy Sutskever, and Ruslan Salakhutdinov. 2014 Dropout: simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929- 1958.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding. CoRR, 0oss/104.09864,
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Trans, Assoc. Comput. Linguistics, 7:625-641.
Jianlin Su, Ahmed Murtadha, Shengfeng Pan, Jing Hou, Jun Sun, Wanwei Huang, Bo Wen, and Yun- feng Liu. 2022. Global Pointer: Novel Efficient Span-based Approach for Named Entity Recognition. ArXiv:2208. 03054 [cs].
Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2018. A broad coveraae challenge corpus for sentence understanding through inference. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL- IITTT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Papers), pages 1112-1122. Association for Computational Linguisties.
M. Therasa and G. Mathivanan. 2022. Survey of ma- chine reading comprehension models and its evalua- tion metrics. 2022 6th International Conference on Computing Methodologies and Communication (ICCMC), pages 1006-1013
Erik F. Tjong Kim Sang and Fien De Meulder. 2003, Introduction to the con1l-2003 shared task: Language- independent named entity recognition. In Proceed- ings of CoNLL-2003, pages 142-147. Edmonton, Canada.
Hang Yan, Junqi Dai, Tuo Ji, Xipeng Qiu, and Zheng Zhang. 202 A unified generative framework for aspect-based sentiment analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference Natural Language Processing (Volume I: Long Papers). pages 2416-2429, Online. Association for Computational Linguistics.
David Wadden, Ulme Wennberg, Yi Luan, and Han- naneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- pages 5784-
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang. and Xipeng Qiu. 20216. A unified generative framework for various NER subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing
guage Processing (EMNLP-IJCNH) 5789, Hong Kong, China. Association for Computa- tional Linguistics.

---

(Volume 1:
Long Papers), pages 5808-5822, Online. Association for Computational Linguistics.
RE
ACE05
CoNLL04
NYT
SciERC
Mirror TANL
100.00
100.00
100.00
100.00
96.91
96.95
100.00
99.64
Ping Yang, Junjie Wang, Ruyi Gan, Xinyu Zhu, Lin Zhang, Ziwei Wu, Xinyu Gao, Jiaxing Zhang, and Tetsuya Sakai. 2022. Zero-shot learners for natural language understanding via unified multiple choice perspective. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 7042-7055, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Table 12: Upper bound of relation extraction with Mir- and TANL position indexing strategies.
B Hyper-parameter Settings
Table 13 shows the hyper-parametee parameters in our exper- iments. For few-shot experiments, we follow Lu et al. (2022) and generate 1-, 5-, 10-shot data with 5 seeds.
Tong Zhu, Xiaoye Qu, Wenliang Chen, Zhefeng Wang, Baoxing Huai, Nicholas Yuan, and Min Zhang, 2022. Efficient document-level event extraction via pseudo- trigger-aware pruned complete graph. Proceed- ings of the Phirty-FFrst International Joint Confer- ence on Artificial Intelligence, IJCAI-22, pages 4552- 4558. International Joint Conferences on Artificial Intelligence Organization. Main Track.
Item
Setting
warmup proportion
0.1
pretraining epochs
3
fine-tuning epochs
20
A Comparisons on Information Indexing Strategies
fine-tuning epoch patience
3
few-shot epochs
200
few-shot epoch patience
10
UIE (Lu et al., 2022) provides the extracted infor- mation's positions based on string matching. How- ever, this strategy is not accurate and contains am- biguities. To investigate the matching accuracy, we take the NER task as an example and use golden en- tity strings to calculate the upper bound Fl scores of different UIE string matching strategies. The table below shows that the upper bounds are quite low on the datasets (<30%). This indicates that ob- taining positions via string matching ineffective and has serious ambiguity problems.
batch size
8
PLM learning rate
2e-5
PLM weight decay
0.1
others learning rate
le-4
max gradient norm
1.0
dp dy dropout
1024
512
0.3
Table 13: Hyper-parameter settings.
C Dataset Statistics
NER
ACE04
ACEOS
CoNLLO3
This section contains detailed statistics for pretrain- ing datasets and fine-tuning datasets. Pretraining data statistics are listed in Table 14, 16, 17, 18 and
Mirror 100.00
100.00
100.00
UlE-first
13.31
14.51
27.67
UTE-longer-first
14.55
16.21
27.97
15. For the sampling number Naxx of each kind of dataset, please refer to Table 2. When collecting pretraining data, refer the datasets mentioned in Therasa and Mathivanan (2022) and Yang et al. (2022). Downstream data statistics are listed Table 19. We also provide direct inference results with the pretrained Mirror model in Table 19,
Table 11:
Upper bound of different string matching strategies on NER.
TANL (Paolini et al., 2021) can provides exact positions in NER since it generates the enclosure tags. However, it still faces the ambiguity problem when two entities have the same string in joint entity relation extraction because the tail entity is generated text corresponding an enclosed head entity (refer section in the TANL paper). We also calculate the upper bound scores relation extraction in a TANL manner, and the results show it does not ideally generate perfect positions.
D
Case Study
We provide some interesting cases across different tasks with the pretrained Mirror w/ Inst. manu- ally evaluate its versatility on various tasks under zero-shot settings. The model inputs & outputs are presented in Table 20.

---

Name
#Instruction
#Instance
Name
#lnstruction
#instance
AnatEM
42
5,861
bc2gm
42
12,500
news
5
5,000
be4chemd
42
20,000
ANLI
29
15,000
bc5cdr
4,560 42
ARC
3,361
3,370
Broad Tweet_Corpp
42
5,334
CoLA
43
5,000
FabNER
42
9,435
CosmosQA
4,483
5,000
Findcee
42
20,000
cos_e
5,000
5,000
GENIA
42
15,023
dbpedia
5,000
HarveyNER
3,967 42
DREAM
3,842
5,000
MultiNERD
42
20,000
hellaswag
20
5,000
NCBIdicase
42
5,432
IMDB
26
5,000
ontoNotes5
42
20,000
MedQA
5,000
5,000
TweetNER7
42
7,103
MNLI
29
5,000
WikiANN en
42
20,000
MRPC
40
3,668
WNUT-16
42
2,394
MultiRC
4,999
5,000
OpenBookQA
4,835
4,957
Total
42
171,609
QASC
4,832
5,000
QNLI
31
5,000
Table 16: Pretraining data statistics on NER. The maxi-
mal sampling number Nmax for each dataset is 20,000.
QQP
40
5,000
RACE
4,482
5,000
RACE-C
4,782
5,000
ReClor
3,368
4,638
Name
#Instruction
#Instance
RTE
29
2,490
ADE_corpus
ADE corpus
3,417
SciQ
4,989
5,000
FewRel
20,000
SNLI
29
5,000
GIDS
8,526
SST-2
26
5,000
kbp37
15,807
Winogrande
20
5,000
New-York-Timenn
20,000
WNLI
31
635
NYTHHHRL
20,000
Total
54,070
134,758
semeval
8,000
WebNLG
5,019
Table 14: Pretraining data statisties on classification. The maximal sampling number Nmax: for each dataset is 5,000. * ANLI contains 3 subsets, so the total number is greater than 5,000.
Wiki-ZSL
23,107
Total
123,876
Table 17: Pretraining data statistics on RE. The maximal sampling number N for each dataset is 20,000.
Name
#Instruction
#Instance
Name
#lnstruction
#Instance
BiPaR
11,524
11,668
ms fos_marco__vv v2.1
20,000
20,000
PHEE
40
2,898
newsqa
19,659
20,000
Total
40
2,898
squad
19,998
20,000
SubjQA
4,060
13,990
Table 15: Pretraining data statistics on Due scarcity of EE datasets, we sample all the instances
Total
75,220
85,658
(N Max = 00).
Table 18: Pretraining data statistics MRC. The maxi- mal sampling number Nmax for each dataset is 20,000.

---

Task
Dataset
Ciaaion
Metne
#Train
#Dev
*Test
Included in PT
Minorzaca
ACF34
Mitchell
Faiily Fl
6,502
NER
ACE3S
Walker (2006)
Entit Micro Fl.
7,299
971
1,000
X
18.70
CONLEAS
Tjung Kim Sung and De Meulder (20033)
Earity Micro Fl
14,041
3,250
3,433
X
66.91
Triplet
2,050
RE
CUNTJ.44
NYT
Ruth and Yik, (20094)
Riedel at al. (2010)
Triplet Micro
Tripla Micro
$6,196
922
5,000
231
5,000
288
X
8
69.67
1.40
Triplet
1,861
ACED5
Walker et al. (2006)
Trigger & Argument Micro Fl
19,216
901
676
X
3,99/0.00
EE
Sayspunichet (2020)
Trigger Aquinon Micra
11,189
2,150.00
14-res
Poniki (2014)
Triphet
1,266
X
14-lup
Poniki (20114)
Triplet Micro F1
906
219
333
01.00
ABSA
(2015)
Triplet
16-rs.
Puniiki et al. (2016)
Tripla Miem
857
210
336
0.00
Discunimons NER
CADEC
Karimi et all (2015)
Enkity Micro Fl
5,540
1,097
1,160
52.34
Hyper
HyperRED
(2022)
Tuple F1
39,840
1,000
Movie
Lis et al. (2013)
Enkit Micro Fl
9,774
2,412
2,12
X
39.24
Farily Fl
1,520
al. (2021)
Fatity
100
45.91
Zero-shot NER
Literature
Liu et al. (2021)
Earity Micro Fl
100
400
416
46.77
(2021)
Eatity
Poliic
Lin st al. (2021)
Fatily Micno E1
199
540
650
67:27
Science
Lin al. (2001)
Enkity Micro Fl
200
450
543
$4.42
SQUAD v2.U
Rapurtar (2018)
40.35/67.39
CoLA
Warstodk et (2019)
Matthew's 's Correlation Coefficiont
8,551
527
63.91
Wang
Accuracy
363,846
40,430
84. 84.84
Williams et (2018)
Accuray
392,702
9,815
85
Clessihicsiio
SST-2
Socher al. (2013)
Accuracy
67,350
873
93.58
Accuracy
RIE
Wang et al. (2019)
Accuracy
2,490
277
85
MRFC
Dolan and Brockett (2005)
Accuracy
3,668
408
1,725
89.23
Table 19: Data statistics on downstream tasks. Included in PT stands for whether the dataset included the data pretraining corpus. Mirroraieea is the model trained on the pretraining corpus.
Classification (Multi-hhoic MRC)
Input
[I] Mirror Mirror on the wall, who's the fairest of them all? [LC] Evil Queen
[LC] Snow White
Output [LC]snow White
Extractive MRC
Input
[1] Mirror Mirror on the wall, who's the fairest of them all? [TP] Evil Queen is jealous of Snow White's beauty.
Output
Snow White
Named Entity Extraction
Input
[I] Mirror Mirror, please help me extract all the model names. [LM] model name [TL] LLaMA and OPT are open-sourced large language models.
Output
[LM]LLaMA, [LM]OPT
Relation Extraction
Input
[I] Mirror Mirror, please help me extract the entity relationship triplet. [LR] break up [TL] The drama surrounding the high- high-profile divorce between Holly- wood actors Johnny Depp and Amber Heard appears be over the couple reportedly reached amicable settlement.
Output ([LR]break up- Amber Heard, Johnny Depp)
Table 20: Case results obtained by the pretrained Mirror w/ Inst. The name of our proposed Mirror is borrowed from the magic mirror Snow White and the Seven Dwarfs. We hope to build a universal model that can help more people solve more problems.

---

Mirror
Mirror can help you deal with wide range of Natural Language Understanding and Information Extraction tasks.
Instruction
Mirror Mirror, please help me extract the entity relationship triplet.
Schema Labels
Split with for multiple inputs
For entities, relations or classification, input ("ent/rellccl ["cls1", "type2"]]
For events and hyper relations, input ("type": ["role1", "role2"]]
("rel": ["break up"]}
Text
The drama surrounding the high-profile divorce between Hollywood actors Johnny Depp and Amber Hea
Background
Reset
Clear Output
Ask Mirror
Output
Item
Predicted
rel
[["break up", "AAmer Heard", [33, 3, [ ("Johnny Depp", [00, ]]]]
Made by Mirror Team w/
Figure 4: Mirror toolkit demonstration. The predicted relation label is converted to label string break up. The positions shown in the predicted results are counted by tokens, so they do not match the input string characters. You can find the pretrained model weights and demo code in the repository and deploy it on your machine: https: //gi thub, com/Spicol 97 /7/Mrror


Reviews:
[{"review_id": "2l0HbrgcrZ", "reviewer": "Reviewer_z1TA", "summary": "", "questions": "* The authors acknowledge certain limitations of the model, particularly its scalability concerns with respect to handling long text or document-level inputs due to the transformer's context size limit. The authors should also point out that the model would not scale well in a setup where the number entity types and relations types are very large (especially for end-to-end RE).\n* Figure 3 caption should be more detailed, where each step is clearly described\n* The complexity of the decoding algorithm (ALG 1) should be clearly stated. What is the speed-up of this model compared to autoregressive UIE models ?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough comments and constructive suggestions! In the following, your questions are first stated and then followed by our responses.\n\n**Q: Due to the pertaining on multiple labelled datasets, the comparison with other UIE approaches is not really fair.**\n\nIt is difficult to pretrain other baseline models with our dataset since they are not designed to cover all the tasks, especially the discontinuous NER, n-ary RE, MRC, and classification tasks. Furthermore, as other models' pretraining datasets are not fully open-sourced, it is hard to pretrain Mirror with those data. Compared to other pretraining datasets, as shown in the table below, Mirror applies less pretraining data while supporting more tasks. While we acknowledge that there might be variations in performance based on pretraining data, we aimed to highlight the versatility and effectiveness of our proposed Mirror framework across a wide range of tasks. We will release our datasets to help readers reproduce our results.\n\n| Methods                | Pre-trainig tasks                                                                                                      | #Total Instance | #NER Instance | #RE Instance | #MRC Instance |\n|------------------------|------------------------------------------------------------------------------------------------------------------------|----------------:|--------------:|-------------:|--------------:|\n| UIE (Lu et al., 2022)  | Distantly-supervised NER and RE; ConceptNet and Wikidata for structure decoding; Causal language modeling on Wikipedia |         Unknown |       Unknown |      Unknown |          None |\n| USM (Lou et al., 2023) | NER, RE, MRC                                                                                                           |            611k |           60k |         356k |          195k |\n| Mirror (ours)          | NER, RE, EE, Cls, MRC                                                                                                  |            519k |          172k |         124k |           86k |\n\n**Q: The authors should also point out that the model would not scale well in a setup where the number of entity types and relations types are very large (especially for end-to-end RE).**\n\nThank you for the suggestion and emphasizing the context limitation concerns we mentioned in the paper. Actually, all schema-guided end-to-end IE systems suffer from context limitation when scaling the number of labels. We agree that the scalability limitations are important to consider in future work, especially in scenarios involving long texts or document-level inputs, as well as tasks with a large number of entity and relation types. We will explicitly mention these limitations in the revised version.\n\n**Q: Figure 3 caption should be more detailed, where each step is clearly described.**\n\nThanks for your suggestion. We will add the following explanation in the revised version: \"Mirror first constructs inputs for each task, then utilizes a pretrained language model to predict the adjacency matrix via the biaffine attention. After that, final results are decoded from the adjacency matrix directly.\"\n\n**Q: The complexity of the decoding algorithm (ALG 1) should be clearly stated. What is the speed-up of this model compared to autoregressive UIE models?**\n\nCompared to the popular UIE model (Lu et al., 2022), our model is up to 32.61 times faster when inference, and the advantage still holds when the batch size increases. The speed tests are conducted with one NVIDIA V100 under the same hardware and software environment.\n\n| batch size | UIE (instances/second) | Mirror (instances/second) | Speed-up |\n|-----------:|-----------------------:|--------------------------:|---------:|\n|          1 |                   0.21 |                      5.68 |    27.24 |\n|          2 |                   0.32 |                     10.56 |    32.61 |\n\n**Q: Could reproduce the results with some difficulty.**\n\nOur hyper-parameter settings are available in section 4.1 and Appendix A. Meanwhile, all our pretraining datasets are publicly available, and the statistics are provided in Appendix B. We will open-source all the code and pretraining datasets for easily reproducing our results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The idea is interesting, the results are strong, and the experiments are well executed with thorough analysis.\n* It is nice to see that instruction tuning is applied to bidirectional transformers, allowing prediction in a single shot in contrast to autoregressive decoding. The model also obtain competitive result under few-shot and zero-shot setting.", "reasons_to_reject": "* Due to the pertaining on multiple labelled datasets, the comparaison with other UIE approaches is not really fair. I expect that other UIE baselines would perform similarly (or better) if pertained using the instruction same dataset.\n", "questions_to_authors": "* The authors acknowledge certain limitations of the model, particularly its scalability concerns with respect to handling long text or document-level inputs due to the transformer's context size limit. The authors should also point out that the model would not scale well in a setup where the number entity types and relations types are very large (especially for end-to-end RE).\n* Figure 3 caption should be more detailed, where each step is clearly described\n* The complexity of the decoding algorithm (ALG 1) should be clearly stated. What is the speed-up of this model compared to autoregressive UIE models ?\n", "paper_topic_and_main_contributions": "This paper introduces a new universal information extraction (UIE) model called Mirror. Unlike autoregressive UIE, this paper proposes a paradigm that treats IE problems as multi-span cyclic graphs extraction. The model is evaluated on a wide variety of task including joint IE, entity extraction, classification and MRC.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VW8Uh914gp", "reviewer": "Reviewer_aSvu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback! In the following, your questions are first stated and then followed by our responses.\n\n**Q: The performance is not strong enough. I think the proposed method significantly underperforms USM (Table 3)**\n- Mirror presents competitive finetuing performance with USM. As shown in Table 3, Mirror outperforms USM on five datasets (i.e., NER-ACE04, RE-ACE05, RE-NYT, RE-SciERC, EE-CASIE-Trigger), and the average result for Mirror is only 0.96%, slightly lower than USM.\n- Mirror significantly surpasses USM on few-shot and zero-shot settings. As presented in Table 5, Mirror outperforms USM on few-shot NER, Event Extraction, and ABSA tasks, and surpasses USM with 9.44% F1 on the zero-shot NER, as illustrated in Table 6.\n- Mirror supports more tasks than USM, such as discontinuous NER, n-ary RE, MRC, and classification. \n\n**Q: Although the paper claims that the proposed method can conduct MRC and classification tasks, it underperforms direct fine-tuning, so I think this claim is trivial.**\n\nSince part of the MRC and classification datasets are included in the pre-training phase, we only report Mirror-direct in Table 7 without fully fine-tuning. After we fully fine-tuning Mirror for 5 epochs (Mirror-FT), we found the performance similar to RoBERTa-large. We will conduct more experiments on MRC and classification tasks in the revised version, addressing your concerns more comprehensively.\n\n|               | CoLA (Mcc) | QQP (Acc) | QNLI (Acc) | RTE (Acc) |\n|---------------|-----------:|----------:|-----------:|----------:|\n| RoBERTa-large |      68.0  |     92.2  |      93.9  |     86.6  |\n| Mirror-direct |      63.9  |     84.8  |      91.6  |     85.9  |\n| Mirror-FT     |      70.2  |     91.5  |      93.6  |     86.6  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. well-designed graph-based IE decoding method.\n2. well-formed layout.\n3. The motivation is clear and persuasive in a certain level.", "reasons_to_reject": "1. The performance is not strong enough\n2. I think the proposed methd significantly underperforms USM (Table 3)\n3. Although the paper claims that the proposed method can conduct MRC and classification tasks, it underperforms direct fine-tuning, so I think this claim is trivial.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes, Mirror, a framework that can handle complex multi-span extraction, n-ary extraction, machine reading comprehension (MRC). It formulates IE tasks as a unified multi-slot tuple extraction problem and transform those tuples into multi-span cyclic graphs. This graph structure is rather flexible and scalable. It can be applied to not only complex IE tasks but also MRC tasks. Extensive experiments on 30 datasets from 8 tasks, and the results show that our model achieves competitive results under few-shot and zero-shot settings.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "T6ankJ9BnB", "reviewer": "Reviewer_59YZ", "summary": "", "questions": "1.\tWhy treat the MRC task's question as an instruction, rather than a label schema?  USM/MRC-EE treat the MRC task's question as a special type of entity/event, and their experiments have proven that this can enhance the entity/event argument model's ability to handle zero-shot scenarios by leveraging the diversity of the question. However, Mirror's approach does not benefit in this way.\nhttps://aclanthology.org/2020.emnlp-main.128.pdf\n2.\tWhat distinguishes Extractive Question Answering (QA) from extractive Machine Reading Comprehension (MRC) tasks, and why is this distinction repeatedly stressed in the paper? Within the framework of Mirror, they represent the same task: given a query, the goal is to predict the boundaries of the answer and extract the answer.\n", "limitations": "", "rating": 2, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful suggestions and elaborate comments! In the following, your questions are first stated and then followed by our responses.\n\n**Q: The comparisons made in Table 1 lack rigor. In theory, MRC and classification tasks can also be accomplished by other frameworks.**\n\nIt is a valid point that the MRC and classification tasks could be theoretically accomplished by multiple frameworks, such as extraction models and flexible generation-based models. However, in the actual usage of these methods, there are many potential problems.\n- For the MRC task: It is slightly different from NER as there is a query per instance without span types. For triple-based universal IE frameworks (e.g., UIE, DeepStruct), the query may be integrated into the input texts directly, while the type-free span is difficult to be represented as a triple. USM is pretrained on MRC datasets, but the results are absent from their paper. So, we tag it as \"theoretically supports MRC\" in Table 1.\n- For DeepStruct and TANL frameworks on the classification task: The task is not officially supported as no relevant experiments are conducted in their papers. As you have said, there are some potential difficulties in supporting the task since we have to set up new input and output formats and devise new corresponding structure decoding strategies. We will revise Table 1 with a more fine-grained comparison to clarify the distinctions and commonalities between the proposed Mirror framework and other approaches.\n\n**Q: Information positions could be obtained from generation-style methods.**\n\nThank you for pointing out the position information and its utilization in UIE and TANL. There is no such issue in Mirror and USM, while it is a common problem in generation-based methods.\n- For string matching in UIE, it is not perfect and may bring ambiguities. To investigate the matching accuracy, we take the NER task as an example and use golden entity strings to calculate the upper bound F1 scores of different UIE string matching strategies. The table below shows that the upper bounds are quite low on the datasets (<30%). This indicates that obtaining positions via string matching is ineffective and has serious ambiguity problems.\n| NER           | ACE04 ↑ | ACE05 ↑ | CoNLL03 ↑ |\n|------------------|--------:|--------:|----------:|\n| Mirror           |  100.00 |  100.00 |    100.00 |\n| UIE-first        |   13.31 |   14.51 |     27.67 |\n| UIE-longer first |   14.55 |   16.21 |     27.97 |\n\n- We acknowledge that TANL can provide some positions since it uses the enclosure tags. However, it still faces the ambiguity problem when two entities have the same string in joint entity relation extraction because the tail entity is a generated text corresponding to an enclosed head entity (refer to section 3 in the TANL paper). We also calculate the upper bound F1 scores of relation extraction in a TANL manner, and the results show it does not ideally generate accurate positions.\n| RE | ACE05 ↑ | CoNLL04 ↑ | NYT ↑  | SciERC ↑ |\n|--------|--------:|----------:|-------:|---------:|\n| Mirror |  100.00 |    100.00 | 100.00 |   100.00 |\n| TANL   |   96.91 |     96.95 |    100.00 |    99.64 |\n\n- About the crossmarks in Table 1, our original intention was to compare unambiguous position generation methods. We will carefully revise our comparison to ensure that the statement accurately reflects the differences in how position information is provided across different frameworks instead of simple checkmarks and crossmarks.\n\n**Q:  It is recommended that the authors conduct ablation experiments on pre-training data according to different task types to clarify the role of each pre-training task part.**\n\nThanks for the suggestion! We conduct ablation experiments on the pretraining data to better understand how different pre-training tasks contribute to the performance. As the table below shows, both classification and MRC data improve the IE performances, and we think Mirror could benefit from the classification task to gather more global features. We will conduct more ablation studies and add these results and corresponding analyses to the revised version. We believe that this will enhance the transparency of our approach and its effectiveness in handling diverse tasks.\n\n|               | NER - CoNLL03 (F1) | Rel - NYT (F1) | MRC - SQuAD v2 (F1) | Cls - MRPC (Acc) | Average |\n|---------------|-------------------:|---------------:|--------------------:|-----------------:|--------:|\n| Mirror-direct |              66.91 |          69.67 |               67.39 |            89.22 |   73.30 |\n| w/o Cls       |              66.82 |          57.03 |               67.14 |             0.00 |   47.75 |\n| w/o IE        |               0.00 |           0.00 |               68.77 |            89.22 |   39.50 |\n| w/o Span      |              66.76 |          54.76 |                0.00 |            87.50 |   52.26 |\n\n**Q: Why treat the MRC task's question as an instruction, rather than a label schema?**\n\nThank you for the question and the reference! We want to ensure every task is instructed, so the instruction section is different from label schemas. In addition, MRC's natural language questions are more similar to instructions rather than discrete labels, so we differentiate those questions from labels when constructing inputs to the model.\n\n**Q: What distinguishes extractive Question Answering (QA) from extractive Machine Reading Comprehension (MRC) tasks, and why is this distinction repeatedly stressed in the paper?**\n\nSorry for the confusion. We intended to emphasize they are the same task. We will address the problem in the revised version with more caution.\n\n**Q: The writing needs to be improved.**\n\nWe appreciate your feedback on the writing style and grammatical errors. We will thoroughly review and improve the writing to ensure the paper is presented in a clear and polished manner.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis paper proposes a framework that is suitable for extraction, MRC, and classification tasks and further introduces an Instruction-enhanced model building on the traditional BERT-Style framework.\n2.\tThe paper conducts detailed experiments on massive datasets and has strong few-shot and zero-shot performance. Moreover, it introduces discontinuous entity extraction for the first time in universal information extraction tasks.\n", "reasons_to_reject": "The comparisons made in Table 1 lack rigor. In Table 1, the authors purport that Mirror, in contrast to other universal information extraction models and structure prediction models, can be applied to not only complex IE tasks but also MRC and classification tasks. Yet, in theory, these tasks can also be accomplished by other frameworks.\nFirstly, all of these frameworks can perform MRC tasks as they are all capable of extracting a single text span given a query. Furthermore, frameworks such as TANL and DeepStruct, despite being structure prediction models, are not inherently incapable of performing classification tasks. TANL and DeepStruct have been tested on relation classification tasks, and by removing entity identifiers, they both can perform text classification, similar to the original T5 model.\n\nThe authors also contend that a significant issue with generation-style methods is their failure to accurately predict the location of structured information, which is critical for NER tasks and fair evaluations. However, both UIE (Lu et al., 2022) and TANL (Paolini et al., 2021) utilize Token Position as the extraction result. UIE employs the string match method to recover the span's position, while TANL generates identifiers encompassed by specific tokens, thereby effectively obtaining location information.\n\nAlthough Mirror serves as a universal framework that can simultaneously handle classification and extraction, I did not see any impact of pre-training with classification data on model performance. Moreover, although classification and extraction are similar in form, they do not focus on the same information. Classification tasks pay more attention to the overall information of the text, while extraction tasks focus more on specific local information. It is recommended that the authors conduct ablation experiments on pre-training data according to different task types to clarify the role of each pre-training task part.\n", "questions_to_authors": "1.\tWhy treat the MRC task's question as an instruction, rather than a label schema?  USM/MRC-EE treat the MRC task's question as a special type of entity/event, and their experiments have proven that this can enhance the entity/event argument model's ability to handle zero-shot scenarios by leveraging the diversity of the question. However, Mirror's approach does not benefit in this way.\nhttps://aclanthology.org/2020.emnlp-main.128.pdf\n2.\tWhat distinguishes Extractive Question Answering (QA) from extractive Machine Reading Comprehension (MRC) tasks, and why is this distinction repeatedly stressed in the paper? Within the framework of Mirror, they represent the same task: given a query, the goal is to predict the boundaries of the answer and extract the answer.\n", "paper_topic_and_main_contributions": "This paper presents a novel framework called Mirror for information extraction (IE) tasks, which reorganizes IE problems as a multi-span cyclic graph extraction problem and devise a non-autoregressive graph decoding algorithm to extract all spans in a single step.\nThis graph structure is incredibly versatile, supporting not only complex IE tasks but also machine reading comprehension and classification tasks. Furthermore, the authors have manually constructed a pre-training corpus containing 57 datasets and have conducted experiments on 30 datasets across 8 downstream tasks, demonstrating the model's compatibility and outperforming or reaching competitive performance under the zero/few-shot settings. This paper also provides a complete code and dataset as well as an interactive interface for future work. This is beneficial for future research.\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1.\t“Extractive Pretrained Language Model” is a weird expression. Although it's evident that the author's intention is to differentiate Mirror from generative methods, the expression \"Extractive Pretrained Language Model\" isn't commonly known or used. A more comprehensible and understandable expression might be \"Bidirectional Encoder style\" or “BERT-style” PLM only.\n2.\tThere are quite a few grammatical errors in the text. The writing needs to be improved.\n"}}]

Author rebuttal:

Meta-review: All reviewers agree that this work has made some interesting contributions to the field of IE by introducing the unified approach that casts various IE problems into a span graph prediction problem. The authors used a non-auto-regressive approach, which to me is the right way to go. There are useful suggestions for strengthening the work with additional experiments, which I suggest the authors incorporate into the final version.
Decision: Accept-Main