UTC-IE: A Unified Token-pair Classification Architecture for Information Extraction
Authors: Hang Yan
Yu Sun
Xiaonan Li
Yunhua Zhou
Xuanjing Huang
Xipeng Qiu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Information extraction
unified classification
Transformer
CNN

Abstract: Information Extraction (IE) spans several tasks with different output structures, such as named entity recognition, relation extraction and event extraction. Previously, those tasks were solved with different models because of diverse task output structures. Through re-examining IE tasks, we find that all of them can be interpreted as extracting spans and span relations. We propose using the start and end token of a span to pinpoint the span in texts, and using the start-to-start and end-to-end token pairs of two spans to determine the relation. Hence, we can unify all IE tasks under the same token-pair classification formulation. Based on the reformulation, we propose a \textbf{U}nified \textbf{T}oken-pair \textbf{C}lassification architecture for \textbf{I}nformation \textbf{E}xtraction (\textbf{UTC-IE}), where we introduce Plusformer on top of the token-pair feature matrix. Specifically, it models axis-aware interaction with plus-shaped self-attention and local interaction with Convolutional Neural Network over token pairs. Experiments show that our approach outperforms task-specific and unified models on all tasks in 10 datasets, and achieves better or comparable results on 2 joint IE datasets. Moreover, UTC-IE speeds up over state-of-the-art models on IE tasks significantly in most datasets, which verifies the effectiveness of our architecture.

Under review as a conference paper at ICLR 2023
UTC-IE: A UNIFIED TOKEN-PAIR CLASSIFICATION ARCHITECTURE FOR INFORMATION EXTRACTION
Anonymous authors Paper under double-blind review
ABSTRACT
Information Extraction (IE) spans several tasks with different output structures, such named entity recognition, relation extraction and event extraction. Pre- viously, those tasks were solved with different models because of diverse task output structures. Through re-examining IE tasks, find that all of them be interpreted as extracting spans and span relations. We propose using the start and end token of span pinpoint the span in texts, and using the start-to-start and end-to-end token pairs of two spans to determine the relation. Hence, we can unify all IE tasks under the same token-patt -airr classification formulation. Based on the reformulation, we propose a Unified Token-par pair Classification architecture for Information Extraction (UTC-IE), where we introduce Plusformer on top of the token-pair feature matrix. Specifically, it models axis-aware interaction with plus- shaped self-attention and local interaction with Convolutional Neural Network over token pairs. Experiments show that our approach outperforms task-specific and unified models on all tasks in 10 datasets, and achieves better or comparable results on 2 joint IE datasets. Moreover, UTC-IE speeds up over state-of the-the-art models on IE tasks significantly in most datasets, which verifies the effectiveness of our architecture.
INTRODUCTION
Information Extraction (IE) aims to identify and classify structured information from unstructured texts (Andersen et al. 1992 Grishman 2019). IE consists of a wide range of tasks, such as named
entity recognition (NER), joint entity relation extraction (RF and event extraction (EE) 2
In the last decade, many paradigms have been proposed to solve IE tasks, such as sequence label-
ing (McCallum & Li| 2003] Huang et al. 2015 Zheng et al., 2017: Yu et al. 2020a). span-based
classification (Jiang et al. |0200 Yu et al. 2020b Wang et al. 1021 Ye al. 2022 MRC-based
methods (Lev et al. 2017 Li et al. [2020] Liu et al. 2020) and generation based methods (Zeng
let al. 2018 Yan et al. 202 individual tasks, but it is desired to have a unified model to solve all IE tasks without designing dedicated modules. Besides, tackling all IE tasks with one model
Hsu al. 2022). The above work mainly concentrates solving
facilitate knowledge sharing between different tasks. Therefore, various attempts have been made to unify all IE tasks with one model structure. [Wadden al. (2019); Lin al. (2020); Nguyen et al. (202I) encode all IE tasks' target structure as graphs and design graph-based methods to predict them; Paolini et al. (2021); ));
Lu et al. (2022) solve general IE tasks in a generative way with a text-to-text or text-to-structure framework. However, basph models tend to be complex to design, and generative models are time-consuming to decode.
In our work, we creatively propose a simple yet effective paradigm for unified IE. Inspired by Diang al. (2020), we re-examine IE tasks and consider that of them are fundamentally span extrac- tion (entity extraction in NER and RE, trigger classification and argument span detection in EE)
Joint entity relation extraction aims extract both entities and relations. In our paper, call it relation extraction (RF) for simplicity. FEvent extraction covers trigger extraction and argument extraction, where we first conduct argument span detection and then conduct argument role classification in our architecture.

---

Under review as a conference paper at ICLR 2023
6x
School of Computer Scance ho the bust teachers
britih auther LX, Bowiing wrate Potter nowsh
gir netalyy
New York
City
Britsh
The
aather
sH
Compater
Science
41
returned
701, rol,
rola
wrste
New
42
Potter
York
Seachers
soveb
git
New City
School
J.. Rowling
Author
Transport
rot,
Artfact
roly
Destimation
(a) Named Entity Recagnition
(b) Relation Extraction
(c) Event Extraction
An illustration of the token-pair decomposition for IE tasks. Each cell represents one token pair, and it can be classified into pre-defined types. e, r, t, a and rol in figures mean entity, relation, event trigger, event argument and event role. For the span extraction, we use the start-to-end and end-to-start token pairs to pinpoint the span, such as entity spans e1; C2, argument spans (,, 02 and trigger span t (cells with pure color). For the relational extraction, we use the start-to-start and end-to-end token pairs represent the relation, such as and rolı, rol2 (cells with gradient color). Therefore, all IE tasks can be decomposed into token pair classifications. After the reformulation, the local dependency and interaction from the plus-shaped orientation (as the orange and blue dotted lines depict) can provide vital information to classify the central token pair.
Figure 1:
or relational extraction 3 (relation extraction in RE and argument role classification in EE). Based on this perspective, we further simplify and unify all IE tasks into token-pair classification tasks. Figure shows how each task can be converted. Specifically, a span is decomposed into start-to-end and end std-tttstart token pairs. As depicted, the entity "School of Computer Science" in Figure T[a is decomposed into indices of (School, Science) and (Science, School). As for detecting the relation between two spans, we convert it into start-to-start and end-to-end token pairs from head mention to tail mention. For example, in IT), the relation "Author" between "J.K. Rowling" and "Harry Potter novels" is decomposed into indices of (J.K., Harry) and (Rowling, novels).
Based on the above decomposition, we propose a Unified Token-pair Classification architecture for Information Extraction (UTC-IE). Specifically, we first apply Biaffine model on top of the pre- trained language model get representations of token pairs. Then we design a novel Transformer to obtain interactions between them. As the plus-shaped dotted lines depicted in Figure [ token pairs horizontal and vertical directions cover vital information for the classification on the central token pair. For span extraction, token pairs in the plus-shaped orientation are either clashing or nested with the central token pair, for example, 62 is contained by e1 in Figure Ta): for relational extraction, the central token pair's 'ss two constituent spans locate in the plus-shaped orientation, such in Figure [bb), r is determined by éj and 62. Therefore, we make one token pair only attend horizontally and incorporated keep
vertically in the token pair feature matrix. In addition, position embeddings the token pairs position-aware Moreover, neighboring token pairs are highly likely to be informative determine the types of the central token pair, so we apply Convolutional Neural Network (CNN) to model the local interaction after the plus-shaped attention. Since the attention map for one token pair is intuitively similar to the plus operator, we name the novel module as Plusformer.
We conduct numerous experiments in two settings, When training separately on each task, our model outperforms previous task-specific and unified models on 10 datasets of all IE tasks. When training a single model simultancously on all IE tasks in one dataset (named as joint IE task), UTC- IE achieves better or comparable results than 2 joint IE baselines. To thoroughly analyze why our UTC-IE architecture is useful in IE tasks under the token-par paradigm, we execute several ablation studies. We observe that CNN module Plusformer plays a significant role in IE tasks because of the abundant local dependency between token pairs after the reformulation. Furthermore, owing to
In this paper, use relational extraction to represent extracting relations hetween spans, which has broader meanings than relation extraction.
2

---

Under review as a conference paper at ICLR 2023
the good parallelism of self-attention and CNN, UTC-IE is one to two orders of magnitude faster than prior unified IE models and some task-specific work. To summarize, key contributions
follows
1. We introduce UTC-IE, which decomposes all IE tasks into token-pair classification tasks. In this way, we can unify all IE tasks under the same task formulation. Henceforth, we can use model to fit all IE tasks without designing task-specific modules. Besides, this unified decomposition is much faster than recently proposed generation- unified frameworks.
2.
After the reformulation of different IE tasks, propose the Plusformer model interac- tion between different token pairs. The plus-shaped self-attention and CNN in Plusformer are well-motivated and effective in the reformulated IE scenario. Experiments 12 IE datasets all achieve ffte-of-tth--as (SOTA) performance which justifies the superiority of Plusformer IE tasks.
3. The reformulation enables us to use one model to fit all IE tasks. Therefore, we can train one model three IE tasks, and results two joint IE datasets show that the proposed unification can effectively benefit each IE task through multi-task learning.
4. Extensive ablation experiments reveal that components in Plusformer are necessary and beneficial. Among them, CNN module in Plusformer be essential to the overall perfor- mance. Analysis shows that this performance gain is well-explained because when refor- mulating IE tasks into token pair classifications, the adjacent token pairs can be informative and CNN can take good ad advantage of the local dependency between them.
TASK DECOMPOSITION AND DECODING
We first introduce how we decompose IE tasks conduct training, and then present the decod- ing procedure for the decomposition. More discussions about the decomposition are presented in Appendix A
2.1 TASK DECOMPOSITION
Formally, given an input sentence of tokens æ= [2,, Fr..], potential token pairs can form a score matrix Y (R)) where S is span classes, R relational classes. We stipulate
When span (s, e) is of type f, then es e = Y [e, s, = 1, where s,e [1, L. and t E [1, [S]] are the start, end token indices and span type;
When the span (51,e1) e1) forms the relation E [[S]+1, [S| [R]] with another span (s2, e2), then (1s $2, r] Y(e1, 62, 1.
NER aims to extract all entities {(s,, es, 4:)}, where 4 E S. and S. is pre-defined entity types. Therefore, in NER, S S. and R
RE aims to extract all relations {((s)) eh 14),, ru (sf, et, (,))),, where the superscript and denotes the head and tail entities, t/ t E Sc, r, E R, and S., R2 are pre-defined entity types, and relation types. Therefore, in RE, S = S. and R = R,-
EE aims extract all events {{(s), es, ti), (s) e rol!).... (s e rot }}, where (s;, c,) means the trigger span, l E S, is the event type, S is pre-defined event types; Sia, Gio E [1, L. are the start and end token indices of an argument span, is the number of arguments of the trigger, to extract argument spans, of the argument and R, is pre-defined role types. We can view role types between the trigger and the arguments relations. Therefore, in EE, S = S, U. and R R..
set the argument type S., and |S 1; rol, R, is the role type
Joint IE aims to jointly extract entities, relations, and events in the text. Extracting entities and relations generally the same those in NER and RE. When extracting events, there is need to extract argument spans purposely because all argument candidates are entities. Therefore, in joint IE, S S, U S. and R R, R...
3

---

Under review as a conference paper at ICLR 2023
LayerNoma
MLP
Plusformer
X
Get.UI
3x3 Conv
Biafline Model
LayerNoms
PLM
PlusAmetiion
Figure 2: An overview of the UTC-IE Model.
2.2 DECODING
The decoding essentially extracts spans and relations from the score matrix Y. If (s,,e,tt (ee 8, t] = and E [S]]. then the span (s,e) e) is of type t. And for two spans (81 61) and (82, e2), if Y $2, $2, = Y e2. r]
and r E [|S| + 1, |S| + [R]], then the span (51, e1) forms relation r with the span (s2, e2). The above decoding is for the ideal situation, where no span clash exists. However, for model's predictions, need to first resolve the conflicts. The decoding with model's predictions will be presented Appendix B
3 METHOD
Figure shows an overview of the architecture. Firstly, we present Biaffine (Dozat Manning, 2017) model based pre-trained language models (PLM). Then, propose novel Transformer- like structure named Plusformer to model interactions between token pairs. Next, we describe loss functions.
3.1
BIAFFINE MODEL
Given input sentence, we first apply a PL. our sentence encoder to obtain the contextualized representation as follows
H [1,, h2 ... hr]
(1)
where H Rxd the PLM's hidden size.
Next, we use the Biaffine mechanism to get features for each token pair as follows
H' H MLP start (H), MLP and (H),
S Jl (H ( H + ( + H + b,
where MLP start, MLP end are multi-layer perceptron layers, H', H E Rldd, W1 E Rdxexd W2 E ox 2d b E R + refers to concatenation; S E IR IxXxx provides features for all possible token pairs, and is the feature dimension size.
3.2 PLUSFORMER
As illustrated in Section [ when modeling the interaction between token pairs, the plus-shaped and local interaction should be beneficial. Therefore, we introduce the axis-aware plus-shaped self- attention and position embeddings conduct plus-shaped interaction, name this self-attention Plusttteniion. Then, we leverage CNN to model local dependencies. We name this whole structure Plusformer.
4

---

Under review as a conference paper at ICLR 2023
PlusAttention. We first apply the self-attention mechanism vertically as follows
Vaswani et al. 2017
horizontally and
z [1, Attention(S]i: W W W S,, [1]: = Altention(S]:::) W S S Sk.j.
QK
Attention K, V) = softmax
Ve V
After the self-attention, where W W W W W WV Rexe Zh 7 RRxxxxe the following method merge Zh Z
S' MLP R @ Z.),
(4)
where S' E LLLxx, We make the plus-shaped self-attention axis-aware by using groups of attention parameters and using concatenation instead of an addition to merge Zh Z"
Position Embeddings. Although the model should be able distinguish between horizontal and vertical directions through axis-aware plus-shaped attention, it still lacks the sense of distances be- tween token pairs and the area the token pair locates. Therefore, we utilize two kinds of position embeddings to enable the model with these abilities.
Rotary Position Embedding (RoPE) (Su et al. 2021) encode the relative distance between two token pairs. is utilized both horizontal and vertical self-attention.
Triangle position embedding is incorporated to mark the position of token pairs in the feature map, which means cells in the upper and lower triangles will use different position embeddings. adds to in Eq. before Attention.
CNN Layer. After the PlusAttention, we apply CNN with kernel size on the S' to help model exploit the local dependency between neighboring token pairs. The formulation is as follows
S" Conv(a (Conv(S')))
(5)
where S" E Lxee and a is the activation function; and the bias term of CNN not used to avoid result inconsistencies for sample when it is in batches of different lengths.
The Plusformer layer will be repeatedly used to interact fully between token pairs. Layer normal- ization (Ba al. 2016 is ignored in the formulation for brevity.
3.3 Loss FUNCTION
Finally, we get final scores as follows
Ys. YR = Sigmoid [x;,;; | [S]:], Ý MLP(S" + S),
(6)
Ys E 2xxxX| YR E [IXLX(X[+1) are scores for span ex- traction and relational extraction, respectively. The +1 in ([R|+) is because we use the adaptive thresholding loss (ATL) from (Zhou et al. 2021) to avoid global threshold in relational extraction.
where Ý LX
For the span extraction, we use the binary cross-entropy (BCE) loss follows
1.
C, = Σ I Y(i.j. ( + +11 Y(i,j, Yl.j..)
fj=1r=l
For the relational extraction, we utilize the ATL as follows
exp(Yr
= Σ
Σ log
ErEPPOTHH exp Y r()
1.j ml
rEPp
exp(YR(i,)) +1]
-log
ΣmENTURTHy exp Y [i
where Pr and Ny denote the positive and negative classes, YRl: [e|::;|R||] is the score for the threshold class TH. Only token pairs with scores higher than their corresponding adaptive thresholds are considered when decoding. We do not use ATL loss for span extraction because we need sort span scores when decoding spans. The total loss L= L1 La is used for optimization.
5

---

Under review as a conference paper at ICLR 2023
Table Overall F1 on single IE tasks. Results UTC-IE are the average of runs, and the subscript means the standard deviation (e.g., 93.452a means 93.45 Datasets marked as have nested entities. Results marked as f are from Yan al. )022). means results from their Github repo or our reproduction. means that the UTC-IE without Plusformer surpasses previous SOTA performance.
Named Entity Recognition
CoNLL03
OntoNotes
ACE04*
ACE05-Ent*
GENTA*
BART-NER Yan et al.
2021
93.24
90.38
86.84
84.74
78.93
TANL Paolim al.
2021)
91.7
89.8
84.9
76.4
W (Li 2022
93.07
90.50
87.431
86.77
80.32
UIE
(Lu et al. 2022
92.99
86.89
85.78
BS (hu & 2022
93.39;
91.51;
87.081
87.20
CNN-NER Yan et al.
2022
87.31
87.42
80.331
UTC-IE
93,4524
91.77s
87.5433
87.75a5
80,45
-Plusformer
92.9813
91.37s
86.5123
86.5920
79.3417
ACE05-Rias Rhat
ACE05-R,,h Rabbett
SciERC
Relation Extraction
Ent.
Rel.
Ent.
Rel.
Ent.
Rel.
TANL
Paolini al.
2021
88.9
63.7
PURE
Zhong
Chon 2021
88.7
63.9
89.7
65.6
66.6
35.6
PFN
Yan ct al.
2021b
89.0
66.8
67.257
37.65a
UIE
(u ct al.
2022
66.06
36.53
UTC-IE
88.82 88. 88.5019
64.94a3
89.87 15
67.794s
69.03
38.77p8
-Plusformer
63.3472
89.80
66.21
68.05
37.1240
ACE05-R
SciERC+
Symmetric Relation Extraction
Ent.
Rel.
Ent.
Rel.
UniRE (Wang al. 2021
88.8
64.3
68.4
36.9
PL-Marker Ye UTC-IE -llusformer
2022
89.8
66.5
69.9
41.6
90.1621
67.4774
69.95
42.51
88.9 88.9829
64.588s
68. 78
39.5 16.51a
ACE05-E
ACE05-E+
ERE-EN
Event Extraction
Trig.
Arg.
Trig.
Arg.
Trig.
Arg.
TANI. Paolini ct al.
2021
68.4
47.6
TEXT2EVENT (Lu et 2021
71.9
53.8
71.8
54.4
59.4
48.3
UIE (Lu et al. 2022
73.36
54.79
DEGREE HHsu el al.
2022
73.3
55.8
70.9
56.3
57.1
49.6
UTC-IE
73.46ga
56.51
73.44s5
57.68
60.20ga
52.51
Plusformer
72.8878
55.41
72.92g4
56.63
59.2877
51.33
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
We conduct experiments on 10 datasets across three IE tasks, including NER, RE, and EE, and on 2 joint IE datasets. We evaluate NER task with CoNLL03 (Sang & Meulder 2003) and OntoNotes (Pradhan et al,, 2013) on flat NER, and with ACE04 (Doddington et al.) 2004). ACE05-Ent (Walker et al. (006)) and GENIA (Kim et al., 2003) on nested NER. As for relation extraction, we use ACE05-R (Wakerr et al., (2066) and SciERC (Luan et al. 2018). Since WWng et al. (202T) and Ye ct al. (2022) consider symmetric relations, which shall massively influence the performance, we name this scenario Symmetric RE with datasets ACE05-R+ and SciERC+ For event extraction, we follow Lin et al. (2020) to perform experiments on three datasets, ACE0S-E, ACE05-EA E+ (Dod- dington al. 2004) and ERE-EN (Song al. 2015). And for joint IE, test ACE05-E+ and ERE-EN. Statistics of all these datasets, implementation details, evaluation metrics, baselines, pre-trained language models and Syper-parameter described in Appendix C

---

Under review as a conference paper at ICLR 2023
Table 2: Results on joint IE. UTC-IEséngle shows results by separately trained model NER, RE and EE, while JoCt shows results by jointly trained model. means that UTC-IE without Plusformer surpasses previous SOTA performance.
ACE05-E+
ERE-EN
Joint IE
Ent.
Rel.
Trig.
Arg.
Ent.
Rel.
Trig.
Arg.
OnelE Lin al. 2020
89.6
58.6
72.8
54.8
87.0
53.2
57.0
46.5
FourlE Nguyen et al. 2021
91.1
63.6
73.3
57.5
87.4
56.1
57.9
48,6
UTC-IE vingle
91.3710
65.00g9
71.9865
56.01
86.35
55.5792
57.01
48 48,2900
UTC-IE joint
91.4820
65.54g0
73.6347
57.6230
87.301s
56.92g0
57.8898
50.91
Plusformer
90.7230
62.947s
72.9962
55.6874
86.9413
54.28p4
57.79gs
48.7251
4.2 RESULTS SINGLE IE TASKS
In this section, we report the UTC-IE performance in each single IE task. Results are shown in Table The table shows that UTC-IE exceeds the previous SOTA models on all IE tasks. Particularly,
UTC-IE averagely improves the entity Fl of NER, the entity FI and relation FI of RE, the entity Fl and the relation Fl symmetric RE for +0.18, 18, +0.71, +1.07, +0.21, +0.94, respectively. And for the EE tasks, UTC-IE increases the trigger F1, argument F1 for +0.35 and +1.67. We highlight that our model significantly helpful for relational extraction, such as relation extraction and argument extraction, which proves the effectiveness of interaction between token pairs. Although the per- formance increment of span extraction is not as significant as that of relational extraction, UTC-IE consistently improves various span extraction tasks.
Besides, we also test UTC-IE without Plusformer. Surprisingly, this simple model surpasses previ- SOTA models on four results marked with & which proves the effectiveness of the task decom- position. The comparison between models with and without Plusformer clearly shows that Plus- former is effective all tested datasets, and the performance improvement ranges from +0.40 (on OntoNotes) to +3.00 (on SciERC*). Notably, the average performance gain of adding Plusformer on symmetric RE (+2.06) more remarkable than that on RE (+1.03). We presume this is because the interaction between token pairs are more beneficial for symmetric relations.
4.3 RESULTS ON JOINT IE TASK
Multi-task learning has be proven to be useful in the IE area (Lin et al. 2020 Nguyen et al. 2021 Since UTC-IE unifies all IE tasks into a token-paie classification scenario, it is natural to test whether one UTC-IE model can benefit from jointly learning all IE tasks. In Table 2 the performance of UTC-IEsingle is from the entity F1 of NER, relation FI of RE, trigger F1 of EE and argument F1 of EE, respectively. Based the comparison between UTC-IE and UTC-IE it is obvious that jointly learning these three tasks consistently improves performance in the 2 joint IE datasets.
Moreover, UTC-IEsjaitt outperforms previous SOTA joint IE models in Table 2 the average per- formance enhancement is +0.69 in ACE05-E+ and +0.75 in ERE-EN. Specifically, UTC-IE,oint relational extraction by +1.30. Thusly, through unifying dif-
increases the average performance ferent IE tasks through our task decomposition, Plusformer can enjoy the benefit of multi- multi-tasking learning, and achieve better performance than previous SOTA models.
4.4 SPEED COMPARISON
To get a sense of the speed superiority of UTC-IE, we compare the inference speed of UTC-IE with previous unified models and task-specific SOTA models. The former comparison is presented Table and the latter locates in the Appendix D Compared with the generative UIE (Lu et al.) 2022), UTC-IE improves Fl from 0.46 (on CoNLL03) to 2.89 (on ACE05-E+), and obtains one order magnitude of speed boost. Compared with OnelE (Lin et al. 2020), UTC-IE fundamentally enhances the performance for relational extractions (e.g.. Rel. and Arg.) with average of 4.47 Fi increment in joint IE. At the same time, UTC-IE is one order of magnitude faster than OneIE. In
*The precision and recall for UTC-IE in these datasets be found in Table Appendix.

---

Under review as a conference paper at ICLR 2023
Table 3: The Fl and efficiency comparison with UIE and OnelE, "Ent.", "Rel." and "Arg." denote F1 of corresponding test sets. "Speed" is measured in "sentence/s" on inference procedure. The improvement shows the changes in performance and speed.
CoNLL03
ACE05-Ent
(CE05-Ralbere
SciERC
ACE05-E+
Single IE
Ent.
Speed
Ent.
Speed
Rel.
Speed
Rel.
Speed
Arg.
Speed
UIE (u al. 2022
92.99
14.5
85.78
8.6
66.06
11.4
36.53
8.7
$4.79
4.0
UTC-IE
93.45
210.3
87.75
304.3
67.79
85.4
38.77
165.7
57.68
88.1
Improvement
+0.46
x14.5
+1.97
x35.4
+1.73
x7.5
+2.24
x19.0
+2.89
x22.0
ACE05-E+
ERE-EN
Joint IE
Ent.
Rel.
Trig.
Arg.
Speed
Ent.
Rel.
Trig.
Arg.
Speed
OnelE (Lin et al. 2020)
89.6
58.6
72.8
54.8
4.8
87.0
53.2
$7.0
46.5
13.5
UTC-IE Improvement
91.48
65.54
73.63
57.62
121.6
87.30
56.92
57.88
50.91
153.5
+1.88
+6.94
+0.83
+2.82
x25.3
+0.30
+3.72
+0.88
+4.41
x11.4
Table 4: Ablation studies in the NER, RE and EE datasets. CNN-IE is similar to UTC-IE except
that it is deprived of the PlusAttention. Underlines mean the most dropped factor. means that the CNN-IE surpasses previous SOTA performance.
ACE05-Ent
ACE05-R bbert bert
ACE05-F+
Ent.
Ent.
Rel.
Trig.
Arg.
UTC-IE
87.7535
88.8212
64,9433
73.4435
57.6878
CNN
87.3922
88.7122
63.55ss
72.9831
56. 36.74gg
positon embeddings
87.5334
88.7320
64.29s6
73.12es
57.02so
axxisaaarre
87.59u7
88.7919
63.91
73.2946
56.87as
CNN-IE
87.45
88.70
64.67 26
73.0499
56.97
a nutshell, compared with previous SOTA models (whether task-specific, unified or joint), UTC-IE achieves substantial performance gain across several datasets with a significant speed boost.
4.5 ABLATION STUDY
To analyze the effectiveness of each component in Plusformer, we ablate each of them and list the outcomes in Table 4 and results on more datasets are presented in Appendix F Besides, we study how many Plusformer layers are suitable in Appendix F.5) Based on the ablation. CNN the most useful component among all IE tasks. The reason behind this improvement is that once token pairs are organized in the square feature map, the spatial correlations between neighboring token pairs become allusive, and CNN excels at exploiting these local interactions. More comprehensive analysis of CNN in Plusformer locates in Appendix F.I To deepen our understanding of UTC- IE, we try another variant of Plusformer where the PlusAttention is discarded, and name this variant CNN-IE, The bottom line of Table 4 shows that the CNN-IE model can surpass or approach previous SOTA performance in almost all datasets, which proves the universality of our proposed task formulation.
However, CNN is not panacea for UTC-IE. From Table 4 removing position embeddings or axis- award from UTC-IE will lead to an average of 0.39 or 0.44 performance degradation, respectively. Moreover, based
the performance of CNN-IE and UTC-IE, the average performance shrinks from 74.53 to 74.17 if the Pttsnttention is deprived of Plusformer, which means the plus-shaped self-attention is a desideratum. In addition, present some intuitive examples and deeper analysis for position embeddings and axir-aware in Appendix F33 and
'Removing axis-aware means using the same self-attention parameters for both directions and adding th and Z" instead of concatenation.
8

---

Under review as a conference paper at ICLR 2023
5 RELATED WORK
Information extraction tasks, which consists of named entity recognition, relation extraction, and event extraction, have long been a fundamental and well-researched task in the natural language pro- cessing (NLP) field. Previous researches mainly only focus on one or two tasks. Recently building joint neural models of unified IE tasks has attracted increasing attention. Some of them incorporate graphs into IE structure. WWdden et al. (2019) propose unified framework called DYGIE++ to extract entities, relations and events by leveraging span representations via span graph updates. Lin et al. (2020) and Nguyen et al. (2021) extend DYGIE++ by incorporating global features to extract cross-task and cross-instance interactions with multi-task learning. In addition to the graph-based models mentioned above, other studies focus tackling general IE by generative models. PPalinn et al.] (202T) construct framework called TANL, which enhances the generation model using aug- mented language methods. Moreover, [Lu et al. (2022) regard IE task a text-to-structtre generation task, and leveraging prompt mechanism.
We unify all IE tasks several token-pair classification tasks, which are fundamentally similar to the span-based methods on the IE task, for the start and end tokens can locate a span. Numerous NER studies emerge on span-based models, which are compatible with both flat and nested entities and perform well (Eberts & Ulges 2020 Yu et al. 2020b Li et al. 2021: Zhu & Li 2022). In addition to entities, the span-baser method is also used in RE Some models (Wang et al. 2021
Ye et al. 2022) only leverage span representations to locate entities and simply calculate the in- teraction between entity pair, while others (Wang et al., 2020: Zhong & Chen 2021) encode span pair information explicitly to extract relations. With regard to event extraction, as far as we know, there is little work on injecting span information into EE explicitly. Wadden et al. (2019) leveraging span representations embedding layer without further interaction. Conceptually, Jiang et al. (2020)* 's work is similar to ours, but they need a two-stage model determine the span type and span relations, respectively. Detailed analysis are depicted in Appendix G Although many span-based IE models exist, they are task-specific and lack interaction between token pairs. Decomposing IE tasks as token-pair classifi- cation and conducting interaction between token pairs can uniformly model span-related knowledge and advance SOTA performance.
general IE, but their model is complicated and only considers span at the
The key component of Plusformer is the plus-shaped attention mechanism, which can make token pairs interact with each other in an efficient way. A similar structure called Axial Transformers (Ho et al. 2019) is proposed in Computer Vision (CV) field, which is designed to deal with data organized as high-dimension tensors. [Tan et al. (2022) incorporate axial attention into relation clas- sification, aiming to improve the performance on two-hop relation. However, CNN was not used in these work, while CNN has been proven to be vital to the IE tasks. Another similar structure named Twin Transformer (Guo et al. 2021) used in CV is very much similar to Plusformer, where they en- code pixel of image from row and column sequentially, and leverage CNN on top of them. But the position embeddings, which is important for IE tasks, are not used in the Twin Transformer. Besides, we want to point out that the usage of plus-shaped attention and CNN originates from the reformula- tion of IE tasks, any other modules which can directly enable interaction between constituent spans of a relation and between adjacent token pairs should be beneficial.
CONCLUSION
In this paper. we decompose NER, RE and EE tasks into token-pair classifications. Through the decomposition, we unify all IE tasks under the same formulation. After scrutinizing the token- pair feature matrix, we find the adjacent and plus-shaped interactions between token pairs should be informative. Therefore, we propose Plusformer, which uses an axis- plus-shaped self-attention followed by CNN layers to help token pairs interact with each other. Experiments in 10 single IE datasets and 2 joint IE datasets all outperform or approach the SOTA performance. Besides, owing to the parallelism of self eelf-attention and CNN, our model's 's inference speed is substantially faster than previous SOTA models in RE and EE. Lastly, most of the previous IE models limit the interaction in the 1-D sequential dimension, while the reformulation of IE tasks opens a new angle to broaden the communication to the 2-D feature matrix.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Peggy M. Andersen, Philip J. Hayes, Steven P. Weinstein, Alison K. Huettner, Linda M. Schmandt, and Irene B. Nirenburg. Automatic extraction of facts from press releases generate news stories. In 3rd Applied Natural Language Processing Conference, ANLP 1992, Trento, Italy, March 31
April 3, 1992 pp- 170-177. ACL, 1992. doi: 10.3115/97444999775331 URL https://
ac velantholoee logcholooy..oo org A92 10247]
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. abs/1607.064555 2016. URL http: wwtg/bgggaaaaooo///..o/ee 06450
CoRR,
George R. Doddington, Alexis Mitchell, Mark A. Przybocki, Lance A. Ramshaw, Stephanie M. Strassel, and Ralph M. Weischedel. The automatic content extraction (ACE) program tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation, LREC 2004, May 26-28, 2004, Lisbon, Portugal. European Language Resources Association, 2004. URL http: www htt.://cco.o....e.eeeeeeeeeeeeee
summarr Summinriess/.hee.
Timothy Dozat and Christopher D. Manning. Deep biaffine attention for neural dependency pars- ing. 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: //openreview www.:ee/fff... wtt.:e/oorrrmm....55......
Markus Eberts and Adrian Ulges. Span-based joint entity and relation extraction with trans- former pre-training. In Giuseppe De Giacomo, Alejandro Catalá, Bistra Dilkina, Michela Mi- lano, Senén Barro, Alberto Bugarín, and Jérôme Lang (eds.), ECAI 2020 24th European Conference on Artificial Intelligence. 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 September 8, 2020 Including 10th Conference on Prestigious Applica- tions of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Applications, pp. 2013. IOS Press, 2020. doi: 10.323//FAA2203211 https: 1tto..oog/10..333//A....
Ralph Grishman. Twenty-five years of information extraction. Nat. Lang. Eng.. 25(6))677- 692, 2019. doi: 10.10177513513249190000 5135132491 9000512
URL
Ruohao Guo. Dantong Niu, Liao Qu, and Zhenbo Li. SOTR: segmenting objects with transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 7137-7146. IEEE, 2021. doi: 10. 109/CCCV48922.2021.. 00707.
http:////oooooooo.......oo.///// /10. www.11000/011922222 20222222221200000000.002
Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many- task model: Growing neural network for multiple NLP tasks. In Martha Palmer, Rebecca Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp. 1923 1933. Association for Computational Linguisties, 2017. doi: 10. 10.18633/11/111-1200 URL /ttp:://doi..oo httppp///oo...orr0...8833///////
Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multi- dimensional transformers. CORR, abs/1912.12188, 2019. URL http:://axii.. .rg. abs/
191 291..7 80
I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee, Scott Miller, Prem Natarajan, Kai-Wei Chang, and Nanyun Peng. DEGREE: A data-efficient generation-based event extraction model. In Ma- rine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 1890-1908, Association for Computational Linguistics, 2022. doi: 1111166//11/2222
naacl-man maiel-main.tee 138.. URL http::ttt///cm / 1tt3:///1111111331////22222 naacl in. 138
Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional LST models for sequence tagging. CoRR, tbs/1558.0199 2015. Utl.httpp//arxtvv.rrr///bb//bb
10

---

Under review as a conference paper at ICLR 2023
Zhengbao Jiang, Wei Xu, Jun Araki, and Graham Neubig. Generalizing natural language anal- ysis through span-relation representations. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, ACI. 2020, Online, July 5-10, 2020, pp. 2120-2133. Association for Compu- tational Linguistics, 2020. doi: 10. 10.16653/1/2220aacllmmo
192. URL https: /tttpp://do..coo .org/
10. 11w11166///222/2 acl main. maain.192 192
Dong Kim, Tomoko Ohta, Yuka Tateisi, and iun'i Tsujii. GENIA corpus a semantically annotated corpus for bio-textmining. In Proceedings of the Eleventh International Conference on Intelligent Systems for Molecular Biology, June 29 July 3, 2003, Brisbane, Australia, pp. 180-182, 2003. URL httt::::/biiiinoormm.a. 1ttp///pppp_1111101111 1/11807etoc
www.suuujouoaaaaoo .rg. httlg//gi/coo content./
Omer Levy, Minjoon Seo, Funsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Roger Levy and Lucia Specia (eds.), Proceedings of the 21st Confer- ence on Computational Natural Language Learning (CoNL.. 2017), Vancouver, Canada, August 3-4, 2017, pp. 333-342. Association for Computational Linguistics, 2017. doi: 10.18653/v1/
K17-1034. URL https: merpi.org//n /ttgg/1000 111.1..55.////111111
Fei Li, Zhichao Lin, Meishan Zhang, and Donghong Ji. A span span-based model for joint over- lapped and discontinuous named entity recognition. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers). Virtual Event, August 1-6, 2021, pp. 4814-4828. Association for Computational Linguistics, 2021. doi: 10.1 00.18653/v1/2000 acc aall-alllonnn. URL
https:/ //doi. /tog/10..000. 11w53811/2222
pecl-longe 372
Jingye Li, Hao Fei, Jiang Liu, Shengqiong Wu, Meishan Zhang, Chong Teng, Donghong Ji, and Fei Li. Unified named entity recognition as word-word relation classification. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2022 Virtual Event February 22 March 1, 2022, pp. 10965-10973. AAAI Press, 2022. URL hhttp::/ aaai. .acaiiorr org/ ndex. wwwpwdwwwppppedeeeeemeeeeewp wwpppppp..../..p article/view21332
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. A unified MRC framework for named entity recognition. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 5849-5859. Association for Computational I Linguistics, 2020. doi: acl ma ...
URI. https:
7/doi. /0lg/100... 18653/v1
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. A joint neural model for information extrac- tion with global features. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 7999-8009, Association for Computational Linguis- tics, 2020. doi: 10.18653/w1/2220.ce aww.://000...mmm.....mmmm 713. URL 2020. wwac-mmaaaa. 713.
Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. Event extraction as machine reading comprehension. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Longuage Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 1641-1651. Association for Computational Linguistics, 2020. doi: 10.18653/ 1/2033//1/222.cooo 10006500///0222mmmmmmmmo//m/ mann--main. 128. URL httpt:ttt//// http::::/ddoi.ooo http://ai..oo.........1.33//////
emmnppmainn.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
net/ fmr d
Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. Text2event: Controllable sequence-to-structure generation for end-to-end event

---

Under review as a conference paper at ICLR 2023
extraction. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Processing, ACL/IJCNLP 2021. (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pp. 2795-2806. Association for Computational Lin- guisties, 2021. doi: 10. 20.1865//1/2221.cco/
URL
w1/2021.ac1-
217
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. Unified structure generation for universal information extraction. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACI. 2022, Dublin, Ireland, May 22-
27, 2022, pp. 5755-5772. Association for Computational Linguisties, 2022. URL https:// aclanthology wwo.org/////. acl- long. 395
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification entities, relations, and coreference for scientific knowledge graph construction. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun 'nn'iiii Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 3219-3232. Association for Computational Linguistics. 2018. doi: 10.18653/v1/ URL https:://///.m. /oi. org/10. 1 8 / v /wtp//////- 011011111111 1360
Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hanmaneh Hajishirzi. A gen- eral framework for information extraction using dynamic span graphs. In Jill Burstein. Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 3036-3046 Association for Computational Linguisties, 2019. doi: (0.18653/v1/n19-115 1308.
URL https: org. 111.1.....111111111 1308
Andrew McCallum and Wei Li. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In Walter Daelemans and Miles Osborne (eds.), Proceedings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NACCL 2003, Edmonton, Canada, May 31 June 1, 2003, pp. 188-191. ACL, 2003. URL https: www. acwweb. wwwgg//aahhllooooooooooooo loyy//lg////// WW00/0333000077700
Minh Van Nguyen, Viet Dac Lai, and Thien Huu Nguyen. Cross-task instance representation in- teractions and label dependencies for joint information extraction with graph convolutional net- works. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Belt- agy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLL 2021, Online, June 6-11, 2021, pp. 27-38 Association for Computational Linguisties. 2021. doi: 10.18653/v1/200 maacl main... 1ttp:/.......101111131131311/220 www.llaaccaaaa....aa. mmain
URL
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cícero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. Structured prediction as transla- tion between augmented natural languages. In 9th International Conference on Learning Rep- resentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL net/ forum -id-S- doooummii=US..
ltttps::/cpenzeet
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. Semeval-2014 task 4: Aspect based sentiment analysis. In Preslav Nakov and Torsten Zesch (eds.), Proceedings of the 8th International Workshop on SSman- tic Evaluation, SemEval @ COLING 2014, Dublin, Ireland, August 23-24, 2014, pp. 27-35. The Association for Computer Linguistics, 2014. doi: 10.3115/v1//14-2000 URL https:
(/cbi.orr/110.3115//11/
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng. Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. Towards robust linguistic analysis using ontonotes. In Julia Hockenmaier and Sebastian Riedel (eds.), Proceedings of the Seventeenth Conference on Computational Natural Language Learning, CoNLL Sofia, Bulgaria. August 8-9, 2013, pp. 143-152. ACI,, 2013. URL httphhttt//aalll.l wow. 351661
12

---

Under review as a conference paper at ICLR 2023
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction the conll-2003 shared task: Language- independent named entity recognition. In Walter Daelemans and Miles Osbore (eds.), Proceed- ings of the Seventh Conference on Natural Language Learning. CoNLL 2003, Held in cooperation with HL.T-NAACI. 2003, Edmonton, Canada, May 31 June 1, 2003, pp. 142-147. ACI., 2003.
URL https: //c.aacaacc. antho www.elaahhhhoooooooooooooo og.. 2gy.orrg/033004193 -
Zhiyi Song, Ann Bies, Stephanie M. Strassel, Tom Riese, Justin Mott, Joe Ellis, Jonathan Wright, Seth Kulick, Neville Ryant, and Xiaoyi Ma. From light to rich ERE: annotation of entities, relations, and events. In Eduard H. Hovy, Teruko Mitamura, and Martha Palmer (eds.), Pro- ceedings of the The 3rd Workshop on EVENTS: Definition, Delection, Coreference, and Rep- resentation, EVENTS@ HLP-NAACL EVENTS@#LP-NAACL 2015. Denver, Colorado, USA, June 4, 2015, pp. 89-98. doi: 10.3115/1/W15-08222
Association for Computational Linguistics, 2015.
URL https:
Gabriel Stanovsky and Ido Dagan. Creating a large benchmark for open information extraction. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 2300-2305. The Association for Computational Linguistics. 2016. doi: 10. 10.18653/v1/ d16-1252. htt.:ttpp:////tt wr0//0...0000 http//1.1886555 1tt.1/116///111--11
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. CoRR, abs/2104.09864 2021. URL Utthtts::://aatt /aattpss//ariii... org/
abs/ abs/2100 /21/2204..0 09864
Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. Document-level relation extraction with adaptive focal loss and knowledge distillation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACI. 2022, Dublin, Ireland, May 22-27, 2022, pp. 1672-1681. Association for Computational Linguistics, 2022. doi: 10.18653//1/022. 12222..iin.........nn....---. acl. URL 2022. findings acl 132
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL http:ttts/:/pooeeedeiiiee neur i pps. wwtppppp.pppcccccccccc /201/hhsh// Bf5ee243547deeee hrrrg22339acee111eb000ecccceaa.3 Abstaaaaatteaaaaaaattttt...aa... Absttact........................ html
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. Entity, relation, and event extraction with contextualized span representations. In Kentaro Inui, Jing Jiang. Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Pro-
cessing, EMNLP- 2019, Hong Kong, China, November 3-7, 2019, pp. 5783-5788. As- sociation for Computational Linguisties, 2019. doi: 10.18653/11/D19-15555 URL https:
7/doi m:0/iorrg/00.. 1tt..0.1111111111111111
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. Ace 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57:45, 2006.
Yijun Wang, Changzhi Sun, Yuanbin Wu, Junchi Yan, Peng Gao, and Guotong Xie. Pre-training entity relation encoder with intra-span and inter-span information. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 1692-1705. Association for Computational Linguistics, 2020. doi: 10. 10.18663/112022.emmm
132. URL
wwt.iii66svvwaee emn ma mamiin.... 132
Yijun Wang, Changzhi Sun, Yuanbin Wu, Hao Zhou, Lei Li, and Junchi Yan. Unire: A uni- fied label space for entity relation extraction. In Chengqing Zong, Fei Xia, Wenjie Li, and
13

---

Under review as a conference paper at ICLR 2023
Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Conference on Natural Language Pro- cessing, ACLIJCNLP 2021. (Volume 1: Long Papers), Virtual Event, August 1-6, 2021. pp. 220- 231. Association for Computational Linguistics, 2021. doi: 10. 10.18653/v12001 acl-long. 19. URL httpp:://do..oo wtt.oo//g1.111111111111111111000 www111166//10000 acl-llllllnnl 19
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. A unified gen- erative framework for various NER subtasks. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/JJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 5808-5822 hal.loo..cllloo5555551 URL
Association for Computational Linguistics, 2021a. doi: acl ong. 451
Hang Yan, Yu Sun, Xiaonan Li, and Xipeng Qiu. An embarrassingly easy but strong baseline for nested named entity recognition. CoRR, abs/2bos 04534, 2022. doi: (0.48550/anXiv.2208..44
URL aattps://do1: http://....01111..5.555/5///.../
100 485508550
2208 04534
Zhiheng Yan, Chong Zhang, Jinlan Fu, Qi Zhang, and Zhongyu Wei. A partition filter network for joint entity and relation extraction. In Maric-Franeine Moens, Xuanjing Huang, Lucia Spe- cia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Re- public, 7-11 November, 2021, pp. 185-197. Association for Computational Linguistics, 2021b. doi: 10. 10.18633/v122211 12665////2221.emllmmmaaatr 17. URL pottps://dott wtt..///...11////2122222222
10.
emn p- ma i in. 17
Deming Ye, Yankai Lin, Peng Li, and Maosong Sun. Packed levitated marker for entity and relation extraction. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.). Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long
Papers), ACL 2022, Dublin, Ireland. May 22-27, 2022, pp. 4904-4917, Association for Compu- tational Linguistics, 2022. https:: ac www.hhlnnnhhogyyyoooooooooooooom org/ 2022 aac- 337
Bowen Yu, Zhenyu Zhang, Xiaobo Shu, Tingwen Liu, Yubin Wang, Bin Wang, and Sujian Li. Joint extraction of entities and relations based novel decomposition strategy. In Giuseppe De Giacomo, Alejandro Catalá, Bistra Dilkina, Michela Milano, Senén Barro, Alberto Bugarín, and Jérôme Lang (eds.), ECAI 2020- 24th European Conference on Artificial Intelligence, 29 August- 8 September 2020, Santiago de Compostela, Spain, August 29 September 8, 2020 Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Applications, pp. 2282 2289. IOS Press, 2020a. doi: 10.3233/FAIA2203556 URL htRLsttts:://// orw.l.rggg//. 221/0/3333////AA2..33
Juntao Yu, Bernd Bohnet, and Massimo Poesio. Named entity recognition dependency parsing- In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 6470-6476. Association for Computational Linguistics, 2020b. doi: 10. 18653/v11 2020.acl 577
URL https::/ org/10 18653/ acl-main. 577
Xiangrong Zeng, Daojian Zeng, Shizhu He. Kang Liu, and Jun Zhao. Extracting relational facts by an end-to-end neural model with copy mechanism. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- fics, ACI. 2018, Melbourne, Australia, July 15-20, 2018, Volume I: Long Papers, pp. 506- 514. Association for Computational Linguisties, 2018. doi: 10. 10.18653/v1/P18-1000 1047. URL wrw./ll...r./P1188 10477
Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. Joint extraction of entities and relations based on a novel tagging scheme. In Regina Barzilay and Min- Yen Kan (eds.). Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver. Canada, July 30 August 4, Volume 1: Long Papers. pp. 1227-1236. Association for Computational Linguistics, 2017. doi: 10.186533v1/117-1111 URL https: 7/doi. o0gll. 11w63//1//112/2/ htt.://///1..1115////////1/
14

---

Under review as a conference paper at ICLR 2023
Zexuan Zhong and Danqi Chen. A frustratingly easy approach for entity and relation extrac- tion. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkami-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguis-
fies: Human Language Technologies, NAACL-HI.T 2021, Online, June 6-11, 2021, pp. 50-61. Association for Computational Linguistics. 2021. doi: 12.18653/112222..maaa .aw.02ln.aal-mmiin. main.. URL
httpp:://oi..rr/10..1233 www.o.o.rg/..... 186.1....663661 v//
naacl main.5
Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. Document-level relation extraction with adaptive thresholding and localized context pooling. In Thirty-Fifth AAAI Conference on Ar- tificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelli- gence, EAA/ 2021, Virtual Event, February 2-9, 2021, pp. 14612-14620 AAAI Press, 2021. URL https: ojs aaai. org/ index. php/ AAAI /tt.a//atiileeeeeew////.
Enwei Zhu and Jinpeng Li. Boundary smoothing for named entity recognition. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 7096-7108. Association for Computational Linguis- tics, 2022. doi: 10. 11211//1//002222lllloaae 2022 acl aacc-llonng 490
URL hnttps://do/ /tttps://oollorr/1....883 106531..1.633311
A DISCUSSION ON TASK DECOMPOSITION
In this section, we will discuss two issues of the decomposition. The first is the inconsistency stipulation about the relation decomposition, the second is the false positive issue when decoding relations.
A.1 THE INCONSISTENCY
As our stipulation in Section if span (8, e) has an expected span type I, both the Y(s, e, and yle, 8,4] are 1. If two spans (81, e1) and (82, ez) have relation this means the relation should also exist between spans (#1, e1) and (e2, 82) (the end-to-start version of the span (#2, e2)). then based stipulation we only define the Y [s1,, $2, r] = and Y|e,, 62, 1, this causes an inconsistency between the stipulations. We ignore ((5 62, and Y [e1, 82, r] make the decoding less cluttered.
the relation, the Y[s1, r2 and Y ((e,, $2, r] should also equal 1. but
A.2 FALSE POSITIVE RELATION
A potential risk of the decomposition and decoding is that it may cause false positive relations.
Given four spans P1 = ($1, (1), = (82, Ps = (83, ea), P4 = (84, 84), if p4 has relation r with P1 and P2. and relation exist between P4 and ps- Then Y(s,, Y[s4, $2; r] = Y(e,,
Y lea, c1, r] 1, = 1. However, if $1 = 83, e2 = C3. Namely, shares start token get Y ( 894,88,77 83, r] Y[e,, ea, 1,, the decoding process will mistakenly think p4 has relation r with P3. However, this situation should be rare, and none is found in the tested datasets.
with Ps and p2 shares end token with p3. Then, based on Y[84, 81, = [ea, e2,
B DECODING WITH MODEL'S PREDICTIONS
In this section, we will detail the decoding process for models' predictions. The process described in Section 2.2 is not directly applicable to models' predictions since spans may conflict with each matrix Ys from Eq.
follow previous work (Yu et al. 2020b) to first filter out spans whose scores are less than 0.5: for the left spans, we sort the spans based on
5 With prediction
their scores, then choose spans in descending order and make sure the span has boundary clash with chosen spans. For relational extraction, we first decode all spans, then we get a binary matrix
'lll IE tasks forbid span boundary clashes.
15

---

Under review as a conference paper at ICLR 2023
YR([,,,, then pair spans to check whether they form relations. Take two spans (1) and ($2, 62) for instance, if YR(S1, [81, $2, r| Yx(ei, E2, (,f1,e2.1 r = 1, we claim the first span has relation r with the second span. For the RE task, pair all entity spans to check if they form relations; for the EE task, we pair the trigger spans and argument spans to check if they form a role relationship; and for the joint IE task, pair entity spans check if they form relations, we pair the trigger spans and entity spans (because all argument spans are entity spans) to check if they form role relationship.
YR =YR
C EXPERIMENTAL. SETTINGS
In this section, we describe all experimental settings in detail, such as the statistics of datasets, baseline models, and more implementation details.
C.I DATASETS
We conduct experiments on 10 single datasets and 2 joint IE datasets, and we detail the statistic of all datasets in Table
#Train
#Dev
#Test
#Ents (#Types)
#Rels (#Types) #Evts (#Types)
CoNLLO3
14,041
3,250
3,453
35.1k (4)
OntoNotes
59,924
8,528
8,262
104.2k (18)
ACE04
6,297
742
824
27.8k (7)
ACE05-Ent
7,178
960
1,051
31.7k (7)
GENIA
15,038
1,765
1,732
56.0k (5)
ACE0S-R
10,051
2,424
2,050
38.3k (7)
7.1k (6)
ACE05-R+
10,051
2,424
2,050
38.3k (7)
7.7k (6)
SciERC
1,861
275
551
8.1k (6)
4.6k (7)
SciERC+
1,861
275
551
(6)
5.5k
ACEO5-E
17,172
923
832
34.5k (7)
5.9k
5.1k (33:22)
ACEOS-E+
19,204
901
676
54.7k (7)
8.7k (6)
5.3k (33:22)
ERE-EN
14,722
1,209
1,163
46.2k (7)
5.9k (5)
7.3k (38:21)
Table 5: Datasets statistics. "#Types" denotes the number of classes. Note that "#Types" in the last column tasks, which are flat NER, nested NER, RE and EE from top to bottom. For the joint IE setting, the "ACE05-E+" and "ERE-EN" are used. In the RE block, datasets following mean that each symmetric relational instance is regarded two directional instances, leading to more relations.
(#event types; #role type) pairs. Every block represents datasets of different
Named entity recognition. We perform experiments on both flat and nested NER benchmarks. In flat NER, adopt CoNLLO3 (Sang & Meulder 2003) and OntoNotes (Pradhan et al. 2013) datasets. nested NER, we experiment on ACE04 (Doddington et al. 2004), ACEOS (Walerr al., 2006) and GENIA (Kim al. 2003). To distinguish ACE05 dataset used in other tasks, name ACE0S in named entity recognition as ACE05-Ent. Specifically, we use the same prepro- cessing and splitting procedure on nested datasets as Yan et al. (2022), for they fix some annotation problems to unify different versions of these datasets and make strictly fair comparison.
Relation extraction. We conduct experiments on two relation extraction datasets, ACE05 (Walker et al., (006)) and (c (Luan et al. 2018). The ACEOS dataset, named as ACE05-R in our paper, is collected from various domains, such as newswire and online forums. The SciERC dataset provides entity, coreference and relation annotations from AI conference/workshop proceedings.
16

---

Under review as a conference paper at ICLR 2023
We follow the data preprocessing in Luan et al. (2019) to split ACE05-R and SciERC into train, dev and test sets.
In typical RE, it is crucial to distinguish which entity comes first (head entity) and which comes next (tail entity). As for symmetric relational instance, the relation exists from both head-to-tail and tail-to-head directions. There one such relation type in ACE05-R and two in SciERC. Some papers (Wang et al. [222]] Ye et al. 2022) regard each symmetric relational example as two directed relations, while others regard them as one relation. We find that this setting will hugely influence the performance. Therefore, we name this setting Symmetric Relation Extraction and name the corresponding datasets ACE05-R* R and SciERC*
Event extraction. We evaluate UTC-IE on two widely used event extraction datasets, ACE2005
Doddington et al., (004) and ERE (Song et al. (015)). Following the prior preprocessing step Wadden et al.] [019;] Lin et al. 2020 Lu et al. 2021) on them, we obtain three datasets, ACE05- E, ACEOCCO--B and ERE-EN. ACE0S-E+ E+ additionally takes relation arguments, pronouns and multi- token event triggers into consideration compared with ACEOS-E. We use the same train/dev/test split as Lu et al. (2021) for all datasets to ensure fair comparison. Furthermore, we still use ACE05-E+ and ERE-EN on joint IE, for they have annotations on all IE tasks.
C.2 BASELINES
TANL (Paolini et al. 2021) and UIE et al. 2022) are both unified information extraction models in the generative way, with different input and output formats. TANL uses T5-base as the backbone model, while UIE uses T5-large. We compare our model with them in every IE task. For TANL, we report single-task results for our model is trained under each task. For UIE we report results with pre-training, which have better performance. In addition to these two baselines, each task also compares with a series of recently proposed task-specific methods as follows.
CNN-IE is the baseline model we design to prove the necessity of PlusAttentoon. The only differ- ence between CNN-IE and UTC-IE is the former ignores the Altsntttntion in Figure We tune the number of CNN layers in CNN-IE from to 6, and the best results are reported.
Named entity recognition. We compare our model's performance on NER with several recently proposed NER methods.
2021a formulates unified NER model as entity span sequence generation task. They use BART large the pre-trained model.
BART-NER
Yan et al.
W'NER (L et al. 2022) formulates unified NER model as word-to-word classification
task. The model employs BioBERT GENIA and BERT-large on other datasets.
BS (Zhu & Li, 2022): authors use span-based NER model as baseline and propose bound- ary smoothing as regularization technique improve model performance. It leverages RoBERTa-base as the base encoder.
CNN-NER (Yan al. (022) utilizes CNN model local spatial correlations between spans and surpass recently proposed methods on nested NER. We report results using RoBERTa-base model.
Relation extraction. For relation extraction, we compare our model with several SOTA models.
UniRE (Wang et al.
2021 jointly extracts entities and relations using table containing
all word pairs.
PURE (Zhong & Chen 1 adopts pipeline approach to solve NER and RE indepen- dently, using distinct contextual representations for entities and relations.
PFN (Yan et al. 2021b) claims that some information should be shared between named entity recognition and relation extraction, while other information should be independent. They propose PFN to model two-way interaction (partition and filter) between two tasks.
PL-Maarer Yee et al. 2022): authors consider interactions between spans and propose PL-Marrker by strategically packing the markers in the encoder.
17

---

Under review as a conference paper at ICLR 2023
Table 6: Overall pre-trained model on all IE baselines. Abbreviations before "L" denote pre- pre-trained model names. Specifically, "BA" means BART, "BE" means BERT, "RoB" means RoBERTa, "ALB" means ALBERT, "DeB" means DeBERTa. The letters after "_" means the size of the model, such as base model ("b"", large model ("I"), xx-large model ("xxl").. The number of parameters of each pre-trained model follows: BE-b 10M), BE-I (340M), RoB-b (125M), ALB-xxl (233M), DeB-I (390M), T5-b (220M), T5-1 (770M), BA-I (406M).
Named Entity Recognition
CoNLLO3
OntoNotes
ACE04*
ACE05-Ent
GENIA
BART-NER Yan et 2021a
BA-I
BA-I
BA-1
BA-I
BA-I
TANL. (Paolini et al. 2021
T5-b
T5-b
T5-b
T5-b
W (Li et al. 2022
BE-1
BE-1
BE-1
BE-1
BioBERT
UIE Lu
2022)
T5-1
T5-1
T5-1
BS 2hu Li 2022
RoB-b
RoB-b
RoB-b
RoB-b
CNN-NER Yan et al.
2022
RoB-b
RoB-b
BioBERT
UTC-IE
RoB-b
RoB-b
RoB-b
RoB-b
BioBERT
Relation Extraction
ACE05-Ro... Rocrt
SCE05-Ratbert
SciERC
TANL
Paolini al. 2021
T5-b
PURE
ZZoong Chen 2021
BE-b
ALB-xxl
SciBERT
PFN
(Yan et al. (2021b)
BE-b
ALB-xxl
SciBERT
UIE
Lu ct al. 2022)
T5-1
T5-1
UTC-IE
1E
BE-b
ALB-xxl
SciBERT
Symmetric Relation Extraction
ACE05-R
SciRC
UniRE (Wang et al.
2021
BE-b
SciBERT
PL-Marker (Ye al. 2022
BE-b
SciBERT
UTC-IE
BE-b
SciBERT
Event Extraction
ACE05-E
ACE05-E+
ERE-EN
TANL (Paolini et al.
2021
T5-b
TEXT2EVENT Lu
2021
T5-1
T5-1
T5-1
UIE Lu al.
T5-1
DEGREE (Hsu ct al. 2022)
BA-1
BA-1
BA-1
UTC-IE
DeB-1
DeB-1
DeB-1
Joint IE
ACE05-E+
ERE-EN
OnelE Ln et al. 2020
BE-1
BE-1
FourIE Nguyen 2021
BE-1
BE-1
UTC-IE
BE-1
BE-1
18

---

Under review as a conference paper at ICLR 2023
Previous models mentioned above different RE datasets. Specifically, UniRE and PL-Marker regard symmetric relations as two directed relations, while other work does not. Besides, these two models utilize
context.
Event extraction. Generative methods are popular in recently proposed event extraction papers.
TEXT2EVENT (Lu ct al. 2021) is sequence-to-struuuuse model which outputs a tree-like event structure with given input sentence. The model uses T5-large the base model.
DEGREE (Hsu et al., 2022) leverages manually designed prompts generate event records in natural language. We report the end-to-end performance of DEGREE instead of the pipeline way. The model leverages BaRT-large as encoder-decoder.
Joint IE. There are only two previous models that consider the joint IE in ACE05-E and ERE-EN datasets.
OnelE (Lin et al. 2020) proposes an end-to-end IE model, which employs global features and type dependency constraint decoding step.
FourlE (Nguyen et al. 202T) further improves the model by incorporating interaction de- pendency representation level and label level.
For fair comparison, we list the pre-trained model used for all baselines and our model on every IE dataset in Table When choosing our pre-trained language model different IE tasks' datasets, we pick the same pre-trained model as the most recently published papers, such BioBERT for GENIA and RoBERTa-base for other NER datasets. For RE and joint IE tasks, we choose the same pre-trained model previous work. For tasks where previous work applied generative pre- trainee model, we choose pre-trained model that has a similar size. For example, in event extraction, we use DeBERTa-large whose number of parameters is 390M, which is closest to BART-large and T5-large used by previous EE papers.
C.3 EVALUATION METRICS
We report micro-F1 on all tasks:
Entity: an entity is correct if its entity type and span offsets match a golden entity. We use "Ent." to represent entity through all tables.
Relation: a relation correct if its type and its head and tail entities are correct, and the offsets and type of entities should also match the golden instance. We use "Rel." to represent relation F1 through all tables.
Event trigger: a trigger is correct if its span offset and event type is correct. We use "Trig." to represent trigger F1 through all tables.
Event argument: an argument is correct its span offset, event type and role type all match the ground truth. We use "Arg." to represent argument F1 through all tables.
C.4 PYPER-PARAMETERS
used in each dataset are listed in Table AdamW opti- mizer (Loshchilov & Hutter 2019) with weight decay le-2 for all datasets. Experiments are con- ducted five times with five different random seeds. We report the performance on test sets based on the model which achieves the best dev results in each dataset. For NER, the best results are calcu- lated by the entity Fl; for RE, the best results are calculated by the sum of entity Fl and relation Fl; for EE, the best results are calculated by the best argument Fl; for joint IE, the best results are calculated by the sum of relation Fl and trigger F1.
The detailed
D COMPLETE RESULTS
We present the complete results of UTC-IE and that without Plusformer's in Table
19

---

Under review as a conference paper at ICLR 2023
Table 7: The hlper-pprameen used in each dataset.
Named Entity Recognition
CoNLL03
OntoNotes
ACE04*
ACE05-Ent
GENIA*
# Epochs
30
10
50
50
5
Learning Rate
le-5
le-5
2e-5
2e-5
7e-6
Batch Size
12
12
48
48
8
# Plusformer Layers
2
2
2
Biaffine Dimension
200
200
200
200
200
Feature Dimension
32
100
100
100
100
Warmup Ratio
0.1
0.1
0.2
0.2
0.1
Relation Extraction
ACE05-Raass
ACE05-R, (.E05--R.bbert
SciERC
ACE05-R
SciERC
# Epochs
100
100
70
100
Learning Rate
3e-5
3e-5
3e-5
3e-5
3e-5
Batch Size
32
32
16
32
16
# Plusformer Layers
3
3
3
Biaffine Dimension
200
200
200
Feature Dimension c
200
200
200
200
200
Warmup Ratio
0.1
0.1
0.1
0.1
0.1
Event Extraction
ACE05-E
ACE05-E+
ERE-EN
# Epochs
70
70
70
Learning Rate
le-5
le-5
le-5
Batch Size
32
32
# Plusformer Layers
3
3
3
Biaffine Dimension d
300
300
300
Feature Dimension
150
150
150
Warmup Ratio
0.1
0.1
0.1
Joint IE
ACE05-F+
ERE-EN
# Epochs
70
Learning Rate
lc-5
3c-5
Batch Size
12
12
# Plusformer Layers
3
3
Biaffine Dimension
300
300
Feature Dimension c
150
150
Warmup Ratio
0.1
0.1
20

---

Under review as a conference paper at ICLR 2023
Table 8: Completed results for precision (P), recall (R) and FI (F) of UTC-IE on different tasks. Bold results represent the most improved metrics on UTC-IE without Plusformer between precision and recall.
Named Entity
CoNLL03
OntoNotes
ACE04*
ACE0S-Ent
GENIA
Extraction
P
R
F
P
R
F
P
R
F
P
R
F
P
R
F
UTC-IE
93.4 93.6 93.5
91.7
91.9
91.8
87.3
87.7
87.5
86.8
88.8
87.8
81.6
79.4
80.5 79.3
-Plusformer
93.0
93.0
93.0
91.0
91.8 91.4
86.8
86.2
86.5
85.8
87.4
86.6
81.6
77.2
Relation Extruction
ACEOS-R,
ACE05-R
Rabert
SciERC
Ent.
Rel.
Ent.
Rel.
Ent.
Rel.
P
R
F
P
R
P
P
R
F
P
R
F
P
R
F
P
R
F
UTC-IE
88.7 89.0 88.8
70.1 60.5 64.9
89.3
90.5
89.9
70.4
65.4
67.8
68.0
70.1
69.0
43.6
34.9
38.8
Plusformer
88.1
88.9 88.5
66.7
58.4
63.3
89.7
90.0
89.8
70.2
62.7
66.2
67.4
68.9
68.1
43.0
32.7
37.1
Symmetric
ACE05-R*
SciERC+
Relation
Ent.
Rel.
Ent.
ReL.
Extraction
P
R
F
P
R
F
P
R
P
R
F
UTC-IE
90.0
90.5
90.2
69.3
65.8
67.5
68.5
71.5
70.0
45.7
39,8
42.5
Plussfomer
87.5
90.5
89.0
67.3
64.0
64.6
68.0
69.6
68.8
43.5
36.2
39.5
ACE05-E
ACEOS-E+
ERE-EN
Event
Ent.
Rel.
Ent.
Rel.
Ent.
Rel.
Extraction
P
R
F
P
R
F
P
R
F
P
R
F
P
R
F
P
R
F
UTC-IE
70,9 76,2 73.5
55,5
57.6
56.5
70,8 76.1 73.4 57.8
57.6
57.7
58.1
62.5
60.2 54.5
50.7
52.5
Plusformer
70.1 76.0 72.9
52.5
58.7
55.4
70.5
75.5
72.9
55.6
57.7
56.6
56.0
63.0
59.3
52.3
50.3
51.3
E
SPEED COMPARISION
We test the speed of other models through their released code. For models, such as OnelE (Lin al. 2020 DEGREE (Hsu et al. (022)) and PL-Marke Ye et al., 2022). they also released a trained model along with their code, and we used their released model test the inference speed. For UIE (u et al. 2022) and BS (Zhu & Li 2022). we trained model with their code. The speed test is conducted one RTX 3090 GPU
as 32 for all models (if the model goes out of memory, we choose the largest batch size that can accommodate the GPU); the test corpus is the test set of each datasets. The speed is measured by the number of sentences in the test divided by the number of seconds that elapsed. And each inference is repeated three times, the average speed reported.
the batch size
The speed comparison can be roughly categorized into two kinds. The first kind is the comparison with previous universal IE models, namely OnelE (Lin et al. 2020) and UIE (Lu al. (022).. and results depicted in Table Compared with UIE in five chosen datasets, UTC-IE is x19.7 faster and improves performance by 1.86 averagely. Besides, for the joint IE task. UTC-IE is 18.4 times faster than OnelE and improves performance by 2.72 average. The second kind is the comparison between UTC-IE and SOTA models targeted for each IE task, and results are presented in Table 9 Compared with previous SOTA models, the average performance increments for entity F1, relation Fl and argument Fl are 0.31, 0.94 and 2.15. In the meantime, UTC-IE speeds up for x x1.0, x5.5 and x101.9 averagely.
In short, using UTC-IE for IE tasks can not only substantially enhance the performance in most cases, but also significantly speed up the inference speed in almost all datasets.
P--Marre used two-stage pipeline to conduct prediction. Therefore, the time is measured by the total seconds elapse finish stages.
21

---

Under review as a conference paper at ICLR 2023
Table 9: The Fl and inference time comparison on UTC-IE and currently SOTA models on each IE task. "Ent.", "Rel." and "Arg." denote F1 of corresponding test sets. "Speed" is measured in "sentence/s" on inference procedure. Improvement shows the changes in performance and speed.
CoNLLO3
ACE0S-Ent
Named Entity Recognition
Ent.
Speed
Ent.
Speed
BS (hu & Li 2022 UTC-IE Improvement
93.39
265.6
87.20
355,4
93.45
285.3
87.75
344.3
+0.06
x1.1
+0.55
ACE05-R*
SciERC+
Symmetric Relation Extraction
Rel.
Speed
Rel.
Speed
PL-Marker Ye al. 2022
66.5
30.1
41.6
26.0
UTC-IE
67.47
173.8
42.51
134.7
Improvement
+0.97
x5.8
+0.91
x5.2
ACE05-E+
ERE-EN
Event Extraction
Arg.
Speed
Arg.
Speed
DEGREE Hssu et al. 2022
56.3
0.8
49.6
1.2
UTC-IE
57.68
88.1
52.51
114.6
Improvement
+1.38
x107.4
+2.91
x96.3
Table 10: Ablation Study for span extraction. Underlines mean the most dropped factor. the CNN-IE surpasses previous SOTA performance.
means
CoNLLO3
ACE05-Ent ACE05-Riard Ent.
SciERC
ACE05-E+
ERE-EN
Ent.
Ent.
Trig.
Trig.
UTC-IE
93.45
87.75
88.8212
69.03aa
73.446s
60.20g4
CNN
93.1011
87,3922
88.7122
68.35
72,9834
58.91a1
position embeddings
93.2511
87.5334
88.7320
68.69ss
73.12es
59.0370
axis-aware
93.2310
87.5927
88.7919
68.5348
73.29
59.56g9
CNN-IE
93.3216
87.45
88.70
68.11
73.0499
59.47
F ABLATION STUDY
For ablation, will choose datasets for each IE task study the effect of each component in Plusformer. We separately list the performance for span extraction (including entity extraction NER and RE, trigger extraction EE) in Table I0 and relational extraction (including relation extraction in RE and argument extraction in EE) in Table [T] Besides, we also study how the performance varies with the change the number of Plusformer layers in Figure 10
F.1 CNN
Based on our ablations in Table and Table 11 the CNN module in Plusformer contributes most the performance enhancement. To reveal why CNN so effective in both span extraction and relational extraction, we first present an intuitive example in Figure to show how CNN helps to extract entities and relations the RE task. Like in Figure 3 a), for NER, the entity e2 can interact with entity and relation F1 through CNN. Besides, for RE, CNN can contribute in two ways. On the one hand, CNN helps the relational token pair to directly gather information from its constituent entities, like the rı in Figure b).. On the other hand, the start-to-start and end-to-end relational token pairs, like two 72 cells, can directly interact with each other through CNN.
To quantitatively present the effectiveness of CNN in UTC-IE, we propose further ablations to show how the distance between the relational token pair and its constituent spans affects the relational F1, and how the distance between start-to-start and end-to-end token pairs affects the relational F1.
22

---

Under review as a conference paper at ICLR 2023
Table 11:
Ablation Study for relational extraction. Underlines mean the most dropped factor.
means that the CNN-IE surpasses previous SOTA performance.
ACE05-Ri Rbort
SciERC Rel.
ACE05-E+
ERE-EN
Rel.
Arg.
Arg.
UTC-IE
64.94a3
38.7798
57.687a
52.5195
CNN
63.55sa
37.5683
56,7499
51.59pp
position embeddings
64.29sa
37.98g9
57.02so
52.0670
axii--aare
63.91
37.7683
56.87pa
51.926p
CNN-IE
64.67 26
37.64
56.97
51.78
David Perkins is
village doctor in California
David Perkins
village doctor in Califoria
David Perkins
T2
T2
C1
F2
Perkins
e2
12
P2
village
Is
Is
doctor
village
village
doctor
doctor
PER-JOB
California
California
CNN Receptive
(a) Center on entity
(b) Center on relation and 1'2
Field
Figure 3: An intuitive example of the influence of CNN on span extraction and relation extraction.
Furthermore, we conduct experiments on UTC-IE with different kernel sizes and choose the most proper size.
F.1.1 DISTANCE BETWEEN THE RELATIONAL TOKEN PAIR AND ITS CONSTITUENT SPANS VS. RELATIONAL
In this section, we will show how the relational F1 (relation Fl in RE and argument F1 in EE) will change when the distance between the relational token pair and its constituent spans varies. For two spans (81, ei) and (s2, e2) (we ignore their diagonally symmetric counterparts, since they will not affect the calculation here), the span relation from (81,61) to (#2, e2) represented by two token pairs (2) and (e1, ez). The distance between the two token pairs and its constituent spans is calculated as follows
d= max 82 - e1l. 1061-)22000 +
where the distance d is named as "Span-Rel-Spee Distance", represents the longest distance be- tween the relational token pairs to their constituent spans. The relation between and the relational F1 is shown in Figure 4 Without CNN, the performance for extracting relations between nearby constituent spans will drop sightly, while less affected for further ones, which proves that CNN is effective for exploiting local dependency to predict relations.
F.1.2 DISTANCE BETWEEN START-TO-STANE AND END-TO-ENS TOKEN PAIRS VS. RELATIONAL Fl
As shown Figure b), b), the distance between the start-to-start and end-to-end relational token pairs is small, the CNN should be helpful. To verify this assumption, we first define the "Inner Relational Distance" as follows, for two spans ($1, G1) and (s2, e2), the relational token pairs are ($1, 82) and (e1, e2), then the distance between two relational token pairs is calculated follows
d max(e1 81, e2 52) + 1,
(10)
23

---

Under review as a conference paper at ICLR 2023
Span Rel Spon Distance Vs. Rel. F1
Span-fielSSpn Distance Vs. Rel. F1
60
60
50
Del.
40
Plusformer
30
Plusformer
W.o. CNN
W.o. CNN
20
1-1
13-15 =16
6-7 10-12 13-15 11
(295) (237) (156) (141) (80) (91) (43) 147) (28)
(33)
(28)
(59)
(1111 (118) (168) (145) (136) (93) (116)
Span Rel Span Distance
Span-Rel Ral-aan Distance
(Number relations gold)
(Number relations gold)
(a) ACE0S-R
(b) SciERC
Spar Rel Span Distance Vs. Arg.
Span Rel Distance
70
60
50
A0g.
50
40
Plusformer
Plusformer
20-
1-1
2-2
3-3
4-4
6-7
8.9
10:12 13-15
216
1-1
2.2
3-3
4-4
6-7
8.9
10-12
13-15 216
176)
(104)
(92)
(69)
(32)
(87)
(61)
(52) (41)
(72)
(161)
(148) (109)
(77)
(66)
(74)
(41)
(54)
(40)
(41)
Span Span
Span Span
(Number arguments in gold)
(Number arguments gold)
(c) ACEOS-E+
(d) ERE-EN
Figure 4: Distance between the relational token pair and its constituent spans (Span-Rel-Sparl Dis- tance) VS. relational F1 when with or without CNN in Plusformer. The upper and lower figures are for RE and EE tasks, respectively. From the results. it is clear that without CNN, the performance of Plusformer will drop when extracting relations (relation for RE and argument for EE) between nearby spans, while the performance less effected for relations with further constituent spans. We conjecture this is because the receptive field of CNN is limited to relatively small distance.
24

---

Under review as a conference paper at ICLR 2023
Inner Relational Distance Vs. Rel. F1
Inner Relational Distonce Vs. Rel. F1
45
C
40
Del
35
20
Plusformer
30
Plusformer
W.o. CNN
W.o. CNN
25
2.2
3.3
4.4
25
1.
2.2
3.3
5.5
6.6
2
(91.))
[193)
(28)
181
(4)
(84)
(365)
[241)
(125)
(74)
(43)
(42)
Inner Relational Distance
Inner Relational Distance
(Number relations gold)
(Number relations gold)
(a) ACE0S-R
(b) SciERC
Relational Distance Arg.
Inner Relational Distance Arg.
60
40
40
20
Plusformer
Plusformer
W.o. CNN
W.o. CNN
2-2
2-2
(547)
(97)
(24)
(15)
(3)
(702)
(89)
(12)
(16)
(11
Inner Relational Distance
inner Relational Distance
(Number arguments gold)
(Number of arguments in gold)
ACE0S-E+
(d) ERE-EN
Figure 5: Distance between two relational token pairs of the same span pair (Inner Relational Dis- tance) VS. relational F1 when with or without CNN Plusformer. The upper and lower figures are for RE and EE tasks, respectively. Since almost all spans are of length of less than 5, CNN is valuable to model the interaction between start-to-start and end-to-end relational pairs.
where reveals the distance between start-to-start and end-to-end token pairs, and it is actually decided by the max constituent span length. And its relation with the relational F1 is shown in Figure It is clear that, most of the start-to-start token pair is near to its end-to-end token pair, and CNN takes advantage of this adjacency to make better predictions.
F.1.3 CNN KERNEL SIZE VS. F1
We study the relation between the kernel size of CNN and Fl performance in Figure 6 We observe that CNN with kernel size 3 obtains the best performance on almost all datasets and tasks. Specif- ically, reducing CNN kernel size to 1 significantly harms the performance will lose the capability of interacting with neighboring token pairs. In contrast, F1 also slightly de- creases with larger CNN kernel size. We presume that CNN with larger kernel size may introduce more noise and harm performance. Therefore, we choose kernel size for all datasets.
all datasets, for CNN
F.2 Is CNN ALL WE NEED?
Since CNN is so effective in the Plusformer, it is natural to ask whether it is enough only to use CNN. Therefore, we conduct experiments on models without the plus-shaped self-attention and named this model CNN-IE. We conduct experiments for CNN-IE in six datasets, and results are listed in Table I0 and Table [I] Only with the CNN module can the model achieve SOTA or near SOTA performance in all six datasets, which depicts the effectiveness of the proposed token-pain decomposition and CNN module. However, still lags behind the UTC-IE model, which reveals the necessity of the PlusAttention.
F.3 POSITION EMBEDDINGS
The RoPE embedding aims help token pairs be aware of the spatial relationships between each other, and the triangle position embedding tries to enable spans to be informed of their areas in the feature map. From Table 10 and Table [1] the position embeddings enhance the span extraction
25

---

Under review as a conference paper at ICLR 2023
Ent. F1
Ent. F1
93.50
87.50
93.25
87.00
CNN Kernel Size
CNN Kernel Size
(a) CoNLLO3
(b) ACE05-Ent
89.00
Frt:
Rei FI
65
Frt. F1
Rel.
Rel F1
69.00
Ent
88.80
38
88.60
68.00
37
Kernel
Kernel
(c) ACEOS-R
(d) SciERC
Ent. F1
Rei.
5B
Ent. F1
Ret. F1
Arg Fl
73.00
60.00
72.00
(e) ACEOS-E+
(I) ERE-EN
Figure 6: The performance varies with the kernel size of CNN. NER, RE and EE results are listed from top to bottom. CNN with kernel size 3 has the best performance over almost all datasets.
and relational extraction for 0.39, 0.64 averagely. Besides, Figure 7 we show that the position embeddings can help the model exploit the distance bias improve the performance of relational extraction.
F.4 AXIS-AWARE PLUS-SHAPED CELF-ATTENTION
Lastly, we study the effect PlusAttention. We present an example to delineate why the axis- aware is valuable for span extraction and relational extraction Figure g From Figure axis- aware should be worthwhile no matter what the task is, span extraction or relational extraction. As expected, from Table I0 and Table [I if discard the axis-aware in Plusformer, the average performance of span extraction and relational extraction diminish 0.28 and 0.86, respectively, which reveals the necessity of axis-aware in the PlusAttention module.
Besides, we show two case studies of the plus-shaped attention in Figure 9 from the test dataset of ACE2005-Em and ACE2005-R. Both cases put larger attention scores on informative token pairs.
The sentences are
F.5 NUMBER OF PLUSFORMER Vs. F1
We study the relation between the number of Plusformer layers and Fl performance in Figure 10 For the NER datasets, we use two layers Plusformer, and for the RE and EE we use three.
G COMPARISON WITH GLAD
Jiang et al. (220)) claims that many NLP tasks can be regarded as the span prediction and prediction of relations between pairs spans (named as span extraction and relational extraction in our paper), which is conceptually similar our insights. However, our work is fundamentally distinct from theirs on both formulation and model architecture. Jiang et al. (2020) classify various NLP tasks into two separate tasks and design different modules for them. To contrast, unify all traditional IE tasks into a single formulation, namely token-pair classification. Therefore, we only need one model for all tasks. Besides, [iing al. (2020) simply use the concatenation of the start and end
26

---

Under review as a conference paper at ICLR 2023
Span-Rei-Span Distance Vs. Rel. F1
Span Rel Span Distance Vs. Rel. F1
60
60
40
40
Plusformer
Plusformer
30
20
W.o. Pos. Emb.
W.o. Eos.Emb.
13-15
(295) (237) (156) (141) (80) (91) (43) [47) (28)
133)
(281
(591
(1211 (118) (168) (145) 1136) (93) (116)
Rela--Re--pann Distance
Reaa--Rl--Sann Distance
(Number of relations in gald)
[Number of relations gold)
(a) ACE05-R
(b) SciERC
Span-Rel-Span Distance Vs. Arg. F1
Span-Rel-Span Distance Arg.
70
60
Arg.
50
ArD.
40
40
Plustormer
Plusformer
Pos Emb.
Pos.Emb.
30
1-1
2-2
3-3
4
5-5
6-7
6-9 13-15 =16
1-1
2-2
3-3
444
5-5
6-7
8.9 10-12 26
176)
(104) 192)
(69)
(32)
(87)
(62)
[52] (41)
[72)
(161)
(148) (109) (77)
[66)
(74)
[41]
(54)
(4D) [41]
Span-Rel-Span
Span-Rel-Span
(Number arguments gold)
(Number arguments gald)
(c) ACE05-EB E+
(d) ERE-EN
Figure 7: Distance between the relational token pair and its constituent spans Span-Rel-Span Dis- Iance) VS. relational Fl when with or without position embeddings in Plusformer. The upper and lower figures are for RE and EE tasks, respectively. Without position embeddings, the relational performance is lower almost in all Sela-Rel--paan distances. We presume this is because, with position embedding, Plusformer can exploit the distance inductive bias determine the relations.
Table 12: Results comparison between a similar work GLAD and UTC-IE on NER, RE, SRL and ABSA. We leverage BERT-base as base model for fair comparison. GLAD performs NER and RE jointly on WLP dataset, and report them separately. We use the same settings as theirs. means that the UTC-IE without Plusformer surpasses previous SOTA performance.
NER WLP
RE
OIE 0IE2016
SRL OntoNotes
ABSA
WLP
SemEval14
GLAD Jiang et al. 2020
78.1
64.7
36.7
83.3
70.8
UTC-IE -Plusformer
82.51
68.57+54
37.90,7
84.90
73.53
79.47 4-2
66.07 1:7
36.73
83.75
83.75
71.80
token representations to represent a span, and for relations, they concatenate the head and tail span representations. Therefore, in their work, the interaction between spans weak. In work, obtain the feature matrix of all token pairs and add well-designed Plusformer module on top of all token pairs, where token pairs can interact with others thoroughly.
In order to prove the superiority of our reformulation and UTCCIE model, we make a fair comparison on several tasks from the GLAD benchmark (Jiang et al.) 2020). We choose 3 additional IE tasks, including Open Information Extraction (OIE). Semantic Role Labeling (SRL) and Aspect Based Sentiment Analysis (ABSA), and NER and RE. We use WLP (Hashimoto al. 2017) on NER and RE, OIE2016 (Stanovsky & Dagan 2016) on OIE, OntoNotes (Pradhan et al. 2013) on SRL and SemEval14 (Pontiki et al. 2014) on ABSA. The detailed experimental settings are the same as those in GLAD, to ensure fair comparison. Results are present in Table [2]
The table shows that UTC-IE outperforms GLAD on all chosen tasks exceedingly, with +2.76 improvement on average. Moreover, we observe that UTC-IE without Plusformer also surpasses GLAD benchmarks on all tasks with +0.84 improvement on average, which proves the superiority of our unified reformulation.
27

---

Under review as a conference paper at ICLR 2023
Pittsburgh
hubs built
Pittsburgh
US
hubs built
Pittsburgh
Pittsburgh
e1
of
X
US
Airway
Airway
hubs
X
hubs
Spans with the same start token with e
Potential head span for Potential tail span for
Spans with the same end token with
x Spans that clash with entity
(a) Span Extraction
(b) Relational Extraction
Figure 8: Examples to show why axis-aware is meaningful for IE tasks. In the left figure, spans in e's vertical direction share the same end token as e except for spans in the lower triangle, since they clash with in the back (because "Airway" the end token of but the start token for these spans); spans in e's horizontal direction have common start token as e, but not spans in the lower triangle, because they clash with in the front (since "US" the token of but the end token for these spans). Therefore, both the axis-aware and triangle position embedding are crucial for spans to figure out their relationships with each other. the right figure, for a relational token pair, the spans from its horizontal direction must be the head span, while the tail span must come vertically. Thusly, axis-aware informative for relational extractions.
Jord:
Din- berg
Wesb
Prthburgh
Afrway
habe
hes the
#irdagg
Iest easa
Jvm-
daw-
bug
lang
chedw
Wint
Owettrr
Hg
1dle
tw
Alroaye
of Airway's hubs
US
Newa Westchater
Ginburg
Wistchoter
ORG-AFF
(b.1) PlusAttention RE's
(b.2) PlusAttention RE's
map
end-to-end
Figure 9: Two case studies of the PlusAttention. The horizontal and vertical attention scores are from the horizontal and vertical self-attentions of last layer of Plusformer. The center cells are with two colors, one for the horizontal attention scores and the other for the vertical attention scores. For NER, the center cell attends more other entities. And for RE, the center relational cell attends more on its constituent entities.
28

---

Under review as a conference paper at ICLR 2023
Ent. Fl
86.00
Ent. F1
93.25
87.50
Plusformer Layers
Plusformer Layers
(a) CoNLL03
(b) ACB0S-EII
F1
Rei Fl
Enc. F1
89.00
Ent.
Rel
Rel
88.75
68.00
# Plustormer Layers
Plustormer Layers
(c) ACE05-R
(d) SciERC
Ent. F1
R1
61.00
Enc. F1
Rel F1
5B
Arg. Fl
Thg..
60.00
73.00
59.00
Plusformer Layers
Plusformer Layers (f) ERE-EN
ACEOS-EF E+
Figure 10: The performance varies with the number of Plusformer layers. NER, RE and EE results are listed from top to bottom. For the NER tasks, the performance peaks at the two layers Plus- former, and for RE and EE, the performance plateaus after three layers of Plusformer.
29


Reviews:
[{"review_id": "trowj2bCizN", "reviewer": "Reviewer_j1ou", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To address the information extraction (IE) setting for multiple tasks, the authors propose to formulate the IE problem as token-pair classification. The proposed formulation represents the spans and span-pair relations through token-pair interactions which is similar to table-filling approaches. The experiment results show benefits compared to prior methods on 10 datasets.", "summary_of_the_review": "Although the proposed approach is relatively simple and can be applied to multiple IE tasks, there are issues with the motivation and novelty of the work. ", "strength_and_weaknesses": "Strengths\n - The speed comparison in section 4.4 empirically supports the computational efficiency of the proposed model.\n\nWeaknesses\n - The motivation for the token-pair reformulation is not convincing as there is no fundamental challenge in unified IE that is being addressed.\n - There is limited novelty as the model combines several existing components (position embeddings, CNN) that are not specific to the token-pair formulation.\n - The setting of unified IE is limited as previous works (e.g. GLAD benchmark) included more tasks such as coreference, parsing and aspect-based sentiment analysis.\n - The SpanRel model (Jiang et al., 2020) is similar to this work in motivation and formulation, but the authors’ comparison is superficial and SpanRel is missing from the experiment results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Tic2QWNIBj", "reviewer": "Reviewer_7KpY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a unified information extraction (UIE) framework based on the token pair classification idea. The UIE solution allows modeling all types of IE tasks such as NER, RE and EE with just one unified model. Also the authors propose a Plusformer on top of the token-pair feature matrix for better feature learning. They show their method advances better efficacy and efficiency. The proposal of UIE simplified with a token pair scheme is technically sound, and also the experiments are solid. However, I'm afraid this paper may suffer from some key flaws, with respect to the topic of UIE, which I list bellow.\n\n\n1. Overall, this paper made limited contributions.\n\na. Token pair classification, aka., table filling/grid tagging, is one of the most typical and long-standing task modeling schemes for the information extraction jobs in NLP community.\nAlso, the paper claims to advance with unifying all IE tasks, yet the idea of token pair classification has been well explored in NER [2], RE [1] and EE [3].\nThus this paper leverages such a method for UIE, which in the perspective of technique, contributes less.\n\nb. The proposed Plusformer is simply the combination of CNN layer with axis-aware addition. The module is designed for modeling the local interaction. However as far as I know, this will fall short on modeling some of the long-range dependences between elements for especially the RE/EE tasks [5].\n\n\n\n2. Problematic/unconvincing motivations\n\na. The authors indicate the issues of the existing works: \"However, these models are either complex to be designed or time-consuming to decode\", which could be problematic. For example, generative seq2seq methods are very easy and straightforward, while graph-based methods enjoy high running efficiency due to high parallel architecture.\n\nb. The motivation and designing detail of the use of token pair in this paper is essentially the same as the recent work [2], while the authors seem to skip [2] without acknowledging it, which is unacceptable.\n\nc. Suspicious unified IE. I don't feel that the task modeling should be called as unified information extraction, since it seems different task types (NER/RE/EE) come with one specific task formalization. [4] models all different IE tasks with just one real single unified formulation under a well-designed structured extraction language/expression.\n\nAnd also the paper writing or technique description is not clear, e.g.,\n1) Y [s, e, t] = Y [e, s, t] , what does the [*] mean?\n2) how to solve the issue of overlapping elements in NER/RE/EE with your formulization?\n3) the task formulization lacks necessary examples for certain IE tasks.\n\n\n3. Another potential problem in the technique method part: one of the key reasons to propose UIE in IE community lies in the capability of knowledge transfer, i.e., pre-training once and transferring everywhere, to naturally support few-shot learning for downstream IE task. But the UIE in this paper seems to fail to well support this advantage; they more likely perform unified IE task modeling, and nothing more. This would limit the real merit of UIE.\n\n\n4. As an extension of token pair based IE method, this work didn't survey the existing works for this line, e.g., differences.\nAlso some critical existing works are overlooked [1,2,3].\n\n\n\n[1] Modeling Joint Entity and Relation Extraction with Table Representation. EMNLP 2014.\n\n[2] Unified Named Entity Recognition as Word-Word Relation Classification. AAAI 2022.\n\n[3] OneEE: A One-Stage Framework for Fast Overlapping and Nested Event Extraction. COLING 2022.\n\n[4] Unified Structure Generation for Universal Information Extraction. ACL 2022.\n\n[5] Graph Convolution over Pruned Dependency Trees Improves Relation Extraction. EMNLP 2018.\n", "summary_of_the_review": "See above.", "strength_and_weaknesses": "See above.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "73GcVPFCbUC", "reviewer": "Reviewer_TuyE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "One existing problem in information extraction is that it consists of a wide range of tasks. Previous attempts to unify all IE tasks with one architecture are either complex or time-consuming to decode. To this end, this paper proposes a simple yet effective paradigm for unified IE by interpreting all IE tasks as span extraction and span relation extraction. Specifically, the paper introduces Plusformer, which models the axis-aware interaction and location interaction on top of the token-pair feature matrix. Experiments on 12 IE tasks show that the proposed approach achieves SOTA performance. The proposed model is also much faster than prior unified IE models", "summary_of_the_review": "The paper proposed an interesting idea to unify all IE tasks into token-pair classification tasks and proposes a relatively new attention mechanism for token-pair feature matrix. Extensive experiments and ablation studies have been performed to demonstrate the superiority of the proposed method on performance and decoding speed. Overall, I believe this paper should be accepted.", "strength_and_weaknesses": "Strengths: \n* The paper proposes an interesting idea to unify all IE tasks into token-pair classification tasks.\n* The paper has performed extensive experiments on 12 IE tasks and showed the superiority of the proposed method on both performance and decoding speed.\n* The paper is overall well written.\n\nWeaknesses:\n* The paper proposed a new attention mechanism for token-pair feature matrix. It would be better if the authors could provide some visualization examples on axis-aware attention and location interaction attention. The authors could also provide qualitative examples of how this new attention mechanism helps with performance.\n* In equation (4), it is unclear the concatenation of Z_h and Z_v is along which axis.\n* In Section 4.3, the author mentioned that “the performance of UTC-IE_single is from trigger/argument F1 of EE”; however, the trig. (and arg. F1) score for UTC-IE_single on ERE-EN in Table 2 is 57.01 (and 48.29), while the trig. (and arg. F1) score for UTC-IE on ERE-EN in Table 1 is 60.20 (and 52.51). According to the appendix, it seems like the difference is due to the usage of different pre-trained models? If so, the authors might want to consider pointing out this in the main content. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject