UTC-IE: A Unified Token-pair Classification Architecture for Information Extraction
Year: 2023

Abstract:
Information Extraction (IE) spans several tasks with different output structures, such as named entity recognition, relation extraction and event extraction. Previously, those tasks were solved with different models because of diverse task output structures. Through re-examining IE tasks, we find that all of them can be interpreted as extracting spans and span relations. They can further be decomposed into tokenpair classification tasks by using the start and end token of a span to pinpoint the span, and using the start-to-start and end-to-end token pairs of two spans to determine the relation. Based on the reformulation, we propose a Unified Token-pair Classification architecture for Information Extraction (UTC-IE), where we introduce Plusformer on top of the tokenpair feature matrix. Specifically, it models axis-aware interaction with plus-shaped selfattention and local interaction with Convolutional Neural Network over token pairs. Experiments show that our approach outperforms task-specific and unified models on all tasks in 10 datasets, and achieves better or comparable results on 2 joint IE datasets. Moreover, UTCIE speeds up over state-of-the-art models on IE tasks significantly in most datasets, which verifies the effectiveness of our architecture.1

heading: 1 Introduction
text: Information Extraction (IE) aims to identify and classify structured information from unstructured texts (Andersen et al., 1992; Grishman, 2019). IE consists of a wide range of tasks, such as named entity recognition (NER), joint entity relation extraction (RE)2 and event extraction (EE) .
In the last decade, many paradigms have been proposed to solve IE tasks, such as sequence label-
∗Equal contribution. †Corresponding author.
1Code is available at https://github.com/yhcc/utcie. 2Joint entity relation extraction aims to extract both entities and relations. In our paper, we call it relation extraction (RE) for simplicity.
ing (McCallum and Li, 2003; Huang et al., 2015; Zheng et al., 2017; Yu et al., 2020a), span-based classification (Jiang et al., 2020; Yu et al., 2020b; Wang et al., 2021; Ye et al., 2022), MRC-based methods (Levy et al., 2017; Li et al., 2020; Liu et al., 2020) and generation-based methods (Zeng et al., 2018; Yan et al., 2021a; Hsu et al., 2022). The above work mainly concentrates on solving individual tasks, but it is desired to unify all IE tasks without designing dedicated modules, as tackling all IE tasks with one model can facilitate knowledge sharing between different tasks. Therefore, various attempts have been made to unify all IE tasks with one model structure. Wadden et al. (2019); Lin et al. (2020); Nguyen et al. (2021) encode all IE tasks’ target structure as graphs and design graph-based methods to predict them; Paolini et al. (2021); Lu et al. (2022) solve general IE tasks in a generative way with text-to-text or text-to-structure frameworks. However, graph-based models tend to be complex to design, while generative models are time-consuming to decode.
In our work, we creatively propose a simple yet effective paradigm for unified IE. Inspired by Jiang et al. (2020), we re-examine IE tasks and consider that all of them are fundamentally span extraction (entity extraction in NER and RE, trigger extraction and argument span detection in EE) or relational extraction3 (relation extraction in RE and argument role classification in EE). Based on this perspective, we further simplify and unify all IE tasks into tokenpair classification tasks. Figure 1 shows how each task can be converted. Specifically, a span is decomposed into start-to-end and end-to-start token pairs. As depicted, the entity “School of Computer Science” in Figure 1(a) is decomposed into indices of (School, Science) and (Science, School). As for
3In this paper, we use relational extraction to encompass the extraction of any kind of relationship or other interaction between spans, which as broader meanings than relation extraction.
4096

heading: Computer
text: detecting the relation between two spans, we convert it into start-to-start and end-to-end token pairs from head mention to tail mention. For example, in Figure 1(b), the relation “Author” between “J.K. Rowling” and “Harry Potter novels” is decomposed into indices of (J.K., Harry) and (Rowling, novels).
Based on the above decomposition, we propose a Unified Token-pair Classification architecture for Information Extraction (UTC-IE). Specifically, we first apply Biaffine model on top of the pre-trained language model (PLM) to get representations of token pairs. Then we design a novel Transformer to obtain interactions between them. As the plusshaped dotted lines depicted in Figure 1, token pairs in horizontal and vertical directions cover vital information for the central token pair. For span extraction, token pairs in the plus-shaped orientation are either clashing or nested with the central token pair, for example, e2 is contained by e1 in Figure 1(a); for relational extraction, the central token pair’s two constituent mentions locate in the plus-shaped orientation, such as in Figure 1(b), r is determined by e1 and e2. Therefore, we make one token pair only attend horizontally and vertically in the token pair feature matrix. Additionally, position embeddings are incorporated to keep the token pairs position-aware. Moreover, neighboring
token pairs are highly likely to be informative to determine the types of the central token pair, so we apply Convolutional Neural Network (CNN) to model the local interaction after the plus-shaped attention. Since the attention map for one token pair is intuitively similar to the plus operator, we name this whole novel module as Plusformer.
We conduct numerous experiments in two settings. When training separately on each task (named as single IE task), our model outperforms previous task-specific and unified models on 10 datasets of all IE tasks. When training a single model simultaneously on all IE tasks in one dataset (named as joint IE task), UTC-IE achieves better or comparable results than 2 joint IE baselines. To thoroughly analyze why UTC-IE is useful under the token-pair paradigm, we execute several ablation studies. We observe that CNN module in Plusformer plays a significant role in IE tasks by the abundant local dependency between token pairs after the reformulation. Besides, owing to the good parallelism of self-attention and CNN, UTC-IE is one to two orders of magnitude faster than prior unified IE models and some task-specific work. To summarize, our key contributions are as follows
1. We introduce UTC-IE, which decomposes all IE tasks into token-pair classification tasks.
In this way, we can unify all single IE tasks under the same task formulation, and use one model to fit all IE tasks without designing taskspecific modules. Besides, this unified decomposition is much faster than recently proposed generation-based unified frameworks.
2. After the reformulation of different IE tasks, we propose the Plusformer to model interaction between different token pairs. The plusshaped self-attention and CNN in Plusformer are well-motivated and effective in the reformulated IE scenario. Experiments in 12 IE datasets all achieve state-of-the-art (SOTA) performance which justifies the superiority of Plusformer in IE tasks.
3. The reformulation enables us to use one model to fit all IE tasks concurrently. Therefore, we can train one model on three IE tasks, and results on two joint IE datasets show that the proposed unification can effectively benefit each IE task through multi-task learning.
4. Extensive ablation experiments reveal that components in Plusformer are necessary and beneficial. Among them, CNN module in Plusformer can be essential to the overall performance. Analysis shows that this performance gain is well-explained because when reformulating IE tasks into token-pair classifications, the adjacent token pairs can be informative and CNN can take good advantage of the local dependency between them.

heading: 2 Task Decomposition and Decoding
text: We first introduce how we decompose IE tasks to conduct training, then present the decoding procedure for decomposition. More discussions about the decomposition are presented in Appendix A.

heading: 2.1 Task Decomposition
text: Formally, given an input sentence of L tokens x= [x1, x2, ..., xL], the potential token pairs can form a score matrix Y∈RL×L×(|S|+|R|), where S is span classes, R is relational classes. We stipulate
• When a span (s, e) is of type t, then Y(s,e,t) = Y(e,s,t) = 1, where s, e ∈ [1, L] and t ∈ [1, |S|] are the start, end token indices and span type;
• When the span (s1, e1) forms the relation r ∈ [|S|+1, |S|+ |R|] with another span (s2, e2), then Y(s1,s2,r) = Y(e1,e2,r) = 1.
NER aims to extract all entities {(si, ei, ti)}, where ti ∈ Se and Se is pre-defined entity types. Therefore, in NER, S = Se and R = ϕ.
RE aims to extract all relations {((shi , ehi , thi ), ri, (sti, eti, tti))}, where the superscript h and t denotes the head and tail entities, thi , t t i ∈ Se, ri ∈ Rr and Se,Rr are pre-defined entity types, and relation types. Therefore, in RE, S = Se and R = Rr.
EE aims to extract all events {{(si, ei, ti), (s1ia, e1ia, rol1i ), . . . , (skia, ekia, rolki )}}, where (si, ei) means the trigger span, ti ∈ St is the event type, St is pre-defined event types; sia, eia ∈ [1, L] are the start and end token indices of an argument span, k is the number of arguments of the trigger. To extract argument spans, we identify argument span into Sa which binarily denotes "has argument / no argument", thus |Sa| = 1; roli ∈ Ro is the role type of the argument and Ro is pre-defined role types. Following the formulation in RE, we can view role types from the trigger to the arguments as relations. Therefore, in EE, S = St ∪ Sa and R = Ro.
Joint IE aims to jointly extract entities, relations, and events in the text. Extracting entities and relations are generally the same as those in NER and RE. When extracting events, there is no need to extract argument spans purposely because all argument candidates are entities. Therefore, in joint IE, S = St ∪ Se and R = Rr ∪Ro.

heading: 2.2 Decoding
text: The decoding essentially extracts spans and relations from the score matrix Y. If Y(s,e,t) = Y(e,s,t) = 1 and t ∈ [1, |S|], then the span (s, e) is of type t. And for two spans (s1, e1) and (s2, e2), if Y(s1,s2,r) = Y(e1,e2,r) = 1 and r ∈ [|S|+1, |S|+ |R|], then the span (s1, e1) forms relation r with the span (s2, e2). The above decoding is for the ideal situation, where no span clash exists. However, for model’s predictions, we need to first resolve the conflicts. The decoding with model’s predictions will be presented in Appendix B.

heading: 3 Method
text: Figure 2 shows an overview of the architecture. Firstly, we present Biaffine (Dozat and Manning, 2017) model based on PLMs. Then, we propose a
novel Transformer-like structure named Plusformer to model interactions between token pairs. Finally, we describe loss functions.

heading: 3.1 Biaffine Model
text: Given an input sentence, we first apply a PLM as our sentence encoder to obtain the contextualized representation as follows
H = [h1, h2, ..., hL] = PLM([x1, x2, ..., xL]), (1)
where H ∈ RL×d, d is the PLM’s hidden size. Next, we use the Biaffine mechanism to get features for each token pair as follows
Hs, He = MLPstart(H), MLPend(H),
Si,j = (H s i ) TW1H e j +W2(H s i ⊕Hej) + v,
(2)
where MLPstart,MLPend are multi-layer perceptron layers, Hs, He ∈ RL×d, W1 ∈ Rd×c×d, W2 ∈ Rc×2d, v ∈ Rc , ⊕ refers to concatenation; S ∈ RL×L×c provides features for all possible token pairs, and c is the feature dimension size.

heading: 3.2 Plusformer
text: As illustrated in Section 1, when modeling the interaction between token pairs, the plus-shaped and local interaction should be beneficial. Therefore, we introduce the axis-aware plus-shaped self-attention and position embeddings to conduct plus-shaped interaction, we name this self-attention PlusAttention. Then, we leverage CNN to model local dependencies. We name this whole structure Plusformer.
PlusAttention. We first apply the self-attention mechanism (Vaswani et al., 2017) horizontally and
vertically as follows
Zhi,: = Attention(Si,:W Q h ,Si,:W K h ,Si,:W V h ), Zv:,j = Attention(S:,jW Q v ,S:,jW K v ,S:,jW V v ), Attention(Q,K,V) = softmax( QKT√
c )V,
(3)
where WQh ,W K h ,W V h , W Q v ,WKv ,W V v ∈ Rc×c, Zh,Zv ∈ RL×L×c. After the self-attention, we use the following method to merge Zh,Zv
S′ = MLP(Zh ⊕ Zv), (4)
where S′ ∈ RL×L×c. We make the plus-shaped self-attention axis-awareness by using two groups of attention parameters and using concatenation instead of an addition to merge Zh,Zv.
Position Embeddings. Although the model should be able to distinguish between horizontal and vertical directions through axis-aware plusshaped attention, it still lacks the sense of distances between token pairs and the area the token pair locates. Hence, we use two kinds of position embeddings to enable the model with these abilities.
• Rotary Position Embedding (RoPE) (Su et al., 2021) can encode the relative distance between two token pairs. It is utilized in both horizontal and vertical self-attention.
• Triangle position embedding is incorporated to mark the position of token pairs in the feature map, indicating whether the cell is in the upper or lower triangles. It adds to S in Eq.(3) before Attention.
CNN Layer. After the PlusAttention, we apply CNN with kernel size 3 × 3 on the S′ to help the model exploit the local dependency between neighboring token pairs. The formulation is as follows
S′′ = Conv(σ(Conv(S′))) (5)
where S′′ ∈ RL×L×c, and σ is the activation function; and the bias term of CNN is not used to avoid result inconsistencies for a sample when it is in batches of different lengths.
The Plusformer layer will be repeatedly used to interact fully between token pairs. Layer normalization (Ba et al., 2016) is ignored in the formulation for brevity.

heading: 3.3 Loss Function
text: Finally, we get final scores as follows
ŶS , ŶR = Sigmoid(Ŷ(:,:,:|S|)), Ŷ(:,:,|S|:),
Ŷ = MLP(S′′ + S), (6)
where ŶS ∈RL×L×|S|, ŶR ∈ RL×L×(|R|+1) are scores for span extraction and relational extraction, respectively; and Ŷ∈RL×L×(|S|+|R|+1). The +1 in (|R|+1) is because we use the adaptive thresholding loss (ATL) from Zhou et al. (2021) to avoid a global threshold in relational extraction.
For the span extraction, we use the binary crossentropy (BCE) loss as follows
L1 = − L∑
i,j=1
|S|∑
r=1
[Y(i,j,r)logŶ(i,j,r)
+ (1−Y(i,j,r))log(1− Ŷ(i,j,r))] (7)
For the relational extraction, we utilize the ATL as follows
L2 =− L∑
i,j=1
∑
r∈PT
log ( exp(ŶR(i,j,r))∑
r′∈PT∪{TH} exp(ŶR(i,j,r′))
)
− log (
exp(ŶR(i,j,|R|+1))∑ r′∈NT∪{TH} exp(ŶR(i,j,r′))
)
(8)
where PT and NT denote the positive and negative classes, ŶR(:,:,|R|+1) is the score for the threshold class TH. Only token pairs with scores higher than their corresponding adaptive thresholds are considered when decoding. We do not use ATL for span extraction because we need to sort span scores when decoding spans. The total loss L=L1+L2 is used for optimization.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: We conduct experiments on 10 datasets across three IE tasks, including NER, RE, and EE, and on 2 joint IE datasets. We evaluate NER task with CoNLL03 (Sang and Meulder, 2003) and OntoNotes (Pradhan et al., 2013) on flat NER, and with ACE04 (Doddington et al., 2004), ACE05-Ent (Walker et al., 2006) and GENIA (Kim et al., 2003) on nested NER. As for relation extraction, we use ACE05R (Walker et al., 2006) and SciERC (Luan et al., 2018). Since Wang et al. (2021) and Ye et al. (2022) consider symmetric relations, which shall massively influence the performance, we name this scenario Symmetric RE with datasets ACE05-R+ and SciERC+. For event extraction, we follow Lin et al. (2020) to perform experiments on three datasets, ACE05-E, ACE05-E+ (Doddington et al., 2004) and ERE-EN (Song et al., 2015). And for joint IE, we test on ACE05-E+ and ERE-EN. Statistics of all these datasets and detailed experimental settings are described in Appendix C.

heading: 4.2 Results on Single IE tasks
text: In this section, we report the UTC-IE performance in each single IE task. Results are shown in Table 1. The complete results for UTC-IE is shown in Appendix D. The table shows that UTC-IE exceeds previous SOTA models on all IE tasks. Particularly, UTC-IE averagely improves the entity F1 of NER, the entity F1 and relation F1 of RE, the entity F1 and the relation F1 of symmetric RE for +0.18, +0.71, +1.07, +0.21, +0.94, respectively. And for the EE tasks, UTC-IE increases the trigger F1, argument F1 for +0.35 and +1.67. We highlight that our model is helpful for relational extraction, such as relation extraction and argument extraction, which proves the effectiveness of interaction between token pairs. Although the performance increment of span extraction is not as significant as that of relational extraction, UTC-IE consistently improves on various span extraction tasks.
Besides, we also test UTC-IE without Plusformer. Surprisingly, this simple model surpasses previous SOTA models on four results marked with ♣, which proves the effectiveness of the task decomposition. The comparison between models with and without Plusformer clearly shows that Plusformer is effective in all tested datasets, and the performance improvement ranges from +0.40 (on OntoNotes) to +3.00 (on SciERC+). Notably,
the average performance gain of adding Plusformer on symmetric RE (+2.06) is more remarkable than that on RE (+1.03). We presume this is because the interaction between token pairs are more beneficial for symmetric relations.

heading: 4.3 Results on Joint IE task
text: Multi-task learning has proven to be useful in the IE area (Lin et al., 2020; Nguyen et al., 2021). Since UTC-IE unifies all IE tasks into a token-pair classification scenario, it is natural to test whether one UTC-IE model can benefit from jointly learning all IE tasks. In Table 2, the performance of UTCIEsingle is from the entity F1 of NER, relation F1 of RE, trigger F1 of EE and argument F1 of EE, respectively. Based on the comparison between UTC-IEsingle and UTC-IEjoint, it is obvious that
jointly learning these three tasks consistently improves performance in the 2 joint IE datasets.
Moreover, UTC-IEjoint outperforms previous SOTA joint IE models in Table 2, the average performance enhancement is +0.69 in ACE05-E+ and +0.75 in ERE-EN. Specifically, UTC-IEjoint increases the average performance of relational extraction by +1.30. Thusly, through unifying different IE tasks through our task decomposition, Plusformer can enjoy the benefit of multi-tasking learning, and achieve better performance than previous SOTA models.

heading: 4.4 Speed Comparison
text: To get a sense of the speed superiority of UTC-IE, we compare the inference speed of UTC-IE with previous unified models on ACE05 series datasets
and with task-specific SOTA models on every IE tasks. The former comparison is presented in Table 3 and the latter locates in the Appendix E. Compared with the generative UIE (Lu et al., 2022), UTC-IE improves F1 from 1.73 (on ACE05-R) to 2.89 (on ACE05-E+), and obtains one order magnitude of speed boost. Compared with OneIE (Lin et al., 2020), UTC-IE fundamentally enhances the performance for relational extractions (e.g., Rel. and Arg.) with an average of 4.47 F1 increment in joint IE. At the same time, UTC-IE is one order of magnitude faster than OneIE. In a nutshell, compared with previous SOTA models (whether taskspecific, unified or joint), UTC-IE achieves substantial performance gain across several datasets with a significant speed boost.

heading: 4.5 Ablation Study
text: To analyze the effectiveness of each component in Plusformer, we ablate each of them and list the outcomes in Table 4, and results on more datasets are presented in Appendix F. Besides, we study how many Plusformer layers are suitable in Appendix F.5. Based on the ablation, CNN is the most useful component among all IE tasks. The reason behind this improvement is that once token pairs are organized in the square feature map, the spatial correlations between neighboring token pairs become allusive, and CNN excels at exploiting these local interactions. More comprehensive analysis of CNN in Plusformer locates in Appendix F.1. To deepen our understanding of UTC-IE, we try an-
other variant of Plusformer where the PlusAttention is discarded, and we name this variant CNN-IE. The bottom line of Table 4 shows that the CNN-IE model can surpass or approach previous SOTA performance in almost all datasets, which proves the universality of our proposed task formulation.
However, CNN is not a panacea for UTC-IE. From Table 4, removing position embeddings or axis-awareness4 from UTC-IE will lead to an average of 0.39 or 0.44 performance degradation, respectively. Moreover, based on the performance of CNN-IE and UTC-IE, the average performance shrinks from 74.53 to 74.17 if the PlusAttention is deprived of Plusformer, which means the plusshaped self-attention is a desideratum. In addition, we present some intuitive examples and deeper analysis for position embeddings and axis-aware in Appendix F.3 and F.4.

heading: 5 Related Work
text: Information extraction tasks, which consist of named entity recognition, relation extraction, and event extraction, have long been a fundamental and well-researched task in the natural language processing (NLP) field. Previous researches mainly only focus on one or two tasks. Recently, building joint neural models of unified IE tasks has attracted increasing attention. Some of them incorporate graphs into IE structure. Wadden et al. (2019) propose a unified framework called DYGIE++ to
4Removing axis-aware means using the same self-attention parameters for both directions and adding Zh and Zv instead of concatenation.
extract entities, relations and events by leveraging span representations via span graph updates. Lin et al. (2020) and Nguyen et al. (2021) extend DYGIE++ by incorporating global features to extract cross-task and cross-instance interactions with multi-task learning. In addition to the graph-based models mentioned above, other studies focus on tackling general IE by generative models. Paolini et al. (2021) construct a framework called TANL, which enhances the generation model using augmented language methods. Moreover, Lu et al. (2022) regard IE task as a text-to-structure generation task, and leveraging prompt mechanism.
We unify all IE tasks as several token-pair classification tasks, which are fundamentally similar to the span-based methods on the IE task, for the start and end tokens can locate a span. Numerous NER studies emerge on span-based models, which are compatible with both flat and nested entities and perform well (Eberts and Ulges, 2020; Yu et al., 2020b; Li et al., 2021; Zhu and Li, 2022). In addition to entities, the span-based method is also used in RE. Some models (Wang et al., 2021; Ye et al., 2022) only leverage span representations to locate entities and simply calculate the interaction between entity pair, while others (Wang et al., 2020; Zhong and Chen, 2021) encode span pair information explicitly to extract relations. With regard to event extraction, as far as we know, there is little work on injecting span information into EE explicitly. Wadden et al. (2019) leverage span representations on general IE, but their model is complicated and only considers span at the embedding layer without further interaction. Conceptually, Jiang et al. (2020)’s work is similar to ours, but they need a two-stage model to determine the span type and span relations, respectively. Detailed analysis are depicted in Appendix G. Although many spanbased IE models exist, they are task-specific and lack interaction between token pairs. Decomposing
IE tasks as token-pair classification and conducting interaction between token pairs can uniformly model span-related knowledge and advance SOTA performance.
The most novel component of Plusformer is the plus-shaped attention mechanism, which can make token pairs interact with each other in an efficient way. A similar structure called Axial Transformers (Ho et al., 2019) is proposed in the Computer Vision (CV) field, which is designed to deal with data organized as high-dimension tensors. Tan et al. (2022) incorporate axial attention into relation classification to improve the performance on two-hop relation. However, CNN was not used in these works, while CNN has been proven to be vital to the IE tasks. Another similar structure named Twin Transformer (Guo et al., 2021) used in CV, where they encode pixels of image from row and column sequentially, and leverage CNN on top of them. But the position embeddings, which are important for IE tasks, are not used in the Twin Transformer. Besides, we want to point out that the usage of plus-shaped attention and CNN originates from the reformulation of IE tasks, any other modules which can directly enable interaction between constituent spans of a relation and between adjacent token pairs should be beneficial.

heading: 6 Conclusion
text: In this paper, we decompose NER, RE and EE tasks into token-pair classifications. Through the decomposition, we unify all IE tasks under the same formulation. After scrutinizing the token-pair feature matrix, we find the adjacent and plus-shaped interactions between token pairs should be informative. Therefore, we propose Plusformer, which uses an axis-aware plus-shaped self-attention followed by CNN layers to help token pairs interact with each other. Experiments on 10 single IE datasets and 2 joint IE datasets all outperform or approach the
SOTA performance. Besides, owing to the parallelism of self-attention and CNN, our model’s inference speed is substantially faster than previous SOTA models in RE and EE. Lastly, most of the previous IE models limit the interaction in the 1-D sequential dimension, while the reformulation of IE tasks opens a new angle to broaden the communication to the 2-D feature matrix.

heading: Limitations
text: While we unify diverse IE tasks into token-pair classification tasks and propose a simple but useful architecture to help token pairs interact with each other in an effective way, there are still several limitations that are worth discussing. Firstly, all modules in our UTC-IE are based on pre-trained language models, and experiments proves that different PLMs may influence the performance on the same dataset. Hence, our model relies on the capability of the PLM, which need a lot of GPU resources to complete the experiments. Additionally, although incorporating PlusAttention instead of self-attention can effectively reduce the memory and computational complexity from O(L4) to O(2L3), it still require a little large computation. Future work can leverage the backbone of our unification and model, and focus on the acceleration on each module.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their insightful comments. We also thank the developers of fastNLP5 and fitlog6. Thank Yuntao Chen for helping us preparing the code for publishing. This work was supported by the National Natural Science Foundation of China (No. 62236004 and No. 62022027) and CCF-Baidu Open Fund.

heading: A Discussion on Task decomposition
text: In this section, we will discuss two issues of the decomposition. The first is the inconsistency stipulation about the relation decomposition, the second is the false positive issue when decoding relations.

heading: A.1 The inconsistency
text: As our stipulation in Section 2, if a span (s, e) has an expected span type t, both the Y(s,e,t) and Y(e,s,t) are 1. If two spans (s1, e1) and (s2, e2) have relation r, this means the relation should also exist between spans (s1, e1) and (e2, s2) (the endto-start version of the span (s2, e2)), then based on our stipulation on the relation, the Y(s1,e2,r) and Y(e1,s2,r) should also equal 1, but we only define the Y(s1,s2,r) = 1 and Y(e1,e2,r) = 1, this causes an inconsistency between the stipulations. We ignore Y(s1,e2,r) and Y(e1,s2,r) to make the decoding less cluttered.

heading: A.2 False Positive Relation
text: A potential risk of the decomposition and decoding is that it may cause false positive relations. Given four spans p1 = (s1, e1), p2 = (s2, e2), p3 = (s3, e3), p4 = (s4, s4), if p4 has relation r with p1 and p2, and no relation exist between p4 and p3, then Y(s4,s1,r) = Y(e4,e1,r) = 1, Y(s4,s2,r) = Y(e4,e2,r) = 1. However, if s1 = s3, e2 = e3. Namely, p1 shares start token with p3 and p2 shares end token with p3. Then, based on Y(s4,s1,r) = Y(e4,e2,r) = 1, we get Y(s4,s3,r) = Y(e4,e3,r) = 1„ the decoding process will mistakenly think p4 has relation r with p3. However, this situation should be rare, and none is found in the tested datasets.

heading: B Decoding with model’s predictions
text: In this section, we will detail the decoding process for models’ predictions. The process described in Section 2.2 is not directly applicable to models’ predictions since spans may conflict with each other7.
7All IE tasks forbid span boundary clashes.
With prediction score matrix ŶS from Eq.(6), we follow previous work (Yu et al., 2020b) to first filter out spans whose scores are less than 0.5; for the remaining spans, we sort the spans based on their scores, then choose spans in descending order and make sure the span has no boundary clash with chosen spans. For relational extraction, we first decode all spans, then we get a binary matrix ȲR = ŶR(:,:,:|R|+1) > ŶR(:,:,|R|+1), then we pair spans to check whether they form relations. Take two spans (s1, e1) and (s2, e2) for instance, if ȲR(s1,s2,r) = ȲR(e1,e2,r) = 1, we claim the first span has relation r with the second span. For the RE task, we pair all entity spans to check if they form relations; for the EE task, we pair the trigger spans and argument spans to check if they form a role relationship; and for the joint IE task, we pair entity spans to check if they form relations, we pair the trigger spans and entity spans (because all argument spans are entity spans) to check if they form a role relationship.

heading: C Experimental Settings
text: In this section, we describe all experimental settings in detail, such as the statistics of datasets, baseline models, and more implementation details.

heading: C.1 Datasets
text: We conduct experiments on 10 single IE datasets and 2 joint IE datasets, and we detail the statistics of all datasets in Table 5.
Named entity recognition. We perform experiments on both flat and nested NER benchmarks. In flat NER, we adopt CoNLL03 (Sang and Meulder, 2003) and OntoNotes8 (Pradhan et al., 2013) datasets. In nested NER, we experiment on ACE049 (Doddington et al., 2004), ACE0510 (Walker et al., 2006) and GENIA (Kim et al., 2003). To distinguish ACE05 dataset used in other tasks, we name ACE05 in named entity recognition as ACE05-Ent. Specifically, we use the same preprocessing and splitting procedure on nested datasets as Yan et al. (2022), for they fix some annotation problems to unify different versions of these datasets and make a strictly fair comparison.
Relation extraction. We conduct experiments on two relation extraction datasets, ACE05 (Walker et al., 2006) and SciERC11 (Luan et al., 2018). The
8https://catalog.ldc.upenn.edu/LDC2013T19 9https://catalog.ldc.upenn.edu/LDC2005T09
10https://catalog.ldc.upenn.edu/LDC2006T06 11http://nlp.cs.washington.edu/sciIE/
ACE05 dataset, named as ACE05-R in our paper, is collected from various domains, such as newswire and online forums. The SciERC dataset provides entity, coreference and relation annotations from AI conference/workshop proceedings. In our experiments, we only use entity and relation annotations. We follow the data preprocessing in Luan et al. (2019) to split ACE05-R and SciERC into train, dev and test sets.
In typical RE, it is crucial to distinguish which entity comes first (head entity) and which comes next (tail entity). As for symmetric relational instance, the relation exists from both head-to-tail and tail-to-head directions. There are one such relation type in ACE05-R and two in SciERC. Some papers (Wang et al., 2021; Ye et al., 2022) regard each symmetric relational example as two directed relations, while others regard them as one relation. We find that this setting will hugely influence the performance. Therefore, we name the setting of having two directed relations as Symmetric Relation Extraction and name the corresponding datasets ACE05-R+ and SciERC+.
Event extraction. We evaluate UTC-IE on two widely used event extraction datasets, ACE2005 (Doddington et al., 2004) and ERE (Song et al., 2015). Following the prior preprocessing step (Wadden et al., 2019; Lin et al., 2020; Lu et al., 2021) on them, we obtain three datasets, ACE05-E, ACE05-E+ and ERE-EN. ACE05-E+ additionally takes relation arguments, pronouns and multi-token event triggers into consideration compared with ACE05-E. We use the same train/dev/test split as Lu et al. (2021) for all datasets to ensure a fair comparison. Furthermore, we still use ACE05-E+ and ERE-EN on joint IE, for they have annotations on all IE tasks.

heading: C.2 Baselines
text: TANL (Paolini et al., 2021) and UIE (Lu et al., 2022) are both unified information extraction models in the generative way, with different input and output formats. TANL uses T5-base as the backbone model, while UIE uses T5-large. We compare our model with them in every IE task. For TANL, we report single-task results for our model is trained under each task. For UIE, we report results with pre-training, which have better performance. In addition to these two baselines, each task also compares with a series of recently proposed task-specific methods as follows.
CNN-IE is the baseline model we design to prove the necessity of PlusAttention. The only difference between CNN-IE and UTC-IE is the former ignores the PlusAttention in Figure 2. We tune the number of CNN layers in CNN-IE from 2 to 6, and the best results are reported.
Named entity recognition. We compare our model’s performance on NER with several recently proposed NER methods.
• BART-NER (Yan et al., 2021a) formulates unified NER model as entity span sequence generation task. They use BART-large as the pre-trained model.
• W2NER (Li et al., 2022) formulates unified NER model as word-to-word classification task. The model employs BioBERT on GENIA and BERT-large on other datasets.
• BS (Zhu and Li, 2022): authors use spanbased NER model as baseline and propose boundary smoothing as a regularization technique to improve model performance. It leverages RoBERTa-base as the base encoder.
• CNN-NER (Yan et al., 2022) utilizes CNN to model local spatial correlations between spans and surpass recently proposed methods on nested NER. We report results using RoBERTa-base model.
Relation extraction. For relation extraction, we compare our model with several SOTA models.
• UniRE (Wang et al., 2021) jointly extracts entities and relations using a table containing all word pairs.
• PURE (Zhong and Chen, 2021) adopts a pipeline approach to solve NER and RE independently, using distinct contextual representations for entities and relations.
• PFN (Yan et al., 2021b) claims that some information should be shared between named entity recognition and relation extraction, while other information should be independent. They propose PFN to model two-way interaction (partition and filter) between two tasks.
• PL-Marker (Ye et al., 2022): authors consider interactions between spans and propose PL-Marker by strategically packing the markers in the encoder.
Previous models mentioned above use different RE datasets. Specifically, UniRE and PL-Marker regard symmetric relations as two directed relations, while other work does not. Besides, these two models utilize cross-sentence context.
Event extraction. Generative methods are popular in recently proposed event extraction papers.
• TEXT2EVENT (Lu et al., 2021) is a sequence-to-structure model which outputs a tree-like event structure with a given input sentence. The model uses T5-large as the base model.
• DEGREE (Hsu et al., 2022) leverages manually designed prompts to generate event records in natural language. We report the end-to-end performance of DEGREE instead of the pipeline way. The model leverages BART-large as encoder-decoder.
Joint IE. There are only two previous models that consider the joint IE in ACE05-E and ERE-EN datasets.
• OneIE (Lin et al., 2020) proposes an end-toend IE model, which employs global features and type dependency constraint at decoding step.
• FourIE (Nguyen et al., 2021) further improves the model by incorporating interaction dependency on representation level and label level.
For a fair comparison, we list the pre-trained model used for all baselines and our model on every IE dataset in Table 6. When choosing our pre-trained language model in different IE tasks’ datasets, we pick the same pre-trained model as the most recently published papers, such as BioBERT for GENIA and RoBERTa-base for other NER datasets. For RE and joint IE tasks, we choose the same pre-trained model as previous work. For tasks where previous work applied a generative pretrained model, we choose pre-trained model that has a similar size. For example, in event extraction, we use DeBERTa-large, whose number of parameters is 390M, which is closest to BART-large and T5-large used by previous EE papers.

heading: C.3 Evaluation Metrics
text: We report micro-F1 on all tasks:
• Entity: an entity is correct if its entity type
and span offsets match a golden entity. We use “Ent.” to represent entity F1 through all tables.
• Relation: a relation is correct if its type and its head and tail entities are correct, and the offsets and type of entities should also match the golden instance. We use “Rel.” to represent relation F1 through all tables.
• Event trigger: a trigger is correct if its span offset and event type is correct. We use “Trig.” to represent trigger F1 through all tables.
• Event argument: an argument is correct if its span offset, event type and role type all match the ground truth. We use “Arg.” to represent argument F1 through all tables.

heading: C.4 Hyper-Parameters
text: The detailed hyper-parameters used in each dataset are listed in Table 7. We use AdamW optimizer (Loshchilov and Hutter, 2019) with weight
decay 1e-2 for all datasets. Experiments are conducted five times with five different random seeds. We report the performance on test sets based on the model which achieves the best dev results in each dataset. For NER, the best results are calculated by the entity F1; for RE, the best results are calculated by the sum of entity F1 and relation F1; for EE, the best results are calculated by the best argument F1; for joint IE, the best results are calculated by the sum of relation F1 and trigger F1.

heading: D Complete Results
text: We present the complete results of UTC-IE and that without Plusformer in Table 8.

heading: E Speed Comparison
text: We test the speed of other models through their released code. For models, such as OneIE (Lin et al., 2020), DEGREE (Hsu et al., 2022) and PL-
Marker12 (Ye et al., 2022), they also released a trained model along with their code, and we used their released model to test the inference speed. For UIE (Lu et al., 2022) and BS (Zhu and Li, 2022), we trained a model with their code. The speed test is conducted in one RTX 3090 GPU and the batch size is set as 32 for all models (if the model goes out of memory, we choose the largest batch size that can accommodate the GPU); the test corpus is the test set of each dataset. The speed is measured by the number of sentences in the test set divided by the number of seconds that elapsed. And each inference is repeated three times, the average speed is reported.
The speed comparison can be roughly categorized into two kinds. The first kind is the comparison with previous universal IE models, namely OneIE (Lin et al., 2020) and UIE (Lu et al., 2022), and results are depicted in Table 3. Compared with UIE in five chosen datasets, UTC-IE is x19.7 faster and improves performance by 1.86 averagely. Besides, for the joint IE task, UTC-IE is 18.4 times faster than OneIE and improves performance by 2.72 on average. The second kind is the comparison between UTC-IE and SOTA models targeted
12PL-Marker used a two-stage pipeline to conduct prediction. Therefore, the time is measured by the total seconds elapse to finish two stages.
for each IE task, and results are presented in Table 9. Compared with previous SOTA models, the average performance increments for entity F1, relation F1 and argument F1 are 0.31, 0.94 and 2.15. In the meantime, UTC-IE speeds up for x1.0, x5.5 and x101.9 averagely.
In short, using UTC-IE for IE tasks can not only substantially enhance the performance in most cases, but also significantly speed up the inference speed in almost all datasets.

heading: F Ablation Study
text: For ablation, we will choose two datasets for each IE task to study the effect of each component in Plusformer. We separately list the performance for span extraction (including entity extraction in NER and RE, trigger extraction in EE) in Table 10 and relational extraction (including relation extraction in RE and argument extraction in EE) in Table 11. Besides, we also study how the performance varies with the change of the number of Plusformer layers in Figure 10.

heading: F.1 CNN
text: Based on our ablations in Table 10 and Table 11, the CNN module in Plusformer contributes most to the performance enhancement. To reveal why CNN is so effective in both span extraction and relational
extraction, we first present an intuitive example in Figure 3 to show how CNN helps to extract entities and relations in the RE task. Like in Figure 3(a), for NER, the entity e2 can interact with entity e1 and relation r1 through CNN. Besides, for RE, CNN can contribute in two ways. On the one hand, CNN helps the relational token pair to directly gather information from its constituent entities, like the r1 in Figure 3(b). On the other hand, the start-to-start and end-to-end relational token pairs, like two r2 cells, can directly interact with each other through CNN.
To quantitatively present the effectiveness of CNN in UTC-IE, we propose further ablations to show how the distance between the relational token pair and its constituent spans affects the relational F1, and how the distance between start-to-start and end-to-end token pairs affects the relational F1. Furthermore, we conduct experiments on UTCIE with different kernel sizes and choose the most proper size.

heading: F.1.1 Distance Between the Relational Token Pair and Its Constituent spans VS. Relational F1
text: In this section, we will show how the relational F1 (relation F1 in RE and argument F1 in EE) will change when the distance between the relational token pair and its constituent spans varies. For two spans (s1, e1) and (s2, e2) (we ignore their diagonally symmetric counterparts, since they will not affect the calculation here), the span relation from (s1, e1) to (s2, e2) is represented by two token pairs (s1, s2) and (e1, e2). The distance between the two token pairs and its constituent spans is calculated as follows
d = max(|s2 − e1|, |s1 − e2|) + 1, (9)
where the distance d is named as “Span-Rel-Span Distance”, it represents the longest distance between the relational token pairs to their constituent spans. The relation between d and the relational F1 is shown in Figure 4. Without CNN, the performance for extracting relations between nearby constituent spans will drop sightly, while less affected for further ones, which proves that CNN is effective for exploiting local dependency to predict relations.

heading: F.1.2 Distance Between Start-to-Start and End-to-End Token Pairs VS. Relational
text: F1
As shown in Figure 3(b), if the distance between the start-to-start and end-to-end relational token pairs is small, the CNN should be helpful. To verify this assumption, we first define the “Inner Relational Distance” as follows, for two spans (s1, e1) and (s2, e2), the relational token pairs are (s1, s2) and (e1, e2), then the distance between two relational token pairs is calculated as follows
d = max(e1 − s1, e2 − s2) + 1, (10)
where d reveals the distance between start-to-start and end-to-end token pairs, and it is actually decided by the max constituent span length. And its relation with the relational F1 is shown in Figure 5. It is clear that, most of the start-to-start token pairs are near to their end-to-end token pairs, and CNN takes advantage of this adjacency to make better predictions.

heading: F.1.3 CNN Kernel Size VS. F1
text: We study the relation between the kernel size of CNN and F1 performance in Figure 6. We observe that CNN with kernel size 3 obtains the best performance on almost all datasets and tasks. Specifically,
reducing CNN kernel size to 1 significantly harms the performance on all datasets, for CNN will lose the capability of interacting with neighboring token pairs. In contrast, F1 also slightly decreases with larger CNN kernel size. We presume that CNN with a larger kernel size may introduce more noise and harm performance. Therefore, we choose kernel size 3 for all datasets.
F.2 Is CNN All We Need?
Since CNN is so effective in the Plusformer, it is natural to ask whether it is enough only to use CNN. Therefore, we conduct experiments on models without the plus-shaped self-attention and named this model CNN-IE. We conduct experiments for CNNIE in six datasets, and results are listed in Table 10 and Table 11. With only the CNN module, the model can achieve SOTA or near SOTA performance in all six datasets, which depicts the effectiveness of the proposed token-pair decomposition and CNN module. However, it still lags behind the UTC-IE model, which reveals the necessity of the PlusAttention.
It is worth noting that CNN-IE is different from CNN-NER (Yan et al., 2022). As for model structures, CNN-IE is one of our baseline models and reserves the general framework of Plusformer, namely self-attention with CNN layers. However, CNN-NER only uses residual CNN layers. As we can see in Table 1, CNN-IE has different results on ACE05-Ent than CNN-NER does. Besides, as for tasks, CNN-NER only formulates the nested NER task and can not transfer to other IE tasks directly. However, our CNN-IE can easily apply to NER, RE and EE. In each IE task on CNN-IE, CNN modules have their specific functions to capture different neighboring token pairs, as depicted in Figure 3.

heading: F.3 Position Embeddings
text: The RoPE embedding aims to help token pairs be aware of the spatial relationships between each other, and the triangle position embedding tries to enable spans to be informed of their areas in the feature map. From Table 10 and Table 11, the position embeddings enhance the span extraction and rela-
tional extraction for 0.39, 0.64 averagely. Besides, in Figure 7, we show that the position embeddings can help the model exploit the distance bias to improve the performance of relational extraction.

heading: F.4 Axis-aware Plus-shaped Self-Attention
text: Lastly, we study the effect of PlusAttention. We present an example to delineate why the axis-aware is valuable for span extraction and relational extraction in Figure 8. From Figure 8, axis-aware should be worthwhile no matter what the task is, span extraction or relational extraction. As expected, from Table 10 and Table 11, if we discard the axis-aware in Plusformer, the average performance of span extraction and relational extraction diminish 0.28 and 0.86, respectively, which reveals the necessity of axis-aware in the PlusAttention module.
Besides, we show two case studies of the plusshaped attention in Figure 9. The sentences are from the test dataset of ACE2005-Ent and ACE2005-R. Both cases put larger attention scores on informative token pairs.

heading: F.5 Number of Plusformer Vs. F1
text: We study the relation between the number of Plusformer layers and F1 performance in Figure 10. For the NER datasets, we use two layers of Plusformer, and for the RE and EE we use three.

heading: G Comparison with GLAD
text: Jiang et al. (2020) claims that many NLP tasks can be regarded as the span prediction and prediction of relations between pairs of spans (named as span extraction and relational extraction in our paper), which is conceptually similar to our insights. However, our work is fundamentally distinct from theirs on both formulation and model architecture. Jiang et al. (2020) classify various NLP tasks into two separate tasks and design different modules for them. To contrast, we unify all traditional IE tasks into a single formulation, namely token-pair classification. Therefore, we only need one model for all tasks. Besides, Jiang et al. (2020) simply use the concatenation of the start and end token representations to represent a span, and for relations, they concatenate the head and tail span representations. Therefore, in their work, the interaction between
spans are weak. In our work, we obtain the feature matrix of all token pairs and add well-designed Plusformer module on top of all token pairs, where token pairs can interact with others thoroughly.
In order to prove the superiority of our reformulation and UTC-IE model, we make a fair comparison on several tasks from the GLAD benchmark (Jiang et al., 2020). We choose 3 additional IE tasks, including Open Information Extraction (OIE), Semantic Role Labeling (SRL) and Aspect Based Sentiment Analysis (ABSA), and NER and RE. We use WLP (Hashimoto et al., 2017) on NER and RE, OIE2016 (Stanovsky and Dagan, 2016) on OIE, OntoNotes (Pradhan et al., 2013) on SRL and SemEval14 (Pontiki et al., 2014) on ABSA. The detailed experimental settings are the same as those in GLAD, to ensure a fair comparison. Results are present in Table 12.
The table shows that UTC-IE outperforms GLAD on all chosen tasks exceedingly, with +2.76 improvement on average. Moreover, we observe that UTC-IE without Plusformer also surpasses GLAD benchmarks on all tasks with +0.84 improvement on average, which proves the superiority of our unified reformulation.
𝑒! US Airways 𝑒" 𝑒# one of US Airway‘s hubs US Airway’s hubs
𝑒!
𝑒" 𝑒#
(a) PlusAttention map on NER
Pittsburgh , one of US Airway ’ s hubs …
Pittsburgh
of
US
Airway
hubs
,
‘s
one
𝑒!
𝑒"
𝑒#
𝑒! JordeenaGinsburg 𝑒" 𝑟 News 12 Westchester ORG-AFF
𝒓
(b.1) PlusAttention map on RE’s start-to-start cell
from News 12
Jordeena
News
12
Westchester
the
Ginsburg
has
from
Jordeena Ginsburg
Westchester has the …
𝑒!
𝑒!
𝑒"
𝑒"
𝒓
(b.2) PlusAttention map on RE’s end-to-end cell
from News 12
Jordeena
News
12
Westchester
the
Ginsburg
has
from
Jordeena Ginsburg
Westchester has the …
𝑒!
𝑒"
𝒓
𝑒! JordeenaGinsburg 𝑒" 𝑟 News 12 Westchester ORG-AFF
𝑒!
𝑒"
𝒓

heading: EE ACE05-E+ ERE-EN
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section "Limitations" (7th section)
3 A2. Did you discuss any potential risks of your work? Section "Limitations" (7th section)
3 A3. Do the abstract and introduction summarize the paper’s main claims? "Abstract" and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix C
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix C and section 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix C and Section 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix C
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future work can leverage the backbone of our unification and model, and focus on the acceleration on each module