# DEERE: Document-Level Event Extraction as Relation Extraction

Authors: Jian Li; Ruijuan Hu; Keliang Zhang; Haiyan Liu; Yanzhou Ma
Journal: Mobile Information Systems
 (2022)

---

## Abstract

The descriptions of complex events usually span sentences, so we need to extract complete event information from the whole document. To address the challenges of document-level event extraction, we propose a novel framework named Document-level Event Extraction as Relation Extraction (DEERE), which is suitable for document-level event extraction tasks without trigger-word labelling. By well-designed task transformation, DEERE remodels event extraction as single-stage relation extraction, which can mitigate error propagation. A long text supported encoder is adopted in the relation extraction model to aware the global context effectively. A fault-tolerant event integration algorithm is designed to improve the prediction accuracy. Experimental results show that our approach advances the SOTA for the ChFinAnn dataset by an average F1-score of 3.7. The code and data are available athttps://github.com/maomaotfntfn/DEERE.


---

## Body

## 1. Introduction

The aim of the Event Extraction (EE) task is to extract structured event information from unstructured text [1]. EE can be divided into Sentence-level Event Extraction (SEE) and Document-level Event Extraction (DEE). Previous research has focused on SEE, but the description of a complex event usually involves multiple sentences, so we need to extract more complete event information from the whole document. SEE no longer meets our needs, and its methods are ill-suited for DEE tasks.The two main challenges of DEE are argument-scattering and multievents. Argument-scattering means that the arguments of an event are scattered in multiple sentences. As shown in Figure1, the arguments of Event-2 are scattered in S18, S21, and S22. Multievents mean that a document includes multiple events, in which there may be arguments that overlap. Depending on the degree of overlap, the relationship between two events can be classified as (1) no arguments overlap, (2) arguments overlap between events of different types, and (3) arguments overlap between events of the same type (e.g., both Event-1 and Event-2 contain the argument Tacheng International). In addition, we will also encounter the DEE task without trigger-word labelling, which can be regarded as another challenge.Figure 1
A sample of document-level event extraction.To address the above-given challenges, the most recent SOTA method, DE-PPN [2], designed an end-to-end model, where a document-level encoder is used to obtain the text representations, and a multigranularity decoder is used to generate events in parallel. DE-PPN encodes each sentence separately and concatenates them into document encoding after max pooling, which does not consider the interaction between sentences and cannot be fully aware of the global context. The extraction process of DE-PPN includes candidate argument identification, event prediction, and role filling. This kind of multi-stage structure is prone to error propagation.In this paper, we propose an event extraction framework named DEERE (short for Document-level Event Extraction as Relation Extraction). The key idea is to transform the complex DEE task into a relatively simple relation extraction task, which can deal with both challenges of argument-scattering and multievents. DEERE adopts a single-stage entity-relation joint extraction model to mitigate error propagation. A long text supported Transformer is used as the text encoder, which can effectively aware of the global context.In summary, our contributions include:(1)
We propose a novel framework (DEERE) based on task transformation, which is suitable for DEE tasks without trigger-word labelling.(2)
We design two key algorithms in the framework. The role selection algorithm can reduce the probability of event arguments overlapping, and the event division algorithm can further deal with the case of arguments overlapping mentioned.(3)
Experimental results show that DEERE significantly outperforms the most recent SOTA method on the widely used DEE dataset, with an average F1-score improvement of 3.7.


## 2. Methodology

As shown in Figure2, the architecture of DEERE includes three modules: task transformation, relation extraction, and event prediction. Labelled events are transformed into well-designed relational triples, which are used as the training data of a relation extraction model. The relation extraction model adopts a long text supported Transformer to encode the whole document. During event prediction, the relational triples extracted from the input text are reorganized into basic events.Figure 2
The overall architecture of DEERE.


### 2.1. Task Transformation

#### 2.1.1. Relational Triplet Creation

The labelled events in the training set are transformed into two kinds of relations: (1) role relation describes the role assignments between arguments, which is designed to resolve the challenge of argument-scattering and multievents without key role overlap; (2) co-event relation describes whether two arguments belong to the same event, which is designed to resolve the challenge of multi-events with key role overlap.Suppose the event typeEk includes m roles, denoted as r1,r2,…,rm; an event instance of Ek also includes m arguments correspondingly, denoted as a1,a2,…,am. Select one from the roles of Ek as the key role, and the argument that plays the key role in a specific event is called the key argument. The key argument of the event is combined with each nonkey argument to form a role relation triple akey,Ek_ri,ai, where akey represents the key argument of a specific Ek event, and ai represents a nonkey argument playing the i-th role in the same event (i≠key).In a document, a key argument may involve several events of the same type. In order to distinguish these events, a subkey role is selected from the nonkey roles of the event type, and the argument that plays the subkey role in a specific event is called the subkey argument. The subkey argument of the event is combined with every other argument to form a coevent relation tripleaskey,co_event,ai, which represents that askey and ai belong to a same event (i≠key andi≠skey).According to the definition, there are multiple types of role relation, whose number is related to the total roles of all the event types. There is only one type of coevent relation, which ignores event types and roles. These relational triples will be used as the training data and prediction targets of the relational extraction model.


#### 2.1.2. Key Role Selection

Based on the above-given task conversion rules, in order to reduce the situation of a key argument involving multiple events of the same type, we hope to select the role that can best distinguish different events as the key role. For a group of events with the same type, we define role discrimination as the average probability that each argument on a role can accurately identify the event to which it belongs.Formally, suppose that a documentd contains n events of Ek, denoted as e1,e2,…,enT; the argument list of ei is denoted as ai1,ai2,…,aim, the argument matrix for all events can be represented as follows:(1)e1e2⋮en=a11a12…a1ma21a22…a2m⋮⋮⋱⋮an1an2…anm.The discrimination of rolerj to events of Ek in single document d can be expressed as follows:(2)discriminationEk,rj,d=∑i=1npei∗qaij.In equation (2), pei represents the probability of the occurrence of event ei, and qaij represents the probability that the argument aij can accurately identify the event to which it belongs. We assume that each event has an equal probability, that is, pei=1/n, so the formula can be simplified as follows:(3)discriminationEk,rj,d=1n∑i=1nqaij.In a group of events of the same type, the higher the repetition rate of arguments on a role, the lower the role discrimination for events. It is not difficult to prove that the summation part in equation (3) is numerically equal to the count of nonrepeated arguments on the role, where arguments with null value are not counted. We can get the following equation:(4)discriminationEk,rj,d=1ncount_distincta1j,a2j…,anj.In equation (4), count_distinct represents the count of nonrepeated arguments, which can be easily obtained by set operation in practice. Suppose that there are T documents in the training set that contain events of Ek, we can calculate the discrimination of role rj in each document dt separately, and regard the average as the global role discrimination:(5)discrimination_globalEk,rj=1T∑t=1TdiscriminationEk,rj,dt.From the role list ofEk, we can select the role with the highest discrimination as the key role and select the one with the second highest discrimination as the subkey role.


### 2.2. Relation Extraction

#### 2.2.1. Relation Extraction Model

The triples obtained from the task transformation will be input to the relation extraction model together with the original text as training data. In theory, any relation extraction model can be applied here. However, since the results of relation extraction will directly affect the results of EE, we adopt the recently proposed GPLinker [3], an entity-relationship joint extraction model based on GlobalPointer [4]. Preliminary experiments show that the performance of GPLinker is slightly better than the Casrel [5] and comparable to the TPLinker [6]. In addition, the GPLinker model has fast training speed, high decoding efficiency, and theoretically no exposure bias.GlobalPointer is essentially a token-pair recognition model, which can be used in nested and nonnested NER. Multilabel categorical cross-entropy is used as the loss function during training. GPLinker converts the extraction of relational triples (subject, predicate, object) into three kinds of token-pair recognition: entity head/tail pair, subject/object heads pair, and subject/object tails pair. Each kind of token-pair is recognized by a specific GlobalPointer, and all GlobalPointer modules share the same text encoder.


#### 2.2.2. Long Text Encoding

In recent years, the relationship extraction SOTA models (Casrel, TPLinker, etc.) are mostly based on BERT [7] or other pretrained language models. The original BERT uses absolute position encoding and can handle a maximum text length of 512 tokens. The text length of document-level extraction tasks is usually beyond the above range. If the long text is truncated or segmented, it will inevitably affect the model’s perception of the context of the full text, which is also a common problem with previous DEE models. For this, we try to use RoFormer [8], a Transformer that uses relative position encoding, to encode every document as a whole. Of course, other pretrained language models that support long text encoding can also be used here.


### 2.3. Event Prediction

Shown as the event prediction module of Figure2, we first extract relational triplets from the input text and then integrate these triplets as basic events. Since the triples are predicted by the relation extraction model, there are inevitably some errors or omissions, which requires the event integration algorithm to have a certain fault tolerance. In this regard, we first construct event-clusters by the key argument and role relation. If an event-cluster contains multiple basic events, it will be further divided according to the subkey argument and coevent relation.


#### 2.3.1. Event-Cluster Construction

Based on the predicted role relation triplets, a special structure called event-cluster is constructed around each key argument. Every event-cluster includes a key argument and related nonkey arguments, all of which belong to the same event type. An event-cluster can be classified as a single event-cluster or compound event-cluster according to the number of basic events included.According to the construction rules, there is only one key argument in an event-cluster, but there may be several arguments on a nonkey role, which is called a multivalued role. The higher the proportion of multi-valued roles in an event-cluster, the more likely it contains multiple events. If there are more than one subkey argument and the proportion of multivalued roles exceeds a certain threshold, it will be judged as a compound event-cluster; otherwise, it will be judged as a single event-cluster. The threshold can be adjusted as a hyperparameter.


#### 2.3.2. Event Division

A compound event-cluster needs to be further divided into basic events. In detail, we first create one basic event for each subkey argument and then assign arguments to other roles of each event according to the predicted coevent relations. If there are more than one candidate arguments for a role, it can be selected by relation strength.We have also considered the maximal clique search algorithm for event division, which requires that any two arguments of the same event must be accurately judged as coevent relation. Assuming that an event hasn arguments, even if only one-way relations are considered, n ∗ (n − 1)/2 coevent relations need to be extracted. The missing of just one relation can lead to serious errors in event division. As shown in Figure 3, there is a complete event with 5 arguments [A, B, C, D, E], if the relation <B, D> is missing, it will be divided into two incomplete events [A, B, C, E] and [A, C, D, E].Figure 3
An error case of event division based on the maximal clique search algorithm.Although the maximal clique search algorithm is more complete in theory, its application conditions are too demanding. In contrast, our proposed algorithm only needsn − 2 coevent relations to divide a basic event, which not only reduces the application conditions of the algorithm but also has stronger fault tolerance. This point will be confirmed in the experiments.


## 2.1. Task Transformation

### 2.1.1. Relational Triplet Creation

The labelled events in the training set are transformed into two kinds of relations: (1) role relation describes the role assignments between arguments, which is designed to resolve the challenge of argument-scattering and multievents without key role overlap; (2) co-event relation describes whether two arguments belong to the same event, which is designed to resolve the challenge of multi-events with key role overlap.Suppose the event typeEk includes m roles, denoted as r1,r2,…,rm; an event instance of Ek also includes m arguments correspondingly, denoted as a1,a2,…,am. Select one from the roles of Ek as the key role, and the argument that plays the key role in a specific event is called the key argument. The key argument of the event is combined with each nonkey argument to form a role relation triple akey,Ek_ri,ai, where akey represents the key argument of a specific Ek event, and ai represents a nonkey argument playing the i-th role in the same event (i≠key).In a document, a key argument may involve several events of the same type. In order to distinguish these events, a subkey role is selected from the nonkey roles of the event type, and the argument that plays the subkey role in a specific event is called the subkey argument. The subkey argument of the event is combined with every other argument to form a coevent relation tripleaskey,co_event,ai, which represents that askey and ai belong to a same event (i≠key andi≠skey).According to the definition, there are multiple types of role relation, whose number is related to the total roles of all the event types. There is only one type of coevent relation, which ignores event types and roles. These relational triples will be used as the training data and prediction targets of the relational extraction model.


### 2.1.2. Key Role Selection

Based on the above-given task conversion rules, in order to reduce the situation of a key argument involving multiple events of the same type, we hope to select the role that can best distinguish different events as the key role. For a group of events with the same type, we define role discrimination as the average probability that each argument on a role can accurately identify the event to which it belongs.Formally, suppose that a documentd contains n events of Ek, denoted as e1,e2,…,enT; the argument list of ei is denoted as ai1,ai2,…,aim, the argument matrix for all events can be represented as follows:(1)e1e2⋮en=a11a12…a1ma21a22…a2m⋮⋮⋱⋮an1an2…anm.The discrimination of rolerj to events of Ek in single document d can be expressed as follows:(2)discriminationEk,rj,d=∑i=1npei∗qaij.In equation (2), pei represents the probability of the occurrence of event ei, and qaij represents the probability that the argument aij can accurately identify the event to which it belongs. We assume that each event has an equal probability, that is, pei=1/n, so the formula can be simplified as follows:(3)discriminationEk,rj,d=1n∑i=1nqaij.In a group of events of the same type, the higher the repetition rate of arguments on a role, the lower the role discrimination for events. It is not difficult to prove that the summation part in equation (3) is numerically equal to the count of nonrepeated arguments on the role, where arguments with null value are not counted. We can get the following equation:(4)discriminationEk,rj,d=1ncount_distincta1j,a2j…,anj.In equation (4), count_distinct represents the count of nonrepeated arguments, which can be easily obtained by set operation in practice. Suppose that there are T documents in the training set that contain events of Ek, we can calculate the discrimination of role rj in each document dt separately, and regard the average as the global role discrimination:(5)discrimination_globalEk,rj=1T∑t=1TdiscriminationEk,rj,dt.From the role list ofEk, we can select the role with the highest discrimination as the key role and select the one with the second highest discrimination as the subkey role.


## 2.1.1. Relational Triplet Creation

The labelled events in the training set are transformed into two kinds of relations: (1) role relation describes the role assignments between arguments, which is designed to resolve the challenge of argument-scattering and multievents without key role overlap; (2) co-event relation describes whether two arguments belong to the same event, which is designed to resolve the challenge of multi-events with key role overlap.Suppose the event typeEk includes m roles, denoted as r1,r2,…,rm; an event instance of Ek also includes m arguments correspondingly, denoted as a1,a2,…,am. Select one from the roles of Ek as the key role, and the argument that plays the key role in a specific event is called the key argument. The key argument of the event is combined with each nonkey argument to form a role relation triple akey,Ek_ri,ai, where akey represents the key argument of a specific Ek event, and ai represents a nonkey argument playing the i-th role in the same event (i≠key).In a document, a key argument may involve several events of the same type. In order to distinguish these events, a subkey role is selected from the nonkey roles of the event type, and the argument that plays the subkey role in a specific event is called the subkey argument. The subkey argument of the event is combined with every other argument to form a coevent relation tripleaskey,co_event,ai, which represents that askey and ai belong to a same event (i≠key andi≠skey).According to the definition, there are multiple types of role relation, whose number is related to the total roles of all the event types. There is only one type of coevent relation, which ignores event types and roles. These relational triples will be used as the training data and prediction targets of the relational extraction model.


## 2.1.2. Key Role Selection

Based on the above-given task conversion rules, in order to reduce the situation of a key argument involving multiple events of the same type, we hope to select the role that can best distinguish different events as the key role. For a group of events with the same type, we define role discrimination as the average probability that each argument on a role can accurately identify the event to which it belongs.Formally, suppose that a documentd contains n events of Ek, denoted as e1,e2,…,enT; the argument list of ei is denoted as ai1,ai2,…,aim, the argument matrix for all events can be represented as follows:(1)e1e2⋮en=a11a12…a1ma21a22…a2m⋮⋮⋱⋮an1an2…anm.The discrimination of rolerj to events of Ek in single document d can be expressed as follows:(2)discriminationEk,rj,d=∑i=1npei∗qaij.In equation (2), pei represents the probability of the occurrence of event ei, and qaij represents the probability that the argument aij can accurately identify the event to which it belongs. We assume that each event has an equal probability, that is, pei=1/n, so the formula can be simplified as follows:(3)discriminationEk,rj,d=1n∑i=1nqaij.In a group of events of the same type, the higher the repetition rate of arguments on a role, the lower the role discrimination for events. It is not difficult to prove that the summation part in equation (3) is numerically equal to the count of nonrepeated arguments on the role, where arguments with null value are not counted. We can get the following equation:(4)discriminationEk,rj,d=1ncount_distincta1j,a2j…,anj.In equation (4), count_distinct represents the count of nonrepeated arguments, which can be easily obtained by set operation in practice. Suppose that there are T documents in the training set that contain events of Ek, we can calculate the discrimination of role rj in each document dt separately, and regard the average as the global role discrimination:(5)discrimination_globalEk,rj=1T∑t=1TdiscriminationEk,rj,dt.From the role list ofEk, we can select the role with the highest discrimination as the key role and select the one with the second highest discrimination as the subkey role.


## 2.2. Relation Extraction

### 2.2.1. Relation Extraction Model

The triples obtained from the task transformation will be input to the relation extraction model together with the original text as training data. In theory, any relation extraction model can be applied here. However, since the results of relation extraction will directly affect the results of EE, we adopt the recently proposed GPLinker [3], an entity-relationship joint extraction model based on GlobalPointer [4]. Preliminary experiments show that the performance of GPLinker is slightly better than the Casrel [5] and comparable to the TPLinker [6]. In addition, the GPLinker model has fast training speed, high decoding efficiency, and theoretically no exposure bias.GlobalPointer is essentially a token-pair recognition model, which can be used in nested and nonnested NER. Multilabel categorical cross-entropy is used as the loss function during training. GPLinker converts the extraction of relational triples (subject, predicate, object) into three kinds of token-pair recognition: entity head/tail pair, subject/object heads pair, and subject/object tails pair. Each kind of token-pair is recognized by a specific GlobalPointer, and all GlobalPointer modules share the same text encoder.


### 2.2.2. Long Text Encoding

In recent years, the relationship extraction SOTA models (Casrel, TPLinker, etc.) are mostly based on BERT [7] or other pretrained language models. The original BERT uses absolute position encoding and can handle a maximum text length of 512 tokens. The text length of document-level extraction tasks is usually beyond the above range. If the long text is truncated or segmented, it will inevitably affect the model’s perception of the context of the full text, which is also a common problem with previous DEE models. For this, we try to use RoFormer [8], a Transformer that uses relative position encoding, to encode every document as a whole. Of course, other pretrained language models that support long text encoding can also be used here.


## 2.2.1. Relation Extraction Model

The triples obtained from the task transformation will be input to the relation extraction model together with the original text as training data. In theory, any relation extraction model can be applied here. However, since the results of relation extraction will directly affect the results of EE, we adopt the recently proposed GPLinker [3], an entity-relationship joint extraction model based on GlobalPointer [4]. Preliminary experiments show that the performance of GPLinker is slightly better than the Casrel [5] and comparable to the TPLinker [6]. In addition, the GPLinker model has fast training speed, high decoding efficiency, and theoretically no exposure bias.GlobalPointer is essentially a token-pair recognition model, which can be used in nested and nonnested NER. Multilabel categorical cross-entropy is used as the loss function during training. GPLinker converts the extraction of relational triples (subject, predicate, object) into three kinds of token-pair recognition: entity head/tail pair, subject/object heads pair, and subject/object tails pair. Each kind of token-pair is recognized by a specific GlobalPointer, and all GlobalPointer modules share the same text encoder.


## 2.2.2. Long Text Encoding

In recent years, the relationship extraction SOTA models (Casrel, TPLinker, etc.) are mostly based on BERT [7] or other pretrained language models. The original BERT uses absolute position encoding and can handle a maximum text length of 512 tokens. The text length of document-level extraction tasks is usually beyond the above range. If the long text is truncated or segmented, it will inevitably affect the model’s perception of the context of the full text, which is also a common problem with previous DEE models. For this, we try to use RoFormer [8], a Transformer that uses relative position encoding, to encode every document as a whole. Of course, other pretrained language models that support long text encoding can also be used here.


## 2.3. Event Prediction

Shown as the event prediction module of Figure2, we first extract relational triplets from the input text and then integrate these triplets as basic events. Since the triples are predicted by the relation extraction model, there are inevitably some errors or omissions, which requires the event integration algorithm to have a certain fault tolerance. In this regard, we first construct event-clusters by the key argument and role relation. If an event-cluster contains multiple basic events, it will be further divided according to the subkey argument and coevent relation.


### 2.3.1. Event-Cluster Construction

Based on the predicted role relation triplets, a special structure called event-cluster is constructed around each key argument. Every event-cluster includes a key argument and related nonkey arguments, all of which belong to the same event type. An event-cluster can be classified as a single event-cluster or compound event-cluster according to the number of basic events included.According to the construction rules, there is only one key argument in an event-cluster, but there may be several arguments on a nonkey role, which is called a multivalued role. The higher the proportion of multi-valued roles in an event-cluster, the more likely it contains multiple events. If there are more than one subkey argument and the proportion of multivalued roles exceeds a certain threshold, it will be judged as a compound event-cluster; otherwise, it will be judged as a single event-cluster. The threshold can be adjusted as a hyperparameter.


### 2.3.2. Event Division

A compound event-cluster needs to be further divided into basic events. In detail, we first create one basic event for each subkey argument and then assign arguments to other roles of each event according to the predicted coevent relations. If there are more than one candidate arguments for a role, it can be selected by relation strength.We have also considered the maximal clique search algorithm for event division, which requires that any two arguments of the same event must be accurately judged as coevent relation. Assuming that an event hasn arguments, even if only one-way relations are considered, n ∗ (n − 1)/2 coevent relations need to be extracted. The missing of just one relation can lead to serious errors in event division. As shown in Figure 3, there is a complete event with 5 arguments [A, B, C, D, E], if the relation <B, D> is missing, it will be divided into two incomplete events [A, B, C, E] and [A, C, D, E].Figure 3
An error case of event division based on the maximal clique search algorithm.Although the maximal clique search algorithm is more complete in theory, its application conditions are too demanding. In contrast, our proposed algorithm only needsn − 2 coevent relations to divide a basic event, which not only reduces the application conditions of the algorithm but also has stronger fault tolerance. This point will be confirmed in the experiments.


## 2.3.1. Event-Cluster Construction

Based on the predicted role relation triplets, a special structure called event-cluster is constructed around each key argument. Every event-cluster includes a key argument and related nonkey arguments, all of which belong to the same event type. An event-cluster can be classified as a single event-cluster or compound event-cluster according to the number of basic events included.According to the construction rules, there is only one key argument in an event-cluster, but there may be several arguments on a nonkey role, which is called a multivalued role. The higher the proportion of multi-valued roles in an event-cluster, the more likely it contains multiple events. If there are more than one subkey argument and the proportion of multivalued roles exceeds a certain threshold, it will be judged as a compound event-cluster; otherwise, it will be judged as a single event-cluster. The threshold can be adjusted as a hyperparameter.


## 2.3.2. Event Division

A compound event-cluster needs to be further divided into basic events. In detail, we first create one basic event for each subkey argument and then assign arguments to other roles of each event according to the predicted coevent relations. If there are more than one candidate arguments for a role, it can be selected by relation strength.We have also considered the maximal clique search algorithm for event division, which requires that any two arguments of the same event must be accurately judged as coevent relation. Assuming that an event hasn arguments, even if only one-way relations are considered, n ∗ (n − 1)/2 coevent relations need to be extracted. The missing of just one relation can lead to serious errors in event division. As shown in Figure 3, there is a complete event with 5 arguments [A, B, C, D, E], if the relation <B, D> is missing, it will be divided into two incomplete events [A, B, C, E] and [A, C, D, E].Figure 3
An error case of event division based on the maximal clique search algorithm.Although the maximal clique search algorithm is more complete in theory, its application conditions are too demanding. In contrast, our proposed algorithm only needsn − 2 coevent relations to divide a basic event, which not only reduces the application conditions of the algorithm but also has stronger fault tolerance. This point will be confirmed in the experiments.


## 3. Experiments

### 3.1. Evaluation Dataset

#### 3.1.1. Dataset

Our proposed method will be evaluated on the ChFinAnn dataset, which is also applied by DCFEE [9], Doc2EDAG [10], and DE-PPN [2]. ChFinAnn is a DEE dataset without trigger-word labelling, which is automatically annotated by distant supervision. The dataset contains 5 financial event types: Equity Freeze (EF), Equity Repurchase (ER), Equity Underweight (EU), Equity Overweight (EO), and Equity Pledge (EP). It includes a total of 32,040 documents, about 30% of which contain multiple events.


#### 3.1.2. Evaluation Metric

We follow the evaluation metrics used in Doc2EDAG and DE-PPN. For each predicted event, the most similar ground truth is selected without replacement to calculate Precision, Recall, and F1-score. Microaveraged role-level scores are considered as the final metric for each event type. For the global performance, the previous literature only reported the macro-averaged F1-score of all event types, which does not consider the imbalance of sample distribution. Therefore, we will additionally report the microaveraged F1-score that can better reflect the practical performance.


### 3.2. Experimental Setup

The head size of the GlobalPointer in the relation extraction model is set to 64. The text encoder adopts the char-based RoFormer (Chinese_roformer_char_L-12_H-768_A-12), whose parameter scale is comparable to BERT and the vocabulary is reduced to 12000. Since most of the documents in the dataset are within 2000 tokens, the maximum encoding length (max_len) of RoFormer is set to 2000. The threshold of multivalued roles proportion for multi-event judgement is set to 0.2. For each event type, we select the role with the highest discrimination as the key role and select the second highest one as the subkey role. The selection results are shown in Table1.Table 1
The result of key and subkey role selection.
Event typeKey roleSubkey roleEquityPledgePledgedSharesPledgeeEquityRepurchaseRepurchasedSharesLowestTradingPriceEquityOverweightTradedSharesEndDateEquityUnderweightTradedSharesEndDateEquityFreezeFrozeSharesLegalInstitutionAll experiments run on a workstation with RTX3090. The model is trained for 20 epochs with the optimizer of exponential moving average Adam and the learning rate of 1e − 5. The best performing model on the development set is saved and its performance on the test set is used as the final test result.


### 3.3. Main Results

#### 3.3.1. Baseline

Our framework DEERE is compared with the previous SOTA methods as follows: DCFEE [9] proposed a DEE method based on key-event detection and argument completion. DCFEE-O and DCFEE-M are the single-event version and multievent version, respectively. Doc2EDAG [10] is an end-to-end model that transforms event table filling into a path expending of entity-based directed acyclic graph. GreedyDec is a simple version of Doc2EDAG, which only fills one event table entry greedily. DE-PPN [2] is the most recent SOTA model that aggregates the document-level context to predict events in parallel. DE-PPN-1 is the simple version that only generates one event.


#### 3.3.2. Compare with SOTA

As shown in Table2, our framework DEERE achieves the best performance in all 5 event types compared to baseline methods. Specifically, DEERE improves 1.0, 5.4, 6.0, 2.1, and 3.5 F1-score over the DE-PPN on the event type EF, ER, EU, EO, and EP, respectively. Table 3 shows that our model gets the highest score in both single-event and multievent parts for every event type. As to global performance, the macro F1-score of DEERE is 3.6 higher than that of DE-PPN (3.7 higher for the single-event part and 3.4 higher for the multi-event part). In addition, the micro F1-score of DEERE on the test set is 83.7 (90.8 for the single-event part and 75.9 for the multi-event part), which is not reported in the previous literature.Table 2
Precision (P), recall (R), and F1-score (F1) for 5 event types.
ModelsEFEREUEOEPPRF1PRF1PRF1PRF1PRF1DCFEE-O66.041.651.184.581.883.162.735.445.351.442.646.664.363.663.9DCFEE-M51.840.745.683.778.080.849.539.944.242.547.544.959.866.462.9GreedyDec79.546.858.983.374.978.968.740.851.269.740.651.385.748.762.1Doc2EDAG77.164.570.291.383.687.380.265.071.882.169.075.080.074.877.3DE-PPN-177.855.864.975.676.476.076.463.769.477.154.363.785.543.057.2DE-PPN78.269.473.589.385.687.669.779.974.481.071.375.883.873.778.4DEERE (ours)79.470.374.594.591.593.083.377.580.481.274.977.983.580.381.9Table 3
F1-score for all event types on single-event (S.) and multievent (M.) parts.
ModelsEFEREUEOEPMacro-avgMicro-avgS.M.S.M.S.M.S.M.S.M.S.M.S&MS.M.S&MDCFEE-O56.046.586.754.148.541.247.745.268.461.161.549.658.0—DCFEE-M48.443.183.853.448.139.647.442.067.060.058.947.755.7—GreedyDec75.940.881.749.862.234.665.729.488.542.374.839.460.5—Doc2EDAG80.061.389.468.477.464.679.469.585.572.582.367.376.3—DE-PPN-182.446.378.353.982.245.678.139.382.838.580.744.766.2—DE-PPN82.163.589.170.579.766.780.669.688.073.283.968.777.9—DEERE (ours)84.865.495.671.284.575.183.771.189.277.687.672.181.590.875.983.7


### 3.4. Ablation Experiments

To verify the effect of the two key mechanisms in our framework, we conducted a series of ablation experiments. As shown in Table4, RoleSelection + means to select the key roles and subkey roles according to the discrimination for events, and RoleSelection-means to select them sequentially from the role list. EventDivision + means to perform the event division operation on event-clusters, EventDivision-means to treat every event-cluster as a single event.Table 4
F1-score of ablation tests on DEERE variants.
RoleSelectionEventDivisionEFEREUEOEPMacro-avgMicro-avg++74.593.080.477.981.981.583.7+−72.792.980.778.481.681.383.5−+72.692.279.875.079.679.881.8−−70.091.974.368.067.674.474.6Test results show that using either of the two mechanisms can improve the performance, but the improvements overlap significantly when both mechanisms are used. The reason is that after using role selection, the situation that the same key argument involves multiple events is greatly reduced, and the effect of event division will be relatively limited. Although the performance of using only role selection on the ChFinAnn dataset is close to that of using both mechanisms simultaneously, it does not mean that event division is dispensable. If the key roles selected in other EE tasks are not so ideal (i.e., the discrimination for events is not high enough), event division will become more important.In addition, we also test the event division method based on maximal clique search. The micro F1-score is 81.3 with role selection and 76.8 without role selection. Compared with our event division method, it drops by 2.4 and 5.0, respectively.


### 3.5. Effect of Maximum Encoding Length

We investigate the influence of the max_len of the text encoder on the performance. As shown in Figure4, with the increase of max_len, the F1-score and Recall increases rapidly, while the Precision is basically flat (with a slight decrease). The model performance no longer improves significantly after the max_len exceeds 2000, which is consistent with our default value. The results affirm the idea of global context awareness and the feasibility of whole document encoding.Figure 4
Effect of maximum encoding length.


### 3.6. Effect of Relation Extraction Model

Our framework does not rely on specific relation extraction models and text encoders, which can be freely chosen. With other settings unchanged, we try to adopt different combinations of relation extraction models and text encoders. The training results are shown in Figure5, where Casrel [5] is a two-stage entity-relationship joint extraction model, and NEZHA [11] is another pretrained language model using relative position encoding. The performance of EE is basically synchronized with that of relation extraction, and the combination of GPLinker and RoFormer obtains the best score. The results show the single-stage relation extraction model (GPLinker) is significantly better than the multistage model (Casrel), while the effect of text encoder changes is much smaller.Figure 5
Effect of relation extraction model.


## 3.1. Evaluation Dataset

### 3.1.1. Dataset

Our proposed method will be evaluated on the ChFinAnn dataset, which is also applied by DCFEE [9], Doc2EDAG [10], and DE-PPN [2]. ChFinAnn is a DEE dataset without trigger-word labelling, which is automatically annotated by distant supervision. The dataset contains 5 financial event types: Equity Freeze (EF), Equity Repurchase (ER), Equity Underweight (EU), Equity Overweight (EO), and Equity Pledge (EP). It includes a total of 32,040 documents, about 30% of which contain multiple events.


### 3.1.2. Evaluation Metric

We follow the evaluation metrics used in Doc2EDAG and DE-PPN. For each predicted event, the most similar ground truth is selected without replacement to calculate Precision, Recall, and F1-score. Microaveraged role-level scores are considered as the final metric for each event type. For the global performance, the previous literature only reported the macro-averaged F1-score of all event types, which does not consider the imbalance of sample distribution. Therefore, we will additionally report the microaveraged F1-score that can better reflect the practical performance.


## 3.1.1. Dataset

Our proposed method will be evaluated on the ChFinAnn dataset, which is also applied by DCFEE [9], Doc2EDAG [10], and DE-PPN [2]. ChFinAnn is a DEE dataset without trigger-word labelling, which is automatically annotated by distant supervision. The dataset contains 5 financial event types: Equity Freeze (EF), Equity Repurchase (ER), Equity Underweight (EU), Equity Overweight (EO), and Equity Pledge (EP). It includes a total of 32,040 documents, about 30% of which contain multiple events.


## 3.1.2. Evaluation Metric

We follow the evaluation metrics used in Doc2EDAG and DE-PPN. For each predicted event, the most similar ground truth is selected without replacement to calculate Precision, Recall, and F1-score. Microaveraged role-level scores are considered as the final metric for each event type. For the global performance, the previous literature only reported the macro-averaged F1-score of all event types, which does not consider the imbalance of sample distribution. Therefore, we will additionally report the microaveraged F1-score that can better reflect the practical performance.


## 3.2. Experimental Setup

The head size of the GlobalPointer in the relation extraction model is set to 64. The text encoder adopts the char-based RoFormer (Chinese_roformer_char_L-12_H-768_A-12), whose parameter scale is comparable to BERT and the vocabulary is reduced to 12000. Since most of the documents in the dataset are within 2000 tokens, the maximum encoding length (max_len) of RoFormer is set to 2000. The threshold of multivalued roles proportion for multi-event judgement is set to 0.2. For each event type, we select the role with the highest discrimination as the key role and select the second highest one as the subkey role. The selection results are shown in Table1.Table 1
The result of key and subkey role selection.
Event typeKey roleSubkey roleEquityPledgePledgedSharesPledgeeEquityRepurchaseRepurchasedSharesLowestTradingPriceEquityOverweightTradedSharesEndDateEquityUnderweightTradedSharesEndDateEquityFreezeFrozeSharesLegalInstitutionAll experiments run on a workstation with RTX3090. The model is trained for 20 epochs with the optimizer of exponential moving average Adam and the learning rate of 1e − 5. The best performing model on the development set is saved and its performance on the test set is used as the final test result.


## 3.3. Main Results

### 3.3.1. Baseline

Our framework DEERE is compared with the previous SOTA methods as follows: DCFEE [9] proposed a DEE method based on key-event detection and argument completion. DCFEE-O and DCFEE-M are the single-event version and multievent version, respectively. Doc2EDAG [10] is an end-to-end model that transforms event table filling into a path expending of entity-based directed acyclic graph. GreedyDec is a simple version of Doc2EDAG, which only fills one event table entry greedily. DE-PPN [2] is the most recent SOTA model that aggregates the document-level context to predict events in parallel. DE-PPN-1 is the simple version that only generates one event.


### 3.3.2. Compare with SOTA

As shown in Table2, our framework DEERE achieves the best performance in all 5 event types compared to baseline methods. Specifically, DEERE improves 1.0, 5.4, 6.0, 2.1, and 3.5 F1-score over the DE-PPN on the event type EF, ER, EU, EO, and EP, respectively. Table 3 shows that our model gets the highest score in both single-event and multievent parts for every event type. As to global performance, the macro F1-score of DEERE is 3.6 higher than that of DE-PPN (3.7 higher for the single-event part and 3.4 higher for the multi-event part). In addition, the micro F1-score of DEERE on the test set is 83.7 (90.8 for the single-event part and 75.9 for the multi-event part), which is not reported in the previous literature.Table 2
Precision (P), recall (R), and F1-score (F1) for 5 event types.
ModelsEFEREUEOEPPRF1PRF1PRF1PRF1PRF1DCFEE-O66.041.651.184.581.883.162.735.445.351.442.646.664.363.663.9DCFEE-M51.840.745.683.778.080.849.539.944.242.547.544.959.866.462.9GreedyDec79.546.858.983.374.978.968.740.851.269.740.651.385.748.762.1Doc2EDAG77.164.570.291.383.687.380.265.071.882.169.075.080.074.877.3DE-PPN-177.855.864.975.676.476.076.463.769.477.154.363.785.543.057.2DE-PPN78.269.473.589.385.687.669.779.974.481.071.375.883.873.778.4DEERE (ours)79.470.374.594.591.593.083.377.580.481.274.977.983.580.381.9Table 3
F1-score for all event types on single-event (S.) and multievent (M.) parts.
ModelsEFEREUEOEPMacro-avgMicro-avgS.M.S.M.S.M.S.M.S.M.S.M.S&MS.M.S&MDCFEE-O56.046.586.754.148.541.247.745.268.461.161.549.658.0—DCFEE-M48.443.183.853.448.139.647.442.067.060.058.947.755.7—GreedyDec75.940.881.749.862.234.665.729.488.542.374.839.460.5—Doc2EDAG80.061.389.468.477.464.679.469.585.572.582.367.376.3—DE-PPN-182.446.378.353.982.245.678.139.382.838.580.744.766.2—DE-PPN82.163.589.170.579.766.780.669.688.073.283.968.777.9—DEERE (ours)84.865.495.671.284.575.183.771.189.277.687.672.181.590.875.983.7


## 3.3.1. Baseline

Our framework DEERE is compared with the previous SOTA methods as follows: DCFEE [9] proposed a DEE method based on key-event detection and argument completion. DCFEE-O and DCFEE-M are the single-event version and multievent version, respectively. Doc2EDAG [10] is an end-to-end model that transforms event table filling into a path expending of entity-based directed acyclic graph. GreedyDec is a simple version of Doc2EDAG, which only fills one event table entry greedily. DE-PPN [2] is the most recent SOTA model that aggregates the document-level context to predict events in parallel. DE-PPN-1 is the simple version that only generates one event.


## 3.3.2. Compare with SOTA

As shown in Table2, our framework DEERE achieves the best performance in all 5 event types compared to baseline methods. Specifically, DEERE improves 1.0, 5.4, 6.0, 2.1, and 3.5 F1-score over the DE-PPN on the event type EF, ER, EU, EO, and EP, respectively. Table 3 shows that our model gets the highest score in both single-event and multievent parts for every event type. As to global performance, the macro F1-score of DEERE is 3.6 higher than that of DE-PPN (3.7 higher for the single-event part and 3.4 higher for the multi-event part). In addition, the micro F1-score of DEERE on the test set is 83.7 (90.8 for the single-event part and 75.9 for the multi-event part), which is not reported in the previous literature.Table 2
Precision (P), recall (R), and F1-score (F1) for 5 event types.
ModelsEFEREUEOEPPRF1PRF1PRF1PRF1PRF1DCFEE-O66.041.651.184.581.883.162.735.445.351.442.646.664.363.663.9DCFEE-M51.840.745.683.778.080.849.539.944.242.547.544.959.866.462.9GreedyDec79.546.858.983.374.978.968.740.851.269.740.651.385.748.762.1Doc2EDAG77.164.570.291.383.687.380.265.071.882.169.075.080.074.877.3DE-PPN-177.855.864.975.676.476.076.463.769.477.154.363.785.543.057.2DE-PPN78.269.473.589.385.687.669.779.974.481.071.375.883.873.778.4DEERE (ours)79.470.374.594.591.593.083.377.580.481.274.977.983.580.381.9Table 3
F1-score for all event types on single-event (S.) and multievent (M.) parts.
ModelsEFEREUEOEPMacro-avgMicro-avgS.M.S.M.S.M.S.M.S.M.S.M.S&MS.M.S&MDCFEE-O56.046.586.754.148.541.247.745.268.461.161.549.658.0—DCFEE-M48.443.183.853.448.139.647.442.067.060.058.947.755.7—GreedyDec75.940.881.749.862.234.665.729.488.542.374.839.460.5—Doc2EDAG80.061.389.468.477.464.679.469.585.572.582.367.376.3—DE-PPN-182.446.378.353.982.245.678.139.382.838.580.744.766.2—DE-PPN82.163.589.170.579.766.780.669.688.073.283.968.777.9—DEERE (ours)84.865.495.671.284.575.183.771.189.277.687.672.181.590.875.983.7


## 3.4. Ablation Experiments

To verify the effect of the two key mechanisms in our framework, we conducted a series of ablation experiments. As shown in Table4, RoleSelection + means to select the key roles and subkey roles according to the discrimination for events, and RoleSelection-means to select them sequentially from the role list. EventDivision + means to perform the event division operation on event-clusters, EventDivision-means to treat every event-cluster as a single event.Table 4
F1-score of ablation tests on DEERE variants.
RoleSelectionEventDivisionEFEREUEOEPMacro-avgMicro-avg++74.593.080.477.981.981.583.7+−72.792.980.778.481.681.383.5−+72.692.279.875.079.679.881.8−−70.091.974.368.067.674.474.6Test results show that using either of the two mechanisms can improve the performance, but the improvements overlap significantly when both mechanisms are used. The reason is that after using role selection, the situation that the same key argument involves multiple events is greatly reduced, and the effect of event division will be relatively limited. Although the performance of using only role selection on the ChFinAnn dataset is close to that of using both mechanisms simultaneously, it does not mean that event division is dispensable. If the key roles selected in other EE tasks are not so ideal (i.e., the discrimination for events is not high enough), event division will become more important.In addition, we also test the event division method based on maximal clique search. The micro F1-score is 81.3 with role selection and 76.8 without role selection. Compared with our event division method, it drops by 2.4 and 5.0, respectively.


## 3.5. Effect of Maximum Encoding Length

We investigate the influence of the max_len of the text encoder on the performance. As shown in Figure4, with the increase of max_len, the F1-score and Recall increases rapidly, while the Precision is basically flat (with a slight decrease). The model performance no longer improves significantly after the max_len exceeds 2000, which is consistent with our default value. The results affirm the idea of global context awareness and the feasibility of whole document encoding.Figure 4
Effect of maximum encoding length.


## 3.6. Effect of Relation Extraction Model

Our framework does not rely on specific relation extraction models and text encoders, which can be freely chosen. With other settings unchanged, we try to adopt different combinations of relation extraction models and text encoders. The training results are shown in Figure5, where Casrel [5] is a two-stage entity-relationship joint extraction model, and NEZHA [11] is another pretrained language model using relative position encoding. The performance of EE is basically synchronized with that of relation extraction, and the combination of GPLinker and RoFormer obtains the best score. The results show the single-stage relation extraction model (GPLinker) is significantly better than the multistage model (Casrel), while the effect of text encoder changes is much smaller.Figure 5
Effect of relation extraction model.


## 4. Related Work

SEE uses only features obtained from intrasentences, and traditional feature engineering-based approaches [12, 13], cannot be adapted to tasks that rely on complex semantic relationships. Recent work on EE is based on deep learning to automatically learn features, mainly using pipeline and joint models. Pipeline methods [14, 15], whether using CNN or RNN, use a pipeline approach to split the extraction process into two separate processes, extracting event trigger words and detecting arguments. This approach inevitably causes error transfer and makes it difficult to capture long-distance dependent information. To reduce error transfer, joint methods [16, 17], consider simultaneous extraction of trigger words and arguments. To address the problem of overlapping roles, pretrained language models [18, 19], are used to model intrasentence and intersentence contextual information, improving the accuracy of the task overall.Early classification models [20, 21], divide DEE into two subtasks: recognition of event descriptors and detecting arguments, using SVM as classifiers. Neural network-based classification models [22, 23], use word embeddings as the input to the decision tree, and then the structured information of the document is obtained through the integration of information.To address the argument-scattering, one solution is to transform the extraction task into a sequence annotation task, which dynamically fuses two different levels of information at the sentence level and the document-level [24]. Another solution is to use a sentence as the event-centered sentence for arguments complementation. DCFEE [9] is based on sequence annotation, main event discovery, and arguments complementation strategy to construct the extraction model, which solves the argument-scattering to a certain extent. For the multievents, Doc2EDAG [10] utilizes an end-to-end approach to integrate arguments scattered across different documents and transforms the document-level event table filling task into an entity-based path expansion task for directed acyclic graphs. The multilayer bidirectional network MLBiNet [25] fuses cross-sentence semantic and associative event information to enhance the discrimination of each event mention. In addition, there are methods [26, 27], that transform EE into other tasks such as reading comprehension and intelligent quizzing.


## 5. Conclusion

We propose a novel framework (DEERE) that cleverly transforms the task of DEE into a relation extraction task. The new SOTA performance on the ChFinAnn dataset illustrates the reasonability of the framework design, and ablation experiments verify the effectiveness of the key mechanisms in the framework. The single-stage relation extraction model can mitigate error propagation, and the event integration algorithm with fault tolerance can compensate for some errors of relation extraction. Increasing the maximum length of the text encoder is beneficial to improve document awareness, but it also requires more computational resources. The ChFinAnn dataset used in our experiments has a considerable scale, but labelled data in other application scenarios are often insufficient. In future work, we will focus on few-shot event extraction and data augmentation for it.



---

*Source: 2742796-2022-08-26.xml*