From Simple to Complex: A Progressive Framework for Document-level Informative Argument Extraction
Authors: Quzhe Huang
Yanxi Zhang
Dongyan Zhao
Venue: EMNLP 2023 Findings (2023)
Keywords: Document-level event extraction
informative argument extraction
simple-to-complex prediction

Abstract: Document-level Event Argument Extraction (EAE) requires the model to extract arguments of multiple events from a single document. Considering the underlying dependencies between these events, recent efforts leverage the idea of "memory", where the results of already predicted events are cached and can be retrieved to help the prediction of upcoming events. These methods extract events according to their appearance order in the document, however, the event that appears in the first sentence does not mean that it is the easiest to extract. Existing methods might introduce noise to the extraction of upcoming events if they rely on an incorrect prediction of previous events. In order to provide more reliable memory, we propose a simple-to-complex progressive framework for document-level EAE. Specifically, we first calculate the difficulty of each event and then, we conduct the extraction following a simple-to-complex order. In this way, the memory will store the most certain results, and the model could use these reliable sources to help the prediction of more difficult events. Experiments on WikiEvents show that our model outperforms SOTA by 1.4\% in F1, indicating the proposed simple-to-complex framework is useful in the EAE task.

From Simple to Complex: A Progressive Framework for Document-level Informative Argument Extraction
Quzhe Huang Yanxi Z X Dongyan Zhao
Wangxuan Institute of Computer Technology, Peking University
School of Intelligence Science and Technology, Peking University *Center for Data Science, Peking University
National Key Laboratory of General Artificial Intelligence zhaady Yēpku. edu. cn zhangyxēstu. pku. edu.cn cn
zhaody
Abstract
Generation-basee document-level EAE methods are widely used in recent works (Li et al., 2021; Du et al., 2022; Du and Ji, 2022; Huang et al., 2022). Among them, one line of studies (Li et al., 2021; Huang et al., 2022) treats each event inde- pendently and ignores the underlying correlations between events in real-world documents. Other works (Du et al., 2022: Du and Ji, 2022) start to con- sider inter-event dependencies and model them by introducing the idea of "memory", where event pre- dictions (e.g., arguments. roles) are cached and can be retrieved to help the prediction of the upcoming events in a document. However, since these meth- ods still use front-to- to-back prediction- eextiction-extractinnn events according to their appearance order in a doc- ument, an event can only rely the predictions of events that appeared before it. Besides, the predic- tion of an event is cached regardless of its quality, whereas false predictions may be cached first, mis- leading the prediction of the following events.
Document-level Event Argument Extraction (EAE) requires the model to extract arguments of multiple events from a single document. Considering the underlying dependencies be- tween these events, recent efforts leverage the idea of "memory". where the results of al- ready predicted events are cached and can be retrieved help the prediction of upcoming events. These methods extract events according their appearance order the document, how- ever, the event that appears in the first sentence does not mean that it is the easiest to extract. Existing methods might introduce noise the extraction of upcoming events if they rely on an incorrect prediction previous events. In order provide more reliable memory, pro- pose simple-to capple-to-coo complex progressive frame- work for document-level EAE. Specifically, we first calculate the difficulty of each event and then, we conduct the extraction following simple-to-comple order. In this way, the mem- ory will store the most certain results, and the model could use these reliable sources to help the prediction of more difficult events. Experi- ments on WIKIEVENTS show that our model outperforms SOTA by 1.4% in F1, indicating the proposed to-opleo framework is useful in the EAE task. The code is avail- able https://github com nomm/zhaggxx8177/ symplee__t_comllex
In general, using current retrieval-based meth- ods to model inter-event dependencies faces two main challenges: (1) front-to-back prediction only partially models inter-event dependencies, where the dependency links from an event to all the events that appeared after it are ignored; (2) incorrect pre- dictions may be cached first and retrieved by the upcoming events.
Considering the challenges, we propose the himple-to-copple framework. First, we calculate the difficulty of each event, where the difficulty is defined as the average probability that the model assigns to the arguments of an event. We also cali- brate the argument probabilities before using them to ensure they truly reflect how certain the model is each argument. Second, reorder events in document from simple to complex and predict them accordingly. In this way, the model could use the simple instance to help the prediction of the difficult ones, no matter whether the simple events appear before or after the difficult ones
1 Introduction
Document-level Event Argument Extraction (EAE) aims at identifying the participants of multiple events from document and classifying them into proper roles (Li et al., 2021; Du et al., 2022; Xu et al., 2022; Huang al., 2022; Yang al., 2023). Understanding events in documents crucial for a line of downstream tasks, such as machine read- ing comprehension (Han et al., 2021) and dialogue systems (Zhang et al., 2020).
Equal Contribution.

---

the original document.
(Section 3.1). Here, we use retrieval to augment model input and apply constrained decoding to im- prove model output, both leveraging inter-event dependencies to benefit prediction. Second, we elaborate on how to define and calculate the diffi- culty of an event, and how to reorder events in a document from simple to complex for simple-to- complex prediction (Section 3.2).
We conduct experiments on widely used bench- marks WIKIEVENTS(Li et al., 2021), and our pro- posed simple-to-complee framework outperforms the previous SOTA by 1.4% in F1, illustrating the effectiveness of our method. Further analy- ses show the calibration of probability is very im- portant when calculating the difficulty of different events and the success of our framework relies the better usage of dependency between an event and the events that appear after it in the document.
3.1
IAE Model
Our memory-enhanced IAE model is based on a generative model. When calculating the difficulty of each event well generating (the arguments of) events in a document from simple to complex, we use the same generative model. In this study, we assume that each event has a prediction order and events in a document are predicted according to that order. After reordering, the prediction order of an event may change and further change the retrieved prediction of that event.
2 Task Definition
In this work, we focus on document-level Informa- tive Argument Extraction' (IAE) (Li et al., 2021), where informative arguments are far more distant than local/uninformativee ones and provide more useful information about an event. We formulate document-level IAE as a generative template-filling task following Li et al. (2021) and Du et al. (2022). Given a document D with triggers marked (using a special token <tgr>), our goal is to extract all the arguments of E to fill in the slots of the event template T.
Model Input & Output The generation of event E in document D is conditioned on the (I) prediction order E {1, 2...,, me): the or- der of predicting (the arguments of) E, where ne denotes the number of events D; (2) event con- text e: the concatenation of E's context words (a continuous span in D close to E's trigger) and E's template; (3) retrieved prediction m the predic- tion of an event appeared before E retrieved from the document memory, a data structure that caches the predictions of already predicted events in D. To sum up, the input of event for the model is:
Each event consists of (1) an event trigger. which has specific type E (we use E represent an event); (2) a series of event arguments, each corresponding to a specific role. In the event ontol- ogy, event types and argument roles are pre-defined, and event templates depicting the relationship be- tween the argument roles of an event are also pro- vided. For example, the template for E Attack in the KAIROS ontology² is:

 m 
 
 T 
 I1, I2, ,
Tn [EOS]
<arg1> attacked <arg2> using <arg3> <arg4> place
where I1, 22,32..... are the context words and T is E's unfilled template, and these two parts form the event context c. The prediction of E is filled template, with each <arg> placcholder replaced by the predicted argument (or not), as shown in Figure If there are multiple arguments for the same slot, the arguments are connected with "and".
where each slot <argx> is a placeholder for ar- guments with specific role. We replace all the <argx>s in a template with a special token <arg> before using them as input. If the model extracts an argument, then <arg> will be replaced by the argument. If no, the placeholder <arg> remains.
Detrieval-Augmentee Generation In the input stage (both for training and testing), we augment our model with similarity- based retrieval follow- ing Du et al. (2022) to make it capable of find- ing argument mentions beyond the context of event, especially informative ones (Li et al., 2021). When predicting the i-th event E, in a document D, the snapshot of the document memory is m {m1, m2...., m22....,,;11,, where m, denotes the pre- diction of the k-th event. We calculate the cosine
3 Method
In this section, we illustrate our framework based on simple-to-complee prediction (Figure 1). First, we introduce our memory-enhancee IAE model
Name mentions are more informative than nominal men- tions, and pronouns are the least informative. 'https: h/ww///wdd 1dc. upenn. wwwpn.pdu/cool......ppp /tbu/collaboratioons/ current-projects

---

arr transported <arg> <arg> from <arg place <arg> place 


 Thailand July forcibly <igr> repatriatd <tgr> more than Lighurs to China [EOS]
D
E
Ez
E3
Eg
E
E2
D
C2
02 2
0/
C2
m
m
R'
m
mg
Generative Model
Generative Model
F2B Memory
0/ = 3
S2C Memory
m
7
arg P2
M3
m
S2C Reordering
Bounded Constraints
77
Transporter
Passanger/Artifact
Destination
Thailand transported Uighurs in <arg> from <arg> place to China place
Figure 1:
Our simple-to to complex progressive framework for document-leves IAE. First, we calculate the difficulty each event document and obtain new prediction order for that event. Second, we reorder events D from simple complex, and predict them accordingly. S2C denotes Simple-to-Complee while F2B denotes Front-to-Back. Here, we plot the process of predicting the arguments of Egg
similarity between E is context c and each predic- tion in m using S-BERT (Reimers and Gurevych, 2019) embeddings, and select the prediction with the highest score as additional input to help the prediction of E;:
another, it may In the example above, "Mike" will never be de- coded as the ATTACKER of Eg once it decoded as the DETAINEE of E., even the prediction of DETAINER is incorrect. To alleviate such er- ror propagation, we disable the constraints when (1) the model is not certain about the prediction of an argument. The certainty of an argument can be measured by the calibrated probability of decoding it. Low probability intuitively implies the model is confident about this prediction, while we have also found that high probability (e.g. 0.8) corresponds to low prediction accuracy, which is shown Figure 3. Therefore, we set both lower and upper bounds for argument probabilities to ex- clude possibly incorrect constraints, and we refer to our pruned constraints bounded constraints. The heuristics of selecting bounds are discussed in Appendix A.3.
(Du et al., 2022).
exp exp.f(e; f(e,, m2)
score(mylcc) mm exp [(,, my)
f(ai, m2) SBERT (: R
m
arg max score( my
(3)
where SBERT() denotes S-BERT encoding, m denotes the retrieved prediction that E, relies on.
Constrained Decoding In the output stage, we introduce argument pair constraints following Du et al. (2022) to constrain the decoding of arguments with conflicting roles. For example, if the DE- TAINEE of event E, is "Mike", then "Mike" can not be decoded as the ATTACKER of another event E₁ (happened after E,). since "Mike" already in jail. Here, "Mike" as DETAINEF, and "Mike" as Ar- TACKER is an argument pair constraint. However, once an incorrect prediction is used to constrain
3.2 Simple-to-Complex Reordering
Since an event usually contains multiple arguments, we reckon the difficulty of an event lies in the av- erage difficulty of predicting its arguments. In this

---

section, we will elaborate on how calculate the difficulty of an event as well as how to reorder events in document by the difficulty and predict them from simple to complex.
2017). We have also found similar problems in our model, which is discussed Section 5.1. There- fore, we should calibrate these probabilities before using them for our Specifically, we adopt temperature scaling (Guo et al., 2017; Desai and Durrett, 2020), simple and effective method for calibration. In this work, the temperature T is selected by minimizing the Ex- pected Calibration Error (ECE) (Pakdaman Naeini et al., 2015) on the validation set, and we denote the temperature with the lowest ECE T".
reordering.
Suppose the set of prediction orders of events in a document D bc = {o1, .2..... )nc) ), where represents the i-th appeared event is the 0;-th to be predicted, then front-to-back prediction satisfies 0r i, ..., Mee Suppose the probabilities of decoding the arguments of the i-th event E; in a document be:
Accounting for calibration, there should be a revision the calculation of p!" Suppose the logits vector of the j-th predicted argument of the i-th event be z then:
arg pi PP: Ps
(1) (2)
(na)
(4)
where n, denotes the number of predicted argu- ments of Ej, and p; the generative model assigns to the j-th argument of the i-th event. The argument probability reflects the certainty of the generative model on predicting an argument, inversely proportional to our desired difficulty. Thus, we define the difficulty of predict- ing the arguments of Eg as:
(f) denotes the probability that
()
( P
max so softmar
(8)
T
where traverses each dimension of z, (5)
Experiments
d = { (1))) (2)
}
(5)
4.1
Dataset and Evaluation Metries
d = 1- - P. .j j- 1.,....,70 ne:
())
We evaluate our framework on WIKIEVENTS (Li et al., 2021) as it annotates all the events in a doc- ument (averagely events per document), while existing document-level datasets such as DocEE (Tong et al., 2022), RAMS (Ebner et al., 2020) and MUC-4 (Sundheim, 1992) only annotate at most 3 events per document. Also, it provides complete coreference annotation for document-level LAE. Its statisties are shown in Table 1.
(6)
where d0d denotes the difficulty of predicting the j-th argument of the i-th event. The difficulty of Eg is defined as the average difficulty of predicting its arguments,
take the average of d and obtain the difficulty of Eg:
deee mean 9
(d."))
If no arguments of E, are predicted, then d." That means E, will be placed to the rear, since it provides no arguments/roles that might benefit prediction. After calculating the difficulty of events in D, obtain o' (o,, oz.,.. ... .. Then, we can predict (the arguments of) events in D from simple to complex according to o'.
Train
Dev
Test
# Event Types
49
35
34
# Argument Types
57
32
44
Documents
206
20
20
set of prediction orders # Sentences
5262
378
492
# Events
3241
345
365
# Arguments
4413
411
556
Events (per doc)
15.73
17.25
18.25
The method described above assumes that the model providing the probabilities is well-calibrated, where confidence (the probability that a model as- signs to prediction) equals or nearly equals ac- curacy (the real correctness of prediction). In other words, high confidence corresponds high accuracy, and vice versa. However, some studies reveal that current Deep Neural Networks (DNNs) are prone to over-confidence, which implies that the model's confidence is not reliable (Guo et al.,
# Tokens (per doc)
789.33
643.75
712.00
Table WIKIEVENTS Statistics
We measure the Argument Identification (Arg-I) and Argument Classification (Arg-C) capabilities of our model following Li al. (2013). If an ar- gument span matches any of the gold informative arguments of the event, the argument is correctly identified. If the semantic role also matches, the

---

Argument Identification (Arg-I)
Argument Classification (Arg- C)
Models
Head Match
Coref Match
Head Match
Coref Match
R
F1
P
F1
R
F1
BERT-CRF
52.71
58,12
43.291
47.70
BART-Gen
58.62
55.64
57.091
62.84
59.64
61.19
54.02
51.27
52.611
57.47
54.55
55.971
w/ M
60.38
57.97
59.15
64.72
62.14
63.40
54.53
52.36
53.42
58.11
55.80
56.93
w/ M+C
61.79
58.88
60.30
66.16
63.04
64.56
55.70
53.08
54.36
59.32
56.52
57.88
w/ M+0 (S2C)
61.61
59.60
60.59
65.73
63.59
64.64
55.81
53.99
54.88
59.18
57.25
58.20
w/ M+O+C* (S2C-CD)
62.57
59.96
61.24
66.73
63.95
65.31
56.52
54.17
55.32
59.92
57.43
58.65'
Table 2: Performance (%) of document-level IAE WIKIEVENTS. M denotes document Memory, denotes original Constraints from Du et al. (2022). O denotes simple-to teae--to-ooppeee reordering, and C' denotes bounded Constraints. denotes cited results, denotes statistical significance compared with (Du al., 2022) (p < 0.05).
argument is considered correctly classified. Fol- lowing previous studies document-level IAE (Li et al., 2021; Du et al., 2022), adopt Head Word Match (Head F1) (Huang and Riloff, 2021) and Coreferential Match (Coref F1) (Ji and Grish- man, 2008) to judge whether the predicted argu- ment span matches the gold argument span. Head Word Match demands the first word of the predicted argument to match that of the gold argument, while Coreferential Match only needs the extracted argu- ment to be coreferential with the gold argument. We report the micro-P/R/F1 averaged three dif- ferent seeds.
methods on WIKIEVENTS to document-
level IAE, with average gain of 1.4% in Fl all four settings.
All models augmented with retrieval (i.e., w/ M) perform better compared with BERT-CRE and raw BART-Gen, showing the importance of modeling inter-event dependencies.
Compared to BART-Gen (w/ M), the addition of simple-to-complex reordering (S2C model) greatly improves FI, where Fl average increases by 1.34 in Arg-I and 1.42 in Arg-C. This improvement can be mainly attributed our simple-to-complex prediction paradigm, since it allows more inter-event dependency links, i.e., from an event to all the events that appeared after it.
4.2 Baselines
We compare our framework with a series of com- petitive baselines: (1) BERT-CRF (Shi and Lin, 2019),, a simple BERT-based model without incor- porating lexical or syntactic features for argument identification and classification.
After applying our bounded constraints (S2C- CD model), there is additional improve- ment in P and FI, which shows that incorrect constraints are effectively pruned.
(2) BART-Gen (Li al., 2021), conditional neural text gener- ation model that generates filled template for each event given the event template and context words. (3) BART-Gen (w/ M+C) (Du et al., 2022), a framework based on BART-Gen, which utilizes retrieval to augment model input and constructs argument pair constraints for decoding. It is the SOTA model on locument-level IAF, but still ex- tracts events according to their appearance order in the document. We also report the results of BART- Gen (w/ M) for comparison.
Analysis
5.1 Is Calibration Necessary?
What to Calibrate? For each argument, we fo- cus on its first token probability, and this proba- bility is what we aim calibrate. The reason for using the first token probabilities is that the gener- ation of the remaining tokens is highly dependent on the first token. As shown Figure 2, 87% the non con-first token probabilities are 2 0.9, while first token probabilities are better distributed, with only 53% of them 2 0.9.
4.3
Main Results
The main results for document-level IAE are pre- sented in Table 2. From the results, we can con-
clude that:
Why to Calibrate? Modern DNNs are prone to over-confidence, which implies that the model's
Our S2C-CD model outperforms all previous

---

Uncelibr ated Pubability Distribution (Fist)
Uncaiirrted Probabiity Distribution (Reraiining
Cellbrated Probebility Ditribution Ifrsk
[00,0.1]
[05,5.21
19 0.2.8.1
(84,071
207.8.8
853.00
250
200
250
300
500
10
20
40
50
Probrabiiiiin
Probatiaties
Probatiloos
Figure 2: UncalibratedCCalibratee probability distribution. The left two diagrams respectively show the uncalibrated first and non-first (remaining) token probability distribution, while diagram on right shows the calibrated first token probability distribution.
Arg-I
Arg-C:
Reliability Diagram
Models
before chlibration
Head
Coref
Head F1
Coref F1
atro ener
S2C
60.59
64.64
54.88
58.20
08
RC
59.15
63.40
53.42
56.93
RU
59.39
63.46
53.65
56.98
0.6
Table 3: Ablation (%) for simple-to-compler reorder- ing. RC denotes Reordering by Calibrated probabilities (simple-t--oomplex reordering), RU denotes Reordering by Uncalibrated probabilities.
0.
fidence) to align them with accuracy before using them for our compe-to-comple reordering.
Figure 3: The reliability diagram before/after calibra- tion. The dashed line represents zero error.
Influence of Uncalibrated Probabilities We conduct an ablation study to further explore what will happen if we reorder events using uncalibrated probabilities. As shown Table 3, if we order events by uncalibrated probabilities, FI only comparable to excluding simple-to-complex re- ordering from our S2C model. The performance is maximized only when the probabilities are cali- brated. Therefore, calibration is essential.
confidence is not reliable (Guo al., 2017). We have also found similar problems in our model. As shown in Figure 2, the first and non-fire non-first token probability distribution both exhibit a severe over- confidence phenomenon before calibration, with most probabilities 2 0.9. This suggests that the model tends to assign a high (i.e., + 1) probabil- ity to nearly all of the arguments, which not truly reflect how sure the model of each argu- ment. Also, over-confidence leads miscalibra- tion, which reflected in the reliability diagrum in Figure 3. The reliability diagram plots the relation between confidence and accuracy, and its defini- tion is discussed in Appendix A.2. As shown in Figure 3, most points on the orange curve (before calibration) are far from the zero error curve where confidence exactly equals accuracy, demonstrating that uncalibrated probabilities are unreliable. After calibration, the first token probability distribution becomes flat (Figure 2) and calibrated (Figure 3). Therefore, we should calibrate probabilities (con-
5.2 Difficulty Calculation Needs Memory?
In this section, we present two possible ways of calculating the difficulty of an event to explore their impact on performance. In Figure 4, we de- fine the first inference as step 1-2 (calculating the difficulty), and the second inference as step (pre- dicting the arguments reordered events).
The first way is utilized in our framework, where the same model for both inferences. When calculating the difficulty the first inference, we also use retrieval. There are two reasons for this. On the one hand, the model input is augmented with retrieval during training, so the input/prompt

---

Arg-I
Arg-C
Intervals
One Model
(m o1
Head F1
Coref FI
Head F1
Coref FI
0
S2C (R1)
60.59
64.64
54.88
58.20
S2C (R2)
60.79
64.65
54.55
57.67
E
Model
Reordering
(m C1 02)
arg Pi
0
Table 4: Performance (%) of using two different ways to calculate difficulty. R1 and R2 denote the results of the first and second way, respectively.
Model 2
Two Models
(m C1. oi) 01 2
performs only comparably with our S2C model. This implies the number of correct and incorrect constraints is nearly equal when we predict events from simple complex. By contrast, bounded constraints perform well, suggesting that the num- ber of incorrect constraints is indeed reduced and the correct constraints are (mostly) kept.
1
Model 1
Reordering
(c,, 01)
arg
P1
0 =
Figure 4: Two ways of calculating the difficulty.
format should be consistent during testing. the other hand, model is trained to predict the ar- guments of an upcoming event conditioned on the prediction of an already predicted event. Therefore, we should calculate "the difficulty of an event con- ditioned on the prediction of an earlier predicted event". However, the retrieved predictions of the event at both inferences usually different.
Arg-1
Arg-C
Models
Head FI
Coref FI
Head
Coref
S2C-CD
61.24
65.31
55.32
58.65
BCs (S2C)
60.59
64.64
54.88
58.20
+ +Oss
60.61
64.69
54.87
58.20
Table 5: Ablation (%) for bounded constraints. BCs denote Bounded Constraints, OCs denote Original Con- straints from Du et al. (2022).
The second way is to use separate models for each inference. At the first inference, we use a model trained without retrieval to calculate the "raw" difficulty of an event (i.e., do not condition on the prediction of an earlier predicted event). At the second inference. we train retrieval-augmentted model. This way removes the possible influence of retrieval on calculating the difficulty of an event, but the training overhead doubles.
Using the steps presented in Appendix A.3, we obtain the lower bound 0.5 and the upper bound 0.8, so we will disable constraint the probability of decoding the argument I 0.5 or 2 0.8. Based on this, we individually analyze the influence of the lower and upper bounds, as shown in Table 6. We have found that whether we remove the lower bound or the upper bound, the performance drops, indicating that both bounds are useful for reducing the number of incorrect constraints.
We conduct an experiment to compare these two ways, shown in Table 4. RI and R2 represent model and two models, respectively. R1 is comparable R2 in Arg-I, while RI is notably better than R2 Arg-C. The results suggest that using one model is generally better as to perfor- mance, so we should calculate the difficulty of an event conditioned on the prediction of an earlier predicted event. Besides, using one model is more time-efficient.
Arg-1
Arg C
Intervals
Coref F1
Head
Coref F1
0.55, 0.8)
61.24
65.31
55.32
58.65
[0.0, 0.8)
60.72
64.81
54.78
58.12
00.5, 1.0)
60.56
64.63
54.81
58.15
Table 6: Influence of the lower and upper bounds.
5.3 Influence of Bounded Constraints
In this section, we first compare our bounded con- straints with those presented in Du et al. (2022), then analyze the impact of the lower and upper bounds individually.
5,4 Case Study
In the case presented in Figure 5, would like to predict the arguments of event F. Damage, which describes the mental damage that Dzhokhar Tsamaev brought to the victims as a bomber.
In Table 5, we observe that when applying the original constraints (Du et al., 2022), the model

---

Victim
[51] Moments before federal judge sentenced him to death, Buston Marathon bomber [Dzhokhar Tsarnaev] rose to his feet Wednesday and apologized to the victims and their loved ones for thesf first time.
front-to -t-bbcckkk
[S3] Amid deep silence
courtróom, the 21-year-old ethnic Chechen said, "I sorry for the lives that taken, the suffering that I've caused you, for the damage that I've done irreparable damage" comple-t-coomplex
Injurer
[S7] In May, after a 12-week trial, Tsarnaev was found guilty of killing three people and injuring 264 in the April 15, 2013, bombing at the race, where and his brother, Tamerian, 26, set off two pressure crossur--cooker bombs near the finish line.
Retrieved
Predkted
front- tront-to-back
Role
Extracted Argument
Role
Extracted Argument
[death]
Damager
Nonal
E Damage [damage]
simple-to-compler
Injure [inuringl
Role
Extracted Argument
Role
Extracted Argument
Injurer
Ozhokhar Tsarnaev
Damager
Dahokhar Tsarnaev
Figure 5: Case study on simple-to-comples reordering.
front-to-back prediction, E can only access the pre- dictions carlier appeared events and retrieves Ej is prediction as additional input. The death of Dzhokhar Tsarnaev in Ej happened after E, wrongly restricting the prediction of "Dzhokhar Tsamaav" as the DAMAGER of E. By contrast, with our comple-to-compler prediction, E has the chance rely on the INJURER argument of a later appeared event Eg and obtain the correct DAM- AGER argument "Dzhokhar Tsarnaev". E2 de- scribes that Dzhokhar Tsarnaev injured 264 people in the bombing as the INJURER, thus bringing men- tal damage to them as the DAMAGER.
Unidentified
Spurious
Misclassified
HM 221 (49.0%)
198 (43.9%)
32 (7.1%)
CM 199 (48.4%) 176 (42.8%)
36 (8.8%)
Table 7: Errors made by our framework under Head Match (HM) and Coref Match (CM).
and their participants usually spread across the document in document-level EAE. We focus on document-level IAE (Li et al., 2021) in this work, which is more practical but more challenging. Li et al. (2021) constructed the WIRIEVENTS dataset and pioneered the research on document-level IAE. Compared with existing document-level EAE datasets such as DocEE (Tong et al., 2022), RAMS (Ebner al., 2020) and MUC-4 (Sundheim, 1992) that only annotate at most 3 events per document, WIKIE
Comparing these two prediction paradigms, prediction is better, mainly because it allows more inter-event depen- dency links. In this example, it is intuitive that E more similar to Σ2, since they respectively depict the physical damage and mental damage Dzhokhar Tsamaev brought to the victims. However, the dependency link from E to E2 is disabled when predicting events from front to back.
find that
annotates all the events in a docu- ment, with an average events per document. Also, provides complete coreference annotation for evaluating document-level IAE.
Recently, generation-based methods have been proposed for document-level EAE. Among them, template generation-based approaches (Li al., 2021; Huang al., 2022; Du al., 2022) are widely utilized. BART-Gen (Li et al., 2021) con- ditioned generation on event templates and con- text words but considered each event independently. Further, Du et al. (2022); Du and Ji (2022) intro- duced the idea of "memory" to document-level EAE, where predictions of already predicted events were utilized additional input. Although these methods can model inter-event dependeneies some extent, they ignore the dependency links from an event to all the events that appeared after a document. Besides, uncertain/false event predic-
5.5 Error Analysis
Table 7 summarizes the error types of our S2C-CD model. The errors mainly come from the inability to recognize argument span (around half), while only about 8% of identified arguments assigned incorrect semantic roles. Therefore, identifying the argument span more important than assigning a more accurate role to alrcady extracted arguments.
6
Related Work
6.1
Document-level EAE
Unlike sentence-level EAE (Li et al., 2014; Du and Cardie, 2020; Xiangyu et al., 2021), events

---

tions may be cached first and retrieved by future events, misleading their prediction.
Acknowledgements
This work is supported by the National Key R&D Program of China (No. 2021YFC3340304) We would like to thank the anonymous reviewers for their helpful comments. We would like to express uppreciation to Yansong Feng for his insightful suggestions on our idea.
6.2 Confidence Calibration
Studies on the calibration of natural language mod- els have been drawing attention recently (Desai and Durrett, 2020; Park and Caragea, 2022; Kim et al., 2023). Among modern calibration approaches. tem- simple and effective method (Desai and Durrett, 2020) which can produce low ECE (Guo et al., 2017; Chen et al., 2023). Due to its low time overhead and low ECE property. we adopt it in our work. Other works focus on methods such as label smoothing (Pereyra et al., 2017) and data augmentation (Hendrycks* al., 2020), but these methods cannot produce as low ECE as temperature scaling (Chen al., 2023). More recent studies started to treat calibration as an additional task, which needs collecting data and training extra models (Ye and Durrett, 2022; Zhang et al., 2021). In order to reduce time overhead, we do not consider these approaches.
perature scaling
References
Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, and Heng Ji. 2023. A close look into the calibra- tion of pre-trained language models. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 1343-1380 1367, Toronto, Canada. Association for Computational Linguistics.
Morris H. Degroot and Stephen E. Fienherg. 1983. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statisti- cian), 32(1-2):12-25
Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Procesxing (EMNLP), pages 295-302, Online. Association for Computational Linguisties.
7 Conclusion
In this work, we propose the idea of simple-to- complex prediction for events in document, where events in document are reordered from simple to complex and predicted accordingly. Besides, we introduce retrieval to augment model input and ap- ply constrained decoding to improve model output. Empirical results and analysis demonstrate that our best model outperforms prior methods by notable margin and our simple-to-comple prediction is beneficial since it allows more inter-event depen- dency links, i.e., from event to all the events appeared after it.
Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 67 1-683, Online. Association for Computational Linguistics.
Xinya Du and Heng Ji. 2022. Retrieval-augmented generative question answering for event argument extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4649 1619-4466, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Xinya Du, Sha Li. and Heng Ji. 2022. Dynamic global memory for document-level argument extraction. Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 5264-5275 Dublin, Ireland. As- sociation for Computational Linguistics.
Limitations
Firstly, our framework requires two inference pro- cesses, where the first inference is calculate the difficulty of events in a document and the second in- ference is to predict the arguments of these events from simple to complex. Secondly, the way of setting lower/upper bounds is a hard pruning strat- egy that disables constraints where the argument probability is too low/high. However, this strategy rigidly excludes constraints for which the model is sufficiently certain or less reliable, without really taking into account the wrong constraints caused by the incorrectly predicted arguments. We leave the problems for future work.
Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. 2020. Multi-sentence ar- gument linking, In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 8057-800 8077, Online. Association for Computational Linguistics.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Wein- berger. 2017. On calibration of modern neural net- works. In Proceedings of the 34th International Conference on Muchine Learning Volume 70, page 1321-1330, Sydney, NSW, Australia. JMLR.org.

---

Rujun Han, I-Hung Hsu, Jiao Sun, Julia Baylon. Qiang Ning. Dan Roth, and Nanyun Peng. 2021. ESTER: A machine reading comprehension dataset for reason- ing about event semantic relations. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 7543-7559, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alexandru Viculescu-Mizi and Rich Caruana. 2005. Predicting good probabilities with supervised learn- ing. In Proceedings of the 22nd International Confer- ence on Machine Learning, page 625-632, New York, NY, USA. Association for Computing Machinery.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated proba- bilities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1).
Dan Hendrycks*, Norman Mu*, Ekin Dogus Cubuk, Barret Zoph. Justin Gilmer, and Balaji Lakshmi- narayanan. 2020. Augmix: A simple method to im- prove robustness and uncertainty under data shift. In International Conference on Learning Representa- fions.
Seo Yeon Park and Comelia Caragea. 2022. On the cal- ibration of pre-trained language models using mixup guided by area under the margin and saliency. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 5364-5374, Dublin, Ireland. As- sociation for Computational Linguistics.
Kuan-Hao Huang, I-Hung Hsu, Prem Natarajan, Kai- Wei Chang, and Nanyun Peng. 2022. Multilin- gual generative language models for zero-shot cross- lingual event argument extraction, In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 4633-4646, Dublin, Ireland. Association for Computational Linguistics.
Gabriel Pereyra, George Tucker. Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. 2017. Regularizing neural networks by penalizing confident output dis- tributions.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing 3982-3992, Hong Kong, China, Association for Com- putational Linguistics.
Ruihong Huang and Ellen RilofT. 2021, Modeling textual cohesion for event extraction. Proceedings of the AAAI Conference Artificial Intelligence, 26(1):1664-161 16700
pages
Heng Ji and Ralph Grishman. 2008 Refining event extraction through cross-document inference. In Pro- ceedings of ACL-08: HLT, pages 254-262, Colum- bus, Ohio. Association for Computational Linguis- tics.
Peng Shi and Jimmy Lin. 2019. Simple BERT Models for Relation Extraction and Semantic Role Labeling. arXiv e-prints, page
Jaeyoung Kim, Donghin Na, Sungchul Choi, and Sung- bin Lim. 2023. Bag of tricks for in-distribution cali- bration of pretrained transformers. In Findings of the Association for Computational Linguisties: EACL 2023, pages 551-563, Dubrovnik, Croatia. Associa- tion for Computational Linguistics.
Beth M. Sundheim. 1992. Overview of the fourth Mes- sage Understanding Evaluation and Conference. Fourth Message Understanding Conference (MUC- 4): Proceedings of Conference Held in McLean, Virginia, June 16-18, 1992.
MeiHan Tong, Bin Xu, Shuai Wang. Mcihuan Han, Yixin Cao, Jiangqi Zhu, Siyu Chen, Lei Hou, and Juanzi Li. 2022. DocE:: A large-scale and fine- grained benchmark for document-leved event extrac- tion. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- giex, pages 3970-3982, Seattle, United States. Asso- ciation for Computational Linguistics.
Qi Li, Heng Ji, Yu Hong, and Sujian Li. 2014. Con- structing information networks using one single model. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1846-1851, Doha, Qatar. Associa- tion for Computational Linguistics.
Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global fea- tures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 73-82, Sofia, Bulgaria. Association for Computational Linguistics.
Xi Xiangyu, Wei Ye, Shikun Zhang, Quanxiu Wang, Huixing Jiang, and Wei Wu. 2021. Capturing event argument interaction via bi-directional entity- recurrent decoder. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 210-219, Online. Association for Computational Linguistics.
Sha Li, Heng Ji, and Jiawei Han. 2021. Document-level event argument extraction by conditional generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional L Linguistics: Human Language Technologies, pages 894-908, Online. Association for Computa- tional Linguistics.
Runxin Xu, Peiyi Wang, Tianyu Liu, Shuang Zeng. Baobao Chang, and Zhifang Sui. 2022. A two-stream

---

AMR-enhanced model for document-leve event ar- gument extraction. Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pages 5025-5036, Seattle, United States. Association for Computational Lin- guistics.
Yuqing Yang, Qipeng Guo, Xiangkun Hu, Yue Zhang, Xipeng Qiu, and Zheng Zhang. 2023. An AMR- based link prediction approach for document-level event argument extraction. In Proceedings the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 12876- 12889, Toronto, Canada. Association for Computa- tional Linguistics.
Xi Ye and Greg Durrett. 2022. Can explanations be use- ful for calibrating black box models? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6199-6212, Dublin, Ireland. Association for Computational Linguistics.
Shujian Zhang, Chengyue Gong, Eunsol Choi. 2021. Knowing more about questions can help: Improving calibration in question answering. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 1958-1970, Online. Association for Computational Linguistics.
Tianran Zhang, Muhao Chen, and Alex A. T. Bui. 2020.
Diagnostic prediction with sequence-of-set represen- tation learning for clinical events. In Artificial Intelli- gence Medicine, pages 348-358, Cham. Springer International Publishing.

---

A Appendix
where denotes the number of predictions. ECE measures how calibrated the model is. The smaller the ECE, the more calibrated the model is.
A.1 Hyperparameters
The hyperparameters used our experiments are shown in Table 8.
A.3 The Heuristies of Selecting Bounds
We select the lower and upper bounds of the ar- gument probabilities according to the probability distribution (Figure 2) and the reliability diagram (Figure 3). The steps are as follows:
Hyperparameter
Value
Train_batch_size
2
eval_batch_sizz
1
learning_rate
3e-5
1. Firstly, calculate the median Nmed of the number of probabilities in each interval (e.g., 1.0) of the calibrated probability distri- bution (Figure 2). If the number of proba- bilities in an interval is 2 Nmed. then is selected as a candidate interval.
accumulate_grad__bttcee batchhes
4
train_epoches
5
warmup_steps
0
weight_decay
0
gpus
Table 8: Hyperparameters.
2. Secondly, merge all candidate intervals form a continuous interval I. If there are multiple intervals, then prune intervals with poorer calibration (c.g.. [0.8, 1.0)) according to the reliability diagram (Figure 3), and only keep one (denoted as I).
A.2 Confidence Calibration
Confidence & Accuracy Confidence is defined the probability that model assigns pre- diction, while accuracy is the real correctness of prediction. In the classification task, "prediction" means the predicted class of a specific instance. In our work, "prediction" means a specific argument.
3. Finally, prune the less calibrated part of J.
Following these steps, we present the process of selecting bounds in our work below. First, we can calculate Nmed = 60.5 according to Figure 2. Af- ter merging candidate intervals, we obtain con- tinuous interval I = [0.3, 0.8). Then, we should prune [0.3, 0.5] because probabilities in this inter- val are less calibrated with low accuracy (Figure 3). To sum up, our final interval is [0.5, 0.8).
Confidence Calibration The goal of confidence calibration is to align the model's posterior prob- abilities (confidence) with empirical likelihoods (accuracy) (Guo et al., 2017). For example, if take 100 instances where the model's prediction receives a posterior probability of 0.8, the model should get 80 of the instances correct.
Reliability Diagrams Usually, calibration is vi- sualized by reliability diagrams (Degroot and Fien- berg, 1983; Niculescu-Mizil and Caruana, 2005). Reliability diagrams treat expected accuracy function of model confidence. To draw the reliabil- ity diagram, usually partition predictions into k disjoint, equally-sized bins (B1, B2, ... Bx}. and calculate the average confidence/accuracy in each bin as an approximation. this paper, we set k 10.
Evaluation Metries We use Expected Calibra- tion Error (ECE) (Pakdaman Nacini et al., 2015) in this work. ECE is the weighted average of all the bins' confidenceeaccurrayy difference in the reliabil- ity diagram:
B n
BCE
[B:) conl(B.) (9)


Reviews:
[{"review_id": "FxlHD5vblX", "reviewer": "Reviewer_Unoe", "summary": "", "questions": "A. The equipment and environment of your experiment? How long does it take for each runs?\n\nB. Are there cases of multiple labels? E.g. an argument role is annotated with two or more entities? Or an entity plays more than one argument role in event? How you deal with that?\n\nC. Are all results averaged over multiple seeds?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful review comments! Firstly, I will reply to the points that you have mentioned in **Reasons To Reject**.\n\n**Reason A:** This paper only conducts experiments on WikiEvents dataset and compares the results with three previous works, which may be insufficient.\n\n**Response:** Our proposed method aims to solve the poblems where there are multiple events in one document and the arguments might be very distant from the trigger. It is a very challenging task and to the best of our knowledge, WikiEvents is the only dataset suitable for this task. Due to the difficulty of annotation, other datasets like DocEE$^{[1]}$, RAMS$^{[2]}$ and MUC-4$^{[3]}$ only annotate one or two events per document and they mostly focus on local extraction, where arguments are very close to the trigger. Instead, WikiEvents has 16 events per document averagely, and focus on the extraction of informative mentions, which are 14*×* distant from the trigger than their uninformative/local counterparts. Therefore, the setting of WikiEvents is suitable for our task, and our experiments on this dataset prove the effectiveness of our framework. Since this task is more challenging compared to traditional document-level EAE, it is less explored with only a few works$^{[4-5]}$.\n\n**Reason B:** Fig.5 only presents the results that the model can predict correctly with simple-to-complex ordering, but if without it cannot. I would prefer to see how the simple-to-complex ordering corrects the prediction, in this case study.\n\n**Response:** In this case, the order of difficulty in predicting the three events is $E_2>E>E_1$. The prediction order before ordering is $E_1\\rightarrow E_2\\rightarrow E$, where $E_2$ is the most difficult event with the low probability and is retrieved as additional input for $E$, misleading the prediction. After ordering, the prediction order is $E_1\\rightarrow E\\rightarrow E_2$, where $E_1$ provides a reliable informative argument \"Dzhokhar Tsarnaev\" to help the prediction of $E$, and $E_2$ is excluded because it is more difficult than $E$.\n\n**Reason C:** The pipeline method of two step inference processes also brings some error propagattions for the final event arguments extraction.\n\n**Response:** Any pipeline will have the error propagation problem. In this work, the quality of ordering determines the final result. Although ordering may bring errors sometimes, the advantages still outweigh the disadvantages, as shown in the table below. This table summarizes the prediction change from BART-Gen (w/ M) to BART-Gen (w/ M+O) on the test set. There are in total 12 cases that increase F1, while only 3 cases harm F1.\n\n| Prediction chang        | Number of cases |\n| ----------------------- | --------------- |\n| **Wrong->Right (good)** | 12              |\n| **Right->Wrong (bad)**  | 3               |\n\n**Reason D:** The introduction section should include a brief description of the proposed methods.\n\n**Response:** Thanks for your suggestion. We will add a brief description of our framework to the introduction section in the final version.\n\n---\n\nSecondly, we would like to respond to your **Questions for The Authors** one by one as follows:\n\n**Question A:** The equipment and environment of your experiment? How long does it take for each runs?\n\n**Answer:** We train and test our model using a single NVIDIA GeForce RTX 3090 GPU. It takes about 550 seconds for training and 320 seconds for inference. There may also be some checkpoint loading overhead, but it's not a high percentage compared to the total time consumption. \n\n**Question B:** Are there cases of multiple labels? E.g. an argument role is annotated with two or more entities? Or an entity plays more than one argument role in event? How you deal with that?\n\n**Answer:** Our method is generation-based, so it is flexible and natural to handle these two situations. If an argument role is annotated with two (or more) entities, the model can predict as \"entity_A and entity_B\" in the corresponding slot of the event template. If an entity plays multiple roles in a specific event, the same entity can be predicted in two (or more) slots of the event template, since generative approaches do not restrict one entity to be decoded only once.\n\n**Question C:** Are all results averaged over multiple seeds?\n\n**Answer:** Yes, we conduct our experiments three times on three different seeds for each setting and report the average micro-F1 in our paper. We will add the variance and $p$-value in the final version, which are provided below. The $p$-value is calculated between our best model and the previous SOTA (both marked in bold in the table below). As is shown in the table, all the $p$-values are less than 0.05, indicating our improvement is significant.\n\n| Models                              | Arg-I Head F1 | Arg-I Coref F1 | Arg-C Head F1 | Arg-C Coref F1 |\n| ----------------------------------- | :-----------: | :------------: | :-----------: | :------------: |\n| BART-Gen w/ M                       |  59.15±0.27   |   63.40±0.35   |  53.42±0.34   |   56.93±0.38   |\n| **BART-Gen w/ M+C (Previous SOTA)** |  60.30±0.11   |   64.56±0.31   |  54.36±0.25   |   57.88±0.34   |\n| BART-Gen w/ M+O (Our Model)         |  60.59±0.23   |   64.64±0.21   |  54.88±0.26   |   58.20±0.39   |\n| **BART-Gen w/ M+O+C’ (Our Model)**  |  61.24±0.14   |   65.31±0.27   |  55.32±0.20   |   58.65±0.20   |\n| $p$-value                           |  0.01088625   |   0.00211084   |  0.00127592   |   0.00825234   |\n\n[1] Tong, M., Xu, B., Wang, S., Han, M., Cao, Y., Zhu, J., ... & Li, J. (2022, July). DocEE: A Large-Scale and Fine-grained Benchmark for Document-level Event Extraction. In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies* (pp. 3970-3982).\n\n[2] Ebner, S., Xia, P., Culkin, R., Rawlins, K., & Van Durme, B. (2020, July). Multi-Sentence Argument Linking. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics* (pp. 8057-8077).\n\n[3] McLean, V. (1992, June). Fourth message understanding conference (MUC-4). In *Proceedings of fourth message understanding conference (MUC-4)*.\n\n[4] Li, S., Ji, H., & Han, J. (2021, June). Document-Level Event Argument Extraction by Conditional Generation. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies* (pp. 894-908).\n\n[5] Du, X., Li, S., & Ji, H. (2022, May). Dynamic Global Memory for Document-level Argument Extraction. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)* (pp. 5264-5275).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a novel simple-to-complex progressive framework for document-level event argument extraction. The paper is well written and clearly organized. The authors conduct comprehensive experiments to validate their argument about simple-to-complex framework. They also experiment and analysis the prediction probabilities bias issues and the entity type constraints issues.", "reasons_to_reject": "A. This paper only conducts experiments on WikiEvents dataset and compares the results with three previous works, which may be insufficient. \n\nB. Fig.5 only presents the results that the model can predict correctly with simple-to-complex ordering, but if without it cannot. I would prefer to see how the simple-to-complex ordering corrects the prediction, in this case study.\n\nC. The pipeline method of two step inference processes also brings some error propagattions for the final event arguments extraction.\n\nD. The introduction section should include a brief description of the proposed methods.", "questions_to_authors": "A. The equipment and environment of your experiment? How long does it take for each runs?\n\nB. Are there cases of multiple labels? E.g. an argument role is annotated with two or more entities? Or an entity plays more than one argument role in event? How you deal with that?\n\nC. Are all results averaged over multiple seeds?", "paper_topic_and_main_contributions": "This paper is about document-level event argument extraction task.\nThe authors argue that extracting events according to their appearance order has ignored the difficulty of each event. They propose a simple-to-complex progressive framework for document-level informative argument extraction.\nExperiment results have shown that their proposed method can help event arguments prediction and have achieved outperform performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZgSXoNll4L", "reviewer": "Reviewer_FLW2", "summary": "", "questions": "Question A: What is the train/dev/test size? \n\nQuestion B: How were the P/R/F1 averaged? Micro, Macro, Instance-based?\n\nQuestion C: Did you conduct any statistical analysis of the results? Are the runs averaged over multiple seeds?\n\nI am open to pushing my soundness score if I am convinced that the results are statistically sound", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your enlightening questions and suggestions! Firstly, we will respond to your questions about the contribution of our work in **Reasons To Reject**. \n\nAlthough we only conduct experiments on WikiEvents, this does not mean that our work is merely incremental. That is because we made a significant improvement on a very challenging task. Our proposed method aims to solve the poblems where there are multiple events in one document and the arguments might be very distant from the trigger. It is a very challenging task and to the best of our knowledge, WikiEvents is the only dataset suitable for this task. Due to the difficulty of annotation, other datasets like DocEE$^{[1]}$, RAMS$^{[2]}$ and MUC-4$^{[3]}$ only annotate one or two events per document and they mostly focus on local extraction, where arguments are very close to the trigger. Instead, WikiEvents has 16 events per document averagely, and focus on the extraction of informative mentions, which are 14*×* distant from the trigger than their uninformative/local counterparts. Therefore, the setting of WikiEvents is suitable for our task, and our experiments on this dataset prove the effectiveness of our framework. Since this task is more challenging compared to traditional document-level EAE, it is less explored with only a few works$^{[4-5]}$. Besides, the improvement of our framework over previous SOTA is statistically significant, and we will provide the results when replying to your Question C.\n\n---\n\nWe have also read your **Questions for The Authors** carefully, and would like to answer them one by one as follows:\n\n**Question A:** What is the train/dev/test size?\n\n**Answer:** WikiEvents statistics are shown in Table 1, where there are 3241, 345, and 365 events in the train/dev/test set, respectively. Besides, there are respectively 4413, 411, and 556 arguments in the train/dev/test set, which is not presented in the paper. We will include this in our final version.\n\n**Question B:** How were the P/R/F1 averaged? Micro, Macro, Instance-based?\n\n**Answer:** We follow previous works$^{[4-5]}$ and calculate micro-P/R/F1 for a fair comparison. In the setting of EAE, this calculation is argument-level (not event/instance-level), where we first calculate P and R for all the arguments, then calculate a general F1.\n\n**Question C:** Did you conduct any statistical analysis of the results? Are the runs averaged over multiple seeds?\n\n**Answer:** Yes, we conduct our experiments three times on three different seeds for each setting and report the average micro-F1 in our paper. We will add the variance and $p$-value in the final version, which are provided below. The $p$-value is calculated between our best model and the previous SOTA (both marked in bold in the table below). As is shown in the table, all the $p$-values are less than 0.05, indicating our improvement is significant. Besides, our best model outperforms the previous SOTA on each seed.\n\n| Models                              | Arg-I Head F1 | Arg-I Coref F1 | Arg-C Head F1 | Arg-C Coref F1 |\n| ----------------------------------- | :-----------: | :------------: | :-----------: | :------------: |\n| BART-Gen w/ M                       |  59.15±0.27   |   63.40±0.35   |  53.42±0.34   |   56.93±0.38   |\n| **BART-Gen w/ M+C (Previous SOTA)** |  60.30±0.11   |   64.56±0.31   |  54.36±0.25   |   57.88±0.34   |\n| BART-Gen w/ M+O (Our Model)         |  60.59±0.23   |   64.64±0.21   |  54.88±0.26   |   58.20±0.39   |\n| **BART-Gen w/ M+O+C’ (Our Model)**  |  61.24±0.14   |   65.31±0.27   |  55.32±0.20   |   58.65±0.20   |\n| $p$-value                           |  0.01088625   |   0.00211084   |  0.00127592   |   0.00825234   |\n\n[1] Tong, M., Xu, B., Wang, S., Han, M., Cao, Y., Zhu, J., ... & Li, J. (2022, July). DocEE: A Large-Scale and Fine-grained Benchmark for Document-level Event Extraction. In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies* (pp. 3970-3982).\n\n[2] McLean, V. (1992, June). Fourth message understanding conference (MUC-4). In *Proceedings of fourth message understanding conference (MUC-4)*.\n\n[3] Ebner, S., Xia, P., Culkin, R., Rawlins, K., & Van Durme, B. (2020, July). Multi-Sentence Argument Linking. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics* (pp. 8057-8077).\n\n[4] Li, S., Ji, H., & Han, J. (2021, June). Document-Level Event Argument Extraction by Conditional Generation. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies* (pp. 894-908).\n\n[5] Du, X., Li, S., & Ji, H. (2022, May). Dynamic Global Memory for Document-level Argument Extraction. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)* (pp. 5264-5275).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method is simple and is motivated and explained well, it also seems to be effective. The empirical analysis is reasonably thorough and attributes the performance gains to the proposed improvements over the baseline method.", "reasons_to_reject": "The improvements stemming from the proposed method over the baseline are only marginal. This in itself is not a problem, however the difference should be analysed with statistical tests, to inspire confidence that the improvements are indeed due to the proposed method rather than chance.\n\nSome minor details regarding the experimental setup are missing (see my questions to the authors).\n\nFinally, the contributions here might be considered too small to warrant acceptance at a flagship conference: The proposed method is an incremental improvement, the score improvements are marginal, the task is rather niche with only 1 dataset available. As such I am not too excited about this submission. However, I think the paper deserves its place in the Findings volume.", "questions_to_authors": "Question A: What is the train/dev/test size? \n\nQuestion B: How were the P/R/F1 averaged? Micro, Macro, Instance-based?\n\nQuestion C: Did you conduct any statistical analysis of the results? Are the runs averaged over multiple seeds?\n\nI am open to pushing my soundness score if I am convinced that the results are statistically sound", "paper_topic_and_main_contributions": "The paper investigates the task of document level event argument extraction: Given a set of event triggers in a document, the task is to identify their corresponding arguments. The paper improves upon an existing generative method for task, by ordering the events to be processed by the observed probability of the generated arguments rather than in order of appearance.\n\nThe suggested method is evaluated in an empirical study and results suggest that it slightly improves upon the state of the art on an existing wikipedia-based event argument extraction dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are some slight grammatical errors, e.g. ll048--052 \"Recent works model [...] and achieves [...]\"\n\nThe manuscript could benefit from a quick run through a grammar checker (e.g. grammarly or Docs/Word) to get rid of those."}}, {"review_id": "ngD5uSHDUb", "reviewer": "Reviewer_CXkm", "summary": "", "questions": "1. Is it reasonable to use similarity to retrieve predicted event templates and arguments to predict probabilities to define the difficulty of event extraction?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! We will answer your questions (both in Reasons To Reject and Questions For The Authors) about our work one by one as follows:\n\n**Q1:** The experimental performance improvement of the proposed model is not obvious, and some significance tests can be added to verify whether the improved effect is credible.\n\n**A1:** Thanks for your suggestions. We conduct our experiments three times for each setting and report the average micro-F1 in our paper. We will add the variance and $p$-value in the final version, which are provided below. The $p$-value is calculated between our best model and the previous SOTA (both marked in bold in the table below). As is shown in the table, all the $p$-values are less than 0.05, indicating our improvement is significant. \n\n| Models                              | Arg-I Head F1 | Arg-I Coref F1 | Arg-C Head F1 | Arg-C Coref F1 |\n| ----------------------------------- | :-----------: | :------------: | :-----------: | :------------: |\n| BART-Gen w/ M                       |  59.15±0.27   |   63.40±0.35   |  53.42±0.34   |   56.93±0.38   |\n| **BART-Gen w/ M+C (Previous SOTA)** |  60.30±0.11   |   64.56±0.31   |  54.36±0.25   |   57.88±0.34   |\n| BART-Gen w/ M+O (Our Model)         |  60.59±0.23   |   64.64±0.21   |  54.88±0.26   |   58.20±0.39   |\n| **BART-Gen w/ M+O+C’ (Our Model)**  |  61.24±0.14   |   65.31±0.27   |  55.32±0.20   |   58.65±0.20   |\n| $p$-value                           |  0.01088625   |   0.00211084   |  0.00127592   |   0.00825234   |\n\n**Q2:** Some core modules are not clearly written, especially the Order Adaptive Argument Pair Constraints part. It is suggested to add examples or descriptions as necessary.\n\n**A2:** Thanks for your suggestions, we will include more detailed information as well as examples about this module. % Below are an introduction of this module?\n\nThe argument pair constraints are used at inference time to constrain the prediction of conflicting entities in a document. For example, if the \"Detainee\" of event $e_1$ is \"Mike\", then \"Mike\" cannot be predicted as the \"Attacker\" of another event $e_2$ (happened after $e_1$) in the same document, since \"Mike\" is already in jail. After \"Mike\" is decoded in $e_1$ as \"Detainee\", we lower the probability of predicting it as the \"Attacker\" of another event $e_2$ to restrict such wrong/contradictory prediction.\n\nHowever, this kind of constraints assume that previously predicted arguments are all correct and can be used to constrain the prediction of next-event arguments, which cannot be true. Once an incorrect prediction is used to constrain another, it may cause more errors sometimes. In the example above, \"Mike\" will never be predicted as the \"Attacker\" of $e_2$ once he is predicted as the \"Detainee\" of $e_1$, even if the prediction of \"Detainee\" is incorrect. To alleviate such error propagation, we only apply the constraints when the model is certain about the prediction of an entity, where the certainty of an entity can be measured by its output probability when decoding. When the output probability of an entity (e.g., \"Mike\") is too low/high, we disable the constraints. Low probability means the model is not sure/certain about this prediction, while high probability implies the prediction is not reliable. As is shown in Figure 4, when the probability is high (e.g., *>*0.8), the accuracy is low. Therefore, these predictions are not reliable, and we should prune them.\n\n**Q3:** Is it reasonable to use similarity to retrieve predicted event templates and arguments to predict probabilities to define the difficulty of event extraction?\n\n**A3:** We think, the question is why do we retrieve event templates and arguments from the document memory as additional input when predicting probabilities as the measure of difficulty? \n\nThe reason is that when calculating the difficulty (first inference) and predicting the final results (second inference), we use the same model, and its objective is to predict the next event conditioned on the retrieved event templates/arguments. In other words, the input/prompt format consistency should be ensured. However, we could also exclude retrieval when predicting the difficulty to eliminate its possible influence in the calculation process. Specifically, we use a separate BART model trained without retrieval to predict probabilities, and use the originally trained BART model (with retrieval) to get the final results. R1 and R2 are the results presented in the paper, and R3 is the result of this experiment. Comparing R2 and R3, we can conclude that the exclusion of retrieval does not have a large impact on model performance, where R2 is even slightly better (especially in Arg-C) than R3, proving that it is reasonable to include retrieval when calculating the difficulty.\n\n| Models                   | Arg-I Head F1  | Arg-I Coref F1 | Arg-C Head F1  | Arg-C Coref F1 |\n| ------------------------ | :------------: | :------------: | :------------: | :------------: |\n| BART-Gen w/ M **(R1)**   |   59.15±0.27   |   63.40±0.35   |   53.42±0.34   |   56.93±0.38   |\n| BART-Gen w/ M+O **(R2)** |   60.59±0.23   |   64.64±0.21   | **54.88**±0.26 | **58.20**±0.39 |\n| BART-Gen w/ M+O **(R3)** | **60.79**±0.23 | **64.65**±0.24 |   54.55±0.22   |   57.67±0.28   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper identifies and locates the error propagation problem of sequential prediction in document-level event argument detection. And trying to distinguish the difficulty of the event to alleviate the error propagation, which has certain reference significance.", "reasons_to_reject": "1. The experimental performance improvement of the proposed model is not obvious, and some significance tests can be added to verify whether the improved effect is credible.\n2. Some core modules are not clearly written, especially the Order Adaptive Argument Pair Constraints part. It is suggested to add examples or descriptions as necessary.", "questions_to_authors": "1. Is it reasonable to use similarity to retrieve predicted event templates and arguments to predict probabilities to define the difficulty of event extraction?", "paper_topic_and_main_contributions": "This paper proposes a new method for document-level informative argument extraction via a simple-to-complex progressive framework. The proposed order of the predicted events in a document by their difficulties to eliminate the error propagation of argument extraction.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a simple-to-complex progressive approach for event argument extraction and the complexities of events are determined by the probability of their generated arguments in the first step. The approach achieves better extraction performance but the improvements are relatively marginal.
Decision: Accept-Findings