DemoSG: Demonstration-enhanced Schema-guided Generation for Low-resource Event Extraction
Authors: Gang Zhao
Xiaocheng Gong
Xinjie Yang
Guanting Dong
Shudong Lu
Si Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Low-resource
Event Extraction
Few-shot
Domain Adaptation
Demonstration-based Learning

Abstract: Most current Event Extraction (EE) methods focus on the high-resource scenario, which requires a large amount of annotated data and can hardly be applied to low-resource domains. To address EE more effectively with limited resources, we propose the Demonstration-enhanced Schema-guided Generation (DemoSG) model, which benefits low-resource EE from two aspects: Firstly, we propose the demonstration-based learning paradigm for EE to fully use the annotated data, which transforms them into demonstrations to illustrate the extraction process and help the model learn effectively. Secondly, we formulate EE as a natural language generation task guided by schema-based prompts,  thereby leveraging label semantics and promoting knowledge transfer in low-resource scenarios. We conduct extensive experiments under in-domain and domain adaptation low-resource settings on three datasets, and study the robustness of DemoSG. The results show that DemoSG significantly outperforms current methods in low-resource scenarios.

DemoSG: Demoostrationnennameee Schema-guided Generation for Low-resource Event Extraction
Gang Zhao, Xiaocheng Gong, Xinjie Yang, Guanting Dong, Shudong Lu and Si Li' School of Artificial Intelligence, Beijing University of Posts and Telecommunications (zhaogang, xiaochengkung, yangxinjie, lushudong, 1isi Jebupt edw. edu. cn
Abstract
Input text
Kelly arrived Seoul from Seijing Friday to brief Toon
Most current Event Extraction (EE) methods focus on the high-resouree scenario, which re- quires large amount of annotated data and can hardly be applied to low-resource domains. To address EE more effectively with lim- ited resources, we propose the Demonstration- enhanced Schema-guided Generation (De- moSG) model, which benefits low-resource EE from two aspects: Firstly, we propose the demonstration-based learning paradigm for EE fully use the annotated data, which trans- forms them into demonstrations to illustrate the extraction process and help the model leam ef- fectively. Secondly, we formulate EE as natu- ral language generation task guided by schema- based prompts, thereby leveraging label seman- tics and promoting knowledge transfer in low- resource scenarios. We conduct extensive ex- periments under in-domain and domain adapta- tion low-resource settings on three datasets, and study the robustness DemoSG. The results show that DemoSG significantly outperforms current methods low-resource scenarios.
Training Data
Schema
Select Demo
Create Promat
Trampper Demo
Tratspeet Pronupt
both expected
Type
arrive at Mars. Event Trigger is arrive. Destination is Mans
Event Trigger [MNSK).
Artifact is (MASK).
Sequence-to Sequence Network
Generation
Extracted
Extracted Transport Records
Event Trigger la ontivd, Artifact Kelly. Origin Belling.
Destination Seoul. Vahicle None.
Figure I: A simplified EE example from the dataset ACE0S-E, and the insight proposed DemoSG. Event triggers, arguments, and roles are highlighted in colors.
Most studies on EE primarily focus on the high- resource scenario (Nguyen al., 2016; Yan et al., 2019; Cui al., 2020; Du and Cardie, 2020a; Ram- poni et al., 2020; Huang and Peng, 2021), which requires a large amount of annotated training data to attain satisfactory performance. However, event annotation is a costly and labor-intensive process, rendering these methods challenging to apply in domains with limited annotated data. Thus, there is growing need to explore EE in low-resource scenarios characterized by scarcity of training examples, which has garnered recent attention.
1 Introduction
Event Extraction (EE) aims to extract event records from unstructured texts, which typically include a trigger indicating the occurrence of the event and multiple arguments of pre-defined roles (Dod- dington et al., 2004a; Ahn, 2006). For instance, the text shown in Figure I describes two records corresponding the Transport and Meet events, re- spectively. The Transport record is triggered by the word "arrived"" and consists of arguments:" Kelly", "Beijing", and "Seoul". Similarly, the Meet record is triggered by "brief" and has two arguments: "Kelly" and "Yoon". EE plays a crucial role in natural language processing as it provides valuable information for various downstream tasks, includ- ing knowledge graph construction (Zhang al., 2020) and question answering (Han al., 2021).
Lu al. (2021) models EE unified sequence- to-structure generation, facilitating knowledge shar- ing between the Event Detection and Argument Ex- traction subtasks. Building upon Lu et al. (2021),
Lu et al. (2022) introduces various pre-training strategies on large-scale datasets to enhance struc- ture generation and improve low-resource EE per- formance. Hsu al. (2022) incorporates sup- plementary information by manually designing a prompt for each event type, encompassing event descriptions and role relations. However, despite their effectiveness, existing methods exhibit cer-
*Corresponding author.

---

tain inadequacies: 1) Sequence-to-struulure Gen- eration ignores the gup between the downstream structure generation and pre-training natural lan- guage generation. 2) Large-scale pre-training needs a lot of computational resources and excess corpus. 3) Manual design of delicate prompts still demands considerable human effort, and the performance of EE is sensitive to prompt design.
multiple ways: Firstly, leveraging label semanties facilitates the extraction and knowledge transfer across different events. For instance, the semantic of the "Destination" role hints that its argument is typically a place, and the similarity between "Ar- tacker" and "Adjudicator" makes it easy to transfer argument extraction knowledge among them, for they both indicate to humans. Secondly, natural lan- guage generation is consistent with the pre-training task of PLM, eliminating the need for excessive pre-training of structural generation methods (Lu et al., 2021, 2022). Furthermore, unlike classifica- tion methods constrained by predefined categories, DemoSG is more flexible and can readily adapt to new events through their demonstrations with- out further fine-tuning, which is referred to the aarameter-agnostee domain adaptation capability. Our contributions can be summarized follows: learn-
In this paper, tackle EE more effectively with limited resources, we propose the Demonstration- enhanced -aiidd Generation (DemoSG) model, benefiting from two aspects: 1) Improving the efficiency of using annotated data. 2) Enhanc- ing the knowledge transfer across different events.
To make full of annotated training data, consider them not only as signals for supervising model learning, but also as task demonstrations to illustrate the extraction process to the model. Specifically, as shown in Figure 1, DemoSG se- lects suitable training example for each event, transforms it into a demonstration in the natural language style, and incorporates the demonstration along with the input text enhance the extraction. We refer this paradigm as demonstration-bast based learning for EE, which draws inspiration from the in-context learning the GPT series (Ouyang et al., 2022). To identify appropriate instances for demon- strating, we devise several selecting strategics that can be categorized into two groups: 1) Demo- oriented selection, which aims to choose examples with the best demonstrating features. 2) Instance- oriented retrieval, which aims to find the examples that are most semantically similar to the input sen- tence. With the enhancement of demonstrations, DemoSG can better understand the EE process and learn effectively with only a few training examples.
1) We propose demonstration- ing paradigm for EE, which automatically creates demonstrations to help understand EE process and learn effectively with only few training examples. 2) We formulate EE to schema-guides natural language generation, which leverages the semantic information of event labels and promotes knowl- edge transfer in low-resource scenarios.
3) We conduct extensive experiments under var- ious low-resource settings on three datasets and study the robustness of DemoSG. The results show that DemoSG significantly outperforms previous methods in low-resource scenarios.
Methodology
In this section, first elucidate some preliminary concepts of low-resource EE in Section 2.1. Next, we present the proposed Demonstration-enhanee Schema-guided Generation model in Section 2.2. Lastly, we provide details regarding the training and inference processes in Section 2.3.
To further enhance the knowledge transfer capa- bility and improve the effectiveness of demonstra- tions, we leverage label semantic information by formulating EE to a seq2seq generation task guided by sasema-based prompts. detail, DemoSG cre- ates a prompt formally similar to the demonstra- tions for each event, which incorporates the event type and roles specified in the event schema. Then, the prompt and eehonned sentence are fed to- gether into the pre-trained language model (PLM) generate natural sentence describing the event records. Finally, DemoSG employs a rule-based de- coding algorithm decode records from the gener- ated sentences. The cidemm--gided sequence gen- eration of DemoSG promotes knowledge transfer in
2.1 Low-resource Event Extraction
X Given a tokenized input sentence X { the event extraction task aims to extract a of event records R [R)) where R, contains a trigger
Ty and several arguments A, = {4 k=1 Each T, or Ajk corresponds to event type E, or event role Oýk predefined in the schema S This paper considers two types of low-resource EE scenarios: In-domain low-resource scenario focuses on the challenge that the amount of training examples is quite limited. Considering the complete train-
Ayl

---

Event Type
Event Trigger
Arguments
Transport
anrved
Artifact: Kelly, Origi: Beijing: Destination:
Negative
Positive Event
Training
Inference
Meet
brief
Entit:: Kelly, Yoon
Demo Selection
Calculating Loss
Decoding Records
Sampling
Demonstration-orieer -rrenteed
Event Trigger is anrived Artifact Kelly, Origin Beijjing Destination Scoul. Vehicle None.
Rafe
Drratn
Context
Transformer Decoder
Instance- oriented
Perain
Transformer Encoder
Retrieval
both expected Mars. Event trigger arrive. Destination Mars...
Event Event Trigger <Maaao.. Art <Masko...
SBERT
Demo
arrived
willing meet the American commander... Event Trigger meet. Entity's commander.
Evont
Text
arrived...
<Masko, Entiity is <Masio:
Demonstration
Input Text
Schema-based Prompt
The overall architecture of DemoSG. The left shows the demonstration selection process, while the right presents the innponstration-enhaneee record generation framework. Different events are distinguished using colors.
Figure 2:
Pal with the schema Sn and subset Dj with S Sp, the objective is to fully use the limited data and train a high- performance model on D, when [Di] < [Dn]. The in-domain low-resource scenario poses challenges to the data utilization efficiency of EE models.
ing dataset Dn ((X,, Ri)
sentences that include event triggers, arguments, and their corresponding event roles.
In detail, given record R. with a trigger T and several arguments A (A,)]A), where each A; corresponds to the role O,, DemoSG transfers R to a sequence Y = {8, } that is designed as "Event trigger is T O₁ is A O2 is Az..." Those followed by the
Domain adaptation scenario pertains to the situation where the target domain lacks examples but large number of source domain examples are available. Given two subsets Dere and Digt, where Swe Sigt 2 and iDiyel < Sreeel our objec- tive is to initially pre-train a source-domain model on D Peres then achieve high performance on target domain subset Digt- Domain adaptation enables the low-resource domain to leverage the knowledge acquired from the well-studied domain, requiring strong knowledge transfer capability of EE models.
roles without any arguments padding token "None". For example, the Transport record in Figure 2 is represented as "Event trig- ger is arrived. Artifact is Kelly. Origin is Beijing. Destination is Seoul. Vehicle is None..... We also consider the following special situations: Multiple records of the same event type can be expressed by concatenating respective sequence representations. Multiple arguments corresponding the same role will be merged together via "&", such as "Artifact is Kelly Yoon". Since DemoSG extracts each type of events separately, event type is no longer required for record representations, which relieves the pressure of model prediction.
2.2 Demonstration-ennaann Schema-guided Generation for Event Extraction
We propose end-to-end model DemoSG, which leverages demonstration-F based learning and schema-guidde generation to address the aforemen- tioned low-resource scenarios. As Figure 2 shows, DemoSG initially constructs the task demonstration and prompt for each event type, and then employs eqquence-to-sequenne network to generate natural sentences that describe the event records.
2.2.2 Event Demonstration Construction
To effectively the limited training examples, not only treat them traditional supervised learning signal, but also transform them to event demonstrations which can bring additional infor- mation and help understand the extraction process.
2.2.1 Unified Sequence Representation of Event Record
The demonstration D, of event type E, natu- ral sentence containing context part and anno- tation part, which constructed by the following steps: Firstly, DemoSG selects or retrieves an ex- ample (X,, R,) from the training set D Drrain, which
To model EE as a cewpueenn------uenneeee generation task, we design a unified representation template to transform event records to unambiguous natural

---

contains records of E, Next, DemoSG transforms records associated with E, to an annotation sen- tence Y following the record representation tem- plate in Section 2.2.1. Note that we employ the uni- fied record template for both demonstrations and model predictions, promoting the cohesive interac- tion between them. Finally, the demonstration D; is constructed by concatenating the context part X; and the annotation part Y,. Given the significance of selecting appropriate examples for construct- ing demonstrations, we propose several selection strategies that can be categorized into two groups: Demonstration-oriented selection aims to pick the examples with the best demonstrating character- istics. Specifically, the training sample associated with more event roles tends to contain more infor- mation of extracting such type of events. And an example with longer text may offer more contextual information for extracting the same event. Based on these considerations, we propose two selection strategies: 1) rich-role strategy selects the exam- ple with the highest number of associated roles for each event. 2) rich-context strategy chooses the example with the longest context for each event. Instance-oriented retrieval focuses retriev- ing examples that are most semantically similar the input sentence, the semantic similarity may enhance the effectiveness of demonstrations. Concretely, similar strategy involves encoding the input sentence X and each example sentence X; using SBERT (Reimers and Gurevych, 2019), fol- lowed by calculating the cosine similarity between their (CLS) embeddings to rank X,. Finally, sim- ilar strategy retrieves the top-ranked example for each event type to construct its demonstration.
2.2.4 Enhanced Sequence-to-sequenee
Generation for Event Extraction
As Figure 2 shows, DemoSG generates the record sequence of each event type individually via com- mon encoder-decoder architecture enhanced by re- spective event demonstration and prompt. Given an input sentence X [ DemoSG first constructs event demonstration D; = {di)} 12r
x1
and schema- prompt P. = {pis 1 for the event type E,, Then, DemoSG concatenates D.. X and P. via the "<SEP>" token, and uses a Trans- former Encoder (Vaswani et al., 2017) to obtain the demoonttration-cnaanne hidden representation:
(1) Subsequently. DemoSG decodes the enhanced rep- resentation H, and generates event record sequence Y; {uty) |41| 1=1 token by token:
H = Encoder([D;; X; P:l)
Ykj, h:j = Decoder( (H,; hut.-.,, h-y- (2) where Decoder(-) represents the transformer de- coder and h.j is the decoder state at the jth step. By iterating above generation process for all event types ( DemoSG finally obtains the com- plete record sequence set Y = -Y))
2.3 Training and Inference
Since DemoSG generates records for cach event type individually, we consider the sentences gen- crated for annotated event types as positive exam- ples, while the sentences generated for unannotated event types serve negative examples. At the train- ing stage, sample negative examples m times the number of positive samples, where m is hyper- parameter. The following negative log- likelihood loss function optimized when training:
L =-
Σ DPUDD
log logp(Y p(||,, Xxraiainn S,0)
(3)
2.2.3
Schema-based Prompt Construction
We design a prompt template for DemoSG ex- ploit the semantic information of event types and roles base on the event schema. Given the event schema& S = EE, N where E, is the event type and O; = {Otg), prompt of F2 is designed as: "Event type is E, Event trigger is <Mask>. On is <Mask>. O₂ is <Maskk...", ", where <Mask> represent the mask token of the PLM. For example, the prompt of afor- mentioned Transport event can be constructed as "Event type is Transport. Event Irigger is <Mask>. Artifact is <Mask>. Origin <Mask>.." " Lever- aging the label semantic information not only helps query relevant triggers and arguments, but also fa- cilitates knowledge transfer across different events.
M
p(Y|X, Dtrain; S)
II [P(vee vei Dirain,
(4)
where represents the model parameters, D Derain is the training set, is the event schema, Dp and D are the positive and sampled negative sets. At the inference phase, DemoSG decodes event records from the generated Y = [Y based deterministic algorithm, and employs string matching to obtain the offsets of event triggers and arguments. Following Lu et al. (2021), when the predicted string appears multiple times in the sentence, we choose all matched offsets for trigger prediction, and the matched offset that is closest to the predicted trigger for argument extraction.
1O are event roles, the
Ne via a rule-

---

Train
Dataset
Statistic
Dev
Test
Fuli(100%)
2%
5%
10%
2shot
5shot
10shot
#Sentts
923
17.172
314
157
302
#Fivents
450
403
4,202
74
206
414
101
234
458
ACE05-EN
#Roles
605
576
4,859
84
225
442
106
247
487
#Sents
901
676
19,216
176
348
158
300
ACEOS-EN+
#Events
468
424
4,419
91
232
455
100
241
448
#Roles
759
689
6,607
114
323
609
140
325
605
2) Text2Event (Lu et al., 2021), which integrates Table 1: Data statistics of ACE05-EN and ACE05-EN+, #Sents, #Events and #Roles indicate the sentence, event record and role numbers. Statistics of the low-resource subsets are the average results of five different sampling.
3
Experiments
event detection and argument extraction into uni- fied structure generation task. 3) UIE (Lu et al., 2022), which improves the low-resource perfor- mance through various pre-training strategies based on Text2Event. 4) DEGREE (Hsu et al., 2022), which incorporates additional information by man- ually designing prompt for each event type.
To assess the effectiveness of DemoSG, we conduct comprehensive experiments under in-domain low- resource, domain adaptation, and high-resource settings. An ablation study applied to explore the impact of each module and the improvement achieved. Furthermore, we also investigate the ro- bustness applying demonstrations of DemoSG.
For the high-resouree experiments, we also compare with the following methods: 1) DY- GIE++ (Wadden et al., 2019), which BERT- based classification model that utilizes the span graph propagation. 2) Joint3EE (Nguyen and Nguyen, 2019), which jointly extracts entities, trig- gers, and arguments based on the shared represen- tations. 3) GAIL (Zhang et al.. 2019), which is joint entity and event extraction model based on inverse reinforcement learning. 4) EEQA (Du and Cardie, 2020b) and MQAEE (Li et al., 2020), which formulate event extraction question an- swering problem using machine reading compre- hension models. 5) TANL (Paolini et al., 2021), which formulates event extraction as a translation task between augmented natural languages.
3.1 Experimental Settings
Datasets. We evaluate our method on two widely used event extraction benchmarks: ACE0S- EN (Wadden et al., 2019) and ACE0S-EN+ (Lin et al., 2020). Both of them contain 33 event types as well as 22 event roles, and derive from ACE2005 (Doddington et al., 2004b) dataset that provides rich annotations of entities, relations and events in English. The full data splits and preprocessing steps for both benchmarks are con- sistent with previous works (Wadden et al., 2019; Lin et al., 2020). Additionally, we utilize the same data sampling strategy as UIE (Lu et al., 2022) for the low-resource settings. Detail statistics of the datasets are shown in Table 1.
We employ the BART-large pre-trained by Lewis et al. (2020) as our seq2seq network, and optimize our model via the Adafac- tor (Shazeer and Stern, 2018 2018) optimizer with a learning rate of 4e-5 and warmup rate of 0.1. The training batch size is set to 16, and the epoch number for the in-domain low-resource experi- ments is set to 90, while 45 for the domain adap- tation and high-resource experiments of DemoSG. Since ONEIE is not specifically designed for low- resource scenarios, we train it for 90 epochs as well in the in-domain setting. The negative exam- ple sampling rate m is set to after the search from (5,799,11,13115) using dev set the high- resource setting, and we find that can balance Recall and Precision scores in practice. We con- duct experiments on the single Nvidia Tesla-V100 GPU. All experiments of baseline methods are con- ducted based on the released code of their orig-
Implementations.
Evaluation Metries. We utilize the same eval- uation metrics as previous event extraction stud- ies (Wadden et al., 2019; Lin et al., 2020; Lu et al.. 2021, 2022): 1) Trigger Classification Miero F1- score (Trig-C): trigger correctly classified its offset and event type align with the gold labels. 2) Argument Classification Micro Fl-score (Arg-C): an argument is correctly classified if its event type, offset, and event role all match the golden ones.
Baselines. We compare our method with the following baselines in low-resource scenarios: I) OnelE (Lin et al., 2020), the current SOTA high-resource method, which extracts globally optimal event records using global features. upe.pen 'https://catalog Idc. www.uu.eee..eeee edu/LDC2806T06

---

ACEISEEN
ACEIS-EN+
Metric Medeh
WEE
2%
DR
107
A
2,bbt
AVE
57.3gx1
61,9gx0
51.5
36.2az
51.0g45
01.3.11
S1.1
58.0gas
61.2.11
05.3g77
Teettiverr
196
300
97.6
35.0aa
58.0
2733
25
32.
10
003
30.4
157
10.8a1a
10
4
9099
31.8
630
C
UIB
42.6135
48.8111
65.3
50.
57.3 57.333
60.1.s
55.8
40.4
45.6 85.
$1.0.15
47.0
52.8
50
56.0
DenoSCip
00
69.1
59.0
60 60.
56.3
65.8u2
65. 55.1gn
64.8
35.6
140.50
63.
50.8
DeroSii
50.7
588
56.6
63.7
20.1
58.2245
66600
56.0
45, 15.5
61.3
57.0
DenuSO,
$8.7asa
10.
00.2
60.0.33
60.5
60.0.66
61.4.2v
00.51.4
4.000
14
2422.4
14.3
7.3zo
184414
20.0
15.2
34.2.00
7.0.14
33.6
18.3215
18.3122
22/511
17.3
14Ruaa
24Auaa
92.9ux
20.7
30.5.n
17.4
10
16.1
167ex
11 1a1
114
1511
31-4
Crg-C
UIE
18.3188
25.8117
816111
25.3
26.2114
27.3
10.
27.9
10
55.9
20.1
06.01a
20.8
F1(9)
DenoSSii
18.8g
35.3
20.5
185
43.3g73
21.
22.
42.0
20.9
22.0gm
38.0
S6.1
Deno8Cio
35.8
36.412
31.6
18.8aa
31 31.2a57
128
83.5 Rana
303.2
24.
38.8
42.5
35.7
20.0gs4
33,3
332.6
22.2,a3
40llas
32.3
33.0
28.3.3
37 1at0
42.5
36500
Table 2: Experimental results in the in-domain low-resource settings, AVE-S (hot) and AVE-R (atio) are average performances across the few-shot and the data-limited settings, respectively. DemoSG R(ote), DemoSG and s represent three variants of DemoSG with different demonstration selecting strategies. We report the means and standard deviations of 5 sampling seeds to mitigate the randomness introduced by data sampling.
inal papers. Considering the influence of sam- pling training examples in low-resource experi- ments, we run 5 times with different sampling seeds and report the average results well as standard deviations. Our code will be available httph::t//// //gi thub. wwt.////aaggaa/////mmmo
AVE-S of two datasets. These results provide com- pelling evidence for the effectiveness of our method in low-resource event extraction.
2) is noteworthy that DemoSG exhibits greater improvement in sho-shot settings compared to data-limited settings, and the extent of improve- ment in data-limited settings increases with the growth of available data. Specifically, for argu- ment extraction, DemoSG achieves +4.8%/46.2% improvement in AVE-R, whereas it achieves higher improvement of +7.4%/49.0%% in AVE-S on the two benchmarks. Furthermore, in the data-limited settings of the two benchmarks, De- moSG demonstrates a +1.1 411.%%/46.5%/+8. +3.9%%/49.7%//46.54 increase in Arg-C F1, respec- tively. These observations suggest that our pro- posed demonstration-basee learning may be more effective when there is more balanced data distri- bution or greater availability of demonstrations.
3.2 In-domain Low-resource Scenario
To verify the effectiveness of DemoSG in in- domain low-resource scenarios, we conduct ex- tensive experiments in several few-shol and data- limited settings, following Lu et al. (2022). For the few-shot experiments, we randomly sample 2/5/10 examples per event type from the original training set, while keeping the dev and test sets unchanged. For the data-limited experiments, we directly sam- ple 2%/5% 51%% from the original training set for model training. Compared with the few-shot set- ting, the data-limited setting has unbalanced data distributions and zero-shot situations, posing chal- lenges to the generalization ability of EE methods.
and
3) All of the proposed demonstration selection strategies achieve strong performance and possess distinct characteristies when compared to each other. The similar retrieving strategy excels in tackling low-resource argument extraction, with 2.7% 2.7%/ higher AVE-S of Arg-C than rich- role on the two benchmarks. And the rich-context strategy tends to have better low- rewoureee event de- tection ability, with a 1.7%/0..8% higher AVE-R of Trig-C than similar retriving on both benchmarks.
Table 2 presents the results of in-domain low- resource experiments. We can observe that:
1) DemoSG shows remarkable superiority over other baselines in the in-domain low-resource sce- narios. For argument extraction, DemoSG con- sistently achieves improvements on few-shot and lata-limited settings compared to baseline methods. For instance, in the 2/5/10-shot settings of ACE05- EN and ACE05-EN+, DemoSG outperforms the highest baseline methods by +7.44%/41..0%/44..555 and +9,4%/44.7%/411 .4% in terms of Arg-C Fl, respectively. Regarding event detection, DemoSG also surpasses the highest baselines by +1.0% on AVE-R of ACEOS-EN+ and +5.1% on
3.3 Domain Adaptation Scenario
To investigate the cross-domain knowledge transfer capability of DemoSG, we perform experiments in both the parameter-adaptee and parameter- agnostic domain adaptation settings.

---

ACEOS-EN
ACEOS-EN+
Model
Parameter-adaptive
Parameter-agnostie Trig-C
Parameter-adaptive
Parameter-aguuotie
Trig-C
Arg-C
Arg-C
Trig-C
Arg-C
Trig-C
Arg-C
OnelE
83.1
58.8
77.2
50.9
Tex12Event
77.3
61.6
12.1
9.7
79.8
59.7
22.1
20.3
DEGREE
81.0
63.4
26.9
18.1
81.2
61.4
33,9
26.3
UIE
81.4
62.9
35.2
11.2
81.3
61.7
41.2
22.5
DemoSGR
82.47
69.2
45.91
36.91
81.4
63.7
44.31
34.91
Table Performance comparison of Trig-C C and Arg-C scores(%) parameter-adqptive and purameter-ugnostic domain adaptation settings. Given that OnelE is classification-bass adapting it to target domains without fine- tuning poses a significant challenge. The marker refers significant test p-value < 0.05 when comparing with
DEGREE.
ACEOS-EN
ACEOS-EN+ Trig-C Arg-C
the generation paradigm via demonstration- based learning, enabling DemoSG adapt to new do- mains and event types without further finetuning model parameters on the target domain. Specifi- cally, DemoSG can directly comprehend the extrac- tion process new event types via demonstrations of the target domain at the inference phase. We regard this ability as the parameter-agnostic do- main adaptation, which can avoid the catastrophic forgetting (Li et al., 2022) and the extra comput- ing cost brought by the finetuning process. For the vurameter-agnostt setting, we first train source domain model on the svc, and then directly eval- uate on the tgt set without parameter finetuning. As Table 3 shows, DemoSG gains a significant im- provement in both event detection and argument extraction on both datasets. Regarding event detec- tion, DemoSG surpasses the highest baseline UIE by ++0.7%/43. 4.1%%%%%......... on Trig- C F1, benefiting from its ability to comprehend target domain extraction through demonstrations. In terms of argument ex- traction, although DEGREE showcases strong per- formance the parameter-agnostic setting by in- corporating role relation information of new event types via manually designed prompts, DemoSG still outperforms DEGREE on both datasets by +18.8%/48.66
Model
Type
Trig-C
Arg-C
DYGIE++
Cls
73.6
$2.5
Joint3EE
Cls
69.8
$2.1
GAIL'
Cls
72.0
52.4
EEQA
Cls
72.4
53.3
MQAEE
Cls
71.7
53.4
TANI,
Gen
68.4
47.6
OnelE
Cls
74.2
57.0
72.5
55.9
Tex12Fvent
Gen
71.9"
53.8"
70.3
53.4
DEGREE
Gen
73.3"
55.8"
70.7
55.7
UIE
Gen
73.4*
54.8"
70.7
52.6
DemoSGR
Gen
73.4
56.0
71.2
56.8
Table Performance comparison in the high-resource setting. "Cls" and "Gen" stand for classification-based and generation-based method. Results marked with are sourced from the original paper, and denotes significant test p-value 0.05 compared with DEGREE,
Parameter-adaptive domain adaptation. Fol- lowing Lu et al. (2021), in the parameter-adaptive setting, we divide each dataset into source do- main subset src and a target domain subset igt. The sFc keeps the examples associated with the top 10 frequent event types, and the fgt keeps the sen- tences associated with the remaining 23 event types. For both src and fgt, we sample 80% examples for model training and use the other 20% for evalua- tion. After the sampling, we first pre-train a source domain model on the src, and then fine-tune the model parameters and evaluate on the tgt set. As shown in Table 3, DemoSG outperforms the base- line methods in argument extraction for the target domain, exhibiting a 45.8%//2.0% improvement in compared to
Arg-C Fl. This outcome not only validates the effectiveness of demonstrations for DemoSG but also suggests that demonstration- based learning a more effective approach to help comprehend the task than intricate prompts.
Arg-C on ACEOS-EN/ACEOS the best-performing baselines. For event detection, DemoSG also achieves the highest performance on ACE05-EN+ and performs competitively with the SOTA method ONEIE on ACE05-EE -.005-EE.110 The re- sults suggest that DemoSG possesses strong do- main adaptation capabilities by leveraging the label semantic information of the event schema.
3.4 High-resource Scenario
To gain insights into our framework, we also evalu- ate our method in the high-resource scenario, where each type of training example is abundant. For the high-resource experiments, train all models on the full training sets and evaluate their performance on the original development and sets.
trrameter-agnostee domain adaptation. Un- like previous event extraction methods, we enhance
According to Table 4, although designed for low-

---

ACEOS-EN
ACEOS-EN+ Parameter-adaptive
Model
5-shot
Parameter-adaptive
High-resource
5-shot
High-resource
Trig-C
Arg-C
Trig-C
Arg-C
Trig-C
Arg-C
Trig-C
Arg-C
Trig-C
Arg-C
Trig-C
Arg-C
D
59.3
34,8
82.4
69.2
73.4
56.0
55.6
34.4
81.4
63.7
71.2
56.8
w/o Demo
-3.7
-5.5
-3.0
-0.
-5.4
-28
4.1
-1.3
weo Schema
.49
-544
-6.9
-5.3
-38
Schema
33.0
-18.4
-0.2
33.9
-13.8
Table 5: Ablation study for the key components of DemoSG R(ote) under the 5-shot, parameter-adaptixe and the high-resource settings. Trig-C and Arg-C represent Trig-C Fl score and Arg-C Fl score, respectively.
resource event extraction, our DemoSG also outper- forms baselines by a certain margin the argument extraction (Arg-C) in the high-resource setting on ACEOS-EN+. In terms of the event detection (Trig- C). DemoSG also achieves competitive results on both benchmarks (73.4% and 71.2%). The above results prove that DemoSG has good generaliza- tion ability in both low-resource and high-resource scenarios. Furthermore, we observe that generative methods perform better than classification based methods in many cases under both the low-rcsourve and high-resource scenarios, which illustrates the correctness and great potential of our choice to adopt the generative event extraction framework.
tions and the schema semantic information leads a substantial performance degradation compared to removing only one of them, particularly -26.5% 21.5% for Arg-C F1 the 5-shot setting.
Effectiveness and Robustness of Demonstration-based Learning
Since demonstrations can influence the low- resource EE performance, we design two variants for DemoSG with different kinds of damage to the demonstrations to explore the effectiveness and robustness of demonstration- based learning:
1) Demonsiration Perturbation. To analyze the influence of incorrect demonstrations, sample 40% of the demonstrations and replace the golden triggers and arguments with random spans within the context. Test Perturbation applies perturbation during the inference phase only, while Train-test Perturbation applies perturbation during both the training and inference phases.
3.5 Ablation Study
To examine the impact of each module of De- moSG and the resulting improvement, per- form ablation experiments three variants of De- moSG in the few-shot, parameter-adaptivy and high-resource settings: 1) w/o Demo, which elimi- nates demonstrations and generates records solely based on concatenated input text and schema-based prompts. 2) w/o Schema, which excludes the uti- lization of schema semantics by replacing all la- bels with irrelevant tokens during prompt construc- tion. 3) w/o SchomS, which removes both demonstrations and schema semantics. From Table 5 we can observe that:
2) Demonstration Dropping. To investigate the robustness of demonstration based learning miss- ing demonstrations, randomly drop 40% of the demonstrations during the training inference phases. Test Drop performs dropping during the inference phase only, while Train-test Drop applics dropping during both training and inference.
We conduct the above experiments based on De- moSG with rich-role strategy under the 5-shot set- ting. From Figure 3, we can observe that:
1) The performance of DemoSG experiences significant drops when removing demonstrations in three different scenarios, particularly -5.5% 3.2% for Arg-C F1 in the 5-shot setting on ACFOS-EN//CCEOS-ENN: This result indicates that demonstration--asee learning plays crucial role in our framework, especially for in-domain low- resource event extraction.
1) Incorrect demonstrations exert detrimental influence on the model performance. Specifically, perturbing the demonstrations during the inference phase leads to a reduction -1.%%% on Arg- C FI for both benchmarks. The reduction further increases to -1.8%/-3. 1.1% when the perturbation is applied during both the training and inference phases. Despite experiencing slight performance drop in the presence of incorrect demonstrations, DemoSG consistently outperforms the robust base- line UIE The results underscore the effectiveness of demonstration--aseeee learning, highlighting the influence of accurate demonstrations in helping model understand the extraction process.
2) Incorporating label semantic information of the event schema is essential for DemoSG, as it significantly influences all settings. For instance, removing the schema semantic information results in noteworthy -5.4/-1.9 decrease in Arg-C F1 for the arrameter-adaptive domain adaptation setting. 3) We observe that removing both demonstra-

---

2) The absence of demonstrations has more significant impact than incorrect demonstrations. Specifically, when dropping demonstrations only during the inference phase, there is a reduction of 8.5% in Arg-C FI, resulting in DemoSG performing worse than UIE on ACE05-EN+. We consider that this phenomenon can be attributed to the Exposure Bias (Schmidt, 2019) between the training and inference phases, which also explains why DemoSG exhibits fewer improvements data- limited settings. is noteworthy that dropping demonstrations during both the training and infer- ence phases leads to a recovery of +3.1 %/+3.6% in Arg-C performance compared to Test Drop. These results suggest that reducing exposure bias could be effective approach to enhance the robustness of demonstration -based learning. We leave further investigation of this topic for future studies.
38
.IE
DemoSC
Train-test Perturbation Pamurbatiinn
36
Train-1est Drop
Drop
36
m
92
30
28
26
24
22
29
ACECS-EN
Figure 3: Experiment result of the robustness study under the 5-shot low-resource setting. DemoSG and its variants are based on rich-role in the experiments.
Demonstration-basel Learning. The idea is originally motivated by the in-context learning ability of the GPT3 (Brown et al., 2020), where the
the model learns by only conditioning demonstration without tuning. However, in-context learning relies the enormous size of PLMs. To make the demonstration effective for small PLMs, Lee et al. (2022) proposes a demonstration- based learning framework for NER, which treats demonstrations as additional contextual informa- tion for the sequence tagging model during training, Zhang et al. (2022) further studies the robustness of demonstration basod learning in the sequence tagging paradigm for NER. Our method differs from these studies from: 1) Our demonstration- enhaced paradigm is designed according to the spe- cific characteristics of event extraction. 2) Our generative framework enhances the flexibility of demonstration-baase learning, providing it with the parameter agnossic Nomain-adaptaan capability.
5 Related Work
Low-resource Event Extraction. Most of the previous EE methods focus on the high-resource scenario (Nguyen et al., 2016; Yan et al., 2019; Cui et al., 2020; Du and Cardie, 2020a; Ramponi et al., 2020; Huang and Peng. 2021), which makes them hardly upplied to new domains where the annotated data is limited. Thus, low-resource EE starts to attract attention recently. Lu et al. (2021) models EE a unified eequence-to-strreuure gen- eration, which shares the knowledge between the Event Detection and Argument Extraction subtasks. However, the gap between the downstream struc- ture generation and pre-trate pre-training natural language generation has not been considered. Based on Lu et al. (2021), Lu et al. (2022) proposes several pre- training strategies on large-scale datasets, which boosts the structure generation and improves the low-resource EE performance. However, large- scale pre-training needs lot of computing re- sources and excess corpus. Hsu et al. (2022) incor- porates additional information via manually design- ing a prompt for each event type, which contains the description of the event and the relation of roles. However, designing such delicate prompts still re- quires quite few human efforts, and the EE per- formance is sensitive to the design. Recently, Lou al. (2023) proposes unified token linking opera- tions improve the knowledge transfer ability. We leave the comparison to future studies since their code and data sampling details are not available.
Conclusion
In this paper, we propose the Demonstration- enhanced Schema-guided Generation (DemoSG) model for low-resource event extraction, which benefits from two aspects: Firstly, we propose the demonstrationn-uust based learning paradigm for EE fully use the annotated data, which transforms them into demonstrations to illustrate the extraction pro- cess and help the model learn effectively. Secondly, we formulate EE as a natural language genera- tion task guided by schema-based prompts, thereby leveraging label semantics and promoting knowl- edge transfer in low-resource scenarios. Extensive experiments and analyses show that DemoSG sig- nificantly outperforms current methods in various low-resource and domain adaptation scenarios, and demonstrate the effectiveness of our method.

---

Limitations
George Doddington, Alexis Mitchell, Mark Przybocki. Lance Ramshaw. Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extrac- tion (ACE) program tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'044) Lishon, Portugal. European Language Resources As- sociation (ELRA).
In this paper, we propose the DemoSG model to facilitate low-resource event extraction. To exploit the additional information of demonstrations and prompts, DemoSG generates records of each event type individually. Though achieving significant im- provement in low-resource scenarios, the individ- ual generation makes DemoËE predict relatively slower than methods that generate all records at once (Lu et al., 2021, 2022).
Xinya Du and Claire Cardie. 2020a. Event extraction by answering (almost) natural questions. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 67 1-633, Online. Association for Computational Linguistics.
Ethics Statement
Xinya Du and Claire Cardie. 2020b. Event extraction by answering (almost) natural questions. arXiv preprint 20X4:2004.1362.
The contributions of this paper are purely methodological, specifically introducing the Cemonstrationn-enaanee Schema-guided Genera- tion model (DemoSG) to enhance the performance of low-resource Event Extraction. Thus, this paper does not have any direct negative social impact.
Rujun Han, I-Hung Hsu, Jiao Sun, Julia Baylon, Qiang Ning, Dan Roth, and Nanyun Peng. 2021. ESTER: A machine reading comprehension dataset for reason- ing about event semantic relations. Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 7543-7559, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Acknowledgements
We sincerely thank all anonymous reviewers for their valuable comments and suggestions. This work was supported by Program for Youth Innova- tive Research Team of BUPT No. 2023QNTD02.
I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee, Scott Miller, Prem Natarajan, Kai- Chang, and Nanyun Peng. 2022. DEGREE: A data-efficient generation- based event extraction model. Pro- ceedings of the 2022 Conference of the North Amer- ican Chupter of the Association for Computational Linguistics: Human Language Technologies, pages 1890-1908, Seattle, United States. Association for Computational Linguistics.
References
David Ahn. 2006. The stages of event extraction. In Proceedings of the Workshop on Annotating and Rea- soning about Time and Events, pages 1-8, Sydney, Australia. Association for Computational Linguistics.
Hoig--siang Huang and Nanyun Peng. 2021. Document-level event extraction with efficient end-to-end learning of cross-event dependencies. In Proceedings of the Third Workshop on Narrative Understanding, pages 36-47, Virtual. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Dong- Ho Lee, Akshen Kadakia, Kangmin Tan, Mahak Agarwal, Xinyu Feng. Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, and Xiang Ren. 2022. Good examples make faster learner: Simple Semonnsratioon-aseee learning for low-resource NER. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2687-2700. Dublin, Ireland. Association for Computational Linguisties.
Shiyao Cui, Bowen Yu, Tingwen Liu, Zhenyu Zhang, Xuebin Wang, and Jinqiao Shi. 2020. Edge-enhanced graph convolution networks for event detection with syntactic relation. In Findings of the Association for Computational Linguistics: EMNLP 2020. pages 2339-2339, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence- to-scee---seeuecee pre-training for natural language generation, translation. and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 787 1871-7880, Online, Association for Computa- tional Linguistics.
George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanic Strassel, and Ralph Weischedel. 2004a. The automatic content extrac- tion (ACE) program- - tasks, data, and evaluation. In Proceedings of the Fowrth International Conference on Language Resources and Evaluation LLREC'04) Lisbon, Portugal. European Language Resources As- sociation (ELRA).

---

Dingcheng Li, Zheng Chen, Eunah Cho, Jie Hao. Xi- aohu Liu, Fan Xing, Chenlei Guo, and Yang Liu. 2022. Overcoming catastrophic forgetting during domain adaptation of seq2seq language generation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computu- tional I. Linguistics: Human Language Technologies, pages $441-5454, Seattle, United States. Association for Computational Linguistics.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Ci- cero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation be- tween augmented natural languages. arXiv preprint arXiv:2101.05779
Alan Ramponi, Rob van der Goot, Rosario Lombardo. and Barbara Plank. 2020. Biomedical event extrac- tion as sequence labeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5357-5367, Online. Association for Computational Linguistics.
Fayuan Li, Weihua Peng, Yuguang Chen, Quan Wang, Lu Pan. Yajuan Lyu, and Yong Zhu. 2020. Event extraction as multi multi-tumm turn question answering. Find- ings of the Association for Computational Linguisties: EMNLP 2020. pages 929-838.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China, Association for Com- putational Linguistics.
Ying Lin, Heng Ji. Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7999-8009, Online. Association for Computational Linguistics.
Jie Lou, Yaojie Lu, Dai Dai, Wei Jia, Hongyu Lin, Xi- anpei Han, Le Sun, and Hua Wu. 2023. Universal information extraction as unified semantic matching. arXiv preprint 200.02230..
Florian Schmidt. 2019. Generalization in generation: A closer look at exposure bias. arXiv preprint 191.v219...00222.
Noam Shazeer and Mitchell Stem. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596 PMLR.
Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021. Text2Event: Controllable sequence-to- structure generation for end-to-end event extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing ( (Volume 1: Long Papers), pages 2795-2806, Online. Association for Computational Linguisties.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Fukasz Kaiser, and IIlia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, NIPS' 17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Yaojic Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Uni- fied structure generation for universal information extraction. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3755-,772, Dublin, Ireland. Association for Computational Linguistics.
David Wadden, Ulme Wennberg, Yi Luan, and Han- naneh Hajishirzi. 2019, Entity, relation, and event extraction with contextualized span representations. arXiv preprint arXiv: 1909.03546.
Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grish- man. 2016. Joint event extraction via recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 300-309, San Diego, Califoria. Association for Computational Linguistics.
Haoran Yan, Xiaolong Jin. Xiangbin Meng, Jiafeng Guo, and Xueqi Cheng. 2019. Event detection with multi- order graph convolution and aggregated attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCN) pages 5766- S770. Hong Kong, China. Association for Computa- tional I. Linguisties.
Trung Minh Nguyen and Thien Huu Nguyen. 2019. One for all: Neural joint modeling of entities and events. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 6851-6858.
Hongming Zhang, Xin Liu, Haojie Pan, Yangqiu Song. and Cane Wing-Ki Leung. 2020. Aser: A large- scale eventuality knowledge graph. WWW '20, page 201-211, New York, NY, USA. Association for Com- puting Machinery.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744
Hongxin Zhang, Yanzhe Zhang, Ruiyi Zhang, and Diyi Yang. 2022. Robustness of demonstration- learning under limited data scenario.

---

Tongtao Zhang, Heng Ji, and Avirup Sil. 2019. Joint en- tity and event extraction with generative adversarial imitation learning. Data Intellignnce, 1(2)999-120.
A Comparison of Model Parameters
We put the comparison the PLMs and the amount model parameters of DemoSG with other low- resource baselines in Table 6.
Model
Backbone PLM
Params
OnelE
BERT-large
308M
DEGREE
BART-lagee
404M
Text2Fvent
T5-large
780M
UIE
TS-large
780M
DemoSG
BART-large
404M
Table 6: Comparison of model parameters.


Reviews:
[{"review_id": "eBCOjiIkkt", "reviewer": "Reviewer_XB14", "summary": "", "questions": "* A. In relation to the need to iterate over all event types during extraction (also reported among limitations), what's the execution time of DemoSG compared to other systems? Can DemoSG deal with large event ontologies with many event types?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you sincerely for recognizing our work and providing valuable suggestions! Below is our detailed response to your concerns:\n\n**Q1:** *In relation to the need to iterate over all event types during extraction (also reported among limitations), what's the execution time of DemoSG compared to other systems? Can DemoSG deal with large event ontologies with many event types?*  \n**A1:** Though mentioned in our Limitation section that utilizing separate decoding processes may marginally increase the inference time, the resulting time delay is acceptable. Please refer to the following table for a comparison of training and prediction times on ACE-05. Our training and inference times align closely with those of the recent DEGREE baseline. In comparison to structured generation techniques (UIE and Text2Event) which have extra costs for additional pre-training, DemoSG can even save the entire pretraining costs while introducing an acceptable increase in inference time due to the generation of natural style.  \nMoreover, DemoSG could deal with large event ontologies with many event types. Note that ACE-05 associates with 33 types of events, which can already be considered as having a biggish event ontology. For situations with even larger event ontologies, DemoSG can also be tailored accordingly via a quick event type recall to identify potential event types, thereby saving inference time significantly.  \n| Model               | Training time (s/example) | Inference time (s/example) |\n|----------------------|--------|----------|\n| Text2Event | 0.4085 | 0.0557   |\n| UIE | 0.3175   | 0.0173   |\n| DEGREE | 0.3012   | 0.4608   |\n| DemoSG | 0.3699   | 0.4800   |  \n\n\\* The above results are obtained under consistent experimental environments, including identical batch sizes and computational resources, etc.\n\n**Q2:** *The ideas underlying the proposed DemoSG approach build on well known techniques (e.g., 'demonstration' w.r.t. in-context learning) and the contribution feel overall incremental in its nature.*  \n**A2:** While our approach draws inspiration from In-context Learning, it's important to emphasize the significant differences. In-context Learning relies on the “emergent” capabilities of large language models with a vast number of parameters (e.g., GPT-3.5 with over 100 billion parameters) and extensive pretraining. It does not necessarily require the inclusion of Demonstrations during training, and the choice of Demonstrations can be relatively arbitrary. However, this approach is not suitable for smaller PLM models that prioritize efficient inference speeds, for their weakness in parameter scale. Our goal is to enable Demonstrations to be effective even on models with relatively fewer parameters (similar to models like BART), and efficiently address low-resource event extraction. Hence, in our design of DemoSG, we take various factors into consideration to ensure its demonstration-based learning compatibility:  \n**1. Introducing Demonstrations during Training:** Different from the In-context Learning of LLMs，we have incorporated demonstrations during the training process, enabling the BART model to acquire the ability of understanding the provided demonstrations.  \n**2. Designing Demonstrations Selection strategies:** We have meticulously tailored the methods of selecting Demonstrations to ensure the inclusion of high-quality examples.  \n**3. Modeling EE as Natural Language Generation:** By transforming the extraction task into a generation task, we unify the forms of demonstrations, prompts, and model outputs, fostering their intrinsic correlations and facilitating the transfer of knowledge across different event types.\n\n**Q3:** Regarding the questions mentioned in \"*Typos Grammar Style And Presentation Improvements*\":  \n**A3:** We greatly appreciate your corrections! We will rectify the relevant problems in the camera-ready version. Regarding the question in T3, we think there are no missing elements in the formula. Because both positive and negative samples are treated uniformly as they undergo the same text generation process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* S1. Solid performance in all considered settings, with substantial improvements in some settings and especially in relation to argument extraction.\n\n* S2. Reproducibility facilitated by adequate description of the approach in the paper and by code provided as part of supplemental material.\n\n* S3. Overall well written paper.", "reasons_to_reject": "* A. In relation to the need to iterate over all event types during extraction (also reported among limitations), what's the execution time of DemoSG compared to other systems? Can DemoSG deal with large event ontologies with many event types?\n\n* B. The ideas underlying the proposed DemoSG approach build on well known techniques (e.g., 'demonstration' w.r.t. in-context learning) and the contribution feel overall incremental in its nature.", "questions_to_authors": "* A. In relation to the need to iterate over all event types during extraction (also reported among limitations), what's the execution time of DemoSG compared to other systems? Can DemoSG deal with large event ontologies with many event types?", "paper_topic_and_main_contributions": "SUMMARY. The paper proposes and evaluates a new event extraction method tailored for low-resource scenarios named \"Demonstration-enhanced Schema-guided Generagion (DemoSG)\". To that purpose, DemoSG relies (i) on the capabilities of a pre-trained (fine-tuned) encoder-decoder LM (BART), (ii) on the inclusion of a 'demonstration' (i.e., example text + verbalized extracted event) of the event type to extract among the LM encoder inputs, and (iii) on leveraging the information in event type and role labels by including them in a 'schema-based prompt'. The decoder output is then matched (via string-matching) to trigger and argument mentions in the input text. DemoSG is evaluated in both in-domain (both low- and high-resource) and domain adaptation settings based on ACE05-EN and ACE05-EN+ datasets, where it is compared to several state-of-the-art systems for both low- and high-resource event extraction, exhibiting overall performance improvements or anyhow competitiveness in the considered scenarios.\n\nCONTRIBUTIONS:\n\n* C1. Proposal of the DemoSG model, distinguished in particular by its 'unified representation template' to verbalize demonstrations and prompt, and the selection strategies to choose the demonstration events.\n\n* C2. Experimental evaluation addressing different low/high-resource and in-domain/domain-adaptation settings, and including additional ablation and robustness (w.r.t. employed demonstration, in appendix) tests for DemoSG.\n\n* C3. Improvements over state-of-the-art methods in many of considered experimental settings.\n\nPOST-REBUTTAL UPDATE. I thank the authors for the clarifications and the additional data in their rebuttal. After reading and considering it (esp. the supplied running times) and in the light of other reviews as well, I increased the *excitement* score from 3 to 4 to better reflect my leaning positively towards this submission.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* T1.  [§2.2.1] \"Multiple arguments correspond to\" -> \"that correspond to\" / \"corresponding to\"\n* T2.  [§2.3] \"that is closet\" -> \"closest\"\n* T3.  [Eq. 3] is something missing here to differentiate between samples from D_P and D_N?\n* T4.  [Eq. 4] p(Y|...) and p(y_i|...) should be conditioned also on input sentence X and parameters \\theta"}}, {"review_id": "HcvoDYmGbQ", "reviewer": "Reviewer_7B7C", "summary": "", "questions": "1. Could you provide more insight into the training time and inference time of the model, including details beyond what is presented in Table 6? Additionally, could you consider including metrics like GMACs or GFLOPs for the inference process of the proposed method to further enhance the analysis?\n\n2. The concept of demonstrations generating records for each event type individually (as mentioned in line 304) is intriguing. However, it's not entirely clear how this approach aids unrelated event types. Has there been any assessment, either qualitative or quantitative, of the retrieval quality of demonstrations in these scenarios?\n\n3. Considering the limitations of the pre-trained language model (BART), how was the optimal number of demonstrations determined? Could you elaborate on the selection process for this number? Also, is there a possibility that the length of the target instance might exceed the model's input limit?\n\n4. The similarity between ACE05 and ACE05+ in Table 5 and Figure 3 raises questions about the selection of these two variants as benchmarks. Could you provide more clarity on why these specific benchmarks were chosen over other well-known event extraction benchmarks, such as wiki-event?\n\n5. The application of rich-role and rich-context strategies (lines 249 - 252) could benefit from an illustrative example for better understanding. Specifically, how are these strategies implemented? Furthermore, considering ACE05's instances with multiple event records of varying types, could you explain how these selection strategies are adapted to handle such complexities?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks greatly for your suggestions, especially for the recommendation of supplementing experiments on other benchmarks! The following presents our supplementary experimental results, and our detailed response to your concerns and questions:\n\n**Q1:** *The lack of experiments on diverse datasets weakens the generalizability of the findings. The similarity between ACE05 and ACE05+ in Table 5 and Figure 3 raises questions about the selection of these two variants as benchmarks. Could you provide more clarity on why these specific benchmarks were chosen over other well-known event extraction benchmarks, such as wiki-event?*  \n**A1:** We fully understand your concerns and share the same perspective that diverse datasets could provide a more comprehensive validation of our method's effectiveness. ACE05 and ACE05+ serve as the two predominant baselines in event extraction, as highlighted in recent literature [1-3] with their distinguishing attributes. In addition to ACE05 and ACE05+, We initially planned to extend our experiments to the ERE-EN [4] dataset for a more comprehensive study. However, acquiring the license for ERE-EN took some time. During the review period, we were able to supplement our work with experiments on the ERE-EN dataset.  \nThe experiments on ERE-EN also present the same trend with ACE05 and ACE05+, showing that DemoSG has a superior low-resource EE ability over baseline methods. Here, we present the salient results of low-resource experiments in the following tables. Comparing DemoSG against UIE, we can observe a 1.6% increase in Arg-C F1 on AVG-R, and a 4.8% increase on AVG-S, which also proves the better low-resource EE ability of out method. The full results on the ERE-EN benchmark will be provided in the camera-ready version.  \n**Trig-C:**  \n|     |Model   |  2-shot|  5-shot|  10-shot | AVE-S     | 2%|  5%|  10% | AVE-R  |\n| ----------- | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | -------- |\n|     |  UIE       |  32.2|  36.5  |  39.5  |  36.1      |  **33.1**  |  43.9  |  49.3  |  42.1  |\n| |  DemoSG-Role  |  32.6  |  43.2  |  **46.5**  |  40.8      |  32.3  |  **44.7**  |  **49.9**  |  **42.3**      |\n| |  DemoSG-Context  |  **36.0**  |  **45.0**  |  46.1  |  **42.4**      |  28.5  |  40.2  |  48.0  |  38.9      |\n| |  DemoSG-Similar  |  35.7  |  42.3  |  44.6  |  40.9      |  32.6  |  43.8  |  47.2  |  41.2      |\n\n**Arg-C:**  \n|   |Model   |  2-shot|  5-shot|  10-shot | AVE-S     | 2%|  5%|  10% | AVE-R  |\n| ----------- | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ | -------- |\n|      |  UIE       |  20.1  |  25.9  |  30.5  |  25.5      |  **19.1** |  29.9 |  35.8 |  28.3  |\n| |  DemoSG-Role  |  18.5  |  **30.9**  |  **38.4**  |  29.3      |  18.2  |  **33.2**  |  38.3  |  **29.9**      |\n| |  DemoSG-Context  |  22.3  |  **30.9**  |  37.8 |  **30.3**      |  17.3  |  31.9  |  38.9  |  29.4      |\n| |  DemoSG-Similar  |  **22.9** |  30.8  |  36.0  |  29.9      |  18.4  |  32.2  |  **39.2** |  **29.9**      |\n\n\n**Q2:** *The computational complexity of the proposed method could potentially hinder its practical applicability. Could you provide more insight into the training time and inference time of the model, including details beyond what is presented in Table 6? Additionally, could you consider including metrics like GMACs or GFLOPs for the inference process of the proposed method to further enhance the analysis?*  \n**A2:** Though mentioned in our Limitation section that utilizing separate decoding processes may marginally increase the inference time, the resulting time delay is acceptable. Our training and inference times align closely with those of the recent DEGREE baseline. In comparison to structured generation techniques (UIE and Text2Event) which have extra costs for additional pre-training, DemoSG can even save the entire pretraining costs while introducing an acceptable increase in inference time due to the generation of natural style. Please refer to the following table for a comparison of training and prediction times on ACE05. Besides, thanks for the suggestion of introducing GMACs and GFLOPs metrics. Due to the tight timeframe for author responses, we will include the analysis of GFLOPS in the camera-ready version.  \n| Model               | Training time (s/example) | Inference time (s/example) |\n|----------------------|--------|----------|\n| Text2Event | 0.4085 | 0.0557   |\n| UIE | 0.3175   | 0.0173   |\n| DEGREE | 0.3012   | 0.4608   |\n| DemoSG | 0.3699   | 0.4800   |\n\n\\* The above results are obtained under consistent experimental environments, including identical batch sizes and computational resources, etc.\n\n**Q3:** *Considering the limitations of the pre-trained language model (BART), how was the optimal number of demonstrations determined? Could you elaborate on the selection process for this number? Also, is there a possibility that the length of the target instance might exceed the model's input limit?*  \n**A3:** We choose only one sample to construct the demonstration. As the pioneering introduction of the demonstration-based learning for EE, our primary exploration focuses on the impact of incorporating demonstrations to enhance data utilization for low-resource event extraction. Hence, the investigation into the optimal number of demonstrations will be deferred to future research endeavors.\nThroughout our practical experimentation, we observed that selecting a single demonstration would not breach the model's input limitations. For employing more demonstrations or dealing with document-level input texts in the future studies, we believe that using PLMs with extrapolatable positional embeddings can offer a resolution.\n\n**Q4:** *The application of rich-role and rich-context strategies (lines 249 - 252) could benefit from an illustrative example for better understanding. Specifically, how are these strategies implemented? Furthermore, considering ACE05's instances with multiple event records of varying types, could you explain how these selection strategies are adapted to handle such complexities?*  \n**A4:** When selecting demonstrations for each type of event, we begin by searching the training dataset for samples containing records of these specific event types. Records pertaining to other event types are filtered out from consideration, yielding a set of candidate samples. Subsequently, in accordance with the criteria of demonstration selection (rich-role, rich-context, similarity), samples are chosen from the candidate set. These selected samples are then structured into demonstration sequences using our proposed unified serialization format. In the scenario where a sample encompasses multiple records belonging to different event types, its demonstration for each event type exclusively takes into account the records associated with that particular event type. This design serves to alleviate the issue of interference among event types.\n\n**Q5:** *The concept of demonstrations generating records for each event type individually (as mentioned in line 304) is intriguing. However, it's not entirely clear how this approach aids unrelated event types. Has there been any assessment, either qualitative or quantitative, of the retrieval quality of demonstrations in these scenarios?*  \n**A5:** For each event type, we solely consider samples that contain records of that particular event type for constructing demonstrations, and eliminate any records of other event types. In this way, the selected demonstrations inherently match the corresponding event types. This approach effectively resolves issues related to unrelated event types and potential interference among diverse event types.\n\n**References**  \n[1] A Joint Neural Model for Information Extraction with Global Features  \n[2] Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction  \n[3] Unified Structure Generation for Universal Information Extraction  \n[4] From light to rich ERE: annotation of entities, relations, and events", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper has a well-organized structure that enhances its comprehensibility.\n\n- The noticeable enhancements in results achieved by the proposed approach contribute to the paper's strength.", "reasons_to_reject": "- The lack of experiments on diverse datasets weakens the generalizability of the findings.\n\n- The computational complexity of the proposed method could potentially hinder its practical applicability.", "questions_to_authors": "1. Could you provide more insight into the training time and inference time of the model, including details beyond what is presented in Table 6? Additionally, could you consider including metrics like GMACs or GFLOPs for the inference process of the proposed method to further enhance the analysis?\n\n2. The concept of demonstrations generating records for each event type individually (as mentioned in line 304) is intriguing. However, it's not entirely clear how this approach aids unrelated event types. Has there been any assessment, either qualitative or quantitative, of the retrieval quality of demonstrations in these scenarios?\n\n3. Considering the limitations of the pre-trained language model (BART), how was the optimal number of demonstrations determined? Could you elaborate on the selection process for this number? Also, is there a possibility that the length of the target instance might exceed the model's input limit?\n\n4. The similarity between ACE05 and ACE05+ in Table 5 and Figure 3 raises questions about the selection of these two variants as benchmarks. Could you provide more clarity on why these specific benchmarks were chosen over other well-known event extraction benchmarks, such as wiki-event?\n\n5. The application of rich-role and rich-context strategies (lines 249 - 252) could benefit from an illustrative example for better understanding. Specifically, how are these strategies implemented? Furthermore, considering ACE05's instances with multiple event records of varying types, could you explain how these selection strategies are adapted to handle such complexities?", "paper_topic_and_main_contributions": "This paper introduces the DemoSG model for low-resource Event Extraction (EE). It employs demonstration-based learning and schema-guided generation to enhance performance in low-resource scenarios. More details on methodology and results are needed, but the approach holds promise for improving EE in resource-scarce contexts.\n\nThe paper acknowledges the drawback related to computation cost; however, it overlooks a potential limitation concerning the input length.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "s1YTCZLbUy", "reviewer": "Reviewer_99nK", "summary": "", "questions": "(Lines 304-310): I wonder what the effects of the negative sampling. Specifically, if there are 33 event types, then for a given mention one positive example is made and 32 negative examples which can be sampled from. How often does the model predict more than one possible event type for a given mention, and how do you select then from candidates?\n\n(Line 320): The conversion of the generated text to a data object. How well does the rule-based deterministic algorithm work in practice? Is this just a literal_eval or something? How often does the model overgenerate for example, brackets or matching parentheses or whatever? This kind of evidence I think adds to the argument for the strengths of generative over classification models.\n\nTo make room for more error analysis and discussion, consider collapsing Section 2.2.1 and 2.2.3, which contain near identical content. Also, a part of the introduction (lines 41-72) is repeated nearly verbatim in the related work section (lines 579-606). \n\nThe researchers don’t take the opportunity to consider future directions for EE. A lot of work in EE (including this one) use data based on ACE2005. Is this satisfactory? What opportunities do LLMs make possible for future EE work?\n\nTo strengthen the related work section, you could also include citations for using LLMs for other kinds structure generation (IE-related as well as syntax, other semantic representations, etc).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your acknowledgement for our work, meticulous review comments and valuable suggestions! We greatly concur with your concerns into the future of event extraction, and are willing to engage in discussions regarding the future of event extraction in the age of LLMs. Below is our detailed response to your questions and concerns:\n\n**Q1:** Towards questions about Figure 1: \"Example in Figure 1. I think this is a generally good figure, but it leaves a lot ambiguous to stand on its own...\"  \n**A1:** Thanks for the feedback. The intention behind the design of Figure 1 is to provide an intuitive overview of the event extraction task and showcase our core innovation. Therefore, some details were simplified, and these details are elaborated upon more thoroughly in Figure 2. Here are our responses to your questions:\n- *\"Is this graph information (including mention spans) part of input?\"* \nThe model takes only the training text as input. The arrows in the figure are included to provide a clearer illustration of the event composition in the example sentences, facilitating readers' understanding of our method. \n- *\"Also, for each training input, you select how many demos?\"* \nFor each type of event, we choose only one demonstration. To the best of our knowledges, we are the first to introduce demonstration-based learning architecture for event extraction. The primary focus of our study is to explore the impact of introducing demonstrations on low-resource event extraction. We will leave the exploration of the optimal number of demonstrations for future work, and we will supplement the discussion in the camera-ready version to provide initial insights into how the number of demonstrations may affect performance.\n- *\"Does the training set look like this:\"* We didn’t include ‘mention’ as input. The actual structure of the training set is as follows: [(transport demo + input text + transport prompt), (meet demo + input text + meet prompt), (negative-event demo + input text + negative-event prompt), ...].We have provided formats of training examples in Figure2.\n- *\"And then at inference time, the model generates data for all event types\"* Yes. During the inference phase, extraction results are generated for each type of event.  \nWe will make according revisions to enhance the clarity of Figure 1 in our camera-ready version.\n\n**Q2:** Towards the misunderstanding about our description: \"*EE is sensitive to prompt design*\":  \n**A2:** When we mention “EE is sensitive to prompt design”, we refer to DEGREE's incorporating supplementary information by manually designing a prompt for each event type, encompassing event descriptions and role relations. For instance, the additional description for the *Justice:Sue* event is *somebody was sued by some other in somewhere. The adjudication was judged by some adjudicator.* This fine-grained design makes the method highly sensitive to the event description, and this has been mentioned in their paper.  \nFurthermore, practically, we believe that the natural language form of prompts, instead of the structured prompts used in Text2Event and UIE, is superior. This is because natural language prompts align better with the generation task guided by the model's pretraining, offering better compatibility with downstream tasks, bridging the gap between pretraining and fine-tuning, and promoting the transfer of pretrained knowledge. This design also allows our method to save a significant amount of further pretraining costs compared to UIE and Text2Event.  \n\n**Q3:** Towards questions about demo construction process: \"*What are the 'best demonstrating characteristics'...*\":  \n**A3:** The term 'best demonstration characteristics' refers to the perspective of selecting samples that possess superior demonstration capabilities. This selection is made from the standpoint of the demonstrations themselves. We have proposed two demonstration-oriented selection strategies: rich-role and rich-context.\n- Rich-role selects the sample involving the most event roles for each type of event as the demonstration for that event type. This strategy takes into consideration that training examples carrying more roles can better illustrate the event extraction process for that event type.\n- Similarly, rich-context selects the sample with the longest text as the demonstration, considering that longer contextual information may be helpful.\n\nOur approach only selects the most suitable sample to construct the demonstration, aiming to explore the benefits of using demonstrations for low-resource event extraction.  \nThe difference between our data-limited and few-shot experimental settings lies in the possibility of demonstration absence in the former (because we cannot guarantee that the training data covers all event types, during testing, the model may encounter event types and their demonstrations it has not seen before). However, the model can still generalize to examples without demonstrations through the event prompts, consistently improving DemoSG's overall performance over the baseline.\nRegarding the impact of demonstrations and the issue of their robustness, we conducted corresponding explorations in Section 3.5 Ablation Study as well as the Effectiveness and Robustness experiments in Appendix B. Our robustness experiments validate that mixing training samples with missing demonstrations during training can enhance the model's robustness and generalization ability in scenarios where demonstrations are missing. \n\n**Q4:** Towards the suggestion about weaknesses analysis:  \n**A4:** In Appendix B, we delve into the robustness of DemoSG, showcasing areas where DemoSG has room for improvement, specifically cases where the model has demonstrations during training but lacks them during prediction. We also provide initial insights into how to address this weakness in future optimizations, namely by mitigating the exposure bias issue via introducing demonstrations during the training process. We will include further analysis of bad cases in the Case Study section of the camera-ready version.\n\n**Q5:** Towards questions about our ablation study:  \n**A5:** We appreciate the insightful question. In our ablation study involving the use of schema semantic information, we removed event type descriptions and replaced the roles in the prompts with role indices. For example, \"Origion is [mask]. Destination is [Mask]…\" was replaced with \"Role1 is [mask]. Role2 is [Mask]…\", and the label semantics from the schema were removed. To ensure a fair ablation experiment, we simultaneously replaced both roles in the demonstrations and the prompts with role indices to maintain their consistency. Hence, we think it is an apples-to-apples comparison. Regarding the significant drop observed when simultaneously ablating schema and semantic information, we believe this phenomenon may be due to the fact that demonstrations and label semantics not only have a cumulative effect but also share some common influence. This may be a crucial factor for the success of generative models in low-resource event extraction tasks.\n\n**Q6:** Towards your concern \"*An additional concern I have is the use (or overuse) of the same benchmark dataset...*\" and the future of EE in the age of LLM:  \n**A6:** As for your additional concern, This is a profound reflection. We agree with your perspective that the event extraction task has been extensively studied on benchmarks like ACE and ACE+, and it requires new and more challenging benchmarks. However, until such a innovative benchmark is proposed, we believe that future research in event extraction can focus on the following aspects:\n- **Low-resource event extraction:** Previous methods have mostly relied on large amounts of data. For the event extraction task that has high annotation costs, addressing event extraction under low-resource scenarios is challenging, which is also the motivation of our study.\n- **Multimodal Event Extraction:** With the increasing trend toward multimodal information, events are increasingly presented in the form of multimodal images and text. For example, news reports about certain events often come with accompanying images. Research on how to combine images to enhance performance in multimodal event extraction is a promising future research direction [1].\n- **Document-level event extraction:** Event extraction from long texts at the document level presents unique challenges (multi-event, argument-scattering [2]), and efficiently capturing cross-document event information is also a significant challenge, even for LLMs.\n- **More elegant integration of LLM capabilities:** LLMs have demonstrated their strong capabilities in multiple tasks. Directly modeling a task as a text generation and conducting instructional supervised fine-tuning on LLMs often yields improvements. However, fine-tuning LLMs consumes significant computational resources. Future research directions should explore more elegant ways to harness the advantages of LLMs. For example, in this paper, we propose the demonstration-based learning architecture for event extraction, bringing the in-context learning ability of LLMs to smaller PLMs (BART), enabling small models to understand event extraction tasks through demonstrations, and enhancing their low-resource event extraction capabilities. We believe that exploring methods that can elegantly combine the strengths of large models will be one of the future directions in the era of LLMs.\n\n**Q7:** Towards your question about effects of negative sampling: \"*I wonder what the effects of the negative sampling...*\":  \n**A7:** Negative sampling is applied exclusively during the model training process to balance the ratio of positive and negative samples. A single sample often contains one or several types of events. For these included event types, the sample is considered a positive sample, while for the event types not included, it is considered a negative sample.  \nConsequently, the total number of negative samples is quite large, and if without negative sampling, the model would tend to predict *no event occurred*, leading to a decrease in Recall scores. During prediction, there is no need for negative sampling.\n\n\n**Q8:** Towards the confusion about the conversion of the generated texts to event records:  \n**A8:** During decoding, we match the defined templates: first, we segment multiple event records based on triggers, and then we locate the corresponding arguments based on the roles within each event record.  \nWe model event extraction as natural language generation rather than structured generation (as in UIE and Text2Event). This approach mitigates the gap between natural language generation tasks during PLM pretraining and downstream fine-tuning tasks. Therefore, our model can learn the output format even in low-resource scenarios without issues such as redundant parentheses. \n\n\n**Q9:** Towards the suggestion for improving the related work section:  \n**A9:** Thank you for the suggestion. We will continue to monitor recent IE-related work that leverages LLMs and make appropriate additions to the related work section.\n\n**References**  \n[1] Cross-media Structured Common Space for Multimedia Event Extraction  \n[2] Document-level event extraction via heterogeneous graph-based interaction model with a tracker", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n\nThe researchers show strong evidence of how to improve efficiency of the use of annotated data using demonstrations and schema-based prompts, thus they show strong evidence of how EE can be improved in low-resource settings. Seq2seq generation for event extraction, or structure generation, appears to work also in cross-domain EE with the additional signal of event participant labels. \n\nTo increase the strength of their argument, the researchers tested the hypothesis of cross-domain event meaning transfer, measuring how well does the model transfer knowledge between domains measuring the “parameter-agnostic domain (sic) adaption capability” of the model (section 3.3), testing how knowledge transfers across different events using schema-guided natural language generation (leveraging similarity between event types / entity types). See domain adaptation scenario (lines 171-181) and results reported (lines 466+). However, it is a missed opportunity to examine what factors are at play in increasing performance here (see below).", "reasons_to_reject": "Example in Figure 1. I think this is a generally good figure, but it leaves a lot ambiguous to stand on its own. For example, the input text is shown with directed edges between events and their arguments. Is this graph information (including mention spans) part of input? Also, for each training input, you select how many demos? Does the color of the boxes around “select demo” examples and “schema” examples mean something (the red vs. the blue)? Is this to create a single positive example, or multiple? Does the training set look like this: [(input text + ?event mention + transport demo + transport schema), (input text + ?event mention + meet demo + meet schema), (input text + ?event mention + negative-event demo + negative-event schema), …]? And then at inference time, the model generates data for all event types? A lot of this could be of been made more clear in the figure or early in the paper. \n\n(Line 72): You say that “EE is sensitive to prompt design”. Did you test any variants of prompts? Are there general best practices for EE prompt design that you found that helps explain some of the variance observed?\n\n(section 2.2.2) Event demonstration construction processes are not clearly defined. What are the “best demonstrating characteristics” (lines 242-243)? How many demonstrations does the model select on average between the demonstration-oriented and the instance-oriented methods? How does this affect the decoding process, i.e. if a demo for a specific event type is not selected for a specific example that is indeed associated with that event type, how well does the model do in generalizing to “unseen” event types in the low-resource setting? What is the impact of this on performance of the “select demo” part of Figure 1? It would help to parse into natural language the functions in the included utils.py, for example, maybe as part of the Appendix.\n\nResults section ignores the model’s weaknesses: there is not an error analysis, making it hard to understand better what still needs to be improved. How well does the model do in special cases? Does model performance reflect mostly good performance on the more frequent event types (say, Attack event?) How well does the model do for arguments associated with multiple arguments? Is all of the input limited to single sentences, or are there cases where arguments need to be extracted across larger contexts? More broadly, where does the model fail more predictably?\n\nThe ablation study is not airtight. The claim that adaptation was due to “leveraging the label semantic information of the event schema” needs some support to be more convincing. It is not clear how this was tested “removing the schema semantic information” (line 570): How is this done in practice, replacing semantic roles like “adjudicator” with an empty ARG label? Is it an apples-to-apples comparison? Also, does the schema-based prompts depend on the demonstration selection? If so, how does this affect the ablation study? And how to account for the big drop in performance when both schema and demos are removed (lines 573-577)?\n\nAn additional concern I have is the use (or overuse) of the same benchmark dataset (which has been around since 2005) which limits how much the event extraction field can be pushed towards new challenges, like reasoning about how participants change as the result of an event sequence, or selecting salient events, or building timelines of events, or summarization or something. What does the research suggest for the next iteration of experiments and the future of event understanding tasks in this day and age of LLMs?", "questions_to_authors": "(Lines 304-310): I wonder what the effects of the negative sampling. Specifically, if there are 33 event types, then for a given mention one positive example is made and 32 negative examples which can be sampled from. How often does the model predict more than one possible event type for a given mention, and how do you select then from candidates?\n\n(Line 320): The conversion of the generated text to a data object. How well does the rule-based deterministic algorithm work in practice? Is this just a literal_eval or something? How often does the model overgenerate for example, brackets or matching parentheses or whatever? This kind of evidence I think adds to the argument for the strengths of generative over classification models.\n\nTo make room for more error analysis and discussion, consider collapsing Section 2.2.1 and 2.2.3, which contain near identical content. Also, a part of the introduction (lines 41-72) is repeated nearly verbatim in the related work section (lines 579-606). \n\nThe researchers don’t take the opportunity to consider future directions for EE. A lot of work in EE (including this one) use data based on ACE2005. Is this satisfactory? What opportunities do LLMs make possible for future EE work?\n\nTo strengthen the related work section, you could also include citations for using LLMs for other kinds structure generation (IE-related as well as syntax, other semantic representations, etc).", "paper_topic_and_main_contributions": "Summary: Building upon previous work in structured generation (Lu et al 2021, Lu et al 2022, Hsu et al 2022), this paper presents work in low-resource event extraction using NLG and schema-based prompts, where NLG involves large language seq2seq models with raw text input and structured representation output for events and event arguments. “Demonstration-enhanced” refers to sampling methods to select the best examples for in-context learning, while “schema-based prompts” are natural language event templates for each event type. By low-resource settings, the researchers mean how many examples the model sees (full supervision vs few-shot in-context learning). Their methods outperform other systems in low-resource scenarios, and the paper gives evidence for generative paradigms outperforming classification methods more generally. The main dataset used are variants of ACE2005, and code will be open source (some code available as attachment). Main contributions are experiments in structure generation and studying event extraction in low resource settings (data efficiency).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Throughout the paper, change “domain adaption” to “domain adaptation”"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces DemoSG to handle event extraction low-resource scenarios. DemoSG leverages BART and incorporates a 'demonstration' of the event type to extract, along with schema-based prompts that include event type and role labels. The BART decoder output is matched to trigger and argument mentions in the input text. The evaluation covers in-domain and domain adaptation settings using ACE05-EN and ACE05-EN+ datasets, demonstrating competitive or improved performance compared to state-of-the-art systems for low- and high-resource event extraction. 



The reviewers raised considerable concerns but the reviewers could convince the reviewers and address many of their concerns.  Additionally, the reviewers also agree that the contribution is exciting and the results are reproducible.
Decision: Accept-Findings