SlowGuess/NAACL_2022_OCR huggingface dataset:
For all of DEGREE, DEGREE(ED), and DEGREE(EAE), we fine-tune the pre-trained BARTlarge (Lewis et al., 2020) with Huggingface package (Wolf et al., 2020). The number of parameters is around 406 millions. We train DEGREE with our machine that equips 128 AMD EPYC 7452 32-Core Processor, 4 NVIDIA A100 GPUs, and 792G RAM. We consider AdamW optimizer (Loshchilov and Hutter, 2019) with learning rate set to  $10^{-5}$  and the weight decay set to  $10^{-5}$ . We set the batch size to 6 for DEGREE(EAE) and 32 for DEGREE(ED) and DEGREE. The number of training epochs is 45. It takes around 2 hours, 18 hours, 22 hours to train DEGREE(EAE), DEGREE(ED), and DEGREE, respectively.
We do hyper-parameter search on  $m$ , the number of negative examples, from  $\{3, 5, 7, 10, 13, 15, 18, 21\}$ , and our preliminary trials show that  $m$  less than 10 are usually less useful. For the learning rate and the weight decay, we tune it based on our preliminary experiment for event argument extraction from  $\{10^{-5}, 10^{-4}\}$ , while they are both fixed to  $10^{-5}$  for all the experiments.
We consider ACE  $2005^{5}$  (Doddington et al., 2004) and  $\mathrm{ERE}^6$  (Song et al., 2015). Both consider LDC User Agreement for Non-Members as the licenses. Both datasets are created for entity, relation, and event extraction while our focus is only event extraction in this paper. In the original ACE 2005 dataset, it contains data for English, Chinese, and Arabic and we only take the English data for our experiment. In the original ERE dataset, it contains data for English, and Chinese and we only take the English data for our experiment as well.
Because both datasets contain event like Justice:Execute and Life:Die, it is possible that some offensive words (e.g., killed) would appear in the passage. Also, some real names may appear in the passage as well (e.g., Palestinian president, Mahmoud Abbas). How to accurately identify these kinds of information is part of the goal of the task. Therefore, we do not take any changes on the datasets for protecting or anonymizing.
We split the training data based on documents, which is a more realistic setup compared to splitting data by instance. Table 11 lists the statistics of ACE05-E, ACE05-E $^{+}$ , and ERE-EN. Specifically, we try to make each proportion of data contain as many event types as possible.
This section describes the implementation details for all baselines we use. We run the experiments with three different random seeds and report the best value.
- DyGIE++: we use their released pre-trained model<sup>8</sup> for evaluation.
- OneIE: we use their provided code<sup>9</sup> to train the model with default parameters.
- BERT_QA: we use their provided code<sup>10</sup> to train the model with default parameters.
- TANL: we use their provided code<sup>11</sup> to train the
4The valid roles for each event type are predefined in the event ontology for each dataset, or can be decided by the user of interest.
$^{5}$ https://catalog.ldc.upenn.edu/LDC2006T06 $^{6}$ https://catalog.ldc.upenn.edu/LDC2020T19 $^{7}$ https://catalog.ldc.upenn.edu/license/ldc-non-members-agreement.pdf $^{8}$ https://github.com/dwadden/dygiepp $^{9}$ http://blender.cs.illinois.edu/software/oneie/ $^{10}$ https://github.com/xinyadu/eeqa $^{11}$ https://github.com/amazon-research/tanl
model. We conduct the experiments with two variations: (1) using their default parameters, and (2) using their default parameters but with more training epochs. We observe that the second variant works better. As a result, we report the number obtained from the second setting.
- Text2Event: we use their official code<sup>12</sup> to train the model with the provided parameter setting.
- dbRNN: we directly report the experimental results from their paper.
- Joint3EE: we directly report the experimental results from their paper.
- MQAEE: we directly report the experimental results from their paper.
- BART-Gen: we report the experimental results from their released appendix.13
$^{12}$ https://github.com/luyaojie/  
Text2Event  
 $^{13}$ https://github.com/raspberryice/  
gen-arg/blob/main/NAACL_2021_Appendix.  
pdf
Event Type	EAE Template
Life:Be-Born	somebody was born in somewhere.
Life:Marry	somebody got married in somewhere.
Life:Divorce	somebody divorced in somewhere.
Life:Injure	somebody or some organization led to some victim injured by some way in some-where.
Life:Die	somebody or some organization led to some victim died by some way in somewhere.
Movement:Transport	something was sent to somewhere from some place by some vehicle. somebody or some organization was responsible for the transport.
Transaction:Transfer-Ownership	someone got something from some seller in somewhere.
Transaction:Transfer-Money	someone paid some other in somewhere.
Business:Start-Org	somebody or some organization launched some organization in somewhere.
Business:Merge-Org	some organization was merged.
Business:Declare-Bankruptcy	some organization declared bankruptcy.
Business:End-Org	some organization dissolved.
Conflict:Attack	some attacker attacked some facility, someone, or some organization by some way in somewhere.
Conflict:Demonstrate	some people or some organization protest at somewhere.
Contact:Meet	some people or some organization met at somewhere.
Contact:Phone-Write	some people or some organization called or texted messages at somewhere.
Personnel:Start-Position	somebody got new job and was hired by some people or some organization in somewhere.
Personnel:End-Position	somebody stopped working for some people or some organization at somewhere.
Personnel:Nominate	somebody was nominated by somebody or some organization to do a job.
Personnel:Elect	somebody was elected a position, and the election was voted by some people or some organization in somewhere.
Justice:Arrest-Jail	somebody was sent to jailed or arrested by somebody or some organization in somewhere.
Justice:Release-Parole	somebody was released by some people or some organization from somewhere.
Justice:Trial-Hearing	somebody, prosecuted by some other, faced a trial in somewhere. The hearing was judged by some adjudicator.
Justice:Charge-Indict	somebody was charged by some other in somewhere. The adjudication was judged by some adjudicator.
Justice:Sue	somebody was sued by some other in somewhere. The adjudication was judged by some adjudicator.
Justice:Convict	somebody was convicted of a crime in somewhere. The adjudication was judged by some adjudicator.
Justice:Sentence	somebody was sentenced to punishment in somewhere. The adjudication was judged by some adjudicator.
Justice:Fine	some people or some organization in somewhere was ordered by some adjudicator to pay a fine.
Justice:Execute	somebody was executed by somebody or some organization at somewhere.
Justice:Extradite	somebody was extradicted to somewhere from some place. somebody or some organization was responsible for the extradition.
Justice:Acquit	somebody was acquitted of the charges by some adjudicator.
Justice:Pardon	somebody received a pardon from some adjudicator.
Justice:Appeal	some other in somewhere appealed the adjudication from some adjudicator.
Table 9: All EAE templates for ACE05-E and ACE05-  $\mathbf{E}^{+}$
Table 10: All EAE templates for ERE-EN.
Dataset	Split	#Docs	#Sents	#Events	#Event Types	#Args	#Arg Types
ACE05-E	Train (full)	529	17172	4202	33	4859	22
Train (1%)	5	103	47	14	65	16
Train (2%)	10	250	77	17	104	16
Train (3%)	15	451	119	23	153	17
Train (5%)	25	649	212	27	228	21
Train (10%)	50	1688	412	28	461	21
Train (20%)	110	3467	823	33	936	22
Train (30%)	160	5429	1368	33	1621	22
Train (50%)	260	8985	2114	33	2426	22
Dev	28	923	450	21	605	22
Test	40	832	403	31	576	20
ACE05-E+	Train (full)	529	19216	4419	33	6607	22
Train (1%)	5	92	49	15	75	16
Train (2%)	10	243	82	19	129	16
Train (3%)	15	434	124	24	203	19
Train (5%)	25	628	219	27	297	21
Train (10%)	50	1915	428	29	629	21
Train (20%)	110	3834	878	33	1284	22
Train (30%)	160	6159	1445	33	2212	22
Train (50%)	260	10104	2231	33	3293	22
Dev	28	901	468	22	759	22
Test	40	676	424	31	689	21
ERE-EN	Train (full)	396	14736	6208	38	8924	21
Train (1%)	4	109	61	14	78	16
Train (2%)	8	228	128	21	183	19
Train (3%)	12	419	179	26	272	19
Train (5%)	20	701	437	31	640	21
Train (10%)	40	1536	618	37	908	21
Train (20%)	80	2848	1231	38	1656	21
Train (30%)	120	4382	1843	38	2632	21
Train (50%)	200	7690	3138	38	4441	21
Dev	31	1209	525	34	730	21
Test	31	1163	551	33	822	21
Table 11: Dataset statistics. Our experiments are conducted in sentences, which were split from documents. In the table, "#Docs" means the number of documents; "#Sents" means the number of sentences, "#Events" means the number of events; "#Event Types" means the number of event types in total; "#Args" means the number of argument in total; "#Arg Types" means the number of argument role types in total.
E Few-Shot and Zero-Shot Event Extraction
In order to further test our models' generaliability, we additionally conduct zero-shot and few-shot experiments on the ACE05-E dataset with DEGREE(ED) and DEGREE(EAE).
Settings. We first select the top  $n$  common event types as "seen" types and use the rest as "unseen/rare" types, where the top common types are listed in Table 12. To simulate a zero-shot scenario, we remove all events with "unseen/rare" types from the training data. To simulate a few-shot scenario, we keep only  $k$  event examples for each "unseen/rare" type (denoted as  $k$ -shot). During the evaluation, we calculate micro F1-scores only for these "unseen/rare" types.
n	Seen Event Types for Training/Development
5	Conflict:Attack, Movement:Transport, Life:Die, Contact:Meet, Personnel:Elect
10	Conflict:Attack, Movement:Transport, Life:Die, Contact:Meet, Personnel:Elect, Life:Injure, Personnel:End-Position, Justice:Trial-Hearing, Contact:Phone-Write, Transaction:Transfer-Money
Table 12: Common event types in ACE05-E.
Compared baselines. We consider the following baselines: (1) BERT_QA (Du and Cardie, 2020) (2) OneIE (Lin et al., 2020) (3) Matching baseline, a proposed baseline that makes trigger predictions by performing string matching between the input passage and the event keywords. (4) Lemmatization baseline, another proposed baseline that performs string matching on lemmatized input passage and the event keywords. (Note: (3) and (4) are baselines only for event detection tasks.)
Experimental results. Figure 4, Table 13, and Table 14 show the results of  $n = 5$  and  $n = 10$ . From the two subfigures in the left column, we see that DEGREE(ED) achieves promising results in the zero-shot setting. In fact, it performs better than BERT_QA trained in the 10-shot setting and OneIE trained in the 5-shot setting. This demonstrates the great potential of DEGREE(ED) to discover new event types. Interestingly, we observe that our two proposed baselines perform surprisingly well, suggesting that the trigger annotations in ACE05-E are actually not diverse. Despite their impressive performance, DEGREE(ED) still outperforms the matching baseline by over  $4.7\%$  absolute trigger classification F1 in both  $n = 5$  and
$n = 10$  cases in zero-shot scenario. Additionally, with only one training instance for each unseen type, DEGREE(ED) can outperform both proposed baselines.
Next, we compare the results for the event argument extraction task. From the two middle subfigures, we observe that when given gold triggers, our model performs much better than all baselines with a large margin. Lastly, we train models for both trigger and argument extraction and report the final argument classification scores in the two right subfigures. We justify that our model has strong generalizability to unseen event types and it can outperform BERT_QA and OneIE even when they are both trained in 5-shot settings.
(a) Results for top common 5 event types.
Figure 4: The zero/few-shot experimental results. Left: The result for the models on event detection task with the scores reported in trigger classification F1. Middle: The models are tested under the scenario of given gold trigger and evaluated with argument classification criterion. Right: The results for the models to perform event extraction task, which aims to predict triggers and their corresponding arguments (we report the argument classification F1).
(b) Results for top common 10 event types.
Event Extraction
Trigger Argument	Common 5	Common 10
Tri-I	Tri-C	Arg-I	Arg-C	Tri-I	Tri-C	Arg-I	Arg-C
Matching Baseline	42.7	42.1	-	-	46.3	46.3	-	-
Lemmatization Baseline	51.5	50.2	-	-	56.6	56.0	-	-
BERT_QA 1-shot	10.0	1.4	1.3	1.3	8.2	1.6	1.1	1.1
BERT_QA 5-shot	14.0	12.6	11.1	10.8	20.8	15.4	14.6	13.9
BERT_QA 10-shot	37.8	33.5	22.9	22.1	32.0	27.8	19.5	18.6
OneIE 1-shot	4.2	4.2	1.5	1.5	4.1	2.7	2.0	2.0
OneIE 5-shot	39.3	38.5	24.8	22.8	41.9	41.9	29.7	27.2
OneIE 10-shot	54.8	53.3	36.0	34.9	61.5	57.8	41.4	39.2
DEGREE(ED) 0-shot	DEGREE(EAE) 0-shot	53.3	46.8	29.6	25.1	60.9	54.5	42.0	31.4
DEGREE(ED) 1-shot	DEGREE(EAE) 1-shot	60.1	53.3	38.8	31.6	61.2	60.9	41.1	34.7
DEGREE(ED) 5-shot	DEGREE(EAE) 5-shot	57.8	55.5	40.6	36.1	65.8	64.8	45.3	42.7
DEGREE(ED) 10-shot	DEGREE(EAE) 10-shot	63.8	61.2	46.0	42.0	72.1	68.8	52.5	48.4
OneIE (Full)	72.7	70.5	52.3	49.9	74.5	73.0	51.2	48.9
DEGREE(ED) (Full)	DEGREE(EAE) (Full)	68.4	66.0	51.9	48.7	72.0	69.8	52.5	49.2
Table 13: Full results of zero/few-shot event extraction on ACE05-E.
Event Argument Extraction
Trigger	Argument	Common 5	Common 10
Tri-I	Tri-C	Arg-I	Arg-C	Tri-I	Tri-C	Arg-I	Arg-C
Gold Triggers	BERT_QA 0-shot	100.0	100.0	55.8	37.9	100.0	100.0	57.2	46.7
Gold Triggers	BERT_QA 1-shot	100.0	100.0	55.8	44.3	100.0	100.0	57.8	47.2
Gold Triggers	BERT_QA 5-shot	100.0	100.0	56.6	49.6	100.0	100.0	59.1	50.6
Gold Triggers	BERT_QA 10-shot	100.0	100.0	58.8	52.9	100.0	100.0	60.5	52.8
Gold Triggers	OneIE 1-shot	100.0	100.0	40.9	36.5	100.0	100.0	48.3	44.2
Gold Triggers	OneIE 5-shot	100.0	100.0	55.6	51.4	100.0	100.0	58.6	55.0
Gold Triggers	OneIE 10-shot	100.0	100.0	59.4	56.7	100.0	100.0	62.0	59.5
Gold Triggers	DEGREE(EAE) 0-shot	100.0	100.0	56.1	48.0	100.0	100.0	66.5	53.3
Gold Triggers	DEGREE(EAE) 1-shot	100.0	100.0	65.2	55.2	100.0	100.0	65.4	54.7
Gold Triggers	DEGREE(EAE) 5-shot	100.0	100.0	70.9	62.2	100.0	100.0	68.0	61.7
Gold Triggers	DEGREE(EAE) 10-shot	100.0	100.0	71.1	64.2	100.0	100.0	71.6	64.3
Gold Triggers	BERT_QA (Full)	100.0	100.0	63.1	57.9	100.0	100.0	62.1	56.5
Gold Triggers	OneIE (Full)	100.0	100.0	70.8	66.4	100.0	100.0	67.9	64.1
Gold Triggers	DEGREE(EAE) (Full)	100.0	100.0	74.5	70.6	100.0	100.0	73.6	68.9
Table 14: Full results of zero/few-shot event argument extraction on ACE05-E.