AniEE: A Dataset of Animal Experimental Literature for Event Extraction
Authors: Dohee Kim
Ra Yoo
Soyoung Yang
Hee Yang
Jaegul Choo
Venue: EMNLP 2023 Findings (2023)
Keywords: Information Extraction
Event Extraction
Named Entity Recognition
Biomedical Corpus
Scientific Literature
Animal Experiments

Abstract: Event extraction (EE), as a crucial information extraction (IE) task, aims to identify event triggers and their associated arguments from unstructured text, subsequently classifying them into pre-defined types and roles. In the biomedical domain, EE is widely used to extract complex structures representing biological events from literature. Due to the complicated semantics and specialized domain knowledge, it is challenging to construct biomedical event extraction datasets. Additionally, most existing biomedical EE datasets primarily focus on cell experiments or the overall experimental procedures. Therefore, we introduce AniEE, an event extraction dataset concentrated on the animal experiment stage. We establish a novel animal experiment customized entity and event scheme in collaboration with domain experts. We then create an expert-annotated high-quality dataset containing discontinuous entities and nested events and evaluate our dataset on the recent outstanding NER and EE models.

AniEE: A Dataset of Animal Experimental Literature for Event Extraction
Dohee Kim o
Ra Yoo
Soyoung Yang KAIST AI
Hee Yang 9 Jaegul Choo
Advanced Institute of Convergence Technology,
Kookmin University
{doheel 121, sy_yang, rayooêsnu. att.ssuu.c.kk. kc, yhee61 www.ookmin..ccoomomoommm ac..miiiiicc.................... ac. kr
acc.aisttaa...... kr
Abstract
Object
Subject
Amount
Schedule
Event extraction (EE), as a crucial information extraction (IE) task, aims identify event trig- gers and their associated arguments from un- structured lext, subsequently classifying them into pre-defined types and roles. In the biomed- ical domain, BE is widely used to extract com- plex structures representing biological events from the literature. Due to the complicated syntactic nature and specialized domain knowl- edge, it is challenging to construct biomedi- cal EE datasets. Additionally, most existing biomedical EE datasets primarily focus on cell experiments the overall experimental proce- dures. Therefore, we introduce AniEE, NER dataset concentrated on the animal ex- periment stage. We establish novel animal experiment customized entity and event scheme collaboration with domain experts. We then create expert-annotated high-quality dataset containing discontinuous entities and nested events and evaluate the recent outstanding NER and EE models our dataset. The dataset is publicly available at: https://gi thub, com/ domyown/ under CC BY-NC-ND 4.0 li- cense.
BNN27 (2, 10, and 50 mg/ /kg]
Lp. for
7 days
SAMPLE
DURATION
ADMINISTRATION
administration to rats
ANIMAL
SUBJECT
Sample Administration
Cause
Object
BNN27 increased the Anti-infammatory
SAMPLE
POSITIVE
MOLECULAR
REGULATION
BIOMARKER
(II.-10 and I1.44
(b) Positive Regulation
Cause
Object
Site
BNN27 reversed the glial activation
SAMPLE
NEGATIVE
ANATOMICAL POSITIVE
NAME
REGULATION
ENTITIES
REGULATION
(c) Negative Regulation
Figure 1: Each example for three event types from
AniEE: SampleAdministratioon, PositiveRegulation, and NegativeReguuation
extract the complex syntactic structures of bio- logical events. Each event consists of an event trigger and arguments, as well the rela- tions between them. Figure (a) illustrates an example of three which includes event trigger "i.p." and Ob- ject, Subject, Schedule, and three Amount ar- guments. Additionally, Figure (b) and (c) de- scribe examples of PositiveRegulation and Neg- ativeRegulation, where "BN27" is the Cause of the events.
1 Introduction
Scientific literature provide a groundwork for further investigations and offer valuable platform for turning academic achievements into tangible industrial outcomes. Especially in the biomedical domain, the volume of litera- ture grows exponentially (Frisoni al., 2021)', posing challenges in locating relevant experi- mental key facts within unstructured text. help researchers discover related literature that fulfills their needs, information extraction (IE) actively utilized. Across diverse IE tasks, event extraction (EE) has been spotlighted to
events,
Among various biomedical literature, animal experiment-relatte articles are one the most difficult texts extract valuable information from. The experimental phases can be gen- erally divided into three stages: within cells, animal experiments, and clinical trials. Com- pared to cell experiments, animal research re-
"qqual Contributions.
'A of January 2021, PubMed alone has a total of 32M articles dating back to the year 1781. In 2020, three papers were published every minute.

---

Task
Sub-task
Number of Types
Count
Dataset
Topic
NER
EE
Disc.Entity
Nest.Event
Entity
Role
Event
Entity
Role
Event
ShARe13
Clinical notes
11,161
CADEC
Medical forum
6,318
GET1
Cell proteins
v
V
2
6
16,976
10,270
14,840
Cell, Animal,
MIEEE
16
8,291
7,588
5,554
Clinical trials
AniEE
Animal
12
22,105
17,538
10.546
(Ours)
Table I:
Comparison of clinical-domain named entity recognition (NER) datasets and biomedical-domain event extraction (EE) datasets: CLEF eHealth Task 2013 (ShARe13) (Danielle et al., 2013), CADEC (Sarvnaz Multi-level Event Extraction (MLEE) (Pyysalo al.,
al., 2015), GENIA (GE11) (Kim et al., 2011), 2012). To the best of knowledge, AniEE is the first dataset animal experiment literature containing both discontinuous entities "Disc.Entity) and nested events (Nest.Event). The number of documents and sentences is described in Appendix Table 12.
quires further due diligence because they must consider ethical guidelines have extensive resource requirements. More importantly, be- fore moving on to further clinical trials with human participants, animal research serves as significant step to evaluate safety and efficacy. Therefore, thorough investigations of previous research essential design animal exper- iments, verifying information such as species, dosage, and duration, and the relations between them, as shown in Figure 1.
tinuous entities and 2) nested events. Existing benchmark datasets not include both discon- tinuous entities and nested events. Therefore, we anticipate our dataset will contribute to the advance of all relevant NER and sub-tasks. We sum up our contributions follows:
We introduce a new high-quality dataset AniEE consisting of 350 animal experi- mental literature annotated by domain ex- perts.
We define novel entity and event scheme tailored to animal experiments.
Despite the importance of experimental infor- mation, EE studies targeting animal experiment literature have rarely been conducted. One of the reasons is, described in Table 12, exist- ing EE datasets contain literature that is either limited to cell experiment stage (Kim et al., 2011; Pyysalo et al., 2013; Ohta et al., 2013) or does not specify a concrete experimental stage (Pyysalo al., 2012). Therefore, an entity and event scheme that do not align with the dataset scope make it difficult to identify specific event triggers and their associated arguments, which are prevalent in animal experiments.
We evaluate the recent NER and EE ap- proaches on dataset dealing with both discontimuous entities and nested events.
Related Work
2.1 Complex Information Extraction
Traditional NER and EE formulate their tasks as a sequence labeling problem (Lample et al., 2016; Liu al., 2018; Lin al., 2019; Cao et al.. 2019), assigning a tag to each token from pre-defined tagging scheme (e.g.. BIO tag- ging scheme). When faced with challenging syntactic scenarios such nested or discontin- uous text structures, the tagging scheme lacks the flexibility address such complexities ade- quately. Consequently, alternative approaches have been explored for each task.
Therefore, we introduce AniEE, named en- tity recognition and event extraction dataset focused animal experiments. We establish a new entity and event scheme designed for the animal experiment stage collaboration with domain experts. Our dataset annotated by biomedical professionals and applied to the de-
latest competitive NER and EE models. scribed in Table I, the novelty of our dataset lies two aspects: addressing both 1) discon-
To handle sophisticated NER sub-tasks. 1) span- based methods (Luan et al., 2019; Wadden et al., 2019; Yu et al., 2020; Yamada et al., 2020), 2) hypergraph- based methods (Lu and Roth, 2015; Muis and Lu, 2016; Katiyar and Cardie, 2018; Wang and Lu, 2018), and 3) se- quence generation based methods (Straková et al., 2019; Yan et al., 2021: I; Fei et al., 2021)
PIn Table 1, note that the number of events, relations, and entities are the sum of train and valid datasets in the case of GE11. GG, and PC because the test datasets of them are not available publicly. Also, if the number of trigger types is not equal to the number of event types in the dataset, we count the trigger type as the event type.

---

Entity type
Frequency
Ratio (%)
Definition
Include both inducers which promote a certain action, as well as inhibitors which suppress certain activity
SampleName
4,566
20.7
SampleType
114
0.5
Refer to the nature of the sample, including extract. oil, and powder
Dosage
497
2.2
Amount of sample administered to animals
Duration
186
0.8
Total period of sample administration to animals, excluding the animal handling period
DusageFrequencc
52
0.2
Interval of sample administration
AnimalSubject
1,199
5.4
Animal species
AnimalStrain
233
1.1
Subtypes or genetic variants of animal specics
AnimalSex
102
0.5
Sex of animal species
Anatomy
3,514
15.9
Body components. such as organs and tissues
BolecurarBiomareer
3,699
16.7
Quantitative or qualitative measurement indicators of cellular-level biological process
Response
6,323
28.6
Physiological changes or responses associated with MolecularBommarker
DiseaseName
1,620
7.3
Target discase investigated in a study
Total
22,105
100.0
Table 2: Definition and frequency of entity types. Ratios presented rounded the second decimal place.
have been proposed. Luan et al. (2019) enumer- ate all possible spans and recognize overlapped entities by span-level classification. Lu and Roth (2015) represent each sentence as a hyper- graph with nodes indicating entity types and boundaries for overlapped NER. Muis and Lu (2016) extend this effort to apply discontinuous NER. Straková et al. (2019) use the sequence- to-sequence model to output a label sequence with multiple labels for single token for over- lapped NER.
As a biomedical domain dataset for EE task, GE11 (Kim al., 201 is focused on events related to the transcription factors in human blood cells. CG (Pyysalo et al., 2013) is con- centrated on the biological processes involved in the development of cancer. MLEE (Pyysalo et al., 2012) extend throughout logical organization from the molecular to the organism level, for both NER and EE tasks. In short, existing EE datasets consist of literature that is either restricted to cell experiments generalized to cover all experimental stages. Hence, introduce novel dataset, AniEE, which aims to extract key animal experimental information.
levels bio-
Due to the syntactic complexity, research has focused on either nested discontinuous enti- ties. Recently, unified NER models (Li et al.. 2020b, 2021, 2022) have been proposed to jointly extract these entity types.
3 Dataset Construction
On the other hand, to address advanced EE sub- tasks, sipeline-based methods (Li et al., 2020a; Sheng et al., 2021) have been introduced which sequentially extract event triggers and argu- ments. However, due to the inherent problem of error propagation in such sequential processes, OneEE (Cao et al., 2022) propose the one-step framework to simultaneously predict the rela- tionship between triggers and arguments.
3.1 Dataset Collection
The raw corpus collected from PubMed, which a widely-used online database contain- ing vast collection of biomedical literature. We collaborated with two senior domain ex- perts to define the search terms, with the aim to crawl diverse set of animal experimental liter- ature. Medical subject headings (MesH), which serve as a hicrarchical search index for the PubMed database, were used to determine the search term. First, we included ([MesHTrrms] AnimalDiseaseModel OR [MesHTerms] Ani- mals) in the search term to retrieve literature that performed animal experiments. Then, to obtain a literature collection on wide range of topics, we set the search terms of ([MesH- Subheading] Physiopathology OR [MesHSub- headingl Chemistry) include the literature which falls under physiopathology or chemistry as broad categories, while we excluded review articles. Thus, the final search terms consisted of ([MesHTerms] AnimalDiseaseMModel OR
2.2 Clinical and Biomedical Datasets
Among the various domains where informa- tion extraction research is conducted, the clini- cal and biomedical domains are highly active fields based on numerous training datasets. In the clinical domain. ShARe13 (Danielle et al., 2013) and CADEC (Sarvnaz et al., 2015) are clinical report datasets including discontinuous drug event and disease entities. In the biomed- ical domain, several datasets (Pyysalo et al., 2009; Ohta et al., 2010) are derived from GF- NIA corpus (Ohta et al., 2002; Kim et al., 2003), including JNLPBA (Kim et al., 2004).

---

Event Type
Argument Role
Definition
Freq.
Ratio (%)
Administration of a specific sample
Object, Subject, Site,
SamplcAdminissrationn
the experimental subject, including injection, oral administration, and topical application
1,364
12.9
Amount, Schedule
Stimulation of biological process system in animals that increases the activity, expression, or response particular target.
PositiveRegulation
Object, Cause, Site
5,811
55.1
Suppression or inhibition of a biological process system in animals, resulting in reduced activity, 3,371
NegativeRegulation
Object, Cause, Site
32.0
expression, response specific target.
Table 3: Definition of event types and their corresponding argument roles. Ratios are presented rounded to the second decimal place.
[MesHTermm] Animals) AND ([MesHSubhead- ing] Physiopathology OR [MesHSubheadding Chemistry) AND (Not Review). We collected the titles and abstracts the literature from the top search results. We then removed the arti- cles without the direct involvement of animal experiments, resulting total of articles.
gument roles' 3, including our two novel event argument roles express the relations between the event trigger and arguments linked to exper- imental conditions (i.c., Dosage, Duration).
Positive and Negative Regulation We an- notate PositiveRegulation and NegativeRegula- tion on the text spans that induce or interfere with biological processes. PositiveRegulation, such as the increase in cancer cells, does not necessarily mean ultimately positive effect, and the same concept applies to NegativeRegu- lation. unique characteristic of our dataset that it contains nested events. Figure (c) de- scribes an example of nested events where the event PositiveRegulation ("activation") is the Object argument of another event NegativeReg- ulation ("reversed").
3.2 Entity and Event Type Scheme
AniEE contains 12 entity types and 3 event types. Table and Table describe the en- tity types and event types in dataset, re- spectively. The event arguments are detailed in Appendix Table 13.
Entity Types Existing benchmark datasets (Ohta et al., 2002; Pyysalo et al., 2012) have typically focused on anatomical information. encompassing various entity types, especially at the cellular level. Given our primary focus on animal experiments,
3.3
Annotation Procedure & Strategy
3.3.1 Annotation Procedure
consolidated these ious types into a single unified category named Anatomy. On the other hand, the animal types are more fine-grate fine-grained: AninalSubject eral identifier (e.g.. "mice") and AnimalStrain a specific identifier (c.g.. "C57BL/6J"), Also, we introduce numerical entities, which are key attributes of the animal experiment design, such Dosage, Duration, and DosageFrequency.
To maintain annotation consistency between annotators, our annotation process consisted of two stages; pilot annotation and expert an- notation. All annotators were instructed with detailed annotation guidelines" 4.
gen-
Pilot Annotation Prior to performing the actual annotation process, a pilot annotation was conducted train the annotators due their varying levels of domain knowledge. It was performed to apply our newly defined scheme 10 pieces of literature, which are extracted if they are related to the animal exper- iment stage from the MLEE corpus (Pyysalo et al., 2012), publicly available biomedical event extraction dataset. During the annotation. developed the annotation guidelines, and the annotators became familiar with the task
Sample Administration We annotate SampleAdiiinistration on the text spans rep- resenting a specific approach administering a sample. It can be verb, such as a typical event trigger, but it can also be non-noun part of speech, including specialized abbrevi- ation (c.g., 'i.c.v") """" or an adverb (c.g., "orally"), When literature explicitly describes method of administering a sample, we prioritize anno- lating specific expressions over generic verbs (c.g.. "administered" and "treated"), as illus- trated in Figure (a). The event has five ar-
'Object refers Theme previous work. "The annotation guidelines and examples are released at http://gihhh.ccm//mmmywww

---

and tagtog a web-based annotation tool with a user-friendly interface.
Statistics
Train
Valid
Test
Number
250
50
50
Avg.Sent
11.6
11.6
10.8
Expert Annotation Six domain experts were recruited as annotators who are masters or Ph.D. candidates in the biomedical field. Two annotators independently annotated each piece of the literature.
Avg. Token
455.5
474
429.3
Avg. Entities/Doe
83.5
86.4
76,1
Avg. Evgnttvet//Dc
30.4
31.3
28.5
Table 4: Statistics of AniEE. The average length of sen- tences per document (Avg.Sent), the average number of tokens per sentence (Avg. Token), the average number of entities (Avg. Entities/D/)))) and events per document (Avg. (ang.Evets/Doo) are reported.
3.3.2 Annotation for Complex Scenarios
Traditional annotation methods focus on contin- uous text spans, making it difficult to annotate certain entities and events due to the complex semantic nature of the animal experiment lit- erature. To address this issue, developed specialized annotation strategies to handle two specific scenarios: 1) the occurrences of dis- continuous entities and 2) the instances where a solitary event trigger indicates several events.
Tasks
R
IAA
NER
0.943
0.947
0.944
0.973
EE
0.687
0.656
0.662
0.586
Table Annotation quality related scores of 10 ran- domly sampled documents. Precision (P), recall (R), and FI score are macro averaged, and inter-annotator agreement (IAA) is Krippendorff's
Discontinous Entity As shown in the Dosage case Figure I (a), for numerical en- tity types, substantial amount of literature list several numbers and then mention their unit only once at the end, resulting in discon- tinuous entities. To minimize the annotators' errors, these entities were subdivided into nu- meric (e.g.. "8") and unit (e.g.. "mg/kg") enti- ties during the annotation process with spe- cial relation type only for mapping the num- ber and unit. We then post-process merge them into single entity (e.g.. "8 mg/kg"). For Dosage, because the daily dosage units can be described, we temporarily segmented the unit entity into two unit sub-entities: Dosage- Unit (c.g., "mg/kg") and Unit (c.g.. "/day"), which were later combined into one (e.g.. "8 mg/kg /day").
between "losartan" and "4.5 mg/kg") and in- struet the annotators. After the annotation pro- cess, post-processing is conducted to produce two distinct events for each sample.
3.4 Dataset Statistics and Characteristics
The AniEE corpus contains a total of 350 ani- mal experimental articles. We split dataset into 4:1:1 for the training, validation, and test sets. Table 4 presents their statistics.
Named Entity Types Table shows the long-tail distribution of the frequency of the entity types in dataset similar to other datasets (Pyysalo et al., 2012; Luo et al., 2022), including Response (28.6%) SampleName (20.7%), MolecularBionarker (16.7%), and Anatomy (15.9%). Our dataset contains 644 discontinuous entities.
Multiple Events Single Event Trig- ger Given an example of "ginsenoside Rbl (35 mg/kg) and losartan (4.5 mg/kg) i.c.v", an event trigger ("i.c.v") has two samples and cor- responding dosages for each sample. Since an event trigger corresponds to only single instance of an Object argument, the example represents one event for each sample, with a total of events. However, prior research has found that these scenarios are challenging extract due to the inherent semantic intricacy, which has consequently been acknowledged as a limitation (Friedrich et al., 2020). In order to accurately extract these events, we introduce supplementary relation type to link each dosage associated with each respective sample (e.g..
Event Types Table presents frequency of event types in our dataset, such as Sam- pleAdministration (12.9%), PositiveRegula- tion (55.1%), 1%), NegativeRegulation (32.0%). In addition, for PositiveRegulation and Neg- ativeRegulation event types, since an event be possible argument for another event, AniEE has 709 (6.7%) nested events.
"BioBERT from biobert-vi.1
co/
dmis co/dmis lab/
www.. www.ttagtog.com

---

IAA To evaluate the annotation quality, we report standard measure of the inter-annotator agreement (IAA) of Krippendorff's 's a (Krippen- dorff, 2011). l), as well precision, recall, and Fl measure. We estimate each score for two tasks: tasks on ten randomly-samplee documents. Each document is independently annotated by two annotators: one Ph.D. can- didate and one master's student. We consider the annotation labels of the Ph.D. candidate as the ground truth and compare them to the other one's. Table shows the IAA results. Krippen- dorff's a are 0.973, and 0,586 for the NER and EE tasks, respectively. Morante et al. (2009) has reported that the IAA scores of Sasaki al. (2008) and GENIA event corpus are 0.49, and 0.56. These low scores were attributed the difficulty of the annotation due to the complex- ity of the EE dataset structure. Compared to these scores, our IAA scores favorable.
Baselines
F1
SpanNER
66.84
71.88
69.27
W2NER
72.24
69.64
70.92
NER
Table NER performance on two baselines:
W2NER (Li et al., 2022) and SpanNER (Li al., 2021). All scores are the percentage (%).
W2NER
P
R
F1
Sample
SampleName
80.67
79.11
79.88
SampleType
66.67
50.00
57.14
Dose
Dosage
77.27
77.27
77.27
Duration
57.14
70.59
63.16
DosageFrequency
60.00
25.00
35.29
Animal
AnimalSubject
89.39
93.57
91.43
AnimalStrain
61.70
90.62
73.41
4 Experiments
AnimalSex
1.0
69.23
81.82
Target
4.1 Settings
Anatomy
71.52
74.27
72.87
MolecularBiomarker
77.06
69.56
73.12
To examine the effectiveness and challenges of AniEE corpus, we conduct experiments on recent superior baseline models for the NER and EE tasks.
Response
58.99
61.02
59.99
DiseaseName 76.35
78.06
77.20
Table 7: NER performance of W2NER (Li et al., 2022)
each entity type. entity types are grouped by Gray categories for readability.
NER Models We evaluate our dataset on unified NER models, which allow extract discontinuous and flat entities. W2NER (L.i al.. 2022) is an unified NER framework by modeling 2D grid matrix of word pairs. Span- NER (Li et al., 2021) proposes a span-bar based model which extracts entity fragments and the relations between fragments to jointly recog- nize both discontinuous and overlapped enti- ties.
better understanding of contextual representa- tions. Since several documents longer than the maximum sequence length BioBERT, we split the document into smaller chunks.
Evaluation Metric Following previous work in the NER and EE tasks (Sheng et al., 2021: Cao et al., 2022), we report Precision (P), Recall (R), and F measure. We measure four evaluation metrics for the EE task. 1) Trig- ger Identification: an event trigger is correctly identified if the predicted trigger matches with ground truth; 2) Trigger Classification: an event trigger correctly classified the identi- fied trigger is identical the right event type; 3) Argument Identification: an event argument correctly identified if the predicted argument aligns with ground truth; 4) Argument Clas- sification: an event argument is correctly iden- tified if the identified argument is assigned the right role. In short, we can organize iden- tification as the task finding the region trigger entity and classification predicting the class of each object.
EE Models Given the nascent nature of the nested event extraction (EE) task, only limited number of approaches have heen introduced in recent years. In our study, we assess our dataset using end-to-end models, wherein both event triggers and arguments are predicted simultane- ously. The current methodologies encompass two major categories: pipeline-based multi- staged approaches and one-staged approaches. CasEE (Sheng et al., 2021) sequentially type detection task, followed by trigger and argu- ment detection task with cascading decoding strategy. OneEE (Cao et al., 2022) simultanc- ously perform event extraction task using word- word relation matrix. All models are sentence- level haselines, but we extend baseline input the document level, feeding multiple sen- tences as input, to predict the events based on

---

Trigger Identification
Trigger Classification
Argument Identification
Argument Classification
Models
P
R
F1
P
F1
R
F1
R
F1
CasEE
70.83
73.54
72.16
67.56
70.23
68.87
54.07
59.19
56.51
53.31
58.54
55.81
OneEE
68.98
34.82
46.28
67.25
33.95
45.12
59.04
29.42
39.47
60.23
26.40
36.71
Table Event extraction performance comparison of two baseline models for the four evaluation metries: trigger identification, trigger classification, argument identification, and argument classification (see Section 4.1).
Implementation Detail We use BioBERT (Lee et al., 2020) as pre- trained language model and adopt AdamW optimizer (lyya and Frank, 2019) with learning rate of 2e-5. The batch size is 8, the hidden size dp 768. We train baselines with 200 epochs.
CasEE
P
R
F1
SampleAAminiistattoon
56.56
64.49
60.26
PositiveRegulation
65.32
68.88
67.05
NegativeRegulation 76.89
74.81
75.83
Table 9: Event extraction performance of the triggers for each event type.
4.2
Experimental Results
CasEE
P
R
F1
4.2.1 Named Entity Recognition
SampleAdmiiistraation
Table shows the precision, recall, and micro- average FI scores of the NER baselines. W2NER slightly outperforms SpanNER based on score.
Object
54.68
69.09
61.04
Subject
61.54
80.00
69.57
Site
83.33
71.43
76.92
Amount
67.80
90.91
77.67
Schedule
83.33
40,00
54.05
Table presents the precision, recall, and FI scores of W2NER for each entity type. We assume that the long-tail frequency distribution and the number of unique expressions within each entity type affect the performance. Head- frequency entity types, such as SampleName and DiseaseName, where their frequencies 4,566 and 1,620, show higher performance than tail entity types, such as SampleTType and DosageFirequecyy. where their frequencies are 114 and 52, respectively. Also, the Fl scores of AnimalSubject and AnimalSex are higher than other entity types. This is because the concepts animal subjects and sex are less specific animal experiments than other entity types, making it easier to leverage the knowledge gained from pretrained language model.
Prediction of Entity Types
PositiveRegulation
Object
47.54
53.49
50.34
Cause
60.33
70.29
64.92
Site
34.29
30.77
32,43
NegativeRegulation
Object
53.47
51.25
52.33
Cause
59.87
73.39
65.94
Site
15.38
11.11
12.90
Table 10: Event extraction performance of the argu- ments for each event type.
Extraction of the Event Types Table 9 shows the precision, recall, and F1 score for each event type. This is the result when the model proceeds to trigger identification and classification jointly. The low accuracy of Sam- plcAdministtation, where the frequency ratio is 12.9%, can explained by the low fre- quency compared with other event types. How- ever, although PositiveRegulation (55.1 155.1) ap- pears more frequently than NegativeReggultiinn (32.0%), the model predicts NegativeRegula- tion more accurately than PositiveRegulation. Hence, we further analyze this observation Section 5.1.
4.2.2 Event Extraction
Table 8 shows the BE results for the four evalu- ation metrics. CasEE consistently outperforms OneEF, except for the precision scores for the argument identification and classification met- rics. Overall, CasEE has higher recall than precision across all metrics, which suggests that the model produces more false positive predictions than false negatives. On the other hand, OneEE has a large gap between precision recall across all metrics, which implies that the model generates lot of false negatives and easily misses gold triggers and entities.
Extraction of the Event Arguments Ta- 10 describes the extraction performance the event arguments for each event type. This is the result when the model predicts the argu-

---

ment identification and classification together. CasEE (Sheng et al., 2021) shows the highest Fl score on the Amount in SampleAdministre tion. This result can be explained by the fact that the mentions of Amount usually consist numeric and a specific unit component, making easier for the model to detect this consistent pattern in unstructured documents.
W2NER
P
R
F1
Sample-Amiinistratioon
60.53
56.56
58,48
PositiveRegulation
71.06
60.28
65.23
NegativeRegulation 82.64
67.34
74.21
Table 11: NER performance scores when detecting the text span of the event triggers. All scores are the per- centage (%)
addition, the scores of Site in Posi- tiveRegulation and NegativeRegulatio sig- nificantly lower compared to SampleAdminis- tration. Site in SampleAdminnstration typically refers to the location of administration and has a low variance. contrast, Site in regulation refers to where the effect occurs and therefore includes a wider range of locations than the location of administration. Therefore, Sites in PositiveRegulation and NegativeRegulation can be descrihed as more difficult to detect com- pared to SampleAdministrattion because of the variety of places where effects occur.
20
Samplekdminnistration NegativeRegulati PositiveRegulation
15
Frequency
Top-k Lemma Cluster
Figure 2: Freqeucny ratio comparison for each event
Recognition of the Event Trigger On the EE task, the model needs to detect the span of the event trigger. We set this task NER, which detects the span of the event trigger mention. As described in Table 11, W2NER (Li et al., 2022) shows progressively higher scores for NegativeRegulation, Posi- tiveRegulation, and Sample AamplleAdministratiion It similar to the event trigger classification per- formance in Table 9; the order of score is the same, Thus, we can analyze that the low NER performance of PositiveRegulation is due to the imbalanced distribution of trigger mentions.
type. Each line represents an event type. We plot the percentage of the event mention distribution (Y-axis) accounted for by the top five lemma clusters (X-axis).
shown in Figure in PositiveRegulation, the top-1 lemma cluster with the lemma "in- duced" accounts for 20% of the overall fre- queney ratio, while the second lemma clus- ter ("increased") accounts for 5.8%. This is distinguishable with other event types, such as Administttmnittttiinnonnnnn and NegativeRegula- tion, where the frequency percentage gradually decreases with each subsequent lemma cluster. Therefore, the low performance of PositiveReg- ulation in Table 9 can be explained by the im- balanced distribution of the trigger mentions.
5 Analysis
5.1
Distribution of the Event Mentions
6 Conclusion
the event trigger classification task, we expected that the model would predict Pos- itiveRegulation more accurately than Nega- tiveRegulation because the frequency of Pos- itiveRegulation is 1.7 times greater than one NegativeRegulaaton. However, the results shown in Table 9 contradict our expectations. To analyze this experiment, we collected the mentions of the event triggers for each event type and extracted the lemmas to group the mentions in a semantic manner' To calculate the frequency ratio of lemma cluster per total mention frequency, summed the frequencies of all the mentions belonging to the cluster.
In order to enhance the efficiency and accuracy of a comprehensive review of existing animal experiment literature, introduce AniEE, event extraction dataset designed specifically for the animal experimentation stage. The dis- tinctiveness of our dataset can be summarized through two key points. To the best of our knowledge, our dataset represents the first event extraction dataset focusing on animal exper- imentation. addition, our dataset encom- passes both discontinuous named entities and nested events within texts at the document level. anticipate that introducing our novel dataset, AniEE, will contribute significantly to advanc- ing document-level event extraction not only in
WWo utilize NLTK WordNet Lemmatizer Top-5 lemma cluster for each event type is described in Appendix A.2

---

the biomedical domain, but also in the natural language processing.
L. Mowery Damielle. Velupillai Sumithra, R. South Brett, M. Christensen Lee, Martínez David, Kelly Liadh, Goeuriot Lorraine, El- hadad Noémie, Pradhan Sameer, K. Savova Guergana, and W. Chapman Wendy. 2013. Task I: Share/clef ehealth evaluation lab 2013. In Conference and Labs of the Evaluation Fo- rum.
Limitations
We acknowledge that our dataset, which con- tains 350 abstracts, not huge due to labor- intensive manual annotation. However, con- sidering the number of annotated events and entities, well our experimental results, the current dataset size is sufficient to develop NER and EE models.
Hao Fei, Donghong Ji, Bobo Li, Yijiang Liu, Yafeng Ren, and Fei Li. 2021. Rethinking boundaries: Find-to-end recognition of discon- tinuous mentions with pointer networks. Pro- ceedings of the AAAI Conference Artificial Intelligence, 35(14): 1775-13377
Ethics Statement
Annemarie Friedrich, Heike Adel, Federico Tomazic, Johannes Hingerl, Renou Benteau, Anika Marusczyk, and Lukas Lange. 2020. The sofc-exp corpus and neural approaches information extraction the materials science domain. CoRR, abs/2006.03039,
As each annotator is a master's or Ph.D. stu- dent, we compensated each annotator reason- ably comparable to a lab stipend. Additionally, we adjusted the workload weekly, accounting for each annotator's schedule.
Giacomo Frisoni, Gianluca Moro, and An- tonella Carbonaro. 2021. A survey event extraction for natural language understanding: Riding the biomedical literature wave. IEEE Access, 9:160721-160755
Acknowledgement
This work was supported by grant (2162MDD)076) from Ministry of Food and Drug Safety in 2023, Institute of Information & communications Technology Planning & Evaluation (ITPP) grant funded by the
Loshchilov llya and Hutter Frank. 2019. De- coupled weight decay regularization. In Inter- national Conference on Learning Representa- fions.
Arzoo Katiyar and Claire Cardie. 2018. Nested named entity recognition revisited. Proceed- ings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computa- tional Linguisties: Human Language Technolo- gies, Volume 1 (Long Papers), pages 861-871, New Orleans, Louisiana. Association for Com- putational Linguisties.
Korea government (MSIT) (No.2019-0-00075.
Artificial Intelligence Graduate School Program (KAIST)), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2022R A2B5502001913)... Also, we would like to express our gratitude to Jiyun Lee, Heejeong Hwang, Jungyeon Yu, and Jungyeon Lee for their participation in this work.
Jin Jin-Dong Kim, Tomoko Ohta, Yuka Tateisi, and Junichi Tsujii. 2003. Genia corpus a se- mantically annotated corpus for bio-textmining. Bioinformatics, 19 Suppl 1:i180-2.
Jin-Dong Kim, Tomoko Ohta, Yoshimasa Tsu- ruoka, Yuka Tateisi, and Nigel Collier. 2004, Introduction to the bio-entity recognition task at jalpba. In Proceedings of the International Joint Workshop on Natural Language Pro- cessing in Biomedicine and Its Applications, JNLPBA '04, page 70-75, USA. Association for Computational Linguistics.
References
Hu Cao, Jingye Li, Fangfang Su, Fei Li, Hao Fei, Shengqiong Wu, Bobo Li, Liang Zhao, and Donghong Ji. 2022. OneEE: one-stage framework for fast overlapping and nested event extraction. In Proceedings of the 29th In- ternational Conference on Computational Lin- guistics, pages 995-1164, Gyeongju, Repub- lic of Korea. International Committee Com- putational Linguistics.
Jin-Dong Kim, Yue Wang. Toshihisa Takagi, and Akinori Yonezawa. 2011. Overview of Genia event task in BioNLP shared task 2011 1. In Proceedings of BioNLP Shared Task 201 I Workshop, pages 7-15, Portland, Oregon, USA. Association for Computational Linguistics.
Yixin Cao, Zikun Hu. Tat-seng Chua, Zhiyuan Liu, and Heng Ji. 2019. Low-resource name tagging learned with weakly labeled data. In Proceedings of the 2019 Conference Em- pirical Methods Natural Language Process- ing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 261-270, Hong Kong, China. Association for Computational Linguistics.
Klaus Krippendorff. 2011. Computing krippen- dorff's alpha-reliabillty.
Guillaume Lample. Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American

---

Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pages 260-270, San Diego, California. Association for Computational Linguistics.
for Computational Linguisties: Human Lan- guage Technologies, Volume 1 (Long and Short Papers), pages 3036-3046, Minneapolis, Min- nesota, Association for Computational Linguis- tics.
Jinhyuk Lee. Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: pre- trained biomedical language representation model for biomedical text mining. Bioinfor- matics, 36(4); 1234-1240.
Ling Luo, Po-Ting Lai. Chih-Hsuaa Hasaa Wei, Cecilia N Arighi, and Zhiyong Lu. 2022.
BioRED: rich biomedical relation extraction dataset. Briefings in Bioinformatics, 23(5). Bbac282.
Fayuan Li, Weihua Peng, Yuguang Chen, Quan Wang, Lu Pan, Yajuan Lyu, and Yong Zhu. 2020a. Event extraction as multi-turn ques- tion answering. In Findings of the Association for Computational Linguistics: EMNLP 2020. pages 829-838, Online. Association for Com- putational Linguisties.
Roser Morante, Vincent Van Asch, and Wal- ter Daclemans. 2009. A memory-based learn- ing approach to event extraction in biomedi- cal texts. In Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task, pages 59-67, Boulder, Colorado. Association for Computational Linguistics.
Fei Li, ZhiChao Lin, Meishan Zhang, and Donghong 2021. span-based model for joint overlapped and discontinuous named en- tity recognition. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 4814 4828, Online. Association for Computational Linguistics.
Aldrian Ohaja Muis and Wei Lu, 2016. Learn- ing to recognize discontiguous entities. In Pro- ceedings of the 2016 Conference on Empiri- cal Methods in Natural Language Processing, pages 75-84, Austin, Texas, Association for Computational Linguistics.
Tomoko Ohta, Sampo Pyysalo, Makoto Miwa, Jin-Dong Kim, and Jun'ichi Tsujii. 2010. Event extraction for post-translatinn cations. In Proceedings of the 2010 Workshop Biomedical Natural Language Processing, pages 19-27, Uppsala, Sweden. Association for Computational Linguistics.
modifi-
Jingye Li, Hao Fei, Jiang Liu, Shenggiong Wu, Meishan Zhang, Chong Teng, Donghong Ji, and Fei Li. 2022. Unified named entity recogni- tion as word-word relation classification. Pro- ceedings of the AAAI Conference on Artificial Intelligence 36(10):10965-10977
Tomoko Ohta, Sampo Pyysalo, Rafal Rak, An- drew Rowley, Hong- Woo Chun, Sang-Jae Jung, Sung-Pil Choi Sophia Ananiadou, and Jun' ichi Tsuji. 2013. Overview of the pathway cura- tion (PC) task of BioNLP shared task 2013. In Proceedings of the BioNLP Shared Task 2013 Workshop. pages 67-75, Sofia, Bulgaria. Asso- ciation for Computational Linguistics.
Xiaoya Li. Jingrong Feng. Yuxian Meng. Qinghong Han, Fei Wu, and Jiwei Li. 2020b. A unified MRC framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Asvociation for Computational Linguistics. pages 9849-5555, Online. Associ- ation for Computational Linguistics.
Tomoko Ohta, Yuka Tateisi, and Jin Dong Kim. 2002. The genia corpus: An annotated research abstract corpus in molecular biology domain. In Proceedings of the Second International Conference on Human Language Technology Research, HLT '02, page 82-86, San Francisco, CA. USA. Morgan Kaufmann Publishers Inc. Sampo Pyysalo, Tomoko Ohta, and Sophia Ananiadou. 2013. Overview of the cancer ge- netics (CG) task of BioNLP shared task 2013. In Proceedings of the BioNLP Shared Task 2013 Workshop. pages 58 66, Sofia, Bulgaria. Association for Computational Linguistics.
Ying Lin, Liyuan Liu, Heng Ji, Dong Yu, and Jiawei Han. 2019. Reliability-wware dynamic feature composition for name tagging. In Pro- ceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 165-174, Florence, Italy, Association for Com- putational Linguistics.
Liyuan Liu, Jingbo Shang, Xiang Ren, Frank F. Xu, Huan Gui, Jian Peng, and Ji- awei Han. 2018 Empower sequence label- ing with task-aware neural language model. AAAI' 8/AAAAA 18/EAAI 18. AAAI Press.
Sampo Pyysalo, Tomoko Ohta, Jin-Dong Kim, and Jun ichi Tsujii. 2009. Static relations: a piece in the biomedical information extraction puzzle. In Proceedings of the BioNIP 2009 Workshop, pages 1-9, Boulder, Colorado. As- sociation for Computational Linguistics.
Wei Lu and Dan Roth. 2015. Joint mention ex- traction and classification with mention hyper- graphs. In Conference on Empirical Methods in Natural Language Processing.
Yi Luan, Dave Wadden, Lubeng He, Amy Shah, Mari Ostendorf, and Hannaneh Ha- jishirzi. 2019. A general framework for infor- mation extraction using dynamic span graphs. Proceedings of the 2019 Conference of the North American Chapter of the Association
Sampo Pyysalo. Tomoko Ohta, Makoto Miwa, Han-Cheol Cho, Jun'ichi Tsujii, and Sophia Ananiadou. 2012. Event extraction across mul- tiple levels of biological organization. Bioin- formatics, 28( (81188557---58.1

---

Karimi Sarvnaz, Metke-Jimenez Alejandro, Kemp Madonna, and Wang Chen. 2013. Cadec: A corpus of adverse drug event annotations. Journal of Biomedical Informatics, 55:73-81.
parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6470-6476, Online. Associ- ation for Computational Linguisties.
Yutaka Sasaki, Paul Thompson, Philip Cotter, John McNaught, and Sophia Ananiadou. 2008. Event frame extraction based gene regula- corpus. In Proceedings of the 22nd Inter- national Conference on Computational Linguis- fics (Coling 2008), pages 761-768, Manchester, UK. Coling 2008 Organizing Committee.
Jiawei Sheng, Shu Guo, Bowen Yu, Qian Li, Yiming Hei, Lihong Wang. Tingwen Liu, and Hongbo Xu. 2021. CasEE: A joint learning framework with cascade decoding for overlap- ping event extraction. In Findings of the As- sociation for Computational Linguistics: ACL- IJCNPP 2021, pages 164-174, Online. Associ- ation for Computational Linguistics.
Jana Straková, Milan Straka, and Jan Hajic. 2019. Neural architectures for nested NER through linearization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5326-5331, Florence, Italy. Association for Computational Linguistics.
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference Natural Language Pro- cessing (EMNLP-IJCNLP) pages 5784-5789, Hong Kong, China. Association for Computa- tional Linguistics.
Bailin Wang and Wei Lu. 2018. Neural segmen- hypergraphs for overlapping mention recog- nition. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 214, Brussels, Belgium. Association for Computational Linguistics.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representa- tions with entity- aware self- attentotnii In Pro- ceedings of the 2020 Conference Empiri- cal Methods in Natural Language Processing (EMNLP), pages 6442 6442-6454, Online. Associa- tion for Computational Linguistics.
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021. A uni- fied generative framework for various NER sub- tasks. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Lin- guistics and the I Ith International Joint Con- ference on Natural Language Processing (Vol- une 1: Long Papers), pages 5808-3 0888, On- line. Association for Computational Linguis- ties.
Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020. Named entity recognition as dependency

---

A Dataset Construction
model prediction. As shown in Table 14, the model predicts the discontinuous entities for the first three examples accurately. However, the model fails to detect the duration entity of the fourth example (i.e., "five days") since it predicts "five consecutive days" as a flat entity. This is because we define Duration number and unit the annotation strategy.
A.1
Dataset Statistic
Count Role
Corpus
Dataset
Event
Entity
Document
Sentence
ShARe13
11,161
298
18,767
CADEC
6,318
1250
7,597
GE11
10,270
14,840
16,976
1,514
14,962
MILER
5,554
7,588
8,291
262
2,607
B.2 Nested Event
AniEE
(Ours)
9,140
14,151
21,973
398
4,581
Similar to discontinuous entities, we color the event triggers given data sample and tag whether CasEE (Sheng et al., 2021) correctly predicts them. Also, we extract the relations between event triggers when one of the triggers is an argument of the other. The rela- tions between triggers are described by a triplet, where the first is the event trigger of the cur- rent example, the second is the argument of the first, and the third the role of the second argument within the event of the first trigger. Table 15 shows two examples of nested events. The model shows incorrect prediction in the first example and correct prediction in the sec- ond example, but the argument roles are the same as for Object.
Table 12: Dataset statistic comparison.
Table 12 shows the dataset comparison with existing benchmarks with corpus information. Also, we describe the detinition and frequency of event arguments in Table 13.
A.2 Top-5 Lemma Cluster of Event Type
As mentioned in Section 5.1, we collect the mentions event triggers and group them by lemma based NLTK WordNet Lemmatizer. For instance, "activated by" is trigger men- tion, cluster "activated" includes it. For each event type, the top-5 frequent lemma clus- ters with their total frequency and ratio (%) are follows:
PositiveRegulation: induced (752, 2%),,), increased 5.8%), increase (89, 2.4%), associated (72, 1.9%). improved (70, 1.9%)
NegaaiveRegulation: reduced (178, 8.1%), decreased (148, 6.7%), anti (103,
4.7%), inhibitor (78, 3.5%), attenuated (69, 3.1%)
SampleAdministration administration (112, 12.0%), treated (98, 10.5%), injec- tion (84, 9.0%), treatment (75, 8.0%), ad- ministered (57, 6.1%)
B Case Study
To further investigate I) discontinuous entities and 2) nested events in our dataset, visualize six samples of our dataset.
B.1 Discontinuous Entity
We extract data samples that contain discon- tinuous entities, color the named entities with each color of their entity type, and lag whether the prediction of this entity is success or fail. W2NER (Li et al., 2022) is utilized to extract

---

Argument Role
Frequency
Ratio (%)
Definition
Samplekkdministratinn
Object
1,366
41.90
material which is used for an event
Subject
690
21.17
An animal experimental subject of an event
Site
240
7.36
Body region where
occurrs
Amount
729
22.36
Quantity measurement of sample
Schedule
215
6.59
A time frame of an event
Total
3,240
100.0
PositiveRegulatie
Object
5,575
Physiological parameters affected by an event
Cause
2,734
30.82
Attribute that influences the modifications of the target factor Physiological region where Object argument is observed
Site
562
6.3
Total
8,871
100.0
Negative
Object
3,267
60.20
Physiological parameters affected by an event "SampleAAminiisrattoo"
Cause
1,888
34.79
Attribute that influences the modifications of the target factor
Site
272
5.0
Physiological region where Object argument is observed
Total
5,427
100.0
Table 13: Definition and frequency of arguments for each event type. We sum up the frequency of arguments for each event type and get the ratio in the event type. Ratios are presented rounded to the second decimal place.
Discontinuous Entity
Example
Prediction
ppm
Mice (Swiss Webster) exposed toluene (0, 2000 4000 ppm, 30 min day)
Success
2000 ppm
mg/kg/day
10 mg/kg/day
Female Sprague-Davee Dawley rats treated orally with ascending methadone dosage schedule (5, 10, 15, 20, 25 and 30 mg/kg/day)
15 mg/kg/day
Success
20 mg/kg/day
25 mg/kg/day
3xTg mice were fod control or Cr-supplemented (3% Cr (w/w)) diet for 8-9 weeks The rats were treated with ScNPs by intraperitoncal injection (0.52009mg ScNP/kg) for five consecutive days
weeks
Success
five days
Fail
Types: Dosage, Duration
Table 14: Event examples of AniEE that contain discontinuous entities. Note that the second column only describes the discontinuous entities, excluding the flat ones, such as "30 min", "9 weeks", and "0.52009mg ScNP/k". The entity types are represented as colors for readability.
Example
Nested Events
Prediction
the mechanical and metabolic disruption of cartilage was prevented in vivo.
(prevented, disruption, Object)
Fail
Protective and anti-inflammatory effect of selenium nano-particles against bleomycin-indeee induced pulmonary injury in male rats
(against, protective, Object)
Success
Types: PositiveRegulation, NegativeRegguation
Table 15: Nested event examples. The event types represented as colors for readability. The relation between the event trigger and its argument is described in relation triplet: (Trigger mention, Argument mention, Relation).


Reviews:
[{"review_id": "OmvfeypI32", "reviewer": "Reviewer_zNXM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the thoroughness invested in your review. We extend our gratitude for all constructive comments. To enhance readability, we have noted the comments in block quotes along with the corresponding responses below:\n\n> The paper lacks details on the annotation process and guidelines.\n\nThe annotation process is described in **Section 3.3. Annotation Procedure and Strategy**, and the annotation guideline has been made accessible through the provided link.\nhttps://docs.google.com/document/d/1BBl4mP9fJzYIRt5zOkEMLhM7__PvfjmDem33Tc1HHoU/edit?usp=sharing\n\n> The authors use NER and EE systems to \"evaluate\" the dataset, but this analysis does not actually show the quality or validity of the dataset.\n\nInter-annotator agreement (IAA) has conventionally been the standard to indicate the dataset quality ([1], [2]). We also measured the quality of our dataset with IAA, which is described in **Section 3.4 Dataset Statistics and Characteristics-IAA** of the paper.\n\n[1] Gavin et al., 2023, Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement\n\n[2] Jean Carletta, 1996, Assessing agreement on classification tasks: the kappa statistic.\n\n> The section on related work should focus more on event extraction datasets and tasks instead of including other works that do not directly align with the dataset and tasks described such as SemEval 2017 Task 10 and the ShARe13 and CADEC datasets.\n\nWe acknowledge the observation that the content of our related work section might not be directly aligned with the event extraction datasets. We appreciate your feedback in this regard. In light of this, we intend to refine the related work section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work describes a new annotated corpus on the specific topic of animal experiments.", "reasons_to_reject": "The paper lacks details on the annotation process and guidelines.\nThe authors use NER and EE systems to \"evaluate\" the dataset, but this analysis does not actually show the quality or validity of the dataset.\nThe section on related work should focus more on event extraction datasets and tasks instead of including other works that do not directly align with the dataset and tasks described such as SemEval 2017 Task 10 and the ShARe13 and CADEC datasets.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper describes a new corpus with annotated entities and events related to animal experiments. The work defined a specific set of entity types and event types according to the requirements in this topic. The paper does not give details of the annotation guidelines and focuses on dataset statistics and on results of applying selected NER and event extraction systems on this corpus.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yiLqlVzrCf", "reviewer": "Reviewer_tSdi", "summary": "", "questions": "1). Lines 204-208: “...encompassing various entity types, especially at the cellular level. Given our primary focus on animal experiments, we consolidate these varied types into a single unified category named Anatomy.” This looks to me as if you are moving from fine-grained types to coarse-grained types. Have you looked into the impact of this on the performance of models? Don’t you think that it’s right for the models to learn fine-grained types, as these can sometimes actually affect the type of event being mentioned in text?\n\n2). Lines 338 - 340: “We consider the annotation labels of the Ph.D. candidate as the ground truth and compare them to the other one’s.” This seems very biassed. Were there further assessments of one’s knowledge, experience, etc. taken into account in addition to just being a Ph.D. or Masters student?\n\n3). Lines 381 - 383: “...we split the document into smaller chunks.” How did you decide on the size of the chunks? In a context where the context needed spans multiple sentences and these sentences fall into different chunks, how do you handle this? Eventually, how you perform chunking has a direct impact on the quality of the representations that the model learns, which affects the performance.\n\n4). In your evaluation as described in lines 384–400, when identifying arguments, do you consider an argument correctly identified if both its span boundary and role are correct?\n\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the thoroughness invested in your review. We extend our gratitude for all constructive comments. To enhance readability, we have noted the comments in block quotes along with the corresponding responses below:\n\n> 1). Lines 204-208: “...encompassing various entity types, especially at the cellular level. Given our primary focus on animal experiments, we consolidate these varied types into a single unified category named Anatomy.” This looks to me as if you are moving from fine-grained types to coarse-grained types. Have you looked into the impact of this on the performance of models? Don’t you think that it’s right for the models to learn fine-grained types, as these can sometimes actually affect the type of event being mentioned in text?\n\nWe appreciate the discerning observation made by the reviewer regarding the transition from fine-grained to coarse-grained entity types. Indeed, **this transition has been made to align with our domain focus on animal experiments**. This schema refinement was strategically designed in collaboration with experts. In discussions with experts, it was determined that the unified “Anatomy” category would be sufficiently helpful for animal experiments, rather than the cellular experiments that have been the focus of previous studies.\n\n> 2). Lines 338 - 340: “We consider the annotation labels of the Ph.D. candidate as the ground truth and compare them to the other one’s.” This seems very biassed. Were there further assessments of one’s knowledge, experience, etc. taken into account in addition to just being a Ph.D. or Masters student?\n\nWe acknowledge the misleading statement. We reported precision, recall, and f1 score to **identify annotators' agreement between Ph.D. and Masters student**, rather than assigning the Ph.D students' annotations as the ground truth solely based on degree. We will revise the statement in the camera-ready.\n\n> 3). Lines 381 - 383: “...we split the document into smaller chunks.” How did you decide on the size of the chunks? In a context where the context needed spans multiple sentences and these sentences fall into different chunks, how do you handle this? Eventually, how you perform chunking has a direct impact on the quality of the representations that the model learns, which affects the performance.\n\nWe acknowledge the potential ambiguity of our wording, which is split into smaller chunks rather than **truncation**. To provide a further clarity, the documents were truncated when they exceeded the predefined maximum input length of the model. In line with the reviewer’s comment,  we endeavored to minimize truncation for sufficiently contextualized inputs to facilitate event extraction. Notably, special consideration was granted to avoid fragmenting event trigger and their corresponding arguments of each event. We will add a detailed truncation process in the camera-ready.\n\n> 4). In your evaluation as described in lines 384–400, when identifying arguments, do you consider an argument correctly identified if both its span boundary and role are correct?\n\nYes, the argument was considered correctly identified if both its text span boundary and role class were predicted correctly.\n\nWe will revise all the typos and presentation improvements in the camera-ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1). They contribute to the community an event extraction dataset concentrated on the animal experiment stage. They provide a detailed description of how the dataset was annotated. They further provide descriptions of the dataset and various statistics on its contents.\n\n2). They run existing work in the field on the dataset to establish benchmarks. \n", "reasons_to_reject": "See the questions below.", "questions_to_authors": "1). Lines 204-208: “...encompassing various entity types, especially at the cellular level. Given our primary focus on animal experiments, we consolidate these varied types into a single unified category named Anatomy.” This looks to me as if you are moving from fine-grained types to coarse-grained types. Have you looked into the impact of this on the performance of models? Don’t you think that it’s right for the models to learn fine-grained types, as these can sometimes actually affect the type of event being mentioned in text?\n\n2). Lines 338 - 340: “We consider the annotation labels of the Ph.D. candidate as the ground truth and compare them to the other one’s.” This seems very biassed. Were there further assessments of one’s knowledge, experience, etc. taken into account in addition to just being a Ph.D. or Masters student?\n\n3). Lines 381 - 383: “...we split the document into smaller chunks.” How did you decide on the size of the chunks? In a context where the context needed spans multiple sentences and these sentences fall into different chunks, how do you handle this? Eventually, how you perform chunking has a direct impact on the quality of the representations that the model learns, which affects the performance.\n\n4). In your evaluation as described in lines 384–400, when identifying arguments, do you consider an argument correctly identified if both its span boundary and role are correct?\n\n", "paper_topic_and_main_contributions": "This work contributes a dataset for Event Extraction (EE) in the biomedical domain. EE is widely used to extract complex structures representing biological events from the literature. Complementing previous pieces of work, the proposed event extraction dataset concentrates on the animal experiment stage. The work establishes a novel animal experiment customised entity and event scheme in collaboration with domain experts to create an expert-annotated high-quality dataset containing discontinuous entities and nested events and evaluate recent outstanding named entity recognition and EE models on the dataset to establish benchmarks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1). Figure 1 should be large enough so that each sentence fills exactly one line for easy illustration, as in (c). In (a) and (b), you have sentences spanning two lines with arrows from one line to the other. This is a confusing representation.\n\n2). Lines 314-315: Please provide a clear explanation to the link between Table 4 and what is in the footnote –BioBert\n\n3). Lines 021 - 022: \"...evaluate our dataset on the recent outstanding NER and EE models.\" or \".. evaluate the recent outstanding NER and EE models on our dataset.\"?"}}, {"review_id": "5bAMOc60WO", "reviewer": "Reviewer_7ePo", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the thoroughness invested in your review. We extend our gratitude for all constructive comments. To enhance readability, we have noted the comments in block quotes along with the corresponding responses below:\n\n> There are a few points where the paper appears to be a bit weaker, although they may not be major weaknesses. One concern is that inter-annotator agreement on the event annotations (0.586) is much lower than for the entities (0.973). The authors cite IAA rates for another related dataset to show that theirs are higher, as justification that theirs are sufficient. But they don’t say anything further about why there was so much more disagreement about the event annotations. Does this suggest an upper bound on the potential accuracy or F1 scores that a model trained on this dataset could achieve for event detection and/or classification? It is possible that events are much more difficult for humans to agree on. But the relative simplicity of their event types gives the impression that it might have been possible to do another round of aligning between the two annotators on the exact definitions of events to be annotated. A bit more discussion of why it wasn’t feasible to get higher agreement might help build confidence in the usefulness of that part of the dataset.\n\nWe appreciate the reviewer's observation regarding the discernible disparity in inter-annotator agreement (IAA) between entity and event annotations. Nonetheless, considering that IAA is strictly a measure of whether the annotation indexes align exactly. it has the potential for enhancement when accounting for instances of partial agreement between annotators.\n\nOn the other hands, annotations for event extraction (EE) task is more difficult than that for named entity recognition (NER) task for the following reasons:\n\n* **Complicated text structure** : It needs to consider inter-sentential relationship since relevant information of an event often spans multiple sentences.\n* **Interactions between event elements** : While NER often deals with tagging of individual entities, EE requires tagging based on semantic interpretations among event trigger and arguments such as Cause and Object.\n\nTherefore, we've struggled our best, but there's still some improvements. Before we release the dataset, we will make one more round of enhancements by post-processing.\n\n> The model experiments at the end of the paper are also not described in much detail. In this case, that is probably fine, since the authors are reproducing existing models from past research, and the focus of their paper is on the new dataset rather than any model innovations. They show results for two models each for the NER and EE tasks, respectively. There isn’t a lot to conclude from those findings, though, other than that the models do seem to be learning.\n\nIn accordance with the reviewer's comment, we will add implementation details to the appendix in the camera-ready. We adopted their best hyper-parameters of the NER and EE models reported in the original paper. We trained the models with one NVIDIA RTX 3090 GPU.\n\n> I’m not entirely sure how to assess whether these results are impressive or not, because no comparative results from similar research are shown. The authors are the first to use their own dataset. But since their goal is to demonstrate the value of their new dataset, it actually might be helpful to show how the same types of models performed on a previous dataset. If the authors get at least similar scores when the models are trained on their own dataset, that could indicate that their dataset is as consistently labeled and useful for training NER and EE models as other datasets have been.\n\nThe objective behind evaluating recent NER and EE models on our dataset is two-fold: firstly, to facilitate our dataset, thereby alleviating the burden on researchers and secondly, to address the potential interpretational challenges encountered by users who are introduced to our dataset for the first time. To train the models, we used only publicly verified open-source code and reproduced experiments on the datasets reported in previous studies to ensure that there are no implementation issues with the models. Then, we evaluated our dataset against the latest NER and EE models to provide baseline scores.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The new dataset seems like a valuable contribution to an area of biomedical IE where there seems to be demand and a potential positive impact on other research. The authors provide useful examples of report excerpts about animal experiments, showing very technical language that might be difficult to parse and extract relevant details from, without labeled data of the same format.\n\nAt the same time, the components of these events are reasonably straightforward, with the authors defining just three event types — Sample Administration (a dosage of something was administered to some animal subject), Positive Regulation (the sample made something else increase) and Negative Regulation (the sample made something else decrease). All of those other “somethings” are extracted entities with defined roles in the event.\n\nThis suggests that information about animal experiments can be organized into consistent patterns, that allow for capturing many relationships between a wide variety of treatments and outcomes, in a way that could be easily quantified and analyzed. As far as event schemas go, this one seems to be appealingly well structured to cover many new types of experiments, and even a non-domain expert can imagine its useful applications.\n\nThe authors also describe a reasonably thorough process of developing their schema and annotating their documents. They demonstrate the flexibility of how they defined their discontinuous entities to extract dosage amounts, units, and per-time-period unit aspects to compose complete dosage information. They also describe a two-stage annotation process, to align their annotators on consistent guidelines. And they report sample inter-annotator agreement for both entities and events. The inter-annotator agreement for entities in particular is very high.\n\nThe experiments, while briefly described in this paper, also appear to show that the dataset can be used to train reasonably useful models for detecting and classifying the annotated entities and events in animal experiments.", "reasons_to_reject": "There are a few points where the paper appears to be a bit weaker, although they may not be major weaknesses. One concern is that inter-annotator agreement on the event annotations (0.586) is much lower than for the entities (0.973). The authors cite IAA rates for another related dataset to show that theirs are higher, as justification that theirs are sufficient. But they don’t say anything further about why there was so much more disagreement about the event annotations.\n\nDoes this suggest an upper bound on the potential accuracy or F1 scores that a model trained on this dataset could achieve for event detection and/or classification? It is possible that events are much more difficult for humans to agree on. But the relative simplicity of their event types gives the impression that it might have been possible to do another round of aligning between the two annotators on the exact definitions of events to be annotated. A bit more discussion of why it wasn’t feasible to get higher agreement might help build confidence in the usefulness of that part of the dataset.\n\nThe model experiments at the end of the paper are also not described in much detail. In this case, that is probably fine, since the authors are reproducing existing models from past research, and the focus of their paper is on the new dataset rather than any model innovations. They show results for two models each for the NER and EE tasks, respectively. There isn’t a lot to conclude from those findings, though, other than that the models do seem to be learning.\n\nI’m not entirely sure how to assess whether these results are impressive or not, because no comparative results from similar research are shown. The authors are the first to use their own dataset. But since their goal is to demonstrate the value of their new dataset, it actually might be helpful to show how the same types of models performed on a previous dataset. If the authors get at least similar scores when the models are trained on their own dataset, that could indicate that their dataset is as consistently labeled and useful for training NER and EE models as other datasets have been.\n\nIt would also be reasonable, however, to expect that different entity and event schemas will lead to different average performance of models trained on them. So model results from past research using other datasets might not be very comparable. Again, I don’t think this is a major issue, because the paper’s main focus is not the model experiments. But here too, a little more discussion of what we should learn from the experiments, or how they might suggest a jumping-off point for other researchers seeking to use this dataset, might help strengthen the paper more.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a new dataset for biomedical NER and EE. It focuses on scientific literature about animal experiments, a stage of clinical research for which the authors indicate there aren’t as many IE resources as cell stage research. The dataset includes 350 documents manually annotated with 22k entities and 10k events.\n\nThe authors developed their own schema of entity and event types relevant for animal experiments, consulting with domain experts. Notably, the dataset contains discontinuous entities and nested events, which the authors found to be important structures for capturing specific attributes of the relevant experiments.\n\nThe paper focuses mainly on the introduction of the dataset, a detailed explanation of the process of collecting and annotating the included examples, and resulting frequencies of events and entities. The paper also includes experimental results from training several existing NER and EE models on the new dataset, showing that the dataset can be used to train models to detect and classify the included entities and events.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This data paper describes a new gold dataset for biomedical NER and EE focused on scientific literature about animal experiments. The dataset  fills a gap in biomedical NLP.  The corpus is annotated for events and entities, according to a schema specifically designed consulting domain experts. The paper offers a description of the corpus creation process, proposes a new annotation schema,  and reports rich descriptive statistics. The two-stage annotation process followed can be robust enough to guarantee quality, provided the annotation guidelines are made clear. IAA is reported for both entities and events. Various existing NER and EE models are also evaluated with the new dataset, and the results support the efficacy of the dataset for NER and EE.  
Both reviewers and authors engaged seriously in reviewing and discussion periods. The authors' responses were accurate and detailed; they acknowledged some of the indicated shortcomings, and provided clarifications and details on many of the points raised. Reviewers generally appreciated the work, but found some serious weaknesses in the original paper.  Particularly in need for attention are the following: 
- an explanation of the differences in inter-annotator  agreement between entities and events is needed. Given the scores, a discussion seem to be beneficial as to whether and how far disagreement is intrinsic to the task or may depend on the annotation guidelines;
- having just two annotators questions the overall validity. This choice should be well-motivated; 
- the complete annotation guidelines were made available only during the discussion phase, and still seem weak in some parts;
- the paper is well-written, but the framing of is not fully convincing, for instance “related works” could be more focused on the specific task addressed in the paper;
- the work would benefit from a comparison with performance on different datasets in the experimental part.
Decision: Accept-Findings