Mention-Agnostic Information Extraction for Ontological Annotation of Biomedical Articles (2024)

Abstract: Biomedical information extraction is crucial for advancing research, enhancing healthcare, and discovering treatments by efficiently analyzing extensive data. Given the extensive amount of biomedical data available, automated information extraction methods are necessary due to manual extraction’s labor-intensive, expertisedependent, and costly nature. In this paper, we propose a novel two-stage system for information extraction where we annotate biomedical articles based on a specific ontology (HOIP). The major challenge is annotating relation between biomedical processes often not explicitly mentioned in text articles. Here, we first predict the candidate processes and then determine the relationships between these processes without relying on mentions. The experimental results show promising outcomes in mentionagnostic process identification using Large Language Models (LLMs). In relation classification, our proposed BERT-based models outperform LLMs significantly. The end-to-end evaluation results suggest the difficulty of this task and room for improvement in both process identification and relation classification.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 457–473 August 16, 2024. ©2024 Association for Computational Linguistics
457

heading: 1 Introduction
text: In the biomedical domain, unraveling the mechanisms underlying various diseases contributes significantly to their treatment and prevention. However, information about these mechanisms is often scattered across articles, presenting challenges. The challenges include the lack of clarity, the implicit nature of background knowledge, and the ad hoc use of vocabularies with variations in notation. Moreover, inherent biological complexity spans molecules, cells, and organs, with external factors such as viruses influencing infection mechanisms.
To address these challenges, organizing knowledge through ontologies is crucial as they provide
*Equal contribution.
a clear framework for consistently structuring entities and their relationships. In the Homeostasis Imbalance Process Ontology (HOIP), manual annotation has been employed to extract and structure knowledge about processes such as cellular senescence and COVID-19 infection mechanisms (Yamagata et al., 2021, 2024). Despite these systematic approaches, manual annotation faces significant challenges due to its high cost and time-consuming nature. These challenges highlight the need for more efficient and consistent (semi-)automated annotation approaches to improve the overall quality and usefulness of ontologies.
In this paper, we propose an application of Natural Language Processing (NLP) as a promising solution. Specifically, assuming automatic annotation of the HOIP ontology as our ultimate goal, we propose a two-stage information extraction (IE) system. Figure 1 shows an overview of our two-stage system. Given an input passage1, the first stage, Process Identification, identifies process entities that are described in the passage or can be inferred using the domain knowledge.2 The entities are represented as unique IDs in the ontology. The entities are then passed to the second stage, Document-level Relation Extraction (DocRE) (Christopoulou et al., 2019; Zhou et al., 2021; Xiao et al., 2022; Zhang et al., 2021; Li et al., 2023), to classify entity pairs into a pre-defined set of interrelations. The system output is represented as a set of triples: {(head entity ID, relation, tail entity ID)}. We develop and evaluate different approaches including supervised models based on BERT (Devlin et al., 2019) and
1In this paper, we use the word “passage” instead of “document” or “paragraph” to describe the input in our task, because the text describing biomedical processes is not necessarily a complete text like an entire paragraph or document.
2Process Identification is similar to Entity Disambiguation (ED), but differs as discussed in the following paragraph. Given an input passage, ED aims to identify entities (IDs) for each given mention, whereas Process Identification aims to identify entities (IDs) without the availability of mentions.
generative methods based on Large Language Models (LLMs) and In-Context Learning (ICL) (Brown et al., 2020; Chowdhery et al., 2022; Wadhwa et al., 2023; Ozyurt et al., 2024) for both process identification and DocRE. The HOIP dataset, a novel manually annotated dataset built based on the HOIP ontology for biomedical IE system development, will be available to the public.
Traditional IE studies (Yu et al., 2020; Wu et al., 2020; Zhou et al., 2021) assume that an entity can appear multiple times in a passage explicitly (such textual instances are called mentions), and derive entity features from these mentions. Mentions are strong indicators in IE, since they directly indicate how entities are described in a text. However, in real-world scenarios including our HOIP dataset, an entity sometimes appears only implicitly. With no availability of mentions, it is not obvious how to induce useful entity features from a passage. This paper proposes multiple approaches that do not require explicit mentions.
Our contributions and findings are summarized as follows:
• We release the HOIP dataset, to facilitate the development and bench-marking of IE models for the real-world ontology. • We develop a mention-agnostic two-stage IE system, which enables to extract structured knowledge described implicitly in text. BERT-based supervised models and LLM-based models are presented for both process identification and DocRE. • Experimental results in process identification suggest that generative models are valuable for lowresource in-domain corpora like the HOIP dataset. • DocRE results suggest that, although mentions are strong indicators, the proposed BERT-based mod-
els outperform LLMs and achieve F1 scores of around 56-59 points even without mention hints. • Evaluation results on the end-to-end system reveal that improvements in both process identification and DocRE are crucial in the current stage. • The HOIP dataset and the source codes are available: https://github.com/norikinishida/ hoip-dataset (dataset), https://github. com/sl-633/bio-process-identifier (process identification), https://github.com/ norikinishida/kapipe (DocRE).

heading: 2 HOIP Dataset
text: Our ultimate goal is to update and improve ontologies by (semi-)automatically extracting entities and their interrelations from articles. As a testbed ontology, we choose the Homeostasis Imbalance Process Ontology (HOIP) (Yamagata et al., 2021, 2024), which focuses on understanding the COVID-19 infectious mechanism (courses).3 To facilitate the development of NLP systems and benchmark the task, we construct and release a new dataset named the HOIP dataset based on the HOIP ontology. The dataset includes passages extracted from PubMed articles describing biomedical processes in the context of COVID-19 infectious courses. Each passage is a brief portion of a PubMed article that describes at least two specific processes. The processes are manually annotated as a set of triples, i.e., {(head entity, relation, tail entity)}. Figure 1 shows a real example in the dataset.
3For the details of the ontology see Appendix A.

heading: 2.1 Data Collection and Enhancement
text: We first stored the HOIP ontology files in an RDF store using Apache Jena Fuseki4, and constructed a SPARQL endpoint. We used SPARQL queries to retrieve the information required for the dataset. The results were then converted to CSV. To optimize the dataset for machine annotation and enhance its clarity, we made several adjustments based on the hierarchical structure of the HOIP ontology. Originally, some process entities included course information, such as “blood vessel damage in severe COVID-19” (the course is in italics), indicating a specific context. We removed these course information from the entities to optimize for machine annotation. Additionally, processes with too fine granularity were deemed unsuitable for machine annotation predictions. Therefore, we prioritized processes that are generalized using superclasses of each process, assigning Gene Ontology (GO) terms (Ashburner et al., 2000). This approach ensures that the annotations are practical for applications for reusability.

heading: 2.2 Dataset Organization
text: In the CSV file generated by the above procedure, each record corresponds to one triple. We combined the triples associated with the same passage (string) and PubMed ID into the same group, and this group was considered to be a single example in the final dataset. We found that there were textual overlaps across the passages. Thus, if a passage dsrc was textually contained in another passage ddst and both passages are associated with the same PubMed ID, the triples Tsrc for dsrc were merged into the triples Tdst for ddst. Finally, we split the entire dataset into training, development, and test sets, ensuring that passages extracted from the same article were not scattered across different splits. The dataset statistics are shown in Table 1.
4https://jena.apache.org/documentation/fuseki2/

heading: 3 Methods for Process Identification
text: In the HOIP dataset, a biological process entity is annotated depending on whether it is mentioned (explicitly or implicitly) in the passage, without specifying the corresponding phrase of the entity in the passage. This makes the dataset more closely match the real-world scenario, but also brings challenges to the automatic process identification – directly employing Named Entity Recognition methods that require the correspondence between a explicit mention (entity text and offsets) and an input text for model training is no longer an option. To address this task, we propose approaches to identify biological processes without prior recognition of mentions that can be matched to terminological expressions of entities in the HOIP ontology. Two distinct approaches are developed: BERT-based supervised methods and LLM-based In-Context Learning (ICL) methods.

heading: 3.1 BERT-based Supervised Approach
text: Considering that 360 unique process names are encompassed in the HOIP dataset, the task of process identification can be approached as a multi-class and multi-label classification problem. For simplification purposes, we convert the task into a binary format, framing it in the following manner: Let D be the set of passages and A be the set of annotated process names. The input sequence is constructed for each passage from D as follows:
[CLS] passage [SEP] name [SEP]
where name denotes a process name ai ∈ A. Then, the task is a binary classification task whether the passage involves the process or not.

heading: 3.2 LLM-based ICL Approach
text: Taking into account the unique characteristics of the dataset and the rapid advancements in the capabilities of LLMs to produce coherent texts in low resource settings (Wang et al., 2023), LLMs are utilized in this study to generate HOIP processes for each passage. We aim to evaluate the model’s performance in low-resource settings characterized by imbalanced data in a specialized domain, and assess the model’s generative capability in producing HOIP ontology terms. • Zero-shot setting: The model is prompted to
list the biological processes present in the text. Following a prompt format being demonstrated effective in many studies (Mishra et al., 2022;
Sclar et al., 2023), the prompt includes task instruction, constraints on the output, the input text. An example of the prompt is shown in Table 8. • Few-shot setting: Following the previously mentioned prompt format, two few-shot strategies are employed through adding demonstrations: the first involves selecting randomly three examples from the development set, while the second is selecting examples based on semantic closeness of process names.

heading: 4 Methods for Document-level RE
text: Given an input passage d, a set of entities for the passage {e1, · · · , eK}, and a pre-defined set of relations R, document-level relation extraction (DocRE) (Christopoulou et al., 2019; Zhou et al., 2021; Xiao et al., 2022; Zhang et al., 2021) aims to predict relations from R ∪ {NA} for entity pairs (ei, ej) (i, j ∈ [1,K]; i ̸= j), where ei and ej denote head and tail entities respectively, and the NA class indicates that the entity pair has no relation.

heading: 4.1 QA-Style DocRE Model
text: Our first approach is to perform DocRE as a Question Answering (QA) task. We first generate questions for each possible triple. The question and the input passage are concatenated and passed to a pre-trained language model for answering.
Question Generation. We first enumerate all possible entity pairs {(ei, ej)}i,j∈[1,K];i ̸=j , and then apply pre-defined template functions {Tr}r∈R to the entity pairs to obtain questions for each possible triple (ei, r, ej): q(ei,r,ej) = Tr(ei, ej). Table 7 shows examples for the pre-defined templates. The input x to our QA model is as follows:
[CLS] question [SEP] passage [SEP]
where question and passage are the word-pieces tokens of q(ei,r,ej) and d, respectively. An example for the input is “[CLS] does immunoglobulin production result in immunoglobulin mediated immune response ? [SEP] within 19 days after sympton onset , 100 % ... [SEP]”.
Answer Classification. Then, we feed the input sequence x into a BERT-based encoder (Devlin et al., 2019; Beltagy et al., 2019) to obtain the contextual embeddings: {hw} Nxtok w=1 = Encoder(x), where Nxtok is the number of tokens in x. We concatenate the output of the last layer for the [CLS] token and the average-pooling embedding to obtain
the passage embedding: h̃ = h1⊕ 1Nxtok ∑Nxtok
w=1 hw, where ⊕ represents the concatenation of vectors. Then, we apply a two-layer feed-forward network and sigmoid activation to the passage embedding to calculate the probability of answer “Yes”.
Loss Function. The network is trained using a binary cross entropy loss to maximize the probability for the correct triples.

heading: 4.2 Mention-Agnostic ATLOP (MA-ATLOP)
text: Our first approach requires solving QAs for all the possible triples. Since the number of possible triples is increased by O(K2) for the number of entities K, this is not efficient. Our second approach is to make predictions over all possible triples in a single forward pass. We extend a traditional and popular DocRE method, ATLOP (Zhou et al., 2021), so as not to rely on explicit mentions. We call this method Mention-Agnostic ATLOP, or MA-ATLOP shortly.
Entity Encoding. We use a BERT-based encoder to encode each entity ei and passage d jointly into a dense vector that takes into account how the entity ei is described in the passage d. Specifically, we first retrieve the canonical names {ni}Ki=1 and the descriptions {si}Ki=1 for the given entities from the ontology using the entity IDs as query. Then, for each entity ei, we construct input xi as follows:
[CLS] name : description [SEP] passage [SEP]
where name, description, and passage are the wordpieces tokens of ni, si, and d, respectively. We apply the encoder to each input xi independently to obtain contextual embeddings: {hi,w} N xi tok
w=1 = Encoder(xi). We take the embedding of the [CLS] token as the entity embedding, i.e., ei = hi,1.
Relation Classification. After obtaining the entity embeddings {ei}Ki=1, we apply two separate FFNNs and tanh activation to map them to different representations for the head/tail entities of triples. Then, we apply a group bilinear classifier (Zheng et al., 2019; Tang et al., 2020) to the head/tail representations of an entity pair (ei, ej). Specifically, we divide both head/tail representations into G contiguous groups and then apply bilinear to each group. They are then summed up to calculate the score for relation r ∈ R ∪ {TH}. Refer to Zhou et al. (2021) for the detail of group bilinear. We follow ATLOP and employ the adaptive-thresholding
class TH. The relations scored higher than the TH class are regarded as positive. If no such relation exists, the NA class is assigned to the entity pair.
Loss Function. We use the adaptive-thresholding loss proposed in ATLOP to push the scores of correct/incorrect relations to be higher/lower than the TH class.
Negative Entity Sampling (NES). In an experimental setting, we assume experts correctly annotate entities. However, in real-world situations, entities are automatically annotated by the systems. Thus, it often happens that entities not described in the passage are included in the given entity list {ei}Ki=1. Our DocRE system must be robust to such noisy (false-positive) entities. Therefore, we propose Negative Entity Sampling (NES), where we sample additional negative entities randomly and add them to the given entities {ei} Kpos i=1 during training. We sample negative entities from all entities in the ontology. Given the number of positive entities Kpos and a hyperparameter ρ > 0, we define the number of sampled negative entities as Kneg = round(ρ×Kpos), where round is the rounding function. For instance, for Kpos = 10 and ρ = 0.5, Kneg is 5. We add a linear layer to the network to classify whether the entity is described in the passage: yenti = σ(FFNNent(ei)). We use a binary cross entropy as an auxiliary loss to maximize yenti for positive entities.

heading: 4.3 LLM and In-Context Learning for DocRE
text: To investigate the effectiveness of LLMs with In-Context Learning (ICL) (Brown et al., 2020; Chowdhery et al., 2022; Wadhwa et al., 2023; Ozyurt et al., 2024) in our task, we compare the LLM-ICL results with the above BERT-based models. Table 9 in Appendix C shows a prompt example we used in our experiments. Specifically, we instruct an LLM to generate a bulleted list of triples for the given passage and the entity list.5 Each entity is represented in the form "* <ID> : <NAME>" and the entity list is presented as a bulleted list. In the prompt, we also use 3 examples randomly sampled from the training set as the fewshot demonstrations. The same demonstrations are used for all test passages. From each bulleted line generated, we extract the head entity ID ei, the relation label r, and the tail entity ID ej using regular
5In our preliminary experiments, we also tried to generate JSON directly by Llama2 13B; however, generating JSON yielded lower DocRE scores consistently than generating text.
expressions. If the extracted entity IDs (ei, ej) and the relation label (r) cannot be found in the given entity list {e1, · · · , eK} and the possible relation classes R, the bulleted line is ignored. We also remove duplicated triples. The resulting triples are then compared with the gold triples for evaluation.

heading: 5 Experiments on Process Identification
text: Binary classification. In the supervised task formulation, each passage is associated with all 360 process names, with binary labels assigned based on the presence of them in the annotations. Numerous negative samples are constructed for each passage. Consequently, we incorporate negative sampling using various ratios of negative to positive samples. The classification task involves finetuning BERT-based models – BioBERT (Lee et al., 2019), SciBERT (Beltagy et al., 2019), and PubMedBERT (Gu et al., 2021) on the training set. For hyper-parameter details see Table 11.
Generative experiments. Two instruction types are utilized: One prompts the model to list all biological processes in the text, while the other instructs it to generate pairs of processes having a relation. This distinction stems from annotation being conducted at the relation level, where only processes involved in relations are annotated. Consequently, other processes may exist in the text but aren’t annotated. We employ Llama2 13B (Touvron et al., 2023) and Llama3 8B (AI@Meta, 2024) on the test set to ensure comparability with the supervised method’s results.

heading: 5.1 Clustering-based Demonstration Selection
text: In the zero-shot setting, process names differ significantly from the provided annotations. In the few-shot scenario, performance is highly sensitive to the chosen demonstrations (Li et al., 2022; Lu et al., 2022; Zhang et al., 2023). To enhance fewshot performance, we introduce a retrieval module based on semantic similarity to cluster the most relevant examples from the annotated processes in the development set.
To achieve this, we use the development set to create 10 clusters with K-means clustering (Lloyd, 1982; MacQueen et al., 1967), based on the annotated process list for each passage. Each list is encoded into a vector using BERT by averaging the last hidden state of the [CLS] token for all labels. Each passage in the test set is assigned a cluster given the last hidden state of the [CLS] token of the
passage to the clustering model. During generation, examples from the assigned cluster are included in the prompt. See Table 10 for details.

heading: 5.2 Evaluation Methods
text: We evaluate three primary aspects of generative process identification. The initial aspect involves the Direct Output assessment, where we directly evaluate the output by comparing it to the annotation. This process aims to weight the model’s ability to generate processes formulated within the knowledge base framework. As for the second aspect – assessing the system’s capability to automatically populate the target knowledge base, we include an ontology alignment-based evaluation which is presented in three steps:
1. Computing embeddings: Let Egenerated be the set of embeddings of the generated processes and Eontology be the set of embeddings of ontology terms calculated by SapBERT (Liu et al., 2021). 2. Selecting top k generated processes: For each xi from Egenerated, calculate sim(xi, yj) for all yj from Eontology. Then, select the k elements with the highest cosine similarity. 3. Computing precision, recall and F1 scores between the list of annotated processes and the flattened list of top k generated processes at a passage level.
The last aspect of evaluation mirrors the second one, with the distinction being that instead of matching with the entire ontology, only the process names used in the dataset are taken into account. This approach is grounded on the assumption that the terms utilized in the dataset annotation are the most prevalent.

heading: 5.3 Results and Discussion
text: Direct output assessment. We compared annotation labels with generated process names using zero-shot, regular few-shot, and ICL few-shot settings. The ICL few-shot method achieved the most exact matches, with 30 compared to 2 for regular few-shot and none for zero-shot, underlining the importance of better selected demonstrations, as in Min et al. (2022). Thus, evaluation using the HOIP ontology and dataset matching will be based on the ICL few-shot setting outputs.
In-Dataset Matching. We report in Table 2 the results of the fine-tuned BERT-based models. The negative ratio significantly influences the overall performances of the models. Results indicate that with fewer negative samples, models are more likely to identify true positives but at the cost of also misclassifying more false positives. This is likely due to semantic similarity among the inputs.
PubMedBERT achieves the highest F1 score, under the optimal negative ratio of 8.
Furthermore, we compare the results of the supervised approach with the top-1 results of Llama2 and Llama3 present in Table 3. Across all negative ratios, the F1 score of BERT-based models exceeds the results of Llama2, even under the lowresource setting. However, this trend changes with Llama3, which outperforms the PubMedBERT result by nearly 2 points. Comparing Llama2 and Llama3 reveals that Llama3 is more effective at generating well-tailored process names, resulting in higher precision. Llama3 generates fewer, but better-quality candidates, enhancing task performance. Llama2 improves with more candidates, increasing chances of correct matches, but accuracy still depends significantly on the quality of these generated candidates.
In-Ontology Matching. Following the same tendency in the In-Dataset setting, matching with better-generated process names proves to be more effective overall. Since the goal of this matching is to automatically populate an ontology and DocRE is the next step in the pipeline, concentrating on finding the correct process names is crucial. This focus will aid the DocRE step in serving as a filtering mechanism, ensuring more accurate and relevant candidate triplets to be added to the ontology.

heading: 6 Experiments on Document-level RE
text: We evaluate our systems on the HOIP dataset and CDR dataset (Li et al., 2016). CDR consists of 1,500 abstracts from PubMed, manually annotated with Chemical or Disease entities and Chemical-Induce-Disease relations between them. Since entity IDs in CDR are MeSH unique IDs (e.g., D006493), and the HOIP ontology, while highly specialized for annotating processes related to COVID-19, does not provide the coverage for general chemical compounds and disease terms as the MeSH controlled vocabulary, we used MeSH instead of the HOIP ontology for CDR. We use precision, recall, and F1 metrics, and report the scores averaged independently over 3 trials with different random seeds. A triple (ei, r, ej) is considered correct when the head entity ID (ei), the tail entity ID (ej), and the relation label (r) are all predicted correctly. We used greedy decoding in the LLM-ICL methods, the results of which do not depend on the seed differences. Table 12 shows the hyperparameters for our DocRE models.

heading: 6.1 Experiments on the CDR Dataset
text: To investigate the importance of mentions in DocRE and how well our models can identify relations without relying on mentions, we first evaluate the models on CDR. We also evaluate a variant of each of our models, which uses the first-appearing annotated mention as the entity name rather than the canonical name retrieved from the MeSH ontology. Although this variant still does not use mention spans, we expect this variant to recognize more easily how the entity is described in the passage than the original model, because the entity names appear at least once in the passage.
Table 4 shows the results. ATLOP exploits mention spans as the direct hints for entity encoding and achieves an F1 score of 69.7. In contrast, our best mention-agnostic model, i.e., MA-ATLOP (+ first mention), achieved an F1 score of 62.3, lower than the ATLOP score by 7.4 points. When there were no mention hints at all, MA-ATLOP and QAModel yielded F1 scores of 59.2 and 58.1, respectively. These results suggest that our models can identify triples more accurately than expected even without mention hints; however, mentions are still crucial in this task. Also, the BERT-based supervised models outperformed the LLM counterparts. MA-ATLOP outperformed QA-Model consistently. Considering that MA-ATLOP also has higher computation efficiency than QA-Model, MA-ATLOP is more suitable for real-world applications. By employing Negative Entity Sampling (NES), when no mention is available, MA-ATLOP improved all metrics slightly, suggesting the effectiveness
of NES. For the "first-mention" setting, NES did not improve the performance, probably due to the discrepancy between the entity-name style between positive entities (mention) and negative entities (ontology-based name).

heading: 6.2 Experiments on the HOIP Dataset
text: The upper block in Table 5 shows the results on the HOIP dataset when using the ground-truth entities. We evaluate the models that do not require mention hints on this dataset. The BERT-based models achieved much higher F1 scores (56.5-59.0) than LLM (17.6). MA-ATLOP also outperformed QAModel by 2.2 points in F1. These results were consistent with the results on CDR, demonstrating the effectiveness of MA-ATLOP in terms of both accuracy and computational efficiency in this task. Negative Entity Sampling improved the precision by 4 points, but decreased the recall. These results suggest that, while NES enhances the filtering capability of MA-ATLOP, NES also has the effect of making the model reluctant about positive predictions, and it would be necessary to develop techniques to avoid such biases.
The above experiments assume that the entities are fully and correctly annotated. This setup is appropriate for a clean measurement of the DocRE system’s performance itself. However, in reality, entities can be predicted automatically. To evaluate the whole system’s performance in the real-world situations, we evaluate our best DocRE model (MAATLOP) on the HOIP dataset with entities predicted by Llama3 (8B).
The lower block in Table 5 shows the results. We first calculated the upper-bound scores for the predicted entities. Specifically, we created a subset of gold triples that can be created based on the predicted entities. Precision, recall, and F1 scores are
100.0, 26.8, and 42.3, respectively. The precision does not depend on the quality of predicted entities. The lower recall suggests that there is much room for improvement in DocRE as the process identification recall improves. MA-ATLOP yielded 7.7, 14.9, and 10.2 scores for precision, recall, and F1, respectively. Compared to the much higher precision (67.2) in the gold-entity setup (Table 5), the results suggests that the current model struggles to filter out noisy triples with irrelevant entities. In summary, both improvements in recall (coverage) and precision (low-noisiness) in process identification and DocRE are needed in the current situation, suggesting the difficulty of this task.

heading: 7 Case Study
text: We performed case study to analyze the system outputs qualitatively. We used the best Llama3 (8B) and MA-ATLOP models for process identification and DocRE, respectively.
Table 6 shows an example with true-positive and false-negative entities and triples. Additional example can be found in Appendix F. For ease of understanding, entities are shown by names, not by IDs. We can observe that entities that are almost explicit in the passages, such as “pyroptosis” (Pyroptosis in the passage), and “pore formation in membrane of other organism” (Formation of pores), were accurately extracted. Triples that are almost explicit based on the context, such as (“pyroptosis” has part, “pore formation in membrane of other organism”) and (“pyroptosis”, has result, “release of DAMP molecules by cell rupture”), were correctly identified by the DocRE system. In contrast, implicit (or knowledge-requiring) entities and triples, such as “binding of pattern recognition receptor to DAMPs”, were not identified by the systems. The entity is derived from the interpretation that these molecules recruit more immune cells, which requires background knowledge of immunology: DAMP molecules must bind to receptors recognized by immune cells to recruit immune cells.
The quality evaluation reveals several insights: (1) Detailing causal relationships in elucidating disease mechanisms often necessitates background knowledge not explicitly mentioned in articles. This background knowledge is sometimes added to intermediate causal entities by manual annotation. The BERT-based supervised models and LLMs have difficulty in obtaining such background knowledge and understanding the task from limited
supervision (labeled data or demonstrations). (2) NLP systems could be complementary to manual annotations. Manual annotation often focuses on the causality of a particular process in one literature and gives priority to further causes and consequences in other literature. Therefore, other processes and causal relationships in the same passage may not be extracted. It is also possible that processes that missed identification due to simple errors due to annotation fatigue are also in the false positive. In such manual annotation issues, NLP analysis could make a significant contribution to the identification of processes.

heading: 8 Related Work
text: Gene Ontology Causal Activity Modeling (GOCAM) defines molecular-level causal relation-
ships (Thomas et al., 2019); however, it lacks granularity and context for COVID-19 infection. Our HOIP dataset is based on the HOIP ontology (Yamagata et al., 2021, 2024), which organizes knowledge about biomedical processes in the context of COVID-19 infectious courses and thus essential for analyzing SARS-CoV-2 infection and progression.
In knowledge acquisition, entities are typically identified by Named Entity Recognition (NER) and Entity Disambiguation (ED). The task of NER is to identify mentions in the given text that represent one of the pre-defined types (e.g., Chemical, Disease) (Yu et al., 2020; Zhu and Li, 2022; Ye et al., 2022). The entity mentions are then passed to ED to link them to the knowledge-base concept IDs that the mentions refer to best (Kolitsas et al., 2018; Wu et al., 2020; Cao et al., 2021; Yamada et al., 2022). These tasks commonly assume that entities are explicitly described in text. In reality, however, entities are not necessarily explicitly described. In this work, we explore mention-agnostic methods for process identification.
The most widely used approach to DocRE is to model entities by a pre-trained Transformer and perform pairwise relation classification. Christopoulou et al. (2019) proposed to model entity dependencies via graphs with nodes of various granularities. Zhou et al. (2021) proposed ATLOP, which models entity-pair contexts for pairwise relation classification. Xiao et al. (2022) introduced evidence modeling for improving ATLOP. Zhang et al. (2021) used U-Net architecture for modeling entity dependencies. These methods commonly rely on mentions and often insert special mentionboundary markers into text to indicate the mention locations to the Transformer. However, Li et al. (2023) showed that these methods are too sensitive to the accuracy of mentions and it is unrealistic to expect perfect mentions in the real-world scenario. In contrast, we propose mention-agnostic DocRE methods and investigate how well the mentionagnostic models can identify relations.

heading: 9 Conclusion
text: To assist ontology-based biological knowledge annotation, this work proposes a new dataset and practicable entity- and relation-level biomedical information extraction methods. We will continue to promote relevant research of semi-automatic annotation and advance practical applications.
Limitations
Despite demonstrating promising outcomes in mention-agnostic process identification and DocRE, our methodology does face limitations. First, our two-stage IE system consists of a cascade of process identification and DocRE, which inevitably suffers from error propagation. The experimental results in the pipeline setting suggest that the DocRE performance is significantly vulnerable to the accuracy of predicted entities. Moreover, the process identification model and the DocRE model are disconnected and cannot interact with each other. Second, our methods have only been evaluated in the domain of the HOIP ontology, and the accuracy in other biomedical domains and ontologies remains unknown. Third, our methodology has not been fully evaluated by domain experts. Although an expert analysis is performed, the analysis is based primarily on just two examples. A more thorough and detailed analysis by specialists is needed. Tackling these limitations remains an intriguing avenue for future research.

heading: Acknowledgments
text: We would like to thank the anonymous reviewers for their thoughtful and insightful comments, which we found very helpful in improving the paper. This work was supported by JSPS KAKENHI Grant Numbers JP22K17959, 21K17815, and JP22H05015, and ANR AIBy4 project (ANR20-THIA-0011), Nantes University.

heading: A HoIP Ontology
text: Understanding the HOIP ontology may be helpful for understanding our HOIP dataset and the task. In this section, we describe the features of the HOIP ontology.
The HOIP ontology is annotated based on COVID-19 related articles in PubMed using Protégé 5.5.06 and the Web Ontology Language (OWL). The COVID-19 infectious processes are manually annotated. Passages corresponding to the annotated terms are also provided. Article identifiers (e.g., PubMed ID (PMID: 25301932), DOI) are also provided using the database crossreference annotation property.
The processes in HOIP consist of a hierarchy. The infectious processes described in the articles and the superclass of each process using Gene Ontology are annotated.
The relationships between processes are annotated using object properties, Causal relationships between processes are primarily annotated using the ‘has result’ relationship. Furthermore, subprocesses of a process are identified using the ‘has part’ relation.
HOIP defines a "COVID-19 infectious course" as a sequence of the abovementioned processes to describe infectious mechanisms. These courses are organized into an is-a (subclass of) hierarchy by severity, ranging from mild to severe. Notably, the "COVID-19 severe course" includes a subclass associated with acute respiratory distress syndrome (ARDS). These COVID-19-specific processes are used as our primary dataset for this study.

heading: B Question Templates
text: Table 7 shows the question templates Tr (r ∈ R) used for QA-Model. In the table, <HEAD> and <TAIL> are replaced by the head and tail entity names, respectively. The entity names are retrieved from the ontology using the entity IDs as query. We
6https://protege.stanford.edu
manually created the question templates for each dataset: CDR and HOIP.

heading: C Prompts
text: Table 8 shows an example of the prompt used in the few-shot setting in process identification. Only examples section is discarded in the zero-shot setting. Table 9 also shows a prompt used in DocRE experiments on the HOIP dataset and the corresponding output by Llama3 (8B). We replaced the ontology name (“HOIP”) and possible relation classes (“hasresult, has-part, ...”) in the prompt template with “MeSH” and “Chemical-Induce-Disease” respectively in CDR experiments. The demonstrations are also different between the datasets.
D ICL Few-Shot Setting in Process Identification
Table 10 exhibits the number of examples per cluster created for ICL in the few-shot setting in process identification.

heading: E Hyperparameters
text: Table 11 shows the hyper-parameters used in the supervised models for process identification. Table 12 also list hyper-parameters used in our DocRE models.

heading: F Another Example of Case Study
text: Table 13 shows another example used in our case study (in Section 7).

heading: Future Work
text: Tackling these limitations remains an intriguing avenue for future research