Intervention extraction in preclinical animal studies of Alzheimer’s Disease: Enhancing regex performance with language model-based filtering (2024)

Abstract: We explore different information extraction tools for annotation of interventions to support automated systematic reviews of preclinical AD animal studies. We compare two PICO (Population, Intervention, Comparison, and Outcome) extraction tools and two promptingbased learning strategies based on Large Language Models (LLMs). Motivated by the high recall of a dictionary-based approach, we define a two-stage method, removing false positives obtained from regexes with a pre-trained LM. With ChatGPT-based filtering using threeshot prompting, our approach reduces almost two-thirds of False Positives compared to the dictionary approach alone, while outperforming knowledge-free instructional prompting.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 486–492 August 16, 2024. ©2024 Association for Computational Linguistics
486

heading: 1 Introduction and Related Work
text: Biomedical information extraction is the task of automatically extracting entities, relations, and events from biomedical literature (Hobbs, 2002; Liu et al., 2016). This information is in turn relevant to writing of systematic reviews, which support evidencebased decision making by identifying, integrating, and assimilating relevant articles on a given clinical question (Methley et al., 2014). A standard framework used for defining review questions is PICO, standing for Population (or Patient), Intervention (or Exposure), Comparison, and Outcome (Cooke et al., 2012).
We examine information extraction in Alzheimer’s Disease (AD), which has affected more than 55 million people around the world1. We focus on detecting the PICO dimension of Intervention in the AD literature, where interventions are typically drugs. This task is sometimes referred to as intervention extraction. It suffers from having low precision compared to extraction of other PICO elements (Hair et al.,
1https://www.alzint.org/about/dementia-factsfigures/dementia-statistics/
2023a). More precise extraction of interventions will support more effective systematic reviewing, and can help to prioritize drugs for clinical trials in literature-based discovery (Pu et al., 2023).
Standard methods for intervention extraction include dictionary-based approaches (Hair et al., 2023b) and machine learning models (Wang et al., 2021; Wei et al., 2024). The recent advent of generative Large Language Models (LLMs) and the prompting-based paradigm for information extraction (Liu et al., 2023) raise questions of how to better leverage them in this task and whether they outperform previous methods. This is particularly interesting in domain-specific scenarios such as AD, where limited data is available for training models (Wang et al., 2023). We address these questions, with two main contributions: 1) we show that while generative LLMs improve intervention extraction precision, they suffer from low recall compared to dictionary-based methods, and 2) we propose a two-stage architecture combining both dictionaries and LLMs that better balances precision and recall and reaches a new state-of-the-art on the AD dataset.

heading: 2 Methods

heading: 2.1 Data
text: We used a manually-curated dataset containing preclinical animal studies in the context of AD (Hair et al., 2023b). This dataset consists of documents comprising title, abstract, and keyword fields for 100 studies. The dataset was created in two steps: 1) a set of regular expression (regex) patterns corresponding to a dictionary of interventions was applied to annotate intervention entities, and 2) a human annotator labeled each tagged entity as “intervention” or “not an intervention”. Figure 1 shows an example of an annotated document, the extracted entities, and the human judgment label for each. The AD dataset may not be perfect since
human annotations were a subset of regex annotations. Intervention entities not being captured by the regex dictionary were out of the scope for the human annotator. However, we used human annotations as a gold standard for this study.
We randomly split the dataset into a training set (5 documents) and a test set (95 documents) (Table 1). The training documents were used for fewshot learning in prompt-based methods. All results are reported on the test set.

heading: 2.2 Baselines
text: We adapted three biomedical entity extraction tools for intervention extraction to use as baselines, detailed in Table 2. The regex-based method of (Hair et al., 2023b) utilizes a customized dictionary based on regular expressions for preclinical AD animal studies. Each publication was tagged with animal models, outcomes, interventions, species, and sexes; here we considered only entities tagged as interventions. The intervention dictionary had a list of 12,447 compounds compiled from DrugBank2 and Alzforum3. Synonyms, alternate spellings, and punctuation differences were captured in regexes (Hair et al., 2023b). This method was used to create
2https://go.drugbank.com/drugs 3https://www.alzforum.org/therapeutics
the dataset employed in our experiments, resulting in maximum recall by design.
Wang et al. (2021) constructed a PICO extraction workflow based on Bidirectional Encoder Representations (BERT; Devlin et al. (2019)) for general preclinical animal studies (not specific to AD). This method had two entity categories that relate to interventions: Intervention and Comparator. Intervention was defined as interventions that reflect clinical practice, while Comparator was defined as a control group, such as no treatment, vehicle/placebo, sham treatment, or another intervention. We treated entities tagged as either Intervention or Comparator entity types as interventions.
Finally, we also used the latest version of PubTator 3.0 (Wei et al., 2024) as an additional baseline, due to its widespread usage in biomedical information extraction. This tool extracted proteins, genetic variants, diseases, and chemicals with a recently developed named entity recognition (NER) model called AIONER (Luo et al., 2023). We treated Pubtator-identified Chemical entities as Interventions. For this entity type, training was based on the NLM-Chem corpus (Islamaj et al., 2021), with ∼5000 unique drug/chemical name annotations in 150 PubMed full-text chemical literature. The PubTator API4 was used to conduct raw processing of input texts for entity extraction.

heading: 2.3 Prompt-based methods
text: Since we had limited labeled data, we prioritized prompt-based models over training machine learning or deep learning models. We followed the framework of (Liu et al., 2023) to design our prompt-based models. We considered four aspects of prompt-based learning for intervention extraction: pre-trained language models (PLMs), prompting templates, answer space, and prompting parameters.
Pre-trained LMs We selected ChatGPT5 and GPT-4 (OpenAI, 2023) as the PLMs.6
Prompting templates We adapted prompting templates previously used for zero-shot gene extraction in biomedical literature (Törnkvist, 2024) to our task for both zero-shot and few-shot learning.
4https://www.ncbi.nlm.nih.gov/research/pubtator/api.html 5https://platform.openai.com/docs/models/gpt-3-5-turbo 6In an effort to employ open-source LMs, we also considered OLMo (Groeneveld et al., 2024). However, we were not able to obtain meaningful answers from this LM.
Our templates7 are described in Appendix B.
Answer space We used text spans from the documents that were recognized as interventions.
Prompting parameters For both models, we set temperature to be 0.7, max_tokens as 50, and top_p as 1. The “temperature" parameter controls randomness, which ranges between 0 to 2.

heading: 2.4 Two-stage filtering method
text: The approach we proposed (Figure 2) was motivated by the maximum recall provided by the regex patterns used to create the dataset. Instead of having a PLM doing the full work of extracting interventions, we proposed using it to filter the false positives obtained from regexes. Precision errors arising from regex-based methods were mainly due to a lack of context: any entity that matched a regex would be recognized as an intervention. We hypothesized that a PLM can contextualize entity
7The prompting templates (Appendix B) used for baselines were different from the templates (Appendix C) in the two-stage filtering method. For baselines, the prompting templates were adapted from Törnkvist (2024). For the two-stage filtering method, the prompting templates were created on our own.
context and filter them out if appropriate, without undermining recall.
We experimented with both zero-shot and fewshot approaches (with examples sampled from the training set), using ChatGPT as the PLM. We tailored our prompts to frame the task as filtering (described in Appendix C). All other parameters were the same as described in Section 2.3.

heading: 3 Results and Discussion
text: Table 3 summarises our results, reporting precision, recall, and F1 scores for all methods. As expected, the regex-based method resulted in perfect recall (since it was employed to develop the dataset in the first place) but low precision. Both BERT-based and PubTator 3.0 approaches did not perform well, likely due to domain differences. The prompt-based methods resulted in slightly better precision, at the price of a large decrease in recall.
Our zero-shot filtering approach outperformed the baselines in F1 score, with only a small decrease in recall. It filtered 30% of the false positives (FPs) of the regex-based method. A three-shot variant, adding three true positive (TP) examples to the prompt, gave even better precision and F1 score, filtering almost two-thirds of all FPs.

heading: 3.1 Further analysis on few-shot prompting
text: We performed additional experiments analyzing the influence of adding positive examples to the prompt in our two-stage method. The regex-only baseline resulted in 67 TPs and 288 FPs: an ideal filtering layer should remove all FPs while keeping the original TPs.
Table 4 shows detailed results using two metrics: the total reduction in FPs and the total reduction in TPs (the latter framed as “TP price”, since this should ideally be zero). In general, the higher the FP reduction, the higher the TP price. However, we did not see any particular trends when increasing
the number of examples, except for an outlier result for 1-shot prompting. Our 3-shot results provided the best balance but more work is required to understand if further increasing the number of examples can result in better performance.
We also tried using negative examples for fewshot learning. However, this did not improve the performance compared to using positive examples only. We report detailed results in Appendix A.

heading: 3.2 Motivating case study
text: We discuss case studies for our motivation for employing the two-stage filtering method. We analyzed the False Positives (FPs) of the regex-based method. As this method annotated every text span matching the intervention dictionary indiscriminately, the 288 FPs came from context recognition errors, i.e. where an intervention term did not describe a relevant intervention in the context of a document. For instance, the potential intervention entity “quercetin" in PMID:36840284 (Table 5) was part of the molecular modeling results of a compound rather than a drug whose effects were directly studied.
One may argue that removing “quercetin" from the regex dictionary would reduce FPs. However, the entity “quercetin" was also used as an intervention in other contexts. As shown in Table 5, PMID:30618732 assessed the effects of “quercetin" as an intervention for treating adult mice with neurodegenerative diseases. Therefore, an ideal method must contextually differentiate usages of the putative entity mentions.

heading: 3.3 PLM response outliers
text: A generative PLM may produce model responses out of the target answer space, requiring further processing. In the Regex+ChatGPT (0-shot) scenario, the model responded with “therapeutic” (cf. “intervention”/“not an intervention”) for a potential intervention entity string “therapeutic" and a given context of PMID:25061594. In the Regex+ChatGPT (3-shot) scenario, the model responded with a copy
of the prompting template for a potential intervention “potassium" for PMID:30548427.
For these two outliers, we reverted to the output of a RegEx phase, i.e. with the label “intervention”.

heading: 4 Conclusion
text: In this work, we proposed a two-stage approach for intervention extraction that combined a regexbased method with a filtering step done by prompting a generative LLM. This approach outperformed strong baselines, including standalone use of LLMs. Effectively, we show that LLMs can augment regex/dictionary-based methods by removing context recognition errors.
Future work involves extending our approach to all PICO entities, beyond just interventions. This will help automate important tasks in the literature review for AD, such as collecting data for systematic reviews, and support creating more precise knowledge graphs for literature-based discovery. The same approach could also be adapted with specific resources and be applied to other datasets and domains, such as clinical trials (Nye et al., 2018). Finally, different strategies to employ LLMs in the filtering step could be investigated, such as finetuning.

heading: A Prompting with negative shot for a two-stage filtering method

heading: B A prompting template for intervention extraction baselines
text: B.1 Zero-shot learning
B.2 Few-shot learning

heading: C A prompting template for a two-stage filtering method
text: C.1 Zero-shot learning
C.2 Few-shot learning

heading: Future Work
text: Future work involves extending our approach to all PICO entities, beyond just interventions. This will help automate important tasks in the literature review for AD, such as collecting data for systematic reviews, and support creating more precise knowledge graphs for literature-based discovery. The same approach could also be adapted with specific resources and be applied to other datasets and domains, such as clinical trials (Nye et al. , 2018). Finally, different strategies to employ LLMs in the filtering step could be investigated, such as finetuning