BioDEX: Large-Scale Biomedical Adverse Drug Event Extraction for Real-World Pharmacovigilance
Authors: Karel D'Oosterlinck
François Remy
Johannes Deleu
Thomas Demeester
Chris Develder
Klim Zaporojets
Aneiss Ghodsi
Simon Ellershaw
Jack Collins
Christopher Potts
Venue: EMNLP 2023 Findings (2023)
Keywords: pharmacovigilance
datasets
healthcare
adverse drug events
TL;DR: We introduce BioDEX, a large resource for Biomedical adverse Drug Event Extraction rooted in the historical biomedical literature (65k abstracts, 19k full-text papers, 256k drug safety reports). We show that models perform well below human experts.

Abstract: Timely and accurate extraction of Adverse Drug Events (ADE) from biomedical literature is paramount for public safety, but involves slow and costly manual labor. We set out to improve drug safety monitoring (pharmacovigilance, PV) through the use of Natural Language Processing (NLP). We introduce BioDEX, a large-scale resource for Biomedical adverse Drug Event eXtraction, rooted in the historical output of drug safety reporting in the U.S. BioDEX consists of 65k abstracts and 19k full-text biomedical papers with 256k associated document-level safety reports created by medical experts. The core features of these reports include the reported weight, age, and biological sex of a patient, a set of drugs taken by the patient, the drug dosages, the reactions experienced, and whether the reaction was life threatening. In this work, we consider the task of predicting the core information of the report given its originating paper. We estimate human performance to be 72.0% F1, whereas our best model achieves 59.1% F1 (62.3 validation), indicating significant headroom. We also begin to explore ways in which these models could help professional PV reviewers. Our code and data are available at https://github.com/KarelDO/BioDEX.

BioDEX: Large-Scale Biomedical Adverse Drug Event Extraction for Real-World Pharmacovigilance
Karel D'Oosterlinckk: ! François Remy', Johannes Deleu', Thomas Demeester' Chris Develder', Klim Zaporojets', Aneiss Ghodsi", Simon Ellershaw', Jack Collins Christopher Potts
Ghent University imee Aarhus University
Parexel Al Labs
"Stanford University
"karel. wtw.:doootrrrncckueeeeeee
Abstract
Medical paper Abstract
Drug Safety Report
Timely and accurate extraction of Adverse Drug Events (ADE) from biomedical litera- ture paramount for public safety. but in- volves slow and costly manual labor. We out to improve drug safety monitoring (phar- macovigilance, PV) through the use Natu- ral Language Processing (NLP). We introduce BioDEX. large-scale resource for Biomedi- cal adverse Drug Event eXtraction, rooted in the historical output of drug safety reporting in the U.S. BioDEX consists of 65k abstracts and 19k full-text biomedical papers with 256k associated document-level safety reports cre- ated by medical experts. The core features of these reports include the reported weight, age, and biological sex of patient, set of drugs taken by the patient, the drug dosages, the reactions experienced, and whether the re- action was life threatening. In this work, we consider the task of predicting the core infor- mation of the report given its originating paper. We estimate human performance to be 72.0% F1, whereas our best model achieves 59.1 F1 (62.3 validation), indicating significant head- room. We also begin to explore ways in which these models could help professional PV re- viewers. Our code and data are available at https://gi www..tt............aeeeeeeema com/
Adult, female, 75kg
Threatening
Drugs taken
Body
Dextrose (40mVH)
Carboplatin (Iniection)
Chlorphanramine before chema)
Reactions experienced: Fibrile neutropenis, Pneumonis
Figure 1: BioDEX consists of 65k PubMed ab- stracts and 19k full text papers, accompanied by 256k document- drug safety reports. The schematic illus- trates the core information that constitutes drug safety report (they often contain much more detailed informa- tion as well). These reports created by pharmacovig- ilance experts and are vital for drug safety monitoring.
and biological sex, list drugs taken by the pa- tient, and a list of adverse reactions experienced and whether they are considered serious.
Drug manufacturers employ teams of experts to contimually triage new papers and submit these re- ports. This is challenging work since it requires experts to survey entire biomedical papers and uti- lize their pre-existing knowledge about drug of interest, its conventional indications, and its known adverse reactions. Furthermore, manufacturers are placed under constant time pressure keep up with the latest publications, since failure to report in a timely manner can lead to hefty fines and compro- mise public safety. This pressure has potential to increase in the near future: there has been a steady acceleration of biomedical research over the last few years (Figure 2), and drug events are consis- tently under-reported (Alatawi and Hansen, 2017).
1 Introduction
In the United States, the Food and Drug Admin- istration (FDA) mandates drug producers mon- itor and report Adverse Drug Events (ADE) de- scribed in the biomedical literature. Such a report, called an Individual Case Safety Report (ICSR), is stored in the FDA Adverse Event Reporting Sys- tem (FAERS; Food and Drug Administration 2017),
In this work, we set out to improve the scal- ability and accuracy of PV using Natural Lan- guage Processing (NLP). As first step, we intro- duce BioDEX, large-scale dataset for document- level Biomedical adverse Drug Event eXtraction. BioDEX consists of biomedical papers with asso- ciated expert-created drug safety reports. These reports were submitted to the FDA between 2012
which is cornerstone resouree for drug safcty re- search, also called pharmacovigilance (PV).
Figure| 1 briefly summarizes the core informa- tion PV workers must extract from papers while constructing these reports. This includes a descrip- tion of the patient in terms of reported weight, age,

---

1,000,000
Diomedicaal peperrs
2 Pharmacovigilance Reporting
year
750,000
Pharmaceutical companies are required to partic- ipate in drug safety reporting for the drugs they produce. Regulations differ across regions of the world. In this work, we focus on the pharmacovigi- lance process defined by U.S. regulations.
500,000
Drag Safety Heports
per year
1960
2000 2020
The reporting process starts with PV litera- ture review stage. Periodically, a vast database of biomedical literature is queried to retrieve new pub- lications that could describe an adverse event with regard to a drug of interest. Conventionally this is done by matching the trade name of the drug or names of its active substances. These queries are designed by experts and depend on the specific use- case, but they always aim for wide coverage; there are strong regulatory fines associated with missing reports, which creates strong incentives for very high recall. Reports can also originate from other modalities such as forms, emails, and social media. In this work, we only focus on reports originating from biomedical publications.
Figure The number of peer-reviewed biomedical papers published each year is accelerating (as indexed in Medline). The total number of drug safety reports originating from articles is on the rise as well, but the trend indicates stagnation (reports submitted to FAERS from 2012 onwards).
and 2022 as part of real-world PV efforts. Thus, BioDEX is grounded in the historical and regula- tory context of drug safety monitoring in the U.S. BioDEX contains PubMed articles published be- tween 1968 and 2022, with 65,648 articles having an abstract available and 19,433 featuring a full- text paper. In total, 256,240 reports are included (there can be multiple reports per article).
Once set of candidate publications is found, triaging process begins. For example, papers that mention serious adverse event should be prior- itized, as these reports need to be submitted in a strict time window. This is often done via another high recall system that matches words such as 'se- rious' and "life threatening' via a lexicon-based approach.
We evaluate the ability of language models (LMs) to fill out the core information of report given a full-text article that is known to describe at least one ADE. We estimate lower bound on human performance to 72.0% F1. Our best model (a fine-tuned FLAN-T5-Large: Chung et al. 2022) attains 59.1% F1, indicating substantial ad- ditional room for improvement while also suggest- ing that models trained on BioDEX are a path to being useful tools for PV workers. Addition- ally, we evaluate the capability of OpenAl's GPT models (text-dan) teeextddavinii-oee 0002, text-davinci-ee 3et-3.5-turbe gpt-4; Brown et al. 2020) but find that they severely struggle with this task, at- taining at most 53.1% F1.
Each resulting publication is investigated by ex- pert PV workers in multi-stage pipeline, which can differ across companies. Typically, the ini- tial flagging of potential ADES is done by non- clinician PV workers. Evidence is flagged and can be mapped to a standardized ontology to introduce uniformity in downstream stages. Subsequently, clinicians review the report and refine the event details before the report is submitted.
In this work, we abstract away the details of this human-based workflow and model the task as tak- ing in biomedical publication and outputting the final pharmacovigilance report. Systems that per- form well this task could go a long way towards automating pharmacovigilance.
Our models can aid drug safety research efforts today. An important use-case for drug safety re- search efficiently finding papers that describe an adverse event with regard to a specific drug or re- action. Conventional search baselines suffer from low precision, since mentioned drugs and reactions are only rarely involved in an adverse event. Our models are specifically trained to extract adverse events, leading to better performance.
Related Work
Biomedical NLP LMs have pushed the frontiers of biomedical NLP. These models generally fol- low the Transformer architecture (Vaswani et al., 2017; Devlin et al., 2018; Radford et al., 2019; Brown et al., 2020; Raffel et al., 2020; Nori et al.,
All our code and data are available as supple- mentary material.

---

2023). LMS, sometimes specifically tailored to- wards the biomedical domain, achieve statc-of-the- art results across a range of biomedical benchmarks (Yasumaga et al., 2022; Luo et al., 2022; Singhal et al., 2022). For example, LMs have achieved single-human performance on PubMedQA (Jin et al., 2019), expert-labeled biomedical question answering task with yes/no/maybe labels. Poten- tially, such models could be useful for PV as well. A key challenge is that PV requires processing en- tire biomedical publications, which PubMedQA does not support but BioDEX does.
cornerstone resource for drug safety research. Pre- vious work has focused on pre-processing FAERS, which can include grounding drug and reaction mentions to medical ontologies and detecting du- plicate reports (Banda et al., 2016; Hauben et al., 2021; Khalcel et al., 2022; Kreimeyer et al., 2022; Hung et al., 2022). In contrast, BioDEX is focused improving the process of entering drug safety reports into FAERS, starting from the biomedical literature.
Xu and Wang (2014) combine both FAERS and biomedical literature for enhanced drug safety sig- nal mining. We go one step further and explicitly link reports from FAERS with their originating doc- uments, which allows us to create document-level drug event extraction task.
Recently, Zhao et al. (2022) introduced PMC- Patients, large-scale dataset for patient-to-patien patient-to-article retrieval built on top of PubMed. BioDEX can be seen as complement- ing this effort; instead of retrieving relevant papers, BioDEX aims to extract structured patient informa- tion from biomedical publications for pharmacovig- ilance purposes. Both the extraction of the infor- mation as well as the retrieval of relevant articles are highly relevant for Evidence-Bass Medicine (EBM; Sackett 1997) and pharmacovigilance
4
The BioDEX Dataset
4.1 Dataset Description
Each entry of BioDEX consists one article and a list of associated reports. Articles and reports both contain many different features and metadata. In this section we limit ourselves to discussing only the most prominent features of our dataset. A full enumeration of all fields is given in Appendix A (for reports) and Appendix B (for articles).
Adverse Drug Event Extraction Previous work has focused on ADE extraction. However, almost all ADE datasets utilize some form of span-level an- notations created by medical experts (Wallace et al., 2016; Roberts et al., 2017: Nye et al., 2018: Kang al., 2019; Dirkson al., 2022). This severely limits the scale of these approaches (Basile al., 2019). Nye et al. (2018) annotate an impressive 5000 abstracts but in part utilize non-expert anno- tations. Wallace et al. (2016) combine document- level resource for Randomized Control Trial re- ports with their supporting literature and use distant supervision derive pseudo span-level labels.
4.1.1 PubMed Articles
Each article contains a title and an abstract. If the full-text paper is openly accessible, it is also included together with its corresponding license. Articles also feature lists of keywords, Medical Subject Headings (MeSH; Lipscomb 2000), and a list of chemical substances mentioned in the publi- cation.
The abstract and article metadata was parsed from the Medline distribution (NLM, 2021) using the pubmed-parser package (Achakulvisut et al.. 2020). If available, the full-text paper was pulled from PubMed Central Open Access Subset (NLM, 2003), using their provided API.'
BioDEX relies on the historical output of safety reporting in the U.S. Thus, it is orders of magnitude larger than these resources without requiring any additional expert labels, and it can automatically be expanded over time when new reports become available. This grounding in historical data entails that BioDEX closely matches the real-world clin- ical and regulatory task of PV. In addition, since we consider adverse drug event extraction at the document-level, we circumvent the need for span- level labels.
4.1.2 Drug Safety Reports
A report contains mennically-relevan information about the described patient in the form of reported patient biological sex, weight, age group, and the age which the event first occurred. Not all in- formation is always present in the reports; this de- pends on what exactly the authors described in their article.
FDA Adverse Event Reporting System The FDA Adverse Event Reporting System (FAERS; Food and Drug Administration 2017) is used as a
'https://www ncbi nlm.mmmmm..mmmmmm....mmm nih, wihh.go/pmc//tols// apenftlist/

---

15,000
Chemstherapy
Report
12,500
Pregnancy
10,000
Reuximab
Immunotharapy
abstract
Children
7,500
Immunosuppresaion
Treatment
5,000
Multiple Myeloma
Rhoumatold Arthitts
2,500
full text
Toxicity
Broast Cancer
'15
Multiple Sclerosis
Epilopsy
Figure 3: Number of BioDEX abstracts and full-text papers published over time. Articles published before 2000 are not visualized, there are only 1,519 of them.
Safety
Methetrexate
Sars-Cov-2
Stroke
Lymphoma
Bevacizumab
Each report features list of drugs, each with their own set fields. Every drug consists one active ingredient. available, the drug may feature additional details such as the product name of the drug, the drug administration route, the (cumula- tive) dosage taken, the action taken with this drug (c.g., dose increased), and whether the drug was considered potential cause of the adverse reaction by the authors not. If provided in the article, the reports can even describe the exact lot number of the drug product taken by the patient.
Kichey Transplantation
Systamic Lupus Erythematosus
Acuta Kidney Injury
Fibrilation
Anticoaguiatton Infixiimbb
Figure 4: Number of occurrences for the 30 most fre- quent keywords BioDEX publications.
percentile:
25th
50th
75
abstract length
Each report also features list of reactions. Each reaction is characterized by entry from the stan- dardized MedDRA ontology (Medical Dictionary for Regulatory Activities; Brown al. 1999), as well as a field describing the outcome (e.g., recov- ered, recovering, fatal).
# characters
825
1,263
1,679
# tokens full-text length
177
275
383
# characters
14,801
19,935
29.531
# tokens
3,761
5,152
7,890
Table I: Abstract and full-text length percentiles of
4.2 Dataset Analysis
BioDEX in number of characters and tokens. Tokeniza- tion done with the OpenAl's tiktoken package, using the vocabulary of the text-davinci--ne model.
BioDEX features articles published between 1968 and 2022, with stark increase in articles from 2013 onwards, corresponding to new PV-related legislation in Europe in 2012 (Fornasier et al., 2018). Figure displays the article distribution starting from 2000, The associated reports all orig- inate from a period between 2012 and 2022.
ure illustrates from which countries the reports originated. Some regions are underrepresented, in- dicating avenue for future work.
Not all report attributes are strietly required and thus show up across BioDEX varying frequencies. For example, the patient sex attribute present in 74.5% reports, while patient age group is only present in 17.9% of reports. Appendix C outlines all attribute frequen- cies.
BioDEX covers broad range of topics. In to- tal 55,951 unique article keywords included. Figure shows the most prominent ones.
The median full-text paper in BioDEX is about 20k characters long. Table displays the quartiles for both the abstract and full-text length in number of characters and tokens. We note that the aver- age full-text paper is much longer than the context window used in many present-day LMs.
4.3 Dataset Creation
BioDEX is created by matching articles parsed from Medline with drug safety reports entered FAERS. To avoid ambiguity, we only consider arti-
While BioDEX rooted in a U.S.-based re- source, other countries are represented as well. Fig-

---

Belgium 1,854
Report occurances (loporshmic stale)
U.S.A. 110,456
Portugal 2,102
19,750
Venezuela
Ghana
Phillopines
India 0,888
Angole
Botswana
Chile
250
Figure 5: Number of drug safety reports in BioDEX originating from given country. Colors follow log scale. A selection of countries are specifically highlighted with their exact number of drug safety reports annotated.
cles with unique PubMed identifier and unique title. Only reports containing an explicit reference supporting paper are considered.
2. Patientsex: The reported biological sex of the patient. for unknown, for male, for female.
Unfortunately, this reference to the supporting literature is not structured. We parse the article title out of this unstructured reference. If find title that exactly matches a title in our of articles. we enter both the article and associated report in BioDEX. Otherwise, we drop the report.
3. Drugs: The set of all active substance names of the drugs discussed in the report. For example: azathioprine, infliximab, mesalamine, prednisolone.
4. Reactions: The set of all reac- tion terms discussed in the report. For example: Epstein-Barr virus infection reactivation, Idiopathic interstitial pneumonia.
When creating BioDEX, we prioritized creating precisiciiioon matches. Future work could expand the size of our dataset by considering phisticated procedure match articles and reports -e.g., by using metadata other than the article titles.
For the Report-Extraction task, we only sider reports where all these attributes are present. While BioDEX reports contain more detailed at- tributes as well, we leave predicting these details as future work.
5 Task and Metrics
In this work, we focus the task of predicting the core information of report given full-text paper, which we call Report-Extraction. Accurate and autonomous extraction of drug safety reports can have a large impact on PV by increasing the quality of safety signals and decreasing the time required to surface new signals.
5.2 The Report-Extracte Dataset
We create new dataset specifically for this task by manipulating BioDEX. First, restrict ourselves to only articles with full-text paper available. Ad- ditionally, we only consider articles with less than 10 associated reports, since we found that the few articles with more were often very large survey pa- pers discussing a broad range of adverse effects. If multiple reports per article are available, one report is sampled to act as the gold label of our task. We leave the task of predicting variable number of reports per publication, which BioDEX supports, as future work.
5.1 Core Reports
We reduce the complexity of the detailed reports by only predicting the 4 core attributes:
1. Serious: The scriousness of the adverse event. Equal to if the adverse event resulted in death, life threatening condition, hospital- ization, disability, congenital anomaly, or any other serious condition. none of the above occurred, equal to 2.
We divide the data into train/test splits by taking articles published before 2021 as training in- stances and the rest as testing instances. This adds

---

article date
can be used to estimate lower-bound on the Inter- Annotator Agreement (IAA).
split
size
min.
max.
For every article with multiple reports available, we randomly validate one core report against an- other. Using our Report report-Extraction Performance, this produces IAA score of 72.04% F1.
train
9,624 (62%)
1990
2020
validation
2,407 (15%)
1985
2020
test
3,628 (23%)
2021
2022
As random baseline, consider validating a core report against another report uniformly sam- pled from the entire dataset. This produces an Fl of 24.28% and serves as lower bar for non-triviat performance. This score is significantly larger than 0% mainly due to high random guessing accuracy on the serious and patientsed attributes.
Table 2: Sizes of the Eoprrt-Exttaction splits and corre- sponding ranges of article publish dates.
a temporal generalization component to our task. Finally, we create val idation split by uniformly holding-out 20% of the training samples.
We deliberately created test scenario that sim- ulates the real-world situation these models will face: they will have been developed on data up to a specific time point and then, by necessity, they will encounter reports from later time periods. is vital that we study how models behave in this challenging scenario.
Experiments and Results
Motivated by the recent success of LLMs, we choose to model Report-Extraction as a sequence- to-sequence problem. Given a full-text paper as input, we train models to predict the core re- port in a stringified format, such as "serious: patientsex: 1 drugs: azathioprine, infliximab, mesalamine, prednisolone reactions: epstein-barr virus infection reactivation, idiopathic interstitial pneumonia".
The resulting dataset sizes and article dates given in Table We distribute this subset of our dataset structured format as well.
5.3 Report-Extraction Performance
To estimate performance, we need to define sim- ilarity metric between two reports. This is achieved by taking weighted average over the 4 at- tribute similarities.? 2 For serious and patientsex, the similarity is the conventional classification ac- curacy. For drugs and reactions, the set preci- sion and recall metrics are used. Every predicted drug or reaction in these sets is either correct or wrong, based on an exact string match. We report the average of all the report-level Fl scores, cal- culated using the weighted attribute precision and recall scores. This is strict metric, since multi- ple correct ways of describing the same drug or reaction are not taken into account. future work, medical ontologies can be used to normalize drug and reaction mentions to create more lenient met- rics.
We report validation results for all models considered. Only the best models are subsequently evaluated on the test split.
6.1 Few-shot In-context Learning
First, we evaluate the few-shot in in-context learning performance
dataset achieved by OpenAl's text-davinci-002.
text-davinci-000 gpt-3. 5-turbo, and gpt-4 models (Brown et al., 2020). A key limitation of in-context learning is that both the few-shot demonstrations and the actual input need to fit in the same context window. Given the average length of our inputs, the context window becomes constraint: most of the full-text papers do not fit the next-davinci-O0 context window of 4,096 tokens (see Table 1). Thus, we aim to maximally utilize the available context window. Given a fixed natural description prompt of the task (see Appendix for the full prompt), we investigate the trade-off between the number of tokens dedicated to in-context demon- strations and the number of tokens of the input pa- per. Since it is prohibitive to include entire papers,
5.4 Inter-Annotator Agreement
A single article can be linked to multiple reports. Often, these reports comment on the same under- lying adverse event but were submitted by inde- pendent people or institutions. These situations
'The weight factors are 1/6 for the serious and patientsex scores, and 1/3 for the drugs and reactions scores.
'Different views of the Report-Extraction task are possi- ble. For example, it could he defined as a set of (multi-lahel) classification tasks.

---

input paper tokens (avg)
REP
Parse percentage
# generation tokens (avg)
context tokens (avg)
model
demos
(% F1)
text-davinci-002
5
2347
44.15
100
41
3871
text-davinci-002
7
1669
45.78
3956
text-davinci-002
10
845
45.91
98
43
3965
text-davinci-002
12
385
45.80
98
36
3968
text-davinci-003
6
2070
48.13
100
50
3968
text-davinci-008
7
1669
50,45
99
47
3968
text-davinci--n0
8
1440
47.16
100
54
3959
3pt-3.5-turbb-031
7
1710
30.55
76
29
3955
gpt-4-0312
7
1710
51.71
100
43
3954
(4k context)
gpt-4-0312
7
3638
49.69
100
43
5925
gpt-4-0312
14
3151
48.00
100
38
7215
Table 3: Few-shot in-context learning results on the BioDEX Report-Extraction Extraction task (validation split). For each model, vary the combination of number of few-shot demos and the amount of tokens dedicated towards the input paper. REP denotes the Report-Ein ExportEExttction Performance. Parse percentage denotes the frequency of times the model formed well-structured generation. Models were evaluated on first 100 examples validation.
we use only the abstracts for the demonstrations and truncate the full-text input paper maximally fill the context window. We use the DSP package to implement all experiments (Khattab et al., 2022). Table 3 summarizes the experiments. We find the optimal trade-off to consist of 7 abstract-level demonstrations, which results in incorporating around 1 1,660 tokens of the final paper.
examples.
6.2 Fine-tuned Models
We further experiment with fine-tuning our own specialized models for the Report-Extraction task. We consider the suite of FLAN-T5 models (Chung et al., 2022), which are based on the encoder- decoder Transformer architecture (Vaswani et al., 2017). Table 4 summarizes the experiments.
On the validation set, this achieves a per- formance of 45.78% F1 for text-davinci-O0 -002. 50.44% F1 for text-davinci -003, and 51.71% Fl for gpt-4.+ While this performance is certainly non-trivial, especially given only 7 labeled exam- ples, it is far from expert-level. We explored using the context window of gpt-4 beyond 4096 tokens, but found no improvements when further scaling the amount of demonstrations or the amount of paper input tokens. The cheaper gpt-3. 5-turbo model performs sub-par and struggles to properly format its generations.
The most successful run consisted of fine-tuning FLAN-T5-Large on source context window of 2048 tokens and a target context window 256 tokens. This achieves 62.28% on validation
Given a fixed context window of 512 or 1,024 to- kens, the larger FLAN-T5-XL model performs better. For given model size, longer context windows im- prove performance. We leave the further scaling of model sizes and context windows as future work.
The best text-davinci-003 and gpt-4-0312 models achieve 50.60% F1 and 53.1 1% F1 on test respectively. We conclude that, at least in our stan- dard use of the methods, few-shot learning achieves non-trivial but unsatisfactory performance our Report-Exttraeet task. See Appendix E for 10
Models were trained for up to 5 epochs with a starting learning rate of 0.0001, linearly sched- uled. We used the Adafactor optimizer with default hyperparameters (Shazeer and Stern, 2018).
We used greedy decoding to form the genera- tions. Beam search decoding, with beam width of 8, did not improve performance. Evaluating FLAN-TS-Large with 2048 source and 256 target tokens on the test split results in 59.1 % F1.
'All default hyperparameter settings were used for the OpenAl API calls. To save on costs, we validate and test on the first 100 examples of the respective splits.

---

# source tokens
# target tokens
REP
Parse Percentage
# generation tokens (avg)
model
(% F1)
FLAN-T5-Large
2048
256
62.28
98.96
59.60
FLAN-T5-Large
2048
128
61.39
99.58
52.96
FLAN-T5-Large
1024
256
55.88
96.05
75.08
FLAN-T5-Large 512
128
50.92
94.72
53.60
FLAN-T5-XL 1024
256
58.32
99.46
48.82
FLAN-T5-XL
512
256
53.19
97.55
64.69
Table 4: Fine-tuning results on the BioDEX Report-Extraction task (val caliontion split). REP denotes the Report- Extraction Performance. Parse percentage denotes the frequency weell-stuucture model outputs.
FLAN-T5-Large
gpt-4-0312 % F1
reaction classificatiin (96
% F1
50
70
80
baseline
nan-tS-large
ser cerioussess
92.90
94.00
(1753)
drug ineffective (232)
patientsex
92.65
93.00
(175)
drugs 61.83
50.99
product use unspproved indication (94)
condition aggravated (71)
reactions
34.13
12.62
toxicity to various egents (65)
maternal exposure during pregnancy (38)
drug ineffective for unapproved indication (34)
Table 5: Per attribute performance of the best FLAN-T5-Large and gpt-4-8312 runs on the validation split.
neuropathy peripheral (31)
dyspncea
pyrexia (55)
anaemia (45)
pneumonia (33)
diarmoee (56)
6.3 Attribute-level Performance
headache (30)
disease progression (43)
Fine-tuned models have better Report-Extraction Performance compared in-context learning mod- els. In table Table 5, we break down the per- formance of the best fine-tuned and in-context model on the validation split per predicted at- tribute. FLAN-T5-Large and gpt-4 attain similar performance when predicting seriousness and patientsex. However, FLAN-T5-Large outper- forms gpt-4 predicting the correct drugs and reactions. While these two attributes are hard to predict in general, we hypothesize the fine-tuned model attains better performance because it was able to learn from more reports during training, allowing it to better captured the specific termi- nology used in these reports as well as their prior distribution.
vomiting (51)
drug resistance (36)
nausea (80)
fatigue (39)
treatment failure (42)
hypotension
sepele (4B)
thrombocytopenia (60)
respinatory failure (40)
neutropenia (66)
acute kidney injury (76)
drug interaction (70)
pancylopenia (29)
febrile neutropenia (39)
drug drug-induced liver injury
Figure 6: Reaction classification performance across the 30 most frequent reactions in the BioDEX validatio set. Baseline performance lighter color, FLAN-T5 in darker color. Support in parentheses. Average per- formance bold. Reactions are sorted by baseline performance.
7 Improving Pharmacovigilance
high cost of errors, human experts will remain vital for effective solutions in the years to come.
Our primary goal is to improve the scalability and accuracy of PV using NLP. The above exper- iments highlighted the potential for LMs to tonomously fill in ADE reports. However, fully au- tonomous drug event reporting systems are unlikely to achieve widespread adoption today. Mainly be- cause the challenging nature of this task and the
However, models still deliver tangible value by augmenting existing expert-based work- flows. Given the vast number of biomedical papers published, it is inereasingly impractical to thor- oughly vet every candidate publication (as is cur- rently being done). Drug manufacturers are looking

---

more efficiently triage the literature to prioritize efforts, as this minimizes risk from regulatory fines and maximizes public safery.
der et al., 2021; Zhang et al., 2020). Thus, the re- sources put forth in this paper should not be naively applied to automate safety reporting. Rather, we suggest that these systems could be integrated as an additional tool at the disposal of PV workers, and we encourage careful study of how to best em- power these experts to work more efficiently and effectively.
Such a triaging system is typically based on a naive lookup: finding all papers that match a drug name is likely to find all papers where that drug engages in an adverse event. Unfortunately, such a system has low precision, causing human effort to be wasted investigating irrelevant papers.
Different countries can face different health is- sues. When we develop biomedical language sys- tems, it is important they work for everyone. Some countries are underrepresented in our dataset. Sub- sequent data collection efforts should focus on these countries to alleviate this issue. Addition- ally, confounders such as patient age and patient sex need to be taken into account to ensure satisfac- tory performance across different demographics.
We find that our model predictions achieve a higher performance at finding adverse events con- cerning specific reactions, compared to the lookup baseline. We measure this through the macro av- erage Fl score on the binary classification task of predicting per paper if a reaction was part of an adverse events. Figure 6 shows the results for the 30 most frequent reactions in the validation split. High-recall baselines still have a valuable place in the PV review process, but our system could be used to more efficiently prioritize effort. Ap- pendix F describes the same experiment for drugs. Future work could utilize all details of BioDEX reports or incorporate the 65k abstract-level dat- apoints during training to further improve utility for PV. For example, BioDEX would support fine- tuning a question answering model for PV.
Acknowledgements
We thank our anonymous reviewers for their in- sightful comments and suggestions. KD gratefully acknowledges funding from the FWO Fundamental Research PhD Fellowship (1 (11632223N). KZ grate- fully acknowledges funding from the Innovation Fund Denmark (IFD) through the Grand Solution project Hospital @ Night.
8 Conclusion
References
We introduced BioDEX, a large-scale document- level Biomedical adverse Drug Event Extraction dataset. BioDEX covers an important and challeng- ing real-world task: extracting detailed drug safety reports from full-text biomedical publications. We find that LLMs struggle to get traction on this task using in-context learning. Fine-tuned models more successful, but expert-level performance re- mains elusive. Nevertheless, our models have the potential to make drug safety research more effi- cient, and we demonstrated their utility in a conven- tional PV use-case. We release all data and models. We hope that BioDEX stimulates new research in the high-impact area of drug safety monitoring.
Titipat Achakulvisut, Daniel Acuna, and Konrad Ko- rding. 2020. Pubmed parser: A python parser for pubmed open-access xml subset and medline xml dataset xml dataset. Journal of Open Source Soft- ware, 5(46):1979.
Yasser M Alatawi and Richard A Hansen. 2017. Empir- ical estimation of reporttep in the us food and drug administration adverse event reporting system (faers). Expert opinion on drug safery, 16(7):761- 767.
Juan M Banda, Lee Evans, Rami Vanguri, Nicholas P Tatonetti, Patrick B Ryan, and Nigam H Shah. 2016. A curated and standardized adverse drug event re- source to accelerate drug safety research. Scientific data, 3(1):1111.
9 Limitations and Ethical Considerations
Anna O Basile, Alexandre Yahi, and Nicholas P Tatonetti. 2019. Artificial intelligence for drug toxic- ity and safety. Trends in pharmacological sciences, 40(9):624 635.
Drug Safety Reporting is an important real-world task. Submitting faulty reports consistently un- derreporting specific adverse events could have pro- found impacts for public safety. LMS are known to make mistakes and fabricate evidence, they are al- most invariably biased towards specific predictions, and they can be prone to adversarial attacks (Ben-
Emily M Bender, Timmit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM confer- ence on fairness, accountability, and transparency. pages 610-623.

---

Elliot G Brown, Louise Wood, and Sue Wood, 1999. The medical dictionary for regulatory activities (med- dra). Drug safery, 20(2): 10-17.0091117
fda's publicly available data. Journal of the American Medical Informatics Association, 23(3):966-44) 600.
Mohammad Ali Khaleel, Amer Hayat Khan, Siti Maisharah Sheikh Ghadzi, Azreen Syazril Adnan, and Qasem M Abdallah. 2022. A standardized dataset of a spontaneous adverse event reporting sys- tem. Healtheare, 10(3):420.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-search predict: Composing retrieval and language mod- cls for knowleedg--nntesiie nlp. arXiv preprint arXiv:2212.1402:
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction finetuuedd language models. arXiv preprint arXiv: 22 (0.1.... 1446.
Kory Kreimeyer, Oanh Dang, Jonathan Spiker, Paula Gish, Jessica Weintraub, Eilcen Wu, Robert Ball, and Taxiarchis Botsis. 2022. Increased confidence in deduplication of drug safety reports with natural language processing of narratives at the us food and drug administration. Frontiers in Drug Safety and Regulation,
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
Anne Dirkson, Suzan Verberne, Gerard van Oort-
merssen, Hans Gelderblom, and Wessel Kraaij. 2022.
How do others cope? extracting coping strategies for adverse drug events from social media. Journal of Biomedical Informatics, page 104228.
Carolyn E Lipscomb. 2000. Medical subject headings (mesh). Bulletin of the Medical Library Association, 88(3):265.
Food and Drug Administration. 2017. FDA AEs reporting system (FAERS) public dashboard. https: //ww. fda. gov/Drugs/ aawaae/eeeeooooaeeeeeeeaaaatrrre
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6).
Surveillance/Adveeeeeerrrr htm. Accessed May 16, 2023.
70093.
Giulia Fornasier, Sara Francescon, Roberto Leone, and Paolo Baldo. 2018. An historical overview over Pharmacovigilance. International journal of clini- cal pharmacy, 40:744-747.
NLM. 2003. PMC Open Access Subset. Internet. Cited 2023 May 23.
NLM. 2021. Medline. Internet. Cited 2023 May 23.
Manfred Hauben, Chen Zou, Steve Bright, and Eric Hung. 2021. More extreme duplication the us fda faers database and a suggested check point for disproportionallty analysis. Pharnacoepidemiology and drug safety, 30(8):1440-144
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabili- ties of gpt-4 on medical challenge problems. arXiv preprint arXiv: 13375.
Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang. lain J Marshall, Ani Nenkova, and Byron C Wal- lace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In Pro- ceedings of the conference. Association for Computa- tional Linguistics. Meeting, volume 2018, page 197. NIH Public Access.
Eric Hung, Manfred Hauben, Henry Essex, Chen Zou, and Steve Bright. 2022. More extreme duplication in FDA adverse event reporting system detected by literature reference normalization and fuzzy string matching. Pharmacoepiddmmooloyy and Drug Safety.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909, 9999066166.
Alee Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 18):9
Tian Kang, Shirui Zou, and Chunhua Weng. 2019. Pre- training recognize pico elements from randomized controlled trial literature. Studies health technol- ogy and informatics, 264:
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research. 21(1):5455-55)
Taha A Kass-Hout, Zhiheng Xu, Matthew Mohebbi, Hans Nelsen, Adam Baker, Jonathan Levine, Elaine Johanson, and Roselie A Bright. 2016. Openfda: an innovative platform providing access to wealth of

---

Kirk Roberts, Dina Semner-Fushman, and Joseph M Tonning. 2017. Overview the tac 2017 adverse reaction extraction from drug labels track. In TAC.
David L Sackett, 1997. Evidence-based medicine. In Seminars Perinatology, volume 21, pages 3-5, El- sevier.
Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. International Conference on Machine Learning. pages 4596-4604. PMLR.
Karan Singhal, Shekoofch Azizi, Tao Tu, S Sara Mah- davi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13136
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Éukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Byron Wallace, Joel Kuiper, Aakash Sharma, Mingxi Zhu, and lain J Marshall. 2016. Extracting pico sen- tences from clinical trial reports using supervised distant supervision. The Journal of Machine Learn- ing Research, 17(1):4572-455
Rong Xu and QuanQiu Wang. 2014. Large-scale com- bining signals from both biomedical literature and the fda adverse event reporting system (faers) to improve post-marketing drug safety signal detection. BMC bioinformatics, 15(1):1-10.
Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. Linkbert: Pretraining language models with document links. arXiv preprint 2003. 15827.
Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. 2020. Adversarial attacks on deep- learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1-11.
Zhengyun Zhao, Qiao Jin, and Sheng Yu. 2022. Pmc- patients: A large-scale dataset of patient notes and relations extracted from reports in pubmed cen- tral. arXiv preprint 1387

---

A BioDEX Report Schema
The following paragraph enumerates the fields present in the drug safety reports and lists possible values if defined. It was adapted from the official description' of the FAERS fields found OpenFDA (Kass-Hout et al., 2016). The dot in field names denotes nesting.
companynumb Identifier for the company providing the report. This is self-assigned. fulfillepeddiecciteriaa Identifies expedited reports (those that were processed within 15 days). Possible values: 1: True, 2: False
occurcountry The name of the country where the event occurred. Possible values: name: Country codes,
link:
Actions taken with the drug. Possible values: 1: Drug withdrawn, 2: Dose reduced, 3: Dose increased, 4: Dose not changed, 5: Unknown, 6: Not applicable
wtw.:aaddddrgetess..cvvisille wtw.:a/bbtteettttteeeeaaaeeeneme Product active ingredient, which may be
different than other drug identifiers (when provided).
Dechallenge outcome information the event abated after product use stopped or the dose was reduced. Only present when this was attempted and the data was provided. Possible values: I: Yes, 2: No, 3: Does not apply
The drug's route of administration. Possible values: 001: Auricular (otic), 002: Buccal, 003: Cutancous, 004: Dental, 005: Endocervical, 006: Endosinusial, 007:
Endotracheal, 008: Epidural, 009: Extra-amniotic, 010: Hemodialysis, 011: Intra corpus cavemosuum
012: Intra-amniotic, 013: Intra-arterial, 014: Intra-articular, 015: Intra-uterine, 016: Intracardiac,
017: Intracavernous, 018: Intracerebral, 019: Intracervical, 020: Intracisternal, 021: Intracorneal, 022:
Intracoronary, 023: Intradermal, 024: Intradiscal (intraspinal), 025: Intrahepatic, 026: Intralesional,
027: Intralymphatic, 028: Intramedullar (bone marrow), 029: Intrameningeal, 030: Intramuscular,
031: Intraocular, 032: Intrapericardial, 033: Intraperitoneal, 034: Intrapleural, 035: Intrasynovial,
036: Intratumor, 037: Intrathecal, 038: Intrathoracic, 039: Intratracheal, 040: Intravenous bolus, 041:
Intravenous drip, 042: Intravenous (not otherwise specified), 043: Intravesical, 044: lontophoresis, 045:
Nasal, 046: Occlusive dressing technique, 047: Ophthalmic, 048: Oral, 049: Oropharingeal, 050: Other, 051: Parenteral, 052: Periarticular, 053: Perineural, 054: Rectal, 055: Respiratory (inhalation), 056:
Retrobulbar, 057: Sunconjunctival, 058: Subcutancous, 059: Subdermal, 060: Sublingual, 061: Topical,
062: Transdermal, 063: Transmammary, 064: Transplacental, 065: Unknown, 066: Urethral, 067: Vaginal
htt.://rwgg.tnnssdggauuuuriiiinn if provided.
Drug authorization or application number (NDA or ANDA),
patient.dr werpeett.rug..iie ittess.dduubaaahhuunbb Drug product lot number, if provided.
Reported role of the drug in the adverse event report. These values are not validated by FDA. Possible values: I: Suspect (the drug was considered by the reporter to be the cause), 2: Concomitant (the drug was reported as being taken along with the suspect drug), 3: Interacting (the drug was considered by the reporter to have interacted with the suspect drug) art.uutteem..duuguuuuutttvvooose experienced, if provided.
weliient.duug..o wrw.eees.dduuuhaaaeeeiiaaattteee
The cumulative dose taken until the first reaction was
The unit for 'dngaamullaivedosagauun Possible values: 001: kg (kilograms), 002: (grams), 003: mg (milligrams), 004: μg (micrograms) The drug's dosage form. There is no standard, but values may include terms like 'tablet' 'solution for injection'.
patient.drug.items. www..addddngiiie.ddddddgssseeee Additional detail about the dosage taken. Frequently unknown, but occasionally including information like a brief textual description of the schedule of administration. matient.drug.oom wrw...item..duugndddaat Date the patient stopped taking the drug.
srriint.rmngiiinndddggededeeooo Encoding format of the field 'drugenddateformaa:: Always set '102' (YYYYMMDD).
Phttps: fda. http:///wwt....eee...//////////a

---

dettent.druggilem.dduuiiiinnnn denddidginiininittiiinn Indication for the drug's use.
The unit for the interval in the field 'druginterval-
dosageumimumb. Possible values: 801: Year, 802: Month, 803: Week, 804: Day, 805: Hour, 806: Minute, 807: Trimester, 810: Cyclical, 811: Trimester, 812: As necessary, 813: Total
Number of units in the field 'drugintervaldooseeedef-
nition'.
Whether the reaction occured after readministration of the drug. Possible values: Yes, No, 3: Unknown
patten..drug.ieenssrrggrrerrrr
ddngrecuractiomeddraaesss The version of MedDRA from which the term in 'drugrecu- raction' is drawn.
patient.drug.iteee
hergggiiennnndnggspppaaelleaenn The number of separate doses that were administered. matient.drug.iteee
frrnggiitemm.duuusfattaaatte Date the patient began taking the drug. Encoding format of the field 'drugstartdate'. Always set to
'102' (YYYYMMDD).
http://cwwintnssdddsssssrrrrrrna The number portion of dosage; when combined with ddrugstuucuuredossguuint the complete dosage information is represented. For example, #300* in '300 mg'.
The unit for the field ddrusstucturedosseemmm) For example, *mg* in '300 mg'. Possible values: 001: kg (kilograms), 002: g (grams), 003: mg (milligrams), 004: μg (micrograms)
which the patient was taking the drug. The interval of the field "dmngreatmeenduraiionnit for
None Possible values: 801: Year, 802: Month, 803:
Week, 804: Day, 805: Hour, 806: Minute
Drug name. This may be the valid trade name of the product (such 'ADVIL' or 'LEVE') the generie name (such 'IBUPROFEN') This field is not systematically normalized. It may contain misspellings idiosyncratic descriptions of drugs, such combination products such those used for birth control.
patient.pttientagegrouu Populated with Patient Age Group code. Possible values: Neonate, 2: Infant, 3: Child, 4: Adolescent, 5: Adult, 6: Elderly patient.paatientonsstage Age of the patient when the event first occured.
patientpaatentoosstagguntt The unit for the interval in the field 'patientonsetage. Possible values: 800:
Decade, 801: Year, 802: Month, 803: Week, 804: Day, 805: Hour
patient.patientsexx The sex of the patient. Possible values: O: Unknown, Male, 2: Female
matient.pattennteeiht The patient weight, in kg (kilograms). ieracciion.iemssrraaiiiiioooooei
Patient reaction, as MedDRA term. Note that these terms are
encoded in British English. For instance, diarrhea is spelled 'diarrohea'. MedDRA a standardized medi- cal terminology. Possible values: name: McdDRA, link: http://www.fitago/ooooootttttttd aanncuureeProdncttabbbeinggggggo 12030..hmm60000mmm
The version of MedDRA from which the term in
traactoomeddrrppt is drawn.
Outcome of the reaction in 'reactionmeddrrapt' the time of last observation. Possible values: Recovered/reeolledd 2: Recovering/resolving. 3: Not recovered/not resolved, 4: Recovereddresolvedd with sequelae (consequent health issues), 5: Fatal, 6: Unknown
paatenttreaction.teess..e..m
antlerttsmmmaaa.aaaraaiientlleee Populated with Case Event Date, when available; does 'NOT include Case Narrative.
Populated with the Literature Reference information, when available. caaerrssoouue-quaaffiiiittt Category of individual who submitted the report. Possible values: 1: Physician, Pharmacist, 3: Other health professional, 4: Lawyer, 5: Consumer or non-health professional prinarrsuureeepporrccountt Country from which the report was submitted.

---

primarysoureecountry Country of the reporter of the event. Possible values; name: Country codes, link:
http://data..kth.orr/ddaad wttp:///aaaa//o////oooottttoot/e
receiptdate Date that the mosst recent_ information in the report was received by FDA.
receivedate Date that the report was _iistt_ received by FDA. If this report has multiple versions, this will be the date the first version was received by FDA. receiver.reciierorggniaall
Name of the organization receiving the report. Because FDA received the report, the value is always 'FDA'.
receiver.receivertype The type of organization receiving the report. The value, '6', only specified it is 'other', otherwise it left blank. Possible values: Other
reporttype Code indicating the circumstances under which the report was generated. Possible values: 1:
Spontancous, 2: Report from study, 3: Other, 4: Not available to sender (unknown)
safetyreportid The 8-digit Safety Report ID number, also known as the case report number case ID. The first digits (before the hyphen) identify individual report and the last digit (after the hyphen) is checksum. This field can be used identify or find specific adverse event report.
safetyreportversion The version number of the "safetyreportid" Multiple versions of the same report may exist, it is generally best to only count the latest report and disregard others. openIiDA will only return the latest version a report.
sender.senerorgg cenderseenerrorganizztion Name of the organization sending the report. Because FDA is providing these reports to you, the value is always 'FDA-Public Use.'
sender.sendertppe The name of the organization sending the report. Because FDA is providing these reports you, the value is always '2'. Possible values: 2: Regulatory authority
serious Seriousness of the adverse event. Possible values: 1: The adverse event resulted in death, a life threatening condition, hospitalization, disability, congenital anomaly, or other serious condition, 2: The adverse event did not result any of the above
veriousnesseongeniaaanmoo This value is '1' if the adverse event resulted in a congenital anomaly, and absent otherwise.
seriousnessdeath This value is '1' if the adverse event resulted in death, and absent otherwise. seriousnessdisabling This value is '1' if the adverse event resulted in disability, and absent otherwise. seriousnesshospialliatoon This value is '1' the adverse event resulted a hospitalization, and absent otherwise.
seriousneslifeehhrateennng This value is '1' if the adverse event resulted in a life threatening condition, and absent otherwise.
seriousnessother This value is '1' if the adverse event resulted in some other serious condition, and absent otherwise.
transmissiondate Date that the record was created. This may be earlier than the date the record was received by the FDA.
B BioDEX Article Schema
The following paragraph enumerates the fields present in the articles. It was adapted from the pubmed-parser (Achakulvisut paciaaaaaaeercaaaaluiiiiiii et al., 2020) documentation.
title Title of the article.
pmid PubMed ID.
issue The Issue of the journal.
pages Pages of the article in the journal publication.
abstract Abstract of the article.
fulltext The full text associated with the article from the PubMed Central Open Access Subset. if available. fulltext license The license associated with the full text paper from the PubMed Central Open Access Subset. if available.
journal Journal of the given paper. authors Authors, each separated by 1:'

---

affiliations The affiliations of the authors.
pubdate Publication date. Defaults to year information only. doi DOI.
medline ta Abbreviation of the journal name.
nim_ ham__nqqu____e _ NLM unique identification.
issn linking ISSN linkage, typically use to link with Web of Science dataset.
country Country extracted from journal information field.
mesh terms List of McSH terms with corresponding McSH ID, each separated by : e.g.
'D000161:Acoussie Stimulation; D000328:Adul; ..."
publication_ typess List of publication type list each separated by ';' e.g. 'D016428:Joumal Article'. chemical list List of chemical terms, each separated by ';'
keywords List of keywords, each separated by ";"
reference String of PMID each separated by ";' or list of references made to the article.
delete Boolean, 'False' means paper got updated so you might have two. pmc PubMed Central ID.
other id Other IDs found, each separated by "
C BioDEX Report Attribute Frequencies
Table 6 contains the frequency of occurrence for the report attributes in BioDEX.

---

attribute
frequency (%)
patientagegroup
17.94
patientonsetage
68.94
paticntonsetageuni
68.94
patientsex
74.58
patientweight
4.78
summary
10.07
drugalministtaaiionrouue
74.32
drugbatchnumb
10.94
drugeunulaiveedsagemmbb
0.40
drugcumullaivedosagunni
0.35
drugenddate
2.73
drugenddateformat
2.73
dmgginteraldooageeeeiitioo
13.93
drugintervaldossgeuninmmmb
13.93
dmugreeurreaddinnisttatiio
17.07
dmugsparaatcdosagenuubb
13.77
drugstartdate
6.15
drugstartdateformmat
6.15
drugtreatmentduration
0.64
diggtraatmenddurationnii
0.64
drugrecurrence
0.63
drugdosageform
17.78
drugdosagetext
59.83
dmngstructureedosgeennb
29.33
drugstructuredossgeumii
29.33
drugauthorizatioommnnb
31.98
actiondrug
76.79
drugadditional
41.12
drugindication
81.85
activesubstance
99.32
reactionoutcome
98.10
Table 6: Frequency of occurrence of each attribute per report. Attributes not mentioned have occurrence of 100%.

---

D Few-Shot Prompt
The prompt below is the one used for the in-context learning experiments. No effort was spent on prompt engineering and the demonstrations randomly sampled from the training set.
Few-Shot Prompt:
Read biomedical paper and extract information about the adverse drug event mentioned by the authors. Return serious value ('1' for serious, '2' for not serious), Return a patientsex value ("1' for male, for female). Return a list drugs taken and reactions experienced.
Follow the following format.
Question: $ 5(What adverse drug event was described in the following context?)
Context: S(biomedical paper that describes adverse drug events)
Answer: S the adverse drug event described in the context)
Question: What adverse drug event was described in the following context?
Context: report the of patient with b-cell prolymphocytic leukemia who successfully treated with the novel humanized monoclonal antibody obinutuzumab. this patient was previously treated with the combination of rituximab and bendamustine and had recurrent infusion reactions. her treatment with rituximab and bendamustine was discontinued when she developed disease progression after cycles therapy. she was then treated with obinutuzumab 1000 mg on day of every cycle and chlorambucil 0.5 mg/kg on days and 15 every 28 days to which she had greater tolerability. after 4 cycles of treatment, she had resolution of her clinical symptoms, massive splenomegaly, and normalization of her white blood cell count.
Answer: serious: patientsex: 2 drugs: bendamustine hydrochloride, rituximab reactions: cytopenia, treatment failure
Question: What adverse drug event was described in the following context?
Context: sarcoid associated pulmonary hypertension (saph) is a common complication of sarcoidosis and is associated with poor prognosis. saph can be due to multiple synergistic mechanisms and current therapeutic strategies treat systemic sarcoidosis and pulmonary hypertension separately. several studies have been performed to develop an effective therapy for saph but have been met with mixed results. the ambition trial successfully treated incident patients with pulmonary arterial hypertension (pah) with the upfront combination of ambrisentan and tadalafil; however combination therapy has not yet been studied in patients with saph. here we report a cohort of patients with newly diagnosed saph who were treated with upfront combination therapy per the ambition study protocol. we report three subjects with newly diagnosed saph who were treated with combination ambrisentan and tadalafil. baseline hemodynamics were compared with those from surveillance right heart catheterization while on therapy. mean follow up period was 17 months. each subject demonstrated clinical and hemodynamic improvement with combination therapy. this series is the first to evaluate upfront combination ambrisentan and tadalafil therapy for treatment of newly diagnosed saph. despite the impressive clinical and hemodynamic improvement, the study is limited by its small size and retrospective nature. while these initial results are promising, further work is needed fully evaluate this regimen for treatment of saph. (sarcoidosis vasc diffuse lung dis 2020; 37 (2): 234-238).
Answer: serious: 1 patientsex: drugs: ambrisentan, infliximab, methotrexate, prednisolone, tadalafil reactions: off label use, urosepsis

---

Question: What adverse drug event was described in the following context?
Context: haloperidol is a typical antipsychotic drug. this drug is still widely used in emergency medicine, psychiatry, and general medicine departments. it is mostly used for acute confusional state, psychotic disorders, agitation, delirium, and aggressive behaviour. overdose of haloperidol can cause sudden deaths. cardiopulmonary arrest related to use of haloperidol had been reported in literature as case reports but are very few. no such cases have been reported in india till now. we report case of cardiac arrest due to the use of haloperidol.
Answer: serious:
patientsex: 1 drugs: haloperidol lactate reactions: cardiac arrest, ventricular tachycardia
Question: What adverse drug event was described in the following context?
Context: neonatal nonoliguric hyperkalemia (nohk) is metabolic abnormality that occurs extremely premature nconates at approximately 24 h after birth and is mainly due to the immature functioning the sodium (na+) Upotassimm (k+) pump. magnesium sulfate is frequently used in obstetrical practice to prevent preterm labor and to treat preeclampsia; this medication can also cause hypermagnesemia and hyperkalemia by a mechanism that is different from that of nohk. herein, we report the first case of very early-onset neonatal hyperkalemia induced by maternal hypermagnesemia. neonate born at 32 weeks of gestation developed hyperkalemia (k+ 6.4 mmol/l) h after birth. the neonate's blood potassium concentration reached 7.0 mmol/1 4 after birth, despite good urine output. the neonate and his mother had severe hypermagnesemiia caused by intravenous infusion of magnesium sulfate given for tocolysis due to pre-term labor. the early-onset hyperkalemia may have been caused by the accumulation of potassium ions transported through the placenta, the shift of potassium ions from the intracellular to the extracellular space in the infant due to the malfunctioning of the na+/K+ pump and the inhibition of renal distal tube potassium ion secretion, there is possibility that these mechanisms were induced by maternal and fetal hypernagnesemia after maternal magnesium sulfate administration. because neonatal hyperkalemia poses significant risk for the development of life-thrread tifeathreatnning cardiac arrhythmia, this case highlights the necessity of maternal blood magnesium monitoring during magnesium sulfate administration and neonatal blood potassium monitoring when there is severe maternal hypernagnesemia delivery.
Answer: serious: patientsex: 2 drugs: magnesium sulfate reactions: exposure during pregnancy, bypermagnessermia, hypocalcaemia, hypotonia, product use in unapproved indication
Question: What adverse drug event was described in the following context?
Context: doxycycline and minocycline are tetracyclines with the potential to cause hepatoxicity. although autoimmune-like hepatitis from minocycline is well-described, doxycycline-induced autoimmune hepatitis (diah) has only been deseribed once. we report case of diah with elevated liver enzymes over 5 times the normal upper limit, elevated immunoglobulin g, and high titers of antismooth muscle antibody and antinuclear antibody. by stopping doxycycline, our patient's 's liver enzymes normalized and immunoglobulin g and autoantibody titers rapidly downtrended. as long-term doxycycline therapy becomes more prevalent to treat acne vulgaris and other skin conditions, diah may become more prevalent and recognized.
Answer: serious: patientsex: 2 drugs: doxycycline hyclate reactions: autoimmune hepatitis
Question: What adverse drug event was described in the following context?
Context: oral mucositis, the most common adverse effect of radiotherapy (rt) and/or chemotherapy is observed in almost 97% of patients with head and neck cancer. although several agents like corticosteroids. lidocaine and vitamins are available for its prevention or management, results are often disappointing. here we report on the effects of a topically applied, highly purified natural deoxyribonucleic acid from sturgeon gonads on three cases of moderate to severe oral mucositis in patients with head and neck cancer. three patients who had undergone rt and/or chemotherapy received an oral spray containing sodium salt-based natural deoxyribonucleic acid (pdm) for grade 3 oral mucositis. treatment continued for one month after the end of rt. no patient reported any allergic reactions. rt and chemotherapy were not interrupted and opioid therapy not given to any patient. pain was relieved about 2-3 days after

---

starting treatment and oral mucositis was reduced g2 within one week. outcomes all cases showed topical use of the sodium salt-based pdmn derived from sturgeon gonads was acceptable and safe when used topically for therapeutic and regenerative purposes.pre present results are encouraging and suggest a more in-depth study is warranted on its use in a larger patient cohort with rt-induced oral mucositis. Answer: serious: patientsex: drugs: cisplatin reactions: candida infection, dehydration, pain, stomatitis, weight decreased
Question: What adverse drug event was described in the following context?
Context: background piperacilii//tzzoaactan is commonly used antibiotic for the empirical treatment of severe diabetic foot infections. one of the most feared complications of this drug is the development of pancytopenia. the aim of this study was to determine whether the use of piperacilin/uzzohactam caused any hematological changes in patients admitted with severe diabetes-related foot infections from a specialist multidisciplinary foot clinic. specifically, looking at whether it caused anemia, leukopenia, neutropenia, or thrombocytopenia. methods 1-year retrospective analysis of patients admitted to a tertiary care center for treatment of diabetes-related foot infection using piperacilin//zzoohctaa. hematological indices, urea and electrolytes, and c-reactive protein (crp) were recorded pretreatment, during treatment, and posttreatment. hbale, vitamin b12, folate, thyroid-stimulatin hormone, and free thyroxin were also analyzed exclude any potential confounders as a cause of pancytopenia. results a total of 154 patients were admitted between jamuary 2016 and 31 december 2016 who received piperacilinntazooactann for severe diabetes-related foot infection. on admission, white cell count and crp were raised and fell significantly within the first 48 h. other hematological factors did not change. five patients developed mild pancytopenia, of which three were unexplained. conclusions in this relatively small cohort, pancytopenia did not occur. as such, piperacilinnuuzoaattmm appeared to have a low risk of adverse hematological outcomes and remains the treatment of choice for severe diabetes-related foot infections.
Answer: serious: patientsex: drugs: piperacillin sodiumitazobaactm sodium reactions: haemoglobin decreased, pancytopenia
Question: What adverse drug event was described the following context?
Context: (fulll-tett paper (as many tokens as possible))
Answer:
E Example Outputs
The table below shows the first 10 examples of the validation split. The inputs are truncated after 2.500 characters. The associated simplified report is given as well as the model predictions our best FLAN-T5-Large and GPT-4 models.

---

BioDEX example
(PMID: 28491911) TITLE: Navigating Long- Term Care. ABSTRACT: Americans over age 65 constitute a larger percentage of the population each year: from 14% in 2010 (40 million elderly) to possibly 20% in 2030 (70 million elderly). In 2015, estimated 66 million people provided care the ill, disabled, and elderly in the United States. In 2000, according to the Centers for Disease Control and Prevention (CDC), 15 million Americans used some form of (ong-term care: adult day care, home health, nursing home, or hospice. In all, 13% of people over 85 years old, compared with 1% of those ages 65 to 74, live in nursing homes in the United States. Transitions of care, among these various levels of care, are common: Nursing home to hospital transfer, one of the best-studied transitions, occurs in more than 25% of nursing home residents per year. This article follows one patient through several levels of care. TEXT: Case: AB Mrs. AB is an 84-year-old Caucasian female with history of hypertension, osteoporosis, type 2 diabetes, dyslipidemia, ostcoarthritis, and persistent depression who presents to the office new patient with worsening ambulation: "I'm just not getting around well." The patient lives in small house above the family farm, the side of a mountain. She describes her difficulty as an unsteadiness, and stiffness, in her knees and hips. She has moderate pain her right hip and in her left knee, especially late the day. On clinical examination, AB has reduced internal and external rotation of hips, right side more affected than the left, with some pain to the maneuvers, and widened knees with some tenderness. A Mini-Mental Status Exam (MMSE) is consistent with mild cognitive impairment, with a score of 24. (Generally, scores of 27-30 are normal. 24-26 suggest mild cognitive impairment, 19-23 mild dementia, 10-18 moderate dementia, and <10 severe dementia.) She is taking 17 different medications, listed in the box below. AB has son, Fred, who lives in the main farmhouse below her house, but AB does not get along with him well: He has diagnosis of bipolar, and they argue frequently. Her other son, Rod, lives in Texas, and has recently been diagnosed with leukemia. Rod helps her with medical decisions For example, he helped her pick her current Medicare part D plan. AB also has one surviving brother, 89 years old, but he is rather debilitated. He lives close by her house, but unable to assist her; fact, she assists him-sh... [Truncated]
target flan-t5-large patientsex
scrious
gpt-4
drugs
target
alendronate sodium, amitriptyline, ascorbic acid, celecoxib, chromic chlo- ridelchrmmimm, cinnamon, diltiazem, ginkgo, glucosamine, glyburide, hydroxyzine hydrochloride, metformin hydrochloride, niacin, paroxetine, pioglitazone, simvastatin, st. john's wort
flan-t5-llagg
alendronate sodium, amitriptyline, celecoxib, glimepiride, hydroxyzine palmitate, niacin, paroxetine, simvastatin
gpt-4
diltiazem xr, simvastatin, amitriptyline, paroxetine, st. john's wort, celecoxib, met- formin, alendronate, glyburide xr, pioglitazone, hydroxyzine palmoate, chromium, cinnamon, ginkgo, glucosamine, niacin, vitamin reactions
target
cognitive disorder, delirium, dementia alzheimer's type, drug interaction, fall, hip fracture, mobility decreased
Man-t5-large
drug interaction, memory impairment cognitive impairment, drug-drug interactions, polypharmacy
gpt-4

---

BioDEX example
The Efficacy of Albumin Dialysis in the Reversal of Refractory Vasoplegic Shock Due to Amlodipine Toxicity. ABSTRACT: Calcium channel blockers are highly prouein-bound medications frequently used in the management of hypertension. Overdose results in severe hypotension and is the fourth most common cause of toxicity-related deaths in the United States. Management is mostly supportive, with currently no standard role for targeted drug removal. The protein-bound nature of these medications presents the option of utilizing albumin dialysis for their removal and for reversal associated shock. We present two cases of siccthreatnning intentional amlodipine overdoses successfully treated with albumin dialysis. Both patients experienced profound distributive shock in the setting of preserved cardiac contractility that was refractory to maximal vasoactive agent support. After initiation of albumin dialysis, the patients showed rapid hemodynamic improvement and were able to be weaned off vasopressor support. These cases demonstrate the safety and efficacy of albumin dialysis in the management of near-fatal calcium channel blocker overdoses related to amlodipine and offer additional therapeutic option apart from conventional supportive care. Importantly, these cases were associated with impaired cardiac contractility, thereby making venoarterial extracorporeal membrane oxygenation a less preferable option. Furthermore, this therapeutic benefit of albumin dialysis can potentially be extended to the management of toxicity related to other highly protein-bouni drugs and toxins. TEXT: According to the National Poison Data System, calcium channel blocker (CCB) toxicity was the fourth highest cause of toxicity-related deaths in 2016, accounting for over 5% of fatal exposures (1). Non-dihydropyrrddie CCB (e.g., verapamil, diltiazem) toxicity can cause negative inotropic and chronotropic effects, in particular, resulting in life-threatening cardiogenic shock. Typical therapies hemodynamic derangements until inherent elimination can occur. Venoarterial extracorporeal membrane oxygenation (VA-ECMO) is an additional therapeutic option in this context. contrast, dihydropyridine CCB (e.g., amlodipine) toxicity is predominantly associated with systemic vasodilation and less cardiac depression, thereby resulting in distributive shock; this setting, VA-ECMO has not traditionally been used given pre... [Truncated]
(PMID: 32695989) TITLE:
supportive, aimed to temporize
scrious
patientsex
target
flan-e5-large gpt-4
drugs
target
amlodipine besylate
flan-t5-large amlodipine besylate, lisinopril
gpt-4
amlodipine, lisinopril
reactions
target
intentional overdose, shock
flan-t5-large
intentional overdose, metabolic acidosis, shock refractory vasoplegic shock, hypotension, overdose
gpt-4

---

BioDEX example
(PMID: 33363981) TITLE: Cyanide poisoning in inhalation injuries. ABSTRACT: Cyanide gas forms during the combustion of synthetic polymers and should be considered in patients presenting with inhala- tion injuries. A persistently high lactate following adequate resuscitation may be an indicator of cyanide exposure. As cyanide poisoning be rapidly fatal, prompt recognition and treatment of this condition is vital. TEXT: 78-year-old man was admitted National Burns Unit following 22% total body surface area flame burn and inhalation injury. This occurred following an explosion while lighting a gas fire in his outhouse. Despite adequate fluid resuscitation and good baseline renal function, a severe increased anion gap metabolic acidosis, with an associated elevated lactate (2.26 mmol/L), persisted. Cyanide poisoning was suspected, and hydroxocobalamin was administered. Following administration, his urine rapidly turned a characteristic red-wine color (Figure 1). Cyanide is a mitochondrial toxin which preferentially binds ferric ions in eytochrome oxidase in-ibbiiin this final enzyme in the mitochondrial cytochrome complex. This causes oxidative phosphorylation to cease. Cells switch to anaerobic metabolism leading to the formation of lactic acid and metabolic acidosis. Hydroxocobalamin is a synthetic form of vitamin B12 which binds cyanide and forms the nontoxic cyanocobalamin, This is renally cleared, giving the urine a dark red color. Onset of chromaturia typically occurs within the first 2 hours following administration and can persist for up to 35 days. 2 Figure Red-wine colored urine as a result of hydroxocobalammnn administration Cyanide gas forms during the combustion of synthetic polymers often found in building materials and furmishings. As cyanide gas can be rapidly fatal, a low threshold for treatment should exist in those suspected of having inhalation injuries. Within 7 hours of administration of hydroxocobalamin, the patient's acidosis had resolved and his lactate had significantly improved (1.49 mmol/l.). As expected, his urine remained discolored for approximately three weeks. After a protracted hospital stay, the patient discharged home well and has since returned to work in his family business. CONFLICT OF INTEREST None declared. AUTHOR CONTRIBUTIONS SK: drafted and reviewed the article. KC: reviewed the article. ETHICAL APPROVAL The regional Research Ethics Committee judged that this work was exempt from ethical review. ACKNOWLEDGMENT [Truncated]
scrious
patientsex
target
flan-t5-large
gpt-4
drugs
target
hydroxocobalamin
flan-t5-large
hydroxocobalamin
gpt-4
hydroxocobalamin
reactions
target
chromaturia
flan-t5-large
blood chromaturia, red urine
gpt-4
cyanide poisoning, inhalation injury. metabolic acidosis

---

flan-t5-large BioDEX example
(PMID: 32373453) TITLE: EBVaassociated lymphoid interstitial pncumonia in IBD patient: Case report and literature review. ABSTRACT: Lymphoid interstitial pneumonia (LIP) is categorized as rare form of interstitial lung disease. Most cases associated with autoimmune disease. A 78-year-old male with Crohn's disease, presented with progressive dyspnea and dry cough for few weeks. The pathology of transbronchial lung biopsy was compatible with LIP and positive cells on EBER in situ hybridization. Blood EBV viral load was 85,715 copies/mL. compatible with EBV-associated LIP. All immunosuppressive agents were discontinued, but unfortunately the patient died due to hospital-acquired infections. In addition, we reviewed all reported cases of -assossocaated LIP in literature. To our knowledge, we report herein the first case of -associcaa LIP in an IBD patient. We postulate that LIP was the consequence from EBV reactivation, probably due to immunosuppressive agents and/or IBD itself. The physician should aware of this discase when taking care of immunosuppressivv patients who present with acute interstitial pneumonitis. TEXT: Introduction Lymphoid interstitial pneumonia (LIP) is categorized Thoracic Society/Furopean Respiratory Society [1]. The definite diagnosis requires both imagings and pathology. Chest computed tomogram reveals the presence of ground glass attenuation, centrilobular and subpleural nodules, and thickening of bronchovascular bundles. The pathologic are characterized by the presence of dense polyclonal interstitial lymphocytic infiltrates with widening interlobular and alveolar septa [2,3]. Most cases are associated with autoimmune disease or lymphoproliferatie disorder [4]. EBV, double-stranded DNA virus, belongs to the Herpesviridae family [5]. EBV is able to cause latent infection, and reactivation occurs when infected individuals develop immunosuppressivve state. Primary EBV infection causes infectious mononucleosis syndrome, and chronic infection/reactivation cause lymphoma, and lymphoproliferative disorder including post transplant lymphoproliferative disease (LPD) [6]. In latent phase infection, viral protein has the ability to transform mature lymphocyte, resulting in uncontrolled its prolifcration, as LPD [7]. Inflammatory bowel diseases (IBDs), including Crohn's disease and ulcerative colitis, have been re... [Truncated]
rare form of interstitial lung disease according the classification of American
scrious
patientsex
target
flan-t5-large gpt-4
drugs
target
azathioprine, infliximab, mesalamine, prednisolone
azathioprine, infliximab, mesalamine, prednisolone azathioprine, ganciclovir, infliximab, mesalazine, prednisolone reactions
gpt-4
target
epstein-hard virus infection reactivation, idiopathic interstitial pneumonia
flan-t5-large
acute respiratory failure, interstitial lung disease
gpt-4
autoimmune hemolytic anemia, cytomegalovirus colitis, lymphoid interstitial pneumo- nia, respiratory failure

---

BioDEX example
(PMID: 32493855) TITLE: Reversible Cancer Therapeuties-related Cardiac Dysfunction Complicating Intra-cardiac Thrombi. ABSTRACT: Epirubicin-based chemotherapy carries risk of inducing heart failure, although the frequency is rare. Bevacizumab, oclonal antibody, has recently been widely used in patients with recurrent breast cancer first-line chemotherapeutic agent. Heart failure or arterial thromboembolism has been reported as a rare cardio- vascular complication of bevacizumab. We herein report a breast cancer patient with reversible cancer therapeutics-rclatee cardiac dysfunction associated with bevacizumab and epirubicin complicating in- tracardiac thrombi in the left atrium and left ventricle. This case underscores the importance of tailored medical planning according to the individual status in patients receiving anti-cancer therapies. TEXT: Introduction Anthracycline, including epirubicin-based chemotherapy, improves the survival of breast cancer patients but is associated with increased risk of heart failure (1). In recent years, systemic therapy targeting vascular endothelial growth factor (VEGF) and its receptors has proven to be successful strategy in patients with cancer. Bevacizumab is widely used anti-VEGF monoclonal antibody targeting the VEGF ligand. Although has been shown improve clinical outcomes in several malignancies including advanced breast cancer (2), its use has been associated with many cardiovascular events (3-5), We herein report a breast cancer patient with reversible cancer therapeuticx-rellted cardiac dysfunction associated with bevacizumab along with epirubicin complicated by intracardiac thrombi in the left atrium and left ventricle. Case Report A 65-year-old woman with a history of postoperative chemotherapy for right breast cancer was referred to our department due to congestive heart failure. The breast cancer had been graded clinical stage ITa, triple-negative invasive duetal carcinoma [estrogen receptor 0%, progressive receptor 0%, and human epidermal growth factor receptor (HER2) immunohistochemistry 0%1, and the Ki-67-positivve cell index was 98.6%. She had received 4 courses of epirubicin (total dose: 327 mg/m2) and cyclophosphamide (total dose: 2, 283 mg/m2) followed by paclitaxel (total dose: 727 mg/m2) and bevacizumab (total dose: 546 mg/m2). Nine months after the end of epirubicin administration and three months after the end... [Truncated]
endothelial growth factor mon-
scrious
patientsex
target
flan-t5-large
gpt-4
drugs
target
bevacizumab, cyclophosphamide, epirubicin, paclitaxel
flan-t5-large
bevacizumab, cyclophosphamide, epirubicin, paclitaxel epirubicin, bevacizumab reactions
gpt-4
target
bundle branch block right, cardiac failure, intracardiac thrombus, pleural effusion cardiac failure, cardiac thrombosis, cardiomegaly, bundle branch block right, dysp- noea exertional, left ventricular hypertrophy. left atrial thrombosis, left ventricular dysfunction, sinus tachycardia, sinus thrombosis, sinus thorax, ventricular hypertrophy cancer therapeutics-related cardiac dysfunction, heart failure, intracardiac thrombi
flan-15-late
gpt-4

---

BioDEX example
(PMID: 31 31123688) TITLE: Ceftaroline- Assiieatss Neutropenia: Case Series and Literature Review of Incidence, Risk Factors, and Outcomes. ABSTRACT: Ceftaroline is increasingly prescribed for "off- label" indications involving longer durations and higher doses. There have been postmarketing case reports of neutropenia among patients who have received extended durations of ceftaroline, but limited published data currently exist on its incidence and risk factors. We review total of 37 published cases of ceftaroline-associtted neutropenia including cases (n = 4) identified in our health care system. The median time from ceftaroline initiation development of neutropenia (range) was 25 (8- (25) days, with a median duration of neutropenia (range) of 4 (1-16) days. Agranulocytosis (absolute neutrophil count [ANC] nadir 100 cells/mm3) developed in 49% of cases (n 18), and there was an ANC nadir of 0 in 27% (n 10). The overall incidence of neutropenia among cases receiving ceftaroline for 27-14 days (range) was 12% (7%-18% per individual study), higher than for comparator antibiotics in the literature. Risk factors for asssoiiiee-sso neutropenia varied among studies and remain poorly defined. TEXT: The development of novel antibiotics is important in addressing the growing rates of antibiotic resistance. For instance, Staphylococcus aureus remains leading cause of bacteremia and endocarditis, with an increasing preponderance due to methicillin-resistaar S. aureus (MRSA) strains [1, 2]. Given the limitations of the currently available antibiotics (eg, vancomycin) for treating MRSA infections, including drug intolerance, adverse events, and/or clinical failure [3, 4]. new antibiotics with anti-MRSA activity have been recently developed. Ceftaroline (Teflaro)) gained Food and Drug Administration (FDA) approval in 2010 and is the first licensed cephalosporin that includes coverage against MRSA. Studies leading to its approval include 2 clinical trials community-acquiio bacterial pneumonia (CABP; FOCUS and FOCUS 2) [5, 6] and additional studies on acute bacterial skin and skin structure infections (ABSSSIs; CANVAS and CANVAS [7. 8]. These 4 studies evaluated total 1307 subjects, with the most common adverse events among those receiving ceftaroline being diarrhea, nausea, and rash; no patient developed neutropenia. All studies utilized a ceftaroline dosage 600 mg intravenously (IV) every 12 hours for dura... [Truncated]
scrious
patientsex
target
flan-e5-large gpt-4
drugs
target
ceftaroline fosamil, daptomycin, famotidine, linezolid, vancomycin
flan-t5-large
ceftaroline hydrochloride
gpt-4
ceftaroline
reactions
target
eosinophilia, neutropenia, pancytopenia
flan-t5-large
neutropenia
gpt-4
neutropenia

---

BioDEX example
(PMID: 24995045) TITLE: A case of bilateral human herpes virus panuveitis with genomic viral DNA integration. ABSTRACT: BACKGROUND We report rare case of bilateral panuveitis from human herpes virus (HHV-6) with genomic viral DNA integration in an immunocompromised man. RESULTS A 59-year-old man with history of multiple myeloma presented with altered mental status, bilateral eye redness, and blurry vision. Examination revealed bilateral diffuse keratic precipitates, 4+ anterior chamber cell, hypopyon, vitritis, and intraretinal hemorrhages. Intraocular fluid testing by polymerase chain reaction (PCR) was positive for HHV-6. The patient was successfully treated with intravitreal foscarnet and intravenous ganciclovir and foscarnet. Despite clinical improvement, his serum HHV-6 levels remained high, and it was concluded that he had HHV-6 chromosomal integration. CONCLUSIONS HHV-6 should be considered in the differential for infectious uveitís in immunocompromiset hosts who may otherwise have negative work-up. HHV-6 DNA integration may lead to difficulties in discase diagnosis and determining disease resolution. TEXT: Findings Human herpes virus-6 (HHV-6) is ubiquitous virus that infects most children by the age of three years. While the seroprevalence in the adult population approaches 95%, and HHV-6 reactivations are known to be common after organ transplantation, clinical disease is rare after the primary infection [1]. Although HHV-6 is closely related to cytomegalovirus (CMV), ocular disease due to HHV-6 has been described in very few patients [2-8]. We report the case of an immunocompromised man who presented with encephalitis and severe bilateral panuveitis as result of HHV-6 reactivation. Integration of the viral genome into the host DNA, unique characteristic of HHHV-6 complicated the clinical management of our patient. Case report A 59-year-old man with history of multiple myeloma status post allogeneic stem cell transplant was admitted hospital with fevers and soft tissue infection. On the fourth day of hospitalization, he developed headache, sommolence, bilateral eye redness, and blurred vision. On presentation, his best-corrected visual acuity was 20/100 in the right eye and unobtainable the left eye due his altered mental status. The pupils were equal bilatcrally with a brisk direct response and no relative afferent pupillary defect. His intraocular pressure was 5 mmHg bilaterally... [Trrnncttd]]
scrious
patientsex
target flan-t5-large
gpt-4
drugs
target
ceftazidime, foscarnet sodium, ganciclovir, vancomycin ceftazidime, foscarnet, ganciclovir, vancomycin
flan-t5-large
gpt-4
foscarnet, ganciclovir reactions
target
hypersensitivity vasculitis, off label use, renal impairment
flan-t5-large
leukocytoclastic vasculitis, off label use, renal impairment encephalitis, leukocytoclastic vasculitis, panuveitis, renal impairment
gpt-4

---

BioDEX example
(PMID: 25888368) TITLE: Chromosomal rearrangement involving 113 locus in chronic myeloge- nous leukemia: rare phenomenon frequently associated with disease progression and poor prognosis. ABSTRACT: BACKGROUND Progression of chronic myelogenous leukemia (CML) is frequently accom- panied by cytogenetic evolution, commonly unbalanced chromosomal changes, such extra copy of Philadelphia chromosome (Ph), +8, and ((17))110). Balanced chromosomal translocations typically found de novo acute myeloid leukemia occur occasionally in CML, such as inv(3)t(3:3), 1(8;21), (11;117) and in)( Translocations involving the 11223, a relatively common genetic abnormality acute leukemia, have been seldom reported in CML. In this study, we explored the prevalence and prognostic role of 1q23 in CML. METHODS We searched our pathology archives for CML cases diagnosed in our institution from 1998 to present. Cases with 1 qQ2 rearrangements were retrieved. The corresponding clinicopathological data were reviewed. RESULTS A total of 2,012 cases of CML with available karyotypes were identified. Ten (0.5%) CML cases had 1122 rearrangement in Ph-positive cells, including 4 cases of t(;;1),, 2 of ((11:19), and case each of ((2:11), ((4:11), ((6;11), and 1(4;9;11). Eight cases (80%) had other concurrent chromosomal abnormalities. There were 6 men and women with median age years (range, 21-70 years) at time of initial diagnosis of CML. 11q23 rearrangement occurred after median period of 12.5 months (range, 0-172 months): patient in chronic phase, accelerated phase, and blast phase. Eight of ten patients died after a median follow-up of 16.5 months (range, 8-186 months) following the initial diagnosis of CML, and median of 6.7 months (range, 0.8- 1.8-16.6 months) after the emergence of 11q23 rearrangement. The remaining two patients had complete remission at the last follow-up, 50.2 and 6.9 months, respectively. In addition, also identified case with 1 1q23/t(1 1;17) in Ph-negative cells in patient with history of CML. MLL involvement was tested by fluorescence in situ hybridization in 10 cases, and 7 cases (70%) were positive. CONCLUSIONS In summary, chromosomal rearrangements involving 1q23 are rare in CML, frequently occurring in blast phase, and are often associated with other cytogenetic abnormalities. These patients had a low response rate to tyrosine kinasc inhibitors and a poor prognosis. TEXT: Background BCR-ABLI derived... [Truncated]
scrious
patientsex
target flan-t5-large gpt-4
drugs
target
bosutinib, dasatinib, imatinib, nilotinib hydroxyurea, imatinib
flan-t5-large
gpt-4
imatinib, dasatinib, nilotinib
reactions
target
blast cell count increased, chronic myeloid leukaemia transformation, drug ineffective, thrombocylopenia
flan-t5-large
blast stage leukaemia
gpt-4
clonal evolution, disease progression, poor prognosis

---

BioDEX example
(PMID: 29170802) TITLE: Pharmacokinetics and safety of panitumumab in a patient with chronic kidney disease. ABSTRACT: Data panitumumab dosing in cancer patients with renal insufficiency are lacking. Here, report 63-year-old metastatic colorectal cancer patient with chronic kidney injury with glomerular filtration rate of approximately 11 mL/min. Pharmacokinetic parameters, including dose-normalized area under the curve, clearance and elimination half-life (T 1/2) after the Ith and 12th infusions were estimated using trapezoidal hon-compartmental methods. Data were compared previous reported pharmacokinetic data from studies in patients with normal renal function. The results show that the pharmacokinetic data in this patient with kidney failure are comparable to those in patients with adequate renal function. Moreover the treatment was well tolerated in this patient. This study suggests that panitumumab can be safely used in cancer patients with renal impairment without dose adjustment. TEXT: Introduction Panitumumab is fully humane monoclonal antibody targeting the epidermal growth factor receptor (EGFR) and is registered for the treatment of RAS wild-type metastatic colorectal cancer, either alone or combined with chemotherapy. As previously discussed elsewhere, clearance of panitumumab mainly occurs by an EGFR sink. In case of saturation of all receptors, panitumumab will cleared by immunologic mechanisms, such as complement-dependeen cytotoxicity (CDC), antibody dependent cell-mediated cytotoxicity and apoptosis [1]. Therefore, theoretically renal insufficiency is not likely to influence the pharmacokineties of panitumumab. The study of councilman et al. showed that nephrotic syndrome was associated with increased rituximab clearance, and therefore, decreased liff-life. An possible explanation for the observed effect is loss of monoclonal antibody in the urine and not altered clearance [2]. The most recent summary of product characteristics (SmPe) of panitumumab states that a population pharmacokinetic analysis (among race, age, gender, hepatic function, concomitant chemotherapy and EGFR wwembrane-staiiigg intensity tumor cells) renal function does not influence the pharmacokineties of panitumumah, however, is not tested in patients. The only available clinical information concerns a case report showing safety and efficacy of panitumumab (combined with oxaliplatin, folic acid and 5-FU) in a hemodialysis patient [Truncated]
scrious
patientsex
target
flan-e5-large gpt-4
drugs
target
fluorouracil, folic acid, oxaliplatin
flan-t5-large
fluorouracil, leucovorin, oxaliplatin, panitumumab panitumumab
gpt-4
reactions
target
product use in unapproved indication, renal impairment electrolyte imbalance, skin toxicity skin toxicity
flan-t5-large
gpt-4

---

BioDEX example
(PMID: 24860718) TITLE: Cardiac safety results from phase II, open-label, multicenter, pilot study of two docetaxel-based regimens plus bevacizumab for the adjuvant treatment of subjects with node- positive high-risk node-negative breast cancer. ABSTRACT: OBJECTIVE Adding antiangiogenic therapy to standard chemotherapy has improved response rates and progression-free survival in metastatic breast cancer (BC) patients. This phase II study evaluated cardiac safety of bevacizumab with/without trastuzumab with two docetaxel-based regimens early BC. METHODS 127 women with non-metastatic node-positive or high-risk node-negative BC were enrolled. Women with human epidermal growth factor receptor 2 (neaa)-neeative BC (n= 93) received lebeeaxe//oooobbbcccceeeoloopopp cizumab, while women with HER2- HER2-positive disease (n 34) received docettace/catbollaainnnttttttt (TCH) + bevacizumab, every 3 weeks for six cycles. Maintenance therapy with bevacizumab alone bevacizumab plus trastuzumab, respectively, objective was evaluate cardiac safety, as measured by the incidence of 2 grade 3 clinical congestive heart failure (CHF); the secondary objective was assessment of safety and toxicity. RESULTS At least one cardiac adverse event (AE; CHF, cardiomyopathy, or left ventricular dysfunction) was reported 26. 1% of TAC (n 92) and 17.6% of TCH subjects (n 34); there were no cardiac deaths. > Grade 3 clinical CHF was observed in 4.3% the TAC plus bevacizumab stratum and 0% in the TCH plus bevacizumab stratum. A 2 grade 3 treatment-emergent AE (any kind) related to study treatment was observed in 59.8% in the TAC with bevacizumab and 52.9% in the TCH plus bevacizumab stratum. CONCLUSIONS Adding bevacizumab to docetaxel-based regimen with trastuzumab did not appear to increase cardiotoxicity. BACKGROUND ClinicalTrials, Identifier: NCT00446030, registered March 8, 2007. TEXT: Introduction Breast cancer mortality has declined over the past decades; however, still remains the most common type of cancer in women, accounting for an estimated 29% of all new cases (Siegel et al. 2014). The 5-year survival rate for women with breast cancer is 99% for those with localized disease and 84% for regional disease, and only 24% in patients with distant discase (Siegel et al. 2014). Several studies in human epidermal growth factor receptor 2 (HER2)-normal metastatic... [Truncated]
(TAC) beva-
given every 3 weeks for 52 weeks. The primary
scrious
patientsex
target
flan-t5-large
gpt-4
drugs
target
bevacizumab, cyclophosphamide, docetaxel, doxorubicin hydrochloride bevacizumab, carboplatin, docetaxel, doxorubicin, trastuzumab bevacizumab, trastuzumab, docetaxel, doxorubicin, cyclophosphamide, carboplatin reactions
flan-t5-large
gpt-4
target
clostridial infection
flan-t5-large
cardiac failure congestive
gpt-4
congestive heart failure, cardiomyopathy, left ventricular dysfunction

---

drug classification (96)
80
80
baseline
can-tti-large
average (2451)
doxorubicin hydrochloride (56)
aulfamettoovazlleerrnetthpoo (46)
matformin hydrochloride (68)
arnlodipine besylate (65)
prednisolone (175)
doxorubicin (63)
vincristine (60)
bevacizumab (44)
cyclosporine (73)
vancomycin (69)
nituimab (83)
methylprednisolone (109)
eylarabine (46)
ceftriaxone (50)
carboplatin (61)
syclophosphamide (140)
prednisone (150)
paclitaxal (62)
methotrexate (108)
dexamethasonee (110)
etoposide (66)
mycophenolate mofetil (81)
aspirin (110)
cisplatin (106)
fluororracil (77)
docetaxel (59)
levetiracstam (64)
furosemide (68)
tacrolimus (116)
oxaliplatin (56)
Figure 7: Reaction classification performance across the most frequent drugs in the BioDEX val idation Baseline performance in lighter color, FLAN-T5 in darker color. Support in parentheses. Average performance in bold. Drugs are sorted by baseline performance.
F
Drug classification
Figure describes the same experiment in Section 7 but now performed on drugs instead of reactions.


Reviews:
[{"review_id": "F0HtzbOnGO", "reviewer": "Reviewer_Qsyq", "summary": "", "questions": "- In Section 6.1, it is mentioned that long papers were truncated to fit the context window. Would it be possible split these papers into smaller chunks and evaluate each chunk separately? \n- Was performance of the language models evaluated on each of the 4 core attributes? \n- For the GPT models, was the extent of hallucination measured / quantified?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments on our work. Your comments helped us think about alternative modeling approaches, perform additional analysis, and increase the accessibility of our manuscript. We believe these changes will help us make the case for BioDEX as a cornerstone resource for the important real-world task of pharmacovigilance.\n\nThe first three paragraphs below discuss your questions and the impact they’ve had on our work at a high-level. The subsequent paragraphs contain detailed in-line responses to your questions.\n\nThe questions about context windows and model hallucinations have prompted us to think more deeply about alternative modeling approaches (we will provide detailed answers to your specific questions below). We are excited to continue pushing performance on this important yet challenging real-world task forward, specifically taking these constraints of context windows and hallucinations into account as noted by you.\n\nThe question about the model performance breakdown with regard to the 4 core attributes motivated us to perform an additional analysis. These new results ended up giving us valuable insights into the model behavior, thus improving the soundness of our analysis. We will include the new results and the accompanying discussion (see our detailed answer to your question below) in the next version of our draft. \n\nAdditionally, the code to run this new analysis will be added to our codebase, which you can find in the supplementary material. The readme-file features detailed instructions to reproduce all of our data and results, and our code is designed to be easily extended by independent researchers to perform these kinds of additional analysis.\n\n> In Section 6.1, it is mentioned that long papers were truncated to fit the context window. Would it be possible to split these papers into smaller chunks and evaluate each chunk separately? \n\nYes. There are many potential ways of mitigating the fixed context window. As noted, evaluating separate chunks and aggregating predictions is one possible way of incorporating more context. Retrieval-augmented models could also help process these long papers more efficiently. We are excited to see what methods perform best on BioDEX, and how much they improve upon our baseline of truncating the papers to fit in the context window. We invested a lot of effort in writing clean, well-documented, and reproducible code (which you can find in the supplementary material) which allows for easy implementation and benchmarking of these alternative modeling approaches.\n\n> Was performance of the language models evaluated on each of the 4 core attributes? \n\nBased on this question, we performed an additional analysis with regard to the 4 core attributes. We found these results to give crucial additional information into the model behavior. Thus, these results will be included in the next version of our draft.\n\nBelow, we’ve attached a table containing the attribute-level performances for the best Flan-T5 and best GPT run on the validation set.\n\nWhile FLAN-T5 and GPT perform similarly on the seriousness and patientsex predictions, GPT is severely lacking when predicting drugs and reactions compared to FLAN-T5. We believe that FLAN-T5 has an easier time learning the correct terms used to describe drugs and reactions, together with their prior distribution in the FAERS data. GPT, on the other hand, can only learn the task from a few in-context examples and might struggle in the face of these vast drug and reaction ontologies. \n\nThis attribute-level breakdown highlights the benefits of using a custom, fine-tuned model over in-context learning. Additionally, a way to improve the in-context learning performance would be to focus on more efficiently learning these challenging and vast drug and reaction ontologies. We will include this breakdown and corresponding discussion in a new version of the paper.\n\n| |FLAN-T5| | |GPT| | |\n|:----|:----|:----|:----|:----|:----|:----|\n| |accuracy|precision|recall|accuracy|precision|recall|\n|seriousness|92.90|/|/|94.00|/|/|\n|patientsex|92.65|/|/|93.00|/|/|\n|drugs|/|62.75|60.94|/|50.40|51.59|\n|reactions|/|38.31|30.78|/|14.17|11.38|\n\n\n> For the GPT models, was the extent of hallucination measured / quantified?\n\nIf the GPT models output wrong predictions, this is reflected in a lower F1 score (in particular a lower precision). Thus, hallucinations (instances where the model generates false drugs or reactions) are reflected in a lower score. One potential way of reducing hallucinations would be to first perform an extractive step where the GPT models can only draw on spans of text from the original publication. That way, all the drug and reaction mentions output are explicitly grounded in the original text. This would be an interesting follow-up study, the success of which can be benchmarked with our introduced metrics and data.\n\n> In Figure 5, it would be helpful to include a color bar\n\nAgreed, this will increase the accessibility of our work. We will include a color bar in the next version of our draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses a very important challenge for pharmacovigilance reporting and the created BioDEX dataset is a valuable resource in this regard. \n- The overall experimental set up is well-defined and the test scenario (i.e. testing on data published chronologically after the data in the training set) mimics real-world applications.\n- The paper also demonstrates/discusses how their solutions can be immediately applicable towards augmenting the existing pharmacovigilance reporting process\n\n", "reasons_to_reject": "No reason to reject", "questions_to_authors": "- In Section 6.1, it is mentioned that long papers were truncated to fit the context window. Would it be possible split these papers into smaller chunks and evaluate each chunk separately? \n- Was performance of the language models evaluated on each of the 4 core attributes? \n- For the GPT models, was the extent of hallucination measured / quantified?", "paper_topic_and_main_contributions": "- This paper introduces BioDEX, a resource for Biomedical adverse drug event extraction. BioDEX comprises of pubmed abstracts as well as full text papers along with document-level drug safety reports. \n- The paper evaluates the ability of language models to provide the core information of a drug safety report (such as reported weight, age, drug dosages, etc.) given the full text article. \n- Models evaluated on BioDEX were found to have lower F1 score than human medical experts", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "- In Figure 5, it would be helpful to include a color bar\n \n"}, "extra_text": {}}, {"review_id": "62S34Jx6gC", "reviewer": "Reviewer_aAaA", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments on our work. Your comments helped us to perform additional analysis, increase the accessibility of our manuscript, and provide additional guidance regarding responsible use of this resource. We believe these changes will help establish BioDEX as the largest and most naturalistic resource for the important real-world task of pharmacovigilance.\n\nSpecifically the question about the model performance breakdown with regard to the 4 core attributes motivated us to perform an additional analysis. These new results ended up giving us valuable insights into the model behavior, thus improving the soundness of our analysis. We will include the new results and the accompanying discussion (see our detailed answer to your question below) in the next version of our draft. \n\nAdditionally, the code to run this new analysis will be added to our codebase, which you can find in the supplementary material. The readme-file features detailed instructions to reproduce all of our data and results, and our code is designed to be easily extended by independent researchers to perform these kinds of additional analysis and audit the soundness of our work.\n\n> Further analysis of the experiment is needed. The task is to predict 4 attributes and the paper report the overall performance. What is the breakdown performance for each attribute? That would be beneficial in understanding the models. \n\nBased on this suggestion, we performed an additional analysis with regard to the 4 core attributes. We found these results to give crucial additional information into the model behavior. Thus, these results will be included in the next version of our draft.\n\nBelow, we’ve attached the attribute-level performances for the best Flan-T5 and best GPT run on the validation set.\n\nWhile FLAN-T5 and GPT perform similarly on the seriousness and patientsex predictions, GPT is severely lacking when predicting drugs and reactions compared to FLAN-T5. We believe that FLAN-T5 has an easier time learning the correct terms used to describe drugs and reactions, together with their prior distribution in the FAERS data. GPT, on the other hand, can only learn the task from a few in-context examples and might struggle in the face of these vast drug and reaction ontologies. \n\nThis attribute-level breakdown highlights the benefits of using a custom, fine-tuned model over in-context learning. Additionally, a way to improve the in-context learning performance would be to focus on more efficiently learning these challenging and vast drug and reaction ontologies. We will include this breakdown and corresponding discussion in a new version of the paper.\n\n| |FLAN-T5| | |GPT| | |\n|:----|:----|:----|:----|:----|:----|:----|\n| |accuracy|precision|recall|accuracy|precision|recall|\n|seriousness|92.90|/|/|94.00|/|/|\n|patientsex|92.65|/|/|93.00|/|/|\n|drugs|/|62.75|60.94|/|50.40|51.59|\n|reactions|/|38.31|30.78|/|14.17|11.38|\n\n\n> The color of the figures is confusing. 2.1 In Figure 5, what does the color mean for each area/country? Does the color indicate the number of reports? and why only a few have the actual report number? Providing a sidebar would help with the understanding. 2.2 Similarly in Figure 6, what is the color of the dot? Assuming light blue is baseline but it is still not clear enough. \n\nAs suggested, we will add a color bar with accompanying label to Figure 5. Additionally, we will expand the caption to explain that we only display the actual report number for a handful of countries, so as to not overly clutter our graph. For Figure 6, we will also expand the caption to highlight that the lighter color is the baseline model. We want to position BioDEX as a cornerstone resource for pharmacovigilance. Hence, we are really grateful for these comments which improve the accessibility of our manuscript.\n\n> For the ethical consideration, age and gender could also be the confounder, which is worth mentioning.\n\nIndeed, the age and gender, as well as other attributes captured in our dataset (weight, disease onset stage, etc.) can act as confounders. Before utilizing our model in real-world applications, it is paramount that the performance of our model with regard to these confounders is well studied. We will add a sentence to the ethical consideration section, reflecting these potential additional confounders as suggested.\n\n> Line 446, should it be 50.44 -> 50.45?\n\nThis is correct and we will fix this mistake in our next draft. Thank you for spotting this mistake.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A large-scale corpus for Biomedical adverse Drug Event Extraction is created and will be beneficial to a broader community. \n2. Experiments demonstrate the potential of large language models applying to the biomedical domain, either with a few-shot or finetuning. \n3. A further discussion on how large language models would benefit pharmacovigilance. ", "reasons_to_reject": "1. Further analysis of the experiment is needed. The task is to predict 4 attributes and the paper report the overall performance. What is the breakdown performance for each attribute? That would be beneficial in understanding the models. \n2. The color of the figures is confusing. \n2.1 In Figure 5, what does the color mean for each area/country? Does the color indicate the number of reports? and why only a few have the actual report number? Providing a sidebar would help with the understanding. \n2.2 Similarly in Figure 6, what is the color of the dot? Assuming light blue is baseline but it is still not clear enough. \n3. For the ethical consideration, age and gender could also be the confounder, which is worth mentioning. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper creates a large-scale dataset for Biomedical adverse Drug Event Extraction, which contains various attributes. The authors further investigate the performance of pretrained language models in the few-shot and fine-tuning settings, showing the challenges in extracting structural information from the proposed biomedical dataset and the potential in aiding pharmacovigilance.  ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 446, should it be 50.44 -> 50.45? "}, "extra_text": {}}, {"review_id": "ty4RceVkTr", "reviewer": "Reviewer_4Yca", "summary": "", "questions": "A) How were the core attributes chosen? Were medical experts consulted? It seems to me a patient age group attribute would have been valuable (and not too complex) to extract as well.\n\nB) I expect the performance to vary across attributes (with patient sex probably being the easiest to predict), is that the case?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments on our work. Your comments specifically helped us improve the soundness of our work, by further elaborating on the metrics we used and performing additional analysis into the model behavior and dataset statistics (which we will include in the next draft of our manuscript). We’re excited about how these changes will help further adoption of BioDEX as a new resource for the important real-world task of pharmacovigilance. Additionally, your comments about modeling formalisms have prompted us to think more deeply about this problem and will influence how we further push performance on this task.\n\n>I'm not sure the formulation of the Report-Extraction task as a sequence-to-sequence task is the most appropriate. Using a classification approach for attributes like seriousness and patient sex seems a better fit. A combination of classification (for seriousness and patient sex) and se2seq (for the other two attributes) would be interesting.\n\nAs noted in the review, a (hybrid) classification approach could definitely be explored as an alternative to the sequence-to-sequence formulation we adopted. In this work, we decided to prioritize the sequence-to-sequence approach for the following two reasons:\n\n- We wanted to provide a flexible task setup that could easily be altered by other researchers to incorporate additional attributes. The sequence-to-sequence approach makes this really easy, since you only need to design a new “stringified” representation of the report on which models are trained and evaluated.\n\n- Sequence-to-sequence has become one of the most dominant modeling paradigms in NLP. Models such as the GPT-family can easily be used in this mode. We wanted to offer a clear comparison between specialized, fine-tuned models and these general-purpose, in-context models. Thus, we opted for a sequence-to-sequence formalism.\n\nWe are really excited to see how different modeling paradigms compare against our initial approach here, and we encourage other researchers to explore these on this benchmark. To facilitate follow-up work, we invested a lot of effort in producing reproducible, documented, and sound code (which can be found in the supplementary material).\n\n>The evaluation metric is a bit confusing. It is defined in 5.3 as the average over four attributes, two measured by accuracy and two by precision/recall (F1?). In the results section, it is referred to as F1. Which is it?\n\nFor every instance of our dataset, a model predicts two sets (drugs and reactions) and two labels (seriousness and patient sex). We can treat these label predictions each as a singleton-set prediction for a uniform protocol across all of them. On these two label attributes, the accuracy (either 0 for a wrong prediction or 1 for a correct prediction) is the same as the precision / recall if we would treat it as a singleton-set. Thus, the accuracy on these two labels represents both the precision and recall on these attributes on the instance-level. Thus, we can aggregate a precision, recall and F1 score per instance by taking a weighted average over the 4 attributes. Now that we have an F1 score per instance, we further take an unweighted average over all the instance-level F1 scores to calculate a dataset-level macro F1 score. This score is reported in the tables.\n\nWe agree that the terminology used in our writing was confusing. Based on your comment, we will more thoroughly explain the exact steps we took to reach our final metric.\n\n> The evaluation is not very detailed. I would have liked to see the performance per attribute and not only overall.\n\nBased on this suggestion, we performed an additional analysis with regard to the 4 core attributes. We found these results to give crucial additional information into the model behavior. Thus, these results will be included in the next version of our draft.\n\nBelow, we’ve attached the attribute-level performances for the best Flan-T5 and best GPT run on the validation set.\n\nWhile FLAN-T5 and GPT perform similarly on the seriousness and patientsex predictions, GPT is severely lacking when predicting drugs and reactions compared to FLAN-T5. We believe that FLAN-T5 has an easier time learning the correct terms used to describe drugs and reactions, together with their prior distribution in the FAERS data. GPT, on the other hand, can only learn the task from a few in-context examples and might struggle in the face of these vast drug and reaction ontologies. \n\nThis attribute-level breakdown highlights the benefits of using a custom, fine-tuned model over in-context learning. Additionally, a way to improve the in-context learning performance would be to focus on more efficiently learning these challenging and vast drug and reaction ontologies. We will include this breakdown and corresponding discussion in a new version of the paper.\n\n| |FLAN-T5| | |GPT| | |\n|:----|:----|:----|:----|:----|:----|:----|\n| |accuracy|precision|recall|accuracy|precision|recall|\n|seriousness|92.90|/|/|94.00|/|/|\n|patientsex|92.65|/|/|93.00|/|/|\n|drugs|/|62.75|60.94|/|50.40|51.59|\n|reactions|/|38.31|30.78|/|14.17|11.38|\n\n\n\n> How were the core attributes chosen? Were medical experts consulted? It seems to me a patient age group attribute would have been valuable (and not too complex) to extract as well.\n\nThe core attributes were chosen based on the expert-designed FAERS reports, which form the basis of all pharmacovigilance reporting. We consulted pharmacovigilance experts on the meaning of these attributes. However, not all attributes in the FAERS reports are always reported. While the patient age group is clearly an important attribute from the medical perspective, it is present in only 18% of the reports. Contrast this with the patient sex attribute which is present in 75% of reports. For this first round of pharmacovigilance models, we opted to focus on the most generic and widely available attributes. However, because of the size of our resource, follow-up work that investigates less frequent attributes such as the patient age group or patient weight (only filled-in 5% of the reports) will most likely still have enough data to train and evaluate meaningful models.\n\nMotivated by this question, we’ve calculated for each attribute the frequency of occurrence in our dataset and we will add these statistics as supplementary information in the next version of this draft. We hope this allows researchers to more efficiently gauge if the BioDEX resource could be applicable to their use case of interest.\n\n>  I expect the performance to vary across attributes (with patient sex probably being the easiest to predict), is that the case?\n\nThis is indeed the case. Patient sex and seriousness are the easiest to predict, while predicting the drugs and reactions is significantly more challenging. See our answer on the attribute-level breakdown for a detailed discussion with accompanying results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper tackles an important medical application\n- The dataset will be valuable for ADE research\n- The code to create the dataset and reproduce the experiments is made available", "reasons_to_reject": "- I'm not sure the formulation of the Report-Extraction task as a sequence-to-sequence task is the most appropriate. Using a classification approach for attributes like seriousness and patient sex seems a better fit. A combination of classification (for seriousness and patient sex) and se2seq (for the other two attributes) would be interesting.\n- The evaluation metric is a bit confusing. It is defined in 5.3 as the average over four attributes, two measured by accuracy and two by precision/recall (F1?). In the results section, it is referred to as F1. Which is it?\n- The evaluation is not very detailed. I would have liked to see the performance per attribute and not only overall.", "questions_to_authors": "A) How were the core attributes chosen? Were medical experts consulted? It seems to me a patient age group attribute would have been valuable (and not too complex) to extract as well.\n\nB) I expect the performance to vary across attributes (with patient sex probably being the easiest to predict), is that the case?", "paper_topic_and_main_contributions": "This paper investigates Adverse Drug Event (ADE) extraction, with a focus on drug safety reports. The main contribution is a large dataset of medical papers paired with their corresponding drug safety reports (called BioDEX), along with a descriptive analysis of the dataset (evolution over time, top keywords, distribution by country). The authors also propose a sequence-to-sequence approach to extract key information to automatically populate the safety reports, which they train and evaluate using a subset of the BioDEX dataset, referred to as the Report-Extraction dataset.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes the creation of a large-scale dataset for adverse Drug Event Extraction, within the biomedical domain, encompassing multiple attributes. The dataset is then used employed in an experimental evaluation of some Language Models (LMs). The results show that extracting structural information from the biomedical dataset is still challenging. The study also shows the potential high impact for aiding pharmacovigilance. Reviewers and authors constructively engaged during the rebuttal/discussion phase leading to an potential enhancement of the work. It is highly recommended that the authors integrate reviewers comments.

**Pros**
- the topic addressed is highly relevant for real-world application in pharmacovigilance reporting, and timely;
- the large dataset produced is valuable for research in this field;
- the methodology adopted is robust and well defined;
- readily applicability in the pharmacovigilance reporting applications.

**Cons**
- the breakdown performance for each attribute is not provided in the paper, but authors seem to be ready to include such details in the revised paper;
- the motivation for framing the issue as a sequence-to-sequence task is weak and should be better motivated.
- the definition of the evaluation metric appears to be unclear
Decision: Accept-Findings