MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation
Authors: Zexue He
Yu Wang
An Yan
Yao Liu
Eric Y Chang
Amilcare Gentili
Julian McAuley
Chun-Nan Hsu
Venue: EMNLP 2023 Main (2023)
Keywords: Curated Datasets
NLP for Healthcare
Pre-trained Language Models
TL;DR: We propose M4, a new medical benchmark for developing and evaluating language models for healthcare, wherein we analyze their effectiveness and limitations.

Abstract: Curated datasets for healthcare are often limited due to the need of human annotations from experts. In this paper, we present MedEval, a multi-level, multi-task, and multi-domain medical benchmark to facilitate the development of language models for healthcare. MedEval is comprehensive and consists of data from several healthcare systems and spans 35 human body regions from 8 examination  modalities. With 22,779 collected sentences and 21,228 reports, we provide expert annotations at multiple levels, offering a granular  potential usage of the data and supporting a wide range of tasks. Moreover, we systematically evaluated 10 generic and domain-specific  language models under zero-shot and finetuning settings, from domain-adapted baselines in healthcare to general-purposed state-of-the-art large language models (e.g., ChatGPT). Our evaluations reveal varying effectiveness of the two categories of language models across different tasks, from which we notice the importance of instruction tuning for few-shot usage of large language models. Our investigation  paves the way toward benchmarking language models for healthcare and provides valuable insights into the strengths and limitations of adopting large language models in medical domains, informing their practical applications and future advancements.

MEDEVAL:
A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation
Zexue HeL, Yu Wang , An Yan Yao Liu Eric Y. Chang
Amilcare Gentili .Julian McAuley Chun-Nan Hsu
University of California, San Diego, La Jolla, CA, United States Veterans Affairs San Diego Healthcare System, San Diego, CA, United States
Veterans Affairs National Artificial Intelligence Institute, Washington, DC, United States {zehe, yuw1 64, ayan, yal004, e8chang Jeccsd. edu {agentili, jmcauley, chunnan]@@csoe edu
Abstract
Ditritution of Rlpor Mozul
Curated datasets for healthcare are often lim- ited due the need of human annotations from experts. In this paper, present MEDEVAL, multi-level, multi-task, and multi-domain med- ical benchmark facilitate the development of language models for healthcare. MEDEVAL is comprehensive and consists of data from sev- eral healthcare systems and spans 35 human body regions from 8 examination modalities. With 22,779 collected sentences and 21 21,228 reports, we provide expert annotations at mul- tiple levels, offering a granular potential us- age of the data and supporting wide range of tasks. Moreover, we systematically evalu- ated 10 generic and domain-specific language models under zero-shot and finetuning settings, from domain-adapted baselines healthcare general- purposed state-of-the-an large lan- guage models (e.g., ChatGPT). Our evalua- tions reveal varying effectiveness the two categories language models across different tasks, from which we notice the importance of instruction tuning for few-shot usage of large language models. Our investigation paves the way toward benchmarking language models for healthcare and provides valuable insights into the strengths and limitations of adopting large language models in medical domains, in- forming their practical applications and future a
Leh
JOHAM
kore
(a) Distribution of modality and body parts in MEDEVAL
Domann Experts
Aittiguos tabal Adsiomml
Blanddggaatt Nowninng
Plazer Clddt Frestchon Peppor Sursmectsaiinn
Document
21,208)
Multi-task expert labels multi-granularity
Figure 1: A summary of the multi-level multi-task and multi-domain medical benchmark (MEDEvAL). Clas- sification tasks are highlighted in green and generation tasks are highlighted red.
ceneral-purposed
ing, and evaluating these models requires high- quality domain-speciie datasets, which are often challenging to obtain. Previous medical datasets have been collected from healtheare-relatee liter- ature (Dernoncourt and Lee, 2017; Gupta et al., 2021; Jin et al., 2019b; Banarescu et al., 2013) or web pages on the Internet (McCreery et al., 2020; Ammar et al., 2018). While these datasets are large, they may lack quality with heterogeneous topics (e.g., scientific literature about nutrition may offer limited help in the decision-make X-ray analysis). On the other hand, high-quality clinical data is typically obtained by annotating records from healthcare systems like MIMIC-XXI (Johnson et al., 2019; Yan et al., 2021a). However, such data either limited in size (Tsatsaronis et al., 2015), or may only cover certain dominant systems and specific domains such as chest X-rays (John-
1 Introduction
process of an
Recent advanced language models, e.g., GPT-3, ChatGPT, and LLaMa (Touvron et al., 2023a), are effective in various general tasks, suggesting their potential to healthcare use cases, such as allevi- ating the burden human experts in decision- making and patient care. However, training, adapt-
'We will release the data set and source codes at https: //gi //github www. SoexuuHHe/MedEviee The data set will also he available at the Department of Veterans A ffairs Open Data Portal https:/ //ww///wwwaoo data.va. va..ov
WW use domain to describee data sets that are different in distribution, caused by the modality of the examination (c.g..

---

son al., 2016) or eye diseases (Otmakhova et al., 2022). Other approaches involve automatically gen- erating medical corpus using templates (Pampari et al., 2018; Pappas et al., 2018) or using language models (Guo et al., 2023; Tang et al., 2023), but they have been noted to be limited in diversity, complexity, and quality (Gupta et al., 2021).
limitations of LLMs in healthcare domains, guid- ing the appropriate use of LLM-assisted healthcare decision-making systems in the future. Overall, our contributions are summarized as:
We propose a large-scale medical benchmark, namely MEDEVAL, with a broad coverage for various tasks and domains to facilitate future research in clinical NLP.
To tackle the aforementioned challenges and facilitate research in clinical NLP, we introduce MEDEVAL, a large-scale medical benchmark with multi-level curated labels for multiple tasks and multiple domains. MEDEVAL comprises 22,779 sentence-level datapoints from radiology reports, including expert-crafted classification labels (c.g.. abnormality identification labels) and ground truth for generation tasks (e.g., disambiguated rewrit- ings). Additionally, we include 21,228 complete reports with axpert-annolated medical codes for dis- ease classification (e.g.. for ankle radiology studies) and golden output for generation tasks (c.g., sum- marization of radiology reports). Besides the abil- ity support multi-tasks at different levels, MEDE- VAL's 's uniqueness also lies in its diverse data cover- age for different body parts (such as chest, foot, and ankle) and different modalities (X-rays, ultrasound, etc.), and the incorporated novel tasks/data that are collected from the U.S. Department of Veterans Affairs (VA) health care system nationwide. To the best of our knowledge, MEDEVAL represents the first expert-curated medical NLP benchmark that is both comprehensive and lcrle-scale. MEDE- VAL will be released to facilitate future research.
We provide expert annotations for multiple tasks with multi- -ranulaartuy,, from sentence classification and rewriting, to report classifi- cation and summarization.
We systematically evaluate various language models, and shed light on the strengths and weaknesses of these models for healtheare ap- plications.
2 Related Work
Existing medical bench- marks are typically collected from the following resources. First, data is crawled from public re- sources, such as biomedical journals and literature like PubMed Central (PMC) or Semantic Scholar Ammar et al. (2018), or healthcare-related web pages such as MQP (McCreery et al., 2020) which was proposed by collecting COVID-19 FAQs from the Internet. Those data are heterogeneous and may lack relevance for assisting certain clinical purpose.
Medical Benchmark
On the other hand, high-quality benchmarks are extensively collected from real-world healtheare systems, such as the Medical Information Mart for Intensive Care databases (MIMIC (Johnson et al., 2016, 2019, 2023)). Several works have been proposed based on their databases. For instance, MIMIC-XXR (Johnson et al., 2019) is dataset consisting of pairs of radiology images and reports of chest X-ray exams. MIMIC PERform Dataset (Charlton et al., 2022) comprises physiological sig- nals related to critically-ill patients. Additionally, Edin et al. (2023) collected document summary pairs labeled with diagnosis and procedure codes from Johnson et al. (2023). Though their collection may be one of the most comprehensive, there are many domains not included. A more recent attempt was made to alleviate the incompleteness concern by introducing M3 (Otmakhova al., 2022), multi-domain medical benchmark that incorporates multi-level expert annotations. By only consider- ing studies on ophthalmology. M3 offers limited help in providing a fully comprehensive solution.
We further conduct comprchensive evaluation of multiple offe--ff-thearl language model base- lines, including domain-adappee PLMs followed by in-domain fine-tuning (e.g., fine-tuned BERT (De- vlin et al., 2018)) and peneraa-purposed LLMs uti- lized with few-shot in-context learning (e.g., Chat- GPT). We evaluate their performance on sentence- level and document-level NLU and NLG tasks. We observe the effectiveness of both categories of models in different healthcare tasks, with sur- prisingly comparable performances from LLMs only using few-shot learning to domain-adaptee PLMs in certain generation tasks. Our compre- hensive evaluation indicates language models are strong candidates in medical tasks whose data is already seen/simin /simila to their training data. Our in- vestigation provides insights into the potentials and
X-ray, CT, Ultrasound, etc) and examined body parts (c.g. chest, abdomen, etc).

---

Muli-LLvd
Malti-Takk
Multi-Demain
Clest
Alnirnutty Meniiication
Asyun gitaral disiinu
Ground Tratk: Alvas!
KIDNEY
Asyatt Suoes ere wond in she eet aishes.
Ground Trait: abternal
Chasification New 253
Chest
Ausiguit
Zxpat: Langs cleas
Grann Tratiz Andioaaar.
Sentence
Ground
Tosch 779
Generation 2,509
Doombiguated Rewriting
Inpar: cnndiuttinal ailonstte atressoiabll. Alronval
Groond Troi:: We ooreal
Foot
Jopar: Ghars fiee flnd ex iloegfied Absonsaf Xe
Groond Groond Trank: fire ilooited.
MEDEVAL
Clost
Clavefication
Report Coding Clavefication
Inpar. Misinal paudly alepace dueare wikee tee ligals any reletariesios...
Ground Trustt Edoss youliee: Presesonsa oxganie All other axbssmn
Free
Jnpar: Transsnneere oxpetions of the fort pcoaned maalographi..... Grund Fruitt Bethorppaty: Puntby Crovenndayy Pootter: All Ofher Segutie
Decument Talal 21,22
Chest
CHEST SHOGLE VIW Putrss ow pocessale desce in
9899177
The jesion has rosibal oleeing the iof congteeent ungle na on 069677 There wil granskomater Greand Troh Acce duage lang fioligs Aes sp sied candor pacessker device dhet sot sons 1900017
Generntion 003
Repect Sammarization
FOOT OR MORE USERS, Preves, (neel Suo flen collated digefiowe sinwmaline No eralnser antooppat woldd
Groand Trush to dhe up five whire war algiffosst aloureatly derergee
Jxpan Theee steas ripteasts
resided callpseyyrroooo buoway flomse ohe Andd rigle fhulo
The old ferture ooplosp and nereal elganee with sox deriiyssg calino formsss ooled Groand Trath, incoppinccy fuacher of doal finat saimass cvnaal svowei rekatiouki valk slghht avouv of old.
Figure 2: Dataset composition of MEDEVAL MEDEVAL is a large-scale benchmark composed of 22,779 report sentences and 21,228 reports, covering multiple exam modalities on diverse body parts.
To complement these existing efforts, we col- lect a large-scale dataset of real medical reports from another healthcare system that offers broader coverage including 35 human body regions from 8 examination modalities (e.g., X-ray, CT, etc.).
tion capabilities of recent LLMs, where biomedical problems are addressed through prompting LLMs in zero-shot few-shot settings. This approach has been utilized in various applications, such medical report summarization (Otmakhova et al., 2022), medical writing (Biswas, 2023), and medi- cal named entity recognition (Hu al., 2023), etc.
Language Models for Healthcare Large pre- trained language models are being widely adopted solve healtheare tasks. One line of research in- volves adapting general language models to the biomedical domain through continuous training on domain-specific data and tasks. For instance, Yan et al. (2021b) enhanced BERT with contrastive learning for chest report generation. ClinicBERT (Huang et al., 2019) was proposed by continuously training BERT clinic notes using masked lan- guage modeling, and Yan al. (2022) developed RadBERT by continuously training BERT on a vast collection of radiology reports. Other adaptations of BERT, such as BioBERT (Lee et al., 2020), Blue- BERT (Peng et al., 2019), SciBERT (Beltagy et al., 2019), and BioMegatron (Shin et al., 2020), in- volved training on large publicly available medical corpora like PubMed or Semantic Scholar. Fur- thermore, LLMs of alternative architectures have also been employed, including BioELMo (Jin et al., 2019a), BioBART (Yuan et al., 2022), and BioMed- RoBERTa (Gururangan et al., 2020a). Another research direction capitalizes on the generaliza-
In this work, we propose MEDEVAL, a multi- level data with curated annotations at various gran- ularity to comprehensively evaluate the strengths and limitations of LMs in healthcare.
Dataset Design
MEDEVAL (shown in Figure 2) is designed with multiple NLU and NLG tasks at both the sentence and document levels, based on medical data col- lected from two different healthcare databases. Our data covers diverse combinations of human body parts and examination modalities. We first intro- duce the data sources where we collected the text input (Section 3.1). Then we present the expert- annotated ground truth labels² created by our med- ical team* (Section 3.2).
We use "ground truth labels" to represent both discrimi- native labels for NLU and golden sentences NLG tasks. *See description of the medical team in Appendix A.3.

---

Discase Codes of MIGEEVLL
3.2
Expert Labels and Evaluation Tasks
Enlarged Candioneddastiaam
Candioneguly
Lang Opacity
Lung Lesion
Edema
Consolidation
Atelectasia
Preumochorax
3.3 Sentence-level Labels
Pleural Effisioo
Pleural Other
Support Deviees
Pacumonia
Diskocation
Detovieensis
Fiscture
Goot
NLU Tasks Identifying sentences with certain diagnostic properties practical use case in a real-world healtheare system. For example, iden- tifying if a report sentence implies an abnormal finding about the patient or not. To test if language models can capture the medical semantics of sin- gle sentences, we first include abnormal sentence identification into our evaluation pool. We use the sentence-level corpora and the associated abnor- mality labels to classify abnormal sentences.
Metatarsus Primas Varus
Swelling
Psoriasis
Enthesopathy
Hammer Toe
Osteomyeliii
Mass
Arthriis
Mlanus
Rheumatood
Cypd
Hardware
Emsiio
Pes Cavass
Coalition
Subluxation
Fracture
Nodule
Rupure
Hallux Valgus
Pneumonia
Arthritis
No Finding
Table Report discase codes covered MEDEVAL.
3.1 Input Data Composition
Sentence-Level Corpora The sentence-level cor- pora used in this study are sourced from two well- constructed datasets: the sentence-level OpenI- annotated dataset (Dsmnnn-Fuushan et al., 2016), which consists of sentences from chest studies, and the VA-annotated dataset (He et al., 2023b), which includes sentences about different body parts ex- amined by different modalities. These datasets have undergone de-identification,, completion of missing terms and uniqueness checks. More de- tails about the data preprocessing is given in Ap- pendix A. We use the officially released versions of the Openl-annotated and VA-annotated datasets. In addition, we provide new annotations for sentence- level tasks on these data sources.
Ambiguous sentences appear in radiology re- ports mainly due to the use of medical jargon whose meaning is different from daily usage, contradic- tory findings within the same sentence, or gram- matical errors that mislead interpretation (He al., 2023b). Accurate identification of such sentences is crucial, as they impede patients' comprehension of diagnostic decisions, leading to potential treat- ment delays and irreparable consequences. To the best of our knowledge, a novel task proposed recently, current LMs may not readily include such task into its pre-training stage. Therefore, eval- uation of this task allows us to investigate how language models perform when the tasks are unfa- miliar. We leverage the report sentences and their associated ambiguous labels, and our medical team re-examined and re-annotated the labels for am- biguous sentences.
Report-Level corpora We collect the raw radiol- ogy reports from two distinct sources: (1) text cor- pus from MIMIC-XRR, which comprises records related to human chests (Johnson et al., 2019), (2) text corpus from the databases of nationwide gov- ernment healthcare system. We randomly collect data points about different body parts and exam modalities, resulting in multiple domains under dif- ferent data distributions. The distribution of the domain is illustrated in Figure 1. The collected data are processed with automatic de-identification. followed by thorough human inspection to verify that no private information about patients doc- tors is disclosed hinted in the text. We also employ an offline paraphrasing tool (Damodaran, 2021) to revise the text data collected from the second source. The paraphrasing is followed by an- other human inspection to filter out any unqualified records where the rewriting deviates significantly from the original report. The resulting data set can be considered "synthesized" and containing no privacy information but retaining realistic clinical conditions as the source data.
NLG Task Expanding beyond the previous am- biguous sentence identification, we include the task of sentence disambiguation as a sentence-level gen- eration task. Proposed in He et al. (2023b), sen- tence disambiguation aims to rewrite an ambiguous sentence in way that its diagnostic findings are more explicitly expressed while at the same time, the original content of the report sentence is faith- fully maintained. This requires rewritten sentences to avoid the change of the original pathological findings or introducing new findings. Similar to ambiguous sentence identification, disambiguated rewriting presents a challenging generation task, not only because both the data and task formula- tion not likely to be covered in the pre-training stage of existing language models, but also because there are two objectives that need be optimized at the same time. this task, based on the am- biguous sentences and their associated diagnostic labels, our medical team manually created the dis-
For each evaluation task, we split the data in a ratio of 7:1:2 for vrait/

---

ambiguated rewritings as the ground truth.
from this category customize fit the tasks of MEDEVAL, with their corresponding training data, For NLU tasks at both levels, we follow the eval- uation setting of Yan et al. (2022) and investigate how: BERTbase (Devlin et al., 2018), RadBERT (Yan et al., 2022), BioBERT (Lee et al., 2020),
3,4 Document-level Labels
NLU Task To access if language models can cap- ture the key findings of a radiology report, we con- sider Report Codes Prediction as an evaluation task. This task involves categorizing reports into specific diagnostic codes based the mentioned patholog- ical findings. Therefore, different from sentence- level abnormality identification, this task requires a multi-label multi-class classification. Our medical team manually labels the medical codes of each report. Detailed information regarding the codes is provided in Table 1. More details about the expert- labeling procedure are provided in Appendix A.
clinicalBERT (Huang et al., 2019), BlueBERT (Peng et al., 2019), and GeMMeddReBBERT--aass (Gururangan al., 2020b) perform MEDEVAL. More details about those models are included in Section
For the sentence-level NLG task, we follow the the setting of He et al. (2023b) by evaluating: (1) style transformer (Dai et al., 2019) which trans- fers the original sentence into a less ambiguous style, (2) PPLM (Dathathri et al., 2020) which adds perturbation to LM to move the (re-)generation towards a less ambiguous direction, (3) DEPEN (He et al., 2021a) which built upon PPLM and only re-generates ambiguous tokens detected be- fore, and (4) MedDEPEN (He et al., 2023b), a biomedical-adaptee DEPEN by introducing con- trastive pre-training. Each work has included a transformer-basee language model. We refer the reader to the original papers for more details.
NLG Task Automatic medical summarization plays a crucial role in healthcare literature, by pro- viding concise summaries, it saves time and manual effort for medical professionals when assessing the effectiveness of medical interventions. In our evalu- ation, we include report summarization as task to assess the generation capability of language models. The impression section in each report serves summary that captures the supportive evidence for clinical decisions. To ensure data quality, we con- duct a manual inspection of all collected <report, impression> pairs, filtering out any pairs where the impression does not align with the corresponding report. It is worth noting that the curated parallel data of reports and summaries provide valuable support for future work in related fields.
For the document-level NLG task, we follow the setting of Yan et al. (2022) and customize previ- ously adapted BERT-based models used before for the summarization task.
3.5.2 Prompted LLMs
We include the following general-purposs large language models to test their generalization in the healthcare domain: (1) GPT3: GPT-style large language models with 175B parameters (Brown et al., 2020). We use davinci-003.
3.5 Evaluated Language Models
We evaluate two categories of language models with MEDEVAL (I) domain-adaptee pre-trained language models (Adapted PLMs), which are train- able models adapted on certain domain data, and (2) general-purpose large language models (Prompted LLMs) which used by zero/few-sho prompting.
(2) Chat- GPT;: GPT-style large language model trained with Reinforcement Learning from Human Feed- back (RLHF). We use GPT3.5 5-ttb.. (3) Vieuna- 7B (Chiang et al., 2023): The finetuned version of LLaMa-7B (Touvron et al., 2023a) with 70K user-shared ChatGPT conversations, which capa- ble of generating more detailed and well-structured answers. (4) BioMedLM' a 2.7B GPT-style lan- guage model trained exclusively on biomedical ab- stracts and papers from The Pile(Gao et al., 2020).
3.5.1 Domain-adapted PLMs
Recent literature found it is effective to adapt pre- trained language models to certain narrow domains such as biomedical text by continued training step on domain-specific data (Gururangan et al., 2020a), following which we take pre-trained (or gener- ally adapted) language model, and test it the MEDEVAL test set. We also fine-tuned the models
We prompt those LLMs under zero/few-shot set- tings, where we randomly select the examples from the training set of each task to compose prompts 5
'The results presented are based on models evaluated as of the time of paper acceptance. We've added more results (e.g. on GPT4, I. a.a, etc.) and will continue to include the newest ones in the Appendix E.
"https://crfm.st" htt.::mmm..saaoorr.ee/////2///// biomedlm.hte html

---

Chest
Miscellanenus Domains
Models
Abnormality
Ambiguity
Abnormalityf
Ambiguity
BERT
0.9791
0.9893
0.9607
0.9749
RadBERT
0,9794
0,9869
0.9640
0.9813
BioBERT
0.9791
0.9862
0.9614
0,9743
Adapted PLMs with Fine-Tuning
ClinicalBRRT
0.9809
0.9874
0.9588
0.9736
BlueBERT
0,9803
0.9867
0.9601
0.9775
BioMed-ReBERII
0.9569
0.9758
0.9776
0.9788
zero-shot ChatGPT
0,9277
0,6584
0.8880
0.5206
few-shot ChatGPT
0,9498
0,5831
0.9099
0.5354
zero-shot GPT-3
0.8762
0.8742
0.8243
0.6448
L.LMs Prompted by 7ero/Few Shot
few-shot GPT-3
0.9215
0.8320
0.9054
0.6371
zero-shot Vicuna-7B
0.6987
0.2130
0.7261
0.3739
few-shot Vicuna-7B
0.8071
0.0785
0.8166
0.2844
zero-shot BioMed 1.M
0.6679
0.3485
0.6273
0.3726
few-shot BioMedLM
0.7905
0.6804
0.7638
0.6804
Table 2: Evaluation (accuracy) over categorics of PI.Ms abnormality identification ambiguity identification tasks (sentence-level NLU). Bold: highest performance. Underlined: lowest.
Chest
Miscellangus Domains
Model
Disumbiguation
Content Distortion
Disambiguation
Content Distortion
BLBU4
BLEU4
A.Arram
AAAra
A.Arram
A.Aonaa
Style Transfer
0.5010
0,0510
27.92
0.3110
0.2350
31.17
Adupted PLMs
PPLM
0,3860
0,1150
57.88
0,2700
0.1460
60.14
with Fine-Tuning
DEPEN
0,5000
0.0520
60.48
0,3530
0.0470
67.86
McdDEPEN
0.4960
0.0320
57.88
0.4810
0.0090
68.88
2cro-shot ChatGPT
0.6337
-0,0297
60.73
0.6539
0.1483
60.64
few-shot CharGPT
0,5875
0.0000
68.92
0.6370
0.0815
67.98
zero-shot GPT-3
0.6799
-0.0132
61.78
0,8022
0.1528
61.05
LLMss Prompted
few-shot GPT-3
0.6139
0.0000
76.33
0.7146
0.0607
77.09
by Zero/Few Shots
zero-shot Vicuna-7B
0.6230
0.0693
66.65
0.6771
0.3653
64.64
few-shot Vicuna-7B
0
0.1914
62,55
0,4811
0.2739
63.72
zcro-shot BioMed LM
0.2211
0.0066
23.40
0.1528
0.1416
24.11
few-shot BioMed LM
0.1386
-0.0262
23,30
0.3933
0.3640
23.48
Table 3: Evaluation on disambiguated rewriting Tasks (sentence-level NLG). We report the disambiguation score, content distortion score (where smaller content distortion indicates higher fidelity), and BLEU4 score. Bold: the best performance. Underlined: the worst.
times. We report the test results with the prompts which obtain optimal results on the validation set. See Appendix C for more details.
Accat). Therefore, higher distortion indicates a lower content fidelity.
Results and Discussion
3.6 Evaluation Metrics
In this section, We first present the results for sentence-level NLU tasks (Ambiguity Identifica- tion and Abnormality Identification) in Table 2, then sentence-level NLG task (Disambiguated Rewriting) in Table 3, finally document-level NLU (Code Prediction) and NLG (Report Summariza- tion) tasks in Table and Table 5.
For NLU tasks, we report classification metries in- cluding accuracy and Fl scores. For NLG tasks, we report BLEU and ROUGE scores with respect to the ground truths labeled by our medical team. For sentence-level generation tasks (i.e., rewriting), to evaluate the objective of disambiguation, we follow the setting of He et al. (2023b) to report accuracy decrements of the ambiguity classifier (AAccam) the disambiguation metric. To eval- uate the rewriting fidelity, we report the content distortion score, which is defined as the decre- ment of the accuracy from an abnormality classifier
The Effectiveness of Instruction Tuning While BioMed LM is the first large language model cus- tomized for the biomedical domain, we observe that it does not outperform adapted PLMs and most prompted LLMs in the majority of tasks. Particu-

---

Chest
Foot
Ankle
Model
avg Accuracy
avg EMR
avg Accuracy
avg. FMR
avg. Accuracy T
avg FMR
BERT
0.8779
0.2263
0,9754
0.5635
0.9787
0.6141
RadBERT
0.8785
0.1941
0,9710
0,4910
0,9773
0.5710
Adapted PLMs
BioBERT
0.8782
0.2400
0,9750
0.5617
0.9801
0.6266
with Fine-Tuning
ClinicaBBERT
0.8780
0.2341
0.9731
0.5372
0,9798
0.6224
BlueBERT
0.8843
0.2380
0,9703
0.5939
0.9761
0.5752
BioMed-ReHFRIa
0.8579
0.1415
0.9692
0.4631
0.9752
0.5522
zero-shot ChatGPT
0,5272
0,1024
0,9621
0.4449
0,9660
0.4491
few-shot ChatGPT
0.6485
0.1951
0,9621
0.4186
0,9690
0.4875
GPT-3
0.2744
0.1424
0,9621
0.4449
0.1887
0.6273
LLMs Prompted
few-shot GPT-3
0.8160
0.1805
0,9617
0.4908
by
pero-shot Vicuna-78
0.8216
0.0672
0.9617
0.4186
0.9691
0.4908
Zero/Few Shots
few-shot Vicuna-7IB
0.8228
0.0782
0.5156
0.1041
0.9122
0.4153
few-shot BioMed LM
0.8320
0.0689
0.9667
0.4664
0.9719
0.4980
Table 4: Evaluation report codes prediction Task (Document-level NLU). We report the average accuracy over all classes of diseases and the exact match rate (EMR) between predictions and labels. Bold: the highest performance. Underlined: the lowest.
Miscellaneous Domains
Model
ROUGE-1
ROUGE-21
ROUGE-L*
Som f BLEU4f
BERT
20.48
7.46
18.57
46.52
RadBERT
20.96
7.63
18.90
47,50
30.77
BioBERT
20.79
7.62
18.78
47.19
30.49
Adapted PLMs
ClinicaBEERT
21.22
7,85
19.18
48.26
30.81
with
BlueBERT
20.83
7.78
18.90
47.51
30.93
Fine-Tuning
BioMed ReBERTs
21.19
ChaiGIT
24.64
22.05
54,66
32.30
few-shot ChaiGPT
24.96
8.43
22,23
55.62
35.43
zero-shot GPT-3
24.06
8.67
21.52
54.24
24.43
Ms Prompted
few-shot GPT-3
24.73
9.14
22.16
56,03
34.72
by
Vicuna-72
20.93
18.71
ZerofFow Shots
few-shot Vicuna-7B
21.42
7.26
19.22
47.90
22.00
acro-shot BioMed LM
17.70
5.11
16.49
39.30
15.43
fow-shot BioMed LM
12.15
3,50
11.22
26.87
20.27
Table 5: Evaluation on report summarization task (Document-leve NLG). We report the Rouge scores and BLEU4 scores. Bold: the highest performance. Underlined: the lowest.
larly, BioMed LM has been found to be the weakest performer in tasks such as sentence identification, disambiguated rewriting, and report summariza- tion. We would like to highlight that, unlike other prompted LLMs such ChatGPT, GPT-3, and Vi- cuna, BioMed LM lacks an Instruction Tuning step its model training. This omission significantly impacts BioMed LM's ability to generate replies following the instructions from the given options. In zero-shot NLU tasks, only 40% of the test cases receive appropriate responses at the sentence level and the qualified rate drops to less than 1% the document level (so did not report the results in Table 4). In few-shot report codes prediction, the recument-bbsed prompts often exceed BioMed LM's maximum threshold of 1024 tokens, result- ing in query errors. In generation tasks, BioMed LM keeps returning irrelevant text. Our manual inspection reveals that the outputs rarely adhere
to the given instructions in prompts or address the queries. This is further supported by the remark- ably low BLEU or ROUGE scores in Table 3 and Table 5. We provide more discussions in Appendix D.1. These findings underscore the significance of Instruction Tuning and establish it as a crucial step when adapting prompted LLMs for specialized applications like healthcare decision maaiii..-making....
In the remainder of this section, we focus on addressing more intriguing questions based on average performance across range of baselines (e.g.. the average accuracy of adapted PLMs versus prompted LLMs), where we exclude BioMed LM from further consideration.
Discussion Task Type and Granularity In this section, we aim determine the proficiency of language models different levels and tasks. To achieve this, we begin by calculating the average accuracy scores of all adapted PLM baselines and

---

Scrtence Mirdiiattonn
100
Repart Oodes Prodicionn
Model Family
shot
NLU (Aocuracy1)
NLG (BLEU;)
DG-
Acc
Individual
Average
Individual
Average
D-shot
0.78
51.22
ChaiGPT
0.79
47.15
Few-sbot
0.79
43.08
malkyy
Amidgaly
GPT-3
O-shot
0.66
0.76
49.08
55.90
Adapied
Prampted
RLM
Prompled
Sewtshat
0.86
(a) Sentence-level NLU
(h) Document-level NI.U
Vicuna-P7
D-shot
0.71
0.73
50.74
50.08
Fow-sbot
0.75
49.42
0-shot
0.72
50.35
Average
Few-sbot
0.80
51.74
Table 7: Average accuracy and BLEU of various LM families with zero/few shots.
Shambiiaadoon Tanrit SLEU
Frude 189-00 100000L
Prongond
ProM
Sentence-level NI.G
(d) Document-level NI.G
Common v.s. Rare Domains In Table 6, we explore the impact of the domain on language mod- els in the healthcare field. We compute the av- crage accuracy of adapted PLMs and prompted LLMs in abnormality identification v.s. ambiguity identification. We consistently observe higher per- formance from both adapted PLMS and prompted LLMs when working with data from the chest do- main compared to miscellaneous domains. This superior performance can be attributed to the sim- ilarity between the chest data we tested and the pre-training data of the language models chest- related healthcare text is widely available in the public domain and can be included in the training corpus of PLMs. Similarly, LMs are expected excel abnormality identification tasks, which are a common research topic in current literature.
Figure 3: Average performance of adapted PLM and prompted L.LM on different tasks and at different levels.
Ahnormality 1
Ambiguity 1
Model
Chest
Miscellaneoous
Chest
Miscellaneous
Adapted PLM
0.9758
0.9526
0.9836
0.9621
Prompted LLM
0,8635
0.8451
0,5399
0.4893
Table 6: Average accuracy of adapted PI.Ms and prompted LLMs in NLU over different domains
prompted LLM baselines in sentence identification tasks. Similarly, we compute the average accuracy of adapted PLM and prompted LLM baselines in document-level code classification task.
First, examining the results presented in Figure 3, we observe that both adapted PLMs and prompted LLMs perform relatively similarly across differ- ent data levels. However, it becomes apparent that adapted PLMs outperform prompted LLMs in NLU tasks, no matter whether it's on the sentence or document level. This suggests that fine-tuning pro- vides more effective means of injecting specific knowledge about narrow domains or tasks. On the other hand, consistently superior performance of prompted LLMs compared adapted PLMs ob- served generation tasks, both the sentence and document levels. This can be attributed to multi- ple advantages of large-scale pre-training such as a larger model size or the benefits HFRL in the LLMs we utilized, such as ChatGPT. These models demonstrate a capability to generate language that is more akin to human-like expressions, thereby achieving better generation scores. These imply that fine-tuning PLM models can be viable choice for NLU tasks, while prompting-bast based L.LMs may be more suitable when healthcare professionals re- quire an AI writer to help their work.
The most challenging scenario arises when both the data and task unseen, specifically in the case of ambiguous identification within the mis- cellaneous domain. In such situations, there are limited or no examples available the public do- main. Therefore, querying language models with (zero) few-shot learning proves to be less effective.
Family of LLMs and Few Shot Learning In this analysis, examine the behavior of different language models (LLMs) with varying numbers of shots across different tasks. We calculate the aver- age accuracy ChatGPT, GPT3, and Vicuna-7B NLU tasks and the average BLEU scores in NLG tasks. Additionally, we consider the average perfor- mance achieved in zero-shot or few-shot settings (Table 7). From the table, it is evident that in most cases, providing additional examples assists LMs in making predictions for NLU tasks. However, in NLG tasks, no consistent trend is observed, indicat- ing the need for further research discover optimal prompts. We do not observe clear advantage any specific LLM family over others, suggesting

---

that the choice of the optimal LLM family for given task may vary on case-by-case basis.
It is important highlight that the responsible and safe usage of biomedical data is critical re- quirement Al for healtheare, especially in the use case of large language models which are noticed to suffer from different kinds of potential harms (Leino et al., 2019; Lloyd, 2018; He et al., 2021b; Xu et al., 2022: He et al., 2022) and weaknesses (Ribeiro et al., 2020; Stuart-Ulin, 2018; He et al., 2023a). Therefore, we strongly recommend that our benchmark be used in conjunction with expert auditing to ensure the highest level of safety in real-world applications.
5 Conclusion
We introduce MEDEVAL, a multi-task, multi-level, and multi-domain medical benchmark designed to serve as a comprehensive testbed for advanced language models. Through extensive evaluation experiments, we thoroughly analyze the capabil- ities and limitations of current LLMs in tackling various medical tasks, such the effectiveness of instruction tuning and the performance disparities between adapted and prompted LMs in NLU and NLG tasks. Our findings provide valuable insights and serve as a handbook for future research in uti- lizing LLMs to enhance healthcare practices.
Acknowledgments
The authors would like to thank NVIDIA corpo- ration for the award of RTX A6000 GPUs CNH through the Applied Research Accelerator Program.
6 Limitations
our efforts to provide a comprehensive testbed for current advanced language models, we have in- cluded multiple tasks. However, we acknowledge that there may be other tasks of interest that could have been analyzed, such as medical named entity recognition, multi-document report summarization, etc. We plan expand the range of test tasks in future iterations of the MEDEVAL benchmark. We'd d like note that due to computing constraints, we were unable to evaluate some large language models such as Vicuna-600 60B or OPT-175B (Zhang et al., 2022). Our evaluation was focused on pop- ular large language models with reasonably large sizes. In future work, we consider addressing this limitation by incorporating these larger language models into our testbed.
References
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavat- ula, Iz Beltagy, Miles Crawford, Doug Downey. Ja- son Dunkelberger, Ahmed Elgohary, Sergey Feld- man, Vu Ha, et al. 2018. Construction of the liter- ature graph in semantic scholar. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies. Volume 3 (Industry Papers), pages 84-91.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguis- ric annotation workshop and interoperability with discourse, pages 178-186.
Iz Beltagy, Kyle Lo. and Arman Cohan. 2019. Scibert: A pretrained language model scientific text. arXiv preprint arXiv:/903. 10676.
7 Ethics Statement
Our data underwent rigorous de- de-identification process and were carefully reviewed by human evaluators following strict anonymization criteria. Moreover, the collection of data from real-world healthcare systems has received the necessary IRB approval (DC VAMC protocol 1736644, VASDHS IRB protocol 200086), ensuring compliance with ethical standards. To further ensure ethical usage, before inputting the data into large language mod- els, including commercial ones like ChatGPT, conducted data synthesis and subjected it to addi- tional human inspection. These steps were taken to address any potential ethical concerns associated with the data.
Som Biswas. 2023. Chatgpt and the future of medical writing.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901.
Peter H Charlton, Kevin Kotzen, Elisa Mejía-Mejía, Philip Aston, Karthik Budidha, Jonathan Mant, Cal- lum Pettit, Joachim A Behar, and Panicos A Kyriacou. 2022. Detecting beats in the photoplethysmogram: benchmarking open-source algorithms. Physiologi- cal Measureeet, 438800800

---

Wei-Lin Chiang. Zhuohan Li. Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing spt-4 with 90%* chatgpt quality.
Suchin Gururangan. Ana Marasovié. Swabha Swayandipta, Kyle Lo, lz Beltagy, Doug Downey, and Noah A Smith. 2020a. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 5997- 6007, Florence, Italy. Association for Computational Linguistics.
Suchin Gururangan, Ana Marasović. Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020b. Don't t stop pretraining: Adapt language models to domains and tasks. In Proceedings of ACI.
Zexue He, Bodhisattwa Prasad Majumder, and Julian McAuley. 2021a. Detect and perturb: Neutral rewrit- ing of biased and sensitive text via gradient-based decoding. In Findings of the Association for Com- putational Linguistics: EMNLP 2021. pages 4173- 4181, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Prithiviraj Damodaran. 2021. Parrot: Paraphrase generation for NLU. https:://itthii com/ wtw.rrraaaamnnaaaaaarrrrrrrrrrae
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language models: simple approach to controlled text generation. In International Conference on Learning Representa- tions.
Zexue He, Bodhisattwa Prasad Majumder, and Julian McAuley. 2021b. Detect and perturb: Neutral rewrit- ing of biased and sensitive text via gradient-basee based decoding. In Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 4173- 4181.
Dina Demner-Fushman, Mare D Kohli, Marc B Rosen- man, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDon- ald. 2016. Preparing a collection of radiology ex- aminations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304-310
Zexue He, Marco Tulio Ribeiro, and Fereshte Khani. 2023a. Targeted data generation: Find- ing and fixing model weaknesses. arXiv preprint arXiv:2305. 17804.
Franck Demonnourt and Ji Young Lee. 2017. Pubmed 200k ret: a dataset for sequential sentence classifica- tion in medical abstracts. IJCNIP 2017, page 308.
Zexue He, Yu Wang, Julian McAuley, and Bod- hisatwwa Prasad Majumder. 2022. Controlling bias exposure for fair interpretable predictions. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 5854-5866, Abu Dhabi, United Arah Emirates. Association for Computational Lin- guistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arviv preprint
Joakim Edin, Alexander Junge, Jakob D Havtom, Lasse Borgholt, Maria Maistro, Tuukka Ruotsalo, and Lars Maalée. 2023, Automated medical coding on mimic- ii and mimic-iv: A critical review and replicability study, arxiv preprint arXiv:2304. 139909
Zexue He, An Yan, Amilcare Gentili, Julian McAuley. and Chun-Nan Hsu. 2023b. "Nothing abnor- mal": Disambiguating medical reports via contrastive knowledge infusion. In Proceedings of the 37th AAAI Conference on Artificial Intelligence. ArXiv preprint DXXiv2305.08300.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint 10X1::201000027
Yan Hu, Igra Ameer, Xu 7uo, Xueqing Peng, Yujia Zhou, Zehan Li. Yiming Li, Jianfu Xiaoqian Jiang. and Hua Xu. 2023. Zero-shot clinical entity recogni- tion using chatgpt. arXiv preprint ar 233: 2303.16416.. 16416.
Zhen Guo, Peiqi Wang. Yanwei Wang, and Shangdi Yu. 2023. Dr. Ilama: Improving small language models in domain-speciife qa via generative data augmenta- tion. arXiv preprint ar 2tt.::20000...
Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint 1904.0590.
Vivek Gupta, Prema Bharti, Pegah Nokhiz, and Harish Karnick. 2021. Sumpubmed: Summarization dataset of pubmed scientific articles. Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the I 1ith International Joint Conference on Natural Language Processing: Stu- dent Research Workshop, pages 292-303.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ileus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya. et al. 2019. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590-597.

---

Qiao Jin, Bhuwan Dhingra, William W Cohen, and Xinghua Lu. 2019a. Probing biomedical embeddings from language models. NAACI. HILT 2019, page 82.
Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018. emrqa: A large corpus for question answering on electronic medical records. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Lunguage Processing, pages 2357-2368.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Co- hen, and Xinghua Lu. 2019b. Pubmedqa: A dataset for biomedical research question answering. In Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- lernational Joint Conference on Natural Language Processing (EMNLP-UCNLP), pages 2567-2577
Dimitris Pappas, Ion Androutsopoulos, and Harris Papa- georgiou. 2018. Bioread: A new dataset for biomed- ical reading comprehension. In Proceedings of the Eleventh International Conference on Language Re- sources and Evaluation (LREC 2018).
Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Benjamin Moody, Brian Gow, wi- H Lehman, et al. 2023. Mimic-iv, a freely accessi- ble electronic health record dataset. Scientific data, 10(1):1.
Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Trans- fer learning in biomedical natural language process- ing: An evaluation of bert and elmo on ten bench- marking datasets. In Proceedings of the 2019 Work- shop on Biomedical Natural Language Processing (BioNLP 2019), pages 58-65.
Alistair FW Johnson, Tom Pollard, Nathaniel R Green- baum, Matthew P Lungren, Chih-ying Deng, Yifan Peng. Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. 2019. Mimic-cxr-jpa.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behay- ioral testing of nlp models with checklist. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912.
large pub- licly available database of labeled chest radiographs. arXiv preprint 191.04190...7042.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, freely accessi- ble critical care database. Scienrific data, 3(1):1-9.
Hoo-Chhng Shin, Yang Zhang, Evelina Bakhturina. Raul Puri, Mostofa Patwary, Mohammad Shoeybi, and Raghav Mani. 2020. Biomegatron: Larger biomedical domain language model. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4706.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformaticc,, (666)):2234-222
Chloe Rose Stuart-Ulin. 2018 Microsoft's politically correct chatbot is even worse than its racist one. Quartz Ideas, 31.
Klas Leino, Matt Fredrikson, Emily Black, Shayak Sen, and Anupam Datta, 2019. Feature-wise bias ampli- fication. In International Conference on Learning Representations.
Ruixiang Tang, Xiaotian Han, Xiaoqian Jiang, and Xia Hu. 2023. Does synthetic data generation of 1lms help clinical text mining? arXiv preprint 043300.0..
Kirsten Lloyd. 2018. Bias amplification in ar- tificial intelligence systems. arXiv preprint arXiv:1809.07842.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977
Renqian Luo, Liai Sun, Yingee Xia, Tao Qin, Sheng Zhang. Hoifung Poon, and Yin Liu. 2022. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6). Bbac409.
Clara H McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, and Xavier Amatriain. 2020. Ef- fective transfer learning for identifying similar ques- tions: matching user questions to covid-19 19 faqs. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3458-3465.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 09288.
George Tsatsaronis, Georgios Balikas, Prodromas Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael Alvers, Dirk Weissenborn. Anastasia Krithara, Sergios Petridis, Dimitris Polychronopou- los, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answer- ing competition. BMC bioinformatics, (6(1):1-28.
Julia Otnakhova, Karin Verspoor, Timothy Baldwin, Antonio Jimeno Yepes, and Jey Han Lau. 2022. M3: Multi-level dataset for multi-document summarisa- tion of medical studies. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2022,
pages 3887-3901.

---

Chaoyi Wu, Weixiong Lin, Xiaoman Zhang. Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. Pme-Ilama: Towards building open-source language models for medicine.
Canwen Xu, Zexue Hc. Zhankui He, and Julian McAuley. 2022. Leashing the inner demons: Self- detoxification for language models. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pages 11530-11537.
An Yan, Zexue He, Xing Lu, Jiang Du, Eric Chang, Amilcare Gentili, Julian McAuley, and Chun-Nan Hsu. 2021a. Weakly supervised contrastive learning for chest x-ray report generation. arXiv preprint arXiv:2109. 12242.
An Yan, Zexue He, Xing Lu, Jiang Du, Eric Chang, Amilcare Gentili, Julian McAuley, and Chun-Nan Hsu. 2021b. Weakly supervised contrastive learning for chest X-ray report generation. In Findings of the Association for Computational Linguisties: EMNLP 2021, pages 4009-4015, Punta Cana, Dominican Re- public. Association for Computational Linguisties.
An Yan, Julian McAuley, Xing Lu, Jiang Du, Eric Y Chang, Amilcare Gentili, and Chun-Nan Hsu. 2022. Radbert: Adapting transformer-bas based language mod- els to radiology. Radiology: Artificial Intelligence, 4(4):e210258.
Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, and Sheng Yu. 2022. Biobart: Pretraining and evaluation of a biomedical generative language model.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-traines transformer language models. arXiv preprint arXiv:2005.01068

---

A Data Preparation
more than 15 years of experience in healthcare and doctor who has more than 10 years of ex- perience serving as a of medical research. We follow standard labeling practices, involving multi- ple rounds of iterative review by different experts until Cohen's kappa coefficient reaches 0.85. Any remaining disagreements are collectively resolved.
A.1 Preprocess of sentence-level corpora OpenI In the original OpenI release, many non-sensitive terms were incorrectly masked as "xxxx" by the de-identification software described (Demner-Fushman et al., 2016). Our medical team manully fills in the missing information based on the context of the reports and additional infor- mation associated with it.
B Implementation details
Models All adapted transformers are imple- mented based on the HuggingFace libraries. All prompted LLMs are implemented with its original release in their official webpage or GitHub.
A.2 Preprocess of the document-level corpra
Manual De-identification Criteria We hire hu- man reviewers to manually inspect the reports after the automatic de-identification tools. According to our criteria, we will discard a datapoint if it con- tains
The model sizes are listed the Section
Experiment Hyperparameterss The configura- tions of querying L.LMs are listed follows:
real names of the patient, or the healthcare professions,
GPT-3
ChatGPT
home address, working address, or locations of the patient healthcare professionals.
Engine
text-davinci-003
gpt-3.5-turbo
Temperature Max Tokens
0
200
200
contact information (e.g., phone number) about the patient, or healthcare professionals.
Return N
1
Top P
1
Frequency Penalty
0
In the second round of human inspection about de-identification, 99.8% of the data are well de- identified in the automatic stage, and 0.2% of the data are discarded.
Presence Penalty
0
Table 8: Configuration of ChatGPT and GPT-3
The configurations for fine-tuning adapted lan- guages models are: learning rate=0.0001, weight decay=0, optimizer=Adam, training epoch= 10, batch size=64, max length=256, All codes are im- plemented with Python3.8 and PyTorch1.7.1 with CUDA 10.1. operated on Ubuntu (16.04.7 LTS) server with 2 NVIDIA GeForce GTS A6000 GPUs. Each has memory of 49GB.
Diseases Code Preparation Before experts ex- amine the disease codes of a report, we first group data by their resources. For reports sourced from MIMIC-XRR, we employ CheXpert (Irvin et al., 2019), rulc-based automatic labeler to generate pseudo-labeld for the diagnostic codes. Each label has three options: positive, negative, and unknown. In the case of reports from the second source, customized rule-based automatic labeler called pyConText NLPX, which generates pseudo-codes according to the keywords of each domain. In the last step, our medical team manually reviews, correct the codes where there is a conflict (e.g.. positive "no finding" appears with certain positive diseases), and writes down the correct codes for each report.
C Prompts used for Querying LLM
C.1 Number of examples in few-shot settings
In our study, we maintain a balanced and unbiased approach by setting the number of examples equal to the number of classes when prompting the lan- guage models (especially in NLU tasks). Addition- ally. we explore alternative numbers of examples, such 1, 5, 7, or 9. The NLU experiment results presented in our paper utilize 2-shot approach. while the NL. results employ a 3-shot approach.
A.3 Medical Team
Our medical expert team consists members, including 2 senior board-certifie radiologists with
Phttps://huggingfaceccc comnodeels

---

C.2 Templates Used in Prompting
it is normal abnormal, wi wioht saying anything else. Sentence: O Label:
For zero-shot settings we design prompts using the following template:
Two-shot classification for Abnormality: We are here to address identification. Given sentence from radiology report, written by radiologist, please tell if this sentence indicates abnormal findings or not. Here are 2 examples.
We are here to address new task, [TASK NAME]. Given [LEVEL NAMEJ, written by radiologist, please [TASK CONTENTJ. Now here is new [LEVEL], tell me [TASK CONTENT], without saying anything else. Input: O Label:
task, abnormal
For few-shot settings we insert examples ex- tracted from the training set of each task into the prompts by:
Sentence: there is likely left basilar opacity. Label: abnormal.
Sentence: unchanged exam without acute abnormality. Label: normal.
We are here to address new task, [TASK NAME]. Given [LEVEL NAME], written by radiologist, please tell [TASK CONTENT]. Here are [N] examples. [EXAMPLE CEXAMPLE N] Now here is new [LEVEL], tell me [TASK CONTENT], without saying anything else. Input: O, Label: The options for the placeholder are:
Now here is new sentence, tell me if it is normal or abnormal, without saying anything else.
Sentence: O Label:
Zero-shot Classification for Ambiguity: Here is a task to classify ambiguous sentences. Given a sentence in a radiology report, written by radiologist, please tell if it is ambiguous. A sentence is defined to be ambiguous because of (1) medical jargon with meanings different from everyday general usage, such as unremarkable; (2) contradictory findings in the same sentence; (3) misleading grammatical errors such as no period between full sentences.
TASK: (abnormal identificatio, ambiguous identification, rewrite an ambiguous sentence to be less ambiguous, summarization
LEVEL NMAE: (sentence, report).
TASK CONTENT:(Tell if the sentence indicates abnormal findings not, Tell it is ambigu- ous, sentence is defined to be ambiguous because of (1) medical jargon with meanings different from everyday general usage, such as unremarkable; (2) contradictory findings in the same sentence; (3) misleading gram- matical errors such as no period between, full sentences, Observed pathology findings, The summary
Now given a new sentence, answer me with "ambiguous" or "unambiguous", without saying anything else. Sentence: O Label:
Two-shot Classification for Ambiguity: Here is a task to classify ambiguous sentences. Given a sentence in a radiology report, written by radiologist, please tell if it is ambiguous. defined to be ambiguous because of (1) medical jargon with meanings different from everyday general usage, such as unremarkable; (2) contradictory findings in the same sentence; (3) misleading gramaatical errors such as no period between full sentences. Here are 2 examples.
N: 3.
sentence is
In the following sections, we provide some exam- ples of the prompts we use to query LLM.
C.3 Sentence-Level Tasks
For sentence-level tasks, there are the following different types prompts for NLU tasks:
Zero-shot Classification for Abnormality We are here to address new task, abnormal identification. Given sentence from radiology report, written by radiologist, please tell if this sentence indicates abnormal findings or not.
Sentence: lungs are unremarkable. Label: ambiguous.
Sentence: unchanged chronic appearance of the left lung. Label: unambiguous.
Now here is a new sentence, tell me if
Now given a new sentence, answer me with

---

"ambiguous" or "unambiguous", wi thout saying anything else.
identifie. Diagnostic: Abnormal. Rewrite: Cardiomegaly and hiatal hernia Without an acute abnormality identified. Now given a new sentence, answer me with its rewrite, without saying anything else. Sentence: O Rewrite:
Sentence: O Label:
Similarly, we will add more examples for the prompts built for more than two-shot situations. For NLG tasks, have the following prompts:
C.4 Document-level Tasks
Zero-shot Rewriting tasks Here is task to rewrite ambiguous sentences to be less ambiguous. Given sentence in radiology report, written by radiologist, please rewrite it to be more explicit about the diagnostic decision reflected in the sentence, however, maintain the main meaning of the original sentence. A sentence is defined to be ambiguous because of (1) medical jargon with meanings different from everyday general usage, such unremarkable; (2) contradictory findings in the same sentence; (3) misleading grammatical errors such as no period between full sentences. Now given a new sentence, answer me with its rewrite, without saying anything else. Sentence: (). Rewrite:
For NLU tasks, we use the following prompts:
Zero-shot Summarization Tasks Here is summarization task. Given radiology report written by radiologist, please write summary of the report. Answer me with its summary only, without saying anything else. Report: O.. Summary:
Three-shot Summarization Tasks Here is summarization task. Given a radiology report written by a radiologist, please write summary of the report. Here are 5 examples.
Report: clinical history 62-year-old male with bilateral bunions. please perform weighted views. a lateral weight bearing view of bilateral feet as well as lateral non-weight bearing view of bilateral feet are studied. three views of the right foot show no fracture dislocation foreign body pathologic calcification swelling. the joint spaces noticeable. there is deformity of the hallux valgus. the angle of pitch is within normal limits. three views of the left foot show no fracture dislocation foreign body pathologic calcification or soft tissue swelling. the joint spaces are not noticeable. hallux valgus minor. the angle of pitch is within normal limits. Summary: bilateral hallux valgus deformities. no acute osseous abnormalities. ssn7312ptc1 no. 1157. Report: there has been no significant change in the patient's condition since the patient's exam which was earlier in the day. the heart size is normal and the lungs are free of disease. on the left base is again noted small granulom. Summary: for a active disease in the chest there is no evidence.
Three-shot Rewriting tasks Here is a task to rewrite ambiguous sentences to be less ambiguous. Given sentence in radiology report, written by radiologist, please rewrite it to be more explicit about the diagnostic decision reflected in the sentence, however, maintain the main meaning of the original sentence. A sentence is defined to be ambiguous because of (1) medical jargon with meanings different from everyday general usage, such unremarkable; (2) contradictory findings in the same sentence; (3) misleading grammatical errors such as no period between full sentences. Here are three examples.
soft tissue not
Sentence: Lungs are unremarkable.
Diagnostic: Normal. Rewrite: Lungs are normal.
Sentence:
The lung volumes are low normal. Diagnostic: Normal. Rewrite: The lung volumes are in the lower range of normal limit.
Sentence:
Cardiomegaly and hiatal hernia without an acute abnormality
Report: the cardiovascular-mediisttt silhouette is normal. it's not unusual

---

for pulmonary vessels. the bones appear to be intact. Summary: chest x-rays within normal ranges. no change in date 2010-06-28.
from BioMed LM in the following:
Sentence Level, NLU-Zero-shot Note that we expect the model to return a single word from "nor- mal", "abnormal", "ambiguous", or "unambigu- ous". However, BioMed LM returns the following output:
Now given a new report, answer me with its summary only, without saying anything else. Report: O Summary:
D More Discussions
Prompt: Given a sentence from a radiology report, written by radiologist, please tell if the following sentence indicates "normal" or "abnormal" findings. Sentence: median sternotomy wires appear intact. Label: Returned Text: median sternotomy wires appear intact.
D.1 Case Studies of BioMed LM
BioMed LM, GPT-sttle LLM trained on PubMed, lacks instruction fine-tuning in its training process. As a result, the model's outputs often lack a unified format, making it challenging to conduct follow-up statistical evaluations. To assess the model's per- formance, we introduce the concept of a qualified rate, which represents the proportion of test cases where the model provides relevant prediction for the given task, such identifying abnormalities by including only one the "normal" or "abnormal" in the response. We report the qualified rate here:
Sentence Level, NLU-Two-shot In this case, we expect the model to return single word "normal", "abnormal", "ambiigouu", "unambiguous", How- ever, the output from BioMed is:
Prompt: Here is task to classify ambiguous sentences. Given sentence in radiology report, written by radiologist, please tell if it is ambiguous, A sentence is defined to be ambiguous because of (1) medical jargon with meanings different from everyday general usage, such as unremarkable; (2) contradictory findings in the same sentence; (3) misleading grammatical errors such as period between full sentences. Here are examples. Sentence: lungs are unremarkable. Label: ambiguous.
Sentence
Report
Zero-shot NLU
40%
1%
Few-shot NLU
92%
85%
Table 9: Qualified Rate of BioMed LM on NLU tasks
In zero-shot settings, BioMed LM struggles to adhere the instructions in the prompt, generat- ing outputs that are freestyle and not aligned with the expected format. In a few-shot setting, partic- ularly document-level tasks, the length of the prompts exceeds the maximum input capacity of BioMed LM (1024 tokens). To address this issue, we employ two solutions: (1) chunking the input into 1024-token segments and (2) discarding test that exceed the maximum length. However, both solutions have drawbacks. The chunked completed input approach leads more "freestyle" outputs, resulting in a lower qualified rate. On the other hand, discarding examples introduces a high variance in the statistics. We show some examples of the "freestyle" outputs at the end of this section. Considering these challenges, we treat BioMed LM exception and exclude it when analyzing results across different categories of LMs. Nonethe- less, we emphasize the importance of further inves- tigation into biomedical LLMs for future research.
Sentence: unchanged chronic appearance of the left lung. Label: unambiguous.
Now given a new sentence, answer me with "ambiguous" or "unambiguous".
Sentence: findings: there is a cystic, approximately 3. with its epicenter in the left submandibular region, with extension into the left submental space, left sublingual space, left base of tongue, and superiorly into the left peripharyngeal space. Label:
2. 6
6.
cm,
Returned Text: cystic mass. Sentence: approximately 3.
there is x 6. 6 cm, mass with its epicenter in the left submandibular region, with extension
cystic,
Example Output of BioMed LM We show some examples unqualified "free-style" outputs

---

into the
Returned Text:
\\[Lingula])
)] scarring.
Sentence Level, NLG-Zero-shot Here we show an unqualified example the ambiguous rewriting task, where the output does not follow the instruc- tion of "without saying anything else".
Document Level, NLU-Zero-shot In this task, we expect the output to have prediction for every disease. However, BioMed LM just repeats the disease code the output.
Prompt: Here is task to rewrite ambiguous sentences to be less ambiguous. Given sentence in radiology report, written by a radiologist, please rewrite it to be more explicit about the diagnostic decision reflected sentence, however, maintain the main meaning of the original sentence. sentence is defined to be ambiguous because of (1) medical jargon with meanings different from everyday general usage, such as unremarkable; (2) contradictory findings in the same sentence; (3) misleading grammatical errors such as no period between full sentences. Now given a new sentence, answer me with its rewrite, without saying anything else. Sentence: aorta appears unchanged as compared to the study. Diagnostic: Abnormal. Rewrite: Returned Text: \!... aorta appears unchanged as compared to the study.
Prompt: Here is a task to classify the observations of the chest reports. Given a radiology report written by radiologist, please predict if every one of the following 14 observations is positive, negative, or unknown. The 14 observations are No Finding, Enlarged Cardiomediastinum, Cardiomegaly, Lung Lesion, Lung Opacity, Edema, Consolidation, Pneumonia, Atelectasis, Pneumothorax, Pleural Effusion, Pleural Other, Fracture, Support Devices. Answer me only with the tuples of each observation and its preidction, without saying anything else. There should be tuples in your answer and tuples are separated by comma. Report: bibasilar the lungs
the
changes in atelectatics. are otherwise clear. the cardiomedical silhouette and the hila are normal. right port-a-cath ends at the cavoatrial junction. there's pneumothorax in this case. No pleural effusion. Mentioned observations:
In this unqualified example, the BioMed LM does not follow the instruction and just output an irrelevant sentence.
Returned Text: No finding, Enlarged Cardiomediastinum, Lung Lesion, Lung Opacity, Edema, Consolidation, Pneumonia, Atelectasis, Pneumothorax, Pleural Effusion, Pleural Other.
Prompt: Here is task to rewrite ambiguous sentences to be less ambiguous. Given sentence in radiology report, written by radiologist, please rewrite it to be more explicit about the diagnostic decision reflected in the sentence, however, maintain the main meaning of the original sentence. sentence is defined be ambiguous because of (1) medical jargon with meanings different from everyday general usage, such as unremarkable; (2) contradictory findings in the same sentence; (3) misleading grammatical errors such as no period between full sentences. Now given new sentence, answer with its rewrite, without saying anything else. Sentence: apparent scarring within the lingula otherwise unremarkable. Diagnostic: Abnormal. Rewrite:
Cardiomegaly,
Document-Level, NLG-Two-shot We show an example of "freestyle" output from BioMed LM Prompt: Here is summriizatin task. Given a radiology report written by a radiologist, please write a summary of the report. Here are 2 examples.
Report: clinical history 62 62-year-old male with bilateral bunions please perform weighted views.
lateral weight bearing view of bilateral feet as well as lateral non-weight bearing view of bilateral feet are studied.
three views of the right foot show
no fracture dislocation foreign body

---

pathologic calcification or soft tissue swelling. the joint spaces are not noticeable. there is a deformity of the hallux valgus. the angle of pitch is within normal limits. three views of the left foot show no fracture dislocation foreign body pathologic calcification or soft tissue swelling. the joint spaces not noticeable. hallux valgus minor. the angle of pitch is within normal limits. Summary: 1 bilateral hallux valgus deformities. 2 no acute osseous abnormalities. ssn7312ptc1job no. 1157.
in our Github repository):
Report: there has been no significant change in the patient's condition since the patient's exam which was earlier in the day. the heart size is normal and the
the lungs are free of disease. left base is again noted small granulom. Summary: for a active disease in the chest there is no evidence.
Report: right and left feet have severe hammertoes. calcaneus on the left shows an achilles spur. the severe hallux valgus configuration of the feet is observed. dominant finding was the degree of luxation and deviation of first metatarsal proximal phalangeal joint..
Summary:
Returned Text: 1 bilateral hammer toes.
2 no acute osseous abnormali ties. ssn7312ptc1jor no. 1157.
Report: the patient has been in the clinic for routine checkup. The patient has
E More Results
We considered popular LLMSs such as GPT3, Chat- GPT, and Vicuna-7B (the instruction finetuned LLaMa) in the main context. But we also keep in mind that LLMs are rapidly developing and we are following up by adding the newest models into our evaluation. In Table 10, Table 11, Table 12, and Table 13, we provide more results with "0 LLaMa2 (Touvron et al., 2023b), LLaMa2-chat (Touvron al., 2023b), GPT-NeoX citegpt-neox- 20b, PMC-LLaMa (Wu al., 2023), BioGPT (Luo al., 2022) etc, which align with the findings al- ready in the paper. We will keep working on follow- ups and add more evaluation results here (and also
4

---

Chest
Miscellaneous Domains
Models
Abnormality
Ambiguityt
Abnormalityt
Ambiguity?
zero-shot BioGPT
0.7521
0.3012
0.6994
0.0543
Adapted PLMs with Fine-Tuning
few-shot BioGPT
0.5966
0.2864
0,5990
0.0672
zero-shot PMC-LLaMa
0.6663
0.2986
0.6189
0.0465
few-shot PMC-LLaMa
0.7660
0.2606
0.7162
0.0574
zero-shol GPT4
0,9299
0.6596
0,9547
0,8462
few-shot GPT4
0,9145
0.7925
0,9417
0.8906
zero-shot GPT-NeoX
0.6371
0.2747
0,5460
0,0651
LLMs Prompted by Zero/Feww Shot
few-shot GPT-NooX
0.5986
0.2578
0,5000
0.0648
zero-shot LLaMa2-chat
0,4626
0.2864
0,4075
0.0679
few-shot LLahhh-chat
0,6551
0.2859
0.5685
0.0679
zero-shot LLaMa2
0.5568
0,3150
0.6474
0.0679
few-shot LLaMa2
0.4728
0.2864
0,4240
0.0679
Table 10: More results on sentence-level NLU tasks.
Chest Content Distortion
Miscellaneous Domains
Models
Disumbiguation AAce...
Disambiguation AAoca
Content Distortion
BLBU4
BLEU4
A.Acca
AAAcss
zero-shot BioGPT
0,8184
0,2971
19.54
0,8623
0.4179
19.03
Adapted PLMs with Fine- Tuning
few shot BioGPT
0,8019
0,3036
22.43
0,8651
0.4382
22.32
zero shot PMC LLMMa
088119
0.4389
19.03
0,9235
0,5393
18.77
PMC-LLaMa
0.5577
0.2244
22.61
0.6719
0.3213
zero-shot GPT4
0.7921
-0.0297
60.61
0,7562
0.1652
GPT4
63.66
zero-shot GPT-NNoo
0,4125
0,0462
38.23
0.4786
0.2225
35.16
Prompted
few-shat GPT-NeoX
.0.056
0.0396
24.65
0.0876
0.2517
25.91
by
zero-shot LLaMa2-chat
0,4059
0.0033
53.23
0,3379
0.1314
$3.20
Zemo/Few Shot
few shot LLaMMa-chatt
0.4521
0.0198
57.26
0.4480
0.1066
52.59
zero-shot L.LaMa2
0.1320
29.95
few-shot Ma2
0.1386
0.257
10.77
0.4044
0.1101
10.18
Table 1I: More results sentence-level NLG tasks.
Foot
Ankle
Models
avg Accurucy!
avg EMR
avg Accuracy"
avg EMR1
avg Accuracy!
avg EMR1
zero-shot BioGPT
0.8276
0.0654
0.9222
0,0010
0.9299
0.0020
Adapted PLMs with Fine-Tuning
few-shot BioGPT
0.7561
0.0030
0.9618
0.4196
0.9688
0.4860
zero-shot PMC-LLaMa
0.8290
0.0663
0.9228
0,0010
0,9206
0.0010
few-shot PMC-LLaMa
0.7549
0.0039
0.9604
0.4044
0.9228
0.0020
zero-shot GPT-NcoX
0.8291
0.0663
0.9238
0.0033
0.9318
0.0194
few-shot GPT-NenX
0.7876
0.0537
0.9617
0.4186
0.9691
0.4908
LLLSs Prompted
zero-shot LLaMa2-chat
0,3192
0.0294
0.96177
0,4186
0.9691
0,4908
by ZerolFeew Shots
few-shot I.I MMa2-chal
0.8192
0.0505
0.9617
0.4186
(19691
0.4908
zero-shot LLaMa2
0.7995
0.0591
0.9518
0,3171
0,9669
0.4597
fow-shot LLaMa2
0.8331
0.0722
0.9594
0.3898
0,9478
0.1
Table 12: More results on document-level NLU tasks.

---

Miscellancous Domains
Models
ROUGE-I1
ROUGE-21
ROUGE-L*
Sum
BLEU4 f
zero-shot BioGPT
18.1577
6.5828
16.9262
41.6667
8.10
Adapted PLMs with Fine-Tuning
few-shot BioGPT
12.3569
3,8041
11.6335
27.7945
zero-shot PMC-LLaMa
11.6813
3.8402
10.8875
26.409
7.52
few-shot PMC-LLaMa
14.5339
3,9377
13.0060
31,4776
8.46
zero-shot GPT4
59.8081
34.6459
55.1671
149.6211
$3.82
few-shot GPT4
16.1813
31.6360
51.5054
139.3228
44.16
zero-shot GPT-NeoX
18,3383
6.6764
17.0308
42.0455
11,01
LLMss Prompted
few-shot GPT-NeoX
12.3496
3.8196
11.6543
27.8235
8.11
by
zero-shot LLaMa2. chat
21.0446
6,9982
18.8494
46,8923
21.97
Zero/Few Shots
few-shot LLaMa2-chat
17.8764
5,6633
15.5410
39,0806
21.54
zero-shot LLaMa2
18.0240
5,8142
16.5904
40,3801
15.37
few-shot LLaMa2
21.4236
7,8157
20.0528
49.2921
Table 13: More results on document-level NLG tasks.


Reviews:
[{"review_id": "547P7jiU7O", "reviewer": "Reviewer_yBca", "summary": "", "questions": "Overall that's an interesting benchmark, and I have several questions for this paper: \n1. Given that this benchmark is tailored for the medical domain, and considering its current size of approximately 22k samples spanning multiple diseases, how scalable or expandable is the M4 benchmark for broader medical applications?\n2. Are there specific medical domains or tasks that the M4 benchmark currently does not cover?\n3. What guided the selection process for the inclusion of the 35 specific human body regions and the 8 examination modalities in the M4 benchmark? Were there any particular criteria or considerations?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive and inspiring review! We are delighted that you appreciate the effort of constructing M4 and its impact on advancing future AI for Healthcare research. Please see our responses below:\n\n**W1: Evaluation depth not clear in abstract** \n>We will revise our abstract accordingly to highlight the comprehensive evaluation outcomes, for example, by adding a conclusive summary of the results. \n \n\n**W2: Potential bias in expert annotation**\n>We acknowledge your concern and agree that biases may be introduced. While some degree of bias might be unavoidable, we have taken measures to minimize its impact. Our work involves large-scale data collection, random sampling, and multi-round annotation with a voting process. \n\n**W3 and Q2: Things that are not covered in M4**\n>As mentioned in our Limitations section, we acknowledge that there are tasks that we were unable to annotate due to budget constraints, resulting in their exclusion from the current M4, such as named entity recognition and multi-document summarization. We also note the absence of data from body parts such as the eye, dermatological, or dental examinations, as they are not significantly collected by our medical system.    \nWe’d like to highlight that M4 is quite expandable and we plan to enhance M4 further by collecting data from additional medical systems and providing corresponding expert labels.\n\n**Q1: Scalability and expandability of M4**\n>M4's distinctiveness lies in its extensive data which covers diverse body domains, multiple medical examination modalities, and varying input/labeling levels. This unique diversity positions M4 for substantial scalability and extensibility across a wide range of medical applications. For example:  \n>1.  Task Expansion: It is flexible for the future inclusion of many engaging tasks by adding corresponding automatic or human annotations into M4.  \n>2.  Broader Impact: The annotated data and tasks within M4 can serve as valuable resources for refining specialized medical tools, such as report paraphrasing or summarization assistants, as well as training medical LLMs, which can serve as foundational models for diverse healthcare applications.\n\n**Q3: Reason for Inclusion of 35 body parts and 8 examination modalities**\n>We randomly sampled reports and sentences from our corpus and reported the number of body parts and modalities included. Therefore, the numbers are supposed to reflect the distributions in the corpus, and in turn mirror the distributions of the real healthcare situation during the specified period at the institution\n\n\n**More LLM results:** \n>We considered popular LLMs such as GPT3, ChatGPT, and Vicuna-7B (the instruction finetuned LLaMa) in our paper. But we also keep in mind that LLMs are rapidly developing and we are following up by adding the newest models into our evaluation.  Here we provide more results with GPT4, LLaMa2, LLaMa2-chat, etc, which align with the findings already in the paper. We will keep working on follow-ups as future work:\n> \n> **Sentence-level NLU tasks**:      \n> |               | Chest        |         | Miscellaneous |      Domains   |\n|:---------------------:|:--------------:|:---------:|:-----------------------:|:---------:|\n|        Model       | Abnormality | Ambiguity| Abnormality | Ambiguity |\n|     zero-shot GPT4    |    0.9299   |   0.6596  |         0.9547        |   0.8462  |\n|     few-shot GPT4     |    0.9145   |   0.7925  |         0.9417        |   0.8906  |\n| zero-shot LLaMa2-chat |    0.4626   |   0.2864  |        0.4075         |   0.0679  |\n|  few-shot LLaMa2-chat |   0.6551    |   0.2859  |         0.5685        |   0.0679  |\n|    zero-shot LLaMa2   |    0.5568   |   0.3150  |        0.6474         |   0.0679  |\n|    few-shot LLaMa2    |    0.4728   |   0.2864  |         0.4240        |   0.0679  |\n> \n> **Sentence-level NLG tasks:**  \n> |                      |                            |        Chest      |           |                 |     Miscellaneous     |          |\n|:------------------------:|:------------------:|:-----------:|:-----------:|:--------------------------:|:-----------:|:-----------:|\n|            Model            | Disambiguation  | Content Distortion | BLEU4    | Disambiguation   | Content Distortion | BLEU4 |\n| zero-shot GPT4         | 0.7921           | -0.0297   | 60.61     | 0.8562                 | 0.2652    | 60.36     |\n| few-shot GPT4          | 0.7591           | 0.0000    | 63.66     | 0.7753                 | 0.0697    | 61.67     |\n| zero-shot LLaMa2-chat  | 0.4059           | 0.0033    | 53.23     | 0.3379                 | 0.1314    | 53.20     |\n| few-shot LLaMa2-chat   | 0.4521           | 0.0198    | 57.26     | 0.4480                 | 0.1066    | 52.59     |\n| zero-shot LLaMa2       | 0.1320           | 0.0165    | 29.95     | 0.0921                 | 0.0157    | 39.60     |\n| few-shot LLaMa2        | 0.1386           | 0.257     | 10.77     | 0.4044                 | 0.1101    | 10.18     |\n> \n>**Document-level NLU tasks:**   \n>|                 |   Chest |         |  Foot|         | Ankle|        |\n|:-----------------------:|:-------------:|:----------:|:-----------:|:----------:|:------------:|:----------:|\n|          Model             | avg accuracy      | avg EMR      | avg accuracy     | avg EMR      | avg accuracy      | avg EMR      |\n| zero-shot LLaMa2-chat  | 0.3192       | 0.0294 | 0.9617       | 0.4186 | 0.9691       | 0.4908 |\n| few-shot LLaMa2-chat   | 0.8192       | 0.0505 | 0.9617       | 0.4186 | 0.9691       | 0.4908 |\n| zero-shot LLaMa2       | 0.7995       | 0.0591 | 0.9518       | 0.3171 | 0.9669       | 0.4597 |\n| few-shot LLaMa2        | 0.8331       | 0.0722 | 0.9594       | 0.3898 | 0.9478       | 0.1135 |\n> \n>(We currently lack the quota to run GPT-4 in this experiment. We are actively working on it and will include the results in the final revision.)\n> \n> **Document-level NLG tasks:**   \n>|                  |              |        |    Miscellaneous    |     Domains     |       |\n|:------------------------:|:--------------------------:|:--------:|:--------:|:----------:|:-------:|\n|            Model            | ROUGE-1                  | ROUGE-2| ROUGE-L| Sum      | BLEU4 |\n|     zero-shot GPT4     |        59.8081        | 34.6459 | 55.1671 | 149.6211 | 53.82 |\n|      few-shot GPT4     |        56.1813        | 31.6360 | 51.5054 | 139.3228 | 44.16 |\n| zero-shot LLaMa2-chat  |        21.0446        |  6.9982 | 18.8494 |  46.8923 | 21.97 |\n|  few-shot LLaMa2-chat  |        17.8764        |  5.6633 | 15.5410 |  39.0806 | 21.54 |\n|    zero-shot LLaMa2    |        18.0240        |  5.8142 | 16.5904 |  40.3801 |  15.37 |\n|    few-shot LLaMa2     |        21.4236        | 7.8157 | 20.0528 |  49.2921 |  18.92 |\n>\n\nWe will include the above discussions in the final revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The M4 dataset is vast, spanning numerous healthcare systems, human body regions, and examination modalities. This ensures a wide application and relevance to multiple medical fields.\n2. The inclusion of expert annotations boosts the credibility and utility of the dataset, addressing the typical challenge in healthcare datasets.\n3.  The authors didn't just provide a dataset but also rigorously evaluated 10 different language models, thus providing a preliminary roadmap for further research. Now there are more and more LLMs : (GPT4, LLaMa, LLaMa2, Bard, etc. ), it will be great if authors put the most recent LLM into evaluation. \n4.  The evaluation reveals practical applications and potential pitfalls of using large language models in the healthcare domain, which can guide real-world applications and be helpful for clinical usage.\n5.  The commitment to release the data and evaluation codes will greatly facilitate the adoption of the benchmark and spur further research.", "reasons_to_reject": "1.  While 10 language models were tested, it's not clear from the abstract how in-depth each evaluation was. Given the variety in datasets, it would be crucial to understand model performance across different tasks and domains.\n2.  As with any curated dataset, there's potential for bias in data selection and expert annotation, which might not be explicitly addressed.\n3. The dataset, while extensive, might still be limited when considering the vastness and diversity of the medical field.\n", "questions_to_authors": "Overall that's an interesting benchmark, and I have several questions for this paper: \n1. Given that this benchmark is tailored for the medical domain, and considering its current size of approximately 22k samples spanning multiple diseases, how scalable or expandable is the M4 benchmark for broader medical applications?\n2. Are there specific medical domains or tasks that the M4 benchmark currently does not cover?\n3. What guided the selection process for the inclusion of the 35 specific human body regions and the 8 examination modalities in the M4 benchmark? Were there any particular criteria or considerations?", "paper_topic_and_main_contributions": "The paper introduces the \"M4\" benchmark, which stands for Multi-level, Multi-task, and Multi-domain Medical benchmark. M4 aims to advance the development of language models for healthcare, addressing the challenge posed by the limited availability of expert-annotated healthcare datasets. This benchmark is extensive, encompassing data from various healthcare systems, covering 35 body regions, and stemming from 8 examination modalities. With a significant amount of sentences and reports, it offers expert annotations at several levels, supporting a diverse range of tasks. The study also rigorously tests 10 language models in zero-shot and finetuning settings, discovering insights into their strengths and limitations in medical applications. The paper promises to release the data and associated evaluation codes to stimulate further research in this domain.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "haINgD6kTd", "reviewer": "Reviewer_7LJK", "summary": "", "questions": "When you are referring to the F1-score metric at line. 389 \"For NLU tasks, we report classification metrics including accuracy and F1 scores.\" are you talking about micro, macro or averaged F1 ?\n\nYou mention L. 430 that \"document-based prompts often exceed BioMed LM’s maximum threshold of 1024 tokens, resulting in query errors.\", but have you considered methods like Fusion-in-Decoder (https://aclanthology.org/2021.eacl-main.74.pdf) to increase context window ?\n\nWhat are the distributions of the documents according to sizes ? Are the documents often exceeding model's maximum input tokens length ?\n\nHave you computed the correlation between human evaluation and automatic metrics like BLUE or ROUGE for the NLG tasks ?\n\nWhy not also using domain specific LLMs like PMC-LLaMa, BioGPT or others ?\n\nHave you tried the Chain-of-Thought (CoT) prompting method on your tasks ?\n\nWhy not using semantic search to collect few-shot examples ?\n\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for the constructive and valuable feedback! Your positive assessment of M4's presentation and the emphasis we gave to privacy and security encourages us. We will address the grammar errors and add the missing reference in our final revision. Here are our responses to your points, which will be incorporated into the final version.\n\n**W1: More open-source generalist LLMs**\n>In our early experiments, we tested LLaMa. Due to the lack of instruction finetuning, we noticed a high failure rate of LLaMa to output valid responses following the instructions. Therefore we instead reported Vicuna (i.e., instruction finetuned LLaMa) results in the paper.     \nWe agree with your comments and include results on newer generalist LLMs (e.g., GPT-NeoX, PMC-LLaMa, etc.) at the end of our response. Please refer to the below Response to Q4. More results can be found in our response to Reviewer yBca.\n\n**Q1: NLU F1 score**\n>We reported the Accuracy and EMR scores for NLU tasks in Tables 2 and 4. We will correct it in L389 in the revision. \n\n**Q2 :Fusion-in-Decoder and Q3 document length distribution**\n> This is a good point. The statistics of the document length distribution are: min = 7 tokens, max = 618 tokens, mean = 84 tokens, std = 48 tokens. We notice the length distribution of few-shot prompts has the following statistics: min = 934 tokens, max=1,545 tokens, mean = 1038 tokens, std=55 tokens. From those, approximately 56.33% of the prompts exceed the 1024-token limit of BioMedLM.    \nWe’d like to note that BioMedLM is a GPT-style model, where Fusion-in-Decoder is not applicable. However, while our current effort primarily aims to create a diverse and vast benchmark for LLMs, we agree with you regarding the significance of techniques/works for long-context language modeling, including Fusion-in-Decoder. We will discuss and encourage more related future work in the revision. \n\n**Q3: human evaluation and automatic metrics**\n>The ground truth of our generation task is provided by experts, forming the basis for automatic metric calculations (such as BLEU, and ROUGE). We didn't carry out separate human evaluations.  \n\n**Q4: More domain-specific LLMs**\n>We appreciate your suggestion. In the last part of the response, we offer additional results involving GPT4, GPT-NeoX-20B, PMC-LLaMa, and BioGPT,  which will be added into the revision. The results align with the observations in the paper.\n\n**Q5 and Q6: Chain-of-Thought (CoT) and semantic search**\n>While our initial goal is to establish a comprehensive medical benchmark and a framework for current LLM evaluation, to which end we only use the default in-context learning setups and basic prompting approaches, we agree with you on the potential of investigating LLM reasoning such as using Chain-of-Thought or exploring different prompting strategies such as using semantic search. Future works based on or inspired by M4 could potentially involve more specialized areas, such as introducing CoT annotations to create M4-CoT or exploring advanced prompting mechanisms. \n\n**More LLM results:**\n> **Sentence-level NLU tasks:** \n>|                       |     Chest    |           | Miscellaneous  |  Domains   |\n|:---------------------:|:-----------:|:---------:|:-------------:|:---------:|\n|         Model         | Abnormality | Ambiguity |  Abnormality  | Ambiguity |\n|     zero-shot GPT4    |    0.9299   |   0.6596  |     0.9547    |   0.8462  |\n|     few-shot GPT4     |    0.9145   |   0.7925  |     0.9417    |   0.8906  |\n|   zero-shot GPT-NeoX  |    0.6371   |   0.2747  |     0.5460    |  0.0651   |\n|   few-shot GPT-NeoX   |    0.5986   |   0.2578  |     0.5000    |   0.0648  |\n|   zero-shot BioGPT    |   0.7521    |   0.3012  |    0.6994     |  0.0543   |\n|    few-shot BioGPT    |   0.5966    |  0.2864   |    0.5990     |   0.0672  |\n|  zero-shot PMC-LLaMa  |   0.6663    |  0.2986   |    0.6189     |   0.0465  |\n|  few-shot PMC-LLaMa   |   0.7660    |   0.2606  |    0.7162     |  0.0574   |\n>   \n> **Sentence-level NLG tasks:**\n>|                     |          |            Chest        |       |  |         Miscellaneous   |       |\n|:-------------------:|:--------------:|:------------------:|:-----:|---------------:|:-------------------:|:-----:|\n|        Model        | Disambiguation | Content Distortion | BLEU4 | Disambiguation | Content Distortion | BLEU4 |\n|   zero-shot GPT4    |     0.7921     |       -0.0297      | 60.61 |     0.7562     |       0.1652       | 60.36 |\n|    few-shot GPT4    |     0.7591     |       0.0000       | 63.66 |     0.7753     |       0.0697       | 67.67 |\n|  zero-shot GPT-NeoX |     0.4125     |       0.0462       | 38.23 |     0.4786     |       0.2225       | 35.16 |\n|  few-shot GPT-NeoX  |     -0.056     |       0.0396       | 24.65 |     0.0876     |       0.2517       | 25.91 |\n|   zero-shot BioGPT  |     0.8184     |       0.2971       | 19.54 |     0.8623     |       0.4179       | 19.03 |\n|   few-shot BioGPT   |     0.8019     |       0.3036       | 22.43 |     0.8651     |       0.4382       | 22.32 |\n| zero-shot PMC-LLaMa |     0.8119     |       0.4389       | 19.03 |     0.9235     |       0.5393       | 18.77 |\n|  few-shot PMC-LLaMa |     0.5577     |       0.2244       | 22.61 |     0.6719     |       0.3213       | 21.07 |\n>\n>It's worth noting that, except for GPT4, achieving an optimal balance between multiple objectives for those LMs (such as BioGPT and PMC-LLaMa) is a challenging task. In other words, they often achieve higher disambiguation scores at the expense of altering the original content significantly during the rewriting.\n>\n> **Document-level NLU tasks**: \n>|                      |     Chest    |         |     Foot     |         |     Ankle    |         |\n|:--------------------:|:------------:|:-------:|:------------:|:-------:|:------------:|:-------:|\n|         Model        | avg Accuracy | avg EMR | avg Accuracy | avg EMR | avg Accuracy | avg EMR |\n|  zero-shot GPT-NeoX  |    0.8291    |  0.0663 |    0.9238    |  0.0033 |    0.9318    |  0.0194 |\n|   few-shot GPT-NeoX  |    0.7876    |  0.0537 |    0.9617    |  0.4186 |    0.9691    |  0.4908 |\n|   zero-shot BioGPT   |    0.8276    |  0.0654 |    0.9222    |  0.0010 |    0.9299    |  0.0020 |\n|   few-shot BioGPT    |    0.7561    |  0.0030 |    0.9618    |  0.4196 |    0.9688    |  0.4860 |\n| zero-shot PMC-LLaMa  |    0.8290    |  0.0663 |    0.9228    |  0.0010 |    0.9206    |  0.0010 |\n|  few-shot PMC-LLaMa  |    0.7549    |  0.0039 |    0.9604    |  0.4044 |    0.9228    |  0.0020 |\n> \n> (We currently lack the quota to run GPT-4 in this experiment. We are actively working on it and will include the results in the final revision.)\n> \n> **Document-level NLG tasks:**\n>|                |         |         | Miscellaneous |  Domains |        |\n|:--------------------:|:-------:|:-------:|:-------------:|:--------:|:------:|\n|             Model         | ROUGE-1 | ROUGE-2 |    ROUGE-L    |    Sum   |  BLEU4 |\n|    zero-shot GPT4    | 59.8081 | 34.6459 |    55.1671    | 149.6211 |  53.82 |\n|     few-shot GPT4    | 56.1813 | 31.6360 |    51.5054    | 139.3228 |  44.16 |\n|  zero-shot GPT-NeoX  | 18.3383 |  6.6764 |    17.0308    |  42.0455 |  11.01 |\n|   few-shot GPT-NeoX  | 12.3496 |  3.8196 |    11.6543    |  27.8235 | 8.11 |\n|   zero-shot BioGPT   | 18.1577 |  6.5828 |    16.9262    |  41.6667 | 8.10 |\n|   few-shot BioGPT    | 12.3569 |  3.8041 |    11.6335    |  27.7945 | 5.26 |\n| zero-shot PMC-LLaMa  | 11.6813 |  3.8402 |    10.8875    |  26.409  |  7.52 |\n|  few-shot PMC-LLaMa  | 14.5339 |  3.9377 |    13.0060    |  31.4776 | 8.46 |\n\n\nWe greatly appreciate your valuable insights and we will incorporate these discussions into our final revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a well written paper which propose a large panel of task and covering most of the SOTA models.\n\nThe paraphrasing process applied for anonymization add a layer of security and leaves less chance of somebody recovering the original document.\n\nThe authors are encouraging reproducibility by giving all the prompts used during the experiments.\n\nThe evaluation automatic methodology of the disambiguation task is very innovative and very interesting.\n\nFew-shot prompting results reported in the tables 1 to 4 are interesting. Models like Vicuna 7B seem to be more often suffering from the additional data than the other models.\n\nThe results show better performances from GPT-3 in few-shot scenario on both NLG and NLU tasks. It would be interesting to see how GPT-4 is performing. Is it following the same trend as GPT-3.5-Turbo or not ? Good question!\n\n", "reasons_to_reject": "It lack of open-source generalist large language models (like LLaMa, GPT-NeoX or/and PMC-LLaMa) to confirm the behavior of those from proprietary ones.\n", "questions_to_authors": "When you are referring to the F1-score metric at line. 389 \"For NLU tasks, we report classification metrics including accuracy and F1 scores.\" are you talking about micro, macro or averaged F1 ?\n\nYou mention L. 430 that \"document-based prompts often exceed BioMed LM’s maximum threshold of 1024 tokens, resulting in query errors.\", but have you considered methods like Fusion-in-Decoder (https://aclanthology.org/2021.eacl-main.74.pdf) to increase context window ?\n\nWhat are the distributions of the documents according to sizes ? Are the documents often exceeding model's maximum input tokens length ?\n\nHave you computed the correlation between human evaluation and automatic metrics like BLUE or ROUGE for the NLG tasks ?\n\nWhy not also using domain specific LLMs like PMC-LLaMa, BioGPT or others ?\n\nHave you tried the Chain-of-Thought (CoT) prompting method on your tasks ?\n\nWhy not using semantic search to collect few-shot examples ?\n\n\n", "paper_topic_and_main_contributions": "The paper is presenting M4, a new biomedical and clinical benchmark meant for PLMs and LLMs evaluation. The authors tried with this benchmark to bring a vast and diversified evaluation protocol by varying data sources, formats, natures and tasks. To make it possible, they collect a set of 5 NLU and NLG tasks from different domains. As part of their works, they introduced a new manually curated dataset based on a set of synthesized clinical notes. Then, they access the performances of eighteen already existing PLMs and LLMs models on this freshly curated benchmark. Finally, they analyze models outputs and metrics to make an in depth analysis of models behaviors.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "L. 32 GPT-3 reference : Language Models are Few-Shot Learners (https://arxiv.org/pdf/2005.14165.pdf)\n ", "Typos_Grammar_Style_and_Presentation_Improvements": "The second example of the Figure 2 have the word \"KIDNEY\" and \"SPINE\" in example four written in uppercase unlike the other ones.\n\nTypo in A.1 \"manully\" rather than \"manually\"."}, "extra_text": {}}, {"review_id": "MOvPOdKTqH", "reviewer": "Reviewer_ZDUq", "summary": "", "questions": "The Open-I and VA datasets were mentioned as sentence-level annotated data. Can the authors provide more details about the original purpose of these datasets, what they were used for, and the form of the data? This applies to the mimic-cxr as well. What were the limitations that led the authors' medical team to re-examine and re-annotate these datasets? A more detailed description of this process is needed.\n\nThe main contribution of this paper lies in the annotation of challenging medical data by experts, making it publicly available to promote advancements in healthcare AI. However, the brief coverage of just about one page for the dataset creation process in the paper is not convincing. An explanation of this section is required.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments. We are encouraged to see your appreciation of our work and its value in the field of AI for Healthcare. We clarify the following details and questions below:\n\n**W1: Lack of details about the main contribution**\n>With respect to confirming de-identification on a report-level basis, we initially obtained raw, anonymized data from our healthcare partners, where personal information was encrypted through automatic software. Subsequently, human reviewers carefully checked each data point to guarantee a higher level of privacy by removing the potential disclosure of personal data, such as names and addresses. Further details and criteria for de-identification are explained in Appendix A.2. More details about the overall annotation process are also explained in the response to W2 below. \n\n\n**W2: Unclear description of the medical team and efforts**\n>Our medical expert team consists of 4 members, including 2 senior board-certified radiologists with more than 15 years of experience in healthcare and a doctor who has more than 10 years of experience serving as a PI of medical research. We follow standard labeling practices, involving multiple rounds of iterative review by different experts until Cohen's kappa coefficient reaches 0.85. Any remaining disagreements are collectively resolved. We will include those details in the revision.\n\n\n**W3 and Q2: Imbalance page assignments about the paper’s focus**\n>The first contribution of M4 is about our data construction with trustworthy expert annotations to facilitate future AI-for-Healthcare research (pages 3-5), and the second is about the systematic evaluation of various language models, via which we aim to shed light on the strengths and pitfalls of these models for healthcare applications (pages 6-8). Specific details about expert labeling and inspection are moved to Appendix A in the submission. Please refer to more details in our previous reply to W2. We will also reorganize our paper in the revision based on your helpful suggestions.\n \n\n\n**Q1: Original OPENI, VA, and MIMIC-CXR datasets**\n> **Original Data and Purpose**:  \n> The OpenI dataset, proposed by [1], contains chest X-ray images and related report sentences. The VA-annotated dataset, originally introduced by [2], comprises general radiology report data, processed and annotated by [3] with abnormalities and ambiguity labels. The MIMIC-CXR [4] database offers x-ray images and corresponding free-text radiology reports for human chests. These datasets are frequently employed for detecting abnormalities or generating medical image reports.\n>\n>**Our Contribution:**   \n>1. For sentence-level tasks, we employ sentences from OpenI and VA datasets, followed by data filtering by our experts to ensure quality. For example, the original sentences in the OpenI dataset are noted to be incomplete due to the oversensitive de-identification software, and repetitive sentences are observed. Our medical experts review each sentence, fill-in the mistakenly-masked tokens (e.g., ''XXXX is clear'' -> ''Lung is clear''), and remove potential redundancy.  For its NLU tasks, we utilize officially released annotations, which our expert team re-examined and resolved annotation inconsistency from previous work. Regarding NLG tasks, our experts wrote down ground-truth sentences for each input. \n> 2. For the report-level aspect, we sample reports from MIMIC-CXR, conducting de-identification sanity checks. Since MIMIC-CXR only contains chest x-ray reports, we collect additional reports from another partner healthcare system which has medical records about different body parts and examination modalities. Our expert team conducted data cleaning on the raw data, (automatic and manual) de-identification, and paraphrasing to guarantee patient privacy. Further expert inspection filters unqualified records where the paraphrases significantly deviate from the original report. Our contribution at this level also encompasses high-quality annotations, where our medical team manually labels medical codes for NLU tasks and creates corresponding summaries for NLG tasks. \n>\n>With the above effort, M4 provides input data with stringent privacy guarantees and high-quality annotations, serving various purposes beyond abnormality identification.   \n>\n> **References:**   \n> [1] Demner-Fushman et al., Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304–310, 2016.    \n> [2] Yan et al., RadBERT: Adapting Transformer-based Language Models to Radiology. Radiol Artif Intell. 2022    \n> [3]  He et al., Disambiguating Medical Reports via Contrastive Knowledge Infusion. AAAI. 2023   \n> [4]  Johnson et al., MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 2019;6.\n\n**Clarification for the title M4:**\n>The four “M”s in the title signify:    \n>1. **M**ulti-level: Incorporating texts from both sentence and report levels;    \n>2. **M**ulti-task: M4's annotated data can be used for diverse tasks from NLG and NLU;   \n>3. **M**ulti-domain: M4 includes varied data domains with different distributions arising from different examination modalities and body parts (see more details in Footnote 2);    \n>4. **M**edical.    \n>\n>We acknowledge your concern and truly appreciate your suggestion. We will incorporate the mention of \"radiology\" into the title, for example, *M4: A Multi-Level, Multi-Task, and Multi-Domain Medical Radiology Report Benchmark for Language Model Evaluation*.\n\n**Reproducibility:**\n> We are happy to address your concern about reproducibility. As we promised in Footnote 1 and aligned with the original purpose of this work (i.e., proposing a public benchmark), we are committed to the release of all data and evaluation codes. This will facilitate the development of future healthcare language models.\n\nWe will include the above details and discussions in the final revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Diverse Task Coverage: The work covers various tasks such as sentence-level, document-level classification, and summarization.\n\nExpert Annotation: Experts have contributed to the annotation.\n\nContribution to Healthcare AI: This research can make a contribution to the field of Healthcare AI.", "reasons_to_reject": "Lack of Details about the Main Contribution: The main contribution of this study lies in having experts directly annotate the challenging clinical areas. Much human annotation was involved, such as confirming de-identification on a report-level basis, or the process through which the medical team annotates. However, there is no detailed explanation of how the crucial human annotation was conducted.\n\nUnclear Description of the Medical Team and Efforts: There is no description of how the medical team was formed(how many? what are their expertise?), or what efforts were made to reduce discrepancies between human annotators.\n\nImbalance in Paper's Focus: Half of the paper seems more focused on solving the data with various models rather than describing the efforts made to obtain high-quality data. The lack of balance between these aspects may detract from the paper's overall effectiveness.", "questions_to_authors": "The Open-I and VA datasets were mentioned as sentence-level annotated data. Can the authors provide more details about the original purpose of these datasets, what they were used for, and the form of the data? This applies to the mimic-cxr as well. What were the limitations that led the authors' medical team to re-examine and re-annotate these datasets? A more detailed description of this process is needed.\n\nThe main contribution of this paper lies in the annotation of challenging medical data by experts, making it publicly available to promote advancements in healthcare AI. However, the brief coverage of just about one page for the dataset creation process in the paper is not convincing. An explanation of this section is required.", "paper_topic_and_main_contributions": "The large-scale medical benchmark, known as M4, encompasses various tasks and domains. It includes not only sentence-level and document-level classification but also other diverse tasks such as summarization. Experts have provided annotations for these tasks, to ensure quality in the benchmark.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The use of \"Multi-Level, Multi-Task, and Multi-Domain Medical\" in the title may lead to misunderstanding, as only radiology reports are utilized. It would seem appropriate to include a reference to radiology-related data in the dataset's name to prevent confusion."}}]

Author rebuttal:

Meta-review: This meta-review assesses the quality of the paper based on three reviews provided by independent reviewers. The paper introduces the M4 benchmark for evaluating language models in the medical domain. It covers a wide range of tasks and domains, including sentence-level and document-level classification, summarization, and more. Expert annotations enhance the quality of the dataset, and it offers insights into the performance of various language models. Below are the reasons to accept and reject, as well as the average Soundness and Excitement scores.

Reasons to Accept:
1. **Broad Task Coverage:** The M4 dataset encompasses various tasks and domains, making it relevant and applicable to multiple medical fields.
2. **Expert Annotations:** The inclusion of expert annotations enhances the credibility and utility of the dataset, addressing a common challenge in healthcare datasets.
3. **Comprehensive Model Evaluation:** The paper evaluates 10 language models, providing valuable insights into their strengths and limitations for medical applications.
4. **Real-world Relevance:** The evaluation sheds light on practical applications and potential pitfalls of using large language models in healthcare, benefiting clinical usage.
5. **Reproducibility Commitment:** The authors' commitment to releasing data and evaluation codes promotes reproducibility and encourages further research in the field.


**Reasons to Reject:**
1. **Limited Evaluation Details:** The abstract lacks clarity on the depth of the evaluation of the 10 language models, especially regarding performance across different tasks and domains.
2. **Potential Data Bias:** The curated dataset may still have biases that are not explicitly addressed, which is a concern in healthcare applications.
3. **Dataset Scope:** Given the vastness and diversity of the medical field, some aspects may not be covered adequately.

The average soundness and excitement scores suggest that the paper has a strong level of soundness in its methodology and arguments and provides exciting insights into the research direction. However, there are concerns about the level of detail in the evaluation and potential data biases, which should be addressed to strengthen the paper's overall quality.
Decision: Accept-Main