Exploring the Boundaries of GPT-4 in Radiology
Authors: Qianchu Liu
Stephanie Hyland
Shruthi Bannur
Kenza Bouzid
Daniel C. Castro
Maria Teodora Wetscherek
Robert Tinn
Harshita Sharma
Fernando Pérez-García
Anton Schwaighofer
Pranav Rajpurkar
Sameer Tajdin Khanna
Hoifung Poon
Naoto Usuyama
Anja Thieme
Aditya V. Nori
Matthew P. Lungren
Ozan Oktay
Javier Alvarez-Valle
Venue: EMNLP 2023 Main (2023)
Keywords: benchmarking GPT-4
radiology
evaluation
large language model
TL;DR: We benchmarked GPT-4 on a range of radiology tasks and found GPT-4 outperforms or is on par with radiology-specific models.

Abstract: The recent success of general-domain large language models (LLMs) has significantly changed the natural language processing paradigm towards a unified foundation model across domains and applications. In this paper, we focus on assessing the performance of GPT-4, the most capable LLM so far, on the text-based applications for radiology reports, comparing against state-of-the-art (SOTA) radiology-specific models. Exploring various prompting strategies, we evaluated GPT-4 on a diverse range of common radiology tasks and we found GPT-4 either outperforms or is on par with current SOTA radiology models. With zero-shot prompting, GPT-4 already obtains substantial gains 
($\approx$ 10\% absolute improvement) over radiology models in temporal sentence similarity classification (accuracy) and natural language inference ($F_1$). For tasks that require learning dataset-specific style or schema (e.g. findings summarisation), GPT-4 improves with example-based prompting and matches supervised SOTA. Our extensive error analysis with a board-certified radiologist shows GPT-4 has a sufficient level of radiology knowledge with only occasional errors in complex context that require nuanced domain knowledge. For findings summarisation, GPT-4 outputs are found to be overall comparable with existing manually-written impressions.

Exploring the Boundaries of GPT-4 in Radiology
Qianchu Liu', Stephanie L. Hyland', Shruthi Bannur', Kenza Bouzid',
Daniel C. Castro', Maria Teodora Wetscherek', Robert Tinn', Harshita Sharma', Fernando Pérez-Garcia' ,Anton Schwaighofer,, Pranav Rajpurkar', Sameer Tajdin Khanna", Hoifung Poon', Naoto Usuyama',
Anja Thieme', Aditya Nori', Matthew P. Lungren', Ozan Oktay' Javier Alvarez-Valle Microsoft Health Futures
2 Harvard University
Abstract
wider range of customised tasks without the need to extensively collect human labels or to perform specialised domain training. Also, with off-the- shelf prompting, applying LLMs is easier than the traditional training pipeline for supervised models.
The recent success of general-domain large language models (LLMs) has significantly changed the natural language processing paradigm towards a unified foundation model across domains and applications. In this paper, we focus on assessing the perfor- mance of GPT-4, the most capable LLM so far, on the text-based applications for radiol- ogy reports, comparing against state-of-the- art (SOTA) ing various prompting strategies, evaluated GPT-4 on a diverse range of common radiology tasks and we found GPT-4 either outperforms or is on par with current SOTA radiology mod- els. With zero-shot prompting, GPT-4 already obtains substantial gains (+s 10% absolute im- provement) over radiology models in tempo- ral sentence similarity classification (accuracy) and natural language inference (Fi). For tasks that require learning datasct-specific style or schema (e.g. findings summarisation). GPT-4 improves with example-based prompting and matches supervised SOTA. Our extensive er- ror analysis with board-certified radiologist shows GPT-4 has a sufficient level of radiol- ogy knowledge with only occasional errors in complex context that require nuanced domain knowledge. For findings summarisation, GPT- 4 outputs are found be overall comparable with existing martally-written written impressions.
While contemporary studies (Nori et al., 2023; Ranjit et al., 2023; Bhayana et al., 2023a) have started to explore the use of GPT-4 in the clinical domain, the readiness of GPT-4 in the radiology workflow remains to be rigorously and systemati- cally tested. In this study, we set the following research questions: (1) How can we evaluate GPT- 4 on its ability to process and understand radiology reports? (2) How can we apply common prompt- ing strategies for GPT-4 across different radiology tasks? (3) How does GPT-4 compare against SOTA radiology-specin models?
models. Explor-
To answer these questions, we established a rig- orous evaluation framework to evaluate GPT-4 on diverse range of common radiology tasks including both language understanding and generation. The evaluation covers sentence-level semantics (natural language inference, sentence similarity classifica- tion), structured information extraction (including entity extraction, disease classification and disease progression classification), and direct application of findings summarisation. We explored various prompting strategies including zero-shot, few-shot, chain-of-thought (CoT)(Wei al., 2022), example selection (Liu al., 2022), and iterative refinement (Ma et al., 2023), and we further experimented with adding self-consistency (Wang et al., 2023) and asking GPT-4 to defer handling uncertain cases to improve the reliability of GPT-4. For each task, we benchmarked GPT-4 with prior GPT-3.5 mod- cls (text-davinee) spective state-of-the-art (SOTA) radiology models. Apart from reporting metric scores, we performed extensive qualitative analysis with board-certified radiologist to understand the model errors by cat- egorising them as ambiguous, label noise, or gen-
1 Introduction
Recently, the emergence of large language mod- els (LLMs) has pushed forward Al performance in many domains; with many GPT-4 (OpenAI, 2023) powered applications achieving and even surpass- ing human performance in many tasks (Bubeck al., 2023; Nori al., 2023). There shift in paradigm towards using a unified general-domain foundation LLM replace domain- and task- specific models. General-domain LLMs enable
and ChatGPT) and the
"Corresponding author: jualvare maallare@@mccrsoftt.co

---

uine model mistakes. We highlight the particular importance of qualitative analysis for open-ended generation tasks such as findings summariastion where GPT-4 may provide alternative solutions. To sum up, our key contributions and findings (in italics) are:
(2023) further tests GPT-4's capabilities beyond language processing towards general intelligence (AGI), exploring tasks such as mathematical prob- lem solving and game playing. Many other studies focus on testing specific capabilities such as rea- soning from LLMs (Liu et al., 2023b; Espejel et al., 2023).
1. Evaluation Framework: We proposed evaluation and error analysis framework to benchmark GPT-4 in radiology. Collaborat- ing with a board-certified radiologist, we pin- pointed the limitations of GPT-4 and the cur- rent task paradigms, directing future evalua- tion pursuits to tackle more intricate and chal- lenging real-world cases and to move beyond mere metric scores.
The evaluation of GPT-4 has also begun to gar- ner interest in the medical field. For example, Lee et al. (2023) discusses the potential advantages and drawbacks of using GPT-4 as an Al chatbot in the medical field. Cheng et al. (2023) investigates pos- sible applications of GPT-4 in biomedical engineer- ing, Nori et al. (2023) evaluates GPT-4 for medical competency examinations and shows GPT-4 perfor- mance is well above the passing score. There have also been few recent studies that evaluate GPT-4 in the radiology domain: Bhayana et al. (2023a.b) show that GPT-4 significantly outperforms GPT-3.5 and exceeds the passing scores on radiology board exams. Other studies have shown great potential from GPT-4 in various radiology applications such as simplifying clinical reports for clinical education (Lyu et al., 2023), extracting structures from radiol- ogy reports (Adams et al., 2023), natural language inference (NLI) (Wu et al., 20236), and generating reports (Ranjit al., 2023). While most of these studies focus on specific application, our study aims for an extensive evaluation to compare GPT-4 against SOTA radiology models, covering diverse tasks and various prompting techniques.
GPT-4 shows a significant level of radiology knowledge. The majority of detected errors are either ambiguous label noise, with few model mistakes requiring nuanced domain knowledge. For findings summarisation, GPT- outputs are often comparable to existing marually-written impressions.
2. Prompting Strategies: We explored and es- tablished good practices for prompting GPT-4 across different radiology tasks.
GPT-4 requires minimal prompting (zero-shot) for tasks with clear instructions (e.g. sentence similarity). However, for tasks needing com- prehension of dataset-specific schema or style (e.g. findings summarisation), which are chal- lenging to articulate in instructions, GPT-4 demands advanced cxample-based prompting.
3. GPTT4 vs. SOTA:
We compared GPT-4 per- formance with task-specific SOTA radiology models for understanding and validating the paradigm shift towards a unified foundation model in the specialised domains.
Beyond prompting GPT-4, continued efforts are being made to adapt LLMs the medical domain via fine-tuning. Med-PaLM and Med-PaLM-2 (Singhal et al., 2022, 2023) improve over PaLM (Chowdhery et al., 2022) and PaLM-2 (Anil et al., 2023) with medical-domain fine-tuning. Yunxiang et al. (2023) and Wu et al. (2023a) further fine-tune the open-source LLaMA model (Touvron et al., 2023) with medical-domain data. Van Veen et al. (2023) adapts LLMs to radiology data with param- eter efficient fine-tuning. While these models offer lightweight alternatives, our study focuses on GPT- as is still by far the best-performing model across many domains and represents the frontier of artificial intelligence (Bubeck et al., 2023).
GPT-4 outperforms or matches performance of task-specific radiology SOTA.
2 Related Work
There have been extensive efforts to benchmark and analyse LLMs in the general-domain. Liang et al. (2023) benchmarks LLMs across broad NLP sce- narios with diverse metrics. Hendrycks al. (2021) measures L.Mss' multitask accuracy across disci- plines. Zheng et al. (2023) explores using LLMs as judge for open-ended questions. Bubeck et al.

---

3 Evaluation Framework
GPT-4 with temperature zero² and self-consistency results with majority voting (indicated by 'sc"). All prompts are presented in Appendix
3.1 Task selection'
We benchmark GPT-4 seven common text-only radiology tasks (Table 1) covering both understand- ing and generation tasks. The two sentence sim- ilarity classification tasks and NLII both require the understanding of sentence-leved semantics in a radiology context, with NLI additionally requir- ing reasoning and logical inference. Structured information extraction tasks (discase classification, disease progression classification, and entity ex- traction) require both superficial entity extraction and inference from with radiology knowledge (e.g. "enlarged heart' implies 'cardiomegaly". For entity extraction, the model must further follow the schema-specific categorisation of entities. Finally, we evaluate GPT-4 on an important part of the radiology workflow: findings summarisation, i.e. condensing detailed descriptions of findings into a clinically actionable impression. These tasks cover different levels of text granularity (sentence-level, word-level, and paragraph-level) and different as- pects of report processing, and hence give us holistic view of how GPT-4 performs processing radiology reports.
3.3 Error analysis with radiologist
The authors did a first pass of the error cases to review easy instances requiring only general syn- tactic and linguistic knowledge (e.g. 'increased pleural effusion' versus 'decreased plcural effu- sion'). We then surfaced the cases where radiology expertise is required to a board-certified radiolo- gist for a second-round review and feedback. For interpretability, we prompted GPT-4 to give an ex- planation after its answer. Reviewing both model answer and reasoning, we categorise each error into: ambiguous,, label noise", or genuine mistake.
4 Experiments
4.1 Sentence similarity classification
Task and model setup In this task, the model receives as input sentence pair and must clas- sify the sentences as having the same, differ- ent meanings. We evaluate the models on two sub-tasks: temporal sentence similarity classifi- cation (MS-CXR-T (Bannur et al., 2023b)) and RadNLI-derived sentence similarity classification. Temporal sentence similarity focuses on temporal changes of discases. For RadNLI, we follow Ban- nur et al. (2023a) to use the subset of bidirectional 'entailment" and 'contradiction' pairs and discard the 'neutral' pairs to convert RadNLI (Miura al., 2021) binary classification task.
3.2 Prompting strategies
Alongside GPT- (gpt-4-32k), we evaluated two earlier GPT-3.5 models: text-davinci- ii00 and ChatGPT (gpt-35-turbo). Model and APT details are in Appendix A. For each task, we started with zero-shot prompting and progressively increased prompt complexity to include random few-shot (a fixed set of random examples), and then similarity- based example selection (Liu et al., 2022). For ex- ample selection, text- text-embeddiig--a ada-002 model to encode the training examples as the candidate pool select n nearest neighbours for each test instance. For NLI, we also explored CoT, as was shown to benefit reasoning tasks (Wei et al., 2022). For find- ings summarisation, we replicated ImpressionGPT (Ma et al., 2023), which adopts dynamic example selection and iterative refinement.
The radiology SOTA for this task is BioViL-T (Bannur et al., 2023a) (a radiology-specite vision- language model trained with temporal multi-modal contrastive learning). The GPT performance is obtained from zero-shot prompting.
OpenAl's general-domaie
Results As shown in Table 2, all the GPT models outperform Bio achieving new SOTA. In particular, GPT-4 significantly outperforms both text dext-davinci-e -003 and ChatGPT on MS-CXR-T, indicating an advanced understanding of disease progression. Error analysis revealed the majority of the GPT-4 (SC) errors are either ambiguous
To test the stability of GPT-4 output, we applied self-consistency (Wang et al., 2023) for sentence similarity, NLI, and disease classification. We re- port mean and standard deviation across five runs of
"The OpenAl API for GPT-4 is non non-deterministic with temperature O. We also explored varying the temperature parameter and found improvement.
"An ambiguous case is when both GPT-4 error output and gold label can arguably correct under different interpreta- tions of labels. For an example, an uncertain pathology could labelled as 'presence' or "absence".
'The majority of our test set comes from data with re- stricted access (c.g., MIMIC (Johnson et al., 2019)).
the label is wrong and model is correct

---

Table Results overview, GPT-4 either outperforms or is on par with previous SOTA. New SOTA is established by GPT-4 sentence similarity and NLI (absolute improvement for accuracy and Fj reported). GPT-4 achieves near-ceiling performance in many tasks with < I % mistake rate (shaded). ImpressionGPT (Ma et al., 2023) requires example selection and iterative example refinement.
Task
Test samples
Prompting GPT-4
GPT-4 performance
Mistake rate
Temporal sentence similarity
361
Zero-shot
New SOTA (f 10% acc.)
0.0%
Sentence similarity (RadNLI)
145
Zero-shot
New SOTA (13% acc.)
0.7%
Natural language inference (RadNLI)
480
Zero-shot + CoT
New SOTA ($ 10% Fi)
5.8%
Disease progression
1326
Zero-shot
On par with SOTA
0.4%
Disease classification
1955
10-shot
On par with SOTA On par with SOTA
0.3%
Entity extraction
100
200-shor*
Findings summarisation
1606 / 576
ImpressionGPT
On par with SOTA
n-shott: similarity-bee example selection with n examples; Mistake rate" (# genuine mistakes] / (# test samples]; f: [MIMIC// [Open-i]
label noise with only model mistake in RadNLI (see Appendix B.1), indicating GPT-4 is achieving near-ceiling performance these tasks.
pertise", and we show that GPT-4 reaches the best performance in both generic and radiology-specineg logical inference. CoT seems to help GPT models particularly to understand the radiology-specific cases. This is because CoT pushes the model to elaborate more on the radiology knowledge rele- vant to the input sentences, therefore giving suf- ficient context for a correct reasoning assessment (see Table B.4). Finally, we highlight that, even for GPT-4, there still a gap performance: the cases that specifically require radiology knowledge are more challenging than the other cases.
Table 2: Zero-shot GPT-4 and GPT-3.5 achieve new SOTA (accuracy) sentence similarity tasks. To test the consistency of GPT-4, we report mean and std. across five runs, and the self-consistency results ('sC').
Model
MS-CXR-T
RadNLI
text-davinci-802
90.3
91.0
ChatGPT
92.0
95.2
GPT-4
97.3 0,2
94.1 #04
GPT-4 (SC)
97.2
93.8
BioViL--T (Bannur al., 2023a)
87.8
90.5
Table 3: GPT performance (macro Fi) on RadNLI with domain analysis. GPT-4 + CoT achieves new SOTA. Mean, std., and self-consistency ('sc') results ported for GPT-4 + CoT across five runs.
4.2
Natural language inference (NLI)
All
need domain expertise? Yes
Task and model setup We assess GPT on the original RadNLI classification dataset (Miura et al., 2021). The model receives input 'premise' and 'hy- pothesis' sentences, and determines their relation: one of "entailment'. 'contradiction'. or 'neutral'.
text-davinci-003
55.9
42.8
60.7
+ CoT
64.9
54.1
68.4
ChaiGPT
45.4
31.5
+ CoT
70.5
65.6
We present GPT performance with zero-shot prompting and COT. We compare GPT models against the current SOTA, radiology-adapted T5 model (DoT5) which was trained on radiology text and general-doman NLI data (Liu et al., 2023a).
GPT-4
87.8
74.0
93.1
+ CoT
89.3 à 0.4
78.9 + 1.4
93.5 : 0.4
+ CoT (sc)
89.2
78.8
93.6
DoT5
(Liu et al., 2023a)
79.8
70.1
86.4
4.3 Disease classification
Results Table shows that GPT-4 with CoT achieves a new SOTA on RadNLI, outperforming DoT5 by 10% in macro Fj. Whereas NLI has tra- ditionally been a challenging task for earlier GPT models, GPT-4 displays a striking improvement, We also observe that CoT greatly helps in this task especially for GPT-3.5.
Task and model setup The evaluation dataset is extracted from Chest ImaGenome (Wu et al., 2021) gold attributes on the sentence level. To fairly com- pare with the SOTA CheXbert (Smit et al., 2020) model, we focus on pleural effusion, atelectasis, pneumonia, and pneumothorax, which are common
We further investigate how GPT-4 performs in cases that require different levels of radiology ex-
"Our categorisation is based on Liu et al. (2023a)'s domain expertise unnotations.

---

pathology names between CheXbert findings and Chest ImaGenome attributes. The output labels are "presence" and "absence" (binary classification) for each pathology. Detailed description the label mapping in Appendix D.
this setup and report mean, standard deviation and majority results in Table E.1.
Table 4: GPT performance on Chest ImaGenome dis-
ease classification.
Besides the CheXbert baseline, we also include the silver annotations from Chest ImaGenome, pro- duced by an ontology-based NLP tool with filtering rules (the Chest ImaGenome gold datasets in fact human-verifien verified silver annotations). To prompt GPT models, we started with zero-shot prompting, and then added 10 in-context examples with both random selection and similarity-base example se- lection. The example candidates are from the Chest ImaGenome silver data.
Model
Micro F,
Macro F1
text text-davinci-000
79.2
79.9
ChatGPT
89.7
85.0
GPT-4
93.0
91.5
GPT-4 (10)
96.6
96.6
GPT-4 (*10)
97.9
97.5
CheXbert
73.6
73.1
Silver
97.8
98.9
(n): number random shots; similarity-baset example selection; Silver: Chest ImaGenome silver annotations.
Results As shown in Table 4, there is progressive improvement from text-davinci-000 to Chat- GPT and then to GPT-4. All the GPT models' zero-shot results outperform CheXbert. We are able to improve GPT-4 zero-shot performance with 10-shot random in-context examples. We achieve further slight improvement with similarity based example selection, approaching the performance of silver annotations.
Table 5: Zero-shot GPT-4 performance after deferring from uncertain cases on Chest ImaGenome dataset: GPT-4 (defer). Its performance is significantly improved from zero-shot GPT-4 (with binary output).
GPT-4 (defer)
GPT-4
Macro Fi Micro F
97.4
93.0
98.6
91.5
Pleural effusion
98.5 [103]
[176]
Atelectasis Pncumonia
99.0 [154]
97.8 [233]
We manually analysed the errors from the GPT-4 (* 10) experiment and found that most (20 out of 30) are ambiguous, with the pathology cast as po- tentially present, rather than being easily labelled present or not. This is particularly the case for pneumonia whose presence is typically only sug- gested by findings in the chest X-ray (See examples of such uncertain cases in Table B.6). The rest of the model errors are cases of label noise and model mistakes. With <1% mistake rate, GPT-4 is approaching cciling performance in this task.
92.3 [16]
75.7 [111]
Pncumothorax
100.0 [17]
97.3 [18]
[n]: number of positive instances each pathology.
4.4 RadGraph entity extraction
Task and model setup This task requires model to extract observation and anatomy entities from radiology reports and determine their pres- ence (present, absent, or uncertain) following the RadGraph schema (Jain et al., 2021). )). To evaluate the extraction, we report micro Fi score counting truc positive when both the extracted entity text and the label are correct. RadGraph provides two datasets: MIMIC (Johnson al., 2019) with both train and test data, and CheXpert (Irvin et al., 2019) (with only test data).
Defer from uncertain cases Given the large amount of uncertain and ambiguous cases in the dataset, we experimented with asking the model to output 'uncertain' alongside the presence and ab- sence labels, and defer from these uncertain cases.? 7 Table shows that GPT-4 achieves very strong performance on those cases for which it is not uncertain. Note that pncumonia classification is dramatically improved and many positive cases of pneumonia are deferred. This aligns with our ob- servation from the dataset that pneumonia is often reported possibility rather than a certain pres- ence. We further test the robustness of GPT-4 in
We compare with the SOTA RadGraph Bench- mark model reported in Jain al. (2021), which is based on DyGlE++ (Wadden et al., 2019) with Pub- MedBERT initializations (Gu et al., 2021). Regard- ing prompting strategy, we started with a randomly selected 1-shot example, and then increased the number of random shots to 10. To push the perfor- mance, leveraged the maximum context window
"This can be seen as an alternative way to allow for uncer- tainties compared with using the class logits (Nori et al., 2023) as the class logits are not available via the API endpoints.
*We first experimented with zero-shot prompting, which re- sulted in many output formatting error. Adding one example resolved the formatting issue.

---

Table 6: GPT performance (micro Fi) on RadGraph entity extraction.
pathology due class imbalance. As MS-CXR-T labels were originally extracted from Chest Im- aGenome, we can also use Chest ImaGenome sil- ver annotations as our baseline. We report GPT performance with zero-shot prompting.
Model
MIMIC
CheXpert
text-davinc1-803 (1)
56.2
49.2
text-davinci-oet (10)
83.2
79.5
ChatGPT (1)
47.1
42.2
Results Table 7 shows that there is again a large jump of performance from GPT-4 compared with the earlier GPT-3.5 models. Zero-shot GPT-4 achieves >95% across all pathologies and is com- parable with Chest ImaGenome silver annotation. Our error analysis reveals that the majority of model errors are either label noise or ambiguous and the small mistake rate (0.4%) reflects the task is nearly solved.
ChatGPT (10)
70.6
67.5
GPT-4 (1)
36.6
25.3
GPT-4 (10)
88.3
84.7
GPT-4 (200)
91.5
88.4
GPT-4 (*200)
92.8
90.0
RadGraph Benchmark
94.3
89.5
(n): number of random shots; #: similarity-bass example selection
of GPT-4, incorporating 200-shot examples with both random selection and similarity- basedd selec- tion. Additionally, we found is helpful to perform GPT inference on individual sentences before com- bining them for report-level output. The in-context examples are also on the sentence level (200-shot sentences roughly corresponds to 40 reports) from the train set.
Table 7: GPT performance MS-CXR-T disease pro- gression (macro accuracy).
Model
PI. eff.
Cons.
PNA
PTX
Edema
fext--dvinci-8@@
92.1
91.8
90.0
96.1
93.6
CharGPT GPT-4
91.0
84.8
84.5
93.0
89.8
98.7
95.7
96.4
99.4
96.8
Silver
98.1
91.8
96.6
100.0
97.6
PNA: pncumonia; PTX: pncumothorax; Pl. eff.: pleural effusion; Cons.: consolidation; Silver: Chest ImaGenome silver annotations.
Results As shown in Table 6, examples are cru- cial for GPT to learn this task. We observe mas- sive jump in performance when increasing the num- ber of examples in the context. GPT-4 with 200 selected examples achieves overall on on-par perfor- mance with RadGraph benchmark: while GPT-4 (*200) underperforms the RadGraph model on the in-domain MIMIC test set, GPT-4 surpasses Rad- Graph Benchmark on the oft-of-domain CheXpert dataset. This indicates GPT-4 could be more ro- bust choice to generalise to out-of-domain datasets. Our error analysis reveals the errors are mostly due GPT-4 failing learn the schema specifics (Ap- pendix B.5). For example, GPT-4 may extract the whole compound word observation term, while the gold annotations break the word down ('mild' and 'moderate"..
4.6 Findings summarisation
Task and model setup The findings summarisa- tion task requires the model summarise the input findings into concise and clinically actionable im- pression section. We evaluate on the MIMIC (John- son et al., 2019) and Open-i "Demner-Fushman et al., 2016) datasets and follow Ma et al. (2023) to report results on the official MIMIC test set and a random split (2400:576 for train:test) for Open-i. For metrics, we report Rougel. (Lin, 2004) and the CheXbert score (Smit et al., 2020) (a radiology- specific factuality metric). We further conduct qualitative comparison study on GPT-4 outputs. For prompting strategies, we started with zero- shot and increased the number of random in- context examples to 10-shot. For GPT-4, we tried adding 100 examples with random selec- tion and similarity-baser selection. Examples are drawn from the respective train set for each dataset. We also replicated ImpressionGPT (Ma et al., 2023) with ChatGPT and GPT-4. ImpressionGPT performs dynamic example selection based on CheX labels and iteratively selects good and bad examples as in-context examples (The imple- mentation details are found in Appendix G).
as the
4.5 Disease progression classification
Task and model setup We evaluate on the tem- poral classification task from MS- -S-CCX-T (Bannur et al., 2023b), which provides progression labels for five pathologies (consolidation, edema, pleural effusion, pneumonia, and pneumothorax) across three progression classes ('improving', 'stable". and 'worsening'). this experiment, the input the radiology report and the outputs are disease pro- gression labels. We report macro accuracy for each

---

We compare with the previous supervised SOTA for this task (Hu et al., 2022) (which adopts graph encoder to model entity relations from findings), as well as with DoT5 (Liu et al., 2023a), a strong zero-shot summarisation baseline.
Table Percentage (%) with which the GPT-4 (Im- pressionGPT) generated impression is equivalent pre- ferred compared with an existing manually-written one according a radiologist.
Manual Imp.
GPT-4 preferred
Sample (n)
Equiv.
Ambig.
preferred
Open-i (80)
28.8
43.8
26.3
1.3
MIMIC (40)
25.0
10.0
57.5
7.5
Results While zero-shot GPT models all outper- form DoTS, we observe that providing examples is crucial for this task: there is consistent and substan- tial improvement when increasing the number of in-context examples for all GPT models. further boost can be achieved when we enable example selection for GPT-4 (#100). The more advanced ImpressionGPT brings the best performance out of GPT-4 and achieves performance comparable with the supervised SOTA.
Equiv. equivalent; Ambig. ambiguous; Manual Imp.: Existing manual impression
Discussion
5.1 Error analysis and GPT-4 consistency
Moving beyond quantitative scores, we manually reviewed GPT-4 errors all the tasks (A de- tailed analysis is shown in Appendix B). We further analysed the consistency of the errors for a selec- tion of tasks and reported the error breakdown in Table 10. We found the majority of the errors are either ambiguous label noise. As example of ambiguity, GPT-4 is extremely strict in identifying paraphrases and argues that one sentence contains minor additional information or slightly different emphasis. fact, for sentence similarity, disease progression, and disease classification tasks, the model mistakes are 1 1% of the test set (Table 1), We believe GPT-4 is achieving near-ceiling perfor- mance on these tasks. For entity extraction and findings summarisation, we found that GPT-4 out- put for many of the error cases is not necessarily wrong, but is offering an alternative to the schema or style in the dataset. This is verified by our quali- tative analysis from Appendix B.5 and Section 4.6),
Qualitative comparison To understand the dif- ferences between GPT-4 output and the manually- written impressions, chose random sample of reports and asked radiologist to compare existing manually-writen impressions with GPT-4 (Impres- sionGPT) output. Table demonstrates that for the majority of the cases (z= 70%), GPT-4 output is either preferred or comparable with the manually- written impression. Tables B.8 and B.9 show exam- ples where GPT-4 outputs are more faithful to the findings than the manually-w written impressions.
Table 8: GPT performance on findings summarisation. ImpressionGPT iteratively refines good and bad exam- ples as in-context examples.
It is important to note that GPT-4 in our current study still makes occasional mistakes. Some mis- takes are unstable actoss runs and can be corrected by self-consistency. Table 10 shows that GPT-4 is mostly consistent, and, for the few cases of incon- sistent output, self-consistency can correct most of the model mistakes that occur in minority runs.' 9 Another helpful strategy is to ask GPT-4 to defer when it is uncertain, as demonstrated by the disease classification experiments (Appendix B.3).
MIMIC
Open-i
Model
R.
CR.
R.
CB.
text-davinci-003
22.9
41.8
14.5
41.9
text-davinci-803 (10)
29.1
43.0
40.5
42.0
CharGPT
20.0
40.5
14.8
39.6
CharGPT (10)
31.0
42.5
40.6
41.0
GPT-4
22.5
39.2
18.0
39.3
GPT-4 (10)
28.5
44.2
42.5
44.9
GPT-4(100)
30.9
44.7
44.2
45.0
GPT-4(>100)
38.4
47.4
59.8
47.3
ChatGPT (ImpressionGPPT
44.7
63.9
58.8
44.8
The remaining model mistakes are mostly cases where nuanced domain knowledge is required. For example, GPT-4 mistakenly equates 'lungs are hy- perinflated but clear' with 'lungs are well-expanded
GPT-4 (ImpressionGPT)
46.0
64.9
64.6
46.5
Hu et al. (2022)
47.1
54.5
64.5
DoT5 (Liu al., 2023a)
11.7
25.8
(n): number of random shots; 9 bassd example selection; R.: RougeL; CB.: CheXbert.
"oote that the overall scores from self-consistency experi- ments (Tables 2, 3 and F, do not refiect this quantitatively due to the noise from the many ambiguous cases.

---

Table 10: Self-consisenncy error analysis for GPT-4, Errors are categorised by whether they are consistent, occurring in minority runs (SC correct) or occurring majority runs (SC incorrect). We further categorise errors into model mistakes and others (ambiguous or label noise). We observe the majority of the errors are consistent and many errors are not model mistakes. Within the cases inconsistent output, self-consistency can correct most of the model mistakes. GPT-4 zero-shot performance is reported in this table (disease classification results are after we defer from the uncertain cases). Error breakdown for other single run experiments are in Table F.I.
Consistent
correct
incorrect
Task
Mistake
Other
Corrected mistake
Other
Mistake
Other
Total
Temporal sentence similarity
0%
72%
10%
0%
0%
18%
Sentence similarity (RadNLI)
11%
78%
0%
11%
9
RadNIII
55%
31%
0%
2%
6%
49
Disease classification
22%
67%
11%
0%
0%
0%
9
All
38%
46%
0%
1%
8%
and clear' in MS-CXR-T. The former indicates an abnormality while the latter is describing nor- mal lungs. We should point out that this mistake does not mean GPT-4 is fundamentally lacking the knowledge. In fact, when asked explicitly about it in isolation (e.g., difference between 'hpperinllatdd and 'well-expanded lungs'), or when we reduce the complexity of the two sentences to 'lungs hy- perinflated" and 'lungs are well-expanded', GPT-4 is able differentiate the two terms (Table B.3).
over task-specific baselines or nearly ceiling per- formance. Disease classification does not fall into this category due to the ambiguity in how to as- sign labels for the uncertain cases. Here, GPT-4 requires 10 examples to achieve comparable near- ceiling performance with previous SOTA. We show that zero-shot GPT-4 can also achieve near-ceiling performance if we defer from uncertain cases (Ta- ble 5) in this task. Another key point note is that GPT-4 better choice than the previous SOTA Chest ImaGenome silver annotations for disease and disease progression classification, as the silver annotations are from rule-based systems that are not available to be re-used for other datasets.
We interpret as nuanced radiology knowledge not being guaranteed to always surface for all contexts with all various prompts. While future prompting strategies might help with these cases, we must acknowledge that potential model mistakes cannot be fully ruled out. Therefore, human in the loop is still required for safety-critical applications.
Different from the above-mentioned tasks, it is not straightforward articulate requirements in the instruction for entity extraction and findings summarisation. For entity extraction, the exact definition of observation and anatomy is schema- specific and in many cases can only be inferred from training examples. For findings summarisa- tion, while there are general mule-of-thumd princi- ples for writing good impression, it is not possible to write down detailed instructions regarding the exact phrasing and style of the impressions in a particular dataset. We call these 'learn-b--exapple tasks. Task-specific supervised models perform competitively on such tasks, as they can explicitly learn an in-domain distribution from all training examples. We found significant improvement of GPT models with increased number of examples compared with zero-shot, and GPT-4 with example selection can match supervised baselines. Future research can explore ways to combine GPT-4 and supervised models (c.g. treating the latter as plug- ins Shen et al. 2023; Xu et al. 2023).
5.2 GPT-4 vs SOTA radiology models
Throughout the experiments, we first observed a significant jump of performance of GPT-4 com- pared with the prior GPT-3.5 (text-davinci-00) and ChatGPT), confirming the findings from previ- ous studies (Nori et al., 2023). We then summarised the overall GPT-4 performance compared with ra- diology SOTA in Table 1. The key finding is that GPT-4 outperforms or is on par with SOTA radiol- ogy models the broad range of tasks considered. We further notice that different tasks require dif- ferent prompting efforts and strategies. For tasks such as sentence similarity, RadNLII and disease progression, the task requirements can be clearly defined in the instruction. (For example, there is clear logical definition for 'entailment', 'neu- tral', and 'contradiction' in NLI). For such 'learn- by-in by-instruction' tasks, a simple zero-shot prompt- ing strategy for GPT-4 can yield significant gains

---

6 Conclusion
Limitations
This study evaluates GPT-4 on a diverse range of common radiology text-based tasks. We found GPT-4 either outperforms or is on par with task- specific radiology models. GPT-4 requires the least prompting effort for the deammby-iinstruction tasks where requirements can be clearly defined in the instruction. Our extensive error analysis shows that although it occasionally fails to surface domain knowledge, GPT-4 has substantial capability in the processing and analysis radiology text, achieving near-ceiling performance in many tasks.
In this paper, we focused on GPT-4 as it is the most capable and the best-performing LLM now across many domains and we would like to establish what best we can do with LLM in radiology. We leave it for future research to test and compare GPT-4 per- formance with other LLMss. In addition, GPT-4 with the current prompting strategies the study already achieves near-ceiling performance in many tasks, we leave an exhaustive experimentation all existing prompting strategies for future research. For example, we have not explored the more re- cently proposed advanced prompting techniques including tree of thought (Yao et al., 2023) and self-critique (Shinn et al., 2023) and encourage future research to apply techniques to help improve the reliability of GPT-4. Also, due resource con- straint, we did not perform self-consistency exhaus- tively for all tasks and for GPT models. That being said, we believe the findings from this paper should already represent what an average user can get out of using GPT models on these tasks. The insights and learnings will be useful for design- ing future prompting strategies for radiology tasks, where particular tasks or error cases will require more prompting efforts.
Our error analysis shows that many the ex- isting radiology tasks contain intrinsic ambiguities and label noise and we call for more quality con- trol when creating evaluation benchmarks in the future. Finally, our qualitative evaluation of the findings summarisation task is limited to single radiologist. This is a subjective assessment that will be influenced by radiologist's own style and preference. The ideal scenario would to ask ra- diologists who participated in the creation of the MIMIC or Open-i dataset to perform the assess- ment so that they have the same styling preference as the dataset. We are also planning to conduct more nuanced qualitative evaluation addressing dif- ferent aspects of the summary in the future.
Ethical Considerations
we would like to assure the readers that the experiments in this study were conducted using Azure Open AI services which have all the compliance requirements
any other Azure Services. Azure Open Al HIPAA compliant and preserves data privacy and compliance of the medical data (e.g.. The data are not available to OpenAI). More details can found in https:

---

/ //wzure. http:///...roooo...cco//////eeee
Rajesh Bhayana, Satheesh Krishna, and Robert R Bleakney. 2023b. Performance of ChatGPT on a radi- ology board-style examination: Insights into current strengths and limitations. Radiology, page 230582.
and
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar. Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with GPT-4 arXiv preprint wwXiv:2333122722
snnuuu/aaasseerrs//usssti//////s All the public datasets used in this paper were also reviewed by MSR (Mi- crosoft Research) IRB (OHRP parent organization number IORG #0008066, IRB #IRB00009672)
hipaa-compliannn
Kunming Cheng, Qiang Guo, Yonghin He, Yanqiu Lu, Shuqin Gu, and Haiyang Wu. 2023. Exploring the potential of GPT-4 in biomedical engineering: the dawn new era. Annals of Biomedicul Engineer- ing. pages 1-9.
under reference numbers RCT4053 and ERP10284. IRB Decision: approved - Not Human Subjects Re- search (per 45846.102(c) 45$46.102((e)55)
Acknowledgments
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. PaLM: Scaling language modeling with pathways. arXiv preprint wrXiv:2204.02311.
We would like to thank the anonymous reviewers and area chairs for their helpful suggestions. We would also like thank Hannah Richardson, Har- sha Nori, Maximilian Ilse and Melissa Bristow for their valuable feedback.
Dina Demmer-Fushmant Mare D Kohli, Marc B Rosen- man, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDon- ald. 2016. Preparing a collection of radiology ex- aminations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304-310,
References
Lisa C Adams, Daniel Truhn, Felix Busch, Avan Kader, Stefan M Nichues, Marcus R Makowski, and Keno K Bressem. 2023. Leveraging GPT-4 for post hoc trans- formation of free-text radiology reports into struc- tured reporting: a multilingual feasibility study. Ra- diology. page 230725.
Jessica López Espejel, El Hassane Ettifouri, Mahaman Sanoussi Yahaya Alassan, EI Mehdi Chouham, and Walid Dahhane. 2023. GPT-3.5 vs GPT-4: Evaluat- ing ChatGPT's reasoning performance in shot learning. arXiv preprint arXiv:2305.1247
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. PaLM 2 technical report. arXiv preprint 2335 23003
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific lan- guage model pretraining for biomedical natural lan- guage processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1-23.
Shruthi Bannur, Stephanie Hyland. Qianchu Liu, Fer- nando Pérez-García, Maximilian llse, Daniel C. Cas- tro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme. Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori, Javier Avarez-Valle, and Ozan Oktay. 2023a. Learn- ing to exploit temporal structure for biomedical vision-language processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 15016-15027.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Mcasuring massive multitask language under- standing. In International Conference on Learning Representations.
Jinpeng Hu, Zhuo Li, Zhihong Chen, Zhuguo Li, Xiang Wan, and Tsung-Hui Chang. 2022. Graph enhanced contrastive learning for radiology findings summa- rization. In Annual Meeting of the Association for Computational Linguistics.
Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fer- nando Pérez-Garcfa Max llse. Daniel Coelho de Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anton Schwaighofer, Maria Teodora Wetscherek, Hannah Richardson, Tristan Naumann, Javier Alvarez Valle, and Ozan Oktay. 2023b. MS- CXR-T: Learning to exploit temporal structure for biomedical vision-language processing. PhysioNet.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Hleus, Chris Chute. Henrik Mark- lund, Behzad Haghgoo, Robyn Ball, Katie Shpan- skaya, et al. 2019. Chexpert: large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the Thirty-Thir AAAI Conference on Artificial Intelligence and Thirty-Fire First Innovative
Rajesh Bhayana, Robert R Bleakney, and Satheesh Kr- ishna. 2023a. GPT-4 radiology: Improvements in advanced reasoning. Radiology, page 230987.

---

Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, pages 590-597.
Chong Ma, Zihao Wu, Jiaqi Wang. Shaochen Xu. Yaonai Wei, Zhengliang Liu, Lei Guo, Xiaoyan Cai. Shu Zhang. Tuo Zhang, et al. 2023. ImpressionGPT: An iterative optimizing framework for radiology re- port summarization with ChatGPT. arXiv preprint arXiv:2304.0844 0844088448.
Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven Truong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng. Curtis Langlotz. et al. 2021. Radgraph: Extracting clinical entities and relations from radiology reports. In Thirty-fithh Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round I).
Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Lan- glotz, and Dan Jurafsky. 2021. Improving factual completeness and consistency of image-to-text textext radi- ology report generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5288-5304.
Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz. Nathaniel R Greenbaum. Matthew P Lungren, Chih- ying Deng. Roger G Mark, and Steven Homg. 2019. MIMIC-CRR, a de-identified publicly avail- able database of chest radiographs with free-text re- ports. Scientific data, 6(1):1-8
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023, Capabilities of GPT-4 medical challenge problems. arXiv preprint arXiv:2303. 13375.
Peter Lec, Sebastien Bubeck, and Joseph Petro. 2023, Benefits, limits, and risks of GPT-4 as an Al chatbot for medicine. New England Journal of Medicine, 388(13):1233-122
OpenAL. 2023. Gpt-4 technical report.
Mercy Ranjit, Gopinath Ganapathy, Ranjit Manuel, and Tanuja Ganu. 2023. Retrieval augmented chest X-ray report generation using OpenAl GPT models.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. 2023, Holistic evaluation language mod- els. Transactions on Machine Learning Research (TMIR).
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- GPT: Solving Al tasks with ChatGPT and its friends in hugging face. arXiv preprint arXiv:2303. 17580.
Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dy- namic memory and self-reflection. arXiv preprint arXiv.2303.11366
Chin-Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81. Barcelona, Spain. Association for Computational Linguistics.
Karan Singhal. Shekoofeh Azizi, Tao Tu, S Sara Mah- davi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022. Large language models encode clinical knowledge. arXiv preprint 2rXiv:2212.13138.
Fangyu Liu, Qianchu Liu, Shruthi Bannur, Fernando Pérez-Gareía, Naoto Usuyama, Sheng Zhang, Tris- tan Naumann, Aditya Nori, Hoifung Poon, Javier Aivare--Vall Ozan Oktay, and Stephanie L. Hy- land. 2023a. Compositional zero-shot domain trans- fer with text-to-text models.
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023. Towards expert-level medical question swering with large language models. arXiv preprint arXiv:2305.096 096117
Hammeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023b. Evaluating the logi- cal reasoning ability of ChatGPT and GPT-4, arXiv preprint 2334::2304.03433
Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pa- reek, Andrew Ng, and Matthew Lungren. 2020. Com- bining automatic labelers and expert annotations for accurate radiology report labeling using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1500-1519, Online. Association for Computa- tional Linguisties.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B
Dolan, Lawrence Carin, and Weizhu Chen. 2022.
What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (Dee- LIO 2022): The 3rd Workshop Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-114.
Qing Lyu, Josh Tan, Michael E Zapadka, Janardhana Ponnatapura, Chuang Niu, Kyle J Myers, Ge Wang, and Christopher T Whitlow. 2023. Translating radi- ology reports into plain language using ChatGPT and GPT-4 with prompt learning: results, limita- tions, and potential. Visual Computing for Industry, Biomedicine, and Ari, 6(1):9.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models, arXiv preprint arXiv:2302.132: 13977.

---

Dave Van Veen, Cara Van Uden, Maayane Attias, Anuj Pareek, Christian Bluethgen, Malgorzata Po- lacin, Wah Chiu, Jean-Benoit Delbrouck, Juan 7am- brano Chaves, Curtis Langlotz, Akshay Chaudhari, and John Pauly. 2023. RadAdapt: Radiology re- port summarization via lightweight domain adapta- tion of large language models. In The 22nd Work- shop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 449 460, Toronto, Canada. Association for Computational Linguistics.
Li Yunxiang. Li Zihan, Zhang Kai, Dan Ruilong, and Zhang You. 2023. ChatDoctor: A medical chat model fine-tuned on aLaMA model using medical do- main knowledge. arXiv preprint arXiv:2303. 14070.
Liammin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging Iim-as aa--as-a-jja as-a-uugee with mt-bench and chatbot arena. arXiv preprint arXiv:2306.0563:
David Wadden, Ulme Wennberg. Luan, and Han- naneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing JEMNLP-IJCNEL pages 5784- 5789, Hong Kong, China. Association for Computa- tional Linguisties.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representa- tions.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, al. 2022. Chain-of-tie thoutht prompting elicits rea- soning in large language models. In Advances in Neural Information Processing Systems.
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang. and Weidi Xie. 2023a. PMC-LLaMA: Further fine- tuning LLaMA on medical papers. arXiv preprint arXiv:2304. 14454.
Joy Wu, Nkechinyere Agu, Ismini Lourentzou, Ar- jun Sharma, Joseph Paguio. Jasper Seth Yao, Ed- ward Christopher Dee, William Mitchell, Satyananda Kashyap, Andrea Giovannini, Leo Anthony Celi, Tanveer Syeda-Man Mahmood, and Mehdi Moradi. 2021, Chest imagenome dataset (version 1.0.0). PhysioNet.
Zihao Wu, Lu Zhang, Chao Cao, Xiaowei Yu, Haixing Dai. Chong Ma. Zhengliang Liu, Lin Zhao, Gang Li, Wei Liu, al. 2023b. Exploring the trade-offs: Unified large language models vs local fine-tuned models for highly-specift, radiology NLI task. arXiv prepriat arXiv:2304.09133
Canwen Xu, Yichong Xu, Shuohang Wang. Yang Liu, Chenguang Zhu, and Julian McAuley. 2023. Small models valuable plug-ins for large language mod- arXiv preprint w.Xiv:2305.0884.
Shunyu Yao, Dian Yu. Jeffrey Zhao, Izhak Shafran, Thomas Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models, arXiy preprint ar0iv:2305 10601.

---

normal. We should point out that this behaviour does mean GPT-4 is fundamentally lacking the knowledge. In fact, if we reduce the complexity of the two sentences to 'lungs are hyperinflated" and ChatGPT (gpt-35-turbo, ver. 0301). We use Azure Cognitive Services API with the ver- sion "2025-03-15-preview.. Model names corre- spond to https: openai. com/docs/ openai.com//ooss models/overview. Regarding to GPT-4 pricing, at the time of conducting this study, it costs 0.06 USD per 1,000 tokens for prompt and 0.12 USD per 1,000 tokens. The actual cost depends the task. For RadNLI with 361 samples, the cost is around USD. The total cost the evaluation conducted in this puper is around 5000 USD. We acknowledge that the cost of GPT-4 is high at the moment. As such, the findings from our paper can save the costs from future researchers who want to investigate similar research questions.
A GPT model and API details
We evaluated GPT-4 (gpt-4-32k, ver. 0314), and earlier GPT-3.5 models: text-davinci-003
and 'lungs are well-expanded', GPT-4 is able to differentiate between the two terms. Therefore, we interpret the error the domain knowledge has not being surfaced in complex context.
Self-consistency does not improve the overall results and most errors are consistent across runs. For MS-CXR out of the 11 error cases are con- sistent. For RadNLI sentence similarity, 8 out of 9 error cases are consistent across runs. For the in- consistent cases, apart from one genuine error from MS-CXR-T which is corrected by self-consistency, all inconsistent cases are ambignous.
B
GPT-4 detailed error analysis
B.1 Sentence similarity
We manually reviewed all GPT-4 errors and found that the errors are mostly ambiguous or label noise and these two tasks can be seen nearly solved by GPT-4. For MS-CXR-T, the majority of the errors required identifying that 'improvement" was syn- onymous with 'decrease' for cases such as edema and lung opacity (all were ground truth paraphrase pairs) (See Table B. Among these cases, GPT-4 does recognise 'improvement" and 'decrease' de- scribe positive change, but argues that 'improve- ment' and 'decrease' describe different aspects of the change. Confirmed with radiologist, GPT-4 reasoming here pedantic but understandable.
In one error however, GPT-4 reasons that 'im- proved' and 'decrease" indicate opposite direc- tion of change, which is a clear error showing the radiology-specific knowledge does not surface from GPT-4 in this case. Fortunately, this error case only the minority across all runs (2 out of cases) and is corrected by Sc.
The finding is similar for the RadNLI sentence similarity task where the majority of the errors are ambiguous cases where GPT-4 flagged that one sen- tence contained slightly more information than the other, hence were not strict paraphrases. There is one genuine error where GPT-4 mistakenly equates 'lungs are hyperinflated but clear' with 'lungs are well-expanded and clear' (See Table B.2). The former indicates an abnormality while the latter is

---

Sentence
interval improvement in mild pulmonary edema
Sentence 2
interval decrease mild pulmonary edema
GPT-4 reasoning
The first sentence describes an improvement in the mild pulmonary edema, which means the condition has gotten better. The second sentence describes a decrease in the mild pulmonary edema, which means the amount or severity of the edema has reduced. Although both sentences indicate a positive change, they describe different aspects of the change in the pulmonary edema.
Error category
ambiguous
Sentence
opacities in the left base have minimally improved.
Sentence 2
opacities in the left base have minimally decreased.
GPT-4 reasoning
The first sentence states that the opacities have improved, which means they have become less severe less noticeable. The second sentence states that the opacities have decreased, which means they have reduced in size number. Although both sentences describe positive change, they convey different aspects of the change opacities.
Error category
ambiguous
Sentence
left basal consolidation has slightly improved.
Sentence 2
left basal consolidation has slightly decreased.
GPT-4 reasoning
The first sentence states that the left basal consolidation has slightly improved, meaning the condition has gotten better. The second sentence states that the left basal consolidation has slightly decreased, which implies that the size or extent of the consolidation has reduced. Although both sentences indicate positive change, they describe different aspects of the change.
Error category
ambiguous
Sentence
mild improvement in right upper lung opacity, possibly a hematoma, status post right upper lobe resection.
Sentence 2
mild decrease in right upper lung opacity, possibly hematoma, status post right upper lobe resection.
GPT-4 reasoning
The two sentences describe different changes in the right upper lung opacity. Sentence mentions "mild improvement, while Sentence mentions "mild decrease." " These changes are opposite nature, with improvement suggesting a positive change and decrease suggesting reduction in opacity.
Error category
genuine mistake
Table B.I: Examples of GPT-4 enrors the MS-CXR-T temporal sentence similarity dataset. Note that all pairs are labelled (ground truth) as paraphrases, but were classified as different by GPT-4, In the first three examples, GPT-4 gives sound reasoning. The last example shows erroneous reasoning where the model thinks improvement and decrease indicate opposite directions of change, but fortunately the last example comes from 2 out of 5 runs and can corrected by celf-connistency.

---

Sentence Sentence 2
Osseous structures are without an acute abnormality.
The osseous structures are grossly unremarkable, although evaluation is limited secondary to body habitus.
GPT-4 output and reasoning
different. While both sentences describe the osseous structures as having any significant issues, Sentencel states that there are no acute abnormalities, while Sentence2 mentions that the evaluation limited due to the patient's body habitus, implying that there might be some limitations in the assessment.
True label
paraphrase
Error category
ambiguous
Sentence
Lungs are hyperinflated but clear.
Sentence 2
Lungs are well-expanded and clear without focal consolidation concerning for pneumonia.
GPT-4 output and reasoning
paraphrase. Both sentences describe lungs that are clear and without any signs pneumonia. The first sentence uses the term "hyperinflated" while the second uses eelldexpp but both essentially convey the same meaning of lungs being expanded and clear.
True label
different
Error category
genuine mistake
Table B.2: Examples of GPT-4 errors the RadNLI-deriven sentence similarity dataset. In the first example, the label is ambiguous and GPT-4 highlights the additional information in the second sentence. In the second example, GPT-4 erroneously equates hyperinflated pathological state) with 'well-expanded lungs.
Sentence 1
Lungs are hyperinflated but clear.
Sentence 2
Lungs are well-expanded and clear without focal consolidation concerning for pneumonia.
GPT-4 output and reasoning
paraphrase. Both sentences describe lungs that are clear and without any signs of pneumonia. The first sentence uses the term "hyperinflated" while the second uses "well-expanded," " but both essentially convey the same meaning of lungs being expanded and clear.
True label
different
Sentence
Lungs are hyperinflated.
Sentence
Lungs are well-expanded.
GPT-4 output and reasoning
different. Although both sentences describe the state of the lungs, they convey different meanings. "Hyperinflated" implies that the lungs are excessively inflated, which could indicate a medical issue. On the other hand, "well- expanded" suggests that the lungs are in healthy state, functioning properly with adequate inflation.
True label
different
Table B.3: An example show that domain knowledge surfaced from GPT-4 when reduce the complexity of the context as in the second example.

---

B.2 RadNLI
with 'uncertain' as well as GPT-4 reasons that 'lin- ear opacities' can possibly reflect atelectasis. This indicates that the model is able to defer in the case of an obvious erroneous output.
We observe higher error rate in NLI compared to sentence similarity. This can be explained by the subtle nature of the task that requires precise logi- cal reasoning in order to correctly recognise textual entailment from neutral examples and contradic- tions. Out of the 45 model "errors" from GPT-4 + CoT (SC), we identified 13 logical errors where the model's reasoning indicated lack of under- standing of entailment as it appeared to look for paraphrases 'the hypothesis does not provide all information in the premise' tionality of entailment 'the premise is part of the hypothesis'. Additionally, we found 18 label-noise examples for which GPT-4 was arguably right. An extra set of 14 eπrors were domain specific such 'unchanged'' does not imply 'normal' and 'en- larged' suggests 'not top normal' in radiology context (See Table B.5). While most of the errors are consistent across run, there are 7 cases of incon- sistent output across runs. For cases, majority run does not align with the gold label but these cases are mostly label noise. For cases, majority cor- rects inconsistent output and all 3 cases are genuine errors from minority runs.
We test self-consistency for the GPT-4 (defer) set up across five runs, the 8 identified errors are consistent. There is an extra error from 1 out of
5 runs where the model makes a genuine mistake: GPT-4 thinks pleural effusion is present based on "there is no pneumothorax, large effusion, or con- gestion." Fortunately, this obvious error can be corrected by majority voting from self-consistency.
confused the direc-
B.4 Disease Progression
The error analysis reveals that GPT-4 is achieving ceiling performance on this task. We manually re- viewed total of 34 errors from all the pathologies and found most of the model mismatches are either due to label noise (7) or the case being ambiguous (21). Many of the ambiguous cases appeared where the pathologies mentioned are uncertain in the first place and therefore harder to assess for change. The genuine errors are either because the model fails to recognise pathologies (e.g. 'hemorrhage" will indicate consolidation) or failed to associate the change word with the pathology due to unfamiliar- ity with radiology-specific phrasing (e.g.. not able to recognise edema is worsening from 'inercased opacities consistent with mild pulmonary edema'). Table B.7 shows the example errors from GPT-4 on disease progression.
We also show in Table B.4 the examples where CoT improves GPT-4 performance on RadNI.I.
B.3 Disease Classification
We analyse the errors for the GPT-4 (defer) setup after defer from uncertain cases. We collect a total of 8 error cases for all pathologies from a single run: there are two cases of label noise, while the other errors are largely ambiguous and uncertain cases where GPT-4 should output 'uncer- tain' but outputs "absence" instead. For example, in "alternatively, could be due to infection, a toxic allergic drug reaction hemorrhage.' GPT-4 should output uncertain for pneumonia rather than absence. There are two genuine eтrors where GPT- 4 fails to identify 'pleural effusion' from 'fluid did not accumulate acutely', and mistakenly assigns the absence label for pleural effusion in the sen- tence 'pleural effusion is nearly resolved'. When we compare the outupt before and after we add the 'uncertain' option to the model, observe an overall improvement in Fi, many of the ambigu- ous errors from binary setup are now assigned with 'uncertain' label. In addition, some of the genuine errors from binary setup (e.g., not understanding 'linear opacities' indicates atelectasis) are assigned
B.5 RadGraph
We examine the GPT-4 (#200) output on MIMIC and collect in total 137 error cases. We categorise these errors into 31 cases where gold entities are not identified from predictions, and 73 cases where predicted entities are not found in gold, and 33 cases where the entity tokens are extracted but the labels are not correct. For the missing gold enti- ties, we found that most of the cases (22 out of 33) are due to GPT breaking up the word differ- ently than the gold annotations: e.g., While GPT-4 extracts the whole phrase "mild-to-moderate" as one observation entity, the gold annotation extracts 'mild' and 'moderate' separately. For the 73 over- predicted entities, we found that the majority of the mismatch is due to inconsistency in the annota- tion. For example, in the sentence 'an esophageal drainage tube passes into the stomach and out of view" GPT extracts 'out of view" as the observa- tion but the gold annotation does not. However, in

---

Premise
Mediastinal and hilar contours are grossly unremarkable.
Hypothesis
Bony structures are unremarkable.
GPT-4 + CoT
The premise talks about the mediastinal and hilar contours being unremarkable, which refers to the structures in the chest area. The hypothesis talks about bony structures being unremarkable, which refers to the bones. These two statements are not directly related to each other, as one talks about chest structures and the other talks about bones. Therefore, there is no clear evidence to support contradict the hypothesis based on the premise. Hence, the relation between the premise and hypothesis is neutral.
REASON:
ANSWER: neutral
GPT-4
ANSWER: entailment
True label
neutral
Premise
The lungs are clear.
Hypothesis
The lungs are well expanded.
GPT-4 + CoT
Both the premise and hypothesis related to the lungs, but they express different information. The premise states that the lungs are clear, which means that there are no signs of disease or infection. On the other hand, the hypothesis states that the lungs are well expanded, which means that they are functioning properly and have good air flow. These two statements are not contradictory, but they do not necessarily entail each other either. It is possible for the lungs to be clear but not well expanded, or for the lungs to be well expanded but not necessarily clear. Therefore, the relation between the premise and hypothesis is neutral.
ANSWER: neutral
GPT-4
ANSWER: entailment
True label
neutral
Table B.4; Examples where CoT improves GPT-4 performance on RadNLI

---

Premise
The heart is mildly enlarged, unchanged.
Hypothesis
The heart top normal in size, which exaggerated by the low lung volumes. REASON:
GPT-4 output and reasoning
The premise states that the heart mildly enlarged and unchanged, while the hypothesis suggests that the heart top normal in size and appears larger due to low lung volumes. These two statements provide different infor- mation about the heart size and do not support or contradict each other directly. ANSWER: neutral
True label
contradiction
Premise
The pulmonary vasculature is normal.
Hypothesis
There no pulmonary edema.
GPT-4 output and reasoning
The premise states that the pulmonary vasculature normal, which implies that there are no abnormalities in the blood vessels of the lungs. The hypothesis states that there is no pulmonary edema, which is a condition where fluid accumulates in the lungs due to issues with the blood vessels or heart. Since a normal pulmonary vasculature would not typically result in pulmonary edema, the hypothesis can be inferred from the premise. ANSWER: entailment
REASON:
Error category
genuine mistake
True label
neutral
Error category
genuine mistake
Premise
Opacities involving the right middle lobe and left lower lobe are again demon- strated and may represent atelectasis or infection in the appropriate setting.
Hypothesis
Minimal left basilar opacity suggests minor atelectasis.
GPT-4 output and
The premise mentions opacities in the right middle lobe and left lower lobe, which could represent atelectasis or infection. The hypothesis, the other hand, talks about minimal left basilar opacity suggesting minor atelectasis. While both sentences discuss opacities and atelectasis, they refer different parts of the lungs. Therefore, the relation between the premise and hypothesis is undetermined.
REASON:
reasoning
ANSWER: neutral
True label
entailment
Error category
genuine mistake
Table B.5: GPT-4 emrors on RadNLI. In the first example, GPT-3 misses that 'enlarged' entails 'not vop-normal' for hear size. second example, GPT-4 fails to understand that normal pulmonary vasculature does not necessarily always indicates normal pulmonary edema. In the last example. GPT-4 fails to understand left basilar opacity and opacities in the left lower lobe refer to the same location.

---

Text
scattered vague opacities bilaterally are consistent with clinical history of septic emboli, unknown whether active scar.
Gold
(pneumonia, absent)
GPT-4 answer and reasoning
(pneumonia, present). The text mentions "scattered vague opacities bilaterally" which are consistent with the clinical history of septic emboli. This suggests the presence of pneumonia.
Error category
ambiguous
Text
no large pleural effusions are seen, although trace effusions will be difficult to exclude.
Gold
(pleural effusion, absent)
GPT-4 answer and reasoning
(pleural effusion, present). The text mentions that large plcural efffusons are seen, but trace effusions difficult to exclude, which implies that small pleural effusions might be present.
Error category
ambiguous
Table B.6: Examples of ambiguous GPT-4 errors on the disease classification binary classification task from Chest ImaGenome Gold dataset.
Text
Small focus of hemorrhage or atelectasis at the base of the postoperative
right lung has cleared...
Gold
(consolidation, improving)
GPT-4 answer and reasoning
(consolidation, stable). Consolidation is not mentioned the input text. The findings discussed include persistent moderate right pneumothorax, small bi- lateral pleural effusions, small focus of hemorrhage or atelectasis, and mild generalized pulmonary edema.
Error category
genuine mistake
Text
Increased interstitial opacity with mediastinal vascular fullness and Kerley B lines is consistent with mild to moderate pulmonary edema
Gold
(edema, worsening)
GPT-4 answer and reasoning
(edema, stable). The text does not provide any information about the change in the edema's condition compared to previous state. Therefore, we cannot determine if it is improving or worsening, and we assume it is stable.
Error category
genuine mistake
Table B.7: Examples of GPT-4 genuine mistakes on the disease progression classification task. the first example, GPT-4 fails to identify that hemorrhage is type of consolidation. In the second example, GPT-4 fails to understand the radiology-specific phrasing that change "consistent with edema also apply to the edema.

---

another gold example in the training data: 'right internal jugular line ends in the right atrium and an upper enteric drain passes into a non-distended stomach and out view :, "out view' was anno- tated as the observation. This reflects the intrinsic ambiguity of annotating what counts as observa- tion and anatomies in the schema. In another 25 cases of over- pver-predicted entities, GPT over-extracts words such 'probable', 'definite' observa- tions. Finally, we found 33 cases where GPT-4 extracts the correct entities but assigns the wrong label, and they are mostly due to the confusion of uncertain labels. Many of these are ambiguous, for example, for the sentence 'No definite focal con- solidation identified', the gold annotation assigns an "observationn-uccrrain label for 'consolidation" but GPT assigns abservation-absem!
This error analysis reveals the intrinsic challenge of learning schema-specific annotations in this task, as there may not be a single clear-cut standard in some cases. GPT output often not wrong but is offering an alternative solution.
B.6 Findings summarisation
Table B.9 and Table B.8 show examples where GPT-4 (ImpressionGPTT outputs are favoured than existing manually-wirt written impressions.

---

Findings
There mild cardiomegaly. Mediastinal contours appear within normal limits. There are small bilateral pleural effusions, left greater than right with left basilar opacities. No pneumothorax. Mild degenerative changes of the thoracic spine.
Existing impression
Cardiomegaly with small bilateral plcural effusions and left basilar atclectasis.
GPT-4 impression
Mild cardiomegaly and small bilateral pleural effusions with left basilar opaci- ties.
Findings
Cardiomegaly is noted. No pleural effusions. No pneumothorax. There is perihilar prominence and interstitial opacification.
Existing impression
Cardiomegaly and mild interstitial pulmonary edema.
GPT-4 impression
Stable cardiomegaly with prominent perihilar opacities which may represent scarring or edema.
Table B.8: Examples where GPT-4 (ImpressionGPP) impressions are favoured over the existing manually-writter impressions on Open-i dataset. In both examples, GPT-4 outputs have more faithful and more complete details the findings.

---

Findings
Single AP upright portable view of the chest was obtained. The patient's overlying chin obscures the medial bilateral upper lobes. The cardiac silhouette remains enlarged. Prominence of the pulmonary arteries is partially imaged and again seen. Evidence of diaphragmatic/pluraal plaques is seen bilaterally suggesting prior asbestos exposure.
Existing impression
The patient's chin overlies the bilateral medial upper lobes, obscuring the view. Given this, the cardiac silhouette is persistently enlarged. There is again prominence of the pulmonary arteries. Pulmonary vascular congestion appears improved.
GPT-4 impression
Cardiomegaly and prominence of pulmonary arteries. Bilateral diaphrag- matic/pleural plaques suggestive of prior asbestos exposure.
Findings
AP upright and lateral views of the chest provided. Vascular stent is seen in the region of the right brachiocephalic vein. The heart is moderately enlarged. There is mild interstitial pulmonary edema. Previously noted ET and NG tubes have been removed. No large pleural effusion. Mediastinal contour is stable. Bony structures are sclerotic which could reflect renal osteodystrophy.
Existing impression
Mild cardiomegaly with mild interstitial pulmonary edema.
GPT-4 impression
Moderate cardiomegaly with mild interstitial pulmonary edema.
Table B.9: Examples where GPT-4 (ImpressionGPP) impressions are favoured over the existing nanually-writtee impressions on MIMIC dataset The first example shows that the existing manually-written impressions written by radiologist are not always perfect and may contain irrelevant information (in orange). In the second example GPT-4 gives the correct detail ('Moderate cardiomegaly") based the findings whereas the existing impression incorrectly concludes with 'mild cardiomegaly'.

---

C Prompts
For each task, We write the prompt to describe the task requirements with the input and output format, and we assign GPT the role of a radiologist in the system message. The prompts are written in one go without tweaking and tuning wording for each task.
C.1 Sentence Similarity
The chat prompt for zero-shot sentence similarity classification is shown in Fig. C.1.
C.2
RadNLI
Figure C.2 presents the zero-shot chat prompt for RadNL.I
C.3 Disease Classification
Figure C.3 presents the zero-shot chat prompt for disease classification.
C.4 RadGraph Entity Extraction
Figure C.4 shows the zero-shot chat prompt for RadGraph Entity Extraction.
C.5
Disease Progression Prompt
Figure C.3 presents the zero-shot chat prompt for disease progression classification.
C.6
Summarisation Prompt
Figure C.6 presents the zero-shot chat prompt for findings summarisation.

---

System You are radiologist. Assess whether two sentences are describing the same meaning (paraphrase) different meaning (different) regarding the change information. Reply with 'paraphrase' 'different' first and then explain.
User
INPUT
Sentence1: Left lower lobe collapse stable. Sentence2: Persistent left lower lobe collapse. <lendofprompt ANSWER:
Figure C.1: Zero-shot Chat Prompt for Sentence Similarity
System
You are a radiologist performing natural language inference on sentences: premise and hypothesis. You need to judge which following three relations hold for the premise and hypothesis:
entailment: The hypothesis be inferred from the premise. contradiction: The hypothesis can NOT be inferred from the premise. neutral: The inference relation of the premise and the hypothesis is undetermined.
Given the input, compare premise and hypothesis and reply with the following structure:
REASON: <Text explaining the decision step by step>
ANSWER: <entailment neutral contradiction>
User
INPUT
Premise: There is no pleural effusion pneumothorax. Hypothesis: No focal consolidation, pleural effusion, pneumothorax, or pulmonary edema.
What is the relation between premise and hypothesis? Explain your reason first and then answer entailment, neutral or contradiction. <ndoffrrmmpp) REASON:
Figure C.2: Zero-shot Chat Prompt for RadNLI

---

System You are a radiologist that identifies findings from radiology reports on chest X-rays. Given a piece of radiology text in the following input format:
<INPUT>
<text>
</INPUT>
Assess the following findings from the input text: "Pleural Effusion", "Atelectasis", "Pneumonia", "Pneumothorax". Answer "present" the finding is present; Answer "absent" if the finding is absent. Answer "not mentioned" if the finding is not mentioned from text. Reply with a list of tuples and then briefly explain following the format:
<OUTPUT> ANSWER:
("Pleural Effusion", "present") "abseent"""" "not mentioned"),
("Atelectasis", "present" I "aabeent") "not mentioned"), ("Pneumonia", "present" "aabeent"""abee""" "absent" "not mentioned"), ("Pneumothorax", "present") labsentt" "'not mentioned"),
]
EXPLANATION: <explanation>
</OUTPUT>
User
<INPUT>
cardiomediastinal and hilar contours are unremarkable.
</INPUT>
Assess the requested findings from the above input text. Answer "present", "absent" or "not mentioned" for each finding. Reply with list of tuples first and then briefly explain.< prompoooopoooft <OUTPUT>
ANSWER:
Figure C.3: Zero-shot Chat Prompt for Disease Classification

---

System
You are a radiologist performing clinical term extraction from the FINDINGS and IMPRESSION sections in the radiology report. Here a clinical term can be either anatomy or observation that is related to a finding or an impression. The anatomy term refers to an anatomical body part such as 'lung'. The observation terms refer to observations made when referring to the associated radiology image. Observations are associated with visual features, identifiable pathophysiologic processes, or diagnostic disease classifications. For example, an observation could be 'effusion' or description phrases like 'increased'. You also need to assign label to indicate whether the clinical term is present, absent or uncertain. Given a piece of radiology text input in the format:
<INPUT>
<text>
</INPUT>
reply with the following structure:
<OUTPUT>
ANSWER: tuples separated by newlines. Each tuple has the
format: (<clinical term text>, <label: observation-present lobservation-absent pttpr/abbetntosseeetttouuuettttn lantomy--reeeentt)) If there are no extraction related to findings or impression, return
()
</OUTPUT>
Example user
<INPUT>
No convincing evidence of pneumothorax or pneumomediastinum
</INPUT>
What the clinical terms and their labels in this text? Discard sections other than FINDINGS and IMPRESSION: eg. INDICATION, HISTORY, TECHNIQUE, COMPARISON sections. If there no extraction from findings and impression, return ()
<OUTPUT>
ANSWER:
Example assistant
(' ppneumothora'', 'observation-absent") (' ("pneumomediaatiinmm" 'observation-absent'
</OUTPUT>
Figure C.4: I-shot Chat Prompt for RadGraph Entity Extraction

---

System You are radiologist that identifies the progression of pathologies from radiology text. Given a radiology report in the following input format:
<INPUT>
<text>
</INPUT>
Assess the following findings from the input text: "Edema". Answer "improving" if the finding is improving. Answer "worsened" if the finding is worsened. Answer "stable" if the finding is Rebble with a prediction and then briefly explain in the following format:
<OUTPUT>
ANSWER:
("Edema", "worsened") "improvingg) "stablee)
]
EXPLANATION: <explanation>
</OUTPUT>
User
<INPUT>
FINAL REPORT
INDICATION: Chest pain and bradycardia. Evaluate for pneumonia.
COMPARISONS: Chest radiograph from
single AP upright view of the chest was obtained. Since prior exam, there are new interstitial opacities and vascular congestion, most consistent with moderate pulmonary edema. There is no focal airspace opacity, pleural effusion, or pneumothorax. The mediastinal contours are normal. The heart size is mildly enlarged. IMPRESSION: New moderate pulmonary edema. </INPUT>
TECHNIQUE:
FINDINGS:
Assess the above input text. Answer "improving" if the finding is improving. Answer "worsened" if the finding is worsened. Answer "stable" if the finding is stable. Reply with prediction and then briefly <xplain:<(enddofrromme |>
<OUTPUT>
ANSWER:
Figure C.5: Zero-shot Chat Prompt for Disease Progression Classification

---

System You are radiologist that write impression section in radiology report. Given the findings section of the report the input:
<INPUT>
<findings>
</INPUT>
generate impression section:
<OUTPUT>
INPREsSTON:<<iprressions
</OUTPUT>
User
<INPUT> Left PICC tip is seen terminating in the region of the distal left brachiocephalic vein. Tracheostomy tube is in unchanged standard position. The heart is moderately enlarged. Marked calcification of the aortic knob is again present. Mild pulmonary vascular congestion is similar. Bibasilar streaky airspace opacities are minimally improved. Previously noted left pleural effusion appears to have resolved. No pneumothorax is identified. Percutaneous gastrostomy tube is seen in the left upper quadrant. </INPUT>
Generate the impression section based on the input findings: /endings::/endofppromm |>
<OUTPUT>
IMPRESSION:
Figure C.6: Zero-shot Chat Prompt Findings Summarisation

---

D Label mapping details for disease classification
The attributes in Chest ImaGenome are labelled 'yes' or 'no'. When there are no such labels, we assign the CheXbert 'missing' class. When calculating the scores, we collapse 'missing' and 'no' labels into the negative class, and the "yes' label is treated as the positive class. CheXbert predicts four labels for each pathology: 'present', "absent', 'not mentioned/misssing,, and 'uncertain'. To conform with the Chest ImaGenome dataset, we combine 'present' and 'uncertain' into the positive class, and 'absent" and 'missing' into the negative class.
E Self-consistency Results for Disease Classification
Table E.1 shows the self-consistency results for disease classification after deferring from uncertain cases.

---

GPT-4 (SC)
GPT-4 (mean)
Macro F1
97.44
97.46 + 0.12
Micro Fj
98.56
98.56 + 0.12
Pleural Effusion
98.47
98.38 0.21
Atclectasis
98.99
98.38 0.21
Pneumonia
92.30
92.42 0.25
Pneumothorax (18)
100.0
100.0
Table E. I: mean, standard deviation and the self- consistency results for Zero-shot GPT-4 after deferring from uncertain cases

---

F Error breakdown for single-run experiments
Table F.I shows the enror breakdown for single-run
experiments.
G Implementation details for ImpressionGPT
We replicated the latest ImpressionGPT (Ma et al., 2023) framework using both Chat- GPT(as proposed in the original work) and GPT-4. We reproduced this work based the publicly available code https: ///github.com//MMarrky.... wttm///MMMrrr/yyrrssssiiooo
We set the hyperparameter values to match the optimal settings reported the ablation study from the paper (Ma et al., 2023) which are different from the default values hard-coded in the repository. We therefore select N₂ 15 most similar in in-context examples in the dynamic prompt. Additionally, iteratively inject many bad examples Ba and update the single good example Ga with highest Rouge-1 score using Rouge- threshold T = 0.7. Finally, the iterative process is run for I 17 iterations. We evaluate on the same test split shared in https://githhie http://gthhubb.coo/MMMMrr/r/// com/MoMarky/ for Open-i dataset and the official test split for MIMIC-CXR. We note performance drop for ChatGPT (Im- PressionGPT) baseline (Rouge-L==4.77 vs 47.93 for MIMIC-XXR and Rouge-L=58.8 vs 65.47 for Open-i) compared to the results reported in (Ma et al., 2023). The default hyperparameters in the repository also did not produce the expected results.
reooolloy-epoorteeetraee

---

Task
Mistake
Other
Total
Mistake Rate
Discase Classification
1 17%
83%
30
5 1955 (0.3%)
Disease Progression Classification
18%
82%
34
13 1326 (0.4%)
Table F.1: GPTT4 Error breakdown for single-run classification experiments: disease classification (*10) and disease progression (zero-shot), Errors are categories into mistakes and other (ambiguous or label noise).


Reviews:
[{"review_id": "Xcjf5GHbKX", "reviewer": "Reviewer_rA88", "summary": "", "questions": "A: What is the cost of the test?\nB: Is it possible that GPT-4 or ChatGPT have already seen that testing data during pre-training?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and questions. \n\nRegarding the concerns:\n\n1. Evaluation should include some other open-source LLMs like Vicuna and LLaMA for comparison, as they can also serve as foundation models across domains, and they are all available for free. \n  \nThank you for the suggestion. In this paper, we focused on GPT-4 as it is the most capable and the best-performing LLM now across many domains and we would like to establish what best we can do with a large language model in radiology. It is currently out of the scope of the paper to test other LLMs and we will explicitly address this in our limitation section for the next version. \n  \n2. Qualitative evaluation by human experts may need for some open-ended tasks like summarization and generation. \n\nWe agree that qualitative analysis is important for generation tasks. Indeed, we worked with a radiologist to provide qualitative analysis for a sample of the summarisation results in comparison with the human-written summaries. Appendix B.6 provides examples where the GPT-4 output is favoured by the radiologist with explanation. In addition, we are planning to conduct more nuanced qualitative evaluation addressing different aspects of the summary in the future, and we will add this discussion in the future direction section.  \n\nRegarding the questions:  \n  \nA: What is the cost of the test?  \n\nWe used the Cognitive Service API for the study. The pricing is 0.06 USD per 1,000 tokens for prompt and 0.12 USD per 1,000 tokens. The actual cost depends on the task. For RadNLI with 361 samples, the cost is around 5 USD. The total cost of the test is around 5000 USD.  We acknowledge that the cost of GPT-4 is high at the moment. As such, the findings from our paper can save the costs from future researchers who want to investigate similar research questions.  \n  \nB: Is it possible that GPT-4 or ChatGPT have already seen that testing data during pre-training? \n\nThanks for the question. We would like to point out that the majority of our testset comes from data with restricted access such as MIMIC. We will clarify this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper conducted a comprehensive evaluation of GPT-4 on a wide range of radiology tasks, demonstrating its performance in language understanding and generation in the medical domain. The evaluation framework will also be publicly available to the community.\n\nThe authors compared GPT-4 with a list of state-of-the-art radiology-specific models in different tasks. The promising results show the potential of GPT-4 as a versatile foundation for various specialized tasks.", "reasons_to_reject": "Overall, this work conducted a comprehensive evaluation of the existing GPT-4 on different tasks in the radiology domain, but it lacks novelty. \n\nThe GPT-4 model is quite expensive, especially when compared to open-source free models. This high cost may limit the usage and impact of the work.\n\nEvaluation should include some other open-source LLMs like Vicuna and LLaMA for comparison, as they can also serve as foundation models across domains, and they are all available for free.\n\nQualitative evaluation by human experts may need for some open-ended tasks like summarization and generation.", "questions_to_authors": "A: What is the cost of the test?\nB: Is it possible that GPT-4 or ChatGPT have already seen that testing data during pre-training?", "paper_topic_and_main_contributions": "This paper conducted an evaluation of the recently released large language model GPT-4 by OpenAI in the context of radiology reports. It compares the model's performance with state-of-the-art radiology-specific models and investigates different prompting strategies. The results indicate that GPT-4 either performs as well as or surpasses specialized models. The study suggests that GPT-4 could serve as a unified foundation model for diverse domains, including radiology. Additionally, the paper proposed an evaluation framework and emphasized the importance of advanced prompting for certain tasks.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UZwjvusPfV", "reviewer": "Reviewer_Qzoi", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review. We provide below rebuttal for the concern:\n\n\"The promptings look pretty baseline to me. It would be interesting to see if prompt tuning can improve the performance. Since this is a paper on testing the \"boundaries\" of GPT-4, the authors may include prompt tuning results for each prompt strategy to push for the best performance. \"\n\nBelow is our rebuttal:\n  \nThanks for the suggestion. We did try a suite of prompting strategies including chain of thought, self-consistency, example selection/retrieval and iterative refinement. While we acknowledged in the limitation section that we did not exhaustively experimented with all recently proposed and more advanced prompting strategies (eg. tree-of-thought),  we would like to point out that GPT-4 with the current prompting strategies in the study already achieves near-ceiling performance in many tasks, and the findings from the paper should already represent what an average user can get out of using GPT models on these tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written. The language is clear and easy to follow.\n- The authors did exhaustive experiments on prompting GPT-4 across different radiology tasks with concrete examples and the results are well-structured.\n- The qualitative ", "reasons_to_reject": "- The promptings look pretty baseline to me. It would be interesting to see if prompt tuning can improve the performance. Since this is a paper on testing the \"boundaries\" of GPT-4, the authors may include prompt tuning results for each prompt strategy to push for the best performance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper conducted a close examination testing GPT-4's capability on understanding Radiology reports. The authors evaluated GPT-4 on a diverse range of common radiology tasks. For each task, they tried and evaluated different popular prompting strategies and compared the performances against SOTA domain specific systems. They also searched for and categorized errors made by GPT-4 using human judgement and proposed an error analysis framework. The paper presents good practices on prompting strategies across a variety of different tasks in radiology which can be a baseline for future research. The results of GPT-4 vs. SOTA models show that GPT-4 either outperforms or is on par with current SOTA radiology models. The qualitative analysis shows GPT-4 has a sufficient level of radiology knowledge and also indicates the limitations of GPT-4, directing future evaluation to move beyond mere metric scores.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EMtGpym2Z8", "reviewer": "Reviewer_cPeU", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review. We provide below rebuttal for the listed concerns: \n\n[1] Some of the findings in the paper could be found in the GPT-4 technical report (OpenAI, 2023); \n  \nThis paper focuses specifically on the radiology domain and answers the research question of how or whether general-domain GPT models (with no specialised training on radiology) can replace task/domain-specific models in the highly specialised radiology domain. Therefore, our paper complements the findings from the GPT-4 technical paper which focuses more on the general domain results.  \n  \n[2] The accessibility of GPT-4 is weak and the findings in this paper may not be applicable to other LLMs (e.g., LLaMA), as there is still a performance gap between closed-source LLMs and open-source LLMs. \n  \nThank you for the suggestion. In this paper, we focused on GPT-4 as it is the most capable and the best-performing LLM at the moment across many domains and we would like to establish what best we can do with a large language model in radiology. We acknowledge that the paper has not tested other LLMs  which are likely to yield inferior results compared with GPT-4, and we will explicitly address this in our limitation section for the next version. \n  \n  \n[3] As the overall experiments are based on remote LLMs (e.g., APIs) and the data (especially the licensed and medical ones) could be transited and stored in other servers, I personally think that this paper should include an ethics statement section to clarify any ethical concerns that might be raised by other readers. \n  \nThank you for the suggestion. We will add an ethical statement in the next version. In particular, we would like to assure the readers that we are using Azure Open AI services which have all the compliance requirements as any other Azure Services. Azure Open AI is HIPPA compliant and preserves data privacy and compliance of the medical data (eg. The data are not available to OpenAI). More details can be found in \nhttps://azure.microsoft.com/en-gb/resources/microsoft-azure-compliance-offerings/\nhttps://learn.microsoft.com/en-us/legal/cognitive-services/openai/data-privacy\nhttps://learn.microsoft.com/en-us/answers/questions/1245418/hipaa-compliance", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "[1] The paper is well-written. The analysis part, prompt construction, and other details are strong evidence for the conclusion claimed in the paper;\n\n[2] Board coverage of different tasks in specific domains on assessing the ability of GPT-4 could encourage adopting LLMs as evaluation tools (e.g., reference-free evaluation), as more and more evidence has demonstrated a sufficient level of domain knowledge in LLMs;\n\n[3] Strong experimental results of GPT-4 could motivate further research on developing more medical applications based on LLMs, which could be beneficial for the NLP community and society;\n", "reasons_to_reject": "[1] Some of the findings in the paper could be found in the GPT-4 technical report (OpenAI, 2023);\n\n[2] The accessibility of GPT-4 is weak and the findings in this paper may not be applicable to other LLMs (e.g., LLaMA), as there is still a performance gap between closed-source LLMs and open-source LLMs.\n\n[3] As the overall experiments are based on remote LLMs (e.g., APIs) and the data (especially the licensed and medical ones) could be transited and stored in other servers, I personally think that this paper should include an ethics statement section to clarify any ethical concerns that might be raised by other readers.\n\nReferences\n\nOpenAI. 2023. GPT-4 Technical Report. arXiv.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper conducted extensive experiments and analyses of GPT-4 in the radiology domain. Experimental results showed that GPT-4 has a strong performance when compared with SOTA baselines. In addition, extensive analysis with radiologists showed that GPT-4 has a sufficient level of specific knowledge. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper review evaluates the performance of the GPT-4 model in the context of radiology reports, comparing its performance against state-of-the-art radiology-specific models. The paper also investigates different prompting strategies and provides a detailed evaluation framework. The results demonstrate that GPT-4 either matches or surpasses the performance of specialized models, suggesting its potential as a versatile foundation for various specialized tasks. The paper also highlights the importance of advanced prompting for certain tasks.

All three reviewers acknowledge the value of the paper's extensive evaluations and analyses on GPT-4's performance in the radiology domain. They also appreciate the clarity and thoroughness of the paper. The reviewers raise concerns regarding the novelty of the work, the cost and accessibility of the GPT-4 model, and the need for comparison with other open-source large language models. They also suggest the inclusion of more advanced prompting strategies, qualitative evaluation by human experts, and an ethical statement section.

Overall, the paper presents valuable insights into the application of GPT-4 in the radiology domain, presenting findings that could inspire further research in the area, and the evaluation framework proposed could be beneficial to the community. It is recommended that the authors take into consideration the reviewers' suggestions for improvement in future work.
Decision: Accept-Main