Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting
Authors: Benjamin Yan
Ruochen Liu
David E Kuo
Subathra Adithan
Eduardo Pontes Reis
Stephen Kwak
Vasantha Kumar Venugopal
Chloe P O'Connell
Agustina Saenz
Pranav Rajpurkar
Michael Moor
Venue: EMNLP 2023 Findings (2023)
Keywords: language model
radiology
report generation
few-shot prompting
in-context learning
radgraph

Abstract: Automatically generated reports from medical images promise to improve the workflow of radiologists. Existing methods consider an image-to-report modeling task by directly generating a fully-fledged report from an image. However, this conflates the content of the report (e.g., findings and their attributes) with its style (e.g., format and choice of words), which can lead to clinically inaccurate reports. To address this, we propose a two-step approach for radiology report generation. First, we extract the content from an image; then, we verbalize the extracted content into a report that matches the style of a specific radiologist. For this, we leverage RadGraph---a graph representation of reports---together with large language models (LLMs). In our quantitative evaluations, we find that our approach leads to beneficial performance. Our human evaluation with clinical raters highlights that the AI-generated reports are indistinguishably tailored to the style of individual radiologist despite leveraging only a few examples as context.

Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting
Benjamin Yan', Ruochen Liu', David E. Kuo', Subathra Adithan', Eduardo Pontes Reis', Stephen Kwak,, Vasantha Kumar Venugopal", Chloe P. O'Connell', Agustina Saenz", Pranav Rajpurkar Michael Moor
Stannorr University, JJawaharlal Institute of Postgraduate Medical Education and Research 3 John Hopkins University, *Aligarh Muslim University. " Harvard Medical School,
Tequal senior authorship
Abstract
rephrase existing text, paving the way for new ap- proaches automated report generation in radiol- ogy. However, despite the promise of LLMs, their application to this task is not without challenges. One principal concern is their tendency to 'hallu- cinate', i.e., to generate false information (even if plausible sounding), which can be particularly problematic in high-stakes settings such when generating reports from medical images.
Automatically generated reports from medi- cal images promise to improve the workflow radiologists. Existing methods consider an image-to-report modeling task by directly gen- erating a fully-fledged report from an image. However, this conflates the content of the re- port (e.g., findings and their attributes) with its style (e.g., g., format and choice of words), which lead to clinically inaccurate reports. To ad- dress this, we propose two-step approach for radiology report generation. First, we extract the content from image; then, verbalize the extracted content into a report that matches the style of a specific radiologist. For this, we leverage RadGraph-a graph representation eeports-tggether together with large language mod- els (LLMs). our quantitative evaluations, find that our approach leads to beneficial per- formance. Our human evaluation with clinical raters highlights that the Al-generated reports are indistinguishably tailored to the style of in- dividual radiologist despite leveraging only few examples context.
Previous attempts automated report genera- tion have largely focused on approaches that aim produce fully-fledged reports directly from medical images (i.c., a image-to-repont modelling task) (Al- laouzi et al., 2018; Chen et al., 2020; Wang et al., 2022a). However, this conflates the content of the report (i.e., the radiology entities and attributes that are described) with its style (i.e., the complement of radiology entities and attributes, everything needed in terms of language, grammar, and struc- ture to formulate a fully-fledged report, on top of the content representation), which limits the flexi- bility and applicability of such models.
This problem is also reflected in the employed evaluation metrics of existing approaches. They frequently optimize for traditional natural language processing (NLP) metrics, such as BLEU, ROUGE, or METEOR, which measure the similarity of the generated report to reference report based on lex- ical overlap. These metrics, while generally useful, may not correlate with the clinical correctness or usefulness of the report ( (Yu et al., 2022). Previous work has demonstrated that optimizing for clinical metrics that measure the content's 's relevance, such as the RadGraph scores (Jain et al., 2021), is crucial in generating reports that accurately represent the image findings. The RadGraph score gauges simi- larity in extracted radiology entities and relations with a ground truth annotation. This emphasizes the report's clinical content as opposed to style or lexical overlap, and makes it an amenable mea- sure of the report's usefulness and completeness
1 Introduction
Generating radiology reports from medical images is a crucial task in the field of medical imaging. For human interpreters to write such reports is not only time-consuming and labor-intensive but also requires high level of expertise (Hartung al., 2020). Furthermore, these reports are often subject to inter-observer variability, potentially compro- mising the consistency and accuracy of the reports' findings. As a result, there growing interest in methods for automated radiology report genera- tion, which can alleviate these issues and improve overall efficiency of the diagnostic process.
Recent advances in large language models (LLMs) have shown great potential for generating high-quality text, with the ability to customize out- puts based on user-specified instructions (Brown et al., 2020b). These models have been utilized to

---

clinical practice.
high-quality report.
Here, we propose to generate radiology reports as a two-step procedure for disentangling report content and style. In the first step, dedicated model is trained to extract pure content from the image. Specifically, it generates structured repre- sentation (called RadGraph) of the entities and at- tributes that are present in the image. In the second step, frozen LLM generates stylized report from this structured representation. Given a few report examples as context, the LLM can on-the-fly fly adapt the report style closely to the style of a target ra- diologist or hospital template for whom the report should be drafted. This model stylization could offer several advantages for radiology workflows: flexibility, in generating reports targeted to their readership, such as ones with less jargon that are more accessible to patients; consistency, in ensur- ing clear communication between a radiologist and the referring physician, who may be accustomed to a particular style of reporting; and emphasis on pre- ferred information, in highlighting findings most relevant to specialist's ss scope of practice (c.g., follow-up up a patch of pneumonia, or the correct location of pacing lead in the heart).
Our approach addresses the limitations of end-to- end image-to to-rroort generation, as the LLM does not need to suggest facts about the image. Instead, it can focus on rephrasing the structured set of enti- ties and attributes that were first derived from the image and infuse the desired style into the report. At prediction time, the LLM in-context learns cerialization-tt-rrpon mapping from few exam- ples of a target report writing style. Our method not only offers a novel solution to the challenges posed by previous approaches but also enhances the cus- tomization and adaptability of radiology reports generated by LLMs.
This paper makes the following contributions:
First, we develop a method for extracting structured content- is, a serialized ver- sion of RadGraph- radiology images.
Second, we propose strategy for generating stylized reports from this extracted content by means of in-context learning from few style- specific example reports.
Third, our overall system (combining content extraction and style generation) achieves com- petitive performance at radiology report gen- eration.
In the first step of our approach, we alter the supervision signal of an image-text model to a se- rialization of the clinical entities (as captured in RadGraph) and their attributes, rather than the full report text. This step ensures the content extraction model focuses only on generating the report's clin- ical content, measured by RadGraph score, rather than optimizing for traditional NLP metrics such as BLEU that may not correlate with the report's clin- ical relevance. By generating the clinical content first, we prioritize the report's clinical usefulness over its stylistic quality, which can be improved and even personalized in the second step.
Fourth, in a human style evaluation clinical ex- perts were not able to distinguish real reports from AI Al-generated ones that were adapted to the writing style of individual radiologists.
Related Works
2.1 Medical Report Generation
Medical report generation (MRG) has seen a re- cent insurgence in the field of medical Al. Early works (Allaouzi et al., 2018) adhere to the meth- ods of image captioning models (Vinyals et al., 2015; Xu al., 2015), leveraging deep CNNs extract image features and RNNs to generate text descriptions in an encoder-decoder fashion. Mean- while, emerging in several works was an auxiliary classification task to predict certain medical abnor- malities, with the aim of more structured guidance for report generation (Shin et al., 2016; Wang et al., 2018: Yin al., 2019; Yuan et al., 2019). Later, the use of the attention mechanism in MRG sys- tems became increasingly prevalent (Jing et al., 2017; Chen et al., 2020). To further bridge visual and linguistic modalities while incorporating med-
For the second step, we leverage GPT-3.5 (a gen- erative LLM developed by OpenAl) to transform the predicted serialization of clinical entities and attributes from the image into a styled radiology report (Brown et al., 2020a). These models have shown great promise in generating high-quality text (Anil et al., 2023), including summaries and paraphrases of existing text (Li et al., 2023a), which we can use to inject hospital-specinf style, as well to enhance readability and understandability. By separating the content generation and style injec- tion steps, we can ensure the model optimizes for the relevant criteria for each step, resulting in a

---

Conventional:
Ours:
Training
Inference
Report examples from Dr.
Reports
Images
Reports
Images
Findings:
The lungs are
The fungs ore cleor No sign of therax.
dlear. No sign of
Sertalized RadGraph
1. Content extrac extraction
New Image
RadGraph
Serialized RadGraph
learning
Report
Lungs clear No presmathorax
Vision encoder
Vision
1 LM loss
LM loss
Serialization
LIM
Text decoder
Text decoder
Generated report
Generated RadGraph Serialization
Green target for supervision
Direct: "image-to-repot"
Indirect: "Image-to-Serialzzation"
Generated report style Dr.
Figure 1: Overview our two-step pipeline for generating radiology reports (right panel) contrasted with the standard 'image-to-repen strategy for directly generating radiology reports from X-ray images (left panel). During training, we consider set of paired chest X-ray images and radiology reports. preprocess the reports by extracting its content in the form graph representation (RadGraph). We serialize the RadGraph into condensed summary of the clinical content of the report. From there, this RadGraph serialization is used as the supervising signal an image captioning model. For illustration purposes, we omit to display the text encoder that augments both captioning models with clinical context. At inference time, generate a serialized RadGraph. By providing few example reports of specific radiologist (and extracting their RadGraphs). an LLM can in-context learn verbalize the structured content extracted from the image into fully-fledged report in the style of the target radiologist.
chest X-ray is fed through the model to
ical domain knowledge, various kinds of knowl- edge graphs have been explored for use (Li et al., 2019, 2023b; Zhao et al., 2021; Zhang et al., 2020; Liu et al., 2021). In our study, we use the classic encoder-decoder architecture, as it is common denominator of many MRG approaches.
rectly trains the model to generate serialized (i.e., text) representation of RadGraph as we decouple the content and style generation and focus solely on the clinical correctness during the content gen- eration stage. Additionally, serialized RadGraphs allow us juxtapose dense content representations with stylized reports which enables style adaptation via in-context learning.
Subsequent studies acknowledge the constraints of traditional natural language generation metrics when assessing medical reports, prompting grow- ing emphasis ensuring clinical accuracy. Rad- Graph (Jain al., 2021) is dataset of entities and relations in full-text chest X-ray radiology reports based on a novel information extraction schema. (Yang et al., 2022) adopts the knowledge graph provided by RadGraph as general knowledge em- bedding. (Wang et al., 2022b) employed clas- sification loss for medical concepts provided by RadGraph. (Delbrouck et al., 2022) improves the factual completeness and correctness of generated radiology reports with well-designed RadGraph reward. Most existing methods develop metrics around RadGraph and integrate that into the objec- tive function; our approach, on the other hand, di-
2.2 Large Language Models
Recent watersheds such as the Transformer chitecture (Vaswani et al., 2017), generative pre- training objectives (Radford et al., 2018), and in- creased computing power have facilitated the train- ing of large language models comprising billions of parameters (Brown et al., 2020a; Chowdhery et al., 2022; Touvron et al., 2023). These advances have significantly burgeoned model capability in tasks such as translation, summarization, and generating long text that closely resembles human language. In 2021, OpenAl announced GPT-3 (Brown et al., 2020a; Floridi and Chiriatti, 2020), gener- ative language model featuring an unprecedented 175 billion parameters. Their study introduces in-

---

1. Clinical Report
2. RadGraph
3. Subgraph Serializations
4. RadGraph Serialization
The lungs are clear. incidentally noted are biiatoral cervical ribs.
FINDINES:
FINDINGS
FANDINGS
biiseral Ahall
carvica AM6 on
FINDINGS
ANATOPP
cleer
IMPRESSION:
: There is no acutte
bisteral cervical
IMPRESSION
cartiopulmonery and uent consolidationns present
cardiopulmonary no corfiuent consolidation
IMPRESSION
IMPRESSION
conflurn 066 DA
no route cardlcpulmonrry process
Average Taken Count: 39.7
Awerage Token
CES-DA
Lotaol
confiucnt concolidation
Overview of the RadGraph serialization. In the RadGraph entities, ANAT-DP,, OBS-DP, and OBS-DA refer to present anatomical, present observational, and absent observational entities, respectively. The token counts refer to the averages across test set.
Figure 2:
context learning, the ability of LLMs to learn to perform task simply by being provided few ex- amples context-withhe
RadGraph are either anatomical (e.g, lungs, cardio- mediastinal, carina) or observational entities (e.g., acute, abnormality). The edges are directed and heterogeneous, capturing three types of relations modify, located at, suggestive of of-between entities, Nodes and edges are automatically obtained via a named entity recognition and relation extraction model on MIMIC-CRR reports, employing the DY- GIE++ framework from Wadden et al. (2019). This embodies the Report RadGraph preprocessing step. content extraction in (Figure 1). We dis- tinguish this from content generation (Section 3.4), which is the transformer- prediction of serial- ized content based on the image.
any parameter updates. LLMs have also demonstrated great potential in the medical domain. For instance, GPT-4 has been employed to post-hoc transform free-text radiol- ogy reports into structured reports (Adams al., 2023). As for LLM--bsed MRG systems, Wang et al. (2023) utilized ChatGPT to generate medi- cal reports based on features extracted by neural networks (e.g.. disease classifier), However, their approach does not exploit in-context learning and thus has limited control over format and style.
3 Methods
3.3 RadGraph Serialization
3.1 Dataset
Our study makes use of the MIMIC-CXR (John- son et al., 2019) and RadGraph (Jain et ul., 1021) datasets. MIMIC-CXR is large dataset of 377 110 chest radiographs imaged at Beth Israel Deaconess Medical Center from 227 835 studies, with free- text medical reports. The RadGraph dataset is pub- licly available and comprises radiology text reports and corresponding knowledge graphs (Figure 2). To preprocess reports, we extract only Findings and Impressions, as other sections contain details that cannot be readily referenced from the image, such as patient demographics or lab results from different procedures. Findings refer the direct observations from the image (c.g., opacity of lungs, catheter placement), while Impression summarizes the most urgent inferences and diagnostically rele- vant findings (e.g. presence of pneumonia).
We serialize cach RadGraph into a structured text representation (Figure 2), which serves as the super- vision label for the content generation model (Sec- tion 3.4). This serialization acts dense con- tent representation whose advantages over the free- text report include conciseness and the pruning of non-semantic content (c.g., style-filler words, ceroolloisssspecife phrasing) from the report. The aim is to focus the model purely on generating the content backbone at this step, and defer the style in- jection later stage of the pipeline (Section 3.5).
To exploit the graph structure of the input Rad- Graph, we first extract the weakly connected com- ponents (step (3) in Figure 2). These are the maxi- mal subgraphs where any two nodes can be reached through a path of undirected links. Each compo- nent is thus separate network of spatially and medically related entities, segmenting the chest X- ray's content into distinct regions of interest that can be serialized in parallel.
3.2 RadGraph
As notation, a RadGraph refers to any knowledge graph within the titular dataset. The nodes of a
For each component, we create a text span of all

---

labelled entities. The keywords and maybe are prepended to absent and uncertain entities, respec- tively. The entity ordering follows the syntax of the corresponding tokens in the report to ensure read- ability and lexical fidelity. When the report con- tains both Findings and Impression sections (Fig- ure 2). we analogously stratify the components based on their referenced location in the free-text report. This crucial the content representation should carefully distinguish between factual im- age information (Findings) and clinical inferences (Impressions), even if strongly supported.
Text Encoder We use a Transformer encoder extract features H {h1, h2,,.. .... hu from the clin- ical document text input with length I consisting of word embeddings {wi, W2, wr} where wi E R is the vector representation of the i-th word in the text, is the embedding dimension and h; E R is the attended features of the i-th word to other words in the input document. The features are then transformed into summarization denoted of disease- related topies (such as pneumonia or atelectasis) to be queried from the document, as proposed (Nguyen et al., 2021),
Q {q1 (2: ..., 9n representing
Within Findings and Impression, the components are concatenated and separated by delimiters. The two sections are combined into a single text, which is the report serialization and densely characterizes the full chest radiograph. In the case where the report is not bipartite (e.g., only Impressions), we unify the components under singular section.
Dust Softmax(QH")
(1)
where matrix Q E Raxe is constructed by verti- cally stacking {q), 42..................... qn3 where each vector qi is initialized with random values and subse- quently refined through the attention process, and H E R is formed by stacking (h1, h.,
3.4 Content Generation Model
For the content generation model, we leverage the encoder-decoder architecture that has been widely employed in image-to-text systems. An image en- coder takes chest X-ray images the input and encodes them into visual feature representation. In parallel, text encoder reads clinical documents, such as doctor indications, and transforms the tex- tual content into dense feature vectors.
F Fused Embedding
We obtain the final, contex- tualized embedding D Naae by entangling the visual embedding Dang and text embedding Dizts
D LayerNorm(Deng + Dint)
(2)
where D will be the input for the report decoder.
Report Decoder We use the Transformer the backbone of our report decoder to generate long, robust text. A total of 12 decoder components are stacked together where each component consists of a masked multi-head self-attention component followed by feed-forward layer. The report gener- ation objective (Equation 3) is defined as the cross- entropy loss between predicted words and ground truth y. Here, we denote Piy as the confidence of selecting the j-th word of vocabulary V in the i-th position in the generated text, and Vij binary indicator of whether the j-th word appears in the i-th position of the ground truth.
The visual and text embeddings are then added together and passed through a LayerNorm oper- ation to form contextualized embeddings. The fused embeddings are then fed into the report de- coder, which generates serialized RadGraph word- by-word. The main architecture is adapted from (Nguyen et al., 2021), but we simplified it by re- moving the classifier and the interpretation module eliminate as many potential confounders as we can since our goal is evaluate the influence of the supervision signal.
Image Encoder We adopt a DenseNet- 121 (Huang et al., 2017) model pre-trained on the Ima- geNet dataset as the image encoder. For each input chest X-ray image T, it extracts feature vector d E RR where is the embedding dimension. If imaging study consists of more than one image. the feature will be obtained via max-pooling across all feature vectors extracted from each image. then transformed into n low-dimensional discase representations Dimg E Raxe
L=
y log((()))))
(3)
3.5 Style Generation Step
We describe the process of prompting a pre-trained LLM to generate reports from the serialization. This enables adapting the generation to a spe- cific style by supplying the LLM with relevant

---

in-context examples. Each example is a pair: seri- alization s; from the RadGraph, and corresponding ground truth report r; under the desired style. model from OpenAI,
lates the overlap in clinical entities and relations ex- tracted by RadGraph from both machine-generater and human-generated reports. In this study, we lay emphasis on clinical metrics because we aim to generate reports in different styles while keeping accurate clinical information instead of reports that lexically match the ground truth in the dataset.
We use the a dialogue-basedi based LLM that accepts a sequence of messages as input, rather than singular text prompt. This is useful for inserting style pairs seamlessly: we relay them back-and-forth con- versation between user and assistant roles, where a user role supplies serialization 8, and an assis- tant responds with target report r; for the language model to learn in-context.
3.7 Experimental Setup
We conduct four experiments to scrutinize each individual step and overall performance of our pro- posed strategy for report generation. This includes:
The chain of K examples (ss, is pref- aced by system role message indicating the LLM should act as the report-generatin assistant, estab- lishing its specific task within the radiology-basee dialogue. The remaining prompt is structured as 81 Tı, 82 T2, ................................ ,KK rk, Seval :. At the end, the model is given an evaluation serialization Seval predicted from the chest X-ray image using our content generation model, cuing the LLM to gen- erate the corresponding report prediction feval next. Note that in the zero-shot case, the prompt just Seval :, with no preceding context examples.
Image to Serialization: We evaluate content generation model in terms of comparing the content of the generated serialization with the ground truth report.
2. Serialization to Report: Conditioning on a strong image-to-serialization model, we eval- uate ALM-based style injection by using the ground-truth RadGraphs as input to the LLM.
3. End-lo-end report generation: We evaluate the pipeline end-to-end, feeding the image and clinical context through the content generation model and the generated serialization through the LLM.
3.6 Evaluation Metrics
We present a comprehensive quantitative evalua- tion of our approach with commonly-used metrics concerming both language fluency and clinical accu- racy. For each metric, we display the mean across the n test reports, well as the 95% confidence interval (z + 1.96
4. Human style evaluation: To evaluate the style quality, we let physicians rate sets of 4 radiol- ogy reports where were written by the same radiologists and 1 was AT-generated by our method following the style of the radiologist. The goal for the physicians is to detect the AI- generated report and to justify their choice.
Vn
Natural Language Generation Metrics (NLG)
As for classical NLG metrics, we compute BLEU- 2 and BERT scores. However, these metrics have relevant limitations due to focusing on lexical sim- ilarity or general (non-clinical) semanties, respec- tively, thereby lacking in the assessment of clinical similarity (for more details, see Section A.5).
Baseline The baseline model is adapted from (Nguyen et al., 2021), sharing the same architecture as our image-to-serializztion model (Section 3.4), but its supervision target is the full report (Findings and Impressions) instead of serialization. The train- ing involves an identical parameter set as above.
Clinical Accuracy Metrics CheXbert vector similarity extends beyond BERT by utilizing CheXbert, a model trained specifically on datasets comprising chest X-rays. It computes the co- sine similarity between the indicator vectors of 14 pathologics that the CheXbert labeler extracts from -eeereeeeegeeeeedd and human-generated radiol- ogy reports. It is designed evaluate radiology- specific information but its evaluation is limited to 14 pathologies. To address this limitation, we also adopt RadGraph Fl (Yu et al., 2022) that calcu-
Additional experimental details (training, hyper- parameter search, and infrastructure) are provided in Section A.I.
4 Results
4.1 Image to Serialization
We train two content generation models, one with the full report as the supervision target, and the other with the serialized RadGraph. In table 2, we present the RadGraph Fl evaluation result on the

---

Method
Examples
RadCliQ (1)
RadGraph FI (f)
CheXhert (1)
BLEU (1)
BERT Score (1)
Baseline
3.553 +: 0.032
0.186 + 0.005
0.352 = 0.008
1184::0.000
0.378 0.006
Ours
0
3.527 + 0.027
0.228 + 0.004
0.394 + 0.008
0.162 + 0.003
0.333 0.004
1
3.512 + 0.028
0.226 + 0.005
0.393 = 0.008
0.170 = 0.003
0.343
0.004
5
3.489 1 0.028
0.224 1 0.004
0.391 1 0.008
0.177 1 0.003
0.354 1 0.004
10
3,485 = 0.028
0.224 + 0.005
0.390 = 0.008
0.180 = 0.003
0.357 + 0.004
Table I: Results for the end-to-end report generation. The baseline is the direct image-to-report transformer, i.e., the content generation model trained to predict reports directly from image. Note that RadCliQ (lower is better) is the most important metric as it best corresponds with radiologists' quality assessment (Yu et al., 2022). Best row is bold. Rows leveraging in-context examples are grayed out, whereas the first two rows the exact same input data.
MIMIC-CX test set, comparing the outputs from both models against the ground truth full report.
image-to-report model) in clinical accuracy met- ries (CheXbert similarity, RadGraph F1, RadCliQ), even in the zero-shot style generation case. This illustrates greater accuracy in extracting radiology content from the chest X-ray, the central focus of separating content generation from style injection. Notably, RadCliQ is composite metric found by Yu et al. (2022) best correlate with quality judge- ment of human radiologists.
The comparison is suitable as RadGraph FI is ag- nostic of general lexical similarity, measuring only overlap in radiology entities and relations extracted from the text. We find the model trained on the se- rialized RadGraph outperforms the model trained on the full report. This verifies our assumption that switching the supervision signal to the serialization would help focus the model on generating clinical content with greater accuracy and saliency.
The primary contribution of more examples is improving lexical and non-clinical NLG met- rics (BLEU, BERT Score), similar to our findings in the Serialization - Report step. However, they remain slightly lower than those of the baseline. A potential explanation that during training, the baseline is directly supervised with full reports, while only the extracted content is available to our content generation model-wile with the report itself synthesized with an external, pre-trained LLM.
Supervision Target RadGraph F1 0.186 + 0.005
Full Report
Serialized RadGraph
0.221 + 0.004
Table 2: RadGraph entity and relation Fl score of mod- els trained on full report versus serialized RadGraph.
4.2 Serialization to Report
4.4 Human Style Evaluation
For selecting examples, we draw randomly from the train split. This avoids patient overlap that could unfairly benefit performance on test cases, We compare the L.M--generted reports against the ground truth reports. Results are provided (Ta- ble 3) for varying numbers of in-context examples. We observe strong performance across all metrics, including 0.722 RadGraph Fl mean in the zero- shot regime. Furthermore, lexical overlap metrics such as BLEU and BERT score saw noticeable im- provement with more examples (20.2% and 15.7% increases, respectively, from to 10 examples). This aligns with the aim of in-context learning to improve the style fidelity of generated reports.
Four board-certified radiologists were instructed to write chest X-ray reports in their usual style from 40 randomly selected chest X-ray images from the MIMIC-CXR test set (Figure 4). As several chest X-rays in the MIMIC-CXR dataset were within normal limits or interval follow ups on intensive care patients, duplicate or cuar-duplicte chest X- ray reports were removed upon manual inspection.
The style generation step described in Section 3.5 was then used to produce Al-generated chest X-ray reports in the style of each of the four ra- diologists. From these radiologist-gennertee and AI Al-generated chest X-ray reports, we created 23 sets of four chest X-ray reports for style evalu- ation by physician evaluators (i.e., the target au- dience for written radiology reports). In each set, three reports (corresponding to three different chest X-rays) from the same radiologist, and one AI- generated report (from a fourth chest X-ray) in the style of that radiologist were presented random
4.3 End-to-End Report Generation
We evaluate the end-to-end performance from chest X-ray to report by concatenating the content gen- eration step with the style generation step. The results are presented in Table 1. We observe that our two-step model surpasses the baseline (direct

---

Data collection
Human evaluation
Images
Radiologists
Reports
Serialized RadGraphs
"Which report is Al-generated?"
1. a)
Ra
Report generation in style of
for image le:
Physician raters:
Content X
Image
why?
Language
Context
Generated serialization
Structure
Fraction identified Al-gen reports:
Instruction,
report
R₂
15%
100m
Prompt
LLM
Random guessing beseline
Figure Overview human expert evaluation of the style generation. panel a, set of radiologist write radiology reports in their style for chest X-ray studies. For each written report, extract the RadGraph representation and serialize it into the text representation. Panel illustrates how we Al-generate a radiology report in the style of specific radiologist, Rg, for an imput image Ig. To in-context learn the style of R1, for k shots we provide context of k input/ output pairs of serialized RadGraphs (dense content representation) together with the corresponding stylized reports as written by Rj. Next, to generate a report for a given image 4g. we feed the image through our mmage-o--serializztion model and the generated RadGraph serialization the final query of the prompt. The LLM is then prompted to return generated report in the style of Ri. Panel e illustrates the human evaluation. Physician raters are provided with problems each consisting of reports, 3 written by one radiologist and Al-generated in the radiologist's style.
indicated with hat)
order. Three physician evaluators were asked to identify the Al-generated report out of the four re- ports and indicate whether report content, language, or report structure contributed to their choice (for more details, refer to Section A.3).
"Chest X-ray PA and lateral views:"). Human eval- uators may use particular heuristics to call report AI -l-generated, when faced with similar appearing reports. However, if they are not reflective of the truth, this may result in accuracy less than random.
A one-sided Z-test was performed for the pro- portion of Al-generated reports correctly identified by the three evaluators with a null hypothesis of 25%, corresponding random chance, and an al- ternative hypothesis of >25%, corresponding Al- generated reports being identified at rate greater than that of random chance. Evaluators A, B, correctly identified out of 23 (21.7%), out of 23 (21.7%), and 4 out of 23 (17.4%) AT-generated chest X-ray reports, respectively, for mean ac- curacy of 20.3%. The one-sided Z-test produced p-values of 0.648, 0.648, and 0.832 for Evaluators A, B, and C respectively, and 0.835 when pooling all evaluators together.
Evaluator (n=23)
Evalustor (n=23)
Evaluator C (n-23)
Pooled (n 69)
20
Correct
Figure 4: Human Style Evaluation: All evaluators iden- tified the Al-generated chest X-ray report at rates lower than 25% (corresponding to random chance).
One potential explanation for why evaluators identified <25% of Al-generated reports (worse than random), is variance in radiologist style. For example, our radiologists would sometimes alter- nate between using parentheses to highlight the chest X-ray view (c.g., "Chest X-ray (PA and lat- eral views):") and not using any parentheses (e.g.,
Discussion
We presented novel approach for radiology report generation that disentangles the report's content from its style. Our experiments showed that our

---

method offers several advantages over the prevail- ing paradigm of direct image-to-report modeling. First, training a content extraction model to predict a serialized RadGraph represention from the input image helps focusing the model on the clinically relevant content which is reflected in improved per- formance metrics (Table 2). Second, when concate- nating the content generation step with the style injection step, observe favourable performance compared to the direct image-to-repont baseline (Ta- ble 1). Third, by in-context learning the radiologist- specific mapping from serialized RadGraph to re- port, our method enables the generation of high quality reports that are tailored to the individual radiologist to the degree of indiscerabiliiyy using just few example reports. To ultimately deter- mine the clinical utility of our method, deployment studies will be an exciting venue for future work. Another promising direction would be expand our two-step report generation paradigm to other modalities such as radiology mammograms and magnetic resonance imaging (MRI).
APIs for prompting models generate radiology reports from our content representations. How- ever, cloud-based services are allowed, including the Azure OpenAl platform, which we use for de- ploying and prompting the gpt-3.5-turbo model. A stipulation is the monetary service costs, which are counted at rate per one thousand tokens. These can pose financial barriers to equitable access to high-end language models (which are typically more expensive), as well usage scale.
Furthermore, as discussed in the Introduction section, the use of large language models is ac- companied by their risks of "hallucination" and generating false or misleading content. These risks can be amplified in the critical setting of medical report generation. To mitigate them, we prompt the LLM not to synthesize medical content, but rather to rephrase existing content into readable, stylized prose. We assist the model through providing con- tent serialization to report pairs as in-context ex- amples. This is our style injection step, which we intentfully separate from the content generation step to reduce the opportunity for LLM hallucina- tion when generating the full report prediction.
6 Limitations
A limitation of our approach is that it relies on the accuracy and effectiveness of the initially extracted RadGraph representation, this serves as a key su- pervision signal in our model. Notably, RadGraphs are extracted at inference time using an automated model rather than manual expert labelling. The model achieves high performance in entity and re- lation extraction (Jain et al., 2021) but is susceptible to error, particularly with report inputs that contain rarer medical entities or ambiguous observations. Furthermore, due to employed LLM being served by third party (OpenAI), reproducing our results comes at the financial costs of using Azure OpenAT's service. Another consequence of relying on this service that we cannot guarantce the de- terministically exact reproduction of our findings, as the served LLM models may change and poten- tially degrade over time- for instance, if models are replaced by distilled versions thereof.
Acknowledgements
This project was supported by AWS promotional credits.
References
Lisa Adams, Daniel Truhn, Felix Busch, Avan Kader, Stefan M Nichues, Marcus R Makowski, and Keno K Bressem. 2023. Leveraging gpt-4 for post hoc trans- formation of free-text radiology reports into struc- tured reporting: a multilingual feasibility study. Ra- diology, (0)e22253
Imane Allaouzi, Ben Ahmed, Badr Benamrou, and Mustapha Ouardouz. 2018. Automatic caption gener- ation for medical images. In Proceedings the 3rd International Conference on Smart City Applications, pages 1-6.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.
7 Ethics Considerations
A principal ethical consideration is the de- identified, credentialed medical data we worked with. In particular, responsible usage policy of the MIMIC-CXR dataset (Johnson et al., 2019) pro- hibits sharing access to third parties. This disquali- fies the use of ChatGPT or large language model
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020a. Language models are few-shot learners. Advances neural information processing systems, 33: 177-1

---

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Firic Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020b. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, NeurlPS 2020, December 6- 12, 2020, viriual.
Chih-ying, Roger G. Mark, and Stephen Horng. 2019. Mimic-cxr, a de- ieeetdeetiie publicly available database of chest radiographs with free-text reports. Scientific Data.
Christy Y Li, Xiaodan I. Liang, Zhiting Hu, and Fric P Xing. 2019. Knowledge-driven encode, retrieve, paraphrase for medical image report generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6666-6673.
Hao Li, Yuping Wu, Viktor Schlegel, Riza Batista- Navarro, Thanh-Tung Nguyen, Abhinav Ramesh Kashyap, Xiaojun Zeng. Daniel Beck, Stefan Win- kler. and Goran Nenadic. 2023a. Pulsar: Pre- training with extracted healthcare terms for sum- marising patients' problems and data augmenta- tion with black-box large language models. arXiv preprimt 23X6v:2306.02754.
Zhihong Chen, Yan Song. Tsung- Chang, and Xi- ang Wan. 2020. Generating radiology reports via memory-driven transformer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1439-1449.
Mingjie Li, Rui Liu, Fuyu Wang, Xiaojun Chang, and Xiaodan Liang. 2023b. Auxiliary signal-guided knowledge encoder-decoder for medical report gen- eration. World Wide Web, 26(1):253-22
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou. 2021. Exploring and distilling posterior and prior knowledge for radiology report generation. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 13753- 13762.
Bean-Benoil Delbrouck, Pierre Chambon, Christian Bluethgen, Emily Tsai, Omar Almusa, and Curtis P Langlotz. 2022. Improving the factual correctness of radiology report generation with semantic rewards. arXiv preprint urXiv:2210.1 12186.
Hoang TN Nguyen, Dong Nie, Taivanbat Badamdorj, Yujie Liu, Yingying Zhu, Jason Truong, and Li Cheng. 2021. Automated generation of accurate & fluent medical x-ray reports. arXiv preprint arXiv: 2008. 12126.
Luciano Floridi and Massimo Chiriatti. 2020. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-694
Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
Michael P Hartung, Ian C Bickle, Frank Gaillard, and Jeffrey Kanne, 2020. How to create great radiol- ogy report. RadioGraphics, (0(6):1658-11)))
Hoo- Hoo-Chang Shin, Kirk Roberts, Le Lu, Dina Demner- Fushman, Jianhua Yao, and Ronald M Summers. 2016. Learning to read chest x-rays: Recurrent neu- ral cascade model for automated image annotation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2497-2506.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected con- volutional networks. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 202.
Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven Q. H. Truong, Du Nguyen Duong, Tan Bui, Pierre J. Chambon, Yuhao Zhang, Matthew P. Lungren, An- drew Y. Ng. Curtis P. Langlotz, and Pranav Rajpurkar. 2021. Radgraph: Extracting clinical entities and re- lations from radiology reports. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Baoyu Jing, Pengtao Xic, and Eric Xing. 2017. On the automatic generation of medical imaging reports. arXiv preprini arXiv:1711.08799,
Oriol Vinyals. Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE con- ference on computer vision and pattern recognition. pages 3156-3164.
Alistair E.W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Deng

---

David Wadden, Ulme Wennberg, Yi Luan, and Han- naneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing (EMNLP-IJCNIPP) pages 5784- 5789, Hong Kong, China. Association for Computa- tional Linguistics.
Jianbo Yuan, Haofu Liao, Rui Luo, and Jiebo Luo. 2019. Automatic radiology report generation based multi-view image fusion and medical concept en- richment. Medical Image Computing and Com- puter Assisted MiterventoonMMICAI 2019: 22nd In- ternational Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part VI 22, pages 721- 729. Springer.
Yixiao 7hang, Xiaosong Wang. Ziyue Xu, Qihang Yu, Alan Yuille, and Daguang Xu. 2020. When radiol- ogy report generation meets knowledge graph. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 12910-12917.
Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. 2023. Chatcad: Interac- tive computer-aided diagnosis on medical image using large language models. arXiv preprint arXiv:230..07257.
Haifeng Zhao, Jie Chen, Lili Huang. Tingting Yang, Wanhai Ding, and Chuanfu Li. 2021. Automatic generation of medical report with knowledge graph. In 2021 10th International Conference on Computing and Pattern Recognition, pages 1-1.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M Summers. 2018. Tienet: Text-image em- bedding network for common thorax disease classifi- cation and reporting in chest x-rays. Proceedings of the IEEE conference on computer vision and pat- tern recognition, pages 9049-9058.
A
Appendix
Zhanyu Wang, Hongwei Han, Lei Wang. Xiu Li, and Luping Zhou, 2022a. Automated radiographic report generation purely on transformer: A multicriteria supervised approach. IEEE Transactions on Medicul Imaging. 41110)22803-288
A.1 Further experimental details
In the following, we provide additional details about our experimental setup including information about the model training and used infrastructures.
Zhanyu Wang. Mingkang Tang, Lei Wang, Xiu Li, and Luping Zhou. 2022h. A medical semantic-assiste assisted transformer for radiographic report generation. In Medical Image Computing and Computer Assisted Intervention-Men MMCCAA 2022: 25th International Con- ference, Singapore, September 18-22, 2022, Proceed- ings, Part III. pages 655 664. Springer.
Training The image-to-serialization model is trained and evaluated on the official train/test split of the MIMIC-CXR (v2.0.0) dataset with 213 501 and 2 799 chest X-ray reports, respectively. The ground-truth RadGraph serialization associated with each study is taken from the RadGraph dataset. The model is trained for 25 epochs, with a batch size of 16, learning rate of 0.0001, an embedding size of 512, a weight decay of 0.001, a dropout rate of 0.1, and 12 transformer blocks in the decoder.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho. Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual atten- tion. In International conference on machine learn- ing, pages 2048-2057. PMLR.
Hyperparameter Search We optimized our hy- perparameters for the content generation model through grid search with the help of WandB. Fig- ure visualizes the performance of the model with a different set of hyperparameters. We search the learning rate from [0.001, 0.0001], embedding di- mension from [256, 512], number of transformer decoder components from [6, 12], and weight de- cay rate from [0.001, 0.0001, 0).
Shuxin Yang. Xian Wu, Shen Ge, S Kevin Zhou, and Li Xiao. 2022, Knowledge matters: Chest radiology report generation with general and specific knowl- edge. Medical Image Analysis, 10:102510.
Changchang Yin, Buyue Qian, Jishang Wei, Xiaoyu Li, Xianli Zhang, Yang Li, and Qinghua Zheng. 2019. Automatic generation of medical imaging diagnostic report with hierarchical recurrent neural network. In 2019 IEEE international conference on data mining (ICDM), pages 728-737. IEEE
Infrastructure
The image- soagee-to-sann serriaiizzzaiinn model was trained on an AWS g5.2xlarge instance with one NVIDIA A10G Tensor Core GPU. It was trained for 25 epochs or roughly 50 hours. Evalua- tion was dispatched on two NVIDIA RTX A4000 GPUs with 16 GB of memory. We use the Azure OpenAl service to access the gpt-3.5-turbe lan- guage model, with a cloud-basee based deployment.
Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururaby Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad, Andrew Y Ng, et al. 2022. Evaluating progress in automatic chest x-ray radiology report generation. medRxiv, pages 2022- 08.

---

COnIec
LUM DAIIEDS
1.9100
12
10
Pveiutter&
..000
1.0010
343
220
1.-
043
1001
Figure 5: Sweeps of our hyperparameter search: we use validation loss the metric to search for best hyperpa- rameters.
10
A.2 Serialization to Report Results
Table 3 shows the quantitative results of our Seri- alization to report generation task as described in Section 4.2.
10
A.3 Human Style Evaluation
Here, we provide further results on the human style evaluation of the generated reports. To reiterate the approach, three physicians were tasked to identify Al-generated report within set of four radiol- ogy reports: three were written by the same radiolo- gist and the fourth one (appearing in random order) was generated using our approach using in-context examples from the same radiologist.
Language
Lanquege
Figure 6: Human Style Evaluation Explanations: When style evaluators attributed their choice of Al-generated report, they attributed it primarily to language and structural differences between human-generated and Al- generated reports.
Usser: Generate a chest x-ray report from the following key words:
Figure 6 shows the cumulative count of expla- nations stratified by clinical evaluator (rows), as well as by whether the selection was correct or in- correct (columns). Language (e.g., word choice, grammar, and/or writing style) was the primary heuristic that evaluators used to decide whether a
<SERIALIZATION 1>
Assistant: <REPORT 1>
User: Generate chest x-ray report from the following key words:
report was buman-generadn or Al-generated. fol- lowed by content (e.g., missing important details including extraneous details), and structure (e.g.. different of numbering section headings).
<SERIALIZATION 2>
Assistant: <REPORT 2>
User: Generate chest x-ray report from the following key words:
A.4 LLM Prompting
Here, we illustrate template of our dialogue- based prompt for the LLM generate report prediction, with 2-shot learning to adapt report style. For notation, <SERIAL.IZATION i> and <REPORT i> are placeholders for the i-th serialization and cor- responding ground truth report, respectively, and <EVAL SERIALIZATION> is the serialization pre-
<EVAL SERIALIZATION?
The LLM will proceed to generate its report pre- diction from the evaluation serialization, which we compare against the corresponding ground truth report in the test set.
dicted from an image in the evaluation set.
A.5 Natural Language Generation Metrics BL.EU-2 is widely used in machine translation tasks; measures the similarity between a can- didate translation and one or more reference trans-
System: You are a helpful assistant that gen- erates chest x-ray reports from key words.

---

Examples
RadCliQ (1)
RadGrrap F1 (1)
CheXbert (1)
BLEU (1)
BERT Score (1)
0
1.357 + 0.030
0.722 0.006
0.843 + 0.007
0.388 + 0.005
0.568 + 0.005
1
1.234 + 0.031
0.723
0.007
0.847 = 0.007
0.423 = 0.005
0.614 0.005
5
1.125 + 0.030
0.731 0.006
0.852 + 0.007
0.460 + 0.006
0.652 0.005
10
1.107 + 0.030
0.733 0.006
0.853 + 0.007
0.466 0.006
0.657 0.005
Table 3: Results for the Serialization + Report generation task, using the ground truth serialization in place of the predicted one. The goal of this experiment is to evaluate the style generation in isolation, assuming very strong content generation step (here the ground truth serialization of RadGraph). Note a lower RadCliQ score is better. The best row bolded. Leveraging more in-context examples leads to better performance across all metrics.
lations by comparing their bigram overlaps. Al- though BLEU-2 is a fast and reliable metric, it possesses few limitations, e.g., it does not take into account synonymous words and the proper of grammar. We thus also adopt the BERT score, recently proposed metric for assessing the quality of seeheneegeenerted text. takes into account the semantic similarity between the generated text and reference text by calculating the contextual embeddings of both texts using BERT and measur- ing their cosine similarity. Nevertheless, BERT is general-purpost metric and not at all optimized for capturing clinical semantics and radiological findings. This is why the included clinical accu- racy metries are most salient our quantitative evaluation of radiology report generation.


Reviews:
[{"review_id": "Ra70Uv3Ivg", "reviewer": "Reviewer_5JVq", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their exemplary and insightful comments.\n> **Is there no loss of information from the Serialization to Report process? What are your thoughts on the reason behind the slight difference in performance, with a RadGraphF1 of 0.221 ± 0.004 for Table1 and 0.228 ± 0.004 for Table2?**  \n\nTable 1 reflects the performance of the predicted serialization against the ground truth report; Table 2  reflects the performance of the predicted report—the predicted serialization paraphrased by an LLM to improve lexical coherence. The slight difference of +0.007 thus results from attuned wording and syntax. This can improve readability to the machine parser involved in calculating RadGraph F1, based on parsed radiological entities. We also provide an expanded table with the other metrics. \n\n| Predicted           | Evaluation Target | RadCliq ($\\downarrow$) | Rad F1 ($\\uparrow$) | Chexbert ($\\uparrow$) | BLEU ($\\uparrow$) | BERT ($\\uparrow$) |\n|---------------------|-------------------|-------------------|------------------|--------------------|----------------|----------------|\n| Pred Serialization  | GT Report         |   3.847 $\\pm$ 0.028    |         0.221 $\\pm$ 0.004           |    0.390 $\\pm$ 0.008              |       0.119 $\\pm$ 0.003         |          0.223 $\\pm$ 0.005      |\n| Pred Report         | GT Report         | 3.530 $\\pm$ 0.027  | 0.228 $\\pm$ 0.004    | 0.395 $\\pm$ 0.008            | 0.162 $\\pm$ 0.003     | 0.332 $\\pm$ 0.004     |\n\nThis supports the claim that there is minimal–if any–clinical accuracy loss (Rad F1, Chexbert) from serialization to report, a step that functions primarily to reword the serialized content into an acceptable prosaic quality, rather than generate content anew (to mitigate hallucinations). Furthermore, the table shows that comparing the fully-fletched predicted report with the ground-truth report results in high NLG metric scores (BLEU, BERT) likely attributed to the increased fluency of the predicted report (as compared to the serialization). This expected behaviour therefore also affects the aggregate metric RadCliQ which is a linear combination of the other metrics. We see that serialized text is–by itself–an inadequate substitute for a fully-fledged report, despite capturing relevant clinical information.\n\n\n> **In the Image Encoder section (Section 3.4), it is mentioned that if there are multiple images in one study, they are aggregated. It would be beneficial to evaluate the test performance based on the number of images present in each study, showcasing the performance for cases where a study contains one image, two images, and so on.**  \n\nWe concur with the benefits of this evaluation, and have added the table below for reports generated under the zero-shot setting (i.e. no in-context learning examples). \n\n| # of Images | n (# of studies) | RadCliq ($\\downarrow$) | Rad F1 ($\\uparrow$) | Chexbert ($\\uparrow$) | BLEU ($\\uparrow$) | BERT ($\\uparrow$) |\n|-------------|------------------|-------------------|------------------|--------------------|----------------|----------------|\n| 1           | 1439             | 3.586          | 0.218         | 0.387           | 0.155       | 0.322       |\n| 2           | 1065             | 3.471          | 0.246         | 0.401         | 0.172       | 0.343       |\n| 3           | 265              | 3.463          | 0.232         | 0.411           | 0.166       | 0.343       |\n| 4           | 29               | 3.533          | 0.217         | 0.378           | 0.180       | 0.352       |\n| 5           | 1                | 2.819          | 0.368         | 0.620           | 0.295       | 0.386       |\n\n\nWe find that multiple images (>1) correspond generally with enhanced evaluation metrics. However, this finding is complicated by the unequal distribution of image quantities—glaringly with the singular 5-image study—as demarcated by the external MIMIC-CXR dataset. \n\n> **In the Content Generation Model section (Section 3.4), it is mentioned that the Text Encoder receives the clinical document as input, but the specific details are not provided.**  \n\nWe thank the reviewer for catching this and will clarify the following in the paper. The text encoder takes in a doctor's indication, which is usually a one-sentence short description of the clinical background information of the subject of the chest x-ray study. One example indication is “x-year-old male with history of recently diagnosed pneumonia presenting with worsening fever and productive cough”. We believe that combining radiology images with the doctor's indications replicates the routine tasks of a radiologist effectively.\nThe text embedding generated by the text encoder (detailed on line 334-351) will be concatenated with the visual embedding through summation and layer normalization, and this fused embedding will be used as the input for the report decoder (detailed on line 352-370).\n\n> **It would be valuable to experiment with both the Baseline and the Proposed Model without the Text Encoder and share the results. This paper does not propose a new architectural design but shows the effectiveness of serialized data. Given this focus, I think that showcasing results across various scenarios would be valuable to illustrate the impact more comprehensively.**  \n\nWe appreciate the reviewer’s suggestion and ran the Image-to-Serialization experiments with both the baseline model and the proposed model without the text encoder (i.e. the model only takes images as input) and present the results below. This table also demonstrates the benefits of using doctor’s indications and the text encoder.  We can also see that our proposed model, using the serialized RadGraph as the supervision target, still achieves a better result compared to the baseline model in the absence of the text encoder. Notably, in the absence of the clinical indication, the benefit of our approach compared to training on the full report is even more pronounced (33.6% gain in RadGraph F1 without indication versus 18.8% gain when using the indication). \n\n|  Supervised Target  | Text Encoder (Clinical Indication) |   RadGraph F1  |\n|:-------------------:|:------------:|:--------------:|\n|     Full Report     | \t No \t | 0.128 $\\pm$ 0.004 |\n| Serialized RadGraph | \t No \t | 0.171 $\\pm$ 0.004 |\n|     Full Report     | \t Yes     | 0.186 $\\pm$ 0.005 |\n| Serialized RadGraph | \t Yes     | 0.221 $\\pm$ 0.004 |\n\n> **In the 4.2 Serialization to Report experiment, the zero-shot performance is observed to be 0.722 RadGraphF1. Despite using ground-truth serialization, the performance is not nearly perfect. Is it plausible to attribute this discrepancy to the fact that the performance of the content extractor (Report -> RadGraph preprocessing) might not be flawless?**\n\nThis is a salient point of inquiry. As mentioned in our limitations section, the RadGraph extraction from the report is based on a pre-trained DYGIE++ model (Jain et. al 2021), which is not flawless despite generally strong performance in relation and entity extraction. Beyond limitations in the content extractor, it is conceivable that the LLM rephrasing could introduce some noise. As an ablative test, we compare the ground-truth serialization directly with the ground-truth report, without the LLM to paraphrase the serialization into a zero-shot report. \n\n| Prediction      | Target   | RadGraph F1 |\n|-----------------|----------|-------------|\n| GT Serialization| GT Report|              0.632 $\\pm$ 0.005 |\n| Zero-Shot Report| GT Report|             0.722 $\\pm$ 0.006 | \n\nWith comparable–even better–performance in RadGraph F1 with zero-shot reports, we can largely exonerate LLM hallucination as a cause of zero-shot losses. Instead, the LLM’s enhancements in report readability and wording were successful at improving clinical coherence. The results suggest the RadGraph content extractor is the rate-limiting step here; an enhanced extractor would be portent for higher accuracy. \n\n\n**We appreciate the reviewer’s suggestions and hope that, in light of our changes to the manuscript and explanations in responses to the reviewer’s suggestions, they would consider raising their score.**\n\n*References*:  \n\nSaahil Jain, et al. 2021. Radgraph: Extracting clinical entities and relations from radiology reports. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n- The authors propose a straightforward yet effective approach that involves converting raw reports into structured data. This process aids in the removal of non-semantic content.\n- They also introduce a style-aware report generation technique and carry out a human evaluation to assess its effectiveness.\n- Easy to follow", "reasons_to_reject": "- Is there no loss of information from the Serialization to Report process? What are your thoughts on the reason behind the slight difference in performance, with a RadGraphF1 of 0.221 ± 0.004 for Table1 and 0.228 ± 0.004 for Table2?\n\n- In the Image Encoder section (Section 3.4), it is mentioned that if there are multiple images in one study, they are aggregated. It would be beneficial to evaluate the test performance based on the number of images present in each study, showcasing the performance for cases where a study contains one image, two images, and so on.\n\n- In the Content Generation Model section (Section 3.4), it is mentioned that the Text Encoder receives the clinical document as input, but the specific details are not provided.\n\n- It would be valuable to experiment with both the Baseline and the Proposed Model without the Text Encoder and share the results.\nThis paper does not propose a new architectural design but shows the effectiveness of serialized data.\nGiven this focus, I think that showcasing results across various scenarios would be valuable to illustrate the impact more comprehensively.\n\n- In the 4.2 Serialization to Report experiment, the zero-shot performance is observed to be 0.722 RadGraphF1.\nDespite using ground-truth serialization, the performance is not nearly perfect.\nIs it plausible to attribute this discrepancy to the fact that the performance of the content extractor (Report -> RadGraph preprocessing) might not be flawless?", "questions_to_authors": "", "paper_topic_and_main_contributions": "\n- The authors introduce a novel method for generating reports emphasizing style awareness.\n- The proposed model follows a two-step approach to separate report content and style.\n- In the initial phase, the model is trained to produce serialized data that captures the clinical content of the report.\n- Subsequently, a Language Model (LLM) is employed to convert the predicted serialized data into a coherent report.\n- Importantly, this transformation process ensures that the resulting report aligns seamlessly with the distinct style preferences of a designated radiologist, achieved through in-context learning techniques.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1E3BHKUaPU", "reviewer": "Reviewer_aXs3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback that helped us substantially improve our manuscript.\n\n> **However, the study lacks a compelling argument as to why considering style is \"clinically important\" in generating clinical reports. The more desirable approach and ultimate goal in creating clinical reports should be to focus on generating \"content\" from images as accurately as possible, minimizing hallucinations, and ensuring the report's factual accuracy.**   \n\nThank you for raising this point. However, we believe there are compelling reasons why explicitly modeling style could benefit radiology workflows:\n\n1. **Disentangling style generation from content generation**: This allows us to train an initial content extraction model that is dedicated to extracting the relevant content – as opposed to being trained to jointly predict a blend of clinical content and the report style (e.g. the wording, formatting, structure etc.). Therefore, we believe that disentangling style from content allows us to focus *more* on content and clinical accuracy – not less.   \n\n2. **Intrinsic value of stylizing reports**: We believe there are a multitude of reasons why it is interesting for a model to stylize radiology reports:  \n\n- *Flexibility*: Stylizing can be used to generate reports that are targeted to the readership (e.g. an orthopedic surgeon who may be interested in different aspects of a Chest X-ray than a pulmonologist). Furthermore, flexibly stylizing reports can also enable reports with less jargon that are more accessible to patients.  \n\n- *Workflow*: for human-in-the-loop radiology report generation, automatically drafting a report that already closely aligns with the specific radiologist's preferred style, makes it easier / smoother for the radiologists to edit and approve the report, thereby potentially saving the radiologists valuable time, and reducing administrative burden. In our human evaluation study (Section 4.4), radiologists could not reliably distinguish between real reports and AI-generated ones in the same style - suggesting our model can closely mimic radiologist preferences. This could improve radiologist productivity and satisfaction.  \n\n- *Local adaptability*: Adapting reports to the expectations of referring physicians can optimize communication of clinical findings. For instance, highlighting findings most relevant to a specialist's scope of practice (e.g., follow-up of a patch of pneumonia or correct location of\npacing lead in the heart). In our model, this is achieved via few-shot in-context learning. Clear communication supports appropriate follow-up care.   \n\n- *Consistency*: Some referring physicians may be accustomed to a particular radiologist's style of reporting. Maintaining this style in generated reports can ensure clear communication between the radiologist and the referring physician.  \n\n- *Emphasis on preferred information*: Different radiologists prioritize different information (e.g. due to specialization). Tailored reports can highlight or emphasize the information that an individual radiologist finds most crucial, and can mimic the radiologist in the way this information is interpreted or highlighted.\n\nIn our revised draft, we will more clearly outline the advantages of our model being able to explicitly stylize a radiology report, and to that end will extend our discussion.\n\n> **The authors appear to have de-prioritized what is fundamentally important in the medical field, focusing on style, which makes their argument for the importance of this research less persuasive.**  \n\nThank you for raising this point. Our narrative is not to simply “focus on style”. Instead, our line of reasoning is the following: We disentangle content from style. This allows for dedicated optimization of clinical content (content extraction), and style generation (via LLM few-shot adaptations). We find that training our model on dense clinical content allows us to improve in clinical accuracy (as opposed to “de-priotizing it”). Moreover, we also evaluate content and style in different settings to scrutinize individual steps our pipeline (e.g., image-to-serialization, serialization-to-report, end-to-end report generation).\nWe will update our introduction to make it more clear to the reader that our paper aims at disentangling content from style in order to optimize *both aspects* separately, which we find leads to improved performance.\n\n> **If the aim was to enhance the scores of widely-used metrics by considering style, the performance difference between the proposed method and the baseline was marginal or even favored the baseline in the results of the Table 2 experiment. Specifically, it was not surprising to see that the scores for RadCliQ and RadGraph were better, given that the method used RadGraph in the training phase. Also, a slight difference was evident in the chexbert result.**\n\nThank you for raising this point; we would like to clarify. First, the RadGraph (Jain et. al 2021) is extracted directly from the report tokens (that is, graph nodes constitute a subset of tokens in the report, and graph edges are based solely on syntactic and morphological relations in the report text), which are equally visible to the baseline model during training—its supervision target being the entire, fully-fledged report. Thus, our method does not have access to additional medical information that would privilege it in measures of clinical accuracy. Rather, the information is organized in a more structured and accessible way. \n\nTable 2 shows that our method outperforms the (Nguyen et al. 2021) baseline across all clinical accuracy metrics: RadCliQ, an aggregate metric that by design is closely aligned with the preference of radiologists (Yu et. al 2023), RadGraph F1, and CheXbert score. The remaining, more classical NLG metrics are substantially less reliable in the clinical setting (for example, changing a single word may change the meaning and implications of a medical paragraph), which is why non-superiority on these additional measures is not clinically relevant. Indeed, (Yu et. al 2023) finds that metrics such as BLEU poorly approximate radiologists’s evaluation of reports, in comparison to clinical accuracy metrics. \n\nFurthermore, in our response to the next point below, we show further comparisons to prior works with convincing performance gains. We will update our result section to better explain the relevance of the clinical metrics (in contrast to metrics like BLEU), and the further comparisons to prior work. \n\n> **Experimental results lack comparisons with other studies. (It's necessary to verify statistically significant results by conducting repeated experiments on the same data set.)**\n\nWe concur that such comparisons are necessary; our paper does include both quantitative and qualitative comparisons to other studies:\n- In our quantitative evaluations (Table 2), we compare our approach to a strong baseline model adapted from (Nguyen et al. 2021). We show that our approach outperforms this prior work in clinical accuracy metrics like RadGraph F1 and CheXbert similarity while achieving competitive performance in natural language metrics (NLG) like BLEU. Furthermore, in accordance with (Yu et al. 2023), NLG metrics such as BLEU have been found to poorly correlate with radiologists’ evaluations of reports compared with clinical accuracy metrics.\n- We compare our approach to further prior works by leveraging directly comparable results from a recently published paper (Yu et al. 2023) (only the 0-shot version of our approach is directly comparable).\n\n|  Method | RadCliq ($\\downarrow$) | RadGraph F1 ($\\uparrow$) | CheXbert ($\\uparrow$) | BLEU ($\\uparrow$) | BERT Score ($\\uparrow$) | \n|-----------------|--------|-----------|----------|--------------|--------|\n| R2Gen |  4.050   |  0.134 | 0.286 | 0.137 | 0.271 | \n| WCL |  3.985 |  0.143 | 0.309 | 0.144 | 0.275 | \n| CvT2DistilGPT2 | 3.903  |  0.154 | 0.335 | 0.143 | 0.280 | \n| **Ours (0-shot)** | **3.527** | **0.228** | **0.394** | **0.162** | **0.333** |\n\nAll methods are evaluated on the test split of the MIMIC-CXR dataset, with identical preprocessing steps to extract findings and impressions for joint generation. **We observe that our method considerably outperforms the contemporary baselines above, across all metrics including those of clinical efficacy and lexical overlap.** We will extend our paper to include these salient further comparisons.\n- In the human evaluation study (Section 4.4), we have radiologists write reports in their own style, and then compare these to AI-generated reports that are adapted to the radiologist’s style (via in-context learning over serialization-report pairs). Showing that human experts cannot reliably distinguish between real radiologist reports and AI-generated ones (that mimic their style) demonstrates the quality of our approach to tailor radiology report generation to individual radiologists.\n\nTo summarize, we believe the current manuscript provides adequate qualitative and quantitative comparisons to prior studies. We find our method outperforms a suite of recent works (elucidated in the table above), as well as the strong (Nguyen et. al 2021) baseline in terms of clinical accuracy, which–we agree–has evaluation primacy.\n\n> **While this study incorporates a few ideas and uses RadGraph and ChatGPT, it does not offer considerable novelty in its methods or contribute significantly to the field.**\n\nThe core novelty of our work lies in the strategy of disentangling content and style through a two-step generation process. Additionally, we make several other novel contributions around supervision, evaluation, and the applicability of large language models. We believe these collectively constitute meaningful advances to the field.\n\nIn particular:  \n1. We propose a new two-step strategy for radiology report generation that disentangles content generation from style injection. Treating these as separate steps is novel.  \n2. For content generation, we supervise on serialized RadGraphs rather than full reports. Using dense structured representations for supervision enables a focus on clinical accuracy.  \n3. We demonstrate that serializing RadGraphs is an effective bridge between vision and language modalities for content extraction.  \n4. Our approach allows flexible conditioning of style based on only a few examples, by leveraging large language models. Style adaptation is not well explored in previous work.  \n5. To our knowledge, we are the first to evaluate style transfer for radiology reports by having clinical experts attempt to detect AI-generated reports. This provides a rigorous human evaluation of style quality.\n\n**In light of these clarifications and the additional context we have provided around the novelty and value of our approach, we kindly request that you reconsider the scores assigned to our paper. We believe the revisions detailed above address the limitations you noted, and help establish the unique contributions and clinical relevance of our work. Specifically, we feel the novelty of our two-step disentangled approach to separate content extraction from style generation, the use of serialized RadGraphs for supervision, and the rigorous human evaluation study all represent meaningful advances.**\n\n**We hope you agree that the additional details provided in this response help to position our work as making valuable contributions to the field of natural language processing for radiology report generation. We would be grateful if you could raise the scores accordingly to better reflect the merits of our approach after accounting for these clarifications. We appreciate you taking the time to re-evaluate our work, and look forward to receiving your feedback.**\n\n\n*References:*\n\n- Feiyang et. al 2023. Evaluating progress in automatic chest X-ray radiology report generation. In Cell Patterns, 2023\n- Jain et. al 2021. Radgraph: Extracting clinical entities and relations from radiology reports. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual. \n- Nguyen et. al 2021. Automated Generation of Accurate & Fluent Medical X-ray Reports. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3552–3569, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors attempted to address the issue of clinical report generation from a new perspective in their methodology, and they achieved results indicating that ChatGPT can reference style well enough to generate reports that are indistinguishable from those written by humans.", "reasons_to_reject": "1. This study doesn't persuasively explain \"why style should be considered in the medical domain\" as proposed by this study, and it doesn't sufficiently explore the clinical importance of such a consideration.\n\n2. Experimental results lack comparisons with other studies. (It's necessary to verify statistically significant results by conducting repeated experiments on the same data set.)\n\n3. While this study incorporates a few ideas and uses RadGraph and ChatGPT, it does not offer considerable novelty in its methods or contribute significantly to the field.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This study sought to utilize a novel approach in report generation tasks, generating short sentences from a RadGraph-based entity-attribute format instead of generating free-format raw text from images. The generated content was subsequently fed into the ChatGPT model, along with a few example reports, to rewrite the clinical report in a congruent style. This method of separating content and style is simple yet interesting.\n\nHowever, the study lacks a compelling argument as to why considering style is \"clinically important\" in generating clinical reports. The more desirable approach and ultimate goal in creating clinical reports should be to focus on generating \"content\" from images as accurately as possible, minimizing hallucinations, and ensuring the report's factual accuracy. Therefore, the aim of research employing artificial intelligence in the medical field should not be to create slightly more natural-style reports using ChatGPT. The authors appear to have de-prioritized what is fundamentally important in the medical field, focusing on style, which makes their argument for the importance of this research less persuasive.\n\nIf the aim was to enhance the scores of widely-used metrics by considering style, the performance difference between the proposed method and the baseline was marginal or even favored the baseline in the results of the Table 2 experiment. Specifically, it was not surprising to see that the scores for RadCliQ and RadGraph were better, given that the method used RadGraph in the training phase. Also, a slight difference was evident in the chexbert result. The absence of comparison with other studies further complicates the understanding of this study's significance.\n\nThe authors attempted to address the problem of generating clinical reports from a new perspective. However, the study lacks an explanation of the clinical significance and implications of this research, and the results are not particularly surprising.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rfxegx1Ned", "reviewer": "Reviewer_SCYV", "summary": "", "questions": "I strongly recommend the authors to address the following concerns.\n1. The work argues that existing methods, which generate reports directly from images, may lead to clinically inaccurate reports due to conflating content and style. Could you give some examples or evidence to prove it?\n2. Why is it more accurate to write a report that matches a physician's style? What is the advantages of usefulness of matching a physician's style?\n3. How to serialize the RadGraph into a condensed summary of the clinical content of the report?\n4. How to ensure/define the order of entities in the generated Serialized RadGraph?\n5. For example, in Figure 2, how to obtain the \"acute cadiopulmonary process\"? How to obtain the \"no\"?\n6. How to obtain the Serialized RadGraph from the Graph?\n7. What is the purpose of introducing a text encoder? If I understand correctly that the input is just the image, what and where is the input text?\n8. What is the full input of GPT-3.5 under both few-shot and zero-shot settings? Could you give an example to illustrate the full input and full output of GPT-3.5?\n9. How many examples are used to prompt GPT-3.5?\n10. Could you give more details about the Image to Serialization task? What are the input and output of this task? What is the meaning of comparing the content of the generated serialization with the ground-truth report? Why is only the F1 score calculated? What are the results of other metrics, e.g., BELU? Therefore, it is important to clarify what are the input and output of this task.\n11. In Table 2, what is the baseline model? Pure Transformer or R2Gen? How many Transformer layers and the number of parameters are used in the baseline? \n12. Besides, while the paper reports beneficial performance, a detailed comparison with more existing image-to-report methods is missing. Including such a comparison would provide a clearer understanding of the proposed approach's advantages.\n13. When evaluating the AI-generated reports, are the physicians given the test set CXR images?\n13. The criteria used in human study is subjective. Authors are encouraged to introduce more information about three clinicians, i.e. working experience and expert title. Senior and junior clinicians may find differences when considering whether the same report is AI-generated or not.\n14. I strongly recommend the author to further evaluate the quality of the generated reports, instead of asking the physician to judge whether the report is AI-generated or not.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their tremendously insightful and meticulous feedback.\n\n> **(1) The work argues that existing methods, which generate reports directly from images, may lead to clinically inaccurate reports due to conflating content and style. Could you give some examples or evidence to prove it?**  \n\n(Nguyen et. al 2021) can be seen as an ablation of our content extraction step, whereas style and content are kept blended together (as opposed to disentangled as in our approach). Given that Nguyen et al.’s approach is less clinically accurate, i.e., in terms of identifying clinical findings (see Table 1 and 2), this provides direct evidence supporting our hypothesis that it is valuable to model content and style separately. \nFor example, here is an excerpt of a findings section of a report as compared to its  RadGraph serialization:  \n\n*Report*:  \n$\\text{The lungs are clear. Incidentally noted are bilateral cervical ribs.}$  \n\n*RadGraph Serialization*:  \n$\\text{lungs clear} $ \\n $ \\text{bilateral cervical ribs}$.\n\nUsing the RadGraph Serialization can be seen as focusing the loss on the subset of tokens that represent the clinically relevant content, whereas a typical loss in prior works would treat each report token as equally important. \n\n*RadGraph Serialization as a dense subset of the report*:     \n$\\text{The \\textcolor{green}{lungs} are \\textcolor{green}{clear}. Incidentally noted are \\textcolor{green}{bilateral cervical ribs}.}$  \n\nIn our revised draft, we will more clearly elaborate on our findings that conflating content and style leads to inferior performance.\n\n\n> **(2) Why is it more accurate to write a report that matches a physician's style? What is the advantages of usefulness of matching a physician's style?**    \n\nThank you for raising this point. We believe there are compelling reasons why explicitly modeling style could benefit radiology workflows:\n\n1. **Disentangling style generation from content generation**: This allows us to train an initial content extraction model that is dedicated to extracting the relevant content – as opposed to being trained to jointly predict a blend of clinical content and the report style (e.g. the wording, formatting, structure etc.). Therefore, we believe that disentangling style from content allows us to focus *more* on content and clinical accuracy – not less.   \n\n2. **Intrinsic value of stylizing reports**: We believe there are a multitude of reasons why it is interesting for a model to stylize radiology reports:  \n\n- *Flexibility*: Stylizing can be used to generate reports that are targeted to the readership (e.g. an orthopedic surgeon who may be interested in different aspects of a Chest X-ray than a pulmonologist). Furthermore, flexibly stylizing reports can also enable reports with less jargon that are more accessible to patients.  \n\n- *Workflow*: for human-in-the-loop radiology report generation, automatically drafting a report that already closely aligns with the specific radiologist's preferred style, makes it easier / smoother for the radiologists to edit and approve the report, thereby potentially saving the radiologists valuable time, and reducing administrative burden. In our human evaluation study (Section 4.4), radiologists could not reliably distinguish between real reports and AI-generated ones in the same style - suggesting our model can closely mimic radiologist preferences. This could improve radiologist productivity and satisfaction.  \n\n- *Local adaptability*: Adapting reports to the expectations of referring physicians can optimize communication of clinical findings. For instance, highlighting findings most relevant to a specialist's scope of practice (e.g., follow-up of a patch of pneumonia or correct location of\npacing lead in the heart). In our model, this is achieved via few-shot in-context learning. Clear communication supports appropriate follow-up care.   \n\n- *Consistency*: Some referring physicians may be accustomed to a particular radiologist's style of reporting. Maintaining this style in generated reports can ensure clear communication between the radiologist and the referring physician.  \n\n- *Emphasis on preferred information*: Different radiologists prioritize different information (e.g. due to specialization). Tailored reports can highlight or emphasize the information that an individual radiologist finds most crucial, and can mimic the radiologist in the way this information is interpreted or highlighted.  \n\nIn our revised draft, we will more clearly outline the advantages of our model being able to explicitly stylize a radiology report, and to that end will extend our discussion.\n\n> **(3) How to serialize the RadGraph into a condensed summary of the clinical content of the report?**   \n\nThis is accounted for in Section 3.3 and Figure 2; in summary, we first partition the RadGraph into weakly connected components, corresponding to subgraphs of radiologically related and linked entities. The keywords *no* and *maybe* are prepended to entities labeled absent and uncertain in the graph, respectively. In each subgraph, a text span is formed from enumerating the entity tokens. The text spans are stratified into *Findings* and *Impressions* sections based on their referenced location in the radiology report, and concatenated into a singular, content-dense, heavily structured text. \n\n> **(4) How to ensure/define the order of entities in the generated Serialized RadGraph?**  \n\nThe order of entities is learned implicitly by training the image-to-serialization transformer on 213,501 reports (Line 870), analogous to how LLMs learn the proper syntax of words merely by means of next-token-prediction. The entity orderings in the training supervision (i.e. ground-truth serialization) are defined by the order of corresponding entity tokens in the report, as explained in Lines 288-290. At inference time, the order in the generated serialization is not regulated directly, and is taken as-is from the model’s prediction–to be later paraphrased by the LLM into a fully-fledged report.  \n\nFor future work, it will be an exciting route to investigate auxiliary loss terms to explicitly penalize the correct order of the sequence of labels (for the report generation task). Beyond the ordering of findings in available report datasets, there is no objective, canonical ordering of findings, especially when we consider graph representations thereof that possess no intrinsic ordering of the nodes and subgraph components.\n\nWe will include these considerations about the Serialized RadGraph ordering in Section 3.3.\n\n> **(5) For example, in Figure 2, how to obtain the \"acute cadiopulmonary process\"? How to obtain the \"no\"?**   \n\nThis is achieved by the RadGraph content extractor model (Jain et. al 2021); entity extraction from the report text is heterogeneous in the sense that multiple types of entities are considered–for instance, present, ambivalent, or absent observational entities. In this case, the “no” is obtained for the model’s extraction of an absent entity. The syntagma “acute cardiopulmonary process” is obtained from the precise report tokens that are extracted as RadGraph entities. \n\n> **(6) How to obtain the Serialized RadGraph from the Graph?**    \n\nThe process of serializing the RadGraph into a text representation is detailed in Section 3.3 and Figure 2 of our manuscript, as well as expounded upon in our response to **Question (3)** above.\n\n> **(7) What is the purpose of introducing a text encoder? If I understand correctly that the input is just the image, what and where is the input text?**  \n\nThe text encoder takes in a doctor's indication, which is usually a one-sentence short description of the clinical background information of the subject of the chest x-ray study. One example indication is “x-year-old male with history of recently diagnosed pneumonia presenting with worsening fever and productive cough”. We believe that combining radiology images with the doctor's indications replicates the routine tasks of a radiologist effectively.   \n\nWe performed an ablation study on both the baseline model and our proposed model without the text encoder (i.e., the model input is just image). The experiments are in the Image-to-Serialization setting. Compared with the performance of models with the text encoder, we can see the benefits of doctor’s indication and the text encoder. Notably, while the clinical indication provides valuable information, we observe that leaving it away makes the relative performance gain of our approach (compared to modeling the full report directly) even more pronounced:\n\n|  Supervised Target  | Text Encoder (Clinical Indication) |   RadGraph F1  |\n|:-------------------:|:------------:|:--------------:|\n| \tFull Report \t|  \tNo  \t| $0.128 \\pm 0.004$ |\n| Serialized RadGraph |  \tNo  \t| $0.171 \\pm 0.004$ |\n| \tFull Report \t|  \tYes \t| $0.186 \\pm 0.005$ |\n| Serialized RadGraph |  \tYes \t| $0.221 \\pm 0.004$ |\n\n> **(8) What is the full input of GPT-3.5 under both few-shot and zero-shot settings? Could you give an example to illustrate the full input and full output of GPT-3.5?**  \n\nThe precise prompt structure for GPT-3.5 input is detailed in Section 3.5 and Appendix A.4. We provide an example to illustrate the full GPT-3.5 input and output in both a zero-shot and few-shot setting. Note that GPT-3.5 is a dialogue-based LLM, and prompts are structured conversationally between a system, user, and assistant.\n\n---\n**Zero-Shot Input / Prompt**\n- System: You are a helpful assistant that generates chest x-ray reports from key words. \n- User: Generate a chest x-ray report from the following key words:  *'FINDINGS: lungs hyperinflated. no focal consolidation. no pleural effusion. no pneumothorax. Cardiac mediastinal silhouettes unremarkable. Hilar contours within normal limits. IMPRESSION: no acute cardiopulmonary process. no focal pneumonia. '*\n\n**Zero-Shot Output:** *'The chest x-ray findings reveal hyperinflated lungs without any focal consolidation, pleural effusion or pneumothorax. The cardiac mediastinal silhouettes are unremarkable and the hilar contours are within normal limits. Based on these findings, there is no acute cardiopulmonary process or focal pneumonia observed.'*\n\n---\n\nHere, the LLM is provided with one paired input & output example of a serialization (supplied by user) and ground truth report (replied by assistant).  A $K$-shot prompt would be provided with $K$ such pairs preceding the evaluation serialization (i.e. generated from the chest X-ray at inference time).\n\n**One-Shot Input / Prompt** \n- System: You are a helpful assistant that generates chest x-ray reports from key words. \n- User: Generate a chest x-ray report from the following key words: *'FINDINGS \\nleft IJ catheter upper portion SVC \\nno pneumothorax \\nlittle change \\n'*\n- Assistant: *'In comparison with the earlier study of this date, there has been placement of a left IJ catheter that extends to the upper portion of the SVC.  No evidence of pneumothorax.  Otherwise, little change. '*\n- User: Generate a chest x-ray report from the following key words: *'FINDINGS: pacemaker leads right atrium right ventricle. position. heart size mildly enlarged. hilar mediastinal contours unremarkable. no focal consolidation. no pleural effusion. no pneumothorax. no pulmonary edema. osseous structures unremarkable. IMPRESSION: Satisfactory position pacemaker leads right atrium right ventricle. no pneumothorax. '*\n\n**One-Shot Output:** *'The chest x-ray reveals pacemaker leads in the right atrium and right ventricle, which are in a satisfactory position. The heart size is mildly enlarged and the hilar mediastinal contours are unremarkable. There is no focal consolidation, pleural effusion, or pneumothorax. Additionally, there is no pulmonary edema and the osseous structures are unremarkable. Overall, the impression is that the pacemaker leads are in a satisfactory position and there is no evidence of pneumothorax.'*\n\n---\n\n> **(9) How many examples are used to prompt GPT-3.5?**  \n\nOur study triages between 0,1,5, and 10 examples to prompt GPT-3.5, as described in Section 4.2. The results for each are provided in Tables 2 and 3 of the manuscript. We will put more emphasis on this detail in our revision by more prominently listing each number of in-context examples.\n\n> **(10) Could you give more details about the Image to Serialization task? What are the input and output of this task? What is the meaning of comparing the content of the generated serialization with the ground-truth report? Why is only the F1 score calculated? What are the results of other metrics, e.g., BELU? Therefore, it is important to clarify what are the input and output of this task.**  \n\nThe input is the radiology images associated with each study, and corresponding doctor’s indication (i.e., text), and the output is structured text that represents the content of the study, where we use serialized RadGraph (see Section 3.2 and 3.3 for how we obtain it) as the signal for supervision.\n\nWhen comparing the *content* of the generated serialization with the ground-truth report, we are looking mainly for *agreement in radiological entities and relations*—for example, the presence of pleural effusion in the lung areas–as a litmus of clinical accuracy. This is precisely what RadGraph F1 calculates; ergo it takes primacy in evaluating *content*. By contrast, NLG metrics such as BLEU and BERT score take into account non-medical and vernacular tokens, making them less ideal for gauging the accuracy of clinical content.\n\nWe provide results for the other metrics below (predicted / generated serialization vs. ground-truth report on the *top row*). \n\n| Predicted           | Evaluation Target | RadCliq ($\\downarrow$) | Rad F1 ($\\uparrow$) | Chexbert ($\\uparrow$) | BLEU ($\\uparrow$) | BERT ($\\uparrow$) |\n|---------------------|-------------------|-------------------|------------------|--------------------|----------------|----------------|\n| Pred Serialization  | GT Report         |   3.847 $\\pm$ 0.028    |         0.221 $\\pm$ 0.004           |    0.390 $\\pm$ 0.008              |       0.119 $\\pm$ 0.003         |          0.223 $\\pm$ 0.005      |\n| Pred Report         | GT Report         | 3.530 $\\pm$ 0.027 | 0.228 $\\pm$ 0.004   | 0.395 $\\pm$ 0.008            | 0.162 $\\pm$ 0.003     | 0.332 $\\pm$ 0.004     |\n\nWe observe that the predicted report–the generated serialization paraphrased by an LLM–is superior in lexical metrics (BLEU, BERT), but possesses similar clinical accuracy scores (RadGraph F1, Chexbert) to the generated serialization. This is an critical observation as it prefigures the LLM’s role as rephrasing the content already present in the generated serialization, rather than creating content anew (to mitigate hallucinations).\n\n\n> **(11) In Table 2, what is the baseline model? Pure Transformer or R2Gen? How many Transformer layers and the number of parameters are used in the baseline?**  \n\nThe strong baseline is adapted from (Nguyen et. al 2021), with a decoder of 12 transformer blocks and an embedding dimension of 256, determined by hyperparameter sweep (detailed in Appendix A.1), and a 121-layer DenseNet vision backbone with 7.8 million parameters. \n\nThe baseline possesses the identical architecture and parameters as our content generation model, as detailed in Section 3.4. The baseline is directly trained to predict the full report, while the novelty of our method is comprised of two steps: first generate clinical content in serialized RadGraph format, and then use an LLM to generate the full report based on outputs from the first step.\n\n> **(12) Besides, while the paper reports beneficial performance, a detailed comparison with more existing image-to-report methods is missing. Including such a comparison would provide a clearer understanding of the proposed approach's advantages.**  \n\nThe manuscript already compares the proposed approach to a strong prior method by Nguyen et al. 2021 (line 456), which was reported to outperform a large battery of over 15 prior methods; see Table 1 in (Nguyen et al. 2021). In addition, we report further results from a recently published paper (Yu et al. 2023), and compare it to our proposed approach (only the 0-shot version of our approach is directly comparable):\n\n|  Method | RadCliq ($\\downarrow$) | RadGraph F1 ($\\uparrow$) | CheXbert ($\\uparrow$) | BLEU ($\\uparrow$) | BERT Score ($\\uparrow$) | \n|-----------------|--------|-----------|----------|--------------|--------|\n| R2Gen |  4.050   |  0.134 | 0.286 | 0.137 | 0.271 | \n| WCL |  3.985 |  0.143 | 0.309 | 0.144 | 0.275 | \n| CvT2DistilGPT2 | 3.903  |  0.154 | 0.335 | 0.143 | 0.280 | \n| **Ours (0-shot)** | **3.527** | **0.228** | **0.394** | **0.162** | **0.333** |\n\nAll methods are evaluated on the test split of the MIMIC-CXR dataset, with identical preprocessing steps to extract findings and impressions. **We observe that our method considerably outperforms the contemporary baselines above, across all metrics including those of clinical efficacy and lexical overlap.** We will extend our paper to include these further comparisons.\n\n> **(13) When evaluating the AI-generated reports, are the physicians given the test set CXR images?**\n\nThank you for raising this point. The “Human Style Evaluation” experiment was designed to assess whether our approach allows for the generation of radiology reports that are indistinguishably tailored to a specific radiologist’s style (e.g., $R_1$). By contrast, for assessing the clinical accuracy of the reports (i.e., whether a model extracts the correct findings from an image), we leverage established metrics such as RadGraph F1. \n\nAs the reports from radiologist $R_1$ were about different patients, we believe that it is more meaningful for the human raters to focus on the reports (and their style) as there is nothing to compare by looking at the non-corresponding images. At the time of designing the experiment, we had been concerned that, if we had shown the images, the physicians would use data points from the image and not from the reports per se to make their decision.\n\nIn summary, the goal of our human style evaluation was to assess the report style in isolation (without judging the report content at the same time), hence we let clinical raters review different reports by the same radiologist (plus an AI-generated one in the same style). \n\n\n> **(14) The criteria used in human study is subjective. Authors are encouraged to introduce more information about three clinicians, i.e. working experience and expert title. Senior and junior clinicians may find differences when considering whether the same report is AI-generated or not.**\n\nThank you for raising this point. In our revised manuscript we will include further details about all the involved medical experts including years of experience and specialty.\n\n> **(15) I strongly recommend the author to further evaluate the quality of the generated reports, instead of asking the physician to judge whether the report is AI-generated or not.**  \n\nThis is an excellent recommendation. As illustrated in Figure 6 and Appendix A.3 of our manuscript, physicians were also asked to attribute their decisions either (1) content (e.g. missing important details or including extraneous details), (2) language (e.g. word choice, grammar, and/or writing style), or (3) structure differences with human-quality reports (e.g. different use of numbering or section headings). In A.3, we also identify and elucidate examples of correctly detected AI-generated reports (i.e. failure modes), for each of the three categories.\n\nIn our revised manuscript, we will further elaborate on these results, and weigh a more comprehensive parcel of human evaluations, such as numerical ratings on the aforementioned categories et plus. \n\n\n\n**We appreciate the reviewer’s suggestions and hope that, in light of our changes to the manuscript and explanations in responses to the reviewer’s suggestions, they would consider raising their score.**\n\n*References:*\n- Feiyang et. al 2023. Evaluating progress in automatic chest X-ray radiology report generation. In Cell Patterns, 2023\n- Jain et. al 2021. Radgraph: Extracting clinical entities and relations from radiology reports. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.\n- Nguyen et. al 2021. Automated Generation of Accurate & Fluent Medical X-ray Reports. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3552–3569, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed approach is sound and the utilization of RadGraph is well-motivated. The idea is interesting.\n\n2. The beneficial performance shown in the evaluations validates the effectiveness of the proposed approach.", "reasons_to_reject": "1. Several claims should be proved.\n- The work argues that existing methods, which generate reports directly from images, may lead to clinically inaccurate reports due to conflating content and style. Could you give some examples or evidence to prove it?\n- Why is it more accurate to write a report that matches a physician's style? What is the advantages of usefulness of matching a physician's style?\n \n2. More model details should be provided and clarified.\n- How to serialize the RadGraph into a condensed summary of the clinical content of the report?\n- How to ensure/define the order of entities in the generated Serialized RadGraph?\n- For example, in Figure 2, how to obtain the \"acute cadiopulmonary process\"? How to obtain the \"no\"?\n- How to obtain the Serialized RadGraph from the Graph?\n- What is the purpose of introducing a text encoder? If I understand correctly that the input is just the image, what and where is the input text?\n- What is the full input of GPT-3.5 under both few-shot and zero-shot settings? Could you give an example to illustrate the full input and full output of GPT-3.5?\n- How many examples are used to prompt GPT-3.5?\n \n3. More experimental details should be provided and clarified.\n- Could you give more details about the Image to Serialization task? What are the input and output of this task? What is the meaning of comparing the content of the generated serialization with the ground-truth report? Why is only the F1 score calculated? What are the results of other metrics, e.g., BELU? Therefore, it is important to clarify what are the input and output of this task.\n- In Table 2, what is the baseline model? Pure Transformer or R2Gen? How many Transformer layers and the number of parameters are used in the baseline? \n- Besides, while the paper reports beneficial performance, a detailed comparison with more existing image-to-report methods is missing. Including such a comparison would provide a clearer understanding of the proposed approach's advantages.\n- When evaluating the AI-generated reports, are the physicians given the test set CXR images?\n- The criteria used in human study is subjective. Authors are encouraged to introduce more information about three clinicians, i.e. working experience and expert title. Senior and junior clinicians may find differences when considering whether the same report is AI-generated or not.\n- I strongly recommend the author to further evaluate the quality of the generated reports, instead of asking the physician to judge whether the report is AI-generated or not.\n\n ", "questions_to_authors": "I strongly recommend the authors to address the following concerns.\n1. The work argues that existing methods, which generate reports directly from images, may lead to clinically inaccurate reports due to conflating content and style. Could you give some examples or evidence to prove it?\n2. Why is it more accurate to write a report that matches a physician's style? What is the advantages of usefulness of matching a physician's style?\n3. How to serialize the RadGraph into a condensed summary of the clinical content of the report?\n4. How to ensure/define the order of entities in the generated Serialized RadGraph?\n5. For example, in Figure 2, how to obtain the \"acute cadiopulmonary process\"? How to obtain the \"no\"?\n6. How to obtain the Serialized RadGraph from the Graph?\n7. What is the purpose of introducing a text encoder? If I understand correctly that the input is just the image, what and where is the input text?\n8. What is the full input of GPT-3.5 under both few-shot and zero-shot settings? Could you give an example to illustrate the full input and full output of GPT-3.5?\n9. How many examples are used to prompt GPT-3.5?\n10. Could you give more details about the Image to Serialization task? What are the input and output of this task? What is the meaning of comparing the content of the generated serialization with the ground-truth report? Why is only the F1 score calculated? What are the results of other metrics, e.g., BELU? Therefore, it is important to clarify what are the input and output of this task.\n11. In Table 2, what is the baseline model? Pure Transformer or R2Gen? How many Transformer layers and the number of parameters are used in the baseline? \n12. Besides, while the paper reports beneficial performance, a detailed comparison with more existing image-to-report methods is missing. Including such a comparison would provide a clearer understanding of the proposed approach's advantages.\n13. When evaluating the AI-generated reports, are the physicians given the test set CXR images?\n13. The criteria used in human study is subjective. Authors are encouraged to introduce more information about three clinicians, i.e. working experience and expert title. Senior and junior clinicians may find differences when considering whether the same report is AI-generated or not.\n14. I strongly recommend the author to further evaluate the quality of the generated reports, instead of asking the physician to judge whether the report is AI-generated or not.", "paper_topic_and_main_contributions": "The paper proposes a two-step approach for generating radiology reports that match a physician's style from medical images. The authors first extract content from images and then verbalize it into a report matching the specific radiologist's style. They leverage RadGraph, a graph representation of reports, along with large language models (LLMs). The paper reports positive quantitative evaluations and highlights that the AI-generated reports are tailored to individual radiologist styles.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a two-step approach to automatically generating radiology reports, where the first stage focuses on extracting content from the image and the second stage verbalizing the content to a specific style. Although the reviewers agreed that the proposed approach is straightforward and convincing, a strong concern remained regarding the fact that the experiment results did not substantiate the core claim of disentangling the content and style of radiology reports, and the potential risk of missing important content while verbalizing it to different styles.
Decision: Accept-Findings