Image-aware Evaluation of Generated Medical Reports
Authors: Gefen Dawidowicz
Elad Hirsch
Ayellet Tal
Venue: NeurIPS 2024 poster (2024)
Keywords: medical report generation
evaluation metric
vision language
Primary area: machine_learning_for_healthcare

Abstract: The paper proposes a novel evaluation metric for automatic medical report generation from X-ray images, VLScore. It aims to overcome the limitations of existing evaluation methods, which either focus solely on textual similarities, ignoring clinical aspects, or concentrate only on a single clinical aspect, the pathology, neglecting all other factors. The key idea of our metric is to measure the similarity between radiology reports while considering the corresponding image. We demonstrate the benefit of our metric through evaluation on a dataset where radiologists marked errors in pairs of reports, showing notable alignment with radiologists' judgments. In addition, we provide a new dataset for evaluating metrics. This dataset includes well-designed perturbations that distinguish between significant modifications (e.g., removal of a diagnosis) and insignificant ones. It highlights the weaknesses in current evaluation metrics and provides a clear framework for analysis.

Image-aware Evaluation of Generated Medical Reports
Gefen Dawidowiez" Elad Hirsch Ayellet Tal
Technion Israel Institute of Technology
Abstract
The paper proposes novel evaluation metric for automatic medical report genera- tion from X-ray images, V LScore. It aims to overcome the limitations of existing evaluation methods, which either focus solely on textual similarities, ignoring clini- cal aspects, or concentrate only on single clinical aspect, the pathology, neglecting all other factors. The key idea of our metric is to measure the similarity between radiology reports while considering the corresponding image. We demonstrate the benefit of our metric through evaluation on dataset where radiologists marked errors in pairs of reports, showing notable alignment with radiologists' judgments. In addition, we provide a new dataset for evaluating metrics. This dataset includes well-designed perturbations that distinguish between significant modifications (e.g.. removal of a diagnosis) and insignificant ones. It highlights the weaknesses current evaluation metrics and provides clear framework for analysis.
Introduction
Medical reports are crucial for summarizing diagnoses seen in X-ray images. These reports facilitate communication among healtheare professionals, ensuring continuity of care and informed decision- making. The process of generating these reports requires expertise and time, typically involving trained radiologist who writes them from scratch. Automatic generation of medical reports therefore has the potential to enhance patient care and streamline administrative processes, ultimately improving healthcare delivery. Consequently, numerous research efforts have been directed toward 30]. These medical report generation models aim to produce coherent and accurate reports based on input images, with a current focus on chest radiology reports.
this goal 3 15
17 27
However, evaluating the generated reports is a challenge in its own right (34]. This evaluation is essential to determine whether new technique truly advances the state of the art. Currently, two families of metrics are being used, language metrics and clinical metrics. Derived from language generation tasks, such as neural machine translation or image captioning, generated medical reports are evaluated with natural language generation (NLG) metrics. In our context, the similarity between the generated report (candidate) the ground-truth report (reference) associated with the input image is measured. To evaluate some of the medical entities that are expressed in the report, clinical efficacy (CE) is measured by applying pre-trained classification network on the ground-truth and output reports to assess their agreement pre-defined pathologies drawbacks, which the current paper attempts to solve, as described hereafter.
13 26 These metrics suffer from
The traditional NLG metrics, namely BLEU 25 METEOR M. ROUGE [20]. and CIDEr 28 are --gram-based and measure the token (or sequence of tokens) overlap between the candidate and reference texts. More recent metrics, such as BERTScore (35), rely on deep textual feature similarity, making them more robust to variations in exact tokens and instead utilizing contextual embeddings. However, these metrics fail to capture several critical nuances in the context of medical reports. In particular, while the reports are typically lengthy and structured, following systematic review
These authors contributed equally to this work
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

GT: "The lungs are clear. The cardionsdiastinal silhouctte is within normal limits. No acule osscous abnormalities." ?
GT: "right hilar and perihilar opacification appears unchanged and suggests site trented malignancy...
2010
Candidate: "Frontal and
Candidate: "left hilar and peribilar opucification appears unchanged and suggests a site of treated malignancy... ..."
lateral views of the chest were obtained There is no pleural effusion or pneumothorax. The heart size normal. Bony structures are intact."
RadGinph 0.022
RedCh
BERTSSore 0.935
0.99
Similarity
Metric
BERTScore 0:39
CheXberi
CheXben 0.973
00
0.501
0.892
0.475
0.997
(a) Same findings
(b) Different location
Figure 1: Vision-language based evaluation of generated reports. Our proposed evaluation metric for reference report (GT) and candidate report takes into account the associated image, thereby overcoming several drawbacks of common metrics that only measure the distance between two texts. Consequently, when comparing two semantically very similar reports that suit an image (a), our metric suggests a high degree of similarity, whereas other metrics are unaware of these similarities and therefore give low scores due to textual differences. In contrast, when comparing two reports that are textually similar but differ by a single word that impacts the location of a finding (b), common metries still provide a very high score, while our metric penalizes for that error.
method, they contain mostly normal findings and general references. The crucial findings may occupy a small portion of the report, sometimes comprising only one or two words (e.g.. "large pleural effusion" vs. "no pleural effusions"). As these findings may be brief, they may not significantly impact the scores, despite their important information. Furthermore, radiologists frequently include in the report information that cannot be extracted from an image, such as the patient's 's history, findings from other sources, or details about treatment. These metrics penalize for such missing information.
CE metrics are oblivious these issues and focus solely on assessing the presence or absence of certain pathologies in the reports. However, due to the difficulty of comprehensively labeling the data, the available pathology classes are limited both in diversity and in detail. Currently, the number of pathologies used for evaluation is only 14, neglecting possible findings beyond these pathologies. Furthermore, only the existence or absence of a pathology is taken into account, while specifics such as locations (e.g.. "left" vs. "right") and severity (e.g.. "small" vs. "mild") are completely disregarded. These limitations constrain the metrics' ability capture overall clinical correctness.
Finally, an additional major drawback in the current situation in this domain is that results are evaluated separately these two types of metrics. This raises evaluation ambiguity: what does it signify if one model demonstrates improved scores in one type of metric but worse scores in the other? For example, the result of [18] is better than (1I in terms of CE, but worse in terms of NLG metrics on MIMIC- CXR dataset [6] Which generative model shall one use?
Indeed, it was demonstrated in 34 that these metrics exhibit low correlation with radiologist judgment. this study, pairs of reference and candidate reports were evaluated by several NLG and CE metrics and compared to radiologist evaluations. To improve the correlation, it was proposed to utilize a graph representation of clinical entities and relations [14), and to use it to compute an additional metric, by measuring the similarity in both the entities and their relationships between reference report and a generated report. However, this graph structure is not unique, as evidenced by the variability between manual annotators, which can potentially lead to inaccurate metrics. Additionally, normal findings may or may not appear in a report, and their presence or absence can influence the result, although both scenarios are valid.
We propose a novel metric to measure the correspondence between two reports. The key idea is that the reports are linked through an associated image, and this image should be taken into account. Hence, we assess both the textual similarity between the reports and the extent to which they convey shared clinical information. By utilizing the visual modality, we address several weaknesses inherent

---

in existing approaches discussed above. These include the separation to two distinct types of metrics, the overemphasis of general long phrases compared important concise phrases in NLG metries, the consideration of information that cannot be inferred from the image, and the limited diversity and detail captured by CE metrics. Figure demonsttaaes the benefits of our proposed metric. When two reports are semantically very similar but significantly different in terms of text (Figure Ip), common metrics assign very low scores, whereas our metric manages to capture the high similarity between the reports. In contrast, when two reports are highly similar textually except for single word that holds significant anatomical importance (e.g., left vs. right), as in Figure Dp, common metrics provide high score, while our metric is more sensitive to this critical change.
Specifically, our approach is based on the idea that measuring similarity between reports in a joint disual-textudl space expresses aspects of similarity and distinctiveness, which cannot be achieved in a single modality. This can be explained by the fact that reports conveying equivalent relevant information, which can be extracted from given image, are expected to be positioned closely together within this space. Furthermore, the learned space ensures that a broad spectrum of pathologies and observations are accommodated, rather than being limited to pre-defined ones. Hence, we compute the proximity of 3 components in this space - the image, the ground-truth report and the generated report. We note that joint embedding space of images and reports has garnered interest in recent years within the field of representation learning in the medical domain, facilitating tasks such as retrieval and classification 22 6 [ 33 37). This is the first time propose that measuring similarities within this space provides a robust framework for evaluating how accurately a generated report describes an image. Additionally, we define new similarity score in this shared space.
Furthermore, in addition to demonstrating a notably strong alignment of our metric with radiologists' judgments on ReXVal [34]. we introduce a new dataset containing tailored perturbations. These are designed to examine the sensitivity of evaluation metrics to significant versus insignificant modifications. Our findings indicate that our metric is sensitive to critical details such pathologies or the location of findings, while remaining robust to errors of low significance, such as the omission of non-clinical phrases.
Hence, our contributions are three-fold:
1. A novel evaluation metric for medical report generation, considering both textual and clinical aspects. Its key idea is to measure report similarity through their corresponding image.
2. Showing that our metric achieves the strongest alignment with radiologists judgment com- pared to previous metrics, well as the highest robustness to the perturbations.
3. A new evaluation dataset containing targeted perturbations designed to reveal weaknesses and benefits evaluation metrics for medical report generation.
2 Related Work
Medical report generation. Medical report generation models aim to produce coherent and accurate medical reports based on input images. Most research in this field focuses on a paired setup, where both images and corresponding reports are available for training. Typically, visual features are encoded using a dedicated encoder module, followed by decoder that generates the output report 3
10
12 15 19
23
30 36). Additional techniques, such as incorporating knowledge
graphs 17 21 36 , planning 10, and reinforcement learning 23 32 are often employed in the
process. Training these models without access to paired data has also garnered attention other
works R 9 22
Evaluation metries for report generation. Evaluation is crucial for assessing the effectiveness of all the above approaches. Traditional metrics used to measure aligmment between two texts (reference and candidate) rely on n-gram overlap and include BLEU (25). METEOR M. ROUGE (0) & CIDEr (28) These metrics consider exact word or phrase alignments, often overlooking paraphrases and failing to account for variations in semantic significance. Specialized metrics such as clinical efficacy (CE) metrics have been developed to measure clinical precision, recall, F1 score, or cosine similarity. These metrics are typically derived from classification models trained to identify specific pathologies 13. 26. While CE metrics assess high-level semantic similarity between reference and generated reports, they may not fully capture the coherence and overall quality of the generated reports. Recently, another metric called RadGraph Fl (34) was proposed, which computes overlap
3

---

between graphs representing clinical entities and relations in both reference and generated reports. In 34 it demonstrated that RadGraph, along with the embedding- based metric BERTScoree [35. exhibits relatively high alignment with radiologists' judgments.
Image-report representation learning. Multi-modal representation learning in the medical domain is usually focused on Vision Language Pre-training (VLP), aiming to align corresponding images and texts in the same representation space. These representations can be utilized for various downstream tasks, such as retrieval, classification and detection. Most of the works in that domain utilize contrastive learning objectives to align the two modalities. Con VIRT 37 proposed a simple yet effective model that utilized a global contrastive objective between the image and the corresponding report. GLoRIA M. LIMITR [6] & LoVT [24] proposed different methods to utilize the local connections between the modalities and incorporated them into an additional contrastive objective. MGCA 299 added prototype module which introduces disease-level information. Biovil [2] focused on the language representations and introduced Nomain-specific language model. MedCLIP (3) proposed to replace the contrastive loss with semantic matching loss based on medical knowledge to eliminate false negatives in contrastive learning.
3 Proposed Metric
The goal of report generation is to automatically create free-text description for a clinical image. The quality of the generated reports should be quantified based on how well the overall information present in the image is expressed in the report. Currently, evaluation metrics measure the similarity between single ground-truth report and the generated report without considering the images, focusing either on language similarity (NLG metries) or clinical efficacy (CE metrics). NLG metrics do not account for the fact that certain sentences or specific words are more crucial than others, although these impact patient diagnosis and treatment more significantly. CE metrics compare pathologies but are limited in the types of pathologies they extract and ignore important pathology properties (c.g., location or severity). Additionally, this approach raises a question: how do choose a model between two given ones when one is superior in terms of NLG and the other in terms of CE?
In the light of the limitations of existing metrics, we seek new metric that quantifies the quality of a generated report, addressing the following criteria: (I) consideration of both textual and clinical aspects, (2) recognition that subtle differences between reports may be significant, (3) giving less weight to information that cannot be extracted from the image (e.g. previous conditions), and (4) capturing multiple possible findings along with their properties such as locations and severity.
Our key idea is that the similarity between radiology reports should consider the corresponding radiology image. We propose to quantify the quality of the generated report with respect to both the information present in the input image and the ground-truth report. To integrate information from two modalities and compute a similarity score, we implement our idea at two steps. First, we project the image and reports into a shared space where similarities and differences are trained to be captured. Second, we propose novel metric that quantifies quality based on the relationships among three components: the image, the generated report, and the ground-truth report, as described below.
Multi-modal embedding model. Medical image-report representation models aim to construet a joint embedding space where corresponding images and reports are positioned close to each other, while con-corresponiing ones are placed farther apart 66 II 37 2 33]. These models learn distinctive features that can be extracted from both the images and the reports. The properties of these embedding models align with desired properties in medical reports evaluation metrics: (1) As the embedding model constructs the embedding space to successfully separate non-matching images and reports, it seeks the differences that matter, even when these are subtle. This is crucial for our purpose, subtle differences between reports can distinguish between excellent report and one that fails to describe the image. Furthermore, the separation is learned rather than being guided by pre-defined criteria (e.g., specific 14 pathologies). Therefore, it enables the quantification of the quality of reports describing a wide range of pathologies and abnormalities. (2) As the model learns to distinguish between examples within the dataset, it also acquires the ability to overlook general information lacking discriminative value. This property is important for an evaluation metric as allows the metric prioritize the extraction of pertinent clinical details from the input image.
Our approach can function with any shared space embedding model. However, we prioritize precise image-report retrieval-oriented models, i.c., finding the exact match from one modality given the

---

other, as this aligns with our goal. Precise retrieval relies on matching fine details, which necessitates shared information, both local and global, within the image and report. Other focuses of embedding models are often on downstream tasks such as classification or segmentation, where the focus on specific pathologies usually suffices. Therefore, we chose to employ the model of [6] which focuses on image-text and text-image retrieval, rather than on other downstream tasks. Its utilization aligns well with the task of evaluating report generation, which entails quantifying the accuracy of diverse descriptions contained within the report. Section demonstrates the model's 's benefit for our needs by showing its superior alignment with radiologists judgment compared to other embedding models.
Report-report similarity evaluation. Given the embedding vectors of an image, a generated report, and a ground-truth report, our aim is to define a similarity score between them. There are various alternatives for such a score, discussed in Section 4.3
We define the similarity as the area of the triangle created by the three high- sigh-dimensional points (i.e., the embedding vectors). Intuitively, this area takes into account the proximity to each other in the embedding space of all pairs of points in this triplet. Formally, for an image i, a generated report g. and a ground-truth report mapped by the embedding model Emò to le Me. respectively, their proximity is computed using the following triangle area formula, utilizing the inner product (;
1 g(i,g,r) - Vie ge, go) (ie Te, ie re) (ic 9es ie Te)
(1)
Since the embedding model is designed to position the image and the two reports that correspond to the same image close to each other in the embedding space, lower values of this measurement indicate a higher level of similarity between the reports and the image.
To normalize the measure to the range of (0, 1], where indicates low similarity and 1 represents high similarity, the final quality score is defined as:
V V'Scorr(i.g.7 = max{1 T(i, g.r. 00
C
where C' is constant. We set it C 890 to suit the maximal value of Equation | (representing the largest triangle area) for the baseline dataset [34], which comprises pairs of diverse quality. Therefore, this choice is compatible across a variety of datasets.
Experimental Analysis
We analyze our proposed metric by comparing its alignment with radiologists' judgment and by using a controlled dataset to demonstrate the domain sensitivity of the various metrics across different error types. Further details on the experiments and findings are provided below.
4.1 Radiologist-juuggemnn data
This experiment aims to assess the agreement between evaluation metrics and the actual ratings of certified radiologists. We expect good evaluation metries to show strong correlation with the assessments of radiologists, who have extensive experience in interpreting medical reports and defining report aceuracy. A low correlation would raise concerns about the validity of the metric, since its scores would diverge from the opinions of radiologists, the experts in this field.
Dataset. The recent work of 34 introduces an evaluation dataset, called Radiology Report Expert Evaluation (ReXVal), comprising 200 pairs of reports. Each pair consists a reference report (drawn from the MIMIC-CRR training set) and candidate report retrieved as the best match from the same dataset, according to one of the following automated metrics: BLEU, BERTScore, CheXbert, and RadGraph F1. Each pair was annotated by 6 board-certified radiologists, who provide the number of errors in the candidate report across several error categories, such as omission of finding or false location of finding. These annotations generate a rating for each pair of reports based on the number of errors identified by the radiologists.
Correlation score. The alignment between the different metric scores (previous metrics & ours) and the radiologist scores assigned to the same reports is quantified using the Kendall rank correlation coefficient (7), as proposed in (34). This coefficient assesses the degree of similarity in the ordering of
5

---

Table I: Evaluation on VeXVal dataset. Compared to previous evaluation metrics, our metric achieves higher correlation (Kendall coefficient 7) with radiologist judgment from the ReX Val dataset. We report the results of BLEU (B), METEOR (M), ROUGE-L (R-L), BERTScore (BS), CheXpert (Cp), CheXbert (Cb), and RadGraph FI (RG).
NLG
CE
OURS VLScore
B-1
B-4
M
R-L
BS
Cp
Cb
RG 0.515
Kendall
0.388
0.345
0.465
0.476
0.511
0.463
0.499
0.718
ranks assigned by the metrics and radiologists. In this context, high correlation coefficient indicates that the ranking produced by a given metric closely resembles the ranking assigned by radiologists. The agreement with radiologists is important indicate the extent to which each metric captures the perceived quality or clinical relevance of the reports as assessed by experts. This means the metric effectively identifies which report-reference pairs considered better worse.
Results. Table shows the correlation between common evaluation metrics and our proposed metric to the judgment of radiologists. Our correlation coefficient (r = 0.718) is much higher than the classical evaluations (e.g., B-4 with T = 0.345) and the recently proposed metric of RadGraph F1 (RG, 0.515). This can be explained by radiologists' focus on the diagnostic significance of textual differences rather than on mere textual overlap. Our approach aims to achieve a similar focus, which contributes to the higher correlation observe.
4.2 Perturbed data
In the medical domain, the single dataset with human judgement is of [34,, which we used in [oction[4.]]eee metric assessment. We propose to add another dateset, a perturbed dataset, which will assess the robustness of the different metrics several properties crucial for quantifying the accuracy of generated medical reports. This dataset of larger scale and contains targeted perturbations that address several weaknesses that may arise in evaluation metrics and are present in generated reports.
Toward this end, we sampled subset of 440 pairs of images and reports from the validation and test sets of LXM, For this subset, we manually applied six types of modifications to the accompanying reports. This resulted in 979 triplets, each includes an image, the ground -ruth report and the perturbed report. The dataset will be released, with samples provided in the supplementals.
In order to quantify the quality of the metrics and assess for which modifications the metrics robust or sensitive, we compare the behavior of each evaluation metric for each modification type. We discuss each pertubation and experiment below.
Removal of pathology sentence vs. insignificant sentence. Different sentences in medical reports carry varying levels of significance. Those that describe the pathologies and diagnosis central to the essence of the report, making their presence crucial in the generated report, while other sentences may be of lower significance. To evaluate the sensitivity of the evaluation metrics to sentences with varying importance, we removed sentences from the ground-truth report, as follows. In the first case, examine the effect of a missing pathology by omitting the sentence that indicates it. Clinically, this difference is dramatic as it represents a major error in image analysis. Consequently, we expect the metries to penalize considerably. In this case, obtained 238 examples.
In the second case, we removed general sentences that are irrelevant to the analysis of the given image. Such sentences could include common phrases that may or may not appear in reports (e.g.. "a chest radiograph") ") or phrases related to the actions taken by the doctor (c.g.. "called the nurse"). We do not expect a model generate these sentences based on a given image. Therefore, aim for the metric to be immune to the presence or absence of these phrases. We obtained 119 such examples.
Table 2 shows the average similarity scores given by each metric for each one of these perturbation types and their difference (A). metric should penalize clinical errors more than general sentences. However, both perturbations influence NLG metrics similarly (the difference in the similarity scores between the two is approximately 0). which shows their weak sensitivity to clinical details. This behavior is expected as these metrics quantify the presence or absence of sequences of words without consideration of context and sentence importance. On the other hand, both CE metrics and our metric aware of the clinical aspects and are affected differently by the two perturbations. For CE metrics,

---

Table 2: Results on perturbed dataset: Removing pathology vs. insignificant sentences. When removing a pathology, the score should drop, while when omitting a general sentence, should change only slightly. NLG metrics result in similar scores in both cases; however, CE and our metric penalize the score accordance with the importance. Our metric shows the highest sensitivity.
NLG
CE
OURS
Sentence Removal
B-1
B-4
M
R-L
BS
Cp
Cb
RG
VI.ccore
Insignificant
0.75
0.74
0.49
0.85
0.89
0.99
0.97
0.93
0.84
Significant (pathology)
0.76
0.74
0.48
0.85
0.88
0.87
0.90
0.82
0.69
-0.01
0.00
0.01
0.00
0.01
0.12
0.07
0.11
0.15
Table 3: Results on perturbed dataset: Modifying a descriptive vs. non-informative word. When altering a clinical descriptor, the score should decrease, while when changing a non-informative word, it should change only slightly. NLG & CE metrics yield similar scores in these cases; Our metric shows high sensitivity. Moreover, all previous metrics set a high score (close to 1) to the perturbed report, suggesting it is near perfect, although critical error was made.
NLG
CE
OURS
Changed Word
B-1
B-4
M
R-I.
BS
Cp
Ch
RG
VL.core
Non-informative
0.98
0.95
0.65
0.98
0.97
0.99
0.99
0.97
0.91
Location
0.98
0.95
0.67
0.98
0.99
1.0
1.0
0.93
0.72
A
0.00
0.00
-0.02
0.00
-0.02
-0.01
-0.01
0.04
0.19
Severity A
0.97
0.94
0.65
0.97
0.99
0.98
0.99
0.92
0.79
0.01
0.01
0.00
0.01
-0.02
0.01
0.00
0.05
0.12
which assess the overlap of clinical entities between two reports, removing the pathology description greatly impact their similarity results. Yet, our metric exhibits the highest sensitivity in this case (the highest A). It is derived from the nature of our metric relying discriminative visual and textual information. This is so since the embedding model utilizes clinical information from both the image and the report, mapping them to close or distant locations in a shared space. General information shared across multiple studies is assigned less weight in this mapping process, as it does not contribute significantly to the differentiation between radiology studies.
Modification of pathology description word vs. non-informative word. In medical reports, single words, often those that describe the pathology, can significantly alter the diagnosis. To assess the sensitivity of the evaluation metrics to changes in such words, we modified a single word describing either the location or severity of the pathology. 'The modifications regarding the location of the pathology include alterations to other anatomical locations (e.g.. basal to apical) and swaps between left and right. As for the severity of the pathology, modified the severity level. To establish bascline for this evaluation, we also replaced non-informative words (c.g.. "the" / "this" / "there") by the [UNK] token. We anticipate that change in the pathology description will exert greater influence on the metrics than a change in non- non-informative words. We obtained 187 examples for the location change, 95 for the severity change and 133 for the non-informative word change.
Table 3 presents the difference (A) of each metric for the significant perturbations (location or severity) relative to the non-informative word change. As before, NLG metrics are unaware of the significance of the words. The scores are similar for both significant and non-significant word perturbations. Additionally, for all these perturbations, most of the scores (all but METEOR) are very close to 1, suggesting almost perfect similarity, although the clinical difference caused by the perturbations (c.g., mislocating finding) are very significant. CE metrics also fail to recognize the significance of these perturbations. The difference to the non-informative word perturbations for these metrics is low, and the absolute scores are relatively high (close to 1), demonstrating the low sensitivity of these metrics to such errors. This is explained by the fact that these metrics often focus on the existence of findings but not their details. Our metric, however, is aware of these subtle differences and their importance, due to the contrastive-batt design of the shared space we operate within, leading penalty in the score compared to the change of non-informative word.
Modification of reports without findings. There is no single structure for medical reports. Conse- quently, a radiology image can be described in various ways depending on factors such as the patient's
7

---

Nudd 10000090 12
Chattert
ViScome
VLSCome
ScSco
Vi
(a) Ours B-4
(b) Ours RG
(c) Ours BS
(d) Ours vs. Cb
Figure 2: Scores of equivalent normal reports. When comparing two reports with no findings. the score is expected to be high, as both convey the same clinical findings with differences only in the writing of the report. Compared to BLEU-4 (a) and RadGraph F1 (b), our metric yields high scores (right-side), while the other metrics yield low scores (bottom-side). BERTScore (c) provides mid-range scores instead of high scores. CheXpert (d) provides higher scores than the others, yet lower scores than our metric, although it was expected to yield scores close to 1 as both reports in each pair contain no findings.
history the radiologist's style and focus. However, the core purpose of the report diagnosing abnormalities and recognizing normal findings, whether explicitly or implicitly should remain consistent. To demonstrate the robustness of metries these variations, we specifically select studies with normal findings only. We created a standardized general report representing studies with no findings by compiling the most common sentences from the dataset associated with reports lacking abnormalities. Then, for each image corresponding to report with no abnormalities, we pair it with this constructed report (see example in Figure Th). In total, we obtain 207 such examples. In this experiment, we expect a metric to exhibit robustness to the exact wording and to discern the clinical meaning of the report. Specifically, the metric score should be relatively high.
Figure 2 compares the scores of our metric to those given by four other metrics. BLEU-4 and RadGraph F1 assign these examples low score, particularly for BLEU-4 the average is 0.15 and for RadGraph F1 is 0.23, whereas BERTSccre yields mid-level results with an average of 0.51. This occurs because BLEU and RadGraph are sensitive the exact wording, while BERTScore exhibits some robustness to words with similar meanings. CheXbert, which is CE metric, is expected to assign very high scores in this case, as all the reports should be classified with findings. Yet, its average is only 0.63. The other CE metric, CheXpert (F1), which directly measures classification accuracy, does behave as expected with an average of 0.91. Our scores are predominantly high, with an average of 0.85, comparable to CheXpert and surpass all other methods, as we do not rely solely on text similarity but rather on information that is mutual to the visual and textual modalities, with semantics captured in the shared space. Other metrics demonstrate undesirable behavior as those depicted Figure 2 BLEU-1 0.33, METEOR 18, ROUGE-L 0.33.
4.3 Ablation study
Similarity measurements. Given three embedding vectors-one for the ground-truth report Te, one for the generated report 9c, and one for the input image T-blable presents the results of several alternative similarity measurements. For all the measures, we examine the correlation with the ranking of radiologists, using the Kendall correlation coefficient. The first measure considers only re and 9e: and the others consider also the image embedding iç. These measures are: (1) The cosine similarity between re and ge. This distance implicitly considers the image as part of the construction of the shared space, but not in the computation of the metric.
(2) The radius the minimal bounding sphere measures the density of the three points in the high dimensional space by estimating the minimal bounding sphere of these three points. (3) Image centered cosine similarity is the cosine similarity between the difference vectors re and ge The difference between this similarity and the vanilla cosine similarity is the origin, i.c. whether is the Ö vector the image vector. (4) Our metrie: the area of the triangle created by the three high- nigh-dimensional points (embedding vectors).
Table Table shows that the results by these measurements, which are all based explicitly or implicitly on connections with visual information, are superior to the previous methods presented in Table [] which are purely textual. Furthermore, the relative position of the image in the embedding space to the two reports is useful for the similarity measurement, as evident by the higher correlation achieved in the image- centred cosine similarity compared to the regular cosine similarity (0.678 vs. 0.555).
8

---

Table 4: Similarity measurements ablation. Our proposed metric, based on triangle area computa- tion, achieves the highest correlation with radiologists ranking (Kendall's r coefficient on ReX Val dataset), compared to other methods to measure the similarity of embeddings in the shared space.
Minimal bounding
Image-centered cosine similarity
Triangle area (ours)
Cosine similarity
sphere
Kendall 7
0,555
0.525
0.678
0.718
Table 5: Embedding models ablation. Comparison of various vision- embedding models to construct the shared space for our metric. LIMITR achieves the highest correlation with radiologists judgment (Kendall's coefficant on ReXlVal dataset).
Con VIRT 37
BioViL 2
MedCLIP
B3
GLoRIA
I LIMITR 6
Kendall т
0.526
0,580
0.636
0.698
0.718
In addition, when considering the Euclidean distances between the points and not only the angles between them, as captured by the triangle area, the correlation boosts (0.718). . The minimal bounding sphere shows relatively low correlation (0.525), possibly due to the exponential growth of its volume in high dimensions. Intuitively, bounding spheres might contain mostly empty volume.
Embedding models. Recall that we used LIMITR [6] our multi-moda embedding model due to its focus on precise retrieval tasks, as well as extracting discriminative representations at both local and global levels. Table ( that this model outperforms other multi-modal embedding models. LIMITR embeddings demonstrate the highest correlation with radiologists' rankings compared to VIRT (37,, BiL (2) MedCLIP [33. and GLoRIA [[]] (the C' values set for these models are 1,4, 0.06, and 1155, respectively). This may be explained by the fact that LIMITR is optimized for retrieval tasks, excelling matching single image with specific report, while the other models prioritize downstream tasks such as classification and segmentation, which may be less precise for capturing nuances relevant for evaluation purposes. Since our metric relies strong alignment between the image and the entire report to assess the quality of generated reports, the emphasis of LIMITR on alignment-focuser tasks aligns well with our evaluation criteria.
Results on report generation models. We have shown the benefit of our metric on evaluation datasets. In the supplementary materials we show similar advantages reports generates by SoTA report generation models, both qualitatively and quantitatively.
Limitations. The constant C' utilized in Eq. 2 is determined based on a representative dataset and applicable to all datasets within the domain. It serves the purpose of normalizing the size of the triangle to the range (0, 1] (if desired), as the size of the triangle is unbounded. In different domains, such as other imaging sources, the value C will need to be determined accordingly.
Societal impact. The high correlation of our metric with radiologists' judgment may result in overreliance on automated systems, which are prone errors. If metric leads increased reliance on automated systems for generating medical reports, there is risk that healthcare professionals may become overly dependent on these technologies. This could potentially lead to a reduction in critical thinking and clinical judgment, as well as a loss of the human touch in patient care. Therefore, while this metric can be used to quantify the quality of automated systems, their utilization in real-world practice should be carefully supervised.
5
Conclusions
This paper introduces a novel evaluation metric, V L.corre, for automatic medical report generation from X-ray images. It addresses shortcomings found in current evaluation methods, which either focus solely on textual similarities, neglecting clinical considerations, or concentrate solely on one clinical aspect, such as pathology, while disregarding all other facets. The main idea behind our metric is to assess the similarity between radiology reports in relation to their corresponding radiology image. The metric measures similarity embeddings, of the image, the ground-truth report, and the generated report. We demonstrate the strength of our metric compared to previous metries both qualitatively and quantitatively; for instance, it improves the correlation with radiologists judgments by 20% on the ReXVal dataset. In addition,
joint visual-textual space, according to the triplet of

---

we propose new dataset for metric evaluation, featuring thoughtfully crafted perturbations that differentiate between substantial modifications (e.g.. omitting a diagnosis) and less significant ones.
Acknowledgements. 2329/22.
We gratefully acknowledge the support of the Israel Science Foundation
References
Banerjee, S., Lavie, A.: METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization (2005)
Boecking, B., Usuyama, N., Bannur, S., Castro, D.C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Sivare-Vaalle J., et al.: Making the most semanties improve biomedical vision language processing. In: European conference on computer vision. pp. 1-21. Springer (2022)
[3] Chen, Z., Shen, Y., Song, Y., Wan, X.: Cross-modal memory networks for radiology report generation. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1tth International Joint Conference on Natural Language Processing. Association for Computational Linguistics (2021)
[4] Chen, Z.. Song. Y., Chang. T.H., Wan, X.:
Generating radiology reports via memory-driver transformer. In: Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP). Association for Computational Linguistics (2020)
[5] Chen, Z.. Song. Y., Chang. T.H., Wan, X.: Generating radiology reports via memory- transformer. Proceedings of 2020 Conference on Empirical Methods Natural Language Processing (EMNLP) 1439-1449 (2020)
[6] Dawidowicz, Hirsch, E., Tal, Limitr: Leveraging local information medical image-text representation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 21165-21173 (2023)
[7] Gu, T., Liu, D., Li, Z., Cai, W.: Complex organ mask guided radiology report generation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 7995 50044404444 (2024)
[8] Hirsch, E., Dawidowicz, G., Tal, A.: Medeycle: Unpaired medical report generation via cycle-eonsistency.
Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1929-1944 (2024)
[9] Hirsch, E., Dawidowicz. G., Tal, A.: Medrat: Unpaired medical report generation via auxiliary tasks. In: European Conference Computer Vision. pp. 18-35. Springer (2025)
[10] Hou, W., Xu, K., Cheng. Y., Li, W., Liu, J.: ORGAN: Observation guided radiology report generation via tree reasoning. In: Rogers, A., Boyd-Graber, J.. Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 8108-8 8122. Association for Computational Linguistics, Toronto, Canada (Jul 2023)
[11] Huang, S.C., Shen, L., Lungren, M.P., Yeung. S.: Gloria: A multimodal global global-local representation learning framework for label-efficient medical image recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3942-399295 (2021)
[12] Huang, Z., Zhang, X., Zhang, S.: Kiut: Knowledge-injee u-transformer for radiology report generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19809 19818 (2023)
[13] Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurca-lleus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings the AAAI conference on artificial intelligence. vol. pp. 590-597 (2019)
[14] Jain. S., Agrawal, A.. Saporta, A.. Truong, S.Q.. Duong, D.N., Bui, T. Chambon. P., Zhang, Y., Lungren, M.P., Ng, Y., et al.: Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:21066 14463 (2021)
[15] Jing, B., Xie, P. Xing, E.: On the automatic generation of medical imaging reports. arXiv preprint 1111:11 (2017)
[16] Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y,, Mark, R.G., Horng. S.: Mimic-exr, de-identified publicly available database of chest radiographs with free-text reports. Scientific data 6(1), 317 (2019)
10

---

[17] Li. C.Y., Liang, X., Hu, Z., Xing, E.P. Knowledge-driven encode. retrieve. paraphrase for medical image report generation. In: Proceedings of the AAAI Conference Artificial Intelligence, vol. 33, pp. 6666-6673 (2019)
[18] Li. M., Lin, B.., Chen, Z.. Lin. H., Liang. X., Chang. X.: Dynamic graph enhanced contrastive learning for chest x-ray report generation. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattem Recognition. pp. 3334-3343 (2023)
[19] Li, Y., Yang, B., Cheng, X., Zhu, Z., Li, H., Zou, Y.: Unify, align and refine: Multi-level semantic alignment radiology report generation. In: Proceedings of the IEEE/CVP International Conference on Computer Vision. pp. 2863-2874 (2023)
[20] Lin, C.Y.: ROUGE: package for automatic evaluation summaries. In: Text Summarization Branches Out. Association for Computational Linguistics (Jul 2004)
[21] Liu, F., Wu, X.. Ge, S., Fan, W., Zou, Exploring and distilling posterior and prior knowledge for radiology report generation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13753-13762 (2021)
[22] Liu, F., You, C., Wu, X., Ge, S., Sun, X., et al.: Auto-encoding knowledge graph for unsupervised medical report generation. Advances in Neural Information Processing Systems 34, 16266-16279 (2021)
[23] Liu, G., Hsu, T.M.H., MeDermott, M., Boag, W., Weng, W.H., Szolovits, P., Ghassemi, M.: Clinically accurate chest x-ray report generation. In: Machine Learning for Healthcare Conference, pp. 249-269. PMLR (2019)
[24] Müller, P., Kaissis, G., Zou, C.. Rueckert, D.: Joint learning of localized representations from medical images and reports. In: European Conference on Computer Vision. pp. 685-701. Springer (2022)
[25] Papineni, K., Roukos, S., Ward, T.. Zhu, WJJ.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (2002)
[26] Smit, A., Jain, S., Rajpurkar, P., Parcek, A., Ng, A.Y., Lungren, M.: Combining automatic labelers and expert annotations for accurate radiology report labeling using bert. In: Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP). pp. 1500-1519 (2020)
[27] Tanida, T., Müller, P., Kaissis, G., Rueckent, D.: Interactive and explainable region-guided radiology report generation. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 7433-7442 (2023)
[28] Vedantam, R., Lawrence Zitnick. C., Parikh, D.: Cider: Consensus-bail image description evaluation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4566-45777 (2015)
[29] Wang, F. Zhou, Y., Wang, S.. Vardhanabhuti, V., Yu, L.: Multi guatiigraanlarity cross- modal alignment for generalized medical visual representation learning. Advances in Neural Information Processing Systems 35, 33536-33549 (2022)
[30] Wang. Bhalerao, A., He, Y: Cross-modal prototype driven network for radiology report generation. Computer EECV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV. pp. 563-579. Springer (2022)
[31] Wang. Z., Liu, L.. Wang, L., Zhou, L.: Metransformer: Radiology report generation by transformer with multiple learnable expert tokens. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11558-11566 (2023)
[32] Wang, Z., Tang, M., Wang. L., Li, X., Zhou, L.: A medical semantic-assiste transformer for radiographic report generation. In: Medical Image Computing and Computer Assisted InterventionMICCAAI 2022. pp- 655-664. Springer (2022)
[33] Wang. Z., Wu, Z., Agarwal, D., Sun, J.: Medclip: Contrastive learning from unpaired medical images and text (2022)
[34] Yu, F., Endo, M., Krishnan, R., Pan, L. Tsai, A., Reis, E.P., Fonseca. E.K.U.N., Lee, H.M.H., Abad, Z.S.H., Ng. A.Y., et al.: Evaluating progress in automatic chest x-ray radiology report generation. Patterns 4(9) (2023)
[35] Zhang, T., Kishore, V., Wu, F. Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert. In: International Conference on Learning Representations (2019)

---

[36] Zhang, Y., Wang, X., Xu, Z., Yu, Q. Yuille, A., Xu, D.: When radiology report generation meets knowledge graph. In: Proceedings of the AAAI Conference Artificial Intelligence. vol. 34, pp. 12910-12917 (2020)
[37] Zhang, Y., Jiang. H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. In: Machine Learning for Healthcare Conference, pp. 2-25. PMLR (2022)
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims and contributions from the abstract and the introduction elaborated and backed with experiments in Sections
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of our paper are discussed in a designated section under Section 4.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 3 provides detailed explanations about method and section includes all the necessary information about the experiments and the used data, to allow reproducibility.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The code for computing our metric and the perturbed dataset will be released.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
The experimental details appear in Section and supplementary materi-
als.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our proposed metric is deterministic.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
13

---

Justification: Addressed in the supplementary materials.
9. Code Of Ethies
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of
Answer: [Yes]
Justification: We follow the ethical guidelines, and we communicate the known or anticipated consequences of research in the societal impact paragraph.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We communicate the known or anticipated consequences of research in the societal impact paragraph.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited the original papers that produced the code package or dataset.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper communicates the details of the dataset/code.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
14


Reviews:
[{"review_id": "aINhni87jZ", "reviewer": "Reviewer_F1XY", "summary": "The paper introduces a new image and language-based metric for generated report evaluation of X-ray images.", "questions": "I have no question/confusion about the technical aspects of the paper.", "limitations": "1.\tThe constant (C) introduced in Eq. 2 should be numerized for each model in Table 5.\n2.\tThe ablation studies should further investigate the impacts of image-related perturbations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe concept of utilizing both semantic and vision-based representations in the form of a similarity score is very interesting.\n2.\tThe supplementary material effectively reflects different clinical cases and showcases the performance of the proposed metric alongside other SOTA models, both for report generation and evaluation metrics.\n3.\tApplying perturbations to the data provides an in-depth investigation of the proposed metric under different conditions.\n4.\tThe ablation studies complement the main findings of the paper and address potential questions proactively.", "weaknesses": "1.\tThe paper claims to introduce VLScore for automatic medical report generation from X-ray images. However, the primary contribution is the metric itself, which is validated in several scenarios. My main concern about the paper is the technical novelty. \n2.\tAll perturbations have been applied at the level of text, but it is intuitively obvious that the removal of such information has a lesser impact on image-report-based metrics. It would be beneficial for the authors to conduct another supplemental study, in which several regions of the image are blurred or removed and their corresponding report is accordingly changed. This would more accurately assess the proposed metric in the context of \"Image-aware evaluation of generated reports.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments and insights, appreciating the novelty, the motivation, and the results.\n\n*Q1:* _the primary contribution is the metric itself ... technical novelty_\n\nThe main contribution is indeed the metric itself. However, the approach to designing it is novel. Furthermore, proper evaluation metrics are crucial for advancing the future development of models for specific tasks, such as medical report generation. It is common for model-based metrics to utilize pre-trained models, as seen in CLIPScore (Hessel et al.) and Inception Score (Salimans et al.), while their novelty lies in the utilization of these models as part of a metric for a given task.\n\n**Q2:** _conduct another supplemental study ... regions of the image are blurred or removed (image-related perturbations)_\n\nThank you for this idea. We conducted the proposed experiment for the pathology \"cardiomegaly,\" characterized by an enlarged heart typically positioned in the central region of chest X-ray images. We tested two modifications: removing a central rectangle and removing a random peripheral rectangle, each occupying 20% of the image size. The modifications yielded a VLScore of 0.78 for central region removal and a VLScore of 0.92 for random peripheral region removal. These results highlight the sensitivity of our metric to different image content. Removing the central region, which included key visual indicators of the pathology, had a greater impact compared to removing other areas, which were less critical to the essence of the report. We will add these results to the supplementary materials.\n\n\n**Q3:** _The constant (C) ... for each model in Table 5_\n\nThe values are: ConVIRT: 124, BioViL: 1, MedCLIP: 0.06, GLoRIA: 1155.\nWe will add them to Table 5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aBS3xms481", "reviewer": "Reviewer_EkCX", "summary": "In this work, the authors introduce a novel metric (VLScore) for evaluating the quality of generated medical reports. The key idea is to measure the similarity between radiology reports while also taking the image itself into account; this is a distinction from prior works, which generally only compare radiology reports without considering the source image. Evaluations demonstrate that the proposed metric aligns closely with the judgments of expert radiologists. The authors also provide a new dataset where they perturb the quality of reports (e.g. by removing a critical diagnosis) to evaluate how metrics react; evaluations on this dataset also demonstrate the utility of the proposed metric.", "questions": "In addition to the concerns listed above in the “weaknesses” section, I have listed some additional questions below: \n\n- The authors are also encouraged to also report comparisons with the RadCliq metric, which is currently state-of-the-art on ReXVal.\n- Ablation Table 4 could benefit from a comparison with a RefCLIPScore-like approach, where the image embeddings are generated from LIMITR rather than CLIP.\n- Can the authors comment on inference time and how expensive it would be to run this approach in this practice?\n- Section 4.2 can benefit from additional implementation details related to the creation of the perturbed dataset. For instance, how were “pathology description words” selected? How were modifications made (e.g. was an external LLM utilized)? Some qualitative examples from the perturbed dataset may be useful here.", "limitations": "Yes, the authors have adequately described the limitations of their work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The approach introduced by the authors addresses an important, high-impact problem. There is a shortage of effective metrics in the medical domain for evaluating the quality of generated text. \n2. A particular strength of this paper is the evaluations on perturbed data. Whereas most existing works evaluate solely on the human-annotated ReXVal dataset, the authors of this work introduce a new dataset with perturbed samples. This dataset can be used to evaluate the extent to which metrics respond to variations in report quality. The dataset is well-designed and has the potential to be useful for future works.", "weaknesses": "1. **Limited in scope**: One weakness of this paper is that the proposed approach is limited in scope. In particular, the method is designed to operate on a single type of medical image (chest X-ray) and is evaluated using a single dataset (MIMIC-CXR).\n    - The proposed method assumes access to a multimodal model sensitive to local features. This is a strong assumption, and such models may not be available for more diverse use cases.\n    - Can the authors comment on whether they expect their approach to generalize to other chest X-ray datasets? Is the approach expected to generalize to other imaging modalities and anatomical features beyond chest X-rays? Would the model need to be retrained to operate on other types of radiology reports?\n2. **Need for finer-grained evaluations**: The paper could benefit from some additional analysis of the proposed metric, particularly with respect to important subgroups such as disease labels. For instance, when evaluating the removal of a pathology sentence vs. insignificant sentence, does the quality of the metric vary for different pathologies?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments and insights, appreciating the novelty, the motivation, the method, and the results.\n\n**Q1:** _Assuming access to multimodal models sensitive to local features_\n\nOur method does not rely on multimodal models sensitive to local features; it only relies on an embedding model for both modalities. For instance, Table 5 compares several embedding models, which do not necessarily utilize local features, such as ConVIRT [36]. However, in the domain of chest X-rays, using local features is beneficial. Therefore, although ConVIRT achieves good results (surpassing RadGraph F1), LIMITR further improves these results.\n\n**Q2:** _generalization to other chest X-ray datasets, imaging modalities, and anatomical features_\n\nWe expect the metric to generalize well to other chest X-ray datasets, given that multimodal models have shown strong generalization across various datasets and tasks. For instance, LIMITR achieved good zero/few-shot classification performance on the RSNA (88% accuracy) and CheXpert (88.7% accuracy) datasets. Additionally, our metric has the potential to generalize across different imaging modalities and anatomical features, either by employing suitable multimodal models or by retraining the proposed models. However, due to the lack of relevant datasets, we cannot verify this at this time.\n\n**Q3:** _finer-grained evaluations ... subgroups such as disease labels_\n\nThank you for this suggestion. We conducted finer-grained evaluations for the removal of pathology experiment, refining the results in Table 2. The average score across all pathologies is 0.69. The scores for each pathology are as follows:\n\n* Atelectasis: 0.77\n* Cardiomegaly: 0.74\n* Consolidation: 0.71\n* Edema: 0.69\n* Enlarged Cardiomediastinum: 0.75\n* Fracture: 0.72\n* Lung Lesion: 0.57\n* Lung Opacity: 0.699\n* Pleural Effusion: 0.69\n* Pleural Other: 0.71\n* Pneumonia: 0.83\n* Pneumothorax: 0.51\n* Support Device: 0.65\n\nWe observe that these scores are similar for most pathologies.\n\n**Q4:** _comparisons with the RadCliq metric_\n\nWe followed your suggestion. Please see Table 1 in the attached PDF of the rebuttal, which shows that VLScore is more sensitive than RadCliq for all perturbations (sentence removal and word change). For example, our metric shows a difference of 0.15 for pathology versus insignificant sentence removal, while RadCliq shows only 0.06. This is expected, as RadCliq is a linear combination of other metrics we report in the paper (BertScore, CheXbert, and RadGraph F1). Additionally, for reports without findings, our metric captures their similarity with an average score of 0.85, whereas RadCliq assigns a low score of 0.434 to these pairs, showcasing the robustness of our metric in this aspect. We will add RadCliq to the tables of the perturbed data.\n\n**Q5:** _Ablation Table 4 ... RefCLIPScore-like approach_\n\nA RefCLIPScore-like approach (using LIMITR embeddings) yields a low correlation with radiologists' judgment on the ReXval dataset, achieving a score of 0.241 compared to 0.718 with our VLScore.\n\n**Q6:** _Inference time_\n\nThe inference time for a single triplet of one image and two reports (ground truth and generated) is 2.30 seconds on a single A6000 GPU. In comparison, other metrics, such as RadGraph F1, take longer to run, with a running time of 3.7 seconds per example.\n\n**Q7:** _additional implementation details related to the creation of the perturbed dataset_\n\nAll the perturbations were manually created by biomedical experts. Specifically, reports with only 1-2 pathologies were identified manually to ensure the modifications were significant enough. The experts identified the pathology sentences, general sentences, pathology descriptive words, and non-informative words. The sentences and words were then modified according to the perturbation manually while maintaining reasonable medical context. Many qualitative examples for each perturbation in the dataset are available in the supplementary materials, which we will refer to in the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iLXdgQf8uj", "reviewer": "Reviewer_6uVW", "summary": "This paper proposed a novel evaluation metric, VLScore, which can more accurately reflect the alignment of generated reports and radiologists’ judgments. It is achieved by measuring the similarity in visual-textual space. To demonstrate the effectiveness of the proposed metrics, a new dataset with specific perturbations (e.g. location, severity) is created.", "questions": "1. What is the running time of VLScore compared to NLG and CE metrics?\n2. Is there any justification for the selection of multi-modal embedding models?\n3. As the paper proposes new evaluation metrics for benchmarking, it will be more convincing to provide the evaluation of existing methods using the proposed VLScore.", "limitations": "The authors adequately addressed the limitations and, if applicable, potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Motivation: Existing report generation methods are benchmarked with natural language generation (NLG) metrics and clinical efficacy (CE) metrics. However, those metrics cannot accurately reflect the medical correctness of the generated reports. Thus, the advancement of this aspect is important.\n2. Identified Problems and Solutions: 1) NLG metrics cannot capture nuanced but significant differences in long reports. The description beyond images and differences of expression will cause a drop in evaluation. 2) CE metrics rely on a trained classification model with 14 disease classes, which means the detailed measurement of the report will be neglected. The proposed similarity measurement of representation space can solve the problems to some extent.\n3. Evaluation: The dataset with tailored perturbations can provide detailed comparisons in different aspects and amplify the differences between the proposed VLScore and the conventional NLG and CE metrics.", "weaknesses": "1. The effectiveness of the proposed metric highly depends on the performance of multi-modal embedding models.\n2. The proposed metrics cannot evaluate the similarities of writing style between generated reports and ground truths as the metrics mainly focus on the similarities in latent space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments and insights, appreciating the novelty, the motivation, the method, and the evaluation.\n\n**Q1:** _dependency on the performance of multi-modal embedding models_\n\nOur evaluation metric requires the selection of an embedding model that maps images and reports to a shared space, where distance can be measured. The existence of such models is a reasonable assumption, as these models exist (Table 5). Accurate embedding models lead to more accurate metrics (see Table 5). The method is general; if improved models are released, this metric can easily be upgraded.\n\n**Q2:** _cannot evaluate the similarities of writing style_\n\nThank you for the insight. Indeed, our metric is robust to style when measuring the similarity between two reports. This is an advantage of the metric, as two experts may naturally produce different reports for the same image—differing in style but not in essence (i.e., the findings). Our approach focuses on the essence rather than the specific wording.\n\n**Q3:** _running time in comparison to other metrics_\n\nThe inference time for a single triplet of one image and two reports (ground truth and generated) is 2.30 seconds on a single A6000 GPU. The running times for the different metrics are as follows:\nRadGraph (CE) 3.7 seconds, \nCheXbert (CE) 0.9 seconds, \nBertScore (NLG) 0.6 seconds, \nBLEU (NLG) 0.001 seconds, \nROUGE (NLG) 0.001 seconds,\nMETEOR (NLG) 0.6 seconds,\nand VLScore (ours - both) 2.3 seconds. \n\n**Q4:** _selection of multi-modal embedding models_\n\nWe experimented with various multi-modal embedding models from recent top-tier venues, which are diverse in architecture and training approach (e.g., including global/local features, different losses, or different encoders). The results are presented in Table 5. We selected the model that showed the best correlation with radiologists' judgment.\n\n**Q5:** _evaluation of existing methods using the proposed VLScore_\n\nBoth qualitative and quantitative evaluations of existing methods are provided in the supplementary materials, in Sections 2 and 3. We will add references to these results in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iZZ6ZmcNbI", "reviewer": "Reviewer_VcyL", "summary": "This paper propose a novel evaluation metric, VLScore, for automatic medical report generation, considering both textual and clinical aspects. It key idea is to map the reports and their corresponding image to the joint visual-textual space and measure the similarity. Experiments demonstrate that they gain better performance than other metrics in both ReXVal dataset and their new proposed dataset.", "questions": "1. I don’t understand the generation procession of the perturbed dataset. Please explain over the length of report and how much content was deleted. I think this greatly affects the final relevance results.\n2. In Figure 1b, the inconsistency in position while everything else is described correctly results in a score of 0.675. If, for the same case, some synonyms are used in the report, or other perturbations are applied, is it possible to obtain a similarly reasonable score?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper’s performance looks good. The results show the proposed method improves the correlation with radiologists’ judgments by 20%.\n2. A series of perturbed data are proposed to demonstrate the robustness of different metrics.", "weaknesses": "1. In terms of motivation, evaluating the quality of radiology report generation through image modality is a far-fetched and unreasonable process. \n2. Images and text need to be mapped to the same feature space, and the evaluation process proposed in the paper relies too heavily on the shared image-text space. However, according to the results reported in LIMITR, the retrieval results are not very high, and I do not have enough confidence that all anomalies mentioned in the report can be corresponded to the images. Additionally, I doubt whether the distance between the reports and the image embedding could reliably measure the similarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments and insights, appreciating the novelty, and the results.\n\n**Q1:** _motivation: evaluating the quality of radiology report generation through image modality..._\n\nAn image contains a wealth of information that can be described textually in many forms. Directly measuring the semantic similarity between two textual reports remains problematic, as highlighted by both NLG and CE metrics, which often fail to accurately assess report quality. Our approach addresses this challenge by incorporating visual information to better evaluate semantic similarity. Finally, the results demonstrate the benefits of our approach, including its strong correlation with radiologists' judgments, as well as its robustness and sensitivity to tailored perturbations.\n\n**Q2:** _relies on the shared image-text space ... reliably measure the similarity_\n\nWhile embedding models are typically evaluated on tasks such as retrieval or classification, we assess their effectiveness based on correlation with radiologists' judgments and their robustness and sensitivity to tailored perturbations. In this context, our proposed use of these models demonstrates their ability to capture essential details in both the image and the report. This effectiveness is highlighted by the results on the perturbed datasets (see Section 4.2). Consequently, our metric shows improvements compared to previous metrics.\n\n**Q3:** _generation process of the perturbed dataset ... how much content was deleted_\n\nThe perturbed dataset was manually created by biomedical experts, who had the option to remove a sentence (either one that describes a pathology or an insignificant one) and to modify a single word (a pathology descriptive word or a general word) for each given report. After modifying a pathology descriptive word or a non-informative word, the length of the report remains unchanged. Removing a pathology sentence resulted in an average length difference of 12.3 words compared to the original length, while removing an insignificant sentence led to an average length difference of 11.8 words. The length of the standardized general report is 29 words, and the average length of reports without findings is 27.4 words. In all cases, the perturbed reports compared had a similar length.\n\n**Q4:** _Figure 1b... synonyms are used...obtain a similarly reasonable score_\n\nOur metric is relatively robust to the exact wording when measuring the similarity between two reports, which was one of our goals. Following your suggestion, we studied the influence of modifying words with synonyms experimentally. Our method demonstrates robustness to these perturbations. For example, in Figure 1b, changing the word \"suggests\" to \"indicates\" results in a VLScore of 0.96; changing \"a site\" to \"an area\" yields a VLScore of 0.95; and changing \"seems\" to 'appears' results in a VLScore of 0.98. These scores, which are very close to 1, indicate the desired robustness of our method. We will add these results to the supplementary materials.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General Response to All Reviewers**

We would like to thank all the reviewers for their hard work and enlightening comments, appreciating the 
motivation (_“an important, high-impact problem”_ [R-EkCX], _“the advancement of this aspect is important”_ [R-6uVW]),
the novelty (_“a new image and language-based metric”_ [R-F1XY], _“a novel metric”_ [R-EkCX], _“a novel evaluation metric”_ [R-6uVW \& R-VcyL]),  the method (_“The concept ... is interesting”_ [R-F1XY], _“introduce a new dataset ... well-designed”_ [R-EkCX], _“The proposed similarity measurement ... can solve the problems”_ [R-6uVW]), and the results (_"provides an in-depth investigation,"_ _"address potential questions proactively"_ [R-F1XY], _"aligns closely with the judgments of expert radiologists"_ [R-EkCX], _"performance looks good,"_ _"demonstrate the robustness of different metrics"_ [R-VcyL]).

We respond to the reviewers' comments, addressing the issues that they raise below.

Meta-review: The four reviewers are divided: two lean towards a score of 6-7, while the other two suggest a score of 2-4. I find the most critical review to be unduly harsh, seemingly disregarding the strong and well-argued rebuttal. The approach for evaluating radiological reports from medical images is intriguing, and I share the reviewers' and authors' views on its rationale and originality. This AC is inclined towards a borderline acceptance, although I acknowledge that the paper's acceptance may depend on its prioritisation relative to other submissions.
Decision: Accept (poster)