RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capability
Authors: Jonggwon Park
Byungmu Yoon
Soobum Kim
Kyoyun Choi
Venue: NeurIPS 2025 poster ()
Keywords: medical imaging
multimodal learning
zero-shot learning
vision-language alignment
explainable ai
Primary area: machine_learning_for_sciences

Abstract: Recent advancements in multimodal models have significantly improved vision-language (VL) alignment in radiology. 
However, existing approaches struggle to effectively utilize complex radiology reports for learning and offer limited interpretability through attention probability visualizations.
To address these challenges, we introduce $\textbf{RadZero}$, a novel framework for VL alignment in chest X-ray with zero-shot multi-task capability.
A key component of our approach is $\textbf{VL-CABS}$ ($\textbf{V}$ision-$\textbf{L}$anguage $\textbf{C}$ross-$\textbf{A}$ttention $\textbf{B}$ased on $\textbf{S}$imilarity), which aligns text embeddings with local image features for interpretable, fine-grained VL reasoning.
RadZero leverages large language models to extract concise semantic sentences from radiology reports and employs multi-positive contrastive training to effectively capture relationships between images and multiple relevant textual descriptions. 
It uses a pre-trained vision encoder with additional trainable Transformer layers, allowing efficient high-resolution image processing. 
By computing similarity between text embeddings and local image patch features, VL-CABS enables zero-shot inference with similarity 
probability for classification, and pixel-level VL similarity maps for grounding and segmentation. 
Experimental results on public chest radiograph benchmarks show that RadZero outperforms state-of-the-art methods in zero-shot classification, grounding, and segmentation. 
Furthermore, VL similarity map analysis highlights the potential of VL-CABS for improving explainability in VL alignment. 
Additionally, qualitative evaluation demonstrates RadZero's capability for open-vocabulary semantic segmentation, further validating its effectiveness in medical imaging.
Code is available at https://github.com/deepnoid-ai/RadZero.

RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot
Capability
Jonggwon Park, Byungmu Yoon, Soobum Kim, Kyoyun Choi
DEEPNOID Inc.
Seoul, South Korea
(jgpark, bmyoon, soobumk, xychoi]]deeppoiit com
Abstract
Recent advancements in multimodal models have significantly improved vision- language (VL.) alignment in radiology. However, existing approaches struggle to effectively utilize complex radiology reports for learning and offer limited inter- pretability through attention probability visualizations. To address these challenges, we introduce RadZero, novel framework for VL alignment in chest X-ray with zero-shot multi-task capability. A key component of our approach is VL-CABS (Vision-Lange) Languu- Cross-Attentin beddings with local image features for interpretable, fine-grained VL reasoning. RadZero leverages large language models to extract concise semantic sentences from radiology reports and employs multi-positin positive contrastive training to effectively capture relationships between images and multiple relevant textual descriptions. It uses a pre-trained vision encoder with additional trainable Transformer layers, allowing efficient high-resolution image processing. By computing similarity be- tween text embeddings and local image patch features, VL-CABS enables zero-shot inference with similarity probability for classification, and pixel-level similar- ity maps for grounding and segmentation. Experimental results public chest radiograph benchmarks show that RadZero outperforms rate-of-the--a art methods zero-shot classification, grounding, and segmentation. Furthermore, VL similarity map analysis highlights the potential of VL-CABS for improving explainability VI. alignment. Additionally, qualitative evaluation demonstrates RadZero's 's capabil- ity for open-vocabulary semantic segmentation, further validating its effectiveness in medical imaging. Code is available
Based on Similarity), which aligns text em-
Introduction
Recent advancements in deep learning have significantly impacted medical imaging, leading merous studies on computer-aided diagnosis [5] g 25 36). However, acquiring high-quality manual annotations remains key challenge. In contrast, vision- laaiong (VL) models (VLMs) in the natural have reduced reliance on manual labeling by learning from image-text pairs without explicit supervision, achieving strong zero-shot performance in tasks like classification and retrieval. Building on this progress, VLMs have been increasingly explored in medical imaging, including chest X-rays (CXRs). Several studies have demonstrated effective representation learning and zero-shot capabilities [19 24 without task-specift annotations.
image domain 29
40 39
37 42 43
Despite these advance, current medical VLMs underutilize the rich semantics of radiology im- age-report pairs. Prior methods-such as word-level alignment 22
clinical entity extraction
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(4)
Pailtr Teat Pronet Theee Aboos
Negative Text Procege The Jange char
Open Y Vecebulary
Scuunitic Segmentution
Abbrots
Aftindia Map
Baselines
Segnealeise
Sixllarit
Stulketyy
Topar Trugy
RadZero
Fladdscale
Segurstates
Figure 1: Comparison of attention maps and the proposed VL similarity map for visualizing VL alignment. (a) While traditional attention maps inevitably exhibit high values at certain points due to softmax activation, the proposed VL similarity maps yield low values for unrelated image-text pair. (b) Their fixed scale, originating from cosine similarity, enables open-vocabulary semantic segmentation through simple thresholding.
19 face limitations, struggling with poor text embedding segmentation and inefficient training due to random sampling. An effective solution should 1) decompose reports into semantically minimal, clinically meaningful sentences, 2) embed each sentence independently, and 3) learn from multiple sentence-image pairs per study to fully leverage supervision.
42 37
and using large language model (LLM) prompts
Moreover, reliable explainability is critical for the clinical adoption of medical VLMs. Attention However, while an attention map [Figure top) can indicate where the model is focusing. does not provide an explanation for why it is attending to those regions. This limitation can be addressed by computing pixel-level image-text similarity, which enables more fine-graines and transparent explanations.
maps are used as explainable features by most of the recent research [37 19 42
To overcome these shortcomings, we propose RadZero, novel VL alignment framework for chest X-ray with zero-shot multi-task capabili- ties. RadZero employs multi-positive contrastive learning (20) to incorporate multiple sentences per image-report pair. RadZero's core innova- tion is
Zero Shet Multi- Task Performance
CARZis
MiSKLIP
Score (N)
Cross-Are Cross-Attention Based on Similarity (VL-CABS), which directly com- putes cosine similarity between text descriptions and local image patches. Unlike traditional at- tention maps, the resulting VL similarity maps offer clearer visual reasoning by maintaining low values for unrelated image-text pairs (Fig- (a), bottom). This enhances interpretability and enables open-vocabulan semantic segmen-
Zero Shot Chonification
Zere-Shet Granding
Zero Groundigg
Lero Segmentation
Figure 2: Zero-shot multi-task performance. Each score is averaged over multiple datasets per task.
tation via simple thresholding. RadZero also supports high- resolution inputs by freezing a pre-trained image encoder [39] and adding trainable Transformer layers (18), further boosting performance fine-grained zero-shot tasks. Experiments on public chest radiograph benchmarks demonstrate that RadZero outperforms state-of-the-an art (SOTA) models in various zero-shot tasks (Figure 2 while qualitative analyses reveal its enhanced explainability and potential for vocabuluyy semantic segmentation.
2
Related Works
2.1 General vision-language alignment
Contrastive learning for -asion-l alignment with large-scale image-text pairs has been actively studied. CLIP 22 demonstrated that directly aligning images and text enables strong zero-shot

---

classification. LiT [39 proposed freezing the pre- pre-trained vision encoder during contrastive training, preserving fine-grained visual features and further enhancing zero-shot performance. dino. txt [T8]
extended this framework by adding Transformer [33 layers top of pre-trained DINOv2 [27
training only a lightweight module while keeping the vision encoder frozen. Additionally, fused
global and patch-averaged embeddings, enabling patch-level similarity computation with text and supporting open-vocabulary semantic segmentation. UniCLIP (20) introduced a multi-positive NCE
(MP-NCE) loss, which independently evaluates multiple positive pairs per image. Building these advances, our approach integrates a frozen, fine-grained vision encoder with trainable Transformer layers, following LiT and dino. txt. We also adopt MP-NCE loss to align images with multiple text representations effectively.
2.2
Vision-language alignment in chest X-ray
Since the introduction of contrastive learning in radiology 43] aligning CXR images with radiology reports has become an actíve research area. GLoRIA 12] focused on local alignment using cross- attention between word-level text embeddings and patch-level image features. MGCA [34 employed both report-level and token-level embeddings to extract multi-granul features, and ViL- (I] similarly relied on token-level embeddings. Nevertheless, segmenting reports into individual words tokens often fails to capture their full semantic meaning.
Due to the complexity of medical image-report relationships, alignment interpretability is essential for clinical use and is commonly addressed using attention maps. MedKLIP [37] and KAD [42. for example, used RadGraph [14] to extract report features and employed attention maps for tasks such as grounding and segmentation. In addition to VL alignment, G2D [2 aggregated attention maps in addition to VL alignment to generate pseudo masks, which were used as pixel-wise pretext supervisory signals during pre- pre-training. CARZero [19 also used attention maps when leveraging cross-attention alignment for zero-shot tasks, incorporating LLM-based prompt alignment to stan- dardize reports. Despite their utility, attention maps have limitations: they often highlight irrelevant regions due to softmax activation, but removing softmax is not ideal as raw logits are unnormalized and uncentered. Additionally, variation in the norms of query and key embeddings leads to incon- sistent similarity values across different image-text pairs. An example and a detailed discussion of the limited explainability of attention maps are provided in Appendix [A] In contrast, approach enhances explainability with VL-CABS, aligning visual patches and text embeddings. The resulting maps offer intuitive and consistent measures of fine-gred grained image-text similarity.
Methods
3.1 Finding-sentence extraction
Radiology reports contain diverse types of information, including clinical history, observations, comparative analysis with prior studies, and diagnostic impressions. Encoding the entire report into a single text embedding often fails capture this complexity. CARZero [19] addressed this by using an LLM to extract relevant sentences and introducing prompt alignment strategy based on the template "There is [disease]" for consistency between training and inference. Similarly, we use LLM to extract such sentences, which we refer to as asniiingg-see findnng-entencess. These are generated using prompt that follows a predefined structure, such as "There is [finding] of [location]," and segmented into minimal semantic units containing the finding name, presence (or uncertainty), and location. The full prompt is provided in Appendix E.2 Each image is paired with multiple finding-sentee sentences during training, as illustrated in Figure 3 (a). For zero-shot inference, we apply prompt alignment by prepending "There is" to text descriptions of findings and anatomical regions.
3.2
Vision-language alignment with similarity based cross-attention
3.2.1
Model architecture
To leverage the advantages of vision encoder pre-training, we adopt the approach of LiT by freezing a pre-trained vision encoder in contrastive learning. In Vision Transformers 9 such as DINOv2 (27), interpolating the positional embeddings allows for increased input image resolution 31.. Building on this property, train our model with high high-resolution images. To embed the output
3

---

(8)
Radiology
Finding Sentence S There oflonion
Simitarity Monix
Imoue
Dmrage 7,
(3
Left remocardins opacity nas substuntially inproved, likely die to atclectasic. Pigtail plaural catheder remointing place and no visibls pueumethocax
Vision Exooder +
Text Ensoder
Raiiology Report
Transfeemet Layars
CL-SABS
Ey CIfaNCC
Findinx
Large Language Model
Text Emrbodding
t
left retrocaniis opacity
Vision Token Sequenes
Similantt Logit
There atelectants
There is pigsall pleural catheter it place
(MP NCE Loss)
There is no preamothoras
Finding-Senttenee Sentence
Similarity Scores
Vision Emieedding
C
(d)
Probubility
Infecence
Zaro-Shat Classifiettion
Patch-Lovel Mg
Inferenee
Zero Shot Grounding Sogmentation
Similarity Map
Pixcl-Levdl GG
Similarity Map
Figure 3: The overall framework of RadZero. (a) Finding-sentenee extraction using LLM. (b) Computation of the similarity logit, Eij between image I; and cos-sim denote weighted sum and cosine similarity, respectively. (c) Computation of MP-NCE loss (C1) and InfoNCE loss (Cr.) from the similarity logit matrix. (d) Zero-shot inference pipeline.
W-sum and
of the vision encoder, we add trainable Transformer layers, as proposed by Jose et al. 18] For the text encoder, extract embeddings for each finding-sentence. The model architecture is illustrated in Figure (b).
pre-trained Sentence-BERR [30.. which is fine-tuned during training. to
3.2.2 Vision-language cross-attention based similarity
We propose VL-CABS (Vision-Language Cross-Attention Based Similarity), a cosine similarity- based cross-attention mechanism for computing similarity logits. By directly employing cosine similarity between the text and visual patch embeddings, we obtain VL similarity scores that are well-defined in range and centered at zero. This consistent scaling allows for fair comparisons across different image-text pairs and significantly enhances explainability through the visualization of VL similarity maps. It also enables single thresholding, offering new possibilities for open-vocabulary semantic segmentation.
The proposed method operates on a mini-batch of size B, pairing each image I, (i I....... B) with N; associated finding-sentences {SSn [ Each image is processed by a vision encoder fa followed by trainable layers fa. yielding sequence of embeddings (ao, U11,.--: Uiz. fa (T.()), where 140 E IRD corresponds to the [CLS] token and (UA ) C RD are patch embeddings, with D denoting the embedding dimension and the total number of patches. Each
Sy is
encoded into sentence-level embedding t" f(()) ) E IR" using text encoder fc.
To compute VL similarity, we (2-normalize all embeddings as 65 = Uik/ / and ET = 17/ /lle" and calculate the scaled cosine similarity between each patch and sentence as siyx =< Gsk. > exp(T) (k: ......I), L),, where 7 is learnable temperature. These scores are converted into attention weights using softmax over the patch index k: dii = exp(>ī))) /// exp()")m) A sentence-spese specific attended vision embedding is computed as the weighted sum oi = El a vak which is then 22-normalized as in mll/ The global similarity logit between image I, and sentence 5th is given by 1h =< b Pr > cxp(r)) The corresponding patch-level similarity map is My = 1111 J.
3.3 Multi-positive contrastive learning
also uses prompt templates for training, suffers from instability due to randomly selecting one sentence for each image at every training step. To utilize all N finding- sentences matched to each image at every step, we adopt multi-positive NCE (MP-NCE) loss 2 which treats positive pairs independently in order to amplify the loss contributions from each positive
Although CARZero 19

---

pair. A visualization of our contrastive loss is shown in Figure 3 (c). Let Ny = =E= N, be the total number of finding-sentence sentences mini-batch. For the i-th image, the number of positive and negative finding-sentende are N; and Nr N₁, respectively. The MP-NCE loss can be computed as follows:
exp(
L
log
NT
N (m))
exp(
Σ m=1
exp(1
For each inding-sentence S. there is one positive image I and B negative images. The corresponding InfoNCE loss 26 is computed as follows:
exp( B
CT
log
(2)
NT
exp(1i) + Σ exp(1)))
jp
The final objective function is the sum of C and Cr: C Cs+
3.4 Zero-shot inference
The similarity logit between an image I and sentence Sj, denoted by 1. is converted into similarity probability itz = ((i)) via a sigmoid function, and used for zero-shot classification.
For grounding and segmentation, reshape the patch-level similarity map Mij 1 Tz) into a vL x vL square map, and resize it to the original image resolution via bilinear interpolation. To account for preprocessing such as padding and resizing, this interpolation is applied accordingly. final element-wise sigmoid activation is applied to obtain the pixel-level similarity map Mi = (bilinear( My)),, which we refer to as the VL similarity map and use for zero-shot grounding and segmentation. The VL similarity map is derived from the cosine similarity between vision patches and text embeddings, and since we do not modify the embedding space beyond applying f2-normalization and adjusting the temperature, it can be directly interpreted as the similarity between each image pixel and the text. The zero-shot inference process is illustrated in Figure (d).
Experiments
4.1 Training dataset
MIMIC-CXI [16] We train our model using the MIMIC-CXR dataset for VL aligmment. MIMIC-
CXR comprises 377,1 CXR images from 227, 227,835 radiographic studies involving 65,379 patients. Each study includes a radiology report and one or more CXR images in either frontal or lateral views. Images are sourced from MIMIC-CXR-JGG [17] and only the 'finding'' and 'impression' sections of reports are extracted using the official codebase All view positions are considered, and the official dataset split is followed. As described in Section 3.1 finding-
extraction is applied, with each study containing average of 6.45 such sentences. Studies without extracted finding-sentents are discarded, resulting in 352,875 training images and 2,852 for validation.
4.2 Test datasets
Open-I (OI) 7 contains 3,851 radiology reports and 7,470 CXR images with multi-label annotations
for 18 diseases. PadChest (PC) 4 comprises 160,868 CXR images from 67,000 patients, with 192 labels showing long-tailed distribution. Following 19), use 39,053 samples annotated by board-certified radiologists. Additionally, PadChest20 (PC20), introduced in [19], serves as a test set for rare disease evaluation, consisting of 20 classes with fewer than 10 samples each. ChestXray14
(CXR14) (35] provides official test set with 22,433 images labeled for 14 diseases. CheXpert (CXP)
13 includes test set of 500 patients' images annotated by five board-certified radiologists. Following we evaluate classification on five observations: atelectasis, cardiomegaly, consolidation, edema,
19
5

---

and pleural effusion. ChestXDet10 (CXD10) 23 a subset of CXR14, contains 542 images with bounding box annotations for 10 diseases in the official test set. SIIM [38 pneumothorax dataset provides segmentation masks for 11. 11,582 CXRs; we adopt the test split from [34], which includes 1,704 images with 458 positives. RSNA 32 pneumonia dataset consists of 29,700 frontal CXRs
with bounding box annotations; we use the test set from [37 containing 5,337 images, including 1,218 positives. MS-CXR [2] consists of 1,153 image-phrase- bounding box triplets, with images sourced from MIMIC-CXR. The bounding boxes annotated to specific phrases in the report enable more detailed grounding, referred to as phrase grounding. For fair evaluation on the test set 167 images released by (6. where each phrase maps single bounding box, exclude these images from the training set described in Sec. 4.1
4.3 Evaluation metrics
AUC, area under the ROC curve, is adopted to evaluate zero-shot classification multi-label test datasets. Pointing game [41). which determines whether the coordinates of the maximum value falls within the corresponding bounding box, employed as the grounding metric. Dice score serves as a standard evaluation metric for segmentation. Following (37] we compute the Dice score using only positive samples and optimize the segmentation threshold on the test Threshold search intervals are 0.01 for sigmoid and 0.001 for softmax, depending on the feature map's activation function. Pixel-wise AUC (Pix-AUC) computes AUC at pixel-level evaluate the quality of the segmentation probability map. To account for both sensitivity and specificity in mask prediction, we incorporate both positive and negative samples. For fine- meeggriied tasks such as grounding and segmentation, predictions are interpolated back to the original image size before evaluation.
maximize the score.
4.4 Implementation details
We adopt XrayDINOv2 [5] as the pre-trained vision encoder, which was trained in unimodal setting using CXR images based on DINOv2 [27]. While the vision encoder was trained with an image resolution of 224, we increase it to 518 for our experiments. The patch size of 14 x 14 leads to 37 x 37 patches, yielding a vision patch length L of 1369. The text encoder MPNet wall-mpnet--bas "aapettbbseev2") [30,, initialized with pre-trained parameters and further
during training. The trainable Transformer layers consist of two randomly initialized layers (K 2), with a hidden dimension of 768, matching the embedding size of both the vision and text encoders. While the vision encoder remains frozen, all other parameters are trainable. Following 22). the learnable temperature parameter is initialized to log(1 1 /0.07). The details of model training be found in Appendix C.1 The
LLM used for extracting finding-sentences is "Llama-3.33-7 3.3ma-3.3-70B--11 3--00-Innstruct" (1O), deployed in a private computing environment.
Results
5.1 Zero-shot evaluation
Classification. Table compares RadZero with SOTA models on public test datasets. For the five datasets evaluated in CARZero [1), we report their published results. For SIIM and RSNA, independently evaluated two open-source models. RadZero achieved new SOTA performance on Ol and PC, irrespective of image resolution. In the long-tailed PC dataset with 192 classes, RadZero
Open-1 (OI)
PadChest (PC)
PadChest20 (PC20)
ChestXray14 (CXR14)
CheXpert (CXP)
ChestXDet10 (CXD10)
Method
SIIM
RSNA
GLoRIA 12
0.589
0.565
0.558
0.610
0.750
0.645
BioViL.T 1
0.702
0.655
0.608
0.729
0.789
0.708
McdKLIP
32
0.759
0.629
0.688
0.726
0.879
0.713
0.897
0.869
KAD 42
0.807
0.750
0.735
0.789
0.905
0.735
CARZero 19
0.838
0.810
0.837
0.811
0.923
0.796
0.924
0.747
RadZero (224px)
Rad7ero
0.847
0.841
0.871
0.804
0.900
0.787
0..924
0.834
Table 1: Zero-shot classification AUROC scores on public CXR datasets. For fair comparison, we also report the results of low-resolution (224x224) version of RadZero.

---

Method
Mean
ATR
CATC
CONS
НЕР
ЕMPH
FIB
FX
MASS
NOD
PTX
GLoRIA [2
0.367
0.479
0.053
0.737
0.528
0.667
0.366
0.013
0.533
0.156
0.143
KAD E2
0.391
0.646
0.132
0.699
0.618
0.644
0.244
0.199
0.267
0.316
0.143
BioViL-T
1
0.438
0.000
0.630
0.504
0.846
0.026
0.500
0.171
MedKLIP
37
0
0.625
0.132
0.837
0.675
0.734
0.305
0.224
0.733
0.312
0.229
CAR7ero 19
0.604
0.782
0.184
0.700
0.457
RudZero (224px)
0.537
0.604
0.211
0.806
0.813
0.795
0.451
0.197
0.767
0.325
0.400
RadZero
0.646
0.368
0.824
0.857
0.872
0.585
0.250
0.767
0.506
0.543
Table 2: be found in Appendix E.I
Zero-shot grounding results (pointing game accuracy) CXD10. Lesion abbreviations can
outperformed CARZero by 3.1 percentage points, demonstrating strong generalization in zero-shot classification. Notable gains are also observed in PC20, which focuses on rare diseases, suggesting that VL-CABS is particularly effective for infrequent conditions. On datasets where RadZero failed to rank first, MedKLIP performed best on RSNA, while CARZero led on CXR14, CXP, and CXD10. However, MedKLIP underperformed on the latter datasets, and CARZero underperformed on RSNA. In contrast, RadZero showed results comparable to the top-performing models across all datasets. Interestingly, the lower-resolution RadZero (224px) even outperformed RadZero: potentially due to the pre-trained vision encoder, as discussed in Sec. 5.4
The representative classification metric shown in Figure 2 is the average AUC across all datasets. RadZero established new SOTA, outperforming CARZero by 1.5 percentage points, a gain at- tributable to our training strategy that incorporates multi multi-positive contrastive learning to enhance the diversity of both positive and negative samples per image.
Grounding. Table 2 presents zero-shot grounding results on CXD10. We adopted the pointing game scores reported by CARZero for all models except Vio, which we evaluated using its re- leased weights. RadZero achieved the highest average score across all diseases, outperforming CARZero by 0.079. Per-lesion analysis showed that RadZero achieved the best performance in all classes except consolidation, indicating that the proposed VL-CABS ef- fectively captures local alignment between text and image patches regardless of disease type. Furthermore, RadZero efficiently supports higher input resolutions, enabling more precise localization.
Method
MS-CXR
BioViL-T [I]
0.719
McdKLIP 37
0.407
CARZero 19
0,749
RadZero (224px)
0.832
Rad/ero
0.844
Table 3: Zero-shot phrase grounding results (pointing game accuracy) on MS-XXR.
Table 3 reports zero-shot phrase grounding results, evaluating align- ment the phrase level in contrast disease-level grounding in Table 2 Pointing game accuracy is used as the evaluation metric. We evaluated all basclines using publicly available models. RadZero achieved the highest score of 0.844, demonstrating accurate interpretation of text phrases. The strong performance of RadZero (224px) suggests that the gains are largely attributable to the effectiveness of VL-CABS rather than the higher input resolution.
Segmentation. Table 4 summarizes zero-shot segmen- tation results SIIM and RSNA. To benchmark against supervised models, we fine-tuned MGCA with varying proportions of training data; percentages in parentheses indicate the amount used.
RSNA Dice
SIIM
Method
Dice
Pix-AUC
GLoRIA 12
0.347
BioVil. 1
0.439'
MedKLIP
37
0.465
0.044
0.648
Among zero-shot models, RadZero achieved the highest Dice scores on both datasets. On SIIM, it outperformed CARZero by 71 71%, demonstrating superior segmentation capability. The smaller margin on RSNA is likely due to its coarser annotations annotations-bbundinng boxes rather than pixel- which
G2D
0.477'
0.051
CAR7ero 19
0.540
0.100
0.856
CARZero (logits)
0.529
0.081
0.928
RadZero (224px)
0.562
0.121
0.943
RadZero
0.546
0.171
0.947
MGCA [24] (1%)
0.313
0.144
0.752
MGCA (10%)
0.571
0.238
0.856
level masks -wich limit the advantages of RadZero's 's fine-grained VL similarity maps. As in phrase grounding, the results of RadZero (224px) suggest that the superior performance is not merely driven by higher resolution.
MGCA (100%)
0.578
0.305
0.976
Table 4: Zero-shot segmentation results, Val- ues with are from 337 and from
RadZero also remained competitive against fine-tuned models. It outperformed MGCA (1%) on both datasets, demonstrating the effectiveness of zero-shot
7

---

6
TTurc undictads
The DURIO Chao
(b)
012
That nolarma
Thire Mareiiideeo
Figure 4: VL similarity maps of CXR images from CXDIO, representing (a) normal, (b) fibro- sis, and (c) effusion in the right lung. The value at the top-right corner represent the similarity probability 7 between each CXR image and the text prompt (bottom-riihtt corner).
Figure 5: Open-vocabulary semantic segmen- tation results: (a), (b) for findings and (c) for anatomical regions. The CXR images and bound- ing box labels are from CXD10. The segmenta- tion thresholds were set to 0.7 for (a) and (b), and 0.4 for (c).
segmentation. Although it did not surpass MGCA (10%) or (100%), RadZero requires no mask labels, enabling broader generalization beyond fixed vocabularies, further discussed in Sec. 5.3
SIIM's detailed annotations also allow for Pix-AUC evaluation. RadZero achieved the highest score, exceeding even MGCA (10%), indicating well-calibrated VL. similarity maps that distinguish positive from negative regions. In contrast, MedKLIP and CARZero, both relying on attention maps, performed worse with scores of 0.648 and 0.856, respectively. For a fair comparison, we also evaluated CARZero's pre-softmax logits (CARZero (logits)), which improved performance but still fell short of RadZero. Notably, CARZero (logits) underperformed its own Dice score (0.081 vs. 100 despite threshold tuning, as expected from the inconsistent scaling of dot-product similarity. In contrast, VL-CABS directly encode pixel-level text-image similarity. allowing low values for negative samples. This contributed to its superior Pix-AUC, as further supported in Sec. 5.2
5.2
VL similarity map analysis
Figure 4 illustrates that RadZero effectively aligned visual and textual representations with VL- CABS. The outputs, VL similarity map M and probability i, offer both interpretable visualizations and quantitative metrics. For the normal image Figure the model assigned low similarity (0.072) the prompt "There is atelectasis" with dark VL similarity map, indicating weak alignment between vision tokens and the text embedding, In contrast, "There is no atelectasis" (0.921) and "The lungs are clear" (0.846) yielded bright activations across lung fields, reflecting strong alignment.
Figure (b) shows fibrosis, and "There is fibrosis" resulted in high similarity (0.970) with strong activations in the affected lung. Prompts indicating normality received much lower scores (0.093 and 0.014) and darker VL similarity maps, clearly distinguishing abnormal from normal descriptions.
Figure E (c) highlights RadZero's 's ability to distinguish anatomical descriptions. For right-sided pleural effusion, the model assigned high similarity (0.907) to "There effusion," " with bright activations in the correct region. Notably, "There is right effusion" (0.946) scored even higher, indicating accurate localization, while "There left effusion" scored much lower (0. 125) and dark VL similarity map, showing that the model correctly distinguishes between left and right lung regions.
Overall, these results underscore the explainability of VL-CABS. The similarity probability is verifiable at the pixel level, enabling spatially grounded explanations. By explicitly revealing how conclusions are derived, RadZero offers enhanced interpretability in the context of disease diagnosis.
5.3 Open-vocabulary semantic segmentation
Figure presents open-vocabulary semantic segmentation results for both findings and anatomical regions. Segmentation masks were generated by thresholding the VL similarity map M for each text
8

---

Metoed
Similarity
Trzinable
Clandicttion
Goounding
Segmentation
Bayers
PC20
CXR14
CXD10
CXD1
CXR
SIIM
dat-product
214
0.853
0.790
0.422
0.075
0.830
0.363
0.505
0.483
[c]
CoG
Transtommer
0.300
0.793
0.539
(23-pp)
COG
Transformer
0.851
0.579
0.785
0.537
RadZZero
CCM
Tuantormer
518
0.847
0.571
0.304
0.757
0.622
01544
Lincas
0.768
0.775
0.764
0.735
Tranformer
224
0.834
0.866
0.377
0.797
0:901
0.770
0.121
0.174
0,01
CAR7em
Transfomer Dec.
234
0.827
0.115
0.877
0.795
0.589
0.770
0.477
0.743
0.022
Table 5: Ablation study of model architecture components. "MP" denotes multi-positiinn
prompt; in cases of overlapping predictions, the prompt with the highest similarity was assigned. In Figure (a), RadZero successfully localized lesions based on text queries, though some segmentation masks extended beyond ground truth boxes, indicating room for improvement. Notably, certain incorrect predictions captured clinically relevant features that were not explicitly annotated: in Figure (b)) a chest tube was reasonably associated with "pneumothorax." " Figure further demonstrates RadZero's 's ability to segment anatomical structures without supervision. inferring approximate spatial regions from text despite imprecise boundaries. These results highlight the potential of VL-CABS for zero-shot ppen-ooabbuaary semantic segmentation and RadZero's 's capacity to align textual descriptions with medical imagery. Additional qualitative examples are provided in Appendix H
5.4 Ablation Studies
Ablation study RadZero components. 1) Similarity function: Comparing (a) and (b) in Table shows the effect of using cosine similarity instead of scaled dot-product (33] for VL alignment. Cosine similarity, which better aligns with inference-time VL similarity maps, improved classification and slightly enhances grounding. 2) Trainable parameters: (b) and (c) compare a linear layer and a two-layer Transformer, with the image encoder frozen. Transformer layers yielded consistent gains across all tasks, particularly in grounding (0.483 + 0.539) and segmentation (0.078 - 0.099). 3) Multi-positivve pairs: The difference between (c) and (d) lay the use of multi-pone positivooeeeee contrastive pairs. (d) improved classification (c.g.. PC20: 0.860 - 0.879) and segmentation (0.099 + 0.121), highlighting the advantage of richer supervision. 4) Image resolution: RadZero and (d) shared the same architecture, except that RadZero used higher resolution inputs (518 vs. 224). This change substantially improved grounding (0.537 - 0.622) and segmentation (0.121 + 0.171), showing the importance of resouurosottion features for spatially localized tasks. In classification, (d) outperformed RadZero: likely due to the vision encoder (XrayDINOv2[3) being pre-trained at 224 pixels, indicating that using an encoder pre-trained at higher resolutions may further enhance the performance.
Comparison among different VL alignment approaches. Table compares RadZero with alternative VI. alignment methods, keeping all settings identical except for VL feature fusion and loss computation. LiT, which uses a [CLS] embedding for alignment, showed limited classification performance and was incapable of grounding or segmentation. dino. txt improved classification through additional Transformer layers, but its mean pooling constrained grounding and segmentation performance. CARZero introduced a cross-attention decoder, enhancing performance on those tasks. However, when compared the RadZero ablations, (b) outperformed CARZero across most metrics, showing that VL-CABS alone is sufficiently effective.
6 Conclusion
In this work, we introduced RadZero, novel VL alignment model for chest X-ray that achieved strong zero-shot performance in classification, grounding, and segmentation. Central RadZero is VL- CABS, which computes image-text similarity at the patch-level to improve interpretability. Combined with multi-positive contrastive training, VL-CABS enabled effective representation learning without pixel-level annotations, and the support for high-resolution inputs further boosted performance. Extensive evaluations on public chest radiograph benchmarks showed that RadZero outperformed SOTA methods. VI similarity map analysis highlighted the enhanced explainability of VL-CABS by providing transparent rationales for how conclusions are derived. Qualitative assessments further demonstrated RadZero's 's potential for vocabuuuyy semantic segmentation.

---

Despite its impressive results, RadZero has limitations that indicate areas for future research. The observed performance degradation on specific datasets emphasizes the necessity of enhancing general- ization capabiliiy. Its reliance on the pre-trained vision encoder may also restrict domain adaptability. In addition, the current study validates the proposed RadZero training framework only on chest X-ray datasets, which limits the scope of its generalization. Future work could explore extending RadZero to other medical imaging modalities such as CT and MRI, demonstrating its potential as a universal vision-language learning framework adaptable to diverse anatomical and visual characteristics. More- over, applying VL-CABS to general imaging domains, for instance in open-vocabulary semantic segmentation, could be meaningful direction toward building more interpretable VLMs.
Acknowledgments and Disclosure of Funding
This work was supported by the Technology Innovation Program (RS- RDS-2025-022210111 Development of Pedical-Specializee Multimodal Hyperscale Generative AI Technology for Global Integration) funded by the Ministry of Trade Industry & Energy (MOTIE, South Korea).
References
[1] Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Pérez-García Maximilian llse. Daniel C. Castro, Benedikt Boccking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori, Javier Alvarez- Valle, and Ozan Oktay. Learning to exploit temporal structure for biomedical vision-language processing. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 15016-15027. June 2023.
[2] Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C. Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez- Valle, Hoifung Poon, and Ozan Oktay. Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing, page 1-21. Springer Nature Switzerland, 2022. ISBN 978303 1 1200555, doi:
3-177788---0--00000000 5_1. URL
http://dx. doi org/ 1007 /978-3 3-031-20059--3 2_1
[3] Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer. Stephanie Hyland, Maria Wetscherek. Tristan Naumann, Aditya Nori, Javier Alvarez-Valle al. Making the most of text semantics improve biomedical vision-language processing. In ECCV, pages 1-21. Springer, 2022.
[4] Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria de la Iglesia- Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical Image Analysis, 66:101797, 797, December 2020. ISSN 1361-8415.
doi: 10.1016 me..001/j.media.220 m00.a.2020.a...mmmmmmm 101797. URL |RL|http: hhtt:://xx...e. doi. 0r...0..11000000 1016/
202d..20022..0 101797
[5] Pierre Chambon, Jean-Benoit Delbrouck, Thomas Sounack, Shih-Cheng Huang, Zhihong Chen, Maya Varma, Steven QH Truong, Chu The Chuong, and Curtis P Langlotz. Chexpert plus: Hundreds of thousands of aligned radiology texts, images and patients, arXiv preprint arXiv:2405. 195388 2024.
[6] Zhihao Chen, Yang Zhou, Anh Tran, Junting Zhao, Liang Wan, Gideon Su Kai Ooi, Lionel Tim-Ee Cheng, Choon Hua Thng. Xinxing Xu, Yong Liu, al. Medical phrase grounding with region-phras context contrastive alignment. International Conference on Medical Image Computing and Computer- Intervention, pages 371-381. Springer, 2023.
[7] Dina Demner-FFushin Marc D Kohli, Mare Rosenman, Steven E Shooshan, Louis Rodriguez. Sameer Antani, George R Thoma, and Clement McDonald. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304-100 Mar 2016. doi: 10. 1.. 0093//amia/ovv000
[8] Kunio Doi. Computer-aided diagnosis medical imaging: Historical review, current status and future potential. Computerized Medical Imaging and Gruphics, 3184)1198-21, 2007. ISSN 0895-61 doi: hups: wttg///01111....oompeceecoo
00000022.002 URL URL / //www//wwwww//o www.:ssceececccccttccce com.science/ artzadee/pi/$09961110000066 Computer-aided Diagnosis (CAD) and Image-guided Decision Support.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, G Heigold. S Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2020.
10

---

[10] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman. Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llaa 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[11] Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205. 142004 2022.
[12] Shih-Cheri Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal global- local representation learning framework for label-efficient medical image recognition. In ICCV, pages 3942-33 3921, 2021.
[13] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Tleus, Chris Chute, Henrik Mark- lund, Behzad Haghgon, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong. Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis Langlotz, Bhavik N. Patel, Matthew P. Lungren. and Andrew Y. Ng. Chexpert: A large chest radiograph dataset with uncertainty labels and expert compari- son. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):550 597)))955999999900 Jul. 2019. doi: 10. 1609/aaai.v33000 /t0///aaiiv33001.33315550 33301590.
URL https://ojs aaai wow.ooog/noioooe index. hhtp:/AAT//rticle//aaaaae
[14] Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong. Tan Bui, Pierre Cham- bon, Yuhao Zhang, Matthew P. Lungren, Andrew Ng. Curtis Langlotz, and Pranav Rajpurkar. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv: 106. 14463, 2021.
[15] Mohammad Jamshidi, Ali Lalbakhsh, Jakub Talla, Zdenök Peroutka, Farimah Hadjilooei, Pedram Lal- bakhsh, Morteza Jamshidi, Luigi La Spada, Mirhamed Mirmozafari, Mojgan Dehghani, Asal Sabet, Saeed Roshani, Sobhan Roshani, Nima Bayat-Makou, Bahare Mohamadzade, Zahra Malek, Alireza Jamshidi, Sarah Kiani, Hamed Masshemi-Dezake and Wahab Mohyuddin. Artificial intelligence and covid-19: Deep learning approaches for diagnosis and treatment. IEEE Access, 8095581 -109595, 2020. doi: 10. 1001119/SSSSSS.O 2020,300197,
[16] Alistair EW Johnson, Tom J Pollard. Seth Berkowitz. Nathaniel R Greenbaum, Matthew P Lungren. Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxt, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1 )(1):317, 2019.
[17] Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng. Zhiyong Lu, Roger G Mark, Seth Berkowitz, and Steven Horng. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.0704. 2019.
[18] Cijo Jose, Théo Moutakanni, Dahyun Kang, Federico Baldassarre, Timothée Darcet, Hu Xu. Daniel Li. Mare Szafraniec, Michaël Ramamonjisoa, Maxime Oquab, et al. Dinov2 meets text: A unified framework for image-and pixel-level vision-language alignment. arXiv preprint arXiv:2412.16334, 2024.
[19] Haoran Lai, Qingsong Yao. Zihang Jiang. Rongsheng Wang, Zhiyang He, Xiaodong Tao, and S Kevin Zhou. Carzero: -ooss-attention alignment for radiology zero-shot classification. In CVPR, pages 11137-11144 2024.
[20] Janghyeon Lec, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. Uniclip: Unified framework for contrastive language-image pre-training. NeurlPS, 15:1008-1019, 2022.
[21] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4) 1240, 09 2019. ISSN 1367-4803. doi: 1...0///boiiffrmatii//22262...2. URL /sttp:://do..orgg// 1093 htt.////oooooooortt////////aaaa
[22] Che Liu, Cheng Ouyang, Sibo Cheng, Anand Shah, Wenjia Bai, and Rossella Arcueci. G2d: From global to dense radiography representation learning via vision-language pre-training. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Ad- vances in Neural Information Procesing Systems, volume 37, pages 14751-14773. Curran Asso- ciates, Inc., 2024. URL httpps::/proceeeinngg www.ipe.rrcc.ooo http:/cccppppeppeeeeeeeeeeeeeeee 1ac1444222aaaabaa444222222222222 horpaaaeeeeeepa-aeeeeee----reeee
Conferrr-ooooeecneee pdf
[23] Jingyu Liu, Jie Lian, and Yizhou Yu. Chestx-det10: Chest x-ray dataset on detection of thoracic abnormali- ties. arXiv preprint arXiv:2006.. 10550, 2020.
[24] Dwarikanath Mahapatra, Behzad Bozorgtabar, and Zongyuan Ge. Medical image classification using generalized zero shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 3344-3353, October 2021.

---

[25] Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Langlotz, and Dan Jurafsky. Improving factual completeness and consistency of image-to-text radiology report generation. In Kristina Toutanova, Anna Rumshisky, Luke Zeit Dilek Hakkani-Tur, Iz Bellagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5288 5304, Online, June 2021. Association for Computational Linguisties. doi: 10. 18.188 1/233/11/221.// .aaacll main.416. URL.http::/ a http://aacaaahoooo....oooo org/ 2021 maacl-main 416/
[26] Aaron van den Oord, Yazhe and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint wwXivv:807.03744 2018
[27] Maxime Oquab, Timothéc Darcet, Théo Moutakanni, Huy V Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL attps://openreviiet zet/foruumiid=aBBStttzZ
[28] Fernando Careia, Harshita Sharma, Sam Bond- Bond-Taylor, Kenza Bouzid, Valentina Salvatelli, Maxim- ilian Isse, Shruthi Bannur, Daniel C. Castro, Anton Schwaighofer, Matthew P. Lungren, Maria Teodora Wetscherek, Noel Codella, Stephanie L. Hyland, Javier Alvarez- Valle, and Ozan Oktay. Exploring scalable medical image encoders beyond text supervision. Nature Machine Intelligence, 7(1): 101):119-138: 19-030, Jamuary 2025. ISSN 2522-5839.
doi: 10. 01986442256-024-00966
URL hhtt:://dx..... doi org/10. 1038/
#42256- 024 0212065-2--0965--- 00965-M
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR, 2021.
[30] Nils Reimers and Tryna Gurevych. Sentence-BERT: Sentence emheddings using Siamese BERT-nctwwrks.. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 9992-399, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/ 1/119-1//111114/// 1410. https:/:: htt:://acllaat aclathology org/D19- 1410/
[31] Min Shi, Fuxiao Liu, Shihao Wang. Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang. Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. Eagle: Exploring the design space for multimodal L.MS with mixture of encoders. In ICIR, 2025. URI htt./attt//ppeerrriiii
[32] George Shih, Carol C Wu. Safwan S Halabi, Mare D Kohli. Luciano M Prevedello, Tessa S Cook, Arjun Sharma, Judith K Amorosa, Veronica Arteaga, Maya nett/oormmmdd-vaWee.. institutes of health chest radiograph dataset with expert annotations of possible pncumonia. Radiology. Artificial intelligence, 1(1), 2019.
et al. Augmenting the national
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors. Advances volume 30. Curran Associates. Inc., 2017. ttt/tttpp//prrceedinng wtw.eeeddinnnnnuuuss.c.m files/ 0tt0:////.1e//eeee////////100000
Advaaces in Neural Information Processing Systems,
[34] Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti, and Lequan Yu. Multi goaliigranullriityy cross- modal alignment for generalized medical visual representation learning. NeurIPS, 35:33536-33549, 2022.
[35] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M. Summers. Chestx-ray8: Hospital-scal scalle chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3462-3471, 2017. doi: 10. /119////R..00......
[36] Yancheng Wang. Rajeev Goel, Utkarsh Nath, Alvin C Silva, Teresa Wu, and Yingzhen Yang. Learning low-rank feature for thorax disease classification. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL htttpt://///eeeteeeeet
[37] Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Medklip: Medical knowledge enhanced language-iman pre- preinigg for x-ray diagnosis. In ICCV, pages 21372-21383, 2023.
12

---

[38]
Anna Zawacki, Carol Wu. George Shih, Julia Elliott. Mikhail Fomitchev, Mohannad Hussain, ParasLakhani.
Phil Culliton, and Shumxing Bao. Siim-acr pneumothorax segmentation. httpa: /ttpa://kaggL: kaggle. com/
competitions/ ssim a secc--nneuotthoo--seeeeeee
2019.
[39] Xiaohua Zhai, Xiao Wang. Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked- locked-image text tuning. In CVPR, pages 18123-18133, 2022.
[40] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. ICCV, pages 11975 11986, 2023.
[41] Jiamming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126(10):1084-1100 2018.
[42] Xiaoman Zhang, Chaoyi
Zhang, Weidi Xie, Yanfeng Wang. Knowledge-enhanced visual- language pre-traine me-training on chest radiology images. Nature Communications, 14(1):4542, 2023.
[43] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pages 2-25, PMLR, 2022.
[44] Hong-Yu Yu Zhou, Xiaoyu Chen, Yinghao Zhang, Ruibang Luo, Liansheng Wang, and Yizhou Yu. Generalized radiograph representation learning via cross- supervissn between images and free-text radiology reports. Nature Machine Intelligence, 4(1) 40, Jan 2022. ISSN 2522-5839. doi: 10. 10388844225--021-00455--99 https:/:s/// org. 1038/ #42256 021 0042559
[45] Hong- Yu Zhou, Chenyu Lian, Liansheng Wang, and Yizhou Yu. Advancing radiograph representation learn- ing with masked record modeling. In The Eleventh International Conference Learning Representations, 2023. URL htt.https: BuR.hatppe///peerrviies net net/fouum?ddd X7U26GMM7
13

---

A Are Attention Maps of Medical VLMs Fully Explainable?
Explainability is critical for clinical deployment of deep learning models, and medical VLMs have adopted attention maps as their de facto interpretable features [37 19 42). However, attention maps alone reveal only where the model is focusing, not why it makes a particular prediction. Without the underlying image-text similarity scores, such heatmaps lack interpretability. Interpreting attention maps for complex text queries can be unintuitive and may depend access to ground-truth labels.
To illustrate the challenges of interpreting attention maps, Figure 6 visualizes the attention outputs of CARZero [9 for different image-text text pairs, with the overlaid values indicating the corresponding sim- ilarity probabilities. (a), the prompt "There is right effusion" produces a heatmap focused on the right lower lung and a high similarity probability (0.560), intuitively linking the attended region to the predicted finding. In (b), when queried with "There is left ef- fusion" on the same image, the attention map still highlights the right lower lung. While we can observe where the model attends, the attention map alone offers no clear explanation for why that region is rel- evant. The image-text similarity probability for the prompt "There is left effusion" is low (0.048), which-when considered alongside the attention map-can be interpreted indicating that the attended right lung region does not support the presence of left-sieed effusion. However, how should we interpret the fact that the model attends to the right lung when queried about left effusion? Making sense of this behavior requires access the ground truth: effusion is present in the right lower lung but absent on the left. With this context, one might infer that the model correctly identifies effuuson on the right, recognizes that it does not match the left-sided query, and therefore assigns a low similarity score.
Thery ls right offisiion
lef officion
0.560
(a)
(b)
Figure 6: Attention maps from CARZero [19]. Image-text text similarity probabilities are ob- tained by applying sigmoid function the classification logits.
This example underscores the need interpret attention maps in conjunction with image-text similarity scores. For complex text queries, meaningful interpretation often depends on knowing the ground truth-a a significant limitation in medical VLMs where such labels are frequently unavailable. This reliance undermines the standalone explainability of attention visualizations. To address these limitations, we propose vision-language cross-attention based on similarity (VL-CABS), framework that enables transparent inspection of the vision- language decision process. Detailed examples and analysis of its application in RadZero are provided Sec. 5.2
B Ablation Studies
We conducted ablation studies to assess the impact of key design choices by selectively modifying parts of our approach. The zero-shot tasks for evaluation included classification (class.), grounding (ground.), phrase grounding (phrase.), and segmentation (seg.). Classification was tested on the PadChest dataset, known for its highly imbalanced (long-tailed) label distribution, with AUC as the evaluation metric. Grounding and phrase grounding were evaluated using the pointing game on ChestXDet10 and MS-CXR test sets, respectively. Segmentation performance was measured by the Dice score on the SIIM dataset. For the ablation study, the default batch size and maximum number of epochs were set to 128 and 10, respectively.
View position. Table 6 shows the performance variation based on the view position of CXR im- ages. We compared two models: one trained exclu- sively on frontal view images from MIMIC-CRR and another trained on both frontal and lateral views. The model trained on all view positions consistently outperformed the frontal-only model, suggesting that it effectively learned to interpret lateral images, enhancing overall robustness.
View Position
class.
ground.
phrase.
seg.
Frontal
0.831
0.604
0.838
0.161
All View
0.841
0.622
0.844
0.171
Table 6: Impact of view position.
14

---

Eacoder
Vision
Image Resolation
class.
ground.
phraze.
seg.
Model
class.
ground.
phorase.
seg.
DINO22 E7]
518
0.825
0.6066
0.814
0,100
Linear
0.826
0.549
0.826
0.100
RadDINO
2X
518
0.850
0.630
0.844
0.144
Transformer layer
0.835
0.158
XraD
224
0.841
0.548
0,832
0.118
2 Transformer layers
0.841
0.622
0.844
0.171
XNOOx2
518
0.841
0.622
0.844
0.171
Table 8: Impact of trainable vision layer.
Table 7: Impact of vision encoder and resolution.
Batch size
class.
ground.
phrase.
Text Encoder
class.
ground.
phrase.
seg.
64
0.835
0.583
0.826
0.165
128
0,840
0.594
0.850
0.177
BioBERT
0.842
0.582
0.832
0.127
256
0.841
0.622
0.844
0.171
MPNet [30
0.841
0.622
0.844
0.171
Table 10: Impact of batch size.
Table 9: Impact of text encoder.
Vision encoder and resolution. Table 7 presents the impact of the vision encoder and image resolution on model performance. We compared DINOv2 (27], RadDINO (28) and XrayDINOv2 [S] using image resolutions of 224 and 518. DINOv2, which was trained on natural images rather than X-rays, exhibited relatively lower performance, as expected due to the domain mismatch. Comparing XrayDINOv2 resolutions of 224 and 518, we observe that higher image resolution improves fine-grained tasks such as grounding and segmentation. RadDINO and XrayDINOv2 showed similar performance, suggesting that our approach is effectively applied models trained with the DINOv2 strategy on chest X-ray images.
Trainable vision layer architecture. Table presents the impact of different trainable layers the image encoder. The commonly used linear layer showed relatively lower performance across tasks. In contrast, two Transformer layers achieved the best results across all tasks. Based on this observation, RadZero was designed with two Transformer layers added to the vision encoder. This improvement is likely due to the Transformer's ability to attend to all patch embeddings, capturing richer semantic information.
Text encoder. Table 9 presents the performance of different text encoders used during training. We compared MPNet (30 and BioBERT (21). where BioBERT was fine-tuned clinical reports by CARZero [9]. While MPNet showed slightly lower performance in classification, it achieved notable improvements phrase grounding and segmentation, demonstrating its effectiveness in tasks requiring fine-grained text-image alignment.
Batch size. Table 10 presents the impact of batch size on model performance during training. To ensure fair comparison, we maintained a consistent total number of training steps by adjusting the number of epochs: for batch size of 64, 10 for 128, and 20 for 256. We observed that batch size of 64 resulted in lower performance across all tasks. While the model trained with batch size of 128 performed reasonably well, its zero-shot grounding performance was notably lower than that of the 256 batch size model. As result, we selected 256 as the final batch size. This trend aligns with the well-known impact of batch size in contrastive learning, where larger batch sizes generally improve representation learning by providing more diverse negative samples, leading to better alignment and discrimination.
C Model Training and Computational Details
C.1 Training Configuration
RadZero is trained for 20 epochs with an early stopping patience of 5 epochs, selecting the best model based on validation loss. We employ the AdamW optimizer with a learning rate of 0.0001, following a cosine decay scheduler, with 50 warm-up steps, weight decay of 0.05, and gradient clipping set to 1.0. Training is conducted with global batch size of 256 using distributed data parallel (DDP) on four Nvidia H100 GPUs for 13 hours.
15

---

C.2
Resolution Trade-off Analysis
RadZero employs high-resolution images of 518 px instead of 224 px, which naturally increases the computational cost. Table [T summarizes the trade-off between performance and resource usage across different image resolutions. The columns with downward arrows (4) indicate that lower values are better.
GPU Memory (GB)
Training GPU hour.
Latency.
Throughput
Cls
Grad
Seg
Method
Training
Inference
(ms/img)
(img/s)
AUC
ACC
DICE
RadZero (224px)
25.13
2.20
40
60.18
733.56
0.852
0.537
0.342
Rad7ero
73.36
2.38
52
70.05
94.93
0.851
0.622
0.359
Table 11: Performance-cosi trade-off across different image resolutions.
Both models were trained using four GPUs with batch size of 64 per device (total batch size of 256). For inference, memory usage and latency were measured with a batch size of 1. Throughput (images per second) was measured using the largest power-of batch size that fits into GPU memory for each model: 4096 for the 224 px model and 256 for the 518 px model. Note that throughput (img/s) and latency (ms/img) are not exact reciprocals because throughput is measured under a large-batch setting where computation is parallelized across samples, while latency reflects the time required to process a single image without such parallelism. All computational cost measurements were conducted on Nvidia H100 GPUs using the ChestXray 14 dataset. The reported performance represents the average over all datasets for each task. The results indicate that the additional computational cost is while trade-off given the substantial improvement in fineggraeedd performance.
D Detailed Classification Analysis
D.1 Per-Finding Classification Performance
Tables 12 14 compare RadZero and CARZero 19 in terms of per-finding classification AUCs on the OpenI. PadChest, and ChestXray 14 datasets, respectively. For PadChest, which has large number of classes, we evaluated on five representative categories commonly used. The full names for each abbreviation are provided in Table 16
Mehusl
Alan
ATE
CAIDS
INF.
MASS
NG0
PIX
BUR
EMPR
F10
HORN
FX
DEPCC
LES
Lazten (1)
1387
1338
02012
20877
1030
0190
0302
0053
8726
8376
234ps)
0.890
8.991
1L910
6/989
8764
04000
Table 12: Class-wise disease classification results on the OpenI dataset.
Method
Mean
ATE
CARD
CONS
EDE
PNA
CarZero [9
0.810
0.835
0.906
0.903
0.971
0.841
RadZero(224px)
0.841
0.839
0.917
0.902
0.973
0.846
RadZero
0.841
0.839
0,920
0.899
0.972
0.831
Table 13: Class-wise disease classification results on the PadChest dataset.
Method
Mean
ATE
CARD
EFF
INTL
MASS
NOD
PNA
PIX
CONS
EDE
EMPH
FIB
PLTTI
HERN
Ca
0.718
0.786
0.807
0.796
0.864
0.857
0.672
0.859
0.742
0.772
0.873
0.784
0.883
0.631
0.907
0.789
0.963
RadZeero
0.804
0.792
0.363
0.854
0.669
0.837
0.727
0.766
0.875
0.784
0.881
0.655
0,306
0.781
0.963
Table 14: Class-wise disease classification results on the NIH ChestXray 14 dataset.
Consistent with the average performance, RadZero generally outperforms or matches CARZero on OpenI and PadChest. On ChestXray14, our model performs comparably or better on most pathologies, but notable drop on emphysema (EMPH) accounts for the overall underperformance on the dataset. However, RadZero outperforms CARZero on EMPH OpenI, indicating that drop is dataset-driven rather than due to lesion-specific modeling issues. These results reaffirm that while RadZero demonstrates strong zero-shot classification, further improvements in generalization remain important direction.
16

---

D.2 Comparison with Supervised Baselines
Table 13 compares our zero-shot classification performance against supervised baselines reported in prior work on the CheXpert and ChestXray14 14 datasets. For the baselines, the percentages in parentheses indicate the proportion training data used for supervised training. Since RadZero is trained solely on MIMIC-CXR, its external test results CheXpert and ChestXray14 are reported as single zero-shot scores rather than separate results for 1%, 10%, and 100% of the training data. Note that the results for CheXpert differ from those presented in Table 1] as a different test set was used. Specifically, we followed the test split adopted [45] to ensure a fair comparison with other models. Although both RadZero and RadZero((224x) underperform some fully supervised models trained with 10% or 100% labeled data, they consistently outperform models trained with 1% supervision. These results demonstrate that RadZero achieves competitive zero-shot classification performance, consistent with the segmentation results in Table
CheXpert
ChestXray 14
Method
(1%)
(I0%)
(100%)
(1%)
(10%)
(100%)
VIRT 43
0.870
0.881
0.881
REFERS 44
0.872
0,881
0.882
М3AE 11
0.862
0.873
0,879
MGCA 33
0.888
0.891
0.897
MRM 45
0.885
0.885
0.887
0.794
0.840
0.859
0.888
0.807
RadZero
0.889
0.804
Table 15: Comparison of zero-shot classification performance RadZero against supervised baselines on CheXpert and ChestXray 14 datasets. Results for models marked with an asterisk (*) are taken from whereas the results for all other models are reported their respective papers. The percentages in parentheses indicate the proportion of training data used for supervised training.
45
E
Additional Details
E.1
Abbreviations
Table T6 lists the abbreviations used in this paper for lesions and anatomical regions. The left column shows the abbreviated terms, and the right column gives their description.
Abbreviation
Description
ATE
Atelectasis
CALC
Calcification
Abbreviation
Description
CARD
Cardiomegaly
UL
Upper Lobe
Calcified Granuloma
Mid Lobe
CONS
Consolidation
Lower Lobe
EDE
Pulmonary Edema
Relav
Right Clavicle
EFF
Blfusion
Lelav
Left Clavicle
EMPH
Emphysema
RULZ
Right Upper Lung Zone
Fibrosis
RMLZ
Right Mid Lung Zone
Fracture
Right Lower Lung
HERN
Hernia
LULZ
Left Upper Lung Zone
INFL
Infiltation
LMLZ
Left Mid Lung Zone
LES
Lesion
LLLZ
Left Lower Lung Zone
LG
Lung Granuloma
RCPA
Right Costophrenic Angle
MASS
Mass
LCPA
Left Costophrenic Angle
NOD
Nodule
HD
Hcmidiaphragm
OPAC
Opacity
RHD
Right Hemidiaphragm
PLTH
Pleural Thickening
LHD
Left Hemidiaphragm
PNA
Pncumonia
Pneumothorax
(b) Anatomical region abbreviations
(a) Lesion abbreviations
Table 16: Abbreviations for lesions and anatomical regions.
17

---

E.2
Prompt for finding-sentena extraction.
As shown in Figure 7 the prompt instructs the LLM to extract clinically relevant minimal semantic units in the form of sentences from radiology reports. Finding-senteneee are standardized through prompt alignment to follow a "There is" format, with a one-shot example enhancing extraction accuracy and guiding the model to identify both findings and their corresponding anatomical locations in structured manner.
You expert medical ascatint Al spcializing in undentanding and inalysing theal aemy malioloy
Your taok extret the motically significant and meaningful findings frrm given chest report focusing idensifying plesses expresions fhat deseribe notable cunditions abnemallttis Not that the report may refarenes previnu studio, Sut only need intergretation hood on the choe Therefore, remove and rewite tome like "new", 'imprvedf "mncheggal' "wonenel" "cumsisten" reflect the curmant data in that indicato the condition ctists alserved this wilhout implying comparion
imagee studiio.
The templaee fermat includes "There in [finding] of [lecation)"
"Thare may be [lasting=[( "There no [foxding] of [Jonsion]"
[finding] represents the extractd key findings from the nadiology report, and [Pecation] represents tee unstomical location mentimee in the repoet. If location provided, do not include in the sutput Adhare stricll to the following IS0N formut for the final sutzu, ssing eumple guidline for the detieed analysi strusture. Do nat powvide explanations nciput ISCN format the reuet does crntain any findings adpu list (etumpic: ("Ending sentmnz DI
[Esample]
INPUT:
edema. Left revecanlias reacily has substanillly impemval, lideyy combination of atclectooi and elTusion more conflum the right ling has: periies, and could he due to aymmettrcally roulving odems, but preumonis should be considerel in the appropeiat clinical sctting Small right pleunal effisien likey unclanged, with pigtail pleural catheter commining place and so visible pacumatlannu.
Cardimegally is ausompanied by improving gulmonary vasenlar congeation and decressing
OUTPUT:
"Ending sentenee i
"There canliennualy with pulloverryy vasculte congettion" 'Thae pulmorery olems", "There is left retrocandiae opucily", "There be alcletisss "There may ciluiin"" "There is right ling lass spacity" 2 "Thrre right basc opucity suggotive of puscill paeumonia', "There be small right plauni ellusion", "There piguail pleunal cathteer is place", "There preumothoras"
Figure 7: Prompt design for extracting finding-sentences with LLM.
F Linguistic Robustness and Generalization Analysis
To evaluate the linguistic robustness of RadZero to variations in report tem- plates and phrasing patterns, we con- ducted series of qualitative analy- ses using similarity search within a vector database of training sentences encoded by RadZero's ss text encoder. In Table 17 (a), given query sen- tence "the lungs are clear", the model retrieved multiple semantically simi- lar sentences with varied phrasing and syntactic structures. These results in- dicate that the encoder captures se- mantic equivalence beyond fixed tex- tual templates, such as recognizing "the chest clear" or "the airways are clear" as close variants.
Query
Retrieved (Top-A)
Similurity
The lungs are clear
1.000
Lungs clear
0.991
hungs are clear
sirways There clear lungs
0.993
0.985
There are clear lungs The chesi is clear
0.983
0.982
is pleural ettusion
right lung
0.993
There is pleural efiuuion the right lower lung
0.978
(b) pleural effusion in the right lower lung
There is pleural effusion of the right lower lung
0.976
There effusion the right lower long
0.968
is pleueal effusion the
ling
is right lower lung pleural effassion
0.953
There fibrexis
1,000
probuble fihrosis
0.957
(c) there fibrueis
There is chronic fibroeis
lung fibrosis
0.927
0.934
There is compunent of fibrosis
0.924
There is fibretic disesse
0.917
(d) pleural effusion in RLL (unseen)
There is pleural effusion with drainage There is slight decreuse in pleural fluid
0.753
0.725
There collecting pleural stadd
0.708
We further tested syntactic diversity using queries with complex structures (c.g.. "pleural effusion in the right lower lung"). As shown in Table 17 (b), the retrieved results included ex- pressions such as "right lower lung pleural effusion" and "pleural effusion of the right lower lung," demonstrating the model's robustness to grammatical variations. Similarly, queries evaluating lexical flexibility (e.g.. "there is fibrosis" in Table T7 (c)) showed that RadZero identifies semantically related expressions such as "chronic fibrosis" and "fibrotic disease," suggesting strong generalization across vocabulary variations. These
Table 17: Examples of similarity search results using RadZero's text encoder. Cell colors indicate cosine simi- larity: high (blue) 2 0.95, medium (yellow) 2 0.85, and low (red) < 0.85.
18

---

findings collectively suggest that RadZero is not confined to specific textual templates or vocabularies and generalizes well across diverse linguistic patterns, thereby mitigating the risk of overfitting to particular report styles.
However, observed that embeddings may be less accurate for report expressions that absent in the training data. One notable case involves abbreviations-for example, "RLL" (right lower lung)-which occurred infrequently in the original reports. As shown in Table 17 (d), such unseen abbreviations tend to yield weaker semantic alignment. To address this limitation, non-standardized or abbreviated expressions can be rewritten into complete finding-sentens sentences using LLM-bbsed normalization, which helps maintain robustness across diverse report formats.
G Statistical Significance of Main Results
To assess run-to-run variability and verify the statistical reliability of the reported performance, we conducted additional experiments across five random seeds for all tasks. For each model and task, we report the mean and standard deviation of the evaluation metrics.
Method
Clacciiation
Grounding
Segmentation
PC
PC20
CXR14
CXP
CXD10
SIIM
RSNA
CNDIU
MS-CXR
SUM
RSNA
RudZero
0.847
0.841
0.871
0.804
0.900
0.787
0.924
0.844
0,622
0.844
0.171
0,546
Mean (t))
B.54K (th)
(.840 (#))
0,809 (t))
0,802 (+)
0,899 (t))
0,787 (+))
0,918
0,843 (=)
0.601 (=))
0.845 (:t)
0.164 0,080
0.549
sid
0.0066
00011
(10140
0.0016
0L0019
0,0043
010063
0,0072
0,030
0.0133
0/023
Table 18: Mean and standard deviation of main results over five runs.
is relatively small, indicating that the observed improvements are consistent and not due to random fluctuations. These results confirm the stability and robustness of RadZero across different random initializations.
As shown in Table 18 the variance
H Additional Visualization Results
Figure 8 depicts segmentation of anatomical re- gions, which, while not perfect, generally align with appropriate locations. Figure presents ex- amples demonstrating RadZero's potential for open-vocabulary semantic segmentation, includ- ing additional lesion types such as mass, fibro- sis, and caleification. Full names of lesion and anatomical region abbreviations are provided in
(n)
Table 16
Figure TO presents VI similarity maps for 10 different findings of the ChestXDet10 dataset, following the pipeline in Sec. 5.2 The brightest regions in the map align well with the bound- (b) ing boxes, even for multiple or small lesions. The similarity probability was above 0.5 for all findings except calcification. While the model correctly localized caleifications, the activated regions appeared as small bright spots, leading lower similarity probability of 0.45 due to the weighted sum calculation. This highlights limitation of RadZero, suggesting the need for further refinement in future work.
Figure 8: Open-vocabulary semantic segmenta- tion for anatomical regions. The CXR images are sourced from Open-I. The segmentation threshold was set to 0.4.
19

---

Figure 9: Open-vocabulay, semantic segmentation for findings. The CXR images and bounding box labels are sourced from ChestXDet1O. The segmentation threshold was set 0.7.
Atelectacis
Caleifiration
Convolidation
Effosion
Emphywrma
8.748
8.458
0.810
0.786
0.963
Fibrests
Frncture
MasS
Nedule
Pnemmotheras
0.768
0.999
0.940
0.747
Figure 10: Visualization of VL similarity maps generated by RadZero for 10 findings the ChestXDet10 dataset. Red boxes indicate ground truth bounding boxes. The similarity probability is shown in the top-right corner of each map.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the paper's s scope by highlight- ing RadZero and its core component, VI.-CABS, which computes text-image similarity for interpretable, fine-grained vision-language alignment. They also clearly describe RadZero's zero-shot capability across multiple tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper explicitly discusses RadZero's limitations in each section including
Sec. 6 Conclusion: performance gaps certain datasets (Sec. (.1), prediction of regions extending beyond the ground truth bounding box in open-vocabulary segmentation (Sec.
5.3) and reliance on pre-trained vision encoder (Sec. 5.4), suggesting directions for future improvement.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex- system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledger in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
21

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This work mainly includes empirical contributions.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed experimental configurations in Sections and Appen- dices C.1
R.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived
well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
22

---

5. Open access to data and code
Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Our experiments all conducted publicly accessible datasets, and the details about the datasets used are described in Sections 4.1 (training data) and (test data). For experiment implementation, follow the official code of exisiting works, all code can be found in their official GitHub repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss:/niis.cc))
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. httpssccccpbccccuucueeee hinooi////dddds//deeuuiiiiiii Sww.///ooeiiissiiiiiiaia..a iimiisssionl asionoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed for experimental configurations in Section 4.4 and Appen-
dices H.2 and for the data split in Sections 4.1 (training data) and 4.2 (test data).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report mean and standard deviation across five random seeds in Appendix G ensuring statistical significance and reliability of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The information on the computation resources is described in Appendix C.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps:/nneurips ptt.eeiiiieeeeerrre.......oo..le httppp/ssss//ppbbc//hccccccaaaaa
Answer: [Yes]
Justification: This research was conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper diseuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The potential societal impacts mentioned in Sections and

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deptoyment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not release data or models that have high risk for misuse. Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual- should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer to Sections 4.1 and 4.2
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the patasets.ppaerssiii paperswi thcode. datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
There no new assets released this work.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work has no human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work has no human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: This work have used a publicly available LLM for data processing. To prevent any potential misuse, the LLM was executed secure, private server.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neura) (ttt::e//eearrrgg.ccc/o/ooeeeeer / for what should or should not be described.
27


Reviews:
[{"review_id": "mc9Ydcv57p", "reviewer": "Reviewer_etwN", "summary": "The paper presents RadZero, a novel vision-language alignment model for radiology, featuring VL-CABS, a similarity-based cross-attention mechanism that aligns text embeddings with local image features for interpretable, fine-grained reasoning. Using large language models, RadZero extracts concise semantic sentences from radiology reports and employs multi-positive contrastive learning to align images with multiple textual descriptions, enabling robust representation learning without pixel-level annotations. With a pre-trained vision encoder and trainable Transformer layers, it efficiently handles high-resolution chest radiographs. RadZero achieves state-of-the-art performance in zero-shot classification, grounding, and segmentation on public benchmarks, with VL similarity maps enhancing explainability and supporting open-vocabulary semantic segmentation.", "questions": "1、 The model’s reliance on pretrained vision and language models may limit adaptability and introduce biases. Can this dependency be reduced? Consider self-supervised learning or partial fine-tuning and compare performance (e.g., accuracy) with and without pretrained weights. \n\n2、Multi-positive contrastive learning risks overfitting to specific report formats. How can this be addressed?\n\n3、 VL-CABS similarity maps enhance interpretability, but are they clinically useful? I suggest a small clinician user study to validate their impact on diagnosis (e.g., diagnostic accuracy). \n\n4、High-resolution image processing increases computational cost. How can efficiency be optimized? Explore model compression (e.g., pruning) and report performance (e.g., accuracy) versus resource use (e.g., inference time).", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**：\n\n1、Enhanced Interpretability: RadZero's VL-CABS mechanism generates pixel-level similarity maps by computing cosine similarity between text and image features, offering transparent and fine-grained visual reasoning that significantly improves clinical interpretability compared to traditional attention maps.\n\n**Weaknesses**：\n\n1、Dependence on Pre-trained Models: RadZero relies on pre-trained vision (e.g., XrayDINOv2) and language models, which may limit adaptability to new data distributions and introduce biases from pre-training data.\n2、Limited Generalization: Despite strong performance on public chest radiograph benchmarks, RadZero underperforms on certain datasets, indicating constrained generalization to diverse or uncommon cases.\n3、Risk of Overfitting: Multi-positive contrastive learning may lead to overfitting to specific report formats or vocabularies if training data lacks diversity, potentially reducing performance on varied report structures.", "was_revised": true, "final_justification": "After careful consideration, I still find myself unconvinced of the paper's novelty. While I acknowledge that the work presents a reasonable application of existing methods (e.g., multi-positive contrastive learning), it does not introduce a fundamentally new algorithm. Thus, I have decided to maintain my original score.", "rebuttal": "We appreciate the constructive and incisive feedback. We hope our responses address the reviewer's concerns.\n\n> Concerns over reliance on pretrained models (e.g., limited adaptability or bias), with suggestions of self-supervised learning or fine-tuning\n\nWe thank the reviewer for highlighting the dependency on pretrained models. Our approach currently uses a domain-specific vision encoder (XrayDINOv2) pretrained on X-ray images. To assess this dependency, we conducted experiments with three vision encoder settings: (1) ViT trained from scratch, (2) frozen DINOv2 pretrained on natural images, and (3) frozen XrayDINOv2. This setup evaluates both the necessity of pretraining and the role of domain specificity.\n\n|Vision encoder|OI|PC|PC20|CXR14|CXP|CXD10|Grnd|Phr-Grnd|SIIM (AUC)|SIIM (Dice)|RSNA (AUC)|RSNA (Dice)|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\nViT from scratch|0.700|0.728|0.693|0.704|0.848|0.643|0.237|0.611|0.773|0.023|0.751|0.379|\nDINOv2|0.841|0.825|0.853|0.791|**0.901**|**0.794**|0.606|0.814|0.908|0.100|**0.856**|0.509|\n**XrayDINOv2 (RadZero)**|**0.847**|**0.841**|**0.871**|**0.804**|0.900|0.787|**0.622**|**0.844**|**0.924**|**0.171**|0.834|**0.546**|\n\n*Grnd* and *Phr-Grnd* refer to grounding and phrase grounding, respectively.\n\nTraining from scratch led to substantial degradation across all metrics, confirming some dependency on pretrained weights for effective learning.\nHowever, domain-specific pretraining is not strictly required: the model with DINOv2 (pretrained on natural images) achieved competitive results and even outperformed RadZero on some metrics. This shows that general-domain models can be effectively adapted to the X-ray domain.\nFurthermore, it suggests that our method can be applied to domains lacking domain-specific pretrained vision encoders, broadening its applicability while mitigating the risk of domain-specific biases.\n\nA similar observation holds for the language encoder. Although MPNet, the language model used in our study, was not pretrained on medical texts, \nit successfully learned vision-language alignment using X-ray image–report pairs. This further supports the adaptability of our method across domains without domain-specific language pretraining.\n\nWe appreciate the reviewer's valuable suggestions on self-supervised learning and fine-tuning. We agree that self-supervised learning is a promising direction and plan to explore it in future work. For fine-tuning, we assessed its impact on both computational efficiency and performance. \nDue to increased computational overhead, memory usage rises and maintaining the same batch size becomes infeasible. Training time also increases. The table below compares memory and training time for freezing vs. fine-tuning the vision encoder under the same setup (batch size 64 on 4×H100 GPUs).\n\n|Vision encoder|Training GPU memory↓ (GB)|Training GPU hour↓|\n|-|-|-|\n|Freeze|18.52 x 4|56|\n|Unfreeze|51.70 x 4|100|\n\nFine-tuning requires approximately three times more memory and nearly twice the training time compared to freezing the vision encoder.\n\n|Vision encoder|OI|PC|PC20|CXR14|CXP|CXD10|Grnd|Phr-Grnd|SIIM (AUC)|SIIM (Dice)|RSNA (AUC)|RSNA (Dice)|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n**Freeze(RadZero)**|0.847|**0.841**|**0.871**|0.804|0.900|**0.787**|**0.622**|**0.844**|0.924|**0.171**|0.834|**0.546**|\nUnfreeze|**0.852**|0.800|0.777|**0.818**|**0.903**|0.779|0.400|0.760|**0.932**|0.113|**0.848**|0.496|\n\nFine-tuning the vision encoder yielded only marginal classification gains over RadZero on some datasets, while performance on fine-grained tasks dropped notably. In particular, the fine-tuned model performed significantly worse on PadChest20, which contains rare classes—suggesting reduced generalization. This aligns with [1], which reports that freezing the vision encoder can enhance generalizability. Given the high computational cost of fine-tuning, limited classification improvements, significant fine-grained degradation, and reduced generalization, we conclude that freezing the vision encoder is a more efficient and effective choice for our setting.\n\n> Multi-positive contrastive learning risks overfitting to specific report formats.\n\nWe appreciate the reviewer’s concern regarding potential overfitting. In our case, this risk is mitigated by the diverse linguistic and syntactic patterns introduced through LLM-based extraction of “finding sentences” during training. To validate robustness, we built a vector database of training sentences using RadZero’s text encoder and conducted similarity searches. For example, given the query \"the lungs are clear,\" the model retrieved varied semantically similar sentences, showing no overfitting to fixed templates. (Similarity scores in parentheses.)\n```\nQuery: the lungs are clear\n1.The lungs are clear (1.000)\n2.Lungs are clear (0.994)\n3.The airways are clear (0.993)\n4.There is clear lungs (0.985)\n5.There are clear lungs (0.983)\n6.The chest is clear (0.982)\n```\nDespite not following the \"[There is ...]\" pattern, the model retrieved semantically similar sentences such as \"chest is clear\" and \"airways are clear\", confirming that the encoder is not restricted to a fixed sentence structure.\n\nWe also tested syntactic diversity with \"pleural effusion in the right lower lung\": \n```\nQuery: pleural effusion in the right lower lung\n1.There is pleural effusion in the right lower lung (0.993)\n...\n5.There is a pleural effusion of the right lower lung (0.965)\n6.There is right lower lung pleural effusion (0.953)\n```\nThe model returned variants like \"right lower lung pleural effusion\" and \"pleural effusion of the right lower lung\": it captured the underlying semantics across different grammatical constructions.\n\nTo test lexical flexibility, we queried \"there is fibrosis\":\n```\nQuery: there is fibrosis\n1.There is fibrosis (1.000)\n2.There is probable fibrosis (0.937)\n3.There is lung fibrosis (0.937)\n4.There is chronic fibrosis (0.934)\n5.There is a component of fibrosis (0.924)\n6.There is fibrotic disease (0.917)\n```\nThe retrieved sentences included \"chronic fibrosis\" and \"fibrotic disease\", indicating the model understands semantics beyond the exact query vocabulary.\n\nThese results suggest that our model is not restricted to specific templates or vocabularies and generalizes well across linguistic variations, reducing the risk of overfitting.\n\nHowever, we acknowledge that the model may not generate well-aligned embeddings for report expressions that were rare or absent in the training data. \nOne known case is abbreviations—for example, the abbreviation “RLL” (right lower lung) was rarely used in the original reports. As a result, embeddings for abbreviations may be less accurate or semantically weaker compared to those of their expanded counterparts.\n\n```\nEdge case: pleural effusion in RLL (unseen format)\n1.There is pleural effusion with drainage (0.753)\n2.There is slight decrease in pleural fluid (0.725)\n3.There is collecting pleural fluid (0.708)\n```\nTo address this issue, non-standardized inputs—such as free-form reports with abbreviations or unusual phrasing—can be rewritten into finding-sentences using an LLM. This helps prevent performance degradation and ensures robust generalization across diverse report styles.\n\nEven so, we believe that such overfitting is unlikely to be directly caused by the multi-positive contrastive learning framework. We are concerned that we may have misunderstood the reviewer’s point, and if the reviewer meant a different aspect of overfitting, we would greatly appreciate further clarification.\n\n> VL-CABS similarity maps enhance interpretability, but are they clinically useful?\n\nWe agree with the reviewer that improved interpretability does not necessarily guarantee clinical utility. We also found this to be an important question and conducted a reader study with clinically trained readers to assess whether the model’s outputs improve readers' understanding of radiographic findings. The evaluation results indicate that our model has the potential to support clinical interpretation, and we are currently preparing a separate clinical paper based on these findings. Due to IRB constraints, we are unable to include detailed outcomes in this discussion, but we sincerely appreciate the reviewer’s understanding as we prepare the manuscript for publication.\n\n> Efficiency–cost trade-offs for high-resolution inputs, along with suggestions of model compression\n\nAs the reviewer correctly pointed out, increasing image resolution leads to higher computational cost. To quantify this trade-off, we compared 224px and 518px models in terms of performance and resource usage. The table below summarizes the results:\n\n|Resolution|Training GPU Memory↓ (GB)|Training GPU hour↓|Inference GPU Memory↓ (GB)|Latency↓ (ms/img)|Throughput (img/s)|Cls (AUC)|Grnd (ACC)|Seg (DICE)|\n|-|-|-|-|-|-|-|-|-|\n|224px|25.13 x 4|40|2.20|60.18|733.56|0.852|0.537|0.342|\n|518px|73.36 x 4|52|2.38|70.05|94.93|0.851|0.622|0.359|\n\nThe reported performance in the table reflects the average across all datasets for each task. These results suggest that the added computational cost is a worthwhile trade-off considering the significant gains in fine-grained performance.\n\nAs suggested by the reviewer, we conducted a model compression experiment via pruning. Specifically, we applied 10% L1-norm-based structured pruning to the MLP layers of the vision encoder.\n\n|Method|Num. of Parameters|Throughput (img/s)|Cls (AUC)|\n|-|-|-|-|\n|**RadZero**|209M|94.93|0.851|\n|Pruning10%|203M|97.16|0.680|\n\nPruning reduced the vision encoder’s parameters from 85.7M to 80.1M and the total model size from 209M to 204M, yielding a modest 2.3% gain in inference speed but noticeable performance degradation. While fine-tuning may recover accuracy, the trade-off may not justify the added cost. We appreciate the reviewer’s suggestion and believe model compression for efficiency remains a promising direction.\n\n[1] Zhai, Xiaohua, et al. \"Lit: Zero-shot transfer with locked-image text tuning.\" CVPR, 2022."}, {"review_id": "Pl1llS5QRH", "reviewer": "Reviewer_n3x7", "summary": "The paper introduces **RadZero**, a vision–language (VL) pre-training framework for chest-radiograph interpretation that targets *zero-shot* classification, grounding, segmentation, and open-vocabulary semantic segmentation.\n\nKey novelties are:\n\n- **VL-CABS (Vision-Language Cross-Attention Based on Similarity):** a patch-level cross-attention mechanism that replaces the usual dot-product with *scaled cosine similarity*, producing well-calibrated similarity maps that double as both logits and interpretable heat-maps.\n- **Multi-positive contrastive training:** every image is paired with *all* “finding-sentences” extracted from its report with an LLM, and the Multi-Positive NCE loss is used to aggregate supervision without random sampling.\n- **High-resolution but lightweight vision backbone:** a frozen XrayDINOv2 encoder is fed 518 px images, while only two extra transformer layers are trained, yielding high spatial fidelity at modest cost.", "questions": "1. **Similarity notation** – consider defining the scaled cosine similarity once as\n    \n    $s_{imk}^{n} = \\frac{\\langle v_{ik},t_j^{n}\\rangle}{\\tau}$\n    \n2. **Compute cost vs. benefit** – Given that higher resolution (518 px) helps grounding but slightly hurts classification (Table 5), would a *mixed-resolution* strategy (224 px in early epochs, 518 px for finetuning) recover both?\n\n3. **Statistical significance** – Please report per-task standard errors or run-seed variance so reviewers can judge whether gains (e.g., =- 0.014 AUC on OI) are robust.", "limitations": "The authors provide a frank discussion (Sec 6) of generalisation gaps and reliance on a frozen 224-px pre-trained encoder. I agree with this assessment and appreciate the transparency. Yes, limitations and societal impacts are adequately addressed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\nLarge-scale ablation isolates the effect of similarity metric, trainable layers, multi-positive loss and resolution; results are consistent across 0-shot tasks. \n\nCombines ideas (LiT freezing, dino.txt layers, MP-NCE) into a cohesive medical VLM.\n\nVL-CABS’ use of cosine similarity as both logit and heat-map is neat and, to my knowledge, novel.\n\n## weakness\n\nHigh-resolution inputs (518 px) improve grounding and segmentation but at a memory/latency cost, and they sometimes degrade classification. The paper lacks a systematic analysis of this trade-off or guidance on when the extra cost is warranted.\n\nRadZero freezes XrayDINOv2 trained at 224 px, then feeds 518 px images.  This mismatch slightly hurts classification accuracy and is acknowledged.", "was_revised": true, "final_justification": "Thank you to the authors for the thoughtful rebuttal. I especially appreciate the effort involved in running new experiments to address my questions: Compute/Performance Trade-off and Statistical Significance. While the authors have been transparent about this limitation and have answered my questions, the underlying technical challenge of using a frozen encoder with a different resolution than it was trained on remains an inherent, unsolved weakness of the proposed method.", "rebuttal": "We appreciate the considerate reviews and raising important questions. We hope our responses address the reviewer's concerns.\n\n> High-resolution inputs (518 px) improve grounding and segmentation but at a memory/latency cost, and they sometimes degrade classification. The paper lacks a systematic analysis of this trade-off or guidance on when the extra cost is warranted.\n\nWe appreciate the reviewer’s suggestion to analyze the trade-off between memory/latency costs and performance. As requested, we conducted detailed profiling:\n\n|**Method**|Training GPU Memory↓ (GB)|Training GPU hour↓|Inference GPU Memory↓ (GB)|Latency↓ (ms/img)|Throughput (img/s)|Cls (AUC)|Grnd (ACC)|Seg (DICE)|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|**RadZero(224px)**|25.13 x 4|40|2.20|60.18|733.56|0.852|0.537|0.342|\n|**RadZero**|73.36 x 4|52|2.38|70.05|94.93|0.851|0.622|0.359|\n\nTraining on four H100 GPUs with batch size 64 per device, GPU memory usage was approximately three times higher for 518 px inputs compared to 224 px, and training time increased by ~30%.\nDuring inference with batch size=1, GPU memory usage was 2.2 GB (224 px) vs 2.38 GB (518 px), and per-image latency was approximately 60ms vs. 70ms, respectively.\nThroughput was measured using the maximum power-of-two batch size supported by an H100 GPU: 4096 for 224 px inputs and 256 for 518 px inputs.\nThe reported performance in the table reflects the average across all datasets for each task—classification, grounding, and segmentation.\nAlthough using higher-resolution inputs increases resource requirements, it substantially improves grounding and segmentation accuracy, while classification performance remains largely unchanged.\nBased on this trade-off, we opted to use 518 px as our final input resolution to maximize fine-grained task performance with manageable overhead.\n\n\n> 1. Similarity notation\n\nWe thank the reviewer for suggesting a clearer notation using angle brackets (`⟨·,·⟩`) to denote scaled cosine similarity. We agree that this change would improve readability and will incorporate it in the revised version.\n\n> 2. Compute cost vs. benefit – Given that higher resolution (518 px) helps grounding but slightly hurts classification (Table 5), would a mixed-resolution strategy (224 px in early epochs, 518 px for finetuning) recover both?\n\nThank you for the interesting suggestion regarding a mixed-resolution training strategy. To investigate this, we conducted an experiment where the model was initially trained with 224 px resolution and then switched to 518 px:\n\n||OI|PC|PC20|CXR14|CXP|CXD10|Grnd|Phr-Grnd|SIIM (AUC)|SIIM (Dice)|RSNA (AUC)|RSNA (Dice)|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n**RadZero**|**0.847**|**0.841**|**0.871**|**0.804**|**0.900**|**0.787**|**0.622**|0.844|**0.924**|**0.171**|**0.834**|**0.546**|\nmixed-resolution|0.845|0.834|0.853|0.798|**0.900**|0.783|0.598|**0.874**|0.921|0.156|0.833|0.542\n\n\n*Grnd* and *Phr-Grnd* refer to grounding and phrase grounding, respectively.\n\nThis approach improved performance on fine-grained tasks compared to RadZero (224px) in Table 5 (e.g., grounding: 0.537 → 0.598; phrase grounding: 0.832 → 0.874; Dice score of SIIM: 0.121 → 0.156). However, classification performance declined across most datasets, falling below the 518 px RadZero baseline. While high-resolution inputs help with localization tasks, the performance drop in classification suggests that switching to 518 px after training with 224 px may lead to overfitting. This indicates that a straightforward mixed-resolution may introduce trade-offs that affect classification performance. Further investigation into more sophisticated mixed-resolution strategies could potentially alleviate this issue.\n\n> 3. Statistical significance – Please report per-task standard errors or run-seed variance so reviewers can judge whether gains (e.g., =- 0.014 AUC on OI) are robust.\n\n\nWe thank the reviewer for pointing out the importance of reporting statistical significance. To address this, we report per-task standard deviations across five random seeds:\n||OI|PC|PC20|CXR14|CXP|CXD10|Grnd|Phr-Grnd|SIIM (AUC)|SIIM (Dice)|RSNA (AUC)|RSNA (Dice)|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n**RadZero**|0.847|0.841|0.871|0.804|0.900|0.787|0.622|0.844|0.924|0.171|0.834|0.546|\nMean $\\pm$ std|0.848 $\\pm$ 0.0016|0.840 $\\pm$ 0.0011|0.869 $\\pm$ 0.0040|0.802 $\\pm$ 0.0016|0.899 $\\pm$ 0.0019|0.787 $\\pm$ 0.0043|0.601 $\\pm$ 0.0130|0.845 $\\pm$ 0.0143|0.918 $\\pm$ 0.0063|0.164 $\\pm$ 0.0080|0.843 $\\pm$ 0.0072|0.549 $\\pm$ 0.0023|\n\nThe results show that the variance across runs is relatively small.\nSpecifically, on the OI dataset, where CARZero achieves an AUC of 0.838, RadZero's performance gain falls outside the expected variance range.\nThis suggests that our improvements are robust rather than due to random fluctuation."}, {"review_id": "r7NIuHuZ81", "reviewer": "Reviewer_7dyc", "summary": "This paper presents RadZero, a vision-language alignment model for radiology that supports zero-shot classification, grounding, and segmentation. It introduces VL-CABS, which uses cosine similarity between text and image patches to produce interpretable similarity maps. The model employs a frozen vision encoder, lightweight Transformer layers, and trains with multi-positive contrastive learning to align each image with multiple report sentences. Experiments on public chest X-ray datasets demonstrate strong performance and improved explainability.", "questions": "1. VL similarity maps are central to your interpretability claims but are not directly supervised during training. Have you considered adding auxiliary losses (e.g., from pseudo labels or phrase-region alignments) to explicitly optimize these maps? If so, what were the results?\n\n2. In Table 1, RadZero underperforms CARZero on CXR14 and CheXpert in zero-shot classification. Could you elaborate on why your model may be less effective on these task-driven benchmarks? Is there a trade-off between RadZero's grounding-focused architecture and global classification performance?\n\n3. You chose to freeze the vision encoder throughout training. Could you clarify the motivation behind this decision? Have you considered finetuning it to further improve alignment quality or performance on classification tasks?", "limitations": "1. VL similarity maps are not directly supervised, which raises concerns about their reliability for fine-grained grounding or segmentation beyond visualization. Adding auxiliary losses or weak supervision could improve alignment quality.\n\n2. The reported improvements in grounding and segmentation rely heavily on higher input resolution (518px), while most baselines use 224px. The potential confounding effect of resolution is not fully controlled or discussed.\n\n3. While the method is positioned as interpretable, the similarity maps are only weakly correlated with clinical annotations, and no radiologist or human validation is reported to confirm interpretability or safety in deployment contexts.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.  The paper extends contrastive learning to align each image with multiple semantically minimal sentences extracted from radiology reports. This multi-positive formulation improves training stability and semantic diversity, which is especially valuable in clinical vision-language tasks with complex text-image relationships.\n\n2. The evaluation includes nearly all major public chest X-ray datasets, covering a wide range of zero-shot tasks (classification, grounding, segmentation, phrase grounding). This makes the claims empirically convincing and the conclusions broadly applicable within the medical vision-language domain.\n\n3. Beyond technical design, the paper demonstrates high quality in writing, reproducibility, and transparency. The authors provide implementation details, ablations, and commit to open-sourcing code and models, which reflects strong community contribution standards.\n\nWeaknesses:\n\n1. The optimization objective remains at the global image-text level; although patch similarities are computed, they are only used for pooling and visualization, not directly supervised. As a result, VL-CABS does not explicitly optimize patch-level alignment.\n\n2. Despite strong design and wide evaluation, the model does not clearly outperform prior methods (e.g., CARZero or MedKLIP) across all tasks and datasets. In particular, its gains in classification are modest, and grounding/segmentation improvements rely partly on higher resolution inputs.\n\n3. While the VL similarity map is a central contribution of the paper, it is never explicitly supervised or optimized. It is only derived post-hoc from global alignment signals, raising concerns about whether it can be reliably used for pixel-level reasoning or segmentation beyond visualization.\n\n4. Many of RadZero’s reported improvements in grounding and segmentation rely on high-resolution inputs (518px), while competing methods are evaluated at 224px. Ablation shows significant performance drop at lower resolution, suggesting that part of the gain may come from resolution scaling rather than architectural advances.", "was_revised": true, "final_justification": "My concerns have been addressed by the authors’ rebuttal. I therefore maintain my original score.", "rebuttal": "We appreciate the insightful comments and positive support with constructive feedback. We hope our responses address the reviewer's concerns.\n\n> Concerns about the lack of direct supervision for VL similarity maps (W1, W3, Q1, L1), along with a suggestion to incorporate auxiliary losses to improve alignment (Q1).\n\nWe agree that incorporating direct supervision—e.g., via pseudo labels or phrase-region alignments—could enhance the quality of VL similarity maps for grounding and segmentation. However, fine-grained annotations are significantly scarcer than image-report pairs, which limits scalability, and the potential inaccuracy of pseudo labels may further introduce noise into the training process. Instead, our goal was to leverage abundant paired data to develop a model with strong zero-shot multi-task capabilities, rather than optimizing for a specific downstream task. Although patch-level alignment is not directly supervised, we demonstrate that our model can still produce meaningful pixel-level similarity maps that enable grounding and open-vocabulary segmentation. In fact, open-vocabulary segmentation is especially challenging to train with direct supervision, which we view as a key strength of our approach. That said, we sincerely appreciate the reviewer’s suggestion to further advance our research by incorporating auxiliary losses. We consider this a valuable direction, and plan to explore the use of such supervision to explicitly guide VL similarity maps in future work.\n\n> Model performance is due to high-resolution input rather than structural advantages (W2, W4, L2)\n\nWe agree that higher-resolution inputs contribute to improved grounding and segmentation performance. However, we would like to clarify that RadZero’s gains are not solely attributable to resolution. The two tables below present selected results from the phrase grounding and segmentation evaluations in Tables 3 and 4 of the main text, including only RadZero (224 px) and a few baseline models for comparison at the same input resolution.\n- Table 3\n|Method|MS-CXR|\n|-|-|\n|BioViL-T[1]|0.719|\n|MedKLIP[2]|0.407|\n|CARZero[3]|0.749|\n|**RadZero(224px)**|**0.832**|\n\n- Table 4\n|Method|RSNA (Dice)|SIIM (Dice)|SIIM (Pix-AUC)|\n|-|-|-|-|\n|MedKLIP[2]|0.465|0.044|0.648|\n|CARZero[3]|0.540|0.100|0.856|\n|**RadZero(224px)**|**0.562**|**0.121**|**0.943**|\n\nRadZero(224px) outperforms prior methods even when evaluated at the same 224px resolution, demonstrating the model’s structural advantages. Building on this, we further enhance performance by utilizing a resolution-adaptive design that enables inference with higher-resolution inputs without incurring substantial computational overhead. Therefore, we believe that while resolution plays a role, the improvements do not rely heavily on it, but rather reflect a combination of architectural design and input fidelity. We also appreciate the reviewer’s comment regarding the potential confounding effect of resolution and acknowledge that, although we reported the results of our model at the same 224px resolution to address this concern, our discussion may have been insufficient, and we apologize for not making the distinction clearer in the main text.\n\n> Concerns about the absence of human validation to assess the clinical reliability of VL similarity maps (W3, L1, L3)\n\nWe fully agree with the reviewer’s concern that, while our experimental results demonstrate the potential of VL similarity maps, their reliability in clinical settings remains uncertain without human validation. We also considered this an important question and conducted a reader study with clinically trained readers to assess whether the model’s outputs provide clinically meaningful visual explanations. The results indicated that our model may aid clinical interpretation, and we are currently preparing a separate clinical paper based on these findings. Due to IRB constraints, we cannot include results here, but we appreciate the reviewer’s understanding as we finalize the manuscript for publication.\n\n\n\n> Could you elaborate on why your model may be less effective on these task-driven benchmarks? Is there a trade-off between RadZero's grounding-focused architecture and global classification performance? (W2, Q2)\n\nUpon examining class-wise performance on OpenI and ChestXray14, we identified a potential cause of underperformance:\n\n### OpenI\n|Method|AVG.|ATE|CARD|EFF|INFL|MASS|NOD|PNA|PTX|EDE|EMPH|FIB|PLTH|HERN|FX|OPAC|LES|CG|LG|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|CarZero[3]|0.838|**0.859** |0.933|**0.938**|**0.776**|0.887|0.612|0.877|0.921|0.900| 0.899|**0.917**|0.822|0.953|**0.726**|**0.784**|**0.976**|0.658|0.621|\n|**RadZero(224px)**|**0.851**|0.850|**0.939**|0.937|0.774|**0.891**|**0.653**|**0.881**|**0.952**|**0.910**|0.925|0.900|**0.837**|**0.989**|0.720|**0.784**|0.970|0.700|0.700|\n|**RadZero**|0.847|0.857|0.933|0.933|0.775|0.886|0.630|0.870|0.949|0.902|**0.926**|0.904|0.820|0.982|0.701|0.781|0.929|**0.731**|**0.734**|\n\n### ChestXray14\n|Method|AVG.|ATE|CARD|EFF|INFL|MASS|NOD|PNA|PTX|CONS|EDE|EMPH|FIB|PLTH|HERN|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|CarZero[3]|**0.811**|**0.819**|0.852|**0.873**|0.670|0.854|0.718|0.737|0.871|**0.786**|**0.884**|**0.808**|0.788|0.770|0.928|\n|**RadZero(224px)**|0.807|0.796|**0.864**|0.857|**0.672**|**0.859**|**0.742**|**0.772**|0.873|0.784|0.883|0.631|**0.807**|**0.789**|**0.963**|\n|**RadZero**|0.804|0.792|0.863|0.854|0.669|0.837|0.727|0.766|**0.875**|0.784|0.881|0.655|0.806|0.781|**0.963**|\n\n(Abbreviations used in the table correspond to the following pathologies: ATE=Atelectasis, CARD=Cardiomegaly, EFF=Effusion, INFL=Infiltration, MASS=Lung Mass, NOD=Lung Nodule, PNA=Pneumonia, PTX=Pneumothorax, EDE=Pulmonary Edema, EMPH=Emphysema, FIB=Fibrosis, PLTH=Pleural Thickening, HERN=Hernia, FX=Fracture, OPAC=Opacity, LES=Lesion, CG=Calcified Granuloma, LG=Lung Granuloma, CONS=Consolidation.)\n\nOn ChestXray14, our model outperforms CARZero on most pathologies, but shows notably lower performance on EMPH, which significantly lowers the overall average. However, the model does not appear to struggle with EMPH classification in general, as it actually outperforms CARZero on this class in the OpenI dataset. This suggests the discrepancy stems from dataset-specific differences rather than a core limitation of our model; still, generalization is an area for improvement.\n\nBased on our ablation study in Table 5, we do not observe a clear trade-off between RadZero’s architecture and global classification performance: under the same training setting, variant (c) of our model outperforms the CARZero-style training approach. This suggests that the structural design of VL-CABS can enhance interpretability without sacrificing classification accuracy.\n\n> Motivation behind freezing the vision encoder and consideration of fine-tuning to enhance alignment or classification performance (Q3)\n\nThank you for raising this important question. Our decision to freeze the vision encoder is motivated by two key factors. First, building on approaches such as LiT [4], which leverage frozen pretrained encoders, and dino.txt [5], a self-supervised model optimized for detailed visual semantics, we aim to align text embeddings with fine-grained visual features extracted by self-supervised models. Unlike vision-language models (VLMs) like CLIP that excel at representing global features but often fall short on fine-grained tasks, self-supervised models preserve spatially dense representations that are better suited for localization and structured alignment. Freezing the vision encoder allows us to preserve fine-grained visual features, while the vision-language alignment enables a language interface through the text encoder—effectively bridging the strengths of self-supervised vision models and VLM approaches.\n\nSecond, freezing the vision encoder significantly reduces computational and memory overhead, which is especially important given the high-resolution inputs used in RadZero. To assess the impact of this trade-off on performance, we conducted an experiment in which we unfreezed the vision encoder: \n\n|Vision encoder|OI|PC|PC20|CXR14|CXP|CXD10|Grnd|Phr-Grnd|SIIM (AUC)|SIIM (Dice)|RSNA (AUC)|RSNA (Dice)|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n**Freeze(RadZero)**|0.847|**0.841**|**0.871**|0.804|0.900|**0.787**|**0.622**|**0.844**|0.924|**0.171**|0.834|**0.546**|\nUnfreeze|**0.852**|0.800|0.777|**0.818**|**0.903**|0.779|0.400|0.760|**0.932**|0.113|**0.848**|0.496|\n\n*Grnd* and *Phr-Grnd* refer to grounding and phrase grounding, respectively.\n\nFine-tuning the vision encoder led to marginal improvements in classification performance over RadZero on some datasets (excluding PadChest and CXD10), but the gains were not significant. In contrast, performance on fine-grained tasks dropped considerably. Notably, on the PadChest20 subset—which contains rare classes—the fine-tuned model performed significantly worse, suggesting reduced generalization. This observation is consistent with findings from [4], which shows that freezing the vision encoder can enhance generalizability. Considering the substantial computational cost of fine-tuning, the marginal performance gains in classification, the significant degradation in fine-grained tasks, and the observed reduction in generalization ability, we conclude that freezing the vision encoder is a more efficient and effective choice for our setting.\n\n[1] Bannur, Shruthi, et al. \"Learning to exploit temporal structure for biomedical vision-language processing.\" CVPR, 2023.\n\n[2] Wu, Chaoyi, et al. \"Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis.\" ICCV, 2023\n\n[3] Lai, Haoran, et al. \"Carzero: Cross-attention alignment for radiology zero-shot classification.\" CVPR, 2024.\n\n[4] Zhai, Xiaohua, et al. \"Lit: Zero-shot transfer with locked-image text tuning.\" CVPR, 2022.\n\n[5] Jose, Cijo, et al. \"Dinov2 meets text: A unified framework for image-and pixel-level vision-language alignment.\" CVPR, 2025."}, {"review_id": "lF6J3E20nr", "reviewer": "Reviewer_ja8q", "summary": "The authors present a framework - RadZero - for vision-language alignment in radiology, focusing on chest x-rays. The framework encompasses several zero-shot downstream tasks, including classification, grounding, and segmentation. The approach is evaluated using several public datasets, achieving promising results.", "questions": "Suggestions that would improve my enthusiasm:\n1. Inclusion of supervised baseline for classification.\n2. Inclusion of error bars.\n3. Reporting of subgroup performance by pathology for classification. \n\nClarifications:\n1. Further details on how the segmentation threshold is chosen\n2. Difference between \"grounding\" and \"phrase grounding\"", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The manuscript is well written and generally easy to follow. \n2. The approach is intuitive and the motivation compared to prior works is logical.\n3. Extensive evaluation is performed across many datasets and tasks, including ablation analyses.\n4. The zero-shot open-vocabulary semantic segmentation results are intriguing, where the model is reasonably able to segment different anatomical regions in addition to performing pathology classification.\n\nWeaknesses\n1. The paper could further be improved by additional comparison to supervised baselines. Such benchmarking is reported for the segmentation results, but not for classification. Reporting such results (for at least a subset of the datasets) would help provide additional context on the level of performance.\n2. While the performance for grounding and segmentation is meaningfully higher than the prior CARZero model, the improvement in classification is less dramatic, with underperformance on some datasets.\n3. As is typical in contrastive learning settings, text phrases from the same image are treated as positive pairs and other images are treated as negatives. However, in the current approach where the text phrase is first standardized to a simplified \"finding-sentence\", for instance \"There is no pneumothorax\", this simple negative/positive assignment seems suboptimal. For instance, a similar finding-sentence may also be present for another image in the training batch, where it seems like this could be automatically determined and be treated instead as a positive example.\n4. In the paper checklist, the authors check \"No\" for the inclusion of error bars b/c \"it would be too computationally expensive\". Error bars could be computed via bootstrapping of the predictions, which is not computationally expensive. \n5. Subgroup performance by pathology is reported for zero-shot grounding but not classification. This would also be helpful to see for classification.\n\nOther clarifications/minor comments:\n1. In Figure 3, is the Text Encoder also frozen or just the Vision Encoder?\n2. The difference between \"grounding\" and \"phrase grounding\" is not entirely clear.\n3. In line 88, \"G2D [21]\" appears twice.\n4. Line 198: \"Following [36], we compute the Dice score using only positive samples and optimize the segmentation threshold to maximize the score.\" - Is this optimization performed on the test data itself? Is it done separately per dataset?", "was_revised": true, "final_justification": "I thank the authors for their clarifications and additional experiments included. I have upgraded by score to 6: Strong Accept due to the added experiments regarding baselines, error bars, and subgroup performance. I do have a remaining concern that the segmentation threshold was chosen based on the test set, as this likely causes overfitting and it should instead be chosen on the validation split. I recommend that the authors make this fix in the final version, but I believe that the volume of other experiments and contributions still warrant a Strong Accept.", "rebuttal": "We appreciate the constructive feedback and positive support. We hope our responses address the reviewer's concerns.\n\n> Inclusion of supervised baseline for classification\n\nThank you for the helpful suggestion. As recommended, we compared our zero-shot classification performance against supervised baselines reported in prior work on a subset of the datasets. For CheXpert, we re-evaluated our model using the same test set adopted by the referenced studies (not the one used in our main results) to ensure a fair comparison. For ChestXray14, we used the supervised results reported in [5]. The table below summarizes the comparison, where percentages in parentheses indicate the proportion of training data used in supervised training.\n\n|Method|Chexpert(1%)|Chexpert(10%)|Chexpert(100%)|ChestXray14(1%)|ChestXray14(10%)|ChestXray14(100%)|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|ConVIRT[1]|0.870|0.881|0.881|-|-|-|\n|REFERS[2]|0.872|0.881|0.882|-|-|-|\n|M3AE[3]|0.862|0.873|0.879|-|-|-|\n|MGCA[4]|0.888|**0.891**|**0.897**|-|-|-|\n|MRM[5]|0.885|0.885|0.887|0.794|**0.840**|**0.859**|\n||||||||\n|**RadZero(224px)**|0.888|0.888|0.888|**0.807**|0.807|0.807|\n|**RadZero**|**0.889**|0.889|0.889|0.804|0.804|0.804|\n\nWhile RadZero underperforms fully supervised models trained with 10% or 100% of labeled data, it consistently outperforms models trained with only 1% supervision. These results demonstrate that RadZero achieves competitive zero-shot classification performance, consistent with the segmentation results in Table 5. We are grateful for your insightful suggestion, which allowed us to further substantiate our findings. The new results will be incorporated into the revised version.\n\n> Positive/negative assignment strategy in contrastive learning\n\nWe appreciate the reviewer’s incisive observation. We agree that, especially with larger batch sizes, it becomes increasingly likely that different images may share the same finding-sentence—particularly common negation phrases such as “There is no pneumothorax.” This raises the risk of mistakenly treating positive pairs as negatives under the standard contrastive framework. One possible solution would be to construct training batches that avoid overlapping finding-sentences, though this becomes challenging as the batch size increases. Alternatively, recent work [6] has proposed using soft labels with negation-aware contrastive learning to mitigate such issues. We believe that incorporating such strategies to reduce erroneous negative pairs caused by shared finding-sentences could lead to more robust training. We thank the reviewer again for this incisive suggestion and plan to explore this direction in future work.\n\n> Inclusion of error bars.\n\nWe thank the reviewer for highlighting the importance of reporting error bars. While we agree that bootstrapping model outputs can provide uncertainty estimates, we considered evaluating the robustness of the proposed training method itself—rather than assessing only the predictions of a fixed model—would offer a more meaningful measure of reliability. To this end, we conducted additional experiments across five random seeds to evaluate run-to-run variability and report per-task standard deviations:\n||OI|PC|PC20|CXR14|CXP|CXD10|Grnd|Phr-Grnd|SIIM (AUC)|SIIM (Dice)|RSNA (AUC)|RSNA (Dice)|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n**RadZero**|0.847|0.841|0.871|0.804|0.900|0.787|0.622|0.844|0.924|0.171|0.834|0.546|\nMean $\\pm$ std|0.848 $\\pm$ 0.0016|0.840 $\\pm$ 0.0011|0.869 $\\pm$ 0.0040|0.802 $\\pm$ 0.0016|0.899 $\\pm$ 0.0019|0.787 $\\pm$ 0.0043|0.601 $\\pm$ 0.0130|0.845 $\\pm$ 0.0143|0.918 $\\pm$ 0.0063|0.164 $\\pm$ 0.0080|0.843 $\\pm$ 0.0072|0.549 $\\pm$ 0.0023|\n\n*Grnd* and *Phr-Grnd* refer to grounding and phrase grounding, respectively.\n\nThe results show that the variance across runs is relatively small. This suggests that our improvements are robust rather than due to random fluctuation.\n\n> Reporting of subgroup performance by pathology for classification.\n\nThank you for the helpful suggestion. To address this, we compared RadZero and CARZero on per-pathology classification performance across three datasets. For PadChest, which has a large number of classes, we evaluated on five representative categories commonly used. Abbreviations used in the table correspond to the following pathologies: ATE=Atelectasis, CARD=Cardiomegaly, EFF=Effusion, INFL=Infiltration, MASS=Lung Mass, NOD=Lung Nodule, PNA=Pneumonia, PTX=Pneumothorax, EDE=Pulmonary Edema, EMPH=Emphysema, FIB=Fibrosis, PLTH=Pleural Thickening, HERN=Hernia, FX=Fracture, OPAC=Opacity, LES=Lesion, CG=Calcified Granuloma, LG=Lung Granuloma, CONS=Consolidation.\n\n- OpenI\n|Method|AVG.|ATE|CARD|EFF|INFL|MASS|NOD|PNA|PTX|EDE|EMPH|FIB|PLTH|HERN|FX|OPAC|LES|CG|LG|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|CarZero[7]|0.838|**0.859** |0.933|**0.938**|**0.776**|0.887|0.612|0.877|0.921|0.900| 0.899|**0.917**|0.822|0.953|**0.726**|**0.784**|**0.976**|0.658|0.621|\n|**RadZero(224px)**|**0.851**|0.850|**0.939**|0.937|0.774|**0.891**|**0.653**|**0.881**|**0.952**|**0.910**|0.925|0.900|**0.837**|**0.989**|0.720|**0.784**|0.970|0.700|0.700|\n|**RadZero**|0.847|0.857|0.933|0.933|0.775|0.886|0.630|0.870|0.949|0.902|**0.926**|0.904|0.820|0.982|0.701|0.781|0.929|**0.731**|**0.734**|\n\n- PadChest\n|Method|AVG.|ATE|CARD|CONS|EDE|PNA|\n|-|-|-|-|-|-|-|\n|CarZero[7]|0.810|0.835|0.906|**0.903**|0.971|0.841|\n|**RadZero(224px)**|**0.841**|**0.839**|0.917|0.902|**0.973**|**0.846**|\n|**RadZero**|**0.841**|**0.839**|**0.920**|0.899|0.972|0.831|\n\n- ChestXray14\n|Method|AVG.|ATE|CARD|EFF|INFL|MASS|NOD|PNA|PTX|CONS|EDE|EMPH|FIB|PLTH|HERN|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|CarZero[7]|**0.811**|**0.819**|0.852|**0.873**|0.670|0.854|0.718|0.737|0.871|**0.786**|**0.884**|**0.808**|0.788|0.770|0.928|\n|**RadZero(224px)**|0.807|0.796|**0.864**|0.857|**0.672**|**0.859**|**0.742**|**0.772**|0.873|0.784|0.883|0.631|**0.807**|**0.789**|**0.963**|\n|**RadZero**|0.804|0.792|0.863|0.854|0.669|0.837|0.727|0.766|**0.875**|0.784|0.881|0.655|0.806|0.781|**0.963**|\n\nConsistent with the average performance, RadZero generally outperforms or matches CARZero[7] on OpenI and PadChest. On ChestXray14, our model performs comparably or better on most pathologies, but a notable drop on EMPH accounts for the overall underperformance on the dataset. However, RadZero outperforms CARZero on EMPH in OpenI, indicating that the drop is dataset-driven rather than due to pathology-specific modeling issues. These results reaffirm that while RadZero demonstrates strong zero-shot classification, further improvements in generalization remain an important direction.\n\n\n### Minor comments and clarifications\n\n> In Figure 3, is the Text Encoder also frozen or just the Vision Encoder? \n\nIn Figure 3(b), only the vision encoder (indicated by the snowflake icon) is frozen, while the text encoder remains trainable. We will clarify this in the final version for better readability and appreciate the reviewer’s attention to this detail.\n\n> Difference between \"grounding\" and \"phrase grounding\"\n\nWe apologize for the ambiguity. In our paper, we use the term *grounding* to refer to locating the presence of a specific pathology (e.g., “pneumothorax”) within an image. This is typically evaluated using datasets such as ChestXDet10, which provide class-wise bounding box annotations. In contrast, *phrase grounding* refers to a more descriptive task where the goal is to localize regions described by arbitrary phrases (e.g., “small right pneumothorax”), which may include detailed modifiers or anatomical references beyond a disease label. The MS-CXR dataset is specifically designed for this task, providing image–phrase–box triplets for evaluation. We appreciate the reviewer for pointing out the lack of clarity, and we will revise the manuscript to include this distinction more clearly.\n\n> In line 88, \"G2D [21]\" appears twice.\n\nWe thank the reviewer for pointing out this oversight. We will correct the duplication in the final version.\n\n> Further details on how the segmentation threshold is chosen\n\nYes, the threshold is optimized on the test set, separately per dataset. We'll clarify this in the revision.\n\n[1] Zhang, Yuhao, et al. \"Contrastive learning of medical visual representations from paired images and text.\" PMLR, 2022\n\n[2] Zhou, Hong-Yu, et al. \"Generalized radiograph representation learning via cross-supervision between images and freetext radiology reports.\" Nature Machine Intelligence, 2022.\n\n[3] Geng, Xinyang, et al. \"Multimodal masked autoencoders learn transferable representations.\" arXiv preprint arXiv:2205.14204, 2022.\n\n[4] Wang, Fuying, et al. \"Multi-granularity crossmodal alignment for generalized medical visual representation learning.\" NeurIPS, 2022.\n\n[5] Zhou, Hong-Yu, et al. \"Advancing radiograph representation learning with masked record modeling.\" ICLR, 2023.\n\n[6] Ko, Hanbin, and Chang-Min Park. \"Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis.\" CVPR, 2025.\n\n[7] Lai, Haoran, et al. \"Carzero: Cross-attention alignment for radiology zero-shot classification.\" CVPR, 2024."}]

Author rebuttal:

Meta-review: Summary: The paper introduces a vision–language alignment model, RadZero, for radiology, demonstrating strong zero-shot performance across classification, grounding, and segmentation tasks. There are three main components that contribute to the strong performance of the model: (i) VL-CABS, which computes image–text similarity at the patch level to improve interpretability, (ii) multi-positive contrastive training, enabling effective representation learning without pixel-level annotations, and (iii) support for high-resolution inputs.

Strengths highlighted by the reviewers include:
(i) Transparent and fine-grained visual reasoning that substantially improves clinical interpretability compared to traditional attention maps.
(ii) Comprehensive ablation studies.
(iii) Cohesive integration of multiple ideas into a unified medical VLM.
(iv) Novel and elegant use of cosine similarity as both logit and heatmap in VL-CABS.
(v) Evaluation across nearly all major public chest X-ray datasets, covering a wide range of zero-shot tasks (classification, grounding, segmentation, phrase grounding), which makes the empirical claims convincing.
(vi) High quality of writing, reproducibility, and transparency.
(vii) Clear motivation and logical positioning compared to prior works.
(viii) Intriguing zero-shot open-vocabulary semantic segmentation results, where the model can segment anatomical regions in addition to pathology classification.

Weaknesses discussed after rebuttal and discussions include:
(i)  Novelty concerns – raised by Reviewer etwN. Multi-positive contrastive learning has been explored previously, while the application to radiology is novel. The paper does not propose a fundamentally new algorithm.
(ii) Frozen encoder limitation – using encoders at resolutions different from those they were trained on remains an inherent, unresolved technical weakness of the method.

Review process summary and Reasons for decision:  The paper received a mix of borderline reject, borderline accept, accept, and strong accept ratings. The borderline reject primarily stems from the lack of fundamentally new algorithmic contributions. However, taking into account the rebuttal, the feedback from other reviewers, and the strong experimental validation, the AC judges that the paper’s strengths — including its empirical contributions to medical VLMs — outweigh the weaknesses. The recommendation is to accept the paper as a poster.

Additional remarks from the Area Chair and expected changes in the final draft:
(i) The AC notes that the proposed idea is generic and not inherently specific to medical imaging. Therefore, the method should ideally have been evaluated on natural image datasets to demonstrate broader applicability. Alternatively, if the method is tailored specifically for X-ray imaging, the authors should clearly highlight the unique challenges of X-ray data and how their approach addresses them.

(ii) As pointed out  by Reviewer etwN, the experiments are limited to chest X-ray/CT datasets, which does not fully support the claim of achieving “Explainable Vision–Language Alignment in Radiology.” The title and narrative of the paper must be revised to avoid the generic "radiology" term for  better aligning with the actual scope of experiments and results.

(iii) The authors are also expected to incorporate all additional explanations and experimental results provided in the rebuttal and discussion into the final draft (or supplementary material if space is limited).

(iv)  Moreover, as promised in the NeurIPS submission portal, the authors are expected to open-source the code and training data or provide clear guidance for adapting the publicly available codebases and datasets, to ensure reproducibility of the reported experimental results.
Decision: Accept (poster)