G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training
Authors: Che Liu
Cheng Ouyang
Sibo Cheng
Anand Shah
Wenjia Bai
Rossella Arcucci
Venue: NeurIPS 2024 poster (2024)
Keywords: Medical Vision-Language Pre-training
dense visual learning
Primary area: machine_learning_for_healthcare
TL;DR: We propose G2D, a novel medical VLP framework that learns global and dense visual features solely from image-text pairs, outperforming existing methods even with only 1% of the training data compared to their use of 100%.

Abstract: Medical imaging tasks require an understanding of subtle and localized visual features due to the inherently detailed and area-specific nature of pathological patterns, which are crucial for clinical diagnosis. Although recent advances in medical vision-language pre-training (VLP) enable models to learn clinically relevant visual features by leveraging both medical images and their associated radiology reports, current medical VLP methods primarily focus on aligning images with entire reports. This focus hinders the learning of dense (pixel-level) visual features and is suboptimal for dense prediction tasks (e.g., medical image segmentation).

To address this challenge, we propose a novel medical VLP framework, named **Global to Dense level representation learning (G2D)**, which aims to learn global and dense visual features simultaneously using only image-text pairs without extra annotations. In particular, G2D designs a **Pseudo Segmentation (PS)** task, which enables the model to learn dense visual features during VLP. Notably, generating PS masks can be performed on the fly during VLP, which does not incur extra trainable parameters. With this simple yet effective idea, G2D achieves superior performance across 5 medical imaging tasks and 25 diseases. Particularly, in the segmentation task which requires dense visual features, **G2D surpasses existing models even with just 1% of the training data for finetuning, compared to 100% used by other models**. The code can be found in https://github.com/cheliu-computation/G2D-NeurIPS24/tree/main.

G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training
Che Liu Cheng Ouyang*52 Sibo Cheng!"
Anand Shah Wenjia Bai2.34 Rossella Arcueci
"Department of Earth Science and Engineering, Imperial College London, UK
Data Science Institute, Imperial College London, UK
Department of Computing, Imperial College London, UK
4 Department of Brain Sciences, Imperial College London, UK
Department of Infectious Disease Epidemiology. Imperial College London, UK 7 Royal Brompton and Harefield Hospitals, UK
Department of Engineering Science, University of Oxford, Oxford, UK
Institute of Clinical Sciences, Imperial College London, UK
CEREA, École des Ponts and EDF R&D, fle-de-France, France. che. 1iu21 Simperial....mmmmmmm wwperrallac... aw..c.a........ uk
Abstract
Medical imaging tasks require understanding of subtle and localized visual features due to the inherently detailed and area-specific nature of pathological patterns, which crucial for clinical diagnosis. Although recent advances in medical vision-language pre-training (VLP) enable models to learn clinically rel- visual features by leveraging both medical images and their associated radi- ology reports, current medical VL.P methods primarily focus on aligning images with entire reports. This focus hinders the learning of dense (pixel-level) visual features and is suboptimal for dense prediction tasks (c.g.. medical image segmen- tation). To address this challenge, we propose a novel medical VLP framework. named Global to Dense level representation learning (G2D), which aims to learn global and dense visual features simultaneously using only image-text pairs with- out extra annotations. In particular, G2D designs Pseudo Segmentation (PS) task, which enables the model to learn dense visual features during VL.. Notably, generating PS masks can be performed the fly during VI.,, which does not incur extra trainable parameters. With this simple yet effective idea, G2D achieves su- perior performance across medical imaging tasks and 25 diseases. Particularly, in the segmentation task which requires dense visual features, G2D surpasses ex- isting models even with just 1% of the training data for finetuning, compared to 100% used by other models. The code can found https//gillub..om/chlluu- CompuuattonnGDDDDDeeASSSS/AAeee
1 Introduction
In medical image analysis, learning global and dense visual representations typically requires labor- intensive and costly image and pixel-level annotations ① 2 Vision-language pre-training (VLP) attempts addressing this by aligning vision and language content using paired datasets 13 5 6). Although existing medical VLP methods excel at learning global visual features [7). they face chal- lenges with dense visual features because the level of detail in text reports does not offer sufficient
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Figure 1: Comparing existing med- ical VLP methods with G2D: a) Alignment-base approaches lack dense (pixel-level) feature learning.
Taxt-guided Aligned
Leariing
Not Aligned with Text!
b) Reconstruction-uused approaches do align with text, resulting in defi- ciency in discriminative and clinically relevant visual features. c) The frame- work of G2D (proposed) learns dense, clinically relevant, text-aligned visual features through derived pscudo masks and image- text alignment. We use red text to highlight the deficiencies of ex- isting methods and blue text to empha- size advantages.
Learning Dense Feature!
Masked Image Reconstruction
Pscudo Mask
f
LUZA
Image-Text Alignment
f
Masked Tout Tisken
F
F
F
e
FL
F
F
(c)
pixel-level supervision for learning these more detailed aspects. Existing medical VLP methods are categorized into two main types, shown in Fig. [
Alignment-based Approaches, which focus aligning images with reports 4 g
6
2 [0]. Although methods like [4] 8 9 align images with entire reports and text tokens, they struggle to learn dense, clinically relevant visual features. This is due to the ambigu- ous supervision targets provided by text tokens, which lack explicit relational pairing with
image regions, as discussed in 22
Approaches, which learn representations by reconstructing masked images or reports using masked modeling techniques [2]. However, they also lack success in capturing dense, clinically relevant visual features, the reconstruction task primarily focuses on low-level patterns (texture, shape) rather than high-level seman- ties 13
Reconstruction-
Despite advancements in medical VLP, limitations still exist. Current alignment approaches align image patches with text tokens in a brute-force manner and possibly cause misalignments when some word tokens (e.g.. 'compatible' or 'unremarkable') lack direct visual counterparts, leading to ambiguous local alignments. Mcanwhile, reconstruction--ast basedd approaches may ignore high-level image semantics. They are designed to recover low-level visual information such as intensity and texture, without accounting for high-level semantics [14 B 13). As result, both approaches perform suboptimally for downstream tasks, such as semantic segmentation and visual grounding, which require learning of granular visual features that aligned with high-level semantics.
While numerous VLP methods are designed to capture dense visual features for natural image datasets (e.g.. ImageNet) they often struggle to transfer directly to medical images because they depend on well-trained object detection model [16] 17 or a well-aligned VL.P model [8 19]. In the medical domain, obtaining such pre-trained models is difficult as objects can be defined in var- ious ways within a single medical image (e.g., based on organs, anatomical structures, or abnormal regions). Additionally, in medical domain, there is lack of foundational VLP models that are both publicly accessible and are trained on sufficiently large image-text pairs that cover diverse medical imaging applications.
In response to the aforementioned challenges, we introduce a novel medical VLP approach termed G2D. This approach is designed to extract global and dense visual representations from radiography along with their associated radiology reports, with improved feature granularity and enriched se- mantic information. Central to our approach is pretext task, Pseudo Segmentation (PS), which is guided by pseudo mask (segmentation target) derived from a carefully refined and filtered attention map. PS encourages the model to learn dense representations through pixel-level pretext task that incorporates high-level semantics. This approach, in contrast to traditional methods that align image patches with text tokens, inherently mitigates the misalignment bias and allows learning of more representative features. Notably, the PS pretext task can be implemented to run concurrently with vision-language alignment, ensuring that the model can be trained end-to-end, contrasting with the two-stage training methods [18].
To evaluate the effectiveness of G2D relative to other state-of-the-art (SOTA) VLP approaches, we deploy the pre-trained model across diverse range of downstream tasks, including medical image classification, semantic segmentation, object detection, as well as zero-shot image classification and

---

visual grounding, on six public large-scale CXR datasets. The experimental results demonstrate the superior performance of G2D over existing VLP approaches on these medical applications. Overall, our contribution is three-fold:
1. We introduce G2D, the first end-to-end encoder-decoder medical VLP approach designed to learn visual representations from the global level down to the dense level, supervised by paired radiology reports and a pixel-wise pretext task.
2. We carefully design a pretext task tailored for medical VLP, pseudo segmentation. It formu- lates pseudo mask segmentation target, allowing the model to learn dense visual repre- sentations in the pretext task which can benefit downstream dense visual tasks in medicine. The pseudo mask can be generated using a parameter- free processor that leverages the at- tention map derived from the visual representation associated with radiology reports.
3. We conduct comprehensive experiments to validate the efficacy of the proposed G2D ap- proach, which outperforms peer approaches across five uni-modal and cross-modal down- stream tasks.
2 Related Works
Alignment-based Medical VLP. Drawing inspiration from (3), aligning images with their corre- sponding textual descriptions in the latent space has led to notable advancements in VLP. Within the CXR domain, while CORV [4] made early attempt employing bidirectional contrastive learning to globally align entire images with their paired reports, there remained room for refine- ment. GLoRIA [8] and MGCA [9] represent advancements
maaoe-report alignment, introducing sophisticated global-local methodologies to the field 8 9 These approaches endeavor to estab- lish correspondences between distinct image and text tokens. However, it is crucial recognize that the granularity of token-level alignment could inadvertently introduce distortions to the medi- cal context, potentially leading to misalignments, as illustrated by [20] 2 Med-UniC (20) utilizes augmented text in VLP training to cultivate language invariance, with the goal of mitigating lin- guistic biases from VLP. Meanwhile, MedKLIP (5) and KAD (21) harness domain-specifie knowl- edge from external annotated datasets to enhance textual information extraction. Notably, these approaches 20 representation learning, which could potentially constrain their generalizability.
are contingent upon external resources or extra data to optimize cross-modal
Reconstruction-baser Medical VLP. Several studies, including [12 I1 22 have employed recon- struction of image and text tokens as a pretext task within VLP. Specifically, MRM 12 endeavors reconstruct the original image from masked version and simultaneously aims to regenerate the original text using both the masked image and text as inputs. Conversely, PRIOR [1] adopts a strategy that focuses on cross-modal representation by reconstructing images and sentences based on complete image and report inputs. An enhancement to the MRM [12) approach is proposed by 22. where token weights are adjusted during the reconstruction phase.
While these methods have demonstrated promising outcomes, the ability of the reconstruction pre- text task to capture high-level semantic representations is limited, shown in [14] [5] [3, and is further challenged by the absence of explicit semantic-related constraints in dense visual represen- tation learning.
Methodology
The central aim of G2D to learn global and dense visual representations from medical images under the supervision of their corresponding radiology reports. As illustrated in Fig Left, G2D integrates two alignment strategies: vision-language alignment (VLA) that learns global represen- tations, and pixel alignment (PA) that focuses granular representation via a pixel-level pretext task, Pseudo Segmentation (PS). The pseudo mask for PS is constructed through a parameter- free mechanism, which is operated alongside VLA. The PS pretext task enables G2D to derive dense representations at both encoder and decoder levels during pre-training. Moreover, the task head of the pretext task facilitates a smoother transfer for the pre-trained encoder to be applied to down- stream segmentation tasks, reducing the gap between the dense visual representation learned from VLP and the needs of downstream dense visual tasks after VLP. This contrasts with previous meth-
3

---

Ep
Psaudo Mask
Pseudo
Mask
Edge Smoothing
LyLa
f
d
Figure 2: Left: Framework of G2D. Right: Pipeline for pseudo mask con- struction. We visualize the constructed pseudo mask and corresponding sen- tence in the radiology report See A.7
P
Filtering
f
F
Body
mask
Aggregated Attention Map
ods 4 8 21 5 6
that typically transfer only the pre-trained encoder, potentially leading to an information gap between the pre-training and downstream tasks.
3.1
Vision-Language Contrastive Learning
We utilise dual-encoder image-text contrastive approach following 4 Given training set consisting of pairs image-text (v,, 44), where 0; V denotes an image and 4 E denotes text report, i 1, 2, 3, .... NN, G2D employs
image encoder F. V + RD to encode the image into an embedding of dimension Dr. and a text encoder Fi C + D to encode the text report into embedding of dimension Dj. The embedded image and text features can be denoted as S = [[v1,[1) 1,,, 12,,)))..... (VN, Iw)), where v; = (v,)) and 1 F((4..
As depicted in Fig. 2 G2D incorporates two alignment strategies: VLA and PA. For VLA, the model aims to learn global visual and text representations by pulling the embeddings of paired image-report samples closer, while distancing embeddings of unpaired samples, using contrastive loss LVLA- The objective of contrastive learning is to predict N positive matched pairs (v,, 4) and N2 - N negative pairs among N N possible image-text pair combinations (3). Subsequently, two non-linear vision and language projectors Py and 'i transform V, and I; into the same dimension d, where v; = (v. 1, = P.(4,), and v,, 1, E Rd. After obtaining image feature vectors [v,],]] and text feature vectors [ with the same dimension d, the contrastive loss LVLA can be formulated as:
N Σ
log exp(v, ΣK exp( v 1 1,/0)/
LVLA = K
2j=1
denotes the temperature hyper-parameter empirically set to 0.07 following 99, and K E N is the batch size.
3.2 Pseudo Segmentation Mask Construction
Notably, although MedSAM 23 claims to build image-mask pairs, it requires box prompt inputs not available in the MIMIC-CXR dataset. Designing a box prompt for each image labor-intensive and unfeasible for this work, we construct the pseudo mask based on attention maps.
Attention Aggregation. Inspired by CLIP (3), we incorporate an attention pooling mechanism in conjunction with the non-linear projector P. to derive a pixel-wise attention map. A dense feature map V, is extracted from the final convolutional layer before the pooling operation in the image encoder Fe, with the dimension C' x H W. Here, C denotes the number of channels, while H and W represent the height and width of the feature maps. Subsequently, we reshape V; into a dimension of HW C. In this way, V; can be interpreted as sequence of pixel embeddings, where each token in this sequence represents the embedding of an individual pixel. The length of this sequence is defined by the number of channels, C. A special token, (CLS). is introduced to aggregate all pixel embeddings through multi-head self-attention (MHSA) [25] K] This process offers an attention score matrix W for each pixel, with dimensions àt x H x W. Here, signifies the

---

attention head number, and h E H, with H. being the total number of attention heads. This attention score matrix characterizes the information exchange between pixels and semantics provided by the text 3 18 and therefore it carries semantic information and is an ideal candidate for constructing the pretext pseudo mask. To derive the pseudo mask, we aggregate W across all attention heads to produce W, as described by:
W; h W
(2)
h
Mask Filtering and Edge Smoothing. After obtaining the aggregated attention map W,, we up- sample it to match the original image dimensions H' x W'. To remove pseudo mask regions in the background, we construct body mask for each CXR image using histogram-baset thresholding approach, following common practice [26] 27]. Subsequently, all attention scores outside the body mask are set to zero. threshold is applied to filter out low attention scores within the body mask, transforming W; into a binary mask. The threshold is determined at the 85% percentile of attention scores from W. The binary pseudo mask Mj is formulated as:
1 0
if w. threshold otherwise
M
wwere
j j= 1,2,3, H k = 1, W
(3)
To smooth the square-like boundary in the mask caused by upsampling, we apply bilateral filtering (BF) 22 M,, resulting in refined pseudo mask M,, as shown in Fig. Right. A comprehensive ablation study discussing the threshold and smoothing operation is presented in Sec. 4.5
3.3 Dense Visual Representation Learning through Pseudo Segmentation in VLP
While the global visual representation can be learned via VLA, dense representation often lacks direct alignment. To tackle this limitation, we introduce an image decoder, denoted as Fu. as shown in Fig. Left. This decoder takes visual feature V, as input and utilises the pscudo mask M, as the supervisory signal for the pretext task. We employ the commonly used soft Dice loss and binary cross-entropy loss
27
to optimise this task. The training loss function for LPA formulated as:
LPA
DDico + LBCE):
2
K H' w'
2 x (M k O Myspk)
DDie
M.s.k + Masa
K "' w' LRCE. ΣΣΣ
Misa log( Mij.k) + (1- M.Jaa) log(1 - Mane)
im1 im1jm1 k=1
with M¹ = Fa(V.)
(4)
The total loss for G2D is the sum of the VLA loss (Eq. I and the PA loss (Eq.
Liotal LVLA LPA
(5)
It is worth noting that the pseudo mask designed pixel-wise pretext supervisory signal. Al- though there is no manual annotation involved, the pseudo mask is constructed from the visual feature of the image encoder, which is pre-trained to align with radiology reports and thus contains clinical knowledge such as anatomical regions mentioned by the reports. In this sense, it can be good surrogate target for learning pixel-wise semantic information. To demonstrate that the pseudo mask serves as meaningful target for dense visual pre-training, we conduct an ablation study to use a perturbed pseudo mask with corrupt semantics for pre-training, and compare to the proposed pseudo mask, as detailed in Table 8 and Sec A.6
5

---

4 Experiments and Analysis
In this section, we compare our approach with SOTA medical VLP techniques. The implementation details and dataset training/test splits are reported in A.3 A.4
Pretraining Dataset and Configuration We utilise the MIMIC-CXR dataset 129 [4]. After prepro- cessing based on established protocols [9] [.. it provides 213,384 image-text pairs for pre-training. For the VLP part, we employ a standard ResNet-50 as the vision encoder Fe and adopt the decoder part of a U-Net as the vision decoder Fa. We adopt ClinicalBERT [30 as the text encoder using configurations described in [5] 21]. In line with 19 []. G2D is pre-trained for 50 epochs across 16 A100 GPUs, each accommodating a batch size of 128. The AdamW optimizer is employed with
learning rate set to 2 x 10-1 and a weight decay of 1 x 10-8, Additionally, a linear warm-up and a cosine annealing scheduler are incorporated in the training process.
4.1 Downstream Task Datasets and Configurations
For downstream tasks, our focus is to evaluate the efficacy of G2D in learning granular visual fea- tures that can be used for localisation, vision-language understanding, and visual recognition tasks. We examine the capability and transferability of the learned cross-modal representations by using them for five distinct medical imaging tasks, covering a spectrum of 25 different diseases.
Medical Image Segmentation. This task utilises the RSNA [11 and SIIM [32] datasets, follow- ing preprocessing guidelines established in p [). We adopt U-Net [1] fine-tuning configurations following 8 9.. The pre-trained vision encoder is frozen, while only the decoder parameters are
updated during fine-tuning. Performance is assessed using the Dice score, following the evaluation protocol in g 19I. It is noteworthy that the original MedKLIP [5] uses a different configuration (updating the vision encoder) compared to other methods (freezing the vision encoder) 4 8 6 Therefore, in these experiments, we reference the results reported in (20]. which reimplemented MedKLIP under setting consistent with all other methods. For fair comparison specifically with MedKLIP, we also reimplement G2D under MedKLIIPs original setting, as reported in the Sec A.5
Medical Object Detection. This task is conducted using the RSNA dataset [31] for Pneumonia Detection and the Object-CXR dataset [33] for Foreign Objects Detection, adhering to preprocessing methods from [9]. We employ YOLOv3 34 for detection, using the pre-trained vision encoder and updating an additional detection head during fine-tuning. We report the mean Average Precision (mAP) with IoU thresholds between 0.7. The setup for this task is in accordance with in (9]
Zero-shot Medical Image Visual Grounding. In accordance with [5]. this task is conducted on the RSNA 31 and SIIM 32 datasets, using the same official data split and evaluation metrics. We employ CXR images as input and utilise the corresponding ground truth label maps for assessing the grounding performance, in terms of recall, IoU, and Dice score.
Zero-shot Medical Image Classification. In compliance with the guidelines set forth in we conduct this task on the RSNA (3I), SIIM [32, CheXpert (35). and CXR14 14 (36 datasets. For the RSNA and SIIM datasets, we employ the test set splits provided by MedKLIIP [1, given that
KAD 21 did not conduct experiments on these two datasets. For the CheXpert and CXR 14 datasets 35 36) we use the official test set splits to ensure fair comparison with KAD 21 It is important to note that MedKLIP 55) creates its own test split rather than using the official test split. Hence, we do not use MedKLIP's splits in our experiments. We report the results using the macro average of AUC, F1, and ACC scores across all diseases.
we use the CXR14 dataset [36]. comprising 112,120 frontal-view X-rays from 30,805 patients, annotated for 14 dis- eases. We adhere to the official split for consistent evaluation, following KAD 2TT]. It is worth noting that MedKLIP does not use the official data split. Hence, we refer to the results reported in KAD [2I] rather than those from the original MedKLIP [5]. To ensure fair comparison with Med- KLIP, we reimplemented G2D for this experiment under the MedKLIP configuration. as detailed in Sec A.5 CXR images are resized to 256 x 256 (1.. During fine-tuning, all model parameters are updated, including the pre-trained vision encoder and linear classifier. The Adam optimizer is used with learning rate of 10-1 and batch size of 64 for 50 epochs. Evaluation is based on
Medical Image Fine-tuned Classification. In alignment with
the AUC score, adhering to the protocol outlined in 8 9 12).

---

Table 1: Results semantic segmentation and object detection. Best results are highlighted in bold, with denoting mAP values < 1%. Methods with x use disease-level annotations. ' indicates object detection not deployable with encoder-decode architecture. The McdKLIP results in this table differ from the original work [5] because McdKLIP fine-tur ine-tuned the encoder in their original study, whereas other methods froze the encoder. To ensure fairness, we reimplemented MedKLIP with the frozen encoder for comparison this table. Additionally, for a fair comparison specifically with MedKLIP, we compare G2D with MedKLIP under its original configuration in Tab andd Sec A.5]
Tasks
Semantic Sogmentation (Diee)
Object Desection (mAP)
Datasets Methods
SIIM
RSNA
RSNA
Object
1%
10%
100%
1%
10%
100%
1%
10%
100%
1%
10%
100%
Random Init
9.0
28.6
54.3
6.9
10.6
18.5
1,0
4,0
8.9
0.5
4.4
ImageNot Init
10.2
35.5
63,5
34.8
39.9
64.0
3.6
8.0
15.7
2.9
8.3
ConVIRT [4]
25.0
43.2
59.9
55.0
67.4
67.5
8.2
15.6
17.9
8.6
15.9
GLoRA 8
35.8
46.9
63,4
59.3
67.5
67.8
9.8
14.8
18.8
10.6
15.6
GLoRTA.MIMIC X
37.4
57.1
64.0
60.3
68.7
68.3
11.6
16.1
24.8
8.90
MGCA [X
49.7
59.3
64.2
63.0
68.3
69.8
12.9
16.8
24.9
12.1
19.2
M-FLAG
52.5
64.8
64.6
69.7
70.5
13.7
25.4
12.4
19.3
MedKLIPP 5
50.2
60.8
63,9
66.2
69.4
71.9
8.9
16.3
24.5
7.1
11.6
Ours (encoder) Ours (encoder-decodee))
62.6
63.1
66.8
70.9
72.6
75.1
15.9
21.7
27.2
13.1
20.4
65.6
66.9
68.4
72,8
73.4
76.9
Medical Image Linear Classification. In strict accordance with the configuration in 8 4 9. this task is conducted on the CheXpert [35), RSNA [31], and COVIDx [37] datasets. We only update a randomly initialized linear classification layer, while the pre- mre-trained vision encoder remains frozen. For fair evaluation, we employ AUC scores on CheXpert and RSNA, along with accuracy metries on COVIDx, as mentioned in g 9 Apart from zero-shot image classification and visual grounding, we fine-tune using 1%, 10%, 100% of the training data for all downstream tasks. Detailed settings, including implementation and data splits, outlined in Sec A.4
4.2 Performance on Visual Localisation Tasks
In Tab I following 16 38). we evaluate G2D alongside other SOTA approaches on two pivotal visual localisation tasks: semantic segmentation and object detection. The aim is to assess the efficacy of the dense visual features learned.
Initially, we transfer only the encoder weights from the pre-trained G2D for the segmentation task, adhering to the protocols of | b). In this setup, our approach consistently achieves the highest performance across all data fractions for both SIM 32 and RSNA datasets 31. To assess the impact of the visual decoder pre-trained with the PS pretext task, we transfer the weights of both the encoder and decoder from G2D for the segmentation task, resulting in striking outcomes. Remark- ably, with just 1% of training data, G2D surpasses the performance of all peer methods, even those trained with a full 100% of training data. This observation underlines the fact that the pixel-level pretext task, PS, significantly improves the quality of dense visual features derived from VLP, which provide advantages for the downstream segmentation task.
In object detection, our method consistently outperforms existing methods across all data fractions for both RSNA and Object-CXe CXRR datasets [3]] 33). Notably, G2D achieves 3.8% mAP on the Object-CXR dataset with just 1% of the data for fine-tuning, significant leap from other methods that scarcely reach a 1% mAP.
These results highlight the efficacy of our proposed model, G2D, and the pretext task, PS, especially in semantic segmentation tasks that rely on dense visual features. PS not only enables G2D to learn visual representations in the encoder-decoder structure but also reduces the gap between pre- training and downstream tasks. By enhancing the encoder's ability to capture global and dense features simultaneously. PS surpasses existing approaches, proving particularly advantageous for object detection tasks that heavily rely on dense features [39]]
4.3 Performance on Vision-Language Understanding
In Tab we evaluate the efficacy of G2D on vision-language understanding tasks, zero-shot visual grounding and zero-shot image classification. For the zero-shot visual grounding task, our proposed method outperforms peer approaches. Specifically, on the SIIM dataset (32, it achieves a leading Dice score of 5.1. This dominance persists in the RSNA dataset (31,, where our method reaches
7

---

Table 2: Comparison between G2D (ours) and various other medical VLP methods in vision-lange langggg under- standing tasks, with the best results emphasized in bold. Methods marked with * utilize extra annotated data during pre training. Y indicates that the original work did not report the results. Notably, KAD 21 does not report ACC for the CheXpert dataset.
(a) Results of zero-shot visual grounding task.
(b) Results of zero-shot image classification task.
Task
Zoro-shot Visual Grounding
Tink Methvals
Zawsliut Image Closiicctiin
Datasets Methods
SIM loU
RSNA IoU
ABC
Pi
ACT
ATC
F)
ACC
AUC
F1
ACC
ACCC
Fi
Recall
Dice
Recall
Dice
GLeRIA 8
BioViL
23.8
1.2
1.7
2.1
85.2
83.3
21.8
34.7
CEARIA
Bieval B
71.5
82.8
56.3
460
76.
71.3
53.4
708
58.2
68.6
69.1
48.5
56.2
51.
65.2
17.4
50.3
68.3
09.3
75.0
51.0
46.3
MedKLIP
35.6
2.1
4.0
86.6
31.7
46.5
MaKKIPP
86.3
63.4
90.0
99.2
08.3
54.3
12.6
244
716
57.9
KAD
13.
818
90.1
61.
Ours
37.7
39
5.1
88.4
33.5
47.7
Our
91.5
99.7 09.3
3311
Table 3: Evaluation of image classification fine-tuning on the CXR14 dataset is conducted, with all metrics presented as AUC scores, where the mean metric is macro-averaged. Best performances are highlighted in bold. Methods marked with utilize annotated data for pre-training. MedKLIIP's results here differ from the original study [5] as did not utilize official test split, unlike KAD [2]]. We use the result of MedKLIP reported by KAD (1], which reimplemented MedKLIP on the official test set for fairness. All results in this table are sourced from KAD [21]. To compare fairly with MedKLIP, we assess G2D against its original
configuration in Tab and See A.5
Pleural Thicken
Data fracion
Akllectsis
Cardiomegaly
Infitration
Pocunonia
Pocumetborox
Comoldation
Emphysena
Mai
Ellosion
Mass
Nodule
Eikema
Filrosis
Method
Random Init
58.1
55.7
57.7
63.6
61.6
55.0
60.2
$7.1
58.2
60.8
63.3
53.4
63.7
56.8
46.0
1%
COnVIRT
ImageNet Inic
4
63.5
64.9
66.2
64.2
72.1
57.0
59.0
58.5
60.0
6888
62.6
62.4
66.8
61.5
68.0
70.7
63.1
64.5
GLLRRIA
59.7
59.7
74.1
64.6
55.9
60.7
55.0
55.8
59.2
BioViL
40
57.9
55.5
56.4
72.2
65.0
56.7
54.6
62.6
56.0
65.7
68.1
51.6
51.3
59.2
36.0
MedKI JP
1
60.9
65.5
59.0
74.5
64.3
55.0
61.1
60.9
59.9
65.9
68.2
53.5
64.8
59.3
40.0
KAD*
21
78.7
77.0
88.2
82.9
69.2
75.1
73.5
86.1
72.7
89.3
74.3
69.2
79.1
78.1
70.2
75.4
74.0
86.5
72.9
90.2
74.4
Random Init
69.1
68.2
76.6
74.6
67.4
62.3
$8.0
63.6
72.8
67.8
78.0
64.7
11.5
65.3
77.1
10%
ConVIRT 4
ImageNet Init
72.6
77.1
70.9
79.8
76.9
69.3
68.4
74.8
69.3
65.6
63.0
82.8
79.3
70.1
67.1
81.4
76.7
74.9
87.1
76.7
72.9
71.9
71.1
81.0
GLeRIA 8
74.3
72.1
80.8
80.0
68.7
73.3
67.5
65.55 8
77.9
67.6
79.7
79.9
78.7
69.3
78.7
BioViL 50
72.7
70.3
78.5
79.0
66.6
71.8
67.1
66.5
76.7
68.4
79.9
76.1
74.8
65.3
76.3
MedKL PP 5
74.8
72.9
80.2
79.3
69.8
71.9
68.1
66.6
79.6
79.5
75.6
71.3
81.9
KAD*
80.7
77.6
71.8
78.3
73.7
87.2
75.0
90.3
80.7
72.3
95.1
Ours
81.1
78.4
89.3
83.7
72.2
7888
72.3
74.1
87.88
75.3
84.0
90.4
80.8
72.5
95.4
Random Init
79.0
75.0
87.9
81.5
69.1
79.8
72.6
70.3
82.6
73.1
83.9
83.5
80.7
75.4
90.3
100%
ConVIRT 4
ImageNet
80.8
77.1
86.7
82.5
70.3
81.8
76.1
71.9
72:2
85.7
74.7
85.4
90.1
81.9
80.9
77.1
90.9
GLoRIA i
80.0
76.0
85.5
81.8
70.0
81.4
71.9
71.5
82.8
73.9
83.2
88.7
81.3
76.7
92.1
BioViL 40
80.0
76.5
87.1
81.4
69.7
81.9
75.2
71.0
84.5
74.2
84.2
87.1
82.1
75.9
88.8
McdKLIIP
76.4
69.7
71.2
83.9
75.1
87.9
81.7
77.7
KAD
82.5
78.5
89.7
81.0
71.3
83.6
77.1
74.0
87.4
75.3
36.0
91.6
82.9
77.8
96.
Ours
83.1
79.9
90.3
84.5
718
84,2
78,0
74.
87.7
75.6
86.9
92.0
83.1
78.2
96.5
Dice score of 47.7, surpassing other SOTA approaches. When examining zero-shot image classifi- cation, our method again shows its superiority across the AUC, F1, and ACC metrics on both the RSNA (31) and SIM datasets [32]. Such consistent and superior outcomes underscore the adapt- ability and effectiveness of G2D in handling vision-language understanding tasks, indicating that integrating PS into G2D can enhance not only uni-moda but also cross-modal tasks.
4.4 Performance on Visual Recognition Tasks
In our final assessment focused on visual recognition, Tab demonstrates our method's consistent supremacy on the CXR14 dataset (36] for fine-tuned disease classification across 1%, 10%, and 100% training data. Similarly, Tab underscores that G2D achieves the highest performance on the CheXpert, RSNA, and COVIDx datasets [35] 31 37 for linear evaluation across all training data ratio. Notably, G2D consistently outperforms even those methods like MedKLIP and KAD [1] that leverage additional disease-level annotations during pre-training stage. This demonstrates G2D's representative visual features, suggesting that enhancing dense representation learning via PS can also improve results in tasks primarily anchored on global representation.
8

---

Table 4: Linear classification results for CheXpert, RSNA, and COVIDx datasets with 1%. 10%, and 100% training data. The best results are highlighted in bold. Methods with * leverage disease-level annotations for pre-training. The evaluation metric follows (9)
Datasets (Metrie) Methods
CheXpert (AUC)
RSNA (AUC)
COVIDx (ACC)
1%
10%
100%
1%
10%
100%
1%
10%
100%
Random Init
56.1
62.6
65.7
58.9
69.4
74.1
50.5
60.3
70.0
ImageNet Init
74,4
79.7
81.4
74,9
74.5
76.3
64.8
78.8
86.3
VIRT [
85.9
86.8
87.3
77.4
80.1
81.3
72.5
82.5
92.0
GLoRIA 8
86.6
87.8
88.1
86.1
88.0
88.6
67.3
77.8
89.0
ALORIA-MIMER g
87.1
88.7
88.0
87.0
89.4
90.2
66.5
80.5
88.8
MGCA 9
87.6
88.0
88.2
88.6
89.1
89.9
72.0
83.5
90.5
MRM 12
88.5
88.5
88.7
91.3
92.7
93.3
79.3
90.8
McdKLIP
5
86.2
86.5
87.7
87.3
88.0
89.3
74.5
85.2
90.3
Ours
89.7
90.4
91.1
92.2
92.9
93.6
76.6
88.2
93.4
Table 5: Results of various ablation experiments. The best results are bolded.
(a) Loss for the decoder. 'None' indi- (b) Threshold for constructing (c) Ablation of the number of di- cates Encoder- Only visual backbones. pseudo segmentation masks. mensions of projectors.
Decoder
SIMM Dice
RSNA mAP
Thnshild
CXR14 AUD
Num Dim
RSNA
AUC
MAP
MAP
Nose
492+1.5
11.711
77.1115
15% percentlle
666117
189188
128
65.6=17
159408
781=12
Reconstraciion
53411.
130109
77.322.1
pencmilc
$3.122
141312
183120
256
64.9199
1611111
783115
Pscudo Seg (Qurs)
65611.7
159:08
79111
mediaa
588816
125623
75661.1
512
64.5113
15.7+1.
760+1.
GMM [X]
59.2315
129514
15.2119
(d) Ablation of multi-hear attention
(c)
Number of attention
(f) Refinement of Pscudo Segmenta- tion Masks
maps aggregation.
heads.
Method
SIM
RSNA
CXR14 AUC
SUM Doc
SIM Dice
RSNA
CNR14
Dice
mAP
mAP
AUC
AUC
w Ageregation
6.76.117
15.9 0.8
7..1:112
63A420
142214
78261.0
who body mark
6144115
153121
784416
w/ Aggregation
62. 12.1222
13.5
77,5e22
647811
151423
who olge smmothinng
64.1812
15221.7
78.5123
buhh (Dan)
656:
159:00
1911112
65.3216
15.4109
78.6199
4.5 Ablation Studies
Pseudo Segmentation vs. Reconstruction. In Tab 5a we evaluate the impact of the proposed PS pretext task in comparison to pixel reconstruction and models without a decoder-level constraint. The model pre-trained with PS outperforms the other two approaches across all three downstream tasks, particularly in semantic segmentation. While the model pre-trained with a pixel reconstruction constraint exhibit improved performance compared to unconstrained variants, such models still un- derperform the model with the PS constraint. These results underscore the effectiveness of decoder- level pretext tasks and suggest that an emphasis on high-level semantics, derived from PS, is more beneficial than focusing on the low-level semanties from pixel reconstruction. The PS potentially reduces the disparity between features learned through VLP and those required by downstream mantic segmentation tasks. It also enables the model to acquire more representative features that are beneficial for various tasks.
Threshold of Pseudo Mask Construction. As shown in Tab 5b performance varies with different thresholds, with the 85% percentile threshold proving most effective across all three downstream tasks. Despite employing the Gaussian Mixture Model (GMM) for pseudo mask creation, as sug- gested by 42). its performance is still surpassed by the 85% percentile approach. This indicates that the original attention map might contain noise, and a higher threshold is beneficial for generating more effective pseudo masks.
Furthermore, Tab 5d highlights the importance of aggregating multi-head attention maps for mask construction. Given the absence of explicit semantic supervision in the PS pretext task, not aggre- gating these maps leads to the creation of multiple pseudo masks. This excess of masks introduce ambiguous training objectives for VLP.
Impact of Mask Refinement. Refinement of the pseudo masks affects the model's 's efficacy, as shown in Tab 5i Performance tends to decrease when either the body mask is omitted or edge smoothing is not applied. However, integrating both these strategies, as we implement in G2D, yields optimal results. This underscores the vital role of pseudo mask refinement in enhancing model performance.
Ablation on Hyperparameters. We further ablate the number of attention heads and projector di- mensionality. Performance improves with more attention heads, peaking at 3 before slightly declin-

---

ing at 4 (e). Optimal segmentation and classification results are achieved with 228dimensional projectors. While 256 dimensions provide slight benefits for object detection, they reduce perfor- mance in other tasks (Tab 5c). Projectors of 512 dimensions do not yield further gains. Thus, we select attention heads and 128 228-dimensional projectors for an optimal balance of complexity and effectiveness.
5
Conclusion
In this study, we introduce G2D, a novel medical VLP framework for learning global and dense-level representations. Our proposed pixel-level pretext task, pseudo segmentation, leverages a refined at- tention map to predict a pseudo mask, capturing dense visual features during VLP without requiring additional trainable parameters for its construction. Our model pretrained with this pretext task achieves superior performance across five diverse medical imaging tasks and outperforms methods pretrained with annotated data [5] 21]. especially in semantic segmentation. Specifically, on the SIIM [32] dataset, G2D, when fine-tuned with only 1% of the training data, outperforms other med- ical VLP approaches that utilize the full 100% training set. We anticipate that G2D will inspire further exploration of novel and clinically-guided pretext tasks for medical VLP.
References
[1] O. Ronneherger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmen- tation," in Medical Image Computing and Computer-Assistee InterventonnMIICAA 2015: 18th Interna- tional Conference, Munich, Germany, October 5-9, 2015, Proceedings, Port III 18. Springer, 2015, pp- 234-241.
[2] C. Liu, S. Cheng, M. Shi, A. Shah, W. Bai, and R. Arcucci, "Imitate: Clinical prior guided hierarchical Ausion-language pre-training," arXiv preprint arXiv:3310007355 2023.
[3] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark et al., "Learning transferable visual models from natural language supervision," in International Conference Machine Learning. PMLR, 2021, pp. 8748-8763.
[4] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz, "Contrastive learning of medical visual representations from paired images and text," arXiv preprint arXiv:2010.00747 2020.
[5] C. Wu, X. Zhang, Y. Zhang, Wang, and W. Xie, "Medklip: Medical knowledge enhanced language- image pre-training," medRxiv, pp. 2023-01, 2023.
[6] C. Liu, Cheng, Chen, M. Qiao, W. Zhang, Shah, W. Bai, and R. Arcucci, "M-flag: Medical vision-language pre-training with frozen language models and latent space geometry optiiiattion" in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2023, pp. 637-647.
[7] E. Tiu, E. Talius, P. Patel, C. P. Langlotz, A. Y. Ng. and P. Rajpurkar, "Expert-level detection of pathologies from unannotated chest x-ray images via self superviee learning." Nature Biomedical Engineering, pp. 1-8, 2022.
[8] S.-C. Huang, L. Shen, M. P. Lungren. and S. Yeung. "Gloria: A multimodal global-local representa- tion learning framework for label-efficient medical image recognition." in Proceedings of the IEEE/CVF International Conference on Computer Vixion, 2021, pp. 3942-3951.
[9] F. Wang, Y. Zhou, S. Wang, V. Vardhanabhuti, and L. Yu, "Multi-granukan cross-modal alignment for generalized medical visual representation learning," arXiv preprint arXiv:2210.0604 2022.
[10] C. Liu, A. Shah, W. Bai, and R. Arcucci, "Utilizing synthetic data for medical vasion-language pre- training: Bypassing the need for real images," arXiv preprint arXiv:3310.07027. 2023.
[11] P. Cheng, L. Lin, J. Lyu, Y. Huang, W. Luo. and X. Tang. "Prior: Prototype representation joint learning from medical images and reports." arXiv preprint arXiv:2307.12577 2023.
[12] H.-Y. Zhou, C. Lian, L. Wang, and Y. Yu, "Advancing radiograph representation learning with masked record modeling," in The Eleventh International Conference on Learning Representations.
[13] Y. Liu, S. Zhang, J. Chen, K. Chen, and D. Lin, "Pixmim: Rethinking pixel reconstruction in masked image modeling," arXiv preprint 23X3::2303002416 2023.
[14) K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, "Masked autoencoders are scalable vision learners." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. pp. 16 0000-6 009.
10

---

[15] Y. Liu, S. Zhang. J. Chen, Z. Yu, K. Chen, and D. Lin, "Improving pixel-based mim by reducing wasted modeling capability,"" in Proceedings of the IEEE/CVF International Conference Computer Vision, 2023, pp. 5361-5372.
[16] L. H. Li, P. Zhang, H. Zhang. J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang et al., "Grounded language-image pre-training." in Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, 2022, pp. 10965-10975.
[17] Y. Gao, J. Liu, Z. Xu, J. Zhang. K. Li, R. Ji, and C. Shen, "Pyramidclip: Hierarchical feature alignment for vision-language model pretraining," Advances in neural information processing systems, vol. 35, pp- 35 959-35 35 970, 2022.
[18] C. Zhou, C. C. Loy, and B. Dai, "Extract free dense lahels from clip," in European Conference on Com- puter Vision. Springer, 2022, pp. 696-712.
[19] H. Luo, J. Bao, Y. Wu, X. Hc, and T. Li, "Segelip: Patch aggregation with Icarnable centers for open- vocabulary semantic segmentation," in International Conference Machine Learning. PMLR, 2023,
pp. 23 033--33444.
[20] Z. Wan, C. Liu, M. Zhang, J. Fu, B. Wang. S. Cheng, L. Ma, C. Quilodrán and R. Arcueci, "Med- unie: Unifying cross- loosslln medical vision-language pre-training by diminishing bias," arXiv preprint arXiv:2305. 19894, 2023.
[21] X. Zhang, C. Wu, Y. Zhang, W. Xie, and Y. Wang. "Knowledge-enhanced visual- language pre-training on chest radiology images," Nature Communications, vol. 14, no. 1. p. 4542, 2023.
[22] Huang, H. Zhou, C. Li, H. Yang. J. Liu, and S. Wang, "Enhancing representation in radiography- reports foundation model: A granular alignment algorithm using masked contrastive learning," arXiv preprint 2xXiv:2399.0544
2023.
[23] J. Ma, Y. He, F. Li, L. Han, C. You, and B. Wang, "Segment anything in medical images." Nature Com- munications, vol. 15, no. 1. p. 654, 2024.
[24] A. E. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, Y. Peng. Z. Lu, R. G. Mark, S. J. Berkowitz, and S. Horng, "Mimic-cxr-jpg, a large publicly available database of labeled chest radio- graphs," arXiv preprint arXiv:1901 07042, 2019.
[25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and I. Polosukhin, "Attention all you need," Advances in neural information processing systems, vol. 30, 2017.
[26] C. Ouyang, C. Bifti, C. Chen, T. Kart, H. Qiu, and D. Rueckert, "Sper-superiisionn with superpixels: Training few-shot medical image segmentation without annotation," " in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16. Springer, 2020, pp. 762-780.
[27] F. Isensee, F. Jaeger, A. Kohl, J. Petersen, and K. H. Maier-Hein, "nnu-net: a self-configuring method for deep learning-hasd hased biomedical image segmentation," Nature methods, vol. 18, no. 2, pp. 203-211, 2021.
[28] C. Tomasi and R. Manduchi, "Bilateral filtering for gray and color images," in Sixth international confer- ence computer vision (IEEE Cat. No. 98CH3627I). IEEE, 1998, pp. 839 846.
[29] A. E. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, Y. Peng. Z. Lu, R. G. Mark, S. J. Berkowitz, and S. Horng, "Mimic-exr- jpr. a large publicly available database of labeled chest radio- graphs." arXiv preprint arXiv:1901 000022, 2019.
[30] E. Alsentzer, J. R. Murphy, W. Boag, W.-H. Weng. D. Jin, T. Naumann, and M. McDermott, "Publicly available clinical bert embeddings." arXiv preprint arXiv:1904.00 03323, 2019.
[31] G. Shih, C. C. Wu, S. S. Halabi. M. D. Kohli, L. M. Prevedello. S. Cook, A. Sharma, J. K. Amorosa, el al., "Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia," Rudiology: Artificial Intelligence, vol. 1, no. 1,
V. Arteaga, M.
p. e180041, 2019.
[32] C. Steven G. Langer, PhD and M. George Shih, MD, "Siim-acr pneumothorax segmentation." 2019.
[33] J. Healthcare, "Object-exr-auoonee detection of foreign objects on chest x-rays," 2020.
[34] Redmon and A. Farhadi, "Yolov3: An incremental improvement," arXiv preprint 104.0:1804.02777.
2018.
[35] J. Irvin, Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya et al., "Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison," in Proceedings of the AAAI conference on artificial intelligence, vol. 33, 2019, pp. 590- 597.
11

---

[36] X. Wang, Y. Peng, L. Lu. Z. Lu, M. Bagheri, and R. M. Summers, "Chestx-ray8: Hospital-scale chest x-ray database and benchmarks weakly-supervised classification and localization of common thorax diseases," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2097-2106.
[37] L. Wang, Z. Q. Lin, and A. Wong. "Covid-net: A tailored deep convolutional neural network design for detection covid-19 19 cases from chest x-ray images." Scientific reports, vol. 10, no. 1. pp. 1-12, 2020.
[38] H. Zhang, P. Zhang, X. Hu, Y.-.. Chen, L. Li, X. Dai, L. Wang, L. Yuan, J.-N. Hwang, and J. Gao, "Glipv2: Unifying localization and vision- laniioge understanding," Advances in Neural Information Pro- cessing Systems, vol. 35, pp. 067-366 080, 2022.
[39] T.-.. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection," in Proceedings of the IEEE conference computer vision and pattern recognition, 2017, pp. 21 17-2125.
[40] B. Boecking, N. Usuyama, S. Bannur, D. C. Castro, A. Schwaighofer, S. Hyland, M. Wetscherek, T. Nau- mann. A. Nori. J. Alvarez-Valle et al., "Making the most of text semantics to improve biomedical vision- language processing." in European conference computer vision. Springer, 2022. pp. 1-21.
[41] C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, "Medklip: Medical knowledge enhanced language- image pre-training," medRxiv, pp. 2023-01 01, 2023.
[42] Dombrowski, Reynaud, M. Baugh, and Kainz. "Foreground backguruud separation through concept distillation from generative image foundation models," in Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, 2023, pp. 988-998.
[43] A. Saporta, Gui, Agrawal, A. Pareek, S. Q. Truong, C. D. Nguyen, V.-D Ngo, J. Seckins, G. Blankenberg, A. Ng er al., "Benchmarking saliency methods for chest x-ray interpretation," Nature Machine Intelligence, vol. 4, no. 10, pp. 867-878, 2022.
12

---

A Appendix supplemental material
A.1 Limitations and Future Work
Our work primarily concentrates on learning dense visual representations from pseudo masks, which are generated from attention masks under language supervision, Due the weak supervision signal, the pseudo masks may not effectively associate each pixel with the corresponding text tokens, poten- tially capping the performance of our method. Currently, our approach involves learning both global and pixel-level representations through VLP. In future studies, we aim to delve into regional vi- sual representations during VLP to establish more precise correlations between specific chest X-ray (CXR) regions and phrases in radiology reports.
A.2 Broader Impacts
Our G2D model offers an effective approach for the automatic diagnosis of chest X-ray abnormali- ties using small amount of annotated data. This can help decrease the burden on radiologists and enhance healthcare in underprivileged regions. However, medical data, such as chest X-rays and radiology reports, might include sensitive or potentially harmful information. We strongly advise thorough examination of the data prior to using our model in real-world applications.
A.3 Pre-training Implementation Details
750
Report:
There is no focal consolidation,
pleural effusion or pneumothorax. Bilateral nodular opacities that most likely represent nipple shadows. The cardiomediastinal silhouette is normal. The imaged upper abdomen is unremarkable. Chronic deformity of the posterior left sixth and seventh ribs are noted. No acute cardiopulmonary process.
Figure 3: An exemplar pair of X-ray image and associated clinical report from the MIMIC-CXR dataset 24
The chest X-ray (CXR) images from the MIMIC-CXR dataset 229 are resized dimensions of 256 x 256 and subsequently center-cropped to 224 x 224, adhering to the procedure described
in 4 8 19, with example shown in Fig 3 The intensity of each image is normalized to range of [0. 1]. During the pre-training stage, we employ data augmentation techniques including random grayscale, random perspective, and auto contrast adjustments, using the PyTorch vision library]]
A.4 Downstream Task Implementation Details
The data split information into train/valid/test sets are described in Tab. 6. For all downstream tasks, except from zero-shot image classification and visual grounding, we train with 1%, 10%, 100% of the training set. The downstream tasks are deployed on a 40G A100 GPU.
https:/yyyaaccoaoyyiiioossbbbb/p html
13

---

Table 6: Details on Data Split: The symbol ' denotes that training/validationn data not required for the zero-shot tasks.
Task
Dataset
Split
Train
Valid
Test
CheXpert
35
35
186,027
5,000
202
Linear Classification
RSNA
31
16,010
5,337
5,337
COVIDx
23988
5998
400
Fine-tuned
CXR14
36
77,872
8,652
25,596
Classification
Image
RSNA
31
8 9
16,010
5.337
5,337
Segmentation
SIIM
8,433
1,807
1,807
Object
RSNA
16,010
5,337
5,337
Detection
ObjectCXXR
3
6,400
1,600
1,000
RSNA
11
5
5,337
Zero-shot shot Image
SIIM
32
5
1,807
Classification
CXR14
16
36
21
25,596
CheXpert
15
43,
21
/
500
Zero-shot Visual
RSNA
31
/
5,337
Grounding
SIMM
32
1,807
A.4.1 Visual Localization
Medical Image Segmentation. For the segmentation tasks on the RSNA 31 and SIIM 32 datasets, we initially employ the vision encoder from the pre-trained model. Additionally, we trans- fer both the vision encoder and decoder from the pre-trained model, and proceed to train the segmen- tation network. We implement early stopping during the training process, limiting it to 50 epochs. A learning rate of 2e-4 and a weight decay of 0.05 are adopted. Adam is utilized as the optimizer, with 31 and 32 values set at 0.9 and 0.999, respectively. For the SIIM (32) dataset, the default batch size is set at 8, while for the RSNA 31 dataset, it is set at 16. All configurations strictly adhere to the protocol provided in (
Medical Image Object Detection. The pneumonia detection task on RSNA 31 and foreign ob- jects detection task on Object-CXR [33) datasets are executed on single A100 GPU. For both datasets, early stopping is implemented during the training process, limited to 50 epochs. AdamW is employed as the optimizer across both datasets. For the RSNA (31 dataset, batch size of 8 is set for 1% data fine-tuning with a learning rate of 2e-4, weight decay of le-6, and B1, B2 values of 0.9 and 0.999, respectively. For 10% and 100% data fine-tuning, the batch size is adjusted to 16, with a learning rate of 5c-4, a weight decay of 1e-6, and the same 31. 32 values. Similarly, for the Object-CXX [33] dataset, a batch size of 8 is set for 1% data fine-tuning, with the identical learning rate, weight decay, and 8 values as the RSNA dataset. For 10% and 100% data fine- aune-tunin, the batch size is adjusted to 16, again with learning rate of 5e-4, a weight decay of 1e-6, and B1, B3 values of 0.9 and 0.999. The IOU and NMS thresholds are set at [0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75] and 0.5, respectively. All configurations are in striet compliance with the protocol delineated in (9.
A.4.2 Vision-Language Understanding
Zero-shot Image Classification. The original CXR image goes through a two-step preprocessing routine. Initially, it resized to the dimension of 256 x 256, and then center cropped to 224 x 224. Following the methodologies outlined in [8 19, all pixel values are normalized to the range [0, 1]. The resulting resized image then fed through a visual encoder, followed by a visual projector to generate the image embedding v: Simultaneously, the prompts are fed into a text encoder to obtain text embeddings 1,. The classification evaluation hinges measuring the cosine similarity between the image and text embeddings for each prompt associated with a specific class. The classification outcome is determined by comparing the cosine similarities. Specifically, if the cosine similarity between the image embedding and the positive prompt (e.g., disease) surpasses that between the image embedding and the corresponding negative prompt (e.g., No disease), the outcome is deemed positive. Conversely, if the reverse holds true, the outcome is negative. The prompt is designed following 7
Zero-shot Visual Grounding. To execute this task, we adhere to the Bio pipeline as described in [40]. The visual grounding task can be regarded as a pixel-level classification task, driven by the
14

---

text prompt and the dense visual embedding. The image is fed into the visual encoder to acquire the dense feature map V; from the final convolutional layer of the image encoder, yielding a shape of C'x H x W. At the same time, the prompt is processed through the text encoder and projected into the cross-modal space, resulting in 1. The cosine similarity between 1, and all elements of V; the channel level generates a similarity map. This map is then resized to match the original image size and utilized as the segmentation results to evaluate the zero-shot grounding performance.
A.4.3 Visual Recognition
We conduct evaluations on the CheXpert 35 RSNA B1, COVIDx 37 and CXR14 datasets (36]. In alignment with previous studies 8 4 9 5 21, linear classification is implemented on CheX-
pert (35), RSNA BI], and COVIDx 37.. Here, we update randomly initialized linear layer while
keeping the visual encoder frozen. We adhere to the official test set partition from 5 21
36 for
fair comparison. During our linear classification task, training performed over 50 epochs with a learning rate of 5e-4, a batch size of 8, employing the Adam W optimizer with parameters: 81 = 0.9 and 82 = 0.999. For the CXR14 14 dataset [36], we follow the experimental setup from [21], employ- ing fine-tuned classification while updating all parameters from the visual encoder and linear layer. Images are resized to 256 x 256 and data augmentation is carried out as recommended in [2[] The AdamW optimizer is utilized with a learning rate of 1 x 10- and batch size of 64 for 50 epochs. The linear classification tasks are executed on a single A100 GPU with 40GB memory, using the vision encoder from our pre-traines traitra model the visual backbone. Fine-tuning, is carried out on the randomly initialized linear layer for 50 epochs with early stopping, maintaining a learning rate of 5e-4 and default batch size of 8. We set AdamW as our optimizer, with B1 of 0.9, (32 of 0.999, and a weight decay rate of le-6.
A.5 Comparison under MedKLIP Configuration
Table 7: Performance of CXRI4 14 Classification Fine-Tuning and Segmentation Results on SIIM and RSNA using the MedKLIP Setting [5].
CXR14 14 (AUC)
RSNA (Dice)
SIIM (Dice)
1%
10%
100%
1%
10%
100%
1%
10%
100%
MedKLIP*
77.2
78.9
83.2
70.6
71.6
75.8
66.6
72.1
79.4
G2D(Ours)
80.4
83.8
86.1
73.8
76.1
76.5
70.6
74.5
82.3
To strictly compare our work with MedKLIP [5], reimplement G2D for fine-tuning on CXR14 classification, as well as SIIM and RSNA segmentation tasks, adhering strictly to the MedKLIP configuration. This approach is necessary because the settings of MedKI.IP differ significantly from the other methods that we compare to, such as 4 g 9 6 211. Specifically, MedKLIP updates both the encoder and decoder during segmentation tasks, whereas the other methods only update the decoder and keep the encoder frozen. Moreover, MedKLIP employs its own customized data split for CXR14 classification, contrasting with KAD 21), which uses the official CXR14 dataset split.
Given these differences, comparing other methods directly under the MedKLIP setting could be seen unfair. Therefore, conducted separate comparison between G2D and MedKLIP using the MedKLIP setting. The results, presented in Tab 7 demonstrate that G2D outperforms MedKLIP across all tasks and data ratios, even within the MedKLIP setting.
A.6 Verifying Pseudo Segmentation with Semantic Meaning
To investigate whether the improvements in G2D come from learning dense visual features through pseudo segmentation (PS) or from treating PS as regularization term during pre-training, we per- turbed the semantic integrity of pseudo masks by randomly shuffling them on sample-wise basis (i.e., making images and pseudo masks unpaired). This operation detaches pseudo masks' semantic connection to the original images, ensuring that the PS task does not learn correct semantic informa- tion, but still provide regularisation to the segmentation as the pseudo mask is relatively smooth. The results are presented in Table G2D with uncorrupted pseudo masks in PS (ours) significantly out- performs the results from the shuffled alternative (unpaired images and pseudo masks), not only in
15

---

Mask Construction
SIIM
RSNA
CXR14
Dice
ALC
Table 8: Perturbation on
Pseudo Mask without Semantic Meuning (shuffled)
50.9+244
7.61:.2
63.742.1
Pseudo Masks.
Pscudo Mask with Semantic Meaning (Ours)
65.6
15.9=088
79.11122
visual localisation task but also in visual recognition task. The improved performance demonstrate that the proposed G2D indeed learns transferable visual features thanks to the semantic information provided by the pseudo masks, rather than merely treating PS as regularization mechanism.
A.7 Pseudo Mask Visualization
Right upper lobe pneumonia or mass. However, given right hilar fullness, mass resulting in post-obstructive pneumonia within the differential.
Mild pulmonary Edema with superimposed left upper lung consolidation. Moderate cardiomegaly again seen as well as tortuosity of the descending thoracic aorta.
Figure 4: Pseudo Mask Visualization. Left: Aggregated attention map. Middle: Constructed pseudo mask for the pseudo segmentation task. Red and blue arrows point to areas related to specific text descriptions. Right: Corresponding radiology report. Red and blue text emphasize regions represented in the pseudo mask.
We visualize the aggregated attention map, pseudo mask, and paired medical reports in Fig In- triguingly, without human annotations, both the attention map and pseudo mask successfully capture image regions corresponding to various report words. The pseudo masks manage to capture impor- tant parts of the image regions related to the highlighted words in the clinical reports, as indicated by the red and blue arrows Fig This suggests that the supervision signal for the PS pretext task is enriched by the clinical knowledge and high-level semantics, which explain why the PS pretext task may be better than the pixel reconstruction pretext task.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims presented in the abstract and introduction accurately repre- sent the contributions and scope the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Refer Section A.I.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificattin, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This work mainly includes empirical contributions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed experimental configurations Sections 4.1, A.3, and A.4. Our code will be released after acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data also be provided via
often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models. it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our experiments are all conducted publicty accessible datasets, and all experiment details are illustrated in Sections 4.1, A.3, and A.4. For experiment implemen- tation, we follow the official code of exisiting works, all code can be found in their official GitHub repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.com
public/guides/CCne wtwpeessooouuumisisiiiiooo conooicylllllooooooooolmoooomlll for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
www.Summiisiiioiiiiioi.ooooo for more details. /nips.cc/ppul wttccc//bbbb. htce/uulic/aae weaiiccgguie//eeeee htte://ooeSSbbmeieee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All experiment details are illustrated Sections 4.1, A.3, and A.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We report the error bars for all ablation studies.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Refer to the first part Sections 4.1, A.3, and A.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This work conducted accordance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there is a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Ys]]
Justification:
Please refer to the Section A.2.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not focus content generation and uses clinically verified datasets for all experiments.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer to Section 4.1.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, pattasets.paperswiiite paperswithcodeee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: There is no new assets released in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: This work has no human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work has no human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "vgnjuXg82b", "reviewer": "Reviewer_iPQX", "summary": "The paper proposes a new medical vision-language model, G2D, which employs vision-language alignment (VLA) and pixel alignment (PA) strategies, combined with a pseudo segmentation (PS) pre-training task, to learn global and dense visual representations from medical images. The VLA strategy is used to learn global representations of images and texts, while the PS task constructs pseudo masks through a parameter-free mechanism to facilitate the learning of dense representations. The method is comprehensively validated across five downstream tasks (image segmentation, object detection, zero-shot image visual grounding, zero-shot image classification, and fine-tuned image classification), demonstrating its effectiveness in handling both unimodal and cross-modal tasks.", "questions": "- How do you detect and correct the errors made by aggregated attention map?", "limitations": "Limitations were discussed in Section A.1", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ The paper is well-written, with the motivation, method, and results clearly presented. A minor concern is the reference format; it should be [1] instead of (1) according to the NeurIPS template.\n\n+ A significant concern with most existing works is that they operate primarily at the Image-Text Retrieval level, similar to the perceptual level of CLIP, and do not effectively capture dense features between modalities. The G2D model addresses this issue by integrating Vision-Language Alignment (VLA) and Pseudo Segmentation (PS) tasks to facilitate simultaneous learning of global and dense visual features. This multi-level feature learning significantly enhances the model's performance in tasks requiring dense feature perception, such as segmentation.\n\n+ During pre-training, the G2D method utilizes only image-text pairs without the need for additional annotated data. By generating pseudo masks on the fly through the PS task, it reduces the cost and complexity associated with data annotation.\n\n+ The G2D method is novel, and the experiments are robust. Experimental results on five medical imaging tasks involving 25 diseases demonstrate that the G2D model outperforms existing models, even with minimal fine-tuning data. Notably, in segmentation tasks requiring dense visual features, G2D achieves excellent results with just 1% of the training data for fine-tuning.", "weaknesses": "Major concerns:\n\n- The attention maps could introduce errors in pseudo mask, and these errors may propagate throughout the training process. To address this, a clear validation strategy needs to be outlined. For instance, in Figure 2, aggregated attention map might incorrectly highlight irrelevant regions. It is essential to establish methods for **detecting** and **measuring** these errors to ensure the reliability of the model. I hope the authors could quantify the errors in aggregated attention map and pseudo mask during the rebuttal period.\n\nMinor concerns:\n\n- The training and validation of the model rely on specific datasets, which may introduce biases and potentially affect the model's generalizability to different datasets.\n\n- It is uncertain whether the method can be effectively extended to vision-language tasks involving 3D imaging (e.g., CT and MRI), presenting a limitation in its current scope of application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedbacks!\n>Detecting and measuring the error of pseudo mask (W1, Q1)\n- In G2D, we aim to design pseudo mask for learning dense visual feature from pseudo segmentation task during medical vision-langauge pre-training (VLP), rather than directly guess the semantic mask for supervised learning. \n- Since the MIMIC-CXR dataset, which is used for pretraining, does not have segmentation mask annotations, it is infeasible to directly evaluate the accuracy of the pseudo mask derived from the G2D method.\n- In Appendix A.7, where we conducted a quality check on several samples and visualized two examples in Figure 4. We observed that G2D is capable of learning the image regions of interest that correspond with the entire report, using only language and pseudo mask supervision. However, a detailed quantitative evaluation would require laborious work by clinicians, and we plan to consider this in future studies.\n\n>The training and validation of the model rely on specific datasets, which may introduce biases and potentially affect the model's generalizability to different datasets.(W2)\n- We compared our approach with well-established works such as GLoRIA (ICCV 2021), MGCA (NeurIPS 2022), MedKIP (ICCV 2023), and KAD (Nature Communications 2023). All these studies utilize the MIMIC-CXR dataset, which is commonly used for 2D medical VLP.\n- To ensure a fair comparison with existing methods, we strictly adhered to their experimental settings [1,2,3,4], using the same datasets for both pretraining and downstream evaluation.\n- Furthermore, due to limitations in publicly accessible datasets, MIMIC-CXR is the only large-scale medical image-text dataset, containing over 200,000 samples, available for implementing medical VLP. We hope the research community will release more publicly available datasets for VLP to reduce bias and enhance model generalizability.\n\n>It is uncertain whether the method can be effectively extended to vision-language tasks involving 3D imaging (e.g., CT and MRI), presenting a limitation in its current scope of application. (W3)\n- Our method can be easily adapted to 3D imaging modalities by replacing the 2D image encoder with a 3D version. However, there is currently no public large-scale 3D image-text dataset comparable to MIMIC-CXR, which has over 200,000 samples, for implementing 3D medical VLP. We note that scaling our proposed framework to native 3D is straightforward because the pseudo masks are derived from attentions (see Section 3.2). We will explore the potential of our work further if such datasets become publicly accessible.\n\n[1] Huang, Shih-Cheng, et al. \"Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition.\" ICCV. 2021\\\n[2] Wang, Fuying, et al. \"Multi-granularity cross-modal alignment for generalized medical visual representation learning.\" NeurIPS 2022\\\n[3] Wu, Chaoyi, et al. \"Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis.\" ICCV 2023\\\n[4] Zhang, Xiaoman, et al. \"Knowledge-enhanced visual-language pre-training on chest radiology images.\" Nature Communications 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jyhquw0TsQ", "reviewer": "Reviewer_bXf3", "summary": "The paper proposes an encoder-decoder medical VLP approach for global-to-dense visual representation learning. Pseudo segmentation is adopted for dense level learning. Rich experiments validate the effectiveness of the proposed method.", "questions": "Is it not applicable to compare with MGCA and MRM in the CXR14 dataset?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The motivation behind the work is clear. Pseudo-segmentation supervision is effective, which is validated by experiments.\n2. The experiments are rich and ablation analysis shows the contributions of each component and design.\n3. The illustrations are clear and easy to understand.\n4. The improvements are consistent and sometimes substantial.", "weaknesses": "1. The comparisons with MGCA and MRM in the CXR14 dataset are not included in Table 3, but Table 4 includes the comparisons with MGCA and MRM. What are the reasons behind this?\n2. Transformer-based vision encoder is not analyzed.\n3. The balance between VLA and PA losses is not analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedbacks!\n>Comparing with MGCA and MRM on CXR14 datasets (W1,Q1)\n- In Table 3, we directly reference the results from the KAD study to ensure a fair comparison, as KAD[1] uses the official data split for CXR14. It's important to note that the KAD[1] study does not include results for MGCA[2] and MRM[3].\n- In the MGCA[2] study, results on CXR14 are not reported, which makes direct comparison challenging. Meanwhile, the MRM study, although it includes experiments on CXR14, uses its own data split rather than the official split provided by CXR14. This leads to potential biases when comparing it with existing methods with official split， such as KAD. Therefore, in Table 3, we only compare our results with those methods that have reported outcomes using the official CXR14 split, as documented by the KAD study.\n- To comprehensively compare our G2D method with MGCA and MRM, we re-implemented both on CXR14 using the official split as employed by KAD. For finetuning, we utilized their officially provided pretrained weights. Additionally, we used the official finetuning code from the MRM GitHub repository, making a single modification: we replaced their own data split with the official split used by KAD. The results are shown in the table below. As the table indicates, G2D substantially outperforms both MGCA and MRM on the CXR14 dataset using the official data split:\n| Methods | CXR14(1%) | CXR14(10%) | CXR14(100%) |\n|-------|----------|----------|----------|\n| MGCA[2]   | 62.4  |  73.9 |  81.2 |\n| MRM[3]     | 64.2  |  74.3 |  81.0 |\n| **G2D(ours)**   | **79.1**  | **81.1**  | **83.1**  |\n\n>Transformer-based vision encoder is not analyzed (W2)\n- We conducted the experiments using a transformer-based vision encoder, specifically the ViT, configured identically to that used in the MRM study [3]. For the vision decoder, we also employed a transformer-based architecture, the same as MRM. The results of G2D with various transformer variants are shown in the table below. As indicated by the table, the performance of G2D does not fluctuate significantly with different backbone types, demonstrating that our method is backbone-agnostic.\n\n| Backbone | Classification(AUC) | Segmentation(Dice) | Detection(mAP) |\n|-------|----------|----------|----------|\n| | CXR14(1%) | SIIM(1%) | ObjectCXR(1%) |\n| G2D(CNN)   |  **79.1** | 65.6  |   **3.8** |\n| G2D(Transformer)   | 78.8  | **65.7**  |  3.5 |\n\n>The balance between VLA and PA losses is not analyzed (W3)\n- We selected a coefficient of 1 for both the VLA and PA losses from the initial development of the project, as we believe a robust method should not require specifically tuned coefficients for each loss.\n- Due to time constraints during rebuttal, we plan to ablate the coefficients of these two losses in future work to comprehensively investigate their contributions to the G2D method.\n\n[1] Zhang, Xiaoman, et al. \"Knowledge-enhanced visual-language pre-training on chest radiology images.\" Nature Communications 2023\\\n[2] Wang, Fuying, et al. \"Multi-granularity cross-modal alignment for generalized medical visual representation learning.\" NeurIPS 2022\\\n[3] Zhou, Hong-Yu, et al. \"Advancing Radiograph Representation Learning with Masked Record Modeling.\" ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QpQD7IdrMu", "reviewer": "Reviewer_SZ6L", "summary": "This manuscript describes a medical vision-language pre-training framework called Global to Dense level representation learning (G2D), that learns global and dense visual features simultaneously with only image-text pairs, by exploiting the aggregated attention map from the vision encoder for a pseudo segmentation pretext task. The improved (frozen) vision encoder is then utilized as part of the model pipeline for a number of downstream tasks (e.g. segmentation, classification)", "questions": "1. The accuracy of the initial aggregated attention map appears possibly non-optimal, given that additional thresholding by body mask is required. As such, it might be considered to quantify the accuracy of these maps, possibly against segmentation ground truth.\n\n2. In Section 3.2, it is stated that a threshold is applied (at 85%) to transform the aggregated attention map into a binary mask, before smoothing. It might be clarified if the need for smoothing (and related smoothing parameters) was empirically determined.\n\n3. In Section 3.3, it is stated that \"This decoder takes visual feature V_i as input and utilises the pseudo mask ˜M_i as the supervisory signal for the pretext task\". It might be clarified as to whether and how specific text can be matched to specific (separate) image regions, as in Figure 4 of Section A.7. In other words, while Figure 4 shows specific text descriptions corresponding to specific image regions, were these correspondences/alignments indicated by the proposed G2D model, or are they external manual observations? A.1 suggests no, but this might be explicitly stated.\n\n4. In Section 4, the choice of ResNet-50 as the encoder over other plausible choices (e.g. U-Net encoder) might be briefly explained.\n\n5. For Table 1, it might be clarified as to what \"encoder-decoder\" refers to - the updating of both encoder and decoder?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Pseudo segmentation pretext task enables dense segmentation during pre-training, and avoids external resources as for alignment-based methods, and limitations on high-level semantic representations in reconstruction-based methods\n - Importance of associating semantic meaning verified via experiment", "weaknesses": "- Unclear if specific sentence/phrase to individual image region alignment is achieved, for dense learning\n - Lack of fine-grained pixel-level evaluation of masks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedbacks!\n> Unclear if specific sentence/phrase to individual image region alignment is achieved, for dense learning (W1)\n- Since the MIMIC-CXR pretraining dataset does not establish a direct relationship between specific sentences or phrases and image regions, direct alignment between them is infeasible. To facilitate the learning of dense visual features, we construct pseudo masks for a pseudo segmentation pretext task, as detailed in Section 3.2.\n- Notably, our method has a significant advantage during medical VLP as it learns dense visual features using only image-text pairs. It does not rely on specific sentence or phrase annotations paired with individual image regions. This means our approach is more generalizable and does not require region-level annotations.\n\n> Evaluation of pseudo masks. (W2, Q1)\n- In the G2D approach, our objective is to enhance medical VLP by designing a pseudo mask for learning dense visual features through a pseudo segmentation task. This method does not rely on guessing the semantic mask as in traditional supervised learning. Since the MIMIC-CXR dataset used for pretraining **lacks pixel-level annotations**, it is impossible to directly assess the accuracy of the pseudo masks created through G2D. However, preliminary quality checks, as detailed in Appendix A.7 and illustrated with two examples in Figure 4, show that G2D successfully identifies image regions that align with the content of the entire report using just language cues and pseudo mask supervision.\n- We acknowledge that the pseudo masks are not perfect. However, adding body masks is one of the simplest and most commonly used operations in medical image processing[1].\n\n> Ablation on smoothing pseudo masks (Q2)\n- We assess the impact of smoothing on the G2D model's performance in Table 5-f by comparing versions with and without smoothing. The results show that the model variant with smoothing outperforms the one without, especially in the SIIM segmentation task. This indicates that smoothing the pseudo mask enhances the learning of representative visual features during VLP particularly for dense visual features.\n- To further investigate the effect of smoothing, we performed an ablation study on the 'window_size' parameter when implementing a bilateral filter. The results are shown below. As the findings indicate, variations in 'window_size' do not substantially affect the quality of VLP, demonstrating that the G2D method is robust across different parameter settings for smoothing operations.\n\n| Window Size | Classification(AUC) | Segmentation(Dice) | Detection(mAP) |\n|-------|----------|----------|----------|\n| | CXR14(1%) | SIIM(1%) | ObjectCXR(1%) |\n| 5$\\times$5   |  79.2 | 65.5  |   3.8 |\n| 7$\\times$7 (ours)   |  79.1 | 65.6  |   3.8 |\n| 10$\\times$10   |  79.1 | 65.6  |   3.7 |\n\n> Clarifying Text-to-Image Region Correspondences in G2D Model (Q3)\n- During pre-training, the pseudo mask for each sample is generated by the G2D model using the attention map, without the need for external manual annotations.\n- Our approach does not attempt to align specific text with specific image regions. Instead, our primary objective is to establish a semantically meaningful target for dense representation learning during pre-training. We utilize the semantic information encoded in the image-text attentions, rather than focusing on precisely predicting downstream segmentation tasks, which is theoretically unrealistic.\n- In Appendix A.7, the image region depicted is the pseudo mask derived from the G2D model, not one annotated by humans.\n- We will clarify this part in the camera-ready version according to your suggestion.\n\n> Choice of U-Net encoder (Q4)\n- ResNet-50 is the most commonly used vision encoder for vision-language pre-training (VLP) methods, such as CLIP.\n- For a fair comparison, we strictly adhere to the protocols established by MGCA, MedKIP, KAD, and others, using ResNet-50 as the image encoder.\n- The latest version of nnU-Net [4] utilizes a ResNet backbone and demonstrates improved performance compared to a vanilla, non-ResNet backbone.\n\n> Clarification for Tab 1 'encoder-decoder' (Q5)\n- We adhere to the protocols established by GloRIA[2] and MGCA[3] by only updating the decoder while keeping the encoder frozen during training. We will update and clarify this part in the camera-ready version of our document.\n\n[1] Imura, Masataka, et al. \"Automatic cropping method of chest radiographs based on adaptive binarization.\" EMBC, 2013.\\\n[2] Huang, Shih-Cheng, et al. \"Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition.\" ICCV. 2021\\\n[3] Wang, Fuying, et al. \"Multi-granularity cross-modal alignment for generalized medical visual representation learning.\" NeurIPS 2022\\\n[4] Isensee, Fabian, et al. \"nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation.\" CoRR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wPn9WWqSQg", "reviewer": "Reviewer_t3s2", "summary": "This paper proposes G2D, a novel vision-language pre-training (VLP) framework for medical imaging that aims to learn both global and dense visual representations from radiography images and their associated radiology reports. The key innovation is a pretext task called Pseudo Segmentation (PS), which uses a pseudo mask derived from attention maps to guide the learning of dense visual features during pre-training. The authors demonstrate that G2D outperforms existing medical VLP approaches on various downstream tasks including classification, segmentation, object detection, and zero-shot visual grounding across multiple medical imaging datasets. Notably, G2D shows strong performance on segmentation tasks even when fine-tuned on very limited data.", "questions": "Major concerns:\nMy primary concern revolves around the authors' claim that current medical VLP methods primarily align images with entire text reports. This assertion appears to be inconsistent with the facts, as evidenced by several papers that have employed local alignment between image regions and text. This factual contradiction significantly undermines the novelty of the present work. For instance:\n\nGLoRIA (Huang et al., ICCV 2021): \"Global-Local Representation Alignment for Improved Visual Recognition in Medical Imaging\"\nThis paper introduced a global-local alignment approach, learning finer-grained representations by aligning image patches with text tokens.\nMGCA (Wang et al., arXiv 2022): \"Multi-Granularity Cross-Modal Alignment for Generalized Medical Visual Representation Learning\"\nThis method employed a multi-granularity alignment strategy, including global, local, and fine-grained levels of alignment.\nBioViL (Boecking et al., ECCV 2022): \"Making the Most of Text Semantics to Improve Biomedical Vision–Language Processing\"\nThis work proposed a method to improve biomedical vision-language processing by leveraging text semantics, which includes local alignment strategies.\nMedKLIP (Wu et al., medRxiv 2023): \"Medical Knowledge Enhanced Language-Image Pre-training\"\nThis approach utilized external knowledge bases to enhance local alignment, achieving more fine-grained image-text matching.\nGiven these existing works, the authors' characterization of the current state of medical VLP appears inaccurate. This misrepresentation significantly weakens the claimed novelty of their approach. The authors should provide a more accurate description of existing methods and clearly articulate how their approach differs from or improves upon these established local alignment strategies.\n\nOther minor concerns:\nHave you explored the quality of the learned representations at different levels of the network? Are there significant differences in the quality of features at different scales?\nHow sensitive is the method to the choice of threshold used in pseudo mask construction? The ablation shows results for a few values, but is there a principled way to choose this threshold?\nHave you investigated the potential of using the pseudo masks generated during pre-training for weakly supervised segmentation tasks?\nHow does the performance of G2D change as the amount of pre-training data is varied? Is there a clear relationship between pre-training data volume and downstream task performance?\nGiven the computational requirements for pre-training, have you explored any techniques for making the approach more efficient, such as progressive training or curriculum learning?", "limitations": "The authors provide a brief discussion of limitations in the appendix, acknowledging potential issues with the weak supervision signal from pseudo masks and the need for further research on regional visual representations. They also touch on broader impacts, mentioning both potential benefits for healthcare and risks associated with sensitive medical data. While these discussions are valuable, they could be expanded to provide more specific insights into the limitations of the current approach and potential mitigation strategies for the identified risks.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Novel approach: The paper introduces an innovative method for learning dense visual representations in medical VLP without requiring pixel-level annotations, addressing a key limitation of existing approaches.\n\nWell-motivated: The authors provide a clear rationale for why learning dense representations is important for medical imaging tasks and why existing VLP methods struggle with this.\n\nComprehensive evaluation: The method is evaluated on a wide range of downstream tasks and datasets, demonstrating its versatility and effectiveness across different medical imaging applications.\n\nStrong results: G2D consistently outperforms existing methods, especially on segmentation tasks where it achieves impressive results with very limited fine-tuning data.\n\nAblation studies: The paper includes thorough ablation experiments to validate key design choices and components of the method.\n\nPotential impact: The proposed approach could significantly reduce the need for large annotated datasets in medical imaging, which is a major bottleneck in the field.", "weaknesses": "Limited theoretical analysis: While the method is empirically strong, there is little theoretical justification for why the pseudo segmentation task leads to improved dense representations.\n\nComplexity of the approach: The method involves several components and processing steps, which may make it challenging to implement and potentially limit its adoption.\n\nComputational resources: The pre-training process appears to be computationally intensive (16 A100 GPUs), which could be a barrier for researchers with limited resources.\n\nGeneralization to other domains: While the focus on medical imaging is valuable, it's unclear how well this approach would generalize to other vision-language domains.\n\nComparison to more recent baselines: Some of the baselines used for comparison (e.g., ConVIRT, GLoRIA) are somewhat older.\n\nComparison to more recent medical VLP methods would strengthen the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the questions!\n>Theoretical analysis for why pseudo segmentation task leads to improve dense representations\n- Methods like ConVIRT, GLoRIA, BioViL, MedKLIP, and KAD primarily use an image encoder to extract visual features, aligning them with text embeddings through contrastive learning. MedKLIP and KAD further leverage external tools to extract entities from medical reports for entity classification during VLP However, these methods are limited in their ability to learn dense visual features due to the lack of pixel-level supervision, relying instead on reports or entities.\n- Methods like MRM employ pixel-level pretext tasks, such as masked image reconstruction, for the encoder-decoder vision model. However, this task lacks high-level semantics, as it merely aims to reconstruct the original pixel values [1,2].\n- In contrast, G2D employs pseudo segmentation with pixel-level targets, known as pseudo masks, enabling the vision model to learn dense visual features. G2D's encoder-decoder architecture allows the decoder to become familiar with these dense features, ensuring effective feature learning and continuity when applied to downstream segmentation tasks, unlike methods requiring a randomly initialized decoder. By using pseudo masks derived from attention maps, which align visual features with the medical report, G2D ensures that the pseudo mask reflects the report's semantics. This approach enables the G2D encoder-decoder vision model to learn dense visual features with high-level semantics.\n\n> Complexity of the approach: The method involves several components and processing steps, which may make it challenging to implement and potentially limit its adoption. \n- We kindly disagree. Our method is simple yet effective because **(1)** G2D does not require annotations and only needs image-text pairs; **(2)** Our pretraining is a one-stage process, similar to MGCA, and more efficient than MedKLIP and KAD, which require specific entity extraction from medical reports; **(3)** We directly construct the pseudo mask from the attention map, without relying on an external network, unlike PyramidCLIP[3] and GLIPv2[4].\n\n>Computational resources\n- We use 16 A100 GPUs solely to accelerate the pre-training stage. To ensure a fair comparison with existing methods, we calculate the contrastive loss on each device independently, rather than aggregating batches across devices. Consequently, adding more devices does not increase the batch size or substantially affect the quality of pre-training.\n- We have reimplemented the VLP stage on 2 RTX3090 GPUs, using the same computational resources as MGCA. Due to the smaller RAM of the RTX3090 compared to the A100, we set the batch size to 64 on each GPU. The results of this implementation are shown below. As the table indicates, the downstream performance changes only marginally when pretraining on 2 RTX3090 GPUs, demonstrating that G2D is effective even with less computational resources.\n\n| GPUs | Classification(AUC) | Segmentation(Dice) | Detection(mAP) |\n|-------|----------|----------|----------|\n| | CXR14(1%) | SIIM(1%) | ObjectCXR(1%) |\n| 2 RTX3090   |  78.7 | 65.3  |   3.7 |\n| 16 A100s (default)   |  79.1 | 65.6  |   3.8 |\n\n>Generalization to other domains\n- We implemented pretraining on the MIMIC-CXR dataset, following the protocols established by MGCA, MedKLIP, KAD, and others.\n- Our assumption on image-text region alignment is common and straightforward. Therefore, we believe our method holds value for other vision-language domains, such as natural and remote sensing imagery, due to the importance of dense visual representations across these areas. However, this work is primarily focused on the medical domain. We plan to further explore the potential of our method across various domains in future studies.\n\n>Comparison to more recent baselines\n- We compare our method to M-FLAG (MICCAI 2023), MedKLIP (ICCV 2023), and KAD (Nature Communications 2023), aligning with the comparison methods used in KAD's original work to ensure a fair comparison.\n\n>Pseudo masks for weakly-supervised segmentation tasks\n- This is an interesting direction; we thank the reviewer for the insight and will investigate it in the future. We postulate that our method can serve as an effective pre-training technique for weakly supervised segmentation because it leverages pseudo masks to learn dense visual features without needing precise annotations.\n\n>Effect of the amount of pretraining data\n- For a fair comparison, we implemented the VLP on the full MIMIC-CXR dataset, following the protocols of MGCA, MedKLIP, MRM, KAD, and others. We acknowledge that the amount of pretraining data is a critical and interesting research question for medical VLP, and we plan to investigate this in our future work.\n\n>Progressive training or curriculum learning\n- Thank you for the advice. We have conducted an ablation study using less computational resources—2 RTX 3090s. As shown in the table above, the results demonstrate that G2D also performs well with fewer computational resources.\n\n\n>Clarification on other medical VLP methods’ limitations in alignment\\\n>Quality of different level visual features\\\n>Sensitivity analysis on threshold used in pseudo maks construction\n- We have added a detailed explanation in the official comment below. Please refer to it for more information.\n\n[1] Liu Y, et al. Improving pixel-based mim by reducing wasted modeling capability, ICCV, 2023.\\\n[2] Liu Y, et al. PixMIM, TMLR, 2023\\\n[3] Gao, Yuting, et al. PyramidCLIP, NeurIPS 2022\\\n[4] Zhang, Haotian, et al. GLIPv2, NeurIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper addresses an interesting problem of learning global and dense visual features via vision-language alignment (VLA) and pixel alignment (PA) strategies. The proposed method demonstrates its superior performance on various medical tasks, i.e., image segmentation, object detection, visual grounding, and image classification (both zero-shot and fine-tuned). The reviewers all agree that the paper is well-written and the proposed pseudo-segmentation pretext task of exploiting the aggregated attention maps from the vision encoder is novel and valuable.  In the initial review comments, the reviewers raised concerns about the quality of pseudo masks, how they will affect the training, the feasibility of being extended to perform in 3D, and some details of other techniques. The authors provided a detailed response and supportive experimental results, and most of the concerns were addressed, resulting in favorable final ratings from the reviewers. Overall, the paper introduced a novel pre-training scheme, promoting dense semantic information in medical images, and demonstrated the superior performance gain of the proposed method, which AC believes could benefit and inspire the related field and thus worth accepting to NeurIPS. The AC encourages the authors to further address the review comments and strengthen the paper.
Decision: Accept (poster)