G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training

Authors: che  liu
cheng  ouyang
sibo  cheng
anand  shah
wenjia  bai
rossella  arcucci
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 23:00:17

Abstract: Medical imaging tasks require an understanding of subtle and localized visual features due to the inherently detailed and area-specific nature of pathological patterns, which are crucial for clinical diagnosis. Although recent advances in medical vision-language pre-training (VLP) enable models to learn clinically relevant visual features by leveraging both medical images and their associated radiology reports, current medical VLP methods primarily focus on aligning images with entire reports. This focus hinders the learning of dense (pixel-level) visual features and is suboptimal for dense prediction tasks (e.g., medical image segmentation).

To address this challenge, we propose a novel medical VLP framework, named **Global to Dense level representation learning (G2D)**, which aims to learn global and dense visual features simultaneously using only image-text pairs without extra annotations. In particular, G2D designs a **Pseudo Segmentation (PS)** task, which enables the model to learn dense visual features during VLP. Notably, generating PS masks can be performed on the fly during VLP, which does not incur extra trainable parameters. With this simple yet effective idea, G2D achieves superior performance across 5 medical imaging tasks and 25 diseases. Particularly, in the segmentation task which requires dense visual features, **G2D surpasses existing models even with just 1% of the training data for finetuning, compared to 100% used by other models**. The code can be found in https://github.com/cheliu-computation/G2D-NeurIPS24/tree/main.

Main paper: 
 introduction: 
 In medical image analysis, learning global and dense visual representations typically requires laborintensive and costly image and pixel-level annotations [1, 2] . Vision-language pre-training (VLP) attempts addressing this by aligning vision and language content using paired datasets [3, 4, 5, 6] . Although existing medical VLP methods excel at learning global visual features [7] , they face challenges with dense visual features because the level of detail in text reports does not offer sufficient b) Reconstruction-based approaches do not align with text, resulting in a deficiency in discriminative and clinically relevant visual features. c) The framework of G2D (proposed) learns dense, clinically relevant, text-aligned visual features through derived pseudo masks and image-text alignment. We use red text to highlight the deficiencies of existing methods and blue text to emphasize our advantages. pixel-level supervision for learning these more detailed aspects. Existing medical VLP methods are categorized into two main types, as shown in Fig. 1: • Alignment-based Approaches, which focus on aligning images with reports [4, 8, 9, 5, 6, 2, 10] . Although methods like [4, 8, 9] align images with entire reports and text tokens, they struggle to learn dense, clinically relevant visual features. This is due to the ambiguous supervision targets provided by text tokens, which lack explicit relational pairing with image regions, as discussed in [2] .• Reconstruction-based Approaches, which learn representations by reconstructing masked images or reports using masked modeling techniques [11, 12] . However, they also lack success in capturing dense, clinically relevant visual features, as the reconstruction task primarily focuses on low-level patterns (texture, shape) rather than high-level semantics [13] . Despite advancements in medical VLP, limitations still exist. Current alignment approaches align image patches with text tokens in a brute-force manner and possibly cause misalignments when some word tokens (e.g., 'compatible' or 'unremarkable') lack direct visual counterparts, leading to ambiguous local alignments. Meanwhile, reconstruction-based approaches may ignore high-level image semantics. They are designed to recover low-level visual information such as intensity and texture, without accounting for high-level semantics [14, 13, 15] . As a result, both approaches perform suboptimally for downstream tasks, such as semantic segmentation and visual grounding, which require learning of granular visual features that are aligned with high-level semantics.While numerous VLP methods are designed to capture dense visual features for natural image datasets (e.g., ImageNet) they often struggle to transfer directly to medical images because they depend on a well-trained object detection model [16, 17] or a well-aligned VLP model [18, 19] . In the medical domain, obtaining such pre-trained models is difficult as objects can be defined in various ways within a single medical image (e.g., based on organs, anatomical structures, or abnormal regions) . Additionally, in medical domain, there is a lack of foundational VLP models that are both publicly accessible and are trained on sufficiently large image-text pairs that cover diverse medical imaging applications.In response to the aforementioned challenges, we introduce a novel medical VLP approach termed G2D. This approach is designed to extract global and dense visual representations from radiography along with their associated radiology reports, with improved feature granularity and enriched semantic information. Central to our approach is a pretext task, Pseudo Segmentation (PS), which is guided by a pseudo mask (segmentation target) derived from a carefully refined and filtered attention map. PS encourages the model to learn dense representations through a pixel-level pretext task that incorporates high-level semantics. This approach, in contrast to traditional methods that align image patches with text tokens, inherently mitigates the misalignment bias and allows learning of more representative features. Notably, the PS pretext task can be implemented to run concurrently with vision-language alignment, ensuring that the model can be trained end-to-end, contrasting with the two-stage training methods [18] .To evaluate the effectiveness of G2D relative to other state-of-the-art (SOTA) VLP approaches, we deploy the pre-trained model across a diverse range of downstream tasks, including medical image classification, semantic segmentation, object detection, as well as zero-shot image classification and visual grounding, on six public large-scale CXR datasets. The experimental results demonstrate the superior performance of G2D over existing VLP approaches on these medical applications. Overall, our contribution is three-fold:1. We introduce G2D, the first end-to-end encoder-decoder medical VLP approach designed to learn visual representations from the global level down to the dense level, supervised by paired radiology reports and a pixel-wise pretext task.2. We carefully design a pretext task tailored for medical VLP, pseudo segmentation. It formulates a pseudo mask as segmentation target, allowing the model to learn dense visual representations in the pretext task which can benefit downstream dense visual tasks in medicine.The pseudo mask can be generated using a parameter-free processor that leverages the attention map derived from the visual representation associated with radiology reports.3. We conduct comprehensive experiments to validate the efficacy of the proposed G2D approach, which outperforms peer approaches across five uni-modal and cross-modal downstream tasks. 
related works: 
 Alignment-based Medical VLP. Drawing inspiration from [3] , aligning images with their corresponding textual descriptions in the latent space has led to notable advancements in VLP. Within the CXR domain, while ConVIRT [4] made an early attempt at employing bidirectional contrastive learning to globally align entire images with their paired reports, there remained room for refinement. GLoRIA [8] and MGCA [9] represent advancements in image-report alignment, introducing sophisticated global-local methodologies to the field [8, 9] . These approaches endeavor to establish correspondences between distinct image and text tokens. However, it is crucial to recognize that the granularity of token-level alignment could inadvertently introduce distortions to the medical context, potentially leading to misalignments, as illustrated by [20, 2] . Med-UniC [20] utilizes augmented text in VLP training to cultivate language invariance, with the goal of mitigating linguistic biases from VLP. Meanwhile, MedKLIP [5] and KAD [21] harness domain-specific knowledge from external annotated datasets to enhance textual information extraction. Notably, these approaches [20, 5, 21] are contingent upon external resources or extra data to optimize cross-modal representation learning, which could potentially constrain their generalizability.Reconstruction-based Medical VLP. Several studies, including [12, 11, 22] , have employed reconstruction of image and text tokens as a pretext task within VLP. Specifically, MRM [12] endeavors to reconstruct the original image from a masked version and simultaneously aims to regenerate the original text using both the masked image and text as inputs. Conversely, PRIOR [11] adopts a strategy that focuses on cross-modal representation by reconstructing images and sentences based on complete image and report inputs. An enhancement to the MRM [12] approach is proposed by [22] , where token weights are adjusted during the reconstruction phase.While these methods have demonstrated promising outcomes, the ability of the reconstruction pretext task to capture high-level semantic representations is limited, as shown in [14, 15, 13] , and is further challenged by the absence of explicit semantic-related constraints in dense visual representation learning. 
methodology: 
 The central aim of G2D is to learn global and dense visual representations from medical images under the supervision of their corresponding radiology reports. As illustrated in Fig 2 Left , G2D integrates two alignment strategies: vision-language alignment (VLA) that learns global representations, and pixel alignment (PA) that focuses on granular representation via a pixel-level pretext task, Pseudo Segmentation (PS). The pseudo mask for PS is constructed through a parameter-free mechanism, which is operated alongside VLA. The PS pretext task enables G2D to derive dense representations at both encoder and decoder levels during pre-training. Moreover, the task head of the pretext task facilitates a smoother transfer for the pre-trained encoder to be applied to downstream segmentation tasks, reducing the gap between the dense visual representation learned from VLP and the needs of downstream dense visual tasks after VLP. This contrasts with previous meth-Figure 2 : Left: Framework of G2D. Right: Pipeline for pseudo mask construction. We visualize the constructed pseudo mask and corresponding sentence in the radiology report in Sec A.7.ods [4, 8, 9, 21, 5, 6] that typically transfer only the pre-trained encoder, potentially leading to an information gap between the pre-training and downstream tasks.We utilise a dual-encoder image-text contrastive approach following [4, 8, 9, 5] . Given a training set S consisting of N pairs of image-text (v i , l i ), where v i ∈ V denotes an image and l i ∈ L denotes a text report, i = 1, 2, 3, ..., N , G2D employs an image encoder F e : V → R Dv to encode the image into an embedding of dimension D v , and a text encoder F l : L → R D l to encode the text report into an embedding of dimension D l . The embedded image and text features can be denoted asS = {(v 1 , l 1 ) , (v 2 , l 2 ) , . . . , (v N , l N )}, where v i = F e (v i ) and l i = F l (l i ).As depicted in Fig. 2 , G2D incorporates two alignment strategies: VLA and PA. For VLA, the model aims to learn global visual and text representations by pulling the embeddings of paired image-report samples closer, while distancing embeddings of unpaired samples, using a contrastive loss L VLA . The objective of contrastive learning is to predict N positive matched pairs (v i , l i ) and N 2 -N negative pairs among N × N possible image-text pair combinations [3] . Subsequently, two non-linear vision and language projectors P v and P l transform v i and l i into the same dimension d, where vi = P v (v i ), li = P l (l i ), and vi , li ∈ R d . After obtaining image feature vectors [v i ] N i=1 and text feature vectors [ li ] N i=1 with the same dimension d, the contrastive loss L VLA can be formulated as:EQUATIONσ denotes the temperature hyper-parameter empirically set to 0.07 following [9] , and K ∈ N is the batch size.Notably, although MedSAM [23] claims to build image-mask pairs, it requires box prompt inputs not available in the MIMIC-CXR [24] dataset. Designing a box prompt for each image is labor-intensive and unfeasible for this work, so we construct the pseudo mask based on attention maps.Attention Aggregation. Inspired by CLIP [3] , we incorporate an attention pooling mechanism in conjunction with the non-linear projector P v to derive a pixel-wise attention map. A dense feature map V i is extracted from the final convolutional layer before the pooling operation in the image encoder Fe, with the dimension C × H × W . Here, C denotes the number of channels, while H and W represent the height and width of the feature maps. Subsequently, we reshape V i into a dimension of HW × C. In this way, V i can be interpreted as a sequence of pixel embeddings, where each token in this sequence represents the embedding of an individual pixel. The length of this sequence is defined by the number of channels, C. A special token, [CLS], is introduced to aggregate all pixel embeddings through multi-head self-attention (MHSA) [25, 3] . This process offers an attention score matrix W h i for each pixel, with dimensions h×H ×W . Here, h signifies the attention head number, and h ∈ H, with H being the total number of attention heads. This attention score matrix characterizes the information exchange between pixels and semantics provided by the text [3, 18] , and therefore it carries semantic information and is an ideal candidate for constructing the pretext pseudo mask. To derive the pseudo mask, we aggregate W h i across all attention heads to produce Ŵi , as described by:EQUATIONMask Filtering and Edge Smoothing. After obtaining the aggregated attention map Ŵi , we upsample it to match the original image dimensions H ′ × W ′ . To remove pseudo mask regions in the background, we construct a body mask for each CXR image using a histogram-based thresholding approach, following common practice [26, 27] . Subsequently, all attention scores outside the body mask are set to zero. A threshold is applied to filter out low attention scores within the body mask, transforming Ŵi into a binary mask. The threshold is determined at the 85% percentile of attention scores from Ŵi . The binary pseudo mask M i is formulated as:M j,k i = 1 if W j,k i ≥ threshold 0 otherwise, whereEQUATIONTo smooth the square-like boundary in the mask caused by upsampling, we apply bilateral filtering (BF) [28] to M i , resulting in a refined pseudo mask Mi , as shown in Fig. 2 Right. A comprehensive ablation study discussing the threshold and smoothing operation is presented in Sec. 4.5.While the global visual representation can be learned via VLA, dense representation often lacks direct alignment. To tackle this limitation, we introduce an image decoder, denoted as F d , as shown in Fig. 2 Left. This decoder takes visual feature V i as input and utilises the pseudo mask Mi as the supervisory signal for the pretext task. We employ the commonly used soft Dice loss and binary cross-entropy loss [27] to optimise this task. The training loss function for L PA is formulated as:EQUATIONThe total loss for G2D is the sum of the VLA loss (Eq. 1) and the PA loss (Eq. 4):EQUATIONIt is worth noting that the pseudo mask is designed as a pixel-wise pretext supervisory signal. Although there is no manual annotation involved, the pseudo mask is constructed from the visual feature of the image encoder, which is pre-trained to align with radiology reports and thus contains clinical knowledge such as anatomical regions mentioned by the reports. In this sense, it can be a good surrogate target for learning pixel-wise semantic information. To demonstrate that the pseudo mask serves as a meaningful target for dense visual pre-training, we conduct an ablation study to use a perturbed pseudo mask with corrupt semantics for pre-training, and compare it to the proposed pseudo mask, as detailed in Table 8 and Sec A.6.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: This work mainly includes empirical contributions.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments and analysis: 
 In this section, we compare our approach with SOTA medical VLP techniques. The implementation details and dataset training/test splits are reported in Sec A.3, A.4.We utilise the MIMIC-CXR dataset [29, 24] . After preprocessing based on established protocols [9, 5] , it provides 213,384 image-text pairs for pre-training.For the VLP part, we employ a standard ResNet-50 as the vision encoder F e and adopt the decoder part of a U-Net as the vision decoder F d . We adopt ClinicalBERT [30] as the text encoder using configurations described in [5, 21] . In line with [9, 8] , G2D is pre-trained for 50 epochs across 16 A100 GPUs, each accommodating a batch size of 128. The AdamW optimizer is employed with a learning rate set to 2 × 10 -4 and a weight decay of 1 × 10 -8 . Additionally, a linear warm-up and a cosine annealing scheduler are incorporated in the training process.For downstream tasks, our focus is to evaluate the efficacy of G2D in learning granular visual features that can be used for localisation, vision-language understanding, and visual recognition tasks. We examine the capability and transferability of the learned cross-modal representations by using them for five distinct medical imaging tasks, covering a spectrum of 25 different diseases.Medical Image Segmentation. This task utilises the RSNA [31] and SIIM [32] datasets, following preprocessing guidelines established in [9, 8] . We adopt U-Net [1] fine-tuning configurations following [8, 9] . The pre-trained vision encoder is frozen, while only the decoder parameters are updated during fine-tuning. Performance is assessed using the Dice score, following the evaluation protocol in [8, 9] . It is noteworthy that the original MedKLIP [5] uses a different configuration (updating the vision encoder) compared to other methods (freezing the vision encoder) [4, 8, 9, 6] . Therefore, in these experiments, we reference the results reported in [20] , which reimplemented MedKLIP under a setting consistent with all other methods. For a fair comparison specifically with MedKLIP, we also reimplement G2D under MedKLIP's original setting, as reported in the Sec A.5.Medical Object Detection. This task is conducted using the RSNA dataset [31] for Pneumonia Detection and the Object-CXR dataset [33] for Foreign Objects Detection, adhering to preprocessing methods from [9] . We employ YOLOv3 [34] for detection, using the pre-trained vision encoder and updating an additional detection head during fine-tuning. We report the mean Average Precision (mAP) with IoU thresholds between 0.4∼0.75. The setup for this task is in accordance with in [9] .Zero-shot Medical Image Visual Grounding. In accordance with [5] , this task is conducted on the RSNA [31] and SIIM [32] datasets, using the same official data split and evaluation metrics. We employ CXR images as input and utilise the corresponding ground truth label maps for assessing the grounding performance, in terms of recall, IoU, and Dice score.Zero-shot Medical Image Classification. In compliance with the guidelines set forth in [5, 21] , we conduct this task on the RSNA [31] , SIIM [32] , CheXpert [35] , and CXR14 [36] datasets. For the RSNA and SIIM datasets, we employ the test set splits provided by MedKLIP [5] , given that KAD [21] did not conduct experiments on these two datasets. For the CheXpert and CXR14 datasets [35, 36] , we use the official test set splits to ensure a fair comparison with KAD [21] . It is important to note that MedKLIP [5] creates its own test split rather than using the official test split. Hence, we do not use MedKLIP's splits in our experiments. We report the results using the macro average of AUC, F1, and ACC scores across all diseases.Medical Image Fine-tuned Classification. In alignment with [5, 21] , we use the CXR14 dataset [36] , comprising 112,120 frontal-view X-rays from 30,805 patients, annotated for 14 diseases. We adhere to the official split for consistent evaluation, following KAD [21] . It is worth noting that MedKLIP does not use the official data split. Hence, we refer to the results reported in KAD [21] rather than those from the original MedKLIP [5] . To ensure a fair comparison with Med-KLIP, we reimplemented G2D for this experiment under the MedKLIP configuration, as detailed in Sec A.5. CXR images are resized to 256 × 256 [21] . During fine-tuning, all model parameters are updated, including the pre-trained vision encoder and linear classifier. The AdamW optimizer is used with a learning rate of 1 × 10 -4 and a batch size of 64 for 50 epochs. Evaluation is based on the AUC score, adhering to the protocol outlined in [8, 9, 12] . Medical Image Linear Classification. In strict accordance with the configuration in [8, 4, 9] , this task is conducted on the CheXpert [35] , RSNA [31] , and COVIDx [37] datasets. We only update a randomly initialized linear classification layer, while the pre-trained vision encoder remains frozen.For fair evaluation, we employ AUC scores on CheXpert and RSNA, along with accuracy metrics on COVIDx, as mentioned in [8, 9] . Apart from zero-shot image classification and visual grounding, we fine-tune using 1%, 10%, 100% of the training data for all downstream tasks. Detailed settings, including implementation and data splits, are outlined in Sec A.4.In Tab 1, following [16, 38] , we evaluate G2D alongside other SOTA approaches on two pivotal visual localisation tasks: semantic segmentation and object detection. The aim is to assess the efficacy of the dense visual features learned.Initially, we transfer only the encoder weights from the pre-trained G2D for the segmentation task, adhering to the protocols of [9, 8, 4, 6] . In this setup, our approach consistently achieves the highest performance across all data fractions for both SIIM [32] and RSNA datasets [31] . To assess the impact of the visual decoder pre-trained with the PS pretext task, we transfer the weights of both the encoder and decoder from G2D for the segmentation task, resulting in striking outcomes. Remarkably, with just 1% of training data, G2D surpasses the performance of all peer methods, even those trained with a full 100% of training data. This observation underlines the fact that the pixel-level pretext task, PS, significantly improves the quality of dense visual features derived from VLP, which provide advantages for the downstream segmentation task.In object detection, our method consistently outperforms existing methods across all data fractions for both RSNA and Object-CXR datasets [31, 33] . Notably, G2D achieves a 3.8% mAP on the Object-CXR dataset with just 1% of the data for fine-tuning, a significant leap from other methods that scarcely reach a 1% mAP.These results highlight the efficacy of our proposed model, G2D, and the pretext task, PS, especially in semantic segmentation tasks that rely on dense visual features. PS not only enables G2D to learn visual representations in the encoder-decoder structure but also reduces the gap between pretraining and downstream tasks. By enhancing the encoder's ability to capture global and dense features simultaneously, PS surpasses existing approaches, proving particularly advantageous for object detection tasks that heavily rely on dense features [39] .In Tab 2, we evaluate the efficacy of G2D on vision-language understanding tasks, zero-shot visual grounding and zero-shot image classification. For the zero-shot visual grounding task, our proposed method outperforms peer approaches. Specifically, on the SIIM dataset [32] , it achieves a leading Dice score of 5.1. This dominance persists in the RSNA dataset [31] , where our method reaches a Dice score of 47.7, surpassing other SOTA approaches. When examining zero-shot image classification, our method again shows its superiority across the AUC, F1, and ACC metrics on both the RSNA [31] and SIIM datasets [32] . Such consistent and superior outcomes underscore the adaptability and effectiveness of G2D in handling vision-language understanding tasks, indicating that integrating PS into G2D can enhance not only uni-modal but also cross-modal tasks.In our final assessment focused on visual recognition, Tab 3 demonstrates our method's consistent supremacy on the CXR14 dataset [36] for fine-tuned disease classification across 1%, 10%, and 100% training data. Similarly, Tab 4 underscores that G2D achieves the highest performance on the CheXpert, RSNA, and COVIDx datasets [35, 31, 37] for linear evaluation across all training data ratio. Notably, G2D consistently outperforms even those methods like MedKLIP and KAD [41] that leverage additional disease-level annotations during pre-training stage. This demonstrates G2D's representative visual features, suggesting that enhancing dense representation learning via PS can also improve results in tasks primarily anchored on global representation. Pseudo Segmentation vs. Reconstruction. In Tab 5a, we evaluate the impact of the proposed PS pretext task in comparison to pixel reconstruction and models without a decoder-level constraint.The model pre-trained with PS outperforms the other two approaches across all three downstream tasks, particularly in semantic segmentation. While the model pre-trained with a pixel reconstruction constraint exhibit improved performance compared to unconstrained variants, such models still underperform the model with the PS constraint. These results underscore the effectiveness of decoderlevel pretext tasks and suggest that an emphasis on high-level semantics, derived from PS, is more beneficial than focusing on the low-level semantics from pixel reconstruction. The PS potentially reduces the disparity between features learned through VLP and those required by downstream semantic segmentation tasks. It also enables the model to acquire more representative features that are beneficial for various tasks. Threshold of Pseudo Mask Construction. As shown in Tab 5b, performance varies with different thresholds, with the 85% percentile threshold proving most effective across all three downstream tasks. Despite employing the Gaussian Mixture Model (GMM) for pseudo mask creation, as suggested by [42] , its performance is still surpassed by the 85% percentile approach. This indicates that the original attention map might contain noise, and a higher threshold is beneficial for generating more effective pseudo masks. Furthermore, Tab 5d highlights the importance of aggregating multi-head attention maps for mask construction. Given the absence of explicit semantic supervision in the PS pretext task, not aggregating these maps leads to the creation of multiple pseudo masks. This excess of masks introduce ambiguous training objectives for VLP. Impact of Mask Refinement. Refinement of the pseudo masks affects the model's efficacy, as shown in Tab 5f. Performance tends to decrease when either the body mask is omitted or edge smoothing is not applied. However, integrating both these strategies, as we implement in G2D, yields optimal results. This underscores the vital role of pseudo mask refinement in enhancing model performance. Ablation on Hyperparameters. We further ablate the number of attention heads and projector dimensionality. Performance improves with more attention heads, peaking at 3 before slightly declin-ing at 4 (Tab 5e). Optimal segmentation and classification results are achieved with 128-dimensional projectors. While 256 dimensions provide slight benefits for object detection, they reduce performance in other tasks (Tab 5c). Projectors of 512 dimensions do not yield further gains. Thus, we select 3 attention heads and 128-dimensional projectors for an optimal balance of complexity and effectiveness.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide detailed experimental configurations in Sections 4.1, A.3, and A.4. Our code will be released after acceptance. 
conclusion: 
 In this study, we introduce G2D, a novel medical VLP framework for learning global and dense-level representations. Our proposed pixel-level pretext task, pseudo segmentation, leverages a refined attention map to predict a pseudo mask, capturing dense visual features during VLP without requiring additional trainable parameters for its construction. Our model pretrained with this pretext task achieves superior performance across five diverse medical imaging tasks and outperforms methods pretrained with annotated data [5, 21] , especially in semantic segmentation. Specifically, on the SIIM [32] dataset, G2D, when fine-tuned with only 1% of the training data, outperforms other medical VLP approaches that utilize the full 100% training set. We anticipate that G2D will inspire further exploration of novel and clinically-guided pretext tasks for medical VLP.A Appendix / supplemental material A.1 Limitations and Future WorkOur work primarily concentrates on learning dense visual representations from pseudo masks, which are generated from attention masks under language supervision. Due to the weak supervision signal, the pseudo masks may not effectively associate each pixel with the corresponding text tokens, potentially capping the performance of our method. Currently, our approach involves learning both global and pixel-level representations through VLP. In future studies, we aim to delve into regional visual representations during VLP to establish more precise correlations between specific chest X-ray (CXR) regions and phrases in radiology reports.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Our experiments are all conducted on publicly accessible datasets, and all experiment details are illustrated in Sections 4.1, A.3, and A.4. For experiment implementation, we follow the official code of exisiting works, all code can be found in their official GitHub repository.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report the error bars for all ablation studies.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This work does not focus on content generation and uses clinically verified datasets for all experiments.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Please refer to Section 4.1.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Our G2D model offers an effective approach for the automatic diagnosis of chest X-ray abnormalities using a small amount of annotated data. This can help decrease the burden on radiologists and enhance healthcare in underprivileged regions. However, medical data, such as chest X-rays and radiology reports, might include sensitive or potentially harmful information. We strongly advise a thorough examination of the data prior to using our model in real-world applications. The data split information into train/valid/test sets are described in Tab. 6. For all downstream tasks, except from zero-shot image classification and visual grounding, we train with 1%, 10%, 100% of the training set. The downstream tasks are deployed on a 40G A100 GPU. For the segmentation tasks on the RSNA [31] and SIIM [32] datasets, we initially employ the vision encoder from the pre-trained model. Additionally, we transfer both the vision encoder and decoder from the pre-trained model, and proceed to train the segmentation network. We implement early stopping during the training process, limiting it to 50 epochs.A learning rate of 2e-4 and a weight decay of 0.05 are adopted. AdamW is utilized as the optimizer, with β 1 and β 2 values set at 0.9 and 0.999, respectively. For the SIIM [32] dataset, the default batch size is set at 8, while for the RSNA [31] dataset, it is set at 16. All configurations strictly adhere to the protocol provided in [9] .Medical Image Object Detection. The pneumonia detection task on RSNA [31] and foreign objects detection task on Object-CXR [33] datasets are executed on a single A100 GPU. For both datasets, early stopping is implemented during the training process, limited to 50 epochs. AdamW is employed as the optimizer across both datasets. For the RSNA [31] dataset, a batch size of 8 is set for 1% data fine-tuning with a learning rate of 2e-4, a weight decay of 1e-6, and β 1 , β 2 values of 0.9 and 0.999, respectively. For 10% and 100% data fine-tuning, the batch size is adjusted to 16, with a learning rate of 5e-4, a weight decay of 1e-6, and the same β 1 , β 2 values. Similarly, for the Object-CXR [33] dataset, a batch size of 8 is set for 1% data fine-tuning, with the identical learning rate, weight decay, and β values as the RSNA dataset. For 10% and 100% data fine-tuning, the batch size is adjusted to 16, again with a learning rate of 5e-4, a weight decay of 1e-6, and β 1 , β 2 values of 0.9 and 0.999. The IOU and NMS thresholds are set at [0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75] and 0.5, respectively. All configurations are in strict compliance with the protocol delineated in [9] .Zero-shot Image Classification. The original CXR image goes through a two-step preprocessing routine. Initially, it is resized to the dimension of 256 × 256, and then center cropped to 224 × 224.Following the methodologies outlined in [8, 9] , all pixel values are normalized to the range [0, 1].The resulting resized image is then fed through a visual encoder, followed by a visual projector to generate the image embedding vi . Simultaneously, the prompts are fed into a text encoder to obtain text embeddings li . The classification evaluation hinges on measuring the cosine similarity between the image and text embeddings for each prompt associated with a specific class. The classification outcome is determined by comparing the cosine similarities. Specifically, if the cosine similarity between the image embedding and the positive prompt (e.g., disease) surpasses that between the image embedding and the corresponding negative prompt (e.g., No disease), the outcome is deemed positive. Conversely, if the reverse holds true, the outcome is negative. The prompt is designed following [7] .Zero-shot Visual Grounding. To execute this task, we adhere to the BioViL pipeline as described in [40] . The visual grounding task can be regarded as a pixel-level classification task, driven by the text prompt and the dense visual embedding. The image is fed into the visual encoder to acquire the dense feature map V i from the final convolutional layer of the image encoder, yielding a shape of C × H × W . At the same time, the prompt is processed through the text encoder and projected into the cross-modal space, resulting in li . The cosine similarity between li and all elements of V i at the channel level generates a similarity map. This map is then resized to match the original image size and utilized as the segmentation results to evaluate the zero-shot grounding performance.We conduct evaluations on the CheXpert [35] , RSNA [31] , COVIDx [37] , and CXR14 datasets [36] .In alignment with previous studies [8, 4, 9, 5, 21] , linear classification is implemented on CheXpert [35] , RSNA [31] , and COVIDx [37] . Here, we update a randomly initialized linear layer while keeping the visual encoder frozen. We adhere to the official test set partition from [5, 21, 36] for a fair comparison. During our linear classification task, training is performed over 50 epochs with a learning rate of 5e-4, a batch size of 8, employing the AdamW optimizer with parameters: β 1 = 0.9 and β 2 = 0.999. For the CXR14 dataset [36] , we follow the experimental setup from [21] , employing fine-tuned classification while updating all parameters from the visual encoder and linear layer.Images are resized to 256 × 256 and data augmentation is carried out as recommended in [21] . The AdamW optimizer is utilized with a learning rate of 1 × 10 -4 and a batch size of 64 for 50 epochs.The linear classification tasks are executed on a single A100 GPU with 40GB memory, using the vision encoder from our pre-trained model as the visual backbone. Fine-tuning is carried out on the randomly initialized linear layer for 50 epochs with early stopping, maintaining a learning rate of 5e-4 and a default batch size of 8. We set AdamW as our optimizer, with β 1 of 0.9, β 2 of 0.999, and a weight decay rate of 1e-6. To strictly compare our work with MedKLIP [5] , we reimplement G2D for fine-tuning on CXR14 classification, as well as SIIM and RSNA segmentation tasks, adhering strictly to the MedKLIP configuration. This approach is necessary because the settings of MedKLIP differ significantly from the other methods that we compare to, such as [4, 8, 9, 6, 21] . Specifically, MedKLIP updates both the encoder and decoder during segmentation tasks, whereas the other methods only update the decoder and keep the encoder frozen. Moreover, MedKLIP employs its own customized data split for CXR14 classification, contrasting with KAD [21] , which uses the official CXR14 dataset split.Given these differences, comparing other methods directly under the MedKLIP setting could be seen as unfair. Therefore, we conducted a separate comparison between G2D and MedKLIP using the MedKLIP setting. The results, presented in Tab 7, demonstrate that G2D outperforms MedKLIP across all tasks and data ratios, even within the MedKLIP setting.To investigate whether the improvements in G2D come from learning dense visual features through pseudo segmentation (PS) or from treating PS as a regularization term during pre-training, we perturbed the semantic integrity of pseudo masks by randomly shuffling them on a sample-wise basis (i.e., making images and pseudo masks unpaired). This operation detaches pseudo masks' semantic connection to the original images, ensuring that the PS task does not learn correct semantic information, but still provide regularisation to the segmentation as the pseudo mask is relatively smooth. The results are presented in Table 8 . G2D with uncorrupted pseudo masks in PS (ours) significantly outperforms the results from the shuffled alternative (unpaired images and pseudo masks), not only in visual localisation task but also in visual recognition task. The improved performance demonstrate that the proposed G2D indeed learns transferable visual features thanks to the semantic information provided by the pseudo masks, rather than merely treating PS as a regularization mechanism. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.38th Conference on Neural Information Processing Systems (NeurIPS 2024).https://pytorch.org/vision/stable/transforms.html 


Decision: The paper addresses an interesting problem of learning global and dense visual features via vision-language alignment (VLA) and pixel alignment (PA) strategies. The proposed method demonstrates its superior performance on various medical tasks, i.e., image segmentation, object detection, visual grounding, and image classification (both zero-shot and fine-tuned). The reviewers all agree that the paper is well-written and the proposed pseudo-segmentation pretext task of exploiting the aggregated attention maps from the vision encoder is novel and valuable.  In the initial review comments, the reviewers raised concerns about the quality of pseudo masks, how they will affect the training, the feasibility of being extended to perform in 3D, and some details of other techniques. The authors provided a detailed response and supportive experimental results, and most of the concerns were addressed, resulting in favorable final ratings from the reviewers. Overall, the paper introduced a novel pre-training scheme, promoting dense semantic information in medical images, and demonstrated the superior performance gain of the proposed method, which AC believes could benefit and inspire the related field and thus worth accepting to NeurIPS. The AC encourages the authors to further address the review comments and strengthen the paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a new medical vision-language model, G2D, which employs vision-language alignment (VLA) and pixel alignment (PA) strategies, combined with a pseudo segmentation (PS) pre-training task, to learn global and dense visual representations from medical images. The VLA strategy is used to learn global representations of images and texts, while the PS task constructs pseudo masks through a parameter-free mechanism to facilitate the learning of dense representations. The method is comprehensively validated across five downstream tasks (image segmentation, object detection, zero-shot image visual grounding, zero-shot image classification, and fine-tuned image classification), demonstrating its effectiveness in handling both unimodal and cross-modal tasks.
strength_weakness: strength_weakness: Major concerns:

- The attention maps could introduce errors in pseudo mask, and these errors may propagate throughout the training process. To address this, a clear validation strategy needs to be outlined. For instance, in Figure 2, aggregated attention map might incorrectly highlight irrelevant regions. It is essential to establish methods for **detecting** and **measuring** these errors to ensure the reliability of the model. I hope the authors could quantify the errors in aggregated attention map and pseudo mask during the rebuttal period.

Minor concerns:

- The training and validation of the model rely on specific datasets, which may introduce biases and potentially affect the model's generalizability to different datasets.

- It is uncertain whether the method can be effectively extended to vision-language tasks involving 3D imaging (e.g., CT and MRI), presenting a limitation in its current scope of application.
questions: questions: - How do you detect and correct the errors made by aggregated attention map?
limitations: limitations: Limitations were discussed in Section A.1
paper_summary: paper_summary: The paper proposes an encoder-decoder medical VLP approach for global-to-dense visual representation learning. Pseudo segmentation is adopted for dense level learning. Rich experiments validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The comparisons with MGCA and MRM in the CXR14 dataset are not included in Table 3, but Table 4 includes the comparisons with MGCA and MRM. What are the reasons behind this?
2. Transformer-based vision encoder is not analyzed.
3. The balance between VLA and PA losses is not analyzed.
questions: questions: Is it not applicable to compare with MGCA and MRM in the CXR14 dataset?
limitations: limitations: N/A
paper_summary: paper_summary: This manuscript describes a medical vision-language pre-training framework called Global to Dense level representation learning (G2D), that learns global and dense visual features simultaneously with only image-text pairs, by exploiting the aggregated attention map from the vision encoder for a pseudo segmentation pretext task. The improved (frozen) vision encoder is then utilized as part of the model pipeline for a number of downstream tasks (e.g. segmentation, classification)
strength_weakness: strength_weakness: - Unclear if specific sentence/phrase to individual image region alignment is achieved, for dense learning
 - Lack of fine-grained pixel-level evaluation of masks
questions: questions: 1. The accuracy of the initial aggregated attention map appears possibly non-optimal, given that additional thresholding by body mask is required. As such, it might be considered to quantify the accuracy of these maps, possibly against segmentation ground truth.

2. In Section 3.2, it is stated that a threshold is applied (at 85%) to transform the aggregated attention map into a binary mask, before smoothing. It might be clarified if the need for smoothing (and related smoothing parameters) was empirically determined.

3. In Section 3.3, it is stated that "This decoder takes visual feature V_i as input and utilises the pseudo mask ˜M_i as the supervisory signal for the pretext task". It might be clarified as to whether and how specific text can be matched to specific (separate) image regions, as in Figure 4 of Section A.7. In other words, while Figure 4 shows specific text descriptions corresponding to specific image regions, were these correspondences/alignments indicated by the proposed G2D model, or are they external manual observations? A.1 suggests no, but this might be explicitly stated.

4. In Section 4, the choice of ResNet-50 as the encoder over other plausible choices (e.g. U-Net encoder) might be briefly explained.

5. For Table 1, it might be clarified as to what "encoder-decoder" refers to - the updating of both encoder and decoder?
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes G2D, a novel vision-language pre-training (VLP) framework for medical imaging that aims to learn both global and dense visual representations from radiography images and their associated radiology reports. The key innovation is a pretext task called Pseudo Segmentation (PS), which uses a pseudo mask derived from attention maps to guide the learning of dense visual features during pre-training. The authors demonstrate that G2D outperforms existing medical VLP approaches on various downstream tasks including classification, segmentation, object detection, and zero-shot visual grounding across multiple medical imaging datasets. Notably, G2D shows strong performance on segmentation tasks even when fine-tuned on very limited data.
strength_weakness: strength_weakness: Limited theoretical analysis: While the method is empirically strong, there is little theoretical justification for why the pseudo segmentation task leads to improved dense representations.

Complexity of the approach: The method involves several components and processing steps, which may make it challenging to implement and potentially limit its adoption.

Computational resources: The pre-training process appears to be computationally intensive (16 A100 GPUs), which could be a barrier for researchers with limited resources.

Generalization to other domains: While the focus on medical imaging is valuable, it's unclear how well this approach would generalize to other vision-language domains.

Comparison to more recent baselines: Some of the baselines used for comparison (e.g., ConVIRT, GLoRIA) are somewhat older.

Comparison to more recent medical VLP methods would strengthen the evaluation.
questions: questions: Major concerns:
My primary concern revolves around the authors' claim that current medical VLP methods primarily align images with entire text reports. This assertion appears to be inconsistent with the facts, as evidenced by several papers that have employed local alignment between image regions and text. This factual contradiction significantly undermines the novelty of the present work. For instance:

GLoRIA (Huang et al., ICCV 2021): "Global-Local Representation Alignment for Improved Visual Recognition in Medical Imaging"
This paper introduced a global-local alignment approach, learning finer-grained representations by aligning image patches with text tokens.
MGCA (Wang et al., arXiv 2022): "Multi-Granularity Cross-Modal Alignment for Generalized Medical Visual Representation Learning"
This method employed a multi-granularity alignment strategy, including global, local, and fine-grained levels of alignment.
BioViL (Boecking et al., ECCV 2022): "Making the Most of Text Semantics to Improve Biomedical Vision–Language Processing"
This work proposed a method to improve biomedical vision-language processing by leveraging text semantics, which includes local alignment strategies.
MedKLIP (Wu et al., medRxiv 2023): "Medical Knowledge Enhanced Language-Image Pre-training"
This approach utilized external knowledge bases to enhance local alignment, achieving more fine-grained image-text matching.
Given these existing works, the authors' characterization of the current state of medical VLP appears inaccurate. This misrepresentation significantly weakens the claimed novelty of their approach. The authors should provide a more accurate description of existing methods and clearly articulate how their approach differs from or improves upon these established local alignment strategies.

Other minor concerns:
Have you explored the quality of the learned representations at different levels of the network? Are there significant differences in the quality of features at different scales?
How sensitive is the method to the choice of threshold used in pseudo mask construction? The ablation shows results for a few values, but is there a principled way to choose this threshold?
Have you investigated the potential of using the pseudo masks generated during pre-training for weakly supervised segmentation tasks?
How does the performance of G2D change as the amount of pre-training data is varied? Is there a clear relationship between pre-training data volume and downstream task performance?
Given the computational requirements for pre-training, have you explored any techniques for making the approach more efficient, such as progressive training or curriculum learning?
limitations: limitations: The authors provide a brief discussion of limitations in the appendix, acknowledging potential issues with the weak supervision signal from pseudo masks and the need for further research on regional visual representations. They also touch on broader impacts, mentioning both potential benefits for healthcare and risks associated with sensitive medical data. While these discussions are valuable, they could be expanded to provide more specific insights into the limitations of the current approach and potential mitigation strategies for the identified risks.

Comments: title: Thank you
The rebuttal addresses my concerns
comment: We thank the authors for their clarifications.
Thank you for your thoughtful review and for considering an upgrade in your rating based on our responses. We are grateful for your suggestion on evaluating pseudo labels without per-voxel annotations and will explore implementing this strategy to further validate our methodology. Your insights are invaluable to enhancing the quality of our work.
I appreciate the authors' detailed responses, which effectively addressed my previous concerns. As a result, I'd like to raise my rating to Weak Accept.

Regarding pseudo label evaluation, per-voxel annotations may not be necessary. Based on the report, if the disease is present in the image and the pseudo labels correctly identify it, this counts as a true positive; otherwise, it's a false negative. Similarly, if the report indicates the image is healthy, the authors could calculate the number of true negatives and false positives for the pseudo labels.

This strategy might be able to evaluate the quality of pseudo labels.
Thank you for taking the time to reassess your concerns and for maintaining your positive feedback. We truly appreciate your thoughtful insights and feedback.
Thank you for your feedback and for considering our response. We appreciate the opportunity to clarify our work and are grateful for your thoughtful review.
Thank you for addressing my concerns! I have no further questions and maintain my original rating.
I thank authors for their efforts in addressing my concerns. After reading the rebuttal, my concern has been addressed. And I will update the final rating.
Continue with the rebuttal

>Clarification on other medical VLP methods’ limitations in alignment
- We have described GLoRIA, MGCA, MedKLIP, and KAD in Section 2 of the main article and will provide a detailed explanation and comparison with G2D in the following sections.
- **Ambiguous Token-level Alignment in GLoRIA and MGCA:** Both GLoRIA and MGCA employ a brute-force approach to align image and text tokens. This token-level alignment might compromise the medical context and lead to misalignments. For example, medical terms such as 'compatible' or 'acute' lack direct visual correlates, making local alignment ambiguous.
- **Global Alignment only in BioViL**: BioViL implements only global alignment, as detailed in their original work (equation 2 and section 2.2). During pre-training, their loss functions include global image-text alignment and masked language modeling on the text side, but they do not incorporate a loss for dense visual representation learning.
- **Loss Functions in MedKLIP**: During pre-training, MedKLIP utilizes an entity classification loss, applying it to all image features for classifying entities, and a contrastive loss where different positional names are treated as negative samples using their name embeddings for contrastive learning. However, these loss functions are not explicitly designed for fine-grained image-text matching.
- **Unique Approach of G2D**: Unlike GloRIA, MGCA, BioViL, MedKLIP, and KAD, which only use an image encoder during pretraining, G2D employs an encoder-decoder architecture to enhance visual representation learning. During VLP, the encoder extracts global visual features aligned with text to learn global visual representations. Additionally, G2D incorporates a decoder that performs pseudo-segmentation tasks using pseudo masks generated by the G2D encoder, independent of external annotations. This decoder leverages features from the image encoder for pseudo-segmentation, enabling both the encoder and decoder to jointly learn dense visual representations.

>Quality of different level visual features
- Most network structures for dense prediction tasks use an encoder-decoder architecture, where the features at the **penultimate layer of the decoder** are used for final pixel-wise prediction. For image classification, the structure usually involves an encoder with a linear classifier, where the features of the **penultimate layer of the encoder** are used for prediction. For both **representative cases**, the effectiveness of the penultimate layer of the encoder or decoder is demonstrated in Section 4. These experiments have shown the effectiveness of our approach in the most general settings (i.e., both classification and dense prediction).

>Sensitivity analysis on threshold used in pseudo maks construction
- We conducted a detailed ablation study on various threshold values when building the pseudo masks. The results are shown below. As the table indicates, the best downstream performance is achieved with an 85th percentile threshold. Increasing the threshold to the 95th percentile does not improve performance, suggesting that an extremely high threshold may lead to over-filtering. Conversely, decreasing the threshold from the 85th to the 25th percentile consistently degrades performance, as a lower threshold causes the pseudo mask to cover most of the image, introducing noise during VLP. Based on these experimental results, we empirically set our threshold to the 85th percentile.
- In the future, we will investigate using an adaptive threshold to filter the attention map.

| Threshold | Classification(AUC) | Segmentation(Dice) | Detection(mAP) |
|-------|----------|----------|----------|
| | CXR14(1%) | SIIM(1%) | ObjectCXR(1%) |
| 95% percentile   |  78.5 | 64.8  |   3.7 |
| 85% percentile (default) |  79.1 | 65.6  |   3.8 |
| 75% percentile |   78.3 | 63.0  |   3.4 |
| 50% percentile (median) |   75.6 | 58.8  |   2.3 |
| 25% percentile |  75.2 | 65.6  |   2.1 |