Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models

Authors: kanchan  poudel
manish  dhakal
prasiddha  bhandari
rabin  adhikari
safal  thapaliya
m  poudel
p  dhakal
r  bhandari
s  adhikari
b  thapaliya
  khanal
dhakal  poudel
adhikari thapaliya bhandari
Venue: MIDL_io_2024_Conference
Published: 2024-01-30 11:08:55

Abstract: Medical image segmentation allows quantifying target structure size and shape, aiding in disease diagnosis, prognosis, surgery planning, and comprehension. Building upon recent advancements in foundation Vision-Language Models (VLMs) from natural image-text pairs, several studies have proposed adapting them to Vision-Language Segmentation Models (VLSMs) that allow using language text as an additional input to segmentation models. Introducing auxiliary information via text with human-in-the-loop prompting during inference opens up unique opportunities, such as open vocabulary segmentation and potentially more robust segmentation models against out-of-distribution data.
Although transfer learning from natural to medical images has been explored for image-only segmentation models, the joint representation of vision-language in segmentation problems remains underexplored. This study introduces the first systematic study on transferring VLSMs to 2D medical images, using carefully curated $11$ datasets encompassing diverse modalities and insightful language prompts and experiments. Our findings demonstrate that although VLSMs show competitive performance compared to image-only models
for segmentation after finetuning in limited medical image datasets, not all VLSMs utilize the additional information from language prompts, with image features playing a dominant role. While VLSMs exhibit enhanced performance in handling pooled datasets with diverse
modalities and show potential robustness to domain shifts compared to conventional segmentation models, our results suggest that novel approaches are required to enable VLSMs to leverage the various auxiliary information available through language prompts. The code and datasets are available at https://github.com/naamiinepal/medvlsm.

Main paper: 
 introduction: 
 Medical image segmentation is crucial for various clinical applications such as diagnosis, prognosis, and surgery planning. The latest supervised segmentation models exhibit promising outcomes across diverse imaging modalities, anatomies, and diseases (Milletari et al., 2016; Havaei et al., 2017; Zhou et al., 2018; Chen et al., 2021; Isensee et al., 2021; Hatamizadeh et al., 2022; Oktay et al., 2022; Wazir and Fraz, 2022) . Despite their success, these models are constrained to predefined foreground classes on specific modalities and 
clip-and biomedclip-based medical vlsms: 
 We create four medical VLSMs using CLIP and BiomedCLIP: (i) Finetuning CLIP-based VLSMs, CLIPSeg (Lüddecke and Ecker, 2022) and CRIS 1 (Wang et al., 2022) , pretrained on natural image-text pairs, and (ii) Building two new VLSMs for the medical domain by adding a decoder to BiomedCLIP, pretrained on medical image-text pairs. The proposed new models are BiomedCLIPSeg-D (with a pretrained CLIPSeg decoder) and Biomed-CLIPSeg (with a randomly initialized decoder of CLIPSeg). A sample from the datasets in our experiments is a triplet of a medical image, a segmentation mask, and a text prompt. Figure 1 displays the overall VLSM architecture.CLIPSeg accommodates both CNN and ViT (Dosovitskiy et al., 2020) backbones, whereas CRIS only supports a CNN-based CLIP backbone. BiomedCLIPSeg-based models include transformer-based backbones for both the encoders. We study CLIPSeg and CRIS in both zero-shot and finetuning while only finetuning for BiomedCLIPSeg-based models as they lack an end-to-end pretrained encoder-decoder.We collected 11 2D medical imaging datasets of diverse modalities, organs, and pathologies covering both radiology and non-radiology images for binary and multi-class segmentation tasks (see Table 1 ). All the datasets are used for finetuning separately or combined (as a single pooled dataset) except the last three endoscopy datasets (ETIS, ColonDB, and CVC300), which are used only as the test split to study domain shift robustness. Although language prompts enable injecting rich information into VLSMs, manually crafting individual image-specific prompts becomes impractical for large-scale evaluations. Thus, we implement an automated prompt generation system for extensive assessments of medical VLSMs. This involves incorporating semantic concepts such as size, position, color, and specific medical attributes like gender, age, and pathology.In addition to automated prompts, we introduce manual prompts that provide general class-level information applicable to all samples within a given dataset. The generated language prompts encapsulate a comprehensive set of attributes and information, comprising: (i) Inspired by Tomar et al. (2022) , number, size, and relative location are derived through image processing on segmentation masks. (ii) Motivated by Qin et al. (2022) , we use shape and color information from VQA queries. (iii) General class information, extracted for photographic images from online medical journals, provides overarching details applicable across different datasets. Notably, Qin et al. (2022) used PubMedBERT (Gu et al., 2021) for this purpose; however, our experiments revealed its unreliability, leading us to manually gather this information from online medical journals (see Table 5 ). (iv) Attributes like age, gender of patients, image quality, cardiac cycle, and tumor type are extracted whenever available, contributing valuable context to the language prompts. There are 14 such attributes, (a1 to a14), which we combined in various ways to build nine distinct prompt types (P1 to P9) for each dataset (Table 9 ; Appendix G). Each prompt type caters to specific attribute combinations, prioritizing the class name as the foundational attribute and enhancing the versatility of the generated prompts.We finetuned VLSMs with minimal hyperparameter changes from the original pretraining settings. AdamW (Loshchilov and Hutter, 2017) optimizer with weight decay of 10 -3 , and initial learning rates of 2 × 10 -3 (CLIPSeg) and 2 × 10 -5 (CRIS) were utilized. Dice loss was used alongside Binary Cross Entropy loss scaled by 0.2. The learning rate was reduced by 10 times if validation loss did not decrease for 5 consecutive epochs. Batch sizes of 128 and 32 were used for CLIPSeg and CRIS, respectively, due to the difference in model sizes2 . 
results: 
 VLSMs adapt better to non-radiology images in Zero-Shot Setting (ZSS). Both CRIS and CLIPSeg barely work in ZSS for radiology images except for CRIS in the BUSI dataset but get a Dice score in the range of 20% -70% for non-radiology datasets, with 67.98% being the highest Dice score for ISIC (Figure 2 ). Adding more attributes to the prompt generally improved performance, but the gain is inconsistent across prompts and datasets.Image-specific-attributes or general descriptions? In the ZSS, CRIS performs better on endoscopy datasets when prompts contain image-specific attributes (size, number, and location; P4, P5, and P6; Figure 2 ), but degrades with non-image-specific attributes added (P7, P8, P9). Interestingly, prompts with general descriptions (P8 and P9) achieve the highest performance on the DFU 2022 dataset, possibly due to pretrained models' familiarity with feet and skin compared to the colon. This highlights the complex relationship between pretraining data, VLSM architecture, and the medical segmentation task.Making prompts richer does not always help during finetuning. Figure 2 shows that the DSC variation across prompt type is minimal in the finetuned setting for all and BiomedCLIPSeg-D model on non-radiology (first two rows) and radiology datasets (last row). Finetuning using the prompts improves performance compared to the empty prompt, particularly in multi-class settings.the models. Prompt with only class name (P1) improves segmentation performance in radiology datasets for all four VLSMs. While CRIS' performance almost saturates after adding the class name and mask shape (P2), the rest of the models have similar performance for all the prompts except P0 with multi-class segmentation (CAMUS and CheXlocalize). BiomedCLIPSeg and BiomedCLIPSeg-D, despite being based on a VLM pretrained on medical data, consistently perform poorly across all prompts compared to CLIP and CLIPSeg. This is likely because it has not been further pretrained for segmentation tasks on a large-scale dataset. Subsequent experiments use better performing CLIPSeg and CRIS to study the impact of individual attributes and robustness of VLSMs3 .When finetuned, CRIS captures some language semantics better than CLIPSeg. We replaced attribute values of the input prompts during inference with random uncommon English words and semantically wrong or opposite values to assess whether VLSMs leverage the language semantics. Figure 3 shows that altering attributes minimally impacts CLIPSeg's performance but notably deteriorates CRIS's. To further investigate CLIPSeg's indifference to attribute values, we provided only the class name(P1) as input during in- ference to the model trained on rich prompts P6; the results were very similar to providing the rich prompts, reinforcing the minimal impact of attributes in CLIPSeg. CRIS's performance decreases notably for attributes like size and location. The decline is more significant when providing semantically opposite values than random uncommon English words, indicating robust semantic learning. A qualitative examination of predicted segmentation masks confirms this trend (Figure 4 ). Finetuned VLSMs comparable to SOTA segmentation models. Table 2 compares VLSMs vs. traditional CNN-based models (Ronneberger et al., 2015; Chen et al., 2018; Zhou et al., 2018) on their ability to learn in two scenarios: when trained on (i) individual specialized datasets or (ii) a pooled dataset that combines diverse datasets into a single Tested Dataset → K v a si r -S E G C li n ic D B B K A I C V C -3 0 0 C V C -C o lo n D B E T IS IS IC D F U C A M U S B U S I C h e Xtraining set. While the segmentation models (CNNs and VLSMs) achieve better on pooled endoscopy datasets than individual endoscopy datasets, performance mainly drops when training on a pooled set comprising all the datasets. VLSMs outperform image-only offthe-shelf CNN-based methods in most cases. We have also compared with the best method reported in the literature for each dataset. 4 The state-of-the-art results 5 are better, although VLSMs seem to have competitive performance.VLSMs adapt better to distribution shifts. To assess the ability of the segmentation models to transfer knowledge learned from one dataset to another similar one, we train the models on each large endoscopy dataset (Kvasir-SEG, ClinicDB, and BKAI) and evaluate them on all endoscopy datasets. Table 3 , shows that VLSMs perform better in all the cases than the conventional models for endoscopic datasets. VLSMs show smaller performance drops than conventional models when trained on a different distribution from the test set.• Left ventricular cavity in two-chamber view of the heart at the end of the diastole cycle.• Left ventricular cavity in two-chamber view in the cardiac ultrasound at the end of the diastole cycle. 
discussion, limitations, and conclusion: 
 VLSMs pretrained on natural images show suboptimal zero-shot accuracy with medical images for practical use but provide a foundation for joint text-image representation. Our study provides intriguing insights into prompt design, attributes' roles, and models' performance when finetuning across diverse datasets. The zero-shot segmentation performance showed improvement across all non-radiology datasets when compared to the radiology datasets. This could be attributed to the non-radiology medical imaging modalities being closer to open-domain images, as well as the potential familiarity with organs such as skin and feet (for ISIC and DFU datasets) during pretraining. The best-performing prompts 4. To ensure a thorough comparison across datasets with diverse modalities and SOTA methods, we report the SOTA for each dataset from literature, apart from implementing a few commonly used CNN baselines. 5. Except for CAMUS and ISIC, may have different training, validation, and test splits due to the unavailability of the standard splits in literature. vary with datasets but often include attributes familiar to models during pretraining. For instance, CRIS trained on RefCOCO (Kazemzadeh et al., 2014) for referring image segmentation captures size, location, and number well. The ability of CRIS to leverage better language semantics than CLIPSeg might be due to (i) CRIS's architecture that focuses on token-level intervention instead of CLIPSeg's sentence-level embedding, and (ii) end-to-end VLSM training of CRIS compared to CLIPSeg's training for segmentation task with frozen CLIP encoder. Interestingly, models based on CLIP performing better than those based on BiomedCLIP (pretrained with image-text pairs of 400 million natural domain versus 15 million medical domain) shows that largescale dataset has the benefit that is hard to achieve with smaller-scale domain-specific data.Our study aims to build insights into how well VLSMs leverage textual information and perform transfer learning in the medical domain. It proposes pragmatic prompt settings and systematic experiments instead of implementing an exhaustive list of VLSMs and only grossly comparing their performance. The four CLIP-based VLSMs cover significant variations in architecture to capture global vs. token level information in prompts, training approach with end-to-end for referring image segmentation vs. finetuning only decoder for segmentation, and based on VLM pretrained on natural vs. medical domain, etc. We focus only on 2D medical images, excluding 3D modalities like MRI or CT scans, as most existing VLSMs are suitable only for 2D images, requiring further research in building 3D VLSMs.While the VLSMs' performance seems on par with image-only architectures, and some of the VLSMs use information injected via text prompts, our results show that further research is needed to develop novel approaches that can better leverage the rich information provided via prompts. Moreover, interesting future directions can explore how these prompts could help build more robust and explainable models against out-of-distribution data. Our work serves as an essential first step in this direction, offering a valuable evaluation framework, datasets enriched with prompts, and fascinating insights for future investigation. The predictions are ordered in the descending order of the probability generated by the model. The model has high uncertainty as the maximum probability is about 0.1. The predictions are almost the same and uninformative, which is more prominent in the radiology datasets.• Left ventricular cavity in two-chamber view of the heart at the end of the diastole cycle of a female.• Left ventricular cavity in two-chamber view in the cardiac ultrasound at the end of the diastole cycle of a female. 
include the age: 
 • Left ventricular cavity in two-chamber view of the heart at the end of the diastole cycle of a forty-six-year-old female.• Left ventricular cavity in two-chamber view in the cardiac ultrasound at the end of the diastole cycle of a forty-six-year-old female. 
include the image quality: 
 • Left ventricular cavity in two-chamber view of the heart at the end of the diastole cycle of a 40-year-old female with poor image quality.• Left ventricular cavity in two-chamber view in the cardiac ultrasound at the end of the diastole cycle of a 40-year-old female with poor image quality. 
include the mask shape: 
 • Left ventricular cavity of triangular shape in two-chamber view of the heart at the end of the diastole cycle of a 40-year-old female with poor image quality.• Left ventricular cavity of triangular shape in two-chamber view in the cardiac ultrasound at the end of the diastole cycle of a 40-year-old female with poor image quality. 
 
 Appendix: All Endoscopy*The location of the polyp is [MASK] .[variable, unknown, varied, unpredictable, uncertain] Polyp is located at [MASK] .[bifurcation, apex, rectum, midline, right] The shape of polyp is [MASK] .[irregular, variable, oval, round, different] Polyp is [MASK] in shape.[oval, irregular, round, spherical, cylindrical] The color of the polyp is [MASK] .[yellow, red, blue, brown, pink] Polyp is [MASK] in color.[yellow, white, red, black, green]The location of skin melanoma is [MASK] .[unknown, variable, unusual, unpredictable, rare] The color of skin melanoma is [MASK] .[red, yellow, brown, black, blue] Skin melanoma is [MASK] in texture.[heterogeneous, variable, soft, irregular, fibrous] Skin cancer is located at [MASK] .[extremities, birth, puberty, adolescence, skin] Skin cancer is [MASK] in texture.[heterogeneous, unique, variable, diverse, distinctive]The location of a diabetic foot ulcer is at [MASK] . [first, rest, ankle, home, foot] Diabetic foot ulcer is located at [MASK] .[ankle, heel, foot, extremities, feet] The location of the foot ulcer is [MASK] .[ankle, knee, first, heel, night] Foot ulcer is located at [MASK] .[ankle, heel, foot, knee, night]The left ventricular cavity is [MASK] in shape.[spherical, triangular, normal, oval, round] The myocardium is [MASK] in shape.[spherical, cylindrical, circular, round, triangular] The left atrium cavity is [MASK] in shape.[oval, round, triangular, spherical, irregular] The left ventricular cavity is located at [MASK] .[diastole, apex, rest, 90 • , 45 • ] The myocardium is located at [MASK] .[rest, apex, risk, diastole, birth] The left atrium cavity is located at [MASK] .[diastole, right, left, 90 [irregular, oval, round, triangular, globular] Enlarged Cardiomediastinum is [MASK] in shape. [oval, triangular, irregular, round, rounded] Cardiomegaly is [MASK] in shape.[irregular, triangular, normal, oval, round] Lung Opacity is [MASK] in shape.[irregular, round, oval, nodular, reticular] Consolidation is [MASK] in shape.[spherical, circular, triangular, irregular, round] Atelectasis is [MASK] in shape.[irregular, oval, triangular, spherical, round] Pleural Effusion is [MASK] in shape.[irregular, round, oval, spherical, solid] *This includes six datasets of endoscopy: Kvasir-SEG, ClinicDB, BKAI, CVC-300, CVC-ColonDB, ETIS Appendix F. ResultsTables 6 and 7 show the results of VLSMs with finetuned the decoder while keeping the encoders frozen.To examine the usage of free-text radiology reports of chest x-rays for segmentation, we utilize 1,141 frontal-view CXRs randomly selected from the MIMIC-CXR database (Johnson et al., 2019a,b; Chen et al., 2022) . This dataset contains the segmentation of lungs, which has been verified manually. We use the free-text radiology reports provided in the MIMIC-CXR Database (Johnson et al., 2019a) as the only prompt (P1), and the results are reported in Table 8 . 7. P6: "number size color shape class name, located in the location of the image"• one medium pink round polyp, located in the top left of the image 8. P7: "class name, which is a general description of the class"• polyp, which is a small lump in the lining of colon 9. P8: "number size color shape class name, which is a general description of the class"• one medium pink round polyp, which is a small lump in the lining of colon 10. P9: "number size color shape class name, which is a general description of the class located in the location of the image "• one medium pink round polyp, which is a small lump in the lining of colon located in the top left of the image For General Description of the class, prompts were built using information about the subject on the internet. Five such descriptions were designed for each dataset, and one random sample was selected each time as the general description of the class attribute whenever the prompts p7, p8, and p9 were used.The templates of prompts for the DFU-2022 (Kendrick et al., 2022) and ISIC (Gutman et al., 2016) datasets used were the same as the above examples for endoscopy images, with class name and general description of the class being different. We used class names skin melanoma and foot ulcer for the two datasets, respectively.The five General Description of the class for each of the three types of photographic datasets used is listed in the table below. The prompts for the Breast Ultrasound Images (BUSI) (Al-Dhabyani et al., 2020) dataset are listed below. • Two medium regular-shaped tumors at the center, left in the breast ultrasound image 6. Tumor Shape• Two medium square-shaped benign tumors at the center, left in the breast ultrasound image• Two medium square-shaped regular tumors at the center, left in the breast ultrasound imageFurther details are in Appendix C.Additionally, we have also trained both the models, keeping their encoders frozen whose results are shown in Appendix F.1.https://github.com/naamiinepal/medvlsm 


Decision: The reviewers agreed unanimously to accept this paper. I align with their decision. This paper provides a comprehensive evaluation study, which can offer valuable insights for peer researchers to continue exploring Vision-Language Models in medical imaging.

Reviews: review: paper_summary: paper_summary: This paper investigates the extension of vision-language models (VLMs) to medical image segmentation (VLSM = vision-language segmentation model). The idea is to include text-based prompts along with the original input image allowing for a human-in-the-loop segmentation. 4 VLSMs derived from CLIP and BiomedCLIP are utilized against 11 datasets with a total of 10 possible prompts (with prompt 0 denoting the absence of a prompt). 3 of the 11 datasets are reserved to investigate the impact of domain shift on the model's behavior. Examples of prompts include prompts indicating the location of the foreground object such as "bottom right" or the size of the background object such as "small" as well as combinations of such concepts. Experiments are performed for both zero-shot and fine-tuning strategies.
main_review: main_review: I am not completely convinced by the reasons given by the authors and am still curious as to how this will fare for standard radiological datasets such as MRI and CT, but overall I am satisfied with the paper.
strength_weakness: strength_weakness: => Most of the datasets 8/11 were non-radiological, and MRI and CT were completely excluded even though a 2D slice-by-slice approach allows for their (potential) inclusion
=> The impact of domain shift is only investigated for non-radiological datasets
questions: questions: Why were most datasets non-radiological?
paper_summary: paper_summary: In this paper, the authors present a study exploring the performance, benefits and shortcomings of using vision-language segmentation models for medical-image segmentation. The authors curate a benchmark comprising 11 diverse medical image segmentation datasets, including different anatomies and image types. The authors then explore the performance of 2 different modality fusion techniques (aggregator vs language conditioning) on 2 different base models: CLIP and BiomedCLIP. Since the datasets are image-only, the authors explore several ways to add language prompts and explore the effect of different strategies on model performance. They find that in the medical domain as well, language conditioning in decoder (CRIS) performs better than using modality aggregation and conditioning (CLIPSeg). The authors show that the VLSM models are more robust to domain shifts than image-only counterparts but require more research to fully utilize the power of language conditioning.
main_review: main_review: The authors addressed some of the concerns outlines in the review but hasn't provided a satisfactory argument for their experiment design and hence I recommend a borderline accept for the paper, but still believe the paper could benefit from a more robust experiment design. rather than a bag of many things.
strength_weakness: strength_weakness: * The use of CLIP models for the study is suboptimal given there are many powerful open source VLM models available in the open source community. BiomedCLIP also has the disadvantage since no decoder is available and hence the authors train it from scratch which is not ideal with such a small dataset.
* Since prompt design is a core contribution of the paper, it would have been beneficial to include more details about it in the main paper rather than the appendix.
questions: questions: 1. What hyperparameters were considered when training these models? Since the comparison with several baselines is presented, it begs the question if the current performance of the VSLM models are due to suboptimal hyperparameters.
2. What is the motivation behind using CLIP as the base model for the paper when compared with other VLM models available in the open source community?
3. How was the decoder in BiomedCLIP variants trained? What was the data used? Did the model converge?
paper_summary: paper_summary: This work is a study on transferring Vision-Language Segmentation Models (VLSMs) to 2D medical images, leveraging advancements in foundation Vision-Language Models (VLMs) used in natural image-text pair analysis. The study evaluates VLSMs, adapted from the popular CLIP and BiomedCLIP models, across 11 carefully selected medical image datasets spanning diverse modalities. The study also highlights VLSMs' enhanced performance and potential robustness against domain shifts when handling pooled datasets from different modalities. The paper details the implementation of automated and manual prompt generation, the finetuning process, and an extensive evaluation of VLSMs' robustness, performance, and adaptability to domain shifts. It concludes with a discussion on the limitations and future directions for leveraging VLSMs in medical image segmentation, emphasizing the need for further research to optimize the integration of textual information for improved segmentation outcomes.
main_review: main_review: I am increasing my initial rating to a strong accept. The authors have provided convincing arguments that helped me make this decision. Several of my concerns have been satisfactorily addressed by the authors.
strength_weakness: strength_weakness: The paper does not compare VLSMs against the latest state-of-the-art image-only segmentation models or other emerging multimodal approaches, potentially limiting the context for evaluating VLSMs' performance.

The study leaves open questions about the optimal ways to integrate textual information into segmentation models, suggesting that further research is needed to develop methodologies that can fully exploit this aspect.

The study exclusively focuses on 2D medical images, excluding 3D modalities such as MRI or CT scans. This limitation restricts the applicability of the findings across the full spectrum of medical imaging technologies. The authors have listed the same as a limitation of the work.
questions: questions: Insights into modality-specific challenges and opportunities could guide future research towards modality-adaptive approaches. Discussion on how different imaging modalities affect the performance of VLSMs would be a valuable information for the reader. 

I think it is important to briefly discuss any ethical considerations related to the use of VLSMs in medical image analysis, including privacy concerns with patient data.

Comments: title: Changes made to the revised paper
The discussion period begins
General response to everyone
Official Comment by Authors (2/2)
Official Comment by Authors (1/2)
Usage of CLIP-only models, and BiomedCLIP training
comment: We sincerely thank all the reviewers, as their comments and feedback allowed us to improve the paper. We have clarified and commented on most of the issues raised by the reviewers. Moreover, we have made minor changes to improve the paper's clarity as suggested by the reviewers, summarized below (also highlighted in red in the revised paper):
- We have updated the result section to clarify the comparison against state-of-the-art image-only segmentation models to address Reviewer t1ep's concerns.
- We have also updated the discussion to include modality-specific insights as suggested by Reviewer t1ep.
- We have updated Appendix C of the supplementary section with details on the hyperparameter's search to address Reviewer doLo's concerns about the choice of hyperparameters.
- We have updated the dice scores with percentage signs when not in the 0-1 range, as suggested by Reviewer PqP8.
BiomedCLIP indeed utilizes different fine-tuning settings compared to CLIPSeg and CRIS. However, BiomedCLIP was interesting to us, as it uses CLIP architecture but is pretrained using 15 million biomedical image-text pairs, which, although smaller than CLIP, is still a significant number. Therefore, we wanted to explore the following questions with BiomedCLIP, whose answers were uncertain before our experiments. 
- How does an encoder trained on large in-domain (medical images) perform compared to an encoder-decoder trained on large out-of-domain images?
- How much do the representations from an encoder pre-trained in the same domain help the decoder adapt from scratch, even with a small dataset?
I would like to thank the authors for addressing the questions in the review.

I am curios as to why BiomedCLIP is included in the paper when
1. It does not have a pretrained decoder and hence does not fit in the transfer learning scenario.
2. Given the finetuning dataset is so small, no conclusions can be drawn for the model.
Dear reviewers and authors,

Thank you for your contribution to MIDL24. The discussion period begins! I encourage all reviewers and authors to participate in the discussion to address questions and clarify uncertainties.

Thank you!
We thank all the reviewers for their valuable and thorough feedback.
It is encouraging that all three reviewers found this study valuable, providing essential insights through a comprehensive set of experiments covering many 2D medical image datasets of diverse modalities, organs, and pathologies.
We respond to the various points and reviewers' feedback in separate comments.
> **I think it is important to briefly discuss any ethical considerations related to using VLSMs in medical image analysis,
including privacy concerns with patient data.**

All the datasets we used were personally unidentifiable, made openly available
for research by original dataset creators, and commonly used by researchers.
The metadata available with the images does not disclose the identity of the
participant whose data was acquired. Since we exclusively use the images and
metadata provided with the datasets, VLSMs do not introduce any new ethical
challenges beyond those already existing in image-only medical image segmentation about privacy and anonymity.

Moreover, as mentioned in *Appendix A: Impact on Society*, using VLSMs in
medical image segmentation contributes to fostering explainability and interoperability, particularly with humans in the loop, as language is more interpretable
than just images. This approach helps minimize ethical challenges associated
with black-box deep learning methods in medical applications.

> **The motivation for segmentation in medical imaging is discussed in the paper; however, the discussion on the benefits
of using VLSM over image-only segmentation models has not
been included.**

We want to emphasize the potential benefits of utilizing VLSMs over image-only
segmentation models, which motivated our research. As stated in the Introduction section, *VLSMs have the potential to allow for auxiliary information
incorporation through language prompts during segmentation. This approach
can enhance interpretability and robustness against domain shift and out-of-
distribution data.* Moreover, we conducted experiments to investigate whether
VLSMs outperform image-only models in adapting to distribution shifts. The
results of the experiments have been presented in the Results under the section
as VLSMs adapt better to distribution shifts. Our findings in the section state
that *VLSMs perform better in all the cases than the conventional models for
endoscopic datasets. The performance drops of VLSMs are smaller than that of
the conventional models when trained in a different distribution than that of the
test set.*
Thank you for your review and valuable feedback on our work.

> **The paper does not compare VLSMs against the latest state-
of-the-art image-only segmentation models or other emerging
multimodal approaches, potentially limiting the context for
evaluating VLSMs’ performance.**

We compare the VLSMs against the latest state-of-the-art image-only segmentation models as baselines. Given the diverse range of data modalities and
variations in the state-of-the-art methods across different datasets, we have reported the best method reported in the literature for each dataset in Table 2,
row 6 to ensure comprehensive comparisons. We will add a line in the main
manuscript to simplify this. Additionally, we implement a
few of the most commonly used image-only medical segmentation models as baselines for further comparisons: UNet [7], UNet++ [8], and DeepLabv3+ [9]
in Tables 2 and 3.

> **The study leaves open questions about the optimal ways to
integrate textual information into segmentation models, suggesting that further research is needed to develop methodologies that can fully exploit this aspect.**

We agree with the reviewer that the study leaves open questions on optimal
ways to integrate textual information into segmentation models, but we beg to
disagree that this is the weakness of this study; instead, we think this is the
strength of this work, which through rigorous and comprehensive experimental
designs showed this gap in the literature and SOTA VLSMs, providing the
community one important direction to explore and evaluate new VLSMs that
gets proposed.

> **The study exclusively focuses on 2D medical images, excluding 3D modalities such as MRI or CT scans. This limitation
restricts the applicability of the findings across the full spectrum of medical imaging technologies. The authors have listed
the same as a limitation of the work.**

At the time of this study, no CLIP-like VLSMs trained natively for 3D modalities exist. While it is true that there are essential imaging modalities such as
MRI and CT scans, and studies on that will be interesting, the effort for developing models that work intrinsically for 3D modalities is outside the scope of
this work. It is important to note that our framework, which includes prompt
design and assessing how VLSM leverages language information, can still be
helpful when 3D VLSMs are proposed in the literature and need evaluation.
While we acknowledge that one could still use 2D slices of 3D images and test
our method, we believe the diverse set of intrinsically 2D images, including
ultrasound, X-rays, and photographic medical images, is already broad enough
while 2D-specific to assess primarily 2D VLSMs. 3D modalities bring additional
nuances when working as slices, such as how the 2D segmentation translates to
3D when looking at each slice independently and how to provide prompts for
multiple slices. We will continue exploring 3D as a future work.

> **Insights into modality-specific challenges and opportunities
could guide future research towards modality-adaptive approaches.
Discussion on how different imaging modalities affect the performance of VLSMs would be valuable information for the
reader.**

We agree that modality-specific challenges and opportunities could be very insightful in guiding future research. There are indeed some insights based on
the results. The zero-shot segmentation performance was better for all the non-
radiology datasets we used than for radiology datasets. The closeness of the non-radiology images and the external body parts (in ISIC and DFU datasets)
with open domain datasets potentially leads to this.
We will update the discussion in our paper to make these more straightforward. Furthermore, while there are potentially more modality-specific details
that we have admittedly missed, generating such insights would necessitate further experiments, which are constrained by the scope of our work and the page
limitations imposed by the conference.

**References**

[7] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks
for biomedical image segmentation,” in Medical Image Computing and Computer-
Assisted Intervention–MICCAI 2015: 18th International Conference, Mu-
nich, Germany, October 5-9, 2015, Proceedings, Part III 18, Springer, 2015,
pp. 234–241.

[8] Z. Zhou, M. M. Rahman Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++:
A nested u-net architecture for medical image segmentation,” in Deep
Learning in Medical Image Analysis and Multimodal Learning for Clini-
cal Decision Support: 4th International Workshop, DLMIA 2018, and 8th
International Workshop, ML-CDS 2018, Held in Conjunction with MIC-
CAI 2018, Granada, Spain, September 20, 2018, Proceedings 4, Springer,
2018, pp. 3–11.

[9] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-
decoder with atrous separable convolution for semantic image segmen-
tation,” in Proceedings of the European conference on computer vision
(ECCV), 2018, pp. 801–818.
> **The use of CLIP models for the study is suboptimal, given that
many powerful open-source VLM models are available in the
open-source community.**

> **What is the motivation behind using
CLIP as the base model for the paper when compared with
other VLM models available in the open-source community?**

The scope of the study is to assess the performance of existing VLSM architectures trained on large-scale, open-domain image segmentation data with language prompts when fine-tuned for the medical domain. While many VLMs
exist in the literature, most are only suitable for downstream tasks like image
classification, image-to-text generation, and image retrieval. There are a limited
number of VLSMs trained on large-scale image segmentation data with language
prompts on top of an existing VLM and are mostly CLIP-based: CRIS [2],
CLIPSeg [3], ZegCLIP [4], DenseCLIP [5]. To channel our efforts to better
experimental designs on a wide range of datasets and prompts, we chose the
scope of our study to be around two of the representative models: CRIS [2]
and CLIPSeg [6]. The four VLSMs we created within the scope offered impressive diversity and representation—architectural variation in leveraging global
level and token level information in prompts, trained end-to-end for referring
image segmentation vs. fine-tuned only decoder with segmentation data, and
VLMs pretrained on natural vs. medical domain. As with all benchmark and
insight-generating studies like ours, there is always a tradeoff: implementing
and experimenting with the exhaustive list of methods vs. picking representative and practical approaches and conducting insightful experiments that would
be valuable to the community working on developing or using the techniques.
We chose the latter option and believe we have provided a valuable and essential first step for building robust medical VLSMs by selecting many carefully
collected datasets, designed prompts, and insightful experiments. With our
source code being public in GitHub and the set of well-defined experiments and
prompt generation, we emphasize our agreement with the reviewer’s comment
that the *The benchmark is a valued contribution to the community to test new
models/methods and easily compare results across different works.*

> **BiomedCLIP also has a disadvantage since no decoder is available, and hence, the authors train it from scratch, which is not
ideal with such a small dataset.**

> **How was the decoder trained
for BiomedCLIP variants? What was the data used? Did the
model converge?**

BiomedCLIP indeed has no trained decoder for segmentation. So, the decoders
were trained entirely during our fine-tuning (BiomedCLIPSeg) or adapted from
CLIPSeg’s pretrained model (BiomedCLIPSeg-D) and fine-tuned further. Experiments with these variants aimed to explore the tradeoff between two aspects: a. the familiarity of the BiomedCLIP encoder with the medical domain,
b. the familiarity of CLIPSeg and CRIS models with the segmentation task and
encoder-decoder interaction for the same, albeit in the natural domain. Given
that the latter approach was observed to be more advantageous, our subsequent
analysis only includes CLIPSeg and CRIS.
The architecture of CLIPSeg’s decoder has been used for decoder training in
BiomedCLIP variants. We trained this decoder in our fine-tuning experiments
using the datasets present in our study until the model converged.

**References**

[2] Z. Wang, Y. Lu, Q. Li, et al., “Cris: Clip-driven referring image segmenta-
tion,” in Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, 2022, pp. 11 686–11 695.

[3] L. H. Li, P. Zhang, H. Zhang, et al., “Grounded language-image pre-training,”
in Proceedings of the IEEE/CVF Conference on Computer Vision and Pat-
tern Recognition, 2022, pp. 10 965–10 975.

[4] Z. Zhou, Y. Lei, B. Zhang, L. Liu, and Y. Liu, “Zegclip: Towards adapting
clip for zero-shot semantic segmentation,” in Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, 2023, pp. 11 175–
11 185.

[5] Y. Rao, W. Zhao, G. Chen, et al., “Denseclip: Language-guided dense pre-
diction with context-aware prompting,” in Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, 2022, pp. 18 082–
18 091.

[6] T. L ̈uddecke and A. Ecker, “Image segmentation using text and image
prompts,” in Proceedings of the IEEE/CVF Conference on Computer Vi-
sion and Pattern Recognition, 2022, pp. 7086–7096.
Thank you for your thorough review and constructive feedback on our work.
We want to emphasize that
our contribution in this paper goes beyond extending existing work on VLM-based object detection Qin et al. [1], which had demonstrated improved object
detection performance with VLMs using automatically generated prompts. We
propose novel ways to probe VLSMs and provide an evaluation framework to
get insights into the nuanced roles of various attributes in language prompts
and their value compared to image information during fine-tuning, which were
not covered in Qin et al. [1]’s work.

> **What hyperparameters were considered when training these models? 
Since the comparison with several baselines is presented, it begs the question of whether the current performance of the VSLM models is due to suboptimal hyperparameters.**

We have reported the hyperparameters used in VLSMs in *Section 2.4: Implementation Details*.
We provide additional information and the hyperparameters used in other baselines we implemented in *Appendix C: Experiment* of the supplementary material due to page limitations on the main body of the paper set by the conference guidelines.
We conducted extensive grid searches and tuning within the limitations of our resources to get good enough hyperparameters that did not introduce vanishing and exploding gradients and performed optimally in most datasets.
The details have been updated in *Appendix C: Experiment*.   
Furthermore, we ensured that all models converged well and provided meaningful results for comparison with baselines.
Despite our efforts, we acknowledge the possibility that our hyperparameters may still be suboptimal, potentially resulting in slightly under-optimized models.
We acknowledge that there is often a tradeoff when working with resource-intensive deep-learning models.
However, we emphasize that we focused on understanding the nuances of fine-tuning VLSMs and language's role in that process.
We present transparently the setup we worked on, provide a detailed appendix, and make the code open-source, allowing the community to assess its value and further build upon this work, for instance, when developing new methods to easily benchmark and analyze the role of the language prompts.

> **Since prompt design is a core contribution of the paper, it
would have been beneficial to include more details in the main
paper rather than the appendix.**

We have included the prompt design process in *Section 2.3: Generating Lan-
guage Prompts.* We can revise the section to make it more straightforward. The
details being moved to the Appendix were due to page limitations. Although
we believe that the flow of the paper, experimental details, their results, and
the main message we wanted to deliver are best managed within the page limit
in the current version, we are open to suggestions and discussion to identify if
any existing sections in the main body can be moved to Appendix to bring back
the details on prompt design.

**References**

[1] Z. Qin, H. H. Yi, Q. Lao, and K. Li, “Medical image understanding with pre-
trained vision language models: A comprehensive study,” in The Eleventh
International Conference on Learning Representations, 2022.
Thank you for your thoughtful review and for recognizing the importance of our work.

> **Most of the datasets 8/11 were non-radiological, and MRI and CT were completely excluded even though a 2D slice-by-slice approach allows for their (potential) inclusion**

While we acknowledge that we could use a 2D slice-by-slice approach on CT and MRI datasets to test our method, the slice-by-slice segmentation still needs to be translated back to 3D volume.
The nuances regarding this, incorporating language have not been studied well and could introduce complications beyond the scope of this work.
However, we acknowledge that investigating this aspect could be a valuable research direction for future work.
Furthermore, we believe that our inclusion of a diverse set of intrinsically 2D images, with a fair balance of radiology as well as non-radiology modalities, provides a comprehensive assessment of primarily 2D VLSMs.

> **The impact of domain shift is only investigated for non-radiological datasets**

Among the datasets we utilized, the endoscopic datasets comprised multiple datasets focused on segmenting the same target, polyps, making them ideal for testing domain shift. 
However, for the radiology datasets selected, we could not find other datasets segmenting the same target organ.
From a different perspective, other datasets should be similar since we utilize radiology reports and metadata to design prompts for radiology datasets.

> **"a Dice score in the range of 20 - 70 for non-radiology datasets, with 67.98 being the highest Dice score for ISIC (Figure 2)." (Page 4)  Dice scores should either be given between 0 and 1, or a percentage sign should be added before the reported scores.**

Thank you for the observation. We have updated the Figure and content accordingly.

> **Why were most datasets non-radiological?**

Of the 8 non-radiology datasets we used, six were endoscopic datasets for polyp segmentation. 
So, in a broader context, we chose three radiology and three non-radiology datasets, maintaining a fair balance.
The selection of multiple endoscopic datasets allowed for more detailed analyses, such as cross-dataset evaluations to test robustness against distribution shifts.
Likewise, it also facilitated the creation of a diverse, pooled dataset.
Therefore, we extended the endoscopic datasets to six different datasets because it enabled the designing wider range of experiments.